2024-07-26T10:34:03.076 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T10:34:03.104 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T10:34:03.203 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818761 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/fsstress}} email: vshankar@redhat.com first_in_suite: false job_id: '7818761' kernel: client: branch: testing flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug cephsqlite: 20 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD73bZ+NJIv+n3HZsB/AeFQublE+WgLNcrrVS9CDBymkdM9xD0NmZGho3uc4f2ZT9CDFJ5YzY+1dt0SkMFAbXC0= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPuZYKMgjOE0VrPrChZqjYZc9MoB9DrDZAhNhXKhKJacNqGRNlJO1X4n2bVI9ByUeM3ue5qwbhgK9rvaatKRam0= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO6kX2vvr8FdlrEDTZMAw6I8sw30+5kqbbcV+z1TZAlbYabU+YIsZsRsEOn24ZtIR20QRn1WQtHRPto9y7CzE0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T10:34:03.203 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-26T10:34:03.204 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-26T10:34:03.204 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T10:34:03.205 INFO:teuthology.task.internal:Checking packages... 2024-07-26T10:34:03.225 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-26T10:34:03.225 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T10:34:03.225 INFO:teuthology.packaging:ref: None 2024-07-26T10:34:03.225 INFO:teuthology.packaging:tag: None 2024-07-26T10:34:03.225 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-26T10:34:03.225 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:34:03.225 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-26T10:34:03.376 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-26T10:34:03.378 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T10:34:03.384 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T10:34:03.384 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T10:34:03.475 INFO:teuthology.task.internal:Saving configuration 2024-07-26T10:34:03.496 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T10:34:03.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T10:34:03.546 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818761', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 10:28:49.802830', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD73bZ+NJIv+n3HZsB/AeFQublE+WgLNcrrVS9CDBymkdM9xD0NmZGho3uc4f2ZT9CDFJ5YzY+1dt0SkMFAbXC0='} 2024-07-26T10:34:03.569 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818761', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 10:28:49.801637', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPuZYKMgjOE0VrPrChZqjYZc9MoB9DrDZAhNhXKhKJacNqGRNlJO1X4n2bVI9ByUeM3ue5qwbhgK9rvaatKRam0='} 2024-07-26T10:34:03.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818761', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 10:28:49.800207', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO6kX2vvr8FdlrEDTZMAw6I8sw30+5kqbbcV+z1TZAlbYabU+YIsZsRsEOn24ZtIR20QRn1WQtHRPto9y7CzE0='} 2024-07-26T10:34:03.589 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T10:34:03.596 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-26T10:34:03.596 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-26T10:34:03.596 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-26T10:34:03.596 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T10:34:03.740 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd3ce903ac0>, signals=[15]) 2024-07-26T10:34:03.740 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T10:34:03.746 INFO:teuthology.task.internal:Opening connections... 2024-07-26T10:34:03.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T10:34:03.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:34:03.827 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-07-26T10:34:03.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:34:03.904 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-07-26T10:34:03.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:34:03.981 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T10:34:03.988 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-07-26T10:34:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-07-26T10:34:04.006 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-07-26T10:34:04.063 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="9" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="9" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el9" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:LOGO="fedora-logo-icon" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2024-07-26T10:34:04.064 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T10:34:04.065 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T10:34:04.065 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T10:34:04.065 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-07-26T10:34:04.085 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-07-26T10:34:04.103 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-07-26T10:34:04.103 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-07-26T10:34:04.161 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-07-26T10:34:04.161 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-07-26T10:34:04.161 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-07-26T10:34:04.161 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T10:34:04.162 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T10:34:04.163 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-07-26T10:34:04.185 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-07-26T10:34:04.204 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-07-26T10:34:04.204 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T10:34:04.262 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T10:34:04.263 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T10:34:04.263 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-07-26T10:34:04.285 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T10:34:04.296 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T10:34:04.313 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T10:34:04.313 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-07-26T10:34:04.315 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-07-26T10:34:04.318 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-07-26T10:34:04.334 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T10:34:04.364 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T10:34:04.364 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-07-26T10:34:04.372 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-07-26T10:34:04.375 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-07-26T10:34:04.478 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T10:34:04.546 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T10:34:04.567 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T10:34:04.567 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-07-26T10:34:04.567 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-07-26T10:34:04.567 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T10:34:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:34:04.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-07-26T10:34:04.775 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T10:34:04.775 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T10:34:04.775 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-07-26T10:34:04.793 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-437.el9.x86_64 2024-07-26T10:34:04.793 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T10:34:04.793 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T10:34:04.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum upgrade -y linux-firmware 2024-07-26T10:34:09.047 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.2 MB 00:02 2024-07-26T10:34:11.908 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-07-26T10:34:17.131 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-07-26T10:34:18.798 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 18 kB 00:00 2024-07-26T10:34:19.944 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-07-26T10:34:25.300 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-07-26T10:34:27.382 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-07-26T10:34:27.387 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-07-26T10:34:27.387 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 1 Package 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:34:27.388 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 383 M 2024-07-26T10:34:27.389 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-07-26T10:34:40.083 INFO:teuthology.orchestra.run.smithi144.stdout:linux-firmware-20240716-145.el9.noarch.rpm 31 MB/s | 383 MB 00:12 2024-07-26T10:34:40.083 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:34:40.084 INFO:teuthology.orchestra.run.smithi144.stdout:Total 30 MB/s | 383 MB 00:12 2024-07-26T10:34:43.163 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-07-26T10:34:43.236 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-07-26T10:34:43.236 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-07-26T10:34:46.498 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-07-26T10:34:46.498 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-07-26T10:34:55.938 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-07-26T10:34:56.283 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T10:34:56.500 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T10:34:57.077 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T10:34:57.077 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T10:34:57.818 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T10:34:57.819 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:34:57.819 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2024-07-26T10:34:57.819 INFO:teuthology.orchestra.run.smithi144.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T10:34:57.819 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:34:57.819 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-07-26T10:34:58.127 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T10:34:58.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T10:34:58.282 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T10:34:58.283 DEBUG:teuthology.orchestra.run.smithi144:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T10:34:58.285 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi144.front.sepia.ceph.com to complete... 2024-07-26T10:35:04.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26 10:35:04 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T10:35:04.563 INFO:teuthology.orchestra.run.smithi144.stderr:FINISHED --2024-07-26 10:35:04-- 2024-07-26T10:35:04.563 INFO:teuthology.orchestra.run.smithi144.stderr:Total wall clock time: 6.3s 2024-07-26T10:35:04.563 INFO:teuthology.orchestra.run.smithi144.stderr:Downloaded: 1 files, 342M in 6.0s (57.1 MB/s) 2024-07-26T10:35:04.566 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T10:35:04.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T10:35:06.634 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying... ######################################## 2024-07-26T10:35:06.775 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing... ######################################## 2024-07-26T10:35:06.825 INFO:teuthology.orchestra.run.smithi144.stdout:Updating / installing... 2024-07-26T10:35:14.817 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T10:35:54.999 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T10:35:55.000 DEBUG:teuthology.orchestra.run.smithi144:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T10:35:55.087 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T10:35:55.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T10:35:55.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T10:35:55.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T10:35:55.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T10:35:55.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T10:35:55.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T10:35:55.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T10:35:55.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T10:35:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T10:35:55.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T10:35:55.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T10:35:55.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T10:35:55.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T10:35:55.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T10:35:55.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T10:35:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T10:35:55.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T10:35:55.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T10:35:55.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T10:35:55.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T10:35:55.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T10:35:55.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T10:35:55.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T10:35:55.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T10:35:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T10:35:55.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T10:35:55.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T10:35:55.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T10:35:55.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T10:35:55.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T10:35:55.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T10:35:55.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T10:35:55.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T10:35:55.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T10:35:55.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T10:35:55.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T10:35:55.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T10:35:55.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T10:35:55.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T10:35:55.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T10:35:55.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T10:35:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T10:35:55.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T10:35:55.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T10:35:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T10:35:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T10:35:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T10:35:55.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T10:35:55.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T10:35:55.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T10:35:55.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T10:35:55.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T10:35:55.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T10:35:55.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T10:35:55.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T10:35:55.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T10:35:55.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T10:35:55.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T10:35:55.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T10:35:55.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T10:35:55.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T10:35:55.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T10:35:55.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T10:35:55.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T10:35:55.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T10:35:55.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T10:35:55.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T10:35:55.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T10:35:55.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T10:35:55.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T10:35:55.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T10:35:55.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T10:35:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T10:35:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T10:35:55.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T10:35:55.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T10:35:55.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T10:35:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T10:35:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T10:35:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T10:35:55.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T10:35:55.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T10:35:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T10:35:55.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T10:35:55.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T10:35:55.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T10:35:55.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T10:35:55.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T10:35:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T10:35:55.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T10:35:55.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T10:35:55.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T10:35:55.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T10:35:55.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T10:35:55.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T10:35:55.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T10:35:55.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T10:35:55.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T10:35:55.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T10:35:55.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T10:35:55.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T10:35:55.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T10:35:55.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T10:35:55.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T10:35:55.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T10:35:55.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T10:35:55.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T10:35:55.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T10:35:55.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T10:35:55.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T10:35:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T10:35:55.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T10:35:55.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T10:35:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T10:35:55.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T10:35:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T10:35:55.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T10:35:55.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T10:35:55.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T10:35:55.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T10:35:55.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T10:35:55.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T10:35:55.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T10:35:55.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T10:35:55.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T10:35:55.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T10:35:55.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T10:35:55.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T10:35:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T10:35:55.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T10:35:55.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T10:35:55.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T10:35:55.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T10:35:55.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T10:35:55.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T10:35:55.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T10:35:55.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T10:35:55.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T10:35:55.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T10:35:55.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T10:35:55.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T10:35:55.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T10:35:55.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T10:35:55.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T10:35:55.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T10:35:55.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T10:35:55.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T10:35:55.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T10:35:55.233 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T10:35:55.233 DEBUG:teuthology.orchestra.run.smithi144:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout:for file in vmlinuz System.map config; do 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: fi 2024-07-26T10:35:55.255 INFO:teuthology.orchestra.run.smithi144.stdout:done 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T10:35:55.256 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2024-07-26T10:35:55.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.06 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 68.el9 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Group : Unspecified 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 8264254 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2024-07-26T10:35:55.330 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : builder@centos.org 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:35:55.331 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T10:35:55.332 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T10:35:55.332 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 6.10.0-g3e9255e90710 2024-07-26T10:35:55.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T10:35:55.912 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-07-26T10:35:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T10:35:57.336 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-07-26T10:35:57.338 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-07-26T10:35:57.353 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.bAweWcb2vr 2024-07-26T10:35:57.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bAweWcb2vr 2024-07-26T10:35:57.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.bAweWcb2vr 2024-07-26T10:35:57.564 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.bAweWcb2vr is 6KB 2024-07-26T10:35:57.612 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.bAweWcb2vr 2024-07-26T10:35:57.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T10:35:57.687 INFO:teuthology.orchestra.run.smithi144.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T10:35:57.688 INFO:teuthology.orchestra.run.smithi144.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T10:35:57.688 INFO:teuthology.orchestra.run.smithi144.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T10:35:57.688 INFO:teuthology.orchestra.run.smithi144.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T10:35:57.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T10:35:57.861 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-07-26T10:36:27.865 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-26T10:36:27.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-07-26T10:36:27.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:36:46.435 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-07-26T10:36:55.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-07-26T10:36:55.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:36:58.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-07-26T10:37:10.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-07-26T10:37:10.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:37:42.398 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-07-26T10:37:42.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-07-26T10:37:42.969 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.10.0-g3e9255e90710"... 2024-07-26T10:37:42.970 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-07-26T10:37:43.008 INFO:teuthology.orchestra.run.smithi144.stdout:6.10.0-g3e9255e90710 2024-07-26T10:37:43.008 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T10:37:43.008 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T10:37:43.008 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T10:37:44.009 DEBUG:teuthology.parallel:result is None 2024-07-26T10:37:44.010 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T10:37:44.017 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T10:37:44.017 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T10:37:44.024 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T10:37:44.027 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T10:37:44.073 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T10:37:44.080 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T10:37:44.090 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T10:37:44.091 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T10:37:44.094 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T10:37:44.097 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T10:37:44.411 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T10:37:44.418 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T10:37:44.418 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T10:37:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:37:44.435 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T10:37:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:37:44.452 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T10:37:44.459 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T10:37:44.479 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T10:37:44.536 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:37:44.556 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:37:44.563 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:37:44.569 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:37:44.601 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:37:44.602 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:37:44.606 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T10:37:44.619 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T10:37:44.620 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T10:37:44.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T10:37:44.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T10:37:44.684 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T10:37:44.697 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T10:37:44.697 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T10:37:44.702 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T10:37:44.705 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T10:37:44.741 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:37:44.865 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:37:44.946 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:37:45.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:37:45.106 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:37:45.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T10:37:45.168 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:37:45.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:37:45.275 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:37:45.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:37:45.361 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:37:45.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T10:37:45.426 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:37:45.471 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:37:45.524 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:37:45.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:37:45.650 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:37:45.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T10:37:45.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-07-26T10:37:45.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-07-26T10:37:45.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-07-26T10:37:45.761 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T10:37:45.792 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T10:37:45.819 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T10:37:46.292 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T10:37:46.300 INFO:teuthology.task.internal:Starting timer... 2024-07-26T10:37:46.300 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T10:37:46.312 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T10:37:46.328 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-26T10:37:46.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2024-07-26T10:37:46.435 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2024-07-26T10:37:46.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-07-26T10:37:46.494 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-07-26T10:37:46.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-07-26T10:37:46.565 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-07-26T10:37:46.567 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T10:37:46.567 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2024-07-26T10:37:46.623 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2024-07-26T10:37:46.623 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-07-26T10:37:46.652 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-07-26T10:37:46.652 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-07-26T10:37:46.688 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-07-26T10:37:46.689 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi144.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-07-26T10:37:46.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T10:37:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:37:46.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T10:37:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:37:46.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T10:37:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:37:46.812 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T10:37:46.812 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2024-07-26T10:37:46.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-07-26T10:37:46.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-07-26T10:37:46.938 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T10:37:47.050 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T10:37:47.053 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T10:37:47.178 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T10:37:47.189 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T10:37:47.190 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi144.front.sepia.ceph.com,smithi160.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T10:44:15.775 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi144.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-07-26T10:44:15.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-07-26T10:44:15.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:44:15.893 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-07-26T10:44:16.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-07-26T10:44:16.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-07-26T10:44:16.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:44:16.127 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-07-26T10:44:16.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-07-26T10:44:16.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-07-26T10:44:16.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:44:16.285 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-07-26T10:44:16.366 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-07-26T10:44:16.367 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T10:44:16.378 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T10:44:16.378 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T10:44:16.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T10:44:16.386 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T10:44:16.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T10:44:16.389 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T10:44:16.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T10:44:16.414 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T10:44:16.427 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T10:44:16.441 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T10:44:16.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-07-26T10:44:16.455 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T10:44:16.459 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-07-26T10:44:16.472 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T10:44:16.475 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T10:44:16.479 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-07-26T10:44:16.485 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T10:44:16.490 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-07-26T10:44:16.503 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T10:44:16.516 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T10:44:16.517 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T10:44:16.544 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-07-26T10:44:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T10:44:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-07-26T10:44:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.572 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-07-26T10:44:16.575 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T10:44:16.575 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-07-26T10:44:16.575 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.575 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.576 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.576 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.591 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2024-07-26T10:44:16.623 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2024-07-26T10:44:16.662 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T10:44:16.701 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T10:44:16.842 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-07-26T10:44:16.851 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T10:44:16.851 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-07-26T10:44:16.851 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.851 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.851 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.851 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:44:16.852 INFO:teuthology.run_tasks:Running task install... 2024-07-26T10:44:16.860 DEBUG:teuthology.task.install:project ceph 2024-07-26T10:44:16.860 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-26T10:44:16.861 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-26T10:44:16.861 INFO:teuthology.task.install:Using flavor: default 2024-07-26T10:44:16.866 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T10:44:16.866 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T10:44:16.867 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T10:44:16.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:44:16.869 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T10:44:16.870 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:44:16.870 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T10:44:16.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:44:17.017 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T10:44:17.017 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T10:44:17.026 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T10:44:17.027 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T10:44:17.031 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T10:44:17.031 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T10:44:17.128 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T10:44:17.128 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:44:17.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T10:44:17.138 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T10:44:17.138 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:44:17.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T10:44:17.144 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T10:44:17.144 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:44:17.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T10:44:17.164 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T10:44:17.164 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T10:44:17.170 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T10:44:17.170 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T10:44:17.221 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T10:44:17.221 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T10:44:17.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T10:44:17.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T10:44:17.300 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T10:44:17.310 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T10:44:17.347 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T10:44:17.363 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-07-26T10:44:17.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-07-26T10:44:17.373 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2024-07-26T10:44:17.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2024-07-26T10:44:17.456 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T10:44:17.534 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2024-07-26T10:44:17.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2024-07-26T10:44:17.599 INFO:teuthology.orchestra.run.smithi160.stdout:45 files removed 2024-07-26T10:44:17.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T10:44:17.670 INFO:teuthology.orchestra.run.smithi183.stdout:45 files removed 2024-07-26T10:44:17.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T10:44:17.862 INFO:teuthology.orchestra.run.smithi144.stdout:45 files removed 2024-07-26T10:44:17.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T10:44:18.203 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 262 kB/s | 83 kB 00:00 2024-07-26T10:44:18.291 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 251 kB/s | 83 kB 00:00 2024-07-26T10:44:18.562 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 76 kB/s | 24 kB 00:00 2024-07-26T10:44:18.625 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 82 kB/s | 24 kB 00:00 2024-07-26T10:44:18.698 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 221 kB/s | 83 kB 00:00 2024-07-26T10:44:18.879 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-07-26T10:44:18.967 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-07-26T10:44:19.182 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 62 kB/s | 24 kB 00:00 2024-07-26T10:44:19.387 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-07-26T10:44:19.608 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-07-26T10:44:20.314 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-07-26T10:44:20.595 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.2 MB 00:01 2024-07-26T10:44:21.372 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-07-26T10:44:23.588 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-07-26T10:44:25.615 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-07-26T10:44:27.733 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 18 kB 00:00 2024-07-26T10:44:28.283 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-07-26T10:44:28.720 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-07-26T10:44:30.242 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 18 kB 00:00 2024-07-26T10:44:31.787 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 16 MB/s | 22 MB 00:01 2024-07-26T10:44:33.959 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-26T10:44:36.212 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:44:36.212 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:44:36.223 INFO:teuthology.orchestra.run.smithi160.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T10:44:36.224 INFO:teuthology.orchestra.run.smithi160.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T10:44:36.225 INFO:teuthology.orchestra.run.smithi160.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T10:44:36.226 INFO:teuthology.orchestra.run.smithi160.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T10:44:36.227 INFO:teuthology.orchestra.run.smithi160.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T10:44:36.229 INFO:teuthology.orchestra.run.smithi160.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T10:44:36.230 INFO:teuthology.orchestra.run.smithi160.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T10:44:36.295 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-07-26T10:44:36.314 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-07-26T10:44:36.314 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-07-26T10:44:36.314 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T10:44:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T10:44:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T10:44:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T10:44:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T10:44:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T10:44:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T10:44:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T10:44:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T10:44:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T10:44:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:44:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T10:44:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T10:44:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T10:44:36.320 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T10:44:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T10:44:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T10:44:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T10:44:36.324 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T10:44:36.325 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T10:44:36.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T10:44:36.327 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T10:44:36.328 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T10:44:36.328 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T10:44:36.328 INFO:teuthology.orchestra.run.smithi160.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T10:44:36.328 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T10:44:36.328 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T10:44:36.328 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T10:44:36.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T10:44:36.330 INFO:teuthology.orchestra.run.smithi160.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout:Install 148 Packages 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 12 Packages 2024-07-26T10:44:36.331 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:44:36.332 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 233 M 2024-07-26T10:44:36.332 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-07-26T10:44:37.010 INFO:teuthology.orchestra.run.smithi160.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-07-26T10:44:37.194 INFO:teuthology.orchestra.run.smithi160.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.6 MB/s | 872 kB 00:00 2024-07-26T10:44:37.260 INFO:teuthology.orchestra.run.smithi160.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.2 MB/s | 148 kB 00:00 2024-07-26T10:44:37.360 INFO:teuthology.orchestra.run.smithi160.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 5.5 MB 00:00 2024-07-26T10:44:37.475 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 20 MB 00:15 2024-07-26T10:44:37.502 INFO:teuthology.orchestra.run.smithi160.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 9.3 MB/s | 2.3 MB 00:00 2024-07-26T10:44:37.536 INFO:teuthology.orchestra.run.smithi160.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 8.7 MB/s | 1.5 MB 00:00 2024-07-26T10:44:37.953 INFO:teuthology.orchestra.run.smithi160.stdout:(7/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 18 MB/s | 20 MB 00:01 2024-07-26T10:44:38.032 INFO:teuthology.orchestra.run.smithi160.stdout:(8/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 9.2 MB/s | 4.9 MB 00:00 2024-07-26T10:44:38.085 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-26T10:44:38.087 INFO:teuthology.orchestra.run.smithi160.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 457 kB/s | 25 kB 00:00 2024-07-26T10:44:38.421 INFO:teuthology.orchestra.run.smithi160.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 22 MB/s | 10 MB 00:00 2024-07-26T10:44:38.596 INFO:teuthology.orchestra.run.smithi160.stdout:(11/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 17 MB/s | 18 MB 00:01 2024-07-26T10:44:38.621 INFO:teuthology.orchestra.run.smithi160.stdout:(12/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 1.1 MB/s | 225 kB 00:00 2024-07-26T10:44:38.638 INFO:teuthology.orchestra.run.smithi160.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 648 kB/s | 27 kB 00:00 2024-07-26T10:44:38.663 INFO:teuthology.orchestra.run.smithi160.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 757 kB/s | 32 kB 00:00 2024-07-26T10:44:38.722 INFO:teuthology.orchestra.run.smithi160.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 9.0 MB/s | 765 kB 00:00 2024-07-26T10:44:38.772 INFO:teuthology.orchestra.run.smithi160.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.4 MB/s | 158 kB 00:00 2024-07-26T10:44:38.797 INFO:teuthology.orchestra.run.smithi160.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.6 MB/s | 127 kB 00:00 2024-07-26T10:44:38.831 INFO:teuthology.orchestra.run.smithi160.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 8.4 MB/s | 503 kB 00:00 2024-07-26T10:44:38.890 INFO:teuthology.orchestra.run.smithi160.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 772 kB/s | 45 kB 00:00 2024-07-26T10:44:38.948 INFO:teuthology.orchestra.run.smithi160.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.3 MB/s | 138 kB 00:00 2024-07-26T10:44:39.007 INFO:teuthology.orchestra.run.smithi160.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.1 MB/s | 188 kB 00:00 2024-07-26T10:44:39.082 INFO:teuthology.orchestra.run.smithi160.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 18 MB/s | 5.3 MB 00:00 2024-07-26T10:44:39.107 INFO:teuthology.orchestra.run.smithi160.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 3.6 MB/s | 372 kB 00:00 2024-07-26T10:44:39.133 INFO:teuthology.orchestra.run.smithi160.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 6.9 MB/s | 355 kB 00:00 2024-07-26T10:44:39.158 INFO:teuthology.orchestra.run.smithi160.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T10:44:39.184 INFO:teuthology.orchestra.run.smithi160.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T10:44:39.242 INFO:teuthology.orchestra.run.smithi160.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.9 MB/s | 170 kB 00:00 2024-07-26T10:44:39.326 INFO:teuthology.orchestra.run.smithi160.stdout:(28/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 18 MB/s | 3.1 MB 00:00 2024-07-26T10:44:39.351 INFO:teuthology.orchestra.run.smithi160.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 316 kB/s | 34 kB 00:00 2024-07-26T10:44:39.378 INFO:teuthology.orchestra.run.smithi160.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.9 MB/s | 149 kB 00:00 2024-07-26T10:44:39.778 INFO:teuthology.orchestra.run.smithi160.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 19 MB/s | 7.4 MB 00:00 2024-07-26T10:44:39.844 INFO:teuthology.orchestra.run.smithi160.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 4.2 MB/s | 284 kB 00:00 2024-07-26T10:44:39.894 INFO:teuthology.orchestra.run.smithi160.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 990 kB/s | 49 kB 00:00 2024-07-26T10:44:39.936 INFO:teuthology.orchestra.run.smithi160.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 395 kB/s | 16 kB 00:00 2024-07-26T10:44:39.995 INFO:teuthology.orchestra.run.smithi160.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 4.6 MB/s | 274 kB 00:00 2024-07-26T10:44:40.162 INFO:teuthology.orchestra.run.smithi160.stdout:(36/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 17 MB/s | 14 MB 00:00 2024-07-26T10:44:40.198 INFO:teuthology.orchestra.run.smithi160.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 3.8 MB/s | 771 kB 00:00 2024-07-26T10:44:40.221 INFO:teuthology.orchestra.run.smithi160.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 312 kB/s | 18 kB 00:00 2024-07-26T10:44:40.371 INFO:teuthology.orchestra.run.smithi160.stdout:(39/160): fuse-2.9.9-16.el9.x86_64.rpm 456 kB/s | 80 kB 00:00 2024-07-26T10:44:40.604 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:44:40.606 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:44:40.623 INFO:teuthology.orchestra.run.smithi144.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T10:44:40.624 INFO:teuthology.orchestra.run.smithi144.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T10:44:40.626 INFO:teuthology.orchestra.run.smithi144.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T10:44:40.627 INFO:teuthology.orchestra.run.smithi144.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T10:44:40.629 INFO:teuthology.orchestra.run.smithi144.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T10:44:40.632 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T10:44:40.633 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T10:44:40.702 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T10:44:40.725 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T10:44:40.726 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T10:44:40.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T10:44:40.728 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-07-26T10:44:40.729 INFO:teuthology.orchestra.run.smithi144.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T10:44:40.730 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T10:44:40.731 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T10:44:40.732 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T10:44:40.732 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T10:44:40.732 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T10:44:40.733 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T10:44:40.734 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T10:44:40.735 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T10:44:40.736 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T10:44:40.737 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T10:44:40.738 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T10:44:40.739 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout:Installing weak dependencies: 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout:Install 148 Packages 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 12 Packages 2024-07-26T10:44:40.740 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:44:40.741 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 233 M 2024-07-26T10:44:40.741 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-07-26T10:44:40.746 INFO:teuthology.orchestra.run.smithi160.stdout:(40/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 17 MB/s | 44 MB 00:02 2024-07-26T10:44:40.771 INFO:teuthology.orchestra.run.smithi160.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 488 kB/s | 195 kB 00:00 2024-07-26T10:44:40.805 INFO:teuthology.orchestra.run.smithi160.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-07-26T10:44:40.913 INFO:teuthology.orchestra.run.smithi160.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-07-26T10:44:40.964 INFO:teuthology.orchestra.run.smithi160.stdout:(44/160): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-07-26T10:44:40.989 INFO:teuthology.orchestra.run.smithi160.stdout:(45/160): python3-ply-3.11-14.el9.noarch.rpm 580 kB/s | 106 kB 00:00 2024-07-26T10:44:41.023 INFO:teuthology.orchestra.run.smithi160.stdout:(46/160): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.2 MB 00:00 2024-07-26T10:44:41.048 INFO:teuthology.orchestra.run.smithi160.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-07-26T10:44:41.207 INFO:teuthology.orchestra.run.smithi160.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 989 kB/s | 215 kB 00:00 2024-07-26T10:44:41.390 INFO:teuthology.orchestra.run.smithi160.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 584 kB/s | 107 kB 00:00 2024-07-26T10:44:41.457 INFO:teuthology.orchestra.run.smithi160.stdout:(50/160): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-07-26T10:44:41.482 INFO:teuthology.orchestra.run.smithi160.stdout:(51/160): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 689 kB 00:00 2024-07-26T10:44:41.532 INFO:teuthology.orchestra.run.smithi160.stdout:(52/160): elfutils-libelf-devel-0.191-4.el9.x86 153 kB/s | 22 kB 00:00 2024-07-26T10:44:41.649 INFO:teuthology.orchestra.run.smithi160.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 1.6 MB/s | 313 kB 00:00 2024-07-26T10:44:41.675 INFO:teuthology.orchestra.run.smithi160.stdout:(54/160): flexiblas-3.0.4-8.el9.x86_64.rpm 195 kB/s | 33 kB 00:00 2024-07-26T10:44:41.728 INFO:teuthology.orchestra.run.smithi144.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-07-26T10:44:41.792 INFO:teuthology.orchestra.run.smithi160.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 18 kB 00:00 2024-07-26T10:44:41.825 INFO:teuthology.orchestra.run.smithi160.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 11 kB 00:00 2024-07-26T10:44:41.920 INFO:teuthology.orchestra.run.smithi144.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.5 MB/s | 872 kB 00:00 2024-07-26T10:44:41.934 INFO:teuthology.orchestra.run.smithi160.stdout:(57/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.9 MB/s | 3.0 MB 00:00 2024-07-26T10:44:41.996 INFO:teuthology.orchestra.run.smithi144.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.0 MB/s | 148 kB 00:00 2024-07-26T10:44:42.001 INFO:teuthology.orchestra.run.smithi160.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 767 kB/s | 160 kB 00:00 2024-07-26T10:44:42.026 INFO:teuthology.orchestra.run.smithi160.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 256 kB/s | 45 kB 00:00 2024-07-26T10:44:42.124 INFO:teuthology.orchestra.run.smithi144.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 8.3 MB/s | 5.5 MB 00:00 2024-07-26T10:44:42.143 INFO:teuthology.orchestra.run.smithi160.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-07-26T10:44:42.168 INFO:teuthology.orchestra.run.smithi160.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 216 kB/s | 25 kB 00:00 2024-07-26T10:44:42.238 INFO:teuthology.orchestra.run.smithi144.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 9.4 MB/s | 2.3 MB 00:00 2024-07-26T10:44:42.260 INFO:teuthology.orchestra.run.smithi160.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 2.2 MB/s | 581 kB 00:00 2024-07-26T10:44:42.296 INFO:teuthology.orchestra.run.smithi144.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 8.9 MB/s | 1.5 MB 00:00 2024-07-26T10:44:42.327 INFO:teuthology.orchestra.run.smithi160.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-07-26T10:44:42.386 INFO:teuthology.orchestra.run.smithi160.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2024-07-26T10:44:42.445 INFO:teuthology.orchestra.run.smithi160.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 304 kB/s | 35 kB 00:00 2024-07-26T10:44:42.470 INFO:teuthology.orchestra.run.smithi160.stdout:(66/160): lua-5.4.4-4.el9.x86_64.rpm 905 kB/s | 188 kB 00:00 2024-07-26T10:44:42.588 INFO:teuthology.orchestra.run.smithi144.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 14 MB/s | 4.9 MB 00:00 2024-07-26T10:44:42.828 INFO:teuthology.orchestra.run.smithi160.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-07-26T10:44:42.937 INFO:teuthology.orchestra.run.smithi160.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 8.6 MB/s | 4.7 MB 00:00 2024-07-26T10:44:43.039 INFO:teuthology.orchestra.run.smithi144.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 20 MB 00:01 2024-07-26T10:44:43.054 INFO:teuthology.orchestra.run.smithi160.stdout:(69/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-07-26T10:44:43.089 INFO:teuthology.orchestra.run.smithi144.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 514 kB/s | 25 kB 00:00 2024-07-26T10:44:43.112 INFO:teuthology.orchestra.run.smithi160.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 761 kB/s | 133 kB 00:00 2024-07-26T10:44:43.146 INFO:teuthology.orchestra.run.smithi160.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 2.8 MB/s | 911 kB 00:00 2024-07-26T10:44:43.296 INFO:teuthology.orchestra.run.smithi160.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 696 kB/s | 87 kB 00:00 2024-07-26T10:44:43.372 INFO:teuthology.orchestra.run.smithi144.stdout:(10/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 17 MB/s | 18 MB 00:01 2024-07-26T10:44:43.397 INFO:teuthology.orchestra.run.smithi160.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-07-26T10:44:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:(11/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 11 MB/s | 10 MB 00:00 2024-07-26T10:44:43.522 INFO:teuthology.orchestra.run.smithi160.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 491 kB/s | 61 kB 00:00 2024-07-26T10:44:43.540 INFO:teuthology.orchestra.run.smithi144.stdout:(12/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 1.3 MB/s | 225 kB 00:00 2024-07-26T10:44:43.565 INFO:teuthology.orchestra.run.smithi144.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 469 kB/s | 27 kB 00:00 2024-07-26T10:44:43.591 INFO:teuthology.orchestra.run.smithi144.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 638 kB/s | 32 kB 00:00 2024-07-26T10:44:43.649 INFO:teuthology.orchestra.run.smithi144.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 9.0 MB/s | 765 kB 00:00 2024-07-26T10:44:43.672 INFO:teuthology.orchestra.run.smithi160.stdout:(75/160): postgresql-server-13.14-1.el9.x86_64. 9.4 MB/s | 5.8 MB 00:00 2024-07-26T10:44:43.683 INFO:teuthology.orchestra.run.smithi144.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.7 MB/s | 158 kB 00:00 2024-07-26T10:44:43.716 INFO:teuthology.orchestra.run.smithi144.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.9 MB/s | 127 kB 00:00 2024-07-26T10:44:43.739 INFO:teuthology.orchestra.run.smithi160.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-07-26T10:44:43.767 INFO:teuthology.orchestra.run.smithi144.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.9 MB/s | 503 kB 00:00 2024-07-26T10:44:43.825 INFO:teuthology.orchestra.run.smithi144.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 787 kB/s | 45 kB 00:00 2024-07-26T10:44:43.872 INFO:teuthology.orchestra.run.smithi160.stdout:(77/160): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-07-26T10:44:43.886 INFO:teuthology.orchestra.run.smithi144.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.4 MB/s | 138 kB 00:00 2024-07-26T10:44:43.889 INFO:teuthology.orchestra.run.smithi160.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 172 kB/s | 26 kB 00:00 2024-07-26T10:44:43.940 INFO:teuthology.orchestra.run.smithi160.stdout:(79/160): python3-jinja2-2.11.3-6.el9.noarch.rp 930 kB/s | 248 kB 00:00 2024-07-26T10:44:43.942 INFO:teuthology.orchestra.run.smithi144.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T10:44:43.982 INFO:teuthology.orchestra.run.smithi160.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-07-26T10:44:44.010 INFO:teuthology.orchestra.run.smithi144.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 5.5 MB/s | 372 kB 00:00 2024-07-26T10:44:44.041 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.5 MB 00:02 2024-07-26T10:44:44.082 INFO:teuthology.orchestra.run.smithi160.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-07-26T10:44:44.110 INFO:teuthology.orchestra.run.smithi144.stdout:(23/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 5.3 MB 00:00 2024-07-26T10:44:44.141 INFO:teuthology.orchestra.run.smithi160.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 198 kB/s | 35 kB 00:00 2024-07-26T10:44:44.160 INFO:teuthology.orchestra.run.smithi144.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 355 kB 00:00 2024-07-26T10:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.1 MB/s | 116 kB 00:00 2024-07-26T10:44:44.232 INFO:teuthology.orchestra.run.smithi160.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-07-26T10:44:44.244 INFO:teuthology.orchestra.run.smithi144.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.0 MB/s | 86 kB 00:00 2024-07-26T10:44:44.303 INFO:teuthology.orchestra.run.smithi144.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.9 MB/s | 170 kB 00:00 2024-07-26T10:44:44.316 INFO:teuthology.orchestra.run.smithi160.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-07-26T10:44:44.353 INFO:teuthology.orchestra.run.smithi144.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 701 kB/s | 34 kB 00:00 2024-07-26T10:44:44.383 INFO:teuthology.orchestra.run.smithi160.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 512 kB/s | 77 kB 00:00 2024-07-26T10:44:44.429 INFO:teuthology.orchestra.run.smithi144.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 3.1 MB 00:00 2024-07-26T10:44:44.462 INFO:teuthology.orchestra.run.smithi144.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.3 MB/s | 149 kB 00:00 2024-07-26T10:44:44.492 INFO:teuthology.orchestra.run.smithi160.stdout:(86/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-07-26T10:44:44.517 INFO:teuthology.orchestra.run.smithi160.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 210 kB/s | 42 kB 00:00 2024-07-26T10:44:44.567 INFO:teuthology.orchestra.run.smithi160.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-07-26T10:44:44.676 INFO:teuthology.orchestra.run.smithi160.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 865 kB/s | 159 kB 00:00 2024-07-26T10:44:44.701 INFO:teuthology.orchestra.run.smithi160.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2024-07-26T10:44:44.751 INFO:teuthology.orchestra.run.smithi160.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.3 MB/s | 279 kB 00:00 2024-07-26T10:44:44.802 INFO:teuthology.orchestra.run.smithi160.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 428 kB/s | 54 kB 00:00 2024-07-26T10:44:44.927 INFO:teuthology.orchestra.run.smithi160.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 277 kB/s | 42 kB 00:00 2024-07-26T10:44:44.986 INFO:teuthology.orchestra.run.smithi160.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 278 kB/s | 44 kB 00:00 2024-07-26T10:44:45.004 INFO:teuthology.orchestra.run.smithi144.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-07-26T10:44:45.063 INFO:teuthology.orchestra.run.smithi144.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 4.8 MB/s | 284 kB 00:00 2024-07-26T10:44:45.086 INFO:teuthology.orchestra.run.smithi160.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-07-26T10:44:45.114 INFO:teuthology.orchestra.run.smithi160.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 289 kB/s | 36 kB 00:00 2024-07-26T10:44:45.130 INFO:teuthology.orchestra.run.smithi144.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 744 kB/s | 49 kB 00:00 2024-07-26T10:44:45.180 INFO:teuthology.orchestra.run.smithi144.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 333 kB/s | 16 kB 00:00 2024-07-26T10:44:45.262 INFO:teuthology.orchestra.run.smithi160.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 271 kB/s | 47 kB 00:00 2024-07-26T10:44:45.322 INFO:teuthology.orchestra.run.smithi160.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 1.6 MB/s | 304 kB 00:00 2024-07-26T10:44:45.356 INFO:teuthology.orchestra.run.smithi144.stdout:(35/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 15 MB/s | 14 MB 00:00 2024-07-26T10:44:45.389 INFO:teuthology.orchestra.run.smithi144.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 1.3 MB/s | 274 kB 00:00 2024-07-26T10:44:45.412 INFO:teuthology.orchestra.run.smithi160.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 386 kB/s | 58 kB 00:00 2024-07-26T10:44:45.456 INFO:teuthology.orchestra.run.smithi144.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 7.6 MB/s | 771 kB 00:00 2024-07-26T10:44:45.479 INFO:teuthology.orchestra.run.smithi160.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 607 kB/s | 96 kB 00:00 2024-07-26T10:44:45.481 INFO:teuthology.orchestra.run.smithi144.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 200 kB/s | 18 kB 00:00 2024-07-26T10:44:45.609 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 18 kB 00:00 2024-07-26T10:44:45.671 INFO:teuthology.orchestra.run.smithi160.stdout:(101/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-07-26T10:44:45.688 INFO:teuthology.orchestra.run.smithi160.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 231 kB/s | 64 kB 00:00 2024-07-26T10:44:45.780 INFO:teuthology.orchestra.run.smithi160.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 149 kB/s | 45 kB 00:00 2024-07-26T10:44:45.797 INFO:teuthology.orchestra.run.smithi160.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 180 kB/s | 22 kB 00:00 2024-07-26T10:44:45.889 INFO:teuthology.orchestra.run.smithi160.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.9 MB/s | 548 kB 00:00 2024-07-26T10:44:45.906 INFO:teuthology.orchestra.run.smithi160.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-07-26T10:44:45.934 INFO:teuthology.orchestra.run.smithi160.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.3 MB/s | 309 kB 00:00 2024-07-26T10:44:45.956 INFO:teuthology.orchestra.run.smithi160.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-26T10:44:45.981 INFO:teuthology.orchestra.run.smithi160.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-26T10:44:45.982 INFO:teuthology.orchestra.run.smithi144.stdout:(39/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 15 MB/s | 44 MB 00:02 2024-07-26T10:44:46.007 INFO:teuthology.orchestra.run.smithi160.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-26T10:44:46.007 INFO:teuthology.orchestra.run.smithi144.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 145 kB/s | 80 kB 00:00 2024-07-26T10:44:46.040 INFO:teuthology.orchestra.run.smithi160.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-07-26T10:44:46.082 INFO:teuthology.orchestra.run.smithi160.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-07-26T10:44:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:(41/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-07-26T10:44:46.124 INFO:teuthology.orchestra.run.smithi160.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-07-26T10:44:46.149 INFO:teuthology.orchestra.run.smithi160.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T10:44:46.157 INFO:teuthology.orchestra.run.smithi144.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-07-26T10:44:46.183 INFO:teuthology.orchestra.run.smithi144.stdout:(43/160): libquadmath-11.4.1-3.el9.x86_64.rpm 978 kB/s | 195 kB 00:00 2024-07-26T10:44:46.217 INFO:teuthology.orchestra.run.smithi144.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-07-26T10:44:46.217 INFO:teuthology.orchestra.run.smithi160.stdout:(115/160): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-07-26T10:44:46.242 INFO:teuthology.orchestra.run.smithi160.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 349 kB/s | 32 kB 00:00 2024-07-26T10:44:46.267 INFO:teuthology.orchestra.run.smithi160.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 282 kB/s | 14 kB 00:00 2024-07-26T10:44:46.275 INFO:teuthology.orchestra.run.smithi144.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 135 kB 00:00 2024-07-26T10:44:46.293 INFO:teuthology.orchestra.run.smithi160.stdout:(118/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.4 MB/s | 863 kB 00:00 2024-07-26T10:44:46.309 INFO:teuthology.orchestra.run.smithi144.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-07-26T10:44:46.342 INFO:teuthology.orchestra.run.smithi144.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-07-26T10:44:46.343 INFO:teuthology.orchestra.run.smithi160.stdout:(119/160): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-07-26T10:44:46.368 INFO:teuthology.orchestra.run.smithi160.stdout:(120/160): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-07-26T10:44:46.393 INFO:teuthology.orchestra.run.smithi144.stdout:(48/160): python3-cryptography-36.0.1-4.el9.x86 4.4 MB/s | 1.2 MB 00:00 2024-07-26T10:44:46.393 INFO:teuthology.orchestra.run.smithi160.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.4 MB/s | 216 kB 00:00 2024-07-26T10:44:46.444 INFO:teuthology.orchestra.run.smithi160.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-07-26T10:44:46.469 INFO:teuthology.orchestra.run.smithi160.stdout:(123/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.7 MB/s | 303 kB 00:00 2024-07-26T10:44:46.493 INFO:teuthology.orchestra.run.smithi144.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 689 kB 00:00 2024-07-26T10:44:46.494 INFO:teuthology.orchestra.run.smithi160.stdout:(124/160): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-07-26T10:44:46.518 INFO:teuthology.orchestra.run.smithi144.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 860 kB/s | 107 kB 00:00 2024-07-26T10:44:46.519 INFO:teuthology.orchestra.run.smithi160.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-07-26T10:44:46.545 INFO:teuthology.orchestra.run.smithi160.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-07-26T10:44:46.552 INFO:teuthology.orchestra.run.smithi144.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 371 kB/s | 22 kB 00:00 2024-07-26T10:44:46.570 INFO:teuthology.orchestra.run.smithi160.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-07-26T10:44:46.594 INFO:teuthology.orchestra.run.smithi144.stdout:(52/160): flexiblas-3.0.4-8.el9.x86_64.rpm 803 kB/s | 33 kB 00:00 2024-07-26T10:44:46.595 INFO:teuthology.orchestra.run.smithi160.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-07-26T10:44:46.596 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-07-26T10:44:46.621 INFO:teuthology.orchestra.run.smithi160.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T10:44:46.627 INFO:teuthology.orchestra.run.smithi144.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-07-26T10:44:46.646 INFO:teuthology.orchestra.run.smithi160.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T10:44:46.670 INFO:teuthology.orchestra.run.smithi144.stdout:(54/160): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-07-26T10:44:46.671 INFO:teuthology.orchestra.run.smithi160.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-07-26T10:44:46.695 INFO:teuthology.orchestra.run.smithi144.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 274 kB/s | 18 kB 00:00 2024-07-26T10:44:46.696 INFO:teuthology.orchestra.run.smithi160.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 11 MB/s | 272 kB 00:00 2024-07-26T10:44:46.720 INFO:teuthology.orchestra.run.smithi144.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-07-26T10:44:46.722 INFO:teuthology.orchestra.run.smithi160.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-07-26T10:44:46.746 INFO:teuthology.orchestra.run.smithi144.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-07-26T10:44:46.747 INFO:teuthology.orchestra.run.smithi160.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-07-26T10:44:46.771 INFO:teuthology.orchestra.run.smithi144.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-07-26T10:44:46.772 INFO:teuthology.orchestra.run.smithi160.stdout:(135/160): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-07-26T10:44:46.798 INFO:teuthology.orchestra.run.smithi160.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 560 kB/s | 28 kB 00:00 2024-07-26T10:44:46.821 INFO:teuthology.orchestra.run.smithi144.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-07-26T10:44:46.823 INFO:teuthology.orchestra.run.smithi160.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-07-26T10:44:46.848 INFO:teuthology.orchestra.run.smithi160.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-26T10:44:46.855 INFO:teuthology.orchestra.run.smithi144.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 6.9 MB/s | 581 kB 00:00 2024-07-26T10:44:46.874 INFO:teuthology.orchestra.run.smithi160.stdout:(139/160): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-07-26T10:44:46.882 INFO:teuthology.orchestra.run.smithi144.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-07-26T10:44:46.899 INFO:teuthology.orchestra.run.smithi160.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-07-26T10:44:46.905 INFO:teuthology.orchestra.run.smithi144.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-07-26T10:44:46.939 INFO:teuthology.orchestra.run.smithi144.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-07-26T10:44:46.941 INFO:teuthology.orchestra.run.smithi160.stdout:(141/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.6 MB/s | 2.0 MB 00:00 2024-07-26T10:44:46.966 INFO:teuthology.orchestra.run.smithi160.stdout:(142/160): python3-typing-extensions-4.4.0-2.el 762 kB/s | 51 kB 00:00 2024-07-26T10:44:46.973 INFO:teuthology.orchestra.run.smithi144.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-07-26T10:44:46.991 INFO:teuthology.orchestra.run.smithi160.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-26T10:44:46.998 INFO:teuthology.orchestra.run.smithi144.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 610 kB/s | 35 kB 00:00 2024-07-26T10:44:47.017 INFO:teuthology.orchestra.run.smithi160.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-07-26T10:44:47.034 INFO:teuthology.orchestra.run.smithi160.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 473 kB/s | 20 kB 00:00 2024-07-26T10:44:47.068 INFO:teuthology.orchestra.run.smithi160.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T10:44:47.091 INFO:teuthology.orchestra.run.smithi144.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.0 MB/s | 3.0 MB 00:00 2024-07-26T10:44:47.092 INFO:teuthology.orchestra.run.smithi160.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-07-26T10:44:47.143 INFO:teuthology.orchestra.run.smithi160.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-07-26T10:44:47.191 INFO:teuthology.orchestra.run.smithi144.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-07-26T10:44:47.324 INFO:teuthology.orchestra.run.smithi144.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-07-26T10:44:47.335 INFO:teuthology.orchestra.run.smithi160.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 3.4 MB 00:00 2024-07-26T10:44:47.374 INFO:teuthology.orchestra.run.smithi144.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 4.9 MB/s | 911 kB 00:00 2024-07-26T10:44:47.385 INFO:teuthology.orchestra.run.smithi160.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 3.1 MB 00:00 2024-07-26T10:44:47.444 INFO:teuthology.orchestra.run.smithi160.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 698 kB/s | 41 kB 00:00 2024-07-26T10:44:47.450 INFO:teuthology.orchestra.run.smithi144.stdout:(70/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-07-26T10:44:47.475 INFO:teuthology.orchestra.run.smithi144.stdout:(71/160): postgresql-private-libs-13.14-1.el9.x 889 kB/s | 133 kB 00:00 2024-07-26T10:44:47.517 INFO:teuthology.orchestra.run.smithi144.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-07-26T10:44:47.536 INFO:teuthology.orchestra.run.smithi160.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 2.6 MB/s | 247 kB 00:00 2024-07-26T10:44:47.559 INFO:teuthology.orchestra.run.smithi144.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-07-26T10:44:47.601 INFO:teuthology.orchestra.run.smithi144.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.5 MB/s | 61 kB 00:00 2024-07-26T10:44:47.611 INFO:teuthology.orchestra.run.smithi160.stdout:(153/160): NetworkManager-1.48.2-2.el9.x86_64.r 4.9 MB/s | 2.3 MB 00:00 2024-07-26T10:44:47.636 INFO:teuthology.orchestra.run.smithi160.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 366 kB/s | 37 kB 00:00 2024-07-26T10:44:47.703 INFO:teuthology.orchestra.run.smithi160.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 2.2 MB/s | 207 kB 00:00 2024-07-26T10:44:47.715 INFO:teuthology.orchestra.run.smithi144.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.1 MB/s | 244 kB 00:00 2024-07-26T10:44:47.737 INFO:teuthology.orchestra.run.smithi160.stdout:(156/160): NetworkManager-libnm-1.48.2-2.el9.x8 4.5 MB/s | 1.8 MB 00:00 2024-07-26T10:44:47.762 INFO:teuthology.orchestra.run.smithi160.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.0 MB/s | 260 kB 00:00 2024-07-26T10:44:47.778 INFO:teuthology.orchestra.run.smithi144.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 4.0 MB/s | 248 kB 00:00 2024-07-26T10:44:47.885 INFO:teuthology.orchestra.run.smithi144.stdout:(77/160): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-07-26T10:44:47.911 INFO:teuthology.orchestra.run.smithi144.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 192 kB/s | 26 kB 00:00 2024-07-26T10:44:47.936 INFO:teuthology.orchestra.run.smithi144.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 387 kB/s | 19 kB 00:00 2024-07-26T10:44:47.962 INFO:teuthology.orchestra.run.smithi160.stdout:(158/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-07-26T10:44:47.969 INFO:teuthology.orchestra.run.smithi144.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-07-26T10:44:47.995 INFO:teuthology.orchestra.run.smithi144.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-07-26T10:44:47.996 INFO:teuthology.orchestra.run.smithi160.stdout:(159/160): openssl-3.2.2-2.el9.x86_64.rpm 4.6 MB/s | 1.4 MB 00:00 2024-07-26T10:44:48.120 INFO:teuthology.orchestra.run.smithi144.stdout:(82/160): python3-babel-2.9.1-2.el9.noarch.rpm 9.9 MB/s | 6.0 MB 00:00 2024-07-26T10:44:48.145 INFO:teuthology.orchestra.run.smithi144.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-07-26T10:44:48.204 INFO:teuthology.orchestra.run.smithi144.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-07-26T10:44:48.221 INFO:teuthology.orchestra.run.smithi144.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-07-26T10:44:48.247 INFO:teuthology.orchestra.run.smithi144.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-07-26T10:44:48.254 INFO:teuthology.orchestra.run.smithi160.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 4.6 MB/s | 2.4 MB 00:00 2024-07-26T10:44:48.260 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:44:48.261 INFO:teuthology.orchestra.run.smithi160.stdout:Total 20 MB/s | 233 MB 00:11 2024-07-26T10:44:48.280 INFO:teuthology.orchestra.run.smithi144.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-07-26T10:44:48.322 INFO:teuthology.orchestra.run.smithi144.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-07-26T10:44:48.348 INFO:teuthology.orchestra.run.smithi144.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-07-26T10:44:48.373 INFO:teuthology.orchestra.run.smithi144.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-07-26T10:44:48.398 INFO:teuthology.orchestra.run.smithi144.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-07-26T10:44:48.448 INFO:teuthology.orchestra.run.smithi144.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 872 kB/s | 42 kB 00:00 2024-07-26T10:44:48.490 INFO:teuthology.orchestra.run.smithi144.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.1 MB/s | 44 kB 00:00 2024-07-26T10:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-07-26T10:44:48.649 INFO:teuthology.orchestra.run.smithi144.stdout:(95/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 5.1 MB 00:00 2024-07-26T10:44:48.675 INFO:teuthology.orchestra.run.smithi144.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 270 kB/s | 36 kB 00:00 2024-07-26T10:44:48.700 INFO:teuthology.orchestra.run.smithi144.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 961 kB/s | 47 kB 00:00 2024-07-26T10:44:48.734 INFO:teuthology.orchestra.run.smithi144.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-07-26T10:44:48.759 INFO:teuthology.orchestra.run.smithi144.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-07-26T10:44:48.784 INFO:teuthology.orchestra.run.smithi144.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-07-26T10:44:48.809 INFO:teuthology.orchestra.run.smithi144.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-07-26T10:44:48.835 INFO:teuthology.orchestra.run.smithi144.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-07-26T10:44:48.935 INFO:teuthology.orchestra.run.smithi144.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-07-26T10:44:49.044 INFO:teuthology.orchestra.run.smithi144.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 5.0 MB/s | 548 kB 00:00 2024-07-26T10:44:49.086 INFO:teuthology.orchestra.run.smithi144.stdout:(105/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-07-26T10:44:49.169 INFO:teuthology.orchestra.run.smithi144.stdout:(106/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.4 MB/s | 309 kB 00:00 2024-07-26T10:44:49.195 INFO:teuthology.orchestra.run.smithi144.stdout:(107/160): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-07-26T10:44:49.220 INFO:teuthology.orchestra.run.smithi144.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-26T10:44:49.254 INFO:teuthology.orchestra.run.smithi144.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-07-26T10:44:49.287 INFO:teuthology.orchestra.run.smithi144.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-07-26T10:44:49.351 INFO:teuthology.orchestra.run.smithi144.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 2.4 MB/s | 151 kB 00:00 2024-07-26T10:44:49.438 INFO:teuthology.orchestra.run.smithi144.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.5 MB/s | 839 kB 00:00 2024-07-26T10:44:49.494 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-07-26T10:44:49.622 INFO:teuthology.orchestra.run.smithi144.stdout:(113/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-07-26T10:44:49.691 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-07-26T10:44:49.691 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-07-26T10:44:49.722 INFO:teuthology.orchestra.run.smithi144.stdout:(114/160): python3-asyncssh-2.13.2-5.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-07-26T10:44:49.756 INFO:teuthology.orchestra.run.smithi144.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.1 MB/s | 32 kB 00:00 2024-07-26T10:44:49.781 INFO:teuthology.orchestra.run.smithi144.stdout:(116/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 275 kB/s | 43 kB 00:00 2024-07-26T10:44:49.806 INFO:teuthology.orchestra.run.smithi144.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 284 kB/s | 14 kB 00:00 2024-07-26T10:44:49.848 INFO:teuthology.orchestra.run.smithi144.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-07-26T10:44:49.890 INFO:teuthology.orchestra.run.smithi144.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-07-26T10:44:49.924 INFO:teuthology.orchestra.run.smithi144.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-07-26T10:44:49.958 INFO:teuthology.orchestra.run.smithi144.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.2 MB/s | 216 kB 00:00 2024-07-26T10:44:49.991 INFO:teuthology.orchestra.run.smithi144.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T10:44:50.017 INFO:teuthology.orchestra.run.smithi144.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 443 kB/s | 11 kB 00:00 2024-07-26T10:44:50.042 INFO:teuthology.orchestra.run.smithi144.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 733 kB/s | 18 kB 00:00 2024-07-26T10:44:50.067 INFO:teuthology.orchestra.run.smithi144.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 964 kB/s | 23 kB 00:00 2024-07-26T10:44:50.084 INFO:teuthology.orchestra.run.smithi144.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 1.2 MB/s | 19 kB 00:00 2024-07-26T10:44:50.101 INFO:teuthology.orchestra.run.smithi144.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 1.2 MB/s | 20 kB 00:00 2024-07-26T10:44:50.193 INFO:teuthology.orchestra.run.smithi144.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-07-26T10:44:50.218 INFO:teuthology.orchestra.run.smithi144.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T10:44:50.252 INFO:teuthology.orchestra.run.smithi144.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T10:44:50.286 INFO:teuthology.orchestra.run.smithi144.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-07-26T10:44:50.328 INFO:teuthology.orchestra.run.smithi144.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-07-26T10:44:50.403 INFO:teuthology.orchestra.run.smithi144.stdout:(133/160): libarrow-9.0.0-11.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-07-26T10:44:50.470 INFO:teuthology.orchestra.run.smithi144.stdout:(134/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-07-26T10:44:50.496 INFO:teuthology.orchestra.run.smithi144.stdout:(135/160): python3-portend-3.1.0-2.el9.noarch.r 98 kB/s | 16 kB 00:00 2024-07-26T10:44:50.521 INFO:teuthology.orchestra.run.smithi144.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 578 kB/s | 28 kB 00:00 2024-07-26T10:44:50.546 INFO:teuthology.orchestra.run.smithi144.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 625 kB/s | 31 kB 00:00 2024-07-26T10:44:50.571 INFO:teuthology.orchestra.run.smithi144.stdout:(138/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 902 kB/s | 90 kB 00:00 2024-07-26T10:44:50.596 INFO:teuthology.orchestra.run.smithi144.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-07-26T10:44:50.622 INFO:teuthology.orchestra.run.smithi144.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 726 kB/s | 36 kB 00:00 2024-07-26T10:44:50.647 INFO:teuthology.orchestra.run.smithi144.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T10:44:50.681 INFO:teuthology.orchestra.run.smithi144.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-07-26T10:44:50.706 INFO:teuthology.orchestra.run.smithi144.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-07-26T10:44:50.731 INFO:teuthology.orchestra.run.smithi144.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 820 kB/s | 20 kB 00:00 2024-07-26T10:44:50.765 INFO:teuthology.orchestra.run.smithi144.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-07-26T10:44:50.799 INFO:teuthology.orchestra.run.smithi144.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-07-26T10:44:50.832 INFO:teuthology.orchestra.run.smithi144.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-07-26T10:44:50.958 INFO:teuthology.orchestra.run.smithi144.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-07-26T10:44:51.091 INFO:teuthology.orchestra.run.smithi144.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 12 MB/s | 3.4 MB 00:00 2024-07-26T10:44:51.142 INFO:teuthology.orchestra.run.smithi144.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-07-26T10:44:51.192 INFO:teuthology.orchestra.run.smithi144.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 832 kB/s | 41 kB 00:00 2024-07-26T10:44:51.267 INFO:teuthology.orchestra.run.smithi144.stdout:(152/160): NetworkManager-libnm-1.48.2-2.el9.x8 10 MB/s | 1.8 MB 00:00 2024-07-26T10:44:51.301 INFO:teuthology.orchestra.run.smithi144.stdout:(153/160): NetworkManager-tui-1.48.2-2.el9.x86_ 2.2 MB/s | 247 kB 00:00 2024-07-26T10:44:51.326 INFO:teuthology.orchestra.run.smithi144.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 636 kB/s | 37 kB 00:00 2024-07-26T10:44:51.351 INFO:teuthology.orchestra.run.smithi144.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 4.1 MB/s | 207 kB 00:00 2024-07-26T10:44:51.402 INFO:teuthology.orchestra.run.smithi144.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 3.4 MB/s | 260 kB 00:00 2024-07-26T10:44:51.469 INFO:teuthology.orchestra.run.smithi144.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 12 MB/s | 1.4 MB 00:00 2024-07-26T10:44:51.528 INFO:teuthology.orchestra.run.smithi144.stdout:(158/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-07-26T10:44:51.545 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-07-26T10:44:51.545 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-07-26T10:44:51.586 INFO:teuthology.orchestra.run.smithi144.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 9.3 MB/s | 1.1 MB 00:00 2024-07-26T10:44:51.636 INFO:teuthology.orchestra.run.smithi144.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-07-26T10:44:51.649 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:44:51.650 INFO:teuthology.orchestra.run.smithi144.stdout:Total 21 MB/s | 233 MB 00:10 2024-07-26T10:44:51.952 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-07-26T10:44:54.221 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:44:54.222 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:44:54.233 INFO:teuthology.orchestra.run.smithi183.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T10:44:54.234 INFO:teuthology.orchestra.run.smithi183.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T10:44:54.235 INFO:teuthology.orchestra.run.smithi183.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T10:44:54.236 INFO:teuthology.orchestra.run.smithi183.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T10:44:54.237 INFO:teuthology.orchestra.run.smithi183.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T10:44:54.240 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T10:44:54.240 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T10:44:54.304 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-07-26T10:44:54.329 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-07-26T10:44:54.329 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-07-26T10:44:54.329 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-07-26T10:44:54.329 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T10:44:54.330 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T10:44:54.331 INFO:teuthology.orchestra.run.smithi183.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:44:54.332 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T10:44:54.333 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T10:44:54.334 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T10:44:54.335 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T10:44:54.335 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T10:44:54.335 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T10:44:54.335 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T10:44:54.335 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T10:44:54.337 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T10:44:54.338 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T10:44:54.338 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T10:44:54.338 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T10:44:54.338 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T10:44:54.338 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T10:44:54.338 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T10:44:54.339 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T10:44:54.339 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T10:44:54.339 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T10:44:54.339 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T10:44:54.339 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T10:44:54.339 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T10:44:54.340 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T10:44:54.341 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T10:44:54.341 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T10:44:54.341 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T10:44:54.341 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T10:44:54.341 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T10:44:54.341 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T10:44:54.342 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T10:44:54.343 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T10:44:54.344 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T10:44:54.345 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T10:44:54.345 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T10:44:54.345 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T10:44:54.345 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T10:44:54.345 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T10:44:54.345 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T10:44:54.346 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T10:44:54.346 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T10:44:54.346 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T10:44:54.346 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T10:44:54.346 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T10:44:54.347 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T10:44:54.347 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T10:44:54.347 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T10:44:54.347 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T10:44:54.347 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T10:44:54.347 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T10:44:54.348 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T10:44:54.348 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T10:44:54.348 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T10:44:54.348 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T10:44:54.348 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T10:44:54.348 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T10:44:54.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T10:44:54.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T10:44:54.351 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T10:44:54.352 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:44:54.353 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-07-26T10:44:54.353 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-07-26T10:44:54.353 INFO:teuthology.orchestra.run.smithi183.stdout:Install 148 Packages 2024-07-26T10:44:54.353 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 12 Packages 2024-07-26T10:44:54.353 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:44:54.354 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 233 M 2024-07-26T10:44:54.355 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-07-26T10:44:54.907 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-07-26T10:44:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-07-26T10:44:55.193 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-07-26T10:44:55.703 INFO:teuthology.orchestra.run.smithi183.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-07-26T10:44:55.953 INFO:teuthology.orchestra.run.smithi183.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 3.4 MB/s | 872 kB 00:00 2024-07-26T10:44:56.037 INFO:teuthology.orchestra.run.smithi183.stdout:(3/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 7.4 MB/s | 5.5 MB 00:00 2024-07-26T10:44:56.062 INFO:teuthology.orchestra.run.smithi183.stdout:(4/160): ceph-immutable-object-cache-19.0.0-532 1.3 MB/s | 148 kB 00:00 2024-07-26T10:44:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 17 MB/s | 2.3 MB 00:00 2024-07-26T10:44:56.363 INFO:teuthology.orchestra.run.smithi183.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 5.1 MB/s | 1.5 MB 00:00 2024-07-26T10:44:56.429 INFO:teuthology.orchestra.run.smithi183.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 19 MB/s | 4.9 MB 00:00 2024-07-26T10:44:56.613 INFO:teuthology.orchestra.run.smithi183.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 20 MB 00:01 2024-07-26T10:44:56.663 INFO:teuthology.orchestra.run.smithi183.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 507 kB/s | 25 kB 00:00 2024-07-26T10:44:56.963 INFO:teuthology.orchestra.run.smithi183.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 19 MB/s | 10 MB 00:00 2024-07-26T10:44:57.028 INFO:teuthology.orchestra.run.smithi183.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.4 MB/s | 225 kB 00:00 2024-07-26T10:44:57.089 INFO:teuthology.orchestra.run.smithi183.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 448 kB/s | 27 kB 00:00 2024-07-26T10:44:57.148 INFO:teuthology.orchestra.run.smithi183.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 545 kB/s | 32 kB 00:00 2024-07-26T10:44:57.239 INFO:teuthology.orchestra.run.smithi183.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 8.2 MB/s | 765 kB 00:00 2024-07-26T10:44:57.298 INFO:teuthology.orchestra.run.smithi183.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.6 MB/s | 158 kB 00:00 2024-07-26T10:44:57.365 INFO:teuthology.orchestra.run.smithi183.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.9 MB/s | 127 kB 00:00 2024-07-26T10:44:57.440 INFO:teuthology.orchestra.run.smithi183.stdout:(17/160): libradosstriper1-19.0.0-5320.ge8f88bb 6.6 MB/s | 503 kB 00:00 2024-07-26T10:44:57.774 INFO:teuthology.orchestra.run.smithi183.stdout:(18/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 16 MB/s | 5.3 MB 00:00 2024-07-26T10:44:57.830 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-07-26T10:44:57.830 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-07-26T10:44:57.832 INFO:teuthology.orchestra.run.smithi183.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 777 kB/s | 45 kB 00:00 2024-07-26T10:44:57.900 INFO:teuthology.orchestra.run.smithi183.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.0 MB/s | 138 kB 00:00 2024-07-26T10:44:57.933 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-07-26T10:44:57.958 INFO:teuthology.orchestra.run.smithi183.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T10:44:58.025 INFO:teuthology.orchestra.run.smithi183.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 5.5 MB/s | 372 kB 00:00 2024-07-26T10:44:58.153 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T10:44:58.197 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T10:44:58.217 INFO:teuthology.orchestra.run.smithi183.stdout:(23/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 9.6 MB/s | 18 MB 00:01 2024-07-26T10:44:58.242 INFO:teuthology.orchestra.run.smithi183.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 1.6 MB/s | 355 kB 00:00 2024-07-26T10:44:58.251 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T10:44:58.267 INFO:teuthology.orchestra.run.smithi183.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T10:44:58.286 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T10:44:58.293 INFO:teuthology.orchestra.run.smithi183.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T10:44:58.312 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T10:44:58.343 INFO:teuthology.orchestra.run.smithi183.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 3.3 MB/s | 170 kB 00:00 2024-07-26T10:44:58.354 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T10:44:58.704 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T10:44:58.727 INFO:teuthology.orchestra.run.smithi183.stdout:(28/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 21 MB/s | 44 MB 00:02 2024-07-26T10:44:58.752 INFO:teuthology.orchestra.run.smithi183.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 84 kB/s | 34 kB 00:00 2024-07-26T10:44:58.778 INFO:teuthology.orchestra.run.smithi183.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.9 MB/s | 149 kB 00:00 2024-07-26T10:44:58.812 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:44:58.836 INFO:teuthology.orchestra.run.smithi183.stdout:(31/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 5.4 MB/s | 3.1 MB 00:00 2024-07-26T10:44:58.921 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:44:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:44:58.953 INFO:teuthology.orchestra.run.smithi183.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 2.4 MB/s | 284 kB 00:00 2024-07-26T10:44:59.012 INFO:teuthology.orchestra.run.smithi183.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 850 kB/s | 49 kB 00:00 2024-07-26T10:44:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:44:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:44:59.070 INFO:teuthology.orchestra.run.smithi183.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 283 kB/s | 16 kB 00:00 2024-07-26T10:44:59.084 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T10:44:59.162 INFO:teuthology.orchestra.run.smithi183.stdout:(35/160): ceph-mgr-diskprediction-local-19.0.0- 19 MB/s | 7.4 MB 00:00 2024-07-26T10:44:59.187 INFO:teuthology.orchestra.run.smithi183.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 274 kB 00:00 2024-07-26T10:44:59.229 INFO:teuthology.orchestra.run.smithi183.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 11 MB/s | 771 kB 00:00 2024-07-26T10:44:59.254 INFO:teuthology.orchestra.run.smithi183.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 273 kB/s | 18 kB 00:00 2024-07-26T10:44:59.270 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T10:44:59.324 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T10:44:59.352 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:44:59.430 INFO:teuthology.orchestra.run.smithi183.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 21 MB/s | 14 MB 00:00 2024-07-26T10:44:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:44:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:44:59.540 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:44:59.562 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:44:59.582 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:44:59.736 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:44:59.828 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:44:59.982 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T10:45:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:45:00.172 INFO:teuthology.orchestra.run.smithi183.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 85 kB/s | 80 kB 00:00 2024-07-26T10:45:00.188 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:45:00.285 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:00.364 INFO:teuthology.orchestra.run.smithi183.stdout:(41/160): libgfortran-11.4.1-3.el9.x86_64.rpm 724 kB/s | 803 kB 00:01 2024-07-26T10:45:00.373 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:00.405 INFO:teuthology.orchestra.run.smithi183.stdout:(42/160): libquadmath-11.4.1-3.el9.x86_64.rpm 200 kB/s | 195 kB 00:00 2024-07-26T10:45:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T10:45:00.735 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T10:45:00.778 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T10:45:00.851 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T10:45:00.896 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T10:45:00.928 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T10:45:00.972 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T10:45:01.005 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T10:45:01.032 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T10:45:01.039 INFO:teuthology.orchestra.run.smithi183.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 292 kB/s | 253 kB 00:00 2024-07-26T10:45:01.101 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T10:45:01.162 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T10:45:01.189 INFO:teuthology.orchestra.run.smithi183.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 136 kB/s | 106 kB 00:00 2024-07-26T10:45:01.194 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T10:45:01.238 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T10:45:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T10:45:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T10:45:01.393 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T10:45:01.420 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T10:45:01.449 INFO:teuthology.orchestra.run.smithi183.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 1.1 MB/s | 1.2 MB 00:01 2024-07-26T10:45:01.489 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T10:45:01.535 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T10:45:01.609 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T10:45:01.749 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T10:45:01.818 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T10:45:01.840 INFO:teuthology.orchestra.run.smithi183.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 169 kB/s | 135 kB 00:00 2024-07-26T10:45:01.998 INFO:teuthology.orchestra.run.smithi183.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 154 kB/s | 125 kB 00:00 2024-07-26T10:45:02.090 INFO:teuthology.orchestra.run.smithi183.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 689 kB 00:00 2024-07-26T10:45:02.257 INFO:teuthology.orchestra.run.smithi183.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 641 kB/s | 107 kB 00:00 2024-07-26T10:45:02.299 INFO:teuthology.orchestra.run.smithi183.stdout:(50/160): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-07-26T10:45:02.324 INFO:teuthology.orchestra.run.smithi183.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 324 kB/s | 22 kB 00:00 2024-07-26T10:45:02.342 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T10:45:02.375 INFO:teuthology.orchestra.run.smithi183.stdout:(52/160): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-07-26T10:45:02.400 INFO:teuthology.orchestra.run.smithi183.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-07-26T10:45:02.431 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T10:45:02.442 INFO:teuthology.orchestra.run.smithi183.stdout:(54/160): python3-urllib3-1.26.5-6.el9.noarch.r 222 kB/s | 215 kB 00:00 2024-07-26T10:45:02.461 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T10:45:02.467 INFO:teuthology.orchestra.run.smithi183.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 272 kB/s | 18 kB 00:00 2024-07-26T10:45:02.497 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T10:45:02.518 INFO:teuthology.orchestra.run.smithi183.stdout:(56/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-07-26T10:45:02.538 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T10:45:02.542 INFO:teuthology.orchestra.run.smithi183.stdout:(57/160): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-07-26T10:45:02.568 INFO:teuthology.orchestra.run.smithi183.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-07-26T10:45:02.592 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T10:45:02.634 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T10:45:02.718 INFO:teuthology.orchestra.run.smithi183.stdout:(59/160): libtool-2.4.6-46.el9.x86_64.rpm 3.8 MB/s | 581 kB 00:00 2024-07-26T10:45:02.769 INFO:teuthology.orchestra.run.smithi183.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 507 kB/s | 25 kB 00:00 2024-07-26T10:45:02.794 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T10:45:02.803 INFO:teuthology.orchestra.run.smithi183.stdout:(61/160): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 662 kB 00:00 2024-07-26T10:45:02.828 INFO:teuthology.orchestra.run.smithi183.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 812 kB/s | 47 kB 00:00 2024-07-26T10:45:02.854 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T10:45:02.886 INFO:teuthology.orchestra.run.smithi183.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-07-26T10:45:02.912 INFO:teuthology.orchestra.run.smithi183.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-07-26T10:45:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T10:45:02.937 INFO:teuthology.orchestra.run.smithi183.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-07-26T10:45:03.001 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T10:45:03.046 INFO:teuthology.orchestra.run.smithi183.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.5 MB/s | 3.0 MB 00:00 2024-07-26T10:45:03.073 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T10:45:03.105 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T10:45:03.217 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T10:45:03.484 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T10:45:03.550 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T10:45:03.571 INFO:teuthology.orchestra.run.smithi183.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 3.0 MB/s | 1.6 MB 00:00 2024-07-26T10:45:03.594 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T10:45:03.644 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T10:45:03.721 INFO:teuthology.orchestra.run.smithi183.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 5.9 MB/s | 4.7 MB 00:00 2024-07-26T10:45:03.771 INFO:teuthology.orchestra.run.smithi183.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-07-26T10:45:03.937 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T10:45:03.974 INFO:teuthology.orchestra.run.smithi183.stdout:(70/160): postgresql-contrib-13.14-1.el9.x86_64 2.2 MB/s | 911 kB 00:00 2024-07-26T10:45:04.011 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T10:45:04.053 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T10:45:04.150 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T10:45:04.186 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T10:45:04.226 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T10:45:04.230 INFO:teuthology.orchestra.run.smithi183.stdout:(71/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-07-26T10:45:04.251 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T10:45:04.276 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T10:45:04.280 INFO:teuthology.orchestra.run.smithi183.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-07-26T10:45:04.342 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T10:45:04.447 INFO:teuthology.orchestra.run.smithi183.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 2.1 MB/s | 1.0 MB 00:00 2024-07-26T10:45:04.468 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T10:45:04.497 INFO:teuthology.orchestra.run.smithi183.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-07-26T10:45:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T10:45:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T10:45:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T10:45:04.599 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-07-26T10:45:04.609 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T10:45:04.640 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T10:45:04.656 INFO:teuthology.orchestra.run.smithi183.stdout:(75/160): postgresql-server-13.14-1.el9.x86_64. 6.5 MB/s | 5.8 MB 00:00 2024-07-26T10:45:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T10:45:04.681 INFO:teuthology.orchestra.run.smithi183.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.3 MB/s | 244 kB 00:00 2024-07-26T10:45:04.683 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T10:45:04.705 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T10:45:04.732 INFO:teuthology.orchestra.run.smithi183.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-07-26T10:45:04.738 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T10:45:04.756 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T10:45:04.757 INFO:teuthology.orchestra.run.smithi183.stdout:(78/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-07-26T10:45:04.780 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T10:45:04.782 INFO:teuthology.orchestra.run.smithi183.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-07-26T10:45:04.808 INFO:teuthology.orchestra.run.smithi183.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-07-26T10:45:04.814 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T10:45:04.833 INFO:teuthology.orchestra.run.smithi183.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-07-26T10:45:04.859 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T10:45:04.901 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T10:45:04.910 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T10:45:04.944 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T10:45:04.952 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T10:45:04.966 INFO:teuthology.orchestra.run.smithi183.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.8 MB/s | 250 kB 00:00 2024-07-26T10:45:04.984 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T10:45:05.046 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T10:45:05.078 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T10:45:05.083 INFO:teuthology.orchestra.run.smithi183.stdout:(83/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-07-26T10:45:05.126 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:45:05.142 INFO:teuthology.orchestra.run.smithi183.stdout:(84/160): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-07-26T10:45:05.192 INFO:teuthology.orchestra.run.smithi183.stdout:(85/160): python3-prettytable-0.7.2-27.el9.noar 846 kB/s | 42 kB 00:00 2024-07-26T10:45:05.326 INFO:teuthology.orchestra.run.smithi183.stdout:(86/160): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-07-26T10:45:05.437 INFO:teuthology.orchestra.run.smithi183.stdout:(87/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-07-26T10:45:05.461 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T10:45:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:(88/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.3 MB/s | 5.1 MB 00:00 2024-07-26T10:45:05.554 INFO:teuthology.orchestra.run.smithi183.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-07-26T10:45:05.570 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:45:05.605 INFO:teuthology.orchestra.run.smithi183.stdout:(90/160): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-07-26T10:45:05.630 INFO:teuthology.orchestra.run.smithi183.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-07-26T10:45:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:(92/160): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-07-26T10:45:05.730 INFO:teuthology.orchestra.run.smithi183.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-07-26T10:45:05.755 INFO:teuthology.orchestra.run.smithi183.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-07-26T10:45:05.756 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:45:05.806 INFO:teuthology.orchestra.run.smithi183.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-07-26T10:45:05.812 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:45:05.846 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T10:45:05.856 INFO:teuthology.orchestra.run.smithi183.stdout:(96/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-07-26T10:45:05.873 INFO:teuthology.orchestra.run.smithi183.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-07-26T10:45:05.890 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T10:45:05.915 INFO:teuthology.orchestra.run.smithi183.stdout:(98/160): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-07-26T10:45:05.923 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T10:45:05.947 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:45:05.973 INFO:teuthology.orchestra.run.smithi183.stdout:(99/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rpm 1.6 MB/s | 96 kB 00:00 2024-07-26T10:45:05.986 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:45:05.999 INFO:teuthology.orchestra.run.smithi183.stdout:(100/160): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-07-26T10:45:06.021 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T10:45:06.024 INFO:teuthology.orchestra.run.smithi183.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-07-26T10:45:06.049 INFO:teuthology.orchestra.run.smithi183.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 896 kB/s | 45 kB 00:00 2024-07-26T10:45:06.059 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T10:45:06.087 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T10:45:06.108 INFO:teuthology.orchestra.run.smithi183.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-07-26T10:45:06.132 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T10:45:06.191 INFO:teuthology.orchestra.run.smithi183.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 6.4 MB/s | 548 kB 00:00 2024-07-26T10:45:06.219 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T10:45:06.225 INFO:teuthology.orchestra.run.smithi183.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.2 MB/s | 309 kB 00:00 2024-07-26T10:45:06.259 INFO:teuthology.orchestra.run.smithi183.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 584 kB/s | 19 kB 00:00 2024-07-26T10:45:06.262 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T10:45:06.292 INFO:teuthology.orchestra.run.smithi183.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-07-26T10:45:06.358 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T10:45:06.359 INFO:teuthology.orchestra.run.smithi183.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-07-26T10:45:06.384 INFO:teuthology.orchestra.run.smithi183.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-26T10:45:06.410 INFO:teuthology.orchestra.run.smithi183.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-26T10:45:06.415 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T10:45:06.435 INFO:teuthology.orchestra.run.smithi183.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-07-26T10:45:06.438 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T10:45:06.442 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:45:06.466 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T10:45:06.485 INFO:teuthology.orchestra.run.smithi183.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-07-26T10:45:06.544 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T10:45:06.544 INFO:teuthology.orchestra.run.smithi183.stdout:(113/160): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-07-26T10:45:06.569 INFO:teuthology.orchestra.run.smithi183.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T10:45:06.594 INFO:teuthology.orchestra.run.smithi183.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T10:45:06.596 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:45:06.620 INFO:teuthology.orchestra.run.smithi183.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 565 kB/s | 14 kB 00:00 2024-07-26T10:45:06.626 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:45:06.645 INFO:teuthology.orchestra.run.smithi183.stdout:(117/160): python3-asyncssh-2.13.2-5.el9.noarch 3.3 MB/s | 548 kB 00:00 2024-07-26T10:45:06.671 INFO:teuthology.orchestra.run.smithi183.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-26T10:45:06.696 INFO:teuthology.orchestra.run.smithi183.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-07-26T10:45:06.711 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:45:06.721 INFO:teuthology.orchestra.run.smithi183.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-07-26T10:45:06.746 INFO:teuthology.orchestra.run.smithi183.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-26T10:45:06.752 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:45:06.780 INFO:teuthology.orchestra.run.smithi183.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-07-26T10:45:06.805 INFO:teuthology.orchestra.run.smithi183.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-07-26T10:45:06.829 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:45:06.830 INFO:teuthology.orchestra.run.smithi183.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 705 kB/s | 18 kB 00:00 2024-07-26T10:45:06.856 INFO:teuthology.orchestra.run.smithi183.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 930 kB/s | 23 kB 00:00 2024-07-26T10:45:06.881 INFO:teuthology.orchestra.run.smithi183.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 778 kB/s | 19 kB 00:00 2024-07-26T10:45:06.906 INFO:teuthology.orchestra.run.smithi183.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 784 kB/s | 20 kB 00:00 2024-07-26T10:45:06.956 INFO:teuthology.orchestra.run.smithi183.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-07-26T10:45:06.982 INFO:teuthology.orchestra.run.smithi183.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T10:45:07.015 INFO:teuthology.orchestra.run.smithi183.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-07-26T10:45:07.042 INFO:teuthology.orchestra.run.smithi183.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T10:45:07.074 INFO:teuthology.orchestra.run.smithi183.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-07-26T10:45:07.100 INFO:teuthology.orchestra.run.smithi183.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-07-26T10:45:07.133 INFO:teuthology.orchestra.run.smithi183.stdout:(134/160): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-07-26T10:45:07.139 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:45:07.159 INFO:teuthology.orchestra.run.smithi183.stdout:(135/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-07-26T10:45:07.184 INFO:teuthology.orchestra.run.smithi183.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-07-26T10:45:07.189 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T10:45:07.209 INFO:teuthology.orchestra.run.smithi183.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-07-26T10:45:07.234 INFO:teuthology.orchestra.run.smithi183.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-26T10:45:07.260 INFO:teuthology.orchestra.run.smithi183.stdout:(139/160): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-07-26T10:45:07.285 INFO:teuthology.orchestra.run.smithi183.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-07-26T10:45:07.310 INFO:teuthology.orchestra.run.smithi183.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 2.0 MB/s | 51 kB 00:00 2024-07-26T10:45:07.315 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T10:45:07.328 INFO:teuthology.orchestra.run.smithi183.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 5.3 MB/s | 90 kB 00:00 2024-07-26T10:45:07.331 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:45:07.352 INFO:teuthology.orchestra.run.smithi183.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-07-26T10:45:07.369 INFO:teuthology.orchestra.run.smithi183.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-07-26T10:45:07.411 INFO:teuthology.orchestra.run.smithi183.stdout:(145/160): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-07-26T10:45:07.445 INFO:teuthology.orchestra.run.smithi183.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-07-26T10:45:07.604 INFO:teuthology.orchestra.run.smithi183.stdout:(147/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.1 MB/s | 16 MB 00:01 2024-07-26T10:45:07.646 INFO:teuthology.orchestra.run.smithi183.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-07-26T10:45:07.676 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T10:45:07.821 INFO:teuthology.orchestra.run.smithi183.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 9.0 MB/s | 3.4 MB 00:00 2024-07-26T10:45:07.860 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:45:07.871 INFO:teuthology.orchestra.run.smithi183.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 12 MB/s | 3.1 MB 00:00 2024-07-26T10:45:07.940 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:45:08.103 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:08.205 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:08.495 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:08.521 INFO:teuthology.orchestra.run.smithi183.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 63 kB/s | 41 kB 00:00 2024-07-26T10:45:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T10:45:08.644 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T10:45:08.656 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T10:45:08.671 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T10:45:08.686 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T10:45:08.704 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T10:45:08.736 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T10:45:08.813 INFO:teuthology.orchestra.run.smithi183.stdout:(152/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.0 MB/s | 2.3 MB 00:01 2024-07-26T10:45:08.840 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T10:45:08.904 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T10:45:08.959 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T10:45:08.963 INFO:teuthology.orchestra.run.smithi183.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.6 MB/s | 1.8 MB 00:01 2024-07-26T10:45:09.022 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T10:45:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T10:45:09.071 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T10:45:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:45:09.111 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T10:45:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:45:09.206 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T10:45:09.228 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T10:45:09.241 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T10:45:09.306 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T10:45:09.345 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T10:45:09.408 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T10:45:09.521 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T10:45:09.522 INFO:teuthology.orchestra.run.smithi183.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 247 kB/s | 247 kB 00:00 2024-07-26T10:45:09.547 INFO:teuthology.orchestra.run.smithi183.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 52 kB/s | 37 kB 00:00 2024-07-26T10:45:09.557 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T10:45:09.620 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T10:45:09.656 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T10:45:09.740 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T10:45:09.770 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:45:09.774 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T10:45:09.833 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T10:45:09.839 INFO:teuthology.orchestra.run.smithi183.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 237 kB/s | 207 kB 00:00 2024-07-26T10:45:09.843 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:45:09.872 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T10:45:09.933 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T10:45:10.117 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T10:45:10.180 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T10:45:10.406 INFO:teuthology.orchestra.run.smithi183.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 294 kB/s | 260 kB 00:00 2024-07-26T10:45:10.615 INFO:teuthology.orchestra.run.smithi183.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 1.3 MB/s | 1.4 MB 00:01 2024-07-26T10:45:11.056 INFO:teuthology.orchestra.run.smithi183.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-07-26T10:45:11.112 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T10:45:11.209 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T10:45:11.262 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T10:45:11.300 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T10:45:11.375 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T10:45:11.421 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T10:45:11.489 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T10:45:11.490 INFO:teuthology.orchestra.run.smithi183.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 1.0 MB/s | 1.1 MB 00:01 2024-07-26T10:45:11.497 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:45:11.497 INFO:teuthology.orchestra.run.smithi183.stdout:Total 14 MB/s | 233 MB 00:17 2024-07-26T10:45:11.696 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T10:45:11.757 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T10:45:11.878 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T10:45:11.950 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T10:45:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T10:45:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T10:45:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T10:45:12.146 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:12.181 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:12.203 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T10:45:12.263 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T10:45:12.326 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T10:45:12.386 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T10:45:12.590 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T10:45:12.649 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T10:45:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T10:45:12.735 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-07-26T10:45:12.746 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:12.779 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:45:12.838 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T10:45:12.875 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:45:12.875 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T10:45:12.875 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T10:45:12.875 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:12.881 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T10:45:12.905 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:45:12.929 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-07-26T10:45:12.929 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-07-26T10:45:12.935 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T10:45:12.972 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T10:45:13.030 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T10:45:13.214 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T10:45:13.411 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T10:45:13.461 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T10:45:13.498 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T10:45:13.542 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T10:45:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T10:45:13.664 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T10:45:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T10:45:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T10:45:13.802 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T10:45:13.876 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T10:45:13.925 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T10:45:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T10:45:14.137 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T10:45:14.182 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T10:45:14.337 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T10:45:14.403 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:45:14.787 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-07-26T10:45:14.788 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-07-26T10:45:15.367 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T10:45:15.429 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T10:45:15.469 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T10:45:15.596 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T10:45:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T10:45:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T10:45:15.834 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T10:45:15.880 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T10:45:16.109 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T10:45:16.143 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T10:45:16.243 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T10:45:17.064 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T10:45:17.225 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T10:45:19.075 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T10:45:19.198 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T10:45:19.234 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T10:45:19.270 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T10:45:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T10:45:19.703 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:45:19.829 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:45:19.878 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T10:45:19.937 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T10:45:20.499 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T10:45:20.539 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:45:20.633 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:45:20.666 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T10:45:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-07-26T10:45:21.402 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T10:45:21.440 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T10:45:21.493 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T10:45:21.527 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T10:45:21.553 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T10:45:21.596 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T10:45:21.946 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T10:45:22.054 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:45:22.180 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:45:22.213 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:45:22.310 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:45:22.342 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:45:22.375 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T10:45:22.572 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T10:45:22.615 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T10:45:22.643 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:45:22.749 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:45:22.778 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:45:22.840 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:45:22.862 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:45:22.883 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:45:23.086 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:45:23.182 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:45:23.294 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:23.363 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:23.365 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T10:45:23.519 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:45:23.563 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:45:23.652 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:23.731 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:23.919 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:45:24.027 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T10:45:24.036 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:24.059 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T10:45:24.083 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:45:24.094 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T10:45:24.168 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T10:45:24.212 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T10:45:24.253 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T10:45:24.262 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:45:24.262 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T10:45:24.262 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T10:45:24.262 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:24.296 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T10:45:24.300 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:45:24.329 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T10:45:24.357 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T10:45:24.426 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T10:45:24.479 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T10:45:24.510 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2024-07-26T10:45:24.540 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:24.545 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T10:45:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T10:45:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T10:45:24.709 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:45:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T10:45:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:45:24.724 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:24.724 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T10:45:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:24.744 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T10:45:24.814 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T10:45:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T10:45:24.934 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T10:45:25.069 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T10:45:25.120 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T10:45:25.651 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T10:45:25.732 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T10:45:25.761 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T10:45:25.796 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T10:45:25.846 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T10:45:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T10:45:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T10:45:26.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T10:45:26.145 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T10:45:26.236 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T10:45:26.302 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T10:45:26.330 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:45:26.356 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:45:26.356 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:26.356 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T10:45:26.356 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:45:26.356 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:45:26.357 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:26.374 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T10:45:26.413 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T10:45:26.449 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T10:45:26.496 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T10:45:26.532 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T10:45:26.575 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T10:45:26.580 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:45:26.603 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:45:26.603 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:26.603 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T10:45:26.603 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:45:26.604 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:45:26.604 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:26.629 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T10:45:26.842 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T10:45:26.878 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T10:45:26.919 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T10:45:27.026 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T10:45:27.060 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T10:45:27.064 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:45:27.087 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:45:27.087 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:27.087 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T10:45:27.087 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:45:27.087 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:45:27.087 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:27.100 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T10:45:27.126 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T10:45:27.151 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T10:45:27.194 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T10:45:27.251 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T10:45:27.299 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T10:45:27.315 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T10:45:27.501 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T10:45:27.548 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T10:45:27.626 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T10:45:27.635 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T10:45:27.683 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T10:45:27.710 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T10:45:27.715 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T10:45:27.745 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T10:45:27.784 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T10:45:27.815 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T10:45:27.845 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T10:45:27.855 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T10:45:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T10:45:27.898 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T10:45:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T10:45:27.961 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T10:45:27.961 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T10:45:27.961 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T10:45:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T10:45:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T10:45:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T10:45:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T10:45:28.267 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T10:45:28.326 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:45:28.419 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:45:28.448 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:45:28.480 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T10:45:28.546 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T10:45:28.571 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:45:29.044 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T10:45:29.098 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T10:45:29.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T10:45:29.231 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T10:45:29.287 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T10:45:29.331 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T10:45:29.420 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T10:45:29.451 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T10:45:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T10:45:29.648 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T10:45:29.726 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T10:45:30.365 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T10:45:30.479 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T10:45:31.687 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T10:45:31.721 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:45:31.785 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:45:31.793 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T10:45:31.820 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T10:45:31.854 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T10:45:32.181 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T10:45:32.216 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:45:32.310 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:45:32.338 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T10:45:32.374 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T10:45:32.752 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:45:32.786 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:45:32.874 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T10:45:32.898 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:45:32.911 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:45:32.983 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:45:33.012 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T10:45:33.085 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T10:45:33.121 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:45:33.144 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:45:33.144 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:33.144 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T10:45:33.145 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:45:33.145 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:45:33.145 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:33.197 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:45:33.208 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:45:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T10:45:33.361 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T10:45:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T10:45:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:45:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:45:34.372 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:34.372 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T10:45:34.372 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:45:34.372 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:45:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:45:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:45:34.439 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:34.439 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T10:45:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:45:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:45:34.506 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:34.506 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T10:45:34.506 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:45:34.507 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:45:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:34.826 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:45:34.848 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:45:34.848 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:34.848 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T10:45:34.849 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:45:34.849 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:45:34.849 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:35.286 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:35.320 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:35.888 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:45:35.931 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:45:36.005 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:45:36.005 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T10:45:36.006 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T10:45:36.006 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:36.024 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:45:36.975 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:45:36.975 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2024-07-26T10:45:36.975 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2024-07-26T10:45:36.976 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2024-07-26T10:45:36.976 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2024-07-26T10:45:36.976 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2024-07-26T10:45:36.976 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /root 2024-07-26T10:45:36.976 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2024-07-26T10:45:36.976 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:37.194 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:45:37.242 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:45:37.243 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:37.243 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T10:45:37.243 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:38.945 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:45:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:45:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T10:45:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:45:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:45:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:39.230 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T10:45:39.270 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:45:39.271 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T10:45:39.311 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T10:45:39.341 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:45:39.341 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:39.341 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T10:45:39.341 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:45:39.342 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:45:39.342 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T10:45:39.371 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T10:45:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:45:39.607 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T10:45:39.648 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T10:45:39.708 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T10:45:39.748 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:45:39.794 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T10:45:39.846 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:45:39.914 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:45:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T10:45:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:45:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:45:39.915 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:40.038 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T10:45:40.098 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T10:45:40.175 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T10:45:40.410 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T10:45:40.444 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:45:40.467 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T10:45:40.502 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:45:40.522 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:45:40.553 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T10:45:40.669 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T10:45:40.844 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T10:45:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T10:45:40.948 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T10:45:40.974 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T10:45:41.367 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T10:45:41.466 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T10:45:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T10:45:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T10:45:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:45:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T10:45:41.662 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T10:45:41.668 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:45:41.687 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:45:41.714 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T10:45:41.714 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:45:41.723 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T10:45:41.782 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:45:41.814 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T10:45:41.814 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:45:41.835 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T10:45:41.879 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:45:41.890 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:45:41.916 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T10:45:41.920 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:45:41.920 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:45:42.109 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:45:42.109 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:45:42.185 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:45:42.210 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:45:42.235 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T10:45:42.300 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:45:45.559 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:45:45.654 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:45:46.751 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:45:46.777 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:45:46.934 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:45:47.113 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T10:45:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:45:47.220 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:45:47.220 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:47.220 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T10:45:47.220 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:45:47.220 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:45:47.220 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:47.280 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:45:47.318 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:45:47.474 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T10:45:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-07-26T10:45:47.647 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:45:47.838 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:45:47.839 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:47.839 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T10:45:47.839 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T10:45:48.475 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T10:45:48.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T10:45:48.477 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T10:45:48.490 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T10:45:48.490 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T10:45:48.491 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T10:45:48.492 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T10:45:48.493 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T10:45:48.494 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T10:45:48.495 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T10:45:48.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T10:45:48.497 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T10:45:48.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T10:45:48.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T10:45:48.500 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T10:45:48.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T10:45:48.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:45:48.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T10:45:48.625 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T10:45:48.657 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:45:48.728 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:45:48.728 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:48.729 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T10:45:48.729 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:45:48.729 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:45:48.729 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:45:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:45:48.846 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:48.846 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T10:45:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:48.909 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:45:49.008 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:45:49.008 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:49.008 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T10:45:49.008 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:45:49.008 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:45:49.009 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:49.409 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:45:49.434 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:45:49.460 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:45:49.460 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:49.460 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T10:45:49.460 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:45:49.460 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:45:49.460 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:49.503 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:45:49.503 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:49.503 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T10:45:49.504 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:45:49.504 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:45:49.504 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:45:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:45:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:45:49.707 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:49.708 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T10:45:49.708 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:45:49.708 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:45:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:50.168 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:45:50.191 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:45:50.191 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:50.191 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T10:45:50.191 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:45:50.191 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:45:50.191 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:50.291 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T10:45:50.372 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T10:45:50.445 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T10:45:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T10:45:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T10:45:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T10:45:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T10:45:50.992 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T10:45:51.027 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T10:45:51.076 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T10:45:51.112 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T10:45:51.363 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T10:45:51.364 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.365 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T10:45:51.366 INFO:teuthology.orchestra.run.smithi160.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T10:45:51.367 INFO:teuthology.orchestra.run.smithi160.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.368 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T10:45:51.369 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T10:45:51.370 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T10:45:51.371 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T10:45:51.372 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T10:45:51.373 INFO:teuthology.orchestra.run.smithi160.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.374 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T10:45:51.375 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T10:45:51.376 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-07-26T10:45:51.377 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-07-26T10:45:51.557 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:45:51.586 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:45:51.618 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T10:45:51.682 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T10:45:51.709 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:45:51.792 DEBUG:teuthology.parallel:result is None 2024-07-26T10:45:54.265 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T10:45:54.317 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T10:45:54.371 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T10:45:54.405 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T10:45:54.442 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T10:45:54.734 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:45:54.785 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T10:45:54.829 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T10:45:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:45:54.896 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T10:45:54.923 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:45:54.951 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:45:55.010 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T10:45:55.885 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:45:55.940 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:45:55.958 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:45:56.006 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:45:56.053 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:45:56.059 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:45:56.223 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T10:45:56.251 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:45:56.274 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:45:56.274 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:56.274 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T10:45:56.274 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:45:56.275 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:45:56.275 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:56.318 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:45:56.329 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:45:56.428 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T10:45:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T10:45:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T10:45:57.437 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T10:45:57.457 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T10:45:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:45:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:45:57.494 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:57.494 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T10:45:57.494 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:45:57.494 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:45:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:57.514 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T10:45:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:45:57.568 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:45:57.569 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:57.569 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T10:45:57.569 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:57.614 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:45:57.636 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:45:57.636 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:57.636 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T10:45:57.636 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:45:57.636 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:45:57.637 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:57.644 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T10:45:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T10:45:57.717 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T10:45:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:45:57.792 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T10:45:57.792 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:45:57.964 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:45:57.966 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:45:57.987 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:45:57.987 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:45:57.987 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T10:45:57.987 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:45:57.987 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:45:57.987 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:45:58.022 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T10:45:58.022 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:45:58.158 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:45:58.205 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T10:45:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:45:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:45:58.585 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:45:58.585 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:45:58.738 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:45:58.783 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:45:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T10:45:58.967 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:45:59.211 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:45:59.211 INFO:teuthology.orchestra.run.smithi144.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T10:45:59.211 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:46:02.360 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T10:46:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T10:46:02.442 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T10:46:02.474 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T10:46:02.501 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T10:46:02.694 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:46:02.743 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T10:46:02.776 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T10:46:02.841 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T10:46:02.871 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:46:02.926 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T10:46:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:46:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:46:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:46:04.376 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T10:46:04.460 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T10:46:04.501 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T10:46:04.584 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T10:46:04.614 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T10:46:04.639 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T10:46:04.665 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:46:04.699 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T10:46:04.700 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:46:04.792 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:46:04.825 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T10:46:04.826 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:46:04.910 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:46:04.935 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T10:46:04.940 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:46:04.940 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:46:05.120 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:46:05.120 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:46:05.212 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:46:05.246 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:46:05.282 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T10:46:05.353 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:46:05.898 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T10:46:05.898 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:46:05.899 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T10:46:05.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T10:46:05.912 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T10:46:05.913 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T10:46:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T10:46:05.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T10:46:05.916 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T10:46:05.917 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T10:46:05.918 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T10:46:05.919 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T10:46:05.920 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T10:46:05.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:46:05.922 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T10:46:08.228 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:46:08.228 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:46:08.228 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2024-07-26T10:46:08.228 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:08.228 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T10:46:08.229 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.230 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.231 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.232 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T10:46:08.233 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.234 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T10:46:08.235 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T10:46:08.236 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T10:46:08.237 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T10:46:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T10:46:08.239 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T10:46:08.240 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T10:46:08.241 INFO:teuthology.orchestra.run.smithi144.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.242 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T10:46:08.243 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T10:46:08.244 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T10:46:08.245 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T10:46:08.246 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T10:46:08.246 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:46:08.246 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-07-26T10:46:08.847 DEBUG:teuthology.parallel:result is None 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T10:46:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T10:46:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T10:46:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T10:46:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T10:46:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T10:46:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T10:46:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T10:46:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T10:46:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T10:46:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T10:46:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T10:46:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T10:46:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T10:46:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T10:46:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T10:46:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T10:46:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:46:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T10:46:14.719 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.720 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.721 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T10:46:14.722 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T10:46:14.723 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T10:46:14.724 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T10:46:14.725 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T10:46:14.726 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T10:46:14.727 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T10:46:14.728 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T10:46:14.729 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T10:46:14.730 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T10:46:14.731 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T10:46:14.732 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T10:46:14.732 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:46:14.732 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-07-26T10:46:15.150 DEBUG:teuthology.parallel:result is None 2024-07-26T10:46:15.151 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:15.300 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T10:46:15.372 INFO:teuthology.orchestra.run.smithi144.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:46:15.373 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:46:15.373 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T10:46:15.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:15.519 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T10:46:15.544 INFO:teuthology.orchestra.run.smithi160.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:46:15.544 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:46:15.544 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T10:46:15.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:15.689 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T10:46:15.711 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:46:15.712 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:46:15.712 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T10:46:15.714 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T10:46:15.714 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:46:15.714 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T10:46:15.807 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:46:15.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T10:46:15.838 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:46:15.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T10:46:15.866 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T10:46:15.866 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:46:15.867 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T10:46:15.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T10:46:16.040 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:46:16.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T10:46:16.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T10:46:16.139 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:46:16.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T10:46:16.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T10:46:16.232 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T10:46:16.232 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:46:16.232 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T10:46:16.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T10:46:16.448 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:46:16.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T10:46:16.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T10:46:16.536 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:46:16.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T10:46:16.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T10:46:16.627 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T10:46:16.628 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:46:16.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T10:46:16.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T10:46:16.802 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:46:16.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T10:46:16.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T10:46:16.889 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:46:16.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T10:46:16.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T10:46:16.981 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-26T10:46:17.079 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug cephsqlite': 20, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug'}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-07-26T10:46:17.079 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:17.080 INFO:tasks.cephadm:Cluster fsid is 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:46:17.080 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-26T10:46:17.080 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.144', 'mon.b': '172.21.15.160', 'mon.c': '172.21.15.183'} 2024-07-26T10:46:17.080 INFO:tasks.cephadm:First mon is mon.a on smithi144 2024-07-26T10:46:17.080 INFO:tasks.cephadm:First mgr is x 2024-07-26T10:46:17.080 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-26T10:46:17.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname $(hostname -s) 2024-07-26T10:46:17.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-07-26T10:46:17.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-07-26T10:46:17.214 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-26T10:46:17.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:17.362 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-26T10:46:17.504 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:17.506 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T10:46:17.506 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T10:46:17.506 DEBUG:teuthology.orchestra.run.smithi144:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:17.943 INFO:teuthology.orchestra.run.smithi144.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 10:46 /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:17.943 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.286 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 10:46 /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.287 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.651 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 10:46 /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.651 DEBUG:teuthology.orchestra.run.smithi144:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.710 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.729 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T10:46:18.760 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-26T10:46:18.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T10:46:18.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T10:46:18.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T10:46:19.605 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:46:19.717 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:46:20.108 INFO:teuthology.orchestra.run.smithi144.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:46:52.807 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-07-26T10:46:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:46:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T10:46:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-07-26T10:46:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T10:46:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-07-26T10:46:52.808 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-07-26T10:46:56.991 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-07-26T10:46:56.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:46:56.992 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T10:46:56.992 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-07-26T10:46:56.992 INFO:teuthology.orchestra.run.smithi160.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T10:46:56.992 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-07-26T10:46:56.992 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-07-26T10:47:06.030 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T10:47:06.030 INFO:teuthology.orchestra.run.smithi144.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:47:06.030 INFO:teuthology.orchestra.run.smithi144.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T10:47:06.030 INFO:teuthology.orchestra.run.smithi144.stdout: "repo_digests": [ 2024-07-26T10:47:06.030 INFO:teuthology.orchestra.run.smithi144.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T10:47:06.031 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-07-26T10:47:06.031 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T10:47:06.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph 2024-07-26T10:47:06.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-07-26T10:47:06.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-07-26T10:47:06.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /etc/ceph 2024-07-26T10:47:06.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-07-26T10:47:06.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-07-26T10:47:06.400 INFO:tasks.cephadm:Writing seed config... 2024-07-26T10:47:06.401 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-07-26T10:47:06.401 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-26T10:47:06.401 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-26T10:47:06.401 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-26T10:47:06.402 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-26T10:47:06.403 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-07-26T10:47:06.404 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:47:06.404 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-26T10:47:06.460 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 493f92c6-4b3c-11ef-bcb7-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-07-26T10:47:06.460 DEBUG:teuthology.orchestra.run.smithi144:mon.a> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a.service 2024-07-26T10:47:06.505 DEBUG:teuthology.orchestra.run.smithi144:mgr.x> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.x.service 2024-07-26T10:47:06.550 INFO:tasks.cephadm:Bootstrapping... 2024-07-26T10:47:06.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.144 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-26T10:47:06.832 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:47:06.833 INFO:teuthology.orchestra.run.smithi144.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.144', '--skip-admin-label'] 2024-07-26T10:47:06.833 INFO:teuthology.orchestra.run.smithi144.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-26T10:47:06.833 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying podman|docker is present... 2024-07-26T10:47:06.888 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stdout 5.1.2 2024-07-26T10:47:06.888 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying lvm2 is present... 2024-07-26T10:47:06.888 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying time synchronization is in place... 2024-07-26T10:47:06.906 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T10:47:06.906 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T10:47:06.921 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T10:47:06.922 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout inactive 2024-07-26T10:47:06.938 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout enabled 2024-07-26T10:47:06.954 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout active 2024-07-26T10:47:06.954 INFO:teuthology.orchestra.run.smithi144.stdout:Unit chronyd.service is enabled and running 2024-07-26T10:47:06.955 INFO:teuthology.orchestra.run.smithi144.stdout:Repeating the final host check... 2024-07-26T10:47:07.007 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stdout 5.1.2 2024-07-26T10:47:07.008 INFO:teuthology.orchestra.run.smithi144.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-26T10:47:07.008 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl is present 2024-07-26T10:47:07.008 INFO:teuthology.orchestra.run.smithi144.stdout:lvcreate is present 2024-07-26T10:47:07.025 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T10:47:07.025 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T10:47:07.042 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T10:47:07.042 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout inactive 2024-07-26T10:47:07.059 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout enabled 2024-07-26T10:47:07.074 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout active 2024-07-26T10:47:07.074 INFO:teuthology.orchestra.run.smithi144.stdout:Unit chronyd.service is enabled and running 2024-07-26T10:47:07.074 INFO:teuthology.orchestra.run.smithi144.stdout:Host looks OK 2024-07-26T10:47:07.074 INFO:teuthology.orchestra.run.smithi144.stdout:Cluster fsid: 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:07.075 INFO:teuthology.orchestra.run.smithi144.stdout:Acquiring lock 140138083730240 on /run/cephadm/493f92c6-4b3c-11ef-bcb7-c7b262605968.lock 2024-07-26T10:47:07.075 INFO:teuthology.orchestra.run.smithi144.stdout:Lock 140138083730240 acquired on /run/cephadm/493f92c6-4b3c-11ef-bcb7-c7b262605968.lock 2024-07-26T10:47:07.075 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying IP 172.21.15.144 port 3300 ... 2024-07-26T10:47:07.075 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying IP 172.21.15.144 port 6789 ... 2024-07-26T10:47:07.076 INFO:teuthology.orchestra.run.smithi144.stdout:Base mon IP(s) is [172.21.15.144:3300, 172.21.15.144:6789], mon addrv is [v2:172.21.15.144:3300,v1:172.21.15.144:6789] 2024-07-26T10:47:07.084 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.144 metric 100 2024-07-26T10:47:07.084 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.144 metric 100 2024-07-26T10:47:07.092 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-26T10:47:07.092 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-07-26T10:47:07.092 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-07-26T10:47:07.099 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-26T10:47:07.099 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-26T10:47:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T10:47:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-07-26T10:47:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2024-07-26T10:47:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T10:47:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:Mon IP `172.21.15.144` is in CIDR network `172.21.0.0/20` 2024-07-26T10:47:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:Mon IP `172.21.15.144` is in CIDR network `172.21.0.0/20` 2024-07-26T10:47:07.101 INFO:teuthology.orchestra.run.smithi144.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-26T10:47:07.101 INFO:teuthology.orchestra.run.smithi144.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-26T10:47:07.102 INFO:teuthology.orchestra.run.smithi144.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:47:07.744 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T10:47:07.744 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:47:07.744 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stderr Getting image source signatures 2024-07-26T10:47:07.744 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-26T10:47:07.745 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-26T10:47:07.745 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T10:47:07.745 INFO:teuthology.orchestra.run.smithi144.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-26T10:47:09.137 INFO:teuthology.orchestra.run.smithi144.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T10:47:09.137 INFO:teuthology.orchestra.run.smithi144.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T10:47:09.137 INFO:teuthology.orchestra.run.smithi144.stdout:Extracting ceph user uid/gid from container image... 2024-07-26T10:47:10.319 INFO:teuthology.orchestra.run.smithi144.stdout:stat: stdout 167 167 2024-07-26T10:47:10.319 INFO:teuthology.orchestra.run.smithi144.stdout:Creating initial keys... 2024-07-26T10:47:11.496 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph-authtool: stdout AQCufqNm9I74MhAA2FD+EOo3Rn+KQgHJ8yToRQ== 2024-07-26T10:47:12.589 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph-authtool: stdout AQCwfqNmwYgCAxAAEUX37NC3NECDN/elRnm4pQ== 2024-07-26T10:47:13.700 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph-authtool: stdout AQCxfqNmJTjeBRAACPGu3M0MctGVced3EajGxg== 2024-07-26T10:47:13.702 INFO:teuthology.orchestra.run.smithi144.stdout:Creating initial monmap... 2024-07-26T10:47:14.825 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T10:47:14.825 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool for a [v2:172.21.15.144:3300,v1:172.21.15.144:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:setting min_mon_release = quincy 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/monmaptool: set fsid to 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:47:14.826 INFO:teuthology.orchestra.run.smithi144.stdout:Creating mon... 2024-07-26T10:47:15.961 INFO:teuthology.orchestra.run.smithi144.stdout:create mon.a on 2024-07-26T10:47:16.296 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-26T10:47:16.528 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-26T10:47:16.811 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968.target. 2024-07-26T10:47:16.811 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968.target. 2024-07-26T10:47:17.200 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a 2024-07-26T10:47:17.200 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Failed to reset failed state of unit ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a.service: Unit ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a.service not loaded. 2024-07-26T10:47:17.482 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968.target.wants/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a.service → /etc/systemd/system/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@.service. 2024-07-26T10:47:18.151 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:17 smithi144 podman[31746]: 2024-07-26 10:47:17.787771743 +0000 UTC m=+0.030913419 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:47:18.151 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:18 smithi144 podman[31746]: 2024-07-26 10:47:18.148510597 +0000 UTC m=+0.391652316 container init 602859ada537311256700904c243ad54203f64d202ce9e87b03e8e50bf2a7866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T10:47:18.314 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T10:47:18.314 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout disabled 2024-07-26T10:47:18.329 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T10:47:18.329 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout inactive 2024-07-26T10:47:18.329 INFO:teuthology.orchestra.run.smithi144.stdout:firewalld.service is not enabled 2024-07-26T10:47:18.329 INFO:teuthology.orchestra.run.smithi144.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T10:47:18.329 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for mon to start... 2024-07-26T10:47:18.329 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for mon... 2024-07-26T10:47:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:18 smithi144 podman[31746]: 2024-07-26 10:47:18.159375086 +0000 UTC m=+0.402516802 container start 602859ada537311256700904c243ad54203f64d202ce9e87b03e8e50bf2a7866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T10:47:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:18 smithi144 bash[31746]: 602859ada537311256700904c243ad54203f64d202ce9e87b03e8e50bf2a7866 2024-07-26T10:47:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:18 smithi144 systemd[1]: Started Ceph mon.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:47:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:18 smithi144 ceph-mon[31765]: mkfs 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:18 smithi144 ceph-mon[31765]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T10:47:19.592 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T10:47:19.592 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: monmap epoch 1 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: last_changed 2024-07-26T10:47:14.260857+0000 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: min_mon_release 19 (squid) 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: election_strategy: 1 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: fsmap 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: mgrmap e1: no daemons active 2024-07-26T10:47:19.593 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:19 smithi144 ceph-mon[31765]: from='client.? 172.21.15.144:0/3951428680' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-26T10:47:19.714 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout cluster: 2024-07-26T10:47:19.715 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout id: 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:19.715 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-26T10:47:19.715 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:19.715 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout services: 2024-07-26T10:47:19.716 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.72641s) 2024-07-26T10:47:19.716 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-26T10:47:19.717 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout data: 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout pgs: 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:19.718 INFO:teuthology.orchestra.run.smithi144.stdout:mon is available 2024-07-26T10:47:19.719 INFO:teuthology.orchestra.run.smithi144.stdout:Assimilating anything we can from ceph.conf... 2024-07-26T10:47:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:20 smithi144 ceph-mon[31765]: from='client.? 172.21.15.144:0/319758980' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T10:47:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:20 smithi144 ceph-mon[31765]: from='client.? 172.21.15.144:0/319758980' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout fsid = 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.144:3300,v1:172.21.15.144:6789] 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T10:47:21.093 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T10:47:21.094 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T10:47:21.095 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T10:47:21.095 INFO:teuthology.orchestra.run.smithi144.stdout:Generating new minimal ceph.conf... 2024-07-26T10:47:22.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:21 smithi144 ceph-mon[31765]: from='client.? 172.21.15.144:0/3084915117' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:22.402 INFO:teuthology.orchestra.run.smithi144.stdout:Restarting the monitor... 2024-07-26T10:47:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:22 smithi144 systemd[1]: Stopping Ceph mon.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:47:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a[31761]: 2024-07-26T10:47:22.863+0000 7fcd0f400640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-26T10:47:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a[31761]: 2024-07-26T10:47:22.863+0000 7fcd0f400640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-26T10:47:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:23 smithi144 podman[32005]: 2024-07-26 10:47:23.028610156 +0000 UTC m=+0.268144086 container died 602859ada537311256700904c243ad54203f64d202ce9e87b03e8e50bf2a7866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-26T10:47:23.676 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:23 smithi144 podman[32005]: 2024-07-26 10:47:23.448273556 +0000 UTC m=+0.687807509 container cleanup 602859ada537311256700904c243ad54203f64d202ce9e87b03e8e50bf2a7866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T10:47:23.676 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:23 smithi144 bash[32005]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a 2024-07-26T10:47:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:23 smithi144 podman[32018]: 2024-07-26 10:47:23.673948166 +0000 UTC m=+0.628664563 container remove 602859ada537311256700904c243ad54203f64d202ce9e87b03e8e50bf2a7866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T10:47:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-26T10:47:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 systemd[1]: Stopped Ceph mon.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:47:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.a.service: Consumed 2.059s CPU time. 2024-07-26T10:47:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 systemd[1]: Starting Ceph mon.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:47:24.936 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 podman[32110]: 2024-07-26 10:47:24.574802179 +0000 UTC m=+0.114670407 container create e91cf4998150589cb819d5c0afd65a3bfee4099790756ba75bacbf727af7c528 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T10:47:24.936 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 podman[32110]: 2024-07-26 10:47:24.491629948 +0000 UTC m=+0.031498158 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:47:24.936 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 podman[32110]: 2024-07-26 10:47:24.811592302 +0000 UTC m=+0.351460536 container init e91cf4998150589cb819d5c0afd65a3bfee4099790756ba75bacbf727af7c528 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T10:47:24.936 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 podman[32110]: 2024-07-26 10:47:24.822503131 +0000 UTC m=+0.362371329 container start e91cf4998150589cb819d5c0afd65a3bfee4099790756ba75bacbf727af7c528 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: pidfile_write: ignore empty --pid-file 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: load: jerasure load: lrc 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: RocksDB version: 7.9.2 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Git sha 0 2024-07-26T10:47:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: DB SUMMARY 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: DB Session ID: EZRAASF6RKC3YB3SIV7M 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: CURRENT file: CURRENT 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 93897 ; 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.error_if_exists: 0 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.create_if_missing: 0 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.paranoid_checks: 1 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T10:47:25.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.env: 0x5601975d6280 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.info_log: 0x5601a9129060 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.statistics: (nil) 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.use_fsync: 0 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_log_file_size: 0 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T10:47:25.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.allow_fallocate: 1 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.use_direct_reads: 0 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.db_log_dir: 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.wal_dir: 2024-07-26T10:47:25.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.write_buffer_manager: 0x5601a912d5e0 2024-07-26T10:47:25.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.unordered_write: 0 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T10:47:25.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.row_cache: None 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.wal_filter: None 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.two_write_queues: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.wal_compression: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.atomic_flush: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T10:47:25.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.log_readahead_size: 0 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_background_jobs: 2 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_background_compactions: -1 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_subcompactions: 1 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T10:47:25.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_open_files: -1 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_background_flushes: -1 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Compression algorithms supported: 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kZSTD supported: 0 2024-07-26T10:47:25.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kXpressCompression supported: 0 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kBZip2Compression supported: 0 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kLZ4Compression supported: 1 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kZlibCompression supported: 1 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: kSnappyCompression supported: 1 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.merge_operator: 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_filter: None 2024-07-26T10:47:25.230 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601a9128c00) 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: cache_index_and_filter_blocks: 1 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: pin_top_level_index_and_filter: 1 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: index_type: 0 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: data_block_index_type: 0 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: index_shortening: 1 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: checksum: 4 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: no_block_cache: 0 2024-07-26T10:47:25.231 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_cache: 0x5601a50ed610 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_cache_name: BinnedLRUCache 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_cache_options: 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: capacity : 536870912 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: num_shard_bits : 4 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: strict_capacity_limit : 0 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: high_pri_pool_ratio: 0.000 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_cache_compressed: (nil) 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: persistent_cache: (nil) 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_size: 4096 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_size_deviation: 10 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_restart_interval: 16 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: index_block_restart_interval: 1 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: metadata_block_size: 4096 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: partition_filters: 0 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: use_delta_encoding: 1 2024-07-26T10:47:25.232 INFO:journalctl@ceph.mon.a.smithi144.stdout: filter_policy: bloomfilter 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: whole_key_filtering: 1 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: verify_compression: 0 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: read_amp_bytes_per_bit: 0 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: format_version: 5 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: enable_index_compression: 1 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: block_align: 0 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: max_auto_readahead_size: 262144 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: prepopulate_block_cache: 0 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: initial_auto_readahead_size: 8192 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression: NoCompression 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T10:47:25.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.num_levels: 7 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T10:47:25.234 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T10:47:25.235 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T10:47:25.236 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T10:47:25.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.inplace_update_support: 0 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T10:47:25.238 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.bloom_locality: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.max_successive_merges: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.ttl: 2592000 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.enable_blob_files: false 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.min_blob_size: 0 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T10:47:25.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1568a86f-8278-4b58-965b-c369f46aee1a 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721990844943699, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721990844945806, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90097, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 88167, "index_size": 250, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11355, "raw_average_key_size": 47, "raw_value_size": 81814, "raw_average_value_size": 345, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721990844, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1568a86f-8278-4b58-965b-c369f46aee1a", "db_session_id": "EZRAASF6RKC3YB3SIV7M", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721990844946032, "job": 1, "event": "recovery_finished"} 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T10:47:25.240 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5601a914ca80 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: DB pointer 0x5601a9eec000 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: ** DB Stats ** 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: ** Compaction Stats [default] ** 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T10:47:25.241 INFO:journalctl@ceph.mon.a.smithi144.stdout: L0 2/0 89.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 59.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: Sum 2/0 89.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 59.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 59.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: ** Compaction Stats [default] ** 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 59.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T10:47:25.242 INFO:journalctl@ceph.mon.a.smithi144.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: Cumulative compaction: 0.00 GB write, 12.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: Interval compaction: 0.00 GB write, 12.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: Block cache BinnedLRUCache@0x5601a50ed610#2 capacity: 512.00 MB usage: 1.28 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: starting mon.a rank 0 at public addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] at bind addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???) e1 preinit fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).mds e1 new map 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).mds e1 print_map 2024-07-26T10:47:25.243 INFO:journalctl@ceph.mon.a.smithi144.stdout: e1 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout: btime 2024-07-26T10:47:18:301336+0000 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout: legacy client fscid: -1 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout: 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout: No filesystems configured 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:24 smithi144 ceph-mon[32129]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: monmap epoch 1 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:25.244 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: last_changed 2024-07-26T10:47:14.260857+0000 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: min_mon_release 19 (squid) 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: election_strategy: 1 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: fsmap 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T10:47:25.245 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:25 smithi144 ceph-mon[32129]: mgrmap e1: no daemons active 2024-07-26T10:47:27.891 INFO:teuthology.orchestra.run.smithi144.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-26T10:47:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:27 smithi144 bash[32110]: e91cf4998150589cb819d5c0afd65a3bfee4099790756ba75bacbf727af7c528 2024-07-26T10:47:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:27 smithi144 systemd[1]: Started Ceph mon.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:47:36.086 INFO:teuthology.orchestra.run.smithi144.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-26T10:47:36.112 INFO:teuthology.orchestra.run.smithi144.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-26T10:47:36.112 INFO:teuthology.orchestra.run.smithi144.stdout:Creating mgr... 2024-07-26T10:47:36.114 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-26T10:47:36.115 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-26T10:47:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:36 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/976182862' entity='client.admin' 2024-07-26T10:47:36.549 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.x 2024-07-26T10:47:36.549 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Failed to reset failed state of unit ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.x.service: Unit ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.x.service not loaded. 2024-07-26T10:47:36.804 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968.target.wants/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@.service. 2024-07-26T10:47:37.470 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:37 smithi144 podman[32351]: 2024-07-26 10:47:37.466699418 +0000 UTC m=+0.398359465 container init d5fcf7416ecbc04a77e62c8aa7c700bb0efaa7fafa0fc70195fe4d5a41a7cec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T10:47:37.590 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T10:47:37.590 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout disabled 2024-07-26T10:47:37.605 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T10:47:37.606 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout inactive 2024-07-26T10:47:37.606 INFO:teuthology.orchestra.run.smithi144.stdout:firewalld.service is not enabled 2024-07-26T10:47:37.606 INFO:teuthology.orchestra.run.smithi144.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T10:47:37.627 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T10:47:37.627 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout disabled 2024-07-26T10:47:37.643 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T10:47:37.643 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout inactive 2024-07-26T10:47:37.643 INFO:teuthology.orchestra.run.smithi144.stdout:firewalld.service is not enabled 2024-07-26T10:47:37.643 INFO:teuthology.orchestra.run.smithi144.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-26T10:47:37.643 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for mgr to start... 2024-07-26T10:47:37.644 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for mgr... 2024-07-26T10:47:37.720 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:37 smithi144 podman[32351]: 2024-07-26 10:47:37.475881129 +0000 UTC m=+0.407541102 container start d5fcf7416ecbc04a77e62c8aa7c700bb0efaa7fafa0fc70195fe4d5a41a7cec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T10:47:37.720 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:37 smithi144 bash[32351]: d5fcf7416ecbc04a77e62c8aa7c700bb0efaa7fafa0fc70195fe4d5a41a7cec2 2024-07-26T10:47:37.720 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:37 smithi144 systemd[1]: Started Ceph mgr.x for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:47:38.693 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:38 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2092633489' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T10:47:39.025 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:39.025 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:47:39.025 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 0 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:39.026 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T10:47:39.027 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T10:47:39.029 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T10:47:18:301336+0000", 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T10:47:39.032 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T10:47:18.302476+0000", 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:47:39.033 INFO:teuthology.orchestra.run.smithi144.stdout:mgr not available, waiting (1/15)... 2024-07-26T10:47:42.104 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:41 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/649411511' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T10:47:42.397 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:42.397 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:47:42.397 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 0 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T10:47:42.398 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.400 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T10:47:42.401 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T10:47:42.404 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.404 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T10:47:42.404 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T10:47:42.404 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T10:47:18:301336+0000", 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.405 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T10:47:18.302476+0000", 2024-07-26T10:47:42.406 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T10:47:42.407 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:42.407 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T10:47:42.407 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:47:42.407 INFO:teuthology.orchestra.run.smithi144.stdout:mgr not available, waiting (2/15)... 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: Activating manager daemon x 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: mgrmap e2: x(active, starting, since 0.0141803s) 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:47:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T10:47:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: Manager daemon x is now available 2024-07-26T10:47:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:47:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:47:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' 2024-07-26T10:47:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:43 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' 2024-07-26T10:47:44.849 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:44 smithi144 ceph-mon[32129]: from='mgr.14100 172.21.15.144:0/2856989630' entity='mgr.x' 2024-07-26T10:47:44.849 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:44 smithi144 ceph-mon[32129]: mgrmap e3: x(active, since 1.02464s) 2024-07-26T10:47:45.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:45 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2705444440' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T10:47:46.070 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:46.070 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:47:46.070 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T10:47:46.070 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T10:47:46.070 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 0 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "quorum_age": 20, 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:46.071 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T10:47:46.072 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T10:47:18:301336+0000", 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T10:47:46.073 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ], 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T10:47:46.074 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T10:47:46.075 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T10:47:18.302476+0000", 2024-07-26T10:47:46.075 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T10:47:46.075 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout }, 2024-07-26T10:47:46.075 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T10:47:46.075 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:47:46.075 INFO:teuthology.orchestra.run.smithi144.stdout:mgr is available 2024-07-26T10:47:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:46 smithi144 ceph-mon[32129]: mgrmap e4: x(active, since 2s) 2024-07-26T10:47:47.562 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:47.562 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T10:47:47.562 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout fsid = 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:47:47.565 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T10:47:47.565 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.144:3300,v1:172.21.15.144:6789] 2024-07-26T10:47:47.565 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T10:47:47.565 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T10:47:47.565 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T10:47:47.565 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T10:47:47.566 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T10:47:47.567 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T10:47:47.567 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T10:47:47.567 INFO:teuthology.orchestra.run.smithi144.stdout:Enabling cephadm module... 2024-07-26T10:47:47.874 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:47 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/240592673' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T10:47:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:48 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1956277177' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-26T10:47:48.970 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ignoring --setuser ceph since I am not root 2024-07-26T10:47:48.970 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ignoring --setgroup ceph since I am not root 2024-07-26T10:47:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:49 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1956277177' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-26T10:47:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:49 smithi144 ceph-mon[32129]: mgrmap e5: x(active, since 5s) 2024-07-26T10:47:50.904 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:50 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/545752136' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T10:47:50.916 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:47:50.917 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-26T10:47:50.917 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T10:47:50.917 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T10:47:50.917 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T10:47:50.917 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:47:50.918 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for the mgr to restart... 2024-07-26T10:47:50.918 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for mgr epoch 5... 2024-07-26T10:47:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: Active manager daemon x restarted 2024-07-26T10:47:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: Activating manager daemon x 2024-07-26T10:47:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: osdmap e2: 0 total, 0 up, 0 in 2024-07-26T10:47:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: mgrmap e6: x(active, starting, since 0.00516889s) 2024-07-26T10:47:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:47:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: Manager daemon x is now available 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:47:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:54 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: Found migration_current of "None". Setting to last migration. 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:47:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:55 smithi144 ceph-mon[32129]: mgrmap e7: x(active, since 1.01916s) 2024-07-26T10:47:56.583 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:47:56.583 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-26T10:47:56.583 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T10:47:56.583 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:47:56.583 INFO:teuthology.orchestra.run.smithi144.stdout:mgr epoch 5 is available 2024-07-26T10:47:56.584 INFO:teuthology.orchestra.run.smithi144.stdout:Setting orchestrator backend to cephadm... 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:47:56.479+0000 7f6007c00640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:10:47:56] ENGINE Error in HTTPServer.serve 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: Traceback (most recent call last): 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._connections.run(self.expiration_interval) 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._run(expiration_interval) 2024-07-26T10:47:56.736 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: s = self.context.wrap_socket( 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: return self.sslsocket_class._create( 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self.do_handshake() 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._sslobj.do_handshake() 2024-07-26T10:47:56.737 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:47:56.738 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:47:56.995 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-mon[32129]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T10:47:56.996 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-mon[32129]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T10:47:56.996 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:56 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:58.007 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: [26/Jul/2024:10:47:56] ENGINE Bus STARTING 2024-07-26T10:47:58.007 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: [26/Jul/2024:10:47:56] ENGINE Serving on http://172.21.15.144:8765 2024-07-26T10:47:58.007 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: [26/Jul/2024:10:47:56] ENGINE Serving on https://172.21.15.144:7150 2024-07-26T10:47:58.007 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: [26/Jul/2024:10:47:56] ENGINE Bus STARTED 2024-07-26T10:47:58.007 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: [26/Jul/2024:10:47:56] ENGINE Error in HTTPServer.serve 2024-07-26T10:47:58.007 INFO:journalctl@ceph.mon.a.smithi144.stdout: Traceback (most recent call last): 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._connections.run(self.expiration_interval) 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._run(expiration_interval) 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: s = self.context.wrap_socket( 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: return self.sslsocket_class._create( 2024-07-26T10:47:58.008 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout: self.do_handshake() 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._sslobj.do_handshake() 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: mgrmap e8: x(active, since 2s) 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:47:58.009 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:57 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:59.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:58 smithi144 ceph-mon[32129]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:47:59.506 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-26T10:47:59.506 INFO:teuthology.orchestra.run.smithi144.stdout:Generating ssh key... 2024-07-26T10:48:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:47:59 smithi144 ceph-mon[32129]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: Generating public/private rsa key pair. 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: Your identification has been saved in /tmp/tmp9r9s1bw9/key 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: Your public key has been saved in /tmp/tmp9r9s1bw9/key.pub 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: The key fingerprint is: 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: SHA256:9no4NsVwj+aCMqDZHiRKGUau8fQROX2joB/mjp3bfC4 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: The key's randomart image is: 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: +---[RSA 3072]----+ 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: | . .o | 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |o +.. o | 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |.+...o o . | 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |o++.o.. . . | 2024-07-26T10:48:00.907 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |ooo+.. S+ o | 2024-07-26T10:48:00.908 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |oo. o . .= . | 2024-07-26T10:48:00.908 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |.+.= . . =. | 2024-07-26T10:48:00.908 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: |o o.*oE B.o | 2024-07-26T10:48:00.908 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: | .. .+o=o= | 2024-07-26T10:48:00.908 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: +----[SHA256]-----+ 2024-07-26T10:48:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:01 smithi144 ceph-mon[32129]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:01 smithi144 ceph-mon[32129]: Generating ssh key... 2024-07-26T10:48:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:01 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:01 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:02.694 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:48:02.695 INFO:teuthology.orchestra.run.smithi144.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-26T10:48:02.696 INFO:teuthology.orchestra.run.smithi144.stdout:Adding key to root@localhost authorized_keys... 2024-07-26T10:48:02.698 INFO:teuthology.orchestra.run.smithi144.stdout:Adding host smithi144... 2024-07-26T10:48:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:02 smithi144 ceph-mon[32129]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:04.060 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:03 smithi144 ceph-mon[32129]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi144", "addr": "172.21.15.144", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:05 smithi144 ceph-mon[32129]: Deploying cephadm binary to smithi144 2024-07-26T10:48:09.050 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout Added host 'smithi144' with addr '172.21.15.144' 2024-07-26T10:48:09.050 INFO:teuthology.orchestra.run.smithi144.stdout:Deploying unmanaged mon service... 2024-07-26T10:48:09.613 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:09 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:09.613 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:09 smithi144 ceph-mon[32129]: Added host smithi144 2024-07-26T10:48:09.613 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:09 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:11.289 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-26T10:48:11.290 INFO:teuthology.orchestra.run.smithi144.stdout:Deploying unmanaged mgr service... 2024-07-26T10:48:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:11 smithi144 ceph-mon[32129]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:11 smithi144 ceph-mon[32129]: Saving service mon spec with placement count:5 2024-07-26T10:48:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:11 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:12.906 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:12 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:12.906 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:12 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:12.906 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:12 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:13.198 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-26T10:48:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:13 smithi144 ceph-mon[32129]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:13 smithi144 ceph-mon[32129]: Saving service mgr spec with placement count:2 2024-07-26T10:48:15.285 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:15 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/430068644' entity='client.admin' 2024-07-26T10:48:15.286 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:15 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:15.286 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:15 smithi144 ceph-mon[32129]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:16.491 INFO:teuthology.orchestra.run.smithi144.stdout:Enabling the dashboard module... 2024-07-26T10:48:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:16 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/474938717' entity='client.admin' 2024-07-26T10:48:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:16 smithi144 ceph-mon[32129]: from='mgr.14118 172.21.15.144:0/1107879481' entity='mgr.x' 2024-07-26T10:48:18.204 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:18 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ignoring --setuser ceph since I am not root 2024-07-26T10:48:18.204 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:18 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ignoring --setgroup ceph since I am not root 2024-07-26T10:48:18.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:17 smithi144 ceph-mon[32129]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:18.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:17 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1520246023' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-26T10:48:19.160 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:18 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1520246023' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-26T10:48:19.160 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:18 smithi144 ceph-mon[32129]: mgrmap e9: x(active, since 23s) 2024-07-26T10:48:20.215 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:19 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/4290767739' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T10:48:20.620 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:48:20.620 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-26T10:48:20.621 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T10:48:20.621 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T10:48:20.621 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T10:48:20.621 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:48:20.621 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for the mgr to restart... 2024-07-26T10:48:20.621 INFO:teuthology.orchestra.run.smithi144.stdout:Waiting for mgr epoch 9... 2024-07-26T10:48:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: Active manager daemon x restarted 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: Activating manager daemon x 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: osdmap e3: 0 total, 0 up, 0 in 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: mgrmap e10: x(active, starting, since 0.00843032s) 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: Manager daemon x is now available 2024-07-26T10:48:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:24 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:48:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:48:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-mon[32129]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T10:48:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-mon[32129]: mgrmap e11: x(active, since 1.01037s) 2024-07-26T10:48:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-mon[32129]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T10:48:25.971 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:48:25.616+0000 7f397cc00640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:10:48:25] ENGINE Error in HTTPServer.serve 2024-07-26T10:48:25.971 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: Traceback (most recent call last): 2024-07-26T10:48:25.971 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:48:25.971 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._connections.run(self.expiration_interval) 2024-07-26T10:48:25.971 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._run(expiration_interval) 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: s = self.context.wrap_socket( 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: return self.sslsocket_class._create( 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self.do_handshake() 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:48:25.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._sslobj.do_handshake() 2024-07-26T10:48:25.973 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:48:25.973 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:48:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:48:26.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:26 smithi144 ceph-mon[32129]: [26/Jul/2024:10:48:25] ENGINE Bus STARTING 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:26 smithi144 ceph-mon[32129]: [26/Jul/2024:10:48:25] ENGINE Serving on http://172.21.15.144:8765 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:26 smithi144 ceph-mon[32129]: [26/Jul/2024:10:48:25] ENGINE Serving on https://172.21.15.144:7150 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:26 smithi144 ceph-mon[32129]: [26/Jul/2024:10:48:25] ENGINE Bus STARTED 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:26 smithi144 ceph-mon[32129]: [26/Jul/2024:10:48:25] ENGINE Error in HTTPServer.serve 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: Traceback (most recent call last): 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._connections.run(self.expiration_interval) 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._run(expiration_interval) 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:48:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: s = self.context.wrap_socket( 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: return self.sslsocket_class._create( 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: self.do_handshake() 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._sslobj.do_handshake() 2024-07-26T10:48:26.373 INFO:journalctl@ceph.mon.a.smithi144.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:48:26.378 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout { 2024-07-26T10:48:26.378 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-26T10:48:26.378 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T10:48:26.378 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout } 2024-07-26T10:48:26.378 INFO:teuthology.orchestra.run.smithi144.stdout:mgr epoch 9 is available 2024-07-26T10:48:26.379 INFO:teuthology.orchestra.run.smithi144.stdout:Generating a dashboard self-signed certificate... 2024-07-26T10:48:27.402 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:27 smithi144 ceph-mon[32129]: mgrmap e12: x(active, since 2s) 2024-07-26T10:48:27.402 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:28.138 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-26T10:48:28.138 INFO:teuthology.orchestra.run.smithi144.stdout:Creating initial admin user... 2024-07-26T10:48:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:28 smithi144 ceph-mon[32129]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:28 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:28 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:28 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:29 smithi144 ceph-mon[32129]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:29 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:30.180 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$YnEDnLmf83dlADLFt8xuueZ/zX1SUceh3ea14tE4LwLXYBXLGAU1i", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1721990909, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-26T10:48:30.180 INFO:teuthology.orchestra.run.smithi144.stdout:Fetching dashboard port number... 2024-07-26T10:48:31.689 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:31 smithi144 ceph-mon[32129]: mgrmap e13: x(active, since 6s) 2024-07-26T10:48:31.689 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:31 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/4054503227' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-26T10:48:31.936 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stdout 8443 2024-07-26T10:48:31.969 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T10:48:31.969 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout disabled 2024-07-26T10:48:31.991 INFO:teuthology.orchestra.run.smithi144.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T10:48:31.991 INFO:teuthology.orchestra.run.smithi144.stdout:systemctl: stdout inactive 2024-07-26T10:48:31.991 INFO:teuthology.orchestra.run.smithi144.stdout:firewalld.service is not enabled 2024-07-26T10:48:31.991 INFO:teuthology.orchestra.run.smithi144.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-26T10:48:31.995 INFO:teuthology.orchestra.run.smithi144.stdout:Ceph Dashboard is now available at: 2024-07-26T10:48:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: URL: https://smithi144.front.sepia.ceph.com:8443/ 2024-07-26T10:48:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: User: admin 2024-07-26T10:48:31.996 INFO:teuthology.orchestra.run.smithi144.stdout: Password: pv5htv2rh8 2024-07-26T10:48:31.996 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:31.996 INFO:teuthology.orchestra.run.smithi144.stdout:Saving cluster configuration to /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config directory 2024-07-26T10:48:33.605 INFO:teuthology.orchestra.run.smithi144.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout:Or, if you are only running a single cluster on this host: 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.606 INFO:teuthology.orchestra.run.smithi144.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: ceph telemetry on 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout:For more information see: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:33.607 INFO:teuthology.orchestra.run.smithi144.stdout:Bootstrap complete. 2024-07-26T10:48:33.712 INFO:tasks.cephadm:Fetching config... 2024-07-26T10:48:33.713 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:48:33.713 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-26T10:48:33.757 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-26T10:48:33.758 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:48:33.758 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-26T10:48:33.835 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-26T10:48:33.836 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:48:33.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-26T10:48:33.950 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-26T10:48:33.950 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:48:33.951 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-26T10:48:34.054 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-26T10:48:34.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T10:48:34.148 INFO:teuthology.orchestra.run.smithi144.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:48:34.172 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:33 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1001658005' entity='client.admin' 2024-07-26T10:48:34.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T10:48:34.237 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:48:34.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T10:48:34.293 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqBKclWrt+cq+CM1lnewr3kFvucSPIAC5qVmrSURivZ6zbkn+j81yZIQ0K5zqI4d1Z7vmbHLOlwg/BTuXy8wOdo35GAA5rd64dmfS0WVlR+xaNBdZaLyk3w+CEbXdLkXqhfxo6/e1gb7YCiRbqgElUqeSxshLBR5rlWFj9/g98KAqiGEHWnnia9WD25/N7CPHdnmhE3EvvFwtxVPZuogZp83QSjQNshgQo96mmFxfmOxSwiiFd2iIp0zPN1p5FSXuhLzb5usZtFtk0NzaAqjEuvWM/vfVFteN8zME2DSExPg8mH82chJ1iLpzmyGzJLSmZe9K1usKqBn308BsSbNN/pdMhYDCtafCs94XLbc4AD9hK9386/8aV3BRF3oSmcZT3KDcn3WqeloRixNadPfjYRGdn78lbjn9WoQF3Nm4bQy4qapFakjyC9DcKdv+Fe0T2ZMCOgRRUOr9V14kxOD5BZrO9CuSvUbF7VmZo71O0i9IQHgKyJzJY8thsoXgVJYE= ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:48:34.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-26T10:48:34.730 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:48:38.530 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-26T10:48:38.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-26T10:48:39.047 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:38 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2911162929' entity='client.admin' 2024-07-26T10:48:39.146 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:48:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:40 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:40 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:40 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:48:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:40 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:40 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:40 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:42.080 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-07-26T10:48:42.081 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:48:42.081 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-07-26T10:48:42.100 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:48:42.100 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:48:42.158 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-07-26T10:48:42.158 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch host add smithi160 2024-07-26T10:48:42.592 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:48:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:48:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:42 smithi144 ceph-mon[32129]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:48:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:43 smithi144 ceph-mon[32129]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:48:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:44 smithi144 ceph-mon[32129]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:48:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:44 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:44 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:44 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:44 smithi144 ceph-mon[32129]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:45 smithi144 ceph-mon[32129]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:46 smithi144 ceph-mon[32129]: Deploying cephadm binary to smithi160 2024-07-26T10:48:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:46 smithi144 ceph-mon[32129]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:48.276 INFO:teuthology.orchestra.run.smithi144.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-07-26T10:48:48.358 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:48 smithi144 ceph-mon[32129]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:49.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T10:48:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:49 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:49 smithi144 ceph-mon[32129]: Added host smithi160 2024-07-26T10:48:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:49 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:49 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:49 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:49.664 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:48:50.914 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:50 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:50.915 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:50 smithi144 ceph-mon[32129]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:51.896 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:48:51.897 INFO:teuthology.orchestra.run.smithi144.stdout:[{"addr": "172.21.15.144", "hostname": "smithi144", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-07-26T10:48:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:52 smithi144 ceph-mon[32129]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:48:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:52 smithi144 ceph-mon[32129]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:52.600 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-07-26T10:48:52.600 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:48:52.600 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-07-26T10:48:52.619 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:48:52.619 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:48:52.676 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-07-26T10:48:52.676 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch host add smithi183 2024-07-26T10:48:53.124 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:48:53.549 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:48:53.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:53 smithi144 ceph-mon[32129]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:48:54.644 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:54 smithi144 ceph-mon[32129]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:48:54.644 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:54 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:54.644 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:54 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:54.644 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:54 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:54.644 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:54 smithi144 ceph-mon[32129]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:56 smithi144 ceph-mon[32129]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:48:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:56 smithi144 ceph-mon[32129]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:57 smithi144 ceph-mon[32129]: Deploying cephadm binary to smithi183 2024-07-26T10:48:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:58 smithi144 ceph-mon[32129]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:48:58.877 INFO:teuthology.orchestra.run.smithi144.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-07-26T10:48:59.751 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T10:48:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:59 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:59 smithi144 ceph-mon[32129]: Added host smithi183 2024-07-26T10:48:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:59 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:59 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:48:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:48:59 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:00.195 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:00 smithi144 ceph-mon[32129]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:00 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:02 smithi144 ceph-mon[32129]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:02.323 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:49:02.324 INFO:teuthology.orchestra.run.smithi144.stdout:[{"addr": "172.21.15.144", "hostname": "smithi144", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-07-26T10:49:03.077 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-26T10:49:03.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd crush tunables default 2024-07-26T10:49:03.515 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:49:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:03 smithi144 ceph-mon[32129]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:05.072 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:05.072 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:05.072 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:05.073 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:05.073 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:05.073 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:05.073 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:04 smithi144 ceph-mon[32129]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:05.998 INFO:teuthology.orchestra.run.smithi144.stderr:adjusted tunables profile to default 2024-07-26T10:49:06.020 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:05 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2081504286' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T10:49:06.620 INFO:tasks.cephadm:Adding mon.a on smithi144 2024-07-26T10:49:06.620 INFO:tasks.cephadm:Adding mon.b on smithi160 2024-07-26T10:49:06.620 INFO:tasks.cephadm:Adding mon.c on smithi183 2024-07-26T10:49:06.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch apply mon '3;smithi144:172.21.15.144=a;smithi160:172.21.15.160=b;smithi183:172.21.15.183=c' 2024-07-26T10:49:06.873 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:07 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2081504286' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T10:49:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:07 smithi144 ceph-mon[32129]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:07 smithi144 ceph-mon[32129]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:08 smithi144 ceph-mon[32129]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:08.609 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2024-07-26T10:49:09.162 DEBUG:teuthology.orchestra.run.smithi160:mon.b> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.b.service 2024-07-26T10:49:09.164 DEBUG:teuthology.orchestra.run.smithi183:mon.c> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mon.c.service 2024-07-26T10:49:09.166 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T10:49:09.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T10:49:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi144:172.21.15.144=a;smithi160:172.21.15.160=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:49:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: Saving service mon spec with placement smithi144:172.21.15.144=a;smithi160:172.21.15.160=b;smithi183:172.21.15.183=c;count:3 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:09.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:09 smithi144 ceph-mon[32129]: Deploying daemon mon.c on smithi183 2024-07-26T10:49:10.204 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:10 smithi144 ceph-mon[32129]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:12 smithi144 ceph-mon[32129]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:13.149 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:49:13.149 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","modified":"2024-07-26T10:47:14.260857Z","created":"2024-07-26T10:47:14.260857Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-26T10:49:13.149 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-07-26T10:49:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:13 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/2668350883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T10:49:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:13 smithi183 systemd[1]: Started Ceph mon.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:49:14.975 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T10:49:14.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T10:49:15.232 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: mon.a calling monitor election 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: mon.c calling monitor election 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:18.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: monmap epoch 2 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: last_changed 2024-07-26T10:49:13.601343+0000 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: min_mon_release 19 (squid) 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: election_strategy: 1 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: fsmap 2024-07-26T10:49:18.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: mgrmap e13: x(active, since 54s) 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: overall HEALTH_OK 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:18.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:18 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: mon.a calling monitor election 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: mon.c calling monitor election 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: monmap epoch 2 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: last_changed 2024-07-26T10:49:13.601343+0000 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: min_mon_release 19 (squid) 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: election_strategy: 1 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: fsmap 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: mgrmap e13: x(active, since 54s) 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: overall HEALTH_OK 2024-07-26T10:49:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:19.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:19.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:19.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:19.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:18 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:19.940 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:49:19.940 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","modified":"2024-07-26T10:49:13.601343Z","created":"2024-07-26T10:47:14.260857Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T10:49:19.940 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-07-26T10:49:19.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:19 smithi183 ceph-mon[37273]: Deploying daemon mon.b on smithi160 2024-07-26T10:49:19.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:19 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:19 smithi144 ceph-mon[32129]: Deploying daemon mon.b on smithi160 2024-07-26T10:49:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:19 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:20 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/3939076186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T10:49:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:20 smithi183 ceph-mon[37273]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:20 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/3939076186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T10:49:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:20 smithi144 ceph-mon[32129]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:21.518 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T10:49:21.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T10:49:21.775 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:49:22.191 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:22 smithi183 ceph-mon[37273]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:22 smithi144 ceph-mon[32129]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:22 smithi160 systemd[1]: Started Ceph mon.b for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:49:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: mon.c calling monitor election 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: mon.a calling monitor election 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: monmap epoch 3 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: last_changed 2024-07-26T10:49:22.527206+0000 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: min_mon_release 19 (squid) 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: election_strategy: 1 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: 2: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: fsmap 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: mgrmap e13: x(active, since 63s) 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: overall HEALTH_OK 2024-07-26T10:49:27.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.888 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.888 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.888 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:27 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-07-26T10:49:27.966 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":3,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","modified":"2024-07-26T10:49:22.527206Z","created":"2024-07-26T10:47:14.260857Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T10:49:27.966 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 3 2024-07-26T10:49:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: mon.c calling monitor election 2024-07-26T10:49:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: mon.a calling monitor election 2024-07-26T10:49:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: monmap epoch 3 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: last_changed 2024-07-26T10:49:22.527206+0000 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: min_mon_release 19 (squid) 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: election_strategy: 1 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: 2: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: fsmap 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: mgrmap e13: x(active, since 63s) 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: overall HEALTH_OK 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:27.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:27 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:28.714 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:28 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:28.714 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:28 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/402319880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T10:49:28.714 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:28 smithi183 ceph-mon[37273]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:28.714 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:28 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:28.715 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-26T10:49:28.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config generate-minimal-conf 2024-07-26T10:49:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:28 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:28 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/402319880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T10:49:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:28 smithi144 ceph-mon[32129]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:28 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.152 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: mon.c calling monitor election 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: mon.a calling monitor election 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: monmap epoch 3 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: last_changed 2024-07-26T10:49:22.527206+0000 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: min_mon_release 19 (squid) 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: election_strategy: 1 2024-07-26T10:49:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: 2: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: fsmap 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: mgrmap e13: x(active, since 63s) 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: overall HEALTH_OK 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/402319880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:29.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:29 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:30.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: mon.b calling monitor election 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: mon.b calling monitor election 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: mon.c calling monitor election 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: mon.a calling monitor election 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: monmap epoch 3 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: last_changed 2024-07-26T10:49:22.527206+0000 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:30.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: min_mon_release 19 (squid) 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: election_strategy: 1 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: 2: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: fsmap 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: mgrmap e13: x(active, since 65s) 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: overall HEALTH_OK 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.447 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.447 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.447 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:30 smithi144 ceph-mon[32129]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: mon.b calling monitor election 2024-07-26T10:49:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: mon.b calling monitor election 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: mon.c calling monitor election 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: mon.a calling monitor election 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: monmap epoch 3 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: last_changed 2024-07-26T10:49:22.527206+0000 2024-07-26T10:49:30.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: min_mon_release 19 (squid) 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: election_strategy: 1 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: 2: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: fsmap 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: mgrmap e13: x(active, since 65s) 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: overall HEALTH_OK 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:30 smithi183 ceph-mon[37273]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: mon.b calling monitor election 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: mon.b calling monitor election 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: mon.c calling monitor election 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: mon.a calling monitor election 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: monmap epoch 3 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: last_changed 2024-07-26T10:49:22.527206+0000 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: created 2024-07-26T10:47:14.260857+0000 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: min_mon_release 19 (squid) 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: election_strategy: 1 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: 0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: 2: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: fsmap 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: mgrmap e13: x(active, since 65s) 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: overall HEALTH_OK 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:30.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:30.812 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:30 smithi160 ceph-mon[36715]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:31.406 INFO:teuthology.orchestra.run.smithi144.stdout:# minimal ceph.conf for 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:31.407 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2024-07-26T10:49:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T10:49:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: mon_host = [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2024-07-26T10:49:31.605 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:49:31.528+0000 7f39c4000640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-07-26T10:49:31.605 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:31.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:31 smithi144 ceph-mon[32129]: Reconfiguring daemon mon.a on smithi144 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T10:49:31.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:31.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:31.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:31.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:31 smithi183 ceph-mon[37273]: Reconfiguring daemon mon.a on smithi144 2024-07-26T10:49:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:31 smithi160 ceph-mon[36715]: Reconfiguring daemon mon.a on smithi144 2024-07-26T10:49:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:32 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2938175197' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:32 smithi183 ceph-mon[37273]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:32.647 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:32 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2938175197' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:32.647 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:32 smithi144 ceph-mon[32129]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:32 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2938175197' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:32 smithi160 ceph-mon[36715]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:33.044 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-26T10:49:33.045 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:49:33.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T10:49:33.119 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:49:33.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:33.240 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:49:33.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T10:49:33.268 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:49:33.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:33.332 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:49:33.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T10:49:33.363 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:49:33.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:33.430 INFO:tasks.cephadm:enabling module snap_schedule 2024-07-26T10:49:33.431 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-07-26T10:49:33.886 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: Reconfiguring mon.b (monmap changed)... 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:34.032 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:33 smithi144 ceph-mon[32129]: Reconfiguring daemon mon.b on smithi160 2024-07-26T10:49:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: Reconfiguring mon.b (monmap changed)... 2024-07-26T10:49:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:34.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:33 smithi160 ceph-mon[36715]: Reconfiguring daemon mon.b on smithi160 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: Reconfiguring mon.b (monmap changed)... 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:33 smithi183 ceph-mon[37273]: Reconfiguring daemon mon.b on smithi160 2024-07-26T10:49:35.043 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:34 smithi144 ceph-mon[32129]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:35.043 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:34 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:35.043 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:34 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:35.043 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:34 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:35.043 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:34 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:35.044 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:34 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:34 smithi160 ceph-mon[36715]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:34 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:34 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:34 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:34 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:35.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:34 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:34 smithi183 ceph-mon[37273]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:34 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:34 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:34 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T10:49:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:34 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T10:49:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:34 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: Reconfiguring mon.c (monmap changed)... 2024-07-26T10:49:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: Reconfiguring daemon mon.c on smithi183 2024-07-26T10:49:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:36.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:36.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:36.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:35 smithi160 ceph-mon[36715]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: Reconfiguring mon.c (monmap changed)... 2024-07-26T10:49:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: Reconfiguring daemon mon.c on smithi183 2024-07-26T10:49:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:36.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:36.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:36.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:35 smithi183 ceph-mon[37273]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: Reconfiguring mon.c (monmap changed)... 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: Reconfiguring daemon mon.c on smithi183 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:35 smithi144 ceph-mon[32129]: from='mgr.14150 172.21.15.144:0/223096039' entity='mgr.x' 2024-07-26T10:49:37.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:36 smithi160 ceph-mon[36715]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:37.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:36 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/395000076' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T10:49:37.124 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:36 smithi144 ceph-mon[32129]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:37.124 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:36 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/395000076' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T10:49:37.124 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:37 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ignoring --setuser ceph since I am not root 2024-07-26T10:49:37.124 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:37 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ignoring --setgroup ceph since I am not root 2024-07-26T10:49:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:36 smithi183 ceph-mon[37273]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:49:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:36 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/395000076' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T10:49:37.479 INFO:tasks.cephadm:Adding mgr.x on smithi144 2024-07-26T10:49:37.480 INFO:tasks.cephadm:Adding mgr.y on smithi160 2024-07-26T10:49:37.480 INFO:tasks.cephadm:Adding mgr.z on smithi183 2024-07-26T10:49:37.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch apply mgr '3;smithi144=x;smithi160=y;smithi183=z' 2024-07-26T10:49:37.725 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:49:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:37 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/395000076' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T10:49:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:37 smithi160 ceph-mon[36715]: mgrmap e14: x(active, since 72s) 2024-07-26T10:49:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:37 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/395000076' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T10:49:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:37 smithi183 ceph-mon[37273]: mgrmap e14: x(active, since 72s) 2024-07-26T10:49:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:37 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/395000076' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T10:49:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:37 smithi144 ceph-mon[32129]: mgrmap e14: x(active, since 72s) 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: Active manager daemon x restarted 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: Activating manager daemon x 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: mgrmap e15: x(active, starting, since 0.0147436s) 2024-07-26T10:49:43.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:43.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T10:49:43.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T10:49:43.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T10:49:43.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T10:49:43.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: Manager daemon x is now available 2024-07-26T10:49:43.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: Active manager daemon x restarted 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: Activating manager daemon x 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: mgrmap e15: x(active, starting, since 0.0147436s) 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T10:49:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T10:49:43.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T10:49:43.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T10:49:43.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: Manager daemon x is now available 2024-07-26T10:49:43.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: Active manager daemon x restarted 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: Activating manager daemon x 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: mgrmap e15: x(active, starting, since 0.0147436s) 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T10:49:43.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T10:49:43.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T10:49:43.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: Manager daemon x is now available 2024-07-26T10:49:43.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:43.830 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mgr update... 2024-07-26T10:49:44.126 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:49:44.126 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:49:44.127 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:49:44.127 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:49:44.127 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:43 smithi160 ceph-mon[36715]: mgrmap e16: x(active, since 1.01327s) 2024-07-26T10:49:44.127 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:44.128 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:49:44.128 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:49:44.128 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:49:44.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:49:44.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:43 smithi183 ceph-mon[37273]: mgrmap e16: x(active, since 1.01327s) 2024-07-26T10:49:44.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:49:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T10:49:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:49:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T10:49:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:43 smithi144 ceph-mon[32129]: mgrmap e16: x(active, since 1.01327s) 2024-07-26T10:49:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:44.406 DEBUG:teuthology.orchestra.run.smithi160:mgr.y> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.y.service 2024-07-26T10:49:44.412 DEBUG:teuthology.orchestra.run.smithi183:mgr.z> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mgr.z.service 2024-07-26T10:49:44.416 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-26T10:49:44.417 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-26T10:49:44.424 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-26T10:49:44.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set client debug_client 20 2024-07-26T10:49:44.473 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-26T10:49:44.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set client debug_ms 1 2024-07-26T10:49:44.524 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-26T10:49:44.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-26T10:49:44.525 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:49:44.461+0000 7f4a5c800640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:10:49:44] ENGINE Error in HTTPServer.serve 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: Traceback (most recent call last): 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._connections.run(self.expiration_interval) 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._run(expiration_interval) 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:49:44.526 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: s = self.context.wrap_socket( 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: return self.sslsocket_class._create( 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self.do_handshake() 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: self._sslobj.do_handshake() 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:49:44.527 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:49:44.575 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-26T10:49:44.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-26T10:49:44.624 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-26T10:49:44.624 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-26T10:49:44.675 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-26T10:49:44.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-26T10:49:44.726 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-26T10:49:44.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-26T10:49:44.785 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-26T10:49:44.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-26T10:49:44.844 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-26T10:49:44.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-26T10:49:44.866 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-mon[32129]: Saving service mgr spec with placement smithi144=x;smithi160=y;smithi183=z;count:3 2024-07-26T10:49:44.866 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:44.866 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:44.877 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-26T10:49:44.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-26T10:49:44.932 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-26T10:49:44.932 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-26T10:49:44.995 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-26T10:49:44.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-26T10:49:45.050 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-26T10:49:45.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-26T10:49:45.117 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-26T10:49:45.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-26T10:49:45.118 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:44 smithi183 ceph-mon[37273]: Saving service mgr spec with placement smithi144=x;smithi160=y;smithi183=z;count:3 2024-07-26T10:49:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:45.181 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-26T10:49:45.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-26T10:49:45.208 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:45.228 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-26T10:49:45.228 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-26T10:49:45.291 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T10:49:45.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-26T10:49:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:44 smithi160 ceph-mon[36715]: Saving service mgr spec with placement smithi144=x;smithi160=y;smithi183=z;count:3 2024-07-26T10:49:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:45.350 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T10:49:45.350 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-26T10:49:45.418 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-07-26T10:49:45.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mgr debug_cephsqlite 20 2024-07-26T10:49:45.471 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-07-26T10:49:45.471 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-07-26T10:49:45.534 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-07-26T10:49:45.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-07-26T10:49:45.595 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-07-26T10:49:45.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-07-26T10:49:45.649 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-26T10:49:45.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-26T10:49:45.719 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T10:49:45.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T10:49:45.777 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-26T10:49:45.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-26T10:49:45.838 DEBUG:tasks.cephadm:set 26 configs 2024-07-26T10:49:45.839 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi144') 2024-07-26T10:49:46.047 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: [26/Jul/2024:10:49:44] ENGINE Bus STARTING 2024-07-26T10:49:46.047 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: [26/Jul/2024:10:49:44] ENGINE Serving on http://172.21.15.144:8765 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: [26/Jul/2024:10:49:44] ENGINE Serving on https://172.21.15.144:7150 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: [26/Jul/2024:10:49:44] ENGINE Bus STARTED 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: [26/Jul/2024:10:49:44] ENGINE Error in HTTPServer.serve 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: Traceback (most recent call last): 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._connections.run(self.expiration_interval) 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._run(expiration_interval) 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:49:46.048 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: s = self.context.wrap_socket( 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: return self.sslsocket_class._create( 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: self.do_handshake() 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: self._sslobj.do_handshake() 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.049 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: mgrmap e17: x(active, since 2s) 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.068 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: [26/Jul/2024:10:49:44] ENGINE Bus STARTING 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: [26/Jul/2024:10:49:44] ENGINE Serving on http://172.21.15.144:8765 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: [26/Jul/2024:10:49:44] ENGINE Serving on https://172.21.15.144:7150 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: [26/Jul/2024:10:49:44] ENGINE Bus STARTED 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: [26/Jul/2024:10:49:44] ENGINE Error in HTTPServer.serve 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:49:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:49:46.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: mgrmap e17: x(active, since 2s) 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: [26/Jul/2024:10:49:44] ENGINE Bus STARTING 2024-07-26T10:49:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: [26/Jul/2024:10:49:44] ENGINE Serving on http://172.21.15.144:8765 2024-07-26T10:49:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: [26/Jul/2024:10:49:44] ENGINE Serving on https://172.21.15.144:7150 2024-07-26T10:49:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: [26/Jul/2024:10:49:44] ENGINE Bus STARTED 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: [26/Jul/2024:10:49:44] ENGINE Error in HTTPServer.serve 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: Traceback (most recent call last): 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: self._connections.run(self.expiration_interval) 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: self._run(expiration_interval) 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: s = self.context.wrap_socket( 2024-07-26T10:49:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout: return self.sslsocket_class._create( 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout: self.do_handshake() 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout: self._sslobj.do_handshake() 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: mgrmap e17: x(active, since 2s) 2024-07-26T10:49:46.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:46.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:46.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:47.386 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.498 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.509 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.515 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.516 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.517 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.524 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:48.541 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:50.278 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:50.278 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:50.278 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:50.279 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:50.279 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:50.279 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:50.279 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:50 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1810353274' entity='client.admin' 2024-07-26T10:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:50 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1810353274' entity='client.admin' 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:50 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1810353274' entity='client.admin' 2024-07-26T10:49:51.181 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:51.187 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:51.188 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:51.203 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:51.207 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:49:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:49:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:49:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi144:/etc/ceph/ceph.conf 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi160:/etc/ceph/ceph.conf 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi183:/etc/ceph/ceph.conf 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi160:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: Updating smithi183:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.396 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:51.396 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T10:49:52.013 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:52.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:52.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:52.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:52.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-07-26T10:49:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: Updating smithi144:/var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T10:49:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T10:49:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:53 smithi160 ceph-mon[36715]: Deploying daemon mgr.z on smithi183 2024-07-26T10:49:53.112 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:53 smithi144 ceph-mon[32129]: Deploying daemon mgr.z on smithi183 2024-07-26T10:49:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:53 smithi183 ceph-mon[37273]: Deploying daemon mgr.z on smithi183 2024-07-26T10:49:53.844 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi144') 2024-07-26T10:49:53.885 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 26 10:49:53 smithi183 systemd[1]: Started Ceph mgr.z for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:49:54.922 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:54.922 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:54.922 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:54.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:54.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:54.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T10:49:54.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:54.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:54.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:54 smithi144 ceph-mon[32129]: Deploying daemon mgr.y on smithi160 2024-07-26T10:49:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:54 smithi160 ceph-mon[36715]: Deploying daemon mgr.y on smithi160 2024-07-26T10:49:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T10:49:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:54 smithi183 ceph-mon[37273]: Deploying daemon mgr.y on smithi160 2024-07-26T10:49:56.059 INFO:journalctl@ceph.mgr.y.smithi160.stdout:Jul 26 10:49:55 smithi160 systemd[1]: Started Ceph mgr.y for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:57.180 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.180 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.181 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.181 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.181 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:57.623 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:57.624 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:57.626 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:57.628 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: Reconfiguring daemon mgr.x on smithi144 2024-07-26T10:49:58.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:58.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:58.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:58.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: Reconfiguring daemon mgr.x on smithi144 2024-07-26T10:49:58.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T10:49:58.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:58.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T10:49:58.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T10:49:58.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:58.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: Reconfiguring daemon mgr.x on smithi144 2024-07-26T10:49:58.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:49:58.850 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:49:59.377 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:59 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T10:49:59.378 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:59 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T10:49:59.378 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:59 smithi144 ceph-mon[32129]: Standby manager daemon z started 2024-07-26T10:49:59.378 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:59 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T10:49:59.378 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:49:59 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T10:49:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:59 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T10:49:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:59 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T10:49:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:59 smithi183 ceph-mon[37273]: Standby manager daemon z started 2024-07-26T10:49:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:59 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T10:49:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:49:59 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T10:49:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:59 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T10:49:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:59 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T10:49:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:59 smithi160 ceph-mon[36715]: Standby manager daemon z started 2024-07-26T10:49:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:59 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T10:49:59.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:49:59 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.183:0/1538620646' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T10:50:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T10:50:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:00 smithi183 ceph-mon[37273]: mgrmap e18: x(active, since 16s), standbys: z 2024-07-26T10:50:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:00 smithi183 ceph-mon[37273]: overall HEALTH_OK 2024-07-26T10:50:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T10:50:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:00 smithi144 ceph-mon[32129]: mgrmap e18: x(active, since 16s), standbys: z 2024-07-26T10:50:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:00 smithi144 ceph-mon[32129]: overall HEALTH_OK 2024-07-26T10:50:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T10:50:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:00 smithi160 ceph-mon[36715]: mgrmap e18: x(active, since 16s), standbys: z 2024-07-26T10:50:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:00 smithi160 ceph-mon[36715]: overall HEALTH_OK 2024-07-26T10:50:01.608 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:01 smithi144 ceph-mon[32129]: Standby manager daemon y started 2024-07-26T10:50:01.609 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:01 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T10:50:01.609 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:01 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T10:50:01.609 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:01 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T10:50:01.609 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:01 smithi144 ceph-mon[32129]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T10:50:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:01 smithi183 ceph-mon[37273]: Standby manager daemon y started 2024-07-26T10:50:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:01 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T10:50:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:01 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T10:50:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:01 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T10:50:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:01 smithi183 ceph-mon[37273]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 -- 172.21.15.144:0/160262969 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fea8c0b3c10 msgr2=0x7fea8c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 --2- 172.21.15.144:0/160262969 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fea8c0b3c10 0x7fea8c0b6120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fea9402aa00 tx=0x7fea94000bc0 comp rx=0 tx=0).stop 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 -- 172.21.15.144:0/160262969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa81057e0 msgr2=0x7feaa819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 --2- 172.21.15.144:0/160262969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa81057e0 0x7feaa819bdb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea9800e9f0 tx=0x7fea9800eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 -- 172.21.15.144:0/160262969 shutdown_connections 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 --2- 172.21.15.144:0/160262969 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fea8c0b3c10 0x7fea8c0b6120 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:01.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 --2- 172.21.15.144:0/160262969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feaa810ba70 0x7feaa81a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:01.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 --2- 172.21.15.144:0/160262969 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa810b120 0x7feaa819c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:01.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 --2- 172.21.15.144:0/160262969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa81057e0 0x7feaa819bdb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:01.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 -- 172.21.15.144:0/160262969 >> 172.21.15.144:0/160262969 conn(0x7feaa8101210 msgr2=0x7feaa8103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:01.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 -- 172.21.15.144:0/160262969 shutdown_connections 2024-07-26T10:50:01.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:01.651+0000 7feaaf000640 1 -- 172.21.15.144:0/160262969 wait complete. 2024-07-26T10:50:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:01 smithi160 ceph-mon[36715]: Standby manager daemon y started 2024-07-26T10:50:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:01 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T10:50:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:01 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T10:50:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:01 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T10:50:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:01 smithi160 ceph-mon[36715]: from='mgr.? 172.21.15.160:0/940904491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T10:50:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T10:50:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:02 smithi183 ceph-mon[37273]: mgrmap e19: x(active, since 18s), standbys: z, y 2024-07-26T10:50:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:02 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/160262969' entity='client.admin' 2024-07-26T10:50:02.707 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T10:50:02.708 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:02 smithi144 ceph-mon[32129]: mgrmap e19: x(active, since 18s), standbys: z, y 2024-07-26T10:50:02.708 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:02 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/160262969' entity='client.admin' 2024-07-26T10:50:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T10:50:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:02 smithi160 ceph-mon[36715]: mgrmap e19: x(active, since 18s), standbys: z, y 2024-07-26T10:50:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:02 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/160262969' entity='client.admin' 2024-07-26T10:50:03.515 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:03 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:03.515 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:03 smithi144 ceph-mon[32129]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:03 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:03 smithi183 ceph-mon[37273]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:03 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:03 smithi160 ceph-mon[36715]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 -- 172.21.15.144:0/1555341159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410ba70 msgr2=0x7f34441122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 --2- 172.21.15.144:0/1555341159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410ba70 0x7f34441122a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f343c009ca0 tx=0x7f343c02a8c0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 -- 172.21.15.144:0/1555341159 shutdown_connections 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 --2- 172.21.15.144:0/1555341159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410ba70 0x7f34441122a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 --2- 172.21.15.144:0/1555341159 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f344410b120 0x7f344410b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 --2- 172.21.15.144:0/1555341159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34441057e0 0x7f3444105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 -- 172.21.15.144:0/1555341159 >> 172.21.15.144:0/1555341159 conn(0x7f34441011f0 msgr2=0x7f3444103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 -- 172.21.15.144:0/1555341159 shutdown_connections 2024-07-26T10:50:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.424+0000 7f3448e00640 1 -- 172.21.15.144:0/1555341159 wait complete. 2024-07-26T10:50:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 Processor -- start 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 -- start start 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34441057e0 0x7f344419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 0x7f344419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344410ba70 0x7f34441a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3444115580 con 0x7f344410b120 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3444115400 con 0x7f34441057e0 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.426+0000 7f3448e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3444115700 con 0x7f344410ba70 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 0x7f344419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f3441e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34441057e0 0x7f344419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 0x7f344419c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37800/0 (socket says 172.21.15.144:37800) 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f3442800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344410ba70 0x7f34441a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 -- 172.21.15.144:0/534990621 learned_addr learned my addr 172.21.15.144:0/534990621 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:05.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 -- 172.21.15.144:0/534990621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344410ba70 msgr2=0x7f34441a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:05.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344410ba70 0x7f34441a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 -- 172.21.15.144:0/534990621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34441057e0 msgr2=0x7f344419bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34441057e0 0x7f344419bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f343be00640 1 -- 172.21.15.144:0/534990621 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34441a1f00 con 0x7f344410b120 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.427+0000 7f3442800640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344410ba70 0x7f34441a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.428+0000 7f343be00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 0x7f344419c530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f342c00e970 tx=0x7f342c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.428+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c00cd80 con 0x7f344410b120 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.428+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(24 keys) ==== 1045+0+0 (secure 0 0 0) 0x7f342c0035f0 con 0x7f344410b120 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.428+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c0044a0 con 0x7f344410b120 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.428+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34441a21f0 con 0x7f344410b120 2024-07-26T10:50:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.428+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34441a8950 con 0x7f344410b120 2024-07-26T10:50:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.430+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3444105bf0 con 0x7f344410b120 2024-07-26T10:50:05.434 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.433+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f342c004000 con 0x7f344410b120 2024-07-26T10:50:05.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.435+0000 7f343aa00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34280b3c10 0x7f34280b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:05.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.435+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f342c0e2300 con 0x7f344410b120 2024-07-26T10:50:05.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.435+0000 7f3441e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34280b3c10 0x7f34280b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:05.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.436+0000 7f3441e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34280b3c10 0x7f34280b6120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3434009bd0 tx=0x7f3434005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:05.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.436+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f342c0037a0 con 0x7f344410b120 2024-07-26T10:50:05.641 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.639+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f344402e0c0 con 0x7f344410b120 2024-07-26T10:50:05.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.647+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v13)=0 v13) ==== 114+0+0 (secure 0 0 0) 0x7f342c0ab540 con 0x7f344410b120 2024-07-26T10:50:05.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.649+0000 7f343aa00640 1 -- 172.21.15.144:0/534990621 <== mon.0 v2:172.21.15.144:3300/0 8 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f342c0aa950 con 0x7f344410b120 2024-07-26T10:50:05.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34280b3c10 msgr2=0x7f34280b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:05.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34280b3c10 0x7f34280b6120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3434009bd0 tx=0x7f3434005f90 comp rx=0 tx=0).stop 2024-07-26T10:50:05.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 msgr2=0x7f344419c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:05.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 0x7f344419c530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f342c00e970 tx=0x7f342c00ee30 comp rx=0 tx=0).stop 2024-07-26T10:50:05.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 shutdown_connections 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34280b3c10 0x7f34280b6120 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344410ba70 0x7f34441a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f344410b120 0x7f344419c530 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 --2- 172.21.15.144:0/534990621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34441057e0 0x7f344419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 >> 172.21.15.144:0/534990621 conn(0x7f34441011f0 msgr2=0x7f34441096a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.651+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 shutdown_connections 2024-07-26T10:50:05.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:05.652+0000 7f3448e00640 1 -- 172.21.15.144:0/534990621 wait complete. 2024-07-26T10:50:06.017 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:05 smithi144 ceph-mon[32129]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:06.017 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:05 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/534990621' entity='client.admin' 2024-07-26T10:50:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:05 smithi183 ceph-mon[37273]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:05 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/534990621' entity='client.admin' 2024-07-26T10:50:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:05 smithi160 ceph-mon[36715]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:05 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/534990621' entity='client.admin' 2024-07-26T10:50:06.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.579+0000 7f18f3e00640 1 -- 172.21.15.144:0/1945727979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 msgr2=0x7f18f4105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.579+0000 7f18f3e00640 1 --2- 172.21.15.144:0/1945727979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 0x7f18f4105a00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009900 tx=0x7f18e402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 -- 172.21.15.144:0/1945727979 shutdown_connections 2024-07-26T10:50:06.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 --2- 172.21.15.144:0/1945727979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410b880 0x7f18f41120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 --2- 172.21.15.144:0/1945727979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410af30 0x7f18f410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 --2- 172.21.15.144:0/1945727979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 0x7f18f4105a00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 -- 172.21.15.144:0/1945727979 >> 172.21.15.144:0/1945727979 conn(0x7f18f4101060 msgr2=0x7f18f41034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 -- 172.21.15.144:0/1945727979 shutdown_connections 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.580+0000 7f18f3e00640 1 -- 172.21.15.144:0/1945727979 wait complete. 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 Processor -- start 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 -- start start 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 0x7f18f419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 0x7f18f419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410b880 0x7f18f41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f18f4115460 con 0x7f18f410b880 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f18f41152e0 con 0x7f18f41055f0 2024-07-26T10:50:06.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.581+0000 7f18f3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18f41155e0 con 0x7f18f410af30 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 0x7f18f419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18f3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410b880 0x7f18f41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 0x7f18f419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48898/0 (socket says 172.21.15.144:48898) 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 -- 172.21.15.144:0/4171602762 learned_addr learned my addr 172.21.15.144:0/4171602762 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18f2a00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 0x7f18f419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 -- 172.21.15.144:0/4171602762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 msgr2=0x7f18f419be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 0x7f18f419be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 -- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410b880 msgr2=0x7f18f41a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410b880 0x7f18f41a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18ea000640 1 -- 172.21.15.144:0/4171602762 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f41a1de0 con 0x7f18f410af30 2024-07-26T10:50:06.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.582+0000 7f18f3400640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410b880 0x7f18f41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.583+0000 7f18ea000640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 0x7f18f419c410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00e9f0 tx=0x7f18dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.583+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00ccc0 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.583+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f18dc0108c0 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.583+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc018730 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.583+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f41a20d0 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.583+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f41a8830 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.584+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f4105a00 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.587+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f18dc010430 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.589+0000 7f18ebe00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18d80b3c10 0x7f18d80b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.589+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f18dc0e27e0 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.589+0000 7f18f2a00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18d80b3c10 0x7f18d80b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.589+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f18dc0ab340 con 0x7f18f410af30 2024-07-26T10:50:06.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.590+0000 7f18f2a00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18d80b3c10 0x7f18d80b6120 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f18e4000c00 tx=0x7f18e4005e80 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 -- 172.21.15.144:0/1534641385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23001035b0 msgr2=0x7f23001039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 --2- 172.21.15.144:0/1534641385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23001035b0 0x7f23001039c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009860 tx=0x7f22f402a340 comp rx=0 tx=0).stop 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 -- 172.21.15.144:0/1534641385 shutdown_connections 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 --2- 172.21.15.144:0/1534641385 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300109840 0x7f2300110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 --2- 172.21.15.144:0/1534641385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2300108ef0 0x7f2300109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 --2- 172.21.15.144:0/1534641385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23001035b0 0x7f23001039c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 -- 172.21.15.144:0/1534641385 >> 172.21.15.144:0/1534641385 conn(0x7f23000fefc0 msgr2=0x7f2300101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.135+0000 7f2308800640 1 -- 172.21.15.144:0/1534641385 shutdown_connections 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.136+0000 7f2308800640 1 -- 172.21.15.144:0/1534641385 wait complete. 2024-07-26T10:50:06.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 Processor -- start 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 -- start start 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23001035b0 0x7f230019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300108ef0 0x7f230019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2300109840 0x7f23001a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f23001131a0 con 0x7f23001035b0 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2300113020 con 0x7f2300108ef0 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.137+0000 7f2308800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2300113320 con 0x7f2300109840 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2305e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23001035b0 0x7f230019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2305400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300108ef0 0x7f230019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2305400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300108ef0 0x7f230019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55380/0 (socket says 172.21.15.144:55380) 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2305e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23001035b0 0x7f230019beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37770/0 (socket says 172.21.15.144:37770) 2024-07-26T10:50:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2306800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2300109840 0x7f23001a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2305400640 1 -- 172.21.15.144:0/3816428092 learned_addr learned my addr 172.21.15.144:0/3816428092 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2306800640 1 -- 172.21.15.144:0/3816428092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300108ef0 msgr2=0x7f230019c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2306800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300108ef0 0x7f230019c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2306800640 1 -- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23001035b0 msgr2=0x7f230019beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2306800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23001035b0 0x7f230019beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.138+0000 7f2306800640 1 -- 172.21.15.144:0/3816428092 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23001a1e20 con 0x7f2300109840 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.139+0000 7f2306800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2300109840 0x7f23001a16f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00ef00 tx=0x7f22fc00c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.139+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc004b20 con 0x7f2300109840 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.139+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(23 keys) ==== 1014+0+0 (secure 0 0 0) 0x7f22fc003dd0 con 0x7f2300109840 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.139+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23001a2110 con 0x7f2300109840 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.139+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0046c0 con 0x7f2300109840 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.139+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23001a8870 con 0x7f2300109840 2024-07-26T10:50:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.140+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2300104830 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.143+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f22fc01d020 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.144+0000 7f22faa00640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22e80b3c10 0x7f22e80b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.144+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f22fc0e28e0 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.144+0000 7f2305e00640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22e80b3c10 0x7f22e80b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.145+0000 7f2305e00640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22e80b3c10 0x7f22e80b6120 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f22f40057c0 tx=0x7f22f4000bc0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.146+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f22fc024030 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.347+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f230019cfe0 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.354+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 7 ==== config(24 keys) ==== 1045+0+0 (secure 0 0 0) 0x7f22fc0aaea0 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.357+0000 7f22faa00640 1 -- 172.21.15.144:0/3816428092 <== mon.1 v2:172.21.15.183:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f22fc0ab390 con 0x7f2300109840 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22e80b3c10 msgr2=0x7f22e80b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22e80b3c10 0x7f22e80b6120 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f22f40057c0 tx=0x7f22f4000bc0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2300109840 msgr2=0x7f23001a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2300109840 0x7f23001a16f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00ef00 tx=0x7f22fc00c740 comp rx=0 tx=0).stop 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 shutdown_connections 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22e80b3c10 0x7f22e80b6120 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2300109840 0x7f23001a16f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2300108ef0 0x7f230019c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 --2- 172.21.15.144:0/3816428092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23001035b0 0x7f230019beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.360+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 >> 172.21.15.144:0/3816428092 conn(0x7f23000fefc0 msgr2=0x7f23001076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.361+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 shutdown_connections 2024-07-26T10:50:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:02.361+0000 7f2308800640 1 -- 172.21.15.144:0/3816428092 wait complete. 2024-07-26T10:50:06.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.795+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f18f410d660 con 0x7f18f410af30 2024-07-26T10:50:06.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.802+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 7 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18dc0aae50 con 0x7f18f410af30 2024-07-26T10:50:06.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.802+0000 7f18ebe00640 1 -- 172.21.15.144:0/4171602762 <== mon.1 v2:172.21.15.183:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f18dc0e1f00 con 0x7f18f410af30 2024-07-26T10:50:06.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18d80b3c10 msgr2=0x7f18d80b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18d80b3c10 0x7f18d80b6120 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f18e4000c00 tx=0x7f18e4005e80 comp rx=0 tx=0).stop 2024-07-26T10:50:06.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 msgr2=0x7f18f419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 0x7f18f419c410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00e9f0 tx=0x7f18dc00eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 shutdown_connections 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18d80b3c10 0x7f18d80b6120 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f410b880 0x7f18f41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18f410af30 0x7f18f419c410 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.805+0000 7f18f3e00640 1 --2- 172.21.15.144:0/4171602762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18f41055f0 0x7f18f419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.806+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 >> 172.21.15.144:0/4171602762 conn(0x7f18f4101060 msgr2=0x7f18f4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:06.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.806+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 shutdown_connections 2024-07-26T10:50:06.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:06.806+0000 7f18f3e00640 1 -- 172.21.15.144:0/4171602762 wait complete. 2024-07-26T10:50:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:06.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:06.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:06.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:06 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:06 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:07.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:07.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:07.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:07.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:07.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:07.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:07.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:06 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 -- 172.21.15.144:0/501185585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410f330 msgr2=0x7f7e5410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 --2- 172.21.15.144:0/501185585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410f330 0x7f7e5410f740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009860 tx=0x7f7e3c02a340 comp rx=0 tx=0).stop 2024-07-26T10:50:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 -- 172.21.15.144:0/501185585 shutdown_connections 2024-07-26T10:50:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 --2- 172.21.15.144:0/501185585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410fc80 0x7f7e541164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 --2- 172.21.15.144:0/501185585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410f330 0x7f7e5410f740 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 --2- 172.21.15.144:0/501185585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e54109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 -- 172.21.15.144:0/501185585 >> 172.21.15.144:0/501185585 conn(0x7f7e54078ff0 msgr2=0x7f7e54107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 -- 172.21.15.144:0/501185585 shutdown_connections 2024-07-26T10:50:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.482+0000 7f7e53200640 1 -- 172.21.15.144:0/501185585 wait complete. 2024-07-26T10:50:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.483+0000 7f7e53200640 1 Processor -- start 2024-07-26T10:50:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 -- start start 2024-07-26T10:50:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e541a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410f330 0x7f7e541a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410fc80 0x7f7e541a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7e54119940 con 0x7f7e5410f330 2024-07-26T10:50:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541197c0 con 0x7f7e5410fc80 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e53200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7e54119ac0 con 0x7f7e541099f0 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e51e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e541a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e51e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e541a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48920/0 (socket says 172.21.15.144:48920) 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e51e00640 1 -- 172.21.15.144:0/3554954317 learned_addr learned my addr 172.21.15.144:0/3554954317 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.484+0000 7f7e4be00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410f330 0x7f7e541a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e52800640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410fc80 0x7f7e541a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e51e00640 1 -- 172.21.15.144:0/3554954317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410fc80 msgr2=0x7f7e541a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e51e00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410fc80 0x7f7e541a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e51e00640 1 -- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410f330 msgr2=0x7f7e541a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e51e00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410f330 0x7f7e541a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e51e00640 1 -- 172.21.15.144:0/3554954317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e541a6360 con 0x7f7e541099f0 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e52800640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410fc80 0x7f7e541a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e4be00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410f330 0x7f7e541a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.485+0000 7f7e51e00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e541a03f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400e9f0 tx=0x7f7e4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.486+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4400ccc0 con 0x7f7e541099f0 2024-07-26T10:50:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.486+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e440054d0 con 0x7f7e541099f0 2024-07-26T10:50:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.486+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44010430 con 0x7f7e541099f0 2024-07-26T10:50:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.486+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e541a6650 con 0x7f7e541099f0 2024-07-26T10:50:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.486+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e541acdb0 con 0x7f7e541099f0 2024-07-26T10:50:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.487+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e5410ac70 con 0x7f7e541099f0 2024-07-26T10:50:07.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.490+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f7e440046c0 con 0x7f7e541099f0 2024-07-26T10:50:07.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.492+0000 7f7e4aa00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7e380b3c10 0x7f7e380b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.492+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7e440e2410 con 0x7f7e541099f0 2024-07-26T10:50:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.492+0000 7f7e4be00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7e380b3c10 0x7f7e380b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.493+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7e440ab5c0 con 0x7f7e541099f0 2024-07-26T10:50:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.493+0000 7f7e4be00640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7e380b3c10 0x7f7e380b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c035040 tx=0x7f7e3c030340 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:07.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.695+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f7e54109e00 con 0x7f7e541099f0 2024-07-26T10:50:07.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.705+0000 7f7e4aa00640 1 -- 172.21.15.144:0/3554954317 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v15)=0 v15) ==== 116+0+0 (secure 0 0 0) 0x7f7e54109e00 con 0x7f7e541099f0 2024-07-26T10:50:07.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7e380b3c10 msgr2=0x7f7e380b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:07.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7e380b3c10 0x7f7e380b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c035040 tx=0x7f7e3c030340 comp rx=0 tx=0).stop 2024-07-26T10:50:07.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 msgr2=0x7f7e541a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:07.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e541a03f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400e9f0 tx=0x7f7e4400eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 shutdown_connections 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7e380b3c10 0x7f7e380b6120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e5410fc80 0x7f7e541a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e5410f330 0x7f7e541a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 --2- 172.21.15.144:0/3554954317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e541099f0 0x7f7e541a03f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.709+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 >> 172.21.15.144:0/3554954317 conn(0x7f7e54078ff0 msgr2=0x7f7e5410d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.710+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 shutdown_connections 2024-07-26T10:50:07.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:07.710+0000 7f7e53200640 1 -- 172.21.15.144:0/3554954317 wait complete. 2024-07-26T10:50:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:07 smithi144 ceph-mon[32129]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:07 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:07 smithi160 ceph-mon[36715]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:07 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:07 smithi183 ceph-mon[37273]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:07 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:09.502 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:09 smithi144 ceph-mon[32129]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:09 smithi160 ceph-mon[36715]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:09 smithi183 ceph-mon[37273]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 -- 172.21.15.144:0/2131680533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c108d00 msgr2=0x7fc65c109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 --2- 172.21.15.144:0/2131680533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c108d00 0x7fc65c109110 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc648009860 tx=0x7fc64802a340 comp rx=0 tx=0).stop 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 -- 172.21.15.144:0/2131680533 shutdown_connections 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 --2- 172.21.15.144:0/2131680533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc65c109650 0x7fc65c10fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 --2- 172.21.15.144:0/2131680533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c108d00 0x7fc65c109110 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 --2- 172.21.15.144:0/2131680533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c1033c0 0x7fc65c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 -- 172.21.15.144:0/2131680533 >> 172.21.15.144:0/2131680533 conn(0x7fc65c0fee10 msgr2=0x7fc65c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.954+0000 7fc662000640 1 -- 172.21.15.144:0/2131680533 shutdown_connections 2024-07-26T10:50:11.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.955+0000 7fc662000640 1 -- 172.21.15.144:0/2131680533 wait complete. 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 Processor -- start 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 -- start start 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc65c1033c0 0x7fc65c19bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 0x7fc65c19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c109650 0x7fc65c1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c1132c0 con 0x7fc65c1033c0 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c113140 con 0x7fc65c109650 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.956+0000 7fc662000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c113440 con 0x7fc65c108d00 2024-07-26T10:50:11.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 0x7fc65c19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc661600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c109650 0x7fc65c1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 0x7fc65c19c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:45780/0 (socket says 172.21.15.144:45780) 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 -- 172.21.15.144:0/807176085 learned_addr learned my addr 172.21.15.144:0/807176085 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc660c00640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc65c1033c0 0x7fc65c19bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 -- 172.21.15.144:0/807176085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c109650 msgr2=0x7fc65c1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c109650 0x7fc65c1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 -- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc65c1033c0 msgr2=0x7fc65c19bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc65c1033c0 0x7fc65c19bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:11.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 -- 172.21.15.144:0/807176085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65c1a1e40 con 0x7fc65c108d00 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.957+0000 7fc65be00640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 0x7fc65c19c470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc648035040 tx=0x7fc648032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.958+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc648037040 con 0x7fc65c108d00 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.958+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc648004050 con 0x7fc65c108d00 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.958+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65c1a20d0 con 0x7fc65c108d00 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.958+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6480035f0 con 0x7fc65c108d00 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.958+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc65c1a8890 con 0x7fc65c108d00 2024-07-26T10:50:11.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.959+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc65c104640 con 0x7fc65c108d00 2024-07-26T10:50:11.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.961+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc648043020 con 0x7fc65c108d00 2024-07-26T10:50:11.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.963+0000 7fc659600640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6440b3c10 0x7fc6440b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:11.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.963+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc648101280 con 0x7fc65c108d00 2024-07-26T10:50:11.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.963+0000 7fc660c00640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6440b3c10 0x7fc6440b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:11.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.964+0000 7fc660c00640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6440b3c10 0x7fc6440b6120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc650005fd0 tx=0x7fc650009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:11.968 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:11.967+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc64804a060 con 0x7fc65c108d00 2024-07-26T10:50:12.134 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:11 smithi144 ceph-mon[32129]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:12.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.171+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fc65c1a8b70 con 0x7fc65c108d00 2024-07-26T10:50:12.182 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.180+0000 7fc659600640 1 -- 172.21.15.144:0/807176085 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v16)=0 v16) ==== 106+0+0 (secure 0 0 0) 0x7fc6480ca430 con 0x7fc65c108d00 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6440b3c10 msgr2=0x7fc6440b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6440b3c10 0x7fc6440b6120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc650005fd0 tx=0x7fc650009260 comp rx=0 tx=0).stop 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 msgr2=0x7fc65c19c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 0x7fc65c19c470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc648035040 tx=0x7fc648032d50 comp rx=0 tx=0).stop 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 shutdown_connections 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6440b3c10 0x7fc6440b6120 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc65c109650 0x7fc65c1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc65c108d00 0x7fc65c19c470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 --2- 172.21.15.144:0/807176085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc65c1033c0 0x7fc65c19bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.187 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 >> 172.21.15.144:0/807176085 conn(0x7fc65c0fee10 msgr2=0x7fc65c10e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:12.187 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.184+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 shutdown_connections 2024-07-26T10:50:12.187 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.185+0000 7fc662000640 1 -- 172.21.15.144:0/807176085 wait complete. 2024-07-26T10:50:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:11 smithi160 ceph-mon[36715]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:12.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 -- 172.21.15.144:0/4158412092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22281041f0 msgr2=0x7f2228104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 --2- 172.21.15.144:0/4158412092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22281041f0 0x7f2228104600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2218009ca0 tx=0x7f221802a8c0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 -- 172.21.15.144:0/4158412092 shutdown_connections 2024-07-26T10:50:12.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 --2- 172.21.15.144:0/4158412092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 0x7f222810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 --2- 172.21.15.144:0/4158412092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2228104b40 0x7f222810d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 --2- 172.21.15.144:0/4158412092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22281041f0 0x7f2228104600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 -- 172.21.15.144:0/4158412092 >> 172.21.15.144:0/4158412092 conn(0x7f22280fdcf0 msgr2=0x7f2228100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:12.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 -- 172.21.15.144:0/4158412092 shutdown_connections 2024-07-26T10:50:12.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.333+0000 7f222e000640 1 -- 172.21.15.144:0/4158412092 wait complete. 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.334+0000 7f222e000640 1 Processor -- start 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 -- start start 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22281041f0 0x7f222819bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 0x7f222819c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 0x7f22281a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f22281132c0 con 0x7f2228104b40 2024-07-26T10:50:12.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2228113140 con 0x7f222810d650 2024-07-26T10:50:12.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222e000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2228113440 con 0x7f22281041f0 2024-07-26T10:50:12.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f221fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 0x7f222819c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:12.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22281041f0 0x7f222819bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:12.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f221fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 0x7f222819c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46238/0 (socket says 172.21.15.144:46238) 2024-07-26T10:50:12.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f221fe00640 1 -- 172.21.15.144:0/2708626124 learned_addr learned my addr 172.21.15.144:0/2708626124 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:12.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.335+0000 7f222d600640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 0x7f22281a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f221fe00640 1 -- 172.21.15.144:0/2708626124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22281041f0 msgr2=0x7f222819bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f221fe00640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22281041f0 0x7f222819bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f221fe00640 1 -- 172.21.15.144:0/2708626124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 msgr2=0x7f22281a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f221fe00640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 0x7f22281a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f221fe00640 1 -- 172.21.15.144:0/2708626124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22281a1e80 con 0x7f2228104b40 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f222d600640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 0x7f22281a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.336+0000 7f221fe00640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 0x7f222819c4b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f221000e9f0 tx=0x7f221000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.337+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000ccc0 con 0x7f2228104b40 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.337+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22100108e0 con 0x7f2228104b40 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.337+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210018860 con 0x7f2228104b40 2024-07-26T10:50:12.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.337+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22281a2170 con 0x7f2228104b40 2024-07-26T10:50:12.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.337+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22281a88d0 con 0x7f2228104b40 2024-07-26T10:50:12.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.338+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2228104600 con 0x7f2228104b40 2024-07-26T10:50:12.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.342+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f2210010430 con 0x7f2228104b40 2024-07-26T10:50:12.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.344+0000 7f2226000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f220c0b3b90 0x7f220c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:12.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.344+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f22100e2760 con 0x7f2228104b40 2024-07-26T10:50:12.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.344+0000 7f222cc00640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f220c0b3b90 0x7f220c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:12.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.345+0000 7f222cc00640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f220c0b3b90 0x7f220c0b60a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2218005e30 tx=0x7f2218009360 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:12.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.346+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f22100ab360 con 0x7f2228104b40 2024-07-26T10:50:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:11 smithi183 ceph-mon[37273]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:12.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.544+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f22281a8b60 con 0x7f2228104b40 2024-07-26T10:50:12.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.552+0000 7f2226000640 1 -- 172.21.15.144:0/2708626124 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v17)=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f22100af8b0 con 0x7f2228104b40 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f220c0b3b90 msgr2=0x7f220c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f220c0b3b90 0x7f220c0b60a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2218005e30 tx=0x7f2218009360 comp rx=0 tx=0).stop 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 msgr2=0x7f222819c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 0x7f222819c4b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f221000e9f0 tx=0x7f221000eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 shutdown_connections 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f220c0b3b90 0x7f220c0b60a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f222810d650 0x7f22281a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2228104b40 0x7f222819c4b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 --2- 172.21.15.144:0/2708626124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22281041f0 0x7f222819bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 >> 172.21.15.144:0/2708626124 conn(0x7f22280fdcf0 msgr2=0x7f2228101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 shutdown_connections 2024-07-26T10:50:12.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:12.555+0000 7f222e000640 1 -- 172.21.15.144:0/2708626124 wait complete. 2024-07-26T10:50:13.383 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:13.390 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:13 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:13.390 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:13 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2708626124' entity='client.admin' 2024-07-26T10:50:13.390 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:13 smithi160 ceph-mon[36715]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:13 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:13 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2708626124' entity='client.admin' 2024-07-26T10:50:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:13 smithi144 ceph-mon[32129]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:13 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:13 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2708626124' entity='client.admin' 2024-07-26T10:50:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:13 smithi183 ceph-mon[37273]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:15 smithi183 ceph-mon[37273]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:16.202 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:15 smithi144 ceph-mon[32129]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:15 smithi160 ceph-mon[36715]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:17.046 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi144') 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.665+0000 7f634f200640 1 -- 172.21.15.144:0/1243150989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350101fe0 msgr2=0x7f63501023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.665+0000 7f634f200640 1 --2- 172.21.15.144:0/1243150989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350101fe0 0x7f63501023f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6340009890 tx=0x7f634002a380 comp rx=0 tx=0).stop 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.665+0000 7f634f200640 1 -- 172.21.15.144:0/1243150989 shutdown_connections 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.665+0000 7f634f200640 1 --2- 172.21.15.144:0/1243150989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350107420 0x7f6350107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.666+0000 7f634f200640 1 --2- 172.21.15.144:0/1243150989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350102930 0x7f6350106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.666+0000 7f634f200640 1 --2- 172.21.15.144:0/1243150989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350101fe0 0x7f63501023f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.666+0000 7f634f200640 1 -- 172.21.15.144:0/1243150989 >> 172.21.15.144:0/1243150989 conn(0x7f63500fdcf0 msgr2=0x7f6350100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.666+0000 7f634f200640 1 -- 172.21.15.144:0/1243150989 shutdown_connections 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.666+0000 7f634f200640 1 -- 172.21.15.144:0/1243150989 wait complete. 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 Processor -- start 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 -- start start 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 0x7f63501a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350102930 0x7f63501a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350107420 0x7f63501a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f63501198e0 con 0x7f6350101fe0 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6350119760 con 0x7f6350107420 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6350119a60 con 0x7f6350102930 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.667+0000 7f634de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 0x7f63501a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 0x7f63501a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46272/0 (socket says 172.21.15.144:46272) 2024-07-26T10:50:17.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 -- 172.21.15.144:0/100474919 learned_addr learned my addr 172.21.15.144:0/100474919 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f6347e00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350102930 0x7f63501a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634e800640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350107420 0x7f63501a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 -- 172.21.15.144:0/100474919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350102930 msgr2=0x7f63501a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350102930 0x7f63501a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 -- 172.21.15.144:0/100474919 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350107420 msgr2=0x7f63501a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350107420 0x7f63501a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 -- 172.21.15.144:0/100474919 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63501a62a0 con 0x7f6350101fe0 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f6347e00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350102930 0x7f63501a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.668+0000 7f634de00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 0x7f63501a0330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6340035040 tx=0x7f6340032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.669+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340037040 con 0x7f6350101fe0 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.669+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6340041070 con 0x7f6350101fe0 2024-07-26T10:50:17.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.669+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63501a6530 con 0x7f6350101fe0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.669+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340003850 con 0x7f6350101fe0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.669+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63501accf0 con 0x7f6350101fe0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.670+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f635006af90 con 0x7f6350101fe0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.673+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f63400039f0 con 0x7f6350101fe0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.674+0000 7f6346a00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f63340b3c10 0x7f63340b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.674+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6340102540 con 0x7f6350101fe0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.674+0000 7f6347e00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f63340b3c10 0x7f63340b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.675+0000 7f6347e00640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f63340b3c10 0x7f63340b6120 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6350107a20 tx=0x7f6338009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:17.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.676+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f63400cb770 con 0x7f6350101fe0 2024-07-26T10:50:17.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.776+0000 7f14bca00640 1 -- 172.21.15.144:0/1519051650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8107be0 msgr2=0x7f14b8107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.776+0000 7f14bca00640 1 --2- 172.21.15.144:0/1519051650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8107be0 0x7f14b8107ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009860 tx=0x7f14a402a340 comp rx=0 tx=0).stop 2024-07-26T10:50:17.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 -- 172.21.15.144:0/1519051650 shutdown_connections 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 --2- 172.21.15.144:0/1519051650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8108530 0x7f14b810ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 --2- 172.21.15.144:0/1519051650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8107be0 0x7f14b8107ff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 --2- 172.21.15.144:0/1519051650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b81026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 -- 172.21.15.144:0/1519051650 >> 172.21.15.144:0/1519051650 conn(0x7f14b80fdcf0 msgr2=0x7f14b8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 -- 172.21.15.144:0/1519051650 shutdown_connections 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.777+0000 7f14bca00640 1 -- 172.21.15.144:0/1519051650 wait complete. 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.778+0000 7f14bca00640 1 Processor -- start 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 -- start start 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b8199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8107be0 0x7f14b819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8108530 0x7f14b819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f14b81121a0 con 0x7f14b81022a0 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f14b8112020 con 0x7f14b8107be0 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.779+0000 7f14bca00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f14b8112320 con 0x7f14b8108530 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b8199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b8199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46288/0 (socket says 172.21.15.144:46288) 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 -- 172.21.15.144:0/1923103640 learned_addr learned my addr 172.21.15.144:0/1923103640 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b5e00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8107be0 0x7f14b819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b7200640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8108530 0x7f14b819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 -- 172.21.15.144:0/1923103640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8108530 msgr2=0x7f14b819f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8108530 0x7f14b819f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 -- 172.21.15.144:0/1923103640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8107be0 msgr2=0x7f14b819a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8107be0 0x7f14b819a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b6800640 1 -- 172.21.15.144:0/1923103640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b819fcd0 con 0x7f14b81022a0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.780+0000 7f14b5e00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8107be0 0x7f14b819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.781+0000 7f14b6800640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b8199d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14a800e9c0 tx=0x7f14a800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.781+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a800cd80 con 0x7f14b81022a0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.781+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a80035f0 con 0x7f14b81022a0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.781+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8010690 con 0x7f14b81022a0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.781+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b819ffc0 con 0x7f14b81022a0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.781+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b81a6720 con 0x7f14b81022a0 2024-07-26T10:50:17.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.782+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b8103520 con 0x7f14b81022a0 2024-07-26T10:50:17.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.786+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f14a8004000 con 0x7f14b81022a0 2024-07-26T10:50:17.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.789+0000 7f14af400640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f149c0b3b90 0x7f149c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:17.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.789+0000 7f14b5e00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f149c0b3b90 0x7f149c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:17.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.789+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f14a80e2420 con 0x7f14b81022a0 2024-07-26T10:50:17.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.789+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14a80a71f0 con 0x7f14b81022a0 2024-07-26T10:50:17.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.790+0000 7f14b5e00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f149c0b3b90 0x7f149c0b60a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f14a4005940 tx=0x7f14a40058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:17.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.890+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f6350106e60 con 0x7f6350101fe0 2024-07-26T10:50:17.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.896+0000 7f6346a00640 1 -- 172.21.15.144:0/100474919 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v19)=0 v19) ==== 122+0+0 (secure 0 0 0) 0x7f63400cb070 con 0x7f6350101fe0 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f63340b3c10 msgr2=0x7f63340b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f63340b3c10 0x7f63340b6120 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6350107a20 tx=0x7f6338009260 comp rx=0 tx=0).stop 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 msgr2=0x7f63501a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 0x7f63501a0330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6340035040 tx=0x7f6340032e90 comp rx=0 tx=0).stop 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 shutdown_connections 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f63340b3c10 0x7f63340b6120 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6350107420 0x7f63501a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6350102930 0x7f63501a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 --2- 172.21.15.144:0/100474919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6350101fe0 0x7f63501a0330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:17.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 >> 172.21.15.144:0/100474919 conn(0x7f63500fdcf0 msgr2=0x7f6350104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:17.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 shutdown_connections 2024-07-26T10:50:17.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.899+0000 7f634f200640 1 -- 172.21.15.144:0/100474919 wait complete. 2024-07-26T10:50:18.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.997+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f14b81a6950 con 0x7f14b81022a0 2024-07-26T10:50:18.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.005+0000 7f14af400640 1 -- 172.21.15.144:0/1923103640 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v20)=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f14a80ab660 con 0x7f14b81022a0 2024-07-26T10:50:18.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f149c0b3b90 msgr2=0x7f149c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:18.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f149c0b3b90 0x7f149c0b60a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f14a4005940 tx=0x7f14a40058d0 comp rx=0 tx=0).stop 2024-07-26T10:50:18.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 msgr2=0x7f14b8199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:18.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b8199d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14a800e9c0 tx=0x7f14a800edc0 comp rx=0 tx=0).stop 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 shutdown_connections 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f149c0b3b90 0x7f149c0b60a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b8108530 0x7f14b819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14b8107be0 0x7f14b819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.008+0000 7f14bca00640 1 --2- 172.21.15.144:0/1923103640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14b81022a0 0x7f14b8199d60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.009+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 >> 172.21.15.144:0/1923103640 conn(0x7f14b80fdcf0 msgr2=0x7f14b81062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.009+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 shutdown_connections 2024-07-26T10:50:18.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:18.009+0000 7f14bca00640 1 -- 172.21.15.144:0/1923103640 wait complete. 2024-07-26T10:50:18.038 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:17 smithi144 ceph-mon[32129]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:18.039 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:17 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:17 smithi160 ceph-mon[36715]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:17 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:17 smithi183 ceph-mon[37273]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:17 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:19.143 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:18 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/100474919' entity='client.admin' 2024-07-26T10:50:19.143 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:18 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1923103640' entity='client.admin' 2024-07-26T10:50:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:18 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/100474919' entity='client.admin' 2024-07-26T10:50:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:18 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1923103640' entity='client.admin' 2024-07-26T10:50:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:18 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/100474919' entity='client.admin' 2024-07-26T10:50:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:18 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1923103640' entity='client.admin' 2024-07-26T10:50:19.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 -- 172.21.15.144:0/2097424011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 msgr2=0x7f82481059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:19.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 --2- 172.21.15.144:0/2097424011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f82481059e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8238009900 tx=0x7f823802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 -- 172.21.15.144:0/2097424011 shutdown_connections 2024-07-26T10:50:19.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 --2- 172.21.15.144:0/2097424011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810b860 0x7f8248112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 --2- 172.21.15.144:0/2097424011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810af10 0x7f824810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 --2- 172.21.15.144:0/2097424011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f82481059e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 -- 172.21.15.144:0/2097424011 >> 172.21.15.144:0/2097424011 conn(0x7f8248101040 msgr2=0x7f8248103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 -- 172.21.15.144:0/2097424011 shutdown_connections 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.090+0000 7f8247200640 1 -- 172.21.15.144:0/2097424011 wait complete. 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.091+0000 7f8247200640 1 Processor -- start 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.091+0000 7f8247200640 1 -- start start 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8247200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f824819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8247200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810af10 0x7f824819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8247200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 0x7f82481a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8247200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8248115530 con 0x7f82481055d0 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8247200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f82481153b0 con 0x7f824810af10 2024-07-26T10:50:19.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8247200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82481156b0 con 0x7f824810b860 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8245e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f824819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 0x7f82481a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8245e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f824819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46254/0 (socket says 172.21.15.144:46254) 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 0x7f82481a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:45842/0 (socket says 172.21.15.144:45842) 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8245e00640 1 -- 172.21.15.144:0/2742710460 learned_addr learned my addr 172.21.15.144:0/2742710460 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f823fe00640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810af10 0x7f824819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 -- 172.21.15.144:0/2742710460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810af10 msgr2=0x7f824819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810af10 0x7f824819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 -- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 msgr2=0x7f824819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f824819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f8246800640 1 -- 172.21.15.144:0/2742710460 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82481a1eb0 con 0x7f824810b860 2024-07-26T10:50:19.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.092+0000 7f823fe00640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810af10 0x7f824819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.093+0000 7f8246800640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 0x7f82481a1780 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8240009fd0 tx=0x7f824000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.093+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824000ce20 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.093+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8240005020 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.093+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82481a21a0 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.093+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240010690 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.093+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82481a8900 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.094+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82481059e0 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.097+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f82400054d0 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.098+0000 7f823ea00640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f822c0b3b90 0x7f822c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.098+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f82400e2770 con 0x7f824810b860 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.098+0000 7f8245e00640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f822c0b3b90 0x7f822c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.099+0000 7f8245e00640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f822c0b3b90 0x7f822c0b60a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8238031440 tx=0x7f82380058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:19.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.102+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f82400ab920 con 0x7f824810b860 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.308+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f824810d050 con 0x7f824810b860 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.319+0000 7f823ea00640 1 -- 172.21.15.144:0/2742710460 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v18)=0 v18) ==== 120+0+0 (secure 0 0 0) 0x7f82400ab220 con 0x7f824810b860 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f822c0b3b90 msgr2=0x7f822c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f822c0b3b90 0x7f822c0b60a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8238031440 tx=0x7f82380058d0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 msgr2=0x7f82481a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 0x7f82481a1780 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8240009fd0 tx=0x7f824000ef10 comp rx=0 tx=0).stop 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 shutdown_connections 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f822c0b3b90 0x7f822c0b60a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f824810b860 0x7f82481a1780 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f824810af10 0x7f824819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 --2- 172.21.15.144:0/2742710460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82481055d0 0x7f824819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 >> 172.21.15.144:0/2742710460 conn(0x7f8248101040 msgr2=0x7f8248103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:19.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 shutdown_connections 2024-07-26T10:50:19.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:17.322+0000 7f8247200640 1 -- 172.21.15.144:0/2742710460 wait complete. 2024-07-26T10:50:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:20 smithi160 ceph-mon[36715]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:20 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/946673340' entity='client.admin' 2024-07-26T10:50:20.320 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:20 smithi144 ceph-mon[32129]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:20.320 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:20 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/946673340' entity='client.admin' 2024-07-26T10:50:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:20 smithi183 ceph-mon[37273]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:20 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/946673340' entity='client.admin' 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.005+0000 7f9e96000640 1 -- 172.21.15.144:0/1896564888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 msgr2=0x7f9e90109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.005+0000 7f9e96000640 1 --2- 172.21.15.144:0/1896564888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e90109130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009900 tx=0x7f9e7802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 -- 172.21.15.144:0/1896564888 shutdown_connections 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 --2- 172.21.15.144:0/1896564888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e90109670 0x7f9e9010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 --2- 172.21.15.144:0/1896564888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e90109130 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 --2- 172.21.15.144:0/1896564888 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e901037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 -- 172.21.15.144:0/1896564888 >> 172.21.15.144:0/1896564888 conn(0x7f9e900fee30 msgr2=0x7f9e90101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 -- 172.21.15.144:0/1896564888 shutdown_connections 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.006+0000 7f9e96000640 1 -- 172.21.15.144:0/1896564888 wait complete. 2024-07-26T10:50:21.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.007+0000 7f9e96000640 1 Processor -- start 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.007+0000 7f9e96000640 1 -- start start 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e96000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e9019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e96000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e9019c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e96000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e90109670 0x7f9e901a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e96000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9e90113330 con 0x7f9e90108d20 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e96000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901131b0 con 0x7f9e901033e0 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e96000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901134b0 con 0x7f9e90109670 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e9019c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e94c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e9019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e94c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e9019bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54130/0 (socket says 172.21.15.144:54130) 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e9019c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43638/0 (socket says 172.21.15.144:43638) 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e94c00640 1 -- 172.21.15.144:0/2135052175 learned_addr learned my addr 172.21.15.144:0/2135052175 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:21.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.008+0000 7f9e95600640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e90109670 0x7f9e901a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.009+0000 7f9e94c00640 1 -- 172.21.15.144:0/2135052175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e90109670 msgr2=0x7f9e901a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.009+0000 7f9e94c00640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e90109670 0x7f9e901a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.009+0000 7f9e94c00640 1 -- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 msgr2=0x7f9e9019c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.009+0000 7f9e94c00640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e9019c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.009+0000 7f9e94c00640 1 -- 172.21.15.144:0/2135052175 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e901a1f50 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.009+0000 7f9e94c00640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e9019bfe0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000ebc0 tx=0x7f9e8000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.010+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80010430 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.010+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e80010a80 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.010+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80018a50 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.010+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e901a2240 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.010+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e901a89a0 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.012+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e901037f0 con 0x7f9e901033e0 2024-07-26T10:50:21.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.014+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f9e80018bf0 con 0x7f9e901033e0 2024-07-26T10:50:21.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.015+0000 7f9e8e000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e740b3c10 0x7f9e740b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:21.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.015+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f9e800e2730 con 0x7f9e901033e0 2024-07-26T10:50:21.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.016+0000 7f9e87e00640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e740b3c10 0x7f9e740b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:21.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.016+0000 7f9e87e00640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e740b3c10 0x7f9e740b6120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9e7802a9e0 tx=0x7f9e78005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:21.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.019+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9e800ab8e0 con 0x7f9e901033e0 2024-07-26T10:50:21.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.228+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f9e901a8c00 con 0x7f9e901033e0 2024-07-26T10:50:21.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.239+0000 7f9e8e000640 1 -- 172.21.15.144:0/2135052175 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v23)=0 v23) ==== 119+0+0 (secure 0 0 0) 0x7f9e800ab1e0 con 0x7f9e901033e0 2024-07-26T10:50:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e740b3c10 msgr2=0x7f9e740b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e740b3c10 0x7f9e740b6120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9e7802a9e0 tx=0x7f9e78005a20 comp rx=0 tx=0).stop 2024-07-26T10:50:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 msgr2=0x7f9e9019bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e9019bfe0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000ebc0 tx=0x7f9e8000c2d0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 shutdown_connections 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e740b3c10 0x7f9e740b6120 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e90109670 0x7f9e901a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e90108d20 0x7f9e9019c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 --2- 172.21.15.144:0/2135052175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e901033e0 0x7f9e9019bfe0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 >> 172.21.15.144:0/2135052175 conn(0x7f9e900fee30 msgr2=0x7f9e90108400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:21.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 shutdown_connections 2024-07-26T10:50:21.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.243+0000 7f9e96000640 1 -- 172.21.15.144:0/2135052175 wait complete. 2024-07-26T10:50:22.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:22 smithi144 ceph-mon[32129]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:22.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:22 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/226495270' entity='client.admin' 2024-07-26T10:50:22.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:22 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:22 smithi160 ceph-mon[36715]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:22 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/226495270' entity='client.admin' 2024-07-26T10:50:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:22 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:22 smithi183 ceph-mon[37273]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:22 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/226495270' entity='client.admin' 2024-07-26T10:50:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:22 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:22.695 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi144') 2024-07-26T10:50:22.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 -- 172.21.15.144:0/3043105211 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 msgr2=0x7f2c38112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 --2- 172.21.15.144:0/3043105211 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c38112270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009ca0 tx=0x7f2c3002a8b0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 -- 172.21.15.144:0/3043105211 shutdown_connections 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 --2- 172.21.15.144:0/3043105211 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c38112270 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 --2- 172.21.15.144:0/3043105211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3810b0f0 0x7f2c3810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 --2- 172.21.15.144:0/3043105211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c381057b0 0x7f2c38105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 -- 172.21.15.144:0/3043105211 >> 172.21.15.144:0/3043105211 conn(0x7f2c38101200 msgr2=0x7f2c38103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 -- 172.21.15.144:0/3043105211 shutdown_connections 2024-07-26T10:50:22.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.739+0000 7f2c3f000640 1 -- 172.21.15.144:0/3043105211 wait complete. 2024-07-26T10:50:22.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.740+0000 7f2c3f000640 1 Processor -- start 2024-07-26T10:50:22.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 -- start start 2024-07-26T10:50:22.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c381057b0 0x7f2c3819c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 0x7f2c3819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c381a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2c38115740 con 0x7f2c381057b0 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381155c0 con 0x7f2c3810ba40 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3f000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381158c0 con 0x7f2c3810b0f0 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c37e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c381057b0 0x7f2c3819c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c2f400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 0x7f2c3819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.741+0000 7f2c3d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c381a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c3d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c381a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54158/0 (socket says 172.21.15.144:54158) 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 0x7f2c3819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35562/0 (socket says 172.21.15.144:35562) 2024-07-26T10:50:22.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c3d000640 1 -- 172.21.15.144:0/2403852462 learned_addr learned my addr 172.21.15.144:0/2403852462 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:22.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 -- 172.21.15.144:0/2403852462 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 msgr2=0x7f2c381a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c381a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 -- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c381057b0 msgr2=0x7f2c3819c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c381057b0 0x7f2c3819c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 -- 172.21.15.144:0/2403852462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c381a2000 con 0x7f2c3810b0f0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c3d000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c381a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.742+0000 7f2c2f400640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 0x7f2c3819c630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2c3806c4f0 tx=0x7f2c2000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.743+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000cc40 con 0x7f2c3810b0f0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.743+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c2000cde0 con 0x7f2c3810b0f0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.743+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c381a22f0 con 0x7f2c3810b0f0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.743+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20018780 con 0x7f2c3810b0f0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.743+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c381a8a50 con 0x7f2c3810b0f0 2024-07-26T10:50:22.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.744+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c38105bc0 con 0x7f2c3810b0f0 2024-07-26T10:50:22.749 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.746+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f2c20010a90 con 0x7f2c3810b0f0 2024-07-26T10:50:22.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.748+0000 7f2c35600640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c1c0b3c10 0x7f2c1c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.748+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2c200e2200 con 0x7f2c3810b0f0 2024-07-26T10:50:22.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.749+0000 7f2c37e00640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c1c0b3c10 0x7f2c1c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.752 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.749+0000 7f2c37e00640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c1c0b3c10 0x7f2c1c0b6120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009bd0 tx=0x7f2c28005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:22.754 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.751+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2c200aae00 con 0x7f2c3810b0f0 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 -- 172.21.15.144:0/1251440744 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f44104080 msgr2=0x7f1f44104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 --2- 172.21.15.144:0/1251440744 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f44104080 0x7f1f44104490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1f34009be0 tx=0x7f1f3402a6d0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 -- 172.21.15.144:0/1251440744 shutdown_connections 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 --2- 172.21.15.144:0/1251440744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f4410d630 0x7f1f4410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 --2- 172.21.15.144:0/1251440744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f44104ab0 0x7f1f4410d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 --2- 172.21.15.144:0/1251440744 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f44104080 0x7f1f44104490 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 -- 172.21.15.144:0/1251440744 >> 172.21.15.144:0/1251440744 conn(0x7f1f440fdcf0 msgr2=0x7f1f44100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 -- 172.21.15.144:0/1251440744 shutdown_connections 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.818+0000 7f1f4a600640 1 -- 172.21.15.144:0/1251440744 wait complete. 2024-07-26T10:50:22.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.819+0000 7f1f4a600640 1 Processor -- start 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 -- start start 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 0x7f1f4419bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f44104ab0 0x7f1f4419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f4410d630 0x7f1f441a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441132c0 con 0x7f1f44104080 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44113140 con 0x7f1f4410d630 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.820+0000 7f1f4a600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44113440 con 0x7f1f44104ab0 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 0x7f1f4419bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f43e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f44104ab0 0x7f1f4419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 0x7f1f4419bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43634/0 (socket says 172.21.15.144:43634) 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 -- 172.21.15.144:0/226495270 learned_addr learned my addr 172.21.15.144:0/226495270 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49c00640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f4410d630 0x7f1f441a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 -- 172.21.15.144:0/226495270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f44104ab0 msgr2=0x7f1f4419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f44104ab0 0x7f1f4419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 -- 172.21.15.144:0/226495270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f4410d630 msgr2=0x7f1f441a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f4410d630 0x7f1f441a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49200640 1 -- 172.21.15.144:0/226495270 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f441a1ea0 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.821+0000 7f1f49c00640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f4410d630 0x7f1f441a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.822+0000 7f1f49200640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 0x7f1f4419bf30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f3402acc0 tx=0x7f1f34003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.822+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f34041020 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.822+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f340318f0 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.822+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f441a2130 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.822+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f441a88f0 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.822+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f34036040 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.824+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f200052c0 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.827+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f1f34044020 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.828+0000 7f1f41600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f280b3c10 0x7f1f280b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.829+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1f34103090 con 0x7f1f44104080 2024-07-26T10:50:22.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.829+0000 7f1f43e00640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f280b3c10 0x7f1f280b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.830+0000 7f1f43e00640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f280b3c10 0x7f1f280b6120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f441014d0 tx=0x7f1f2c009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:20.835+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1f3404b030 con 0x7f1f44104080 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.047+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f1f200055b0 con 0x7f1f44104080 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.054+0000 7f1f41600640 1 -- 172.21.15.144:0/226495270 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v22)=0 v22) ==== 116+0+0 (secure 0 0 0) 0x7f1f340cc240 con 0x7f1f44104080 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f280b3c10 msgr2=0x7f1f280b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f280b3c10 0x7f1f280b6120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f441014d0 tx=0x7f1f2c009260 comp rx=0 tx=0).stop 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 msgr2=0x7f1f4419bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 0x7f1f4419bf30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f3402acc0 tx=0x7f1f34003bf0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 shutdown_connections 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f280b3c10 0x7f1f280b6120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f4410d630 0x7f1f441a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f44104ab0 0x7f1f4419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 --2- 172.21.15.144:0/226495270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f44104080 0x7f1f4419bf30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.056+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 >> 172.21.15.144:0/226495270 conn(0x7f1f440fdcf0 msgr2=0x7f1f44101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:22.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.057+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 shutdown_connections 2024-07-26T10:50:22.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:21.057+0000 7f1f4a600640 1 -- 172.21.15.144:0/226495270 wait complete. 2024-07-26T10:50:22.953 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.950+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f2c3802e0c0 con 0x7f2c3810b0f0 2024-07-26T10:50:22.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.958+0000 7f2c35600640 1 -- 172.21.15.144:0/2403852462 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v25)=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f2c200af350 con 0x7f2c3810b0f0 2024-07-26T10:50:22.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c1c0b3c10 msgr2=0x7f2c1c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c1c0b3c10 0x7f2c1c0b6120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009bd0 tx=0x7f2c28005f90 comp rx=0 tx=0).stop 2024-07-26T10:50:22.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 msgr2=0x7f2c3819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:22.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 0x7f2c3819c630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2c3806c4f0 tx=0x7f2c2000ee50 comp rx=0 tx=0).stop 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 shutdown_connections 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c1c0b3c10 0x7f2c1c0b6120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c3810ba40 0x7f2c381a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3810b0f0 0x7f2c3819c630 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 --2- 172.21.15.144:0/2403852462 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c381057b0 0x7f2c3819c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 >> 172.21.15.144:0/2403852462 conn(0x7f2c38101200 msgr2=0x7f2c3810a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.961+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 shutdown_connections 2024-07-26T10:50:22.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.962+0000 7f2c3f000640 1 -- 172.21.15.144:0/2403852462 wait complete. 2024-07-26T10:50:23.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.621+0000 7ff089e00640 1 -- 172.21.15.144:0/3628810133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 msgr2=0x7ff084109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:23.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 --2- 172.21.15.144:0/3628810133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff084109e00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff078009900 tx=0x7ff07802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 -- 172.21.15.144:0/3628810133 shutdown_connections 2024-07-26T10:50:23.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 --2- 172.21.15.144:0/3628810133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410fc80 0x7ff0841164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 --2- 172.21.15.144:0/3628810133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff08410f330 0x7ff08410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 --2- 172.21.15.144:0/3628810133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff084109e00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 -- 172.21.15.144:0/3628810133 >> 172.21.15.144:0/3628810133 conn(0x7ff084078ff0 msgr2=0x7ff084107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 -- 172.21.15.144:0/3628810133 shutdown_connections 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.622+0000 7ff089e00640 1 -- 172.21.15.144:0/3628810133 wait complete. 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.623+0000 7ff089e00640 1 Processor -- start 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.623+0000 7ff089e00640 1 -- start start 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.623+0000 7ff089e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff0841a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff089e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 0x7ff0841a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff089e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff08410fc80 0x7ff0841a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff089e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff084119940 con 0x7ff08410f330 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff089e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0841197c0 con 0x7ff0841099f0 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff089e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff084119ac0 con 0x7ff08410fc80 2024-07-26T10:50:23.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 0x7ff0841a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff088a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff0841a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 0x7ff0841a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46308/0 (socket says 172.21.15.144:46308) 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 -- 172.21.15.144:0/946673340 learned_addr learned my addr 172.21.15.144:0/946673340 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff089400640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff08410fc80 0x7ff0841a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 -- 172.21.15.144:0/946673340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff08410fc80 msgr2=0x7ff0841a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff08410fc80 0x7ff0841a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 -- 172.21.15.144:0/946673340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 msgr2=0x7ff0841a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff0841a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff083e00640 1 -- 172.21.15.144:0/946673340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0841a62c0 con 0x7ff08410f330 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.624+0000 7ff088a00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff0841a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.625+0000 7ff083e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 0x7ff0841a08f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff07000e9f0 tx=0x7ff07000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.625+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07000ccc0 con 0x7ff08410f330 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.625+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0700108e0 con 0x7ff08410f330 2024-07-26T10:50:23.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.625+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070018860 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.625+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0841a65b0 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.625+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0841acd10 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.626+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff084109e00 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.629+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7ff070010430 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.631+0000 7ff081600640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff06c0b3b90 0x7ff06c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.631+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff0700e2840 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.631+0000 7ff088a00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff06c0b3b90 0x7ff06c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.632+0000 7ff088a00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff06c0b3b90 0x7ff06c0b60a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff078031440 tx=0x7ff078032ed0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.632+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff0700ab3a0 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.840+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7ff084111ab0 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.846+0000 7ff081600640 1 -- 172.21.15.144:0/946673340 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v21)=0 v21) ==== 127+0+0 (secure 0 0 0) 0x7ff0700af8f0 con 0x7ff08410f330 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff06c0b3b90 msgr2=0x7ff06c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff06c0b3b90 0x7ff06c0b60a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff078031440 tx=0x7ff078032ed0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 msgr2=0x7ff0841a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:23.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 0x7ff0841a08f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff07000e9f0 tx=0x7ff07000eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 shutdown_connections 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff06c0b3b90 0x7ff06c0b60a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff08410fc80 0x7ff0841a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08410f330 0x7ff0841a08f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 --2- 172.21.15.144:0/946673340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0841099f0 0x7ff0841a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.850+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 >> 172.21.15.144:0/946673340 conn(0x7ff084078ff0 msgr2=0x7ff084079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.851+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 shutdown_connections 2024-07-26T10:50:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:19.851+0000 7ff089e00640 1 -- 172.21.15.144:0/946673340 wait complete. 2024-07-26T10:50:23.704 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:23 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:23.704 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:23 smithi144 ceph-mon[32129]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:23.704 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:23 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:23 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:23 smithi160 ceph-mon[36715]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:23 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:23 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:23 smithi183 ceph-mon[37273]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:23 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:24.184 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:24.185 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:24.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 -- 172.21.15.144:0/3574160131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 msgr2=0x7f26f01022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 --2- 172.21.15.144:0/3574160131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f01022a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009860 tx=0x7f26e002a3d0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 -- 172.21.15.144:0/3574160131 shutdown_connections 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 --2- 172.21.15.144:0/3574160131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 0x7f26f0111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 --2- 172.21.15.144:0/3574160131 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f01027e0 0x7f26f010f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 --2- 172.21.15.144:0/3574160131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f01022a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 -- 172.21.15.144:0/3574160131 >> 172.21.15.144:0/3574160131 conn(0x7f26f00fdcf0 msgr2=0x7f26f0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 -- 172.21.15.144:0/3574160131 shutdown_connections 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.146+0000 7f26ef200640 1 -- 172.21.15.144:0/3574160131 wait complete. 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 Processor -- start 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 -- start start 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f019bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:24.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f01027e0 0x7f26f019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 0x7f26f01a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01154c0 con 0x7f26f010fa40 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0115340 con 0x7f26f01027e0 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.148+0000 7f26ef200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0115640 con 0x7f26f0101e90 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f019bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f019bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35550/0 (socket says 172.21.15.144:35550) 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 -- 172.21.15.144:0/1716803538 learned_addr learned my addr 172.21.15.144:0/1716803538 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26e7e00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f01027e0 0x7f26f019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 -- 172.21.15.144:0/1716803538 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f01027e0 msgr2=0x7f26f019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f01027e0 0x7f26f019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ee800640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 0x7f26f01a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 -- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 msgr2=0x7f26f01a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:24.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 0x7f26f01a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ede00640 1 -- 172.21.15.144:0/1716803538 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f01a1ee0 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.149+0000 7f26ee800640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 0x7f26f01a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.150+0000 7f26ede00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f019bf70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26e0005940 tx=0x7f26e0030fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.150+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0040070 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.150+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26e0005570 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.150+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f01a2170 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.150+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e00046c0 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.150+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f01a8930 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.151+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f01022a0 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.154+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f26e0043070 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.155+0000 7f26e6a00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3c10 0x7f26d40b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.155+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f26e0102c00 con 0x7f26f0101e90 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.155+0000 7f26e7e00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3c10 0x7f26d40b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.156+0000 7f26e7e00640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3c10 0x7f26d40b6120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26f0103ba0 tx=0x7f26d8009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:24.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.160+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f26e00cbe40 con 0x7f26f0101e90 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.379+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f26f019d0a0 con 0x7f26f0101e90 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.389+0000 7f26e6a00640 1 -- 172.21.15.144:0/1716803538 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v24)=0 v24) ==== 120+0+0 (secure 0 0 0) 0x7f26e00cb740 con 0x7f26f0101e90 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3c10 msgr2=0x7f26d40b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3c10 0x7f26d40b6120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26f0103ba0 tx=0x7f26d8009260 comp rx=0 tx=0).stop 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 msgr2=0x7f26f019bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f019bf70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26e0005940 tx=0x7f26e0030fa0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 shutdown_connections 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3c10 0x7f26d40b6120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f010fa40 0x7f26f01a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f01027e0 0x7f26f019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 --2- 172.21.15.144:0/1716803538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f0101e90 0x7f26f019bf70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 >> 172.21.15.144:0/1716803538 conn(0x7f26f00fdcf0 msgr2=0x7f26f0111bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 shutdown_connections 2024-07-26T10:50:24.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:22.392+0000 7f26ef200640 1 -- 172.21.15.144:0/1716803538 wait complete. 2024-07-26T10:50:25.885 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:25 smithi183 ceph-mon[37273]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:25 smithi144 ceph-mon[32129]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:25 smithi160 ceph-mon[36715]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:27.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 -- 172.21.15.144:0/4121002260 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba81033e0 msgr2=0x7faba81037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 --2- 172.21.15.144:0/4121002260 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba81033e0 0x7faba81037f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab98009900 tx=0x7fab9802a410 comp rx=0 tx=0).stop 2024-07-26T10:50:27.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 -- 172.21.15.144:0/4121002260 shutdown_connections 2024-07-26T10:50:27.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 --2- 172.21.15.144:0/4121002260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8109670 0x7faba810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 --2- 172.21.15.144:0/4121002260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faba8108d20 0x7faba8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 --2- 172.21.15.144:0/4121002260 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba81033e0 0x7faba81037f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 -- 172.21.15.144:0/4121002260 >> 172.21.15.144:0/4121002260 conn(0x7faba80fee30 msgr2=0x7faba8101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:27.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 -- 172.21.15.144:0/4121002260 shutdown_connections 2024-07-26T10:50:27.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.343+0000 7faba7200640 1 -- 172.21.15.144:0/4121002260 wait complete. 2024-07-26T10:50:27.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.344+0000 7faba7200640 1 Processor -- start 2024-07-26T10:50:27.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.344+0000 7faba7200640 1 -- start start 2024-07-26T10:50:27.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba7200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faba81033e0 0x7faba819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba7200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 0x7faba819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba8109670 0x7faba81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba7200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7faba8113390 con 0x7faba81033e0 2024-07-26T10:50:27.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba7200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faba8113210 con 0x7faba8109670 2024-07-26T10:50:27.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba7200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faba8113510 con 0x7faba8108d20 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faba81033e0 0x7faba819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 0x7faba819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 0x7faba819c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35596/0 (socket says 172.21.15.144:35596) 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba5400640 1 -- 172.21.15.144:0/640888493 learned_addr learned my addr 172.21.15.144:0/640888493 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.345+0000 7faba6800640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba8109670 0x7faba81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba6800640 1 -- 172.21.15.144:0/640888493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 msgr2=0x7faba819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba6800640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 0x7faba819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba6800640 1 -- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faba81033e0 msgr2=0x7faba819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba6800640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faba81033e0 0x7faba819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba6800640 1 -- 172.21.15.144:0/640888493 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba81a1eb0 con 0x7faba8109670 2024-07-26T10:50:27.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba5400640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 0x7faba819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:27.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.346+0000 7faba6800640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba8109670 0x7faba81a1780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faba0009fd0 tx=0x7faba000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:27.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.348+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba000ce20 con 0x7faba8109670 2024-07-26T10:50:27.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.348+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba0005020 con 0x7faba8109670 2024-07-26T10:50:27.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.348+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba81a21a0 con 0x7faba8109670 2024-07-26T10:50:27.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.348+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0010690 con 0x7faba8109670 2024-07-26T10:50:27.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.348+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba81a8900 con 0x7faba8109670 2024-07-26T10:50:27.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.349+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba81037f0 con 0x7faba8109670 2024-07-26T10:50:27.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.352+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7faba0004b20 con 0x7faba8109670 2024-07-26T10:50:27.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.353+0000 7fab9ea00640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fab8c0b3ce0 0x7fab8c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.353+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7faba00e2630 con 0x7faba8109670 2024-07-26T10:50:27.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.353+0000 7faba5e00640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fab8c0b3ce0 0x7fab8c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.354+0000 7faba5e00640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fab8c0b3ce0 0x7fab8c0b61f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab98005ab0 tx=0x7fab98005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:27.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.357+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faba00ab7e0 con 0x7faba8109670 2024-07-26T10:50:27.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.562+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7faba81a8cd0 con 0x7faba8109670 2024-07-26T10:50:27.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.573+0000 7fab9ea00640 1 -- 172.21.15.144:0/640888493 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v27)=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7faba00ab0e0 con 0x7faba8109670 2024-07-26T10:50:27.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.576+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fab8c0b3ce0 msgr2=0x7fab8c0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.576+0000 7faba7200640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fab8c0b3ce0 0x7fab8c0b61f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab98005ab0 tx=0x7fab98005a20 comp rx=0 tx=0).stop 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.576+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba8109670 msgr2=0x7faba81a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.576+0000 7faba7200640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba8109670 0x7faba81a1780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faba0009fd0 tx=0x7faba000ef10 comp rx=0 tx=0).stop 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 shutdown_connections 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fab8c0b3ce0 0x7fab8c0b61f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faba8109670 0x7faba81a1780 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faba8108d20 0x7faba819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 --2- 172.21.15.144:0/640888493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faba81033e0 0x7faba819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 >> 172.21.15.144:0/640888493 conn(0x7faba80fee30 msgr2=0x7faba8105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:27.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 shutdown_connections 2024-07-26T10:50:27.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:27.577+0000 7faba7200640 1 -- 172.21.15.144:0/640888493 wait complete. 2024-07-26T10:50:27.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.524+0000 7ff8b7e00640 1 -- 172.21.15.144:0/137882895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b81099f0 msgr2=0x7ff8b8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.524+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/137882895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b81099f0 0x7ff8b8109e00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009900 tx=0x7ff8a802a410 comp rx=0 tx=0).stop 2024-07-26T10:50:27.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 -- 172.21.15.144:0/137882895 shutdown_connections 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/137882895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8b810fc80 0x7ff8b81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/137882895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810f330 0x7ff8b810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/137882895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b81099f0 0x7ff8b8109e00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 -- 172.21.15.144:0/137882895 >> 172.21.15.144:0/137882895 conn(0x7ff8b8078ff0 msgr2=0x7ff8b8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 -- 172.21.15.144:0/137882895 shutdown_connections 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.525+0000 7ff8b7e00640 1 -- 172.21.15.144:0/137882895 wait complete. 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.526+0000 7ff8b7e00640 1 Processor -- start 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.526+0000 7ff8b7e00640 1 -- start start 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.526+0000 7ff8b7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8b81099f0 0x7ff8b81a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 0x7ff8b81a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810fc80 0x7ff8b81a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b8119940 con 0x7ff8b810f330 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81197c0 con 0x7ff8b81099f0 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b8119ac0 con 0x7ff8b810fc80 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 0x7ff8b81a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 0x7ff8b81a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43678/0 (socket says 172.21.15.144:43678) 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b6000640 1 -- 172.21.15.144:0/1649325436 learned_addr learned my addr 172.21.15.144:0/1649325436 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7400640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810fc80 0x7ff8b81a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b6a00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8b81099f0 0x7ff8b81a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7400640 1 -- 172.21.15.144:0/1649325436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8b81099f0 msgr2=0x7ff8b81a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7400640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8b81099f0 0x7ff8b81a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7400640 1 -- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 msgr2=0x7ff8b81a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7400640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 0x7ff8b81a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b7400640 1 -- 172.21.15.144:0/1649325436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b81a62e0 con 0x7ff8b810fc80 2024-07-26T10:50:27.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.527+0000 7ff8b6000640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 0x7ff8b81a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.528+0000 7ff8b7400640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810fc80 0x7ff8b81a5bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000ec50 tx=0x7ff8b000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.528+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0003b70 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.528+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8b0003d10 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.528+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b81a65d0 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.528+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0010830 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.528+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b81acd30 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.530+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b8109e00 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.531+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7ff8b0003fa0 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.533+0000 7ff8af400640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff89c0b3c10 0x7ff89c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.533+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff8b00e2550 con 0x7ff8b810fc80 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.533+0000 7ff8b6a00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff89c0b3c10 0x7ff89c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.534+0000 7ff8b6a00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff89c0b3c10 0x7ff89c0b6120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8031440 tx=0x7ff8a8009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:27.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.537+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff8b00ab700 con 0x7ff8b810fc80 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.743+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7ff8b8111ab0 con 0x7ff8b810fc80 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.753+0000 7ff8af400640 1 -- 172.21.15.144:0/1649325436 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v26)=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7ff8b00ab000 con 0x7ff8b810fc80 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff89c0b3c10 msgr2=0x7ff89c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff89c0b3c10 0x7ff89c0b6120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8031440 tx=0x7ff8a8009560 comp rx=0 tx=0).stop 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810fc80 msgr2=0x7ff8b81a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810fc80 0x7ff8b81a5bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000ec50 tx=0x7ff8b000c6a0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 shutdown_connections 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff89c0b3c10 0x7ff89c0b6120 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8b810fc80 0x7ff8b81a5bb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b810f330 0x7ff8b81a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 --2- 172.21.15.144:0/1649325436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8b81099f0 0x7ff8b81a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.756+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 >> 172.21.15.144:0/1649325436 conn(0x7ff8b8078ff0 msgr2=0x7ff8b8108010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:27.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.757+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 shutdown_connections 2024-07-26T10:50:27.986 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:26.757+0000 7ff8b7e00640 1 -- 172.21.15.144:0/1649325436 wait complete. 2024-07-26T10:50:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:29.262 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi144') 2024-07-26T10:50:29.263 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi144') 2024-07-26T10:50:30.186 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:29 smithi144 ceph-mon[32129]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:29 smithi160 ceph-mon[36715]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:29 smithi183 ceph-mon[37273]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:31.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 -- 172.21.15.144:0/3334237779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410f330 msgr2=0x7f047410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 --2- 172.21.15.144:0/3334237779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410f330 0x7f047410f740 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0460009900 tx=0x7f046002a410 comp rx=0 tx=0).stop 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 -- 172.21.15.144:0/3334237779 shutdown_connections 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 --2- 172.21.15.144:0/3334237779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f047410fc80 0x7f04741164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 --2- 172.21.15.144:0/3334237779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410f330 0x7f047410f740 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 --2- 172.21.15.144:0/3334237779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04741099f0 0x7f0474109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 -- 172.21.15.144:0/3334237779 >> 172.21.15.144:0/3334237779 conn(0x7f0474078ff0 msgr2=0x7f0474107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 -- 172.21.15.144:0/3334237779 shutdown_connections 2024-07-26T10:50:31.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.725+0000 7f0479000640 1 -- 172.21.15.144:0/3334237779 wait complete. 2024-07-26T10:50:31.729 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.727+0000 7f0479000640 1 Processor -- start 2024-07-26T10:50:31.729 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.727+0000 7f0479000640 1 -- start start 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.727+0000 7f0479000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04741099f0 0x7f04741a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0479000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 0x7f04741a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0479000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410fc80 0x7f04741a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0479000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f0474119990 con 0x7f04741099f0 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0479000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0474119810 con 0x7f047410fc80 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0473400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04741099f0 0x7f04741a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0479000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0474119b10 con 0x7f047410f330 2024-07-26T10:50:31.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 0x7f04741a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 0x7f04741a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34566/0 (socket says 172.21.15.144:34566) 2024-07-26T10:50:31.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 -- 172.21.15.144:0/2071223177 learned_addr learned my addr 172.21.15.144:0/2071223177 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:31.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0473e00640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410fc80 0x7f04741a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 -- 172.21.15.144:0/2071223177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410fc80 msgr2=0x7f04741a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410fc80 0x7f04741a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 -- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04741099f0 msgr2=0x7f04741a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04741099f0 0x7f04741a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0472a00640 1 -- 172.21.15.144:0/2071223177 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04741a6350 con 0x7f047410f330 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0473400640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04741099f0 0x7f04741a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.728+0000 7f0473e00640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410fc80 0x7f04741a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.729+0000 7f0472a00640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 0x7f04741a0980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0460005ab0 tx=0x7f0460003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.729+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046003f070 con 0x7f047410f330 2024-07-26T10:50:31.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.729+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0460031b50 con 0x7f047410f330 2024-07-26T10:50:31.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.729+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460035030 con 0x7f047410f330 2024-07-26T10:50:31.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.729+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04741a65e0 con 0x7f047410f330 2024-07-26T10:50:31.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.729+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04741acda0 con 0x7f047410f330 2024-07-26T10:50:31.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.731+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f047410ac70 con 0x7f047410f330 2024-07-26T10:50:31.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.733+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f0460042070 con 0x7f047410f330 2024-07-26T10:50:31.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.734+0000 7f046be00640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f04580b3c10 0x7f04580b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.734+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f0460102610 con 0x7f047410f330 2024-07-26T10:50:31.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.734+0000 7f0473400640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f04580b3c10 0x7f04580b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.735+0000 7f0473400640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f04580b3c10 0x7f04580b6120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0464009fc0 tx=0x7f0464005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:31.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.738+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f04600cb7c0 con 0x7f047410f330 2024-07-26T10:50:31.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.712+0000 7f59d6200640 1 -- 172.21.15.144:0/3746906300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010f330 msgr2=0x7f59d010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.712+0000 7f59d6200640 1 --2- 172.21.15.144:0/3746906300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010f330 0x7f59d010f740 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009900 tx=0x7f59c002a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 -- 172.21.15.144:0/3746906300 shutdown_connections 2024-07-26T10:50:31.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 --2- 172.21.15.144:0/3746906300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59d010fc80 0x7f59d01164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 --2- 172.21.15.144:0/3746906300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010f330 0x7f59d010f740 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 --2- 172.21.15.144:0/3746906300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59d01099f0 0x7f59d0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 -- 172.21.15.144:0/3746906300 >> 172.21.15.144:0/3746906300 conn(0x7f59d0078ff0 msgr2=0x7f59d0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 -- 172.21.15.144:0/3746906300 shutdown_connections 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.713+0000 7f59d6200640 1 -- 172.21.15.144:0/3746906300 wait complete. 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.714+0000 7f59d6200640 1 Processor -- start 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.714+0000 7f59d6200640 1 -- start start 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.714+0000 7f59d6200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59d01099f0 0x7f59d01a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d6200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59d010f330 0x7f59d01a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d6200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 0x7f59d01a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d6200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0119990 con 0x7f59d01099f0 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d6200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0119810 con 0x7f59d010f330 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d6200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0119b10 con 0x7f59d010fc80 2024-07-26T10:50:31.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d4e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59d01099f0 0x7f59d01a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 0x7f59d01a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59c7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59d010f330 0x7f59d01a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 0x7f59d01a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34524/0 (socket says 172.21.15.144:34524) 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 -- 172.21.15.144:0/50849277 learned_addr learned my addr 172.21.15.144:0/50849277 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 -- 172.21.15.144:0/50849277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59d010f330 msgr2=0x7f59d01a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59d010f330 0x7f59d01a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 -- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59d01099f0 msgr2=0x7f59d01a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59d01099f0 0x7f59d01a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.715+0000 7f59d5800640 1 -- 172.21.15.144:0/50849277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d01a62d0 con 0x7f59d010fc80 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.716+0000 7f59d5800640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 0x7f59d01a5ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f59c800eb10 tx=0x7f59c800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.716+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8004b20 con 0x7f59d010fc80 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.716+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59c8004cc0 con 0x7f59d010fc80 2024-07-26T10:50:31.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.716+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d01a65c0 con 0x7f59d010fc80 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.716+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8010670 con 0x7f59d010fc80 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.716+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d01acd20 con 0x7f59d010fc80 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.718+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d0109e00 con 0x7f59d010fc80 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.720+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f59c8005020 con 0x7f59d010fc80 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.722+0000 7f59ce000640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f59b00b3c10 0x7f59b00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.722+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f59c80e2590 con 0x7f59d010fc80 2024-07-26T10:50:31.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.722+0000 7f59d4e00640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f59b00b3c10 0x7f59b00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.723+0000 7f59d4e00640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f59b00b3c10 0x7f59b00b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009bd0 tx=0x7f59b8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.726+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59c80ab780 con 0x7f59d010fc80 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.941+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f59d01114c0 con 0x7f59d010fc80 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.950+0000 7f59ce000640 1 -- 172.21.15.144:0/50849277 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v28)=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7f59c80ab080 con 0x7f59d010fc80 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f59b00b3c10 msgr2=0x7f59b00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f59b00b3c10 0x7f59b00b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009bd0 tx=0x7f59b8005f90 comp rx=0 tx=0).stop 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 msgr2=0x7f59d01a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 0x7f59d01a5ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f59c800eb10 tx=0x7f59c800c2d0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 shutdown_connections 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f59b00b3c10 0x7f59b00b6120 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59d010fc80 0x7f59d01a5ba0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59d010f330 0x7f59d01a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 --2- 172.21.15.144:0/50849277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59d01099f0 0x7f59d01a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 >> 172.21.15.144:0/50849277 conn(0x7f59d0078ff0 msgr2=0x7f59d0107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:31.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 shutdown_connections 2024-07-26T10:50:31.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:30.953+0000 7f59d6200640 1 -- 172.21.15.144:0/50849277 wait complete. 2024-07-26T10:50:31.940 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.938+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f0474111a70 con 0x7f047410f330 2024-07-26T10:50:31.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.948+0000 7f046be00640 1 -- 172.21.15.144:0/2071223177 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f04600cb0c0 con 0x7f047410f330 2024-07-26T10:50:31.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.951+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f04580b3c10 msgr2=0x7f04580b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f04580b3c10 0x7f04580b6120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0464009fc0 tx=0x7f0464005f90 comp rx=0 tx=0).stop 2024-07-26T10:50:31.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 msgr2=0x7f04741a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:31.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 0x7f04741a0980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0460005ab0 tx=0x7f0460003b90 comp rx=0 tx=0).stop 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 shutdown_connections 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f04580b3c10 0x7f04580b6120 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f047410fc80 0x7f04741a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f047410f330 0x7f04741a0980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 --2- 172.21.15.144:0/2071223177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04741099f0 0x7f04741a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.952+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 >> 172.21.15.144:0/2071223177 conn(0x7f0474078ff0 msgr2=0x7f04740797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.953+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 shutdown_connections 2024-07-26T10:50:31.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.953+0000 7f0479000640 1 -- 172.21.15.144:0/2071223177 wait complete. 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 -- 172.21.15.144:0/3530530529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241042b0 msgr2=0x7fc4241046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 --2- 172.21.15.144:0/3530530529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241042b0 0x7fc4241046c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc418009900 tx=0x7fc41802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 -- 172.21.15.144:0/3530530529 shutdown_connections 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 --2- 172.21.15.144:0/3530530529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410d860 0x7fc42410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 --2- 172.21.15.144:0/3530530529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc424104ce0 0x7fc42410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 --2- 172.21.15.144:0/3530530529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241042b0 0x7fc4241046c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 -- 172.21.15.144:0/3530530529 >> 172.21.15.144:0/3530530529 conn(0x7fc424079160 msgr2=0x7fc424079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 -- 172.21.15.144:0/3530530529 shutdown_connections 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.171+0000 7fc42b600640 1 -- 172.21.15.144:0/3530530529 wait complete. 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.172+0000 7fc42b600640 1 Processor -- start 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.172+0000 7fc42b600640 1 -- start start 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc42b600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4241042b0 0x7fc424199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc42b600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 0x7fc42419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc42b600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410d860 0x7fc42419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc42b600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc4241134e0 con 0x7fc424104ce0 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc42b600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc424113360 con 0x7fc4241042b0 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc42b600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc424113660 con 0x7fc42410d860 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc423e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 0x7fc42419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:32.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc423e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 0x7fc42419a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57066/0 (socket says 172.21.15.144:57066) 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc423e00640 1 -- 172.21.15.144:0/2873442568 learned_addr learned my addr 172.21.15.144:0/2873442568 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc428c00640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4241042b0 0x7fc424199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.173+0000 7fc429600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410d860 0x7fc42419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc423e00640 1 -- 172.21.15.144:0/2873442568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410d860 msgr2=0x7fc42419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc423e00640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410d860 0x7fc42419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc423e00640 1 -- 172.21.15.144:0/2873442568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4241042b0 msgr2=0x7fc424199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc423e00640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4241042b0 0x7fc424199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc423e00640 1 -- 172.21.15.144:0/2873442568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42419fe20 con 0x7fc424104ce0 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc423e00640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 0x7fc42419a450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc41400e9a0 tx=0x7fc41400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41400cd80 con 0x7fc424104ce0 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4140035f0 con 0x7fc424104ce0 2024-07-26T10:50:32.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4241a0110 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc414010690 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.174+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241a6870 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.175+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4241046c0 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.178+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc414004000 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.179+0000 7fc421600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc40c0b3c10 0x7fc40c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.180+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc4140e2350 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.180+0000 7fc428c00640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc40c0b3c10 0x7fc40c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.180+0000 7fc428c00640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc40c0b3c10 0x7fc40c0b6120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc418031440 tx=0x7fc418005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.181+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc4140042f0 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.382+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7fc42419afe0 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.387+0000 7fc421600640 1 -- 172.21.15.144:0/2873442568 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v29)=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fc4140ab590 con 0x7fc424104ce0 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc40c0b3c10 msgr2=0x7fc40c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:32.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc40c0b3c10 0x7fc40c0b6120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc418031440 tx=0x7fc418005a20 comp rx=0 tx=0).stop 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 msgr2=0x7fc42419a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 0x7fc42419a450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc41400e9a0 tx=0x7fc41400ee60 comp rx=0 tx=0).stop 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 shutdown_connections 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc40c0b3c10 0x7fc40c0b6120 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410d860 0x7fc42419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc424104ce0 0x7fc42419a450 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 --2- 172.21.15.144:0/2873442568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4241042b0 0x7fc424199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.391+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 >> 172.21.15.144:0/2873442568 conn(0x7fc424079160 msgr2=0x7fc424107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.392+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 shutdown_connections 2024-07-26T10:50:32.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:31.392+0000 7fc42b600640 1 -- 172.21.15.144:0/2873442568 wait complete. 2024-07-26T10:50:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:31 smithi144 ceph-mon[32129]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:31 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:31 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2873442568' entity='client.admin' 2024-07-26T10:50:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:31 smithi160 ceph-mon[36715]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:31 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:31 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2873442568' entity='client.admin' 2024-07-26T10:50:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:31 smithi183 ceph-mon[37273]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:31 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:31 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2873442568' entity='client.admin' 2024-07-26T10:50:33.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:32 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:33.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:33.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:33.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:33.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:32 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:33.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.374+0000 7fc762c00640 1 -- 172.21.15.144:0/1245566837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104080 msgr2=0x7fc75c104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.374+0000 7fc762c00640 1 --2- 172.21.15.144:0/1245566837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104080 0x7fc75c104490 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc750009900 tx=0x7fc75002a410 comp rx=0 tx=0).stop 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.375+0000 7fc762c00640 1 -- 172.21.15.144:0/1245566837 shutdown_connections 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.375+0000 7fc762c00640 1 --2- 172.21.15.144:0/1245566837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.375+0000 7fc762c00640 1 --2- 172.21.15.144:0/1245566837 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc75c104ab0 0x7fc75c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.375+0000 7fc762c00640 1 --2- 172.21.15.144:0/1245566837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104080 0x7fc75c104490 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc750009900 tx=0x7fc75002a410 comp rx=0 tx=0).stop 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.375+0000 7fc762c00640 1 -- 172.21.15.144:0/1245566837 >> 172.21.15.144:0/1245566837 conn(0x7fc75c0fdcf0 msgr2=0x7fc75c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:33.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.375+0000 7fc762c00640 1 -- 172.21.15.144:0/1245566837 shutdown_connections 2024-07-26T10:50:33.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.376+0000 7fc762c00640 1 -- 172.21.15.144:0/1245566837 wait complete. 2024-07-26T10:50:33.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.377+0000 7fc762c00640 1 Processor -- start 2024-07-26T10:50:33.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 -- start start 2024-07-26T10:50:33.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104ab0 0x7fc75c101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:33.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c1024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:33.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc75c102a10 0x7fc75c1a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:33.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c1132e0 con 0x7fc75c104ab0 2024-07-26T10:50:33.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c113160 con 0x7fc75c102a10 2024-07-26T10:50:33.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.378+0000 7fc762c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c113460 con 0x7fc75c10d630 2024-07-26T10:50:33.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c1024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:33.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c1024d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34582/0 (socket says 172.21.15.144:34582) 2024-07-26T10:50:33.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 -- 172.21.15.144:0/4075056770 learned_addr learned my addr 172.21.15.144:0/4075056770 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:33.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc762200640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc75c102a10 0x7fc75c1a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:33.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc761800640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104ab0 0x7fc75c101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:33.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 -- 172.21.15.144:0/4075056770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc75c102a10 msgr2=0x7fc75c1a7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:33.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc75c102a10 0x7fc75c1a7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 -- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104ab0 msgr2=0x7fc75c101f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:33.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104ab0 0x7fc75c101f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.379+0000 7fc760e00640 1 -- 172.21.15.144:0/4075056770 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c1a85b0 con 0x7fc75c10d630 2024-07-26T10:50:33.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc760e00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c1024d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc74800ea20 tx=0x7fc74800eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc761800640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104ab0 0x7fc75c101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74800cd00 con 0x7fc75c10d630 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7480108e0 con 0x7fc75c10d630 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75c1a88a0 con 0x7fc75c10d630 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7480187e0 con 0x7fc75c10d630 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.380+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75c1a8de0 con 0x7fc75c10d630 2024-07-26T10:50:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.381+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75c104490 con 0x7fc75c10d630 2024-07-26T10:50:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:32 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:33.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.384+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc748010430 con 0x7fc75c10d630 2024-07-26T10:50:33.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.385+0000 7fc75a000640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7440b3ce0 0x7fc7440b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:33.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.385+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc7480e2540 con 0x7fc75c10d630 2024-07-26T10:50:33.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.385+0000 7fc761800640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7440b3ce0 0x7fc7440b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:33.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.386+0000 7fc761800640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7440b3ce0 0x7fc7440b61f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc750009230 tx=0x7fc750032ce0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:33.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.387+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc7480ab770 con 0x7fc75c10d630 2024-07-26T10:50:33.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.600+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fc75c103290 con 0x7fc75c10d630 2024-07-26T10:50:33.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.609+0000 7fc75a000640 1 -- 172.21.15.144:0/4075056770 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7fc7480ab070 con 0x7fc75c10d630 2024-07-26T10:50:33.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7440b3ce0 msgr2=0x7fc7440b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:33.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7440b3ce0 0x7fc7440b61f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc750009230 tx=0x7fc750032ce0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 msgr2=0x7fc75c1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:33.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c1024d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc74800ea20 tx=0x7fc74800eee0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 shutdown_connections 2024-07-26T10:50:33.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7440b3ce0 0x7fc7440b61f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc75c102a10 0x7fc75c1a7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c10d630 0x7fc75c1024d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 --2- 172.21.15.144:0/4075056770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75c104ab0 0x7fc75c101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:33.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.614+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 >> 172.21.15.144:0/4075056770 conn(0x7fc75c0fdcf0 msgr2=0x7fc75c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:33.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.615+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 shutdown_connections 2024-07-26T10:50:33.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.615+0000 7fc762c00640 1 -- 172.21.15.144:0/4075056770 wait complete. 2024-07-26T10:50:34.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 -- 172.21.15.144:0/432399082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b8115150 msgr2=0x7f29b8117570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 --2- 172.21.15.144:0/432399082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b8115150 0x7f29b8117570 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009ca0 tx=0x7f29b002a8c0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 -- 172.21.15.144:0/432399082 shutdown_connections 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 --2- 172.21.15.144:0/432399082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b8115150 0x7f29b8117570 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 --2- 172.21.15.144:0/432399082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29b81127f0 0x7f29b8114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 --2- 172.21.15.144:0/432399082 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 0x7f29b810be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 -- 172.21.15.144:0/432399082 >> 172.21.15.144:0/432399082 conn(0x7f29b8070d20 msgr2=0x7f29b8073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 -- 172.21.15.144:0/432399082 shutdown_connections 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.849+0000 7f29bd000640 1 -- 172.21.15.144:0/432399082 wait complete. 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.850+0000 7f29bd000640 1 Processor -- start 2024-07-26T10:50:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 -- start start 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 0x7f29b81a2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 0x7f29b81a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29b8115150 0x7f29b81a7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f29b811ac40 con 0x7f29b81127f0 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f29b811aac0 con 0x7f29b810ba20 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29bd000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f29b811adc0 con 0x7f29b8115150 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29b5400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 0x7f29b81a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29b5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 0x7f29b81a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29b5400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 0x7f29b81a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57104/0 (socket says 172.21.15.144:57104) 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29b5400640 1 -- 172.21.15.144:0/295954736 learned_addr learned my addr 172.21.15.144:0/295954736 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.851+0000 7f29b6800640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29b8115150 0x7f29b81a7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5e00640 1 -- 172.21.15.144:0/295954736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29b8115150 msgr2=0x7f29b81a7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:34.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5e00640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29b8115150 0x7f29b81a7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5e00640 1 -- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 msgr2=0x7f29b81a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5e00640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 0x7f29b81a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5e00640 1 -- 172.21.15.144:0/295954736 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b81a86b0 con 0x7f29b810ba20 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5400640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 0x7f29b81a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29b5e00640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 0x7f29b81a2740 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e9f0 tx=0x7f29a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.852+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a800cd80 con 0x7f29b810ba20 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.853+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b81a89a0 con 0x7f29b810ba20 2024-07-26T10:50:34.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.853+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29a80035f0 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.853+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8010690 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.853+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b81af100 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.854+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b810be30 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.856+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f29a80037a0 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.858+0000 7f29aea00640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f299c0b3b90 0x7f299c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.858+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f29a80e2420 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.858+0000 7f29b5400640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f299c0b3b90 0x7f299c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.859+0000 7f29b5400640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f299c0b3b90 0x7f299c0b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f29b81146e0 tx=0x7f29a0005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:33.859+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f29a80ab660 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.065+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f29b81af450 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.073+0000 7f29aea00640 1 -- 172.21.15.144:0/295954736 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v32)=0 v32) ==== 121+0+0 (secure 0 0 0) 0x7f29a80aaf60 con 0x7f29b810ba20 2024-07-26T10:50:34.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.076+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f299c0b3b90 msgr2=0x7f299c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.076+0000 7f29bd000640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f299c0b3b90 0x7f299c0b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f29b81146e0 tx=0x7f29a0005f90 comp rx=0 tx=0).stop 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.076+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 msgr2=0x7f29b81a2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.076+0000 7f29bd000640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 0x7f29b81a2740 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e9f0 tx=0x7f29a800eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 shutdown_connections 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f299c0b3b90 0x7f299c0b60a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29b8115150 0x7f29b81a7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29b81127f0 0x7f29b81a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 --2- 172.21.15.144:0/295954736 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29b810ba20 0x7f29b81a2740 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 >> 172.21.15.144:0/295954736 conn(0x7f29b8070d20 msgr2=0x7f29b8073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 shutdown_connections 2024-07-26T10:50:34.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:34.077+0000 7f29bd000640 1 -- 172.21.15.144:0/295954736 wait complete. 2024-07-26T10:50:34.172 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:33 smithi144 ceph-mon[32129]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:34.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:33 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:34.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:34.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:34.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:34.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:33 smithi160 ceph-mon[36715]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:33 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:34.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:33 smithi183 ceph-mon[37273]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:33 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:35.160 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:35 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:35.160 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:35 smithi144 ceph-mon[32129]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:35 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:35 smithi183 ceph-mon[37273]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:35 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:35 smithi160 ceph-mon[36715]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:37 smithi144 ceph-mon[32129]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:37 smithi160 ceph-mon[36715]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:37 smithi183 ceph-mon[37273]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:39.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 -- 172.21.15.144:0/3560944838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b01041f0 msgr2=0x7fa5b0104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:39.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 --2- 172.21.15.144:0/3560944838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b01041f0 0x7fa5b0104600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009900 tx=0x7fa5a402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 -- 172.21.15.144:0/3560944838 shutdown_connections 2024-07-26T10:50:39.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 --2- 172.21.15.144:0/3560944838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5b010d650 0x7fa5b010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 --2- 172.21.15.144:0/3560944838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b0104b40 0x7fa5b010d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 --2- 172.21.15.144:0/3560944838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b01041f0 0x7fa5b0104600 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 -- 172.21.15.144:0/3560944838 >> 172.21.15.144:0/3560944838 conn(0x7fa5b00fdcd0 msgr2=0x7fa5b0100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 -- 172.21.15.144:0/3560944838 shutdown_connections 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.502+0000 7fa5b6400640 1 -- 172.21.15.144:0/3560944838 wait complete. 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 Processor -- start 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 -- start start 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b01041f0 0x7fa5b0199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5b0104b40 0x7fa5b019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b010d650 0x7fa5b019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0113310 con 0x7fa5b010d650 2024-07-26T10:50:39.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0113190 con 0x7fa5b0104b40 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b6400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0113490 con 0x7fa5b01041f0 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.504+0000 7fa5b5a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b010d650 0x7fa5b019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b01041f0 0x7fa5b0199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b010d650 0x7fa5b019f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57114/0 (socket says 172.21.15.144:57114) 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5afe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5b0104b40 0x7fa5b019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5a00640 1 -- 172.21.15.144:0/122956098 learned_addr learned my addr 172.21.15.144:0/122956098 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 -- 172.21.15.144:0/122956098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5b0104b40 msgr2=0x7fa5b019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5b0104b40 0x7fa5b019a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 -- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b010d650 msgr2=0x7fa5b019f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b010d650 0x7fa5b019f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 -- 172.21.15.144:0/122956098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b019fc90 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.505+0000 7fa5b5000640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b01041f0 0x7fa5b0199d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa5a402a9e0 tx=0x7fa5a4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.506+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a403f070 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.506+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5a4031b50 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.506+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b019ff20 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.506+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4035030 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.506+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b01a66e0 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.507+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b006ba90 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.509+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fa5a4042070 con 0x7fa5b01041f0 2024-07-26T10:50:39.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.511+0000 7fa5ad600640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa5980b3b90 0x7fa5980b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:39.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.511+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa5a41025e0 con 0x7fa5b01041f0 2024-07-26T10:50:39.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.511+0000 7fa5afe00640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa5980b3b90 0x7fa5980b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:39.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.512+0000 7fa5afe00640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa5980b3b90 0x7fa5980b60a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5b010fa90 tx=0x7fa59c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:39.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.515+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa5a40cb140 con 0x7fa5b01041f0 2024-07-26T10:50:39.722 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.720+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fa5b01a68b0 con 0x7fa5b01041f0 2024-07-26T10:50:39.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.730+0000 7fa5ad600640 1 -- 172.21.15.144:0/122956098 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v33)=0 v33) ==== 108+0+0 (secure 0 0 0) 0x7fa5a40cf690 con 0x7fa5b01041f0 2024-07-26T10:50:39.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.732+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa5980b3b90 msgr2=0x7fa5980b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.732+0000 7fa5b6400640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa5980b3b90 0x7fa5980b60a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5b010fa90 tx=0x7fa59c005f90 comp rx=0 tx=0).stop 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b01041f0 msgr2=0x7fa5b0199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b01041f0 0x7fa5b0199d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa5a402a9e0 tx=0x7fa5a4003b90 comp rx=0 tx=0).stop 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 shutdown_connections 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa5980b3b90 0x7fa5980b60a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b010d650 0x7fa5b019f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5b0104b40 0x7fa5b019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 --2- 172.21.15.144:0/122956098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5b01041f0 0x7fa5b0199d20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:39.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 >> 172.21.15.144:0/122956098 conn(0x7fa5b00fdcd0 msgr2=0x7fa5b0107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:39.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 shutdown_connections 2024-07-26T10:50:39.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.733+0000 7fa5b6400640 1 -- 172.21.15.144:0/122956098 wait complete. 2024-07-26T10:50:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:39 smithi144 ceph-mon[32129]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:39 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:39 smithi160 ceph-mon[36715]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:39 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:39 smithi183 ceph-mon[37273]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:39 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:40.413 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.873+0000 7f4351200640 1 -- 172.21.15.144:0/2974756443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c109670 msgr2=0x7f434c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.873+0000 7f4351200640 1 --2- 172.21.15.144:0/2974756443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c109670 0x7f434c10fea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4344009ca0 tx=0x7f434402a8c0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 -- 172.21.15.144:0/2974756443 shutdown_connections 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 --2- 172.21.15.144:0/2974756443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c109670 0x7f434c10fea0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 --2- 172.21.15.144:0/2974756443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c108d20 0x7f434c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 --2- 172.21.15.144:0/2974756443 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c1033e0 0x7f434c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 -- 172.21.15.144:0/2974756443 >> 172.21.15.144:0/2974756443 conn(0x7f434c0fee70 msgr2=0x7f434c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 -- 172.21.15.144:0/2974756443 shutdown_connections 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.874+0000 7f4351200640 1 -- 172.21.15.144:0/2974756443 wait complete. 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.875+0000 7f4351200640 1 Processor -- start 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.875+0000 7f4351200640 1 -- start start 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.875+0000 7f4351200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 0x7f434c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:40.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.875+0000 7f4351200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c108d20 0x7f434c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f4351200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c109670 0x7f434c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f4351200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f434c113320 con 0x7f434c108d20 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f4351200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1131a0 con 0x7f434c109670 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f4351200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1134a0 con 0x7f434c1033e0 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434aa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c108d20 0x7f434c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 0x7f434c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 0x7f434c199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34218/0 (socket says 172.21.15.144:34218) 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 -- 172.21.15.144:0/3830012746 learned_addr learned my addr 172.21.15.144:0/3830012746 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434be00640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c109670 0x7f434c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 -- 172.21.15.144:0/3830012746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c109670 msgr2=0x7f434c19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c109670 0x7f434c19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 -- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c108d20 msgr2=0x7f434c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c108d20 0x7f434c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.876+0000 7f434b400640 1 -- 172.21.15.144:0/3830012746 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c19fc60 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.877+0000 7f434b400640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 0x7f434c199cf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f433c00ea60 tx=0x7f433c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.877+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c00cde0 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.877+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c0035f0 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.877+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c19ff50 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.877+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c010690 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.877+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434c075020 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.878+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f434c1037f0 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.880+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f433c0037a0 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.882+0000 7f4343e00640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3c10 0x7f43300b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.882+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f433c0e2440 con 0x7f434c1033e0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.882+0000 7f434aa00640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3c10 0x7f43300b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:40.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.883+0000 7f434aa00640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3c10 0x7f43300b6120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f43380095f0 tx=0x7f4338012040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.883+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f433c0ab5f0 con 0x7f434c1033e0 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.091+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f434c19ae20 con 0x7f434c1033e0 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.099+0000 7f4343e00640 1 -- 172.21.15.144:0/3830012746 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v34)=0 v34) ==== 129+0+0 (secure 0 0 0) 0x7f434c19ae20 con 0x7f434c1033e0 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3c10 msgr2=0x7f43300b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3c10 0x7f43300b6120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f43380095f0 tx=0x7f4338012040 comp rx=0 tx=0).stop 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 msgr2=0x7f434c199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 0x7f434c199cf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f433c00ea60 tx=0x7f433c00ef20 comp rx=0 tx=0).stop 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 shutdown_connections 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3c10 0x7f43300b6120 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c109670 0x7f434c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c108d20 0x7f434c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 --2- 172.21.15.144:0/3830012746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c1033e0 0x7f434c199cf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 >> 172.21.15.144:0/3830012746 conn(0x7f434c0fee70 msgr2=0x7f434c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:40.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 shutdown_connections 2024-07-26T10:50:40.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.102+0000 7f4351200640 1 -- 172.21.15.144:0/3830012746 wait complete. 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 -- 172.21.15.144:0/1153491300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea8108d20 msgr2=0x7feea8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 --2- 172.21.15.144:0/1153491300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea8108d20 0x7feea8109130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee94009900 tx=0x7fee9402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 -- 172.21.15.144:0/1153491300 shutdown_connections 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 --2- 172.21.15.144:0/1153491300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8109670 0x7feea810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 --2- 172.21.15.144:0/1153491300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea8108d20 0x7feea8109130 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 --2- 172.21.15.144:0/1153491300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea81033e0 0x7feea81037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 -- 172.21.15.144:0/1153491300 >> 172.21.15.144:0/1153491300 conn(0x7feea80fee30 msgr2=0x7feea8101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 -- 172.21.15.144:0/1153491300 shutdown_connections 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.977+0000 7feeadc00640 1 -- 172.21.15.144:0/1153491300 wait complete. 2024-07-26T10:50:41.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.979+0000 7feeadc00640 1 Processor -- start 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.979+0000 7feeadc00640 1 -- start start 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.979+0000 7feeadc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 0x7feea819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.979+0000 7feeadc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8108d20 0x7feea819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.979+0000 7feeadc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea8109670 0x7feea81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feeadc00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7feea8113330 con 0x7feea8108d20 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feeadc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feea81131b0 con 0x7feea8109670 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feeadc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feea81134b0 con 0x7feea81033e0 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8108d20 0x7feea819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 0x7feea819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8108d20 0x7feea819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58846/0 (socket says 172.21.15.144:58846) 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7400640 1 -- 172.21.15.144:0/110784168 learned_addr learned my addr 172.21.15.144:0/110784168 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 0x7feea819bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34222/0 (socket says 172.21.15.144:34222) 2024-07-26T10:50:41.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feead200640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea8109670 0x7feea81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 -- 172.21.15.144:0/110784168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea8109670 msgr2=0x7feea81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea8109670 0x7feea81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 -- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8108d20 msgr2=0x7feea819c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8108d20 0x7feea819c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feea7e00640 1 -- 172.21.15.144:0/110784168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea81a1e90 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.980+0000 7feead200640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea8109670 0x7feea81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.981+0000 7feea7e00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 0x7feea819bf20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee9800e9f0 tx=0x7fee9800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.981+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9800cd80 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.981+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee980035f0 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.981+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea81a2180 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.981+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98010670 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.981+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea81a88e0 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.982+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea81037f0 con 0x7feea81033e0 2024-07-26T10:50:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.984+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fee98004020 con 0x7feea81033e0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.986+0000 7feea4c00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fee8c0b3c10 0x7fee8c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.986+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fee980e22c0 con 0x7feea81033e0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.986+0000 7feea7400640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fee8c0b3c10 0x7fee8c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.987+0000 7feea7400640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fee8c0b3c10 0x7fee8c0b6120 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee94006010 tx=0x7fee94005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:39.990+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fee980ab470 con 0x7feea81033e0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.215+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7feea819d050 con 0x7feea81033e0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.222+0000 7feea4c00640 1 -- 172.21.15.144:0/110784168 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v35)=0 v35) ==== 125+0+0 (secure 0 0 0) 0x7fee980aad70 con 0x7feea81033e0 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fee8c0b3c10 msgr2=0x7fee8c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fee8c0b3c10 0x7fee8c0b6120 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee94006010 tx=0x7fee94005a20 comp rx=0 tx=0).stop 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 msgr2=0x7feea819bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 0x7feea819bf20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee9800e9f0 tx=0x7fee9800eeb0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 shutdown_connections 2024-07-26T10:50:41.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fee8c0b3c10 0x7fee8c0b6120 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feea8109670 0x7feea81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feea8108d20 0x7feea819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 --2- 172.21.15.144:0/110784168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feea81033e0 0x7feea819bf20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:41.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 >> 172.21.15.144:0/110784168 conn(0x7feea80fee30 msgr2=0x7feea810e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:41.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 shutdown_connections 2024-07-26T10:50:41.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:40.225+0000 7feeadc00640 1 -- 172.21.15.144:0/110784168 wait complete. 2024-07-26T10:50:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' 2024-07-26T10:50:41.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:41 smithi183 ceph-mon[37273]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:41.408 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:41.408 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:41.408 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:41.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:41.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:41.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' 2024-07-26T10:50:41.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:41 smithi144 ceph-mon[32129]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:41.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:41.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:50:41.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' 2024-07-26T10:50:41.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:41 smithi160 ceph-mon[36715]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:42.550 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.550 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.550 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.550 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.551 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.552 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '493f92c6-4b3c-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi144') 2024-07-26T10:50:42.553 INFO:tasks.cephadm:Deploying OSDs... 2024-07-26T10:50:42.553 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:50:42.553 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T10:50:42.609 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T10:50:42.609 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-07-26T10:50:42.658 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T10:50:42.658 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-07-26 10:48:37.902608078 +0000 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-07-26 10:43:26.332011695 +0000 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-07-26 10:43:26.332011695 +0000 2024-07-26T10:50:42.659 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-07-26 10:43:26.332011695 +0000 2024-07-26T10:50:42.659 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T10:50:42.733 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-07-26T10:50:42.733 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-07-26T10:50:42.733 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000371063 s, 1.4 MB/s 2024-07-26T10:50:42.737 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T10:50:42.787 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-07-26 10:48:37.978606852 +0000 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-07-26 10:43:26.875002948 +0000 2024-07-26T10:50:42.834 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-07-26 10:43:26.875002948 +0000 2024-07-26T10:50:42.835 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-07-26 10:43:26.875002948 +0000 2024-07-26T10:50:42.835 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T10:50:42.959 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-07-26T10:50:42.959 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-07-26T10:50:42.959 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000360981 s, 1.4 MB/s 2024-07-26T10:50:42.963 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T10:50:43.018 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-07-26T10:50:43.069 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T10:50:43.069 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:43.069 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-07-26T10:50:43.070 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:43.070 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:43.070 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-07-26 10:48:38.059605544 +0000 2024-07-26T10:50:43.070 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-07-26 10:43:27.474993283 +0000 2024-07-26T10:50:43.070 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-07-26 10:43:27.474993283 +0000 2024-07-26T10:50:43.070 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-07-26 10:43:27.474993283 +0000 2024-07-26T10:50:43.070 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T10:50:43.146 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-07-26T10:50:43.147 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-07-26T10:50:43.147 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000399108 s, 1.3 MB/s 2024-07-26T10:50:43.151 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T10:50:43.205 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-07-26T10:50:43.253 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T10:50:43.253 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:43.253 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-07-26T10:50:43.253 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:43.253 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:43.253 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-07-26 10:48:38.130604398 +0000 2024-07-26T10:50:43.254 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-07-26 10:43:28.086983424 +0000 2024-07-26T10:50:43.254 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-07-26 10:43:28.086983424 +0000 2024-07-26T10:50:43.254 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-07-26 10:43:28.086983424 +0000 2024-07-26T10:50:43.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T10:50:43.328 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-07-26T10:50:43.329 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-07-26T10:50:43.329 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00034077 s, 1.5 MB/s 2024-07-26T10:50:43.332 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T10:50:43.389 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-07-26T10:50:43.389 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T10:50:43.408 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T10:50:43.409 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-07-26T10:50:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T10:50:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-07-26 10:49:55.401677274 +0000 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-07-26 10:41:50.477044341 +0000 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-07-26 10:41:50.477044341 +0000 2024-07-26T10:50:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-07-26 10:41:50.477044341 +0000 2024-07-26T10:50:43.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T10:50:43.537 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-07-26T10:50:43.538 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-07-26T10:50:43.538 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000290786 s, 1.8 MB/s 2024-07-26T10:50:43.539 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T10:50:43.598 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-07-26T10:50:43.657 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T10:50:43.657 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:43.657 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-07-26T10:50:43.657 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:43.658 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:43.658 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-07-26 10:49:55.402677259 +0000 2024-07-26T10:50:43.658 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-07-26 10:41:50.813039200 +0000 2024-07-26T10:50:43.658 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-07-26 10:41:50.813039200 +0000 2024-07-26T10:50:43.658 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-07-26 10:41:50.813039200 +0000 2024-07-26T10:50:43.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T10:50:43.722 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-07-26T10:50:43.722 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-07-26T10:50:43.722 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000228688 s, 2.2 MB/s 2024-07-26T10:50:43.724 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T10:50:43.781 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-07-26T10:50:43.837 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T10:50:43.837 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:43.837 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-07-26T10:50:43.838 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:43.838 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:43.838 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-07-26 10:49:55.402677259 +0000 2024-07-26T10:50:43.838 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-07-26 10:41:51.147034090 +0000 2024-07-26T10:50:43.838 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-07-26 10:41:51.147034090 +0000 2024-07-26T10:50:43.838 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-07-26 10:41:51.147034090 +0000 2024-07-26T10:50:43.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T10:50:43.901 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-07-26T10:50:43.901 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-07-26T10:50:43.901 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000227754 s, 2.2 MB/s 2024-07-26T10:50:43.902 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T10:50:43.958 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-07-26 10:49:55.402677259 +0000 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-07-26 10:41:51.481028980 +0000 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-07-26 10:41:51.481028980 +0000 2024-07-26T10:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-07-26 10:41:51.481028980 +0000 2024-07-26T10:50:44.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T10:50:44.076 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:43 smithi160 ceph-mon[36715]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:44.078 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-07-26T10:50:44.078 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-07-26T10:50:44.079 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000221078 s, 2.3 MB/s 2024-07-26T10:50:44.080 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T10:50:44.137 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-07-26T10:50:44.137 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T10:50:44.156 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T10:50:44.157 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-07-26 10:49:53.147549191 +0000 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-07-26 10:41:50.483376241 +0000 2024-07-26T10:50:44.215 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-07-26 10:41:50.483376241 +0000 2024-07-26T10:50:44.216 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-07-26 10:41:50.483376241 +0000 2024-07-26T10:50:44.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T10:50:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:43 smithi144 ceph-mon[32129]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:44.280 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:43 smithi183 ceph-mon[37273]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:44.284 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-07-26T10:50:44.284 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-07-26T10:50:44.284 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000258378 s, 2.0 MB/s 2024-07-26T10:50:44.286 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T10:50:44.343 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-07-26 10:49:53.148549164 +0000 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-07-26 10:41:50.819366931 +0000 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-07-26 10:41:50.819366931 +0000 2024-07-26T10:50:44.400 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-07-26 10:41:50.819366931 +0000 2024-07-26T10:50:44.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T10:50:44.465 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-07-26T10:50:44.465 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-07-26T10:50:44.465 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000232167 s, 2.2 MB/s 2024-07-26T10:50:44.466 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T10:50:44.524 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-07-26 10:49:53.148549164 +0000 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-07-26 10:41:51.155357624 +0000 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-07-26 10:41:51.155357624 +0000 2024-07-26T10:50:44.581 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-07-26 10:41:51.155357624 +0000 2024-07-26T10:50:44.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T10:50:44.645 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-07-26T10:50:44.645 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-07-26T10:50:44.645 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000214855 s, 2.4 MB/s 2024-07-26T10:50:44.646 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T10:50:44.703 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-07-26T10:50:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-07-26 10:49:53.148549164 +0000 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-07-26 10:41:51.485348482 +0000 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-07-26 10:41:51.485348482 +0000 2024-07-26T10:50:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-07-26 10:41:51.485348482 +0000 2024-07-26T10:50:44.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T10:50:44.821 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-07-26T10:50:44.821 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-07-26T10:50:44.821 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000225595 s, 2.3 MB/s 2024-07-26T10:50:44.822 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T10:50:44.879 INFO:tasks.cephadm:Deploying osd.0 on smithi144 with /dev/vg_nvme/lv_4... 2024-07-26T10:50:44.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T10:50:45.337 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:46.129 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:45 smithi144 ceph-mon[32129]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:45 smithi160 ceph-mon[36715]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:45 smithi183 ceph-mon[37273]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:47.542 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T10:50:47.733 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T10:50:47.774 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-07-26T10:50:47.774 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-07-26T10:50:47.775 INFO:teuthology.orchestra.run.smithi144.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.032772 s, 320 MB/s 2024-07-26T10:50:47.775 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-07-26T10:50:48.190 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:47 smithi144 ceph-mon[32129]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:47 smithi160 ceph-mon[36715]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:47 smithi183 ceph-mon[37273]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:48.512 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi144:vg_nvme/lv_4 2024-07-26T10:50:48.951 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:50:50.101 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:49 smithi144 ceph-mon[32129]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:49 smithi160 ceph-mon[36715]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:49 smithi183 ceph-mon[37273]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.901+0000 7fe145c00640 1 -- 172.21.15.144:0/4277643796 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe1401033e0 msgr2=0x7fe1401037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.901+0000 7fe145c00640 1 --2- 172.21.15.144:0/4277643796 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe1401033e0 0x7fe1401037f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe130009be0 tx=0x7fe13002a6c0 comp rx=0 tx=0).stop 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 -- 172.21.15.144:0/4277643796 shutdown_connections 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 --2- 172.21.15.144:0/4277643796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe140109670 0x7fe14010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 --2- 172.21.15.144:0/4277643796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe140108d20 0x7fe140109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 --2- 172.21.15.144:0/4277643796 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe1401033e0 0x7fe1401037f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:50.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 -- 172.21.15.144:0/4277643796 >> 172.21.15.144:0/4277643796 conn(0x7fe1400fee70 msgr2=0x7fe1401012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:50:50.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 -- 172.21.15.144:0/4277643796 shutdown_connections 2024-07-26T10:50:50.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.902+0000 7fe145c00640 1 -- 172.21.15.144:0/4277643796 wait complete. 2024-07-26T10:50:50.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.903+0000 7fe145c00640 1 Processor -- start 2024-07-26T10:50:50.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.903+0000 7fe145c00640 1 -- start start 2024-07-26T10:50:50.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1401033e0 0x7fe140075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:50.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe140108d20 0x7fe140075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:50.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe140109670 0x7fe140075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:50.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe140113340 con 0x7fe140108d20 2024-07-26T10:50:50.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401131c0 con 0x7fe140109670 2024-07-26T10:50:50.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401134c0 con 0x7fe1401033e0 2024-07-26T10:50:50.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe13fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1401033e0 0x7fe140075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:50.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe13fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1401033e0 0x7fe140075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42956/0 (socket says 172.21.15.144:42956) 2024-07-26T10:50:50.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe137400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe140108d20 0x7fe140075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:50.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe145200640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe140109670 0x7fe140075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:50.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.904+0000 7fe13fe00640 1 -- 172.21.15.144:0/2359771224 learned_addr learned my addr 172.21.15.144:0/2359771224 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe145200640 1 -- 172.21.15.144:0/2359771224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1401033e0 msgr2=0x7fe140075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe145200640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1401033e0 0x7fe140075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe145200640 1 -- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe140108d20 msgr2=0x7fe140075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe145200640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe140108d20 0x7fe140075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe145200640 1 -- 172.21.15.144:0/2359771224 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1401a8050 con 0x7fe140109670 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe145200640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe140109670 0x7fe140075bc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe138005fd0 tx=0x7fe13800ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.905+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13800ccc0 con 0x7fe140109670 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.906+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1380035f0 con 0x7fe140109670 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.906+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1401a8340 con 0x7fe140109670 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.906+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13800ce60 con 0x7fe140109670 2024-07-26T10:50:50.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.906+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401a8880 con 0x7fe140109670 2024-07-26T10:50:50.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.907+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1401037f0 con 0x7fe140109670 2024-07-26T10:50:50.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.909+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fe138004000 con 0x7fe140109670 2024-07-26T10:50:50.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.911+0000 7fe13d600640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1240b3c10 0x7fe1240b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:50:50.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.911+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fe1380e22b0 con 0x7fe140109670 2024-07-26T10:50:50.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.911+0000 7fe13fe00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1240b3c10 0x7fe1240b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:50.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.912+0000 7fe13fe00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1240b3c10 0x7fe1240b6120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe130009310 tx=0x7fe130009280 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:50:50.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:50.915+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe138009510 con 0x7fe140109670 2024-07-26T10:50:51.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:50:51.121+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe140063230 con 0x7fe1240b3c10 2024-07-26T10:50:52.115 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:51 smithi144 ceph-mon[32129]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:52.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:50:52.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:50:52.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:51 smithi160 ceph-mon[36715]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:50:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:50:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:51 smithi183 ceph-mon[37273]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:50:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:50:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:53.215 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:52 smithi144 ceph-mon[32129]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:50:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:52 smithi160 ceph-mon[36715]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:50:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:52 smithi183 ceph-mon[37273]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:50:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:53 smithi144 ceph-mon[32129]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:53 smithi160 ceph-mon[36715]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:53 smithi183 ceph-mon[37273]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:54 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1761361502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a20cb98e-b483-474f-aafe-5145286b6145"}]: dispatch 2024-07-26T10:50:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:54 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1761361502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a20cb98e-b483-474f-aafe-5145286b6145"}]': finished 2024-07-26T10:50:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:50:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:54 smithi160 ceph-mon[36715]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T10:50:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:54 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1761361502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a20cb98e-b483-474f-aafe-5145286b6145"}]: dispatch 2024-07-26T10:50:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:54 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1761361502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a20cb98e-b483-474f-aafe-5145286b6145"}]': finished 2024-07-26T10:50:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:50:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:54 smithi144 ceph-mon[32129]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T10:50:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:54 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1761361502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a20cb98e-b483-474f-aafe-5145286b6145"}]: dispatch 2024-07-26T10:50:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:54 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1761361502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a20cb98e-b483-474f-aafe-5145286b6145"}]': finished 2024-07-26T10:50:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:50:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:54 smithi183 ceph-mon[37273]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T10:50:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:55 smithi144 ceph-mon[32129]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:56.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:55 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1909958226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:50:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:55 smithi160 ceph-mon[36715]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:55 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1909958226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:50:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:55 smithi183 ceph-mon[37273]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:55 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1909958226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:50:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:57 smithi144 ceph-mon[32129]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:57 smithi160 ceph-mon[36715]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:50:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:57 smithi183 ceph-mon[37273]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:50:59 smithi144 ceph-mon[32129]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:50:59 smithi160 ceph-mon[36715]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:50:59 smithi183 ceph-mon[37273]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:02.211 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:01 smithi144 ceph-mon[32129]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:01 smithi160 ceph-mon[36715]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:01 smithi183 ceph-mon[37273]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:03 smithi144 ceph-mon[32129]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T10:51:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:03 smithi160 ceph-mon[36715]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T10:51:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:03 smithi183 ceph-mon[37273]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T10:51:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:04 smithi144 ceph-mon[32129]: Deploying daemon osd.0 on smithi144 2024-07-26T10:51:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:04 smithi160 ceph-mon[36715]: Deploying daemon osd.0 on smithi144 2024-07-26T10:51:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:04 smithi183 ceph-mon[37273]: Deploying daemon osd.0 on smithi144 2024-07-26T10:51:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:05 smithi144 ceph-mon[32129]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:05 smithi160 ceph-mon[36715]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:05 smithi183 ceph-mon[37273]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:08.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:07 smithi144 ceph-mon[32129]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:07 smithi160 ceph-mon[36715]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:07 smithi183 ceph-mon[37273]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:10.213 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:09 smithi144 ceph-mon[32129]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:09 smithi160 ceph-mon[36715]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:09 smithi183 ceph-mon[37273]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:12.111 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:11 smithi144 ceph-mon[32129]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:12.111 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:12.111 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:12.111 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:11 smithi160 ceph-mon[36715]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:11 smithi183 ceph-mon[37273]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:12.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:14.233 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:13 smithi144 ceph-mon[32129]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:13 smithi160 ceph-mon[36715]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:13 smithi183 ceph-mon[37273]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T10:51:15.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:15 smithi160 ceph-mon[36715]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T10:51:15.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:15 smithi183 ceph-mon[37273]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T10:51:15.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T10:51:15.898 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:15 smithi144 ceph-mon[32129]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T10:51:16.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.306+0000 7fe13d600640 1 -- 172.21.15.144:0/2359771224 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe140063230 con 0x7fe1240b3c10 2024-07-26T10:51:16.308 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 0 on host 'smithi144' 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.309+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1240b3c10 msgr2=0x7fe1240b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.309+0000 7fe145c00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1240b3c10 0x7fe1240b6120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe130009310 tx=0x7fe130009280 comp rx=0 tx=0).stop 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.309+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe140109670 msgr2=0x7fe140075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.309+0000 7fe145c00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe140109670 0x7fe140075bc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe138005fd0 tx=0x7fe13800ed20 comp rx=0 tx=0).stop 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 shutdown_connections 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1240b3c10 0x7fe1240b6120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe140109670 0x7fe140075bc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe140108d20 0x7fe140075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 --2- 172.21.15.144:0/2359771224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1401033e0 0x7fe140075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 >> 172.21.15.144:0/2359771224 conn(0x7fe1400fee70 msgr2=0x7fe140101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:51:16.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 shutdown_connections 2024-07-26T10:51:16.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:51:16.310+0000 7fe145c00640 1 -- 172.21.15.144:0/2359771224 wait complete. 2024-07-26T10:51:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T10:51:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T10:51:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:51:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:16 smithi160 ceph-mon[36715]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:16 smithi183 ceph-mon[37273]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:51:16.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:16 smithi144 ceph-mon[32129]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:51:17.079 DEBUG:teuthology.orchestra.run.smithi144:osd.0> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.0.service 2024-07-26T10:51:17.084 INFO:tasks.cephadm:Deploying osd.1 on smithi160 with /dev/vg_nvme/lv_4... 2024-07-26T10:51:17.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T10:51:17.338 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:51:17.545 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-0[52093]: 2024-07-26T10:51:17.488+0000 7f4b5d200640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T10:51:17.546 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-0[52093]: 2024-07-26T10:51:17.497+0000 7f4b58200640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:51:17.802 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:17 smithi160 ceph-mon[36715]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:17.802 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:17 smithi160 ceph-mon[36715]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:51:17.802 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:17.802 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:17 smithi160 ceph-mon[36715]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T10:51:17.803 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:17 smithi183 ceph-mon[37273]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:17 smithi183 ceph-mon[37273]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:51:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:17 smithi183 ceph-mon[37273]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T10:51:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:17.977 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-mon[32129]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T10:51:17.977 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-mon[32129]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:51:17.977 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:17.977 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-mon[32129]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T10:51:17.977 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:19.045 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T10:51:19.153 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T10:51:19.166 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-07-26T10:51:19.166 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-07-26T10:51:19.166 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114133 s, 919 MB/s 2024-07-26T10:51:19.167 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-07-26T10:51:19.769 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:19 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:51:19.770 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:19 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:51:19.770 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:19.770 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:19 smithi160 ceph-mon[36715]: osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] boot 2024-07-26T10:51:19.770 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:19 smithi160 ceph-mon[36715]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T10:51:19.770 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:19 smithi160 ceph-mon[36715]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:19.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_4 2024-07-26T10:51:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:19 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:51:19.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:19 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:51:19.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:19.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:19 smithi183 ceph-mon[37273]: osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] boot 2024-07-26T10:51:19.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:19 smithi183 ceph-mon[37273]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T10:51:19.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:19 smithi183 ceph-mon[37273]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:19 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:51:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:19 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:51:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T10:51:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:19 smithi144 ceph-mon[32129]: osd.0 [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] boot 2024-07-26T10:51:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:19 smithi144 ceph-mon[32129]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T10:51:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:19 smithi144 ceph-mon[32129]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:20.053 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:51:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:20 smithi144 ceph-mon[32129]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T10:51:20.744 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:20 smithi183 ceph-mon[37273]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T10:51:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:20 smithi160 ceph-mon[36715]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 -- 172.21.15.160:0/1345075823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c109820 msgr2=0x7f612c110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 --2- 172.21.15.160:0/1345075823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c109820 0x7f612c110050 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6124009ca0 tx=0x7f612402a8c0 comp rx=0 tx=0).stop 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 -- 172.21.15.160:0/1345075823 shutdown_connections 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 --2- 172.21.15.160:0/1345075823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c109820 0x7f612c110050 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 --2- 172.21.15.160:0/1345075823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f612c108ed0 0x7f612c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 --2- 172.21.15.160:0/1345075823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c103590 0x7f612c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:21.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 -- 172.21.15.160:0/1345075823 >> 172.21.15.160:0/1345075823 conn(0x7f612c07b280 msgr2=0x7f612c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 -- 172.21.15.160:0/1345075823 shutdown_connections 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 -- 172.21.15.160:0/1345075823 wait complete. 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 Processor -- start 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.504+0000 7f613385e640 1 -- start start 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f612c103590 0x7f612c199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 0x7f612c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c109820 0x7f612c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113560 con 0x7f612c109820 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1133e0 con 0x7f612c108ed0 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1136e0 con 0x7f612c103590 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 0x7f612c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 0x7f612c19a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:47404/0 (socket says 172.21.15.160:47404) 2024-07-26T10:51:21.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 -- 172.21.15.160:0/77717383 learned_addr learned my addr 172.21.15.160:0/77717383 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f61315fa640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f612c103590 0x7f612c199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 -- 172.21.15.160:0/77717383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f612c103590 msgr2=0x7f612c199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f612c103590 0x7f612c199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 -- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c109820 msgr2=0x7f612c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c109820 0x7f612c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 -- 172.21.15.160:0/77717383 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c19ff00 con 0x7f612c108ed0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f6130df9640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 0x7f612c19a530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f611c00d750 tx=0x7f611c00db50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c017070 con 0x7f612c108ed0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f611c009820 con 0x7f612c108ed0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c1a01f0 con 0x7f612c108ed0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.505+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c004c40 con 0x7f612c108ed0 2024-07-26T10:51:21.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.506+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c1a6950 con 0x7f612c108ed0 2024-07-26T10:51:21.510 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.506+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612c1039a0 con 0x7f612c108ed0 2024-07-26T10:51:21.510 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.508+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f611c0099d0 con 0x7f612c108ed0 2024-07-26T10:51:21.510 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.508+0000 7f61227fc640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f61180b3c10 0x7f61180b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:21.510 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.509+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7f611c0e1fe0 con 0x7f612c108ed0 2024-07-26T10:51:21.511 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.509+0000 7f61315fa640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f61180b3c10 0x7f61180b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:21.511 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.510+0000 7f61315fa640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f61180b3c10 0x7f61180b6120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6128009990 tx=0x7f6128009110 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:51:21.513 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.512+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f611c0ab030 con 0x7f612c108ed0 2024-07-26T10:51:21.695 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:21.693+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f612c063230 con 0x7f61180b3c10 2024-07-26T10:51:21.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:21.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:21.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:21.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:21.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:21.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:22.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:22 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi144 2024-07-26T10:51:22.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:22 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 17089M 2024-07-26T10:51:22.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:22 smithi160 ceph-mon[36715]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:51:22.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:51:22.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:51:22.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:22 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi144 2024-07-26T10:51:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:22 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 17089M 2024-07-26T10:51:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:22 smithi183 ceph-mon[37273]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:51:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:51:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:51:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:22 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi144 2024-07-26T10:51:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:22 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 17089M 2024-07-26T10:51:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:22 smithi144 ceph-mon[32129]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:51:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:51:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:51:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:23.712 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:23 smithi160 ceph-mon[36715]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:23 smithi183 ceph-mon[37273]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:23 smithi144 ceph-mon[32129]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:24 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/3585290035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]: dispatch 2024-07-26T10:51:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:24 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]: dispatch 2024-07-26T10:51:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:24 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]': finished 2024-07-26T10:51:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:24 smithi160 ceph-mon[36715]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T10:51:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:24 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/3585290035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]: dispatch 2024-07-26T10:51:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:24 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]: dispatch 2024-07-26T10:51:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:24 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]': finished 2024-07-26T10:51:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:24 smithi183 ceph-mon[37273]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T10:51:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:24 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/3585290035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]: dispatch 2024-07-26T10:51:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:24 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]: dispatch 2024-07-26T10:51:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:24 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "938ca752-2db7-47b3-aaad-b31ed684607e"}]': finished 2024-07-26T10:51:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:24 smithi144 ceph-mon[32129]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T10:51:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:25 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/1886354732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:51:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:25 smithi160 ceph-mon[36715]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:25 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/1886354732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:51:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:25 smithi183 ceph-mon[37273]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:25 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/1886354732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:51:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:25 smithi144 ceph-mon[32129]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:27 smithi144 ceph-mon[32129]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:27 smithi160 ceph-mon[36715]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:27 smithi183 ceph-mon[37273]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:30.203 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:29 smithi160 ceph-mon[36715]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:29 smithi144 ceph-mon[32129]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:29 smithi183 ceph-mon[37273]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T10:51:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T10:51:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T10:51:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:31 smithi144 ceph-mon[32129]: Deploying daemon osd.1 on smithi160 2024-07-26T10:51:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:31 smithi144 ceph-mon[32129]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:32.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:31 smithi160 ceph-mon[36715]: Deploying daemon osd.1 on smithi160 2024-07-26T10:51:32.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:31 smithi160 ceph-mon[36715]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:31 smithi183 ceph-mon[37273]: Deploying daemon osd.1 on smithi160 2024-07-26T10:51:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:31 smithi183 ceph-mon[37273]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:34.183 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:33 smithi160 ceph-mon[36715]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:33 smithi144 ceph-mon[32129]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:33 smithi183 ceph-mon[37273]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:35 smithi160 ceph-mon[36715]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:35 smithi144 ceph-mon[32129]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:35 smithi183 ceph-mon[37273]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:38.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:37 smithi183 ceph-mon[37273]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:37 smithi144 ceph-mon[32129]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:37 smithi160 ceph-mon[36715]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:38.546 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.544+0000 7f61227fc640 1 -- 172.21.15.160:0/77717383 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f612c063230 con 0x7f61180b3c10 2024-07-26T10:51:38.546 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 1 on host 'smithi160' 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.546+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f61180b3c10 msgr2=0x7f61180b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.546+0000 7f613385e640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f61180b3c10 0x7f61180b6120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6128009990 tx=0x7f6128009110 comp rx=0 tx=0).stop 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.546+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 msgr2=0x7f612c19a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.546+0000 7f613385e640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 0x7f612c19a530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f611c00d750 tx=0x7f611c00db50 comp rx=0 tx=0).stop 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 shutdown_connections 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f61180b3c10 0x7f61180b6120 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:38.548 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c109820 0x7f612c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:38.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f612c108ed0 0x7f612c19a530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:38.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 --2- 172.21.15.160:0/77717383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f612c103590 0x7f612c199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:38.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 >> 172.21.15.160:0/77717383 conn(0x7f612c07b280 msgr2=0x7f612c106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:51:38.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 shutdown_connections 2024-07-26T10:51:38.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:51:38.547+0000 7f613385e640 1 -- 172.21.15.160:0/77717383 wait complete. 2024-07-26T10:51:39.136 DEBUG:teuthology.orchestra.run.smithi160:osd.1> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.1.service 2024-07-26T10:51:39.142 INFO:tasks.cephadm:Deploying osd.2 on smithi183 with /dev/vg_nvme/lv_4... 2024-07-26T10:51:39.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T10:51:39.404 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:51:39.692 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:39.692 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:39.692 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:39 smithi183 ceph-mon[37273]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:39 smithi160 ceph-mon[36715]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:39.810 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:51:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-1[42588]: 2024-07-26T10:51:39.697+0000 7f0ca2e33640 -1 osd.1 0 log_to_monitors true 2024-07-26T10:51:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:39 smithi144 ceph-mon[32129]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:40.881 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:40.881 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:40.881 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:40 smithi183 ceph-mon[37273]: from='osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T10:51:40.881 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:40 smithi183 ceph-mon[37273]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T10:51:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:40 smithi144 ceph-mon[32129]: from='osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T10:51:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:40 smithi144 ceph-mon[32129]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T10:51:41.051 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:41.051 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:41.051 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:40 smithi160 ceph-mon[36715]: from='osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T10:51:41.052 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:40 smithi160 ceph-mon[36715]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T10:51:41.118 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T10:51:41.201 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T10:51:41.212 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-07-26T10:51:41.212 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-07-26T10:51:41.212 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114944 s, 912 MB/s 2024-07-26T10:51:41.212 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-07-26T10:51:41.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-07-26T10:51:41.863 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:41 smithi183 ceph-mon[37273]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T10:51:41.864 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:41.864 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:41 smithi183 ceph-mon[37273]: from='osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:51:41.864 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:41 smithi183 ceph-mon[37273]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T10:51:41.864 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:41 smithi183 ceph-mon[37273]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:51:41.864 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:41 smithi183 ceph-mon[37273]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:41 smithi144 ceph-mon[32129]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T10:51:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:41 smithi144 ceph-mon[32129]: from='osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:51:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:41 smithi144 ceph-mon[32129]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T10:51:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:41 smithi144 ceph-mon[32129]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:51:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:41 smithi144 ceph-mon[32129]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:41 smithi160 ceph-mon[36715]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T10:51:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:41 smithi160 ceph-mon[36715]: from='osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:51:42.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:41 smithi160 ceph-mon[36715]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T10:51:42.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:41 smithi160 ceph-mon[36715]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:51:42.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:41 smithi160 ceph-mon[36715]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:42.087 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:51:42.508 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-1[42588]: 2024-07-26T10:51:42.346+0000 7f0c9edb5640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T10:51:42.508 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-1[42588]: 2024-07-26T10:51:42.350+0000 7f0c96bab640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:51:42.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-mon[36715]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:51:42.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-mon[36715]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T10:51:42.792 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.792 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:42 smithi160 ceph-mon[36715]: from='osd.1 ' entity='osd.1' 2024-07-26T10:51:42.873 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:42 smithi183 ceph-mon[37273]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:51:42.874 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.874 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:42 smithi183 ceph-mon[37273]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T10:51:42.874 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.874 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:42 smithi183 ceph-mon[37273]: from='osd.1 ' entity='osd.1' 2024-07-26T10:51:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:42 smithi144 ceph-mon[32129]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:51:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:42 smithi144 ceph-mon[32129]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T10:51:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:42 smithi144 ceph-mon[32129]: from='osd.1 ' entity='osd.1' 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.342+0000 7f2b071f4640 1 -- 172.21.15.183:0/2973738168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 msgr2=0x7f2b00108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.342+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2973738168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b00108200 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009900 tx=0x7f2aec02a410 comp rx=0 tx=0).stop 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.342+0000 7f2b071f4640 1 -- 172.21.15.183:0/2973738168 shutdown_connections 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.342+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2973738168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b00108740 0x7f2b0010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2973738168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b00108200 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2973738168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b001024b0 0x7f2b001028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 -- 172.21.15.183:0/2973738168 >> 172.21.15.183:0/2973738168 conn(0x7f2b00079170 msgr2=0x7f2b00079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 -- 172.21.15.183:0/2973738168 shutdown_connections 2024-07-26T10:51:43.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 -- 172.21.15.183:0/2973738168 wait complete. 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 Processor -- start 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 -- start start 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b001024b0 0x7f2b0019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b0019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.343+0000 7f2b071f4640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b00108740 0x7f2b0019b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2b071f4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00112470 con 0x7f2b001024b0 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2b071f4640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001122f0 con 0x7f2b00108740 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2b071f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001125f0 con 0x7f2b00107df0 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b0019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b0019b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:41782/0 (socket says 172.21.15.183:41782) 2024-07-26T10:51:43.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 -- 172.21.15.183:0/2097733627 learned_addr learned my addr 172.21.15.183:0/2097733627 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2b05791640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b00108740 0x7f2b0019b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 -- 172.21.15.183:0/2097733627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b00108740 msgr2=0x7f2b0019b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b00108740 0x7f2b0019b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 -- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b001024b0 msgr2=0x7f2b0019ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b001024b0 0x7f2b0019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 -- 172.21.15.183:0/2097733627 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b001a60d0 con 0x7f2b00107df0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afffff640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b0019b380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2aec031a90 tx=0x7f2aec004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec03f030 con 0x7f2b00107df0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2aec003cb0 con 0x7f2b00107df0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b001a6360 con 0x7f2b00107df0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.344+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec035070 con 0x7f2b00107df0 2024-07-26T10:51:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.345+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b001a6910 con 0x7f2b00107df0 2024-07-26T10:51:43.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.346+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b00103730 con 0x7f2b00107df0 2024-07-26T10:51:43.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.348+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f2aec042030 con 0x7f2b00107df0 2024-07-26T10:51:43.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.348+0000 7f2afdffb640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2af00b3c40 0x7f2af00b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:51:43.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.349+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2244+0+0 (secure 0 0 0) 0x7f2aec102d90 con 0x7f2b00107df0 2024-07-26T10:51:43.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.349+0000 7f2b04f90640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2af00b3c40 0x7f2af00b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:43.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.350+0000 7f2b04f90640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2af00b3c40 0x7f2af00b6150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b0006c0d0 tx=0x7f2af4006c90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:51:43.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.352+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2aec0cbc70 con 0x7f2b00107df0 2024-07-26T10:51:43.535 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:51:43.533+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2b00063230 con 0x7f2af00b3c40 2024-07-26T10:51:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:51:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:51:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi160 2024-07-26T10:51:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 17088M 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:43.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] boot 2024-07-26T10:51:43.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T10:51:43.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:51:43.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:51:43.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi160 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 17088M 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] boot 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:51:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:44.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi160 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 17088M 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:44.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: osd.1 [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] boot 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:51:44.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:45.089 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:44 smithi183 ceph-mon[37273]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:51:45.089 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:44 smithi183 ceph-mon[37273]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T10:51:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:44 smithi144 ceph-mon[32129]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:51:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:44 smithi144 ceph-mon[32129]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T10:51:45.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:44 smithi160 ceph-mon[36715]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:51:45.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:44 smithi160 ceph-mon[36715]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T10:51:46.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:45 smithi183 ceph-mon[37273]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:46.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:45 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/3480165428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]: dispatch 2024-07-26T10:51:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:45 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]: dispatch 2024-07-26T10:51:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:45 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]': finished 2024-07-26T10:51:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:45 smithi183 ceph-mon[37273]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T10:51:46.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:51:46.163 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:45 smithi160 ceph-mon[36715]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:46.163 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:45 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/3480165428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]: dispatch 2024-07-26T10:51:46.163 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:45 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]: dispatch 2024-07-26T10:51:46.163 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:45 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]': finished 2024-07-26T10:51:46.163 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:45 smithi160 ceph-mon[36715]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T10:51:46.163 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:51:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:45 smithi144 ceph-mon[32129]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:45 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/3480165428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]: dispatch 2024-07-26T10:51:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:45 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]: dispatch 2024-07-26T10:51:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:45 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91ba96d0-9d5c-4128-98cf-517ac651af19"}]': finished 2024-07-26T10:51:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:45 smithi144 ceph-mon[32129]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T10:51:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:51:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:46 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/73542898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:51:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:46 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/73542898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:51:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:46 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/73542898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:51:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:47 smithi183 ceph-mon[37273]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:47 smithi144 ceph-mon[32129]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:47 smithi160 ceph-mon[36715]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:49.957 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:49 smithi183 ceph-mon[37273]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:49 smithi144 ceph-mon[32129]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:49 smithi160 ceph-mon[36715]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:52.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:51 smithi183 ceph-mon[37273]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:51 smithi144 ceph-mon[32129]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:51 smithi160 ceph-mon[36715]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T10:51:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T10:51:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T10:51:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:54.068 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:53 smithi183 ceph-mon[37273]: Deploying daemon osd.2 on smithi183 2024-07-26T10:51:54.068 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:53 smithi183 ceph-mon[37273]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:53 smithi144 ceph-mon[32129]: Deploying daemon osd.2 on smithi183 2024-07-26T10:51:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:53 smithi144 ceph-mon[32129]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:53 smithi160 ceph-mon[36715]: Deploying daemon osd.2 on smithi183 2024-07-26T10:51:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:53 smithi160 ceph-mon[36715]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:55 smithi160 ceph-mon[36715]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:55 smithi183 ceph-mon[37273]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:55 smithi144 ceph-mon[32129]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:57 smithi183 ceph-mon[37273]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:57 smithi144 ceph-mon[32129]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:57 smithi160 ceph-mon[36715]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:51:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:51:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:59 smithi183 ceph-mon[37273]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:51:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:59 smithi144 ceph-mon[32129]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:51:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:59 smithi160 ceph-mon[36715]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:00.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:00.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:00.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:51:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:00.727 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.725+0000 7f2afdffb640 1 -- 172.21.15.183:0/2097733627 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2b00063230 con 0x7f2af00b3c40 2024-07-26T10:52:00.727 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 2 on host 'smithi183' 2024-07-26T10:52:00.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2af00b3c40 msgr2=0x7f2af00b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:00.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2af00b3c40 0x7f2af00b6150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b0006c0d0 tx=0x7f2af4006c90 comp rx=0 tx=0).stop 2024-07-26T10:52:00.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 msgr2=0x7f2b0019b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:00.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b0019b380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2aec031a90 tx=0x7f2aec004270 comp rx=0 tx=0).stop 2024-07-26T10:52:00.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 shutdown_connections 2024-07-26T10:52:00.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2af00b3c40 0x7f2af00b6150 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b00108740 0x7f2b0019b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b00107df0 0x7f2b0019b380 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 --2- 172.21.15.183:0/2097733627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b001024b0 0x7f2b0019ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 >> 172.21.15.183:0/2097733627 conn(0x7f2b00079170 msgr2=0x7f2b00104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:52:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 shutdown_connections 2024-07-26T10:52:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:52:00.727+0000 7f2b071f4640 1 -- 172.21.15.183:0/2097733627 wait complete. 2024-07-26T10:52:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:01.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:01.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:01.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:01.324 DEBUG:teuthology.orchestra.run.smithi183:osd.2> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.2.service 2024-07-26T10:52:01.367 INFO:tasks.cephadm:Deploying osd.3 on smithi144 with /dev/vg_nvme/lv_3... 2024-07-26T10:52:01.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T10:52:01.820 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:52:02.167 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:01 smithi144 ceph-mon[32129]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:02.167 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:01 smithi144 ceph-mon[32129]: from='osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T10:52:02.167 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:01 smithi144 ceph-mon[32129]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T10:52:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:01 smithi160 ceph-mon[36715]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:01 smithi160 ceph-mon[36715]: from='osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T10:52:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:01 smithi160 ceph-mon[36715]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T10:52:02.330 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:01 smithi183 ceph-mon[37273]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:02.330 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:01 smithi183 ceph-mon[37273]: from='osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T10:52:02.330 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:01 smithi183 ceph-mon[37273]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T10:52:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:02 smithi144 ceph-mon[32129]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T10:52:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:02 smithi144 ceph-mon[32129]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T10:52:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:02 smithi144 ceph-mon[32129]: from='osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:52:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:02 smithi144 ceph-mon[32129]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:52:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:02 smithi160 ceph-mon[36715]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T10:52:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:03.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:02 smithi160 ceph-mon[36715]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T10:52:03.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:02 smithi160 ceph-mon[36715]: from='osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:52:03.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:02 smithi160 ceph-mon[36715]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:52:03.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-mon[37273]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T10:52:03.323 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:03.323 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-mon[37273]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T10:52:03.323 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-mon[37273]: from='osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:52:03.323 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-mon[37273]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:52:03.323 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-2[43666]: 2024-07-26T10:52:02.933+0000 7f98dd724640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T10:52:03.323 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:02 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-2[43666]: 2024-07-26T10:52:02.937+0000 7f98d9522640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:52:04.102 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T10:52:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:03 smithi144 ceph-mon[32129]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:03 smithi144 ceph-mon[32129]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:52:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:03 smithi144 ceph-mon[32129]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T10:52:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.279 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T10:52:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:03 smithi160 ceph-mon[36715]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:03 smithi160 ceph-mon[36715]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:52:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:03 smithi160 ceph-mon[36715]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T10:52:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.319 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-07-26T10:52:04.319 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-07-26T10:52:04.320 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0308908 s, 339 MB/s 2024-07-26T10:52:04.320 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-07-26T10:52:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:03 smithi183 ceph-mon[37273]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:03 smithi183 ceph-mon[37273]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:52:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:03 smithi183 ceph-mon[37273]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T10:52:04.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] boot 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi183 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 17088M 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:04.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:04.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:05.108 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi144:vg_nvme/lv_3 2024-07-26T10:52:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] boot 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi183 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 17088M 2024-07-26T10:52:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:05.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:05.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: osd.2 [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] boot 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi183 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 17088M 2024-07-26T10:52:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:05.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:05.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:05.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:05.541 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:52:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:05 smithi144 ceph-mon[32129]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T10:52:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:05 smithi144 ceph-mon[32129]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T10:52:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:05 smithi160 ceph-mon[36715]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T10:52:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:05 smithi160 ceph-mon[36715]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T10:52:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:05 smithi183 ceph-mon[37273]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T10:52:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T10:52:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:05 smithi183 ceph-mon[37273]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T10:52:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T10:52:06.884 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56276]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56276]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56276]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56276]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56282]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56282]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:06.885 INFO:journalctl@ceph.osd.0.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56282]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.015 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46263]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T10:52:07.015 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.015 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46263]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.015 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46263]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.015 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46266]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T10:52:07.016 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46266]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.016 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46266]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.016 INFO:journalctl@ceph.osd.1.smithi160.stdout:Jul 26 10:52:06 smithi160 sudo[46266]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47259]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T10:52:07.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47259]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47259]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47259]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47262]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T10:52:07.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47262]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47262]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47262]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.072 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47253]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T10:52:07.072 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47253]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.072 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47253]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.072 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47253]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.072 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47256]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T10:52:07.073 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47256]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.073 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47256]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.073 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 26 10:52:06 smithi183 sudo[47256]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56285]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T10:52:07.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56285]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56285]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56285]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56288]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T10:52:07.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.224 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56288]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:06 smithi144 sudo[56288]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T10:52:07.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T10:52:07.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:07.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.226 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46269]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46269]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46269]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46272]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46272]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46272]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 sudo[46272]: pam_unix(sudo:session): session closed for user root 2024-07-26T10:52:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T10:52:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T10:52:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T10:52:07.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T10:52:07.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T10:52:07.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.901+0000 7f8b29400640 1 -- 172.21.15.144:0/3282027783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b241108d0 msgr2=0x7f8b24110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.901+0000 7f8b29400640 1 --2- 172.21.15.144:0/3282027783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b241108d0 0x7f8b24110ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009900 tx=0x7f8b1402a410 comp rx=0 tx=0).stop 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 -- 172.21.15.144:0/3282027783 shutdown_connections 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 --2- 172.21.15.144:0/3282027783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b2406c100 0x7f8b2410f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 --2- 172.21.15.144:0/3282027783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406b6a0 0x7f8b2406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 --2- 172.21.15.144:0/3282027783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b241108d0 0x7f8b24110ce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 -- 172.21.15.144:0/3282027783 >> 172.21.15.144:0/3282027783 conn(0x7f8b2406e360 msgr2=0x7f8b24070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 -- 172.21.15.144:0/3282027783 shutdown_connections 2024-07-26T10:52:07.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.902+0000 7f8b29400640 1 -- 172.21.15.144:0/3282027783 wait complete. 2024-07-26T10:52:07.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.903+0000 7f8b29400640 1 Processor -- start 2024-07-26T10:52:07.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.903+0000 7f8b29400640 1 -- start start 2024-07-26T10:52:07.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b29400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 0x7f8b241a8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b29400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406c100 0x7f8b241a9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b29400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b241108d0 0x7f8b241ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b29400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241222f0 con 0x7f8b2406c100 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b29400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8b24122170 con 0x7f8b2406b6a0 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b29400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8b24122470 con 0x7f8b241108d0 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 0x7f8b241a8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 0x7f8b241a8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47206/0 (socket says 172.21.15.144:47206) 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 -- 172.21.15.144:0/343489215 learned_addr learned my addr 172.21.15.144:0/343489215 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b19e00640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406c100 0x7f8b241a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b23200640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b241108d0 0x7f8b241ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 -- 172.21.15.144:0/343489215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b241108d0 msgr2=0x7f8b241ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b241108d0 0x7f8b241ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 -- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406c100 msgr2=0x7f8b241a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406c100 0x7f8b241a9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b22800640 1 -- 172.21.15.144:0/343489215 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b241aec30 con 0x7f8b2406b6a0 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.904+0000 7f8b19e00640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406c100 0x7f8b241a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:52:07.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.905+0000 7f8b22800640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 0x7f8b241a8cc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005940 tx=0x7f8b140040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:52:07.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.905+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1403f030 con 0x7f8b2406b6a0 2024-07-26T10:52:07.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.905+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b140044c0 con 0x7f8b2406b6a0 2024-07-26T10:52:07.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.905+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14035070 con 0x7f8b2406b6a0 2024-07-26T10:52:07.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.905+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b241aeec0 con 0x7f8b2406b6a0 2024-07-26T10:52:07.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.905+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b241b5680 con 0x7f8b2406b6a0 2024-07-26T10:52:07.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.906+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b2410f4a0 con 0x7f8b2406b6a0 2024-07-26T10:52:07.910 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.908+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8b14042030 con 0x7f8b2406b6a0 2024-07-26T10:52:07.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.910+0000 7f8b1be00640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8b080b3c20 0x7f8b080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:07.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.910+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3452+0+0 (secure 0 0 0) 0x7f8b14102df0 con 0x7f8b2406b6a0 2024-07-26T10:52:07.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.910+0000 7f8b19e00640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8b080b3c20 0x7f8b080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:07.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.911+0000 7f8b19e00640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8b080b3c20 0x7f8b080b6130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b2406bf20 tx=0x7f8b0c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:52:07.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:07.914+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8b140cb6a0 con 0x7f8b2406b6a0 2024-07-26T10:52:08.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:08.127+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8b2410eb90 con 0x7f8b080b3c20 2024-07-26T10:52:08.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:08 smithi183 ceph-mon[37273]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T10:52:08.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:08 smithi183 ceph-mon[37273]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T10:52:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:08 smithi144 ceph-mon[32129]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T10:52:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:08 smithi144 ceph-mon[32129]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T10:52:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:08 smithi160 ceph-mon[36715]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T10:52:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:08 smithi160 ceph-mon[36715]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T10:52:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:09 smithi183 ceph-mon[37273]: from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:52:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:52:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:52:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:09.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:09 smithi183 ceph-mon[37273]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:09 smithi144 ceph-mon[32129]: from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:52:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:52:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:52:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:09 smithi144 ceph-mon[32129]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:09 smithi160 ceph-mon[36715]: from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:52:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:52:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:52:09.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:09.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:09 smithi160 ceph-mon[36715]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:12.197 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:11 smithi144 ceph-mon[32129]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:12.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:11 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3490123972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]: dispatch 2024-07-26T10:52:12.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:11 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]: dispatch 2024-07-26T10:52:12.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:11 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]': finished 2024-07-26T10:52:12.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:12.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:11 smithi144 ceph-mon[32129]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T10:52:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:11 smithi160 ceph-mon[36715]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:11 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3490123972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]: dispatch 2024-07-26T10:52:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:11 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]: dispatch 2024-07-26T10:52:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:11 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]': finished 2024-07-26T10:52:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:11 smithi160 ceph-mon[36715]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T10:52:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:11 smithi183 ceph-mon[37273]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:11 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3490123972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]: dispatch 2024-07-26T10:52:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:11 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]: dispatch 2024-07-26T10:52:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:11 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e932bb8c-4b48-4cf1-8f02-4c151c98a009"}]': finished 2024-07-26T10:52:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:12.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:11 smithi183 ceph-mon[37273]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T10:52:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:12 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1620267700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:52:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:12 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1620267700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:52:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:12 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1620267700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:52:14.211 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:13 smithi160 ceph-mon[36715]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:13 smithi144 ceph-mon[32129]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:13 smithi183 ceph-mon[37273]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:16.182 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:15 smithi144 ceph-mon[32129]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:15 smithi160 ceph-mon[36715]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:15 smithi183 ceph-mon[37273]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:18.132 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:17 smithi144 ceph-mon[32129]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:17 smithi160 ceph-mon[36715]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:17 smithi183 ceph-mon[37273]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:19 smithi144 ceph-mon[32129]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:19 smithi160 ceph-mon[36715]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:19 smithi183 ceph-mon[37273]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T10:52:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T10:52:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T10:52:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:21 smithi144 ceph-mon[32129]: Deploying daemon osd.3 on smithi144 2024-07-26T10:52:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:21 smithi144 ceph-mon[32129]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:21 smithi160 ceph-mon[36715]: Deploying daemon osd.3 on smithi144 2024-07-26T10:52:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:21 smithi160 ceph-mon[36715]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:21 smithi183 ceph-mon[37273]: Deploying daemon osd.3 on smithi144 2024-07-26T10:52:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:21 smithi183 ceph-mon[37273]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:23 smithi144 ceph-mon[32129]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:23 smithi160 ceph-mon[36715]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:23 smithi183 ceph-mon[37273]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:25 smithi144 ceph-mon[32129]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:25 smithi160 ceph-mon[36715]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:25 smithi183 ceph-mon[37273]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:27 smithi144 ceph-mon[32129]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:27 smithi160 ceph-mon[36715]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:27 smithi183 ceph-mon[37273]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:29 smithi144 ceph-mon[32129]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:29 smithi160 ceph-mon[36715]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:29 smithi183 ceph-mon[37273]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:32.239 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:31 smithi144 ceph-mon[32129]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:31 smithi160 ceph-mon[36715]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:31 smithi183 ceph-mon[37273]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T10:52:33.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:33.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:33.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:34.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T10:52:34.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:34.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:34.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:34.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.158+0000 7f8b1be00640 1 -- 172.21.15.144:0/343489215 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8b2410eb90 con 0x7f8b080b3c20 2024-07-26T10:52:34.162 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 3 on host 'smithi144' 2024-07-26T10:52:34.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8b080b3c20 msgr2=0x7f8b080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:34.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8b080b3c20 0x7f8b080b6130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b2406bf20 tx=0x7f8b0c005f90 comp rx=0 tx=0).stop 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 msgr2=0x7f8b241a8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 0x7f8b241a8cc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005940 tx=0x7f8b140040c0 comp rx=0 tx=0).stop 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 shutdown_connections 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8b080b3c20 0x7f8b080b6130 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b241108d0 0x7f8b241ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b2406c100 0x7f8b241a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 --2- 172.21.15.144:0/343489215 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b2406b6a0 0x7f8b241a8cc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 >> 172.21.15.144:0/343489215 conn(0x7f8b2406e360 msgr2=0x7f8b2410d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.162+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 shutdown_connections 2024-07-26T10:52:34.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:52:34.163+0000 7f8b29400640 1 -- 172.21.15.144:0/343489215 wait complete. 2024-07-26T10:52:34.884 DEBUG:teuthology.orchestra.run.smithi144:osd.3> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.3.service 2024-07-26T10:52:34.888 INFO:tasks.cephadm:Deploying osd.4 on smithi160 with /dev/vg_nvme/lv_3... 2024-07-26T10:52:34.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T10:52:35.074 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T10:52:35.074 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:35.074 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T10:52:35.074 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:52:35.074 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:35.074 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:35.075 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T10:52:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T10:52:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:52:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:35.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:35.149 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:52:35.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T10:52:35.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:35.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T10:52:35.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:52:35.271 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:35.271 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:35.271 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] boot 2024-07-26T10:52:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:35 smithi160 ceph-mon[36715]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] boot 2024-07-26T10:52:36.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:35 smithi183 ceph-mon[37273]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: from='osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: osd.3 [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] boot 2024-07-26T10:52:36.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:35 smithi144 ceph-mon[32129]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T10:52:36.933 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T10:52:37.013 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T10:52:37.027 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-07-26T10:52:37.027 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-07-26T10:52:37.027 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114518 s, 916 MB/s 2024-07-26T10:52:37.027 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-07-26T10:52:37.325 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:37 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:52:37.325 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:37 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:52:37.325 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:37 smithi160 ceph-mon[36715]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T10:52:37.325 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:37 smithi160 ceph-mon[36715]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:37 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:52:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:37 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:52:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:37 smithi183 ceph-mon[37273]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T10:52:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:37 smithi183 ceph-mon[37273]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:37 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:52:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:37 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:52:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:37 smithi144 ceph-mon[32129]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T10:52:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:37 smithi144 ceph-mon[32129]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:37.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_3 2024-07-26T10:52:37.881 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:52:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:38 smithi144 ceph-mon[32129]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T10:52:38.301 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:38 smithi160 ceph-mon[36715]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T10:52:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:38 smithi183 ceph-mon[37273]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T10:52:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:39 smithi144 ceph-mon[32129]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-07-26T10:52:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:39 smithi160 ceph-mon[36715]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-07-26T10:52:39.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.342+0000 7ff5886bb640 1 -- 172.21.15.160:0/1801017302 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 msgr2=0x7ff580108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:39.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.342+0000 7ff5886bb640 1 --2- 172.21.15.160:0/1801017302 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff580108220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff570009900 tx=0x7ff57002a410 comp rx=0 tx=0).stop 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- 172.21.15.160:0/1801017302 shutdown_connections 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 --2- 172.21.15.160:0/1801017302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 0x7ff58010ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 --2- 172.21.15.160:0/1801017302 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff580108220 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 --2- 172.21.15.160:0/1801017302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5801024d0 0x7ff5801028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- 172.21.15.160:0/1801017302 >> 172.21.15.160:0/1801017302 conn(0x7ff580079180 msgr2=0x7ff580079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- 172.21.15.160:0/1801017302 shutdown_connections 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- 172.21.15.160:0/1801017302 wait complete. 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 Processor -- start 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- start start 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5801024d0 0x7ff58019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff58019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 0x7ff58019f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff5801135c0 con 0x7ff5801024d0 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff580113440 con 0x7ff580107e10 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.343+0000 7ff5886bb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff580113740 con 0x7ff580108760 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff58019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff58019a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:41558/0 (socket says 172.21.15.160:41558) 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 -- 172.21.15.160:0/2626173923 learned_addr learned my addr 172.21.15.160:0/2626173923 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff586c58640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 0x7ff58019f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 -- 172.21.15.160:0/2626173923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 msgr2=0x7ff58019f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 0x7ff58019f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff586457640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5801024d0 0x7ff58019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 -- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5801024d0 msgr2=0x7ff58019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5801024d0 0x7ff58019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 -- 172.21.15.160:0/2626173923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5801a0000 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff586c58640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 0x7ff58019f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff585c56640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff58019a630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff57002a930 tx=0x7ff570004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57003f030 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff570003cb0 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5801a0290 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570035070 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.344+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5801a6a50 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.345+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff580102960 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.347+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7ff570042030 con 0x7ff580107e10 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.347+0000 7ff5777fe640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff56c0b3c20 0x7ff56c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:52:39.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.347+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3884+0+0 (secure 0 0 0) 0x7ff570103600 con 0x7ff580107e10 2024-07-26T10:52:39.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.348+0000 7ff586457640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff56c0b3c20 0x7ff56c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:39.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.348+0000 7ff586457640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff56c0b3c20 0x7ff56c0b6130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00a810 tx=0x7ff57c008040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:52:39.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.350+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff5700cbda0 con 0x7ff580107e10 2024-07-26T10:52:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:39 smithi183 ceph-mon[37273]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-07-26T10:52:39.535 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:39.533+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff580063230 con 0x7ff56c0b3c20 2024-07-26T10:52:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi144 2024-07-26T10:52:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 8544M 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:52:40.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi144 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 8544M 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:52:40.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:52:40.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:52:40.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi144 2024-07-26T10:52:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 8544M 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:52:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:41 smithi160 ceph-mon[36715]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T10:52:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:41 smithi144 ceph-mon[32129]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T10:52:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:41 smithi183 ceph-mon[37273]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T10:52:42.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:42 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/3149485203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]: dispatch 2024-07-26T10:52:42.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:42 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]: dispatch 2024-07-26T10:52:42.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:42 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]': finished 2024-07-26T10:52:42.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:52:42.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:42 smithi160 ceph-mon[36715]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T10:52:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:42 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/3149485203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]: dispatch 2024-07-26T10:52:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:42 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]: dispatch 2024-07-26T10:52:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:42 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]': finished 2024-07-26T10:52:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:52:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:42 smithi144 ceph-mon[32129]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T10:52:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:42 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/3149485203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]: dispatch 2024-07-26T10:52:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:42 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]: dispatch 2024-07-26T10:52:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:42 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6f12400-2f76-4c60-88e2-d06235484603"}]': finished 2024-07-26T10:52:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:52:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:42 smithi183 ceph-mon[37273]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T10:52:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:43 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/2574864467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:52:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:43 smithi144 ceph-mon[32129]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-07-26T10:52:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:43 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/2574864467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:52:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:43 smithi160 ceph-mon[36715]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-07-26T10:52:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:43 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/2574864467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:52:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:43 smithi183 ceph-mon[37273]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-07-26T10:52:46.125 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:45 smithi160 ceph-mon[36715]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T10:52:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:45 smithi144 ceph-mon[32129]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T10:52:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:45 smithi183 ceph-mon[37273]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T10:52:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:47 smithi144 ceph-mon[32129]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-07-26T10:52:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:47 smithi160 ceph-mon[36715]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-07-26T10:52:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:47 smithi183 ceph-mon[37273]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-07-26T10:52:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T10:52:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T10:52:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T10:52:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:49 smithi144 ceph-mon[32129]: Deploying daemon osd.4 on smithi160 2024-07-26T10:52:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:49 smithi144 ceph-mon[32129]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:50.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:49 smithi160 ceph-mon[36715]: Deploying daemon osd.4 on smithi160 2024-07-26T10:52:50.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:49 smithi160 ceph-mon[36715]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:49 smithi183 ceph-mon[37273]: Deploying daemon osd.4 on smithi160 2024-07-26T10:52:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:49 smithi183 ceph-mon[37273]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:52.202 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:51 smithi160 ceph-mon[36715]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:51 smithi144 ceph-mon[32129]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:51 smithi183 ceph-mon[37273]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:53 smithi144 ceph-mon[32129]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:54.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:53 smithi160 ceph-mon[36715]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:54.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:54.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:54.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:53 smithi183 ceph-mon[37273]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:55 smithi144 ceph-mon[32129]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:55 smithi160 ceph-mon[36715]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:55 smithi183 ceph-mon[37273]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:57.451 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 4 on host 'smithi160' 2024-07-26T10:52:57.452 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.449+0000 7ff5777fe640 1 -- 172.21.15.160:0/2626173923 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff580063230 con 0x7ff56c0b3c20 2024-07-26T10:52:57.454 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff56c0b3c20 msgr2=0x7ff56c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:57.454 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff56c0b3c20 0x7ff56c0b6130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00a810 tx=0x7ff57c008040 comp rx=0 tx=0).stop 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 msgr2=0x7ff58019a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff58019a630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff57002a930 tx=0x7ff570004270 comp rx=0 tx=0).stop 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 shutdown_connections 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff56c0b3c20 0x7ff56c0b6130 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff580108760 0x7ff58019f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff580107e10 0x7ff58019a630 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 --2- 172.21.15.160:0/2626173923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5801024d0 0x7ff58019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 >> 172.21.15.160:0/2626173923 conn(0x7ff580079180 msgr2=0x7ff580105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 shutdown_connections 2024-07-26T10:52:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:52:57.452+0000 7ff5886bb640 1 -- 172.21.15.160:0/2626173923 wait complete. 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:57.774 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:57.775 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.775 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:57.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:52:58.064 DEBUG:teuthology.orchestra.run.smithi160:osd.4> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.4.service 2024-07-26T10:52:58.067 INFO:tasks.cephadm:Deploying osd.5 on smithi183 with /dev/vg_nvme/lv_3... 2024-07-26T10:52:58.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T10:52:58.323 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:52:58.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:58 smithi183 ceph-mon[37273]: from='osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T10:52:58.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:58 smithi183 ceph-mon[37273]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T10:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:58 smithi144 ceph-mon[32129]: from='osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T10:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:58 smithi144 ceph-mon[32129]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T10:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:58 smithi160 ceph-mon[36715]: from='osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T10:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:58 smithi160 ceph-mon[36715]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T10:52:59.722 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:59 smithi183 ceph-mon[37273]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T10:52:59.722 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:52:59.722 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:59 smithi183 ceph-mon[37273]: from='osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:52:59.722 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:59 smithi183 ceph-mon[37273]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T10:52:59.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:59 smithi183 ceph-mon[37273]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:52:59.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:52:59 smithi183 ceph-mon[37273]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:52:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:59 smithi144 ceph-mon[32129]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T10:52:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:52:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:59 smithi144 ceph-mon[32129]: from='osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:52:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:59 smithi144 ceph-mon[32129]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T10:52:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:59 smithi144 ceph-mon[32129]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:52:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:52:59 smithi144 ceph-mon[32129]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:53:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:59 smithi160 ceph-mon[36715]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T10:53:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:59 smithi160 ceph-mon[36715]: from='osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:53:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:59 smithi160 ceph-mon[36715]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T10:53:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:59 smithi160 ceph-mon[36715]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:53:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:52:59 smithi160 ceph-mon[36715]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:53:00.075 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T10:53:00.194 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T10:53:00.207 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-07-26T10:53:00.207 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-07-26T10:53:00.207 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113095 s, 927 MB/s 2024-07-26T10:53:00.207 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-07-26T10:53:00.560 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-4[49050]: 2024-07-26T10:53:00.408+0000 7f79dd55f640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T10:53:00.560 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-4[49050]: 2024-07-26T10:53:00.416+0000 7f79d5355640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:53:00.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-mon[36715]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:53:00.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:00.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-mon[36715]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T10:53:00.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-mon[36715]: from='osd.4 ' entity='osd.4' 2024-07-26T10:53:00.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:00.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-07-26T10:53:00.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:00 smithi183 ceph-mon[37273]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:53:00.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:00.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:00 smithi183 ceph-mon[37273]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T10:53:00.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:00 smithi183 ceph-mon[37273]: from='osd.4 ' entity='osd.4' 2024-07-26T10:53:00.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:00 smithi144 ceph-mon[32129]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:53:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:00 smithi144 ceph-mon[32129]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T10:53:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:00 smithi144 ceph-mon[32129]: from='osd.4 ' entity='osd.4' 2024-07-26T10:53:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:01.169 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:53:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:53:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:53:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi160 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 8544M 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:01.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] boot 2024-07-26T10:53:01.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:01 smithi183 ceph-mon[37273]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T10:53:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:53:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi160 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 8544M 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:01.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:01.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:01.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:01.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] boot 2024-07-26T10:53:01.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:01 smithi144 ceph-mon[32129]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi160 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 8544M 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: osd.4 [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] boot 2024-07-26T10:53:02.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:01 smithi160 ceph-mon[36715]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T10:53:02.699 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 -- 172.21.15.183:0/1351960178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 msgr2=0x7fdf1c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 --2- 172.21.15.183:0/1351960178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c10ef50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009ca0 tx=0x7fdf1402a8c0 comp rx=0 tx=0).stop 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 -- 172.21.15.183:0/1351960178 shutdown_connections 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 --2- 172.21.15.183:0/1351960178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c10ef50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 --2- 172.21.15.183:0/1351960178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdf1c107dd0 0x7fdf1c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 --2- 172.21.15.183:0/1351960178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf1c102490 0x7fdf1c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 -- 172.21.15.183:0/1351960178 >> 172.21.15.183:0/1351960178 conn(0x7fdf1c079160 msgr2=0x7fdf1c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 -- 172.21.15.183:0/1351960178 shutdown_connections 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.695+0000 7fdf21a06640 1 -- 172.21.15.183:0/1351960178 wait complete. 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.696+0000 7fdf21a06640 1 Processor -- start 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.696+0000 7fdf21a06640 1 -- start start 2024-07-26T10:53:02.700 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.696+0000 7fdf21a06640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf1c102490 0x7fdf1c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdf1c107dd0 0x7fdf1c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1134f0 con 0x7fdf1c102490 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c113370 con 0x7fdf1c107dd0 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c113670 con 0x7fdf1c108720 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:55724/0 (socket says 172.21.15.183:55724) 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 -- 172.21.15.183:0/2966391817 learned_addr learned my addr 172.21.15.183:0/2966391817 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1a7fc640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdf1c107dd0 0x7fdf1c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 -- 172.21.15.183:0/2966391817 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdf1c107dd0 msgr2=0x7fdf1c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdf1c107dd0 0x7fdf1c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 -- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf1c102490 msgr2=0x7fdf1c199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:02.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1affd640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf1c102490 0x7fdf1c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf1c102490 0x7fdf1c199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 -- 172.21.15.183:0/2966391817 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1c19fe30 con 0x7fdf1c108720 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf1b7fe640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c19f700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf14005cc0 tx=0x7fdf1402f380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf14003bf0 con 0x7fdf1c108720 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf14003d90 con 0x7fdf1c108720 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1c1a00c0 con 0x7fdf1c108720 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1402fbc0 con 0x7fdf1c108720 2024-07-26T10:53:02.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.697+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf1c1a6880 con 0x7fdf1c108720 2024-07-26T10:53:02.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.699+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1c103710 con 0x7fdf1c108720 2024-07-26T10:53:02.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.700+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fdf1402fd60 con 0x7fdf1c108720 2024-07-26T10:53:02.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.701+0000 7fdf20a04640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdf040b3ca0 0x7fdf040b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:02.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.701+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4316+0+0 (secure 0 0 0) 0x7fdf14104550 con 0x7fdf1c108720 2024-07-26T10:53:02.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.701+0000 7fdf1affd640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdf040b3ca0 0x7fdf040b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:02.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.703+0000 7fdf1affd640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdf040b3ca0 0x7fdf040b61b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00a810 tx=0x7fdf0c008040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:53:02.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.704+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdf140ccbd0 con 0x7fdf1c108720 2024-07-26T10:53:02.888 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:02.887+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fdf1c063230 con 0x7fdf040b3ca0 2024-07-26T10:53:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:03 smithi183 ceph-mon[37273]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T10:53:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:03 smithi183 ceph-mon[37273]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:03 smithi183 ceph-mon[37273]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:53:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:53:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:53:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:03 smithi144 ceph-mon[32129]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T10:53:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:03 smithi144 ceph-mon[32129]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:03 smithi144 ceph-mon[32129]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:53:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:53:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:53:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:03 smithi160 ceph-mon[36715]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T10:53:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:03 smithi160 ceph-mon[36715]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:03 smithi160 ceph-mon[36715]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:53:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:53:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:53:04.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/2292784617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]: dispatch 2024-07-26T10:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]: dispatch 2024-07-26T10:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]': finished 2024-07-26T10:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T10:53:06.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:05 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/373960073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/2292784617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]: dispatch 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]: dispatch 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]': finished 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T10:53:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:05 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/373960073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:53:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/2292784617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]: dispatch 2024-07-26T10:53:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]: dispatch 2024-07-26T10:53:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b45f883-af04-42bd-a38e-38e88cc49515"}]': finished 2024-07-26T10:53:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T10:53:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:05 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/373960073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:53:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:07 smithi144 ceph-mon[32129]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:07 smithi160 ceph-mon[36715]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:07 smithi183 ceph-mon[37273]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:09 smithi144 ceph-mon[32129]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:10.258 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:09 smithi183 ceph-mon[37273]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:09 smithi160 ceph-mon[36715]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:12.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:11 smithi183 ceph-mon[37273]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:11 smithi144 ceph-mon[32129]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:11 smithi160 ceph-mon[36715]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T10:53:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T10:53:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:13.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T10:53:13.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:14.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:13 smithi183 ceph-mon[37273]: Deploying daemon osd.5 on smithi183 2024-07-26T10:53:14.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:13 smithi183 ceph-mon[37273]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:13 smithi144 ceph-mon[32129]: Deploying daemon osd.5 on smithi183 2024-07-26T10:53:14.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:13 smithi144 ceph-mon[32129]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:13 smithi160 ceph-mon[36715]: Deploying daemon osd.5 on smithi183 2024-07-26T10:53:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:13 smithi160 ceph-mon[36715]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:15 smithi144 ceph-mon[32129]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:15 smithi160 ceph-mon[36715]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:16.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:15 smithi183 ceph-mon[37273]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:17 smithi144 ceph-mon[32129]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:17 smithi160 ceph-mon[36715]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:18.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:18.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:17 smithi183 ceph-mon[37273]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:19 smithi160 ceph-mon[36715]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:20.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:20.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:19 smithi183 ceph-mon[37273]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:19 smithi144 ceph-mon[32129]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:20.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.766+0000 7fdf20a04640 1 -- 172.21.15.183:0/2966391817 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdf1c063230 con 0x7fdf040b3ca0 2024-07-26T10:53:20.769 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 5 on host 'smithi183' 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdf040b3ca0 msgr2=0x7fdf040b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdf040b3ca0 0x7fdf040b61b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00a810 tx=0x7fdf0c008040 comp rx=0 tx=0).stop 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 msgr2=0x7fdf1c19f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c19f700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf14005cc0 tx=0x7fdf1402f380 comp rx=0 tx=0).stop 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 shutdown_connections 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdf040b3ca0 0x7fdf040b61b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:20.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c108720 0x7fdf1c19f700 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:20.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdf1c107dd0 0x7fdf1c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:20.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 --2- 172.21.15.183:0/2966391817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf1c102490 0x7fdf1c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:20.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.768+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 >> 172.21.15.183:0/2966391817 conn(0x7fdf1c079160 msgr2=0x7fdf1c104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:53:20.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.769+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 shutdown_connections 2024-07-26T10:53:20.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:53:20.769+0000 7fdf21a06640 1 -- 172.21.15.183:0/2966391817 wait complete. 2024-07-26T10:53:21.300 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:21.300 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:21.300 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:21.327 DEBUG:teuthology.orchestra.run.smithi183:osd.5> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.5.service 2024-07-26T10:53:21.370 INFO:tasks.cephadm:Deploying osd.6 on smithi144 with /dev/vg_nvme/lv_2... 2024-07-26T10:53:21.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T10:53:21.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:21.449 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:21.449 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:21.827 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:53:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:21 smithi160 ceph-mon[36715]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:21 smithi160 ceph-mon[36715]: from='osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T10:53:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:21 smithi160 ceph-mon[36715]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T10:53:22.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:21 smithi144 ceph-mon[32129]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:22.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:21 smithi144 ceph-mon[32129]: from='osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T10:53:22.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:21 smithi144 ceph-mon[32129]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T10:53:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:21 smithi183 ceph-mon[37273]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:21 smithi183 ceph-mon[37273]: from='osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T10:53:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:21 smithi183 ceph-mon[37273]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: from='osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:22 smithi144 ceph-mon[32129]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T10:53:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T10:53:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: from='osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:53:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T10:53:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:53:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:53:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:22 smithi160 ceph-mon[36715]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: from='osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-mon[37273]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T10:53:23.377 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 26 10:53:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-5[50119]: 2024-07-26T10:53:22.991+0000 7ff30195b640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T10:53:23.378 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 26 10:53:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-5[50119]: 2024-07-26T10:53:22.999+0000 7ff2fd759640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:53:24.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:23 smithi183 ceph-mon[37273]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:24.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:24.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:23 smithi183 ceph-mon[37273]: osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] boot 2024-07-26T10:53:24.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:23 smithi183 ceph-mon[37273]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T10:53:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:23 smithi160 ceph-mon[36715]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:23 smithi160 ceph-mon[36715]: osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] boot 2024-07-26T10:53:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:23 smithi160 ceph-mon[36715]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T10:53:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:23 smithi144 ceph-mon[32129]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T10:53:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:23 smithi144 ceph-mon[32129]: osd.5 [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] boot 2024-07-26T10:53:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:23 smithi144 ceph-mon[32129]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T10:53:24.542 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T10:53:24.718 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T10:53:24.757 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-07-26T10:53:24.757 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-07-26T10:53:24.757 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0317944 s, 330 MB/s 2024-07-26T10:53:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-07-26T10:53:25.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:53:25.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:53:25.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.171 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:25 smithi144 ceph-mon[32129]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T10:53:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:25 smithi160 ceph-mon[36715]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:25 smithi183 ceph-mon[37273]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T10:53:25.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi144:vg_nvme/lv_2 2024-07-26T10:53:26.083 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:53:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:26 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi183 2024-07-26T10:53:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:26 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 8544M 2024-07-26T10:53:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:26 smithi160 ceph-mon[36715]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:26 smithi160 ceph-mon[36715]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T10:53:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:26 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi183 2024-07-26T10:53:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:26 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 8544M 2024-07-26T10:53:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:26 smithi183 ceph-mon[37273]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:26 smithi183 ceph-mon[37273]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T10:53:26.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:26 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi183 2024-07-26T10:53:26.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:26 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 8544M 2024-07-26T10:53:26.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:26 smithi144 ceph-mon[32129]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:26.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:26 smithi144 ceph-mon[32129]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T10:53:28.197 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:28 smithi144 ceph-mon[32129]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:28 smithi160 ceph-mon[36715]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:28 smithi183 ceph-mon[37273]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 -- 172.21.15.144:0/2707343828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b8101a50 msgr2=0x7f94b810fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 --2- 172.21.15.144:0/2707343828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b8101a50 0x7f94b810fdc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009cd0 tx=0x7f94b002a8f0 comp rx=0 tx=0).stop 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 -- 172.21.15.144:0/2707343828 shutdown_connections 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 --2- 172.21.15.144:0/2707343828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b8101a50 0x7f94b810fdc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 --2- 172.21.15.144:0/2707343828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 0x7f94b8101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 --2- 172.21.15.144:0/2707343828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b81090f0 0x7f94b8109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 -- 172.21.15.144:0/2707343828 >> 172.21.15.144:0/2707343828 conn(0x7f94b80780b0 msgr2=0x7f94b80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 -- 172.21.15.144:0/2707343828 shutdown_connections 2024-07-26T10:53:28.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.401+0000 7f94bce00640 1 -- 172.21.15.144:0/2707343828 wait complete. 2024-07-26T10:53:28.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 Processor -- start 2024-07-26T10:53:28.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 -- start start 2024-07-26T10:53:28.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 0x7f94b8102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b8101a50 0x7f94b8103100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 0x7f94b8103640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f94b8113350 con 0x7f94b81090f0 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94b81131d0 con 0x7f94b8101a50 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.402+0000 7f94bce00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f94b81134d0 con 0x7f94b8101100 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 0x7f94b8103640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 0x7f94b8103640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:36488/0 (socket says 172.21.15.144:36488) 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7e00640 1 -- 172.21.15.144:0/1868740515 learned_addr learned my addr 172.21.15.144:0/1868740515 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7400640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 0x7f94b8102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:28.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b6a00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b8101a50 0x7f94b8103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:28.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7400640 1 -- 172.21.15.144:0/1868740515 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b8101a50 msgr2=0x7f94b8103100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:28.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7400640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b8101a50 0x7f94b8103100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:28.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7400640 1 -- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 msgr2=0x7f94b8103640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7400640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 0x7f94b8103640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7400640 1 -- 172.21.15.144:0/1868740515 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b81a8020 con 0x7f94b8101100 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b7e00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 0x7f94b8103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.403+0000 7f94b6a00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b8101a50 0x7f94b8103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.404+0000 7f94b7400640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 0x7f94b8102bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94a800e9f0 tx=0x7f94a800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.404+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a800cd80 con 0x7f94b8101100 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.404+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94a80035f0 con 0x7f94b8101100 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.404+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b81a8310 con 0x7f94b8101100 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.404+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8010670 con 0x7f94b8101100 2024-07-26T10:53:28.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.404+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b81a8850 con 0x7f94b8101100 2024-07-26T10:53:28.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.405+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b8101510 con 0x7f94b8101100 2024-07-26T10:53:28.410 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.408+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f94a80037a0 con 0x7f94b8101100 2024-07-26T10:53:28.412 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.409+0000 7f94afe00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f949c0b3c20 0x7f949c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:53:28.412 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.409+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4748+0+0 (secure 0 0 0) 0x7f94a80e3160 con 0x7f94b8101100 2024-07-26T10:53:28.412 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.409+0000 7f94b6a00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f949c0b3c20 0x7f949c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:28.413 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.410+0000 7f94b6a00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f949c0b3c20 0x7f949c0b6130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f94b8108be0 tx=0x7f94a4005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:53:28.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.413+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f94a80ab500 con 0x7f94b8101100 2024-07-26T10:53:28.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:28.620+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f94b8063230 con 0x7f949c0b3c20 2024-07-26T10:53:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:53:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:53:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:29.344 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:53:29.344 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:53:29.344 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:53:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:53:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:30 smithi160 ceph-mon[36715]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:53:30.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:30 smithi160 ceph-mon[36715]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-26T10:53:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:30 smithi183 ceph-mon[37273]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:53:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:30 smithi183 ceph-mon[37273]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-26T10:53:30.388 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:30 smithi144 ceph-mon[32129]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:53:30.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:30 smithi144 ceph-mon[32129]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-26T10:53:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:32 smithi160 ceph-mon[36715]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-07-26T10:53:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:32 smithi183 ceph-mon[37273]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-07-26T10:53:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:32 smithi144 ceph-mon[32129]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-07-26T10:53:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:33 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2193412167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aedef072-4034-47cb-bab6-11f1fd44d82a"}]: dispatch 2024-07-26T10:53:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:33 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2193412167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aedef072-4034-47cb-bab6-11f1fd44d82a"}]': finished 2024-07-26T10:53:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:33 smithi160 ceph-mon[36715]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T10:53:33.359 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:33 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2193412167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aedef072-4034-47cb-bab6-11f1fd44d82a"}]: dispatch 2024-07-26T10:53:33.359 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:33 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2193412167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aedef072-4034-47cb-bab6-11f1fd44d82a"}]': finished 2024-07-26T10:53:33.359 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:33.359 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:33 smithi144 ceph-mon[32129]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T10:53:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:33 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2193412167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aedef072-4034-47cb-bab6-11f1fd44d82a"}]: dispatch 2024-07-26T10:53:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:33 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2193412167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aedef072-4034-47cb-bab6-11f1fd44d82a"}]': finished 2024-07-26T10:53:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:33 smithi183 ceph-mon[37273]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T10:53:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:34 smithi160 ceph-mon[36715]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-26T10:53:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:34 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1391525434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:53:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:34 smithi183 ceph-mon[37273]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-26T10:53:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:34 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1391525434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:53:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:34 smithi144 ceph-mon[32129]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-26T10:53:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:34 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1391525434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:53:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:36 smithi144 ceph-mon[32129]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-26T10:53:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:36 smithi160 ceph-mon[36715]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-26T10:53:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:36 smithi183 ceph-mon[37273]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-26T10:53:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:38 smithi160 ceph-mon[36715]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:53:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:38 smithi183 ceph-mon[37273]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:53:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:38 smithi144 ceph-mon[32129]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:53:40.304 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:40 smithi144 ceph-mon[32129]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:40 smithi160 ceph-mon[36715]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:40 smithi183 ceph-mon[37273]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:42 smithi160 ceph-mon[36715]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T10:53:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:42.350 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:42 smithi144 ceph-mon[32129]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:42.350 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T10:53:42.350 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:42 smithi183 ceph-mon[37273]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T10:53:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:43.328 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:43 smithi144 ceph-mon[32129]: Deploying daemon osd.6 on smithi144 2024-07-26T10:53:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:43 smithi183 ceph-mon[37273]: Deploying daemon osd.6 on smithi144 2024-07-26T10:53:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:43 smithi160 ceph-mon[36715]: Deploying daemon osd.6 on smithi144 2024-07-26T10:53:44.358 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:44 smithi144 ceph-mon[32129]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:44 smithi183 ceph-mon[37273]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:44 smithi160 ceph-mon[36715]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:45.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:45 smithi144 ceph-mon[32129]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:45 smithi183 ceph-mon[37273]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:45 smithi160 ceph-mon[36715]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:47 smithi144 ceph-mon[32129]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:48.263 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:47 smithi160 ceph-mon[36715]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:47 smithi183 ceph-mon[37273]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:50.100 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:49 smithi144 ceph-mon[32129]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:49 smithi160 ceph-mon[36715]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:49 smithi183 ceph-mon[37273]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:51.008 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:51.008 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:51.008 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:52.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:51 smithi144 ceph-mon[32129]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:51 smithi160 ceph-mon[36715]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:51 smithi183 ceph-mon[37273]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:53 smithi144 ceph-mon[32129]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:53 smithi160 ceph-mon[36715]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:53 smithi183 ceph-mon[37273]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:54 smithi144 ceph-mon[32129]: from='osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T10:53:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:54 smithi144 ceph-mon[32129]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T10:53:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:54 smithi160 ceph-mon[36715]: from='osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T10:53:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:54 smithi160 ceph-mon[36715]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T10:53:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:54 smithi183 ceph-mon[37273]: from='osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T10:53:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:54 smithi183 ceph-mon[37273]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T10:53:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:55.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.630+0000 7f94afe00640 1 -- 172.21.15.144:0/1868740515 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f94b8063230 con 0x7f949c0b3c20 2024-07-26T10:53:55.632 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 6 on host 'smithi144' 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.634+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f949c0b3c20 msgr2=0x7f949c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.634+0000 7f94bce00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f949c0b3c20 0x7f949c0b6130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f94b8108be0 tx=0x7f94a4005f90 comp rx=0 tx=0).stop 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 msgr2=0x7f94b8102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 0x7f94b8102bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94a800e9f0 tx=0x7f94a800eeb0 comp rx=0 tx=0).stop 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 shutdown_connections 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f949c0b3c20 0x7f949c0b6130 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:55.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94b81090f0 0x7f94b8103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:55.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94b8101a50 0x7f94b8103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:55.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 --2- 172.21.15.144:0/1868740515 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94b8101100 0x7f94b8102bc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:53:55.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 >> 172.21.15.144:0/1868740515 conn(0x7f94b80780b0 msgr2=0x7f94b80ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:53:55.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 shutdown_connections 2024-07-26T10:53:55.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:53:55.635+0000 7f94bce00640 1 -- 172.21.15.144:0/1868740515 wait complete. 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:56.356 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:56.356 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T10:53:56.356 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:56.356 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:53:56.357 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T10:53:56.357 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:53:56.357 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:56.357 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:56.357 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:53:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:56.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:56.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:53:56.420 DEBUG:teuthology.orchestra.run.smithi144:osd.6> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.6.service 2024-07-26T10:53:56.427 INFO:tasks.cephadm:Deploying osd.7 on smithi160 with /dev/vg_nvme/lv_2... 2024-07-26T10:53:56.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T10:53:56.686 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:53:56.982 INFO:journalctl@ceph.osd.6.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-6[65086]: 2024-07-26T10:53:56.855+0000 7fabd6000640 -1 osd.6 0 waiting for initial osdmap 2024-07-26T10:53:56.983 INFO:journalctl@ceph.osd.6.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-6[65086]: 2024-07-26T10:53:56.887+0000 7fabcc000640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:53:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:56 smithi160 ceph-mon[36715]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:53:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:57.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:56 smithi160 ceph-mon[36715]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T10:53:57.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:57.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:56 smithi160 ceph-mon[36715]: from='osd.6 ' entity='osd.6' 2024-07-26T10:53:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:56 smithi183 ceph-mon[37273]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:53:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:56 smithi183 ceph-mon[37273]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T10:53:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:57.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:56 smithi183 ceph-mon[37273]: from='osd.6 ' entity='osd.6' 2024-07-26T10:53:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-mon[32129]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:53:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-mon[32129]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T10:53:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:56 smithi144 ceph-mon[32129]: from='osd.6 ' entity='osd.6' 2024-07-26T10:53:58.062 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:53:58.063 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:53:58.063 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:58.063 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:58.063 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:58.063 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] boot 2024-07-26T10:53:58.063 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:57 smithi144 ceph-mon[32129]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T10:53:58.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:53:58.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:53:58.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:58.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:58.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:58.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] boot 2024-07-26T10:53:58.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:57 smithi160 ceph-mon[36715]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T10:53:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:53:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:53:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T10:53:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T10:53:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: osd.6 [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] boot 2024-07-26T10:53:58.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:57 smithi183 ceph-mon[37273]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T10:53:58.676 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T10:53:58.771 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T10:53:58.784 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-07-26T10:53:58.784 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-07-26T10:53:58.784 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111015 s, 945 MB/s 2024-07-26T10:53:58.784 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-07-26T10:53:59.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_2 2024-07-26T10:53:59.728 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:54:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:59 smithi160 ceph-mon[36715]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:53:59 smithi160 ceph-mon[36715]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T10:54:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:59 smithi183 ceph-mon[37273]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:53:59 smithi183 ceph-mon[37273]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T10:54:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:59 smithi144 ceph-mon[32129]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:53:59 smithi144 ceph-mon[32129]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T10:54:01.239 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 -- 172.21.15.160:0/2961880277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 msgr2=0x7fe6a41081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:01.239 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 --2- 172.21.15.160:0/2961880277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a41081e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009900 tx=0x7fe68c02a410 comp rx=0 tx=0).stop 2024-07-26T10:54:01.239 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 -- 172.21.15.160:0/2961880277 shutdown_connections 2024-07-26T10:54:01.239 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 --2- 172.21.15.160:0/2961880277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6a4108720 0x7fe6a410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:01.239 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 --2- 172.21.15.160:0/2961880277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a41081e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 --2- 172.21.15.160:0/2961880277 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6a4102490 0x7fe6a41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 -- 172.21.15.160:0/2961880277 >> 172.21.15.160:0/2961880277 conn(0x7fe6a4079160 msgr2=0x7fe6a4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 -- 172.21.15.160:0/2961880277 shutdown_connections 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 -- 172.21.15.160:0/2961880277 wait complete. 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 Processor -- start 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.235+0000 7fe6a957e640 1 -- start start 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6a4102490 0x7fe6a419ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a419b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6a4108720 0x7fe6a419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41123f0 con 0x7fe6a4108720 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a4112270 con 0x7fe6a4107dd0 2024-07-26T10:54:01.240 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a4112570 con 0x7fe6a4102490 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a419b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a419b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:57116/0 (socket says 172.21.15.160:57116) 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 -- 172.21.15.160:0/3589775401 learned_addr learned my addr 172.21.15.160:0/3589775401 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a2ffd640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6a4102490 0x7fe6a419ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 -- 172.21.15.160:0/3589775401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6a4102490 msgr2=0x7fe6a419ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6a4102490 0x7fe6a419ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 -- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6a4108720 msgr2=0x7fe6a419b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6a4108720 0x7fe6a419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 -- 172.21.15.160:0/3589775401 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a41a5ff0 con 0x7fe6a4107dd0 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe69bfff640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a419b220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe68c006010 tx=0x7fe68c004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c03f030 con 0x7fe6a4107dd0 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe68c003aa0 con 0x7fe6a4107dd0 2024-07-26T10:54:01.241 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c035070 con 0x7fe6a4107dd0 2024-07-26T10:54:01.242 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.236+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a41a6280 con 0x7fe6a4107dd0 2024-07-26T10:54:01.242 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.237+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a41a6790 con 0x7fe6a4107dd0 2024-07-26T10:54:01.242 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.238+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a41028a0 con 0x7fe6a4107dd0 2024-07-26T10:54:01.243 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.239+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe68c042030 con 0x7fe6a4107dd0 2024-07-26T10:54:01.243 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.240+0000 7fe6a0ff9640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe6900b3c20 0x7fe6900b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:01.243 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.240+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5180+0+0 (secure 0 0 0) 0x7fe68c1038a0 con 0x7fe6a4107dd0 2024-07-26T10:54:01.243 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.240+0000 7fe6a2ffd640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe6900b3c20 0x7fe6900b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:01.243 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.241+0000 7fe6a2ffd640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe6900b3c20 0x7fe6900b6130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe6a406b450 tx=0x7fe694005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:54:01.245 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.243+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe68c0cbb30 con 0x7fe6a4107dd0 2024-07-26T10:54:01.430 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:01.428+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe6a4063230 con 0x7fe6900b3c20 2024-07-26T10:54:02.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:02.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi144 2024-07-26T10:54:02.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 5696M 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:02.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi144 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 5696M 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:02.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi144 2024-07-26T10:54:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 5696M 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:03 smithi160 ceph-mon[36715]: from='client.24488 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:03 smithi183 ceph-mon[37273]: from='client.24488 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:03 smithi144 ceph-mon[32129]: from='client.24488 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:04 smithi160 ceph-mon[36715]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:04 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/192904775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]: dispatch 2024-07-26T10:54:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:04 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]: dispatch 2024-07-26T10:54:04.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:04 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]': finished 2024-07-26T10:54:04.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:04.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:04 smithi160 ceph-mon[36715]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T10:54:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:04 smithi183 ceph-mon[37273]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:04 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/192904775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]: dispatch 2024-07-26T10:54:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:04 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]: dispatch 2024-07-26T10:54:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:04 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]': finished 2024-07-26T10:54:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:04 smithi183 ceph-mon[37273]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T10:54:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:04 smithi144 ceph-mon[32129]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:04 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/192904775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]: dispatch 2024-07-26T10:54:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:04 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]: dispatch 2024-07-26T10:54:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:04 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23dda395-05ca-4eef-8655-74edaefbfe90"}]': finished 2024-07-26T10:54:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:04 smithi144 ceph-mon[32129]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T10:54:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:05 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/3909765679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:05 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/3909765679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:05 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/3909765679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:06 smithi160 ceph-mon[36715]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:06.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:06 smithi144 ceph-mon[32129]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:06 smithi183 ceph-mon[37273]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:08 smithi160 ceph-mon[36715]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:08 smithi183 ceph-mon[37273]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:08 smithi144 ceph-mon[32129]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:10.183 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:10 smithi160 ceph-mon[36715]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:10 smithi183 ceph-mon[37273]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:10 smithi144 ceph-mon[32129]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:11.301 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T10:54:11.302 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T10:54:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T10:54:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:12 smithi160 ceph-mon[36715]: Deploying daemon osd.7 on smithi160 2024-07-26T10:54:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:12 smithi160 ceph-mon[36715]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:12 smithi183 ceph-mon[37273]: Deploying daemon osd.7 on smithi160 2024-07-26T10:54:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:12 smithi183 ceph-mon[37273]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:12 smithi144 ceph-mon[32129]: Deploying daemon osd.7 on smithi160 2024-07-26T10:54:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:12 smithi144 ceph-mon[32129]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:14.252 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:14 smithi160 ceph-mon[36715]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:14 smithi183 ceph-mon[37273]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:14 smithi144 ceph-mon[32129]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:16.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:16 smithi160 ceph-mon[36715]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:16.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:16.362 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:16.362 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:16 smithi183 ceph-mon[37273]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:16.456 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:16 smithi144 ceph-mon[32129]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:16.456 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:16.456 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:16.456 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:18.282 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:18 smithi160 ceph-mon[36715]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:18 smithi183 ceph-mon[37273]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:18 smithi144 ceph-mon[32129]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:19.585 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 7 on host 'smithi160' 2024-07-26T10:54:19.585 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.582+0000 7fe6a0ff9640 1 -- 172.21.15.160:0/3589775401 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe6a4063230 con 0x7fe6900b3c20 2024-07-26T10:54:19.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe6900b3c20 msgr2=0x7fe6900b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:19.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe6900b3c20 0x7fe6900b6130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe6a406b450 tx=0x7fe694005f90 comp rx=0 tx=0).stop 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 msgr2=0x7fe6a419b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a419b220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe68c006010 tx=0x7fe68c004060 comp rx=0 tx=0).stop 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 shutdown_connections 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe6900b3c20 0x7fe6900b6130 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6a4108720 0x7fe6a419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6a4107dd0 0x7fe6a419b220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 --2- 172.21.15.160:0/3589775401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6a4102490 0x7fe6a419ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 >> 172.21.15.160:0/3589775401 conn(0x7fe6a4079160 msgr2=0x7fe6a4103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 shutdown_connections 2024-07-26T10:54:19.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:54:19.585+0000 7fe6a957e640 1 -- 172.21.15.160:0/3589775401 wait complete. 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T10:54:20.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:19 smithi160 ceph-mon[36715]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T10:54:20.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:19 smithi183 ceph-mon[37273]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T10:54:20.168 DEBUG:teuthology.orchestra.run.smithi160:osd.7> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.7.service 2024-07-26T10:54:20.170 INFO:tasks.cephadm:Deploying osd.8 on smithi183 with /dev/vg_nvme/lv_2... 2024-07-26T10:54:20.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T10:54:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:20.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:20.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:20.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T10:54:20.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:19 smithi144 ceph-mon[32129]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T10:54:20.429 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:54:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:21 smithi183 ceph-mon[37273]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T10:54:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:21 smithi183 ceph-mon[37273]: from='osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:54:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:21 smithi183 ceph-mon[37273]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T10:54:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:21 smithi183 ceph-mon[37273]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:54:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:21 smithi183 ceph-mon[37273]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:21.931 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-mon[36715]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T10:54:21.931 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:21.931 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-mon[36715]: from='osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:54:21.931 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-mon[36715]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T10:54:21.931 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-mon[36715]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:54:21.931 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-mon[36715]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:21.931 INFO:journalctl@ceph.osd.7.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-7[55840]: 2024-07-26T10:54:21.587+0000 7f03b89ae640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T10:54:21.932 INFO:journalctl@ceph.osd.7.smithi160.stdout:Jul 26 10:54:21 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-7[55840]: 2024-07-26T10:54:21.598+0000 7f03b47ac640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:54:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:21 smithi144 ceph-mon[32129]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T10:54:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:21 smithi144 ceph-mon[32129]: from='osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:54:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:21 smithi144 ceph-mon[32129]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T10:54:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:21 smithi144 ceph-mon[32129]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:54:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:21 smithi144 ceph-mon[32129]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-07-26T10:54:22.238 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T10:54:22.323 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T10:54:22.335 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-07-26T10:54:22.335 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-07-26T10:54:22.336 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111348 s, 942 MB/s 2024-07-26T10:54:22.336 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-07-26T10:54:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:22 smithi183 ceph-mon[37273]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:54:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:22 smithi183 ceph-mon[37273]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T10:54:22.940 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:22 smithi160 ceph-mon[36715]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:54:22.940 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:22.940 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:22 smithi160 ceph-mon[36715]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T10:54:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:22 smithi144 ceph-mon[32129]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:54:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:22 smithi144 ceph-mon[32129]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T10:54:23.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-07-26T10:54:23.326 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] boot 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.761 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.762 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:23 smithi183 ceph-mon[37273]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T10:54:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] boot 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:23.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:23.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:23.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:23.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:23 smithi144 ceph-mon[32129]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: osd.7 [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] boot 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:24.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:24.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:23 smithi160 ceph-mon[36715]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T10:54:24.792 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:24 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi160 2024-07-26T10:54:24.792 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:24 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 5696M 2024-07-26T10:54:24.792 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:24 smithi183 ceph-mon[37273]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T10:54:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:24 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi160 2024-07-26T10:54:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:24 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 5696M 2024-07-26T10:54:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:24 smithi144 ceph-mon[32129]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T10:54:25.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 -- 172.21.15.183:0/61727231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce301024c0 msgr2=0x7fce301028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:25.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 --2- 172.21.15.183:0/61727231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce301024c0 0x7fce301028d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce20009900 tx=0x7fce2002a410 comp rx=0 tx=0).stop 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 -- 172.21.15.183:0/61727231 shutdown_connections 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 --2- 172.21.15.183:0/61727231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce30108750 0x7fce3010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 --2- 172.21.15.183:0/61727231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce30107e00 0x7fce30108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 --2- 172.21.15.183:0/61727231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce301024c0 0x7fce301028d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 -- 172.21.15.183:0/61727231 >> 172.21.15.183:0/61727231 conn(0x7fce30079180 msgr2=0x7fce30079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 -- 172.21.15.183:0/61727231 shutdown_connections 2024-07-26T10:54:25.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.003+0000 7fce348c5640 1 -- 172.21.15.183:0/61727231 wait complete. 2024-07-26T10:54:25.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.004+0000 7fce348c5640 1 Processor -- start 2024-07-26T10:54:25.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.004+0000 7fce348c5640 1 -- start start 2024-07-26T10:54:25.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.004+0000 7fce348c5640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce301024c0 0x7fce3019c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:25.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.004+0000 7fce348c5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce30107e00 0x7fce3019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.004+0000 7fce348c5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 0x7fce301a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce348c5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fce301135a0 con 0x7fce30107e00 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce348c5640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fce30113420 con 0x7fce301024c0 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce348c5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fce30113720 con 0x7fce30108750 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 0x7fce301a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 0x7fce301a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:40004/0 (socket says 172.21.15.183:40004) 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 -- 172.21.15.183:0/955830062 learned_addr learned my addr 172.21.15.183:0/955830062 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2e59c640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce301024c0 0x7fce3019c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 -- 172.21.15.183:0/955830062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce301024c0 msgr2=0x7fce3019c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce301024c0 0x7fce3019c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 -- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce30107e00 msgr2=0x7fce3019c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce30107e00 0x7fce3019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 -- 172.21.15.183:0/955830062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce301a2140 con 0x7fce30108750 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2e59c640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce301024c0 0x7fce3019c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce2ed9d640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 0x7fce301a1a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fce2800b970 tx=0x7fce2800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28019070 con 0x7fce30108750 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce301a2430 con 0x7fce30108750 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce28014440 con 0x7fce30108750 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce280075d0 con 0x7fce30108750 2024-07-26T10:54:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.005+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce301a8b90 con 0x7fce30108750 2024-07-26T10:54:25.011 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.007+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce30103740 con 0x7fce30108750 2024-07-26T10:54:25.012 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.008+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fce2801c070 con 0x7fce30108750 2024-07-26T10:54:25.012 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.008+0000 7fce277fe640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fce1c0b3c20 0x7fce1c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:25.013 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.009+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5612+0+0 (secure 0 0 0) 0x7fce280e2840 con 0x7fce30108750 2024-07-26T10:54:25.013 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.009+0000 7fce2e59c640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fce1c0b3c20 0x7fce1c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:25.013 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.010+0000 7fce2e59c640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fce1c0b3c20 0x7fce1c0b6130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce20031440 tx=0x7fce20005d50 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:54:25.014 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.013+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fce280aa920 con 0x7fce30108750 2024-07-26T10:54:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:24 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi160 2024-07-26T10:54:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:24 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 5696M 2024-07-26T10:54:25.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:24 smithi160 ceph-mon[36715]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T10:54:25.196 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:25.194+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fce30111180 con 0x7fce1c0b3c20 2024-07-26T10:54:25.882 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:25 smithi183 ceph-mon[37273]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:25.882 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:25.882 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:25.882 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:25 smithi144 ceph-mon[32129]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:25 smithi160 ceph-mon[36715]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:26.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:26.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:26.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:26 smithi183 ceph-mon[37273]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:26 smithi144 ceph-mon[32129]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:26 smithi160 ceph-mon[36715]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:27 smithi144 ceph-mon[32129]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:27 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/3673945332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]: dispatch 2024-07-26T10:54:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:27 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]: dispatch 2024-07-26T10:54:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:27 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]': finished 2024-07-26T10:54:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:27.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:27 smithi144 ceph-mon[32129]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T10:54:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:27 smithi160 ceph-mon[36715]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:27 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/3673945332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]: dispatch 2024-07-26T10:54:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:27 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]: dispatch 2024-07-26T10:54:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:27 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]': finished 2024-07-26T10:54:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:27 smithi160 ceph-mon[36715]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T10:54:28.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:27 smithi183 ceph-mon[37273]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:28.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:27 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/3673945332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]: dispatch 2024-07-26T10:54:28.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:27 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]: dispatch 2024-07-26T10:54:28.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:27 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0afc0c4-8c60-46bf-a684-f418e6d1ec5a"}]': finished 2024-07-26T10:54:28.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:28.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:27 smithi183 ceph-mon[37273]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T10:54:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:28 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/4157632693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:28 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/4157632693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:28 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/4157632693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:29 smithi144 ceph-mon[32129]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T10:54:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:29 smithi160 ceph-mon[36715]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T10:54:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:29 smithi183 ceph-mon[37273]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T10:54:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:31 smithi183 ceph-mon[37273]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T10:54:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:31 smithi144 ceph-mon[32129]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T10:54:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:31 smithi160 ceph-mon[36715]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T10:54:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:33 smithi183 ceph-mon[37273]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-26T10:54:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:33 smithi144 ceph-mon[32129]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-26T10:54:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:33 smithi160 ceph-mon[36715]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-26T10:54:35.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T10:54:35.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T10:54:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T10:54:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:36.244 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:35 smithi183 ceph-mon[37273]: Deploying daemon osd.8 on smithi183 2024-07-26T10:54:36.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:35 smithi183 ceph-mon[37273]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:54:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:35 smithi160 ceph-mon[36715]: Deploying daemon osd.8 on smithi183 2024-07-26T10:54:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:35 smithi160 ceph-mon[36715]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:54:36.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:35 smithi144 ceph-mon[32129]: Deploying daemon osd.8 on smithi183 2024-07-26T10:54:36.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:35 smithi144 ceph-mon[32129]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:54:38.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:37 smithi183 ceph-mon[37273]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:54:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:37 smithi160 ceph-mon[36715]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:54:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:37 smithi144 ceph-mon[32129]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:54:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:39 smithi160 ceph-mon[36715]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-07-26T10:54:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:40.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:39 smithi183 ceph-mon[37273]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-07-26T10:54:40.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:40.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:40.323 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:39 smithi144 ceph-mon[32129]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-07-26T10:54:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:42.274 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:41 smithi183 ceph-mon[37273]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:41 smithi160 ceph-mon[36715]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:41 smithi144 ceph-mon[32129]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:43.187 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.185+0000 7fce277fe640 1 -- 172.21.15.183:0/955830062 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fce30111180 con 0x7fce1c0b3c20 2024-07-26T10:54:43.188 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 8 on host 'smithi183' 2024-07-26T10:54:43.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fce1c0b3c20 msgr2=0x7fce1c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:43.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fce1c0b3c20 0x7fce1c0b6130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce20031440 tx=0x7fce20005d50 comp rx=0 tx=0).stop 2024-07-26T10:54:43.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 msgr2=0x7fce301a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:43.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 0x7fce301a1a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fce2800b970 tx=0x7fce2800be30 comp rx=0 tx=0).stop 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 shutdown_connections 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fce1c0b3c20 0x7fce1c0b6130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce30108750 0x7fce301a1a10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce30107e00 0x7fce3019c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 --2- 172.21.15.183:0/955830062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce301024c0 0x7fce3019c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 >> 172.21.15.183:0/955830062 conn(0x7fce30079180 msgr2=0x7fce30110a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 shutdown_connections 2024-07-26T10:54:43.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:54:43.187+0000 7fce348c5640 1 -- 172.21.15.183:0/955830062 wait complete. 2024-07-26T10:54:43.920 DEBUG:teuthology.orchestra.run.smithi183:osd.8> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.8.service 2024-07-26T10:54:43.922 INFO:tasks.cephadm:Deploying osd.9 on smithi144 with /dev/vg_nvme/lv_1... 2024-07-26T10:54:43.923 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T10:54:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:43 smithi183 ceph-mon[37273]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:43 smithi183 ceph-mon[37273]: from='osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T10:54:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:43 smithi183 ceph-mon[37273]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T10:54:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:43 smithi160 ceph-mon[36715]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:43 smithi160 ceph-mon[36715]: from='osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T10:54:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:43 smithi160 ceph-mon[36715]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T10:54:44.384 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:54:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:43 smithi144 ceph-mon[32129]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:43 smithi144 ceph-mon[32129]: from='osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T10:54:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:43 smithi144 ceph-mon[32129]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: from='osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T10:54:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-mon[37273]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:54:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T10:54:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: from='osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:54:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T10:54:45.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:45 smithi144 ceph-mon[32129]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:54:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T10:54:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: from='osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:54:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T10:54:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:45 smithi160 ceph-mon[36715]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:54:45.458 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-8[56629]: 2024-07-26T10:54:45.193+0000 7f4a96f23640 -1 osd.8 0 waiting for initial osdmap 2024-07-26T10:54:45.458 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 26 10:54:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-8[56629]: 2024-07-26T10:54:45.204+0000 7f4a92d21640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:54:46.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:46 smithi144 ceph-mon[32129]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:46.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:46 smithi144 ceph-mon[32129]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:54:46.227 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:46.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:46 smithi144 ceph-mon[32129]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T10:54:46.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:46 smithi160 ceph-mon[36715]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:46 smithi160 ceph-mon[36715]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:54:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:46 smithi160 ceph-mon[36715]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T10:54:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:46 smithi183 ceph-mon[37273]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T10:54:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:46 smithi183 ceph-mon[37273]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:54:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:46 smithi183 ceph-mon[37273]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T10:54:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:46.842 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T10:54:47.008 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T10:54:47.056 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-07-26T10:54:47.056 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-07-26T10:54:47.057 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0396321 s, 265 MB/s 2024-07-26T10:54:47.057 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-07-26T10:54:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:54:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:54:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] boot 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi183 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 5696M 2024-07-26T10:54:47.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:47.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:47.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:54:47.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:54:47.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:47.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] boot 2024-07-26T10:54:47.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi183 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.445 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 5696M 2024-07-26T10:54:47.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:47.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:47.446 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: osd.8 [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] boot 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi183 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 5696M 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:47.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi144:vg_nvme/lv_1 2024-07-26T10:54:48.414 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:54:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:48 smithi144 ceph-mon[32129]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T10:54:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:48.477 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:48 smithi160 ceph-mon[36715]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T10:54:48.477 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:48.478 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:48.478 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:48.478 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:48 smithi183 ceph-mon[37273]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T10:54:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:54:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:49 smithi160 ceph-mon[36715]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T10:54:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:49 smithi160 ceph-mon[36715]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:49.586 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:49 smithi144 ceph-mon[32129]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T10:54:49.586 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:49 smithi144 ceph-mon[32129]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:49 smithi183 ceph-mon[37273]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T10:54:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:49 smithi183 ceph-mon[37273]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:50.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 -- 172.21.15.144:0/2063574044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe010af10 msgr2=0x7fcbe010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:50.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2063574044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe010af10 0x7fcbe010b320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009900 tx=0x7fcbcc02a3f0 comp rx=0 tx=0).stop 2024-07-26T10:54:50.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 -- 172.21.15.144:0/2063574044 shutdown_connections 2024-07-26T10:54:50.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2063574044 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcbe010b860 0x7fcbe0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:50.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2063574044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe010af10 0x7fcbe010b320 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:50.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2063574044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe01055d0 0x7fcbe01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:50.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.649+0000 7fcbdf200640 1 -- 172.21.15.144:0/2063574044 >> 172.21.15.144:0/2063574044 conn(0x7fcbe0101040 msgr2=0x7fcbe0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:54:50.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.650+0000 7fcbdf200640 1 -- 172.21.15.144:0/2063574044 shutdown_connections 2024-07-26T10:54:50.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.650+0000 7fcbdf200640 1 -- 172.21.15.144:0/2063574044 wait complete. 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 Processor -- start 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 -- start start 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 0x7fcbe019bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcbe010af10 0x7fcbe019c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe010b860 0x7fcbe01a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe0115520 con 0x7fcbe01055d0 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe01153a0 con 0x7fcbe010af10 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.651+0000 7fcbdf200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe01156a0 con 0x7fcbe010b860 2024-07-26T10:54:50.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbdde00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 0x7fcbe019bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbdde00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 0x7fcbe019bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53004/0 (socket says 172.21.15.144:53004) 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbdd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcbe010af10 0x7fcbe019c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbdde00640 1 -- 172.21.15.144:0/2411864039 learned_addr learned my addr 172.21.15.144:0/2411864039 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbde800640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe010b860 0x7fcbe01a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbde800640 1 -- 172.21.15.144:0/2411864039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcbe010af10 msgr2=0x7fcbe019c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbde800640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcbe010af10 0x7fcbe019c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbde800640 1 -- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 msgr2=0x7fcbe019bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbde800640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 0x7fcbe019bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:54:50.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbde800640 1 -- 172.21.15.144:0/2411864039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe01a1f20 con 0x7fcbe010b860 2024-07-26T10:54:50.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.652+0000 7fcbdde00640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 0x7fcbe019bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:54:50.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.653+0000 7fcbde800640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe010b860 0x7fcbe01a17f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800ec50 tx=0x7fcbd800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:54:50.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.653+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8003b70 con 0x7fcbe010b860 2024-07-26T10:54:50.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.653+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe01a2210 con 0x7fcbe010b860 2024-07-26T10:54:50.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.653+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd8003d10 con 0x7fcbe010b860 2024-07-26T10:54:50.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.653+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8010830 con 0x7fcbe010b860 2024-07-26T10:54:50.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.653+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe01a8970 con 0x7fcbe010b860 2024-07-26T10:54:50.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.654+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe01059e0 con 0x7fcbe010b860 2024-07-26T10:54:50.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.657+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fcbd8004b40 con 0x7fcbe010b860 2024-07-26T10:54:50.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.658+0000 7fcbd6a00640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcbc40b3c20 0x7fcbc40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:54:50.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.658+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6044+0+0 (secure 0 0 0) 0x7fcbd80e3980 con 0x7fcbe010b860 2024-07-26T10:54:50.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.658+0000 7fcbdde00640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcbc40b3c20 0x7fcbc40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:50.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.659+0000 7fcbdde00640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcbc40b3c20 0x7fcbc40b6130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0011550 tx=0x7fcbd0005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:54:50.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.660+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcbd80ab8a0 con 0x7fcbe010b860 2024-07-26T10:54:50.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:54:50.867+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fcbe0063230 con 0x7fcbc40b3c20 2024-07-26T10:54:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:51.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:54:51.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:54:51.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:51 smithi144 ceph-mon[32129]: from='client.24526 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:51 smithi144 ceph-mon[32129]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:51 smithi160 ceph-mon[36715]: from='client.24526 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:51 smithi160 ceph-mon[36715]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:51 smithi183 ceph-mon[37273]: from='client.24526 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:54:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:51 smithi183 ceph-mon[37273]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:54:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:53 smithi144 ceph-mon[32129]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T10:54:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:53 smithi160 ceph-mon[36715]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T10:54:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:53 smithi183 ceph-mon[37273]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T10:54:55.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:54 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3999799489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]: dispatch 2024-07-26T10:54:55.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:54 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]: dispatch 2024-07-26T10:54:55.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:54 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]': finished 2024-07-26T10:54:55.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:54:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:54 smithi144 ceph-mon[32129]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T10:54:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:54 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3999799489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]: dispatch 2024-07-26T10:54:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:54 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]: dispatch 2024-07-26T10:54:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:54 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]': finished 2024-07-26T10:54:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:54:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:54 smithi160 ceph-mon[36715]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T10:54:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:54 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3999799489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]: dispatch 2024-07-26T10:54:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:54 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]: dispatch 2024-07-26T10:54:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:54 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f586925-f5c7-495c-a472-0d3e6af7dfdf"}]': finished 2024-07-26T10:54:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:54:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:54 smithi183 ceph-mon[37273]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T10:54:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:55 smithi144 ceph-mon[32129]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-07-26T10:54:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:55 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2066471627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:55 smithi160 ceph-mon[36715]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-07-26T10:54:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:55 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2066471627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:55 smithi183 ceph-mon[37273]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-07-26T10:54:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:55 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2066471627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:54:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:57 smithi144 ceph-mon[32129]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T10:54:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:57 smithi160 ceph-mon[36715]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T10:54:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:57 smithi183 ceph-mon[37273]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T10:55:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:54:59 smithi160 ceph-mon[36715]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:54:59 smithi183 ceph-mon[37273]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:54:59 smithi144 ceph-mon[32129]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:02.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:01 smithi144 ceph-mon[32129]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:01 smithi160 ceph-mon[36715]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:01 smithi183 ceph-mon[37273]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:03 smithi160 ceph-mon[36715]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:03 smithi183 ceph-mon[37273]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:03 smithi144 ceph-mon[32129]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:05.277 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T10:55:05.277 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T10:55:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T10:55:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:06 smithi144 ceph-mon[32129]: Deploying daemon osd.9 on smithi144 2024-07-26T10:55:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:06 smithi144 ceph-mon[32129]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:06 smithi160 ceph-mon[36715]: Deploying daemon osd.9 on smithi144 2024-07-26T10:55:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:06 smithi160 ceph-mon[36715]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:06 smithi183 ceph-mon[37273]: Deploying daemon osd.9 on smithi144 2024-07-26T10:55:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:06 smithi183 ceph-mon[37273]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:08 smithi144 ceph-mon[32129]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:08 smithi160 ceph-mon[36715]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:08 smithi183 ceph-mon[37273]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:10 smithi160 ceph-mon[36715]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:10 smithi183 ceph-mon[37273]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:10 smithi144 ceph-mon[32129]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:12 smithi160 ceph-mon[36715]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:12 smithi183 ceph-mon[37273]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:12 smithi144 ceph-mon[32129]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:14.088 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:14 smithi144 ceph-mon[32129]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:14.088 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:14.088 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:14.088 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:14.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:14 smithi160 ceph-mon[36715]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:14.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:14.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:14.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:14 smithi183 ceph-mon[37273]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:16 smithi160 ceph-mon[36715]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:16 smithi183 ceph-mon[37273]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:16 smithi144 ceph-mon[32129]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:17.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:17.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T10:55:17.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T10:55:17.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:17.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:17.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:18.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.088+0000 7fcbd6a00640 1 -- 172.21.15.144:0/2411864039 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcbe0063230 con 0x7fcbc40b3c20 2024-07-26T10:55:18.090 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 9 on host 'smithi144' 2024-07-26T10:55:18.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.091+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcbc40b3c20 msgr2=0x7fcbc40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:18.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.091+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcbc40b3c20 0x7fcbc40b6130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0011550 tx=0x7fcbd0005f90 comp rx=0 tx=0).stop 2024-07-26T10:55:18.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe010b860 msgr2=0x7fcbe01a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe010b860 0x7fcbe01a17f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800ec50 tx=0x7fcbd800c6a0 comp rx=0 tx=0).stop 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 shutdown_connections 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcbc40b3c20 0x7fcbc40b6130 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe010b860 0x7fcbe01a17f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcbe010af10 0x7fcbe019c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 --2- 172.21.15.144:0/2411864039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe01055d0 0x7fcbe019bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 >> 172.21.15.144:0/2411864039 conn(0x7fcbe0101040 msgr2=0x7fcbe0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 shutdown_connections 2024-07-26T10:55:18.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:55:18.092+0000 7fcbdf200640 1 -- 172.21.15.144:0/2411864039 wait complete. 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T10:55:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:55:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T10:55:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:55:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:18.942 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:18.943 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:19.027 DEBUG:teuthology.orchestra.run.smithi144:osd.9> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.9.service 2024-07-26T10:55:19.032 INFO:tasks.cephadm:Deploying osd.10 on smithi160 with /dev/vg_nvme/lv_1... 2024-07-26T10:55:19.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T10:55:19.292 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:55:19.681 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:19 smithi160 ceph-mon[36715]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:55:19.682 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:19 smithi160 ceph-mon[36715]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T10:55:19.682 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:19.682 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:19.682 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:19 smithi160 ceph-mon[36715]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:19.682 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:19 smithi160 ceph-mon[36715]: from='osd.9 ' entity='osd.9' 2024-07-26T10:55:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-mon[32129]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:55:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-mon[32129]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T10:55:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:19.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:19.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-mon[32129]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:19.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-mon[32129]: from='osd.9 ' entity='osd.9' 2024-07-26T10:55:19.721 INFO:journalctl@ceph.osd.9.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-9[71641]: 2024-07-26T10:55:19.413+0000 7fe214600640 -1 osd.9 0 waiting for initial osdmap 2024-07-26T10:55:19.721 INFO:journalctl@ceph.osd.9.smithi144.stdout:Jul 26 10:55:19 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-9[71641]: 2024-07-26T10:55:19.481+0000 7fe20a600640 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:55:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:19 smithi183 ceph-mon[37273]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-07-26T10:55:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:19 smithi183 ceph-mon[37273]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T10:55:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:19 smithi183 ceph-mon[37273]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-07-26T10:55:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:19 smithi183 ceph-mon[37273]: from='osd.9 ' entity='osd.9' 2024-07-26T10:55:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:20 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:55:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:20 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:55:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:20 smithi144 ceph-mon[32129]: osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] boot 2024-07-26T10:55:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:20 smithi144 ceph-mon[32129]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T10:55:20.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:20 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:55:20.752 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:20 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:55:20.752 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:20.752 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:20.752 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:20 smithi160 ceph-mon[36715]: osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] boot 2024-07-26T10:55:20.752 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:20 smithi160 ceph-mon[36715]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T10:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:20 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:20 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T10:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:20 smithi183 ceph-mon[37273]: osd.9 [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] boot 2024-07-26T10:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:20 smithi183 ceph-mon[37273]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T10:55:21.123 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T10:55:21.202 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T10:55:21.215 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-07-26T10:55:21.215 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-07-26T10:55:21.215 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011415 s, 919 MB/s 2024-07-26T10:55:21.215 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-07-26T10:55:21.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:21 smithi160 ceph-mon[36715]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:21.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_1 2024-07-26T10:55:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:21 smithi183 ceph-mon[37273]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:21 smithi144 ceph-mon[32129]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:22.072 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T10:55:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:22 smithi160 ceph-mon[36715]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T10:55:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:22 smithi183 ceph-mon[37273]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T10:55:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:22 smithi144 ceph-mon[32129]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 -- 172.21.15.160:0/3615880570 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 msgr2=0x7f685c102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 --2- 172.21.15.160:0/3615880570 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c102920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f684c009900 tx=0x7f684c02a410 comp rx=0 tx=0).stop 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 -- 172.21.15.160:0/3615880570 shutdown_connections 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 --2- 172.21.15.160:0/3615880570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f685c108750 0x7f685c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 --2- 172.21.15.160:0/3615880570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c107e00 0x7f685c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 --2- 172.21.15.160:0/3615880570 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c102920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 -- 172.21.15.160:0/3615880570 >> 172.21.15.160:0/3615880570 conn(0x7f685c079190 msgr2=0x7f685c0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 -- 172.21.15.160:0/3615880570 shutdown_connections 2024-07-26T10:55:23.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.686+0000 7f6861fee640 1 -- 172.21.15.160:0/3615880570 wait complete. 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 Processor -- start 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 -- start start 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c107e00 0x7f685c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f685c108750 0x7f685c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f685c112420 con 0x7f685c107e00 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1122a0 con 0x7f685c102510 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f6861fee640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1125a0 con 0x7f685c108750 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.687+0000 7f685b7fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:47466/0 (socket says 172.21.15.160:47466) 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 -- 172.21.15.160:0/3344828569 learned_addr learned my addr 172.21.15.160:0/3344828569 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-07-26T10:55:23.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 -- 172.21.15.160:0/3344828569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f685c108750 msgr2=0x7f685c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f685c108750 0x7f685c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 -- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c107e00 msgr2=0x7f685c199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c107e00 0x7f685c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 -- 172.21.15.160:0/3344828569 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c19ee60 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f685b7fe640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c198ef0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f684c006010 tx=0x7f684c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c03f030 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685c19f0f0 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f684c003800 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c035070 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.688+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685c1a58b0 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.689+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685c103740 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.692+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f684c003cb0 con 0x7f685c102510 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.692+0000 7f6858ff9640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f68440b3c20 0x7f68440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:23.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.692+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6476+0+0 (secure 0 0 0) 0x7f684c1041c0 con 0x7f685c102510 2024-07-26T10:55:23.700 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.693+0000 7f685affd640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f68440b3c20 0x7f68440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:23.700 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.694+0000 7f685affd640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f68440b3c20 0x7f68440b6130 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f685c100750 tx=0x7f6850006c90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:55:23.700 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.696+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f684c0cbf40 con 0x7f685c102510 2024-07-26T10:55:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:23 smithi144 ceph-mon[32129]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:23 smithi160 ceph-mon[36715]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:23.882 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:23.880+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f685c063230 con 0x7f68440b3c20 2024-07-26T10:55:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:23 smithi183 ceph-mon[37273]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:55:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi144 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 4272M 2024-07-26T10:55:24.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:24.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:24.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi144 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 4272M 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:24.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:25.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:55:25.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:55:25.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:55:25.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:25.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi144 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 4272M 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:25.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:25.040 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:25 smithi183 ceph-mon[37273]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:25.969 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:25 smithi160 ceph-mon[36715]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:25 smithi144 ceph-mon[32129]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:26 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/1006383136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]: dispatch 2024-07-26T10:55:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:26 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]: dispatch 2024-07-26T10:55:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:26 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]': finished 2024-07-26T10:55:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:26 smithi160 ceph-mon[36715]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T10:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:26 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/1006383136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]: dispatch 2024-07-26T10:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:26 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]: dispatch 2024-07-26T10:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:26 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]': finished 2024-07-26T10:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:26 smithi183 ceph-mon[37273]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T10:55:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:26 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/1006383136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]: dispatch 2024-07-26T10:55:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:26 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]: dispatch 2024-07-26T10:55:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:26 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fac8da53-9c38-4368-9278-612c04a06498"}]': finished 2024-07-26T10:55:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:26 smithi144 ceph-mon[32129]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T10:55:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:27 smithi183 ceph-mon[37273]: from='client.? 172.21.15.160:0/1111781903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:55:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:27 smithi183 ceph-mon[37273]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:27 smithi144 ceph-mon[32129]: from='client.? 172.21.15.160:0/1111781903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:55:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:27 smithi144 ceph-mon[32129]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:27 smithi160 ceph-mon[36715]: from='client.? 172.21.15.160:0/1111781903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:55:28.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:27 smithi160 ceph-mon[36715]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:30.267 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:29 smithi160 ceph-mon[36715]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:29 smithi183 ceph-mon[37273]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:29 smithi144 ceph-mon[32129]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:31 smithi160 ceph-mon[36715]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:31 smithi183 ceph-mon[37273]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:31 smithi144 ceph-mon[32129]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:34.120 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:33 smithi160 ceph-mon[36715]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:34.121 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T10:55:34.121 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:33 smithi183 ceph-mon[37273]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T10:55:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:33 smithi144 ceph-mon[32129]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T10:55:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:35.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:34 smithi160 ceph-mon[36715]: Deploying daemon osd.10 on smithi160 2024-07-26T10:55:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:34 smithi183 ceph-mon[37273]: Deploying daemon osd.10 on smithi160 2024-07-26T10:55:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:34 smithi144 ceph-mon[32129]: Deploying daemon osd.10 on smithi160 2024-07-26T10:55:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:35 smithi160 ceph-mon[36715]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:36.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:35 smithi144 ceph-mon[32129]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:35 smithi183 ceph-mon[37273]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:38.076 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:38 smithi144 ceph-mon[32129]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:38.143 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:38 smithi160 ceph-mon[36715]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:38 smithi183 ceph-mon[37273]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:39.005 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:39.005 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:39.005 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:40.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:40 smithi183 ceph-mon[37273]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:40 smithi160 ceph-mon[36715]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:40 smithi144 ceph-mon[32129]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:42.064 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 10 on host 'smithi160' 2024-07-26T10:55:42.064 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.061+0000 7f6858ff9640 1 -- 172.21.15.160:0/3344828569 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f685c063230 con 0x7f68440b3c20 2024-07-26T10:55:42.066 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f68440b3c20 msgr2=0x7f68440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:42.066 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f68440b3c20 0x7f68440b6130 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f685c100750 tx=0x7f6850006c90 comp rx=0 tx=0).stop 2024-07-26T10:55:42.066 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 msgr2=0x7f685c198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:42.066 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c198ef0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f684c006010 tx=0x7f684c0042f0 comp rx=0 tx=0).stop 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 shutdown_connections 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f68440b3c20 0x7f68440b6130 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f685c108750 0x7f685c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c107e00 0x7f685c199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 --2- 172.21.15.160:0/3344828569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f685c102510 0x7f685c198ef0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.063+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 >> 172.21.15.160:0/3344828569 conn(0x7f685c079190 msgr2=0x7f685c100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.064+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 shutdown_connections 2024-07-26T10:55:42.067 INFO:teuthology.orchestra.run.smithi160.stderr:2024-07-26T10:55:42.064+0000 7f6861fee640 1 -- 172.21.15.160:0/3344828569 wait complete. 2024-07-26T10:55:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:42 smithi160 ceph-mon[36715]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:42 smithi183 ceph-mon[37273]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:42 smithi144 ceph-mon[32129]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:43.086 DEBUG:teuthology.orchestra.run.smithi160:osd.10> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.10.service 2024-07-26T10:55:43.089 INFO:tasks.cephadm:Deploying osd.11 on smithi183 with /dev/vg_nvme/lv_1... 2024-07-26T10:55:43.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T10:55:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:43 smithi160 ceph-mon[36715]: from='osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T10:55:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:43 smithi160 ceph-mon[36715]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T10:55:43.349 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:55:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:43 smithi183 ceph-mon[37273]: from='osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T10:55:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:43 smithi183 ceph-mon[37273]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T10:55:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:43 smithi144 ceph-mon[32129]: from='osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T10:55:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:43 smithi144 ceph-mon[32129]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T10:55:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T10:55:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: from='osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:55:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T10:55:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:55:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T10:55:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: from='osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:55:44.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T10:55:44.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:55:44.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:44.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: from='osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:44.986 INFO:journalctl@ceph.osd.10.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-10[62538]: 2024-07-26T10:55:44.819+0000 7f5236f73640 -1 osd.10 0 waiting for initial osdmap 2024-07-26T10:55:44.986 INFO:journalctl@ceph.osd.10.smithi160.stdout:Jul 26 10:55:44 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-10[62538]: 2024-07-26T10:55:44.832+0000 7f522e554640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:55:45.201 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T10:55:45.258 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:55:45.258 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.258 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T10:55:45.258 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.259 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: from='osd.10 ' entity='osd.10' 2024-07-26T10:55:45.259 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:45.259 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.259 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] boot 2024-07-26T10:55:45.259 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:45 smithi160 ceph-mon[36715]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T10:55:45.310 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: from='osd.10 ' entity='osd.10' 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:45.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] boot 2024-07-26T10:55:45.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:45 smithi183 ceph-mon[37273]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T10:55:45.328 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-07-26T10:55:45.329 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-07-26T10:55:45.329 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0161492 s, 649 MB/s 2024-07-26T10:55:45.329 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: from='osd.10 ' entity='osd.10' 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-07-26T10:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T10:55:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: osd.10 [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] boot 2024-07-26T10:55:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:45 smithi144 ceph-mon[32129]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T10:55:45.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2024-07-26T10:55:46.227 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:55:46.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:46 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:55:46.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:46 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:55:46.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:46 smithi183 ceph-mon[37273]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T10:55:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:46 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:55:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:46 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:55:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:46 smithi144 ceph-mon[32129]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T10:55:46.504 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:46 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:55:46.504 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:46 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:55:46.505 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:46 smithi160 ceph-mon[36715]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T10:55:47.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T10:55:47.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi160 2024-07-26T10:55:47.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 4272M 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:47.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:47.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:47.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:47 smithi183 ceph-mon[37273]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:55:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi160 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 4272M 2024-07-26T10:55:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:47 smithi144 ceph-mon[32129]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:55:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi160 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 4272M 2024-07-26T10:55:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:55:47.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:47 smithi160 ceph-mon[36715]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:55:47.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.871+0000 7f67f537a640 1 -- 172.21.15.183:0/623958369 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 msgr2=0x7f67f01028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.871+0000 7f67f537a640 1 --2- 172.21.15.183:0/623958369 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f01028a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009900 tx=0x7f67e002a410 comp rx=0 tx=0).stop 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 -- 172.21.15.183:0/623958369 shutdown_connections 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 --2- 172.21.15.183:0/623958369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67f0108720 0x7f67f010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 --2- 172.21.15.183:0/623958369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67f0107dd0 0x7f67f01081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 --2- 172.21.15.183:0/623958369 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f01028a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 -- 172.21.15.183:0/623958369 >> 172.21.15.183:0/623958369 conn(0x7f67f0079190 msgr2=0x7f67f00795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 -- 172.21.15.183:0/623958369 shutdown_connections 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 -- 172.21.15.183:0/623958369 wait complete. 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 Processor -- start 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.872+0000 7f67f537a640 1 -- start start 2024-07-26T10:55:47.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67f537a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f00ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67f537a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67f0107dd0 0x7f67f00ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67f537a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67f0108720 0x7f67f00ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67f537a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f67f0112440 con 0x7f67f0108720 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67f537a640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f67f01122c0 con 0x7f67f0107dd0 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67f537a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f67f01125c0 con 0x7f67f0102490 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f00ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f00ff080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:33226/0 (socket says 172.21.15.183:33226) 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 -- 172.21.15.183:0/4103627585 learned_addr learned my addr 172.21.15.183:0/4103627585 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67ee7fc640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67f0107dd0 0x7f67f00ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 -- 172.21.15.183:0/4103627585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67f0107dd0 msgr2=0x7f67f00ff5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67f0107dd0 0x7f67f00ff5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 -- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67f0108720 msgr2=0x7f67f00ffb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:55:47.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67f0108720 0x7f67f00ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 -- 172.21.15.183:0/4103627585 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f01a8200 con 0x7f67f0102490 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.873+0000 7f67eeffd640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f00ff080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67e0005940 tx=0x7f67e00042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.874+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e003f030 con 0x7f67f0102490 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.874+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f01a8490 con 0x7f67f0102490 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.874+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67e0003cb0 con 0x7f67f0102490 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.874+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f01a89c0 con 0x7f67f0102490 2024-07-26T10:55:47.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.874+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0035070 con 0x7f67f0102490 2024-07-26T10:55:47.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.875+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f0103710 con 0x7f67f0102490 2024-07-26T10:55:47.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.876+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f67e0042030 con 0x7f67f0102490 2024-07-26T10:55:47.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.877+0000 7f67d7fff640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f67d80b3c20 0x7f67d80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:47.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.877+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6909+0+0 (secure 0 0 0) 0x7f67e0104390 con 0x7f67f0102490 2024-07-26T10:55:47.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.878+0000 7f67ee7fc640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f67d80b3c20 0x7f67d80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:47.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.879+0000 7f67ee7fc640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f67d80b3c20 0x7f67d80b6130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f67e4005d00 tx=0x7f67e4005c50 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:55:47.882 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:47.880+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f67e00cbec0 con 0x7f67f0102490 2024-07-26T10:55:48.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:55:48.061+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f67f0063230 con 0x7f67d80b3c20 2024-07-26T10:55:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:48 smithi183 ceph-mon[37273]: from='client.24565 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:55:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:55:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:55:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:48 smithi144 ceph-mon[32129]: from='client.24565 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:55:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:55:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:55:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:48 smithi160 ceph-mon[36715]: from='client.24565 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:55:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T10:55:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T10:55:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:49.878 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:49 smithi183 ceph-mon[37273]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-07-26T10:55:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:49 smithi144 ceph-mon[32129]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-07-26T10:55:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:49 smithi160 ceph-mon[36715]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-07-26T10:55:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:50 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/1005439585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]: dispatch 2024-07-26T10:55:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:50 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]: dispatch 2024-07-26T10:55:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:50 smithi183 ceph-mon[37273]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]': finished 2024-07-26T10:55:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:55:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:50 smithi183 ceph-mon[37273]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T10:55:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:50 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/1005439585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]: dispatch 2024-07-26T10:55:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:50 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]: dispatch 2024-07-26T10:55:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:50 smithi144 ceph-mon[32129]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]': finished 2024-07-26T10:55:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:55:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:50 smithi144 ceph-mon[32129]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T10:55:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:50 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/1005439585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]: dispatch 2024-07-26T10:55:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:50 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]: dispatch 2024-07-26T10:55:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:50 smithi160 ceph-mon[36715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f00c381-b270-461b-a3be-ae5822a8affb"}]': finished 2024-07-26T10:55:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:55:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:50 smithi160 ceph-mon[36715]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T10:55:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:51 smithi183 ceph-mon[37273]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-07-26T10:55:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:51 smithi183 ceph-mon[37273]: from='client.? 172.21.15.183:0/3127101684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:55:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:51 smithi144 ceph-mon[32129]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-07-26T10:55:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:51 smithi144 ceph-mon[32129]: from='client.? 172.21.15.183:0/3127101684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:55:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:51 smithi160 ceph-mon[36715]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-07-26T10:55:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:51 smithi160 ceph-mon[36715]: from='client.? 172.21.15.183:0/3127101684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T10:55:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:53 smithi160 ceph-mon[36715]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-07-26T10:55:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:53 smithi183 ceph-mon[37273]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-07-26T10:55:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:53 smithi144 ceph-mon[32129]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-07-26T10:55:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:55 smithi160 ceph-mon[36715]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T10:55:56.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:55 smithi183 ceph-mon[37273]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T10:55:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:55 smithi144 ceph-mon[32129]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T10:55:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:57 smithi160 ceph-mon[36715]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T10:55:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:58.325 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:57 smithi183 ceph-mon[37273]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:58.325 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T10:55:58.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:57 smithi144 ceph-mon[32129]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T10:55:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T10:55:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:59.088 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:55:59 smithi183 ceph-mon[37273]: Deploying daemon osd.11 on smithi183 2024-07-26T10:55:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:55:59 smithi160 ceph-mon[36715]: Deploying daemon osd.11 on smithi183 2024-07-26T10:55:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:55:59 smithi144 ceph-mon[32129]: Deploying daemon osd.11 on smithi183 2024-07-26T10:56:00.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:00 smithi183 ceph-mon[37273]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:00 smithi160 ceph-mon[36715]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:00 smithi144 ceph-mon[32129]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:02 smithi160 ceph-mon[36715]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:02.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:02 smithi183 ceph-mon[37273]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:02 smithi144 ceph-mon[32129]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:03.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:03.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:03.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:04 smithi160 ceph-mon[36715]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:04 smithi183 ceph-mon[37273]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:04 smithi144 ceph-mon[32129]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:06 smithi160 ceph-mon[36715]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:06 smithi183 ceph-mon[37273]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:06.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:06.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:06.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:06 smithi144 ceph-mon[32129]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:06.685 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.683+0000 7f67d7fff640 1 -- 172.21.15.183:0/4103627585 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f67f0063230 con 0x7f67d80b3c20 2024-07-26T10:56:06.686 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 11 on host 'smithi183' 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f67d80b3c20 msgr2=0x7f67d80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f67d80b3c20 0x7f67d80b6130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f67e4005d00 tx=0x7f67e4005c50 comp rx=0 tx=0).stop 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 msgr2=0x7f67f00ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f00ff080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67e0005940 tx=0x7f67e00042f0 comp rx=0 tx=0).stop 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 shutdown_connections 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f67d80b3c20 0x7f67d80b6130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67f0108720 0x7f67f00ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67f0107dd0 0x7f67f00ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 --2- 172.21.15.183:0/4103627585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67f0102490 0x7f67f00ff080 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:06.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 >> 172.21.15.183:0/4103627585 conn(0x7f67f0079190 msgr2=0x7f67f006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:06.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 shutdown_connections 2024-07-26T10:56:06.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:06.686+0000 7f67f537a640 1 -- 172.21.15.183:0/4103627585 wait complete. 2024-07-26T10:56:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:07 smithi183 ceph-mon[37273]: from='osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T10:56:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:07 smithi183 ceph-mon[37273]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T10:56:07.307 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:07 smithi144 ceph-mon[32129]: from='osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T10:56:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:07 smithi144 ceph-mon[32129]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T10:56:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:07 smithi160 ceph-mon[36715]: from='osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T10:56:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:07 smithi160 ceph-mon[36715]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T10:56:07.318 DEBUG:teuthology.orchestra.run.smithi183:osd.11> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@osd.11.service 2024-07-26T10:56:07.320 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-26T10:56:07.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd stat -f json 2024-07-26T10:56:07.784 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:08 smithi183 ceph-mon[37273]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:08 smithi183 ceph-mon[37273]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T10:56:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:08 smithi183 ceph-mon[37273]: from='osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:56:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:08 smithi183 ceph-mon[37273]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T10:56:08.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:08 smithi183 ceph-mon[37273]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:56:08.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:08 smithi144 ceph-mon[32129]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:08.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:08 smithi144 ceph-mon[32129]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T10:56:08.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:08.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:08 smithi144 ceph-mon[32129]: from='osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:56:08.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:08 smithi144 ceph-mon[32129]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T10:56:08.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:08 smithi144 ceph-mon[32129]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:56:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:08 smithi160 ceph-mon[36715]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:08 smithi160 ceph-mon[36715]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T10:56:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:08 smithi160 ceph-mon[36715]: from='osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:56:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:08 smithi160 ceph-mon[36715]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T10:56:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:08 smithi160 ceph-mon[36715]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-07-26T10:56:09.693 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-11[63192]: 2024-07-26T10:56:09.426+0000 7ff566334640 -1 osd.11 0 waiting for initial osdmap 2024-07-26T10:56:09.693 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-osd-11[63192]: 2024-07-26T10:56:09.442+0000 7ff55d915640 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T10:56:09.788 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:09 smithi144 ceph-mon[32129]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:56:09.788 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:09 smithi144 ceph-mon[32129]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T10:56:09.788 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:09.788 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:09.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:09 smithi144 ceph-mon[32129]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:09.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:09 smithi144 ceph-mon[32129]: from='osd.11 ' entity='osd.11' 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 -- 172.21.15.144:0/2709838458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10f330 msgr2=0x7fef8c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 --2- 172.21.15.144:0/2709838458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10f330 0x7fef8c10f740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fef74009900 tx=0x7fef7402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 -- 172.21.15.144:0/2709838458 shutdown_connections 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 --2- 172.21.15.144:0/2709838458 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10fc80 0x7fef8c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 --2- 172.21.15.144:0/2709838458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10f330 0x7fef8c10f740 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 --2- 172.21.15.144:0/2709838458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.992+0000 7fef8b200640 1 -- 172.21.15.144:0/2709838458 >> 172.21.15.144:0/2709838458 conn(0x7fef8c078ff0 msgr2=0x7fef8c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.993+0000 7fef8b200640 1 -- 172.21.15.144:0/2709838458 shutdown_connections 2024-07-26T10:56:09.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.993+0000 7fef8b200640 1 -- 172.21.15.144:0/2709838458 wait complete. 2024-07-26T10:56:09.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.994+0000 7fef8b200640 1 Processor -- start 2024-07-26T10:56:09.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.994+0000 7fef8b200640 1 -- start start 2024-07-26T10:56:09.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.995+0000 7fef8b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c1a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:09.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.995+0000 7fef8b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10f330 0x7fef8c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:09.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.995+0000 7fef8b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10fc80 0x7fef8c1a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:09.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.995+0000 7fef8b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c119940 con 0x7fef8c10fc80 2024-07-26T10:56:09.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.995+0000 7fef8b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c1197c0 con 0x7fef8c10f330 2024-07-26T10:56:09.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.995+0000 7fef8b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c119ac0 con 0x7fef8c1099f0 2024-07-26T10:56:09.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef89e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c1a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:09.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef89400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10f330 0x7fef8c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:09.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef89e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c1a0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:59570/0 (socket says 172.21.15.144:59570) 2024-07-26T10:56:09.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef89e00640 1 -- 172.21.15.144:0/3478816319 learned_addr learned my addr 172.21.15.144:0/3478816319 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:09.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef8a800640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10fc80 0x7fef8c1a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:09.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef8a800640 1 -- 172.21.15.144:0/3478816319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 msgr2=0x7fef8c1a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:09.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef8a800640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c1a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:09.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef8a800640 1 -- 172.21.15.144:0/3478816319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10f330 msgr2=0x7fef8c1a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:09.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef8a800640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10f330 0x7fef8c1a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:09.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef8a800640 1 -- 172.21.15.144:0/3478816319 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c1a62c0 con 0x7fef8c10fc80 2024-07-26T10:56:09.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef89e00640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c1a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.996+0000 7fef89400640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10f330 0x7fef8c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.997+0000 7fef8a800640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10fc80 0x7fef8c1a5b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e970 tx=0x7fef7c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.997+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00ccc0 con 0x7fef8c10fc80 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.997+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c1a65b0 con 0x7fef8c10fc80 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.997+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef7c005020 con 0x7fef8c10fc80 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.997+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c1acd10 con 0x7fef8c10fc80 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.997+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c01c070 con 0x7fef8c10fc80 2024-07-26T10:56:10.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:09.999+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8c109e00 con 0x7fef8c10fc80 2024-07-26T10:56:10.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.001+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fef7c0049a0 con 0x7fef8c10fc80 2024-07-26T10:56:10.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.003+0000 7fef82a00640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef6c0b3c20 0x7fef6c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:10.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.004+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7062+0+0 (secure 0 0 0) 0x7fef7c0e36a0 con 0x7fef8c10fc80 2024-07-26T10:56:10.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.004+0000 7fef89e00640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef6c0b3c20 0x7fef6c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:10.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.004+0000 7fef89e00640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef6c0b3c20 0x7fef6c0b6130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fef8400e840 tx=0x7fef84009800 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:10.009 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.008+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fef7c0ab130 con 0x7fef8c10fc80 2024-07-26T10:56:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-mon[37273]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:56:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-mon[37273]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T10:56:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-mon[37273]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:09 smithi183 ceph-mon[37273]: from='osd.11 ' entity='osd.11' 2024-07-26T10:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:09 smithi160 ceph-mon[36715]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-07-26T10:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:09 smithi160 ceph-mon[36715]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T10:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:09 smithi160 ceph-mon[36715]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T10:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:09 smithi160 ceph-mon[36715]: from='osd.11 ' entity='osd.11' 2024-07-26T10:56:10.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.206+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fef8c1114e0 con 0x7fef8c10fc80 2024-07-26T10:56:10.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.207+0000 7fef82a00640 1 -- 172.21.15.144:0/3478816319 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7fef7c0aaa30 con 0x7fef8c10fc80 2024-07-26T10:56:10.209 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:10.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef6c0b3c20 msgr2=0x7fef6c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:10.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef6c0b3c20 0x7fef6c0b6130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fef8400e840 tx=0x7fef84009800 comp rx=0 tx=0).stop 2024-07-26T10:56:10.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10fc80 msgr2=0x7fef8c1a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10fc80 0x7fef8c1a5b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e970 tx=0x7fef7c00ee30 comp rx=0 tx=0).stop 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 shutdown_connections 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef6c0b3c20 0x7fef6c0b6130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef8c10fc80 0x7fef8c1a5b90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef8c10f330 0x7fef8c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 --2- 172.21.15.144:0/3478816319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef8c1099f0 0x7fef8c1a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 >> 172.21.15.144:0/3478816319 conn(0x7fef8c078ff0 msgr2=0x7fef8c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.210+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 shutdown_connections 2024-07-26T10:56:10.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:10.211+0000 7fef8b200640 1 -- 172.21.15.144:0/3478816319 wait complete. 2024-07-26T10:56:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: purged_snaps scrub starts 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: purged_snaps scrub ok 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3478816319' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] boot 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:10.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: purged_snaps scrub starts 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: purged_snaps scrub ok 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3478816319' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] boot 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:11.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.106 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1721991345,"num_in_osds":12,"osd_in_since":1721991350,"num_remapped_pgs":0} 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: purged_snaps scrub starts 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: purged_snaps scrub ok 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3478816319' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: osd.11 [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] boot 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T10:56:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:11.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:11.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:11 smithi183 ceph-mon[37273]: Detected new or changed devices on smithi183 2024-07-26T10:56:11.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:11 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 4272M 2024-07-26T10:56:11.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:11 smithi183 ceph-mon[37273]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:11 smithi144 ceph-mon[32129]: Detected new or changed devices on smithi183 2024-07-26T10:56:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:11 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 4272M 2024-07-26T10:56:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:11 smithi144 ceph-mon[32129]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:11 smithi160 ceph-mon[36715]: Detected new or changed devices on smithi183 2024-07-26T10:56:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:11 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 4272M 2024-07-26T10:56:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:11 smithi160 ceph-mon[36715]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:12.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd stat -f json 2024-07-26T10:56:12.559 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:12 smithi144 ceph-mon[32129]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T10:56:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:12 smithi160 ceph-mon[36715]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T10:56:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:12 smithi183 ceph-mon[37273]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T10:56:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:13 smithi144 ceph-mon[32129]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:13 smithi160 ceph-mon[36715]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:13 smithi183 ceph-mon[37273]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:14.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 -- 172.21.15.144:0/1474852510 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410b860 msgr2=0x7f2964112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:14.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 --2- 172.21.15.144:0/1474852510 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410b860 0x7f2964112090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f295c009900 tx=0x7f295c02a3f0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 -- 172.21.15.144:0/1474852510 shutdown_connections 2024-07-26T10:56:14.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 --2- 172.21.15.144:0/1474852510 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410b860 0x7f2964112090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 --2- 172.21.15.144:0/1474852510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f296410af10 0x7f296410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 --2- 172.21.15.144:0/1474852510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f29641059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 -- 172.21.15.144:0/1474852510 >> 172.21.15.144:0/1474852510 conn(0x7f2964101040 msgr2=0x7f2964103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:14.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 -- 172.21.15.144:0/1474852510 shutdown_connections 2024-07-26T10:56:14.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.873+0000 7f296a600640 1 -- 172.21.15.144:0/1474852510 wait complete. 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 Processor -- start 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 -- start start 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f296419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410af10 0x7f296419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f296410b860 0x7f29641a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2964115530 con 0x7f29641055d0 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f29641153b0 con 0x7f296410af10 2024-07-26T10:56:14.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f296a600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f29641156b0 con 0x7f296410b860 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f2969200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f296419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.875+0000 7f2969200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f296419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49510/0 (socket says 172.21.15.144:49510) 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 -- 172.21.15.144:0/2691007781 learned_addr learned my addr 172.21.15.144:0/2691007781 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969c00640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f296410b860 0x7f29641a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2963e00640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410af10 0x7f296419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 -- 172.21.15.144:0/2691007781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f296410b860 msgr2=0x7f29641a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f296410b860 0x7f29641a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 -- 172.21.15.144:0/2691007781 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410af10 msgr2=0x7f296419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:14.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410af10 0x7f296419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 -- 172.21.15.144:0/2691007781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29641a1e70 con 0x7f29641055d0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2963e00640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410af10 0x7f296419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.876+0000 7f2969200640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f296419bf00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f295400ebc0 tx=0x7f295400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.877+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954010630 con 0x7f29641055d0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.877+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29641a2160 con 0x7f29641055d0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.877+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2954010c80 con 0x7f29641055d0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.877+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954018b60 con 0x7f29641055d0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.877+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29641a88c0 con 0x7f29641055d0 2024-07-26T10:56:14.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.878+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29641059e0 con 0x7f29641055d0 2024-07-26T10:56:14.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.881+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f2954018d00 con 0x7f29641055d0 2024-07-26T10:56:14.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.882+0000 7f2961600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f29480b3c20 0x7f29480b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:14.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.882+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f29540e4380 con 0x7f29641055d0 2024-07-26T10:56:14.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.883+0000 7f2963e00640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f29480b3c20 0x7f29480b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.883+0000 7f2963e00640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f29480b3c20 0x7f29480b6130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f296410a870 tx=0x7f294c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:14.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:14.884+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f29540ab6b0 con 0x7f29641055d0 2024-07-26T10:56:15.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.087+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f296402e0c0 con 0x7f29641055d0 2024-07-26T10:56:15.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.088+0000 7f2961600640 1 -- 172.21.15.144:0/2691007781 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f29540afc00 con 0x7f29641055d0 2024-07-26T10:56:15.090 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f29480b3c20 msgr2=0x7f29480b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f29480b3c20 0x7f29480b6130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f296410a870 tx=0x7f294c005f90 comp rx=0 tx=0).stop 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 msgr2=0x7f296419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f296419bf00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f295400ebc0 tx=0x7f295400c6a0 comp rx=0 tx=0).stop 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 shutdown_connections 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f29480b3c20 0x7f29480b6130 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f296410b860 0x7f29641a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f296410af10 0x7f296419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 --2- 172.21.15.144:0/2691007781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29641055d0 0x7f296419bf00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:15.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.091+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 >> 172.21.15.144:0/2691007781 conn(0x7f2964101040 msgr2=0x7f2964103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:15.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.092+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 shutdown_connections 2024-07-26T10:56:15.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:15.092+0000 7f296a600640 1 -- 172.21.15.144:0/2691007781 wait complete. 2024-07-26T10:56:15.916 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1721991370,"num_in_osds":12,"osd_in_since":1721991350,"num_remapped_pgs":0} 2024-07-26T10:56:15.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:56:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:15 smithi144 ceph-mon[32129]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:15 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2691007781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T10:56:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:15 smithi160 ceph-mon[36715]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:15 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2691007781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T10:56:16.357 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:15 smithi183 ceph-mon[37273]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:15 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2691007781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T10:56:18.265 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:17 smithi144 ceph-mon[32129]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:17 smithi160 ceph-mon[36715]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:17 smithi183 ceph-mon[37273]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:18.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 -- 172.21.15.144:0/1308255409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c10fc80 msgr2=0x7f3d0c1164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:18.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 --2- 172.21.15.144:0/1308255409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c10fc80 0x7f3d0c1164b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009be0 tx=0x7f3d0402a6d0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 -- 172.21.15.144:0/1308255409 shutdown_connections 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 --2- 172.21.15.144:0/1308255409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c10fc80 0x7f3d0c1164b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 --2- 172.21.15.144:0/1308255409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c10f330 0x7f3d0c10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 --2- 172.21.15.144:0/1308255409 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 -- 172.21.15.144:0/1308255409 >> 172.21.15.144:0/1308255409 conn(0x7f3d0c078ff0 msgr2=0x7f3d0c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 -- 172.21.15.144:0/1308255409 shutdown_connections 2024-07-26T10:56:18.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.585+0000 7f3d11400640 1 -- 172.21.15.144:0/1308255409 wait complete. 2024-07-26T10:56:18.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.586+0000 7f3d11400640 1 Processor -- start 2024-07-26T10:56:18.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.586+0000 7f3d11400640 1 -- start start 2024-07-26T10:56:18.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d11400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c1a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:18.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d11400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c10f330 0x7f3d0c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:18.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d11400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d0c10fc80 0x7f3d0c1a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:18.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d11400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c119940 con 0x7f3d0c10f330 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d11400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1197c0 con 0x7f3d0c10fc80 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d11400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c119ac0 con 0x7f3d0c1099f0 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d0be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c1a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d0be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c1a02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:59610/0 (socket says 172.21.15.144:59610) 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d0be00640 1 -- 172.21.15.144:0/1163082980 learned_addr learned my addr 172.21.15.144:0/1163082980 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d03400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c10f330 0x7f3d0c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.587+0000 7f3d10a00640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d0c10fc80 0x7f3d0c1a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d0be00640 1 -- 172.21.15.144:0/1163082980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d0c10fc80 msgr2=0x7f3d0c1a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d0be00640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d0c10fc80 0x7f3d0c1a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d0be00640 1 -- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c10f330 msgr2=0x7f3d0c1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d0be00640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c10f330 0x7f3d0c1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d0be00640 1 -- 172.21.15.144:0/1163082980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0c1a6220 con 0x7f3d0c1099f0 2024-07-26T10:56:18.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d0be00640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c1a02b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00e9c0 tx=0x7f3cfc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:18.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00cde0 con 0x7f3d0c1099f0 2024-07-26T10:56:18.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d0c1a6510 con 0x7f3d0c1099f0 2024-07-26T10:56:18.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cfc0035f0 con 0x7f3d0c1099f0 2024-07-26T10:56:18.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc010690 con 0x7f3d0c1099f0 2024-07-26T10:56:18.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.588+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d0c1acc70 con 0x7f3d0c1099f0 2024-07-26T10:56:18.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.589+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d0c109e00 con 0x7f3d0c1099f0 2024-07-26T10:56:18.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.592+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f3cfc004000 con 0x7f3d0c1099f0 2024-07-26T10:56:18.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.593+0000 7f3d09600640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3cf00b3c20 0x7f3cf00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:18.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.594+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f3cfc0e3cf0 con 0x7f3d0c1099f0 2024-07-26T10:56:18.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.594+0000 7f3d03400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3cf00b3c20 0x7f3cf00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:18.597 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.594+0000 7f3d03400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3cf00b3c20 0x7f3cf00b6130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c10fa50 tx=0x7f3cf4005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:18.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.595+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3cfc0ab700 con 0x7f3d0c1099f0 2024-07-26T10:56:18.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.798+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d0c1a13e0 con 0x7f3d0c1099f0 2024-07-26T10:56:18.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.800+0000 7f3d09600640 1 -- 172.21.15.144:0/1163082980 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18847 (secure 0 0 0) 0x7f3cfc0ab000 con 0x7f3d0c1099f0 2024-07-26T10:56:18.805 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:18.806 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":72,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:56:11.695117+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:56:18.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3cf00b3c20 msgr2=0x7f3cf00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:18.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3cf00b3c20 0x7f3cf00b6130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c10fa50 tx=0x7f3cf4005f90 comp rx=0 tx=0).stop 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 msgr2=0x7f3d0c1a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c1a02b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00e9c0 tx=0x7f3cfc00ee80 comp rx=0 tx=0).stop 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 shutdown_connections 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3cf00b3c20 0x7f3cf00b6130 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d0c10fc80 0x7f3d0c1a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c10f330 0x7f3d0c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 --2- 172.21.15.144:0/1163082980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d0c1099f0 0x7f3d0c1a02b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 >> 172.21.15.144:0/1163082980 conn(0x7f3d0c078ff0 msgr2=0x7f3d0c0796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 shutdown_connections 2024-07-26T10:56:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:18.803+0000 7f3d11400640 1 -- 172.21.15.144:0/1163082980 wait complete. 2024-07-26T10:56:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:19 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1163082980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:56:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:19 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1163082980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:56:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:19 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1163082980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:56:19.602 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T10:52:04.947795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T10:56:19.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-26T10:56:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:20 smithi160 ceph-mon[36715]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:20 smithi183 ceph-mon[37273]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:20.394 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:20 smithi144 ceph-mon[32129]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 -- 172.21.15.144:0/3425888676 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810ba40 msgr2=0x7f5b18112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 --2- 172.21.15.144:0/3425888676 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810ba40 0x7f5b18112270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009ca0 tx=0x7f5b1402a8c0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 -- 172.21.15.144:0/3425888676 shutdown_connections 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 --2- 172.21.15.144:0/3425888676 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810ba40 0x7f5b18112270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 --2- 172.21.15.144:0/3425888676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b1810b0f0 0x7f5b1810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 --2- 172.21.15.144:0/3425888676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b181057b0 0x7f5b18105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 -- 172.21.15.144:0/3425888676 >> 172.21.15.144:0/3425888676 conn(0x7f5b181011c0 msgr2=0x7f5b18103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 -- 172.21.15.144:0/3425888676 shutdown_connections 2024-07-26T10:56:22.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.256+0000 7f5b20800640 1 -- 172.21.15.144:0/3425888676 wait complete. 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 Processor -- start 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 -- start start 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b181057b0 0x7f5b1819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810b0f0 0x7f5b1819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b1810ba40 0x7f5b181a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5b18115760 con 0x7f5b1810ba40 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5b181155e0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b20800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b181158e0 con 0x7f5b181057b0 2024-07-26T10:56:22.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.258+0000 7f5b1e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b1810ba40 0x7f5b181a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:22.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b1810ba40 0x7f5b181a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52294/0 (socket says 172.21.15.144:52294) 2024-07-26T10:56:22.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1e800640 1 -- 172.21.15.144:0/1168356831 learned_addr learned my addr 172.21.15.144:0/1168356831 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:22.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810b0f0 0x7f5b1819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1de00640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b181057b0 0x7f5b1819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 -- 172.21.15.144:0/1168356831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b181057b0 msgr2=0x7f5b1819c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b181057b0 0x7f5b1819c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 -- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b1810ba40 msgr2=0x7f5b181a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b1810ba40 0x7f5b181a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 -- 172.21.15.144:0/1168356831 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b181a2020 con 0x7f5b1810b0f0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.259+0000 7f5b1d400640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810b0f0 0x7f5b1819c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b1806c3a0 tx=0x7f5b0800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.260+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800cc40 con 0x7f5b1810b0f0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.260+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0800cde0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.260+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08004a00 con 0x7f5b1810b0f0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.260+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b181a2310 con 0x7f5b1810b0f0 2024-07-26T10:56:22.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.260+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b181a8a70 con 0x7f5b1810b0f0 2024-07-26T10:56:22.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.261+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b18105bc0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.263 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:22 smithi144 ceph-mon[32129]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:22.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.264+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f5b08004ba0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.266+0000 7f5b12a00640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b000b3ca0 0x7f5b000b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:22.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.266+0000 7f5b1de00640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b000b3ca0 0x7f5b000b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:22.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.266+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f5b080e3770 con 0x7f5b1810b0f0 2024-07-26T10:56:22.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.267+0000 7f5b1de00640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b000b3ca0 0x7f5b000b61b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c005d00 tx=0x7f5b0c00b410 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:22.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.269+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5b080ab0f0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:22 smithi160 ceph-mon[36715]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:22.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:22 smithi183 ceph-mon[37273]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:22.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.465+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b1802e0c0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.465+0000 7f5b12a00640 1 -- 172.21.15.144:0/1168356831 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f5b080aa9f0 con 0x7f5b1810b0f0 2024-07-26T10:56:22.468 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 1 2024-07-26T10:56:22.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b000b3ca0 msgr2=0x7f5b000b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:22.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b000b3ca0 0x7f5b000b61b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c005d00 tx=0x7f5b0c00b410 comp rx=0 tx=0).stop 2024-07-26T10:56:22.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810b0f0 msgr2=0x7f5b1819c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:22.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810b0f0 0x7f5b1819c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b1806c3a0 tx=0x7f5b0800ee50 comp rx=0 tx=0).stop 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 shutdown_connections 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b000b3ca0 0x7f5b000b61b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b1810ba40 0x7f5b181a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b1810b0f0 0x7f5b1819c650 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 --2- 172.21.15.144:0/1168356831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b181057b0 0x7f5b1819c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 >> 172.21.15.144:0/1168356831 conn(0x7f5b181011c0 msgr2=0x7f5b1810a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.468+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 shutdown_connections 2024-07-26T10:56:22.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:22.469+0000 7f5b20800640 1 -- 172.21.15.144:0/1168356831 wait complete. 2024-07-26T10:56:23.153 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:23 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1168356831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.a on smithi144 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.d on smithi144 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.g on smithi144 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.j on smithi144 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.b on smithi160 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.e on smithi160 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.h on smithi160 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.k on smithi160 2024-07-26T10:56:23.229 INFO:tasks.cephadm:Adding mds.c on smithi183 2024-07-26T10:56:23.230 INFO:tasks.cephadm:Adding mds.f on smithi183 2024-07-26T10:56:23.230 INFO:tasks.cephadm:Adding mds.i on smithi183 2024-07-26T10:56:23.230 INFO:tasks.cephadm:Adding mds.l on smithi183 2024-07-26T10:56:23.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph orch apply mds all '12;smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l' 2024-07-26T10:56:23.256 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:23 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1168356831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T10:56:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:23 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1168356831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T10:56:23.491 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T10:56:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:24 smithi160 ceph-mon[36715]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:24 smithi183 ceph-mon[37273]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:24 smithi144 ceph-mon[32129]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 -- 172.21.15.183:0/2126045261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414103590 msgr2=0x7f44141039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2126045261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414103590 0x7f44141039a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4410009900 tx=0x7f441002a410 comp rx=0 tx=0).stop 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 -- 172.21.15.183:0/2126045261 shutdown_connections 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2126045261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4414109820 0x7f4414110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2126045261 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4414108ed0 0x7f44141092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2126045261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414103590 0x7f44141039a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.112+0000 7f441c0d3640 1 -- 172.21.15.183:0/2126045261 >> 172.21.15.183:0/2126045261 conn(0x7f441407b280 msgr2=0x7f441407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 -- 172.21.15.183:0/2126045261 shutdown_connections 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 -- 172.21.15.183:0/2126045261 wait complete. 2024-07-26T10:56:25.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 Processor -- start 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 -- start start 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4414103590 0x7f4414075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 0x7f4414075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4414109820 0x7f4414075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4414113560 con 0x7f4414103590 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f44141133e0 con 0x7f4414109820 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.113+0000 7f441c0d3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f44141136e0 con 0x7f4414108ed0 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 0x7f4414075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 0x7f4414075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:49874/0 (socket says 172.21.15.183:49874) 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 -- 172.21.15.183:0/2517522028 learned_addr learned my addr 172.21.15.183:0/2517522028 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441a670640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4414109820 0x7f4414075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:25.129 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 -- 172.21.15.183:0/2517522028 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4414109820 msgr2=0x7f4414075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4414109820 0x7f4414075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 -- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4414103590 msgr2=0x7f4414075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4414103590 0x7f4414075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 -- 172.21.15.183:0/2517522028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44141a8270 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441966e640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 0x7f4414075680 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f440400d750 tx=0x7f440400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404017070 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44040054d0 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44141a8560 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404004c30 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.114+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44141a8aa0 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.115+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44141039a0 con 0x7f4414108ed0 2024-07-26T10:56:25.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.117+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4404004dd0 con 0x7f4414108ed0 2024-07-26T10:56:25.131 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.118+0000 7f440affd640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f44000b3ca0 0x7f44000b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:25.131 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.118+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f44040e3610 con 0x7f4414108ed0 2024-07-26T10:56:25.131 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.118+0000 7f4419e6f640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f44000b3ca0 0x7f44000b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:25.131 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.119+0000 7f4419e6f640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f44000b3ca0 0x7f44000b61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f441002a930 tx=0x7f4410031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:25.131 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.121+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f44040ab0b0 con 0x7f4414108ed0 2024-07-26T10:56:25.304 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.302+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}) -- 0x7f4414063330 con 0x7f44000b3ca0 2024-07-26T10:56:25.323 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mds.all update... 2024-07-26T10:56:25.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.320+0000 7f440affd640 1 -- 172.21.15.183:0/2517522028 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f4414063330 con 0x7f44000b3ca0 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f44000b3ca0 msgr2=0x7f44000b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f44000b3ca0 0x7f44000b61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f441002a930 tx=0x7f4410031400 comp rx=0 tx=0).stop 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 msgr2=0x7f4414075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 0x7f4414075680 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f440400d750 tx=0x7f440400dc10 comp rx=0 tx=0).stop 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 shutdown_connections 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f44000b3ca0 0x7f44000b61b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4414109820 0x7f4414075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4414108ed0 0x7f4414075680 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 --2- 172.21.15.183:0/2517522028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4414103590 0x7f4414075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 >> 172.21.15.183:0/2517522028 conn(0x7f441407b280 msgr2=0x7f44141070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 shutdown_connections 2024-07-26T10:56:25.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-07-26T10:56:25.323+0000 7f441c0d3640 1 -- 172.21.15.183:0/2517522028 wait complete. 2024-07-26T10:56:25.989 DEBUG:teuthology.orchestra.run.smithi144:mds.a> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service 2024-07-26T10:56:25.995 DEBUG:teuthology.orchestra.run.smithi144:mds.d> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service 2024-07-26T10:56:26.042 DEBUG:teuthology.orchestra.run.smithi144:mds.g> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.g.service 2024-07-26T10:56:26.089 DEBUG:teuthology.orchestra.run.smithi144:mds.j> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service 2024-07-26T10:56:26.135 DEBUG:teuthology.orchestra.run.smithi160:mds.b> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.b.service 2024-07-26T10:56:26.138 DEBUG:teuthology.orchestra.run.smithi160:mds.e> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.e.service 2024-07-26T10:56:26.181 DEBUG:teuthology.orchestra.run.smithi160:mds.h> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.h.service 2024-07-26T10:56:26.224 DEBUG:teuthology.orchestra.run.smithi160:mds.k> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service 2024-07-26T10:56:26.267 DEBUG:teuthology.orchestra.run.smithi183:mds.c> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service 2024-07-26T10:56:26.269 DEBUG:teuthology.orchestra.run.smithi183:mds.f> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service 2024-07-26T10:56:26.285 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:26 smithi160 ceph-mon[36715]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:26.285 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:26.285 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:26.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:26 smithi183 ceph-mon[37273]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:26.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:26.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:26.289 DEBUG:teuthology.orchestra.run.smithi183:mds.i> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service 2024-07-26T10:56:26.332 DEBUG:teuthology.orchestra.run.smithi183:mds.l> sudo journalctl -f -n 0 -u ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service 2024-07-26T10:56:26.375 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-26T10:56:26.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:56:26.451 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:26 smithi144 ceph-mon[32129]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:26.451 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:26.451 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:26.824 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='client.24616 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: Saving service mds.all spec with placement smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-07-26T10:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:27.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:27.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:27.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:27.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='client.24616 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:56:27.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: Saving service mds.all spec with placement smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-07-26T10:56:27.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:27.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:27.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:27.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:27.274 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:27.274 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:27.274 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='client.24616 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: Saving service mds.all spec with placement smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:28.277 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:28 smithi144 ceph-mon[32129]: Deploying daemon mds.l on smithi183 2024-07-26T10:56:28.277 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:28 smithi144 ceph-mon[32129]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:28.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:28 smithi183 ceph-mon[37273]: Deploying daemon mds.l on smithi183 2024-07-26T10:56:28.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:28 smithi183 ceph-mon[37273]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:28.304 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 systemd[1]: Starting Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:28 smithi160 ceph-mon[36715]: Deploying daemon mds.l on smithi183 2024-07-26T10:56:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:28 smithi160 ceph-mon[36715]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:28.555 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 podman[68000]: 2024-07-26 10:56:28.204395362 +0000 UTC m=+0.016829977 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:28.555 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 podman[68000]: 2024-07-26 10:56:28.317474533 +0000 UTC m=+0.129909141 container create 44e1580f9f8c8e46d7c7e73e39f1d57926ec04d03844f54906967e118bcffeb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T10:56:28.556 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 podman[68000]: 2024-07-26 10:56:28.554067448 +0000 UTC m=+0.366502060 container init 44e1580f9f8c8e46d7c7e73e39f1d57926ec04d03844f54906967e118bcffeb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T10:56:28.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 podman[68000]: 2024-07-26 10:56:28.557050088 +0000 UTC m=+0.369484700 container start 44e1580f9f8c8e46d7c7e73e39f1d57926ec04d03844f54906967e118bcffeb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T10:56:28.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 ceph-mds[68019]: -- 172.21.15.183:0/3378084671 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56250e737860 con 0x5625128d9600 2024-07-26T10:56:28.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: starting mds.l at 2024-07-26T10:56:28.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 bash[68000]: 44e1580f9f8c8e46d7c7e73e39f1d57926ec04d03844f54906967e118bcffeb8 2024-07-26T10:56:28.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 10:56:28 smithi183 systemd[1]: Started Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:29.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 -- 172.21.15.144:0/4114873586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c10f330 msgr2=0x7fb12c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:29.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 --2- 172.21.15.144:0/4114873586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c10f330 0x7fb12c10f740 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb114009900 tx=0x7fb11402a410 comp rx=0 tx=0).stop 2024-07-26T10:56:29.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 -- 172.21.15.144:0/4114873586 shutdown_connections 2024-07-26T10:56:29.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 --2- 172.21.15.144:0/4114873586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb12c10fc80 0x7fb12c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 --2- 172.21.15.144:0/4114873586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c10f330 0x7fb12c10f740 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 --2- 172.21.15.144:0/4114873586 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c1099f0 0x7fb12c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 -- 172.21.15.144:0/4114873586 >> 172.21.15.144:0/4114873586 conn(0x7fb12c078ff0 msgr2=0x7fb12c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:29.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 -- 172.21.15.144:0/4114873586 shutdown_connections 2024-07-26T10:56:29.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.172+0000 7fb12b200640 1 -- 172.21.15.144:0/4114873586 wait complete. 2024-07-26T10:56:29.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.173+0000 7fb12b200640 1 Processor -- start 2024-07-26T10:56:29.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.173+0000 7fb12b200640 1 -- start start 2024-07-26T10:56:29.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 0x7fb12c1a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:29.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb12c10f330 0x7fb12c1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:29.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c10fc80 0x7fb12c1a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c119940 con 0x7fb12c1099f0 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c1197c0 con 0x7fb12c10fc80 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c119ac0 con 0x7fb12c10f330 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb129e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 0x7fb12c1a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb129e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 0x7fb12c1a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52326/0 (socket says 172.21.15.144:52326) 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb129e00640 1 -- 172.21.15.144:0/2593580078 learned_addr learned my addr 172.21.15.144:0/2593580078 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb129400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb12c10f330 0x7fb12c1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.174+0000 7fb12a800640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c10fc80 0x7fb12c1a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb129e00640 1 -- 172.21.15.144:0/2593580078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb12c10f330 msgr2=0x7fb12c1a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb129e00640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb12c10f330 0x7fb12c1a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb129e00640 1 -- 172.21.15.144:0/2593580078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c10fc80 msgr2=0x7fb12c1a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb129e00640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c10fc80 0x7fb12c1a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb129e00640 1 -- 172.21.15.144:0/2593580078 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c1a62e0 con 0x7fb12c1099f0 2024-07-26T10:56:29.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb12a800640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c10fc80 0x7fb12c1a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb129e00640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 0x7fb12c1a0370 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00e9f0 tx=0x7fb11c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00cd80 con 0x7fb12c1099f0 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.175+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb11c0108e0 con 0x7fb12c1099f0 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.176+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c018980 con 0x7fb12c1099f0 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.176+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c1a65d0 con 0x7fb12c1099f0 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.176+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c1acd30 con 0x7fb12c1099f0 2024-07-26T10:56:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.177+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c109e00 con 0x7fb12c1099f0 2024-07-26T10:56:29.182 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.179+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb11c010430 con 0x7fb12c1099f0 2024-07-26T10:56:29.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.181+0000 7fb122a00640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb1100b3c20 0x7fb1100b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:29.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.181+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fb11c0e4100 con 0x7fb12c1099f0 2024-07-26T10:56:29.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.181+0000 7fb129400640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb1100b3c20 0x7fb1100b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:29.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.182+0000 7fb129400640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb1100b3c20 0x7fb1100b6130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb114009230 tx=0x7fb114031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:29.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.183+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb11c0ab430 con 0x7fb12c1099f0 2024-07-26T10:56:29.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.421+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb12c1a14a0 con 0x7fb12c1099f0 2024-07-26T10:56:29.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.422+0000 7fb122a00640 1 -- 172.21.15.144:0/2593580078 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7fb11c0af980 con 0x7fb12c1099f0 2024-07-26T10:56:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:29.426 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":2,"btime":"2024-07-26T10:56:28:595196+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-26T10:56:29.426 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 2 2024-07-26T10:56:29.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb1100b3c20 msgr2=0x7fb1100b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:29.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb1100b3c20 0x7fb1100b6130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb114009230 tx=0x7fb114031400 comp rx=0 tx=0).stop 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 msgr2=0x7fb12c1a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 0x7fb12c1a0370 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00e9f0 tx=0x7fb11c00eeb0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 shutdown_connections 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb1100b3c20 0x7fb1100b6130 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb12c10fc80 0x7fb12c1a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb12c10f330 0x7fb12c1a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.425+0000 7fb12b200640 1 --2- 172.21.15.144:0/2593580078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb12c1099f0 0x7fb12c1a0370 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.426+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 >> 172.21.15.144:0/2593580078 conn(0x7fb12c078ff0 msgr2=0x7fb12c108010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.426+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 shutdown_connections 2024-07-26T10:56:29.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:29.426+0000 7fb12b200640 1 -- 172.21.15.144:0/2593580078 wait complete. 2024-07-26T10:56:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T10:56:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:boot 2024-07-26T10:56:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: fsmap 1 up:standby 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: Deploying daemon mds.k on smithi160 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:29.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:29 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2593580078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:boot 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: fsmap 1 up:standby 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: Deploying daemon mds.k on smithi160 2024-07-26T10:56:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:29.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:29 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2593580078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T10:56:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:boot 2024-07-26T10:56:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: fsmap 1 up:standby 2024-07-26T10:56:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T10:56:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: Deploying daemon mds.k on smithi160 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:29.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:29 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2593580078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:30.809 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:30 smithi160 systemd[1]: Starting Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:31.157 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:30 smithi160 podman[66927]: 2024-07-26 10:56:30.790568265 +0000 UTC m=+0.017343734 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:31.157 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:30 smithi160 podman[66927]: 2024-07-26 10:56:30.910584901 +0000 UTC m=+0.137360364 container create b5a2932745cb17c2f235dbf175655c5e0b6b1ec77a98146225f3454366b42571 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T10:56:31.157 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:31 smithi160 podman[66927]: 2024-07-26 10:56:31.15539291 +0000 UTC m=+0.382168374 container init b5a2932745cb17c2f235dbf175655c5e0b6b1ec77a98146225f3454366b42571 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T10:56:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:31 smithi144 ceph-mon[32129]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:31 smithi160 ceph-mon[36715]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:31.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:31 smithi160 podman[66927]: 2024-07-26 10:56:31.158666135 +0000 UTC m=+0.385441589 container start b5a2932745cb17c2f235dbf175655c5e0b6b1ec77a98146225f3454366b42571 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-07-26T10:56:31.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:31 smithi160 ceph-mds[66946]: -- 172.21.15.160:0/2610458712 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564152db7860 con 0x564156f59600 2024-07-26T10:56:31.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:31 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: starting mds.k at 2024-07-26T10:56:31.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:31 smithi160 bash[66927]: b5a2932745cb17c2f235dbf175655c5e0b6b1ec77a98146225f3454366b42571 2024-07-26T10:56:31.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 10:56:31 smithi160 systemd[1]: Started Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:31 smithi183 ceph-mon[37273]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:32.227 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:56:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: Deploying daemon mds.j on smithi144 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:boot 2024-07-26T10:56:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:32 smithi144 ceph-mon[32129]: fsmap 2 up:standby 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:32.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: Deploying daemon mds.j on smithi144 2024-07-26T10:56:32.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T10:56:32.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:boot 2024-07-26T10:56:32.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:32 smithi160 ceph-mon[36715]: fsmap 2 up:standby 2024-07-26T10:56:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: Deploying daemon mds.j on smithi144 2024-07-26T10:56:32.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T10:56:32.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:boot 2024-07-26T10:56:32.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:32 smithi183 ceph-mon[37273]: fsmap 2 up:standby 2024-07-26T10:56:33.425 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:33 smithi144 ceph-mon[32129]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:33.502 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:33 smithi183 ceph-mon[37273]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:33 smithi160 ceph-mon[36715]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:34.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:34 smithi144 systemd[1]: Starting Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:35.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 podman[76932]: 2024-07-26 10:56:35.060635925 +0000 UTC m=+0.050396105 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:35.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 podman[76932]: 2024-07-26 10:56:35.165130285 +0000 UTC m=+0.154890487 container create f17952baa7f056250f0c392b5d038f75160695505bfa29a9ea979e6686c53194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T10:56:35.662 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 podman[76932]: 2024-07-26 10:56:35.419679486 +0000 UTC m=+0.409439678 container init f17952baa7f056250f0c392b5d038f75160695505bfa29a9ea979e6686c53194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T10:56:35.663 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 podman[76932]: 2024-07-26 10:56:35.429691534 +0000 UTC m=+0.419451722 container start f17952baa7f056250f0c392b5d038f75160695505bfa29a9ea979e6686c53194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T10:56:35.663 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 bash[76932]: f17952baa7f056250f0c392b5d038f75160695505bfa29a9ea979e6686c53194 2024-07-26T10:56:35.663 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 systemd[1]: Started Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:35.663 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 ceph-mds[76969]: -- 172.21.15.144:0/2744772073 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565216b4bc20 con 0x56521aceda80 2024-07-26T10:56:35.663 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 10:56:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: starting mds.j at 2024-07-26T10:56:36.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 -- 172.21.15.144:0/741577512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 msgr2=0x7fdd80075570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:36.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 --2- 172.21.15.144:0/741577512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd80075570 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009900 tx=0x7fdd7002a3f0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 -- 172.21.15.144:0/741577512 shutdown_connections 2024-07-26T10:56:36.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 --2- 172.21.15.144:0/741577512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd8010fbc0 0x7fdd80111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 --2- 172.21.15.144:0/741577512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 0x7fdd8010f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 --2- 172.21.15.144:0/741577512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd80075570 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 -- 172.21.15.144:0/741577512 >> 172.21.15.144:0/741577512 conn(0x7fdd800ff030 msgr2=0x7fdd80101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:36.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 -- 172.21.15.144:0/741577512 shutdown_connections 2024-07-26T10:56:36.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.190+0000 7fdd86e00640 1 -- 172.21.15.144:0/741577512 wait complete. 2024-07-26T10:56:36.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.191+0000 7fdd86e00640 1 Processor -- start 2024-07-26T10:56:36.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.191+0000 7fdd86e00640 1 -- start start 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd86e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd8019c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd86e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 0x7fdd8019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd86e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd8010fbc0 0x7fdd801a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd86e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fdd80115730 con 0x7fdd80075160 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd86e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801155b0 con 0x7fdd8010fbc0 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd86e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801158b0 con 0x7fdd80075ab0 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd7fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd8019c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd7fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd8019c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54318/0 (socket says 172.21.15.144:54318) 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd7fe00640 1 -- 172.21.15.144:0/2285957553 learned_addr learned my addr 172.21.15.144:0/2285957553 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd77e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 0x7fdd8019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.192+0000 7fdd84e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd8010fbc0 0x7fdd801a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd77e00640 1 -- 172.21.15.144:0/2285957553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd8010fbc0 msgr2=0x7fdd801a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd77e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd8010fbc0 0x7fdd801a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd77e00640 1 -- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 msgr2=0x7fdd8019c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd77e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd8019c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd77e00640 1 -- 172.21.15.144:0/2285957553 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd801a2070 con 0x7fdd80075ab0 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd77e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 0x7fdd8019c6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdd8006c3a0 tx=0x7fdd6800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:36.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.193+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800cc40 con 0x7fdd80075ab0 2024-07-26T10:56:36.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.194+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd6800cde0 con 0x7fdd80075ab0 2024-07-26T10:56:36.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.194+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd801a2360 con 0x7fdd80075ab0 2024-07-26T10:56:36.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.194+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68004a00 con 0x7fdd80075ab0 2024-07-26T10:56:36.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.194+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd801a8ac0 con 0x7fdd80075ab0 2024-07-26T10:56:36.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.195+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd80075570 con 0x7fdd80075ab0 2024-07-26T10:56:36.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.197+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fdd68004ba0 con 0x7fdd80075ab0 2024-07-26T10:56:36.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.199+0000 7fdd7d600640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdd640b3c20 0x7fdd640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:36.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.199+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fdd680e3940 con 0x7fdd80075ab0 2024-07-26T10:56:36.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.199+0000 7fdd7fe00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdd640b3c20 0x7fdd640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:36.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.200+0000 7fdd7fe00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdd640b3c20 0x7fdd640b6130 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd70031440 tx=0x7fdd700058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:36.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.200+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdd680ab2c0 con 0x7fdd80075ab0 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:36.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:36.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.437 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:36.437 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:36.437 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:36.437 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:36.437 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:36.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.472+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd8019d230 con 0x7fdd80075ab0 2024-07-26T10:56:36.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.473+0000 7fdd7d600640 1 -- 172.21.15.144:0/2285957553 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3094 (secure 0 0 0) 0x7fdd680aabc0 con 0x7fdd80075ab0 2024-07-26T10:56:36.476 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:36.476 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":4,"btime":"2024-07-26T10:56:35:991958+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-07-26T10:56:36.476 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 4 2024-07-26T10:56:36.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdd640b3c20 msgr2=0x7fdd640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdd640b3c20 0x7fdd640b6130 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd70031440 tx=0x7fdd700058d0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 msgr2=0x7fdd8019c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 0x7fdd8019c6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdd8006c3a0 tx=0x7fdd6800ee50 comp rx=0 tx=0).stop 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 shutdown_connections 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdd640b3c20 0x7fdd640b6130 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd8010fbc0 0x7fdd801a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd80075ab0 0x7fdd8019c6a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 --2- 172.21.15.144:0/2285957553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd80075160 0x7fdd8019c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.476+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 >> 172.21.15.144:0/2285957553 conn(0x7fdd800ff030 msgr2=0x7fdd80101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.477+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 shutdown_connections 2024-07-26T10:56:36.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:36.477+0000 7fdd86e00640 1 -- 172.21.15.144:0/2285957553 wait complete. 2024-07-26T10:56:37.127 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:37 smithi144 ceph-mon[32129]: Deploying daemon mds.d on smithi144 2024-07-26T10:56:37.127 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T10:56:37.127 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:37 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:boot 2024-07-26T10:56:37.127 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:37 smithi144 ceph-mon[32129]: fsmap 3 up:standby 2024-07-26T10:56:37.128 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:37 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2285957553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:37 smithi160 ceph-mon[36715]: Deploying daemon mds.d on smithi144 2024-07-26T10:56:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T10:56:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:37 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:boot 2024-07-26T10:56:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:37 smithi160 ceph-mon[36715]: fsmap 3 up:standby 2024-07-26T10:56:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:37 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2285957553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:37 smithi183 ceph-mon[37273]: Deploying daemon mds.d on smithi144 2024-07-26T10:56:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T10:56:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:37 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:boot 2024-07-26T10:56:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:37 smithi183 ceph-mon[37273]: fsmap 3 up:standby 2024-07-26T10:56:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:37 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2285957553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:38 smithi160 ceph-mon[36715]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:38.385 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:38 smithi144 ceph-mon[32129]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:38 smithi183 ceph-mon[37273]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:39.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:56:39.875 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 podman[77319]: 2024-07-26 10:56:39.584541455 +0000 UTC m=+0.176160533 container create 6f685a8b539069587345dda2a9e5e78669dfe7aa1a7ed28515ac2cc1a5fc3d4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T10:56:39.875 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 podman[77319]: 2024-07-26 10:56:39.872114068 +0000 UTC m=+0.463733181 container init 6f685a8b539069587345dda2a9e5e78669dfe7aa1a7ed28515ac2cc1a5fc3d4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T10:56:40.081 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:40 smithi144 ceph-mon[32129]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:40.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 podman[77319]: 2024-07-26 10:56:39.883058257 +0000 UTC m=+0.474677337 container start 6f685a8b539069587345dda2a9e5e78669dfe7aa1a7ed28515ac2cc1a5fc3d4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T10:56:40.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 bash[77319]: 6f685a8b539069587345dda2a9e5e78669dfe7aa1a7ed28515ac2cc1a5fc3d4d 2024-07-26T10:56:40.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 systemd[1]: Started Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:40.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 ceph-mds[77362]: -- 172.21.15.144:0/112818193 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ca199d860 con 0x564ca5b3f600 2024-07-26T10:56:40.221 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 10:56:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: starting mds.d at 2024-07-26T10:56:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:40 smithi160 ceph-mon[36715]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:40 smithi183 ceph-mon[37273]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:boot 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: fsmap 4 up:standby 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:41.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:41.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:41.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:boot 2024-07-26T10:56:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: fsmap 4 up:standby 2024-07-26T10:56:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T10:56:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:41.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:41.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:41.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:boot 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: fsmap 4 up:standby 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:41.389 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 -- 172.21.15.144:0/4090086174 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c1042b0 msgr2=0x7f906c1046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 --2- 172.21.15.144:0/4090086174 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c1042b0 0x7f906c1046c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f905c009900 tx=0x7f905c02a410 comp rx=0 tx=0).stop 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 -- 172.21.15.144:0/4090086174 shutdown_connections 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 --2- 172.21.15.144:0/4090086174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c10d860 0x7f906c10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 --2- 172.21.15.144:0/4090086174 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f906c104ce0 0x7f906c10d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 --2- 172.21.15.144:0/4090086174 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c1042b0 0x7f906c1046c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 -- 172.21.15.144:0/4090086174 >> 172.21.15.144:0/4090086174 conn(0x7f906c079160 msgr2=0x7f906c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.293+0000 7f9073000640 1 -- 172.21.15.144:0/4090086174 shutdown_connections 2024-07-26T10:56:42.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.294+0000 7f9073000640 1 -- 172.21.15.144:0/4090086174 wait complete. 2024-07-26T10:56:42.297 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:42 smithi144 ceph-mon[32129]: Deploying daemon mds.e on smithi160 2024-07-26T10:56:42.297 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:42 smithi144 ceph-mon[32129]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:42.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.295+0000 7f9073000640 1 Processor -- start 2024-07-26T10:56:42.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.295+0000 7f9073000640 1 -- start start 2024-07-26T10:56:42.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9073000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c1042b0 0x7f906c19c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:42.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9073000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f906c104ce0 0x7f906c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:42.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9073000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c10d860 0x7f906c1a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:42.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9073000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f906c1134e0 con 0x7f906c1042b0 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9073000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f906c113360 con 0x7f906c104ce0 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9073000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f906c113660 con 0x7f906c10d860 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f906be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c1042b0 0x7f906c19c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f906b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f906c104ce0 0x7f906c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f9071000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c10d860 0x7f906c1a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f906be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c1042b0 0x7f906c19c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49830/0 (socket says 172.21.15.144:49830) 2024-07-26T10:56:42.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.296+0000 7f906be00640 1 -- 172.21.15.144:0/474670293 learned_addr learned my addr 172.21.15.144:0/474670293 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9071000640 1 -- 172.21.15.144:0/474670293 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f906c104ce0 msgr2=0x7f906c19c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9071000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f906c104ce0 0x7f906c19c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9071000640 1 -- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c1042b0 msgr2=0x7f906c19c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9071000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c1042b0 0x7f906c19c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9071000640 1 -- 172.21.15.144:0/474670293 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c1a2040 con 0x7f906c10d860 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9071000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c10d860 0x7f906c1a1910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9064009fd0 tx=0x7f906400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906400ce20 con 0x7f906c10d860 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9064005020 con 0x7f906c10d860 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906c1a2330 con 0x7f906c10d860 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064010690 con 0x7f906c10d860 2024-07-26T10:56:42.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.297+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f906c1a8a90 con 0x7f906c10d860 2024-07-26T10:56:42.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.299+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f906c1046c0 con 0x7f906c10d860 2024-07-26T10:56:42.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.301+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f9064004b20 con 0x7f906c10d860 2024-07-26T10:56:42.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.303+0000 7f9068c00640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f90500b3c20 0x7f90500b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:42.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.303+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f90640e3f40 con 0x7f906c10d860 2024-07-26T10:56:42.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.303+0000 7f906be00640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f90500b3c20 0x7f90500b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:42.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.304+0000 7f906be00640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f90500b3c20 0x7f90500b6130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f905c000c00 tx=0x7f905c005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:42.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.306+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f90640ab8c0 con 0x7f906c10d860 2024-07-26T10:56:42.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:42 smithi160 ceph-mon[36715]: Deploying daemon mds.e on smithi160 2024-07-26T10:56:42.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:42 smithi160 ceph-mon[36715]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:42.310 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 systemd[1]: Starting Ceph mds.e for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:42 smithi183 ceph-mon[37273]: Deploying daemon mds.e on smithi160 2024-07-26T10:56:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:42 smithi183 ceph-mon[37273]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.538+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f906c1a8d70 con 0x7f906c10d860 2024-07-26T10:56:42.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.539+0000 7f9068c00640 1 -- 172.21.15.144:0/474670293 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3922 (secure 0 0 0) 0x7f90640ab1c0 con 0x7f906c10d860 2024-07-26T10:56:42.541 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:42.541 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":5,"btime":"2024-07-26T10:56:40:022409+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-07-26T10:56:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 5 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f90500b3c20 msgr2=0x7f90500b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f90500b3c20 0x7f90500b6130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f905c000c00 tx=0x7f905c005a20 comp rx=0 tx=0).stop 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c10d860 msgr2=0x7f906c1a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c10d860 0x7f906c1a1910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9064009fd0 tx=0x7f906400ef10 comp rx=0 tx=0).stop 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 shutdown_connections 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f90500b3c20 0x7f90500b6130 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f906c10d860 0x7f906c1a1910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f906c104ce0 0x7f906c19c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 --2- 172.21.15.144:0/474670293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c1042b0 0x7f906c19c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:42.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 >> 172.21.15.144:0/474670293 conn(0x7f906c079160 msgr2=0x7f906c10c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:42.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 shutdown_connections 2024-07-26T10:56:42.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:42.542+0000 7f9073000640 1 -- 172.21.15.144:0/474670293 wait complete. 2024-07-26T10:56:42.679 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 podman[67244]: 2024-07-26 10:56:42.275260445 +0000 UTC m=+0.017255859 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:42.679 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 podman[67244]: 2024-07-26 10:56:42.390040426 +0000 UTC m=+0.132035872 container create dadfa41b89a319d630acd5d4e20a1305f1cf043fe10454812a8e40b2ce9696c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T10:56:42.679 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 podman[67244]: 2024-07-26 10:56:42.678017667 +0000 UTC m=+0.420013075 container init dadfa41b89a319d630acd5d4e20a1305f1cf043fe10454812a8e40b2ce9696c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T10:56:43.021 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 podman[67244]: 2024-07-26 10:56:42.680848662 +0000 UTC m=+0.422844078 container start dadfa41b89a319d630acd5d4e20a1305f1cf043fe10454812a8e40b2ce9696c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T10:56:43.021 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 ceph-mds[67263]: -- 172.21.15.160:0/1827296603 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b5ce3f860 con 0x564b60fe1600 2024-07-26T10:56:43.021 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-e[67259]: starting mds.e at 2024-07-26T10:56:43.021 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 bash[67244]: dadfa41b89a319d630acd5d4e20a1305f1cf043fe10454812a8e40b2ce9696c6 2024-07-26T10:56:43.021 INFO:journalctl@ceph.mds.e.smithi160.stdout:Jul 26 10:56:42 smithi160 systemd[1]: Started Ceph mds.e for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/474670293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:43.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:43.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:43.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/474670293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/474670293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:43.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:44 smithi160 ceph-mon[36715]: Deploying daemon mds.f on smithi183 2024-07-26T10:56:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:44 smithi160 ceph-mon[36715]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T10:56:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:44 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6834/405635522,v1:172.21.15.160:6835/405635522] up:boot 2024-07-26T10:56:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:44 smithi160 ceph-mon[36715]: fsmap 5 up:standby 2024-07-26T10:56:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:44 smithi183 ceph-mon[37273]: Deploying daemon mds.f on smithi183 2024-07-26T10:56:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:44 smithi183 ceph-mon[37273]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T10:56:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:44 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6834/405635522,v1:172.21.15.160:6835/405635522] up:boot 2024-07-26T10:56:44.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:44 smithi183 ceph-mon[37273]: fsmap 5 up:standby 2024-07-26T10:56:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:44 smithi144 ceph-mon[32129]: Deploying daemon mds.f on smithi183 2024-07-26T10:56:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:44 smithi144 ceph-mon[32129]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T10:56:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:44 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6834/405635522,v1:172.21.15.160:6835/405635522] up:boot 2024-07-26T10:56:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:44 smithi144 ceph-mon[32129]: fsmap 5 up:standby 2024-07-26T10:56:45.098 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:44 smithi183 systemd[1]: Starting Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:45.335 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:56:45.361 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 podman[68314]: 2024-07-26 10:56:44.999057613 +0000 UTC m=+0.017376282 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:45.361 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 podman[68314]: 2024-07-26 10:56:45.104934061 +0000 UTC m=+0.123252767 container create 1f1bd016736e6340599f54ac8fd835123bf13f4660757e184c3978cadabe1c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-07-26T10:56:45.361 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 podman[68314]: 2024-07-26 10:56:45.324473657 +0000 UTC m=+0.342792329 container init 1f1bd016736e6340599f54ac8fd835123bf13f4660757e184c3978cadabe1c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-07-26T10:56:45.361 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 podman[68314]: 2024-07-26 10:56:45.327237026 +0000 UTC m=+0.345555689 container start 1f1bd016736e6340599f54ac8fd835123bf13f4660757e184c3978cadabe1c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-07-26T10:56:45.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 ceph-mds[68334]: -- 172.21.15.183:0/300019510 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf483fd860 con 0x55bf4c59f180 2024-07-26T10:56:45.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: starting mds.f at 2024-07-26T10:56:45.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 bash[68314]: 1f1bd016736e6340599f54ac8fd835123bf13f4660757e184c3978cadabe1c2b 2024-07-26T10:56:45.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 10:56:45 smithi183 systemd[1]: Started Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:45.801 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:46.207 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:47.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:47 smithi183 ceph-mon[37273]: Deploying daemon mds.c on smithi183 2024-07-26T10:56:47.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T10:56:47.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:47 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:boot 2024-07-26T10:56:47.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:47 smithi183 ceph-mon[37273]: fsmap 6 up:standby 2024-07-26T10:56:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:47 smithi144 ceph-mon[32129]: Deploying daemon mds.c on smithi183 2024-07-26T10:56:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T10:56:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:47 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:boot 2024-07-26T10:56:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:47 smithi144 ceph-mon[32129]: fsmap 6 up:standby 2024-07-26T10:56:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:47 smithi160 ceph-mon[36715]: Deploying daemon mds.c on smithi183 2024-07-26T10:56:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T10:56:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:47 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:boot 2024-07-26T10:56:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:47 smithi160 ceph-mon[36715]: fsmap 6 up:standby 2024-07-26T10:56:47.589 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 systemd[1]: Starting Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:47.841 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 podman[68619]: 2024-07-26 10:56:47.490267526 +0000 UTC m=+0.016642935 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:47.842 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 podman[68619]: 2024-07-26 10:56:47.620183196 +0000 UTC m=+0.146558651 container create 34383d6ad4a10d76bae1a62e2e7d74b3cae237812d13822ddf43c6846341c8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T10:56:47.842 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 podman[68619]: 2024-07-26 10:56:47.840219687 +0000 UTC m=+0.366595110 container init 34383d6ad4a10d76bae1a62e2e7d74b3cae237812d13822ddf43c6846341c8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3289134379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d01040c0 msgr2=0x7fc9d01044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3289134379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d01040c0 0x7fc9d01044d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009860 tx=0x7fc9c002a340 comp rx=0 tx=0).stop 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3289134379 shutdown_connections 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3289134379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9d010d670 0x7fc9d010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3289134379 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d0104af0 0x7fc9d010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3289134379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d01040c0 0x7fc9d01044d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.078+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3289134379 >> 172.21.15.144:0/3289134379 conn(0x7fc9d00fdcf0 msgr2=0x7fc9d0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.079+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3289134379 shutdown_connections 2024-07-26T10:56:48.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.079+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3289134379 wait complete. 2024-07-26T10:56:48.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.080+0000 7fc9d4c00640 1 Processor -- start 2024-07-26T10:56:48.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.080+0000 7fc9d4c00640 1 -- start start 2024-07-26T10:56:48.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9d4c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 0x7fc9d0101fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:48.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9d4c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9d0104af0 0x7fc9d0102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:48.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9d4c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d010d670 0x7fc9d0102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:48.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9d4c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01132a0 con 0x7fc9d0104af0 2024-07-26T10:56:48.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9d4c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0113120 con 0x7fc9d01040c0 2024-07-26T10:56:48.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9d4c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0113420 con 0x7fc9d010d670 2024-07-26T10:56:48.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 0x7fc9d0101fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:48.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9d0104af0 0x7fc9d0102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:48.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 0x7fc9d0101fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:45578/0 (socket says 172.21.15.144:45578) 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 -- 172.21.15.144:0/3686759489 learned_addr learned my addr 172.21.15.144:0/3686759489 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cfe00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d010d670 0x7fc9d0102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 -- 172.21.15.144:0/3686759489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d010d670 msgr2=0x7fc9d0102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d010d670 0x7fc9d0102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 -- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9d0104af0 msgr2=0x7fc9d0102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9d0104af0 0x7fc9d0102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.081+0000 7fc9cf400640 1 -- 172.21.15.144:0/3686759489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d01a7f90 con 0x7fc9d01040c0 2024-07-26T10:56:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.082+0000 7fc9cf400640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 0x7fc9d0101fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00303f0 tx=0x7fc9c0032eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:48.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.086+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0040020 con 0x7fc9d01040c0 2024-07-26T10:56:48.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.086+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9c0030450 con 0x7fc9d01040c0 2024-07-26T10:56:48.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.086+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d01a8220 con 0x7fc9d01040c0 2024-07-26T10:56:48.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.087+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0037040 con 0x7fc9d01040c0 2024-07-26T10:56:48.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.087+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d01a8750 con 0x7fc9d01040c0 2024-07-26T10:56:48.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.088+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d01044d0 con 0x7fc9d01040c0 2024-07-26T10:56:48.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.093+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc9c0043020 con 0x7fc9d01040c0 2024-07-26T10:56:48.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.095+0000 7fc9c7e00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9b40b3ca0 0x7fc9b40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:48.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.095+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fc9c0074080 con 0x7fc9d01040c0 2024-07-26T10:56:48.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.095+0000 7fc9cea00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9b40b3ca0 0x7fc9b40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:48.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.096+0000 7fc9cea00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9b40b3ca0 0x7fc9b40b61b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc9d010fc20 tx=0x7fc9b8009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:48.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.100+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc9c004a020 con 0x7fc9d01040c0 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:48.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 podman[68619]: 2024-07-26 10:56:47.843315774 +0000 UTC m=+0.369691180 container start 34383d6ad4a10d76bae1a62e2e7d74b3cae237812d13822ddf43c6846341c8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T10:56:48.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 ceph-mds[68637]: -- 172.21.15.183:0/2077854764 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56297a115860 con 0x56297e2b7180 2024-07-26T10:56:48.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: starting mds.c at 2024-07-26T10:56:48.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 bash[68619]: 34383d6ad4a10d76bae1a62e2e7d74b3cae237812d13822ddf43c6846341c8fb 2024-07-26T10:56:48.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 10:56:47 smithi183 systemd[1]: Started Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:48.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.338+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9d01032d0 con 0x7fc9d01040c0 2024-07-26T10:56:48.340 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:48.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:48.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.339+0000 7fc9c7e00640 1 -- 172.21.15.144:0/3686759489 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6397 (secure 0 0 0) 0x7fc9d01032d0 con 0x7fc9d01040c0 2024-07-26T10:56:48.342 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:48.342 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":8,"btime":"2024-07-26T10:56:48:081624+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24677,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-07-26T10:56:48.342 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 8 2024-07-26T10:56:48.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9b40b3ca0 msgr2=0x7fc9b40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:48.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9b40b3ca0 0x7fc9b40b61b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc9d010fc20 tx=0x7fc9b8009260 comp rx=0 tx=0).stop 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 msgr2=0x7fc9d0101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 0x7fc9d0101fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00303f0 tx=0x7fc9c0032eb0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 shutdown_connections 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9b40b3ca0 0x7fc9b40b61b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9d010d670 0x7fc9d0102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9d0104af0 0x7fc9d0102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 --2- 172.21.15.144:0/3686759489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9d01040c0 0x7fc9d0101fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 >> 172.21.15.144:0/3686759489 conn(0x7fc9d00fdcf0 msgr2=0x7fc9d0107220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.342+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 shutdown_connections 2024-07-26T10:56:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:48.343+0000 7fc9d4c00640 1 -- 172.21.15.144:0/3686759489 wait complete. 2024-07-26T10:56:48.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:48.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.482 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:48.482 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:48.482 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:48.482 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:48.482 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:49 smithi144 ceph-mon[32129]: Deploying daemon mds.h on smithi160 2024-07-26T10:56:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:boot 2024-07-26T10:56:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:49 smithi144 ceph-mon[32129]: fsmap 7 up:standby 2024-07-26T10:56:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T10:56:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:49 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3686759489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:49 smithi183 ceph-mon[37273]: Deploying daemon mds.h on smithi160 2024-07-26T10:56:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:boot 2024-07-26T10:56:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:49 smithi183 ceph-mon[37273]: fsmap 7 up:standby 2024-07-26T10:56:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T10:56:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:49 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3686759489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:49.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:49 smithi160 ceph-mon[36715]: Deploying daemon mds.h on smithi160 2024-07-26T10:56:49.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:boot 2024-07-26T10:56:49.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:49 smithi160 ceph-mon[36715]: fsmap 7 up:standby 2024-07-26T10:56:49.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T10:56:49.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:49 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3686759489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:50.059 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:49 smithi160 systemd[1]: Starting Ceph mds.h for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:50 smithi183 ceph-mon[37273]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:50 smithi144 ceph-mon[32129]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:50.490 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 podman[67547]: 2024-07-26 10:56:50.086791538 +0000 UTC m=+0.017417926 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:50.490 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 podman[67547]: 2024-07-26 10:56:50.202830925 +0000 UTC m=+0.133457310 container create 300270b2af953dff6ab40f0d9c18777e2d2b61146c390542f3dd0d8ad217ccfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-26T10:56:50.490 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:50 smithi160 ceph-mon[36715]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:50.751 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 podman[67547]: 2024-07-26 10:56:50.489044381 +0000 UTC m=+0.419670780 container init 300270b2af953dff6ab40f0d9c18777e2d2b61146c390542f3dd0d8ad217ccfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-07-26T10:56:50.751 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 podman[67547]: 2024-07-26 10:56:50.492041798 +0000 UTC m=+0.422668191 container start 300270b2af953dff6ab40f0d9c18777e2d2b61146c390542f3dd0d8ad217ccfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-07-26T10:56:50.752 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 ceph-mds[67567]: -- 172.21.15.160:0/1571750536 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cae335b860 con 0x55cae74fd180 2024-07-26T10:56:50.752 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: starting mds.h at 2024-07-26T10:56:50.752 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 bash[67547]: 300270b2af953dff6ab40f0d9c18777e2d2b61146c390542f3dd0d8ad217ccfe 2024-07-26T10:56:50.752 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 10:56:50 smithi160 systemd[1]: Started Ceph mds.h for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:51.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: Deploying daemon mds.i on smithi183 2024-07-26T10:56:51.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:51.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T10:56:51.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:boot 2024-07-26T10:56:51.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:51 smithi144 ceph-mon[32129]: fsmap 8 up:standby 2024-07-26T10:56:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: Deploying daemon mds.i on smithi183 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:boot 2024-07-26T10:56:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:51 smithi160 ceph-mon[36715]: fsmap 8 up:standby 2024-07-26T10:56:51.837 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: Deploying daemon mds.i on smithi183 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T10:56:51.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:boot 2024-07-26T10:56:51.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:51 smithi183 ceph-mon[37273]: fsmap 8 up:standby 2024-07-26T10:56:53.135 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 podman[68919]: 2024-07-26 10:56:53.026221478 +0000 UTC m=+0.127599804 container create 73dd7ee98bd64c9841f78a45039e53411b63ccfca0d38a3194c136c8c6c41815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-07-26T10:56:53.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 podman[68919]: 2024-07-26 10:56:53.270609212 +0000 UTC m=+0.371987505 container init 73dd7ee98bd64c9841f78a45039e53411b63ccfca0d38a3194c136c8c6c41815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T10:56:53.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 podman[68919]: 2024-07-26 10:56:53.273508317 +0000 UTC m=+0.374886598 container start 73dd7ee98bd64c9841f78a45039e53411b63ccfca0d38a3194c136c8c6c41815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T10:56:53.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 ceph-mds[68938]: -- 172.21.15.183:0/1708937068 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559ac5567c20 con 0x559ac9709a80 2024-07-26T10:56:53.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: starting mds.i at 2024-07-26T10:56:53.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 bash[68919]: 73dd7ee98bd64c9841f78a45039e53411b63ccfca0d38a3194c136c8c6c41815 2024-07-26T10:56:53.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 10:56:53 smithi183 systemd[1]: Started Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.025+0000 7fa104e00640 1 -- 172.21.15.144:0/1397954852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100102930 msgr2=0x7fa100106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.025+0000 7fa104e00640 1 --2- 172.21.15.144:0/1397954852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100102930 0x7fa100106e60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009930 tx=0x7fa0ec02a420 comp rx=0 tx=0).stop 2024-07-26T10:56:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 -- 172.21.15.144:0/1397954852 shutdown_connections 2024-07-26T10:56:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 --2- 172.21.15.144:0/1397954852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa100107420 0x7fa100107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 --2- 172.21.15.144:0/1397954852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100102930 0x7fa100106e60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 --2- 172.21.15.144:0/1397954852 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100101fe0 0x7fa1001023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 -- 172.21.15.144:0/1397954852 >> 172.21.15.144:0/1397954852 conn(0x7fa1000fdcf0 msgr2=0x7fa100100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 -- 172.21.15.144:0/1397954852 shutdown_connections 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.026+0000 7fa104e00640 1 -- 172.21.15.144:0/1397954852 wait complete. 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.027+0000 7fa104e00640 1 Processor -- start 2024-07-26T10:56:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.027+0000 7fa104e00640 1 -- start start 2024-07-26T10:56:54.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa104e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa100101fe0 0x7fa1001a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:54.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa104e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 0x7fa1001a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:54.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa104e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100107420 0x7fa1001a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:54.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa104e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001198d0 con 0x7fa100107420 2024-07-26T10:56:54.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa104e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa100119750 con 0x7fa100102930 2024-07-26T10:56:54.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa104e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa100119a50 con 0x7fa100101fe0 2024-07-26T10:56:54.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa0fea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 0x7fa1001a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:54.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa0ffe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100107420 0x7fa1001a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:54.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa0fea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 0x7fa1001a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:43304/0 (socket says 172.21.15.144:43304) 2024-07-26T10:56:54.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa0fea00640 1 -- 172.21.15.144:0/3873022779 learned_addr learned my addr 172.21.15.144:0/3873022779 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:56:54.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.028+0000 7fa0ff400640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa100101fe0 0x7fa1001a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.029+0000 7fa0fea00640 1 -- 172.21.15.144:0/3873022779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa100101fe0 msgr2=0x7fa1001a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.029+0000 7fa0fea00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa100101fe0 0x7fa1001a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.029+0000 7fa0fea00640 1 -- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100107420 msgr2=0x7fa1001a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.029+0000 7fa0fea00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100107420 0x7fa1001a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.029+0000 7fa0fea00640 1 -- 172.21.15.144:0/3873022779 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1001a62d0 con 0x7fa100102930 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.029+0000 7fa0fea00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 0x7fa1001a0900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec031440 tx=0x7fa0ec003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.030+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec03f070 con 0x7fa100102930 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.030+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1001a6560 con 0x7fa100102930 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.030+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0ec031b50 con 0x7fa100102930 2024-07-26T10:56:54.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.030+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec035030 con 0x7fa100102930 2024-07-26T10:56:54.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.030+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1001acd20 con 0x7fa100102930 2024-07-26T10:56:54.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.031+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa100106e60 con 0x7fa100102930 2024-07-26T10:56:54.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.033+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fa0ec042070 con 0x7fa100102930 2024-07-26T10:56:54.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.035+0000 7fa0f7e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0e40b3c20 0x7fa0e40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:54.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.035+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fa0ec103e70 con 0x7fa100102930 2024-07-26T10:56:54.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.035+0000 7fa0ff400640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0e40b3c20 0x7fa0e40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:54.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.036+0000 7fa0ff400640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0e40b3c20 0x7fa0e40b6130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009bd0 tx=0x7fa0f0005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:54.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.039+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa0ec0cb7f0 con 0x7fa100102930 2024-07-26T10:56:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:54.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.288+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1001acf20 con 0x7fa100102930 2024-07-26T10:56:54.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.289+0000 7fa0f7e00640 1 -- 172.21.15.144:0/3873022779 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8055 (secure 0 0 0) 0x7fa0ec0cb0f0 con 0x7fa100102930 2024-07-26T10:56:54.291 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:56:54.291 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":10,"btime":"2024-07-26T10:56:53:998361+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24677,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24691,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24701,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-07-26T10:56:54.292 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 10 2024-07-26T10:56:54.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0e40b3c20 msgr2=0x7fa0e40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0e40b3c20 0x7fa0e40b6130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009bd0 tx=0x7fa0f0005f90 comp rx=0 tx=0).stop 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 msgr2=0x7fa1001a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 0x7fa1001a0900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec031440 tx=0x7fa0ec003b90 comp rx=0 tx=0).stop 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 shutdown_connections 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0e40b3c20 0x7fa0e40b6130 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa100107420 0x7fa1001a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa100102930 0x7fa1001a0900 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 --2- 172.21.15.144:0/3873022779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa100101fe0 0x7fa1001a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.292+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 >> 172.21.15.144:0/3873022779 conn(0x7fa1000fdcf0 msgr2=0x7fa100104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.293+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 shutdown_connections 2024-07-26T10:56:54.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:56:54.293+0000 7fa104e00640 1 -- 172.21.15.144:0/3873022779 wait complete. 2024-07-26T10:56:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:55.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:55 smithi144 ceph-mon[32129]: Deploying daemon mds.g on smithi144 2024-07-26T10:56:55.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T10:56:55.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:55 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:boot 2024-07-26T10:56:55.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:55 smithi144 ceph-mon[32129]: fsmap 9 up:standby 2024-07-26T10:56:55.309 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:55 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3873022779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:55 smithi160 ceph-mon[36715]: Deploying daemon mds.g on smithi144 2024-07-26T10:56:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T10:56:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:55 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:boot 2024-07-26T10:56:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:55 smithi160 ceph-mon[36715]: fsmap 9 up:standby 2024-07-26T10:56:55.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:55 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3873022779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:55 smithi183 ceph-mon[37273]: Deploying daemon mds.g on smithi144 2024-07-26T10:56:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T10:56:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:55 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:boot 2024-07-26T10:56:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:55 smithi183 ceph-mon[37273]: fsmap 9 up:standby 2024-07-26T10:56:55.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:55 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3873022779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:56:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:56 smithi160 ceph-mon[36715]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:56.332 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:56 smithi144 ceph-mon[32129]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:56 smithi183 ceph-mon[37273]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:57.377 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 systemd[1]: Starting Ceph mds.g for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:56:57.701 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 podman[78065]: 2024-07-26 10:56:57.285490715 +0000 UTC m=+0.040733162 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:56:57.701 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 podman[78065]: 2024-07-26 10:56:57.426778258 +0000 UTC m=+0.182020665 container create 833e9cdd0f01f26406c552e6f636a600d1700c5600924c7aa7f24433eeca739e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-07-26T10:56:57.701 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 podman[78065]: 2024-07-26 10:56:57.69779214 +0000 UTC m=+0.453034623 container init 833e9cdd0f01f26406c552e6f636a600d1700c5600924c7aa7f24433eeca739e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-g, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T10:56:57.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:56:57.970 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 podman[78065]: 2024-07-26 10:56:57.707987518 +0000 UTC m=+0.463229928 container start 833e9cdd0f01f26406c552e6f636a600d1700c5600924c7aa7f24433eeca739e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-g, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-26T10:56:57.970 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 bash[78065]: 833e9cdd0f01f26406c552e6f636a600d1700c5600924c7aa7f24433eeca739e 2024-07-26T10:56:57.970 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 systemd[1]: Started Ceph mds.g for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:56:57.970 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 ceph-mds[78085]: -- 172.21.15.144:0/790710182 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c80bb7c20 con 0x564c84d59a80 2024-07-26T10:56:57.970 INFO:journalctl@ceph.mds.g.smithi144.stdout:Jul 26 10:56:57 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-g[78081]: starting mds.g at 2024-07-26T10:56:58.196 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:56:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:58.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:58.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:56:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:56:58.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:59 smithi160 ceph-mon[36715]: Deploying daemon mds.a on smithi144 2024-07-26T10:56:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T10:56:59.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:59 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] up:boot 2024-07-26T10:56:59.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:56:59 smithi160 ceph-mon[36715]: fsmap 10 up:standby 2024-07-26T10:56:59.363 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:59 smithi144 ceph-mon[32129]: Deploying daemon mds.a on smithi144 2024-07-26T10:56:59.363 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T10:56:59.363 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:59 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] up:boot 2024-07-26T10:56:59.363 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:56:59 smithi144 ceph-mon[32129]: fsmap 10 up:standby 2024-07-26T10:56:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:59 smithi183 ceph-mon[37273]: Deploying daemon mds.a on smithi144 2024-07-26T10:56:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T10:56:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:59 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] up:boot 2024-07-26T10:56:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:56:59 smithi183 ceph-mon[37273]: fsmap 10 up:standby 2024-07-26T10:57:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:00 smithi144 ceph-mon[32129]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:00 smithi183 ceph-mon[37273]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:00 smithi160 ceph-mon[36715]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:01.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 -- 172.21.15.144:0/2388089312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8108730 msgr2=0x7f53d810ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 --2- 172.21.15.144:0/2388089312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8108730 0x7f53d810ef60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ca0 tx=0x7f53d002a8c0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 -- 172.21.15.144:0/2388089312 shutdown_connections 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 --2- 172.21.15.144:0/2388089312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8108730 0x7f53d810ef60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 --2- 172.21.15.144:0/2388089312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53d8107de0 0x7f53d81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 --2- 172.21.15.144:0/2388089312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d81024a0 0x7f53d81028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 -- 172.21.15.144:0/2388089312 >> 172.21.15.144:0/2388089312 conn(0x7f53d8079160 msgr2=0x7f53d8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 -- 172.21.15.144:0/2388089312 shutdown_connections 2024-07-26T10:57:01.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.614+0000 7f53de200640 1 -- 172.21.15.144:0/2388089312 wait complete. 2024-07-26T10:57:01.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.615+0000 7f53de200640 1 Processor -- start 2024-07-26T10:57:01.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.615+0000 7f53de200640 1 -- start start 2024-07-26T10:57:01.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53de200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53d81024a0 0x7f53d819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:01.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53de200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8107de0 0x7f53d819c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:01.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53de200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 0x7f53d81a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53de200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f53d8113510 con 0x7f53d8107de0 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53de200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f53d8113390 con 0x7f53d81024a0 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53de200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53d8113690 con 0x7f53d8108730 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53cea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8107de0 0x7f53d819c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 0x7f53d81a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 0x7f53d81a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:32768/0 (socket says 172.21.15.144:32768) 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53d7e00640 1 -- 172.21.15.144:0/3629666984 learned_addr learned my addr 172.21.15.144:0/3629666984 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.616+0000 7f53cea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8107de0 0x7f53d819c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37340/0 (socket says 172.21.15.144:37340) 2024-07-26T10:57:01.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.617+0000 7f53d7400640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53d81024a0 0x7f53d819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:01.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.617+0000 7f53d7e00640 1 -- 172.21.15.144:0/3629666984 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53d81024a0 msgr2=0x7f53d819c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:01.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.617+0000 7f53d7e00640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53d81024a0 0x7f53d819c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.617+0000 7f53d7e00640 1 -- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8107de0 msgr2=0x7f53d819c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:01.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.617+0000 7f53d7e00640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8107de0 0x7f53d819c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.617+0000 7f53d7e00640 1 -- 172.21.15.144:0/3629666984 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d81a1f90 con 0x7f53d8108730 2024-07-26T10:57:01.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.618+0000 7f53d7e00640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 0x7f53d81a1860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f53d002f540 tx=0x7f53d00316f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:01.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.618+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0003b70 con 0x7f53d8108730 2024-07-26T10:57:01.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.618+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53d0003d10 con 0x7f53d8108730 2024-07-26T10:57:01.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.618+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d81a2220 con 0x7f53d8108730 2024-07-26T10:57:01.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.618+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d002f990 con 0x7f53d8108730 2024-07-26T10:57:01.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.618+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d81a89e0 con 0x7f53d8108730 2024-07-26T10:57:01.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.620+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d8103720 con 0x7f53d8108730 2024-07-26T10:57:01.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.622+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f53d002fb30 con 0x7f53d8108730 2024-07-26T10:57:01.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.623+0000 7f53d4c00640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53bc0b3ca0 0x7f53bc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:01.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.624+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f53d0104e30 con 0x7f53d8108730 2024-07-26T10:57:01.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.624+0000 7f53d7400640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53bc0b3ca0 0x7f53bc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:01.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.624+0000 7f53d7400640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53bc0b3ca0 0x7f53bc0b61b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009bd0 tx=0x7f53c8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:01.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.627+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f53d00cc7b0 con 0x7f53d8108730 2024-07-26T10:57:01.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.892+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53d81a8c40 con 0x7f53d8108730 2024-07-26T10:57:01.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.893+0000 7f53d4c00640 1 -- 172.21.15.144:0/3629666984 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8884 (secure 0 0 0) 0x7f53d00cc0b0 con 0x7f53d8108730 2024-07-26T10:57:01.896 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:01.896 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":11,"btime":"2024-07-26T10:56:58:035116+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24677,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24691,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24701,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24703,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-07-26T10:57:01.896 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 11 2024-07-26T10:57:01.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53bc0b3ca0 msgr2=0x7f53bc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:01.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53bc0b3ca0 0x7f53bc0b61b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009bd0 tx=0x7f53c8005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:01.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 msgr2=0x7f53d81a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:01.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 0x7f53d81a1860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f53d002f540 tx=0x7f53d00316f0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 shutdown_connections 2024-07-26T10:57:01.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53bc0b3ca0 0x7f53bc0b61b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53d8108730 0x7f53d81a1860 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53d8107de0 0x7f53d819c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 --2- 172.21.15.144:0/3629666984 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53d81024a0 0x7f53d819c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 >> 172.21.15.144:0/3629666984 conn(0x7f53d8079160 msgr2=0x7f53d8103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 shutdown_connections 2024-07-26T10:57:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:01.896+0000 7f53de200640 1 -- 172.21.15.144:0/3629666984 wait complete. 2024-07-26T10:57:02.151 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:02 smithi144 ceph-mon[32129]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:02.151 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:02 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3629666984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:02 smithi183 ceph-mon[37273]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:02 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3629666984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:02 smithi160 ceph-mon[36715]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:02 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3629666984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:02.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 10:57:02 smithi144 systemd[1]: Started Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: Deploying daemon mds.b on smithi160 2024-07-26T10:57:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:04.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T10:57:04.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] up:boot 2024-07-26T10:57:04.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:03 smithi160 ceph-mon[36715]: fsmap 11 up:standby 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: Deploying daemon mds.b on smithi160 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T10:57:04.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] up:boot 2024-07-26T10:57:04.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:03 smithi183 ceph-mon[37273]: fsmap 11 up:standby 2024-07-26T10:57:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: Deploying daemon mds.b on smithi160 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] up:boot 2024-07-26T10:57:04.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:03 smithi144 ceph-mon[32129]: fsmap 11 up:standby 2024-07-26T10:57:05.007 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:04 smithi160 systemd[1]: Starting Ceph mds.b for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T10:57:05.199 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:57:05.288 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 podman[67860]: 2024-07-26 10:57:04.907678703 +0000 UTC m=+0.017614319 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T10:57:05.289 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 podman[67860]: 2024-07-26 10:57:05.025616031 +0000 UTC m=+0.135551638 container create decd0104d935384312348803fcc9a89f27815def9e83d5379e3334612dece527 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T10:57:05.289 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 podman[67860]: 2024-07-26 10:57:05.286871586 +0000 UTC m=+0.396807192 container init decd0104d935384312348803fcc9a89f27815def9e83d5379e3334612dece527 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-07-26T10:57:05.559 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 podman[67860]: 2024-07-26 10:57:05.2896706 +0000 UTC m=+0.399606208 container start decd0104d935384312348803fcc9a89f27815def9e83d5379e3334612dece527 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-07-26T10:57:05.559 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 ceph-mds[67879]: -- 172.21.15.160:0/151550668 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560057ffd860 con 0x56005c19fa80 2024-07-26T10:57:05.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: starting mds.b at 2024-07-26T10:57:05.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 bash[67860]: decd0104d935384312348803fcc9a89f27815def9e83d5379e3334612dece527 2024-07-26T10:57:05.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:05 smithi160 systemd[1]: Started Ceph mds.b for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T10:57:05.672 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T10:57:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:07 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] up:boot 2024-07-26T10:57:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:07 smithi144 ceph-mon[32129]: fsmap 12 up:standby 2024-07-26T10:57:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T10:57:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:07 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] up:boot 2024-07-26T10:57:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:07 smithi160 ceph-mon[36715]: fsmap 12 up:standby 2024-07-26T10:57:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T10:57:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:07 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] up:boot 2024-07-26T10:57:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:07 smithi183 ceph-mon[37273]: fsmap 12 up:standby 2024-07-26T10:57:07.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 -- 172.21.15.144:0/1411492327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5360108730 msgr2=0x7f536010ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:07.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 --2- 172.21.15.144:0/1411492327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5360108730 0x7f536010ef60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5358009900 tx=0x7f535802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:57:07.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 -- 172.21.15.144:0/1411492327 shutdown_connections 2024-07-26T10:57:07.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 --2- 172.21.15.144:0/1411492327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5360108730 0x7f536010ef60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:07.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 --2- 172.21.15.144:0/1411492327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360107de0 0x7f53601081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:07.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 --2- 172.21.15.144:0/1411492327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53601024f0 0x7f5360102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:07.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 -- 172.21.15.144:0/1411492327 >> 172.21.15.144:0/1411492327 conn(0x7f5360079150 msgr2=0x7f5360079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:07.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 -- 172.21.15.144:0/1411492327 shutdown_connections 2024-07-26T10:57:07.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.863+0000 7f5365600640 1 -- 172.21.15.144:0/1411492327 wait complete. 2024-07-26T10:57:07.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.864+0000 7f5365600640 1 Processor -- start 2024-07-26T10:57:07.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 -- start start 2024-07-26T10:57:07.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 0x7f5360199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5360107de0 0x7f536019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360108730 0x7f536019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5360112430 con 0x7f5360108730 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f53601122b0 con 0x7f53601024f0 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5365600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53601125b0 con 0x7f5360107de0 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f535ea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 0x7f5360199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f535ea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 0x7f5360199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56232/0 (socket says 172.21.15.144:56232) 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f535ea00640 1 -- 172.21.15.144:0/2631790323 learned_addr learned my addr 172.21.15.144:0/2631790323 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f5356000640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5360107de0 0x7f536019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.865+0000 7f535f400640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360108730 0x7f536019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535ea00640 1 -- 172.21.15.144:0/2631790323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5360107de0 msgr2=0x7f536019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:07.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535ea00640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5360107de0 0x7f536019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535ea00640 1 -- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360108730 msgr2=0x7f536019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535ea00640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360108730 0x7f536019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535ea00640 1 -- 172.21.15.144:0/2631790323 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536019fe20 con 0x7f53601024f0 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535f400640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360108730 0x7f536019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f535ea00640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 0x7f5360199eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5350009d50 tx=0x7f535000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350003b70 con 0x7f53601024f0 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5350003d10 con 0x7f53601024f0 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53601a0110 con 0x7f53601024f0 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53500108a0 con 0x7f53601024f0 2024-07-26T10:57:07.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.866+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53601a6870 con 0x7f53601024f0 2024-07-26T10:57:07.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.868+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5360102900 con 0x7f53601024f0 2024-07-26T10:57:07.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.870+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f5350004000 con 0x7f53601024f0 2024-07-26T10:57:07.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.871+0000 7f5357e00640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53440b3c20 0x7f53440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:07.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.872+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f53500e40b0 con 0x7f53601024f0 2024-07-26T10:57:07.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.872+0000 7f5356000640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53440b3c20 0x7f53440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:07.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.872+0000 7f5356000640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53440b3c20 0x7f53440b6130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5360079790 tx=0x7f5348005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:07.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:07.875+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f53500aba30 con 0x7f53601024f0 2024-07-26T10:57:08.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.119+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f536019afe0 con 0x7f53601024f0 2024-07-26T10:57:08.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.120+0000 7f5357e00640 1 -- 172.21.15.144:0/2631790323 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10542 (secure 0 0 0) 0x7f53500ab330 con 0x7f53601024f0 2024-07-26T10:57:08.122 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:08.122 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":13,"btime":"2024-07-26T10:57:06:003088+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24677,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24691,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24701,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24703,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24715,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-07-26T10:57:08.122 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 13 2024-07-26T10:57:08.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53440b3c20 msgr2=0x7f53440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:08.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53440b3c20 0x7f53440b6130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5360079790 tx=0x7f5348005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:08.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 msgr2=0x7f5360199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:08.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 0x7f5360199eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5350009d50 tx=0x7f535000c6a0 comp rx=0 tx=0).stop 2024-07-26T10:57:08.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 shutdown_connections 2024-07-26T10:57:08.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53440b3c20 0x7f53440b6130 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:08.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5360108730 0x7f536019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:08.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5360107de0 0x7f536019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:08.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 --2- 172.21.15.144:0/2631790323 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53601024f0 0x7f5360199eb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:08.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 >> 172.21.15.144:0/2631790323 conn(0x7f5360079150 msgr2=0x7f5360101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:08.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 shutdown_connections 2024-07-26T10:57:08.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:08.123+0000 7f5365600640 1 -- 172.21.15.144:0/2631790323 wait complete. 2024-07-26T10:57:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:08 smithi160 ceph-mon[36715]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.381 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:08 smithi144 ceph-mon[32129]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:08.381 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.381 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.381 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.381 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.381 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:08 smithi183 ceph-mon[37273]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:09.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-26T10:57:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:09 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2631790323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:09 smithi183 ceph-mon[37273]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:09 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2631790323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:09 smithi144 ceph-mon[32129]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:09 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2631790323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:09 smithi160 ceph-mon[36715]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:09.660 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:11.642 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.643 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.643 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:11.643 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:11.643 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.643 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:11.643 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:11 smithi144 ceph-mon[32129]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:11.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:11 smithi183 ceph-mon[37273]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:11 smithi160 ceph-mon[36715]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 -- 172.21.15.144:0/4280080672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c60108ef0 msgr2=0x7f2c60109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4280080672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c60108ef0 0x7f2c60109300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009900 tx=0x7f2c4c02a3f0 comp rx=0 tx=0).stop 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 -- 172.21.15.144:0/4280080672 shutdown_connections 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4280080672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c60109840 0x7f2c60110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4280080672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c60108ef0 0x7f2c60109300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4280080672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c601035b0 0x7f2c601039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.838+0000 7f2c64e00640 1 -- 172.21.15.144:0/4280080672 >> 172.21.15.144:0/4280080672 conn(0x7f2c600ff000 msgr2=0x7f2c60101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.839+0000 7f2c64e00640 1 -- 172.21.15.144:0/4280080672 shutdown_connections 2024-07-26T10:57:11.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.839+0000 7f2c64e00640 1 -- 172.21.15.144:0/4280080672 wait complete. 2024-07-26T10:57:11.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.840+0000 7f2c64e00640 1 Processor -- start 2024-07-26T10:57:11.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.840+0000 7f2c64e00640 1 -- start start 2024-07-26T10:57:11.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c64e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 0x7f2c6019c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:11.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c64e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c60108ef0 0x7f2c6019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:11.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c64e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c60109840 0x7f2c601a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:11.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c64e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601134b0 con 0x7f2c60108ef0 2024-07-26T10:57:11.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c64e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2c60113330 con 0x7f2c60109840 2024-07-26T10:57:11.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c64e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2c60113630 con 0x7f2c601035b0 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 0x7f2c6019c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 0x7f2c6019c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:36234/0 (socket says 172.21.15.144:36234) 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 -- 172.21.15.144:0/4124195896 learned_addr learned my addr 172.21.15.144:0/4124195896 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5d400640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c60108ef0 0x7f2c6019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5e800640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c60109840 0x7f2c601a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 -- 172.21.15.144:0/4124195896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c60109840 msgr2=0x7f2c601a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c60109840 0x7f2c601a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 -- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c60108ef0 msgr2=0x7f2c6019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c60108ef0 0x7f2c6019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5de00640 1 -- 172.21.15.144:0/4124195896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c601a20f0 con 0x7f2c601035b0 2024-07-26T10:57:11.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.841+0000 7f2c5d400640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c60108ef0 0x7f2c6019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:57:11.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.842+0000 7f2c5de00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 0x7f2c6019c180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000e9f0 tx=0x7f2c5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:11.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.842+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5000cd80 con 0x7f2c601035b0 2024-07-26T10:57:11.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.842+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c601a23e0 con 0x7f2c601035b0 2024-07-26T10:57:11.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.842+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c500108e0 con 0x7f2c601035b0 2024-07-26T10:57:11.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.842+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c50018980 con 0x7f2c601035b0 2024-07-26T10:57:11.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.842+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c601a8b40 con 0x7f2c601035b0 2024-07-26T10:57:11.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.844+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c601039c0 con 0x7f2c601035b0 2024-07-26T10:57:11.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.846+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f2c50010430 con 0x7f2c601035b0 2024-07-26T10:57:11.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.848+0000 7f2c56a00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c440b3c20 0x7f2c440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:11.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.848+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f2c500e3d80 con 0x7f2c601035b0 2024-07-26T10:57:11.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.848+0000 7f2c5d400640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c440b3c20 0x7f2c440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:11.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.849+0000 7f2c5d400640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c440b3c20 0x7f2c440b6130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009230 tx=0x7f2c4c000bc0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:11.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:11.852+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2c500ab700 con 0x7f2c601035b0 2024-07-26T10:57:12.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.060+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2c601a8f30 con 0x7f2c601035b0 2024-07-26T10:57:12.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.060+0000 7f2c56a00640 1 -- 172.21.15.144:0/4124195896 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f2c500ab000 con 0x7f2c601035b0 2024-07-26T10:57:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:12.063 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2024-07-26T10:57:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-07-26T10:57:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "poolnum": 1, 2024-07-26T10:57:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "poolname": ".mgr" 2024-07-26T10:57:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T10:57:12.063 INFO:teuthology.orchestra.run.smithi144.stdout:] 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.063+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c440b3c20 msgr2=0x7f2c440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.063+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c440b3c20 0x7f2c440b6130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009230 tx=0x7f2c4c000bc0 comp rx=0 tx=0).stop 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.063+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 msgr2=0x7f2c6019c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.063+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 0x7f2c6019c180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000e9f0 tx=0x7f2c5000eeb0 comp rx=0 tx=0).stop 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 shutdown_connections 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2c440b3c20 0x7f2c440b6130 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c60109840 0x7f2c601a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c60108ef0 0x7f2c6019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 --2- 172.21.15.144:0/4124195896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c601035b0 0x7f2c6019c180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 >> 172.21.15.144:0/4124195896 conn(0x7f2c600ff000 msgr2=0x7f2c601075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 shutdown_connections 2024-07-26T10:57:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:12.064+0000 7f2c64e00640 1 -- 172.21.15.144:0/4124195896 wait complete. 2024-07-26T10:57:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:12 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/4124195896' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T10:57:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:12 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/4124195896' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T10:57:12.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.865 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:12.865 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:12 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/4124195896' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T10:57:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:13.444 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T10:57:13.445 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T10:57:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:13 smithi144 ceph-mon[32129]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:13 smithi183 ceph-mon[37273]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:13.917 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:13 smithi160 ceph-mon[36715]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3327047809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0108d20 msgr2=0x7fcfe0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3327047809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0108d20 0x7fcfe0109130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009900 tx=0x7fcfcc02a410 comp rx=0 tx=0).stop 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3327047809 shutdown_connections 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3327047809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfe0109670 0x7fcfe010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3327047809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0108d20 0x7fcfe0109130 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3327047809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 0x7fcfe01037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3327047809 >> 172.21.15.144:0/3327047809 conn(0x7fcfe00fee70 msgr2=0x7fcfe01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3327047809 shutdown_connections 2024-07-26T10:57:16.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.016+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3327047809 wait complete. 2024-07-26T10:57:16.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.017+0000 7fcfe6a00640 1 Processor -- start 2024-07-26T10:57:16.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 -- start start 2024-07-26T10:57:16.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 0x7fcfe019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfe0108d20 0x7fcfe019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 0x7fcfe01a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0113350 con 0x7fcfe0109670 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe01131d0 con 0x7fcfe0108d20 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe01134d0 con 0x7fcfe01033e0 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 0x7fcfe01a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 0x7fcfe019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.018+0000 7fcfe4c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfe0108d20 0x7fcfe019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 0x7fcfe01a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54512/0 (socket says 172.21.15.144:54512) 2024-07-26T10:57:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe6000640 1 -- 172.21.15.144:0/3554736307 learned_addr learned my addr 172.21.15.144:0/3554736307 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:16.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe5600640 1 -- 172.21.15.144:0/3554736307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfe0108d20 msgr2=0x7fcfe019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:16.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe5600640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfe0108d20 0x7fcfe019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:16.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe5600640 1 -- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 msgr2=0x7fcfe01a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:16.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe5600640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 0x7fcfe01a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe5600640 1 -- 172.21.15.144:0/3554736307 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe01a1ed0 con 0x7fcfe01033e0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe6000640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 0x7fcfe01a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.019+0000 7fcfe5600640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 0x7fcfe019bf60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9f0 tx=0x7fcfd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.020+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400ccc0 con 0x7fcfe01033e0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.020+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd4005020 con 0x7fcfe01033e0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.020+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe01a21c0 con 0x7fcfe01033e0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.020+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4004a00 con 0x7fcfe01033e0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.020+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe01a8920 con 0x7fcfe01033e0 2024-07-26T10:57:16.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.021+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe01037f0 con 0x7fcfe01033e0 2024-07-26T10:57:16.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.023+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fcfd4004ba0 con 0x7fcfe01033e0 2024-07-26T10:57:16.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.025+0000 7fcfde000640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcfc80b3ca0 0x7fcfc80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:16.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.025+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fcfd40e3830 con 0x7fcfe01033e0 2024-07-26T10:57:16.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.025+0000 7fcfe4c00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcfc80b3ca0 0x7fcfc80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:16.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.026+0000 7fcfe4c00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcfc80b3ca0 0x7fcfc80b61b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc02aa00 tx=0x7fcfcc005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:16.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.029+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcfd40ab1b0 con 0x7fcfe01033e0 2024-07-26T10:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:16 smithi144 ceph-mon[32129]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:16.234+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcfe019d090 con 0x7fcfe01033e0 2024-07-26T10:57:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:16 smithi160 ceph-mon[36715]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:16 smithi183 ceph-mon[37273]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:57:17.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.032+0000 7fcfde000640 1 -- 172.21.15.144:0/3554736307 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fcfd40aaab0 con 0x7fcfe01033e0 2024-07-26T10:57:17.039 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'cephfs_metadata' created 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcfc80b3ca0 msgr2=0x7fcfc80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcfc80b3ca0 0x7fcfc80b61b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc02aa00 tx=0x7fcfcc005a20 comp rx=0 tx=0).stop 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 msgr2=0x7fcfe019bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 0x7fcfe019bf60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9f0 tx=0x7fcfd400eeb0 comp rx=0 tx=0).stop 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 shutdown_connections 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcfc80b3ca0 0x7fcfc80b61b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe0109670 0x7fcfe01a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfe0108d20 0x7fcfe019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 --2- 172.21.15.144:0/3554736307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfe01033e0 0x7fcfe019bf60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:17.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.040+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 >> 172.21.15.144:0/3554736307 conn(0x7fcfe00fee70 msgr2=0x7fcfe0108390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:17.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.041+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 shutdown_connections 2024-07-26T10:57:17.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:17.042+0000 7fcfe6a00640 1 -- 172.21.15.144:0/3554736307 wait complete. 2024-07-26T10:57:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:17 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3554736307' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T10:57:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:17 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T10:57:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:17 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3554736307' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T10:57:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:17 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T10:57:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:17 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3554736307' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T10:57:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:17 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T10:57:18.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-07-26T10:57:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:18 smithi144 ceph-mon[32129]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:18 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T10:57:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:18 smithi144 ceph-mon[32129]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T10:57:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:18 smithi160 ceph-mon[36715]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:18 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T10:57:18.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:18 smithi160 ceph-mon[36715]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T10:57:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:18 smithi183 ceph-mon[37273]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:18 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T10:57:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:18 smithi183 ceph-mon[37273]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T10:57:18.466 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:19 smithi144 ceph-mon[32129]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T10:57:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:19 smithi160 ceph-mon[36715]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T10:57:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:19 smithi183 ceph-mon[37273]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T10:57:20.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:20 smithi144 ceph-mon[32129]: pgmap v290: 33 pgs: 6 creating+peering, 2 active+clean, 25 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:20.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:20 smithi144 ceph-mon[32129]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:20.170 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:20 smithi144 ceph-mon[32129]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T10:57:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:20 smithi160 ceph-mon[36715]: pgmap v290: 33 pgs: 6 creating+peering, 2 active+clean, 25 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:20.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:20 smithi160 ceph-mon[36715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:20.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:20 smithi160 ceph-mon[36715]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T10:57:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:20 smithi183 ceph-mon[37273]: pgmap v290: 33 pgs: 6 creating+peering, 2 active+clean, 25 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:20 smithi183 ceph-mon[37273]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:20 smithi183 ceph-mon[37273]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T10:57:20.772 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 -- 172.21.15.144:0/1219002584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae481040c0 msgr2=0x7fae481044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:20.772 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 --2- 172.21.15.144:0/1219002584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae481040c0 0x7fae481044d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009900 tx=0x7fae3c02a3f0 comp rx=0 tx=0).stop 2024-07-26T10:57:20.772 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 -- 172.21.15.144:0/1219002584 shutdown_connections 2024-07-26T10:57:20.772 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 --2- 172.21.15.144:0/1219002584 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae4810d670 0x7fae4810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 --2- 172.21.15.144:0/1219002584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae48104af0 0x7fae4810d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 --2- 172.21.15.144:0/1219002584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae481040c0 0x7fae481044d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 -- 172.21.15.144:0/1219002584 >> 172.21.15.144:0/1219002584 conn(0x7fae480fdcf0 msgr2=0x7fae48100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 -- 172.21.15.144:0/1219002584 shutdown_connections 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.770+0000 7fae4e600640 1 -- 172.21.15.144:0/1219002584 wait complete. 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.771+0000 7fae4e600640 1 Processor -- start 2024-07-26T10:57:20.773 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 -- start start 2024-07-26T10:57:20.774 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 0x7fae4819beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:20.774 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae48104af0 0x7fae4819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:20.774 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae4810d670 0x7fae481a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:20.774 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fae481132a0 con 0x7fae48104af0 2024-07-26T10:57:20.775 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fae48113120 con 0x7fae481040c0 2024-07-26T10:57:20.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4e600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fae48113420 con 0x7fae4810d670 2024-07-26T10:57:20.778 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 0x7fae4819beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:20.778 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae47e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae48104af0 0x7fae4819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:20.778 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 0x7fae4819beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42758/0 (socket says 172.21.15.144:42758) 2024-07-26T10:57:20.778 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.772+0000 7fae4d200640 1 -- 172.21.15.144:0/1420396257 learned_addr learned my addr 172.21.15.144:0/1420396257 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:20.778 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4dc00640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae4810d670 0x7fae481a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:20.779 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4d200640 1 -- 172.21.15.144:0/1420396257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae4810d670 msgr2=0x7fae481a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:20.779 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4d200640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae4810d670 0x7fae481a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:20.779 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4d200640 1 -- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae48104af0 msgr2=0x7fae4819c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:20.779 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4d200640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae48104af0 0x7fae4819c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:20.779 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4d200640 1 -- 172.21.15.144:0/1420396257 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae481a1e20 con 0x7fae481040c0 2024-07-26T10:57:20.779 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.773+0000 7fae4d200640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 0x7fae4819beb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae3c005940 tx=0x7fae3c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.774+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c03f030 con 0x7fae481040c0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.774+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae481a20b0 con 0x7fae481040c0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.774+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae3c0044c0 con 0x7fae481040c0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.774+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c035070 con 0x7fae481040c0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.774+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae481a8870 con 0x7fae481040c0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.775+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae481044d0 con 0x7fae481040c0 2024-07-26T10:57:20.780 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.777+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fae3c042030 con 0x7fae481040c0 2024-07-26T10:57:20.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.779+0000 7fae45600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae300b3ca0 0x7fae300b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:20.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.779+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7712+0+0 (secure 0 0 0) 0x7fae3c103fa0 con 0x7fae481040c0 2024-07-26T10:57:20.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.779+0000 7fae47e00640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae300b3ca0 0x7fae300b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:20.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.780+0000 7fae47e00640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae300b3ca0 0x7fae300b61b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fae48101540 tx=0x7fae34005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:20.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.785+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fae3c0cb180 con 0x7fae481040c0 2024-07-26T10:57:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:20.984+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fae481a8aa0 con 0x7fae481040c0 2024-07-26T10:57:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.079+0000 7fae45600640 1 -- 172.21.15.144:0/1420396257 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) ==== 171+0+0 (secure 0 0 0) 0x7fae3c0cf6d0 con 0x7fae481040c0 2024-07-26T10:57:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'cephfs_data' created 2024-07-26T10:57:21.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.085+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae300b3ca0 msgr2=0x7fae300b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.085+0000 7fae4e600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae300b3ca0 0x7fae300b61b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fae48101540 tx=0x7fae34005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.085+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 msgr2=0x7fae4819beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.085+0000 7fae4e600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 0x7fae4819beb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae3c005940 tx=0x7fae3c0040c0 comp rx=0 tx=0).stop 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 shutdown_connections 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae300b3ca0 0x7fae300b61b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae4810d670 0x7fae481a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:21.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae48104af0 0x7fae4819c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:21.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 --2- 172.21.15.144:0/1420396257 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae481040c0 0x7fae4819beb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:21.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 >> 172.21.15.144:0/1420396257 conn(0x7fae480fdcf0 msgr2=0x7fae48101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:21.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.086+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 shutdown_connections 2024-07-26T10:57:21.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:21.088+0000 7fae4e600640 1 -- 172.21.15.144:0/1420396257 wait complete. 2024-07-26T10:57:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:21 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1420396257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T10:57:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:21 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T10:57:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:21 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1420396257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T10:57:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:21 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T10:57:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:21 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1420396257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T10:57:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:21 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T10:57:22.003 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T10:57:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:22 smithi144 ceph-mon[32129]: pgmap v292: 33 pgs: 6 creating+peering, 18 active+clean, 9 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:22 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T10:57:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:22 smithi144 ceph-mon[32129]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T10:57:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:22 smithi183 ceph-mon[37273]: pgmap v292: 33 pgs: 6 creating+peering, 18 active+clean, 9 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:22 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T10:57:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:22 smithi183 ceph-mon[37273]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T10:57:22.483 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:22 smithi160 ceph-mon[36715]: pgmap v292: 33 pgs: 6 creating+peering, 18 active+clean, 9 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:22 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T10:57:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:22 smithi160 ceph-mon[36715]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T10:57:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:23 smithi183 ceph-mon[37273]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T10:57:23.462 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:23 smithi144 ceph-mon[32129]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T10:57:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:23 smithi160 ceph-mon[36715]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T10:57:24.201 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:24 smithi144 ceph-mon[32129]: pgmap v295: 97 pgs: 9 creating+peering, 38 active+clean, 50 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:24.201 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:24 smithi144 ceph-mon[32129]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T10:57:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:24 smithi183 ceph-mon[37273]: pgmap v295: 97 pgs: 9 creating+peering, 38 active+clean, 50 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:24 smithi183 ceph-mon[37273]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T10:57:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:24 smithi160 ceph-mon[36715]: pgmap v295: 97 pgs: 9 creating+peering, 38 active+clean, 50 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:24 smithi160 ceph-mon[36715]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 -- 172.21.15.144:0/2806545637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c1055d0 msgr2=0x7f433c1059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 --2- 172.21.15.144:0/2806545637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c1055d0 0x7f433c1059e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4330009900 tx=0x7f433002a410 comp rx=0 tx=0).stop 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 -- 172.21.15.144:0/2806545637 shutdown_connections 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 --2- 172.21.15.144:0/2806545637 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f433c10b860 0x7f433c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 --2- 172.21.15.144:0/2806545637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c10af10 0x7f433c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 --2- 172.21.15.144:0/2806545637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c1055d0 0x7f433c1059e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.753+0000 7f4342200640 1 -- 172.21.15.144:0/2806545637 >> 172.21.15.144:0/2806545637 conn(0x7f433c101040 msgr2=0x7f433c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:24.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.754+0000 7f4342200640 1 -- 172.21.15.144:0/2806545637 shutdown_connections 2024-07-26T10:57:24.757 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.754+0000 7f4342200640 1 -- 172.21.15.144:0/2806545637 wait complete. 2024-07-26T10:57:24.757 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 Processor -- start 2024-07-26T10:57:24.757 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 -- start start 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 0x7f433c19bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c10af10 0x7f433c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f433c10b860 0x7f433c1a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f433c115510 con 0x7f433c10af10 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f433c115390 con 0x7f433c10b860 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.755+0000 7f4342200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f433c115690 con 0x7f433c1055d0 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 0x7f433c19bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f433be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c10af10 0x7f433c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 0x7f433c19bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:46804/0 (socket says 172.21.15.144:46804) 2024-07-26T10:57:24.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4341800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f433c10b860 0x7f433c1a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:24.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 -- 172.21.15.144:0/2696395913 learned_addr learned my addr 172.21.15.144:0/2696395913 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:24.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 -- 172.21.15.144:0/2696395913 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f433c10b860 msgr2=0x7f433c1a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:24.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f433c10b860 0x7f433c1a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:24.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 -- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c10af10 msgr2=0x7f433c19c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c10af10 0x7f433c19c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 -- 172.21.15.144:0/2696395913 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433c1a1e90 con 0x7f433c1055d0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.756+0000 7f4340e00640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 0x7f433c19bf20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4330005940 tx=0x7f43300040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.757+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433003f030 con 0x7f433c1055d0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.757+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f433c1a2120 con 0x7f433c1055d0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.757+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43300044c0 con 0x7f433c1055d0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.757+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330035070 con 0x7f433c1055d0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.757+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f433c1a88e0 con 0x7f433c1055d0 2024-07-26T10:57:24.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.758+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f433c1059e0 con 0x7f433c1055d0 2024-07-26T10:57:24.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.761+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4330042030 con 0x7f433c1055d0 2024-07-26T10:57:24.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.762+0000 7f4339600640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43240b3c20 0x7f43240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:24.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.762+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8094+0+0 (secure 0 0 0) 0x7f43301041d0 con 0x7f433c1055d0 2024-07-26T10:57:24.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.762+0000 7f433be00640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43240b3c20 0x7f43240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:24.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.763+0000 7f433be00640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43240b3c20 0x7f43240b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f433c10aa70 tx=0x7f4328005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:24.765 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:24.764+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f43300cb210 con 0x7f433c1055d0 2024-07-26T10:57:25.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.000+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f433c19d050 con 0x7f433c1055d0 2024-07-26T10:57:25.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.135+0000 7f4339600640 1 -- 172.21.15.144:0/2696395913 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f43300cf760 con 0x7f433c1055d0 2024-07-26T10:57:25.138 INFO:teuthology.orchestra.run.smithi144.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T10:57:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.141+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43240b3c20 msgr2=0x7f43240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.141+0000 7f4342200640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43240b3c20 0x7f43240b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f433c10aa70 tx=0x7f4328005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.141+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 msgr2=0x7f433c19bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.141+0000 7f4342200640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 0x7f433c19bf20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4330005940 tx=0x7f43300040c0 comp rx=0 tx=0).stop 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 shutdown_connections 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43240b3c20 0x7f43240b6130 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f433c10b860 0x7f433c1a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f433c10af10 0x7f433c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 --2- 172.21.15.144:0/2696395913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f433c1055d0 0x7f433c19bf20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 >> 172.21.15.144:0/2696395913 conn(0x7f433c101040 msgr2=0x7f433c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.142+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 shutdown_connections 2024-07-26T10:57:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:25.143+0000 7f4342200640 1 -- 172.21.15.144:0/2696395913 wait complete. 2024-07-26T10:57:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:25 smithi183 ceph-mon[37273]: pgmap v297: 97 pgs: 6 creating+peering, 66 active+clean, 25 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:25 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2696395913' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T10:57:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:25 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T10:57:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:25 smithi183 ceph-mon[37273]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T10:57:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:25 smithi144 ceph-mon[32129]: pgmap v297: 97 pgs: 6 creating+peering, 66 active+clean, 25 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:25 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2696395913' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T10:57:25.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:25 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T10:57:25.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:25 smithi144 ceph-mon[32129]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T10:57:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:25 smithi160 ceph-mon[36715]: pgmap v297: 97 pgs: 6 creating+peering, 66 active+clean, 25 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:25 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2696395913' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T10:57:25.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:25 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T10:57:25.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:25 smithi160 ceph-mon[36715]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T10:57:25.946 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-07-26T10:57:25.946 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] up:active 2024-07-26T10:57:26.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:26 smithi144 ceph-mon[32129]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:57:26.309 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T10:57:26.018+0000 7f775865d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] up:active 2024-07-26T10:57:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:26 smithi160 ceph-mon[36715]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:57:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T10:57:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T10:57:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T10:57:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T10:57:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:26.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] up:active 2024-07-26T10:57:26.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:26 smithi183 ceph-mon[37273]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:57:26.400 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:27 smithi144 ceph-mon[32129]: pgmap v299: 97 pgs: 97 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 348 B/s wr, 4 op/s 2024-07-26T10:57:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:27 smithi183 ceph-mon[37273]: pgmap v299: 97 pgs: 97 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 348 B/s wr, 4 op/s 2024-07-26T10:57:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:27 smithi160 ceph-mon[36715]: pgmap v299: 97 pgs: 97 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 348 B/s wr, 4 op/s 2024-07-26T10:57:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:28 smithi183 ceph-mon[37273]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T10:57:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:28 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T10:57:28.411 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:28 smithi144 ceph-mon[32129]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T10:57:28.412 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:28 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T10:57:28.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:28 smithi160 ceph-mon[36715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T10:57:28.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:28 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T10:57:28.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 -- 172.21.15.144:0/443752897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 msgr2=0x7f58401059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 --2- 172.21.15.144:0/443752897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f58401059e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5830009900 tx=0x7f583002a3f0 comp rx=0 tx=0).stop 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 -- 172.21.15.144:0/443752897 shutdown_connections 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 --2- 172.21.15.144:0/443752897 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f584010b860 0x7f5840112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 --2- 172.21.15.144:0/443752897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 0x7f584010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 --2- 172.21.15.144:0/443752897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f58401059e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 -- 172.21.15.144:0/443752897 >> 172.21.15.144:0/443752897 conn(0x7f5840101040 msgr2=0x7f5840103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 -- 172.21.15.144:0/443752897 shutdown_connections 2024-07-26T10:57:28.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.735+0000 7f583fe00640 1 -- 172.21.15.144:0/443752897 wait complete. 2024-07-26T10:57:28.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.736+0000 7f583fe00640 1 Processor -- start 2024-07-26T10:57:28.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 -- start start 2024-07-26T10:57:28.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f584019bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:28.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 0x7f584019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:28.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f584010b860 0x7f58401a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5840115560 con 0x7f584010af10 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f58401153e0 con 0x7f584010b860 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583fe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f58401156e0 con 0x7f58401055d0 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583e000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 0x7f584019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583ea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f584019bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.737+0000 7f583ea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f584019bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:46814/0 (socket says 172.21.15.144:46814) 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 -- 172.21.15.144:0/2580102286 learned_addr learned my addr 172.21.15.144:0/2580102286 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:28.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583f400640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f584010b860 0x7f58401a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:28.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 -- 172.21.15.144:0/2580102286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f584010b860 msgr2=0x7f58401a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:28.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f584010b860 0x7f58401a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:28.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 -- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 msgr2=0x7f584019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:28.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 0x7f584019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:28.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 -- 172.21.15.144:0/2580102286 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58401a1ee0 con 0x7f58401055d0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583e000640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 0x7f584019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.738+0000 7f583ea00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f584019bf70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5830005ab0 tx=0x7f5830003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.739+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583003f070 con 0x7f58401055d0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.739+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5830031d60 con 0x7f58401055d0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.739+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830035030 con 0x7f58401055d0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.739+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58401a2170 con 0x7f58401055d0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.739+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58401a8930 con 0x7f58401055d0 2024-07-26T10:57:28.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.740+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5840106850 con 0x7f58401055d0 2024-07-26T10:57:28.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.743+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f5830042070 con 0x7f58401055d0 2024-07-26T10:57:28.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.744+0000 7f5837400640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58240b3ca0 0x7f58240b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:28.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.745+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f5830104a80 con 0x7f58401055d0 2024-07-26T10:57:28.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.745+0000 7f583e000640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58240b3ca0 0x7f58240b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:28.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.745+0000 7f583e000640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58240b3ca0 0x7f58240b61b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f584010ab10 tx=0x7f582c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:28.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.746+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f58300cba40 con 0x7f58401055d0 2024-07-26T10:57:28.953 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:28.950+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f58401059e0 con 0x7f58401055d0 2024-07-26T10:57:29.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.150+0000 7f5837400640 1 -- 172.21.15.144:0/2580102286 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) ==== 157+0+0 (secure 0 0 0) 0x7f58300cff90 con 0x7f58401055d0 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58240b3ca0 msgr2=0x7f58240b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58240b3ca0 0x7f58240b61b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f584010ab10 tx=0x7f582c005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 msgr2=0x7f584019bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f584019bf70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5830005ab0 tx=0x7f5830003c80 comp rx=0 tx=0).stop 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 shutdown_connections 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58240b3ca0 0x7f58240b61b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f584010b860 0x7f58401a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f584010af10 0x7f584019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 --2- 172.21.15.144:0/2580102286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58401055d0 0x7f584019bf70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.155+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 >> 172.21.15.144:0/2580102286 conn(0x7f5840101040 msgr2=0x7f5840103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.156+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 shutdown_connections 2024-07-26T10:57:29.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:29.156+0000 7f583fe00640 1 -- 172.21.15.144:0/2580102286 wait complete. 2024-07-26T10:57:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:29 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2580102286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T10:57:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:29 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T10:57:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:29 smithi144 ceph-mon[32129]: pgmap v300: 97 pgs: 97 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s wr, 4 op/s 2024-07-26T10:57:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:29 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2580102286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T10:57:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:29 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T10:57:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:29 smithi160 ceph-mon[36715]: pgmap v300: 97 pgs: 97 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s wr, 4 op/s 2024-07-26T10:57:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:29 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2580102286' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T10:57:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:29 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T10:57:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:29 smithi183 ceph-mon[37273]: pgmap v300: 97 pgs: 97 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s wr, 4 op/s 2024-07-26T10:57:29.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T10:57:30.153 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:30 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T10:57:30.153 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:30 smithi144 ceph-mon[32129]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T10:57:30.392 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:30 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T10:57:30.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:30 smithi160 ceph-mon[36715]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T10:57:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:30 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T10:57:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:30 smithi183 ceph-mon[37273]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T10:57:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:31 smithi160 ceph-mon[36715]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-07-26T10:57:31.423 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:31 smithi144 ceph-mon[32129]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-07-26T10:57:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:31 smithi183 ceph-mon[37273]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-07-26T10:57:32.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.752+0000 7fc98e000640 1 -- 172.21.15.144:0/2875516534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc988108d30 msgr2=0x7fc988109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:32.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.752+0000 7fc98e000640 1 --2- 172.21.15.144:0/2875516534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc988108d30 0x7fc988109140 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc970009ca0 tx=0x7fc97002a8c0 comp rx=0 tx=0).stop 2024-07-26T10:57:32.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 -- 172.21.15.144:0/2875516534 shutdown_connections 2024-07-26T10:57:32.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 --2- 172.21.15.144:0/2875516534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98810feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:32.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 --2- 172.21.15.144:0/2875516534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc988108d30 0x7fc988109140 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:32.756 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 --2- 172.21.15.144:0/2875516534 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9881033f0 0x7fc988103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:32.757 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 -- 172.21.15.144:0/2875516534 >> 172.21.15.144:0/2875516534 conn(0x7fc9880fee60 msgr2=0x7fc9881012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:32.757 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 -- 172.21.15.144:0/2875516534 shutdown_connections 2024-07-26T10:57:32.757 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.753+0000 7fc98e000640 1 -- 172.21.15.144:0/2875516534 wait complete. 2024-07-26T10:57:32.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 Processor -- start 2024-07-26T10:57:32.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 -- start start 2024-07-26T10:57:32.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9881033f0 0x7fc988199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:32.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc988108d30 0x7fc98819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:32.758 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc9881132f0 con 0x7fc9881033f0 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc988113170 con 0x7fc988108d30 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.754+0000 7fc98e000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc988113470 con 0x7fc988109680 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98819f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56476/0 (socket says 172.21.15.144:56476) 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 -- 172.21.15.144:0/3996419745 learned_addr learned my addr 172.21.15.144:0/3996419745 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc987e00640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc988108d30 0x7fc98819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98cc00640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9881033f0 0x7fc988199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 -- 172.21.15.144:0/3996419745 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc988108d30 msgr2=0x7fc98819a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc988108d30 0x7fc98819a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 -- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9881033f0 msgr2=0x7fc988199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9881033f0 0x7fc988199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:32.759 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98d600640 1 -- 172.21.15.144:0/3996419745 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98819fcb0 con 0x7fc988109680 2024-07-26T10:57:32.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.755+0000 7fc98cc00640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9881033f0 0x7fc988199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:57:32.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.756+0000 7fc98d600640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98819f580 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc98000e970 tx=0x7fc98000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:32.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.756+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98000ccc0 con 0x7fc988109680 2024-07-26T10:57:32.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.756+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98819ffa0 con 0x7fc988109680 2024-07-26T10:57:32.760 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.756+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc98000ce60 con 0x7fc988109680 2024-07-26T10:57:32.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.756+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980010430 con 0x7fc988109680 2024-07-26T10:57:32.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.756+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881a6700 con 0x7fc988109680 2024-07-26T10:57:32.761 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.757+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc988103800 con 0x7fc988109680 2024-07-26T10:57:32.762 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.760+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc980004fb0 con 0x7fc988109680 2024-07-26T10:57:32.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.761+0000 7fc985600640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9680b3c10 0x7fc9680b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:32.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.761+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8492+0+0 (secure 0 0 0) 0x7fc9800e44d0 con 0x7fc988109680 2024-07-26T10:57:32.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.761+0000 7fc98cc00640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9680b3c10 0x7fc9680b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:32.765 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.762+0000 7fc98cc00640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9680b3c10 0x7fc9680b6120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc978009c30 tx=0x7fc978005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:32.765 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.762+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc980055600 con 0x7fc988109680 2024-07-26T10:57:32.967 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:32.964+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc98819ae70 con 0x7fc988109680 2024-07-26T10:57:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:33 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3996419745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T10:57:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:33 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T10:57:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:33 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3996419745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T10:57:33.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:33 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T10:57:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:33 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3996419745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T10:57:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:33 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T10:57:34.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.043+0000 7fc985600640 1 -- 172.21.15.144:0/3996419745 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) ==== 228+0+0 (secure 0 0 0) 0x7fc9800ab9c0 con 0x7fc988109680 2024-07-26T10:57:34.048 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'cephfs_data_ec' created 2024-07-26T10:57:34.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.050+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9680b3c10 msgr2=0x7fc9680b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:34.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.050+0000 7fc98e000640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9680b3c10 0x7fc9680b6120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc978009c30 tx=0x7fc978005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:34.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 msgr2=0x7fc98819f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:34.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98819f580 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc98000e970 tx=0x7fc98000ed70 comp rx=0 tx=0).stop 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 shutdown_connections 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9680b3c10 0x7fc9680b6120 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc988109680 0x7fc98819f580 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc988108d30 0x7fc98819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 --2- 172.21.15.144:0/3996419745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9881033f0 0x7fc988199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 >> 172.21.15.144:0/3996419745 conn(0x7fc9880fee60 msgr2=0x7fc988101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.051+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 shutdown_connections 2024-07-26T10:57:34.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:34.052+0000 7fc98e000640 1 -- 172.21.15.144:0/3996419745 wait complete. 2024-07-26T10:57:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:34 smithi160 ceph-mon[36715]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T10:57:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:34 smithi160 ceph-mon[36715]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T10:57:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:34 smithi183 ceph-mon[37273]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T10:57:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:34 smithi183 ceph-mon[37273]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T10:57:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:34 smithi144 ceph-mon[32129]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T10:57:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:34 smithi144 ceph-mon[32129]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T10:57:35.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-07-26T10:57:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:35 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T10:57:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:35 smithi144 ceph-mon[32129]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T10:57:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:35 smithi144 ceph-mon[32129]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T10:57:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:35 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T10:57:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:35 smithi183 ceph-mon[37273]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T10:57:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:35 smithi183 ceph-mon[37273]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T10:57:35.487 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:35 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T10:57:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:35 smithi160 ceph-mon[36715]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T10:57:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:35 smithi160 ceph-mon[36715]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T10:57:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:36 smithi144 ceph-mon[32129]: pgmap v306: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-07-26T10:57:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:36 smithi144 ceph-mon[32129]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:36 smithi144 ceph-mon[32129]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T10:57:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:36 smithi183 ceph-mon[37273]: pgmap v306: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-07-26T10:57:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:36 smithi183 ceph-mon[37273]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:36 smithi183 ceph-mon[37273]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T10:57:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:36 smithi160 ceph-mon[36715]: pgmap v306: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-07-26T10:57:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:36 smithi160 ceph-mon[36715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T10:57:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:36 smithi160 ceph-mon[36715]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 -- 172.21.15.144:0/3087423687 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc10af10 msgr2=0x7f0dcc10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 --2- 172.21.15.144:0/3087423687 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc10af10 0x7f0dcc10b320 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009860 tx=0x7f0db802a340 comp rx=0 tx=0).stop 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 -- 172.21.15.144:0/3087423687 shutdown_connections 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 --2- 172.21.15.144:0/3087423687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dcc10b860 0x7f0dcc112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 --2- 172.21.15.144:0/3087423687 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc10af10 0x7f0dcc10b320 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 --2- 172.21.15.144:0/3087423687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:37.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 -- 172.21.15.144:0/3087423687 >> 172.21.15.144:0/3087423687 conn(0x7f0dcc101040 msgr2=0x7f0dcc103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:37.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 -- 172.21.15.144:0/3087423687 shutdown_connections 2024-07-26T10:57:37.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.994+0000 7f0dd2400640 1 -- 172.21.15.144:0/3087423687 wait complete. 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.995+0000 7f0dd2400640 1 Processor -- start 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.995+0000 7f0dd2400640 1 -- start start 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd2400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd2400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc10af10 0x7f0dcc19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd2400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dcc10b860 0x7f0dcc1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd2400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc1154f0 con 0x7f0dcc10b860 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd2400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc115370 con 0x7f0dcc1055d0 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd2400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc115670 con 0x7f0dcc10af10 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd1a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dcc10b860 0x7f0dcc1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd1000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd1000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc19bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36270/0 (socket says 172.21.15.144:36270) 2024-07-26T10:57:37.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd1000640 1 -- 172.21.15.144:0/2651241419 learned_addr learned my addr 172.21.15.144:0/2651241419 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:38.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dcbe00640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc10af10 0x7f0dcc19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:38.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.996+0000 7f0dd1000640 1 -- 172.21.15.144:0/2651241419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc10af10 msgr2=0x7f0dcc19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd1000640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc10af10 0x7f0dcc19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd1000640 1 -- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dcc10b860 msgr2=0x7f0dcc1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd1000640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dcc10b860 0x7f0dcc1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd1000640 1 -- 172.21.15.144:0/2651241419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc1a1eb0 con 0x7f0dcc1055d0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd1000640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc19bf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000e9f0 tx=0x7f0dc000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc000cd80 con 0x7f0dcc1055d0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc1a21a0 con 0x7f0dcc1055d0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dc00054d0 con 0x7f0dcc1055d0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0010690 con 0x7f0dcc1055d0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.997+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dcc1a8900 con 0x7f0dcc1055d0 2024-07-26T10:57:38.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:37.998+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dcc1059e0 con 0x7f0dcc1055d0 2024-07-26T10:57:38.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.001+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f0dc0004e30 con 0x7f0dcc1055d0 2024-07-26T10:57:38.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.003+0000 7f0dc9600640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0db40b3ca0 0x7f0db40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:38.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.003+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8985+0+0 (secure 0 0 0) 0x7f0dc00e4530 con 0x7f0dcc1055d0 2024-07-26T10:57:38.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.003+0000 7f0dcbe00640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0db40b3ca0 0x7f0db40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:38.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.004+0000 7f0dcbe00640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0db40b3ca0 0x7f0db40b61b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0db8005940 tx=0x7f0db80058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:38.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.004+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0dc0055600 con 0x7f0dcc1055d0 2024-07-26T10:57:38.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:38.210+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f0dcc19d070 con 0x7f0dcc1055d0 2024-07-26T10:57:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:38 smithi183 ceph-mon[37273]: pgmap v309: 129 pgs: 4 creating+peering, 11 unknown, 114 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:38 smithi144 ceph-mon[32129]: pgmap v309: 129 pgs: 4 creating+peering, 11 unknown, 114 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:38 smithi160 ceph-mon[36715]: pgmap v309: 129 pgs: 4 creating+peering, 11 unknown, 114 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:39.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.102+0000 7f0dc9600640 1 -- 172.21.15.144:0/2651241419 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7f0dc00ab830 con 0x7f0dcc1055d0 2024-07-26T10:57:39.107 INFO:teuthology.orchestra.run.smithi144.stderr:set pool 4 allow_ec_overwrites to true 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0db40b3ca0 msgr2=0x7f0db40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0db40b3ca0 0x7f0db40b61b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0db8005940 tx=0x7f0db80058d0 comp rx=0 tx=0).stop 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 msgr2=0x7f0dcc19bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc19bf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000e9f0 tx=0x7f0dc000edf0 comp rx=0 tx=0).stop 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 shutdown_connections 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0db40b3ca0 0x7f0db40b61b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dcc10b860 0x7f0dcc1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc10af10 0x7f0dcc19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.105+0000 7f0dd2400640 1 --2- 172.21.15.144:0/2651241419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0dcc1055d0 0x7f0dcc19bf40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.106+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 >> 172.21.15.144:0/2651241419 conn(0x7f0dcc101040 msgr2=0x7f0dcc103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:39.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.106+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 shutdown_connections 2024-07-26T10:57:39.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:39.106+0000 7f0dd2400640 1 -- 172.21.15.144:0/2651241419 wait complete. 2024-07-26T10:57:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:39 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2651241419' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T10:57:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:39 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T10:57:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:39 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2651241419' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T10:57:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:39 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T10:57:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:39 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2651241419' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T10:57:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:39 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T10:57:39.987 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-07-26T10:57:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:40 smithi144 ceph-mon[32129]: pgmap v310: 129 pgs: 4 creating+peering, 4 unknown, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:40 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T10:57:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:40 smithi144 ceph-mon[32129]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T10:57:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:40 smithi183 ceph-mon[37273]: pgmap v310: 129 pgs: 4 creating+peering, 4 unknown, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:40 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T10:57:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:40 smithi183 ceph-mon[37273]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T10:57:40.428 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:40 smithi160 ceph-mon[36715]: pgmap v310: 129 pgs: 4 creating+peering, 4 unknown, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:40 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T10:57:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:40 smithi160 ceph-mon[36715]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T10:57:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:41 smithi144 ceph-mon[32129]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:41 smithi160 ceph-mon[36715]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:41 smithi183 ceph-mon[37273]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:42.539 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 -- 172.21.15.144:0/397278661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810f330 msgr2=0x7fe15810f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 --2- 172.21.15.144:0/397278661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810f330 0x7fe15810f740 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe144009900 tx=0x7fe14402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 -- 172.21.15.144:0/397278661 shutdown_connections 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 --2- 172.21.15.144:0/397278661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15810fc80 0x7fe1581164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 --2- 172.21.15.144:0/397278661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810f330 0x7fe15810f740 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 --2- 172.21.15.144:0/397278661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1581099f0 0x7fe158109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 -- 172.21.15.144:0/397278661 >> 172.21.15.144:0/397278661 conn(0x7fe158078ff0 msgr2=0x7fe158107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 -- 172.21.15.144:0/397278661 shutdown_connections 2024-07-26T10:57:42.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.537+0000 7fe15ea00640 1 -- 172.21.15.144:0/397278661 wait complete. 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.538+0000 7fe15ea00640 1 Processor -- start 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.538+0000 7fe15ea00640 1 -- start start 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15ea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 0x7fe1581a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15ea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15810f330 0x7fe1581a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15ea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810fc80 0x7fe1581a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15ea00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe158119940 con 0x7fe15810f330 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15ea00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe1581197c0 con 0x7fe15810fc80 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15ea00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe158119ac0 con 0x7fe1581099f0 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15cc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15810f330 0x7fe1581a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:42.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 0x7fe1581a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 0x7fe1581a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:41528/0 (socket says 172.21.15.144:41528) 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15d600640 1 -- 172.21.15.144:0/3915393545 learned_addr learned my addr 172.21.15.144:0/3915393545 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.539+0000 7fe15e000640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810fc80 0x7fe1581a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15cc00640 1 -- 172.21.15.144:0/3915393545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 msgr2=0x7fe1581a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15cc00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 0x7fe1581a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15cc00640 1 -- 172.21.15.144:0/3915393545 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810fc80 msgr2=0x7fe1581a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15cc00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810fc80 0x7fe1581a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15cc00640 1 -- 172.21.15.144:0/3915393545 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1581a62e0 con 0x7fe15810f330 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15d600640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 0x7fe1581a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:57:42.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15cc00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15810f330 0x7fe1581a0910 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe14402a910 tx=0x7fe144031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe14403f070 con 0x7fe15810f330 2024-07-26T10:57:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe144004490 con 0x7fe15810f330 2024-07-26T10:57:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.540+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1581a6570 con 0x7fe15810f330 2024-07-26T10:57:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.541+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144035070 con 0x7fe15810f330 2024-07-26T10:57:42.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.541+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1581acd30 con 0x7fe15810f330 2024-07-26T10:57:42.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.542+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe15810ac70 con 0x7fe15810f330 2024-07-26T10:57:42.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.545+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe14403b460 con 0x7fe15810f330 2024-07-26T10:57:42.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.546+0000 7fe156000640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1400b3c20 0x7fe1400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:42.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.546+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8985+0+0 (secure 0 0 0) 0x7fe144104590 con 0x7fe15810f330 2024-07-26T10:57:42.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.547+0000 7fe15d600640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1400b3c20 0x7fe1400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:42.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.547+0000 7fe15d600640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1400b3c20 0x7fe1400b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0095f0 tx=0x7fe14c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:42.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.549+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe1440cb2a0 con 0x7fe15810f330 2024-07-26T10:57:42.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.789+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe158111a70 con 0x7fe15810f330 2024-07-26T10:57:42.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.843+0000 7fe156000640 1 -- 172.21.15.144:0/3915393545 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7fe1440cf7f0 con 0x7fe15810f330 2024-07-26T10:57:42.846 INFO:teuthology.orchestra.run.smithi144.stderr:added data pool 4 to fsmap 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.848+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1400b3c20 msgr2=0x7fe1400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.848+0000 7fe15ea00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1400b3c20 0x7fe1400b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0095f0 tx=0x7fe14c005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.848+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15810f330 msgr2=0x7fe1581a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.848+0000 7fe15ea00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15810f330 0x7fe1581a0910 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe14402a910 tx=0x7fe144031a20 comp rx=0 tx=0).stop 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.849+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 shutdown_connections 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.849+0000 7fe15ea00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1400b3c20 0x7fe1400b6130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.849+0000 7fe15ea00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe15810fc80 0x7fe1581a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.849+0000 7fe15ea00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15810f330 0x7fe1581a0910 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.849+0000 7fe15ea00640 1 --2- 172.21.15.144:0/3915393545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1581099f0 0x7fe1581a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:42.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.849+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 >> 172.21.15.144:0/3915393545 conn(0x7fe158078ff0 msgr2=0x7fe158108010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:42.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.850+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 shutdown_connections 2024-07-26T10:57:42.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:42.850+0000 7fe15ea00640 1 -- 172.21.15.144:0/3915393545 wait complete. 2024-07-26T10:57:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:42 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3915393545' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T10:57:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:42 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3915393545' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T10:57:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:42 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3915393545' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T10:57:43.701 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:57:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3915393545' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-07-26T10:57:43.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T10:57:43.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:43 smithi144 ceph-mon[32129]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3915393545' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T10:57:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T10:57:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:57:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T10:57:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T10:57:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:43 smithi183 ceph-mon[37273]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:44.154 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3915393545' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-07-26T10:57:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T10:57:44.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:43 smithi160 ceph-mon[36715]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T10:57:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T10:57:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-07-26T10:57:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-07-26T10:57:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-07-26T10:57:45.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:44 smithi183 ceph-mon[37273]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T10:57:45.148 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T10:57:45.148 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-07-26T10:57:45.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:44 smithi144 ceph-mon[32129]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T10:57:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T10:57:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-07-26T10:57:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:44 smithi160 ceph-mon[36715]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T10:57:46.085 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:45 smithi144 ceph-mon[32129]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T10:57:46.085 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:45 smithi144 ceph-mon[32129]: pgmap v317: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:45 smithi160 ceph-mon[36715]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T10:57:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:45 smithi160 ceph-mon[36715]: pgmap v317: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:45 smithi183 ceph-mon[37273]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T10:57:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:45 smithi183 ceph-mon[37273]: pgmap v317: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 -- 172.21.15.144:0/1054202793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 msgr2=0x7f7954112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 --2- 172.21.15.144:0/1054202793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 0x7f7954112090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f794c009ca0 tx=0x7f794c02a8c0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 -- 172.21.15.144:0/1054202793 shutdown_connections 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 --2- 172.21.15.144:0/1054202793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 0x7f7954112090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 --2- 172.21.15.144:0/1054202793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 --2- 172.21.15.144:0/1054202793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79541055d0 0x7f79541059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 -- 172.21.15.144:0/1054202793 >> 172.21.15.144:0/1054202793 conn(0x7f7954101040 msgr2=0x7f7954103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 -- 172.21.15.144:0/1054202793 shutdown_connections 2024-07-26T10:57:46.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.386+0000 7f7953e00640 1 -- 172.21.15.144:0/1054202793 wait complete. 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.388+0000 7f7953e00640 1 Processor -- start 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.388+0000 7f7953e00640 1 -- start start 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.388+0000 7f7953e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79541055d0 0x7f795419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.388+0000 7f7953e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7953e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 0x7f79541a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7953e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7954115580 con 0x7f795410b860 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7953e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7954115400 con 0x7f79541055d0 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7953e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7954115700 con 0x7f795410af10 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7953400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 0x7f79541a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:46.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:41540/0 (socket says 172.21.15.144:41540) 2024-07-26T10:57:46.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 -- 172.21.15.144:0/3189430715 learned_addr learned my addr 172.21.15.144:0/3189430715 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:46.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952a00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79541055d0 0x7f795419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 -- 172.21.15.144:0/3189430715 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79541055d0 msgr2=0x7f795419bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79541055d0 0x7f795419bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 -- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 msgr2=0x7f79541a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 0x7f79541a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.389+0000 7f7952000640 1 -- 172.21.15.144:0/3189430715 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79541a1ee0 con 0x7f795410af10 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.390+0000 7f7952000640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795419c510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f794000e9c0 tx=0x7f794000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.390+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794000ccc0 con 0x7f795410af10 2024-07-26T10:57:46.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.390+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79400054d0 con 0x7f795410af10 2024-07-26T10:57:46.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.390+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79541a21d0 con 0x7f795410af10 2024-07-26T10:57:46.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.390+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940010430 con 0x7f795410af10 2024-07-26T10:57:46.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.390+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79541a8930 con 0x7f795410af10 2024-07-26T10:57:46.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.391+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79541059e0 con 0x7f795410af10 2024-07-26T10:57:46.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.394+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f7940005000 con 0x7f795410af10 2024-07-26T10:57:46.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.395+0000 7f794b400640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79380b3c20 0x7f79380b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:46.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.396+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f79400e45a0 con 0x7f795410af10 2024-07-26T10:57:46.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.396+0000 7f7952a00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79380b3c20 0x7f79380b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:46.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.396+0000 7f7952a00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79380b3c20 0x7f79380b6130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f79440095f0 tx=0x7f7944005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:46.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.397+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7940055600 con 0x7f795410af10 2024-07-26T10:57:46.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.633+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f795419d0a0 con 0x7f795410af10 2024-07-26T10:57:46.636 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.635+0000 7f794b400640 1 -- 172.21.15.144:0/3189430715 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11860 (secure 0 0 0) 0x7f795419d0a0 con 0x7f795410af10 2024-07-26T10:57:46.637 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:46.637 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":16,"btime":"2024-07-26T10:57:42:841265+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24677,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24691,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24701,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24703,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24715,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:57:42.790991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24728,"qdb_cluster":[24728]},"id":1}]} 2024-07-26T10:57:46.638 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 16 2024-07-26T10:57:46.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.637+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79380b3c20 msgr2=0x7f79380b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:46.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.637+0000 7f7953e00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79380b3c20 0x7f79380b6130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f79440095f0 tx=0x7f7944005f90 comp rx=0 tx=0).stop 2024-07-26T10:57:46.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.637+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 msgr2=0x7f795419c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:46.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.637+0000 7f7953e00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795419c510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f794000e9c0 tx=0x7f794000ee80 comp rx=0 tx=0).stop 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 shutdown_connections 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79380b3c20 0x7f79380b6130 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795410b860 0x7f79541a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f795410af10 0x7f795419c510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 --2- 172.21.15.144:0/3189430715 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79541055d0 0x7f795419bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 >> 172.21.15.144:0/3189430715 conn(0x7f7954101040 msgr2=0x7f795410a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 shutdown_connections 2024-07-26T10:57:46.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:46.638+0000 7f7953e00640 1 -- 172.21.15.144:0/3189430715 wait complete. 2024-07-26T10:57:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:46 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3189430715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:46 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3189430715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:46 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3189430715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:47.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:57:47.889 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:48.193 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:47 smithi144 ceph-mon[32129]: pgmap v318: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 96 B/s, 0 objects/s recovering 2024-07-26T10:57:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:47 smithi160 ceph-mon[36715]: pgmap v318: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 96 B/s, 0 objects/s recovering 2024-07-26T10:57:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:47 smithi183 ceph-mon[37273]: pgmap v318: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 96 B/s, 0 objects/s recovering 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 -- 172.21.15.144:0/73923365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010f330 msgr2=0x7f4d4010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 --2- 172.21.15.144:0/73923365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010f330 0x7f4d4010f740 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c02a3f0 comp rx=0 tx=0).stop 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 -- 172.21.15.144:0/73923365 shutdown_connections 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 --2- 172.21.15.144:0/73923365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d4010fc80 0x7f4d401164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 --2- 172.21.15.144:0/73923365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010f330 0x7f4d4010f740 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 --2- 172.21.15.144:0/73923365 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d40109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:49.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.934+0000 7f4d46a00640 1 -- 172.21.15.144:0/73923365 >> 172.21.15.144:0/73923365 conn(0x7f4d40078ff0 msgr2=0x7f4d40107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:49.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.935+0000 7f4d46a00640 1 -- 172.21.15.144:0/73923365 shutdown_connections 2024-07-26T10:57:49.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.935+0000 7f4d46a00640 1 -- 172.21.15.144:0/73923365 wait complete. 2024-07-26T10:57:49.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 Processor -- start 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 -- start start 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d401a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d4010f330 0x7f4d401a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010fc80 0x7f4d401a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4d40119940 con 0x7f4d4010fc80 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4d401197c0 con 0x7f4d401099f0 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.936+0000 7f4d46a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4d40119ac0 con 0x7f4d4010f330 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d401a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d46000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010fc80 0x7f4d401a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d401a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:40410/0 (socket says 172.21.15.144:40410) 2024-07-26T10:57:49.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 -- 172.21.15.144:0/489879982 learned_addr learned my addr 172.21.15.144:0/489879982 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:49.940 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d44c00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d4010f330 0x7f4d401a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:49.940 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 -- 172.21.15.144:0/489879982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d4010f330 msgr2=0x7f4d401a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d4010f330 0x7f4d401a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 -- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010fc80 msgr2=0x7f4d401a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010fc80 0x7f4d401a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d45600640 1 -- 172.21.15.144:0/489879982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d401a62e0 con 0x7f4d401099f0 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.937+0000 7f4d46000640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010fc80 0x7f4d401a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.938+0000 7f4d45600640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d401a0370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000ebc0 tx=0x7f4d3000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.938+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30010630 con 0x7f4d401099f0 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.938+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d401a65d0 con 0x7f4d401099f0 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.938+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d30010c80 con 0x7f4d401099f0 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.938+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30018b60 con 0x7f4d401099f0 2024-07-26T10:57:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.938+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d401acd30 con 0x7f4d401099f0 2024-07-26T10:57:49.942 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.939+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d40109e00 con 0x7f4d401099f0 2024-07-26T10:57:49.943 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.942+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4d30018d00 con 0x7f4d401099f0 2024-07-26T10:57:49.945 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.943+0000 7f4d3e000640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d240b3c20 0x7f4d240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:49.945 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.944+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f4d300e4b40 con 0x7f4d401099f0 2024-07-26T10:57:49.945 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.944+0000 7f4d44c00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d240b3c20 0x7f4d240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:49.946 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.944+0000 7f4d44c00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d240b3c20 0x7f4d240b6130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009230 tx=0x7f4d2c031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:49.946 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:49.945+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4d300ab6b0 con 0x7f4d401099f0 2024-07-26T10:57:50.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.148+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d401a14a0 con 0x7f4d401099f0 2024-07-26T10:57:50.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.150+0000 7f4d3e000640 1 -- 172.21.15.144:0/489879982 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25047 (secure 0 0 0) 0x7f4d300afc00 con 0x7f4d401099f0 2024-07-26T10:57:50.153 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:50.153 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":88,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:57:44.875941+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:57:50.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d240b3c20 msgr2=0x7f4d240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:50.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d240b3c20 0x7f4d240b6130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009230 tx=0x7f4d2c031400 comp rx=0 tx=0).stop 2024-07-26T10:57:50.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 msgr2=0x7f4d401a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:50.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d401a0370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000ebc0 tx=0x7f4d3000c6a0 comp rx=0 tx=0).stop 2024-07-26T10:57:50.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 shutdown_connections 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d240b3c20 0x7f4d240b6130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d4010fc80 0x7f4d401a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d4010f330 0x7f4d401a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 --2- 172.21.15.144:0/489879982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d401099f0 0x7f4d401a0370 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 >> 172.21.15.144:0/489879982 conn(0x7f4d40078ff0 msgr2=0x7f4d400797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 shutdown_connections 2024-07-26T10:57:50.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:50.154+0000 7f4d46a00640 1 -- 172.21.15.144:0/489879982 wait complete. 2024-07-26T10:57:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:50 smithi160 ceph-mon[36715]: pgmap v319: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 95 B/s, 0 objects/s recovering 2024-07-26T10:57:50.363 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:50 smithi144 ceph-mon[32129]: pgmap v319: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 95 B/s, 0 objects/s recovering 2024-07-26T10:57:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:50 smithi183 ceph-mon[37273]: pgmap v319: 129 pgs: 2 peering, 127 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 95 B/s, 0 objects/s recovering 2024-07-26T10:57:50.942 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:57:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:51 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/489879982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:57:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:51 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/489879982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:57:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:51 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/489879982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:57:51.406 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:52 smithi144 ceph-mon[32129]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 72 B/s, 0 objects/s recovering 2024-07-26T10:57:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:52 smithi160 ceph-mon[36715]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 72 B/s, 0 objects/s recovering 2024-07-26T10:57:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:52 smithi183 ceph-mon[37273]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 72 B/s, 0 objects/s recovering 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 -- 172.21.15.144:0/2360272279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 msgr2=0x7f8dd80755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2360272279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd80755f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009ca0 tx=0x7f8dc802a8c0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 -- 172.21.15.144:0/2360272279 shutdown_connections 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2360272279 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 0x7f8dd8111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2360272279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 0x7f8dd810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2360272279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd80755f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 -- 172.21.15.144:0/2360272279 >> 172.21.15.144:0/2360272279 conn(0x7f8dd80fee70 msgr2=0x7f8dd81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 -- 172.21.15.144:0/2360272279 shutdown_connections 2024-07-26T10:57:53.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.656+0000 7f8ddd400640 1 -- 172.21.15.144:0/2360272279 wait complete. 2024-07-26T10:57:53.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.658+0000 7f8ddd400640 1 Processor -- start 2024-07-26T10:57:53.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.658+0000 7f8ddd400640 1 -- start start 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd819c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 0x7f8dd819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddd400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 0x7f8dd81a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddd400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd81154f0 con 0x7f8dd810f940 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddd400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd8115370 con 0x7f8dd8075b30 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddd400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd8115670 con 0x7f8dd80751e0 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd819c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dcf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 0x7f8dd819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd819c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:45544/0 (socket says 172.21.15.144:45544) 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dd7e00640 1 -- 172.21.15.144:0/2712971612 learned_addr learned my addr 172.21.15.144:0/2712971612 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8ddca00640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 0x7f8dd81a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dcf400640 1 -- 172.21.15.144:0/2712971612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 msgr2=0x7f8dd819c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dcf400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd819c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.659+0000 7f8dcf400640 1 -- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 msgr2=0x7f8dd81a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8dcf400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 0x7f8dd81a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8dcf400640 1 -- 172.21.15.144:0/2712971612 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd81a1f70 con 0x7f8dd8075b30 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8ddca00640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 0x7f8dd81a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8dcf400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 0x7f8dd819c5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e9c0 tx=0x7f8dc000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000ccc0 con 0x7f8dd8075b30 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dc00035f0 con 0x7f8dd8075b30 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd81a2260 con 0x7f8dd8075b30 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000ce60 con 0x7f8dd8075b30 2024-07-26T10:57:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.660+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd81a89c0 con 0x7f8dd8075b30 2024-07-26T10:57:53.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.662+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd80755f0 con 0x7f8dd8075b30 2024-07-26T10:57:53.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.664+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8dc0004000 con 0x7f8dd8075b30 2024-07-26T10:57:53.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.665+0000 7f8dd5600640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8dbc0b3ca0 0x7f8dbc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:53.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.666+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f8dc00e44b0 con 0x7f8dd8075b30 2024-07-26T10:57:53.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.666+0000 7f8dd7e00640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8dbc0b3ca0 0x7f8dbc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:53.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.667+0000 7f8dd7e00640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8dbc0b3ca0 0x7f8dbc0b61b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8004af0 tx=0x7f8dc8009360 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:53.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.667+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8dc00ab6d0 con 0x7f8dd8075b30 2024-07-26T10:57:53.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.870+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8dd819d130 con 0x7f8dd8075b30 2024-07-26T10:57:53.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.872+0000 7f8dd5600640 1 -- 172.21.15.144:0/2712971612 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25047 (secure 0 0 0) 0x7f8dd819d130 con 0x7f8dd8075b30 2024-07-26T10:57:53.875 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:53.875 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":88,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:57:44.875941+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:57:53.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8dbc0b3ca0 msgr2=0x7f8dbc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8dbc0b3ca0 0x7f8dbc0b61b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8004af0 tx=0x7f8dc8009360 comp rx=0 tx=0).stop 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 msgr2=0x7f8dd819c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 0x7f8dd819c5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e9c0 tx=0x7f8dc000ee80 comp rx=0 tx=0).stop 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 shutdown_connections 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8dbc0b3ca0 0x7f8dbc0b61b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dd810f940 0x7f8dd81a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dd8075b30 0x7f8dd819c5a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.875+0000 7f8ddd400640 1 --2- 172.21.15.144:0/2712971612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dd80751e0 0x7f8dd819c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.876+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 >> 172.21.15.144:0/2712971612 conn(0x7f8dd80fee70 msgr2=0x7f8dd8108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.876+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 shutdown_connections 2024-07-26T10:57:53.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:53.876+0000 7f8ddd400640 1 -- 172.21.15.144:0/2712971612 wait complete. 2024-07-26T10:57:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:54 smithi144 ceph-mon[32129]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 64 B/s, 0 objects/s recovering 2024-07-26T10:57:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:54 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2712971612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:57:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:54 smithi160 ceph-mon[36715]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 64 B/s, 0 objects/s recovering 2024-07-26T10:57:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:54 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2712971612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:57:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:54 smithi183 ceph-mon[37273]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 64 B/s, 0 objects/s recovering 2024-07-26T10:57:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:54 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2712971612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:57:54.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:57:55.183 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:57:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:56 smithi160 ceph-mon[36715]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 57 B/s, 0 objects/s recovering 2024-07-26T10:57:56.346 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:56 smithi144 ceph-mon[32129]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 57 B/s, 0 objects/s recovering 2024-07-26T10:57:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:56 smithi183 ceph-mon[37273]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 57 B/s, 0 objects/s recovering 2024-07-26T10:57:57.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.379+0000 7f8dce600640 1 -- 172.21.15.144:0/652271780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc810b860 msgr2=0x7f8dc8112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:57.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.379+0000 7f8dce600640 1 --2- 172.21.15.144:0/652271780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc810b860 0x7f8dc8112090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009be0 tx=0x7f8dc002a6c0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 -- 172.21.15.144:0/652271780 shutdown_connections 2024-07-26T10:57:57.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 --2- 172.21.15.144:0/652271780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc810b860 0x7f8dc8112090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 --2- 172.21.15.144:0/652271780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc810af10 0x7f8dc810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 --2- 172.21.15.144:0/652271780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc81055d0 0x7f8dc81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 -- 172.21.15.144:0/652271780 >> 172.21.15.144:0/652271780 conn(0x7f8dc8101040 msgr2=0x7f8dc8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 -- 172.21.15.144:0/652271780 shutdown_connections 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.380+0000 7f8dce600640 1 -- 172.21.15.144:0/652271780 wait complete. 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.381+0000 7f8dce600640 1 Processor -- start 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.381+0000 7f8dce600640 1 -- start start 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dce600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 0x7f8dc819bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:57.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dce600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc810af10 0x7f8dc819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:57.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dce600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc810b860 0x7f8dc81a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:57.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dce600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8115510 con 0x7f8dc810b860 2024-07-26T10:57:57.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dce600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8115390 con 0x7f8dc81055d0 2024-07-26T10:57:57.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dce600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8115690 con 0x7f8dc810af10 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 0x7f8dc819bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dbfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc810af10 0x7f8dc819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 0x7f8dc819bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:40448/0 (socket says 172.21.15.144:40448) 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 -- 172.21.15.144:0/4191879272 learned_addr learned my addr 172.21.15.144:0/4191879272 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcdc00640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc810b860 0x7f8dc81a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 -- 172.21.15.144:0/4191879272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc810af10 msgr2=0x7f8dc819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc810af10 0x7f8dc819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 -- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc810b860 msgr2=0x7f8dc81a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc810b860 0x7f8dc81a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dcd200640 1 -- 172.21.15.144:0/4191879272 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc81a1e30 con 0x7f8dc81055d0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.382+0000 7f8dbfe00640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc810af10 0x7f8dc819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.383+0000 7f8dcd200640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 0x7f8dc819bec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8db800ea60 tx=0x7f8db800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.383+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db800cde0 con 0x7f8dc81055d0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.383+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8db80035f0 con 0x7f8dc81055d0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.383+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc81a2120 con 0x7f8dc81055d0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.383+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db8010690 con 0x7f8dc81055d0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.383+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc81a8880 con 0x7f8dc81055d0 2024-07-26T10:57:57.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.384+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc81059e0 con 0x7f8dc81055d0 2024-07-26T10:57:57.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.387+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8db80037a0 con 0x7f8dc81055d0 2024-07-26T10:57:57.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.388+0000 7f8dc6000640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8da80b3ca0 0x7f8da80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:57:57.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.389+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f8db80e4750 con 0x7f8dc81055d0 2024-07-26T10:57:57.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.389+0000 7f8dbfe00640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8da80b3ca0 0x7f8da80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:57.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.389+0000 7f8dbfe00640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8da80b3ca0 0x7f8da80b61b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8db00096c0 tx=0x7f8db0009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:57:57.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.390+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8db80ab910 con 0x7f8dc81055d0 2024-07-26T10:57:57.636 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.634+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dc819cff0 con 0x7f8dc81055d0 2024-07-26T10:57:57.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.636+0000 7f8dc6000640 1 -- 172.21.15.144:0/4191879272 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11860 (secure 0 0 0) 0x7f8db80ab210 con 0x7f8dc81055d0 2024-07-26T10:57:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:57:57.638 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":16,"btime":"2024-07-26T10:57:42:841265+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24677,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24691,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24701,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24703,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24715,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:57:42.790991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24728,"qdb_cluster":[24728]},"id":1}]} 2024-07-26T10:57:57.638 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 16 2024-07-26T10:57:57.641 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8da80b3ca0 msgr2=0x7f8da80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:57.641 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8da80b3ca0 0x7f8da80b61b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8db00096c0 tx=0x7f8db0009260 comp rx=0 tx=0).stop 2024-07-26T10:57:57.641 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 msgr2=0x7f8dc819bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:57:57.641 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 0x7f8dc819bec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8db800ea60 tx=0x7f8db800ef20 comp rx=0 tx=0).stop 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 shutdown_connections 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8da80b3ca0 0x7f8da80b61b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc810b860 0x7f8dc81a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8dc810af10 0x7f8dc819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 --2- 172.21.15.144:0/4191879272 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8dc81055d0 0x7f8dc819bec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.639+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 >> 172.21.15.144:0/4191879272 conn(0x7f8dc8101040 msgr2=0x7f8dc8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.640+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 shutdown_connections 2024-07-26T10:57:57.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:57:57.640+0000 7f8dce600640 1 -- 172.21.15.144:0/4191879272 wait complete. 2024-07-26T10:57:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:58 smithi160 ceph-mon[36715]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 48 B/s, 0 objects/s recovering 2024-07-26T10:57:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:57:58 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/4191879272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:58.384 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:58 smithi144 ceph-mon[32129]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 48 B/s, 0 objects/s recovering 2024-07-26T10:57:58.384 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:57:58 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/4191879272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:58 smithi183 ceph-mon[37273]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 48 B/s, 0 objects/s recovering 2024-07-26T10:57:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:57:58 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/4191879272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:57:58.462 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:57:58.934 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:00.200 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:00 smithi144 ceph-mon[32129]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:00 smithi160 ceph-mon[36715]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:00 smithi183 ceph-mon[37273]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 -- 172.21.15.144:0/1336250611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 msgr2=0x7f463410ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 --2- 172.21.15.144:0/1336250611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 0x7f463410ed60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f462c009860 tx=0x7f462c02a340 comp rx=0 tx=0).stop 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 -- 172.21.15.144:0/1336250611 shutdown_connections 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 --2- 172.21.15.144:0/1336250611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 0x7f463410ed60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 --2- 172.21.15.144:0/1336250611 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4634107be0 0x7f4634107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 --2- 172.21.15.144:0/1336250611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46341022a0 0x7f46341026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 -- 172.21.15.144:0/1336250611 >> 172.21.15.144:0/1336250611 conn(0x7f46340fdcf0 msgr2=0x7f4634100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 -- 172.21.15.144:0/1336250611 shutdown_connections 2024-07-26T10:58:01.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.065+0000 7f463ae00640 1 -- 172.21.15.144:0/1336250611 wait complete. 2024-07-26T10:58:01.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.067+0000 7f463ae00640 1 Processor -- start 2024-07-26T10:58:01.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.067+0000 7f463ae00640 1 -- start start 2024-07-26T10:58:01.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463ae00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 0x7f4634111e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:01.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463ae00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4634107be0 0x7f4634112390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:01.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463ae00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 0x7f46341128d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:01.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463ae00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f463406ad60 con 0x7f4634108530 2024-07-26T10:58:01.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463ae00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f463406abe0 con 0x7f4634107be0 2024-07-26T10:58:01.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463ae00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f463406aee0 con 0x7f46341022a0 2024-07-26T10:58:01.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f463a400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 0x7f46341128d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f4639a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 0x7f4634111e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f4639000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4634107be0 0x7f4634112390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f4639a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 0x7f4634111e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58604/0 (socket says 172.21.15.144:58604) 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f4639000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4634107be0 0x7f4634112390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:34578/0 (socket says 172.21.15.144:34578) 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.068+0000 7f4639a00640 1 -- 172.21.15.144:0/1412007259 learned_addr learned my addr 172.21.15.144:0/1412007259 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4639a00640 1 -- 172.21.15.144:0/1412007259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4634107be0 msgr2=0x7f4634112390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4639a00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4634107be0 0x7f4634112390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4639a00640 1 -- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 msgr2=0x7f46341128d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4639a00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 0x7f46341128d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4639a00640 1 -- 172.21.15.144:0/1412007259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46341a7ff0 con 0x7f46341022a0 2024-07-26T10:58:01.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4639a00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 0x7f4634111e50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f462800ebc0 tx=0x7f462800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:01.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628010430 con 0x7f46341022a0 2024-07-26T10:58:01.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4628010a80 con 0x7f46341022a0 2024-07-26T10:58:01.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628018a50 con 0x7f46341022a0 2024-07-26T10:58:01.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.069+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46341a82e0 con 0x7f46341022a0 2024-07-26T10:58:01.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.070+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46341a87d0 con 0x7f46341022a0 2024-07-26T10:58:01.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.071+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46341026b0 con 0x7f46341022a0 2024-07-26T10:58:01.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.073+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4628010c30 con 0x7f46341022a0 2024-07-26T10:58:01.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.074+0000 7f4632000640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f461c0b3ca0 0x7f461c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:01.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.075+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f46280e4710 con 0x7f46341022a0 2024-07-26T10:58:01.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.075+0000 7f4639000640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f461c0b3ca0 0x7f461c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.075+0000 7f4639000640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f461c0b3ca0 0x7f461c0b61b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4634107a00 tx=0x7f4620009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:01.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.078+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f46280ab8d0 con 0x7f46341022a0 2024-07-26T10:58:01.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.278+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46341a8bf0 con 0x7f46341022a0 2024-07-26T10:58:01.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.280+0000 7f4632000640 1 -- 172.21.15.144:0/1412007259 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25047 (secure 0 0 0) 0x7f46280ab1d0 con 0x7f46341022a0 2024-07-26T10:58:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:58:01.283 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":88,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:57:44.875941+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:58:01.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f461c0b3ca0 msgr2=0x7f461c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:01.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f461c0b3ca0 0x7f461c0b61b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4634107a00 tx=0x7f4620009260 comp rx=0 tx=0).stop 2024-07-26T10:58:01.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 msgr2=0x7f4634111e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:01.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 0x7f4634111e50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f462800ebc0 tx=0x7f462800c2d0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 shutdown_connections 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f461c0b3ca0 0x7f461c0b61b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4634108530 0x7f46341128d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4634107be0 0x7f4634112390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 --2- 172.21.15.144:0/1412007259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46341022a0 0x7f4634111e50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 >> 172.21.15.144:0/1412007259 conn(0x7f46340fdcf0 msgr2=0x7f46341059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.283+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 shutdown_connections 2024-07-26T10:58:01.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:01.284+0000 7f463ae00640 1 -- 172.21.15.144:0/1412007259 wait complete. 2024-07-26T10:58:02.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T10:57:25.002914+0000', 'modified': '2024-07-26T10:57:42.790991+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24728}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24728': {'gid': 24728, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.160:6839/1074049609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6838', 'nonce': 1074049609}, {'type': 'v1', 'addr': '172.21.15.160:6839', 'nonce': 1074049609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24728, 'qdb_cluster': [24728]} 2024-07-26T10:58:02.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T10:58:02.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T10:58:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:02 smithi183 ceph-mon[37273]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:02 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1412007259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:02 smithi144 ceph-mon[32129]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:02 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1412007259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:02.546 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:02 smithi160 ceph-mon[36715]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:02 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1412007259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:04 smithi144 ceph-mon[32129]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:04 smithi183 ceph-mon[37273]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:04 smithi160 ceph-mon[36715]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:04.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 -- 172.21.15.144:0/3196507164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e41055f0 msgr2=0x7f03e4105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 --2- 172.21.15.144:0/3196507164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e41055f0 0x7f03e4105a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009900 tx=0x7f03d402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 -- 172.21.15.144:0/3196507164 shutdown_connections 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 --2- 172.21.15.144:0/3196507164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e410b880 0x7f03e41120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 --2- 172.21.15.144:0/3196507164 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03e410af30 0x7f03e410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 --2- 172.21.15.144:0/3196507164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e41055f0 0x7f03e4105a00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 -- 172.21.15.144:0/3196507164 >> 172.21.15.144:0/3196507164 conn(0x7f03e41010a0 msgr2=0x7f03e41034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 -- 172.21.15.144:0/3196507164 shutdown_connections 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.790+0000 7f03e8e00640 1 -- 172.21.15.144:0/3196507164 wait complete. 2024-07-26T10:58:04.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 Processor -- start 2024-07-26T10:58:04.794 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 -- start start 2024-07-26T10:58:04.794 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e41055f0 0x7f03e419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e410af30 0x7f03e419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03e410b880 0x7f03e41a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4115550 con 0x7f03e410af30 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41153d0 con 0x7f03e410b880 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.792+0000 7f03e8e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41156d0 con 0x7f03e41055f0 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e410af30 0x7f03e419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e41055f0 0x7f03e419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e410af30 0x7f03e419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48346/0 (socket says 172.21.15.144:48346) 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e2a00640 1 -- 172.21.15.144:0/2761955923 learned_addr learned my addr 172.21.15.144:0/2761955923 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:04.795 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03e410b880 0x7f03e41a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3400640 1 -- 172.21.15.144:0/2761955923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03e410b880 msgr2=0x7f03e41a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3400640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03e410b880 0x7f03e41a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3400640 1 -- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e410af30 msgr2=0x7f03e419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3400640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e410af30 0x7f03e419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.793+0000 7f03e3400640 1 -- 172.21.15.144:0/2761955923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e41a1f30 con 0x7f03e41055f0 2024-07-26T10:58:04.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.794+0000 7f03e3400640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e41055f0 0x7f03e419bfc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f03d402a910 tx=0x7f03d4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:04.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.794+0000 7f03dbe00640 1 -- 172.21.15.144:0/2761955923 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d403f030 con 0x7f03e41055f0 2024-07-26T10:58:04.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.794+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e41a21c0 con 0x7f03e41055f0 2024-07-26T10:58:04.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.794+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e41a8980 con 0x7f03e41055f0 2024-07-26T10:58:04.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.794+0000 7f03dbe00640 1 -- 172.21.15.144:0/2761955923 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d4003aa0 con 0x7f03e41055f0 2024-07-26T10:58:04.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.794+0000 7f03dbe00640 1 -- 172.21.15.144:0/2761955923 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4049060 con 0x7f03e41055f0 2024-07-26T10:58:04.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.797+0000 7f03f1054740 1 Processor -- start 2024-07-26T10:58:04.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 -- start start 2024-07-26T10:58:04.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d302760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d302ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 0x56309d3031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56309d278950 con 0x56309cf340a0 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56309d267e50 con 0x56309cf3c280 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03f1054740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56309d26be90 con 0x56309cf5cf60 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03dbe00640 1 -- 172.21.15.144:0/2761955923 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f03d4042030 con 0x7f03e41055f0 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03e2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 0x56309d3031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d302760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03e2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 0x56309d3031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:34612/0 (socket says 172.21.15.144:34612) 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.798+0000 7f03e2a00640 1 -- 172.21.15.144:0/2496892386 learned_addr learned my addr 172.21.15.144:0/2496892386 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3400640 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d302ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3e00640 1 -- 172.21.15.144:0/2496892386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 msgr2=0x56309d3031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3e00640 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 0x56309d3031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3e00640 1 -- 172.21.15.144:0/2496892386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 msgr2=0x56309d302ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3e00640 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d302ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3e00640 1 -- 172.21.15.144:0/2496892386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56309d20f0f0 con 0x56309cf5cf60 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e2a00640 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 0x56309d3031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:04.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.799+0000 7f03e3e00640 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d302760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009eb0 tx=0x7f03dc010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7ef1044c07170ee server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.800+0000 7f03d8c00640 1 -- 172.21.15.144:0/2496892386 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc016070 con 0x56309cf5cf60 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.800+0000 7f03d8c00640 1 -- 172.21.15.144:0/2496892386 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03dc004d20 con 0x56309cf5cf60 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.800+0000 7f03d8c00640 1 -- 172.21.15.144:0/2496892386 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc012780 con 0x56309cf5cf60 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.800+0000 7f03dbe00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f03c80b3ca0 0x7f03c80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.800+0000 7f03dbe00640 1 -- 172.21.15.144:0/2761955923 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f03d4103d80 con 0x7f03e41055f0 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.800+0000 7f03e3400640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f03c80b3ca0 0x7f03c80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 -- 172.21.15.144:0/2496892386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 msgr2=0x56309d302760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d302760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009eb0 tx=0x7f03dc010740 comp rx=0 tx=0).stop 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 -- 172.21.15.144:0/2496892386 shutdown_connections 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf3c280 0x56309d3031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d302ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 --2- 172.21.15.144:0/2496892386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d302760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 -- 172.21.15.144:0/2496892386 >> 172.21.15.144:0/2496892386 conn(0x56309cf51350 msgr2=0x56309d300650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 -- 172.21.15.144:0/2496892386 shutdown_connections 2024-07-26T10:58:04.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03f1054740 1 -- 172.21.15.144:0/2496892386 wait complete. 2024-07-26T10:58:04.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.801+0000 7f03e3400640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f03c80b3ca0 0x7f03c80b61b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03d40c6970 tx=0x7f03d4000bc0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.802+0000 7f03f1054740 1 -- start start 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.802+0000 7f03f1054740 1 Processor -- start 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d3171c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d3177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d31c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56309d0cb5f0 con 0x56309cf3c280 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56309d204520 con 0x56309cf5cf60 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56309d310f40 con 0x56309cf340a0 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03f1054740 10 client.0 _ll_register_callbacks cb 0x56309d213ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d3171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d3171c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58628/0 (socket says 172.21.15.144:58628) 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03e3e00640 1 -- 172.21.15.144:0/1118018680 learned_addr learned my addr 172.21.15.144:0/1118018680 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03e2a00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d3177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.803+0000 7f03e3e00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d31c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e3e00640 1 -- 172.21.15.144:0/1118018680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 msgr2=0x56309d3171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e3e00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d3171c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e3e00640 1 -- 172.21.15.144:0/1118018680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 msgr2=0x56309d3177a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e3e00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d3177a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e3e00640 1 -- 172.21.15.144:0/1118018680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56309cfc8540 con 0x56309cf5cf60 2024-07-26T10:58:04.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e2a00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d3177a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03e3e00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d31c7c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009c70 tx=0x7f03dc010fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7bb7ddeadd2269 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.804+0000 7f03cd600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.805+0000 7f03cd600640 1 -- 172.21.15.144:0/1118018680 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc016040 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.805+0000 7f03cd600640 1 -- 172.21.15.144:0/1118018680 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f03dc013df0 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.805+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03dc004d20 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.805+0000 7f03cd600640 1 -- 172.21.15.144:0/1118018680 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f03dc003970 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.805+0000 7f03f1054740 10 client.24824 fetch_fsmap learned FSMap version 16 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.805+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x56309d325710 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03cd600640 1 -- 172.21.15.144:0/1118018680 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03dc004d20 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03cd600640 1 -- 172.21.15.144:0/1118018680 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 16) ==== 8350+0+0 (secure 0 0 0) 0x7f03dc028040 con 0x56309cf5cf60 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03f1054740 10 client.24824 fetch_fsmap finished waiting for FSMap version 16 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03f1054740 10 client.24824 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03f1054740 20 client.24824 populate_metadata read hostname 'smithi144' 2024-07-26T10:58:04.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03f1054740 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309cf1b8e0 0x56309d328f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03f1054740 4 client.24824 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T10:58:04.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56309d27a8a0 con 0x56309cf1b8e0 2024-07-26T10:58:04.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.806+0000 7f03e2a00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309cf1b8e0 0x56309d328f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.807+0000 7f03e2a00640 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309cf1b8e0 0x56309d328f50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7a92e2449871fb2 server_cookie=28e7353e2314cb2d in_seq=0 out_seq=0 2024-07-26T10:58:04.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.807+0000 7f03cd600640 10 client.24824 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03cd600640 1 -- 172.21.15.144:0/1118018680 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56309d27a8a0 con 0x56309cf1b8e0 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03cd600640 10 client.24824 handle_command_reply: tid=0 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 client.24824 shutdown 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 msgr2=0x56309d31c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d31c7c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009c70 tx=0x7f03dc010fd0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 shutdown_connections 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309cf1b8e0 0x56309d328f50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d31c7c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03cd600640 0 client.24824 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T10:58:04.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d3177a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 --2- 172.21.15.144:0/1118018680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d3171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 >> 172.21.15.144:0/1118018680 conn(0x56309cf51350 msgr2=0x56309d3058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:04.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 shutdown_connections 2024-07-26T10:58:04.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 1 -- 172.21.15.144:0/1118018680 wait complete. 2024-07-26T10:58:04.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.808+0000 7f03f1054740 20 client.24824 trim_cache size 0 max 16384 2024-07-26T10:58:04.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 Processor -- start 2024-07-26T10:58:04.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 -- start start 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d328880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d310260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d3164b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56309d327290 con 0x56309cf3c280 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56309d327960 con 0x56309cf5cf60 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.828+0000 7f03f1054740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56309d328030 con 0x56309cf340a0 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d310260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d310260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48372/0 (socket says 172.21.15.144:48372) 2024-07-26T10:58:04.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d328880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 -- 172.21.15.144:0/3553333361 learned_addr learned my addr 172.21.15.144:0/3553333361 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e3e00640 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d3164b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 -- 172.21.15.144:0/3553333361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 msgr2=0x56309d328880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d328880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 -- 172.21.15.144:0/3553333361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 msgr2=0x56309d3164b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d3164b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.829+0000 7f03e2a00640 1 -- 172.21.15.144:0/3553333361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56309d310a70 con 0x56309cf3c280 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.830+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d310260 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x56309d329cf0 tx=0x7f03d000b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7ea6b27feceb66 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.830+0000 7f03cfe00640 1 -- 172.21.15.144:0/3553333361 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0005220 con 0x56309cf3c280 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.830+0000 7f03cfe00640 1 -- 172.21.15.144:0/3553333361 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d0004c50 con 0x56309cf3c280 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.830+0000 7f03cfe00640 1 -- 172.21.15.144:0/3553333361 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d00266c0 con 0x56309cf3c280 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 -- 172.21.15.144:0/3553333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 msgr2=0x56309d310260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d310260 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x56309d329cf0 tx=0x7f03d000b440 comp rx=0 tx=0).stop 2024-07-26T10:58:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 -- 172.21.15.144:0/3553333361 shutdown_connections 2024-07-26T10:58:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cf5cf60 0x56309d3164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf3c280 0x56309d310260 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 --2- 172.21.15.144:0/3553333361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf340a0 0x56309d328880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 -- 172.21.15.144:0/3553333361 >> 172.21.15.144:0/3553333361 conn(0x56309cf51350 msgr2=0x56309d326b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 -- 172.21.15.144:0/3553333361 shutdown_connections 2024-07-26T10:58:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.831+0000 7f03f1054740 1 -- 172.21.15.144:0/3553333361 wait complete. 2024-07-26T10:58:04.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.832+0000 7f03f1054740 1 -- start start 2024-07-26T10:58:04.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.832+0000 7f03f1054740 1 Processor -- start 2024-07-26T10:58:04.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d30f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d30f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cef0c90 0x56309d30fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56309d328dd0 con 0x56309cf340a0 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56309d328700 con 0x56309cef0c90 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56309d3294a0 con 0x56309cf5cf60 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03f1054740 10 client.0 _ll_register_callbacks cb 0x7f03dc012660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d30f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d30f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48382/0 (socket says 172.21.15.144:48382) 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 -- 172.21.15.144:0/3898816830 learned_addr learned my addr 172.21.15.144:0/3898816830 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d30f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cef0c90 0x56309d30fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 -- 172.21.15.144:0/3898816830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 msgr2=0x56309d30f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d30f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 -- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cef0c90 msgr2=0x56309d30fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T10:58:04.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cef0c90 0x56309d30fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e3e00640 1 -- 172.21.15.144:0/3898816830 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56309d319260 con 0x56309cf340a0 2024-07-26T10:58:04.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.833+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d30f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:04.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.834+0000 7f03e3e00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d30f250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0053b0 tx=0x7f03dc02b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6578565de933e3f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:04.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.834+0000 7f03cd600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T10:58:04.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.834+0000 7f03cd600640 1 -- 172.21.15.144:0/3898816830 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc043060 con 0x56309cf340a0 2024-07-26T10:58:04.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.834+0000 7f03cd600640 1 -- 172.21.15.144:0/3898816830 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f03dc022030 con 0x56309cf340a0 2024-07-26T10:58:04.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.834+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d0004c50 con 0x56309cf340a0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.834+0000 7f03cd600640 1 -- 172.21.15.144:0/3898816830 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f03dc04a050 con 0x56309cf340a0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 10 client.14685 fetch_fsmap learned FSMap version 16 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x56309d391200 con 0x56309cf340a0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03cd600640 1 -- 172.21.15.144:0/3898816830 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d0004c50 con 0x56309cf340a0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03cd600640 1 -- 172.21.15.144:0/3898816830 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 16) ==== 8350+0+0 (secure 0 0 0) 0x7f03dc00f040 con 0x56309cf340a0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 10 client.14685 fetch_fsmap finished waiting for FSMap version 16 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 10 client.14685 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 20 client.14685 populate_metadata read hostname 'smithi144' 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309d3936f0 0x56309d394b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 4 client.14685 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.835+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "status"}) -- 0x56309d276340 con 0x56309d3936f0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.836+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309d3936f0 0x56309d394b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.836+0000 7f03e2a00640 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309d3936f0 0x56309d394b60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68d9f4475702a551 server_cookie=92b3901bcaa85e1e in_seq=0 out_seq=0 2024-07-26T10:58:04.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.836+0000 7f03cd600640 10 client.14685 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T10:58:04.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03cd600640 1 -- 172.21.15.144:0/3898816830 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56309d276340 con 0x56309d3936f0 2024-07-26T10:58:04.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03cd600640 10 client.14685 handle_command_reply: tid=0 2024-07-26T10:58:04.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 client.14685 shutdown 2024-07-26T10:58:04.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 msgr2=0x56309d30f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d30f250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0053b0 tx=0x7f03dc02b040 comp rx=0 tx=0).stop 2024-07-26T10:58:04.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 shutdown_connections 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56309d3936f0 0x56309d394b60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56309cef0c90 0x56309d30fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56309cf5cf60 0x56309d30f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 --2- 172.21.15.144:0/3898816830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56309cf340a0 0x56309d30f250 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 >> 172.21.15.144:0/3898816830 conn(0x56309cf51350 msgr2=0x56309d326b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03cd600640 0 client.14685 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 shutdown_connections 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 1 -- 172.21.15.144:0/3898816830 wait complete. 2024-07-26T10:58:04.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.837+0000 7f03f1054740 20 client.14685 trim_cache size 0 max 16384 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 0, 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24728, 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 39.702509188999997, 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 16, 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 88, 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 85, 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 59.512320514999999 2024-07-26T10:58:04.842 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f03c80b3ca0 msgr2=0x7f03c80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f03c80b3ca0 0x7f03c80b61b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03d40c6970 tx=0x7f03d4000bc0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e41055f0 msgr2=0x7f03e419bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e41055f0 0x7f03e419bfc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f03d402a910 tx=0x7f03d4004270 comp rx=0 tx=0).stop 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 shutdown_connections 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f03c80b3ca0 0x7f03c80b61b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03e410b880 0x7f03e41a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e410af30 0x7f03e419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 --2- 172.21.15.144:0/2761955923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03e41055f0 0x7f03e419bfc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.838+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 >> 172.21.15.144:0/2761955923 conn(0x7f03e41010a0 msgr2=0x7f03e41034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.839+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 shutdown_connections 2024-07-26T10:58:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:04.839+0000 7f03e8e00640 1 -- 172.21.15.144:0/2761955923 wait complete. 2024-07-26T10:58:05.682 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T10:58:05.682 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:05.683 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T10:58:06.148 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:06 smithi183 ceph-mon[37273]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:06 smithi144 ceph-mon[32129]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:06 smithi160 ceph-mon[36715]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:08 smithi144 ceph-mon[32129]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:08 smithi183 ceph-mon[37273]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:08.419 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 -- 172.21.15.144:0/2901882426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148109670 msgr2=0x7ff14810fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 --2- 172.21.15.144:0/2901882426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148109670 0x7ff14810fea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff144009ca0 tx=0x7ff14402a8c0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 -- 172.21.15.144:0/2901882426 shutdown_connections 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 --2- 172.21.15.144:0/2901882426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148109670 0x7ff14810fea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 --2- 172.21.15.144:0/2901882426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff148108d20 0x7ff148109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 --2- 172.21.15.144:0/2901882426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff1481037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.417+0000 7ff14f200640 1 -- 172.21.15.144:0/2901882426 >> 172.21.15.144:0/2901882426 conn(0x7ff1480fee70 msgr2=0x7ff1481012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.418+0000 7ff14f200640 1 -- 172.21.15.144:0/2901882426 shutdown_connections 2024-07-26T10:58:08.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.418+0000 7ff14f200640 1 -- 172.21.15.144:0/2901882426 wait complete. 2024-07-26T10:58:08.421 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 Processor -- start 2024-07-26T10:58:08.421 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 -- start start 2024-07-26T10:58:08.422 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff148075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:08.422 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148108d20 0x7ff148075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:08.422 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff148109670 0x7ff148075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:08.422 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff148113370 con 0x7ff148108d20 2024-07-26T10:58:08.422 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff1481131f0 con 0x7ff148109670 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.419+0000 7ff14f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff1481134f0 con 0x7ff1481033e0 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff148075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff148075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58656/0 (socket says 172.21.15.144:58656) 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 -- 172.21.15.144:0/291531290 learned_addr learned my addr 172.21.15.144:0/291531290 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14d400640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148108d20 0x7ff148075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14e800640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff148109670 0x7ff148075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 -- 172.21.15.144:0/291531290 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff148109670 msgr2=0x7ff148075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff148109670 0x7ff148075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 -- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148108d20 msgr2=0x7ff148075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148108d20 0x7ff148075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.420+0000 7ff14de00640 1 -- 172.21.15.144:0/291531290 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1481a8020 con 0x7ff1481033e0 2024-07-26T10:58:08.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.421+0000 7ff14de00640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff148075140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00ea60 tx=0x7ff13c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:08.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.421+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c00cde0 con 0x7ff1481033e0 2024-07-26T10:58:08.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.421+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff13c0035f0 con 0x7ff1481033e0 2024-07-26T10:58:08.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.421+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1481a8310 con 0x7ff1481033e0 2024-07-26T10:58:08.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.421+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c010690 con 0x7ff1481033e0 2024-07-26T10:58:08.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.421+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1481a88a0 con 0x7ff1481033e0 2024-07-26T10:58:08.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.422+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1481037f0 con 0x7ff1481033e0 2024-07-26T10:58:08.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.425+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7ff13c0037a0 con 0x7ff1481033e0 2024-07-26T10:58:08.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.426+0000 7ff142a00640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff1300b3ca0 0x7ff1300b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:08.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.426+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7ff13c0e4430 con 0x7ff1481033e0 2024-07-26T10:58:08.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.426+0000 7ff14d400640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff1300b3ca0 0x7ff1300b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:08.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.427+0000 7ff14d400640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff1300b3ca0 0x7ff1300b61b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff148102720 tx=0x7ff138005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:08.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.428+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff13c0ab5f0 con 0x7ff1481033e0 2024-07-26T10:58:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:08 smithi160 ceph-mon[36715]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:08.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.663+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff148076440 con 0x7ff1481033e0 2024-07-26T10:58:08.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.664+0000 7ff142a00640 1 -- 172.21.15.144:0/291531290 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7ff13c0aaef0 con 0x7ff1481033e0 2024-07-26T10:58:08.666 INFO:teuthology.orchestra.run.smithi144.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T10:58:08.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff1300b3ca0 msgr2=0x7ff1300b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff1300b3ca0 0x7ff1300b61b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff148102720 tx=0x7ff138005f90 comp rx=0 tx=0).stop 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 msgr2=0x7ff148075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff148075140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00ea60 tx=0x7ff13c00ef20 comp rx=0 tx=0).stop 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 shutdown_connections 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff1300b3ca0 0x7ff1300b61b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff148109670 0x7ff148075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff148108d20 0x7ff148075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.667+0000 7ff14f200640 1 --2- 172.21.15.144:0/291531290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1481033e0 0x7ff148075140 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.668+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 >> 172.21.15.144:0/291531290 conn(0x7ff1480fee70 msgr2=0x7ff148108340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.668+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 shutdown_connections 2024-07-26T10:58:08.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:08.668+0000 7ff14f200640 1 -- 172.21.15.144:0/291531290 wait complete. 2024-07-26T10:58:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:09 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/291531290' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T10:58:09.430 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:09 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/291531290' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T10:58:09.494 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T10:58:09.494 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi144.front.sepia.ceph.com 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T10:58:09.495 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T10:58:09.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:09.495 DEBUG:teuthology.orchestra.run.smithi144:> ip addr 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2024-07-26T10:58:09.550 INFO:teuthology.orchestra.run.smithi144.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:22 brd ff:ff:ff:ff:ff:ff 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f0 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:23 brd ff:ff:ff:ff:ff:ff 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f1 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5e brd ff:ff:ff:ff:ff:ff 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5f brd ff:ff:ff:ff:ff:ff 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: inet 172.21.15.144/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft 41970sec preferred_lft 41970sec 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2024-07-26T10:58:09.551 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2024-07-26T10:58:09.552 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add name ceph-brx type bridge 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr flush dev ceph-brx 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set ceph-brx up 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T10:58:09.553 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T10:58:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:09 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/291531290' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T10:58:09.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T10:58:10.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:10.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:10.050 DEBUG:teuthology.orchestra.run.smithi144:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T10:58:10.124 INFO:teuthology.orchestra.run.smithi144.stdout:1 2024-07-26T10:58:10.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:10.128 DEBUG:teuthology.orchestra.run.smithi144:> route 2024-07-26T10:58:10.175 INFO:teuthology.orchestra.run.smithi144.stdout:Kernel IP routing table 2024-07-26T10:58:10.175 INFO:teuthology.orchestra.run.smithi144.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T10:58:10.176 INFO:teuthology.orchestra.run.smithi144.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T10:58:10.176 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T10:58:10.176 INFO:teuthology.orchestra.run.smithi144.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T10:58:10.177 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T10:58:10.177 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T10:58:10.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T10:58:10.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T10:58:10.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T10:58:10.177 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T10:58:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:10 smithi144 ceph-mon[32129]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:10.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T10:58:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:10 smithi183 ceph-mon[37273]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:10 smithi160 ceph-mon[36715]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:10.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:10.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:10.891 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2024-07-26T10:58:10.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:10.990 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list-id 2024-07-26T10:58:11.042 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T10:58:11.042 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T10:58:11.042 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:11.042 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-07-26T10:58:11.042 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T10:58:11.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T10:58:11.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:11.224 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-07-26T10:58:11.224 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T10:58:11.224 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T10:58:11.224 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-07-26T10:58:11.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T10:58:11.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-07-26T10:58:11.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-07-26T10:58:11.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-07-26T10:58:11.225 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T10:58:11.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T10:58:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:11.531 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T10:58:11.531 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T10:58:11.531 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set brx.0 up 2024-07-26T10:58:11.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T10:58:11.532 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T10:58:11.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T10:58:11.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:11.724 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-07-26T10:58:11.724 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T10:58:11.724 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:11.770 INFO:teuthology.orchestra.run.smithi144.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T10:58:11.771 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T10:58:11.771 DEBUG:teuthology.orchestra.run.smithi144:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:11.861 DEBUG:teuthology.orchestra.run.smithi144:> sudo modprobe fuse 2024-07-26T10:58:11.929 DEBUG:teuthology.orchestra.run.smithi144:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/proc 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/sys 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/dev 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/security 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/shm 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/pts 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/run 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/cgroup 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/pstore 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/bpf 2024-07-26T10:58:11.982 INFO:teuthology.orchestra.run.smithi144.stdout:/ 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/selinux 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/hugepages 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/mqueue 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/debug 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/tracing 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/fuse/connections 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/config 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/run/user/1000 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/ceph 2024-07-26T10:58:11.983 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/02076a820b90069c139eb087a15b14ff865c78b285923942dec49d6a2b2af33b/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/0d7558e1f28e8e9d2db35162c1147c3eebfb32e0614e6d5cdb62784f8e8afb22/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/run/user/0 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/5a97feda7837184ad9c11f7a3f259cc2e2fbd7d4de7939a18aa7625db9e5f9a7/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/c445407c54295d8b77d43b7eb9e84a0338ef3eb49380520c96634aed31ef2160/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/77f7d03e123ea53bdab62061f4f999fa2a7b8bd8e89104f4bc4d21e927d4935e/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/e8ad1a02a5178bddd18176d8d337064615ccf2f22159e8d16d9763394d920f49/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/3cb584ed809c76e40be492e3b8563c3dca6cf5d0bd644dfb14020de7fc999f8e/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/d50f12bfec63327e92ddd7ec3bbb20d0863f94a8983adf730a6a8a538919ac51/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/06284a7c5917ce6faa65be7111d2b2451dda9c214768201afdd239954314857d/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/ca9af11468981dabdfd149345993af483671a146f76ab9474d164c82a45f4121/merged 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:11.984 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:11.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:11.985 DEBUG:teuthology.orchestra.run.smithi144:> ls /sys/fs/fuse/connections 2024-07-26T10:58:12.030 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-07-26T10:58:12.030 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-07-26T10:58:12.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo modprobe fuse 2024-07-26T10:58:12.146 DEBUG:teuthology.orchestra.run.smithi144:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T10:58:12.187 INFO:teuthology.orchestra.run.smithi144.stdout:/proc 2024-07-26T10:58:12.187 INFO:teuthology.orchestra.run.smithi144.stdout:/sys 2024-07-26T10:58:12.187 INFO:teuthology.orchestra.run.smithi144.stdout:/dev 2024-07-26T10:58:12.187 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/security 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/shm 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/pts 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/run 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/cgroup 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/pstore 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/bpf 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/ 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/selinux 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/hugepages 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/mqueue 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/debug 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/tracing 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/fuse/connections 2024-07-26T10:58:12.188 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/config 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/run/user/1000 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/ceph 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/02076a820b90069c139eb087a15b14ff865c78b285923942dec49d6a2b2af33b/merged 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/0d7558e1f28e8e9d2db35162c1147c3eebfb32e0614e6d5cdb62784f8e8afb22/merged 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/run/user/0 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/5a97feda7837184ad9c11f7a3f259cc2e2fbd7d4de7939a18aa7625db9e5f9a7/merged 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/c445407c54295d8b77d43b7eb9e84a0338ef3eb49380520c96634aed31ef2160/merged 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/77f7d03e123ea53bdab62061f4f999fa2a7b8bd8e89104f4bc4d21e927d4935e/merged 2024-07-26T10:58:12.189 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/e8ad1a02a5178bddd18176d8d337064615ccf2f22159e8d16d9763394d920f49/merged 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/3cb584ed809c76e40be492e3b8563c3dca6cf5d0bd644dfb14020de7fc999f8e/merged 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/d50f12bfec63327e92ddd7ec3bbb20d0863f94a8983adf730a6a8a538919ac51/merged 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/06284a7c5917ce6faa65be7111d2b2451dda9c214768201afdd239954314857d/merged 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/ca9af11468981dabdfd149345993af483671a146f76ab9474d164c82a45f4121/merged 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:12.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:12.191 DEBUG:teuthology.orchestra.run.smithi144:> ls /sys/fs/fuse/connections 2024-07-26T10:58:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:12 smithi144 ceph-mon[32129]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:12.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi144.stderr:2024-07-26T10:58:12.340+0000 7f489d320480 -1 init, newargv = 0x7f4884004ba0 newargc=15 2024-07-26T10:58:12.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi144.stderr:ceph-fuse[82259]: starting ceph client 2024-07-26T10:58:12.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi144.stderr:ceph-fuse[82259]: starting fuse 2024-07-26T10:58:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:12 smithi183 ceph-mon[37273]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:12 smithi160 ceph-mon[36715]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:13.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo modprobe fuse 2024-07-26T10:58:13.455 DEBUG:teuthology.orchestra.run.smithi144:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/proc 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/sys 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/dev 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/security 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/shm 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/pts 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/run 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/cgroup 2024-07-26T10:58:13.509 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/pstore 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/bpf 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/ 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/selinux 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/hugepages 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/mqueue 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/debug 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/tracing 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/fs/fuse/connections 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/sys/kernel/config 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T10:58:13.510 INFO:teuthology.orchestra.run.smithi144.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/run/user/1000 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/ceph 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/02076a820b90069c139eb087a15b14ff865c78b285923942dec49d6a2b2af33b/merged 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/0d7558e1f28e8e9d2db35162c1147c3eebfb32e0614e6d5cdb62784f8e8afb22/merged 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/run/user/0 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/5a97feda7837184ad9c11f7a3f259cc2e2fbd7d4de7939a18aa7625db9e5f9a7/merged 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/c445407c54295d8b77d43b7eb9e84a0338ef3eb49380520c96634aed31ef2160/merged 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/77f7d03e123ea53bdab62061f4f999fa2a7b8bd8e89104f4bc4d21e927d4935e/merged 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/e8ad1a02a5178bddd18176d8d337064615ccf2f22159e8d16d9763394d920f49/merged 2024-07-26T10:58:13.511 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/3cb584ed809c76e40be492e3b8563c3dca6cf5d0bd644dfb14020de7fc999f8e/merged 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/d50f12bfec63327e92ddd7ec3bbb20d0863f94a8983adf730a6a8a538919ac51/merged 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/06284a7c5917ce6faa65be7111d2b2451dda9c214768201afdd239954314857d/merged 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/var/lib/containers/storage/overlay/ca9af11468981dabdfd149345993af483671a146f76ab9474d164c82a45f4121/merged 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T10:58:13.512 INFO:teuthology.orchestra.run.smithi144.stdout:/home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:13.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:13.513 DEBUG:teuthology.orchestra.run.smithi144:> ls /sys/fs/fuse/connections 2024-07-26T10:58:13.562 INFO:teuthology.orchestra.run.smithi144.stdout:107 2024-07-26T10:58:13.563 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [107] 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> sudo stdin-killer -- python3 -c ' 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> import glob 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> import re 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> import os 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> import subprocess 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> def _find_admin_socket(client_name): 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> files = glob.glob(asok_path) 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-07-26T10:58:13.563 DEBUG:teuthology.orchestra.run.smithi144:> 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> # Given a non-glob path, it better be there 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> if "*" not in asok_path: 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> assert(len(files) == 1) 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> return files[0] 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> for f in files: 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> if os.path.exists("/proc/{0}".format(pid)): 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> contents = proc_f.read() 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> if mountpoint in contents: 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> return f 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> print(_find_admin_socket("client.admin")) 2024-07-26T10:58:13.564 DEBUG:teuthology.orchestra.run.smithi144:> ' 2024-07-26T10:58:13.725 INFO:teuthology.orchestra.run.smithi144.stdout:/var/run/ceph/ceph-client.admin.82259.asok 2024-07-26T10:58:13.730 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:13.743 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.82259.asok 2024-07-26T10:58:13.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:13.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.82259.asok status 2024-07-26T10:58:14.035 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T10:58:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "metadata": { 2024-07-26T10:58:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "ceph_sha1": "e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528", 2024-07-26T10:58:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "entity_id": "admin", 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "hostname": "smithi144", 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "pid": "82259", 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "root": "/" 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "dentry_count": 0, 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "dentry_pinned_count": 0, 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24842, 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "inst": { 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "name": { 2024-07-26T10:58:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "type": "client", 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "num": 24842 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "addr": { 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "type": "v1", 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "addr": "172.21.15.144:0", 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "nonce": 2187348764 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "addr": { 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "type": "v1", 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "addr": "172.21.15.144:0", 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "nonce": 2187348764 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-07-26T10:58:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "inst_str": "client.24842 172.21.15.144:0/2187348764", 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "addr_str": "172.21.15.144:0/2187348764", 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "inode_count": 1, 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "mds_epoch": 16, 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "osd_epoch": 88, 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "osd_epoch_barrier": 0, 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "blocklisted": false, 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs" 2024-07-26T10:58:14.038 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T10:58:14.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:14.060 DEBUG:teuthology.orchestra.run.smithi144:> stat --file-system '--printf=%T 2024-07-26T10:58:14.060 DEBUG:teuthology.orchestra.run.smithi144:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:14.112 INFO:teuthology.orchestra.run.smithi144.stdout:fuseblk 2024-07-26T10:58:14.112 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:14.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:14.113 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:14.195 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-07-26T10:58:14.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T10:58:14.338 INFO:teuthology.orchestra.run.smithi144.stdout:# file: . 2024-07-26T10:58:14.339 INFO:teuthology.orchestra.run.smithi144.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-07-26T10:58:14.339 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:58:14.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T10:58:14.350 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:58:14.350 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T10:58:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:14 smithi183 ceph-mon[37273]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:14.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T10:58:14.467 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:14 smithi144 ceph-mon[32129]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:14 smithi160 ceph-mon[36715]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:14.873 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:16 smithi183 ceph-mon[37273]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:16.434 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:16 smithi144 ceph-mon[32129]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:16.434 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:16 smithi160 ceph-mon[36715]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T10:58:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 -- 172.21.15.144:0/3340399343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79680751e0 msgr2=0x7f79680755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 --2- 172.21.15.144:0/3340399343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79680751e0 0x7f79680755f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795802a410 comp rx=0 tx=0).stop 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 -- 172.21.15.144:0/3340399343 shutdown_connections 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 --2- 172.21.15.144:0/3340399343 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f796810f940 0x7f7968111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 --2- 172.21.15.144:0/3340399343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 0x7f796810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 --2- 172.21.15.144:0/3340399343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79680751e0 0x7f79680755f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 -- 172.21.15.144:0/3340399343 >> 172.21.15.144:0/3340399343 conn(0x7f79680fee70 msgr2=0x7f79681012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:16.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 -- 172.21.15.144:0/3340399343 shutdown_connections 2024-07-26T10:58:16.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.957+0000 7f796dc00640 1 -- 172.21.15.144:0/3340399343 wait complete. 2024-07-26T10:58:16.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.958+0000 7f796dc00640 1 Processor -- start 2024-07-26T10:58:16.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.958+0000 7f796dc00640 1 -- start start 2024-07-26T10:58:16.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796dc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79680751e0 0x7f796819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:16.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796dc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 0x7f796819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:16.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796dc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f796810f940 0x7f79681a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:16.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796dc00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f79681154d0 con 0x7f7968075b30 2024-07-26T10:58:16.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796dc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7968115350 con 0x7f79680751e0 2024-07-26T10:58:16.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796dc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7968115650 con 0x7f796810f940 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f795f400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 0x7f796819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f7967e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79680751e0 0x7f796819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f795f400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 0x7f796819c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52564/0 (socket says 172.21.15.144:52564) 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f796d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f796810f940 0x7f79681a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.959+0000 7f795f400640 1 -- 172.21.15.144:0/483684994 learned_addr learned my addr 172.21.15.144:0/483684994 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796d200640 1 -- 172.21.15.144:0/483684994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79680751e0 msgr2=0x7f796819bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796d200640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79680751e0 0x7f796819bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796d200640 1 -- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 msgr2=0x7f796819c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796d200640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 0x7f796819c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796d200640 1 -- 172.21.15.144:0/483684994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79681a1e70 con 0x7f796810f940 2024-07-26T10:58:16.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796d200640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f796810f940 0x7f79681a1740 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7960009fd0 tx=0x7f796000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:16.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000ce20 con 0x7f796810f940 2024-07-26T10:58:16.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7960005020 con 0x7f796810f940 2024-07-26T10:58:16.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.960+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79681a2160 con 0x7f796810f940 2024-07-26T10:58:16.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.961+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960010690 con 0x7f796810f940 2024-07-26T10:58:16.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.961+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681a88c0 con 0x7f796810f940 2024-07-26T10:58:16.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.962+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79680755f0 con 0x7f796810f940 2024-07-26T10:58:16.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.964+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f7960048080 con 0x7f796810f940 2024-07-26T10:58:16.967 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.965+0000 7f7965600640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f794c0b3c20 0x7f794c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:16.968 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.966+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f79600e4a10 con 0x7f796810f940 2024-07-26T10:58:16.968 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.966+0000 7f7967e00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f794c0b3c20 0x7f794c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:16.968 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.967+0000 7f7967e00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f794c0b3c20 0x7f794c0b6130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7958031440 tx=0x7f7958009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:16.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:16.967+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f79600abbd0 con 0x7f796810f940 2024-07-26T10:58:17.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.187+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f796819d030 con 0x7f796810f940 2024-07-26T10:58:17.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.187+0000 7f7965600640 1 -- 172.21.15.144:0/483684994 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+901 (secure 0 0 0) 0x7f79600ab4d0 con 0x7f796810f940 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6801/1820197496 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/288744986 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/563407546 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6800/1828713044 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6801/1828713044 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/2087175124 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/3434609332 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/2275934374 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/2481813561 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6800/2854801065 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/1824026059 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6801/2854801065 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6800/1820197496 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/3586311140 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/3284599906 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:17.191 INFO:teuthology.orchestra.run.smithi144.stderr:listed 15 entries 2024-07-26T10:58:17.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.191+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f794c0b3c20 msgr2=0x7f794c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:17.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.191+0000 7f796dc00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f794c0b3c20 0x7f794c0b6130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7958031440 tx=0x7f7958009560 comp rx=0 tx=0).stop 2024-07-26T10:58:17.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.191+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f796810f940 msgr2=0x7f79681a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:17.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.191+0000 7f796dc00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f796810f940 0x7f79681a1740 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7960009fd0 tx=0x7f796000ef10 comp rx=0 tx=0).stop 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 shutdown_connections 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f794c0b3c20 0x7f794c0b6130 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f796810f940 0x7f79681a1740 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7968075b30 0x7f796819c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 --2- 172.21.15.144:0/483684994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79680751e0 0x7f796819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 >> 172.21.15.144:0/483684994 conn(0x7f79680fee70 msgr2=0x7f79681094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 shutdown_connections 2024-07-26T10:58:17.194 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:17.192+0000 7f796dc00640 1 -- 172.21.15.144:0/483684994 wait complete. 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/483684994' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/483684994' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:17.958 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:58:17.958 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T10:58:18.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T10:58:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/483684994' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T10:58:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:18.879 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:20 smithi144 ceph-mon[32129]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:20 smithi160 ceph-mon[36715]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:20 smithi183 ceph-mon[37273]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:21.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 -- 172.21.15.144:0/3834624478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84101a70 msgr2=0x7f0c84101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:21.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 --2- 172.21.15.144:0/3834624478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84101a70 0x7f0c84101e80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400aa10 tx=0x7f0c7402b720 comp rx=0 tx=0).stop 2024-07-26T10:58:21.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 -- 172.21.15.144:0/3834624478 shutdown_connections 2024-07-26T10:58:21.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 --2- 172.21.15.144:0/3834624478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c84107d00 0x7f0c8410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 --2- 172.21.15.144:0/3834624478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c841073b0 0x7f0c841077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 --2- 172.21.15.144:0/3834624478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84101a70 0x7f0c84101e80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 -- 172.21.15.144:0/3834624478 >> 172.21.15.144:0/3834624478 conn(0x7f0c840774d0 msgr2=0x7f0c84077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 -- 172.21.15.144:0/3834624478 shutdown_connections 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.076+0000 7f0c83e00640 1 -- 172.21.15.144:0/3834624478 wait complete. 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 Processor -- start 2024-07-26T10:58:21.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 -- start start 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c84101a70 0x7f0c840fe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c841073b0 0x7f0c840febf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 0x7f0c840ff130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f0c84111190 con 0x7f0c84101a70 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0c84111010 con 0x7f0c84107d00 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.078+0000 7f0c83e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0c84111310 con 0x7f0c841073b0 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 0x7f0c840ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:21.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c82a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c84101a70 0x7f0c840fe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:21.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c82000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c841073b0 0x7f0c840febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:21.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 0x7f0c840ff130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36818/0 (socket says 172.21.15.144:36818) 2024-07-26T10:58:21.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 -- 172.21.15.144:0/1995732019 learned_addr learned my addr 172.21.15.144:0/1995732019 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 -- 172.21.15.144:0/1995732019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c841073b0 msgr2=0x7f0c840febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c841073b0 0x7f0c840febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 -- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c84101a70 msgr2=0x7f0c840fe6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c84101a70 0x7f0c840fe6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c83400640 1 -- 172.21.15.144:0/1995732019 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c841a7fc0 con 0x7f0c84107d00 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.079+0000 7f0c82000640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c841073b0 0x7f0c840febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.080+0000 7f0c83400640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 0x7f0c840ff130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00eb10 tx=0x7f0c7c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:21.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.080+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c004b20 con 0x7f0c84107d00 2024-07-26T10:58:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.080+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c7c004cc0 con 0x7f0c84107d00 2024-07-26T10:58:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.080+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c010670 con 0x7f0c84107d00 2024-07-26T10:58:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.080+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c841a82b0 con 0x7f0c84107d00 2024-07-26T10:58:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.080+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c841a8770 con 0x7f0c84107d00 2024-07-26T10:58:21.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.081+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c84101e80 con 0x7f0c84107d00 2024-07-26T10:58:21.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.084+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f0c7c005020 con 0x7f0c84107d00 2024-07-26T10:58:21.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.085+0000 7f0c7b400640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0c680b3ca0 0x7f0c680b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:21.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.085+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f0c7c0e4600 con 0x7f0c84107d00 2024-07-26T10:58:21.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.085+0000 7f0c82a00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0c680b3ca0 0x7f0c680b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:21.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.086+0000 7f0c82a00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0c680b3ca0 0x7f0c680b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0c74005ab0 tx=0x7f0c74005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:21.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.089+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0c7c0ab7c0 con 0x7f0c84107d00 2024-07-26T10:58:21.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.306+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0c8402e0c0 con 0x7f0c84107d00 2024-07-26T10:58:21.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.307+0000 7f0c7b400640 1 -- 172.21.15.144:0/1995732019 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+901 (secure 0 0 0) 0x7f0c7c0ab0c0 con 0x7f0c84107d00 2024-07-26T10:58:21.309 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6801/1820197496 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:21.309 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/288744986 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/563407546 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6800/1828713044 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6801/1828713044 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/2087175124 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/3434609332 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/2275934374 2024-07-27T10:47:54.806379+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/2481813561 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6800/2854801065 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/1824026059 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6801/2854801065 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:6800/1820197496 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:21.310 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/3586311140 2024-07-27T10:48:24.009859+0000 2024-07-26T10:58:21.311 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144:0/3284599906 2024-07-27T10:49:42.802348+0000 2024-07-26T10:58:21.311 INFO:teuthology.orchestra.run.smithi144.stderr:listed 15 entries 2024-07-26T10:58:21.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.310+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0c680b3ca0 msgr2=0x7f0c680b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.310+0000 7f0c83e00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0c680b3ca0 0x7f0c680b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0c74005ab0 tx=0x7f0c74005a20 comp rx=0 tx=0).stop 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.310+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 msgr2=0x7f0c840ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.310+0000 7f0c83e00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 0x7f0c840ff130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00eb10 tx=0x7f0c7c00c2d0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 shutdown_connections 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0c680b3ca0 0x7f0c680b61b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84107d00 0x7f0c840ff130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c841073b0 0x7f0c840febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 --2- 172.21.15.144:0/1995732019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c84101a70 0x7f0c840fe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 >> 172.21.15.144:0/1995732019 conn(0x7f0c840774d0 msgr2=0x7f0c841055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 shutdown_connections 2024-07-26T10:58:21.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:21.311+0000 7f0c83e00640 1 -- 172.21.15.144:0/1995732019 wait complete. 2024-07-26T10:58:22.099 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi144.front.sepia.ceph.com... 2024-07-26T10:58:22.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:22.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-07-26T10:58:22.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi144.stderr:ceph-fuse[82259]: fuse finished with error 0 and tester_r 0 2024-07-26T10:58:22.188 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T10:58:22.256 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T10:58:22.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T10:58:22.257 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-07-26T10:58:22.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:58:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:22 smithi183 ceph-mon[37273]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:22 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1995732019' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T10:58:22.413 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:22 smithi144 ceph-mon[32129]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:22.413 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:22.413 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:22.413 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:22 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1995732019' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T10:58:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:22 smithi160 ceph-mon[36715]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:22 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1995732019' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T10:58:22.801 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:23.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:23.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:23.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:58:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:24 smithi144 ceph-mon[32129]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:24 smithi183 ceph-mon[37273]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:24 smithi160 ceph-mon[36715]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:25.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 -- 172.21.15.144:0/665546985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf410af30 msgr2=0x7f8cf410b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:25.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 --2- 172.21.15.144:0/665546985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf410af30 0x7f8cf410b340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009860 tx=0x7f8ce002a340 comp rx=0 tx=0).stop 2024-07-26T10:58:25.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 -- 172.21.15.144:0/665546985 shutdown_connections 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 --2- 172.21.15.144:0/665546985 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410b880 0x7f8cf41120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 --2- 172.21.15.144:0/665546985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf410af30 0x7f8cf410b340 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 --2- 172.21.15.144:0/665546985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf41055f0 0x7f8cf4105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 -- 172.21.15.144:0/665546985 >> 172.21.15.144:0/665546985 conn(0x7f8cf4101060 msgr2=0x7f8cf41034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 -- 172.21.15.144:0/665546985 shutdown_connections 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.098+0000 7f8cfb000640 1 -- 172.21.15.144:0/665546985 wait complete. 2024-07-26T10:58:25.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.099+0000 7f8cfb000640 1 Processor -- start 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.099+0000 7f8cfb000640 1 -- start start 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfb000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 0x7f8cf419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfb000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410af30 0x7f8cf419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfb000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf410b880 0x7f8cf41a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfb000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41154f0 con 0x7f8cf410b880 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfb000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf4115370 con 0x7f8cf410af30 2024-07-26T10:58:25.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfb000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf4115670 con 0x7f8cf41055f0 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cf9c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 0x7f8cf419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cfa600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf410b880 0x7f8cf41a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cf9c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 0x7f8cf419bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54176/0 (socket says 172.21.15.144:54176) 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cf9c00640 1 -- 172.21.15.144:0/2703319723 learned_addr learned my addr 172.21.15.144:0/2703319723 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.100+0000 7f8cf9200640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410af30 0x7f8cf419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cfa600640 1 -- 172.21.15.144:0/2703319723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 msgr2=0x7f8cf419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cfa600640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 0x7f8cf419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cfa600640 1 -- 172.21.15.144:0/2703319723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410af30 msgr2=0x7f8cf419c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:25.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cfa600640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410af30 0x7f8cf419c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cfa600640 1 -- 172.21.15.144:0/2703319723 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf41a1f10 con 0x7f8cf410b880 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cf9c00640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 0x7f8cf419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cf9200640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410af30 0x7f8cf419c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cfa600640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf410b880 0x7f8cf41a17e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00e970 tx=0x7f8cec00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.101+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec00ccc0 con 0x7f8cf410b880 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.102+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf41a2200 con 0x7f8cf410b880 2024-07-26T10:58:25.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.102+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cec0035f0 con 0x7f8cf410b880 2024-07-26T10:58:25.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.102+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec00ce60 con 0x7f8cf410b880 2024-07-26T10:58:25.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.102+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf41a8960 con 0x7f8cf410b880 2024-07-26T10:58:25.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.103+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf4105a00 con 0x7f8cf410b880 2024-07-26T10:58:25.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.106+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8cec004020 con 0x7f8cf410b880 2024-07-26T10:58:25.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.107+0000 7f8cf2000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8cdc0b3ca0 0x7f8cdc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:25.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.107+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f8cec0e4260 con 0x7f8cf410b880 2024-07-26T10:58:25.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.107+0000 7f8cf9c00640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8cdc0b3ca0 0x7f8cdc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:25.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.108+0000 7f8cf9c00640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8cdc0b3ca0 0x7f8cdc0b61b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800e8b0 tx=0x7f8ce80095c0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:25.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.109+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8cec0ab510 con 0x7f8cf410b880 2024-07-26T10:58:25.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.310+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8cf402e0c0 con 0x7f8cf410b880 2024-07-26T10:58:25.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.314+0000 7f8cf2000640 1 -- 172.21.15.144:0/2703319723 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25047 (secure 0 0 0) 0x7f8cec0aae10 con 0x7f8cf410b880 2024-07-26T10:58:25.317 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:58:25.317 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":88,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:57:44.875941+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:58:25.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8cdc0b3ca0 msgr2=0x7f8cdc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:25.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8cdc0b3ca0 0x7f8cdc0b61b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800e8b0 tx=0x7f8ce80095c0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf410b880 msgr2=0x7f8cf41a17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf410b880 0x7f8cf41a17e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00e970 tx=0x7f8cec00ee30 comp rx=0 tx=0).stop 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 shutdown_connections 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8cdc0b3ca0 0x7f8cdc0b61b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cf410b880 0x7f8cf41a17e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8cf410af30 0x7f8cf419c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 --2- 172.21.15.144:0/2703319723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8cf41055f0 0x7f8cf419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.317+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 >> 172.21.15.144:0/2703319723 conn(0x7f8cf4101060 msgr2=0x7f8cf4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.318+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 shutdown_connections 2024-07-26T10:58:25.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:25.318+0000 7f8cfb000640 1 -- 172.21.15.144:0/2703319723 wait complete. 2024-07-26T10:58:26.206 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:58:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:26 smithi183 ceph-mon[37273]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:26 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2703319723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:26 smithi144 ceph-mon[32129]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:26 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2703319723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:26 smithi160 ceph-mon[36715]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T10:58:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:26 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2703319723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:26.663 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:28.367 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:28 smithi144 ceph-mon[32129]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T10:58:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:28 smithi183 ceph-mon[37273]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T10:58:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:28 smithi160 ceph-mon[36715]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T10:58:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 -- 172.21.15.144:0/4100538816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c810a250 msgr2=0x7f91c810a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 --2- 172.21.15.144:0/4100538816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c810a250 0x7f91c810a660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009900 tx=0x7f91b802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:58:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 -- 172.21.15.144:0/4100538816 shutdown_connections 2024-07-26T10:58:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 --2- 172.21.15.144:0/4100538816 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8112050 0x7f91c8118910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 --2- 172.21.15.144:0/4100538816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91c8111700 0x7f91c8111b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 --2- 172.21.15.144:0/4100538816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c810a250 0x7f91c810a660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 -- 172.21.15.144:0/4100538816 >> 172.21.15.144:0/4100538816 conn(0x7f91c8074110 msgr2=0x7f91c8074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 -- 172.21.15.144:0/4100538816 shutdown_connections 2024-07-26T10:58:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.915+0000 7f91ce600640 1 -- 172.21.15.144:0/4100538816 wait complete. 2024-07-26T10:58:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.916+0000 7f91ce600640 1 Processor -- start 2024-07-26T10:58:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 -- start start 2024-07-26T10:58:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91c810a250 0x7f91c810b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 0x7f91c810b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c8112050 0x7f91c810bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f91c811bd50 con 0x7f91c8112050 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f91c811bbd0 con 0x7f91c8111700 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91ce600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f91c811bed0 con 0x7f91c810a250 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 0x7f91c810b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 0x7f91c810b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36840/0 (socket says 172.21.15.144:36840) 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.917+0000 7f91bfe00640 1 -- 172.21.15.144:0/1938834656 learned_addr learned my addr 172.21.15.144:0/1938834656 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91c7400640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91c810a250 0x7f91c810b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91c7e00640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c8112050 0x7f91c810bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91bfe00640 1 -- 172.21.15.144:0/1938834656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91c810a250 msgr2=0x7f91c810b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91bfe00640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91c810a250 0x7f91c810b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91bfe00640 1 -- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c8112050 msgr2=0x7f91c810bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91bfe00640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c8112050 0x7f91c810bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91bfe00640 1 -- 172.21.15.144:0/1938834656 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c81b09a0 con 0x7f91c8111700 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.918+0000 7f91bfe00640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 0x7f91c810b730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91b000e9c0 tx=0x7f91b000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.919+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b000ccc0 con 0x7f91c8111700 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.919+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c81b0c90 con 0x7f91c8111700 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.919+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b00054d0 con 0x7f91c8111700 2024-07-26T10:58:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.919+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0010430 con 0x7f91c8111700 2024-07-26T10:58:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.919+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c81b11a0 con 0x7f91c8111700 2024-07-26T10:58:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.920+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c810a660 con 0x7f91c8111700 2024-07-26T10:58:28.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.923+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f91b0005000 con 0x7f91c8111700 2024-07-26T10:58:28.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.924+0000 7f91c4c00640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f91ac0b3c20 0x7f91ac0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:28.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.924+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f91b00e4600 con 0x7f91c8111700 2024-07-26T10:58:28.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.924+0000 7f91c7400640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f91ac0b3c20 0x7f91ac0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:28.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.925+0000 7f91c7400640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f91ac0b3c20 0x7f91ac0b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f91b8005ab0 tx=0x7f91b8005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:28.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:28.925+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f91b0055600 con 0x7f91c8111700 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.129+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91c810c4f0 con 0x7f91c8111700 2024-07-26T10:58:29.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.131+0000 7f91c4c00640 1 -- 172.21.15.144:0/1938834656 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25047 (secure 0 0 0) 0x7f91c810c4f0 con 0x7f91c8111700 2024-07-26T10:58:29.134 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:58:29.134 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":88,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:57:44.875941+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f91ac0b3c20 msgr2=0x7f91ac0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f91ac0b3c20 0x7f91ac0b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f91b8005ab0 tx=0x7f91b8005a20 comp rx=0 tx=0).stop 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 msgr2=0x7f91c810b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 0x7f91c810b730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91b000e9c0 tx=0x7f91b000ee80 comp rx=0 tx=0).stop 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 shutdown_connections 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f91ac0b3c20 0x7f91ac0b6130 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c8112050 0x7f91c810bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f91c8111700 0x7f91c810b730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:29.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 --2- 172.21.15.144:0/1938834656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91c810a250 0x7f91c810b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:29.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 >> 172.21.15.144:0/1938834656 conn(0x7f91c8074110 msgr2=0x7f91c810f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:29.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 shutdown_connections 2024-07-26T10:58:29.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:29.135+0000 7f91ce600640 1 -- 172.21.15.144:0/1938834656 wait complete. 2024-07-26T10:58:30.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-26T10:58:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:30 smithi144 ceph-mon[32129]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:30 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1938834656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:30 smithi183 ceph-mon[37273]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:30 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1938834656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:30.523 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:30 smithi160 ceph-mon[36715]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:30 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1938834656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:58:32.357 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:32 smithi144 ceph-mon[32129]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:32 smithi183 ceph-mon[37273]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:32 smithi160 ceph-mon[36715]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 -- 172.21.15.144:0/2215130404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 msgr2=0x7f1d3c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 --2- 172.21.15.144:0/2215130404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 0x7f1d3c10f740 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009900 tx=0x7f1d2802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:58:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 -- 172.21.15.144:0/2215130404 shutdown_connections 2024-07-26T10:58:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 --2- 172.21.15.144:0/2215130404 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 0x7f1d3c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 --2- 172.21.15.144:0/2215130404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 0x7f1d3c10f740 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 --2- 172.21.15.144:0/2215130404 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 0x7f1d3c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 -- 172.21.15.144:0/2215130404 >> 172.21.15.144:0/2215130404 conn(0x7f1d3c078ff0 msgr2=0x7f1d3c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 -- 172.21.15.144:0/2215130404 shutdown_connections 2024-07-26T10:58:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.682+0000 7f1d42800640 1 -- 172.21.15.144:0/2215130404 wait complete. 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.683+0000 7f1d42800640 1 Processor -- start 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.683+0000 7f1d42800640 1 -- start start 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d42800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 0x7f1d3c1a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d42800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 0x7f1d3c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d42800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 0x7f1d3c1a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d42800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c119940 con 0x7f1d3c10f330 2024-07-26T10:58:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d42800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c1197c0 con 0x7f1d3c10fc80 2024-07-26T10:58:32.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d42800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c119ac0 con 0x7f1d3c1099f0 2024-07-26T10:58:32.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d40a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 0x7f1d3c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:32.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d41e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 0x7f1d3c1a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:32.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d41400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 0x7f1d3c1a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:32.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d41e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 0x7f1d3c1a5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:37474/0 (socket says 172.21.15.144:37474) 2024-07-26T10:58:32.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.684+0000 7f1d41e00640 1 -- 172.21.15.144:0/755094235 learned_addr learned my addr 172.21.15.144:0/755094235 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d41400640 1 -- 172.21.15.144:0/755094235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 msgr2=0x7f1d3c1a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d41400640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 0x7f1d3c1a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d41400640 1 -- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 msgr2=0x7f1d3c1a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d41400640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 0x7f1d3c1a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d41400640 1 -- 172.21.15.144:0/755094235 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c1a62c0 con 0x7f1d3c1099f0 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d41400640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 0x7f1d3c1a0350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000e9f0 tx=0x7f1d3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:32.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3000ccc0 con 0x7f1d3c1099f0 2024-07-26T10:58:32.690 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c1a65b0 con 0x7f1d3c1099f0 2024-07-26T10:58:32.690 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d300108e0 con 0x7f1d3c1099f0 2024-07-26T10:58:32.690 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.685+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30018880 con 0x7f1d3c1099f0 2024-07-26T10:58:32.690 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.686+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3c1acd10 con 0x7f1d3c1099f0 2024-07-26T10:58:32.690 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.687+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d3c109e00 con 0x7f1d3c1099f0 2024-07-26T10:58:32.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.689+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f1d30010430 con 0x7f1d3c1099f0 2024-07-26T10:58:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.691+0000 7f1d3a000640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1d240b3ca0 0x7f1d240b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.691+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f1d300e4840 con 0x7f1d3c1099f0 2024-07-26T10:58:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.691+0000 7f1d40a00640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1d240b3ca0 0x7f1d240b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.692+0000 7f1d40a00640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1d240b3ca0 0x7f1d240b61b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009230 tx=0x7f1d28031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:32.695 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.692+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1d300ab3b0 con 0x7f1d3c1099f0 2024-07-26T10:58:32.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:32.933+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1d3c02e0c0 con 0x7f1d3c1099f0 2024-07-26T10:58:33.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.099+0000 7f1d3a000640 1 -- 172.21.15.144:0/755094235 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f1d300af900 con 0x7f1d3c1099f0 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1d240b3ca0 msgr2=0x7f1d240b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1d240b3ca0 0x7f1d240b61b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009230 tx=0x7f1d28031400 comp rx=0 tx=0).stop 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 msgr2=0x7f1d3c1a0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 0x7f1d3c1a0350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000e9f0 tx=0x7f1d3000eeb0 comp rx=0 tx=0).stop 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 shutdown_connections 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1d240b3ca0 0x7f1d240b61b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:33.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.103+0000 7f1d42800640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1d3c10fc80 0x7f1d3c1a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:33.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.104+0000 7f1d42800640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d3c10f330 0x7f1d3c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:33.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.104+0000 7f1d42800640 1 --2- 172.21.15.144:0/755094235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d3c1099f0 0x7f1d3c1a0350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:33.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.104+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 >> 172.21.15.144:0/755094235 conn(0x7f1d3c078ff0 msgr2=0x7f1d3c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:33.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.104+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 shutdown_connections 2024-07-26T10:58:33.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:33.104+0000 7f1d42800640 1 -- 172.21.15.144:0/755094235 wait complete. 2024-07-26T10:58:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:33 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/755094235' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T10:58:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:33 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T10:58:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:33 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/755094235' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T10:58:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:33 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T10:58:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:33 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/755094235' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T10:58:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:33 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T10:58:33.967 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-07-26T10:58:33.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-07-26T10:58:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:34 smithi144 ceph-mon[32129]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:34 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T10:58:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:34 smithi144 ceph-mon[32129]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:58:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:34 smithi183 ceph-mon[37273]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:34 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T10:58:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:34 smithi183 ceph-mon[37273]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:58:34.437 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:34 smithi160 ceph-mon[36715]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:34 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T10:58:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:34 smithi160 ceph-mon[36715]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:58:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:36 smithi183 ceph-mon[37273]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:36.395 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:36 smithi144 ceph-mon[32129]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:36 smithi160 ceph-mon[36715]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:36.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 -- 172.21.15.144:0/1056850063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104c80 msgr2=0x7fb0fc10d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 --2- 172.21.15.144:0/1056850063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104c80 0x7fb0fc10d3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009860 tx=0x7fb0e402a340 comp rx=0 tx=0).stop 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 -- 172.21.15.144:0/1056850063 shutdown_connections 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 --2- 172.21.15.144:0/1056850063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0fc10d8e0 0x7fb0fc10fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 --2- 172.21.15.144:0/1056850063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104c80 0x7fb0fc10d3a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 --2- 172.21.15.144:0/1056850063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104330 0x7fb0fc104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 -- 172.21.15.144:0/1056850063 >> 172.21.15.144:0/1056850063 conn(0x7fb0fc079150 msgr2=0x7fb0fc079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 -- 172.21.15.144:0/1056850063 shutdown_connections 2024-07-26T10:58:36.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.598+0000 7fb103a00640 1 -- 172.21.15.144:0/1056850063 wait complete. 2024-07-26T10:58:36.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.599+0000 7fb103a00640 1 Processor -- start 2024-07-26T10:58:36.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 -- start start 2024-07-26T10:58:36.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104330 0x7fb0fc19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 0x7fb0fc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0fc10d8e0 0x7fb0fc1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc113490 con 0x7fb0fc10d8e0 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc113310 con 0x7fb0fc104c80 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb103a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc113610 con 0x7fb0fc104330 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb0f3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 0x7fb0fc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb0f3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 0x7fb0fc19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:37486/0 (socket says 172.21.15.144:37486) 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb0f3e00640 1 -- 172.21.15.144:0/1833404558 learned_addr learned my addr 172.21.15.144:0/1833404558 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.600+0000 7fb101a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0fc10d8e0 0x7fb0fc1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:36.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb101000640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104330 0x7fb0fc19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb0f3e00640 1 -- 172.21.15.144:0/1833404558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104330 msgr2=0x7fb0fc19c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb0f3e00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104330 0x7fb0fc19c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb0f3e00640 1 -- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0fc10d8e0 msgr2=0x7fb0fc1a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb0f3e00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0fc10d8e0 0x7fb0fc1a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb0f3e00640 1 -- 172.21.15.144:0/1833404558 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc1a2190 con 0x7fb0fc104c80 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.601+0000 7fb0f3e00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 0x7fb0fc19c7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4035040 tx=0x7fb0e4032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.602+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4037040 con 0x7fb0fc104c80 2024-07-26T10:58:36.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.602+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0fc1a2420 con 0x7fb0fc104c80 2024-07-26T10:58:36.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.602+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0e4030c10 con 0x7fb0fc104c80 2024-07-26T10:58:36.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.602+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40035f0 con 0x7fb0fc104c80 2024-07-26T10:58:36.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.602+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0fc1a28a0 con 0x7fb0fc104c80 2024-07-26T10:58:36.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.603+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0fc104740 con 0x7fb0fc104c80 2024-07-26T10:58:36.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.606+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb0e4043020 con 0x7fb0fc104c80 2024-07-26T10:58:36.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.607+0000 7fb0fa000640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb0e00b3c20 0x7fb0e00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:36.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.607+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fb0e4103410 con 0x7fb0fc104c80 2024-07-26T10:58:36.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.607+0000 7fb101000640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb0e00b3c20 0x7fb0e00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:36.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.608+0000 7fb101000640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb0e00b3c20 0x7fb0e00b6130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009bd0 tx=0x7fb0ec005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:36.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.609+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb0e404a060 con 0x7fb0fc104c80 2024-07-26T10:58:36.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:36.848+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fb0fc1a8e60 con 0x7fb0fc104c80 2024-07-26T10:58:37.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.121+0000 7fb0fa000640 1 -- 172.21.15.144:0/1833404558 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fb0e40328c0 con 0x7fb0fc104c80 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb0e00b3c20 msgr2=0x7fb0e00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb0e00b3c20 0x7fb0e00b6130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009bd0 tx=0x7fb0ec005f90 comp rx=0 tx=0).stop 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 msgr2=0x7fb0fc19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 0x7fb0fc19c7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4035040 tx=0x7fb0e4032d50 comp rx=0 tx=0).stop 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 shutdown_connections 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb0e00b3c20 0x7fb0e00b6130 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0fc10d8e0 0x7fb0fc1a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0fc104c80 0x7fb0fc19c7c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 --2- 172.21.15.144:0/1833404558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0fc104330 0x7fb0fc19c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.124+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 >> 172.21.15.144:0/1833404558 conn(0x7fb0fc079150 msgr2=0x7fb0fc10ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.125+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 shutdown_connections 2024-07-26T10:58:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:37.125+0000 7fb103a00640 1 -- 172.21.15.144:0/1833404558 wait complete. 2024-07-26T10:58:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:37 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1833404558' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-07-26T10:58:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:37 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-07-26T10:58:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:37 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1833404558' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-07-26T10:58:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:37 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-07-26T10:58:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:37 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1833404558' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-07-26T10:58:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:37 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-07-26T10:58:38.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-07-26T10:58:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:38 smithi183 ceph-mon[37273]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T10:58:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T10:58:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-07-26T10:58:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T10:58:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:58:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-07-26T10:58:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:38 smithi144 ceph-mon[32129]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T10:58:38.543 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T10:58:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T10:58:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-07-26T10:58:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T10:58:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T10:58:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-07-26T10:58:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:38 smithi160 ceph-mon[36715]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] up:active 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-07-26T10:58:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-07-26T10:58:39.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:39 smithi144 ceph-mon[32129]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-07-26T10:58:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] up:active 2024-07-26T10:58:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T10:58:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T10:58:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T10:58:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-07-26T10:58:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-07-26T10:58:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-07-26T10:58:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:39 smithi160 ceph-mon[36715]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-07-26T10:58:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] up:active 2024-07-26T10:58:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T10:58:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T10:58:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T10:58:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-07-26T10:58:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-07-26T10:58:39.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-07-26T10:58:39.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:39 smithi183 ceph-mon[37273]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-07-26T10:58:40.379 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:40 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] up:active 2024-07-26T10:58:40.380 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:40 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-07-26T10:58:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:40 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] up:active 2024-07-26T10:58:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:40 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-07-26T10:58:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:40 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] up:active 2024-07-26T10:58:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:40 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-07-26T10:58:40.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 -- 172.21.15.144:0/1489979741 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 msgr2=0x7fb43c10d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 --2- 172.21.15.144:0/1489979741 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c10d0f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb428009860 tx=0x7fb42802a340 comp rx=0 tx=0).stop 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 -- 172.21.15.144:0/1489979741 shutdown_connections 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 --2- 172.21.15.144:0/1489979741 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 0x7fb43c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 --2- 172.21.15.144:0/1489979741 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c10d0f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 --2- 172.21.15.144:0/1489979741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 0x7fb43c104490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 -- 172.21.15.144:0/1489979741 >> 172.21.15.144:0/1489979741 conn(0x7fb43c0fdcf0 msgr2=0x7fb43c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 -- 172.21.15.144:0/1489979741 shutdown_connections 2024-07-26T10:58:40.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.930+0000 7fb443000640 1 -- 172.21.15.144:0/1489979741 wait complete. 2024-07-26T10:58:40.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.933+0000 7fb443000640 1 Processor -- start 2024-07-26T10:58:40.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.933+0000 7fb443000640 1 -- start start 2024-07-26T10:58:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.933+0000 7fb443000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 0x7fb43c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.933+0000 7fb443000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.933+0000 7fb443000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 0x7fb43c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb443000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c113280 con 0x7fb43c104080 2024-07-26T10:58:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb443000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c113100 con 0x7fb43c104ab0 2024-07-26T10:58:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb443000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c113400 con 0x7fb43c10d630 2024-07-26T10:58:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 0x7fb43c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb442600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 0x7fb43c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb442600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 0x7fb43c19f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56742/0 (socket says 172.21.15.144:56742) 2024-07-26T10:58:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb442600640 1 -- 172.21.15.144:0/2137951392 learned_addr learned my addr 172.21.15.144:0/2137951392 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441200640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:40.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441200640 1 -- 172.21.15.144:0/2137951392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 msgr2=0x7fb43c19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:40.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441200640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 0x7fb43c19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:40.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441200640 1 -- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 msgr2=0x7fb43c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:40.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441200640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 0x7fb43c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:40.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.934+0000 7fb441200640 1 -- 172.21.15.144:0/2137951392 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c19fc60 con 0x7fb43c104ab0 2024-07-26T10:58:40.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.935+0000 7fb441c00640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 0x7fb43c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.935+0000 7fb441200640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c19a290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb42802a860 tx=0x7fb428032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.935+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428040020 con 0x7fb43c104ab0 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.935+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb428030e10 con 0x7fb43c104ab0 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.935+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c19fef0 con 0x7fb43c104ab0 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.935+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428037040 con 0x7fb43c104ab0 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.936+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c1a66b0 con 0x7fb43c104ab0 2024-07-26T10:58:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.937+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43c06af30 con 0x7fb43c104ab0 2024-07-26T10:58:40.941 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.939+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb428043020 con 0x7fb43c104ab0 2024-07-26T10:58:40.942 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.941+0000 7fb43a000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb4240b3c20 0x7fb4240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:40.943 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.941+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fb428104390 con 0x7fb43c104ab0 2024-07-26T10:58:40.943 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.941+0000 7fb441c00640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb4240b3c20 0x7fb4240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:40.943 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.942+0000 7fb441c00640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb4240b3c20 0x7fb4240b6130 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb4300096c0 tx=0x7fb430009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:40.944 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:40.942+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb42804a020 con 0x7fb43c104ab0 2024-07-26T10:58:41.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:41.183+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb43c1a6880 con 0x7fb43c104ab0 2024-07-26T10:58:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:41 smithi144 ceph-mon[32129]: pgmap v345: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 3 op/s 2024-07-26T10:58:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:41 smithi160 ceph-mon[36715]: pgmap v345: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 3 op/s 2024-07-26T10:58:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:41 smithi183 ceph-mon[37273]: pgmap v345: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 3 op/s 2024-07-26T10:58:42.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.178+0000 7fb43a000640 1 -- 172.21.15.144:0/2137951392 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7fb4280cb550 con 0x7fb43c104ab0 2024-07-26T10:58:42.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.182+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb4240b3c20 msgr2=0x7fb4240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:42.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.182+0000 7fb443000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb4240b3c20 0x7fb4240b6130 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb4300096c0 tx=0x7fb430009260 comp rx=0 tx=0).stop 2024-07-26T10:58:42.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.182+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 msgr2=0x7fb43c19a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.182+0000 7fb443000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c19a290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb42802a860 tx=0x7fb428032d50 comp rx=0 tx=0).stop 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 shutdown_connections 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb4240b3c20 0x7fb4240b6130 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb43c10d630 0x7fb43c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb43c104ab0 0x7fb43c19a290 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 --2- 172.21.15.144:0/2137951392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c104080 0x7fb43c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 >> 172.21.15.144:0/2137951392 conn(0x7fb43c0fdcf0 msgr2=0x7fb43c107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:42.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 shutdown_connections 2024-07-26T10:58:42.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:42.183+0000 7fb443000640 1 -- 172.21.15.144:0/2137951392 wait complete. 2024-07-26T10:58:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:42 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2137951392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T10:58:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:42 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T10:58:42.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:42 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2137951392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T10:58:42.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:42 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T10:58:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:42 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2137951392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T10:58:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:42 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T10:58:43.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-26T10:58:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:43 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T10:58:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:43 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-07-26T10:58:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:43 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T10:58:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:43 smithi144 ceph-mon[32129]: pgmap v346: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-07-26T10:58:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:43 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T10:58:43.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:43 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-07-26T10:58:43.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:43 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T10:58:43.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:43 smithi160 ceph-mon[36715]: pgmap v346: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-07-26T10:58:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:43 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T10:58:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:43 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-07-26T10:58:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:43 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T10:58:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:43 smithi183 ceph-mon[37273]: pgmap v346: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-07-26T10:58:43.670 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-07-26T10:58:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-07-26T10:58:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-07-26T10:58:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-07-26T10:58:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-07-26T10:58:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-07-26T10:58:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-07-26T10:58:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T10:58:45.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-07-26T10:58:45.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-07-26T10:58:45.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-07-26T10:58:45.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:45 smithi144 ceph-mon[32129]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T10:58:45.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:45 smithi144 ceph-mon[32129]: pgmap v348: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-07-26T10:58:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-07-26T10:58:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-07-26T10:58:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-07-26T10:58:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:45 smithi160 ceph-mon[36715]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T10:58:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:45 smithi160 ceph-mon[36715]: pgmap v348: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-07-26T10:58:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-07-26T10:58:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-07-26T10:58:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-07-26T10:58:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:45 smithi183 ceph-mon[37273]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T10:58:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:45 smithi183 ceph-mon[37273]: pgmap v348: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s rd, 2.4 KiB/s wr, 8 op/s 2024-07-26T10:58:45.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.987+0000 7f731b200640 1 -- 172.21.15.144:0/3199300785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c075800 msgr2=0x7f731c075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:45.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.987+0000 7f731b200640 1 --2- 172.21.15.144:0/3199300785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c075800 0x7f731c075c10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7308009860 tx=0x7f730802a340 comp rx=0 tx=0).stop 2024-07-26T10:58:45.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 -- 172.21.15.144:0/3199300785 shutdown_connections 2024-07-26T10:58:45.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 --2- 172.21.15.144:0/3199300785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c113cb0 0x7f731c116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:45.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 --2- 172.21.15.144:0/3199300785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c075800 0x7f731c075c10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:45.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 --2- 172.21.15.144:0/3199300785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c076db0 0x7f731c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:45.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 -- 172.21.15.144:0/3199300785 >> 172.21.15.144:0/3199300785 conn(0x7f731c0fee70 msgr2=0x7f731c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:45.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 -- 172.21.15.144:0/3199300785 shutdown_connections 2024-07-26T10:58:45.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.988+0000 7f731b200640 1 -- 172.21.15.144:0/3199300785 wait complete. 2024-07-26T10:58:45.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 Processor -- start 2024-07-26T10:58:45.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 -- start start 2024-07-26T10:58:45.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c075800 0x7f731c1a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:45.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c076db0 0x7f731c1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 0x7f731c1a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f731c1198f0 con 0x7f731c113cb0 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f731c119770 con 0x7f731c076db0 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.989+0000 7f731b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f731c119a70 con 0x7f731c075800 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 0x7f731c1a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f7319400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c076db0 0x7f731c1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 0x7f731c1a5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43858/0 (socket says 172.21.15.144:43858) 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 -- 172.21.15.144:0/246247432 learned_addr learned my addr 172.21.15.144:0/246247432 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f7319e00640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c075800 0x7f731c1a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 -- 172.21.15.144:0/246247432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c075800 msgr2=0x7f731c1a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c075800 0x7f731c1a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 -- 172.21.15.144:0/246247432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c076db0 msgr2=0x7f731c1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c076db0 0x7f731c1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:45.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f731a800640 1 -- 172.21.15.144:0/246247432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731c1a6250 con 0x7f731c113cb0 2024-07-26T10:58:45.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.990+0000 7f7319e00640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c075800 0x7f731c1a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:45.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.991+0000 7f731a800640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 0x7f731c1a5b20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f731400e970 tx=0x7f731400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:45.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.991+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731400ccc0 con 0x7f731c113cb0 2024-07-26T10:58:45.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.991+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f731400ce60 con 0x7f731c113cb0 2024-07-26T10:58:45.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.991+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f731c1a6540 con 0x7f731c113cb0 2024-07-26T10:58:45.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.991+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7314010430 con 0x7f731c113cb0 2024-07-26T10:58:45.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.991+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f731c1acca0 con 0x7f731c113cb0 2024-07-26T10:58:45.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.992+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f731c075c10 con 0x7f731c113cb0 2024-07-26T10:58:45.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.995+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f7314004f90 con 0x7f731c113cb0 2024-07-26T10:58:46.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.998+0000 7f7312a00640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73000b3c20 0x7f73000b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:46.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.998+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f73140e4780 con 0x7f731c113cb0 2024-07-26T10:58:46.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.998+0000 7f7319e00640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73000b3c20 0x7f73000b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:46.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:45.999+0000 7f7319e00640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73000b3c20 0x7f73000b6130 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f730c005fd0 tx=0x7f730c0095c0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:46.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:46.000+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f73140ab9d0 con 0x7f731c113cb0 2024-07-26T10:58:46.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:46.242+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f731c1a1410 con 0x7f731c113cb0 2024-07-26T10:58:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:46 smithi160 ceph-mon[36715]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T10:58:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:46 smithi183 ceph-mon[37273]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T10:58:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:46 smithi144 ceph-mon[32129]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T10:58:47.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.236+0000 7f7312a00640 1 -- 172.21.15.144:0/246247432 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f731c1a1410 con 0x7f731c113cb0 2024-07-26T10:58:47.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73000b3c20 msgr2=0x7f73000b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:47.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73000b3c20 0x7f73000b6130 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f730c005fd0 tx=0x7f730c0095c0 comp rx=0 tx=0).stop 2024-07-26T10:58:47.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 msgr2=0x7f731c1a5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:47.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 0x7f731c1a5b20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f731400e970 tx=0x7f731400ed70 comp rx=0 tx=0).stop 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 shutdown_connections 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73000b3c20 0x7f73000b6130 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f731c113cb0 0x7f731c1a5b20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f731c076db0 0x7f731c1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 --2- 172.21.15.144:0/246247432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731c075800 0x7f731c1a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.241+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 >> 172.21.15.144:0/246247432 conn(0x7f731c0fee70 msgr2=0x7f731c10fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.242+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 shutdown_connections 2024-07-26T10:58:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:47.242+0000 7f731b200640 1 -- 172.21.15.144:0/246247432 wait complete. 2024-07-26T10:58:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:47 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/246247432' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T10:58:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:47 smithi160 ceph-mon[36715]: pgmap v350: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.0 KiB/s wr, 13 op/s 2024-07-26T10:58:47.622 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:47 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/246247432' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T10:58:47.622 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:47 smithi144 ceph-mon[32129]: pgmap v350: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.0 KiB/s wr, 13 op/s 2024-07-26T10:58:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:47 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/246247432' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T10:58:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:47 smithi183 ceph-mon[37273]: pgmap v350: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.0 KiB/s wr, 13 op/s 2024-07-26T10:58:48.063 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-07-26T10:58:48.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-26T10:58:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:48 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/246247432' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T10:58:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:48 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T10:58:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:48 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/246247432' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T10:58:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:48 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T10:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:48 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/246247432' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T10:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:48 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T10:58:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:49 smithi144 ceph-mon[32129]: pgmap v351: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-07-26T10:58:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:49 smithi160 ceph-mon[36715]: pgmap v351: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-07-26T10:58:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:49 smithi183 ceph-mon[37273]: pgmap v351: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-07-26T10:58:50.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 -- 172.21.15.144:0/767171343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c108d20 msgr2=0x7f8a2c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 --2- 172.21.15.144:0/767171343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c108d20 0x7f8a2c109130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009ca0 tx=0x7f8a2002a8c0 comp rx=0 tx=0).stop 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 -- 172.21.15.144:0/767171343 shutdown_connections 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 --2- 172.21.15.144:0/767171343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c109670 0x7f8a2c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 --2- 172.21.15.144:0/767171343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c108d20 0x7f8a2c109130 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 --2- 172.21.15.144:0/767171343 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c1033e0 0x7f8a2c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 -- 172.21.15.144:0/767171343 >> 172.21.15.144:0/767171343 conn(0x7f8a2c0fee70 msgr2=0x7f8a2c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 -- 172.21.15.144:0/767171343 shutdown_connections 2024-07-26T10:58:50.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.925+0000 7f8a32800640 1 -- 172.21.15.144:0/767171343 wait complete. 2024-07-26T10:58:50.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.926+0000 7f8a32800640 1 Processor -- start 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 -- start start 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c1033e0 0x7f8a2c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c108d20 0x7f8a2c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 0x7f8a2c19f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c113300 con 0x7f8a2c1033e0 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c113180 con 0x7f8a2c109670 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a32800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c113480 con 0x7f8a2c108d20 2024-07-26T10:58:50.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a31400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c1033e0 0x7f8a2c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a31e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 0x7f8a2c19f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a31400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c1033e0 0x7f8a2c199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52938/0 (socket says 172.21.15.144:52938) 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a31400640 1 -- 172.21.15.144:0/2111925959 learned_addr learned my addr 172.21.15.144:0/2111925959 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.927+0000 7f8a31e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 0x7f8a2c19f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51152/0 (socket says 172.21.15.144:51152) 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a30a00640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c108d20 0x7f8a2c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a31e00640 1 -- 172.21.15.144:0/2111925959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c108d20 msgr2=0x7f8a2c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a31e00640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c108d20 0x7f8a2c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a31e00640 1 -- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c1033e0 msgr2=0x7f8a2c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a31e00640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c1033e0 0x7f8a2c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a31e00640 1 -- 172.21.15.144:0/2111925959 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c19fca0 con 0x7f8a2c109670 2024-07-26T10:58:50.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a30a00640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c108d20 0x7f8a2c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:58:50.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a31e00640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 0x7f8a2c19f570 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400e970 tx=0x7f8a2400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:50.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.928+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2400ccc0 con 0x7f8a2c109670 2024-07-26T10:58:50.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.929+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2c19ff90 con 0x7f8a2c109670 2024-07-26T10:58:50.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.929+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a24005020 con 0x7f8a2c109670 2024-07-26T10:58:50.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.929+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24004a00 con 0x7f8a2c109670 2024-07-26T10:58:50.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.929+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2c1a66f0 con 0x7f8a2c109670 2024-07-26T10:58:50.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.930+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2c1037f0 con 0x7f8a2c109670 2024-07-26T10:58:50.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.932+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8a24004ba0 con 0x7f8a2c109670 2024-07-26T10:58:50.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.934+0000 7f8a2a000640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8a140b3c20 0x7f8a140b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:50.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.934+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8a240e4190 con 0x7f8a2c109670 2024-07-26T10:58:50.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.934+0000 7f8a31400640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8a140b3c20 0x7f8a140b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:50.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.935+0000 7f8a31400640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8a140b3c20 0x7f8a140b6130 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009c90 tx=0x7f8a1c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:50.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:50.935+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8a240ab380 con 0x7f8a2c109670 2024-07-26T10:58:51.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.161+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8a2c063230 con 0x7f8a140b3c20 2024-07-26T10:58:51.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.200+0000 7f8a2a000640 1 -- 172.21.15.144:0/2111925959 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8a2c063230 con 0x7f8a140b3c20 2024-07-26T10:58:51.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8a140b3c20 msgr2=0x7f8a140b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8a140b3c20 0x7f8a140b6130 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009c90 tx=0x7f8a1c005f90 comp rx=0 tx=0).stop 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 msgr2=0x7f8a2c19f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 0x7f8a2c19f570 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400e970 tx=0x7f8a2400ee30 comp rx=0 tx=0).stop 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 shutdown_connections 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8a140b3c20 0x7f8a140b6130 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a2c109670 0x7f8a2c19f570 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2c108d20 0x7f8a2c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.203+0000 7f8a32800640 1 --2- 172.21.15.144:0/2111925959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a2c1033e0 0x7f8a2c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.204+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 >> 172.21.15.144:0/2111925959 conn(0x7f8a2c0fee70 msgr2=0x7f8a2c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.204+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 shutdown_connections 2024-07-26T10:58:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:51.204+0000 7f8a32800640 1 -- 172.21.15.144:0/2111925959 wait complete. 2024-07-26T10:58:51.470 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:58:51 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:58:51.173+0000 7f4a6be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T10:58:51.471 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:58:51 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:58:51.173+0000 7f4a6be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T10:58:51.471 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:58:51 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:58:51.173+0000 7f4a6be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T10:58:51.471 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:58:51 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:58:51.174+0000 7f4a6be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T10:58:51.471 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 10:58:51 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T10:58:51.174+0000 7f4a6be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T10:58:52.074 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-26T10:58:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:52 smithi160 ceph-mon[36715]: pgmap v352: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 127 B/s wr, 8 op/s 2024-07-26T10:58:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:52 smithi183 ceph-mon[37273]: pgmap v352: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 127 B/s wr, 8 op/s 2024-07-26T10:58:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:52 smithi144 ceph-mon[32129]: pgmap v352: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 127 B/s wr, 8 op/s 2024-07-26T10:58:52.524 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:53 smithi183 ceph-mon[37273]: from='client.24896 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:58:53.457 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:53 smithi144 ceph-mon[32129]: from='client.24896 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:58:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:53 smithi160 ceph-mon[36715]: from='client.24896 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:58:54.383 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:54 smithi144 ceph-mon[32129]: pgmap v353: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 116 B/s wr, 7 op/s 2024-07-26T10:58:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:54 smithi183 ceph-mon[37273]: pgmap v353: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 116 B/s wr, 7 op/s 2024-07-26T10:58:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:54 smithi160 ceph-mon[36715]: pgmap v353: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 116 B/s wr, 7 op/s 2024-07-26T10:58:54.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 -- 172.21.15.144:0/4014589493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10b860 msgr2=0x7f150c112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:54.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 --2- 172.21.15.144:0/4014589493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10b860 0x7f150c112090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1504009ca0 tx=0x7f150402a8c0 comp rx=0 tx=0).stop 2024-07-26T10:58:54.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 -- 172.21.15.144:0/4014589493 shutdown_connections 2024-07-26T10:58:54.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 --2- 172.21.15.144:0/4014589493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10b860 0x7f150c112090 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:54.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 --2- 172.21.15.144:0/4014589493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10af10 0x7f150c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:54.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 --2- 172.21.15.144:0/4014589493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:54.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 -- 172.21.15.144:0/4014589493 >> 172.21.15.144:0/4014589493 conn(0x7f150c101040 msgr2=0x7f150c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:54.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 -- 172.21.15.144:0/4014589493 shutdown_connections 2024-07-26T10:58:54.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.798+0000 7f150b200640 1 -- 172.21.15.144:0/4014589493 wait complete. 2024-07-26T10:58:54.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.800+0000 7f150b200640 1 Processor -- start 2024-07-26T10:58:54.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.800+0000 7f150b200640 1 -- start start 2024-07-26T10:58:54.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.800+0000 7f150b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c19bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:54.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.800+0000 7f150b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 0x7f150c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:54.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.800+0000 7f150b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10b860 0x7f150c1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:54.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.800+0000 7f150b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f150c115510 con 0x7f150c10b860 2024-07-26T10:58:54.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f150b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f150c115390 con 0x7f150c10af10 2024-07-26T10:58:54.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f150b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f150c115690 con 0x7f150c1055d0 2024-07-26T10:58:54.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c19bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:54.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 0x7f150c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:54.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c19bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:49924/0 (socket says 172.21.15.144:49924) 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 0x7f150c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51190/0 (socket says 172.21.15.144:51190) 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 -- 172.21.15.144:0/58844375 learned_addr learned my addr 172.21.15.144:0/58844375 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f150a800640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10b860 0x7f150c1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 -- 172.21.15.144:0/58844375 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 msgr2=0x7f150c19c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 0x7f150c19c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 -- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10b860 msgr2=0x7f150c1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10b860 0x7f150c1a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509e00640 1 -- 172.21.15.144:0/58844375 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c1a1e30 con 0x7f150c1055d0 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f1509400640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 0x7f150c19c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.801+0000 7f150a800640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10b860 0x7f150c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:58:54.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.802+0000 7f1509e00640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c19bec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00e9f0 tx=0x7f14fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:54.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.802+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00cd80 con 0x7f150c1055d0 2024-07-26T10:58:54.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.802+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c1a2120 con 0x7f150c1055d0 2024-07-26T10:58:54.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.802+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14fc0108e0 con 0x7f150c1055d0 2024-07-26T10:58:54.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.802+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc018980 con 0x7f150c1055d0 2024-07-26T10:58:54.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.802+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c1a8880 con 0x7f150c1055d0 2024-07-26T10:58:54.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.803+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c1059e0 con 0x7f150c1055d0 2024-07-26T10:58:54.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.806+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f14fc010430 con 0x7f150c1055d0 2024-07-26T10:58:54.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.807+0000 7f1502a00640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14f00b3cb0 0x7f14f00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:54.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.808+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f14fc0e4b60 con 0x7f150c1055d0 2024-07-26T10:58:54.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.808+0000 7f1509400640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14f00b3cb0 0x7f14f00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:54.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.808+0000 7f1509400640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14f00b3cb0 0x7f14f00b61c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f150c10aa30 tx=0x7f14f8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:54.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:54.809+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14fc0abb60 con 0x7f150c1055d0 2024-07-26T10:58:55.036 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.034+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f150c063230 con 0x7f14f00b3cb0 2024-07-26T10:58:55.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.076+0000 7f1502a00640 1 -- 172.21.15.144:0/58844375 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f150c063230 con 0x7f14f00b3cb0 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.079+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14f00b3cb0 msgr2=0x7f14f00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.079+0000 7f150b200640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14f00b3cb0 0x7f14f00b61c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f150c10aa30 tx=0x7f14f8005f90 comp rx=0 tx=0).stop 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.079+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 msgr2=0x7f150c19bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c19bec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00e9f0 tx=0x7f14fc00eeb0 comp rx=0 tx=0).stop 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 shutdown_connections 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14f00b3cb0 0x7f14f00b61c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150c10b860 0x7f150c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f150c10af10 0x7f150c19c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 --2- 172.21.15.144:0/58844375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f150c1055d0 0x7f150c19bec0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 >> 172.21.15.144:0/58844375 conn(0x7f150c101040 msgr2=0x7f150c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:55.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 shutdown_connections 2024-07-26T10:58:55.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:55.080+0000 7f150b200640 1 -- 172.21.15.144:0/58844375 wait complete. 2024-07-26T10:58:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:55 smithi183 ceph-mon[37273]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T10:58:55.444 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:55 smithi144 ceph-mon[32129]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T10:58:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:55 smithi160 ceph-mon[36715]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T10:58:55.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-26T10:58:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:56 smithi144 ceph-mon[32129]: pgmap v354: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T10:58:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:56 smithi144 ceph-mon[32129]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:58:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:56 smithi183 ceph-mon[37273]: pgmap v354: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T10:58:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:56 smithi183 ceph-mon[37273]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:58:56.460 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:58:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:56 smithi160 ceph-mon[36715]: pgmap v354: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T10:58:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:56 smithi160 ceph-mon[36715]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:58:58.358 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:58:58 smithi144 ceph-mon[32129]: pgmap v355: 129 pgs: 129 active+clean; 604 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-07-26T10:58:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:58:58 smithi183 ceph-mon[37273]: pgmap v355: 129 pgs: 129 active+clean; 604 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-07-26T10:58:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:58:58 smithi160 ceph-mon[36715]: pgmap v355: 129 pgs: 129 active+clean; 604 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-07-26T10:58:58.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 -- 172.21.15.144:0/4220947501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 msgr2=0x7f6be4102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 --2- 172.21.15.144:0/4220947501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be4102250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009900 tx=0x7f6bd402a480 comp rx=0 tx=0).stop 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 -- 172.21.15.144:0/4220947501 shutdown_connections 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 --2- 172.21.15.144:0/4220947501 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be410f9f0 0x7f6be4111e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 --2- 172.21.15.144:0/4220947501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be4102790 0x7f6be410f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 --2- 172.21.15.144:0/4220947501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be4102250 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 -- 172.21.15.144:0/4220947501 >> 172.21.15.144:0/4220947501 conn(0x7f6be40fdd20 msgr2=0x7f6be4100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 -- 172.21.15.144:0/4220947501 shutdown_connections 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.730+0000 7f6be9000640 1 -- 172.21.15.144:0/4220947501 wait complete. 2024-07-26T10:58:58.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 Processor -- start 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 -- start start 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be4102790 0x7f6be419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be410f9f0 0x7f6be41a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6be41154f0 con 0x7f6be4101e40 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6be4115370 con 0x7f6be4102790 2024-07-26T10:58:58.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.732+0000 7f6be9000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6be4115670 con 0x7f6be410f9f0 2024-07-26T10:58:58.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:58.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53000/0 (socket says 172.21.15.144:53000) 2024-07-26T10:58:58.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be3400640 1 -- 172.21.15.144:0/3090237664 learned_addr learned my addr 172.21.15.144:0/3090237664 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:58:58.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be3e00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be410f9f0 0x7f6be41a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be2a00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be4102790 0x7f6be419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be2a00640 1 -- 172.21.15.144:0/3090237664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be410f9f0 msgr2=0x7f6be41a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be2a00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be410f9f0 0x7f6be41a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be2a00640 1 -- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 msgr2=0x7f6be419bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be2a00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be419bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be2a00640 1 -- 172.21.15.144:0/3090237664 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be41a1e90 con 0x7f6be4102790 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be3e00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be410f9f0 0x7f6be41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.733+0000 7f6be3400640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be419bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.734+0000 7f6be2a00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be4102790 0x7f6be419c4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0005fd0 tx=0x7f6bd000dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:58.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.734+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0009c30 con 0x7f6be4102790 2024-07-26T10:58:58.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.734+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be41a2180 con 0x7f6be4102790 2024-07-26T10:58:58.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.734+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd0009dd0 con 0x7f6be4102790 2024-07-26T10:58:58.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.734+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0003e20 con 0x7f6be4102790 2024-07-26T10:58:58.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.734+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be41a88e0 con 0x7f6be4102790 2024-07-26T10:58:58.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.736+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be4102250 con 0x7f6be4102790 2024-07-26T10:58:58.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.738+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f6bd0004000 con 0x7f6be4102790 2024-07-26T10:58:58.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.739+0000 7f6bdbe00640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6bc80b3cb0 0x7f6bc80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:58.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.739+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6bd00e3e50 con 0x7f6be4102790 2024-07-26T10:58:58.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.740+0000 7f6be3400640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6bc80b3cb0 0x7f6bc80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:58.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.740+0000 7f6be3400640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6bc80b3cb0 0x7f6bc80b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40096c0 tx=0x7f6bd4005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:58.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.743+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6bd0017020 con 0x7f6be4102790 2024-07-26T10:58:58.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.977+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f6be4063230 con 0x7f6bc80b3cb0 2024-07-26T10:58:58.985 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.983+0000 7f6bdbe00640 1 -- 172.21.15.144:0/3090237664 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6be4063230 con 0x7f6bc80b3cb0 2024-07-26T10:58:58.985 INFO:teuthology.orchestra.run.smithi144.stdout:/volumes/qa/sv_0/71061eed-03dc-4b80-a2df-4f39011840ef 2024-07-26T10:58:58.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6bc80b3cb0 msgr2=0x7f6bc80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:58.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6bc80b3cb0 0x7f6bc80b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40096c0 tx=0x7f6bd4005a20 comp rx=0 tx=0).stop 2024-07-26T10:58:58.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be4102790 msgr2=0x7f6be419c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:58:58.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be4102790 0x7f6be419c4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0005fd0 tx=0x7f6bd000dde0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 shutdown_connections 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6bc80b3cb0 0x7f6bc80b61c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6be410f9f0 0x7f6be41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6be4102790 0x7f6be419c4c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 --2- 172.21.15.144:0/3090237664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be4101e40 0x7f6be419bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 >> 172.21.15.144:0/3090237664 conn(0x7f6be40fdd20 msgr2=0x7f6be4111b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 shutdown_connections 2024-07-26T10:58:58.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:58:58.986+0000 7f6be9000640 1 -- 172.21.15.144:0/3090237664 wait complete. 2024-07-26T10:58:59.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-26T10:59:00.273 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:00 smithi183 ceph-mon[37273]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:00 smithi183 ceph-mon[37273]: pgmap v356: 129 pgs: 129 active+clean; 605 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-07-26T10:59:00.425 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:00 smithi144 ceph-mon[32129]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:00.425 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:00 smithi144 ceph-mon[32129]: pgmap v356: 129 pgs: 129 active+clean; 605 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-07-26T10:59:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:00 smithi160 ceph-mon[36715]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:00 smithi160 ceph-mon[36715]: pgmap v356: 129 pgs: 129 active+clean; 605 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-07-26T10:59:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:02 smithi144 ceph-mon[32129]: pgmap v357: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-07-26T10:59:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:02 smithi183 ceph-mon[37273]: pgmap v357: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-07-26T10:59:02.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 -- 172.21.15.144:0/302856523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f4077370 msgr2=0x7f46f4077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:02.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 --2- 172.21.15.144:0/302856523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f4077370 0x7f46f4077780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009900 tx=0x7f46e402a3f0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 -- 172.21.15.144:0/302856523 shutdown_connections 2024-07-26T10:59:02.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 --2- 172.21.15.144:0/302856523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46f41141a0 0x7f46f41165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 --2- 172.21.15.144:0/302856523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46f4077cc0 0x7f46f4113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 --2- 172.21.15.144:0/302856523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f4077370 0x7f46f4077780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 -- 172.21.15.144:0/302856523 >> 172.21.15.144:0/302856523 conn(0x7f46f41011f0 msgr2=0x7f46f4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:02.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 -- 172.21.15.144:0/302856523 shutdown_connections 2024-07-26T10:59:02.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.484+0000 7f46faa00640 1 -- 172.21.15.144:0/302856523 wait complete. 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.485+0000 7f46faa00640 1 Processor -- start 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.485+0000 7f46faa00640 1 -- start start 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46faa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46f4077370 0x7f46f41a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46faa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46f4077cc0 0x7f46f41a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46faa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 0x7f46f41a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46faa00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4119b20 con 0x7f46f41141a0 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46faa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41199a0 con 0x7f46f4077cc0 2024-07-26T10:59:02.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46faa00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4119ca0 con 0x7f46f4077370 2024-07-26T10:59:02.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46f8a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 0x7f46f41a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:02.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46f8a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 0x7f46f41a5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54714/0 (socket says 172.21.15.144:54714) 2024-07-26T10:59:02.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46f8a00640 1 -- 172.21.15.144:0/1803531053 learned_addr learned my addr 172.21.15.144:0/1803531053 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:02.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46f3e00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46f4077370 0x7f46f41a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:02.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.486+0000 7f46ebe00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46f4077cc0 0x7f46f41a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f8a00640 1 -- 172.21.15.144:0/1803531053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46f4077370 msgr2=0x7f46f41a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f8a00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46f4077370 0x7f46f41a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f8a00640 1 -- 172.21.15.144:0/1803531053 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46f4077cc0 msgr2=0x7f46f41a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f8a00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46f4077cc0 0x7f46f41a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f8a00640 1 -- 172.21.15.144:0/1803531053 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f41a6460 con 0x7f46f41141a0 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f8a00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 0x7f46f41a5d30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00eb10 tx=0x7f46ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec004b20 con 0x7f46f41141a0 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f41a6750 con 0x7f46f41141a0 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.487+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec004cc0 con 0x7f46f41141a0 2024-07-26T10:59:02.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.488+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec010670 con 0x7f46f41141a0 2024-07-26T10:59:02.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.488+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41aceb0 con 0x7f46f41141a0 2024-07-26T10:59:02.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.489+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f4077780 con 0x7f46f41141a0 2024-07-26T10:59:02.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.491+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f46ec005000 con 0x7f46f41141a0 2024-07-26T10:59:02.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.493+0000 7f46f1600640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f46d80b3d30 0x7f46d80b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:02.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.493+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f46ec0e4990 con 0x7f46f41141a0 2024-07-26T10:59:02.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.494+0000 7f46f3e00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f46d80b3d30 0x7f46d80b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:02.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.494+0000 7f46f3e00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f46d80b3d30 0x7f46d80b6240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f46e4031440 tx=0x7f46e4009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:02.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.496+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f46ec055600 con 0x7f46f41141a0 2024-07-26T10:59:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:02 smithi160 ceph-mon[36715]: pgmap v357: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-07-26T10:59:02.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.722+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f46f4063230 con 0x7f46d80b3d30 2024-07-26T10:59:02.763 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.761+0000 7f46f1600640 1 -- 172.21.15.144:0/1803531053 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f46f4063230 con 0x7f46d80b3d30 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f46d80b3d30 msgr2=0x7f46d80b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f46d80b3d30 0x7f46d80b6240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f46e4031440 tx=0x7f46e4009560 comp rx=0 tx=0).stop 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 msgr2=0x7f46f41a5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 0x7f46f41a5d30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00eb10 tx=0x7f46ec00c2d0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 shutdown_connections 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f46d80b3d30 0x7f46d80b6240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46f41141a0 0x7f46f41a5d30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46f4077cc0 0x7f46f41a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.764+0000 7f46faa00640 1 --2- 172.21.15.144:0/1803531053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46f4077370 0x7f46f41a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.765+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 >> 172.21.15.144:0/1803531053 conn(0x7f46f41011f0 msgr2=0x7f46f4103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.765+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 shutdown_connections 2024-07-26T10:59:02.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:02.765+0000 7f46faa00640 1 -- 172.21.15.144:0/1803531053 wait complete. 2024-07-26T10:59:03.726 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-26T10:59:04.181 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:04 smithi183 ceph-mon[37273]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:04 smithi183 ceph-mon[37273]: pgmap v358: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:04 smithi144 ceph-mon[32129]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:04 smithi144 ceph-mon[32129]: pgmap v358: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:04 smithi160 ceph-mon[36715]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:04 smithi160 ceph-mon[36715]: pgmap v358: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:06 smithi144 ceph-mon[32129]: pgmap v359: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-07-26T10:59:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:06 smithi183 ceph-mon[37273]: pgmap v359: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-07-26T10:59:06.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 -- 172.21.15.144:0/949839646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 msgr2=0x7fd5441023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:06.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 --2- 172.21.15.144:0/949839646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441023f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd534009930 tx=0x7fd53402a440 comp rx=0 tx=0).stop 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 -- 172.21.15.144:0/949839646 shutdown_connections 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 --2- 172.21.15.144:0/949839646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd544107420 0x7fd544107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 --2- 172.21.15.144:0/949839646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 0x7fd544106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 --2- 172.21.15.144:0/949839646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441023f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 -- 172.21.15.144:0/949839646 >> 172.21.15.144:0/949839646 conn(0x7fd5440fdcf0 msgr2=0x7fd544100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 -- 172.21.15.144:0/949839646 shutdown_connections 2024-07-26T10:59:06.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.441+0000 7fd543200640 1 -- 172.21.15.144:0/949839646 wait complete. 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.442+0000 7fd543200640 1 Processor -- start 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.442+0000 7fd543200640 1 -- start start 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd543200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd543200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 0x7fd5441a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd543200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd544107420 0x7fd5441a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd543200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441198c0 con 0x7fd544101fe0 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd543200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd544119740 con 0x7fd544107420 2024-07-26T10:59:06.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd543200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd544119a40 con 0x7fd544102930 2024-07-26T10:59:06.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd541e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:06.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd541e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441a0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54738/0 (socket says 172.21.15.144:54738) 2024-07-26T10:59:06.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd541e00640 1 -- 172.21.15.144:0/2610941488 learned_addr learned my addr 172.21.15.144:0/2610941488 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:06.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd541400640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 0x7fd5441a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:06.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.443+0000 7fd542800640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd544107420 0x7fd5441a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541400640 1 -- 172.21.15.144:0/2610941488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd544107420 msgr2=0x7fd5441a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541400640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd544107420 0x7fd5441a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541400640 1 -- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 msgr2=0x7fd5441a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541400640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541400640 1 -- 172.21.15.144:0/2610941488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5441a6300 con 0x7fd544102930 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541e00640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.444+0000 7fd541400640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 0x7fd5441a0930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd54406a4c0 tx=0x7fd53000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.445+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53000ccc0 con 0x7fd544102930 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.445+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5441a65f0 con 0x7fd544102930 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.445+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5300035f0 con 0x7fd544102930 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.445+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53000ce60 con 0x7fd544102930 2024-07-26T10:59:06.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.445+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441acd50 con 0x7fd544102930 2024-07-26T10:59:06.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.446+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd544106e60 con 0x7fd544102930 2024-07-26T10:59:06.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.451+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fd530004000 con 0x7fd544102930 2024-07-26T10:59:06.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.452+0000 7fd53aa00640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd5280b3cb0 0x7fd5280b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:06.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.452+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd5300e4650 con 0x7fd544102930 2024-07-26T10:59:06.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.453+0000 7fd541e00640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd5280b3cb0 0x7fd5280b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:06.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.453+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd5300e4ad0 con 0x7fd544102930 2024-07-26T10:59:06.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.454+0000 7fd541e00640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd5280b3cb0 0x7fd5280b61c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd534000c00 tx=0x7fd5340058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:06 smithi160 ceph-mon[36715]: pgmap v359: 129 pgs: 129 active+clean; 609 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-07-26T10:59:06.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.681+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fd544106550 con 0x7fd5280b3cb0 2024-07-26T10:59:06.689 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.688+0000 7fd53aa00640 1 -- 172.21.15.144:0/2610941488 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd544106550 con 0x7fd5280b3cb0 2024-07-26T10:59:06.690 INFO:teuthology.orchestra.run.smithi144.stdout:/volumes/qa/sv_1/fe73673a-5e07-410e-a5c2-219a3360d93b 2024-07-26T10:59:06.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.690+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd5280b3cb0 msgr2=0x7fd5280b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:06.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.690+0000 7fd543200640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd5280b3cb0 0x7fd5280b61c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd534000c00 tx=0x7fd5340058d0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.690+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 msgr2=0x7fd5441a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:06.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.690+0000 7fd543200640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 0x7fd5441a0930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd54406a4c0 tx=0x7fd53000ed90 comp rx=0 tx=0).stop 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 shutdown_connections 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd5280b3cb0 0x7fd5280b61c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd544107420 0x7fd5441a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd544102930 0x7fd5441a0930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 --2- 172.21.15.144:0/2610941488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd544101fe0 0x7fd5441a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 >> 172.21.15.144:0/2610941488 conn(0x7fd5440fdcf0 msgr2=0x7fd544104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:06.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 shutdown_connections 2024-07-26T10:59:06.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:06.691+0000 7fd543200640 1 -- 172.21.15.144:0/2610941488 wait complete. 2024-07-26T10:59:07.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T10:59:07.926 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:08 smithi183 ceph-mon[37273]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:08 smithi183 ceph-mon[37273]: pgmap v360: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-07-26T10:59:08.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:08 smithi144 ceph-mon[32129]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:08.410 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:08 smithi144 ceph-mon[32129]: pgmap v360: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-07-26T10:59:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:08 smithi160 ceph-mon[36715]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T10:59:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:08 smithi160 ceph-mon[36715]: pgmap v360: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-07-26T10:59:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:10 smithi144 ceph-mon[32129]: pgmap v361: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 -- 172.21.15.144:0/2163888920 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 msgr2=0x7fc37010f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 --2- 172.21.15.144:0/2163888920 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 0x7fc37010f400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc358009900 tx=0x7fc35802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 -- 172.21.15.144:0/2163888920 shutdown_connections 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 --2- 172.21.15.144:0/2163888920 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc37010f940 0x7fc370111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 --2- 172.21.15.144:0/2163888920 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 0x7fc37010f400 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 --2- 172.21.15.144:0/2163888920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc3700755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 -- 172.21.15.144:0/2163888920 >> 172.21.15.144:0/2163888920 conn(0x7fc3700fee70 msgr2=0x7fc3701012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:10.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 -- 172.21.15.144:0/2163888920 shutdown_connections 2024-07-26T10:59:10.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.221+0000 7fc376000640 1 -- 172.21.15.144:0/2163888920 wait complete. 2024-07-26T10:59:10.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.222+0000 7fc376000640 1 Processor -- start 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.222+0000 7fc376000640 1 -- start start 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc376000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc37019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc376000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 0x7fc37019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc376000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc37010f940 0x7fc3701a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc376000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc370115530 con 0x7fc3700751e0 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc376000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc3701153b0 con 0x7fc370075b30 2024-07-26T10:59:10.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc376000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc3701156b0 con 0x7fc37010f940 2024-07-26T10:59:10.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc374c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc37019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc374c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc37019bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53430/0 (socket says 172.21.15.144:53430) 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc374c00640 1 -- 172.21.15.144:0/2270385312 learned_addr learned my addr 172.21.15.144:0/2270385312 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc36fe00640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 0x7fc37019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.223+0000 7fc375600640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc37010f940 0x7fc3701a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc374c00640 1 -- 172.21.15.144:0/2270385312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc37010f940 msgr2=0x7fc3701a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc374c00640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc37010f940 0x7fc3701a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc374c00640 1 -- 172.21.15.144:0/2270385312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 msgr2=0x7fc37019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc374c00640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 0x7fc37019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc374c00640 1 -- 172.21.15.144:0/2270385312 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3701a1f30 con 0x7fc3700751e0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc374c00640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc37019bfc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc36800ebc0 tx=0x7fc36800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc368010630 con 0x7fc3700751e0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.224+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc368010c80 con 0x7fc3700751e0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.225+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc368018b60 con 0x7fc3700751e0 2024-07-26T10:59:10.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.225+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3701a2220 con 0x7fc3700751e0 2024-07-26T10:59:10.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.225+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3701a8980 con 0x7fc3700751e0 2024-07-26T10:59:10.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.226+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3700766f0 con 0x7fc3700751e0 2024-07-26T10:59:10.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.229+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fc368018d00 con 0x7fc3700751e0 2024-07-26T10:59:10.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.230+0000 7fc36d600640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc3540b3d30 0x7fc3540b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:10.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.230+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc3680e47a0 con 0x7fc3700751e0 2024-07-26T10:59:10.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.230+0000 7fc36fe00640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc3540b3d30 0x7fc3540b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:10.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.231+0000 7fc36fe00640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc3540b3d30 0x7fc3540b6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc358031440 tx=0x7fc358005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:10.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.235+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc3680ab1b0 con 0x7fc3700751e0 2024-07-26T10:59:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:10 smithi183 ceph-mon[37273]: pgmap v361: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-07-26T10:59:10.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.469+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3701a8bf0 con 0x7fc3700751e0 2024-07-26T10:59:10.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.470+0000 7fc36d600640 1 -- 172.21.15.144:0/2270385312 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11931 (secure 0 0 0) 0x7fc3680af700 con 0x7fc3700751e0 2024-07-26T10:59:10.473 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:10.473 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":25,"btime":"2024-07-26T10:58:47:222852+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:58:46.244128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T10:59:10.473 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T10:59:10.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.473+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc3540b3d30 msgr2=0x7fc3540b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:10.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.473+0000 7fc376000640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc3540b3d30 0x7fc3540b6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc358031440 tx=0x7fc358005a20 comp rx=0 tx=0).stop 2024-07-26T10:59:10.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.473+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 msgr2=0x7fc37019bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:10.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.473+0000 7fc376000640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc37019bfc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc36800ebc0 tx=0x7fc36800c6a0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 shutdown_connections 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc3540b3d30 0x7fc3540b6240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc37010f940 0x7fc3701a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc370075b30 0x7fc37019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 --2- 172.21.15.144:0/2270385312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3700751e0 0x7fc37019bfc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 >> 172.21.15.144:0/2270385312 conn(0x7fc3700fee70 msgr2=0x7fc370100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 shutdown_connections 2024-07-26T10:59:10.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:10.474+0000 7fc376000640 1 -- 172.21.15.144:0/2270385312 wait complete. 2024-07-26T10:59:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:10 smithi160 ceph-mon[36715]: pgmap v361: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-07-26T10:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:11 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2270385312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:11 smithi144 ceph-mon[32129]: pgmap v362: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-07-26T10:59:11.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:59:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:11 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2270385312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:59:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:11 smithi183 ceph-mon[37273]: pgmap v362: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-07-26T10:59:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:11 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2270385312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T10:59:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:11 smithi160 ceph-mon[36715]: pgmap v362: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-07-26T10:59:11.804 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:13.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 -- 172.21.15.144:0/3215718522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c0751e0 msgr2=0x7f562c0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:13.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 --2- 172.21.15.144:0/3215718522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c0751e0 0x7f562c0755f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f561c009ca0 tx=0x7f561c02a8c0 comp rx=0 tx=0).stop 2024-07-26T10:59:13.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 -- 172.21.15.144:0/3215718522 shutdown_connections 2024-07-26T10:59:13.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 --2- 172.21.15.144:0/3215718522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f562c10fb00 0x7f562c111f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:13.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 --2- 172.21.15.144:0/3215718522 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c075b30 0x7f562c10f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:13.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 --2- 172.21.15.144:0/3215718522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c0751e0 0x7f562c0755f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:13.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 -- 172.21.15.144:0/3215718522 >> 172.21.15.144:0/3215718522 conn(0x7f562c0ff010 msgr2=0x7f562c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:13.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 -- 172.21.15.144:0/3215718522 shutdown_connections 2024-07-26T10:59:13.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.884+0000 7f5634000640 1 -- 172.21.15.144:0/3215718522 wait complete. 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.885+0000 7f5634000640 1 Processor -- start 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.885+0000 7f5634000640 1 -- start start 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5634000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 0x7f562c19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5634000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f562c075b30 0x7f562c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5634000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c10fb00 0x7f562c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5634000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f562c115700 con 0x7f562c10fb00 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5634000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f562c115580 con 0x7f562c0751e0 2024-07-26T10:59:13.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5634000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f562c115880 con 0x7f562c075b30 2024-07-26T10:59:13.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5631600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 0x7f562c19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:13.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5631600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 0x7f562c19c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55826/0 (socket says 172.21.15.144:55826) 2024-07-26T10:59:13.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5630c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f562c075b30 0x7f562c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:13.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5632000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c10fb00 0x7f562c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:13.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.886+0000 7f5631600640 1 -- 172.21.15.144:0/4041402642 learned_addr learned my addr 172.21.15.144:0/4041402642 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5632000640 1 -- 172.21.15.144:0/4041402642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f562c075b30 msgr2=0x7f562c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5632000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f562c075b30 0x7f562c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5632000640 1 -- 172.21.15.144:0/4041402642 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 msgr2=0x7f562c19c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5632000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 0x7f562c19c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5632000640 1 -- 172.21.15.144:0/4041402642 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562c1a2000 con 0x7f562c10fb00 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5631600640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 0x7f562c19c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.887+0000 7f5632000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c10fb00 0x7f562c1a18d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f562400e970 tx=0x7f562400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:13.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.888+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562400ccc0 con 0x7f562c10fb00 2024-07-26T10:59:13.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.888+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5624005020 con 0x7f562c10fb00 2024-07-26T10:59:13.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.888+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562c1a22f0 con 0x7f562c10fb00 2024-07-26T10:59:13.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.888+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624004a00 con 0x7f562c10fb00 2024-07-26T10:59:13.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.888+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f562c1a8a50 con 0x7f562c10fb00 2024-07-26T10:59:13.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.889+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562c0755f0 con 0x7f562c10fb00 2024-07-26T10:59:13.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.891+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f5624004ba0 con 0x7f562c10fb00 2024-07-26T10:59:13.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.893+0000 7f562a000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f56100b3cb0 0x7f56100b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:13.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.893+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f56240e4270 con 0x7f562c10fb00 2024-07-26T10:59:13.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.893+0000 7f5631600640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f56100b3cb0 0x7f56100b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:13.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.894+0000 7f5631600640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f56100b3cb0 0x7f56100b61c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f561c000f80 tx=0x7f561c005d80 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:13.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:13.895+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f56240ab300 con 0x7f562c10fb00 2024-07-26T10:59:14.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.095+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f562c19d1c0 con 0x7f562c10fb00 2024-07-26T10:59:14.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.099+0000 7f562a000640 1 -- 172.21.15.144:0/4041402642 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7f56240aac00 con 0x7f562c10fb00 2024-07-26T10:59:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:14.102 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:59:14.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f56100b3cb0 msgr2=0x7f56100b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:14.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f56100b3cb0 0x7f56100b61c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f561c000f80 tx=0x7f561c005d80 comp rx=0 tx=0).stop 2024-07-26T10:59:14.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c10fb00 msgr2=0x7f562c1a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:14.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c10fb00 0x7f562c1a18d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f562400e970 tx=0x7f562400ee30 comp rx=0 tx=0).stop 2024-07-26T10:59:14.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 shutdown_connections 2024-07-26T10:59:14.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f56100b3cb0 0x7f56100b61c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:14.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f562c10fb00 0x7f562c1a18d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:14.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f562c075b30 0x7f562c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:14.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 --2- 172.21.15.144:0/4041402642 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f562c0751e0 0x7f562c19c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:14.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 >> 172.21.15.144:0/4041402642 conn(0x7f562c0ff010 msgr2=0x7f562c107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:14.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 shutdown_connections 2024-07-26T10:59:14.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:14.102+0000 7f5634000640 1 -- 172.21.15.144:0/4041402642 wait complete. 2024-07-26T10:59:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:14 smithi183 ceph-mon[37273]: pgmap v363: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:14 smithi144 ceph-mon[32129]: pgmap v363: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:14 smithi160 ceph-mon[36715]: pgmap v363: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:14.912 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T10:59:14.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T10:59:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:15 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/4041402642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:15 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/4041402642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:15.393 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:15 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/4041402642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:16 smithi144 ceph-mon[32129]: pgmap v364: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:16 smithi183 ceph-mon[37273]: pgmap v364: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:16 smithi160 ceph-mon[36715]: pgmap v364: 129 pgs: 129 active+clean; 637 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T10:59:17.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 -- 172.21.15.144:0/1292926544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2081057b0 msgr2=0x7fe208105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:17.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 --2- 172.21.15.144:0/1292926544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2081057b0 0x7fe208105bc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009ca0 tx=0x7fe1f802a8c0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 -- 172.21.15.144:0/1292926544 shutdown_connections 2024-07-26T10:59:17.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 --2- 172.21.15.144:0/1292926544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe20810ba40 0x7fe208112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 --2- 172.21.15.144:0/1292926544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 0x7fe20810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 --2- 172.21.15.144:0/1292926544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2081057b0 0x7fe208105bc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 -- 172.21.15.144:0/1292926544 >> 172.21.15.144:0/1292926544 conn(0x7fe208101240 msgr2=0x7fe208103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:17.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 -- 172.21.15.144:0/1292926544 shutdown_connections 2024-07-26T10:59:17.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.647+0000 7fe20ca00640 1 -- 172.21.15.144:0/1292926544 wait complete. 2024-07-26T10:59:17.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.648+0000 7fe20ca00640 1 Processor -- start 2024-07-26T10:59:17.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 -- start start 2024-07-26T10:59:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe2081057b0 0x7fe20819c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 0x7fe20819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 0x7fe2081a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe208115740 con 0x7fe20810ba40 2024-07-26T10:59:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081155c0 con 0x7fe2081057b0 2024-07-26T10:59:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe20ca00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081158c0 con 0x7fe20810b0f0 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe206800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 0x7fe2081a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe205e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe2081057b0 0x7fe20819c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe206800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 0x7fe2081a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53468/0 (socket says 172.21.15.144:53468) 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe206800640 1 -- 172.21.15.144:0/3642284924 learned_addr learned my addr 172.21.15.144:0/3642284924 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.649+0000 7fe1fd400640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 0x7fe20819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe206800640 1 -- 172.21.15.144:0/3642284924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 msgr2=0x7fe20819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe206800640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 0x7fe20819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe206800640 1 -- 172.21.15.144:0/3642284924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe2081057b0 msgr2=0x7fe20819c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe206800640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe2081057b0 0x7fe20819c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe206800640 1 -- 172.21.15.144:0/3642284924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2081a2080 con 0x7fe20810ba40 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe1fd400640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 0x7fe20819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.650+0000 7fe206800640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 0x7fe2081a1950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e970 tx=0x7fe20000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.651+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20000ccc0 con 0x7fe20810ba40 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.651+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20000ce60 con 0x7fe20810ba40 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.651+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2081a2370 con 0x7fe20810ba40 2024-07-26T10:59:17.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.651+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200010430 con 0x7fe20810ba40 2024-07-26T10:59:17.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.651+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2081a8ad0 con 0x7fe20810ba40 2024-07-26T10:59:17.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.652+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe208105bc0 con 0x7fe20810ba40 2024-07-26T10:59:17.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.655+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fe200004fb0 con 0x7fe20810ba40 2024-07-26T10:59:17.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.658+0000 7fe1ff400640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1ec0b3cb0 0x7fe1ec0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:17.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.658+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fe2000e4890 con 0x7fe20810ba40 2024-07-26T10:59:17.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.658+0000 7fe205e00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1ec0b3cb0 0x7fe1ec0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:17.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.659+0000 7fe205e00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1ec0b3cb0 0x7fe1ec0b61c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006010 tx=0x7fe1f8009820 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:17.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.660+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe200055600 con 0x7fe20810ba40 2024-07-26T10:59:17.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.857+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe208063230 con 0x7fe1ec0b3cb0 2024-07-26T10:59:17.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.879+0000 7fe1ff400640 1 -- 172.21.15.144:0/3642284924 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340385 (secure 0 0 0) 0x7fe208063230 con 0x7fe1ec0b3cb0 2024-07-26T10:59:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:17.888 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1ec0b3cb0 msgr2=0x7fe1ec0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1ec0b3cb0 0x7fe1ec0b61c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006010 tx=0x7fe1f8009820 comp rx=0 tx=0).stop 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 msgr2=0x7fe2081a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 0x7fe2081a1950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e970 tx=0x7fe20000ed70 comp rx=0 tx=0).stop 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 shutdown_connections 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe1ec0b3cb0 0x7fe1ec0b61c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20810ba40 0x7fe2081a1950 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe20810b0f0 0x7fe20819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 --2- 172.21.15.144:0/3642284924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe2081057b0 0x7fe20819c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:17.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 >> 172.21.15.144:0/3642284924 conn(0x7fe208101240 msgr2=0x7fe208103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:17.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 shutdown_connections 2024-07-26T10:59:17.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:17.888+0000 7fe20ca00640 1 -- 172.21.15.144:0/3642284924 wait complete. 2024-07-26T10:59:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:18 smithi144 ceph-mon[32129]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-07-26T10:59:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:18 smithi183 ceph-mon[37273]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-07-26T10:59:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:18 smithi160 ceph-mon[36715]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-07-26T10:59:18.734 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-07-26T10:59:17.013725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658015,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":306,"num_read_kb":251,"num_write":294,"num_write_kb":2269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380088,"kb_used_data":14288,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561736,"statfs":{"total":1151940427776,"available":1151551217664,"internally_reserved":0,"allocated":14630912,"data_stored":8372003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34322,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":75,"num_write":25,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006777"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210147+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:58:45.210147+0000","last_peered":"2024-07-26T10:58:45.210147+0000","last_clean":"2024-07-26T10:58:45.210147+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:58:45.210147+0000","last_undegraded":"2024-07-26T10:58:45.210147+0000","last_fullsized":"2024-07-26T10:58:45.210147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211280+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T10:58:45.211280+0000","last_peered":"2024-07-26T10:58:45.211280+0000","last_clean":"2024-07-26T10:58:45.211280+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T10:58:45.211280+0000","last_undegraded":"2024-07-26T10:58:45.211280+0000","last_fullsized":"2024-07-26T10:58:45.211280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205192+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:58:45.205192+0000","last_peered":"2024-07-26T10:58:45.205192+0000","last_clean":"2024-07-26T10:58:45.205192+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:58:45.205192+0000","last_undegraded":"2024-07-26T10:58:45.205192+0000","last_fullsized":"2024-07-26T10:58:45.205192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211579+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:58:45.211579+0000","last_peered":"2024-07-26T10:58:45.211579+0000","last_clean":"2024-07-26T10:58:45.211579+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:58:45.211579+0000","last_undegraded":"2024-07-26T10:58:45.211579+0000","last_fullsized":"2024-07-26T10:58:45.211579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659488+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T10:58:45.659488+0000","last_peered":"2024-07-26T10:58:45.659488+0000","last_clean":"2024-07-26T10:58:45.659488+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T10:58:45.659488+0000","last_undegraded":"2024-07-26T10:58:45.659488+0000","last_fullsized":"2024-07-26T10:58:45.659488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211908+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T10:58:45.211908+0000","last_peered":"2024-07-26T10:58:45.211908+0000","last_clean":"2024-07-26T10:58:45.211908+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T10:58:45.211908+0000","last_undegraded":"2024-07-26T10:58:45.211908+0000","last_fullsized":"2024-07-26T10:58:45.211908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211725+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:58:45.211725+0000","last_peered":"2024-07-26T10:58:45.211725+0000","last_clean":"2024-07-26T10:58:45.211725+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:58:45.211725+0000","last_undegraded":"2024-07-26T10:58:45.211725+0000","last_fullsized":"2024-07-26T10:58:45.211725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394783+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T10:58:45.394783+0000","last_peered":"2024-07-26T10:58:45.394783+0000","last_clean":"2024-07-26T10:58:45.394783+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T10:58:45.394783+0000","last_undegraded":"2024-07-26T10:58:45.394783+0000","last_fullsized":"2024-07-26T10:58:45.394783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212272+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212272+0000","last_peered":"2024-07-26T10:58:45.212272+0000","last_clean":"2024-07-26T10:58:45.212272+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212272+0000","last_undegraded":"2024-07-26T10:58:45.212272+0000","last_fullsized":"2024-07-26T10:58:45.212272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395246+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395246+0000","last_peered":"2024-07-26T10:58:45.395246+0000","last_clean":"2024-07-26T10:58:45.395246+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395246+0000","last_undegraded":"2024-07-26T10:58:45.395246+0000","last_fullsized":"2024-07-26T10:58:45.395246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395593+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T10:58:45.395593+0000","last_peered":"2024-07-26T10:58:45.395593+0000","last_clean":"2024-07-26T10:58:45.395593+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T10:58:45.395593+0000","last_undegraded":"2024-07-26T10:58:45.395593+0000","last_fullsized":"2024-07-26T10:58:45.395593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212052+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212052+0000","last_peered":"2024-07-26T10:58:45.212052+0000","last_clean":"2024-07-26T10:58:45.212052+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212052+0000","last_undegraded":"2024-07-26T10:58:45.212052+0000","last_fullsized":"2024-07-26T10:58:45.212052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210799+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T10:58:45.210799+0000","last_peered":"2024-07-26T10:58:45.210799+0000","last_clean":"2024-07-26T10:58:45.210799+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T10:58:45.210799+0000","last_undegraded":"2024-07-26T10:58:45.210799+0000","last_fullsized":"2024-07-26T10:58:45.210799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211096+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:58:45.211096+0000","last_peered":"2024-07-26T10:58:45.211096+0000","last_clean":"2024-07-26T10:58:45.211096+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:58:45.211096+0000","last_undegraded":"2024-07-26T10:58:45.211096+0000","last_fullsized":"2024-07-26T10:58:45.211096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212084+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:58:45.212084+0000","last_peered":"2024-07-26T10:58:45.212084+0000","last_clean":"2024-07-26T10:58:45.212084+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:58:45.212084+0000","last_undegraded":"2024-07-26T10:58:45.212084+0000","last_fullsized":"2024-07-26T10:58:45.212084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210731+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T10:58:45.210731+0000","last_peered":"2024-07-26T10:58:45.210731+0000","last_clean":"2024-07-26T10:58:45.210731+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T10:58:45.210731+0000","last_undegraded":"2024-07-26T10:58:45.210731+0000","last_fullsized":"2024-07-26T10:58:45.210731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205302+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T10:58:45.205302+0000","last_peered":"2024-07-26T10:58:45.205302+0000","last_clean":"2024-07-26T10:58:45.205302+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T10:58:45.205302+0000","last_undegraded":"2024-07-26T10:58:45.205302+0000","last_fullsized":"2024-07-26T10:58:45.205302+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.398918+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:58:45.398918+0000","last_peered":"2024-07-26T10:58:45.398918+0000","last_clean":"2024-07-26T10:58:45.398918+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:58:45.398918+0000","last_undegraded":"2024-07-26T10:58:45.398918+0000","last_fullsized":"2024-07-26T10:58:45.398918+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394934+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T10:58:45.394934+0000","last_peered":"2024-07-26T10:58:45.394934+0000","last_clean":"2024-07-26T10:58:45.394934+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T10:58:45.394934+0000","last_undegraded":"2024-07-26T10:58:45.394934+0000","last_fullsized":"2024-07-26T10:58:45.394934+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210014+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:58:45.210014+0000","last_peered":"2024-07-26T10:58:45.210014+0000","last_clean":"2024-07-26T10:58:45.210014+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:58:45.210014+0000","last_undegraded":"2024-07-26T10:58:45.210014+0000","last_fullsized":"2024-07-26T10:58:45.210014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211546+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T10:58:45.211546+0000","last_peered":"2024-07-26T10:58:45.211546+0000","last_clean":"2024-07-26T10:58:45.211546+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T10:58:45.211546+0000","last_undegraded":"2024-07-26T10:58:45.211546+0000","last_fullsized":"2024-07-26T10:58:45.211546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211224+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T10:58:45.211224+0000","last_peered":"2024-07-26T10:58:45.211224+0000","last_clean":"2024-07-26T10:58:45.211224+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T10:58:45.211224+0000","last_undegraded":"2024-07-26T10:58:45.211224+0000","last_fullsized":"2024-07-26T10:58:45.211224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394716+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:58:45.394716+0000","last_peered":"2024-07-26T10:58:45.394716+0000","last_clean":"2024-07-26T10:58:45.394716+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:58:45.394716+0000","last_undegraded":"2024-07-26T10:58:45.394716+0000","last_fullsized":"2024-07-26T10:58:45.394716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212508+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:58:45.212508+0000","last_peered":"2024-07-26T10:58:45.212508+0000","last_clean":"2024-07-26T10:58:45.212508+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:58:45.212508+0000","last_undegraded":"2024-07-26T10:58:45.212508+0000","last_fullsized":"2024-07-26T10:58:45.212508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659287+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T10:58:45.659287+0000","last_peered":"2024-07-26T10:58:45.659287+0000","last_clean":"2024-07-26T10:58:45.659287+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T10:58:45.659287+0000","last_undegraded":"2024-07-26T10:58:45.659287+0000","last_fullsized":"2024-07-26T10:58:45.659287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212680+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:58:45.212680+0000","last_peered":"2024-07-26T10:58:45.212680+0000","last_clean":"2024-07-26T10:58:45.212680+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:58:45.212680+0000","last_undegraded":"2024-07-26T10:58:45.212680+0000","last_fullsized":"2024-07-26T10:58:45.212680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205392+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:58:45.205392+0000","last_peered":"2024-07-26T10:58:45.205392+0000","last_clean":"2024-07-26T10:58:45.205392+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:58:45.205392+0000","last_undegraded":"2024-07-26T10:58:45.205392+0000","last_fullsized":"2024-07-26T10:58:45.205392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211913+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T10:58:45.211913+0000","last_peered":"2024-07-26T10:58:45.211913+0000","last_clean":"2024-07-26T10:58:45.211913+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T10:58:45.211913+0000","last_undegraded":"2024-07-26T10:58:45.211913+0000","last_fullsized":"2024-07-26T10:58:45.211913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:09.157867+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T10:59:09.157867+0000","last_peered":"2024-07-26T10:59:09.157867+0000","last_clean":"2024-07-26T10:59:09.157867+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T10:59:09.157867+0000","last_undegraded":"2024-07-26T10:59:09.157867+0000","last_fullsized":"2024-07-26T10:59:09.157867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208653+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:58:45.208653+0000","last_peered":"2024-07-26T10:58:45.208653+0000","last_clean":"2024-07-26T10:58:45.208653+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:58:45.208653+0000","last_undegraded":"2024-07-26T10:58:45.208653+0000","last_fullsized":"2024-07-26T10:58:45.208653+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397128+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T10:58:45.397128+0000","last_peered":"2024-07-26T10:58:45.397128+0000","last_clean":"2024-07-26T10:58:45.397128+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T10:58:45.397128+0000","last_undegraded":"2024-07-26T10:58:45.397128+0000","last_fullsized":"2024-07-26T10:58:45.397128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211738+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:58:45.211738+0000","last_peered":"2024-07-26T10:58:45.211738+0000","last_clean":"2024-07-26T10:58:45.211738+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:58:45.211738+0000","last_undegraded":"2024-07-26T10:58:45.211738+0000","last_fullsized":"2024-07-26T10:58:45.211738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395666+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:58:45.395666+0000","last_peered":"2024-07-26T10:58:45.395666+0000","last_clean":"2024-07-26T10:58:45.395666+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:58:45.395666+0000","last_undegraded":"2024-07-26T10:58:45.395666+0000","last_fullsized":"2024-07-26T10:58:45.395666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395766+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T10:58:45.395766+0000","last_peered":"2024-07-26T10:58:45.395766+0000","last_clean":"2024-07-26T10:58:45.395766+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T10:58:45.395766+0000","last_undegraded":"2024-07-26T10:58:45.395766+0000","last_fullsized":"2024-07-26T10:58:45.395766+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208881+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:58:45.208881+0000","last_peered":"2024-07-26T10:58:45.208881+0000","last_clean":"2024-07-26T10:58:45.208881+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:58:45.208881+0000","last_undegraded":"2024-07-26T10:58:45.208881+0000","last_fullsized":"2024-07-26T10:58:45.208881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210980+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:58:45.210980+0000","last_peered":"2024-07-26T10:58:45.210980+0000","last_clean":"2024-07-26T10:58:45.210980+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:58:45.210980+0000","last_undegraded":"2024-07-26T10:58:45.210980+0000","last_fullsized":"2024-07-26T10:58:45.210980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395794+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T10:58:45.395794+0000","last_peered":"2024-07-26T10:58:45.395794+0000","last_clean":"2024-07-26T10:58:45.395794+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T10:58:45.395794+0000","last_undegraded":"2024-07-26T10:58:45.395794+0000","last_fullsized":"2024-07-26T10:58:45.395794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210920+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:58:45.210920+0000","last_peered":"2024-07-26T10:58:45.210920+0000","last_clean":"2024-07-26T10:58:45.210920+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:58:45.210920+0000","last_undegraded":"2024-07-26T10:58:45.210920+0000","last_fullsized":"2024-07-26T10:58:45.210920+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396573+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:58:45.396573+0000","last_peered":"2024-07-26T10:58:45.396573+0000","last_clean":"2024-07-26T10:58:45.396573+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:58:45.396573+0000","last_undegraded":"2024-07-26T10:58:45.396573+0000","last_fullsized":"2024-07-26T10:58:45.396573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395303+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T10:58:45.395303+0000","last_peered":"2024-07-26T10:58:45.395303+0000","last_clean":"2024-07-26T10:58:45.395303+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T10:58:45.395303+0000","last_undegraded":"2024-07-26T10:58:45.395303+0000","last_fullsized":"2024-07-26T10:58:45.395303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:11.255285+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T10:59:11.255285+0000","last_peered":"2024-07-26T10:59:11.255285+0000","last_clean":"2024-07-26T10:59:11.255285+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T10:59:11.255285+0000","last_undegraded":"2024-07-26T10:59:11.255285+0000","last_fullsized":"2024-07-26T10:59:11.255285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396965+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:58:45.396965+0000","last_peered":"2024-07-26T10:58:45.396965+0000","last_clean":"2024-07-26T10:58:45.396965+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:58:45.396965+0000","last_undegraded":"2024-07-26T10:58:45.396965+0000","last_fullsized":"2024-07-26T10:58:45.396965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395960+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T10:58:45.395960+0000","last_peered":"2024-07-26T10:58:45.395960+0000","last_clean":"2024-07-26T10:58:45.395960+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T10:58:45.395960+0000","last_undegraded":"2024-07-26T10:58:45.395960+0000","last_fullsized":"2024-07-26T10:58:45.395960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210523+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:58:45.210523+0000","last_peered":"2024-07-26T10:58:45.210523+0000","last_clean":"2024-07-26T10:58:45.210523+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:58:45.210523+0000","last_undegraded":"2024-07-26T10:58:45.210523+0000","last_fullsized":"2024-07-26T10:58:45.210523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.399431+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:45.399431+0000","last_peered":"2024-07-26T10:58:45.399431+0000","last_clean":"2024-07-26T10:58:45.399431+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:45.399431+0000","last_undegraded":"2024-07-26T10:58:45.399431+0000","last_fullsized":"2024-07-26T10:58:45.399431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210623+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T10:58:45.210623+0000","last_peered":"2024-07-26T10:58:45.210623+0000","last_clean":"2024-07-26T10:58:45.210623+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T10:58:45.210623+0000","last_undegraded":"2024-07-26T10:58:45.210623+0000","last_fullsized":"2024-07-26T10:58:45.210623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212541+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:58:45.212541+0000","last_peered":"2024-07-26T10:58:45.212541+0000","last_clean":"2024-07-26T10:58:45.212541+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:58:45.212541+0000","last_undegraded":"2024-07-26T10:58:45.212541+0000","last_fullsized":"2024-07-26T10:58:45.212541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":121,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210602+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210602+0000","last_peered":"2024-07-26T10:58:45.210602+0000","last_clean":"2024-07-26T10:58:45.210602+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210602+0000","last_undegraded":"2024-07-26T10:58:45.210602+0000","last_fullsized":"2024-07-26T10:58:45.210602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.398754+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:58:45.398754+0000","last_peered":"2024-07-26T10:58:45.398754+0000","last_clean":"2024-07-26T10:58:45.398754+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:58:45.398754+0000","last_undegraded":"2024-07-26T10:58:45.398754+0000","last_fullsized":"2024-07-26T10:58:45.398754+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211486+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T10:58:45.211486+0000","last_peered":"2024-07-26T10:58:45.211486+0000","last_clean":"2024-07-26T10:58:45.211486+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T10:58:45.211486+0000","last_undegraded":"2024-07-26T10:58:45.211486+0000","last_fullsized":"2024-07-26T10:58:45.211486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205785+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:58:45.205785+0000","last_peered":"2024-07-26T10:58:45.205785+0000","last_clean":"2024-07-26T10:58:45.205785+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:58:45.205785+0000","last_undegraded":"2024-07-26T10:58:45.205785+0000","last_fullsized":"2024-07-26T10:58:45.205785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394314+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:58:45.394314+0000","last_peered":"2024-07-26T10:58:45.394314+0000","last_clean":"2024-07-26T10:58:45.394314+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:58:45.394314+0000","last_undegraded":"2024-07-26T10:58:45.394314+0000","last_fullsized":"2024-07-26T10:58:45.394314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211236+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T10:58:45.211236+0000","last_peered":"2024-07-26T10:58:45.211236+0000","last_clean":"2024-07-26T10:58:45.211236+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T10:58:45.211236+0000","last_undegraded":"2024-07-26T10:58:45.211236+0000","last_fullsized":"2024-07-26T10:58:45.211236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396372+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:58:45.396372+0000","last_peered":"2024-07-26T10:58:45.396372+0000","last_clean":"2024-07-26T10:58:45.396372+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:58:45.396372+0000","last_undegraded":"2024-07-26T10:58:45.396372+0000","last_fullsized":"2024-07-26T10:58:45.396372+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658809+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:58:45.658809+0000","last_peered":"2024-07-26T10:58:45.658809+0000","last_clean":"2024-07-26T10:58:45.658809+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:58:45.658809+0000","last_undegraded":"2024-07-26T10:58:45.658809+0000","last_fullsized":"2024-07-26T10:58:45.658809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211382+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T10:58:45.211382+0000","last_peered":"2024-07-26T10:58:45.211382+0000","last_clean":"2024-07-26T10:58:45.211382+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T10:58:45.211382+0000","last_undegraded":"2024-07-26T10:58:45.211382+0000","last_fullsized":"2024-07-26T10:58:45.211382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397505+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:58:45.397505+0000","last_peered":"2024-07-26T10:58:45.397505+0000","last_clean":"2024-07-26T10:58:45.397505+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:58:45.397505+0000","last_undegraded":"2024-07-26T10:58:45.397505+0000","last_fullsized":"2024-07-26T10:58:45.397505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395507+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:58:45.395507+0000","last_peered":"2024-07-26T10:58:45.395507+0000","last_clean":"2024-07-26T10:58:45.395507+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:58:45.395507+0000","last_undegraded":"2024-07-26T10:58:45.395507+0000","last_fullsized":"2024-07-26T10:58:45.395507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396126+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T10:58:45.396126+0000","last_peered":"2024-07-26T10:58:45.396126+0000","last_clean":"2024-07-26T10:58:45.396126+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T10:58:45.396126+0000","last_undegraded":"2024-07-26T10:58:45.396126+0000","last_fullsized":"2024-07-26T10:58:45.396126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211685+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:58:45.211685+0000","last_peered":"2024-07-26T10:58:45.211685+0000","last_clean":"2024-07-26T10:58:45.211685+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:58:45.211685+0000","last_undegraded":"2024-07-26T10:58:45.211685+0000","last_fullsized":"2024-07-26T10:58:45.211685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395832+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T10:58:45.395832+0000","last_peered":"2024-07-26T10:58:45.395832+0000","last_clean":"2024-07-26T10:58:45.395832+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T10:58:45.395832+0000","last_undegraded":"2024-07-26T10:58:45.395832+0000","last_fullsized":"2024-07-26T10:58:45.395832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210974+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:58:45.210974+0000","last_peered":"2024-07-26T10:58:45.210974+0000","last_clean":"2024-07-26T10:58:45.210974+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:58:45.210974+0000","last_undegraded":"2024-07-26T10:58:45.210974+0000","last_fullsized":"2024-07-26T10:58:45.210974+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397198+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397198+0000","last_peered":"2024-07-26T10:58:45.397198+0000","last_clean":"2024-07-26T10:58:45.397198+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397198+0000","last_undegraded":"2024-07-26T10:58:45.397198+0000","last_fullsized":"2024-07-26T10:58:45.397198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210333+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T10:58:45.210333+0000","last_peered":"2024-07-26T10:58:45.210333+0000","last_clean":"2024-07-26T10:58:45.210333+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T10:58:45.210333+0000","last_undegraded":"2024-07-26T10:58:45.210333+0000","last_fullsized":"2024-07-26T10:58:45.210333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:15.263422+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T10:59:15.263422+0000","last_peered":"2024-07-26T10:59:15.263422+0000","last_clean":"2024-07-26T10:59:15.263422+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T10:59:15.263422+0000","last_undegraded":"2024-07-26T10:59:15.263422+0000","last_fullsized":"2024-07-26T10:59:15.263422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208051+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T10:58:45.208051+0000","last_peered":"2024-07-26T10:58:45.208051+0000","last_clean":"2024-07-26T10:58:45.208051+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T10:58:45.208051+0000","last_undegraded":"2024-07-26T10:58:45.208051+0000","last_fullsized":"2024-07-26T10:58:45.208051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210991+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T10:58:45.210991+0000","last_peered":"2024-07-26T10:58:45.210991+0000","last_clean":"2024-07-26T10:58:45.210991+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T10:58:45.210991+0000","last_undegraded":"2024-07-26T10:58:45.210991+0000","last_fullsized":"2024-07-26T10:58:45.210991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395620+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:58:45.395620+0000","last_peered":"2024-07-26T10:58:45.395620+0000","last_clean":"2024-07-26T10:58:45.395620+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:58:45.395620+0000","last_undegraded":"2024-07-26T10:58:45.395620+0000","last_fullsized":"2024-07-26T10:58:45.395620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395383+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:58:45.395383+0000","last_peered":"2024-07-26T10:58:45.395383+0000","last_clean":"2024-07-26T10:58:45.395383+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:58:45.395383+0000","last_undegraded":"2024-07-26T10:58:45.395383+0000","last_fullsized":"2024-07-26T10:58:45.395383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396333+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T10:58:45.396333+0000","last_peered":"2024-07-26T10:58:45.396333+0000","last_clean":"2024-07-26T10:58:45.396333+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T10:58:45.396333+0000","last_undegraded":"2024-07-26T10:58:45.396333+0000","last_fullsized":"2024-07-26T10:58:45.396333+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205552+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:58:45.205552+0000","last_peered":"2024-07-26T10:58:45.205552+0000","last_clean":"2024-07-26T10:58:45.205552+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:58:45.205552+0000","last_undegraded":"2024-07-26T10:58:45.205552+0000","last_fullsized":"2024-07-26T10:58:45.205552+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395470+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395470+0000","last_peered":"2024-07-26T10:58:45.395470+0000","last_clean":"2024-07-26T10:58:45.395470+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395470+0000","last_undegraded":"2024-07-26T10:58:45.395470+0000","last_fullsized":"2024-07-26T10:58:45.395470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211550+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T10:58:45.211550+0000","last_peered":"2024-07-26T10:58:45.211550+0000","last_clean":"2024-07-26T10:58:45.211550+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T10:58:45.211550+0000","last_undegraded":"2024-07-26T10:58:45.211550+0000","last_fullsized":"2024-07-26T10:58:45.211550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:14.247031+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T10:59:14.247031+0000","last_peered":"2024-07-26T10:59:14.247031+0000","last_clean":"2024-07-26T10:59:14.247031+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T10:59:14.247031+0000","last_undegraded":"2024-07-26T10:59:14.247031+0000","last_fullsized":"2024-07-26T10:59:14.247031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63412,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":75,"num_read_kb":99,"num_write":34,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208021+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:58:45.208021+0000","last_peered":"2024-07-26T10:58:45.208021+0000","last_clean":"2024-07-26T10:58:45.208021+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:58:45.208021+0000","last_undegraded":"2024-07-26T10:58:45.208021+0000","last_fullsized":"2024-07-26T10:58:45.208021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394854+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T10:58:45.394854+0000","last_peered":"2024-07-26T10:58:45.394854+0000","last_clean":"2024-07-26T10:58:45.394854+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T10:58:45.394854+0000","last_undegraded":"2024-07-26T10:58:45.394854+0000","last_fullsized":"2024-07-26T10:58:45.394854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396288+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:58:45.396288+0000","last_peered":"2024-07-26T10:58:45.396288+0000","last_clean":"2024-07-26T10:58:45.396288+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:58:45.396288+0000","last_undegraded":"2024-07-26T10:58:45.396288+0000","last_fullsized":"2024-07-26T10:58:45.396288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212370+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:58:45.212370+0000","last_peered":"2024-07-26T10:58:45.212370+0000","last_clean":"2024-07-26T10:58:45.212370+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:58:45.212370+0000","last_undegraded":"2024-07-26T10:58:45.212370+0000","last_fullsized":"2024-07-26T10:58:45.212370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659486+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T10:58:45.659486+0000","last_peered":"2024-07-26T10:58:45.659486+0000","last_clean":"2024-07-26T10:58:45.659486+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T10:58:45.659486+0000","last_undegraded":"2024-07-26T10:58:45.659486+0000","last_fullsized":"2024-07-26T10:58:45.659486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211388+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:58:45.211388+0000","last_peered":"2024-07-26T10:58:45.211388+0000","last_clean":"2024-07-26T10:58:45.211388+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:58:45.211388+0000","last_undegraded":"2024-07-26T10:58:45.211388+0000","last_fullsized":"2024-07-26T10:58:45.211388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207003+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:58:45.207003+0000","last_peered":"2024-07-26T10:58:45.207003+0000","last_clean":"2024-07-26T10:58:45.207003+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:58:45.207003+0000","last_undegraded":"2024-07-26T10:58:45.207003+0000","last_fullsized":"2024-07-26T10:58:45.207003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394873+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T10:58:45.394873+0000","last_peered":"2024-07-26T10:58:45.394873+0000","last_clean":"2024-07-26T10:58:45.394873+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T10:58:45.394873+0000","last_undegraded":"2024-07-26T10:58:45.394873+0000","last_fullsized":"2024-07-26T10:58:45.394873+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:13.245665+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T10:59:13.245665+0000","last_peered":"2024-07-26T10:59:13.245665+0000","last_clean":"2024-07-26T10:59:13.245665+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T10:59:13.245665+0000","last_undegraded":"2024-07-26T10:59:13.245665+0000","last_fullsized":"2024-07-26T10:59:13.245665+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208888+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:58:45.208888+0000","last_peered":"2024-07-26T10:58:45.208888+0000","last_clean":"2024-07-26T10:58:45.208888+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:58:45.208888+0000","last_undegraded":"2024-07-26T10:58:45.208888+0000","last_fullsized":"2024-07-26T10:58:45.208888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658974+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:58:45.658974+0000","last_peered":"2024-07-26T10:58:45.658974+0000","last_clean":"2024-07-26T10:58:45.658974+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:58:45.658974+0000","last_undegraded":"2024-07-26T10:58:45.658974+0000","last_fullsized":"2024-07-26T10:58:45.658974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207341+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T10:58:45.207341+0000","last_peered":"2024-07-26T10:58:45.207341+0000","last_clean":"2024-07-26T10:58:45.207341+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T10:58:45.207341+0000","last_undegraded":"2024-07-26T10:58:45.207341+0000","last_fullsized":"2024-07-26T10:58:45.207341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395654+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:58:45.395654+0000","last_peered":"2024-07-26T10:58:45.395654+0000","last_clean":"2024-07-26T10:58:45.395654+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:58:45.395654+0000","last_undegraded":"2024-07-26T10:58:45.395654+0000","last_fullsized":"2024-07-26T10:58:45.395654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395538+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:58:45.395538+0000","last_peered":"2024-07-26T10:58:45.395538+0000","last_clean":"2024-07-26T10:58:45.395538+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:58:45.395538+0000","last_undegraded":"2024-07-26T10:58:45.395538+0000","last_fullsized":"2024-07-26T10:58:45.395538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210311+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T10:58:45.210311+0000","last_peered":"2024-07-26T10:58:45.210311+0000","last_clean":"2024-07-26T10:58:45.210311+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T10:58:45.210311+0000","last_undegraded":"2024-07-26T10:58:45.210311+0000","last_fullsized":"2024-07-26T10:58:45.210311+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396052+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:58:45.396052+0000","last_peered":"2024-07-26T10:58:45.396052+0000","last_clean":"2024-07-26T10:58:45.396052+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:58:45.396052+0000","last_undegraded":"2024-07-26T10:58:45.396052+0000","last_fullsized":"2024-07-26T10:58:45.396052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395567+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:58:45.395567+0000","last_peered":"2024-07-26T10:58:45.395567+0000","last_clean":"2024-07-26T10:58:45.395567+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:58:45.395567+0000","last_undegraded":"2024-07-26T10:58:45.395567+0000","last_fullsized":"2024-07-26T10:58:45.395567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207651+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T10:58:45.207651+0000","last_peered":"2024-07-26T10:58:45.207651+0000","last_clean":"2024-07-26T10:58:45.207651+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T10:58:45.207651+0000","last_undegraded":"2024-07-26T10:58:45.207651+0000","last_fullsized":"2024-07-26T10:58:45.207651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205489+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205489+0000","last_peered":"2024-07-26T10:58:45.205489+0000","last_clean":"2024-07-26T10:58:45.205489+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205489+0000","last_undegraded":"2024-07-26T10:58:45.205489+0000","last_fullsized":"2024-07-26T10:58:45.205489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:12.244737+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T10:59:12.244737+0000","last_peered":"2024-07-26T10:59:12.244737+0000","last_clean":"2024-07-26T10:59:12.244737+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T10:59:12.244737+0000","last_undegraded":"2024-07-26T10:59:12.244737+0000","last_fullsized":"2024-07-26T10:59:12.244737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211262+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T10:58:45.211262+0000","last_peered":"2024-07-26T10:58:45.211262+0000","last_clean":"2024-07-26T10:58:45.211262+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T10:58:45.211262+0000","last_undegraded":"2024-07-26T10:58:45.211262+0000","last_fullsized":"2024-07-26T10:58:45.211262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205703+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205703+0000","last_peered":"2024-07-26T10:58:45.205703+0000","last_clean":"2024-07-26T10:58:45.205703+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205703+0000","last_undegraded":"2024-07-26T10:58:45.205703+0000","last_fullsized":"2024-07-26T10:58:45.205703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394757+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T10:58:45.394757+0000","last_peered":"2024-07-26T10:58:45.394757+0000","last_clean":"2024-07-26T10:58:45.394757+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T10:58:45.394757+0000","last_undegraded":"2024-07-26T10:58:45.394757+0000","last_fullsized":"2024-07-26T10:58:45.394757+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205577+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T10:58:45.205577+0000","last_peered":"2024-07-26T10:58:45.205577+0000","last_clean":"2024-07-26T10:58:45.205577+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T10:58:45.205577+0000","last_undegraded":"2024-07-26T10:58:45.205577+0000","last_fullsized":"2024-07-26T10:58:45.205577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205339+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T10:58:45.205339+0000","last_peered":"2024-07-26T10:58:45.205339+0000","last_clean":"2024-07-26T10:58:45.205339+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T10:58:45.205339+0000","last_undegraded":"2024-07-26T10:58:45.205339+0000","last_fullsized":"2024-07-26T10:58:45.205339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659046+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T10:58:45.659046+0000","last_peered":"2024-07-26T10:58:45.659046+0000","last_clean":"2024-07-26T10:58:45.659046+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T10:58:45.659046+0000","last_undegraded":"2024-07-26T10:58:45.659046+0000","last_fullsized":"2024-07-26T10:58:45.659046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397091+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T10:58:45.397091+0000","last_peered":"2024-07-26T10:58:45.397091+0000","last_clean":"2024-07-26T10:58:45.397091+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T10:58:45.397091+0000","last_undegraded":"2024-07-26T10:58:45.397091+0000","last_fullsized":"2024-07-26T10:58:45.397091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395386+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T10:58:45.395386+0000","last_peered":"2024-07-26T10:58:45.395386+0000","last_clean":"2024-07-26T10:58:45.395386+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T10:58:45.395386+0000","last_undegraded":"2024-07-26T10:58:45.395386+0000","last_fullsized":"2024-07-26T10:58:45.395386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205978+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T10:58:45.205978+0000","last_peered":"2024-07-26T10:58:45.205978+0000","last_clean":"2024-07-26T10:58:45.205978+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T10:58:45.205978+0000","last_undegraded":"2024-07-26T10:58:45.205978+0000","last_fullsized":"2024-07-26T10:58:45.205978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211764+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T10:58:45.211764+0000","last_peered":"2024-07-26T10:58:45.211764+0000","last_clean":"2024-07-26T10:58:45.211764+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T10:58:45.211764+0000","last_undegraded":"2024-07-26T10:58:45.211764+0000","last_fullsized":"2024-07-26T10:58:45.211764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395147+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T10:58:45.395147+0000","last_peered":"2024-07-26T10:58:45.395147+0000","last_clean":"2024-07-26T10:58:45.395147+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T10:58:45.395147+0000","last_undegraded":"2024-07-26T10:58:45.395147+0000","last_fullsized":"2024-07-26T10:58:45.395147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209328+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T10:58:45.209328+0000","last_peered":"2024-07-26T10:58:45.209328+0000","last_clean":"2024-07-26T10:58:45.209328+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T10:58:45.209328+0000","last_undegraded":"2024-07-26T10:58:45.209328+0000","last_fullsized":"2024-07-26T10:58:45.209328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394527+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T10:58:45.394527+0000","last_peered":"2024-07-26T10:58:45.394527+0000","last_clean":"2024-07-26T10:58:45.394527+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T10:58:45.394527+0000","last_undegraded":"2024-07-26T10:58:45.394527+0000","last_fullsized":"2024-07-26T10:58:45.394527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396686+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T10:58:45.396686+0000","last_peered":"2024-07-26T10:58:45.396686+0000","last_clean":"2024-07-26T10:58:45.396686+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T10:58:45.396686+0000","last_undegraded":"2024-07-26T10:58:45.396686+0000","last_fullsized":"2024-07-26T10:58:45.396686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395536+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T10:58:45.395536+0000","last_peered":"2024-07-26T10:58:45.395536+0000","last_clean":"2024-07-26T10:58:45.395536+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T10:58:45.395536+0000","last_undegraded":"2024-07-26T10:58:45.395536+0000","last_fullsized":"2024-07-26T10:58:45.395536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205860+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T10:58:45.205860+0000","last_peered":"2024-07-26T10:58:45.205860+0000","last_clean":"2024-07-26T10:58:45.205860+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T10:58:45.205860+0000","last_undegraded":"2024-07-26T10:58:45.205860+0000","last_fullsized":"2024-07-26T10:58:45.205860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209337+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T10:58:45.209337+0000","last_peered":"2024-07-26T10:58:45.209337+0000","last_clean":"2024-07-26T10:58:45.209337+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T10:58:45.209337+0000","last_undegraded":"2024-07-26T10:58:45.209337+0000","last_fullsized":"2024-07-26T10:58:45.209337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396018+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T10:58:45.396018+0000","last_peered":"2024-07-26T10:58:45.396018+0000","last_clean":"2024-07-26T10:58:45.396018+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T10:58:45.396018+0000","last_undegraded":"2024-07-26T10:58:45.396018+0000","last_fullsized":"2024-07-26T10:58:45.396018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394191+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T10:58:45.394191+0000","last_peered":"2024-07-26T10:58:45.394191+0000","last_clean":"2024-07-26T10:58:45.394191+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T10:58:45.394191+0000","last_undegraded":"2024-07-26T10:58:45.394191+0000","last_fullsized":"2024-07-26T10:58:45.394191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658489+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T10:58:45.658489+0000","last_peered":"2024-07-26T10:58:45.658489+0000","last_clean":"2024-07-26T10:58:45.658489+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T10:58:45.658489+0000","last_undegraded":"2024-07-26T10:58:45.658489+0000","last_fullsized":"2024-07-26T10:58:45.658489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658432+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T10:58:45.658432+0000","last_peered":"2024-07-26T10:58:45.658432+0000","last_clean":"2024-07-26T10:58:45.658432+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T10:58:45.658432+0000","last_undegraded":"2024-07-26T10:58:45.658432+0000","last_fullsized":"2024-07-26T10:58:45.658432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396923+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T10:58:45.396923+0000","last_peered":"2024-07-26T10:58:45.396923+0000","last_clean":"2024-07-26T10:58:45.396923+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T10:58:45.396923+0000","last_undegraded":"2024-07-26T10:58:45.396923+0000","last_fullsized":"2024-07-26T10:58:45.396923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.213030+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T10:58:45.213030+0000","last_peered":"2024-07-26T10:58:45.213030+0000","last_clean":"2024-07-26T10:58:45.213030+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T10:58:45.213030+0000","last_undegraded":"2024-07-26T10:58:45.213030+0000","last_fullsized":"2024-07-26T10:58:45.213030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211415+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T10:58:45.211415+0000","last_peered":"2024-07-26T10:58:45.211415+0000","last_clean":"2024-07-26T10:58:45.211415+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T10:58:45.211415+0000","last_undegraded":"2024-07-26T10:58:45.211415+0000","last_fullsized":"2024-07-26T10:58:45.211415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396247+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T10:58:45.396247+0000","last_peered":"2024-07-26T10:58:45.396247+0000","last_clean":"2024-07-26T10:58:45.396247+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T10:58:45.396247+0000","last_undegraded":"2024-07-26T10:58:45.396247+0000","last_fullsized":"2024-07-26T10:58:45.396247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395006+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T10:58:45.395006+0000","last_peered":"2024-07-26T10:58:45.395006+0000","last_clean":"2024-07-26T10:58:45.395006+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T10:58:45.395006+0000","last_undegraded":"2024-07-26T10:58:45.395006+0000","last_fullsized":"2024-07-26T10:58:45.395006+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209162+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T10:58:45.209162+0000","last_peered":"2024-07-26T10:58:45.209162+0000","last_clean":"2024-07-26T10:58:45.209162+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T10:58:45.209162+0000","last_undegraded":"2024-07-26T10:58:45.209162+0000","last_fullsized":"2024-07-26T10:58:45.209162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210440+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T10:58:45.210440+0000","last_peered":"2024-07-26T10:58:45.210440+0000","last_clean":"2024-07-26T10:58:45.210440+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T10:58:45.210440+0000","last_undegraded":"2024-07-26T10:58:45.210440+0000","last_fullsized":"2024-07-26T10:58:45.210440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395504+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T10:58:45.395504+0000","last_peered":"2024-07-26T10:58:45.395504+0000","last_clean":"2024-07-26T10:58:45.395504+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T10:58:45.395504+0000","last_undegraded":"2024-07-26T10:58:45.395504+0000","last_fullsized":"2024-07-26T10:58:45.395504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211081+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T10:58:45.211081+0000","last_peered":"2024-07-26T10:58:45.211081+0000","last_clean":"2024-07-26T10:58:45.211081+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T10:58:45.211081+0000","last_undegraded":"2024-07-26T10:58:45.211081+0000","last_fullsized":"2024-07-26T10:58:45.211081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212942+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T10:58:45.212942+0000","last_peered":"2024-07-26T10:58:45.212942+0000","last_clean":"2024-07-26T10:58:45.212942+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T10:58:45.212942+0000","last_undegraded":"2024-07-26T10:58:45.212942+0000","last_fullsized":"2024-07-26T10:58:45.212942+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658849+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T10:58:45.658849+0000","last_peered":"2024-07-26T10:58:45.658849+0000","last_clean":"2024-07-26T10:58:45.658849+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T10:58:45.658849+0000","last_undegraded":"2024-07-26T10:58:45.658849+0000","last_fullsized":"2024-07-26T10:58:45.658849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659284+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T10:58:45.659284+0000","last_peered":"2024-07-26T10:58:45.659284+0000","last_clean":"2024-07-26T10:58:45.659284+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T10:58:45.659284+0000","last_undegraded":"2024-07-26T10:58:45.659284+0000","last_fullsized":"2024-07-26T10:58:45.659284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395922+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T10:58:45.395922+0000","last_peered":"2024-07-26T10:58:45.395922+0000","last_clean":"2024-07-26T10:58:45.395922+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T10:58:45.395922+0000","last_undegraded":"2024-07-26T10:58:45.395922+0000","last_fullsized":"2024-07-26T10:58:45.395922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67413,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":225,"num_read_kb":182,"num_write":169,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":202239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678054,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874284,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37036,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708116,"statfs":{"total":95995035648,"available":95957110784,"internally_reserved":0,"allocated":1740800,"data_stored":1194432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037809,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234039,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430267,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593793,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378711,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":594631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542237,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":594525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":64034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T10:59:18.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T10:59:19.237 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:19 smithi183 ceph-mon[37273]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:59:19.388 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:19 smithi144 ceph-mon[32129]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:59:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:19 smithi160 ceph-mon[36715]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:59:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:20 smithi183 ceph-mon[37273]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-07-26T10:59:20.420 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:20 smithi144 ceph-mon[32129]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-07-26T10:59:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:20 smithi160 ceph-mon[36715]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-07-26T10:59:21.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 -- 172.21.15.144:0/763520191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c075b30 msgr2=0x7f9e9c10f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:21.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 --2- 172.21.15.144:0/763520191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c075b30 0x7f9e9c10f3e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009900 tx=0x7f9e8802a3f0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 -- 172.21.15.144:0/763520191 shutdown_connections 2024-07-26T10:59:21.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 --2- 172.21.15.144:0/763520191 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c10f920 0x7f9e9c111d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 --2- 172.21.15.144:0/763520191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c075b30 0x7f9e9c10f3e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 --2- 172.21.15.144:0/763520191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c0751e0 0x7f9e9c0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 -- 172.21.15.144:0/763520191 >> 172.21.15.144:0/763520191 conn(0x7f9e9c0fee10 msgr2=0x7f9e9c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:21.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 -- 172.21.15.144:0/763520191 shutdown_connections 2024-07-26T10:59:21.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.426+0000 7f9e9be00640 1 -- 172.21.15.144:0/763520191 wait complete. 2024-07-26T10:59:21.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.427+0000 7f9e9be00640 1 Processor -- start 2024-07-26T10:59:21.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.427+0000 7f9e9be00640 1 -- start start 2024-07-26T10:59:21.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c0751e0 0x7f9e9c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:21.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c075b30 0x7f9e9c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:21.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c10f920 0x7f9e9c1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:21.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9be00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1154f0 con 0x7f9e9c10f920 2024-07-26T10:59:21.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9be00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c115370 con 0x7f9e9c075b30 2024-07-26T10:59:21.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c115670 con 0x7f9e9c0751e0 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c10f920 0x7f9e9c1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9a000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c075b30 0x7f9e9c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c10f920 0x7f9e9c1a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59516/0 (socket says 172.21.15.144:59516) 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.428+0000 7f9e9b400640 1 -- 172.21.15.144:0/3869652406 learned_addr learned my addr 172.21.15.144:0/3869652406 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9aa00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c0751e0 0x7f9e9c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9a000640 1 -- 172.21.15.144:0/3869652406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c0751e0 msgr2=0x7f9e9c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9a000640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c0751e0 0x7f9e9c19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9a000640 1 -- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c10f920 msgr2=0x7f9e9c1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9a000640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c10f920 0x7f9e9c1a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9a000640 1 -- 172.21.15.144:0/3869652406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c1a1ed0 con 0x7f9e9c075b30 2024-07-26T10:59:21.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.429+0000 7f9e9aa00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c0751e0 0x7f9e9c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:21.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.430+0000 7f9e9a000640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c075b30 0x7f9e9c19c500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9e8802a910 tx=0x7f9e88003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:21.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.430+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8803f070 con 0x7f9e9c075b30 2024-07-26T10:59:21.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.430+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c1a2160 con 0x7f9e9c075b30 2024-07-26T10:59:21.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.430+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e88031b60 con 0x7f9e9c075b30 2024-07-26T10:59:21.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.430+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88035030 con 0x7f9e9c075b30 2024-07-26T10:59:21.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.430+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c1a8920 con 0x7f9e9c075b30 2024-07-26T10:59:21.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.432+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9c0755f0 con 0x7f9e9c075b30 2024-07-26T10:59:21.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.434+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f9e88042070 con 0x7f9e9c075b30 2024-07-26T10:59:21.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.435+0000 7f9e93400640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e800b3cb0 0x7f9e800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:21.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.436+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f9e88104790 con 0x7f9e9c075b30 2024-07-26T10:59:21.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.436+0000 7f9e9aa00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e800b3cb0 0x7f9e800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:21.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.436+0000 7f9e9aa00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e800b3cb0 0x7f9e800b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0095f0 tx=0x7f9e8c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:21.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.440+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9e880cb140 con 0x7f9e9c075b30 2024-07-26T10:59:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:21 smithi160 ceph-mon[36715]: pgmap v367: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-07-26T10:59:21.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.634+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9e9c109bd0 con 0x7f9e800b3cb0 2024-07-26T10:59:21.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:21 smithi183 ceph-mon[37273]: pgmap v367: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-07-26T10:59:21.639 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:21 smithi144 ceph-mon[32129]: pgmap v367: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-07-26T10:59:21.645 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.643+0000 7f9e93400640 1 -- 172.21.15.144:0/3869652406 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340362 (secure 0 0 0) 0x7f9e9c109bd0 con 0x7f9e800b3cb0 2024-07-26T10:59:21.647 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:21.650 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e800b3cb0 msgr2=0x7f9e800b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e800b3cb0 0x7f9e800b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0095f0 tx=0x7f9e8c005f90 comp rx=0 tx=0).stop 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c075b30 msgr2=0x7f9e9c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c075b30 0x7f9e9c19c500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9e8802a910 tx=0x7f9e88003c10 comp rx=0 tx=0).stop 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 shutdown_connections 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9e800b3cb0 0x7f9e800b61c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e9c10f920 0x7f9e9c1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e9c075b30 0x7f9e9c19c500 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 --2- 172.21.15.144:0/3869652406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e9c0751e0 0x7f9e9c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:21.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 >> 172.21.15.144:0/3869652406 conn(0x7f9e9c0fee10 msgr2=0x7f9e9c1094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:21.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 shutdown_connections 2024-07-26T10:59:21.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:21.650+0000 7f9e9be00640 1 -- 172.21.15.144:0/3869652406 wait complete. 2024-07-26T10:59:22.453 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-07-26T10:59:21.015263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662280,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":332,"num_read_kb":271,"num_write":295,"num_write_kb":2274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380100,"kb_used_data":14300,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561724,"statfs":{"total":1151940427776,"available":1151551205376,"internally_reserved":0,"allocated":14643200,"data_stored":8384798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19033,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":60,"num_write":12,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005880"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210147+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:58:45.210147+0000","last_peered":"2024-07-26T10:58:45.210147+0000","last_clean":"2024-07-26T10:58:45.210147+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:58:45.210147+0000","last_undegraded":"2024-07-26T10:58:45.210147+0000","last_fullsized":"2024-07-26T10:58:45.210147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211280+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T10:58:45.211280+0000","last_peered":"2024-07-26T10:58:45.211280+0000","last_clean":"2024-07-26T10:58:45.211280+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T10:58:45.211280+0000","last_undegraded":"2024-07-26T10:58:45.211280+0000","last_fullsized":"2024-07-26T10:58:45.211280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205192+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:58:45.205192+0000","last_peered":"2024-07-26T10:58:45.205192+0000","last_clean":"2024-07-26T10:58:45.205192+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:58:45.205192+0000","last_undegraded":"2024-07-26T10:58:45.205192+0000","last_fullsized":"2024-07-26T10:58:45.205192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211762+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:58:45.211762+0000","last_peered":"2024-07-26T10:58:45.211762+0000","last_clean":"2024-07-26T10:58:45.211762+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:58:45.211762+0000","last_undegraded":"2024-07-26T10:58:45.211762+0000","last_fullsized":"2024-07-26T10:58:45.211762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659488+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T10:58:45.659488+0000","last_peered":"2024-07-26T10:58:45.659488+0000","last_clean":"2024-07-26T10:58:45.659488+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T10:58:45.659488+0000","last_undegraded":"2024-07-26T10:58:45.659488+0000","last_fullsized":"2024-07-26T10:58:45.659488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211908+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T10:58:45.211908+0000","last_peered":"2024-07-26T10:58:45.211908+0000","last_clean":"2024-07-26T10:58:45.211908+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T10:58:45.211908+0000","last_undegraded":"2024-07-26T10:58:45.211908+0000","last_fullsized":"2024-07-26T10:58:45.211908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211725+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:58:45.211725+0000","last_peered":"2024-07-26T10:58:45.211725+0000","last_clean":"2024-07-26T10:58:45.211725+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:58:45.211725+0000","last_undegraded":"2024-07-26T10:58:45.211725+0000","last_fullsized":"2024-07-26T10:58:45.211725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394783+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T10:58:45.394783+0000","last_peered":"2024-07-26T10:58:45.394783+0000","last_clean":"2024-07-26T10:58:45.394783+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T10:58:45.394783+0000","last_undegraded":"2024-07-26T10:58:45.394783+0000","last_fullsized":"2024-07-26T10:58:45.394783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212272+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212272+0000","last_peered":"2024-07-26T10:58:45.212272+0000","last_clean":"2024-07-26T10:58:45.212272+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212272+0000","last_undegraded":"2024-07-26T10:58:45.212272+0000","last_fullsized":"2024-07-26T10:58:45.212272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395246+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395246+0000","last_peered":"2024-07-26T10:58:45.395246+0000","last_clean":"2024-07-26T10:58:45.395246+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395246+0000","last_undegraded":"2024-07-26T10:58:45.395246+0000","last_fullsized":"2024-07-26T10:58:45.395246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395593+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T10:58:45.395593+0000","last_peered":"2024-07-26T10:58:45.395593+0000","last_clean":"2024-07-26T10:58:45.395593+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T10:58:45.395593+0000","last_undegraded":"2024-07-26T10:58:45.395593+0000","last_fullsized":"2024-07-26T10:58:45.395593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212052+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212052+0000","last_peered":"2024-07-26T10:58:45.212052+0000","last_clean":"2024-07-26T10:58:45.212052+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212052+0000","last_undegraded":"2024-07-26T10:58:45.212052+0000","last_fullsized":"2024-07-26T10:58:45.212052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210799+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T10:58:45.210799+0000","last_peered":"2024-07-26T10:58:45.210799+0000","last_clean":"2024-07-26T10:58:45.210799+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T10:58:45.210799+0000","last_undegraded":"2024-07-26T10:58:45.210799+0000","last_fullsized":"2024-07-26T10:58:45.210799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211096+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:58:45.211096+0000","last_peered":"2024-07-26T10:58:45.211096+0000","last_clean":"2024-07-26T10:58:45.211096+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:58:45.211096+0000","last_undegraded":"2024-07-26T10:58:45.211096+0000","last_fullsized":"2024-07-26T10:58:45.211096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212084+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:58:45.212084+0000","last_peered":"2024-07-26T10:58:45.212084+0000","last_clean":"2024-07-26T10:58:45.212084+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:58:45.212084+0000","last_undegraded":"2024-07-26T10:58:45.212084+0000","last_fullsized":"2024-07-26T10:58:45.212084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210731+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T10:58:45.210731+0000","last_peered":"2024-07-26T10:58:45.210731+0000","last_clean":"2024-07-26T10:58:45.210731+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T10:58:45.210731+0000","last_undegraded":"2024-07-26T10:58:45.210731+0000","last_fullsized":"2024-07-26T10:58:45.210731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205302+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T10:58:45.205302+0000","last_peered":"2024-07-26T10:58:45.205302+0000","last_clean":"2024-07-26T10:58:45.205302+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T10:58:45.205302+0000","last_undegraded":"2024-07-26T10:58:45.205302+0000","last_fullsized":"2024-07-26T10:58:45.205302+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.398918+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:58:45.398918+0000","last_peered":"2024-07-26T10:58:45.398918+0000","last_clean":"2024-07-26T10:58:45.398918+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:58:45.398918+0000","last_undegraded":"2024-07-26T10:58:45.398918+0000","last_fullsized":"2024-07-26T10:58:45.398918+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394934+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T10:58:45.394934+0000","last_peered":"2024-07-26T10:58:45.394934+0000","last_clean":"2024-07-26T10:58:45.394934+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T10:58:45.394934+0000","last_undegraded":"2024-07-26T10:58:45.394934+0000","last_fullsized":"2024-07-26T10:58:45.394934+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210014+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:58:45.210014+0000","last_peered":"2024-07-26T10:58:45.210014+0000","last_clean":"2024-07-26T10:58:45.210014+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:58:45.210014+0000","last_undegraded":"2024-07-26T10:58:45.210014+0000","last_fullsized":"2024-07-26T10:58:45.210014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211546+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T10:58:45.211546+0000","last_peered":"2024-07-26T10:58:45.211546+0000","last_clean":"2024-07-26T10:58:45.211546+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T10:58:45.211546+0000","last_undegraded":"2024-07-26T10:58:45.211546+0000","last_fullsized":"2024-07-26T10:58:45.211546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211224+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T10:58:45.211224+0000","last_peered":"2024-07-26T10:58:45.211224+0000","last_clean":"2024-07-26T10:58:45.211224+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T10:58:45.211224+0000","last_undegraded":"2024-07-26T10:58:45.211224+0000","last_fullsized":"2024-07-26T10:58:45.211224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394716+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:58:45.394716+0000","last_peered":"2024-07-26T10:58:45.394716+0000","last_clean":"2024-07-26T10:58:45.394716+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:58:45.394716+0000","last_undegraded":"2024-07-26T10:58:45.394716+0000","last_fullsized":"2024-07-26T10:58:45.394716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212508+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:58:45.212508+0000","last_peered":"2024-07-26T10:58:45.212508+0000","last_clean":"2024-07-26T10:58:45.212508+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:58:45.212508+0000","last_undegraded":"2024-07-26T10:58:45.212508+0000","last_fullsized":"2024-07-26T10:58:45.212508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659287+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T10:58:45.659287+0000","last_peered":"2024-07-26T10:58:45.659287+0000","last_clean":"2024-07-26T10:58:45.659287+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T10:58:45.659287+0000","last_undegraded":"2024-07-26T10:58:45.659287+0000","last_fullsized":"2024-07-26T10:58:45.659287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212680+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:58:45.212680+0000","last_peered":"2024-07-26T10:58:45.212680+0000","last_clean":"2024-07-26T10:58:45.212680+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:58:45.212680+0000","last_undegraded":"2024-07-26T10:58:45.212680+0000","last_fullsized":"2024-07-26T10:58:45.212680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205392+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:58:45.205392+0000","last_peered":"2024-07-26T10:58:45.205392+0000","last_clean":"2024-07-26T10:58:45.205392+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:58:45.205392+0000","last_undegraded":"2024-07-26T10:58:45.205392+0000","last_fullsized":"2024-07-26T10:58:45.205392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211913+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T10:58:45.211913+0000","last_peered":"2024-07-26T10:58:45.211913+0000","last_clean":"2024-07-26T10:58:45.211913+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T10:58:45.211913+0000","last_undegraded":"2024-07-26T10:58:45.211913+0000","last_fullsized":"2024-07-26T10:58:45.211913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:09.157867+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T10:59:09.157867+0000","last_peered":"2024-07-26T10:59:09.157867+0000","last_clean":"2024-07-26T10:59:09.157867+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T10:59:09.157867+0000","last_undegraded":"2024-07-26T10:59:09.157867+0000","last_fullsized":"2024-07-26T10:59:09.157867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208653+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:58:45.208653+0000","last_peered":"2024-07-26T10:58:45.208653+0000","last_clean":"2024-07-26T10:58:45.208653+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:58:45.208653+0000","last_undegraded":"2024-07-26T10:58:45.208653+0000","last_fullsized":"2024-07-26T10:58:45.208653+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397128+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T10:58:45.397128+0000","last_peered":"2024-07-26T10:58:45.397128+0000","last_clean":"2024-07-26T10:58:45.397128+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T10:58:45.397128+0000","last_undegraded":"2024-07-26T10:58:45.397128+0000","last_fullsized":"2024-07-26T10:58:45.397128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211738+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:58:45.211738+0000","last_peered":"2024-07-26T10:58:45.211738+0000","last_clean":"2024-07-26T10:58:45.211738+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:58:45.211738+0000","last_undegraded":"2024-07-26T10:58:45.211738+0000","last_fullsized":"2024-07-26T10:58:45.211738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395666+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:58:45.395666+0000","last_peered":"2024-07-26T10:58:45.395666+0000","last_clean":"2024-07-26T10:58:45.395666+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:58:45.395666+0000","last_undegraded":"2024-07-26T10:58:45.395666+0000","last_fullsized":"2024-07-26T10:58:45.395666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395766+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T10:58:45.395766+0000","last_peered":"2024-07-26T10:58:45.395766+0000","last_clean":"2024-07-26T10:58:45.395766+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T10:58:45.395766+0000","last_undegraded":"2024-07-26T10:58:45.395766+0000","last_fullsized":"2024-07-26T10:58:45.395766+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208881+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:58:45.208881+0000","last_peered":"2024-07-26T10:58:45.208881+0000","last_clean":"2024-07-26T10:58:45.208881+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:58:45.208881+0000","last_undegraded":"2024-07-26T10:58:45.208881+0000","last_fullsized":"2024-07-26T10:58:45.208881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210980+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:58:45.210980+0000","last_peered":"2024-07-26T10:58:45.210980+0000","last_clean":"2024-07-26T10:58:45.210980+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:58:45.210980+0000","last_undegraded":"2024-07-26T10:58:45.210980+0000","last_fullsized":"2024-07-26T10:58:45.210980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395794+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T10:58:45.395794+0000","last_peered":"2024-07-26T10:58:45.395794+0000","last_clean":"2024-07-26T10:58:45.395794+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T10:58:45.395794+0000","last_undegraded":"2024-07-26T10:58:45.395794+0000","last_fullsized":"2024-07-26T10:58:45.395794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210920+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:58:45.210920+0000","last_peered":"2024-07-26T10:58:45.210920+0000","last_clean":"2024-07-26T10:58:45.210920+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:58:45.210920+0000","last_undegraded":"2024-07-26T10:58:45.210920+0000","last_fullsized":"2024-07-26T10:58:45.210920+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396573+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:58:45.396573+0000","last_peered":"2024-07-26T10:58:45.396573+0000","last_clean":"2024-07-26T10:58:45.396573+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:58:45.396573+0000","last_undegraded":"2024-07-26T10:58:45.396573+0000","last_fullsized":"2024-07-26T10:58:45.396573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395303+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T10:58:45.395303+0000","last_peered":"2024-07-26T10:58:45.395303+0000","last_clean":"2024-07-26T10:58:45.395303+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T10:58:45.395303+0000","last_undegraded":"2024-07-26T10:58:45.395303+0000","last_fullsized":"2024-07-26T10:58:45.395303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:16.264611+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T10:59:16.264611+0000","last_peered":"2024-07-26T10:59:16.264611+0000","last_clean":"2024-07-26T10:59:16.264611+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T10:59:16.264611+0000","last_undegraded":"2024-07-26T10:59:16.264611+0000","last_fullsized":"2024-07-26T10:59:16.264611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396965+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:58:45.396965+0000","last_peered":"2024-07-26T10:58:45.396965+0000","last_clean":"2024-07-26T10:58:45.396965+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:58:45.396965+0000","last_undegraded":"2024-07-26T10:58:45.396965+0000","last_fullsized":"2024-07-26T10:58:45.396965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395960+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T10:58:45.395960+0000","last_peered":"2024-07-26T10:58:45.395960+0000","last_clean":"2024-07-26T10:58:45.395960+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T10:58:45.395960+0000","last_undegraded":"2024-07-26T10:58:45.395960+0000","last_fullsized":"2024-07-26T10:58:45.395960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210600+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:58:45.210600+0000","last_peered":"2024-07-26T10:58:45.210600+0000","last_clean":"2024-07-26T10:58:45.210600+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:58:45.210600+0000","last_undegraded":"2024-07-26T10:58:45.210600+0000","last_fullsized":"2024-07-26T10:58:45.210600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.399431+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:45.399431+0000","last_peered":"2024-07-26T10:58:45.399431+0000","last_clean":"2024-07-26T10:58:45.399431+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:45.399431+0000","last_undegraded":"2024-07-26T10:58:45.399431+0000","last_fullsized":"2024-07-26T10:58:45.399431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210623+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T10:58:45.210623+0000","last_peered":"2024-07-26T10:58:45.210623+0000","last_clean":"2024-07-26T10:58:45.210623+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T10:58:45.210623+0000","last_undegraded":"2024-07-26T10:58:45.210623+0000","last_fullsized":"2024-07-26T10:58:45.210623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212541+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:58:45.212541+0000","last_peered":"2024-07-26T10:58:45.212541+0000","last_clean":"2024-07-26T10:58:45.212541+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:58:45.212541+0000","last_undegraded":"2024-07-26T10:58:45.212541+0000","last_fullsized":"2024-07-26T10:58:45.212541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":121,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210602+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210602+0000","last_peered":"2024-07-26T10:58:45.210602+0000","last_clean":"2024-07-26T10:58:45.210602+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210602+0000","last_undegraded":"2024-07-26T10:58:45.210602+0000","last_fullsized":"2024-07-26T10:58:45.210602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.398754+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:58:45.398754+0000","last_peered":"2024-07-26T10:58:45.398754+0000","last_clean":"2024-07-26T10:58:45.398754+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:58:45.398754+0000","last_undegraded":"2024-07-26T10:58:45.398754+0000","last_fullsized":"2024-07-26T10:58:45.398754+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211486+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T10:58:45.211486+0000","last_peered":"2024-07-26T10:58:45.211486+0000","last_clean":"2024-07-26T10:58:45.211486+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T10:58:45.211486+0000","last_undegraded":"2024-07-26T10:58:45.211486+0000","last_fullsized":"2024-07-26T10:58:45.211486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205785+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:58:45.205785+0000","last_peered":"2024-07-26T10:58:45.205785+0000","last_clean":"2024-07-26T10:58:45.205785+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:58:45.205785+0000","last_undegraded":"2024-07-26T10:58:45.205785+0000","last_fullsized":"2024-07-26T10:58:45.205785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394314+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:58:45.394314+0000","last_peered":"2024-07-26T10:58:45.394314+0000","last_clean":"2024-07-26T10:58:45.394314+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:58:45.394314+0000","last_undegraded":"2024-07-26T10:58:45.394314+0000","last_fullsized":"2024-07-26T10:58:45.394314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211236+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T10:58:45.211236+0000","last_peered":"2024-07-26T10:58:45.211236+0000","last_clean":"2024-07-26T10:58:45.211236+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T10:58:45.211236+0000","last_undegraded":"2024-07-26T10:58:45.211236+0000","last_fullsized":"2024-07-26T10:58:45.211236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396372+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:58:45.396372+0000","last_peered":"2024-07-26T10:58:45.396372+0000","last_clean":"2024-07-26T10:58:45.396372+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:58:45.396372+0000","last_undegraded":"2024-07-26T10:58:45.396372+0000","last_fullsized":"2024-07-26T10:58:45.396372+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658809+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:58:45.658809+0000","last_peered":"2024-07-26T10:58:45.658809+0000","last_clean":"2024-07-26T10:58:45.658809+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:58:45.658809+0000","last_undegraded":"2024-07-26T10:58:45.658809+0000","last_fullsized":"2024-07-26T10:58:45.658809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211382+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T10:58:45.211382+0000","last_peered":"2024-07-26T10:58:45.211382+0000","last_clean":"2024-07-26T10:58:45.211382+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T10:58:45.211382+0000","last_undegraded":"2024-07-26T10:58:45.211382+0000","last_fullsized":"2024-07-26T10:58:45.211382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397505+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:58:45.397505+0000","last_peered":"2024-07-26T10:58:45.397505+0000","last_clean":"2024-07-26T10:58:45.397505+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:58:45.397505+0000","last_undegraded":"2024-07-26T10:58:45.397505+0000","last_fullsized":"2024-07-26T10:58:45.397505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395507+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:58:45.395507+0000","last_peered":"2024-07-26T10:58:45.395507+0000","last_clean":"2024-07-26T10:58:45.395507+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:58:45.395507+0000","last_undegraded":"2024-07-26T10:58:45.395507+0000","last_fullsized":"2024-07-26T10:58:45.395507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396126+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T10:58:45.396126+0000","last_peered":"2024-07-26T10:58:45.396126+0000","last_clean":"2024-07-26T10:58:45.396126+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T10:58:45.396126+0000","last_undegraded":"2024-07-26T10:58:45.396126+0000","last_fullsized":"2024-07-26T10:58:45.396126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211685+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:58:45.211685+0000","last_peered":"2024-07-26T10:58:45.211685+0000","last_clean":"2024-07-26T10:58:45.211685+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:58:45.211685+0000","last_undegraded":"2024-07-26T10:58:45.211685+0000","last_fullsized":"2024-07-26T10:58:45.211685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395832+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T10:58:45.395832+0000","last_peered":"2024-07-26T10:58:45.395832+0000","last_clean":"2024-07-26T10:58:45.395832+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T10:58:45.395832+0000","last_undegraded":"2024-07-26T10:58:45.395832+0000","last_fullsized":"2024-07-26T10:58:45.395832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210974+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:58:45.210974+0000","last_peered":"2024-07-26T10:58:45.210974+0000","last_clean":"2024-07-26T10:58:45.210974+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:58:45.210974+0000","last_undegraded":"2024-07-26T10:58:45.210974+0000","last_fullsized":"2024-07-26T10:58:45.210974+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397198+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397198+0000","last_peered":"2024-07-26T10:58:45.397198+0000","last_clean":"2024-07-26T10:58:45.397198+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397198+0000","last_undegraded":"2024-07-26T10:58:45.397198+0000","last_fullsized":"2024-07-26T10:58:45.397198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210333+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T10:58:45.210333+0000","last_peered":"2024-07-26T10:58:45.210333+0000","last_clean":"2024-07-26T10:58:45.210333+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T10:58:45.210333+0000","last_undegraded":"2024-07-26T10:58:45.210333+0000","last_fullsized":"2024-07-26T10:58:45.210333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:15.263422+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T10:59:15.263422+0000","last_peered":"2024-07-26T10:59:15.263422+0000","last_clean":"2024-07-26T10:59:15.263422+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T10:59:15.263422+0000","last_undegraded":"2024-07-26T10:59:15.263422+0000","last_fullsized":"2024-07-26T10:59:15.263422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208051+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T10:58:45.208051+0000","last_peered":"2024-07-26T10:58:45.208051+0000","last_clean":"2024-07-26T10:58:45.208051+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T10:58:45.208051+0000","last_undegraded":"2024-07-26T10:58:45.208051+0000","last_fullsized":"2024-07-26T10:58:45.208051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210991+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T10:58:45.210991+0000","last_peered":"2024-07-26T10:58:45.210991+0000","last_clean":"2024-07-26T10:58:45.210991+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T10:58:45.210991+0000","last_undegraded":"2024-07-26T10:58:45.210991+0000","last_fullsized":"2024-07-26T10:58:45.210991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395620+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:58:45.395620+0000","last_peered":"2024-07-26T10:58:45.395620+0000","last_clean":"2024-07-26T10:58:45.395620+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:58:45.395620+0000","last_undegraded":"2024-07-26T10:58:45.395620+0000","last_fullsized":"2024-07-26T10:58:45.395620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395383+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:58:45.395383+0000","last_peered":"2024-07-26T10:58:45.395383+0000","last_clean":"2024-07-26T10:58:45.395383+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:58:45.395383+0000","last_undegraded":"2024-07-26T10:58:45.395383+0000","last_fullsized":"2024-07-26T10:58:45.395383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396333+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T10:58:45.396333+0000","last_peered":"2024-07-26T10:58:45.396333+0000","last_clean":"2024-07-26T10:58:45.396333+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T10:58:45.396333+0000","last_undegraded":"2024-07-26T10:58:45.396333+0000","last_fullsized":"2024-07-26T10:58:45.396333+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205552+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:58:45.205552+0000","last_peered":"2024-07-26T10:58:45.205552+0000","last_clean":"2024-07-26T10:58:45.205552+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:58:45.205552+0000","last_undegraded":"2024-07-26T10:58:45.205552+0000","last_fullsized":"2024-07-26T10:58:45.205552+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395470+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395470+0000","last_peered":"2024-07-26T10:58:45.395470+0000","last_clean":"2024-07-26T10:58:45.395470+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395470+0000","last_undegraded":"2024-07-26T10:58:45.395470+0000","last_fullsized":"2024-07-26T10:58:45.395470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211550+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T10:58:45.211550+0000","last_peered":"2024-07-26T10:58:45.211550+0000","last_clean":"2024-07-26T10:58:45.211550+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T10:58:45.211550+0000","last_undegraded":"2024-07-26T10:58:45.211550+0000","last_fullsized":"2024-07-26T10:58:45.211550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'34","reported_seq":164,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:19.255969+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T10:59:19.255969+0000","last_peered":"2024-07-26T10:59:19.255969+0000","last_clean":"2024-07-26T10:59:19.255969+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T10:59:19.255969+0000","last_undegraded":"2024-07-26T10:59:19.255969+0000","last_fullsized":"2024-07-26T10:59:19.255969+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67677,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":86,"num_read_kb":109,"num_write":35,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208021+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:58:45.208021+0000","last_peered":"2024-07-26T10:58:45.208021+0000","last_clean":"2024-07-26T10:58:45.208021+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:58:45.208021+0000","last_undegraded":"2024-07-26T10:58:45.208021+0000","last_fullsized":"2024-07-26T10:58:45.208021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394854+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T10:58:45.394854+0000","last_peered":"2024-07-26T10:58:45.394854+0000","last_clean":"2024-07-26T10:58:45.394854+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T10:58:45.394854+0000","last_undegraded":"2024-07-26T10:58:45.394854+0000","last_fullsized":"2024-07-26T10:58:45.394854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396288+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:58:45.396288+0000","last_peered":"2024-07-26T10:58:45.396288+0000","last_clean":"2024-07-26T10:58:45.396288+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:58:45.396288+0000","last_undegraded":"2024-07-26T10:58:45.396288+0000","last_fullsized":"2024-07-26T10:58:45.396288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212370+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:58:45.212370+0000","last_peered":"2024-07-26T10:58:45.212370+0000","last_clean":"2024-07-26T10:58:45.212370+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:58:45.212370+0000","last_undegraded":"2024-07-26T10:58:45.212370+0000","last_fullsized":"2024-07-26T10:58:45.212370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659486+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T10:58:45.659486+0000","last_peered":"2024-07-26T10:58:45.659486+0000","last_clean":"2024-07-26T10:58:45.659486+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T10:58:45.659486+0000","last_undegraded":"2024-07-26T10:58:45.659486+0000","last_fullsized":"2024-07-26T10:58:45.659486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211388+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:58:45.211388+0000","last_peered":"2024-07-26T10:58:45.211388+0000","last_clean":"2024-07-26T10:58:45.211388+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:58:45.211388+0000","last_undegraded":"2024-07-26T10:58:45.211388+0000","last_fullsized":"2024-07-26T10:58:45.211388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207003+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:58:45.207003+0000","last_peered":"2024-07-26T10:58:45.207003+0000","last_clean":"2024-07-26T10:58:45.207003+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:58:45.207003+0000","last_undegraded":"2024-07-26T10:58:45.207003+0000","last_fullsized":"2024-07-26T10:58:45.207003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394873+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T10:58:45.394873+0000","last_peered":"2024-07-26T10:58:45.394873+0000","last_clean":"2024-07-26T10:58:45.394873+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T10:58:45.394873+0000","last_undegraded":"2024-07-26T10:58:45.394873+0000","last_fullsized":"2024-07-26T10:58:45.394873+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:18.254689+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T10:59:18.254689+0000","last_peered":"2024-07-26T10:59:18.254689+0000","last_clean":"2024-07-26T10:59:18.254689+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T10:59:18.254689+0000","last_undegraded":"2024-07-26T10:59:18.254689+0000","last_fullsized":"2024-07-26T10:59:18.254689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208888+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:58:45.208888+0000","last_peered":"2024-07-26T10:58:45.208888+0000","last_clean":"2024-07-26T10:58:45.208888+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:58:45.208888+0000","last_undegraded":"2024-07-26T10:58:45.208888+0000","last_fullsized":"2024-07-26T10:58:45.208888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658974+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:58:45.658974+0000","last_peered":"2024-07-26T10:58:45.658974+0000","last_clean":"2024-07-26T10:58:45.658974+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:58:45.658974+0000","last_undegraded":"2024-07-26T10:58:45.658974+0000","last_fullsized":"2024-07-26T10:58:45.658974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207341+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T10:58:45.207341+0000","last_peered":"2024-07-26T10:58:45.207341+0000","last_clean":"2024-07-26T10:58:45.207341+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T10:58:45.207341+0000","last_undegraded":"2024-07-26T10:58:45.207341+0000","last_fullsized":"2024-07-26T10:58:45.207341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395654+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:58:45.395654+0000","last_peered":"2024-07-26T10:58:45.395654+0000","last_clean":"2024-07-26T10:58:45.395654+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:58:45.395654+0000","last_undegraded":"2024-07-26T10:58:45.395654+0000","last_fullsized":"2024-07-26T10:58:45.395654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395538+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:58:45.395538+0000","last_peered":"2024-07-26T10:58:45.395538+0000","last_clean":"2024-07-26T10:58:45.395538+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:58:45.395538+0000","last_undegraded":"2024-07-26T10:58:45.395538+0000","last_fullsized":"2024-07-26T10:58:45.395538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210311+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T10:58:45.210311+0000","last_peered":"2024-07-26T10:58:45.210311+0000","last_clean":"2024-07-26T10:58:45.210311+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T10:58:45.210311+0000","last_undegraded":"2024-07-26T10:58:45.210311+0000","last_fullsized":"2024-07-26T10:58:45.210311+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396052+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:58:45.396052+0000","last_peered":"2024-07-26T10:58:45.396052+0000","last_clean":"2024-07-26T10:58:45.396052+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:58:45.396052+0000","last_undegraded":"2024-07-26T10:58:45.396052+0000","last_fullsized":"2024-07-26T10:58:45.396052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395567+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:58:45.395567+0000","last_peered":"2024-07-26T10:58:45.395567+0000","last_clean":"2024-07-26T10:58:45.395567+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:58:45.395567+0000","last_undegraded":"2024-07-26T10:58:45.395567+0000","last_fullsized":"2024-07-26T10:58:45.395567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207651+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T10:58:45.207651+0000","last_peered":"2024-07-26T10:58:45.207651+0000","last_clean":"2024-07-26T10:58:45.207651+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T10:58:45.207651+0000","last_undegraded":"2024-07-26T10:58:45.207651+0000","last_fullsized":"2024-07-26T10:58:45.207651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205489+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205489+0000","last_peered":"2024-07-26T10:58:45.205489+0000","last_clean":"2024-07-26T10:58:45.205489+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205489+0000","last_undegraded":"2024-07-26T10:58:45.205489+0000","last_fullsized":"2024-07-26T10:58:45.205489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:17.253824+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T10:59:17.253824+0000","last_peered":"2024-07-26T10:59:17.253824+0000","last_clean":"2024-07-26T10:59:17.253824+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T10:59:17.253824+0000","last_undegraded":"2024-07-26T10:59:17.253824+0000","last_fullsized":"2024-07-26T10:59:17.253824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211262+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T10:58:45.211262+0000","last_peered":"2024-07-26T10:58:45.211262+0000","last_clean":"2024-07-26T10:58:45.211262+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T10:58:45.211262+0000","last_undegraded":"2024-07-26T10:58:45.211262+0000","last_fullsized":"2024-07-26T10:58:45.211262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205703+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205703+0000","last_peered":"2024-07-26T10:58:45.205703+0000","last_clean":"2024-07-26T10:58:45.205703+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205703+0000","last_undegraded":"2024-07-26T10:58:45.205703+0000","last_fullsized":"2024-07-26T10:58:45.205703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394757+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T10:58:45.394757+0000","last_peered":"2024-07-26T10:58:45.394757+0000","last_clean":"2024-07-26T10:58:45.394757+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T10:58:45.394757+0000","last_undegraded":"2024-07-26T10:58:45.394757+0000","last_fullsized":"2024-07-26T10:58:45.394757+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205577+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T10:58:45.205577+0000","last_peered":"2024-07-26T10:58:45.205577+0000","last_clean":"2024-07-26T10:58:45.205577+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T10:58:45.205577+0000","last_undegraded":"2024-07-26T10:58:45.205577+0000","last_fullsized":"2024-07-26T10:58:45.205577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205339+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T10:58:45.205339+0000","last_peered":"2024-07-26T10:58:45.205339+0000","last_clean":"2024-07-26T10:58:45.205339+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T10:58:45.205339+0000","last_undegraded":"2024-07-26T10:58:45.205339+0000","last_fullsized":"2024-07-26T10:58:45.205339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659046+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T10:58:45.659046+0000","last_peered":"2024-07-26T10:58:45.659046+0000","last_clean":"2024-07-26T10:58:45.659046+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T10:58:45.659046+0000","last_undegraded":"2024-07-26T10:58:45.659046+0000","last_fullsized":"2024-07-26T10:58:45.659046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397091+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T10:58:45.397091+0000","last_peered":"2024-07-26T10:58:45.397091+0000","last_clean":"2024-07-26T10:58:45.397091+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T10:58:45.397091+0000","last_undegraded":"2024-07-26T10:58:45.397091+0000","last_fullsized":"2024-07-26T10:58:45.397091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395386+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T10:58:45.395386+0000","last_peered":"2024-07-26T10:58:45.395386+0000","last_clean":"2024-07-26T10:58:45.395386+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T10:58:45.395386+0000","last_undegraded":"2024-07-26T10:58:45.395386+0000","last_fullsized":"2024-07-26T10:58:45.395386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205978+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T10:58:45.205978+0000","last_peered":"2024-07-26T10:58:45.205978+0000","last_clean":"2024-07-26T10:58:45.205978+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T10:58:45.205978+0000","last_undegraded":"2024-07-26T10:58:45.205978+0000","last_fullsized":"2024-07-26T10:58:45.205978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211764+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T10:58:45.211764+0000","last_peered":"2024-07-26T10:58:45.211764+0000","last_clean":"2024-07-26T10:58:45.211764+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T10:58:45.211764+0000","last_undegraded":"2024-07-26T10:58:45.211764+0000","last_fullsized":"2024-07-26T10:58:45.211764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395147+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T10:58:45.395147+0000","last_peered":"2024-07-26T10:58:45.395147+0000","last_clean":"2024-07-26T10:58:45.395147+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T10:58:45.395147+0000","last_undegraded":"2024-07-26T10:58:45.395147+0000","last_fullsized":"2024-07-26T10:58:45.395147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209328+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T10:58:45.209328+0000","last_peered":"2024-07-26T10:58:45.209328+0000","last_clean":"2024-07-26T10:58:45.209328+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T10:58:45.209328+0000","last_undegraded":"2024-07-26T10:58:45.209328+0000","last_fullsized":"2024-07-26T10:58:45.209328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394527+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T10:58:45.394527+0000","last_peered":"2024-07-26T10:58:45.394527+0000","last_clean":"2024-07-26T10:58:45.394527+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T10:58:45.394527+0000","last_undegraded":"2024-07-26T10:58:45.394527+0000","last_fullsized":"2024-07-26T10:58:45.394527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396686+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T10:58:45.396686+0000","last_peered":"2024-07-26T10:58:45.396686+0000","last_clean":"2024-07-26T10:58:45.396686+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T10:58:45.396686+0000","last_undegraded":"2024-07-26T10:58:45.396686+0000","last_fullsized":"2024-07-26T10:58:45.396686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395536+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T10:58:45.395536+0000","last_peered":"2024-07-26T10:58:45.395536+0000","last_clean":"2024-07-26T10:58:45.395536+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T10:58:45.395536+0000","last_undegraded":"2024-07-26T10:58:45.395536+0000","last_fullsized":"2024-07-26T10:58:45.395536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205860+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T10:58:45.205860+0000","last_peered":"2024-07-26T10:58:45.205860+0000","last_clean":"2024-07-26T10:58:45.205860+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T10:58:45.205860+0000","last_undegraded":"2024-07-26T10:58:45.205860+0000","last_fullsized":"2024-07-26T10:58:45.205860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209337+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T10:58:45.209337+0000","last_peered":"2024-07-26T10:58:45.209337+0000","last_clean":"2024-07-26T10:58:45.209337+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T10:58:45.209337+0000","last_undegraded":"2024-07-26T10:58:45.209337+0000","last_fullsized":"2024-07-26T10:58:45.209337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396018+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T10:58:45.396018+0000","last_peered":"2024-07-26T10:58:45.396018+0000","last_clean":"2024-07-26T10:58:45.396018+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T10:58:45.396018+0000","last_undegraded":"2024-07-26T10:58:45.396018+0000","last_fullsized":"2024-07-26T10:58:45.396018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394191+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T10:58:45.394191+0000","last_peered":"2024-07-26T10:58:45.394191+0000","last_clean":"2024-07-26T10:58:45.394191+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T10:58:45.394191+0000","last_undegraded":"2024-07-26T10:58:45.394191+0000","last_fullsized":"2024-07-26T10:58:45.394191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658489+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T10:58:45.658489+0000","last_peered":"2024-07-26T10:58:45.658489+0000","last_clean":"2024-07-26T10:58:45.658489+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T10:58:45.658489+0000","last_undegraded":"2024-07-26T10:58:45.658489+0000","last_fullsized":"2024-07-26T10:58:45.658489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658432+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T10:58:45.658432+0000","last_peered":"2024-07-26T10:58:45.658432+0000","last_clean":"2024-07-26T10:58:45.658432+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T10:58:45.658432+0000","last_undegraded":"2024-07-26T10:58:45.658432+0000","last_fullsized":"2024-07-26T10:58:45.658432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396923+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T10:58:45.396923+0000","last_peered":"2024-07-26T10:58:45.396923+0000","last_clean":"2024-07-26T10:58:45.396923+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T10:58:45.396923+0000","last_undegraded":"2024-07-26T10:58:45.396923+0000","last_fullsized":"2024-07-26T10:58:45.396923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.213030+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T10:58:45.213030+0000","last_peered":"2024-07-26T10:58:45.213030+0000","last_clean":"2024-07-26T10:58:45.213030+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T10:58:45.213030+0000","last_undegraded":"2024-07-26T10:58:45.213030+0000","last_fullsized":"2024-07-26T10:58:45.213030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211415+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T10:58:45.211415+0000","last_peered":"2024-07-26T10:58:45.211415+0000","last_clean":"2024-07-26T10:58:45.211415+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T10:58:45.211415+0000","last_undegraded":"2024-07-26T10:58:45.211415+0000","last_fullsized":"2024-07-26T10:58:45.211415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396247+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T10:58:45.396247+0000","last_peered":"2024-07-26T10:58:45.396247+0000","last_clean":"2024-07-26T10:58:45.396247+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T10:58:45.396247+0000","last_undegraded":"2024-07-26T10:58:45.396247+0000","last_fullsized":"2024-07-26T10:58:45.396247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395006+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T10:58:45.395006+0000","last_peered":"2024-07-26T10:58:45.395006+0000","last_clean":"2024-07-26T10:58:45.395006+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T10:58:45.395006+0000","last_undegraded":"2024-07-26T10:58:45.395006+0000","last_fullsized":"2024-07-26T10:58:45.395006+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209162+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T10:58:45.209162+0000","last_peered":"2024-07-26T10:58:45.209162+0000","last_clean":"2024-07-26T10:58:45.209162+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T10:58:45.209162+0000","last_undegraded":"2024-07-26T10:58:45.209162+0000","last_fullsized":"2024-07-26T10:58:45.209162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210440+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T10:58:45.210440+0000","last_peered":"2024-07-26T10:58:45.210440+0000","last_clean":"2024-07-26T10:58:45.210440+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T10:58:45.210440+0000","last_undegraded":"2024-07-26T10:58:45.210440+0000","last_fullsized":"2024-07-26T10:58:45.210440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395504+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T10:58:45.395504+0000","last_peered":"2024-07-26T10:58:45.395504+0000","last_clean":"2024-07-26T10:58:45.395504+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T10:58:45.395504+0000","last_undegraded":"2024-07-26T10:58:45.395504+0000","last_fullsized":"2024-07-26T10:58:45.395504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211081+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T10:58:45.211081+0000","last_peered":"2024-07-26T10:58:45.211081+0000","last_clean":"2024-07-26T10:58:45.211081+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T10:58:45.211081+0000","last_undegraded":"2024-07-26T10:58:45.211081+0000","last_fullsized":"2024-07-26T10:58:45.211081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212942+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T10:58:45.212942+0000","last_peered":"2024-07-26T10:58:45.212942+0000","last_clean":"2024-07-26T10:58:45.212942+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T10:58:45.212942+0000","last_undegraded":"2024-07-26T10:58:45.212942+0000","last_fullsized":"2024-07-26T10:58:45.212942+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658849+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T10:58:45.658849+0000","last_peered":"2024-07-26T10:58:45.658849+0000","last_clean":"2024-07-26T10:58:45.658849+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T10:58:45.658849+0000","last_undegraded":"2024-07-26T10:58:45.658849+0000","last_fullsized":"2024-07-26T10:58:45.658849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659284+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T10:58:45.659284+0000","last_peered":"2024-07-26T10:58:45.659284+0000","last_clean":"2024-07-26T10:58:45.659284+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T10:58:45.659284+0000","last_undegraded":"2024-07-26T10:58:45.659284+0000","last_fullsized":"2024-07-26T10:58:45.659284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395922+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T10:58:45.395922+0000","last_peered":"2024-07-26T10:58:45.395922+0000","last_clean":"2024-07-26T10:58:45.395922+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T10:58:45.395922+0000","last_undegraded":"2024-07-26T10:58:45.395922+0000","last_fullsized":"2024-07-26T10:58:45.395922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71678,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":170,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":215034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678055,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874285,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1198697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037810,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234039,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790024,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378712,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":598896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542238,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":598790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T10:59:22.458 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T10:59:22.459 INFO:tasks.cephadm:Setting up client nodes... 2024-07-26T10:59:22.459 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-26T10:59:22.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:22 smithi144 ceph-mon[32129]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:59:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:22 smithi160 ceph-mon[36715]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:59:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:22 smithi183 ceph-mon[37273]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T10:59:22.960 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:23.404 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:23.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:23 smithi144 ceph-mon[32129]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-07-26T10:59:23.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:23.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:23 smithi160 ceph-mon[36715]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-07-26T10:59:23.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:23.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:23.580 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:23 smithi183 ceph-mon[37273]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-07-26T10:59:25.404 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.404 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.404 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.404 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.404 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:25 smithi144 ceph-mon[32129]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-07-26T10:59:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:25 smithi160 ceph-mon[36715]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-07-26T10:59:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:25 smithi183 ceph-mon[37273]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-07-26T10:59:25.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 -- 172.21.15.144:0/3435107108 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 msgr2=0x7f7d1c110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:25.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 --2- 172.21.15.144:0/3435107108 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c110070 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009be0 tx=0x7f7d1402a6d0 comp rx=0 tx=0).stop 2024-07-26T10:59:25.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 -- 172.21.15.144:0/3435107108 shutdown_connections 2024-07-26T10:59:25.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 --2- 172.21.15.144:0/3435107108 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c110070 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:25.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 --2- 172.21.15.144:0/3435107108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:25.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 --2- 172.21.15.144:0/3435107108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c1035b0 0x7f7d1c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:25.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 -- 172.21.15.144:0/3435107108 >> 172.21.15.144:0/3435107108 conn(0x7f7d1c0ff000 msgr2=0x7f7d1c101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:25.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 -- 172.21.15.144:0/3435107108 shutdown_connections 2024-07-26T10:59:25.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.711+0000 7f7d23a00640 1 -- 172.21.15.144:0/3435107108 wait complete. 2024-07-26T10:59:25.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 Processor -- start 2024-07-26T10:59:25.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 -- start start 2024-07-26T10:59:25.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1c1035b0 0x7f7d1c10dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:25.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c10a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:25.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c10a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:25.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c113510 con 0x7f7d1c1035b0 2024-07-26T10:59:25.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c113390 con 0x7f7d1c109840 2024-07-26T10:59:25.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.713+0000 7f7d23a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c113690 con 0x7f7d1c108ef0 2024-07-26T10:59:25.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c10a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:25.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d21000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1c1035b0 0x7f7d1c10dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:25.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c10a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:59004/0 (socket says 172.21.15.144:59004) 2024-07-26T10:59:25.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 -- 172.21.15.144:0/1199875532 learned_addr learned my addr 172.21.15.144:0/1199875532 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d21a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c10a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 -- 172.21.15.144:0/1199875532 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 msgr2=0x7f7d1c10a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c10a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 -- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1c1035b0 msgr2=0x7f7d1c10dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1c1035b0 0x7f7d1c10dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d1be00640 1 -- 172.21.15.144:0/1199875532 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c1ac610 con 0x7f7d1c108ef0 2024-07-26T10:59:25.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.714+0000 7f7d21a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c10a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.715+0000 7f7d1be00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c10a040 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800e9c0 tx=0x7f7d0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.715+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0800ccc0 con 0x7f7d1c108ef0 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.715+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d080108e0 con 0x7f7d1c108ef0 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.715+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1c1ac840 con 0x7f7d1c108ef0 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.715+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d080187e0 con 0x7f7d1c108ef0 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.715+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1c1acdd0 con 0x7f7d1c108ef0 2024-07-26T10:59:25.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.717+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d1c1039c0 con 0x7f7d1c108ef0 2024-07-26T10:59:25.721 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.719+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f7d08010430 con 0x7f7d1c108ef0 2024-07-26T10:59:25.722 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.720+0000 7f7d19600640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7d040b3d30 0x7f7d040b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:25.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.721+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f7d080e45a0 con 0x7f7d1c108ef0 2024-07-26T10:59:25.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.721+0000 7f7d21000640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7d040b3d30 0x7f7d040b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:25.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.722+0000 7f7d21000640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7d040b3d30 0x7f7d040b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c10dab0 tx=0x7f7d10005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:25.726 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:25.724+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7d080ab5a0 con 0x7f7d1c108ef0 2024-07-26T10:59:26.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.023+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f7d1c1ad150 con 0x7f7d1c108ef0 2024-07-26T10:59:26.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.032+0000 7f7d19600640 1 -- 172.21.15.144:0/1199875532 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v33) ==== 170+0+59 (secure 0 0 0) 0x7f7d080aaea0 con 0x7f7d1c108ef0 2024-07-26T10:59:26.034 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2024-07-26T10:59:26.034 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCOgaNmO46NARAAYtgcQCtEt/4u+JYLqD6euQ== 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7d040b3d30 msgr2=0x7f7d040b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7d040b3d30 0x7f7d040b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c10dab0 tx=0x7f7d10005f90 comp rx=0 tx=0).stop 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 msgr2=0x7f7d1c10a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c10a040 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800e9c0 tx=0x7f7d0800ee80 comp rx=0 tx=0).stop 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 shutdown_connections 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7d040b3d30 0x7f7d040b6240 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:26.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d1c109840 0x7f7d1c10a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:26.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d1c108ef0 0x7f7d1c10a040 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:26.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 --2- 172.21.15.144:0/1199875532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1c1035b0 0x7f7d1c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:26.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.036+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 >> 172.21.15.144:0/1199875532 conn(0x7f7d1c0ff000 msgr2=0x7f7d1c10e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:26.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.037+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 shutdown_connections 2024-07-26T10:59:26.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:26.037+0000 7f7d23a00640 1 -- 172.21.15.144:0/1199875532 wait complete. 2024-07-26T10:59:26.426 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:26 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1199875532' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T10:59:26.426 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:26 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T10:59:26.426 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:26 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T10:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:26 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1199875532' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T10:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:26 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T10:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:26 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T10:59:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:26 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1199875532' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T10:59:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:26 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T10:59:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:26 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T10:59:26.852 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T10:59:26.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-26T10:59:26.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T10:59:26.964 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph config log 1 --format=json 2024-07-26T10:59:27.464 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:27.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:27 smithi144 ceph-mon[32129]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-07-26T10:59:27.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:27.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:27.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:27 smithi160 ceph-mon[36715]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-07-26T10:59:27.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:27.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:27.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:27 smithi183 ceph-mon[37273]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-07-26T10:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T10:59:29.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 -- 172.21.15.144:0/4195525517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410b860 msgr2=0x7ff674112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 --2- 172.21.15.144:0/4195525517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410b860 0x7ff674112090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009ca0 tx=0x7ff66c02a8b0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 -- 172.21.15.144:0/4195525517 shutdown_connections 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 --2- 172.21.15.144:0/4195525517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410b860 0x7ff674112090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 --2- 172.21.15.144:0/4195525517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff67410af10 0x7ff67410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 --2- 172.21.15.144:0/4195525517 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6741055d0 0x7ff6741059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 -- 172.21.15.144:0/4195525517 >> 172.21.15.144:0/4195525517 conn(0x7ff674101040 msgr2=0x7ff674103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 -- 172.21.15.144:0/4195525517 shutdown_connections 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.620+0000 7ff679000640 1 -- 172.21.15.144:0/4195525517 wait complete. 2024-07-26T10:59:29.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.621+0000 7ff679000640 1 Processor -- start 2024-07-26T10:59:29.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 -- start start 2024-07-26T10:59:29.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 0x7ff67419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410af10 0x7ff67419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff67410b860 0x7ff6741a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff674115560 con 0x7ff6741055d0 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff6741153e0 con 0x7ff67410b860 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff679000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff6741156e0 con 0x7ff67410af10 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.622+0000 7ff673400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 0x7ff67419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 0x7ff67419beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59556/0 (socket says 172.21.15.144:59556) 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 -- 172.21.15.144:0/1682451202 learned_addr learned my addr 172.21.15.144:0/1682451202 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673e00640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff67410b860 0x7ff6741a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff672a00640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410af10 0x7ff67419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 -- 172.21.15.144:0/1682451202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410af10 msgr2=0x7ff67419c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410af10 0x7ff67419c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 -- 172.21.15.144:0/1682451202 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff67410b860 msgr2=0x7ff6741a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:29.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff67410b860 0x7ff6741a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 -- 172.21.15.144:0/1682451202 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6741a1e20 con 0x7ff6741055d0 2024-07-26T10:59:29.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.623+0000 7ff673400640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 0x7ff67419beb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff66400e9f0 tx=0x7ff66400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:29.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.624+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66400cd80 con 0x7ff6741055d0 2024-07-26T10:59:29.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.624+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6741a2110 con 0x7ff6741055d0 2024-07-26T10:59:29.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.624+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6640108e0 con 0x7ff6741055d0 2024-07-26T10:59:29.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.624+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff664018980 con 0x7ff6741055d0 2024-07-26T10:59:29.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.624+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6741a8870 con 0x7ff6741055d0 2024-07-26T10:59:29.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.625+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6741059e0 con 0x7ff6741055d0 2024-07-26T10:59:29.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.628+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7ff664010430 con 0x7ff6741055d0 2024-07-26T10:59:29.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.629+0000 7ff66be00640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff6580b3cb0 0x7ff6580b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:29.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.629+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7ff6640e4ab0 con 0x7ff6741055d0 2024-07-26T10:59:29.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.629+0000 7ff672a00640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff6580b3cb0 0x7ff6580b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:29.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.630+0000 7ff672a00640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff6580b3cb0 0x7ff6580b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff674104890 tx=0x7ff660005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:29.633 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.632+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff6640ab460 con 0x7ff6741055d0 2024-07-26T10:59:29.839 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.836+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7ff67419cfe0 con 0x7ff6741055d0 2024-07-26T10:59:29.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.837+0000 7ff66be00640 1 -- 172.21.15.144:0/1682451202 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v54) ==== 86+0+133 (secure 0 0 0) 0x7ff6640af9b0 con 0x7ff6741055d0 2024-07-26T10:59:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff6580b3cb0 msgr2=0x7ff6580b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff6580b3cb0 0x7ff6580b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff674104890 tx=0x7ff660005f90 comp rx=0 tx=0).stop 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 msgr2=0x7ff67419beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 0x7ff67419beb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff66400e9f0 tx=0x7ff66400eeb0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 shutdown_connections 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff6580b3cb0 0x7ff6580b61c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff67410b860 0x7ff6741a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff67410af10 0x7ff67419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 --2- 172.21.15.144:0/1682451202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6741055d0 0x7ff67419beb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:29.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 >> 172.21.15.144:0/1682451202 conn(0x7ff674101040 msgr2=0x7ff6741094f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:29.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 shutdown_connections 2024-07-26T10:59:29.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:29.840+0000 7ff679000640 1 -- 172.21.15.144:0/1682451202 wait complete. 2024-07-26T10:59:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:30 smithi144 ceph-mon[32129]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T10:59:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:30 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1682451202' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T10:59:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:30 smithi183 ceph-mon[37273]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T10:59:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:30 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1682451202' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T10:59:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:30 smithi160 ceph-mon[36715]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T10:59:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:30 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1682451202' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T10:59:30.652 INFO:teuthology.orchestra.run.smithi144.stdout:[{"version":54,"timestamp":"2024-07-26T10:57:05.472452+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-26T10:59:30.652 INFO:tasks.ceph_manager:config epoch is 54 2024-07-26T10:59:30.652 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T10:59:30.652 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-26T10:59:30.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph mgr dump --format=json 2024-07-26T10:59:31.116 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:32 smithi183 ceph-mon[37273]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T10:59:32.440 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:32 smithi144 ceph-mon[32129]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T10:59:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:32 smithi160 ceph-mon[36715]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T10:59:33.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 -- 172.21.15.144:0/3161095153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 msgr2=0x7fa0241120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 --2- 172.21.15.144:0/3161095153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 0x7fa0241120b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009ca0 tx=0x7fa01c02a8c0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 -- 172.21.15.144:0/3161095153 shutdown_connections 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 --2- 172.21.15.144:0/3161095153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 0x7fa0241120b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 --2- 172.21.15.144:0/3161095153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa02410af30 0x7fa02410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 --2- 172.21.15.144:0/3161095153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0241055f0 0x7fa024105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 -- 172.21.15.144:0/3161095153 >> 172.21.15.144:0/3161095153 conn(0x7fa024101060 msgr2=0x7fa0241034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:33.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 -- 172.21.15.144:0/3161095153 shutdown_connections 2024-07-26T10:59:33.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.313+0000 7fa028e00640 1 -- 172.21.15.144:0/3161095153 wait complete. 2024-07-26T10:59:33.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.314+0000 7fa028e00640 1 Processor -- start 2024-07-26T10:59:33.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.314+0000 7fa028e00640 1 -- start start 2024-07-26T10:59:33.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa028e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0241055f0 0x7fa02419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:33.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa028e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 0x7fa02419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:33.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa028e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 0x7fa0241a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:33.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa028e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa024115530 con 0x7fa02410af30 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa028e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241153b0 con 0x7fa0241055f0 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa028e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241156b0 con 0x7fa02410b880 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa01be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 0x7fa02419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa01be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 0x7fa02419c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38412/0 (socket says 172.21.15.144:38412) 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa023400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0241055f0 0x7fa02419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa01be00640 1 -- 172.21.15.144:0/1144900873 learned_addr learned my addr 172.21.15.144:0/1144900873 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.315+0000 7fa023e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 0x7fa0241a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa023400640 1 -- 172.21.15.144:0/1144900873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 msgr2=0x7fa0241a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa023400640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 0x7fa0241a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa023400640 1 -- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 msgr2=0x7fa02419c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa023400640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 0x7fa02419c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa023400640 1 -- 172.21.15.144:0/1144900873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0241a1e30 con 0x7fa0241055f0 2024-07-26T10:59:33.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa01be00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 0x7fa02419c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:33.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.316+0000 7fa023400640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0241055f0 0x7fa02419bec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa01400e9f0 tx=0x7fa01400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:33.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.317+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01400ccc0 con 0x7fa0241055f0 2024-07-26T10:59:33.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.317+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0241a2120 con 0x7fa0241055f0 2024-07-26T10:59:33.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.317+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0140108e0 con 0x7fa0241055f0 2024-07-26T10:59:33.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.317+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014018880 con 0x7fa0241055f0 2024-07-26T10:59:33.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.317+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a8880 con 0x7fa0241055f0 2024-07-26T10:59:33.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.318+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa024105a00 con 0x7fa0241055f0 2024-07-26T10:59:33.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.321+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fa014010430 con 0x7fa0241055f0 2024-07-26T10:59:33.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.322+0000 7fa020c00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0080b3cb0 0x7fa0080b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:33.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.322+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa0140e4a80 con 0x7fa0241055f0 2024-07-26T10:59:33.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.323+0000 7fa01be00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0080b3cb0 0x7fa0080b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:33.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.323+0000 7fa01be00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0080b3cb0 0x7fa0080b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa024104760 tx=0x7fa00c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:33.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.324+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa0140ab430 con 0x7fa0241055f0 2024-07-26T10:59:33.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.581+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa02402e0c0 con 0x7fa0241055f0 2024-07-26T10:59:33.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.588+0000 7fa020c00640 1 -- 172.21.15.144:0/1144900873 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+291043 (secure 0 0 0) 0x7fa0140af980 con 0x7fa0241055f0 2024-07-26T10:59:33.596 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.593+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0080b3cb0 msgr2=0x7fa0080b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.593+0000 7fa028e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0080b3cb0 0x7fa0080b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa024104760 tx=0x7fa00c005f90 comp rx=0 tx=0).stop 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0241055f0 msgr2=0x7fa02419bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0241055f0 0x7fa02419bec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa01400e9f0 tx=0x7fa01400eeb0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 shutdown_connections 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa0080b3cb0 0x7fa0080b61c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa02410b880 0x7fa0241a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa02410af30 0x7fa02419c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 --2- 172.21.15.144:0/1144900873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0241055f0 0x7fa02419bec0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 >> 172.21.15.144:0/1144900873 conn(0x7fa024101060 msgr2=0x7fa02410a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:33.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 shutdown_connections 2024-07-26T10:59:33.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:33.594+0000 7fa028e00640 1 -- 172.21.15.144:0/1144900873 wait complete. 2024-07-26T10:59:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:34 smithi144 ceph-mon[32129]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:34 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1144900873' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T10:59:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:34 smithi183 ceph-mon[37273]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:34 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1144900873' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T10:59:34.398 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":21,"flags":0,"active_gid":24109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2154879590},{"type":"v1","addr":"172.21.15.144:6801","nonce":2154879590}]},"active_addr":"172.21.15.144:6801/2154879590","active_change":"2024-07-26T10:49:42.802679+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24142,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24146,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.144:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":523873921}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":746580902}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3789413478}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1533168795}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1331137509}]}]} 2024-07-26T10:59:34.406 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-26T10:59:34.406 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-26T10:59:34.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:59:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:34 smithi160 ceph-mon[36715]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:34 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1144900873' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T10:59:34.862 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:36 smithi144 ceph-mon[32129]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:36 smithi183 ceph-mon[37273]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:36 smithi160 ceph-mon[36715]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 -- 172.21.15.144:0/2669210506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5324109670 msgr2=0x7f532410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 --2- 172.21.15.144:0/2669210506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5324109670 0x7f532410fea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f531c009860 tx=0x7f531c02a340 comp rx=0 tx=0).stop 2024-07-26T10:59:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 -- 172.21.15.144:0/2669210506 shutdown_connections 2024-07-26T10:59:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 --2- 172.21.15.144:0/2669210506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5324109670 0x7f532410fea0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 --2- 172.21.15.144:0/2669210506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 0x7f5324109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 --2- 172.21.15.144:0/2669210506 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53241033e0 0x7f53241037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 -- 172.21.15.144:0/2669210506 >> 172.21.15.144:0/2669210506 conn(0x7f53240fee70 msgr2=0x7f53241012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 -- 172.21.15.144:0/2669210506 shutdown_connections 2024-07-26T10:59:37.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.124+0000 7f532a600640 1 -- 172.21.15.144:0/2669210506 wait complete. 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.125+0000 7f532a600640 1 Processor -- start 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.125+0000 7f532a600640 1 -- start start 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f532a600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 0x7f532419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f532a600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 0x7f532419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f532a600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5324109670 0x7f53241a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f532a600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5324113340 con 0x7f53241033e0 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f532a600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f53241131c0 con 0x7f5324109670 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f532a600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53241134c0 con 0x7f5324108d20 2024-07-26T10:59:37.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f5329200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 0x7f532419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:37.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f5329200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 0x7f532419bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38416/0 (socket says 172.21.15.144:38416) 2024-07-26T10:59:37.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f5329200640 1 -- 172.21.15.144:0/2417383063 learned_addr learned my addr 172.21.15.144:0/2417383063 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:37.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f5323e00640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 0x7f532419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:37.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.126+0000 7f5329c00640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5324109670 0x7f53241a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5323e00640 1 -- 172.21.15.144:0/2417383063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5324109670 msgr2=0x7f53241a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5323e00640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5324109670 0x7f53241a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5323e00640 1 -- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 msgr2=0x7f532419bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5323e00640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 0x7f532419bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5323e00640 1 -- 172.21.15.144:0/2417383063 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53241a1ec0 con 0x7f5324108d20 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5329200640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 0x7f532419bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5323e00640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 0x7f532419c4f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f532406b3c0 tx=0x7f530c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c00cc40 con 0x7f5324108d20 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f530c00cde0 con 0x7f5324108d20 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.127+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c004a00 con 0x7f5324108d20 2024-07-26T10:59:37.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.128+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241a21b0 con 0x7f5324108d20 2024-07-26T10:59:37.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.128+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241a8910 con 0x7f5324108d20 2024-07-26T10:59:37.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.129+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53241037f0 con 0x7f5324108d20 2024-07-26T10:59:37.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.132+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f530c004ba0 con 0x7f5324108d20 2024-07-26T10:59:37.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.133+0000 7f5321600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53080b3d30 0x7f53080b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:37.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.134+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f530c0e4240 con 0x7f5324108d20 2024-07-26T10:59:37.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.134+0000 7f5329200640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53080b3d30 0x7f53080b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:37.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.134+0000 7f5329200640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53080b3d30 0x7f53080b6240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f531400e860 tx=0x7f5314010040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:37.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.135+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f530c0ab2d0 con 0x7f5324108d20 2024-07-26T10:59:37.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.338+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f532419d080 con 0x7f5324108d20 2024-07-26T10:59:37.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.341+0000 7f5321600640 1 -- 172.21.15.144:0/2417383063 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7f532419d080 con 0x7f5324108d20 2024-07-26T10:59:37.343 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:37.344 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:59:37.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53080b3d30 msgr2=0x7f53080b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:37.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53080b3d30 0x7f53080b6240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f531400e860 tx=0x7f5314010040 comp rx=0 tx=0).stop 2024-07-26T10:59:37.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 msgr2=0x7f532419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:37.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 0x7f532419c4f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f532406b3c0 tx=0x7f530c00ee50 comp rx=0 tx=0).stop 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 shutdown_connections 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53080b3d30 0x7f53080b6240 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5324109670 0x7f53241a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5324108d20 0x7f532419c4f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.343+0000 7f532a600640 1 --2- 172.21.15.144:0/2417383063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53241033e0 0x7f532419bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.344+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 >> 172.21.15.144:0/2417383063 conn(0x7f53240fee70 msgr2=0x7f5324108360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.344+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 shutdown_connections 2024-07-26T10:59:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:37.344+0000 7f532a600640 1 -- 172.21.15.144:0/2417383063 wait complete. 2024-07-26T10:59:38.121 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-26T10:59:38.122 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T10:59:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:38 smithi183 ceph-mon[37273]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:38 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2417383063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:38 smithi144 ceph-mon[32129]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:38 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2417383063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:38.548 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:38 smithi160 ceph-mon[36715]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:38 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2417383063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:40.160 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:40 smithi144 ceph-mon[32129]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:40 smithi183 ceph-mon[37273]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:40 smithi160 ceph-mon[36715]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:40.722 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1586097432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410af10 msgr2=0x7ff2f410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:40.722 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1586097432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410af10 0x7ff2f410b320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009900 tx=0x7ff2e002a3f0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.722 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1586097432 shutdown_connections 2024-07-26T10:59:40.722 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1586097432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f410b860 0x7ff2f4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1586097432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410af10 0x7ff2f410b320 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1586097432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f41055d0 0x7ff2f41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1586097432 >> 172.21.15.144:0/1586097432 conn(0x7ff2f4101040 msgr2=0x7ff2f4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1586097432 shutdown_connections 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.720+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1586097432 wait complete. 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.721+0000 7ff2f9e00640 1 Processor -- start 2024-07-26T10:59:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 -- start start 2024-07-26T10:59:40.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 0x7ff2f419bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:40.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f410af10 0x7ff2f419c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410b860 0x7ff2f41a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f4115520 con 0x7ff2f410af10 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41153a0 con 0x7ff2f410b860 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41156a0 con 0x7ff2f41055d0 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f8a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 0x7ff2f419bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f410af10 0x7ff2f419c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:40.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.722+0000 7ff2f9400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410b860 0x7ff2f41a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:40.726 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 0x7ff2f419bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48204/0 (socket says 172.21.15.144:48204) 2024-07-26T10:59:40.726 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 -- 172.21.15.144:0/1252029320 learned_addr learned my addr 172.21.15.144:0/1252029320 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 -- 172.21.15.144:0/1252029320 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410b860 msgr2=0x7ff2f41a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410b860 0x7ff2f41a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 -- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f410af10 msgr2=0x7ff2f419c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f410af10 0x7ff2f419c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 -- 172.21.15.144:0/1252029320 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f41a1f20 con 0x7ff2f41055d0 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.723+0000 7ff2f8a00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 0x7ff2f419bfb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e9f0 tx=0x7ff2e800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.724+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800ccc0 con 0x7ff2f41055d0 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.724+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e80108e0 con 0x7ff2f41055d0 2024-07-26T10:59:40.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.724+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f41a2210 con 0x7ff2f41055d0 2024-07-26T10:59:40.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.724+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8018880 con 0x7ff2f41055d0 2024-07-26T10:59:40.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.724+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f41a8970 con 0x7ff2f41055d0 2024-07-26T10:59:40.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.725+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f41059e0 con 0x7ff2f41055d0 2024-07-26T10:59:40.729 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.728+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7ff2e8010430 con 0x7ff2f41055d0 2024-07-26T10:59:40.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.729+0000 7ff2f1600640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff2dc0b3db0 0x7ff2dc0b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:40.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.730+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7ff2e80e4ad0 con 0x7ff2f41055d0 2024-07-26T10:59:40.731 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.730+0000 7ff2f3e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff2dc0b3db0 0x7ff2dc0b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:40.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.730+0000 7ff2f3e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff2dc0b3db0 0x7ff2dc0b62c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2e002a9e0 tx=0x7ff2e0005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:40.732 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.731+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff2e80abad0 con 0x7ff2f41055d0 2024-07-26T10:59:40.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.928+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2f402e0c0 con 0x7ff2f41055d0 2024-07-26T10:59:40.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.930+0000 7ff2f1600640 1 -- 172.21.15.144:0/1252029320 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7ff2e80ab3d0 con 0x7ff2f41055d0 2024-07-26T10:59:40.932 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T10:59:40.933 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T10:59:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff2dc0b3db0 msgr2=0x7ff2dc0b62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff2dc0b3db0 0x7ff2dc0b62c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2e002a9e0 tx=0x7ff2e0005a20 comp rx=0 tx=0).stop 2024-07-26T10:59:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 msgr2=0x7ff2f419bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:40.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 0x7ff2f419bfb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e9f0 tx=0x7ff2e800edf0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 shutdown_connections 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff2dc0b3db0 0x7ff2dc0b62c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff2f410b860 0x7ff2f41a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f410af10 0x7ff2f419c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 --2- 172.21.15.144:0/1252029320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2f41055d0 0x7ff2f419bfb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 >> 172.21.15.144:0/1252029320 conn(0x7ff2f4101040 msgr2=0x7ff2f4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 shutdown_connections 2024-07-26T10:59:40.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:40.933+0000 7ff2f9e00640 1 -- 172.21.15.144:0/1252029320 wait complete. 2024-07-26T10:59:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:41 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1252029320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:41 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1252029320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:41 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1252029320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T10:59:41.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-26T10:59:41.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-26T10:59:41.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-26T10:59:41.765 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-26T10:59:41.765 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-26T10:59:41.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-26T10:59:41.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-26T10:59:41.767 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-26T10:59:41.767 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-26T10:59:41.767 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-26T10:59:41.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-26T10:59:41.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-26T10:59:42.225 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:42 smithi144 ceph-mon[32129]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:42 smithi183 ceph-mon[37273]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:42 smithi160 ceph-mon[36715]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:42.775 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:42.894 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:44.356 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:44 smithi183 ceph-mon[37273]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:44.455 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:44 smithi144 ceph-mon[32129]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:44 smithi160 ceph-mon[36715]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:44.929 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:44.941 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:44.944 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:45.062 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:45.063 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:45.064 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:45.367 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:45.371 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:45.373 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T10:59:46.319 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:46 smithi144 ceph-mon[32129]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:46 smithi183 ceph-mon[37273]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:46 smithi160 ceph-mon[36715]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:48 smithi183 ceph-mon[37273]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:48.458 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:48 smithi144 ceph-mon[32129]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:48 smithi160 ceph-mon[36715]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:50 smithi144 ceph-mon[32129]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:50 smithi160 ceph-mon[36715]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:50 smithi183 ceph-mon[37273]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:51 smithi144 ceph-mon[32129]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:51 smithi160 ceph-mon[36715]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:51 smithi183 ceph-mon[37273]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:54 smithi144 ceph-mon[32129]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:54 smithi183 ceph-mon[37273]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.491+0000 7f8102c00640 1 -- 172.21.15.144:0/1204850300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 msgr2=0x7f80fc111f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.491+0000 7f8102c00640 1 --2- 172.21.15.144:0/1204850300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 0x7f80fc111f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009be0 tx=0x7f80f402a6d0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.491+0000 7f8102c00640 1 -- 172.21.15.144:0/1204850300 shutdown_connections 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.491+0000 7f8102c00640 1 --2- 172.21.15.144:0/1204850300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 0x7f80fc111f90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.491+0000 7f8102c00640 1 --2- 172.21.15.144:0/1204850300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80fc075ad0 0x7f80fc10f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.492+0000 7f8102c00640 1 --2- 172.21.15.144:0/1204850300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 0x7f80fc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.492+0000 7f8102c00640 1 -- 172.21.15.144:0/1204850300 >> 172.21.15.144:0/1204850300 conn(0x7f80fc0ff000 msgr2=0x7f80fc101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.492+0000 7f8102c00640 1 -- 172.21.15.144:0/1204850300 shutdown_connections 2024-07-26T10:59:54.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.492+0000 7f8102c00640 1 -- 172.21.15.144:0/1204850300 wait complete. 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.493+0000 7f8102c00640 1 Processor -- start 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.493+0000 7f8102c00640 1 -- start start 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.493+0000 7f8102c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 0x7f80fc19c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.493+0000 7f8102c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80fc075ad0 0x7f80fc19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f8102c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 0x7f80fc1a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f8102c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1156e0 con 0x7f80fc075ad0 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f8102c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc115560 con 0x7f80fc075180 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f8102c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc115860 con 0x7f80fc10fb70 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fb400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80fc075ad0 0x7f80fc19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 0x7f80fc19c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:54.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 0x7f80fc19c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54196/0 (socket says 172.21.15.144:54196) 2024-07-26T10:59:54.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 -- 172.21.15.144:0/1180703548 learned_addr learned my addr 172.21.15.144:0/1180703548 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:54.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f8100c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 0x7f80fc1a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 -- 172.21.15.144:0/1180703548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 msgr2=0x7f80fc1a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80fc10fb70 0x7f80fc1a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 -- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80fc075ad0 msgr2=0x7f80fc19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80fc075ad0 0x7f80fc19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fbe00640 1 -- 172.21.15.144:0/1180703548 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc1a1fc0 con 0x7f80fc075180 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.494+0000 7f80fb400640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80fc075ad0 0x7f80fc19c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.495+0000 7f80fbe00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 0x7f80fc19c050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e9f0 tx=0x7f80ec00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.495+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec00cd80 con 0x7f80fc075180 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.495+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc1a22b0 con 0x7f80fc075180 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.495+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80ec0054d0 con 0x7f80fc075180 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.495+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec010690 con 0x7f80fc075180 2024-07-26T10:59:54.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.495+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc1a8a10 con 0x7f80fc075180 2024-07-26T10:59:54.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.497+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80fc10a0e0 con 0x7f80fc075180 2024-07-26T10:59:54.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.499+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f80ec004e30 con 0x7f80fc075180 2024-07-26T10:59:54.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.501+0000 7f80f8c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f80e00b3cb0 0x7f80e00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:54.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.501+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f80ec0e4af0 con 0x7f80fc075180 2024-07-26T10:59:54.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.501+0000 7f80fb400640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f80e00b3cb0 0x7f80e00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:54.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.501+0000 7f80f8c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] conn(0x7f80e00bf2d0 0x7f80e00c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:54.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.501+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 --> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80ec015e80 con 0x7f80e00bf2d0 2024-07-26T10:59:54.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.501+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f80ec0e4ea0 con 0x7f80fc075180 2024-07-26T10:59:54.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.502+0000 7f8100c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] conn(0x7f80e00bf2d0 0x7f80e00c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:54.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.502+0000 7f80fb400640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f80e00b3cb0 0x7f80e00b61c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f80e80096c0 tx=0x7f80e8009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:54.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.502+0000 7f8100c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] conn(0x7f80e00bf2d0 0x7f80e00c17c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:54.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.503+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== osd.1 v2:172.21.15.160:6800/3146107361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80ec015e80 con 0x7f80e00bf2d0 2024-07-26T10:59:54.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.519+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 --> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80fc076690 con 0x7f80e00bf2d0 2024-07-26T10:59:54.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.519+0000 7f80f8c00640 1 -- 172.21.15.144:0/1180703548 <== osd.1 v2:172.21.15.160:6800/3146107361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f80fc076690 con 0x7f80e00bf2d0 2024-07-26T10:59:54.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] conn(0x7f80e00bf2d0 msgr2=0x7f80e00c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:54.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] conn(0x7f80e00bf2d0 0x7f80e00c17c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f80e00b3cb0 msgr2=0x7f80e00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:54.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f80e00b3cb0 0x7f80e00b61c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f80e80096c0 tx=0x7f80e8009260 comp rx=0 tx=0).stop 2024-07-26T10:59:54.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 msgr2=0x7f80fc19c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:54.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 --2- 172.21.15.144:0/1180703548 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f80fc075180 0x7f80fc19c050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e9f0 tx=0x7f80ec00edf0 comp rx=0 tx=0).stop 2024-07-26T10:59:54.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8100c00640 1 -- 172.21.15.144:0/1180703548 reap_dead start 2024-07-26T10:59:54.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 shutdown_connections 2024-07-26T10:59:54.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 >> 172.21.15.144:0/1180703548 conn(0x7f80fc0ff000 msgr2=0x7f80fc109680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:54.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.520+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 shutdown_connections 2024-07-26T10:59:54.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:54.521+0000 7f8102c00640 1 -- 172.21.15.144:0/1180703548 wait complete. 2024-07-26T10:59:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:54 smithi160 ceph-mon[36715]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T10:59:56.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 -- 172.21.15.144:0/3931432630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410b880 msgr2=0x7f39b41120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 --2- 172.21.15.144:0/3931432630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410b880 0x7f39b41120b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009be0 tx=0x7f39ac02a6c0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 -- 172.21.15.144:0/3931432630 shutdown_connections 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 --2- 172.21.15.144:0/3931432630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410b880 0x7f39b41120b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 --2- 172.21.15.144:0/3931432630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b410af30 0x7f39b410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 --2- 172.21.15.144:0/3931432630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39b41055f0 0x7f39b4105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 -- 172.21.15.144:0/3931432630 >> 172.21.15.144:0/3931432630 conn(0x7f39b4101040 msgr2=0x7f39b4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 -- 172.21.15.144:0/3931432630 shutdown_connections 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.991+0000 7f39b9800640 1 -- 172.21.15.144:0/3931432630 wait complete. 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.992+0000 7f39b9800640 1 Processor -- start 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 -- start start 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b41055f0 0x7f39b419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410af30 0x7f39b419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39b410b880 0x7f39b41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4115530 con 0x7f39b410b880 2024-07-26T10:59:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41153b0 con 0x7f39b410af30 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b9800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41156b0 con 0x7f39b41055f0 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b41055f0 0x7f39b419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39b410b880 0x7f39b41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b3400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410af30 0x7f39b419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b41055f0 0x7f39b419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:50096/0 (socket says 172.21.15.144:50096) 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.993+0000 7f39b3e00640 1 -- 172.21.15.144:0/3367267707 learned_addr learned my addr 172.21.15.144:0/3367267707 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b3e00640 1 -- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410af30 msgr2=0x7f39b419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b3e00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39b410af30 0x7f39b419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b3e00640 1 -- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39b410b880 msgr2=0x7f39b41a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b3e00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39b410b880 0x7f39b41a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b3e00640 1 -- 172.21.15.144:0/3367267707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b41a1e10 con 0x7f39b41055f0 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b3e00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b41055f0 0x7f39b419bea0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e9f0 tx=0x7f39a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.994+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a400cd80 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.995+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b41a2100 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.995+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a40108e0 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.995+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4018980 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.995+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a8860 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.995+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f39b4063580 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:55.998+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f39a4010430 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.000+0000 7f39b0c00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f39980b3cb0 0x7f39980b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.000+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f39a40e4d20 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.000+0000 7f39b3400640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f39980b3cb0 0x7f39980b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.000+0000 7f39b0c00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] conn(0x7f39980bf2d0 0x7f39980c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.000+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 --> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39980c1e90 con 0x7f39980bf2d0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.000+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f39a40e50d0 con 0x7f39b41055f0 2024-07-26T10:59:56.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.001+0000 7f39b8e00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] conn(0x7f39980bf2d0 0x7f39980c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.001+0000 7f39b3400640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f39980b3cb0 0x7f39980b61c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f39b410a870 tx=0x7f39a0009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.001+0000 7f39b8e00640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] conn(0x7f39980bf2d0 0x7f39980c17c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.002+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== osd.7 v2:172.21.15.160:6816/602437452 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39980c1e90 con 0x7f39980bf2d0 2024-07-26T10:59:56.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.018+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 --> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f39b40639b0 con 0x7f39980bf2d0 2024-07-26T10:59:56.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.019+0000 7f39b0c00640 1 -- 172.21.15.144:0/3367267707 <== osd.7 v2:172.21.15.160:6816/602437452 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f39b40639b0 con 0x7f39980bf2d0 2024-07-26T10:59:56.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.019+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] conn(0x7f39980bf2d0 msgr2=0x7f39980c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.019+0000 7f39b9800640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] conn(0x7f39980bf2d0 0x7f39980c17c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f39980b3cb0 msgr2=0x7f39980b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f39980b3cb0 0x7f39980b61c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f39b410a870 tx=0x7f39a0009260 comp rx=0 tx=0).stop 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b41055f0 msgr2=0x7f39b419bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 --2- 172.21.15.144:0/3367267707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39b41055f0 0x7f39b419bea0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e9f0 tx=0x7f39a400eeb0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b8e00640 1 -- 172.21.15.144:0/3367267707 reap_dead start 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 shutdown_connections 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 >> 172.21.15.144:0/3367267707 conn(0x7f39b4101040 msgr2=0x7f39b4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 shutdown_connections 2024-07-26T10:59:56.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.020+0000 7f39b9800640 1 -- 172.21.15.144:0/3367267707 wait complete. 2024-07-26T10:59:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:56 smithi183 ceph-mon[37273]: pgmap v384: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:56.410 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:56 smithi144 ceph-mon[32129]: pgmap v384: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:56 smithi160 ceph-mon[36715]: pgmap v384: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.612+0000 7f1187e00640 1 -- 172.21.15.144:0/1458586532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188105ad0 msgr2=0x7f1188105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.612+0000 7f1187e00640 1 --2- 172.21.15.144:0/1458586532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188105ad0 0x7f1188105ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1170009900 tx=0x7f117002a3f0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 -- 172.21.15.144:0/1458586532 shutdown_connections 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 --2- 172.21.15.144:0/1458586532 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188106420 0x7f118810feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 --2- 172.21.15.144:0/1458586532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188105ad0 0x7f1188105ee0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 --2- 172.21.15.144:0/1458586532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1188101880 0x7f1188101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 -- 172.21.15.144:0/1458586532 >> 172.21.15.144:0/1458586532 conn(0x7f11880774d0 msgr2=0x7f1188077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 -- 172.21.15.144:0/1458586532 shutdown_connections 2024-07-26T10:59:56.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.613+0000 7f1187e00640 1 -- 172.21.15.144:0/1458586532 wait complete. 2024-07-26T10:59:56.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.614+0000 7f1187e00640 1 Processor -- start 2024-07-26T10:59:56.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.614+0000 7f1187e00640 1 -- start start 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188101880 0x7f118819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188105ad0 0x7f118819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1188106420 0x7f11881a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f11881132b0 con 0x7f1188101880 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1188113130 con 0x7f1188105ad0 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1188113430 con 0x7f1188106420 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188105ad0 0x7f118819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188101880 0x7f118819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188105ad0 0x7f118819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54224/0 (socket says 172.21.15.144:54224) 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 -- 172.21.15.144:0/2564911907 learned_addr learned my addr 172.21.15.144:0/2564911907 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:56.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1187400640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1188106420 0x7f11881a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 -- 172.21.15.144:0/2564911907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1188106420 msgr2=0x7f11881a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1188106420 0x7f11881a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 -- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188101880 msgr2=0x7f118819bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188101880 0x7f118819bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.615+0000 7f1186000640 1 -- 172.21.15.144:0/2564911907 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11881a1e90 con 0x7f1188105ad0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.616+0000 7f1186a00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1188101880 0x7f118819bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.616+0000 7f1186000640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188105ad0 0x7f118819c4c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1170005940 tx=0x7f1170004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.616+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117003f030 con 0x7f1188105ad0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.616+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11881a2120 con 0x7f1188105ad0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.616+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1170004440 con 0x7f1188105ad0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.616+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170035070 con 0x7f1188105ad0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.617+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11881a88e0 con 0x7f1188105ad0 2024-07-26T10:59:56.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.617+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1188107640 con 0x7f1188105ad0 2024-07-26T10:59:56.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.620+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f1170042030 con 0x7f1188105ad0 2024-07-26T10:59:56.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.622+0000 7f118ca00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f116c0b3cb0 0x7f116c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.622+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f1170104c10 con 0x7f1188105ad0 2024-07-26T10:59:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.622+0000 7f1186a00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f116c0b3cb0 0x7f116c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.622+0000 7f118ca00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] conn(0x7f116c0bf2d0 0x7f116c0c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.622+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 --> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f116c0c1e90 con 0x7f116c0bf2d0 2024-07-26T10:59:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.622+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f1170104fc0 con 0x7f1188105ad0 2024-07-26T10:59:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.623+0000 7f1187400640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] conn(0x7f116c0bf2d0 0x7f116c0c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.623+0000 7f1186a00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f116c0b3cb0 0x7f116c0b61c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f11800099a0 tx=0x7f1180009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.624+0000 7f1187400640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] conn(0x7f116c0bf2d0 0x7f116c0c17c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.625+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== osd.3 v2:172.21.15.144:6810/1981116547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f116c0c1e90 con 0x7f116c0bf2d0 2024-07-26T10:59:56.645 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.643+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 --> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f118806a5c0 con 0x7f116c0bf2d0 2024-07-26T10:59:56.646 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.644+0000 7f118ca00640 1 -- 172.21.15.144:0/2564911907 <== osd.3 v2:172.21.15.144:6810/1981116547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f118806a5c0 con 0x7f116c0bf2d0 2024-07-26T10:59:56.646 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.644+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] conn(0x7f116c0bf2d0 msgr2=0x7f116c0c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.646 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.644+0000 7f1187e00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] conn(0x7f116c0bf2d0 0x7f116c0c17c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f116c0b3cb0 msgr2=0x7f116c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f116c0b3cb0 0x7f116c0b61c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f11800099a0 tx=0x7f1180009500 comp rx=0 tx=0).stop 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188105ad0 msgr2=0x7f118819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 --2- 172.21.15.144:0/2564911907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1188105ad0 0x7f118819c4c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1170005940 tx=0x7f1170004040 comp rx=0 tx=0).stop 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187400640 1 -- 172.21.15.144:0/2564911907 reap_dead start 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 shutdown_connections 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 >> 172.21.15.144:0/2564911907 conn(0x7f11880774d0 msgr2=0x7f1188106be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 shutdown_connections 2024-07-26T10:59:56.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.645+0000 7f1187e00640 1 -- 172.21.15.144:0/2564911907 wait complete. 2024-07-26T10:59:56.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 -- 172.21.15.144:0/3919668079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c118330 msgr2=0x7fdb2c11a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 --2- 172.21.15.144:0/3919668079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c118330 0x7fdb2c11a770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009be0 tx=0x7fdb2402a6c0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 -- 172.21.15.144:0/3919668079 shutdown_connections 2024-07-26T10:59:56.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 --2- 172.21.15.144:0/3919668079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c118330 0x7fdb2c11a770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 --2- 172.21.15.144:0/3919668079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb2c10a740 0x7fdb2c117df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 --2- 172.21.15.144:0/3919668079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c109df0 0x7fdb2c10a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 -- 172.21.15.144:0/3919668079 >> 172.21.15.144:0/3919668079 conn(0x7fdb2c070d20 msgr2=0x7fdb2c073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 -- 172.21.15.144:0/3919668079 shutdown_connections 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.830+0000 7fdb31800640 1 -- 172.21.15.144:0/3919668079 wait complete. 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.831+0000 7fdb31800640 1 Processor -- start 2024-07-26T10:59:56.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 -- start start 2024-07-26T10:59:56.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c109df0 0x7fdb2c1a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb2c10a740 0x7fdb2c1a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c118330 0x7fdb2c1aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c11dee0 con 0x7fdb2c10a740 2024-07-26T10:59:56.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c11dd60 con 0x7fdb2c118330 2024-07-26T10:59:56.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb31800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c11e060 con 0x7fdb2c109df0 2024-07-26T10:59:56.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb2b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c118330 0x7fdb2c1aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb2a000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb2c10a740 0x7fdb2c1a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb2b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c118330 0x7fdb2c1aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54252/0 (socket says 172.21.15.144:54252) 2024-07-26T10:59:56.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.832+0000 7fdb2b400640 1 -- 172.21.15.144:0/3676520418 learned_addr learned my addr 172.21.15.144:0/3676520418 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:56.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2aa00640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c109df0 0x7fdb2c1a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2b400640 1 -- 172.21.15.144:0/3676520418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c109df0 msgr2=0x7fdb2c1a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2b400640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c109df0 0x7fdb2c1a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2b400640 1 -- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb2c10a740 msgr2=0x7fdb2c1a4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2b400640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb2c10a740 0x7fdb2c1a4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2b400640 1 -- 172.21.15.144:0/3676520418 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c1aa880 con 0x7fdb2c118330 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.833+0000 7fdb2b400640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c118330 0x7fdb2c1aa150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdb24004af0 tx=0x7fdb24003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.834+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24041020 con 0x7fdb2c118330 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.834+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb24003e10 con 0x7fdb2c118330 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.834+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24036040 con 0x7fdb2c118330 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.834+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c1aab10 con 0x7fdb2c118330 2024-07-26T10:59:56.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.834+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c1b12d0 con 0x7fdb2c118330 2024-07-26T10:59:56.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.835+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb2c063500 con 0x7fdb2c118330 2024-07-26T10:59:56.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.838+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fdb24044020 con 0x7fdb2c118330 2024-07-26T10:59:56.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.839+0000 7fdb23400640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdb100b3d30 0x7fdb100b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.840+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fdb241051a0 con 0x7fdb2c118330 2024-07-26T10:59:56.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.840+0000 7fdb2aa00640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdb100b3d30 0x7fdb100b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.840+0000 7fdb23400640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] conn(0x7fdb100bf350 0x7fdb100c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:56.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.840+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 --> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb2403de50 con 0x7fdb100bf350 2024-07-26T10:59:56.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.840+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdb241055f0 con 0x7fdb2c118330 2024-07-26T10:59:56.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.840+0000 7fdb2a000640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] conn(0x7fdb100bf350 0x7fdb100c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:56.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.841+0000 7fdb2aa00640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdb100b3d30 0x7fdb100b6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009bd0 tx=0x7fdb1c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.841+0000 7fdb2a000640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] conn(0x7fdb100bf350 0x7fdb100c1840 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:56.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.842+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== osd.0 v2:172.21.15.144:6802/3520329112 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb2403de50 con 0x7fdb100bf350 2024-07-26T10:59:56.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.860+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 --> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb2c063900 con 0x7fdb100bf350 2024-07-26T10:59:56.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.861+0000 7fdb23400640 1 -- 172.21.15.144:0/3676520418 <== osd.0 v2:172.21.15.144:6802/3520329112 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb2c063900 con 0x7fdb100bf350 2024-07-26T10:59:56.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] conn(0x7fdb100bf350 msgr2=0x7fdb100c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] conn(0x7fdb100bf350 0x7fdb100c1840 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdb100b3d30 msgr2=0x7fdb100b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdb100b3d30 0x7fdb100b6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009bd0 tx=0x7fdb1c005f90 comp rx=0 tx=0).stop 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c118330 msgr2=0x7fdb2c1aa150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 --2- 172.21.15.144:0/3676520418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdb2c118330 0x7fdb2c1aa150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdb24004af0 tx=0x7fdb24003c20 comp rx=0 tx=0).stop 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb2b400640 1 -- 172.21.15.144:0/3676520418 reap_dead start 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 shutdown_connections 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 >> 172.21.15.144:0/3676520418 conn(0x7fdb2c070d20 msgr2=0x7fdb2c11a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 shutdown_connections 2024-07-26T10:59:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:56.862+0000 7fdb31800640 1 -- 172.21.15.144:0/3676520418 wait complete. 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 -- 172.21.15.144:0/2150553755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0751e0 msgr2=0x7f2ebc0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/2150553755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc0755f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009900 tx=0x7f2eb002a3f0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 -- 172.21.15.144:0/2150553755 shutdown_connections 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/2150553755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ebc10f940 0x7f2ebc111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/2150553755 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc075b30 0x7f2ebc10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/2150553755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc0755f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.169+0000 7f2ec2a00640 1 -- 172.21.15.144:0/2150553755 >> 172.21.15.144:0/2150553755 conn(0x7f2ebc0fee70 msgr2=0x7f2ebc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.170+0000 7f2ec2a00640 1 -- 172.21.15.144:0/2150553755 shutdown_connections 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.170+0000 7f2ec2a00640 1 -- 172.21.15.144:0/2150553755 wait complete. 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 Processor -- start 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 -- start start 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc19bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ebc075b30 0x7f2ebc19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc10f940 0x7f2ebc1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc115520 con 0x7f2ebc10f940 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc1153a0 con 0x7f2ebc0751e0 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.171+0000 7f2ec2a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc1156a0 con 0x7f2ebc075b30 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc19bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:57.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec0c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ebc075b30 0x7f2ebc19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc19bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54272/0 (socket says 172.21.15.144:54272) 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 -- 172.21.15.144:0/3219025762 learned_addr learned my addr 172.21.15.144:0/3219025762 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec2000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc10f940 0x7f2ebc1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 -- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ebc075b30 msgr2=0x7f2ebc19c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ebc075b30 0x7f2ebc19c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 -- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc10f940 msgr2=0x7f2ebc1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc10f940 0x7f2ebc1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec1600640 1 -- 172.21.15.144:0/3219025762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc1a1e40 con 0x7f2ebc0751e0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.172+0000 7f2ec0c00640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ebc075b30 0x7f2ebc19c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.173+0000 7f2ec1600640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc19bed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0005ab0 tx=0x7f2eb0003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.173+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb003f070 con 0x7f2ebc0751e0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.173+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eb0031b60 con 0x7f2ebc0751e0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.173+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ebc1a20d0 con 0x7f2ebc0751e0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.173+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0035030 con 0x7f2ebc0751e0 2024-07-26T10:59:57.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.173+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ebc1a8890 con 0x7f2ebc0751e0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.174+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ebc109eb0 con 0x7f2ebc0751e0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.176+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f2eb0042070 con 0x7f2ebc0751e0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.178+0000 7f2eba000640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2ea40b3cb0 0x7f2ea40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.178+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f2eb0104fe0 con 0x7f2ebc0751e0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.178+0000 7f2ec0c00640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2ea40b3cb0 0x7f2ea40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.179+0000 7f2eba000640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] conn(0x7f2ea40bf2d0 0x7f2ea40c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.179+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 --> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ea40c1e90 con 0x7f2ea40bf2d0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.179+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2eb0105430 con 0x7f2ebc0751e0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.179+0000 7f2ec2000640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] conn(0x7f2ea40bf2d0 0x7f2ea40c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.179+0000 7f2ec0c00640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2ea40b3cb0 0x7f2ea40b61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc111b90 tx=0x7f2ea8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:57.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.180+0000 7f2ec2000640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] conn(0x7f2ea40bf2d0 0x7f2ea40c17c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.181+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== osd.5 v2:172.21.15.183:6808/1593211139 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ea40c1e90 con 0x7f2ea40bf2d0 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.196+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 --> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ebc0766f0 con 0x7f2ea40bf2d0 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2eba000640 1 -- 172.21.15.144:0/3219025762 <== osd.5 v2:172.21.15.183:6808/1593211139 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2ebc0766f0 con 0x7f2ea40bf2d0 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] conn(0x7f2ea40bf2d0 msgr2=0x7f2ea40c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] conn(0x7f2ea40bf2d0 0x7f2ea40c17c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2ea40b3cb0 msgr2=0x7f2ea40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2ea40b3cb0 0x7f2ea40b61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc111b90 tx=0x7f2ea8005f90 comp rx=0 tx=0).stop 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc0751e0 msgr2=0x7f2ebc19bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T10:59:57.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2a00640 1 --2- 172.21.15.144:0/3219025762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2ebc0751e0 0x7f2ebc19bed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0005ab0 tx=0x7f2eb0003c10 comp rx=0 tx=0).stop 2024-07-26T10:59:57.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.197+0000 7f2ec2000640 1 -- 172.21.15.144:0/3219025762 reap_dead start 2024-07-26T10:59:57.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.198+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 shutdown_connections 2024-07-26T10:59:57.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.198+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 >> 172.21.15.144:0/3219025762 conn(0x7f2ebc0fee70 msgr2=0x7f2ebc109450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T10:59:57.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.198+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 shutdown_connections 2024-07-26T10:59:57.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T10:59:57.198+0000 7f2ec2a00640 1 -- 172.21.15.144:0/3219025762 wait complete. 2024-07-26T10:59:58.179 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 10:59:58 smithi144 ceph-mon[32129]: pgmap v385: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 10:59:58 smithi183 ceph-mon[37273]: pgmap v385: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:59:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 10:59:58 smithi160 ceph-mon[36715]: pgmap v385: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:00.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:00 smithi144 ceph-mon[32129]: pgmap v386: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:00.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:00 smithi144 ceph-mon[32129]: overall HEALTH_OK 2024-07-26T11:00:00.267 INFO:teuthology.orchestra.run.smithi144.stdout:206158430276 2024-07-26T11:00:00.268 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-26T11:00:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:00 smithi183 ceph-mon[37273]: pgmap v386: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:00 smithi183 ceph-mon[37273]: overall HEALTH_OK 2024-07-26T11:00:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:00 smithi160 ceph-mon[36715]: pgmap v386: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:00 smithi160 ceph-mon[36715]: overall HEALTH_OK 2024-07-26T11:00:01.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.355+0000 7f88bb200640 1 -- 172.21.15.144:0/3198973167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 msgr2=0x7f88b410fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:01.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.355+0000 7f88bb200640 1 --2- 172.21.15.144:0/3198973167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 0x7f88b410fb90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009ca0 tx=0x7f88b002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 -- 172.21.15.144:0/3198973167 shutdown_connections 2024-07-26T11:00:01.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 --2- 172.21.15.144:0/3198973167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 0x7f88b410fb90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 --2- 172.21.15.144:0/3198973167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88b4104b40 0x7f88b410d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 --2- 172.21.15.144:0/3198973167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 0x7f88b4104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 -- 172.21.15.144:0/3198973167 >> 172.21.15.144:0/3198973167 conn(0x7f88b40fdd80 msgr2=0x7f88b41001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 -- 172.21.15.144:0/3198973167 shutdown_connections 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.356+0000 7f88bb200640 1 -- 172.21.15.144:0/3198973167 wait complete. 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.357+0000 7f88bb200640 1 Processor -- start 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.357+0000 7f88bb200640 1 -- start start 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88bb200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 0x7f88b419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88bb200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88b4104b40 0x7f88b419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88bb200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 0x7f88b41a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88bb200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f88b4113310 con 0x7f88b4104110 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88bb200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f88b4113190 con 0x7f88b410d6c0 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88bb200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f88b4113490 con 0x7f88b4104b40 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88b9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 0x7f88b419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:01.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88b9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 0x7f88b419be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38936/0 (socket says 172.21.15.144:38936) 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88b9e00640 1 -- 172.21.15.144:0/363440942 learned_addr learned my addr 172.21.15.144:0/363440942 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88ba800640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 0x7f88b41a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.358+0000 7f88b9400640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88b4104b40 0x7f88b419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88b9e00640 1 -- 172.21.15.144:0/363440942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88b4104b40 msgr2=0x7f88b419c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88b9e00640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88b4104b40 0x7f88b419c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88b9e00640 1 -- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 msgr2=0x7f88b41a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88b9e00640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 0x7f88b41a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88b9e00640 1 -- 172.21.15.144:0/363440942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b41a1df0 con 0x7f88b4104110 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88ba800640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b410d6c0 0x7f88b41a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88b9e00640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 0x7f88b419be80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88a800ea60 tx=0x7f88a800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a800cde0 con 0x7f88b4104110 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b41a20e0 con 0x7f88b4104110 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a80035f0 con 0x7f88b4104110 2024-07-26T11:00:01.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.359+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8010690 con 0x7f88b4104110 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.360+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b41a8840 con 0x7f88b4104110 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.361+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88b410d140 con 0x7f88b4104110 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.364+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f88a80037a0 con 0x7f88b4104110 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.365+0000 7f88aea00640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f889c0b3cb0 0x7f889c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.365+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f88a80e4990 con 0x7f88b4104110 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.365+0000 7f88b9400640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f889c0b3cb0 0x7f889c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.366+0000 7f88aea00640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] conn(0x7f889c0bf2d0 0x7f889c0c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.366+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 --> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88a8022e50 con 0x7f889c0bf2d0 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.366+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f88a80e4d40 con 0x7f88b4104110 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.366+0000 7f88ba800640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] conn(0x7f889c0bf2d0 0x7f889c0c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.366+0000 7f88b9400640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f889c0b3cb0 0x7f889c0b61c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f88b410f990 tx=0x7f88a4005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.367+0000 7f88ba800640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] conn(0x7f889c0bf2d0 0x7f889c0c17c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:01.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.368+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== osd.10 v2:172.21.15.160:6824/949715799 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88a8022e50 con 0x7f889c0bf2d0 2024-07-26T11:00:01.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.383+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 --> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88b406b990 con 0x7f889c0bf2d0 2024-07-26T11:00:01.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88aea00640 1 -- 172.21.15.144:0/363440942 <== osd.10 v2:172.21.15.160:6824/949715799 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f88b406b990 con 0x7f889c0bf2d0 2024-07-26T11:00:01.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] conn(0x7f889c0bf2d0 msgr2=0x7f889c0c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:01.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88bb200640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] conn(0x7f889c0bf2d0 0x7f889c0c17c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:01.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f889c0b3cb0 msgr2=0x7f889c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88bb200640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f889c0b3cb0 0x7f889c0b61c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f88b410f990 tx=0x7f88a4005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 msgr2=0x7f88b419be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.384+0000 7f88bb200640 1 --2- 172.21.15.144:0/363440942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b4104110 0x7f88b419be80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88a800ea60 tx=0x7f88a800ef20 comp rx=0 tx=0).stop 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.385+0000 7f88ba800640 1 -- 172.21.15.144:0/363440942 reap_dead start 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.385+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 shutdown_connections 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.385+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 >> 172.21.15.144:0/363440942 conn(0x7f88b40fdd80 msgr2=0x7f88b410c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.385+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 shutdown_connections 2024-07-26T11:00:01.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:01.385+0000 7f88bb200640 1 -- 172.21.15.144:0/363440942 wait complete. 2024-07-26T11:00:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:02 smithi144 ceph-mon[32129]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:02 smithi183 ceph-mon[37273]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:02 smithi160 ceph-mon[36715]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:02.850 INFO:teuthology.orchestra.run.smithi144.stdout:38654705769 2024-07-26T11:00:02.851 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-26T11:00:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:04 smithi183 ceph-mon[37273]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T11:00:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:04 smithi144 ceph-mon[32129]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T11:00:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:04 smithi160 ceph-mon[36715]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T11:00:04.634 INFO:teuthology.orchestra.run.smithi144.stdout:158913790032 2024-07-26T11:00:04.634 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-26T11:00:05.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 -- 172.21.15.144:0/263018445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac1057e0 msgr2=0x7f53ac105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:05.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 --2- 172.21.15.144:0/263018445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac1057e0 0x7f53ac105bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009900 tx=0x7f53a002a410 comp rx=0 tx=0).stop 2024-07-26T11:00:05.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 -- 172.21.15.144:0/263018445 shutdown_connections 2024-07-26T11:00:05.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 --2- 172.21.15.144:0/263018445 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53ac10ba70 0x7f53ac1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 --2- 172.21.15.144:0/263018445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53ac10b120 0x7f53ac10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 --2- 172.21.15.144:0/263018445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac1057e0 0x7f53ac105bf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 -- 172.21.15.144:0/263018445 >> 172.21.15.144:0/263018445 conn(0x7f53ac1011f0 msgr2=0x7f53ac103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 -- 172.21.15.144:0/263018445 shutdown_connections 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.634+0000 7f53b3a00640 1 -- 172.21.15.144:0/263018445 wait complete. 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.635+0000 7f53b3a00640 1 Processor -- start 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 -- start start 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53ac1057e0 0x7f53ac19c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac10b120 0x7f53ac19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53ac10ba70 0x7f53ac1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac115790 con 0x7f53ac10b120 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac115610 con 0x7f53ac10ba70 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b3a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac115910 con 0x7f53ac1057e0 2024-07-26T11:00:05.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53abe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac10b120 0x7f53ac19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53abe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac10b120 0x7f53ac19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38962/0 (socket says 172.21.15.144:38962) 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53abe00640 1 -- 172.21.15.144:0/117236372 learned_addr learned my addr 172.21.15.144:0/117236372 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.636+0000 7f53b1a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53ac10ba70 0x7f53ac1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53b1000640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53ac1057e0 0x7f53ac19c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53abe00640 1 -- 172.21.15.144:0/117236372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53ac1057e0 msgr2=0x7f53ac19c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53abe00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53ac1057e0 0x7f53ac19c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53abe00640 1 -- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53ac10ba70 msgr2=0x7f53ac1a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53abe00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53ac10ba70 0x7f53ac1a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53abe00640 1 -- 172.21.15.144:0/117236372 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac1a20d0 con 0x7f53ac10b120 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53b1000640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53ac1057e0 0x7f53ac19c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.637+0000 7f53b1a00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53ac10ba70 0x7f53ac1a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:05.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.638+0000 7f53abe00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac10b120 0x7f53ac19c700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f539800e9f0 tx=0x7f539800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.638+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539800ccc0 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.638+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53980108e0 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.638+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac1a23c0 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.638+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398018860 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.638+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a8b20 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.640+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f53ac063540 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.643+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f5398010430 con 0x7f53ac10b120 2024-07-26T11:00:05.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.646+0000 7f53a9600640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53940b3cb0 0x7f53940b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.646+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f53980e4d10 con 0x7f53ac10b120 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.646+0000 7f53b1000640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53940b3cb0 0x7f53940b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.647+0000 7f53a9600640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] conn(0x7f53940bf2d0 0x7f53940c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.647+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 --> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53940c1e90 con 0x7f53940bf2d0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.647+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f53980e50c0 con 0x7f53ac10b120 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.647+0000 7f53b1a00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] conn(0x7f53940bf2d0 0x7f53940c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.647+0000 7f53b1000640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53940b3cb0 0x7f53940b61c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f53a0005ab0 tx=0x7f53a0005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.648+0000 7f53b1a00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] conn(0x7f53940bf2d0 0x7f53940c17c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.649+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== osd.4 v2:172.21.15.160:6808/2452808824 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53940c1e90 con 0x7f53940bf2d0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.665+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 --> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f53ac0639c0 con 0x7f53940bf2d0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53a9600640 1 -- 172.21.15.144:0/117236372 <== osd.4 v2:172.21.15.160:6808/2452808824 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f53ac0639c0 con 0x7f53940bf2d0 2024-07-26T11:00:05.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] conn(0x7f53940bf2d0 msgr2=0x7f53940c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b3a00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] conn(0x7f53940bf2d0 0x7f53940c17c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53940b3cb0 msgr2=0x7f53940b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b3a00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f53940b3cb0 0x7f53940b61c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f53a0005ab0 tx=0x7f53a0005a20 comp rx=0 tx=0).stop 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac10b120 msgr2=0x7f53ac19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b3a00640 1 --2- 172.21.15.144:0/117236372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53ac10b120 0x7f53ac19c700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f539800e9f0 tx=0x7f539800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.666+0000 7f53b1a00640 1 -- 172.21.15.144:0/117236372 reap_dead start 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.667+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 shutdown_connections 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.667+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 >> 172.21.15.144:0/117236372 conn(0x7f53ac1011f0 msgr2=0x7f53ac104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.667+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 shutdown_connections 2024-07-26T11:00:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.667+0000 7f53b3a00640 1 -- 172.21.15.144:0/117236372 wait complete. 2024-07-26T11:00:06.071 INFO:teuthology.orchestra.run.smithi144.stdout:111669149786 2024-07-26T11:00:06.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-26T11:00:06.127 INFO:teuthology.orchestra.run.smithi144.stdout:279172874294 2024-07-26T11:00:06.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-26T11:00:06.324 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:06 smithi144 ceph-mon[32129]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:00:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:06 smithi183 ceph-mon[37273]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 -- 172.21.15.144:0/588336790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 msgr2=0x7fddec10b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 --2- 172.21.15.144:0/588336790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 0x7fddec10b500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009900 tx=0x7fddd802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 -- 172.21.15.144:0/588336790 shutdown_connections 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 --2- 172.21.15.144:0/588336790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fddec10ba40 0x7fddec112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 --2- 172.21.15.144:0/588336790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 0x7fddec10b500 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 --2- 172.21.15.144:0/588336790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 0x7fddec105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 -- 172.21.15.144:0/588336790 >> 172.21.15.144:0/588336790 conn(0x7fddec101200 msgr2=0x7fddec103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:06.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 -- 172.21.15.144:0/588336790 shutdown_connections 2024-07-26T11:00:06.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.516+0000 7fddf4200640 1 -- 172.21.15.144:0/588336790 wait complete. 2024-07-26T11:00:06.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.517+0000 7fddf4200640 1 Processor -- start 2024-07-26T11:00:06.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.517+0000 7fddf4200640 1 -- start start 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf4200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 0x7fddec19c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf4200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 0x7fddec19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf4200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fddec10ba40 0x7fddec1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf4200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fddec1156d0 con 0x7fddec1057b0 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf4200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fddec115550 con 0x7fddec10b0f0 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf4200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fddec115850 con 0x7fddec10ba40 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf1800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 0x7fddec19c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf1800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 0x7fddec19c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38986/0 (socket says 172.21.15.144:38986) 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf1800640 1 -- 172.21.15.144:0/405159497 learned_addr learned my addr 172.21.15.144:0/405159497 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:06.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf0e00640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 0x7fddec19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.518+0000 7fddf2200640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fddec10ba40 0x7fddec1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf1800640 1 -- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fddec10ba40 msgr2=0x7fddec1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf1800640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fddec10ba40 0x7fddec1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf1800640 1 -- 172.21.15.144:0/405159497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 msgr2=0x7fddec19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf1800640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddec10b0f0 0x7fddec19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf1800640 1 -- 172.21.15.144:0/405159497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec1a2050 con 0x7fddec1057b0 2024-07-26T11:00:06.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf1800640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 0x7fddec19c0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdde000ea60 tx=0x7fdde000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:06.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde000cde0 con 0x7fddec1057b0 2024-07-26T11:00:06.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddec1a2340 con 0x7fddec1057b0 2024-07-26T11:00:06.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdde00035f0 con 0x7fddec1057b0 2024-07-26T11:00:06.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0010690 con 0x7fddec1057b0 2024-07-26T11:00:06.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.519+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddec1a8aa0 con 0x7fddec1057b0 2024-07-26T11:00:06.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.520+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fddec063540 con 0x7fddec1057b0 2024-07-26T11:00:06.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.523+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fdde00037a0 con 0x7fddec1057b0 2024-07-26T11:00:06.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.525+0000 7fddea000640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fddd40b3cb0 0x7fddd40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.525+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fdde00e4990 con 0x7fddec1057b0 2024-07-26T11:00:06.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.525+0000 7fddf0e00640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fddd40b3cb0 0x7fddd40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.525+0000 7fddea000640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] conn(0x7fddd40bf2d0 0x7fddd40c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.525+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 --> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdde0022e50 con 0x7fddd40bf2d0 2024-07-26T11:00:06.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.525+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdde00e4d40 con 0x7fddec1057b0 2024-07-26T11:00:06.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.526+0000 7fddf2200640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] conn(0x7fddd40bf2d0 0x7fddd40c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.526+0000 7fddf0e00640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fddd40b3cb0 0x7fddd40b61c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009230 tx=0x7fddd8031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:06.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.526+0000 7fddf2200640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] conn(0x7fddd40bf2d0 0x7fddd40c17c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:06.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.528+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== osd.11 v2:172.21.15.183:6824/2003058932 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdde0022e50 con 0x7fddd40bf2d0 2024-07-26T11:00:06.539 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 -- 172.21.15.144:0/3957690797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb670108d20 msgr2=0x7fb670109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.539 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 --2- 172.21.15.144:0/3957690797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb670108d20 0x7fb670109130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009900 tx=0x7fb65c02a410 comp rx=0 tx=0).stop 2024-07-26T11:00:06.539 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 -- 172.21.15.144:0/3957690797 shutdown_connections 2024-07-26T11:00:06.539 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 --2- 172.21.15.144:0/3957690797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 0x7fb67010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.539 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 --2- 172.21.15.144:0/3957690797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb670108d20 0x7fb670109130 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 --2- 172.21.15.144:0/3957690797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6701033e0 0x7fb6701037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 -- 172.21.15.144:0/3957690797 >> 172.21.15.144:0/3957690797 conn(0x7fb6700fee70 msgr2=0x7fb6701012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 -- 172.21.15.144:0/3957690797 shutdown_connections 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.724+0000 7fb66f200640 1 -- 172.21.15.144:0/3957690797 wait complete. 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.726+0000 7fb66f200640 1 Processor -- start 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.726+0000 7fb66f200640 1 -- start start 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6701033e0 0x7fb67019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb670108d20 0x7fb67019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 0x7fb6701a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fb670113350 con 0x7fb670109670 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb6701131d0 con 0x7fb6701033e0 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb6701134d0 con 0x7fb670108d20 2024-07-26T11:00:06.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 0x7fb6701a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66de00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6701033e0 0x7fb67019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 0x7fb6701a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38972/0 (socket says 172.21.15.144:38972) 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66e800640 1 -- 172.21.15.144:0/1081997400 learned_addr learned my addr 172.21.15.144:0/1081997400 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.727+0000 7fb66d400640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb670108d20 0x7fb67019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66e800640 1 -- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb670108d20 msgr2=0x7fb67019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66e800640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb670108d20 0x7fb67019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66e800640 1 -- 172.21.15.144:0/1081997400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6701033e0 msgr2=0x7fb67019bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66e800640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6701033e0 0x7fb67019bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66e800640 1 -- 172.21.15.144:0/1081997400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6701a1ed0 con 0x7fb670109670 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66de00640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6701033e0 0x7fb67019bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66e800640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 0x7fb6701a17a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb66800eb10 tx=0x7fb66800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668004b20 con 0x7fb670109670 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6701a21c0 con 0x7fb670109670 2024-07-26T11:00:06.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb668004cc0 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668010670 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.728+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6701a8920 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.729+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb67010fb90 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.732+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fb668005000 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.735+0000 7fb666a00640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb6540b3cb0 0x7fb6540b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.735+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fb6680e4ed0 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.735+0000 7fb66de00640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb6540b3cb0 0x7fb6540b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.736+0000 7fb666a00640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] conn(0x7fb6540bf2d0 0x7fb6540c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.736+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 --> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb66801aec0 con 0x7fb6540bf2d0 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.736+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb6680e5280 con 0x7fb670109670 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.736+0000 7fb66d400640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] conn(0x7fb6540bf2d0 0x7fb6540c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.736+0000 7fb66de00640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb6540b3cb0 0x7fb6540b61c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb660009fc0 tx=0x7fb660005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:06.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.737+0000 7fb66d400640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] conn(0x7fb6540bf2d0 0x7fb6540c17c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.738+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== osd.8 v2:172.21.15.183:6816/2430686973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb66801aec0 con 0x7fb6540bf2d0 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.754+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 --> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb670104660 con 0x7fb6540bf2d0 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.755+0000 7fb666a00640 1 -- 172.21.15.144:0/1081997400 <== osd.8 v2:172.21.15.183:6816/2430686973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb670104660 con 0x7fb6540bf2d0 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.755+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] conn(0x7fb6540bf2d0 msgr2=0x7fb6540c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.755+0000 7fb66f200640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] conn(0x7fb6540bf2d0 0x7fb6540c17c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb6540b3cb0 msgr2=0x7fb6540b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb6540b3cb0 0x7fb6540b61c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb660009fc0 tx=0x7fb660005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 msgr2=0x7fb6701a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 --2- 172.21.15.144:0/1081997400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb670109670 0x7fb6701a17a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb66800eb10 tx=0x7fb66800c2d0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66e800640 1 -- 172.21.15.144:0/1081997400 reap_dead start 2024-07-26T11:00:06.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 shutdown_connections 2024-07-26T11:00:06.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 >> 172.21.15.144:0/1081997400 conn(0x7fb6700fee70 msgr2=0x7fb67010e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:06.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 shutdown_connections 2024-07-26T11:00:06.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:05.756+0000 7fb66f200640 1 -- 172.21.15.144:0/1081997400 wait complete. 2024-07-26T11:00:06.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.543+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 --> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fddec063970 con 0x7fddd40bf2d0 2024-07-26T11:00:06.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddea000640 1 -- 172.21.15.144:0/405159497 <== osd.11 v2:172.21.15.183:6824/2003058932 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fddec063970 con 0x7fddd40bf2d0 2024-07-26T11:00:06.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] conn(0x7fddd40bf2d0 msgr2=0x7fddd40c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddf4200640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] conn(0x7fddd40bf2d0 0x7fddd40c17c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fddd40b3cb0 msgr2=0x7fddd40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddf4200640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fddd40b3cb0 0x7fddd40b61c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009230 tx=0x7fddd8031400 comp rx=0 tx=0).stop 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 msgr2=0x7fddec19c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.544+0000 7fddf4200640 1 --2- 172.21.15.144:0/405159497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddec1057b0 0x7fddec19c0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdde000ea60 tx=0x7fdde000ef20 comp rx=0 tx=0).stop 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.545+0000 7fddf2200640 1 -- 172.21.15.144:0/405159497 reap_dead start 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.545+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 shutdown_connections 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.545+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 >> 172.21.15.144:0/405159497 conn(0x7fddec101200 msgr2=0x7fddec103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.545+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 shutdown_connections 2024-07-26T11:00:06.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:06.545+0000 7fddf4200640 1 -- 172.21.15.144:0/405159497 wait complete. 2024-07-26T11:00:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:06 smithi160 ceph-mon[36715]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:00:07.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 -- 172.21.15.144:0/2463186994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104230 msgr2=0x7f10ac104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:07.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 --2- 172.21.15.144:0/2463186994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104230 0x7f10ac104640 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f109c009900 tx=0x7f109c02a400 comp rx=0 tx=0).stop 2024-07-26T11:00:07.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 -- 172.21.15.144:0/2463186994 shutdown_connections 2024-07-26T11:00:07.223 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 --2- 172.21.15.144:0/2463186994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ac10d690 0x7f10ac10fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 --2- 172.21.15.144:0/2463186994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10ac104b80 0x7f10ac10d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 --2- 172.21.15.144:0/2463186994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104230 0x7f10ac104640 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 -- 172.21.15.144:0/2463186994 >> 172.21.15.144:0/2463186994 conn(0x7f10ac0fdd70 msgr2=0x7f10ac1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:07.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 -- 172.21.15.144:0/2463186994 shutdown_connections 2024-07-26T11:00:07.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.221+0000 7f10ab200640 1 -- 172.21.15.144:0/2463186994 wait complete. 2024-07-26T11:00:07.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.222+0000 7f10ab200640 1 Processor -- start 2024-07-26T11:00:07.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 -- start start 2024-07-26T11:00:07.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ac104230 0x7f10ac101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:07.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104b80 0x7f10ac1024b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:07.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10ac10d690 0x7f10ac1029f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac113300 con 0x7f10ac104230 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac113180 con 0x7f10ac104b80 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10ab200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac113480 con 0x7f10ac10d690 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10a3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104b80 0x7f10ac1024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10a9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ac104230 0x7f10ac101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10aa800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10ac10d690 0x7f10ac1029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10a3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104b80 0x7f10ac1024b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:44554/0 (socket says 172.21.15.144:44554) 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.223+0000 7f10a3e00640 1 -- 172.21.15.144:0/3590446973 learned_addr learned my addr 172.21.15.144:0/3590446973 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a3e00640 1 -- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10ac10d690 msgr2=0x7f10ac1029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a3e00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10ac10d690 0x7f10ac1029f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a3e00640 1 -- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ac104230 msgr2=0x7f10ac101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:07.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a3e00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ac104230 0x7f10ac101f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a3e00640 1 -- 172.21.15.144:0/3590446973 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac1a7fb0 con 0x7f10ac104b80 2024-07-26T11:00:07.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10aa800640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10ac10d690 0x7f10ac1029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:07.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a3e00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104b80 0x7f10ac1024b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f109400e9f0 tx=0x7f109400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.224+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109400ccc0 con 0x7f10ac104b80 2024-07-26T11:00:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.225+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10940108e0 con 0x7f10ac104b80 2024-07-26T11:00:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.225+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094018860 con 0x7f10ac104b80 2024-07-26T11:00:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.225+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac1a82a0 con 0x7f10ac104b80 2024-07-26T11:00:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.225+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac1a8790 con 0x7f10ac104b80 2024-07-26T11:00:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.226+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10ac107bd0 con 0x7f10ac104b80 2024-07-26T11:00:07.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.228+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f1094010430 con 0x7f10ac104b80 2024-07-26T11:00:07.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.230+0000 7f10a2a00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10900b3cb0 0x7f10900b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:07.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.230+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f10940e4ce0 con 0x7f10ac104b80 2024-07-26T11:00:07.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.230+0000 7f10a9e00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10900b3cb0 0x7f10900b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:07.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.230+0000 7f10a2a00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] conn(0x7f10900bf2d0 0x7f10900c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:07.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.231+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 --> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10900c1e90 con 0x7f10900bf2d0 2024-07-26T11:00:07.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.231+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f10940e5090 con 0x7f10ac104b80 2024-07-26T11:00:07.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.231+0000 7f10aa800640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] conn(0x7f10900bf2d0 0x7f10900c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:07.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.231+0000 7f10a9e00640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10900b3cb0 0x7f10900b61c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f109c000c00 tx=0x7f109c031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:07.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.232+0000 7f10aa800640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] conn(0x7f10900bf2d0 0x7f10900c17c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:07.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.233+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== osd.2 v2:172.21.15.183:6800/2239917243 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10900c1e90 con 0x7f10900bf2d0 2024-07-26T11:00:07.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.248+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 --> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f10ac06b950 con 0x7f10900bf2d0 2024-07-26T11:00:07.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10a2a00640 1 -- 172.21.15.144:0/3590446973 <== osd.2 v2:172.21.15.183:6800/2239917243 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f10ac06b950 con 0x7f10900bf2d0 2024-07-26T11:00:07.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] conn(0x7f10900bf2d0 msgr2=0x7f10900c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:07.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10ab200640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] conn(0x7f10900bf2d0 0x7f10900c17c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10900b3cb0 msgr2=0x7f10900b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10ab200640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10900b3cb0 0x7f10900b61c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f109c000c00 tx=0x7f109c031400 comp rx=0 tx=0).stop 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104b80 msgr2=0x7f10ac1024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10ab200640 1 --2- 172.21.15.144:0/3590446973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10ac104b80 0x7f10ac1024b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f109400e9f0 tx=0x7f109400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.249+0000 7f10aa800640 1 -- 172.21.15.144:0/3590446973 reap_dead start 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.250+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 shutdown_connections 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.250+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 >> 172.21.15.144:0/3590446973 conn(0x7f10ac0fdd70 msgr2=0x7f10ac1071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.250+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 shutdown_connections 2024-07-26T11:00:07.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:07.250+0000 7f10ab200640 1 -- 172.21.15.144:0/3590446973 wait complete. 2024-07-26T11:00:07.720 INFO:teuthology.orchestra.run.smithi144.stdout:60129542246 2024-07-26T11:00:07.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-26T11:00:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:08 smithi144 ceph-mon[32129]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:08 smithi183 ceph-mon[37273]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:08 smithi160 ceph-mon[36715]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:08.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 -- 172.21.15.144:0/2400273598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be4108d20 msgr2=0x7f7be4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:08.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 --2- 172.21.15.144:0/2400273598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be4108d20 0x7f7be4109130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009900 tx=0x7f7bd002a410 comp rx=0 tx=0).stop 2024-07-26T11:00:08.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 -- 172.21.15.144:0/2400273598 shutdown_connections 2024-07-26T11:00:08.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 --2- 172.21.15.144:0/2400273598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7be4109670 0x7f7be410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:08.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 --2- 172.21.15.144:0/2400273598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be4108d20 0x7f7be4109130 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:08.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 --2- 172.21.15.144:0/2400273598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be41033e0 0x7f7be41037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:08.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 -- 172.21.15.144:0/2400273598 >> 172.21.15.144:0/2400273598 conn(0x7f7be40fee70 msgr2=0x7f7be41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:08.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 -- 172.21.15.144:0/2400273598 shutdown_connections 2024-07-26T11:00:08.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.830+0000 7f7beaa00640 1 -- 172.21.15.144:0/2400273598 wait complete. 2024-07-26T11:00:08.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 Processor -- start 2024-07-26T11:00:08.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 -- start start 2024-07-26T11:00:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be41033e0 0x7f7be4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be4108d20 0x7f7be419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7be4109670 0x7f7be419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7be4113330 con 0x7f7be41033e0 2024-07-26T11:00:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7be41131b0 con 0x7f7be4109670 2024-07-26T11:00:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.832+0000 7f7beaa00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7be41134b0 con 0x7f7be4108d20 2024-07-26T11:00:08.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.833+0000 7f7be8c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be4108d20 0x7f7be419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:08.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.833+0000 7f7be8c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be4108d20 0x7f7be419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:44318/0 (socket says 172.21.15.144:44318) 2024-07-26T11:00:08.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.833+0000 7f7be8c00640 1 -- 172.21.15.144:0/3901910019 learned_addr learned my addr 172.21.15.144:0/3901910019 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.833+0000 7f7be9600640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be41033e0 0x7f7be4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.833+0000 7f7bea000640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7be4109670 0x7f7be419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.833+0000 7f7be8c00640 1 -- 172.21.15.144:0/3901910019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7be4109670 msgr2=0x7f7be419f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be8c00640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7be4109670 0x7f7be419f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be8c00640 1 -- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be41033e0 msgr2=0x7f7be4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be8c00640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be41033e0 0x7f7be4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be8c00640 1 -- 172.21.15.144:0/3901910019 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be419fc90 con 0x7f7be4108d20 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be9600640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7be41033e0 0x7f7be4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:08.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7bea000640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7be4109670 0x7f7be419f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be8c00640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be4108d20 0x7f7be419a2c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0005ab0 tx=0x7f7bd0003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd003f070 con 0x7f7be4108d20 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be419ff20 con 0x7f7be4108d20 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.834+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd0031b50 con 0x7f7be4108d20 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.835+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0035030 con 0x7f7be4108d20 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.835+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be41a66e0 con 0x7f7be4108d20 2024-07-26T11:00:08.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.835+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7be4063540 con 0x7f7be4108d20 2024-07-26T11:00:08.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.838+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f7bd0042070 con 0x7f7be4108d20 2024-07-26T11:00:08.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.840+0000 7f7be2000640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7bcc0b3cb0 0x7f7bcc0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:08.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.840+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f7bd01050c0 con 0x7f7be4108d20 2024-07-26T11:00:08.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.840+0000 7f7be9600640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7bcc0b3cb0 0x7f7bcc0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:08.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.840+0000 7f7be2000640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] conn(0x7f7bcc0bf2d0 0x7f7bcc0c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:08.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.840+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 --> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bcc0c1e90 con 0x7f7bcc0bf2d0 2024-07-26T11:00:08.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.840+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f7bd0105470 con 0x7f7be4108d20 2024-07-26T11:00:08.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.841+0000 7f7bea000640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] conn(0x7f7bcc0bf2d0 0x7f7bcc0c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:08.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.841+0000 7f7be9600640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7bcc0b3cb0 0x7f7bcc0b61c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009fc0 tx=0x7f7bd8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:08.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.841+0000 7f7bea000640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] conn(0x7f7bcc0bf2d0 0x7f7bcc0c17c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:08.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.842+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== osd.9 v2:172.21.15.144:6826/2364379216 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7bcc0c1e90 con 0x7f7bcc0bf2d0 2024-07-26T11:00:08.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.858+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 --> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7be4063970 con 0x7f7bcc0bf2d0 2024-07-26T11:00:08.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.858+0000 7f7be2000640 1 -- 172.21.15.144:0/3901910019 <== osd.9 v2:172.21.15.144:6826/2364379216 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7be4063970 con 0x7f7bcc0bf2d0 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] conn(0x7f7bcc0bf2d0 msgr2=0x7f7bcc0c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] conn(0x7f7bcc0bf2d0 0x7f7bcc0c17c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7bcc0b3cb0 msgr2=0x7f7bcc0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7bcc0b3cb0 0x7f7bcc0b61c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009fc0 tx=0x7f7bd8005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be4108d20 msgr2=0x7f7be419a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 --2- 172.21.15.144:0/3901910019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7be4108d20 0x7f7be419a2c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0005ab0 tx=0x7f7bd0003b90 comp rx=0 tx=0).stop 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7bea000640 1 -- 172.21.15.144:0/3901910019 reap_dead start 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 shutdown_connections 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 >> 172.21.15.144:0/3901910019 conn(0x7f7be40fee70 msgr2=0x7f7be4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 shutdown_connections 2024-07-26T11:00:08.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:08.859+0000 7f7beaa00640 1 -- 172.21.15.144:0/3901910019 wait complete. 2024-07-26T11:00:09.566 INFO:teuthology.orchestra.run.smithi144.stdout:81604378722 2024-07-26T11:00:09.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-26T11:00:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:10 smithi144 ceph-mon[32129]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:10 smithi183 ceph-mon[37273]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:10.436 INFO:teuthology.orchestra.run.smithi144.stdout:137438953558 2024-07-26T11:00:10.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-26T11:00:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:10 smithi160 ceph-mon[36715]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:10.953 INFO:teuthology.orchestra.run.smithi144.stdout:257698037820 2024-07-26T11:00:10.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-26T11:00:11.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 -- 172.21.15.144:0/2032369836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 msgr2=0x7f6ca0105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:11.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 --2- 172.21.15.144:0/2032369836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca0105bc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009900 tx=0x7f6c9002a410 comp rx=0 tx=0).stop 2024-07-26T11:00:11.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 -- 172.21.15.144:0/2032369836 shutdown_connections 2024-07-26T11:00:11.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 --2- 172.21.15.144:0/2032369836 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ca010ba40 0x7f6ca0112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 --2- 172.21.15.144:0/2032369836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ca010b0f0 0x7f6ca010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 --2- 172.21.15.144:0/2032369836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca0105bc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 -- 172.21.15.144:0/2032369836 >> 172.21.15.144:0/2032369836 conn(0x7f6ca0101240 msgr2=0x7f6ca0103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 -- 172.21.15.144:0/2032369836 shutdown_connections 2024-07-26T11:00:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.195+0000 7f6ca6800640 1 -- 172.21.15.144:0/2032369836 wait complete. 2024-07-26T11:00:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.197+0000 7f6ca6800640 1 Processor -- start 2024-07-26T11:00:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.197+0000 7f6ca6800640 1 -- start start 2024-07-26T11:00:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6ca6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca019c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6ca6800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ca010b0f0 0x7f6ca019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6ca6800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ca010ba40 0x7f6ca01a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6ca6800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01156f0 con 0x7f6ca01057b0 2024-07-26T11:00:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6ca6800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0115570 con 0x7f6ca010b0f0 2024-07-26T11:00:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6ca6800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0115870 con 0x7f6ca010ba40 2024-07-26T11:00:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6c9f400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca019c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6c9f400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca019c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58520/0 (socket says 172.21.15.144:58520) 2024-07-26T11:00:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6c9f400640 1 -- 172.21.15.144:0/893711848 learned_addr learned my addr 172.21.15.144:0/893711848 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.198+0000 7f6c97e00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ca010b0f0 0x7f6ca019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9fe00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ca010ba40 0x7f6ca01a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9f400640 1 -- 172.21.15.144:0/893711848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ca010ba40 msgr2=0x7f6ca01a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9f400640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ca010ba40 0x7f6ca01a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9f400640 1 -- 172.21.15.144:0/893711848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ca010b0f0 msgr2=0x7f6ca019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9f400640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ca010b0f0 0x7f6ca019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9f400640 1 -- 172.21.15.144:0/893711848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca01a2070 con 0x7f6ca01057b0 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.199+0000 7f6c9f400640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca019c100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005940 tx=0x7f6c900040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.200+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9003f030 con 0x7f6ca01057b0 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.200+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c900044c0 con 0x7f6ca01057b0 2024-07-26T11:00:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.200+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca01a2300 con 0x7f6ca01057b0 2024-07-26T11:00:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.200+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90035070 con 0x7f6ca01057b0 2024-07-26T11:00:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.200+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca0077230 con 0x7f6ca01057b0 2024-07-26T11:00:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.201+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ca0063500 con 0x7f6ca01057b0 2024-07-26T11:00:11.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.204+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f6c90042030 con 0x7f6ca01057b0 2024-07-26T11:00:11.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.206+0000 7f6c9cc00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c840b3cb0 0x7f6c840b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.207+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6c90104d20 con 0x7f6ca01057b0 2024-07-26T11:00:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.207+0000 7f6c97e00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c840b3cb0 0x7f6c840b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.207+0000 7f6c9cc00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] conn(0x7f6c840bf2d0 0x7f6c840c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.207+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 --> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c840c1e90 con 0x7f6c840bf2d0 2024-07-26T11:00:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.207+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6c901050d0 con 0x7f6ca01057b0 2024-07-26T11:00:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.208+0000 7f6c9fe00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] conn(0x7f6c840bf2d0 0x7f6c840c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.208+0000 7f6c97e00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c840b3cb0 0x7f6c840b61c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ca010ac50 tx=0x7f6c88016040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.208+0000 7f6c9fe00640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] conn(0x7f6c840bf2d0 0x7f6c840c17c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.209+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== osd.6 v2:172.21.15.144:6818/1312925195 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c840c1e90 con 0x7f6c840bf2d0 2024-07-26T11:00:11.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.226+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 --> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ca0063900 con 0x7f6c840bf2d0 2024-07-26T11:00:11.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.226+0000 7f6c9cc00640 1 -- 172.21.15.144:0/893711848 <== osd.6 v2:172.21.15.144:6818/1312925195 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6ca0063900 con 0x7f6c840bf2d0 2024-07-26T11:00:11.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] conn(0x7f6c840bf2d0 msgr2=0x7f6c840c17c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:11.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] conn(0x7f6c840bf2d0 0x7f6c840c17c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c840b3cb0 msgr2=0x7f6c840b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:11.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c840b3cb0 0x7f6c840b61c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ca010ac50 tx=0x7f6c88016040 comp rx=0 tx=0).stop 2024-07-26T11:00:11.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 msgr2=0x7f6ca019c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:11.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 --2- 172.21.15.144:0/893711848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ca01057b0 0x7f6ca019c100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005940 tx=0x7f6c900040c0 comp rx=0 tx=0).stop 2024-07-26T11:00:11.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6c9fe00640 1 -- 172.21.15.144:0/893711848 reap_dead start 2024-07-26T11:00:11.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 shutdown_connections 2024-07-26T11:00:11.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 >> 172.21.15.144:0/893711848 conn(0x7f6ca0101240 msgr2=0x7f6ca0103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:11.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 shutdown_connections 2024-07-26T11:00:11.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:11.227+0000 7f6ca6800640 1 -- 172.21.15.144:0/893711848 wait complete. 2024-07-26T11:00:11.300 INFO:teuthology.orchestra.run.smithi144.stdout:304942678065 2024-07-26T11:00:11.300 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-26T11:00:12.197 INFO:teuthology.orchestra.run.smithi144.stdout:184683593805 2024-07-26T11:00:12.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-26T11:00:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:12 smithi144 ceph-mon[32129]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:12 smithi160 ceph-mon[36715]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:12 smithi183 ceph-mon[37273]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:12.637 INFO:teuthology.orchestra.run.smithi144.stdout:231928234049 2024-07-26T11:00:12.637 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-26T11:00:12.771 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.771 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.778 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.792 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.793 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.795 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.804 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.808 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.814 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:12.815 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:14.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:14 smithi183 ceph-mon[37273]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:00:14.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:14 smithi144 ceph-mon[32129]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:00:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:14 smithi160 ceph-mon[36715]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:00:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:15 smithi144 ceph-mon[32129]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-07-26T11:00:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:15 smithi144 ceph-mon[32129]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-07-26T11:00:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:15 smithi160 ceph-mon[36715]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-07-26T11:00:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:15 smithi160 ceph-mon[36715]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-07-26T11:00:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:15 smithi183 ceph-mon[37273]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-07-26T11:00:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:15 smithi183 ceph-mon[37273]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-07-26T11:00:18.068 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:18.353 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:18 smithi144 ceph-mon[32129]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:18 smithi183 ceph-mon[37273]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:18 smithi160 ceph-mon[36715]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:20.368 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:20 smithi144 ceph-mon[32129]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:20 smithi183 ceph-mon[37273]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:20 smithi160 ceph-mon[36715]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:22.278 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:22 smithi144 ceph-mon[32129]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-07-26T11:00:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:22 smithi183 ceph-mon[37273]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-07-26T11:00:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:22 smithi160 ceph-mon[36715]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-07-26T11:00:23.141 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:23 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1103625981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T11:00:23.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 -- 172.21.15.144:0/2318647911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b410b120 msgr2=0x7f32b410b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 --2- 172.21.15.144:0/2318647911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b410b120 0x7f32b410b530 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009900 tx=0x7f32a002a410 comp rx=0 tx=0).stop 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 -- 172.21.15.144:0/2318647911 shutdown_connections 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 --2- 172.21.15.144:0/2318647911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b410ba70 0x7f32b41122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 --2- 172.21.15.144:0/2318647911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b410b120 0x7f32b410b530 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 --2- 172.21.15.144:0/2318647911 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b41057e0 0x7f32b4105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.186+0000 7f32bb000640 1 -- 172.21.15.144:0/2318647911 >> 172.21.15.144:0/2318647911 conn(0x7f32b4101210 msgr2=0x7f32b4103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.187+0000 7f32bb000640 1 -- 172.21.15.144:0/2318647911 shutdown_connections 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.187+0000 7f32bb000640 1 -- 172.21.15.144:0/2318647911 wait complete. 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.188+0000 7f32bb000640 1 Processor -- start 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.188+0000 7f32bb000640 1 -- start start 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32bb000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 0x7f32b419c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:23.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32bb000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b410b120 0x7f32b419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32bb000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b410ba70 0x7f32b41a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32bb000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4115750 con 0x7f32b41057e0 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32bb000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f32b41155d0 con 0x7f32b410b120 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32bb000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f32b41158d0 con 0x7f32b410ba70 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32b3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 0x7f32b419c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32b3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 0x7f32b419c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59600/0 (socket says 172.21.15.144:59600) 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32b3e00640 1 -- 172.21.15.144:0/1103625981 learned_addr learned my addr 172.21.15.144:0/1103625981 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32b3400640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b410b120 0x7f32b419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.189+0000 7f32b9000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b410ba70 0x7f32b41a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3e00640 1 -- 172.21.15.144:0/1103625981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b410ba70 msgr2=0x7f32b41a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:23.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3e00640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b410ba70 0x7f32b41a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3e00640 1 -- 172.21.15.144:0/1103625981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b410b120 msgr2=0x7f32b419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3e00640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b410b120 0x7f32b419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3e00640 1 -- 172.21.15.144:0/1103625981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b41a2170 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3400640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b410b120 0x7f32b419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b3e00640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 0x7f32b419c200 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32a400e9f0 tx=0x7f32a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a400cd80 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32a40108e0 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b41a2460 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.190+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4018980 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.191+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b41a8bc0 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.192+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b4105bf0 con 0x7f32b41057e0 2024-07-26T11:00:23.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.194+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f32a4010430 con 0x7f32b41057e0 2024-07-26T11:00:23.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.196+0000 7f32b0c00640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f32980b3ca0 0x7f32980b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:23.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.196+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f32a40e43e0 con 0x7f32b41057e0 2024-07-26T11:00:23.320 INFO:teuthology.orchestra.run.smithi144.stdout:279172874298 2024-07-26T11:00:23.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.196+0000 7f32b3400640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f32980b3ca0 0x7f32980b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:23.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.197+0000 7f32b3400640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f32980b3ca0 0x7f32980b61b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f32a002aa00 tx=0x7f32a0005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:23.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.200+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f32a40aaef0 con 0x7f32b41057e0 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.416+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f32b402e0c0 con 0x7f32b41057e0 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.416+0000 7f32b0c00640 1 -- 172.21.15.144:0/1103625981 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f32a40af440 con 0x7f32b41057e0 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f32980b3ca0 msgr2=0x7f32980b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f32980b3ca0 0x7f32980b61b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f32a002aa00 tx=0x7f32a0005a20 comp rx=0 tx=0).stop 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 msgr2=0x7f32b419c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 0x7f32b419c200 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32a400e9f0 tx=0x7f32a400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 shutdown_connections 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f32980b3ca0 0x7f32980b61b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b410ba70 0x7f32b41a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32b410b120 0x7f32b419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 --2- 172.21.15.144:0/1103625981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32b41057e0 0x7f32b419c200 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 >> 172.21.15.144:0/1103625981 conn(0x7f32b4101210 msgr2=0x7f32b410a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.419+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 shutdown_connections 2024-07-26T11:00:23.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:22.420+0000 7f32bb000640 1 -- 172.21.15.144:0/1103625981 wait complete. 2024-07-26T11:00:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:23 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1103625981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T11:00:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:23 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1103625981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T11:00:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:24 smithi183 ceph-mon[37273]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:24.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:24 smithi160 ceph-mon[36715]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:24 smithi144 ceph-mon[32129]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 -- 172.21.15.144:0/2636224561 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 msgr2=0x7f35781037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 --2- 172.21.15.144:0/2636224561 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f35781037d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f356c009900 tx=0x7f356c02a410 comp rx=0 tx=0).stop 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 -- 172.21.15.144:0/2636224561 shutdown_connections 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 --2- 172.21.15.144:0/2636224561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 0x7f357810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 --2- 172.21.15.144:0/2636224561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 0x7f3578109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 --2- 172.21.15.144:0/2636224561 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f35781037d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 -- 172.21.15.144:0/2636224561 >> 172.21.15.144:0/2636224561 conn(0x7f35780fee10 msgr2=0x7f3578101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:24.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.805+0000 7f357f000640 1 -- 172.21.15.144:0/2636224561 shutdown_connections 2024-07-26T11:00:24.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.806+0000 7f357f000640 1 -- 172.21.15.144:0/2636224561 wait complete. 2024-07-26T11:00:24.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.807+0000 7f357f000640 1 Processor -- start 2024-07-26T11:00:24.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.807+0000 7f357f000640 1 -- start start 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.807+0000 7f357f000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f3578075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.807+0000 7f357f000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 0x7f3578075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.807+0000 7f357f000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 0x7f3578075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357f000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3578113370 con 0x7f3578109650 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357f000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f35781131f0 con 0x7f35781033c0 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357f000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f35781134f0 con 0x7f3578108d00 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357dc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f3578075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:24.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 0x7f3578075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357dc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f3578075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:46594/0 (socket says 172.21.15.144:46594) 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357dc00640 1 -- 172.21.15.144:0/604848180 learned_addr learned my addr 172.21.15.144:0/604848180 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357e600640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 0x7f3578075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357d200640 1 -- 172.21.15.144:0/604848180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 msgr2=0x7f3578075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357d200640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f3578075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357d200640 1 -- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 msgr2=0x7f3578075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357d200640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 0x7f3578075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357d200640 1 -- 172.21.15.144:0/604848180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35781a8020 con 0x7f3578108d00 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357e600640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 0x7f3578075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.808+0000 7f357dc00640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f3578075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:24.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.809+0000 7f357d200640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 0x7f3578075680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f357806c240 tx=0x7f356400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:24.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.809+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356400cc40 con 0x7f3578108d00 2024-07-26T11:00:24.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.809+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35781a8310 con 0x7f3578108d00 2024-07-26T11:00:24.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.809+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f356400cde0 con 0x7f3578108d00 2024-07-26T11:00:24.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.809+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564004a00 con 0x7f3578108d00 2024-07-26T11:00:24.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.809+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35781a8800 con 0x7f3578108d00 2024-07-26T11:00:24.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.810+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35781037d0 con 0x7f3578108d00 2024-07-26T11:00:24.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.813+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f3564004ba0 con 0x7f3578108d00 2024-07-26T11:00:24.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.814+0000 7f3576000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f35600b3c20 0x7f35600b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:24.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.815+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f35640e4200 con 0x7f3578108d00 2024-07-26T11:00:24.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.815+0000 7f357dc00640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f35600b3c20 0x7f35600b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:24.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.816+0000 7f357dc00640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f35600b3c20 0x7f35600b6130 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f356c005ec0 tx=0x7f356c005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:24.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:24.816+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f35640ab3f0 con 0x7f3578108d00 2024-07-26T11:00:25.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.033+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3578076440 con 0x7f3578108d00 2024-07-26T11:00:25.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.033+0000 7f3576000640 1 -- 172.21.15.144:0/604848180 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3578076440 con 0x7f3578108d00 2024-07-26T11:00:25.036 INFO:teuthology.orchestra.run.smithi144.stdout:158913790037 2024-07-26T11:00:25.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f35600b3c20 msgr2=0x7f35600b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:25.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f35600b3c20 0x7f35600b6130 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f356c005ec0 tx=0x7f356c005a20 comp rx=0 tx=0).stop 2024-07-26T11:00:25.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 msgr2=0x7f3578075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:25.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 0x7f3578075680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f357806c240 tx=0x7f356400ee50 comp rx=0 tx=0).stop 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 shutdown_connections 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f35600b3c20 0x7f35600b6130 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3578109650 0x7f3578075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3578108d00 0x7f3578075680 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 --2- 172.21.15.144:0/604848180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35781033c0 0x7f3578075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.036+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 >> 172.21.15.144:0/604848180 conn(0x7f35780fee10 msgr2=0x7f3578101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.037+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 shutdown_connections 2024-07-26T11:00:25.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:25.037+0000 7f357f000640 1 -- 172.21.15.144:0/604848180 wait complete. 2024-07-26T11:00:25.210 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:25 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/604848180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T11:00:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:25 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/604848180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T11:00:25.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:25 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/604848180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T11:00:26.304 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:26 smithi144 ceph-mon[32129]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.400+0000 7fd399000640 1 -- 172.21.15.144:0/3137519605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 msgr2=0x7fd394104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.400+0000 7fd399000640 1 --2- 172.21.15.144:0/3137519605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd394104600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd384009be0 tx=0x7fd38402a6d0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 -- 172.21.15.144:0/3137519605 shutdown_connections 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 --2- 172.21.15.144:0/3137519605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd39410d650 0x7fd39410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 --2- 172.21.15.144:0/3137519605 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 0x7fd39410d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 --2- 172.21.15.144:0/3137519605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd394104600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 -- 172.21.15.144:0/3137519605 >> 172.21.15.144:0/3137519605 conn(0x7fd3940fdcf0 msgr2=0x7fd394100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 -- 172.21.15.144:0/3137519605 shutdown_connections 2024-07-26T11:00:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.401+0000 7fd399000640 1 -- 172.21.15.144:0/3137519605 wait complete. 2024-07-26T11:00:26.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.402+0000 7fd399000640 1 Processor -- start 2024-07-26T11:00:26.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.402+0000 7fd399000640 1 -- start start 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd399000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd39419bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd399000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 0x7fd39419c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd399000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd39410d650 0x7fd3941a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd399000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd3941132c0 con 0x7fd39410d650 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd399000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd394113140 con 0x7fd394104b40 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd399000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd394113440 con 0x7fd3941041f0 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd393400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd39419bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd393e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd39410d650 0x7fd3941a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd38aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 0x7fd39419c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd393400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd39419bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:37450/0 (socket says 172.21.15.144:37450) 2024-07-26T11:00:26.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd393400640 1 -- 172.21.15.144:0/3115534658 learned_addr learned my addr 172.21.15.144:0/3115534658 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:26.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd38aa00640 1 -- 172.21.15.144:0/3115534658 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 msgr2=0x7fd39419bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:26.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd38aa00640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd39419bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.403+0000 7fd38aa00640 1 -- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd39410d650 msgr2=0x7fd3941a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:26.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd38aa00640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd39410d650 0x7fd3941a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd38aa00640 1 -- 172.21.15.144:0/3115534658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3941a1f20 con 0x7fd394104b40 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd38aa00640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 0x7fd39419c550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd39406a230 tx=0x7fd37c00ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c00cc90 con 0x7fd394104b40 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd37c00ce30 con 0x7fd394104b40 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c010430 con 0x7fd394104b40 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3941a2210 con 0x7fd394104b40 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.404+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3941a8970 con 0x7fd394104b40 2024-07-26T11:00:26.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.406+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd394104600 con 0x7fd394104b40 2024-07-26T11:00:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.408+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fd37c003680 con 0x7fd394104b40 2024-07-26T11:00:26.411 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.409+0000 7fd390c00640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3780b3c20 0x7fd3780b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:26.411 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.410+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd37c0e42f0 con 0x7fd394104b40 2024-07-26T11:00:26.411 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.410+0000 7fd393400640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3780b3c20 0x7fd3780b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:26.412 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.410+0000 7fd393400640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3780b3c20 0x7fd3780b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3840099a0 tx=0x7fd384009280 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:26.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.413+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd37c009510 con 0x7fd394104b40 2024-07-26T11:00:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:26 smithi160 ceph-mon[36715]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:26.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.622+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd3941a8c30 con 0x7fd394104b40 2024-07-26T11:00:26.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.623+0000 7fd390c00640 1 -- 172.21.15.144:0/3115534658 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd37c0ab450 con 0x7fd394104b40 2024-07-26T11:00:26.625 INFO:teuthology.orchestra.run.smithi144.stdout:81604378726 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3780b3c20 msgr2=0x7fd3780b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3780b3c20 0x7fd3780b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3840099a0 tx=0x7fd384009280 comp rx=0 tx=0).stop 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 msgr2=0x7fd39419c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 0x7fd39419c550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd39406a230 tx=0x7fd37c00ed90 comp rx=0 tx=0).stop 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 shutdown_connections 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3780b3c20 0x7fd3780b6130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd39410d650 0x7fd3941a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd394104b40 0x7fd39419c550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 --2- 172.21.15.144:0/3115534658 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3941041f0 0x7fd39419bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:26.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 >> 172.21.15.144:0/3115534658 conn(0x7fd3940fdcf0 msgr2=0x7fd394101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:26.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 shutdown_connections 2024-07-26T11:00:26.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:26.626+0000 7fd399000640 1 -- 172.21.15.144:0/3115534658 wait complete. 2024-07-26T11:00:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:26 smithi183 ceph-mon[37273]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:00:27.190 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:27 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3115534658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 -- 172.21.15.144:0/1623628978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c104b80 msgr2=0x7fe92c10d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 --2- 172.21.15.144:0/1623628978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c104b80 0x7fe92c10d0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe918009860 tx=0x7fe91802a340 comp rx=0 tx=0).stop 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 -- 172.21.15.144:0/1623628978 shutdown_connections 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 --2- 172.21.15.144:0/1623628978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c10d690 0x7fe92c10fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 --2- 172.21.15.144:0/1623628978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c104b80 0x7fe92c10d0d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 --2- 172.21.15.144:0/1623628978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104230 0x7fe92c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.308+0000 7fe932600640 1 -- 172.21.15.144:0/1623628978 >> 172.21.15.144:0/1623628978 conn(0x7fe92c0fdd50 msgr2=0x7fe92c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.309+0000 7fe932600640 1 -- 172.21.15.144:0/1623628978 shutdown_connections 2024-07-26T11:00:27.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.309+0000 7fe932600640 1 -- 172.21.15.144:0/1623628978 wait complete. 2024-07-26T11:00:27.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.310+0000 7fe932600640 1 Processor -- start 2024-07-26T11:00:27.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.310+0000 7fe932600640 1 -- start start 2024-07-26T11:00:27.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.310+0000 7fe932600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c104230 0x7fe92c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:27.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.310+0000 7fe932600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104b80 0x7fe92c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.310+0000 7fe932600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c10d690 0x7fe92c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.310+0000 7fe932600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c1132e0 con 0x7fe92c104b80 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe932600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c113160 con 0x7fe92c10d690 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe932600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c113460 con 0x7fe92c104230 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe92be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104b80 0x7fe92c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe92be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104b80 0x7fe92c19c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59696/0 (socket says 172.21.15.144:59696) 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c104230 0x7fe92c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe92be00640 1 -- 172.21.15.144:0/3726365110 learned_addr learned my addr 172.21.15.144:0/3726365110 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:27.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931c00640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c10d690 0x7fe92c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:27.314 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931200640 1 -- 172.21.15.144:0/3726365110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c10d690 msgr2=0x7fe92c1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:27.314 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931200640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c10d690 0x7fe92c1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931200640 1 -- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104b80 msgr2=0x7fe92c19c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:27.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931200640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104b80 0x7fe92c19c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.311+0000 7fe931200640 1 -- 172.21.15.144:0/3726365110 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c1a1e80 con 0x7fe92c104230 2024-07-26T11:00:27.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.312+0000 7fe931200640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c104230 0x7fe92c19bf10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe920005fd0 tx=0x7fe92000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:27.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.312+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92000cd00 con 0x7fe92c104230 2024-07-26T11:00:27.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.312+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9200035f0 con 0x7fe92c104230 2024-07-26T11:00:27.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.312+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe920010690 con 0x7fe92c104230 2024-07-26T11:00:27.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.312+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92c1a2170 con 0x7fe92c104230 2024-07-26T11:00:27.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.312+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92c1a88d0 con 0x7fe92c104230 2024-07-26T11:00:27.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.313+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe92c104640 con 0x7fe92c104230 2024-07-26T11:00:27.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.316+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fe920003a80 con 0x7fe92c104230 2024-07-26T11:00:27.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.317+0000 7fe929600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe9140b3ca0 0x7fe9140b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:27.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.318+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fe9200e4320 con 0x7fe92c104230 2024-07-26T11:00:27.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.318+0000 7fe92be00640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe9140b3ca0 0x7fe9140b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:27.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.319+0000 7fe92be00640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe9140b3ca0 0x7fe9140b61b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe9180057c0 tx=0x7fe9180058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:27.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.319+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe9200ab510 con 0x7fe92c104230 2024-07-26T11:00:27.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.544+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fe92c19d040 con 0x7fe92c104230 2024-07-26T11:00:27.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.545+0000 7fe929600640 1 -- 172.21.15.144:0/3726365110 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe9200aae10 con 0x7fe92c104230 2024-07-26T11:00:27.547 INFO:teuthology.orchestra.run.smithi144.stdout:257698037824 2024-07-26T11:00:27.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.547+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe9140b3ca0 msgr2=0x7fe9140b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:27.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.547+0000 7fe932600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe9140b3ca0 0x7fe9140b61b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe9180057c0 tx=0x7fe9180058d0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.547+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c104230 msgr2=0x7fe92c19bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.547+0000 7fe932600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c104230 0x7fe92c19bf10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe920005fd0 tx=0x7fe92000ee50 comp rx=0 tx=0).stop 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 shutdown_connections 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe9140b3ca0 0x7fe9140b61b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe92c10d690 0x7fe92c1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe92c104b80 0x7fe92c19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 --2- 172.21.15.144:0/3726365110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe92c104230 0x7fe92c19bf10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 >> 172.21.15.144:0/3726365110 conn(0x7fe92c0fdd50 msgr2=0x7fe92c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:27.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 shutdown_connections 2024-07-26T11:00:27.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.548+0000 7fe932600640 1 -- 172.21.15.144:0/3726365110 wait complete. 2024-07-26T11:00:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:27 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3115534658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T11:00:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:27 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3115534658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T11:00:28.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.187+0000 7faf79600640 1 -- 172.21.15.144:0/4074336775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 msgr2=0x7faf74105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.187+0000 7faf79600640 1 --2- 172.21.15.144:0/4074336775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf74105bc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faf64009900 tx=0x7faf6402a410 comp rx=0 tx=0).stop 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 -- 172.21.15.144:0/4074336775 shutdown_connections 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 --2- 172.21.15.144:0/4074336775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf7410ba40 0x7faf74112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 --2- 172.21.15.144:0/4074336775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410b0f0 0x7faf7410b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 --2- 172.21.15.144:0/4074336775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf74105bc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 -- 172.21.15.144:0/4074336775 >> 172.21.15.144:0/4074336775 conn(0x7faf74101240 msgr2=0x7faf74103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 -- 172.21.15.144:0/4074336775 shutdown_connections 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.188+0000 7faf79600640 1 -- 172.21.15.144:0/4074336775 wait complete. 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.189+0000 7faf79600640 1 Processor -- start 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 -- start start 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf7419c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:28.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf7410b0f0 0x7faf7419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410ba40 0x7faf741a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7faf74115710 con 0x7faf741057b0 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faf74115590 con 0x7faf7410b0f0 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.190+0000 7faf79600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf74115890 con 0x7faf7410ba40 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf72a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf7419c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf72a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf7419c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59664/0 (socket says 172.21.15.144:59664) 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf72a00640 1 -- 172.21.15.144:0/1106517679 learned_addr learned my addr 172.21.15.144:0/1106517679 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf73400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410ba40 0x7faf741a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:28.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf72000640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf7410b0f0 0x7faf7419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stdout:206158430282 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf73400640 1 -- 172.21.15.144:0/1106517679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf7410b0f0 msgr2=0x7faf7419c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf73400640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf7410b0f0 0x7faf7419c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf73400640 1 -- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 msgr2=0x7faf7419c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf73400640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf7419c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.191+0000 7faf73400640 1 -- 172.21.15.144:0/1106517679 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf741a20b0 con 0x7faf7410ba40 2024-07-26T11:00:28.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.192+0000 7faf73400640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410ba40 0x7faf741a1980 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009fd0 tx=0x7faf6c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.192+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c00ce20 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.192+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf6c005020 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.192+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf741a23a0 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.192+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c010690 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.192+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf741a8b00 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.193+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf74105bc0 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.197+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7faf6c004b20 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.199+0000 7faf6b400640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faf580b3ca0 0x7faf580b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.199+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7faf6c0e4770 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.199+0000 7faf72a00640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faf580b3ca0 0x7faf580b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.200+0000 7faf72a00640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faf580b3ca0 0x7faf580b61b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faf64031440 tx=0x7faf640058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.203+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faf6c0ab8d0 con 0x7faf7410ba40 2024-07-26T11:00:28.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.409+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7faf7410d230 con 0x7faf7410ba40 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.410+0000 7faf6b400640 1 -- 172.21.15.144:0/1106517679 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faf6c0ab1d0 con 0x7faf7410ba40 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faf580b3ca0 msgr2=0x7faf580b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faf580b3ca0 0x7faf580b61b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faf64031440 tx=0x7faf640058d0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410ba40 msgr2=0x7faf741a1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410ba40 0x7faf741a1980 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009fd0 tx=0x7faf6c00ef10 comp rx=0 tx=0).stop 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 shutdown_connections 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faf580b3ca0 0x7faf580b61b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7410ba40 0x7faf741a1980 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf7410b0f0 0x7faf7419c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 --2- 172.21.15.144:0/1106517679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf741057b0 0x7faf7419c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 >> 172.21.15.144:0/1106517679 conn(0x7faf74101240 msgr2=0x7faf74103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 shutdown_connections 2024-07-26T11:00:28.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:27.413+0000 7faf79600640 1 -- 172.21.15.144:0/1106517679 wait complete. 2024-07-26T11:00:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:28 smithi144 ceph-mon[32129]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-07-26T11:00:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:28 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1106517679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T11:00:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:28 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3726365110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T11:00:28.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:28 smithi160 ceph-mon[36715]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-07-26T11:00:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:28 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1106517679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T11:00:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:28 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3726365110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T11:00:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:28 smithi183 ceph-mon[37273]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-07-26T11:00:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:28 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1106517679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T11:00:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:28 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3726365110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T11:00:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 -- 172.21.15.144:0/1675414800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc108d20 msgr2=0x7f1fdc109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/1675414800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc108d20 0x7f1fdc109130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009900 tx=0x7f1fc802a400 comp rx=0 tx=0).stop 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 -- 172.21.15.144:0/1675414800 shutdown_connections 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/1675414800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc109670 0x7f1fdc10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/1675414800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc108d20 0x7f1fdc109130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/1675414800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 -- 172.21.15.144:0/1675414800 >> 172.21.15.144:0/1675414800 conn(0x7f1fdc0fee70 msgr2=0x7f1fdc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 -- 172.21.15.144:0/1675414800 shutdown_connections 2024-07-26T11:00:29.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.873+0000 7f1fe1a00640 1 -- 172.21.15.144:0/1675414800 wait complete. 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.875+0000 7f1fe1a00640 1 Processor -- start 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.875+0000 7f1fe1a00640 1 -- start start 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.875+0000 7f1fe1a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.875+0000 7f1fe1a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fdc108d20 0x7f1fdc19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.875+0000 7f1fe1a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc109670 0x7f1fdc19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fe1a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1fdc113330 con 0x7f1fdc1033e0 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fe1a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1fdc1131b0 con 0x7f1fdc108d20 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fe1a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1fdc1134b0 con 0x7f1fdc109670 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59708/0 (socket says 172.21.15.144:59708) 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fe1000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc109670 0x7f1fdc19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:29.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 -- 172.21.15.144:0/2642897866 learned_addr learned my addr 172.21.15.144:0/2642897866 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdb400640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fdc108d20 0x7f1fdc19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 -- 172.21.15.144:0/2642897866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc109670 msgr2=0x7f1fdc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc109670 0x7f1fdc19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 -- 172.21.15.144:0/2642897866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fdc108d20 msgr2=0x7f1fdc19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fdc108d20 0x7f1fdc19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.876+0000 7f1fdbe00640 1 -- 172.21.15.144:0/2642897866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc19fcd0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fe1000640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc109670 0x7f1fdc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fdbe00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc199d60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00ea60 tx=0x7f1fcc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc00cd20 con 0x7f1fdc1033e0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fcc0035f0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fdc19ffc0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc010690 con 0x7f1fdc1033e0 2024-07-26T11:00:29.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.877+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fdc075020 con 0x7f1fdc1033e0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.879+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fdc1037f0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.882+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f1fcc0037a0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.884+0000 7f1fd8c00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1fc00b3c20 0x7f1fc00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.884+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f1fcc0e43f0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.884+0000 7f1fdb400640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1fc00b3c20 0x7f1fc00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.885+0000 7f1fdb400640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1fc00b3c20 0x7f1fc00b6130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1fc802a9f0 tx=0x7f1fc8005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:28.888+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1fcc0ab550 con 0x7f1fdc1033e0 2024-07-26T11:00:29.288 INFO:teuthology.orchestra.run.smithi144.stdout:184683593808 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.093+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1fdc10b4b0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.094+0000 7f1fd8c00640 1 -- 172.21.15.144:0/2642897866 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1fdc10b4b0 con 0x7f1fdc1033e0 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1fc00b3c20 msgr2=0x7f1fc00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1fc00b3c20 0x7f1fc00b6130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1fc802a9f0 tx=0x7f1fc8005a20 comp rx=0 tx=0).stop 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 msgr2=0x7f1fdc199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc199d60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00ea60 tx=0x7f1fcc00ee60 comp rx=0 tx=0).stop 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 shutdown_connections 2024-07-26T11:00:29.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1fc00b3c20 0x7f1fc00b6130 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc109670 0x7f1fdc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fdc108d20 0x7f1fdc19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 --2- 172.21.15.144:0/2642897866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fdc1033e0 0x7f1fdc199d60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:29.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.097+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 >> 172.21.15.144:0/2642897866 conn(0x7f1fdc0fee70 msgr2=0x7f1fdc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:29.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.098+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 shutdown_connections 2024-07-26T11:00:29.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:29.098+0000 7f1fe1a00640 1 -- 172.21.15.144:0/2642897866 wait complete. 2024-07-26T11:00:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:00:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:29 smithi144 ceph-mon[32129]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:29 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2642897866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T11:00:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:00:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:29 smithi160 ceph-mon[36715]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:29 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2642897866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T11:00:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:00:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:29 smithi183 ceph-mon[37273]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:29 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2642897866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T11:00:30.314 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 -- 172.21.15.144:0/1922224172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410b860 msgr2=0x7fbe34112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:30.314 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1922224172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410b860 0x7fbe34112090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009ca0 tx=0x7fbe2c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.314 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 -- 172.21.15.144:0/1922224172 shutdown_connections 2024-07-26T11:00:30.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1922224172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410b860 0x7fbe34112090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1922224172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe3410af10 0x7fbe3410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1922224172 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe341059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 -- 172.21.15.144:0/1922224172 >> 172.21.15.144:0/1922224172 conn(0x7fbe34101040 msgr2=0x7fbe34103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:30.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 -- 172.21.15.144:0/1922224172 shutdown_connections 2024-07-26T11:00:30.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.311+0000 7fbe38c00640 1 -- 172.21.15.144:0/1922224172 wait complete. 2024-07-26T11:00:30.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.312+0000 7fbe38c00640 1 Processor -- start 2024-07-26T11:00:30.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.312+0000 7fbe38c00640 1 -- start start 2024-07-26T11:00:30.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe38c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe3419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe38c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 0x7fbe3419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe38c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe3410b860 0x7fbe341a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe38c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbe34115510 con 0x7fbe3410af10 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe38c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbe34115390 con 0x7fbe341055d0 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe38c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe34115690 con 0x7fbe3410b860 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe33400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe3419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe32a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 0x7fbe3419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe33400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe3419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:44516/0 (socket says 172.21.15.144:44516) 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe33400640 1 -- 172.21.15.144:0/1033070934 learned_addr learned my addr 172.21.15.144:0/1033070934 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:30.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe32a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 0x7fbe3419c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57546/0 (socket says 172.21.15.144:57546) 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.313+0000 7fbe33e00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe3410b860 0x7fbe341a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe33400640 1 -- 172.21.15.144:0/1033070934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe3410b860 msgr2=0x7fbe341a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe33400640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe3410b860 0x7fbe341a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe33400640 1 -- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 msgr2=0x7fbe3419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe33400640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 0x7fbe3419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe33400640 1 -- 172.21.15.144:0/1033070934 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe341a1e70 con 0x7fbe341055d0 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe32a00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 0x7fbe3419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe33400640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe3419bf00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400e9f0 tx=0x7fbe2400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2400cd80 con 0x7fbe341055d0 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe341a2160 con 0x7fbe341055d0 2024-07-26T11:00:30.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe240108e0 con 0x7fbe341055d0 2024-07-26T11:00:30.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.314+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24018980 con 0x7fbe341055d0 2024-07-26T11:00:30.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.315+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe341a88c0 con 0x7fbe341055d0 2024-07-26T11:00:30.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.316+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe341059e0 con 0x7fbe341055d0 2024-07-26T11:00:30.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.318+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fbe24010430 con 0x7fbe341055d0 2024-07-26T11:00:30.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.320+0000 7fbe2be00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbe180b3ca0 0x7fbe180b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:30.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.320+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fbe240e4900 con 0x7fbe341055d0 2024-07-26T11:00:30.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.320+0000 7fbe32a00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbe180b3ca0 0x7fbe180b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:30.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.321+0000 7fbe32a00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbe180b3ca0 0x7fbe180b61b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe341048f0 tx=0x7fbe20005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:30.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.321+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbe240ab410 con 0x7fbe341055d0 2024-07-26T11:00:30.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.525+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fbe3402e0c0 con 0x7fbe341055d0 2024-07-26T11:00:30.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.526+0000 7fbe2be00640 1 -- 172.21.15.144:0/1033070934 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fbe240af960 con 0x7fbe341055d0 2024-07-26T11:00:30.530 INFO:teuthology.orchestra.run.smithi144.stdout:304942678069 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.529+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbe180b3ca0 msgr2=0x7fbe180b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.529+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbe180b3ca0 0x7fbe180b61b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe341048f0 tx=0x7fbe20005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.529+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 msgr2=0x7fbe3419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.529+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe3419bf00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400e9f0 tx=0x7fbe2400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 shutdown_connections 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbe180b3ca0 0x7fbe180b61b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe3410b860 0x7fbe341a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe3410af10 0x7fbe3419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 --2- 172.21.15.144:0/1033070934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe341055d0 0x7fbe3419bf00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:30.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 >> 172.21.15.144:0/1033070934 conn(0x7fbe34101040 msgr2=0x7fbe3410a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:30.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 shutdown_connections 2024-07-26T11:00:30.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:30.530+0000 7fbe38c00640 1 -- 172.21.15.144:0/1033070934 wait complete. 2024-07-26T11:00:31.110 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:30 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1033070934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T11:00:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:30 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1033070934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T11:00:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:30 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1033070934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T11:00:31.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037820 got 257698037824 for osd.9 2024-07-26T11:00:31.442 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:31.557 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678065 got 304942678069 for osd.11 2024-07-26T11:00:31.557 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:31.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 -- 172.21.15.144:0/1190683148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb81033e0 msgr2=0x7f6cb81037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 --2- 172.21.15.144:0/1190683148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb81033e0 0x7f6cb81037f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009ca0 tx=0x7f6ca802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 -- 172.21.15.144:0/1190683148 shutdown_connections 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 --2- 172.21.15.144:0/1190683148 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb8109670 0x7f6cb810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 --2- 172.21.15.144:0/1190683148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 0x7f6cb8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 --2- 172.21.15.144:0/1190683148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb81033e0 0x7f6cb81037f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.296+0000 7f6cbd000640 1 -- 172.21.15.144:0/1190683148 >> 172.21.15.144:0/1190683148 conn(0x7f6cb80fee70 msgr2=0x7f6cb81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.297+0000 7f6cbd000640 1 -- 172.21.15.144:0/1190683148 shutdown_connections 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.297+0000 7f6cbd000640 1 -- 172.21.15.144:0/1190683148 wait complete. 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 Processor -- start 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 -- start start 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb81033e0 0x7f6cb819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 0x7f6cb819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb8109670 0x7f6cb819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb80753b0 con 0x7f6cb8109670 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8075230 con 0x7f6cb81033e0 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.298+0000 7f6cbd000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8075530 con 0x7f6cb8108d20 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb81033e0 0x7f6cb819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb8109670 0x7f6cb819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb81033e0 0x7f6cb819c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:44540/0 (socket says 172.21.15.144:44540) 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 0x7f6cb819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb7400640 1 -- 172.21.15.144:0/2931482450 learned_addr learned my addr 172.21.15.144:0/2931482450 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 -- 172.21.15.144:0/2931482450 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb81033e0 msgr2=0x7f6cb819c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb81033e0 0x7f6cb819c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 -- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb8109670 msgr2=0x7f6cb819e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb8109670 0x7f6cb819e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 -- 172.21.15.144:0/2931482450 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb819ecb0 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.299+0000 7f6cb6a00640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 0x7f6cb819c890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6cb806b3c0 tx=0x7f6ca000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.300+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca000ccc0 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.300+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb819efa0 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.300+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ca0009e40 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.300+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0004a00 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.300+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb81a8940 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.301+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb81037f0 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.303+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f6ca0005020 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.305+0000 7f6cafe00640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c9c0b3c20 0x7f6c9c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.305+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6ca00e4190 con 0x7f6cb8108d20 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.305+0000 7f6cb7400640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c9c0b3c20 0x7f6c9c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.306+0000 7f6cb7400640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c9c0b3c20 0x7f6c9c0b6130 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8000f80 tx=0x7f6ca8005d80 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:31.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.306+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6ca00ab380 con 0x7f6cb8108d20 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.510+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6cb819d0c0 con 0x7f6cb8108d20 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.510+0000 7f6cafe00640 1 -- 172.21.15.144:0/2931482450 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6ca00aac80 con 0x7f6cb8108d20 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stdout:60129542253 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c9c0b3c20 msgr2=0x7f6c9c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c9c0b3c20 0x7f6c9c0b6130 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8000f80 tx=0x7f6ca8005d80 comp rx=0 tx=0).stop 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 msgr2=0x7f6cb819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:31.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 0x7f6cb819c890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6cb806b3c0 tx=0x7f6ca000ed90 comp rx=0 tx=0).stop 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 shutdown_connections 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6c9c0b3c20 0x7f6c9c0b6130 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb8109670 0x7f6cb819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cb8108d20 0x7f6cb819c890 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 --2- 172.21.15.144:0/2931482450 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cb81033e0 0x7f6cb819c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 >> 172.21.15.144:0/2931482450 conn(0x7f6cb80fee70 msgr2=0x7f6cb8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.515+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 shutdown_connections 2024-07-26T11:00:31.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:31.516+0000 7f6cbd000640 1 -- 172.21.15.144:0/2931482450 wait complete. 2024-07-26T11:00:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:31 smithi144 ceph-mon[32129]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:31 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2931482450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T11:00:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:31 smithi183 ceph-mon[37273]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:31 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2931482450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T11:00:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:31 smithi160 ceph-mon[36715]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:31 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2931482450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T11:00:32.655 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 -- 172.21.15.144:0/316964847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c10af10 msgr2=0x7efe0c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 --2- 172.21.15.144:0/316964847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c10af10 0x7efe0c10b320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009900 tx=0x7efdf402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 -- 172.21.15.144:0/316964847 shutdown_connections 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 --2- 172.21.15.144:0/316964847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c10b860 0x7efe0c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 --2- 172.21.15.144:0/316964847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c10af10 0x7efe0c10b320 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 --2- 172.21.15.144:0/316964847 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe0c1055d0 0x7efe0c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 -- 172.21.15.144:0/316964847 >> 172.21.15.144:0/316964847 conn(0x7efe0c101040 msgr2=0x7efe0c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 -- 172.21.15.144:0/316964847 shutdown_connections 2024-07-26T11:00:32.656 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.652+0000 7efe0b200640 1 -- 172.21.15.144:0/316964847 wait complete. 2024-07-26T11:00:32.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.653+0000 7efe0b200640 1 Processor -- start 2024-07-26T11:00:32.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.653+0000 7efe0b200640 1 -- start start 2024-07-26T11:00:32.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 0x7efe0c19bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:32.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c10af10 0x7efe0c19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe0c10b860 0x7efe0c1a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c115540 con 0x7efe0c10af10 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c1153c0 con 0x7efe0c10b860 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c1156c0 con 0x7efe0c1055d0 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe01400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c10af10 0x7efe0c19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe09e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 0x7efe0c19bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe09e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 0x7efe0c19bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:33810/0 (socket says 172.21.15.144:33810) 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe09e00640 1 -- 172.21.15.144:0/3310317078 learned_addr learned my addr 172.21.15.144:0/3310317078 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.654+0000 7efe0a800640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe0c10b860 0x7efe0c1a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe09e00640 1 -- 172.21.15.144:0/3310317078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe0c10b860 msgr2=0x7efe0c1a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:32.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe09e00640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe0c10b860 0x7efe0c1a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe09e00640 1 -- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c10af10 msgr2=0x7efe0c19c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:32.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe09e00640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c10af10 0x7efe0c19c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe09e00640 1 -- 172.21.15.144:0/3310317078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c1a1ec0 con 0x7efe0c1055d0 2024-07-26T11:00:32.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe09e00640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 0x7efe0c19bf50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00eb90 tx=0x7efdfc00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc004b20 con 0x7efe0c1055d0 2024-07-26T11:00:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdfc004cc0 con 0x7efe0c1055d0 2024-07-26T11:00:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c1a21b0 con 0x7efe0c1055d0 2024-07-26T11:00:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.655+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc010690 con 0x7efe0c1055d0 2024-07-26T11:00:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.656+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0c1a8910 con 0x7efe0c1055d0 2024-07-26T11:00:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.657+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0c1059e0 con 0x7efe0c1055d0 2024-07-26T11:00:32.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.659+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7efdfc005000 con 0x7efe0c1055d0 2024-07-26T11:00:32.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.661+0000 7efe03400640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efdf00b3ca0 0x7efdf00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:32.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.661+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7efdfc0e4910 con 0x7efe0c1055d0 2024-07-26T11:00:32.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.661+0000 7efe01400640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efdf00b3ca0 0x7efdf00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:32.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.662+0000 7efe01400640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efdf00b3ca0 0x7efdf00b61b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efdf402a9e0 tx=0x7efdf4005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:32.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.662+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efdfc055600 con 0x7efe0c1055d0 2024-07-26T11:00:32.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.863+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7efe0c02e0c0 con 0x7efe0c1055d0 2024-07-26T11:00:32.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.864+0000 7efe03400640 1 -- 172.21.15.144:0/3310317078 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efdfc0aba70 con 0x7efe0c1055d0 2024-07-26T11:00:32.867 INFO:teuthology.orchestra.run.smithi144.stdout:38654705776 2024-07-26T11:00:32.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.866+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efdf00b3ca0 msgr2=0x7efdf00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:32.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efdf00b3ca0 0x7efdf00b61b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efdf402a9e0 tx=0x7efdf4005a20 comp rx=0 tx=0).stop 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 msgr2=0x7efe0c19bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 0x7efe0c19bf50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00eb90 tx=0x7efdfc00ef90 comp rx=0 tx=0).stop 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 shutdown_connections 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efdf00b3ca0 0x7efdf00b61b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe0c10b860 0x7efe0c1a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c10af10 0x7efe0c19c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 --2- 172.21.15.144:0/3310317078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe0c1055d0 0x7efe0c19bf50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:32.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 >> 172.21.15.144:0/3310317078 conn(0x7efe0c101040 msgr2=0x7efe0c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:32.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 shutdown_connections 2024-07-26T11:00:32.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:32.867+0000 7efe0b200640 1 -- 172.21.15.144:0/3310317078 wait complete. 2024-07-26T11:00:32.927 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:32 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3310317078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T11:00:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:32 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3310317078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T11:00:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:32 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3310317078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T11:00:34.006 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:33 smithi144 ceph-mon[32129]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:00:34.134 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593805 got 184683593808 for osd.6 2024-07-26T11:00:34.134 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:33 smithi160 ceph-mon[36715]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:00:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:33 smithi183 ceph-mon[37273]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:00:34.969 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378722 got 81604378726 for osd.2 2024-07-26T11:00:34.969 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:35.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:34 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2940293123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T11:00:35.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542246 got 60129542253 for osd.1 2024-07-26T11:00:35.265 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:34 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2940293123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T11:00:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:34 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2940293123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T11:00:35.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 -- 172.21.15.144:0/1309674667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 msgr2=0x7fc424105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:35.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 --2- 172.21.15.144:0/1309674667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc424105bc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc414009ca0 tx=0x7fc41402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 -- 172.21.15.144:0/1309674667 shutdown_connections 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 --2- 172.21.15.144:0/1309674667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410ba40 0x7fc424112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 --2- 172.21.15.144:0/1309674667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410b0f0 0x7fc42410b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 --2- 172.21.15.144:0/1309674667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc424105bc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 -- 172.21.15.144:0/1309674667 >> 172.21.15.144:0/1309674667 conn(0x7fc424101240 msgr2=0x7fc424103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 -- 172.21.15.144:0/1309674667 shutdown_connections 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.992+0000 7fc42bc00640 1 -- 172.21.15.144:0/1309674667 wait complete. 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.993+0000 7fc42bc00640 1 Processor -- start 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.993+0000 7fc42bc00640 1 -- start start 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.993+0000 7fc42bc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc42419c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc42bc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410b0f0 0x7fc42419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:35.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc42bc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 0x7fc4241a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc42bc00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc424115740 con 0x7fc4241057b0 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc42bc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc4241155c0 con 0x7fc42410b0f0 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc42bc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc4241158c0 con 0x7fc42410ba40 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc42419c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 0x7fc4241a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 0x7fc4241a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:33830/0 (socket says 172.21.15.144:33830) 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc42419c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57574/0 (socket says 172.21.15.144:57574) 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 -- 172.21.15.144:0/2940293123 learned_addr learned my addr 172.21.15.144:0/2940293123 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc423e00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410b0f0 0x7fc42419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 -- 172.21.15.144:0/2940293123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410b0f0 msgr2=0x7fc42419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410b0f0 0x7fc42419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 -- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 msgr2=0x7fc42419c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:35.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc42419c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.994+0000 7fc429c00640 1 -- 172.21.15.144:0/2940293123 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4241a2140 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.995+0000 7fc429c00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 0x7fc4241a1a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9b0 tx=0x7fc41c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.995+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00cd80 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.995+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc41c0035f0 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.995+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4241a2430 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.995+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c010690 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.995+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241a8b90 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.996+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc424105bc0 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:33.999+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fc41c004020 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.000+0000 7fc421600640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc4080b3c20 0x7fc4080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.000+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc41c0e4510 con 0x7fc42410ba40 2024-07-26T11:00:35.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.000+0000 7fc429200640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc4080b3c20 0x7fc4080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:35.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.001+0000 7fc429200640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc4080b3c20 0x7fc4080b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc414006010 tx=0x7fc414009820 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:35.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.001+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc41c0ab670 con 0x7fc42410ba40 2024-07-26T11:00:35.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.219+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc42402e0c0 con 0x7fc42410ba40 2024-07-26T11:00:35.529 INFO:teuthology.orchestra.run.smithi144.stdout:111669149793 2024-07-26T11:00:35.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.220+0000 7fc421600640 1 -- 172.21.15.144:0/2940293123 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc41c0aaf70 con 0x7fc42410ba40 2024-07-26T11:00:35.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc4080b3c20 msgr2=0x7fc4080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc4080b3c20 0x7fc4080b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc414006010 tx=0x7fc414009820 comp rx=0 tx=0).stop 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 msgr2=0x7fc4241a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 0x7fc4241a1a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9b0 tx=0x7fc41c00ee70 comp rx=0 tx=0).stop 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 shutdown_connections 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc4080b3c20 0x7fc4080b6130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc42410ba40 0x7fc4241a1a10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc42410b0f0 0x7fc42419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 --2- 172.21.15.144:0/2940293123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4241057b0 0x7fc42419c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 >> 172.21.15.144:0/2940293123 conn(0x7fc424101240 msgr2=0x7fc424103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 shutdown_connections 2024-07-26T11:00:35.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:34.223+0000 7fc42bc00640 1 -- 172.21.15.144:0/2940293123 wait complete. 2024-07-26T11:00:35.818 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430276 got 206158430282 for osd.7 2024-07-26T11:00:35.819 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:36.047 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:35 smithi144 ceph-mon[32129]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:35 smithi160 ceph-mon[36715]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:35 smithi183 ceph-mon[37273]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:36.428 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874294 got 279172874298 for osd.10 2024-07-26T11:00:36.428 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:36.501 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:36.540 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790032 got 158913790037 for osd.5 2024-07-26T11:00:36.540 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 -- 172.21.15.144:0/648107870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa34010af30 msgr2=0x7fa34010b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 --2- 172.21.15.144:0/648107870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa34010af30 0x7fa34010b340 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa330009900 tx=0x7fa33002a400 comp rx=0 tx=0).stop 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 -- 172.21.15.144:0/648107870 shutdown_connections 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 --2- 172.21.15.144:0/648107870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010b880 0x7fa3401120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 --2- 172.21.15.144:0/648107870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa34010af30 0x7fa34010b340 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 --2- 172.21.15.144:0/648107870 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa3401055f0 0x7fa340105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 -- 172.21.15.144:0/648107870 >> 172.21.15.144:0/648107870 conn(0x7fa340101060 msgr2=0x7fa3401034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 -- 172.21.15.144:0/648107870 shutdown_connections 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.251+0000 7fa345e00640 1 -- 172.21.15.144:0/648107870 wait complete. 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.252+0000 7fa345e00640 1 Processor -- start 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.252+0000 7fa345e00640 1 -- start start 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3401055f0 0x7fa34019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 0x7fa34019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa34010b880 0x7fa3401a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa3401154f0 con 0x7fa34010af30 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa340115370 con 0x7fa34010b880 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa340115670 con 0x7fa3401055f0 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa33fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 0x7fa34019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:36.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa33fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 0x7fa34019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57592/0 (socket says 172.21.15.144:57592) 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa33fe00640 1 -- 172.21.15.144:0/2899129122 learned_addr learned my addr 172.21.15.144:0/2899129122 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa345400640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa34010b880 0x7fa3401a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.253+0000 7fa344a00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3401055f0 0x7fa34019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33fe00640 1 -- 172.21.15.144:0/2899129122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3401055f0 msgr2=0x7fa34019bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33fe00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3401055f0 0x7fa34019bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33fe00640 1 -- 172.21.15.144:0/2899129122 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa34010b880 msgr2=0x7fa3401a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33fe00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa34010b880 0x7fa3401a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33fe00640 1 -- 172.21.15.144:0/2899129122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3401a1e90 con 0x7fa34010af30 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa344a00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3401055f0 0x7fa34019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33fe00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 0x7fa34019c4c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa330006010 tx=0x7fa330004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33003f030 con 0x7fa34010af30 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stdout:137438953564 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.254+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa330003aa0 con 0x7fa34010af30 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.255+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3401a2120 con 0x7fa34010af30 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.255+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3401a88e0 con 0x7fa34010af30 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.255+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa330035070 con 0x7fa34010af30 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.256+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa340105a00 con 0x7fa34010af30 2024-07-26T11:00:36.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.259+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fa330042030 con 0x7fa34010af30 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.261+0000 7fa33d600640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3280b3c20 0x7fa3280b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.262+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa330104650 con 0x7fa34010af30 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.262+0000 7fa344a00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3280b3c20 0x7fa3280b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.263+0000 7fa344a00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3280b3c20 0x7fa3280b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa334005d00 tx=0x7fa33400b410 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.264+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa3300cb1c0 con 0x7fa34010af30 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.465+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa34010d630 con 0x7fa34010af30 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.468+0000 7fa33d600640 1 -- 172.21.15.144:0/2899129122 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa3300cf710 con 0x7fa34010af30 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3280b3c20 msgr2=0x7fa3280b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3280b3c20 0x7fa3280b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa334005d00 tx=0x7fa33400b410 comp rx=0 tx=0).stop 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 msgr2=0x7fa34019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 0x7fa34019c4c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa330006010 tx=0x7fa330004060 comp rx=0 tx=0).stop 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 shutdown_connections 2024-07-26T11:00:36.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3280b3c20 0x7fa3280b6130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa34010b880 0x7fa3401a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa34010af30 0x7fa34019c4c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 --2- 172.21.15.144:0/2899129122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3401055f0 0x7fa34019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 >> 172.21.15.144:0/2899129122 conn(0x7fa340101060 msgr2=0x7fa340103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 shutdown_connections 2024-07-26T11:00:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:36.471+0000 7fa345e00640 1 -- 172.21.15.144:0/2899129122 wait complete. 2024-07-26T11:00:37.135 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:36 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2899129122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T11:00:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:36 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2899129122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T11:00:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:36 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2899129122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T11:00:37.438 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705769 got 38654705776 for osd.0 2024-07-26T11:00:37.438 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:37.952 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149786 got 111669149793 for osd.3 2024-07-26T11:00:37.952 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:37 smithi144 ceph-mon[32129]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:37 smithi160 ceph-mon[36715]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:37 smithi183 ceph-mon[37273]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:38.422 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953558 got 137438953564 for osd.4 2024-07-26T11:00:38.422 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:40.309 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:40 smithi144 ceph-mon[32129]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:40 smithi183 ceph-mon[37273]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:40 smithi160 ceph-mon[36715]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.599+0000 7f00a0a00640 1 -- 172.21.15.144:0/2622390800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c10b860 msgr2=0x7f009c112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.599+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2622390800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c10b860 0x7f009c112090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0094009ca0 tx=0x7f009402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 -- 172.21.15.144:0/2622390800 shutdown_connections 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2622390800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c10b860 0x7f009c112090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2622390800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c10af10 0x7f009c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2622390800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c1055d0 0x7f009c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 -- 172.21.15.144:0/2622390800 >> 172.21.15.144:0/2622390800 conn(0x7f009c101040 msgr2=0x7f009c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:40.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 -- 172.21.15.144:0/2622390800 shutdown_connections 2024-07-26T11:00:40.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.600+0000 7f00a0a00640 1 -- 172.21.15.144:0/2622390800 wait complete. 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 Processor -- start 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 -- start start 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 0x7f009c19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c10af10 0x7f009c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c10b860 0x7f009c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f009c115530 con 0x7f009c1055d0 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1153b0 con 0x7f009c10af10 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.601+0000 7f00a0a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1156b0 con 0x7f009c10b860 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 0x7f009c19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:40.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 0x7f009c19bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38830/0 (socket says 172.21.15.144:38830) 2024-07-26T11:00:40.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 -- 172.21.15.144:0/2299200708 learned_addr learned my addr 172.21.15.144:0/2299200708 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:40.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009b200640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c10b860 0x7f009c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:40.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f0093e00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c10af10 0x7f009c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 -- 172.21.15.144:0/2299200708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c10b860 msgr2=0x7f009c1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c10b860 0x7f009c1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 -- 172.21.15.144:0/2299200708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c10af10 msgr2=0x7f009c19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c10af10 0x7f009c19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f009a800640 1 -- 172.21.15.144:0/2299200708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c1a1e50 con 0x7f009c1055d0 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.602+0000 7f0093e00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c10af10 0x7f009c19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.603+0000 7f009a800640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 0x7f009c19bee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f008c00e9f0 tx=0x7f008c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.603+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c00cd80 con 0x7f009c1055d0 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.603+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f008c0108e0 con 0x7f009c1055d0 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.603+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c018980 con 0x7f009c1055d0 2024-07-26T11:00:40.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.603+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c1a2140 con 0x7f009c1055d0 2024-07-26T11:00:40.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.603+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c1a88a0 con 0x7f009c1055d0 2024-07-26T11:00:40.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.605+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009c1059e0 con 0x7f009c1055d0 2024-07-26T11:00:40.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.610+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f008c010430 con 0x7f009c1055d0 2024-07-26T11:00:40.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.612+0000 7f0093400640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f00800b3ca0 0x7f00800b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:40.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.612+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f008c0e4900 con 0x7f009c1055d0 2024-07-26T11:00:40.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.612+0000 7f0093e00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f00800b3ca0 0x7f00800b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:40.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.612+0000 7f0093e00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f00800b3ca0 0x7f00800b61b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f009c104820 tx=0x7f0084005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:40.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.613+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f008c0ab410 con 0x7f009c1055d0 2024-07-26T11:00:40.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.815+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f009c02e0c0 con 0x7f009c1055d0 2024-07-26T11:00:40.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.815+0000 7f0093400640 1 -- 172.21.15.144:0/2299200708 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f008c0af960 con 0x7f009c1055d0 2024-07-26T11:00:40.818 INFO:teuthology.orchestra.run.smithi144.stdout:231928234056 2024-07-26T11:00:40.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f00800b3ca0 msgr2=0x7f00800b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:40.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f00800b3ca0 0x7f00800b61b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f009c104820 tx=0x7f0084005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:40.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 msgr2=0x7f009c19bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:40.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 0x7f009c19bee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f008c00e9f0 tx=0x7f008c00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 shutdown_connections 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f00800b3ca0 0x7f00800b61b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c10b860 0x7f009c1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f009c10af10 0x7f009c19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 --2- 172.21.15.144:0/2299200708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c1055d0 0x7f009c19bee0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 >> 172.21.15.144:0/2299200708 conn(0x7f009c101040 msgr2=0x7f009c10a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 shutdown_connections 2024-07-26T11:00:40.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:40.818+0000 7f00a0a00640 1 -- 172.21.15.144:0/2299200708 wait complete. 2024-07-26T11:00:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:41 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2299200708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T11:00:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:41 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2299200708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T11:00:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:41 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2299200708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T11:00:41.640 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234049 got 231928234056 for osd.8 2024-07-26T11:00:41.640 DEBUG:teuthology.parallel:result is None 2024-07-26T11:00:41.640 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T11:00:41.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T11:00:42.113 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:42 smithi183 ceph-mon[37273]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:42.450 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:42 smithi144 ceph-mon[32129]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:42 smithi160 ceph-mon[36715]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 -- 172.21.15.144:0/106985557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c10f940 msgr2=0x7ff03c111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 --2- 172.21.15.144:0/106985557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c10f940 0x7ff03c111d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff034009be0 tx=0x7ff03402a6d0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 -- 172.21.15.144:0/106985557 shutdown_connections 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 --2- 172.21.15.144:0/106985557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c10f940 0x7ff03c111d80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 --2- 172.21.15.144:0/106985557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff03c075b30 0x7ff03c10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 --2- 172.21.15.144:0/106985557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c0751e0 0x7ff03c0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.372+0000 7ff042000640 1 -- 172.21.15.144:0/106985557 >> 172.21.15.144:0/106985557 conn(0x7ff03c0fee70 msgr2=0x7ff03c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.373+0000 7ff042000640 1 -- 172.21.15.144:0/106985557 shutdown_connections 2024-07-26T11:00:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.373+0000 7ff042000640 1 -- 172.21.15.144:0/106985557 wait complete. 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 Processor -- start 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 -- start start 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff03c0751e0 0x7ff03c19bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c075b30 0x7ff03c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 0x7ff03c1a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1154d0 con 0x7ff03c0751e0 2024-07-26T11:00:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c115350 con 0x7ff03c075b30 2024-07-26T11:00:44.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.374+0000 7ff042000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c115650 con 0x7ff03c10f940 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff040c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff03c0751e0 0x7ff03c19bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 0x7ff03c1a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 0x7ff03c1a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:46418/0 (socket says 172.21.15.144:46418) 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 -- 172.21.15.144:0/1098641417 learned_addr learned my addr 172.21.15.144:0/1098641417 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff03be00640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c075b30 0x7ff03c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 -- 172.21.15.144:0/1098641417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c075b30 msgr2=0x7ff03c19c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c075b30 0x7ff03c19c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 -- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff03c0751e0 msgr2=0x7ff03c19bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff03c0751e0 0x7ff03c19bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 -- 172.21.15.144:0/1098641417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c1a1e90 con 0x7ff03c10f940 2024-07-26T11:00:44.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff03be00640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c075b30 0x7ff03c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.375+0000 7ff041600640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 0x7ff03c1a1760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff034004af0 tx=0x7ff034003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.376+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034041020 con 0x7ff03c10f940 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.376+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff034003e10 con 0x7ff03c10f940 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.376+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c1a2120 con 0x7ff03c10f940 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.376+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034036040 con 0x7ff03c10f940 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.376+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c1a88e0 con 0x7ff03c10f940 2024-07-26T11:00:44.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.377+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03c0766f0 con 0x7ff03c10f940 2024-07-26T11:00:44.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.379+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7ff034044020 con 0x7ff03c10f940 2024-07-26T11:00:44.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.381+0000 7ff039600640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff0240b3c20 0x7ff0240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:44.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.381+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7ff034104e90 con 0x7ff03c10f940 2024-07-26T11:00:44.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.381+0000 7ff040c00640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff0240b3c20 0x7ff0240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:44.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.382+0000 7ff040c00640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff0240b3c20 0x7ff0240b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff030009bd0 tx=0x7ff030005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:44.384 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:44 smithi144 ceph-mon[32129]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:44 smithi183 ceph-mon[37273]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:44.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.385+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff03404b020 con 0x7ff03c10f940 2024-07-26T11:00:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:44 smithi160 ceph-mon[36715]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:44.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.579+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff03c109b80 con 0x7ff0240b3c20 2024-07-26T11:00:44.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.587+0000 7ff039600640 1 -- 172.21.15.144:0/1098641417 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340475 (secure 0 0 0) 0x7ff03c109b80 con 0x7ff0240b3c20 2024-07-26T11:00:44.596 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:00:44.597 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.595+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff0240b3c20 msgr2=0x7ff0240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff0240b3c20 0x7ff0240b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff030009bd0 tx=0x7ff030005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 msgr2=0x7ff03c1a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 0x7ff03c1a1760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff034004af0 tx=0x7ff034003c20 comp rx=0 tx=0).stop 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 shutdown_connections 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff0240b3c20 0x7ff0240b6130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff03c10f940 0x7ff03c1a1760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff03c075b30 0x7ff03c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 --2- 172.21.15.144:0/1098641417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff03c0751e0 0x7ff03c19bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 >> 172.21.15.144:0/1098641417 conn(0x7ff03c0fee70 msgr2=0x7ff03c109470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 shutdown_connections 2024-07-26T11:00:44.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:44.596+0000 7ff042000640 1 -- 172.21.15.144:0/1098641417 wait complete. 2024-07-26T11:00:45.487 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-07-26T11:00:43.057946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662670,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":824,"num_read_kb":518,"num_write":302,"num_write_kb":2279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380092,"kb_used_data":14300,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561732,"statfs":{"total":1151940427776,"available":1151551213568,"internally_reserved":0,"allocated":14643200,"data_stored":8385968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005864"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210208+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:58:45.210208+0000","last_peered":"2024-07-26T10:58:45.210208+0000","last_clean":"2024-07-26T10:58:45.210208+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:58:45.210208+0000","last_undegraded":"2024-07-26T10:58:45.210208+0000","last_fullsized":"2024-07-26T10:58:45.210208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.965187+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T11:00:12.965187+0000","last_peered":"2024-07-26T11:00:12.965187+0000","last_clean":"2024-07-26T11:00:12.965187+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T11:00:12.965187+0000","last_undegraded":"2024-07-26T11:00:12.965187+0000","last_fullsized":"2024-07-26T11:00:12.965187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205284+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:58:45.205284+0000","last_peered":"2024-07-26T10:58:45.205284+0000","last_clean":"2024-07-26T10:58:45.205284+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:58:45.205284+0000","last_undegraded":"2024-07-26T10:58:45.205284+0000","last_fullsized":"2024-07-26T10:58:45.205284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211762+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:58:45.211762+0000","last_peered":"2024-07-26T10:58:45.211762+0000","last_clean":"2024-07-26T10:58:45.211762+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:58:45.211762+0000","last_undegraded":"2024-07-26T10:58:45.211762+0000","last_fullsized":"2024-07-26T10:58:45.211762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.995842+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T11:00:12.995842+0000","last_peered":"2024-07-26T11:00:12.995842+0000","last_clean":"2024-07-26T11:00:12.995842+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T11:00:12.995842+0000","last_undegraded":"2024-07-26T11:00:12.995842+0000","last_fullsized":"2024-07-26T11:00:12.995842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211982+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T10:58:45.211982+0000","last_peered":"2024-07-26T10:58:45.211982+0000","last_clean":"2024-07-26T10:58:45.211982+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T10:58:45.211982+0000","last_undegraded":"2024-07-26T10:58:45.211982+0000","last_fullsized":"2024-07-26T10:58:45.211982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211853+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:58:45.211853+0000","last_peered":"2024-07-26T10:58:45.211853+0000","last_clean":"2024-07-26T10:58:45.211853+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:58:45.211853+0000","last_undegraded":"2024-07-26T10:58:45.211853+0000","last_fullsized":"2024-07-26T10:58:45.211853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986457+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T11:00:12.986457+0000","last_peered":"2024-07-26T11:00:12.986457+0000","last_clean":"2024-07-26T11:00:12.986457+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T11:00:12.986457+0000","last_undegraded":"2024-07-26T11:00:12.986457+0000","last_fullsized":"2024-07-26T11:00:12.986457+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212336+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212336+0000","last_peered":"2024-07-26T10:58:45.212336+0000","last_clean":"2024-07-26T10:58:45.212336+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212336+0000","last_undegraded":"2024-07-26T10:58:45.212336+0000","last_fullsized":"2024-07-26T10:58:45.212336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395309+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395309+0000","last_peered":"2024-07-26T10:58:45.395309+0000","last_clean":"2024-07-26T10:58:45.395309+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395309+0000","last_undegraded":"2024-07-26T10:58:45.395309+0000","last_fullsized":"2024-07-26T10:58:45.395309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952593+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T11:00:12.952593+0000","last_peered":"2024-07-26T11:00:12.952593+0000","last_clean":"2024-07-26T11:00:12.952593+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T11:00:12.952593+0000","last_undegraded":"2024-07-26T11:00:12.952593+0000","last_fullsized":"2024-07-26T11:00:12.952593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212134+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212134+0000","last_peered":"2024-07-26T10:58:45.212134+0000","last_clean":"2024-07-26T10:58:45.212134+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212134+0000","last_undegraded":"2024-07-26T10:58:45.212134+0000","last_fullsized":"2024-07-26T10:58:45.212134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969521+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T11:00:12.969521+0000","last_peered":"2024-07-26T11:00:12.969521+0000","last_clean":"2024-07-26T11:00:12.969521+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T11:00:12.969521+0000","last_undegraded":"2024-07-26T11:00:12.969521+0000","last_fullsized":"2024-07-26T11:00:12.969521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211215+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:58:45.211215+0000","last_peered":"2024-07-26T10:58:45.211215+0000","last_clean":"2024-07-26T10:58:45.211215+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:58:45.211215+0000","last_undegraded":"2024-07-26T10:58:45.211215+0000","last_fullsized":"2024-07-26T10:58:45.211215+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212254+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:58:45.212254+0000","last_peered":"2024-07-26T10:58:45.212254+0000","last_clean":"2024-07-26T10:58:45.212254+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:58:45.212254+0000","last_undegraded":"2024-07-26T10:58:45.212254+0000","last_fullsized":"2024-07-26T10:58:45.212254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.997353+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T11:00:12.997353+0000","last_peered":"2024-07-26T11:00:12.997353+0000","last_clean":"2024-07-26T11:00:12.997353+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T11:00:12.997353+0000","last_undegraded":"2024-07-26T11:00:12.997353+0000","last_fullsized":"2024-07-26T11:00:12.997353+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205391+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T10:58:45.205391+0000","last_peered":"2024-07-26T10:58:45.205391+0000","last_clean":"2024-07-26T10:58:45.205391+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T10:58:45.205391+0000","last_undegraded":"2024-07-26T10:58:45.205391+0000","last_fullsized":"2024-07-26T10:58:45.205391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:02.756153+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:59:02.756153+0000","last_peered":"2024-07-26T10:59:02.756153+0000","last_clean":"2024-07-26T10:59:02.756153+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:59:02.756153+0000","last_undegraded":"2024-07-26T10:59:02.756153+0000","last_fullsized":"2024-07-26T10:59:02.756153+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.962814+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T11:00:12.962814+0000","last_peered":"2024-07-26T11:00:12.962814+0000","last_clean":"2024-07-26T11:00:12.962814+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T11:00:12.962814+0000","last_undegraded":"2024-07-26T11:00:12.962814+0000","last_fullsized":"2024-07-26T11:00:12.962814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210173+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:58:45.210173+0000","last_peered":"2024-07-26T10:58:45.210173+0000","last_clean":"2024-07-26T10:58:45.210173+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:58:45.210173+0000","last_undegraded":"2024-07-26T10:58:45.210173+0000","last_fullsized":"2024-07-26T10:58:45.210173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211643+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T10:58:45.211643+0000","last_peered":"2024-07-26T10:58:45.211643+0000","last_clean":"2024-07-26T10:58:45.211643+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T10:58:45.211643+0000","last_undegraded":"2024-07-26T10:58:45.211643+0000","last_fullsized":"2024-07-26T10:58:45.211643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.909039+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T11:00:12.909039+0000","last_peered":"2024-07-26T11:00:12.909039+0000","last_clean":"2024-07-26T11:00:12.909039+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T11:00:12.909039+0000","last_undegraded":"2024-07-26T11:00:12.909039+0000","last_fullsized":"2024-07-26T11:00:12.909039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394785+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:58:45.394785+0000","last_peered":"2024-07-26T10:58:45.394785+0000","last_clean":"2024-07-26T10:58:45.394785+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:58:45.394785+0000","last_undegraded":"2024-07-26T10:58:45.394785+0000","last_fullsized":"2024-07-26T10:58:45.394785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212686+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:58:45.212686+0000","last_peered":"2024-07-26T10:58:45.212686+0000","last_clean":"2024-07-26T10:58:45.212686+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:58:45.212686+0000","last_undegraded":"2024-07-26T10:58:45.212686+0000","last_fullsized":"2024-07-26T10:58:45.212686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.990650+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T11:00:12.990650+0000","last_peered":"2024-07-26T11:00:12.990650+0000","last_clean":"2024-07-26T11:00:12.990650+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T11:00:12.990650+0000","last_undegraded":"2024-07-26T11:00:12.990650+0000","last_fullsized":"2024-07-26T11:00:12.990650+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212754+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:58:45.212754+0000","last_peered":"2024-07-26T10:58:45.212754+0000","last_clean":"2024-07-26T10:58:45.212754+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:58:45.212754+0000","last_undegraded":"2024-07-26T10:58:45.212754+0000","last_fullsized":"2024-07-26T10:58:45.212754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205500+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:58:45.205500+0000","last_peered":"2024-07-26T10:58:45.205500+0000","last_clean":"2024-07-26T10:58:45.205500+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:58:45.205500+0000","last_undegraded":"2024-07-26T10:58:45.205500+0000","last_fullsized":"2024-07-26T10:58:45.205500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982080+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T11:00:12.982080+0000","last_peered":"2024-07-26T11:00:12.982080+0000","last_clean":"2024-07-26T11:00:12.982080+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T11:00:12.982080+0000","last_undegraded":"2024-07-26T11:00:12.982080+0000","last_fullsized":"2024-07-26T11:00:12.982080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:30.594697+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T10:59:30.594697+0000","last_peered":"2024-07-26T10:59:30.594697+0000","last_clean":"2024-07-26T10:59:30.594697+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T10:59:30.594697+0000","last_undegraded":"2024-07-26T10:59:30.594697+0000","last_fullsized":"2024-07-26T10:59:30.594697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208811+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:58:45.208811+0000","last_peered":"2024-07-26T10:58:45.208811+0000","last_clean":"2024-07-26T10:58:45.208811+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:58:45.208811+0000","last_undegraded":"2024-07-26T10:58:45.208811+0000","last_fullsized":"2024-07-26T10:58:45.208811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986960+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.986960+0000","last_peered":"2024-07-26T11:00:12.986960+0000","last_clean":"2024-07-26T11:00:12.986960+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T11:00:12.986960+0000","last_undegraded":"2024-07-26T11:00:12.986960+0000","last_fullsized":"2024-07-26T11:00:12.986960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211817+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:58:45.211817+0000","last_peered":"2024-07-26T10:58:45.211817+0000","last_clean":"2024-07-26T10:58:45.211817+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:58:45.211817+0000","last_undegraded":"2024-07-26T10:58:45.211817+0000","last_fullsized":"2024-07-26T10:58:45.211817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395714+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:58:45.395714+0000","last_peered":"2024-07-26T10:58:45.395714+0000","last_clean":"2024-07-26T10:58:45.395714+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:58:45.395714+0000","last_undegraded":"2024-07-26T10:58:45.395714+0000","last_fullsized":"2024-07-26T10:58:45.395714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951920+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.951920+0000","last_peered":"2024-07-26T11:00:12.951920+0000","last_clean":"2024-07-26T11:00:12.951920+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T11:00:12.951920+0000","last_undegraded":"2024-07-26T11:00:12.951920+0000","last_fullsized":"2024-07-26T11:00:12.951920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208937+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:58:45.208937+0000","last_peered":"2024-07-26T10:58:45.208937+0000","last_clean":"2024-07-26T10:58:45.208937+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:58:45.208937+0000","last_undegraded":"2024-07-26T10:58:45.208937+0000","last_fullsized":"2024-07-26T10:58:45.208937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211173+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:58:45.211173+0000","last_peered":"2024-07-26T10:58:45.211173+0000","last_clean":"2024-07-26T10:58:45.211173+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:58:45.211173+0000","last_undegraded":"2024-07-26T10:58:45.211173+0000","last_fullsized":"2024-07-26T10:58:45.211173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.976047+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T11:00:12.976047+0000","last_peered":"2024-07-26T11:00:12.976047+0000","last_clean":"2024-07-26T11:00:12.976047+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T11:00:12.976047+0000","last_undegraded":"2024-07-26T11:00:12.976047+0000","last_fullsized":"2024-07-26T11:00:12.976047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210980+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:58:45.210980+0000","last_peered":"2024-07-26T10:58:45.210980+0000","last_clean":"2024-07-26T10:58:45.210980+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:58:45.210980+0000","last_undegraded":"2024-07-26T10:58:45.210980+0000","last_fullsized":"2024-07-26T10:58:45.210980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396866+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:58:45.396866+0000","last_peered":"2024-07-26T10:58:45.396866+0000","last_clean":"2024-07-26T10:58:45.396866+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:58:45.396866+0000","last_undegraded":"2024-07-26T10:58:45.396866+0000","last_fullsized":"2024-07-26T10:58:45.396866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950069+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T11:00:12.950069+0000","last_peered":"2024-07-26T11:00:12.950069+0000","last_clean":"2024-07-26T11:00:12.950069+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T11:00:12.950069+0000","last_undegraded":"2024-07-26T11:00:12.950069+0000","last_fullsized":"2024-07-26T11:00:12.950069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":171,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:41.450304+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T11:00:41.450304+0000","last_peered":"2024-07-26T11:00:41.450304+0000","last_clean":"2024-07-26T11:00:41.450304+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T11:00:41.450304+0000","last_undegraded":"2024-07-26T11:00:41.450304+0000","last_fullsized":"2024-07-26T11:00:41.450304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397096+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:58:45.397096+0000","last_peered":"2024-07-26T10:58:45.397096+0000","last_clean":"2024-07-26T10:58:45.397096+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:58:45.397096+0000","last_undegraded":"2024-07-26T10:58:45.397096+0000","last_fullsized":"2024-07-26T10:58:45.397096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.983082+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T11:00:12.983082+0000","last_peered":"2024-07-26T11:00:12.983082+0000","last_clean":"2024-07-26T11:00:12.983082+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T11:00:12.983082+0000","last_undegraded":"2024-07-26T11:00:12.983082+0000","last_fullsized":"2024-07-26T11:00:12.983082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210600+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:58:45.210600+0000","last_peered":"2024-07-26T10:58:45.210600+0000","last_clean":"2024-07-26T10:58:45.210600+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:58:45.210600+0000","last_undegraded":"2024-07-26T10:58:45.210600+0000","last_fullsized":"2024-07-26T10:58:45.210600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.399431+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:45.399431+0000","last_peered":"2024-07-26T10:58:45.399431+0000","last_clean":"2024-07-26T10:58:45.399431+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:45.399431+0000","last_undegraded":"2024-07-26T10:58:45.399431+0000","last_fullsized":"2024-07-26T10:58:45.399431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.971525+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T11:00:12.971525+0000","last_peered":"2024-07-26T11:00:12.971525+0000","last_clean":"2024-07-26T11:00:12.971525+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T11:00:12.971525+0000","last_undegraded":"2024-07-26T11:00:12.971525+0000","last_fullsized":"2024-07-26T11:00:12.971525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212762+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:58:45.212762+0000","last_peered":"2024-07-26T10:58:45.212762+0000","last_clean":"2024-07-26T10:58:45.212762+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:58:45.212762+0000","last_undegraded":"2024-07-26T10:58:45.212762+0000","last_fullsized":"2024-07-26T10:58:45.212762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210654+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210654+0000","last_peered":"2024-07-26T10:58:45.210654+0000","last_clean":"2024-07-26T10:58:45.210654+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210654+0000","last_undegraded":"2024-07-26T10:58:45.210654+0000","last_fullsized":"2024-07-26T10:58:45.210654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:55.070655+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:58:55.070655+0000","last_peered":"2024-07-26T10:58:55.070655+0000","last_clean":"2024-07-26T10:58:55.070655+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:58:55.070655+0000","last_undegraded":"2024-07-26T10:58:55.070655+0000","last_fullsized":"2024-07-26T10:58:55.070655+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955554+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T11:00:12.955554+0000","last_peered":"2024-07-26T11:00:12.955554+0000","last_clean":"2024-07-26T11:00:12.955554+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T11:00:12.955554+0000","last_undegraded":"2024-07-26T11:00:12.955554+0000","last_fullsized":"2024-07-26T11:00:12.955554+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205872+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:58:45.205872+0000","last_peered":"2024-07-26T10:58:45.205872+0000","last_clean":"2024-07-26T10:58:45.205872+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:58:45.205872+0000","last_undegraded":"2024-07-26T10:58:45.205872+0000","last_fullsized":"2024-07-26T10:58:45.205872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394373+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:58:45.394373+0000","last_peered":"2024-07-26T10:58:45.394373+0000","last_clean":"2024-07-26T10:58:45.394373+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:58:45.394373+0000","last_undegraded":"2024-07-26T10:58:45.394373+0000","last_fullsized":"2024-07-26T10:58:45.394373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980621+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T11:00:12.980621+0000","last_peered":"2024-07-26T11:00:12.980621+0000","last_clean":"2024-07-26T11:00:12.980621+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T11:00:12.980621+0000","last_undegraded":"2024-07-26T11:00:12.980621+0000","last_fullsized":"2024-07-26T11:00:12.980621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396426+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:58:45.396426+0000","last_peered":"2024-07-26T10:58:45.396426+0000","last_clean":"2024-07-26T10:58:45.396426+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:58:45.396426+0000","last_undegraded":"2024-07-26T10:58:45.396426+0000","last_fullsized":"2024-07-26T10:58:45.396426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658938+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:58:45.658938+0000","last_peered":"2024-07-26T10:58:45.658938+0000","last_clean":"2024-07-26T10:58:45.658938+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:58:45.658938+0000","last_undegraded":"2024-07-26T10:58:45.658938+0000","last_fullsized":"2024-07-26T10:58:45.658938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954729+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T11:00:12.954729+0000","last_peered":"2024-07-26T11:00:12.954729+0000","last_clean":"2024-07-26T11:00:12.954729+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T11:00:12.954729+0000","last_undegraded":"2024-07-26T11:00:12.954729+0000","last_fullsized":"2024-07-26T11:00:12.954729+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397589+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:58:45.397589+0000","last_peered":"2024-07-26T10:58:45.397589+0000","last_clean":"2024-07-26T10:58:45.397589+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:58:45.397589+0000","last_undegraded":"2024-07-26T10:58:45.397589+0000","last_fullsized":"2024-07-26T10:58:45.397589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395573+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:58:45.395573+0000","last_peered":"2024-07-26T10:58:45.395573+0000","last_clean":"2024-07-26T10:58:45.395573+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:58:45.395573+0000","last_undegraded":"2024-07-26T10:58:45.395573+0000","last_fullsized":"2024-07-26T10:58:45.395573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945788+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T11:00:12.945788+0000","last_peered":"2024-07-26T11:00:12.945788+0000","last_clean":"2024-07-26T11:00:12.945788+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T11:00:12.945788+0000","last_undegraded":"2024-07-26T11:00:12.945788+0000","last_fullsized":"2024-07-26T11:00:12.945788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211738+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:58:45.211738+0000","last_peered":"2024-07-26T10:58:45.211738+0000","last_clean":"2024-07-26T10:58:45.211738+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:58:45.211738+0000","last_undegraded":"2024-07-26T10:58:45.211738+0000","last_fullsized":"2024-07-26T10:58:45.211738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395592+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:58:45.395592+0000","last_peered":"2024-07-26T10:58:45.395592+0000","last_clean":"2024-07-26T10:58:45.395592+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:58:45.395592+0000","last_undegraded":"2024-07-26T10:58:45.395592+0000","last_fullsized":"2024-07-26T10:58:45.395592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942946+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T11:00:12.942946+0000","last_peered":"2024-07-26T11:00:12.942946+0000","last_clean":"2024-07-26T11:00:12.942946+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T11:00:12.942946+0000","last_undegraded":"2024-07-26T11:00:12.942946+0000","last_fullsized":"2024-07-26T11:00:12.942946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211049+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:58:45.211049+0000","last_peered":"2024-07-26T10:58:45.211049+0000","last_clean":"2024-07-26T10:58:45.211049+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:58:45.211049+0000","last_undegraded":"2024-07-26T10:58:45.211049+0000","last_fullsized":"2024-07-26T10:58:45.211049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397349+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397349+0000","last_peered":"2024-07-26T10:58:45.397349+0000","last_clean":"2024-07-26T10:58:45.397349+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397349+0000","last_undegraded":"2024-07-26T10:58:45.397349+0000","last_fullsized":"2024-07-26T10:58:45.397349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949820+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T11:00:12.949820+0000","last_peered":"2024-07-26T11:00:12.949820+0000","last_clean":"2024-07-26T11:00:12.949820+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T11:00:12.949820+0000","last_undegraded":"2024-07-26T11:00:12.949820+0000","last_fullsized":"2024-07-26T11:00:12.949820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":174,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:40.448508+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T11:00:40.448508+0000","last_peered":"2024-07-26T11:00:40.448508+0000","last_clean":"2024-07-26T11:00:40.448508+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T11:00:40.448508+0000","last_undegraded":"2024-07-26T11:00:40.448508+0000","last_fullsized":"2024-07-26T11:00:40.448508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208214+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T10:58:45.208214+0000","last_peered":"2024-07-26T10:58:45.208214+0000","last_clean":"2024-07-26T10:58:45.208214+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T10:58:45.208214+0000","last_undegraded":"2024-07-26T10:58:45.208214+0000","last_fullsized":"2024-07-26T10:58:45.208214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987813+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T11:00:12.987813+0000","last_peered":"2024-07-26T11:00:12.987813+0000","last_clean":"2024-07-26T11:00:12.987813+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T11:00:12.987813+0000","last_undegraded":"2024-07-26T11:00:12.987813+0000","last_fullsized":"2024-07-26T11:00:12.987813+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395671+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:58:45.395671+0000","last_peered":"2024-07-26T10:58:45.395671+0000","last_clean":"2024-07-26T10:58:45.395671+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:58:45.395671+0000","last_undegraded":"2024-07-26T10:58:45.395671+0000","last_fullsized":"2024-07-26T10:58:45.395671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395441+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:58:45.395441+0000","last_peered":"2024-07-26T10:58:45.395441+0000","last_clean":"2024-07-26T10:58:45.395441+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:58:45.395441+0000","last_undegraded":"2024-07-26T10:58:45.395441+0000","last_fullsized":"2024-07-26T10:58:45.395441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.984849+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T11:00:12.984849+0000","last_peered":"2024-07-26T11:00:12.984849+0000","last_clean":"2024-07-26T11:00:12.984849+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T11:00:12.984849+0000","last_undegraded":"2024-07-26T11:00:12.984849+0000","last_fullsized":"2024-07-26T11:00:12.984849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205619+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:58:45.205619+0000","last_peered":"2024-07-26T10:58:45.205619+0000","last_clean":"2024-07-26T10:58:45.205619+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:58:45.205619+0000","last_undegraded":"2024-07-26T10:58:45.205619+0000","last_fullsized":"2024-07-26T10:58:45.205619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954477+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T11:00:12.954477+0000","last_peered":"2024-07-26T11:00:12.954477+0000","last_clean":"2024-07-26T11:00:12.954477+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T11:00:12.954477+0000","last_undegraded":"2024-07-26T11:00:12.954477+0000","last_fullsized":"2024-07-26T11:00:12.954477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":329,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:39.378597+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T11:00:39.378597+0000","last_peered":"2024-07-26T11:00:39.378597+0000","last_clean":"2024-07-26T11:00:39.378597+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T11:00:39.378597+0000","last_undegraded":"2024-07-26T11:00:39.378597+0000","last_fullsized":"2024-07-26T11:00:39.378597+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67926,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":247,"num_read_kb":190,"num_write":38,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208081+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:58:45.208081+0000","last_peered":"2024-07-26T10:58:45.208081+0000","last_clean":"2024-07-26T10:58:45.208081+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:58:45.208081+0000","last_undegraded":"2024-07-26T10:58:45.208081+0000","last_fullsized":"2024-07-26T10:58:45.208081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951888+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T11:00:12.951888+0000","last_peered":"2024-07-26T11:00:12.951888+0000","last_clean":"2024-07-26T11:00:12.951888+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T11:00:12.951888+0000","last_undegraded":"2024-07-26T11:00:12.951888+0000","last_fullsized":"2024-07-26T11:00:12.951888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396347+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:58:45.396347+0000","last_peered":"2024-07-26T10:58:45.396347+0000","last_clean":"2024-07-26T10:58:45.396347+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:58:45.396347+0000","last_undegraded":"2024-07-26T10:58:45.396347+0000","last_fullsized":"2024-07-26T10:58:45.396347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212446+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:58:45.212446+0000","last_peered":"2024-07-26T10:58:45.212446+0000","last_clean":"2024-07-26T10:58:45.212446+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:58:45.212446+0000","last_undegraded":"2024-07-26T10:58:45.212446+0000","last_fullsized":"2024-07-26T10:58:45.212446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987806+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T11:00:12.987806+0000","last_peered":"2024-07-26T11:00:12.987806+0000","last_clean":"2024-07-26T11:00:12.987806+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T11:00:12.987806+0000","last_undegraded":"2024-07-26T11:00:12.987806+0000","last_fullsized":"2024-07-26T11:00:12.987806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211576+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:58:45.211576+0000","last_peered":"2024-07-26T10:58:45.211576+0000","last_clean":"2024-07-26T10:58:45.211576+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:58:45.211576+0000","last_undegraded":"2024-07-26T10:58:45.211576+0000","last_fullsized":"2024-07-26T10:58:45.211576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207082+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:58:45.207082+0000","last_peered":"2024-07-26T10:58:45.207082+0000","last_clean":"2024-07-26T10:58:45.207082+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:58:45.207082+0000","last_undegraded":"2024-07-26T10:58:45.207082+0000","last_fullsized":"2024-07-26T10:58:45.207082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950324+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T11:00:12.950324+0000","last_peered":"2024-07-26T11:00:12.950324+0000","last_clean":"2024-07-26T11:00:12.950324+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T11:00:12.950324+0000","last_undegraded":"2024-07-26T11:00:12.950324+0000","last_fullsized":"2024-07-26T11:00:12.950324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":166,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:38.376885+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T11:00:38.376885+0000","last_peered":"2024-07-26T11:00:38.376885+0000","last_clean":"2024-07-26T11:00:38.376885+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T11:00:38.376885+0000","last_undegraded":"2024-07-26T11:00:38.376885+0000","last_fullsized":"2024-07-26T11:00:38.376885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209023+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:58:45.209023+0000","last_peered":"2024-07-26T10:58:45.209023+0000","last_clean":"2024-07-26T10:58:45.209023+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:58:45.209023+0000","last_undegraded":"2024-07-26T10:58:45.209023+0000","last_fullsized":"2024-07-26T10:58:45.209023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659120+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:58:45.659120+0000","last_peered":"2024-07-26T10:58:45.659120+0000","last_clean":"2024-07-26T10:58:45.659120+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:58:45.659120+0000","last_undegraded":"2024-07-26T10:58:45.659120+0000","last_fullsized":"2024-07-26T10:58:45.659120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949031+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T11:00:12.949031+0000","last_peered":"2024-07-26T11:00:12.949031+0000","last_clean":"2024-07-26T11:00:12.949031+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T11:00:12.949031+0000","last_undegraded":"2024-07-26T11:00:12.949031+0000","last_fullsized":"2024-07-26T11:00:12.949031+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395737+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:58:45.395737+0000","last_peered":"2024-07-26T10:58:45.395737+0000","last_clean":"2024-07-26T10:58:45.395737+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:58:45.395737+0000","last_undegraded":"2024-07-26T10:58:45.395737+0000","last_fullsized":"2024-07-26T10:58:45.395737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395949+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:58:45.395949+0000","last_peered":"2024-07-26T10:58:45.395949+0000","last_clean":"2024-07-26T10:58:45.395949+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:58:45.395949+0000","last_undegraded":"2024-07-26T10:58:45.395949+0000","last_fullsized":"2024-07-26T10:58:45.395949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988233+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T11:00:12.988233+0000","last_peered":"2024-07-26T11:00:12.988233+0000","last_clean":"2024-07-26T11:00:12.988233+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T11:00:12.988233+0000","last_undegraded":"2024-07-26T11:00:12.988233+0000","last_fullsized":"2024-07-26T11:00:12.988233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396116+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:58:45.396116+0000","last_peered":"2024-07-26T10:58:45.396116+0000","last_clean":"2024-07-26T10:58:45.396116+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:58:45.396116+0000","last_undegraded":"2024-07-26T10:58:45.396116+0000","last_fullsized":"2024-07-26T10:58:45.396116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395653+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:58:45.395653+0000","last_peered":"2024-07-26T10:58:45.395653+0000","last_clean":"2024-07-26T10:58:45.395653+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:58:45.395653+0000","last_undegraded":"2024-07-26T10:58:45.395653+0000","last_fullsized":"2024-07-26T10:58:45.395653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949369+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T11:00:12.949369+0000","last_peered":"2024-07-26T11:00:12.949369+0000","last_clean":"2024-07-26T11:00:12.949369+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T11:00:12.949369+0000","last_undegraded":"2024-07-26T11:00:12.949369+0000","last_fullsized":"2024-07-26T11:00:12.949369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205546+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205546+0000","last_peered":"2024-07-26T10:58:45.205546+0000","last_clean":"2024-07-26T10:58:45.205546+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205546+0000","last_undegraded":"2024-07-26T10:58:45.205546+0000","last_fullsized":"2024-07-26T10:58:45.205546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:37.376182+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T11:00:37.376182+0000","last_peered":"2024-07-26T11:00:37.376182+0000","last_clean":"2024-07-26T11:00:37.376182+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T11:00:37.376182+0000","last_undegraded":"2024-07-26T11:00:37.376182+0000","last_fullsized":"2024-07-26T11:00:37.376182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.938191+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T11:00:12.938191+0000","last_peered":"2024-07-26T11:00:12.938191+0000","last_clean":"2024-07-26T11:00:12.938191+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T11:00:12.938191+0000","last_undegraded":"2024-07-26T11:00:12.938191+0000","last_fullsized":"2024-07-26T11:00:12.938191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205755+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205755+0000","last_peered":"2024-07-26T10:58:45.205755+0000","last_clean":"2024-07-26T10:58:45.205755+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205755+0000","last_undegraded":"2024-07-26T10:58:45.205755+0000","last_fullsized":"2024-07-26T10:58:45.205755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942683+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T11:00:12.942683+0000","last_peered":"2024-07-26T11:00:12.942683+0000","last_clean":"2024-07-26T11:00:12.942683+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T11:00:12.942683+0000","last_undegraded":"2024-07-26T11:00:12.942683+0000","last_fullsized":"2024-07-26T11:00:12.942683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945771+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T11:00:12.945771+0000","last_peered":"2024-07-26T11:00:12.945771+0000","last_clean":"2024-07-26T11:00:12.945771+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T11:00:12.945771+0000","last_undegraded":"2024-07-26T11:00:12.945771+0000","last_fullsized":"2024-07-26T11:00:12.945771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.975117+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T11:00:12.975117+0000","last_peered":"2024-07-26T11:00:12.975117+0000","last_clean":"2024-07-26T11:00:12.975117+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T11:00:12.975117+0000","last_undegraded":"2024-07-26T11:00:12.975117+0000","last_fullsized":"2024-07-26T11:00:12.975117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992678+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T11:00:12.992678+0000","last_peered":"2024-07-26T11:00:12.992678+0000","last_clean":"2024-07-26T11:00:12.992678+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T11:00:12.992678+0000","last_undegraded":"2024-07-26T11:00:12.992678+0000","last_fullsized":"2024-07-26T11:00:12.992678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.996426+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T11:00:12.996426+0000","last_peered":"2024-07-26T11:00:12.996426+0000","last_clean":"2024-07-26T11:00:12.996426+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T11:00:12.996426+0000","last_undegraded":"2024-07-26T11:00:12.996426+0000","last_fullsized":"2024-07-26T11:00:12.996426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.940149+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T11:00:12.940149+0000","last_peered":"2024-07-26T11:00:12.940149+0000","last_clean":"2024-07-26T11:00:12.940149+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T11:00:12.940149+0000","last_undegraded":"2024-07-26T11:00:12.940149+0000","last_fullsized":"2024-07-26T11:00:12.940149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.941781+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T11:00:12.941781+0000","last_peered":"2024-07-26T11:00:12.941781+0000","last_clean":"2024-07-26T11:00:12.941781+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T11:00:12.941781+0000","last_undegraded":"2024-07-26T11:00:12.941781+0000","last_fullsized":"2024-07-26T11:00:12.941781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.973121+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T11:00:12.973121+0000","last_peered":"2024-07-26T11:00:12.973121+0000","last_clean":"2024-07-26T11:00:12.973121+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T11:00:12.973121+0000","last_undegraded":"2024-07-26T11:00:12.973121+0000","last_fullsized":"2024-07-26T11:00:12.973121+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967499+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T11:00:12.967499+0000","last_peered":"2024-07-26T11:00:12.967499+0000","last_clean":"2024-07-26T11:00:12.967499+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T11:00:12.967499+0000","last_undegraded":"2024-07-26T11:00:12.967499+0000","last_fullsized":"2024-07-26T11:00:12.967499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981644+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T11:00:12.981644+0000","last_peered":"2024-07-26T11:00:12.981644+0000","last_clean":"2024-07-26T11:00:12.981644+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T11:00:12.981644+0000","last_undegraded":"2024-07-26T11:00:12.981644+0000","last_fullsized":"2024-07-26T11:00:12.981644+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981194+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T11:00:12.981194+0000","last_peered":"2024-07-26T11:00:12.981194+0000","last_clean":"2024-07-26T11:00:12.981194+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T11:00:12.981194+0000","last_undegraded":"2024-07-26T11:00:12.981194+0000","last_fullsized":"2024-07-26T11:00:12.981194+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.999046+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T11:00:12.999046+0000","last_peered":"2024-07-26T11:00:12.999046+0000","last_clean":"2024-07-26T11:00:12.999046+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T11:00:12.999046+0000","last_undegraded":"2024-07-26T11:00:12.999046+0000","last_fullsized":"2024-07-26T11:00:12.999046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946601+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T11:00:12.946601+0000","last_peered":"2024-07-26T11:00:12.946601+0000","last_clean":"2024-07-26T11:00:12.946601+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T11:00:12.946601+0000","last_undegraded":"2024-07-26T11:00:12.946601+0000","last_fullsized":"2024-07-26T11:00:12.946601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947146+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T11:00:12.947146+0000","last_peered":"2024-07-26T11:00:12.947146+0000","last_clean":"2024-07-26T11:00:12.947146+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T11:00:12.947146+0000","last_undegraded":"2024-07-26T11:00:12.947146+0000","last_fullsized":"2024-07-26T11:00:12.947146+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945412+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T11:00:12.945412+0000","last_peered":"2024-07-26T11:00:12.945412+0000","last_clean":"2024-07-26T11:00:12.945412+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T11:00:12.945412+0000","last_undegraded":"2024-07-26T11:00:12.945412+0000","last_fullsized":"2024-07-26T11:00:12.945412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974857+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T11:00:12.974857+0000","last_peered":"2024-07-26T11:00:12.974857+0000","last_clean":"2024-07-26T11:00:12.974857+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T11:00:12.974857+0000","last_undegraded":"2024-07-26T11:00:12.974857+0000","last_fullsized":"2024-07-26T11:00:12.974857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992807+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T11:00:12.992807+0000","last_peered":"2024-07-26T11:00:12.992807+0000","last_clean":"2024-07-26T11:00:12.992807+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T11:00:12.992807+0000","last_undegraded":"2024-07-26T11:00:12.992807+0000","last_fullsized":"2024-07-26T11:00:12.992807+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992005+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T11:00:12.992005+0000","last_peered":"2024-07-26T11:00:12.992005+0000","last_clean":"2024-07-26T11:00:12.992005+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T11:00:12.992005+0000","last_undegraded":"2024-07-26T11:00:12.992005+0000","last_fullsized":"2024-07-26T11:00:12.992005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994336+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T11:00:12.994336+0000","last_peered":"2024-07-26T11:00:12.994336+0000","last_clean":"2024-07-26T11:00:12.994336+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T11:00:12.994336+0000","last_undegraded":"2024-07-26T11:00:12.994336+0000","last_fullsized":"2024-07-26T11:00:12.994336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987691+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T11:00:12.987691+0000","last_peered":"2024-07-26T11:00:12.987691+0000","last_clean":"2024-07-26T11:00:12.987691+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T11:00:12.987691+0000","last_undegraded":"2024-07-26T11:00:12.987691+0000","last_fullsized":"2024-07-26T11:00:12.987691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951850+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T11:00:12.951850+0000","last_peered":"2024-07-26T11:00:12.951850+0000","last_clean":"2024-07-26T11:00:12.951850+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T11:00:12.951850+0000","last_undegraded":"2024-07-26T11:00:12.951850+0000","last_fullsized":"2024-07-26T11:00:12.951850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955325+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T11:00:12.955325+0000","last_peered":"2024-07-26T11:00:12.955325+0000","last_clean":"2024-07-26T11:00:12.955325+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T11:00:12.955325+0000","last_undegraded":"2024-07-26T11:00:12.955325+0000","last_fullsized":"2024-07-26T11:00:12.955325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.948540+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T11:00:12.948540+0000","last_peered":"2024-07-26T11:00:12.948540+0000","last_clean":"2024-07-26T11:00:12.948540+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T11:00:12.948540+0000","last_undegraded":"2024-07-26T11:00:12.948540+0000","last_fullsized":"2024-07-26T11:00:12.948540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945953+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T11:00:12.945953+0000","last_peered":"2024-07-26T11:00:12.945953+0000","last_clean":"2024-07-26T11:00:12.945953+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T11:00:12.945953+0000","last_undegraded":"2024-07-26T11:00:12.945953+0000","last_fullsized":"2024-07-26T11:00:12.945953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980776+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T11:00:12.980776+0000","last_peered":"2024-07-26T11:00:12.980776+0000","last_clean":"2024-07-26T11:00:12.980776+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T11:00:12.980776+0000","last_undegraded":"2024-07-26T11:00:12.980776+0000","last_fullsized":"2024-07-26T11:00:12.980776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950356+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T11:00:12.950356+0000","last_peered":"2024-07-26T11:00:12.950356+0000","last_clean":"2024-07-26T11:00:12.950356+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T11:00:12.950356+0000","last_undegraded":"2024-07-26T11:00:12.950356+0000","last_fullsized":"2024-07-26T11:00:12.950356+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946913+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T11:00:12.946913+0000","last_peered":"2024-07-26T11:00:12.946913+0000","last_clean":"2024-07-26T11:00:12.946913+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T11:00:12.946913+0000","last_undegraded":"2024-07-26T11:00:12.946913+0000","last_fullsized":"2024-07-26T11:00:12.946913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967346+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T11:00:12.967346+0000","last_peered":"2024-07-26T11:00:12.967346+0000","last_clean":"2024-07-26T11:00:12.967346+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T11:00:12.967346+0000","last_undegraded":"2024-07-26T11:00:12.967346+0000","last_fullsized":"2024-07-26T11:00:12.967346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951202+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T11:00:12.951202+0000","last_peered":"2024-07-26T11:00:12.951202+0000","last_clean":"2024-07-26T11:00:12.951202+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T11:00:12.951202+0000","last_undegraded":"2024-07-26T11:00:12.951202+0000","last_fullsized":"2024-07-26T11:00:12.951202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989483+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T11:00:12.989483+0000","last_peered":"2024-07-26T11:00:12.989483+0000","last_clean":"2024-07-26T11:00:12.989483+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T11:00:12.989483+0000","last_undegraded":"2024-07-26T11:00:12.989483+0000","last_fullsized":"2024-07-26T11:00:12.989483+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989961+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T11:00:12.989961+0000","last_peered":"2024-07-26T11:00:12.989961+0000","last_clean":"2024-07-26T11:00:12.989961+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T11:00:12.989961+0000","last_undegraded":"2024-07-26T11:00:12.989961+0000","last_fullsized":"2024-07-26T11:00:12.989961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974704+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T11:00:12.974704+0000","last_peered":"2024-07-26T11:00:12.974704+0000","last_clean":"2024-07-26T11:00:12.974704+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T11:00:12.974704+0000","last_undegraded":"2024-07-26T11:00:12.974704+0000","last_fullsized":"2024-07-26T11:00:12.974704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72068,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":743,"num_read_kb":449,"num_write":177,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":216204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678072,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874302,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1199087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037827,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234057,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430286,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593812,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790041,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378729,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":599145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542255,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":599039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:00:45.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T11:00:45.985 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:46 smithi144 ceph-mon[32129]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:00:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:46 smithi144 ceph-mon[32129]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:46 smithi160 ceph-mon[36715]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:00:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:46 smithi160 ceph-mon[36715]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:46 smithi183 ceph-mon[37273]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:00:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:46 smithi183 ceph-mon[37273]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.275+0000 7f0387200640 1 -- 172.21.15.144:0/405933611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 msgr2=0x7f038810b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.275+0000 7f0387200640 1 --2- 172.21.15.144:0/405933611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 0x7f038810b310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0374009900 tx=0x7f037402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 -- 172.21.15.144:0/405933611 shutdown_connections 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 --2- 172.21.15.144:0/405933611 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f038810b850 0x7f0388112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 --2- 172.21.15.144:0/405933611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 0x7f038810b310 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 --2- 172.21.15.144:0/405933611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f03881059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 -- 172.21.15.144:0/405933611 >> 172.21.15.144:0/405933611 conn(0x7f0388101010 msgr2=0x7f0388103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:48.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 -- 172.21.15.144:0/405933611 shutdown_connections 2024-07-26T11:00:48.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.276+0000 7f0387200640 1 -- 172.21.15.144:0/405933611 wait complete. 2024-07-26T11:00:48.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.277+0000 7f0387200640 1 Processor -- start 2024-07-26T11:00:48.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.277+0000 7f0387200640 1 -- start start 2024-07-26T11:00:48.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0387200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f038819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:48.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0387200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 0x7f038819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:48.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0387200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f038810b850 0x7f03881a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0387200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f0388115530 con 0x7f038810af00 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0387200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f03881153b0 con 0x7f03881055c0 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0387200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f03881156b0 con 0x7f038810b850 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0385e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f038819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0385e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f038819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56894/0 (socket says 172.21.15.144:56894) 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.278+0000 7f0385e00640 1 -- 172.21.15.144:0/3570475720 learned_addr learned my addr 172.21.15.144:0/3570475720 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385400640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 0x7f038819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:48.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0386800640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f038810b850 0x7f03881a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:48.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385e00640 1 -- 172.21.15.144:0/3570475720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f038810b850 msgr2=0x7f03881a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:48.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385e00640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f038810b850 0x7f03881a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385e00640 1 -- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 msgr2=0x7f038819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:48.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385e00640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 0x7f038819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385e00640 1 -- 172.21.15.144:0/3570475720 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03881a1eb0 con 0x7f03881055c0 2024-07-26T11:00:48.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.279+0000 7f0385e00640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f038819bf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f037800ea60 tx=0x7f037800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:48.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.280+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037800cde0 con 0x7f03881055c0 2024-07-26T11:00:48.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.280+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03780035f0 con 0x7f03881055c0 2024-07-26T11:00:48.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.280+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03881a21a0 con 0x7f03881055c0 2024-07-26T11:00:48.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.280+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378010690 con 0x7f03881055c0 2024-07-26T11:00:48.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.280+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03881a8900 con 0x7f03881055c0 2024-07-26T11:00:48.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.281+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03881059d0 con 0x7f03881055c0 2024-07-26T11:00:48.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.283+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f03780037a0 con 0x7f03881055c0 2024-07-26T11:00:48.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.285+0000 7f037ea00640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f036c0b3ca0 0x7f036c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:48.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.285+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f03780e4370 con 0x7f03881055c0 2024-07-26T11:00:48.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.285+0000 7f0385400640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f036c0b3ca0 0x7f036c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:48.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.286+0000 7f0385400640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f036c0b3ca0 0x7f036c0b61b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f037402a9e0 tx=0x7f0374005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:48.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.289+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f03780ab4d0 con 0x7f03881055c0 2024-07-26T11:00:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:48 smithi144 ceph-mon[32129]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:48.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.484+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f038810d690 con 0x7f036c0b3ca0 2024-07-26T11:00:48.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.500+0000 7f037ea00640 1 -- 172.21.15.144:0/3570475720 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340475 (secure 0 0 0) 0x7f038810d690 con 0x7f036c0b3ca0 2024-07-26T11:00:48.507 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:00:48.509 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T11:00:48.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f036c0b3ca0 msgr2=0x7f036c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:48.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f036c0b3ca0 0x7f036c0b61b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f037402a9e0 tx=0x7f0374005a20 comp rx=0 tx=0).stop 2024-07-26T11:00:48.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 msgr2=0x7f038819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f038819bf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f037800ea60 tx=0x7f037800ef20 comp rx=0 tx=0).stop 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 shutdown_connections 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f036c0b3ca0 0x7f036c0b61b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f038810b850 0x7f03881a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f038810af00 0x7f038819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 --2- 172.21.15.144:0/3570475720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f03881055c0 0x7f038819bf40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.506+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 >> 172.21.15.144:0/3570475720 conn(0x7f0388101010 msgr2=0x7f0388103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.507+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 shutdown_connections 2024-07-26T11:00:48.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:48.507+0000 7f0387200640 1 -- 172.21.15.144:0/3570475720 wait complete. 2024-07-26T11:00:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:48 smithi160 ceph-mon[36715]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:48 smithi183 ceph-mon[37273]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:49.351 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-07-26T11:00:47.059565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662670,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":849,"num_read_kb":528,"num_write":302,"num_write_kb":2279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380092,"kb_used_data":14300,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561732,"statfs":{"total":1151940427776,"available":1151551213568,"internally_reserved":0,"allocated":14643200,"data_stored":8385968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006069"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210208+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:58:45.210208+0000","last_peered":"2024-07-26T10:58:45.210208+0000","last_clean":"2024-07-26T10:58:45.210208+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:58:45.210208+0000","last_undegraded":"2024-07-26T10:58:45.210208+0000","last_fullsized":"2024-07-26T10:58:45.210208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.965187+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T11:00:12.965187+0000","last_peered":"2024-07-26T11:00:12.965187+0000","last_clean":"2024-07-26T11:00:12.965187+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T11:00:12.965187+0000","last_undegraded":"2024-07-26T11:00:12.965187+0000","last_fullsized":"2024-07-26T11:00:12.965187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205284+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:58:45.205284+0000","last_peered":"2024-07-26T10:58:45.205284+0000","last_clean":"2024-07-26T10:58:45.205284+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:58:45.205284+0000","last_undegraded":"2024-07-26T10:58:45.205284+0000","last_fullsized":"2024-07-26T10:58:45.205284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211762+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:58:45.211762+0000","last_peered":"2024-07-26T10:58:45.211762+0000","last_clean":"2024-07-26T10:58:45.211762+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:58:45.211762+0000","last_undegraded":"2024-07-26T10:58:45.211762+0000","last_fullsized":"2024-07-26T10:58:45.211762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.995842+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T11:00:12.995842+0000","last_peered":"2024-07-26T11:00:12.995842+0000","last_clean":"2024-07-26T11:00:12.995842+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T11:00:12.995842+0000","last_undegraded":"2024-07-26T11:00:12.995842+0000","last_fullsized":"2024-07-26T11:00:12.995842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211982+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T10:58:45.211982+0000","last_peered":"2024-07-26T10:58:45.211982+0000","last_clean":"2024-07-26T10:58:45.211982+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T10:58:45.211982+0000","last_undegraded":"2024-07-26T10:58:45.211982+0000","last_fullsized":"2024-07-26T10:58:45.211982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211853+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:58:45.211853+0000","last_peered":"2024-07-26T10:58:45.211853+0000","last_clean":"2024-07-26T10:58:45.211853+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:58:45.211853+0000","last_undegraded":"2024-07-26T10:58:45.211853+0000","last_fullsized":"2024-07-26T10:58:45.211853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986457+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T11:00:12.986457+0000","last_peered":"2024-07-26T11:00:12.986457+0000","last_clean":"2024-07-26T11:00:12.986457+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T11:00:12.986457+0000","last_undegraded":"2024-07-26T11:00:12.986457+0000","last_fullsized":"2024-07-26T11:00:12.986457+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212336+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212336+0000","last_peered":"2024-07-26T10:58:45.212336+0000","last_clean":"2024-07-26T10:58:45.212336+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212336+0000","last_undegraded":"2024-07-26T10:58:45.212336+0000","last_fullsized":"2024-07-26T10:58:45.212336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395309+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395309+0000","last_peered":"2024-07-26T10:58:45.395309+0000","last_clean":"2024-07-26T10:58:45.395309+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395309+0000","last_undegraded":"2024-07-26T10:58:45.395309+0000","last_fullsized":"2024-07-26T10:58:45.395309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952593+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T11:00:12.952593+0000","last_peered":"2024-07-26T11:00:12.952593+0000","last_clean":"2024-07-26T11:00:12.952593+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T11:00:12.952593+0000","last_undegraded":"2024-07-26T11:00:12.952593+0000","last_fullsized":"2024-07-26T11:00:12.952593+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212134+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212134+0000","last_peered":"2024-07-26T10:58:45.212134+0000","last_clean":"2024-07-26T10:58:45.212134+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212134+0000","last_undegraded":"2024-07-26T10:58:45.212134+0000","last_fullsized":"2024-07-26T10:58:45.212134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969521+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T11:00:12.969521+0000","last_peered":"2024-07-26T11:00:12.969521+0000","last_clean":"2024-07-26T11:00:12.969521+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T11:00:12.969521+0000","last_undegraded":"2024-07-26T11:00:12.969521+0000","last_fullsized":"2024-07-26T11:00:12.969521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211215+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:58:45.211215+0000","last_peered":"2024-07-26T10:58:45.211215+0000","last_clean":"2024-07-26T10:58:45.211215+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:58:45.211215+0000","last_undegraded":"2024-07-26T10:58:45.211215+0000","last_fullsized":"2024-07-26T10:58:45.211215+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212254+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:58:45.212254+0000","last_peered":"2024-07-26T10:58:45.212254+0000","last_clean":"2024-07-26T10:58:45.212254+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:58:45.212254+0000","last_undegraded":"2024-07-26T10:58:45.212254+0000","last_fullsized":"2024-07-26T10:58:45.212254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.997353+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T11:00:12.997353+0000","last_peered":"2024-07-26T11:00:12.997353+0000","last_clean":"2024-07-26T11:00:12.997353+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T11:00:12.997353+0000","last_undegraded":"2024-07-26T11:00:12.997353+0000","last_fullsized":"2024-07-26T11:00:12.997353+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205391+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T10:58:45.205391+0000","last_peered":"2024-07-26T10:58:45.205391+0000","last_clean":"2024-07-26T10:58:45.205391+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T10:58:45.205391+0000","last_undegraded":"2024-07-26T10:58:45.205391+0000","last_fullsized":"2024-07-26T10:58:45.205391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:02.756153+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:59:02.756153+0000","last_peered":"2024-07-26T10:59:02.756153+0000","last_clean":"2024-07-26T10:59:02.756153+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:59:02.756153+0000","last_undegraded":"2024-07-26T10:59:02.756153+0000","last_fullsized":"2024-07-26T10:59:02.756153+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.962814+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T11:00:12.962814+0000","last_peered":"2024-07-26T11:00:12.962814+0000","last_clean":"2024-07-26T11:00:12.962814+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T11:00:12.962814+0000","last_undegraded":"2024-07-26T11:00:12.962814+0000","last_fullsized":"2024-07-26T11:00:12.962814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210173+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:58:45.210173+0000","last_peered":"2024-07-26T10:58:45.210173+0000","last_clean":"2024-07-26T10:58:45.210173+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:58:45.210173+0000","last_undegraded":"2024-07-26T10:58:45.210173+0000","last_fullsized":"2024-07-26T10:58:45.210173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211643+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T10:58:45.211643+0000","last_peered":"2024-07-26T10:58:45.211643+0000","last_clean":"2024-07-26T10:58:45.211643+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T10:58:45.211643+0000","last_undegraded":"2024-07-26T10:58:45.211643+0000","last_fullsized":"2024-07-26T10:58:45.211643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.909039+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T11:00:12.909039+0000","last_peered":"2024-07-26T11:00:12.909039+0000","last_clean":"2024-07-26T11:00:12.909039+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T11:00:12.909039+0000","last_undegraded":"2024-07-26T11:00:12.909039+0000","last_fullsized":"2024-07-26T11:00:12.909039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394785+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:58:45.394785+0000","last_peered":"2024-07-26T10:58:45.394785+0000","last_clean":"2024-07-26T10:58:45.394785+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:58:45.394785+0000","last_undegraded":"2024-07-26T10:58:45.394785+0000","last_fullsized":"2024-07-26T10:58:45.394785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212686+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:58:45.212686+0000","last_peered":"2024-07-26T10:58:45.212686+0000","last_clean":"2024-07-26T10:58:45.212686+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:58:45.212686+0000","last_undegraded":"2024-07-26T10:58:45.212686+0000","last_fullsized":"2024-07-26T10:58:45.212686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.990650+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T11:00:12.990650+0000","last_peered":"2024-07-26T11:00:12.990650+0000","last_clean":"2024-07-26T11:00:12.990650+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T11:00:12.990650+0000","last_undegraded":"2024-07-26T11:00:12.990650+0000","last_fullsized":"2024-07-26T11:00:12.990650+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212754+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:58:45.212754+0000","last_peered":"2024-07-26T10:58:45.212754+0000","last_clean":"2024-07-26T10:58:45.212754+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:58:45.212754+0000","last_undegraded":"2024-07-26T10:58:45.212754+0000","last_fullsized":"2024-07-26T10:58:45.212754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205500+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:58:45.205500+0000","last_peered":"2024-07-26T10:58:45.205500+0000","last_clean":"2024-07-26T10:58:45.205500+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:58:45.205500+0000","last_undegraded":"2024-07-26T10:58:45.205500+0000","last_fullsized":"2024-07-26T10:58:45.205500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982080+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T11:00:12.982080+0000","last_peered":"2024-07-26T11:00:12.982080+0000","last_clean":"2024-07-26T11:00:12.982080+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T11:00:12.982080+0000","last_undegraded":"2024-07-26T11:00:12.982080+0000","last_fullsized":"2024-07-26T11:00:12.982080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:30.594697+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T10:59:30.594697+0000","last_peered":"2024-07-26T10:59:30.594697+0000","last_clean":"2024-07-26T10:59:30.594697+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T10:59:30.594697+0000","last_undegraded":"2024-07-26T10:59:30.594697+0000","last_fullsized":"2024-07-26T10:59:30.594697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208811+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:58:45.208811+0000","last_peered":"2024-07-26T10:58:45.208811+0000","last_clean":"2024-07-26T10:58:45.208811+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:58:45.208811+0000","last_undegraded":"2024-07-26T10:58:45.208811+0000","last_fullsized":"2024-07-26T10:58:45.208811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986960+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.986960+0000","last_peered":"2024-07-26T11:00:12.986960+0000","last_clean":"2024-07-26T11:00:12.986960+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T11:00:12.986960+0000","last_undegraded":"2024-07-26T11:00:12.986960+0000","last_fullsized":"2024-07-26T11:00:12.986960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211817+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:58:45.211817+0000","last_peered":"2024-07-26T10:58:45.211817+0000","last_clean":"2024-07-26T10:58:45.211817+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:58:45.211817+0000","last_undegraded":"2024-07-26T10:58:45.211817+0000","last_fullsized":"2024-07-26T10:58:45.211817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395714+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:58:45.395714+0000","last_peered":"2024-07-26T10:58:45.395714+0000","last_clean":"2024-07-26T10:58:45.395714+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:58:45.395714+0000","last_undegraded":"2024-07-26T10:58:45.395714+0000","last_fullsized":"2024-07-26T10:58:45.395714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951920+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.951920+0000","last_peered":"2024-07-26T11:00:12.951920+0000","last_clean":"2024-07-26T11:00:12.951920+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T11:00:12.951920+0000","last_undegraded":"2024-07-26T11:00:12.951920+0000","last_fullsized":"2024-07-26T11:00:12.951920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208937+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:58:45.208937+0000","last_peered":"2024-07-26T10:58:45.208937+0000","last_clean":"2024-07-26T10:58:45.208937+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:58:45.208937+0000","last_undegraded":"2024-07-26T10:58:45.208937+0000","last_fullsized":"2024-07-26T10:58:45.208937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211173+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:58:45.211173+0000","last_peered":"2024-07-26T10:58:45.211173+0000","last_clean":"2024-07-26T10:58:45.211173+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:58:45.211173+0000","last_undegraded":"2024-07-26T10:58:45.211173+0000","last_fullsized":"2024-07-26T10:58:45.211173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.976047+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T11:00:12.976047+0000","last_peered":"2024-07-26T11:00:12.976047+0000","last_clean":"2024-07-26T11:00:12.976047+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T11:00:12.976047+0000","last_undegraded":"2024-07-26T11:00:12.976047+0000","last_fullsized":"2024-07-26T11:00:12.976047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210980+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:58:45.210980+0000","last_peered":"2024-07-26T10:58:45.210980+0000","last_clean":"2024-07-26T10:58:45.210980+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:58:45.210980+0000","last_undegraded":"2024-07-26T10:58:45.210980+0000","last_fullsized":"2024-07-26T10:58:45.210980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396866+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:58:45.396866+0000","last_peered":"2024-07-26T10:58:45.396866+0000","last_clean":"2024-07-26T10:58:45.396866+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:58:45.396866+0000","last_undegraded":"2024-07-26T10:58:45.396866+0000","last_fullsized":"2024-07-26T10:58:45.396866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950069+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T11:00:12.950069+0000","last_peered":"2024-07-26T11:00:12.950069+0000","last_clean":"2024-07-26T11:00:12.950069+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T11:00:12.950069+0000","last_undegraded":"2024-07-26T11:00:12.950069+0000","last_fullsized":"2024-07-26T11:00:12.950069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":171,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:41.450304+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T11:00:41.450304+0000","last_peered":"2024-07-26T11:00:41.450304+0000","last_clean":"2024-07-26T11:00:41.450304+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T11:00:41.450304+0000","last_undegraded":"2024-07-26T11:00:41.450304+0000","last_fullsized":"2024-07-26T11:00:41.450304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397096+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:58:45.397096+0000","last_peered":"2024-07-26T10:58:45.397096+0000","last_clean":"2024-07-26T10:58:45.397096+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:58:45.397096+0000","last_undegraded":"2024-07-26T10:58:45.397096+0000","last_fullsized":"2024-07-26T10:58:45.397096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.983082+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T11:00:12.983082+0000","last_peered":"2024-07-26T11:00:12.983082+0000","last_clean":"2024-07-26T11:00:12.983082+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T11:00:12.983082+0000","last_undegraded":"2024-07-26T11:00:12.983082+0000","last_fullsized":"2024-07-26T11:00:12.983082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210600+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:58:45.210600+0000","last_peered":"2024-07-26T10:58:45.210600+0000","last_clean":"2024-07-26T10:58:45.210600+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:58:45.210600+0000","last_undegraded":"2024-07-26T10:58:45.210600+0000","last_fullsized":"2024-07-26T10:58:45.210600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.399431+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:45.399431+0000","last_peered":"2024-07-26T10:58:45.399431+0000","last_clean":"2024-07-26T10:58:45.399431+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:45.399431+0000","last_undegraded":"2024-07-26T10:58:45.399431+0000","last_fullsized":"2024-07-26T10:58:45.399431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.971525+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T11:00:12.971525+0000","last_peered":"2024-07-26T11:00:12.971525+0000","last_clean":"2024-07-26T11:00:12.971525+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T11:00:12.971525+0000","last_undegraded":"2024-07-26T11:00:12.971525+0000","last_fullsized":"2024-07-26T11:00:12.971525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212762+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:58:45.212762+0000","last_peered":"2024-07-26T10:58:45.212762+0000","last_clean":"2024-07-26T10:58:45.212762+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:58:45.212762+0000","last_undegraded":"2024-07-26T10:58:45.212762+0000","last_fullsized":"2024-07-26T10:58:45.212762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210654+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210654+0000","last_peered":"2024-07-26T10:58:45.210654+0000","last_clean":"2024-07-26T10:58:45.210654+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210654+0000","last_undegraded":"2024-07-26T10:58:45.210654+0000","last_fullsized":"2024-07-26T10:58:45.210654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:55.070655+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:58:55.070655+0000","last_peered":"2024-07-26T10:58:55.070655+0000","last_clean":"2024-07-26T10:58:55.070655+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:58:55.070655+0000","last_undegraded":"2024-07-26T10:58:55.070655+0000","last_fullsized":"2024-07-26T10:58:55.070655+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955554+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T11:00:12.955554+0000","last_peered":"2024-07-26T11:00:12.955554+0000","last_clean":"2024-07-26T11:00:12.955554+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T11:00:12.955554+0000","last_undegraded":"2024-07-26T11:00:12.955554+0000","last_fullsized":"2024-07-26T11:00:12.955554+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205872+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:58:45.205872+0000","last_peered":"2024-07-26T10:58:45.205872+0000","last_clean":"2024-07-26T10:58:45.205872+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:58:45.205872+0000","last_undegraded":"2024-07-26T10:58:45.205872+0000","last_fullsized":"2024-07-26T10:58:45.205872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.394373+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:58:45.394373+0000","last_peered":"2024-07-26T10:58:45.394373+0000","last_clean":"2024-07-26T10:58:45.394373+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:58:45.394373+0000","last_undegraded":"2024-07-26T10:58:45.394373+0000","last_fullsized":"2024-07-26T10:58:45.394373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980621+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T11:00:12.980621+0000","last_peered":"2024-07-26T11:00:12.980621+0000","last_clean":"2024-07-26T11:00:12.980621+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T11:00:12.980621+0000","last_undegraded":"2024-07-26T11:00:12.980621+0000","last_fullsized":"2024-07-26T11:00:12.980621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396426+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:58:45.396426+0000","last_peered":"2024-07-26T10:58:45.396426+0000","last_clean":"2024-07-26T10:58:45.396426+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:58:45.396426+0000","last_undegraded":"2024-07-26T10:58:45.396426+0000","last_fullsized":"2024-07-26T10:58:45.396426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.658938+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:58:45.658938+0000","last_peered":"2024-07-26T10:58:45.658938+0000","last_clean":"2024-07-26T10:58:45.658938+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:58:45.658938+0000","last_undegraded":"2024-07-26T10:58:45.658938+0000","last_fullsized":"2024-07-26T10:58:45.658938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954729+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T11:00:12.954729+0000","last_peered":"2024-07-26T11:00:12.954729+0000","last_clean":"2024-07-26T11:00:12.954729+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T11:00:12.954729+0000","last_undegraded":"2024-07-26T11:00:12.954729+0000","last_fullsized":"2024-07-26T11:00:12.954729+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397589+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:58:45.397589+0000","last_peered":"2024-07-26T10:58:45.397589+0000","last_clean":"2024-07-26T10:58:45.397589+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:58:45.397589+0000","last_undegraded":"2024-07-26T10:58:45.397589+0000","last_fullsized":"2024-07-26T10:58:45.397589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395573+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:58:45.395573+0000","last_peered":"2024-07-26T10:58:45.395573+0000","last_clean":"2024-07-26T10:58:45.395573+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:58:45.395573+0000","last_undegraded":"2024-07-26T10:58:45.395573+0000","last_fullsized":"2024-07-26T10:58:45.395573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945788+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T11:00:12.945788+0000","last_peered":"2024-07-26T11:00:12.945788+0000","last_clean":"2024-07-26T11:00:12.945788+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T11:00:12.945788+0000","last_undegraded":"2024-07-26T11:00:12.945788+0000","last_fullsized":"2024-07-26T11:00:12.945788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211738+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:58:45.211738+0000","last_peered":"2024-07-26T10:58:45.211738+0000","last_clean":"2024-07-26T10:58:45.211738+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:58:45.211738+0000","last_undegraded":"2024-07-26T10:58:45.211738+0000","last_fullsized":"2024-07-26T10:58:45.211738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395592+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:58:45.395592+0000","last_peered":"2024-07-26T10:58:45.395592+0000","last_clean":"2024-07-26T10:58:45.395592+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:58:45.395592+0000","last_undegraded":"2024-07-26T10:58:45.395592+0000","last_fullsized":"2024-07-26T10:58:45.395592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942946+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T11:00:12.942946+0000","last_peered":"2024-07-26T11:00:12.942946+0000","last_clean":"2024-07-26T11:00:12.942946+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T11:00:12.942946+0000","last_undegraded":"2024-07-26T11:00:12.942946+0000","last_fullsized":"2024-07-26T11:00:12.942946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211049+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:58:45.211049+0000","last_peered":"2024-07-26T10:58:45.211049+0000","last_clean":"2024-07-26T10:58:45.211049+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:58:45.211049+0000","last_undegraded":"2024-07-26T10:58:45.211049+0000","last_fullsized":"2024-07-26T10:58:45.211049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397349+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397349+0000","last_peered":"2024-07-26T10:58:45.397349+0000","last_clean":"2024-07-26T10:58:45.397349+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397349+0000","last_undegraded":"2024-07-26T10:58:45.397349+0000","last_fullsized":"2024-07-26T10:58:45.397349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949820+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T11:00:12.949820+0000","last_peered":"2024-07-26T11:00:12.949820+0000","last_clean":"2024-07-26T11:00:12.949820+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T11:00:12.949820+0000","last_undegraded":"2024-07-26T11:00:12.949820+0000","last_fullsized":"2024-07-26T11:00:12.949820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":179,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:45.460727+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T11:00:45.460727+0000","last_peered":"2024-07-26T11:00:45.460727+0000","last_clean":"2024-07-26T11:00:45.460727+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T11:00:45.460727+0000","last_undegraded":"2024-07-26T11:00:45.460727+0000","last_fullsized":"2024-07-26T11:00:45.460727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208214+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T10:58:45.208214+0000","last_peered":"2024-07-26T10:58:45.208214+0000","last_clean":"2024-07-26T10:58:45.208214+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T10:58:45.208214+0000","last_undegraded":"2024-07-26T10:58:45.208214+0000","last_fullsized":"2024-07-26T10:58:45.208214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987813+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T11:00:12.987813+0000","last_peered":"2024-07-26T11:00:12.987813+0000","last_clean":"2024-07-26T11:00:12.987813+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T11:00:12.987813+0000","last_undegraded":"2024-07-26T11:00:12.987813+0000","last_fullsized":"2024-07-26T11:00:12.987813+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395671+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:58:45.395671+0000","last_peered":"2024-07-26T10:58:45.395671+0000","last_clean":"2024-07-26T10:58:45.395671+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:58:45.395671+0000","last_undegraded":"2024-07-26T10:58:45.395671+0000","last_fullsized":"2024-07-26T10:58:45.395671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395441+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:58:45.395441+0000","last_peered":"2024-07-26T10:58:45.395441+0000","last_clean":"2024-07-26T10:58:45.395441+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:58:45.395441+0000","last_undegraded":"2024-07-26T10:58:45.395441+0000","last_fullsized":"2024-07-26T10:58:45.395441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.984849+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T11:00:12.984849+0000","last_peered":"2024-07-26T11:00:12.984849+0000","last_clean":"2024-07-26T11:00:12.984849+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T11:00:12.984849+0000","last_undegraded":"2024-07-26T11:00:12.984849+0000","last_fullsized":"2024-07-26T11:00:12.984849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205619+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:58:45.205619+0000","last_peered":"2024-07-26T10:58:45.205619+0000","last_clean":"2024-07-26T10:58:45.205619+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:58:45.205619+0000","last_undegraded":"2024-07-26T10:58:45.205619+0000","last_fullsized":"2024-07-26T10:58:45.205619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954477+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T11:00:12.954477+0000","last_peered":"2024-07-26T11:00:12.954477+0000","last_clean":"2024-07-26T11:00:12.954477+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T11:00:12.954477+0000","last_undegraded":"2024-07-26T11:00:12.954477+0000","last_fullsized":"2024-07-26T11:00:12.954477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":339,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:44.386482+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T11:00:44.386482+0000","last_peered":"2024-07-26T11:00:44.386482+0000","last_clean":"2024-07-26T11:00:44.386482+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T11:00:44.386482+0000","last_undegraded":"2024-07-26T11:00:44.386482+0000","last_fullsized":"2024-07-26T11:00:44.386482+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67926,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":257,"num_read_kb":195,"num_write":38,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.208081+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:58:45.208081+0000","last_peered":"2024-07-26T10:58:45.208081+0000","last_clean":"2024-07-26T10:58:45.208081+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:58:45.208081+0000","last_undegraded":"2024-07-26T10:58:45.208081+0000","last_fullsized":"2024-07-26T10:58:45.208081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951888+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T11:00:12.951888+0000","last_peered":"2024-07-26T11:00:12.951888+0000","last_clean":"2024-07-26T11:00:12.951888+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T11:00:12.951888+0000","last_undegraded":"2024-07-26T11:00:12.951888+0000","last_fullsized":"2024-07-26T11:00:12.951888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396347+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:58:45.396347+0000","last_peered":"2024-07-26T10:58:45.396347+0000","last_clean":"2024-07-26T10:58:45.396347+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:58:45.396347+0000","last_undegraded":"2024-07-26T10:58:45.396347+0000","last_fullsized":"2024-07-26T10:58:45.396347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212446+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:58:45.212446+0000","last_peered":"2024-07-26T10:58:45.212446+0000","last_clean":"2024-07-26T10:58:45.212446+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:58:45.212446+0000","last_undegraded":"2024-07-26T10:58:45.212446+0000","last_fullsized":"2024-07-26T10:58:45.212446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987806+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T11:00:12.987806+0000","last_peered":"2024-07-26T11:00:12.987806+0000","last_clean":"2024-07-26T11:00:12.987806+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T11:00:12.987806+0000","last_undegraded":"2024-07-26T11:00:12.987806+0000","last_fullsized":"2024-07-26T11:00:12.987806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.211576+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:58:45.211576+0000","last_peered":"2024-07-26T10:58:45.211576+0000","last_clean":"2024-07-26T10:58:45.211576+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:58:45.211576+0000","last_undegraded":"2024-07-26T10:58:45.211576+0000","last_fullsized":"2024-07-26T10:58:45.211576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.207082+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:58:45.207082+0000","last_peered":"2024-07-26T10:58:45.207082+0000","last_clean":"2024-07-26T10:58:45.207082+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:58:45.207082+0000","last_undegraded":"2024-07-26T10:58:45.207082+0000","last_fullsized":"2024-07-26T10:58:45.207082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950324+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T11:00:12.950324+0000","last_peered":"2024-07-26T11:00:12.950324+0000","last_clean":"2024-07-26T11:00:12.950324+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T11:00:12.950324+0000","last_undegraded":"2024-07-26T11:00:12.950324+0000","last_fullsized":"2024-07-26T11:00:12.950324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":171,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:43.384855+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T11:00:43.384855+0000","last_peered":"2024-07-26T11:00:43.384855+0000","last_clean":"2024-07-26T11:00:43.384855+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T11:00:43.384855+0000","last_undegraded":"2024-07-26T11:00:43.384855+0000","last_fullsized":"2024-07-26T11:00:43.384855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.209023+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:58:45.209023+0000","last_peered":"2024-07-26T10:58:45.209023+0000","last_clean":"2024-07-26T10:58:45.209023+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:58:45.209023+0000","last_undegraded":"2024-07-26T10:58:45.209023+0000","last_fullsized":"2024-07-26T10:58:45.209023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.659120+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:58:45.659120+0000","last_peered":"2024-07-26T10:58:45.659120+0000","last_clean":"2024-07-26T10:58:45.659120+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:58:45.659120+0000","last_undegraded":"2024-07-26T10:58:45.659120+0000","last_fullsized":"2024-07-26T10:58:45.659120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949031+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T11:00:12.949031+0000","last_peered":"2024-07-26T11:00:12.949031+0000","last_clean":"2024-07-26T11:00:12.949031+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T11:00:12.949031+0000","last_undegraded":"2024-07-26T11:00:12.949031+0000","last_fullsized":"2024-07-26T11:00:12.949031+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395737+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:58:45.395737+0000","last_peered":"2024-07-26T10:58:45.395737+0000","last_clean":"2024-07-26T10:58:45.395737+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:58:45.395737+0000","last_undegraded":"2024-07-26T10:58:45.395737+0000","last_fullsized":"2024-07-26T10:58:45.395737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395949+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:58:45.395949+0000","last_peered":"2024-07-26T10:58:45.395949+0000","last_clean":"2024-07-26T10:58:45.395949+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:58:45.395949+0000","last_undegraded":"2024-07-26T10:58:45.395949+0000","last_fullsized":"2024-07-26T10:58:45.395949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988233+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T11:00:12.988233+0000","last_peered":"2024-07-26T11:00:12.988233+0000","last_clean":"2024-07-26T11:00:12.988233+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T11:00:12.988233+0000","last_undegraded":"2024-07-26T11:00:12.988233+0000","last_fullsized":"2024-07-26T11:00:12.988233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.396116+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:58:45.396116+0000","last_peered":"2024-07-26T10:58:45.396116+0000","last_clean":"2024-07-26T10:58:45.396116+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:58:45.396116+0000","last_undegraded":"2024-07-26T10:58:45.396116+0000","last_fullsized":"2024-07-26T10:58:45.396116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395653+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:58:45.395653+0000","last_peered":"2024-07-26T10:58:45.395653+0000","last_clean":"2024-07-26T10:58:45.395653+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:58:45.395653+0000","last_undegraded":"2024-07-26T10:58:45.395653+0000","last_fullsized":"2024-07-26T10:58:45.395653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949369+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T11:00:12.949369+0000","last_peered":"2024-07-26T11:00:12.949369+0000","last_clean":"2024-07-26T11:00:12.949369+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T11:00:12.949369+0000","last_undegraded":"2024-07-26T11:00:12.949369+0000","last_fullsized":"2024-07-26T11:00:12.949369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205546+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205546+0000","last_peered":"2024-07-26T10:58:45.205546+0000","last_clean":"2024-07-26T10:58:45.205546+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205546+0000","last_undegraded":"2024-07-26T10:58:45.205546+0000","last_fullsized":"2024-07-26T10:58:45.205546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:42.384000+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T11:00:42.384000+0000","last_peered":"2024-07-26T11:00:42.384000+0000","last_clean":"2024-07-26T11:00:42.384000+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T11:00:42.384000+0000","last_undegraded":"2024-07-26T11:00:42.384000+0000","last_fullsized":"2024-07-26T11:00:42.384000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.938191+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T11:00:12.938191+0000","last_peered":"2024-07-26T11:00:12.938191+0000","last_clean":"2024-07-26T11:00:12.938191+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T11:00:12.938191+0000","last_undegraded":"2024-07-26T11:00:12.938191+0000","last_fullsized":"2024-07-26T11:00:12.938191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205755+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205755+0000","last_peered":"2024-07-26T10:58:45.205755+0000","last_clean":"2024-07-26T10:58:45.205755+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205755+0000","last_undegraded":"2024-07-26T10:58:45.205755+0000","last_fullsized":"2024-07-26T10:58:45.205755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942683+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T11:00:12.942683+0000","last_peered":"2024-07-26T11:00:12.942683+0000","last_clean":"2024-07-26T11:00:12.942683+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T11:00:12.942683+0000","last_undegraded":"2024-07-26T11:00:12.942683+0000","last_fullsized":"2024-07-26T11:00:12.942683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945771+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T11:00:12.945771+0000","last_peered":"2024-07-26T11:00:12.945771+0000","last_clean":"2024-07-26T11:00:12.945771+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T11:00:12.945771+0000","last_undegraded":"2024-07-26T11:00:12.945771+0000","last_fullsized":"2024-07-26T11:00:12.945771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.975117+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T11:00:12.975117+0000","last_peered":"2024-07-26T11:00:12.975117+0000","last_clean":"2024-07-26T11:00:12.975117+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T11:00:12.975117+0000","last_undegraded":"2024-07-26T11:00:12.975117+0000","last_fullsized":"2024-07-26T11:00:12.975117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992678+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T11:00:12.992678+0000","last_peered":"2024-07-26T11:00:12.992678+0000","last_clean":"2024-07-26T11:00:12.992678+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T11:00:12.992678+0000","last_undegraded":"2024-07-26T11:00:12.992678+0000","last_fullsized":"2024-07-26T11:00:12.992678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.996426+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T11:00:12.996426+0000","last_peered":"2024-07-26T11:00:12.996426+0000","last_clean":"2024-07-26T11:00:12.996426+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T11:00:12.996426+0000","last_undegraded":"2024-07-26T11:00:12.996426+0000","last_fullsized":"2024-07-26T11:00:12.996426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.940149+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T11:00:12.940149+0000","last_peered":"2024-07-26T11:00:12.940149+0000","last_clean":"2024-07-26T11:00:12.940149+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T11:00:12.940149+0000","last_undegraded":"2024-07-26T11:00:12.940149+0000","last_fullsized":"2024-07-26T11:00:12.940149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.941781+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T11:00:12.941781+0000","last_peered":"2024-07-26T11:00:12.941781+0000","last_clean":"2024-07-26T11:00:12.941781+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T11:00:12.941781+0000","last_undegraded":"2024-07-26T11:00:12.941781+0000","last_fullsized":"2024-07-26T11:00:12.941781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.973121+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T11:00:12.973121+0000","last_peered":"2024-07-26T11:00:12.973121+0000","last_clean":"2024-07-26T11:00:12.973121+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T11:00:12.973121+0000","last_undegraded":"2024-07-26T11:00:12.973121+0000","last_fullsized":"2024-07-26T11:00:12.973121+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967499+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T11:00:12.967499+0000","last_peered":"2024-07-26T11:00:12.967499+0000","last_clean":"2024-07-26T11:00:12.967499+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T11:00:12.967499+0000","last_undegraded":"2024-07-26T11:00:12.967499+0000","last_fullsized":"2024-07-26T11:00:12.967499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981644+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T11:00:12.981644+0000","last_peered":"2024-07-26T11:00:12.981644+0000","last_clean":"2024-07-26T11:00:12.981644+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T11:00:12.981644+0000","last_undegraded":"2024-07-26T11:00:12.981644+0000","last_fullsized":"2024-07-26T11:00:12.981644+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981194+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T11:00:12.981194+0000","last_peered":"2024-07-26T11:00:12.981194+0000","last_clean":"2024-07-26T11:00:12.981194+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T11:00:12.981194+0000","last_undegraded":"2024-07-26T11:00:12.981194+0000","last_fullsized":"2024-07-26T11:00:12.981194+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.999046+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T11:00:12.999046+0000","last_peered":"2024-07-26T11:00:12.999046+0000","last_clean":"2024-07-26T11:00:12.999046+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T11:00:12.999046+0000","last_undegraded":"2024-07-26T11:00:12.999046+0000","last_fullsized":"2024-07-26T11:00:12.999046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946601+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T11:00:12.946601+0000","last_peered":"2024-07-26T11:00:12.946601+0000","last_clean":"2024-07-26T11:00:12.946601+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T11:00:12.946601+0000","last_undegraded":"2024-07-26T11:00:12.946601+0000","last_fullsized":"2024-07-26T11:00:12.946601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947146+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T11:00:12.947146+0000","last_peered":"2024-07-26T11:00:12.947146+0000","last_clean":"2024-07-26T11:00:12.947146+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T11:00:12.947146+0000","last_undegraded":"2024-07-26T11:00:12.947146+0000","last_fullsized":"2024-07-26T11:00:12.947146+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945412+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T11:00:12.945412+0000","last_peered":"2024-07-26T11:00:12.945412+0000","last_clean":"2024-07-26T11:00:12.945412+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T11:00:12.945412+0000","last_undegraded":"2024-07-26T11:00:12.945412+0000","last_fullsized":"2024-07-26T11:00:12.945412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974857+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T11:00:12.974857+0000","last_peered":"2024-07-26T11:00:12.974857+0000","last_clean":"2024-07-26T11:00:12.974857+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T11:00:12.974857+0000","last_undegraded":"2024-07-26T11:00:12.974857+0000","last_fullsized":"2024-07-26T11:00:12.974857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992807+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T11:00:12.992807+0000","last_peered":"2024-07-26T11:00:12.992807+0000","last_clean":"2024-07-26T11:00:12.992807+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T11:00:12.992807+0000","last_undegraded":"2024-07-26T11:00:12.992807+0000","last_fullsized":"2024-07-26T11:00:12.992807+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992005+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T11:00:12.992005+0000","last_peered":"2024-07-26T11:00:12.992005+0000","last_clean":"2024-07-26T11:00:12.992005+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T11:00:12.992005+0000","last_undegraded":"2024-07-26T11:00:12.992005+0000","last_fullsized":"2024-07-26T11:00:12.992005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994336+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T11:00:12.994336+0000","last_peered":"2024-07-26T11:00:12.994336+0000","last_clean":"2024-07-26T11:00:12.994336+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T11:00:12.994336+0000","last_undegraded":"2024-07-26T11:00:12.994336+0000","last_fullsized":"2024-07-26T11:00:12.994336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987691+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T11:00:12.987691+0000","last_peered":"2024-07-26T11:00:12.987691+0000","last_clean":"2024-07-26T11:00:12.987691+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T11:00:12.987691+0000","last_undegraded":"2024-07-26T11:00:12.987691+0000","last_fullsized":"2024-07-26T11:00:12.987691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951850+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T11:00:12.951850+0000","last_peered":"2024-07-26T11:00:12.951850+0000","last_clean":"2024-07-26T11:00:12.951850+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T11:00:12.951850+0000","last_undegraded":"2024-07-26T11:00:12.951850+0000","last_fullsized":"2024-07-26T11:00:12.951850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955325+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T11:00:12.955325+0000","last_peered":"2024-07-26T11:00:12.955325+0000","last_clean":"2024-07-26T11:00:12.955325+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T11:00:12.955325+0000","last_undegraded":"2024-07-26T11:00:12.955325+0000","last_fullsized":"2024-07-26T11:00:12.955325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.948540+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T11:00:12.948540+0000","last_peered":"2024-07-26T11:00:12.948540+0000","last_clean":"2024-07-26T11:00:12.948540+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T11:00:12.948540+0000","last_undegraded":"2024-07-26T11:00:12.948540+0000","last_fullsized":"2024-07-26T11:00:12.948540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945953+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T11:00:12.945953+0000","last_peered":"2024-07-26T11:00:12.945953+0000","last_clean":"2024-07-26T11:00:12.945953+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T11:00:12.945953+0000","last_undegraded":"2024-07-26T11:00:12.945953+0000","last_fullsized":"2024-07-26T11:00:12.945953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980776+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T11:00:12.980776+0000","last_peered":"2024-07-26T11:00:12.980776+0000","last_clean":"2024-07-26T11:00:12.980776+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T11:00:12.980776+0000","last_undegraded":"2024-07-26T11:00:12.980776+0000","last_fullsized":"2024-07-26T11:00:12.980776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950356+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T11:00:12.950356+0000","last_peered":"2024-07-26T11:00:12.950356+0000","last_clean":"2024-07-26T11:00:12.950356+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T11:00:12.950356+0000","last_undegraded":"2024-07-26T11:00:12.950356+0000","last_fullsized":"2024-07-26T11:00:12.950356+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946913+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T11:00:12.946913+0000","last_peered":"2024-07-26T11:00:12.946913+0000","last_clean":"2024-07-26T11:00:12.946913+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T11:00:12.946913+0000","last_undegraded":"2024-07-26T11:00:12.946913+0000","last_fullsized":"2024-07-26T11:00:12.946913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967346+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T11:00:12.967346+0000","last_peered":"2024-07-26T11:00:12.967346+0000","last_clean":"2024-07-26T11:00:12.967346+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T11:00:12.967346+0000","last_undegraded":"2024-07-26T11:00:12.967346+0000","last_fullsized":"2024-07-26T11:00:12.967346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951202+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T11:00:12.951202+0000","last_peered":"2024-07-26T11:00:12.951202+0000","last_clean":"2024-07-26T11:00:12.951202+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T11:00:12.951202+0000","last_undegraded":"2024-07-26T11:00:12.951202+0000","last_fullsized":"2024-07-26T11:00:12.951202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989483+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T11:00:12.989483+0000","last_peered":"2024-07-26T11:00:12.989483+0000","last_clean":"2024-07-26T11:00:12.989483+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T11:00:12.989483+0000","last_undegraded":"2024-07-26T11:00:12.989483+0000","last_fullsized":"2024-07-26T11:00:12.989483+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989961+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T11:00:12.989961+0000","last_peered":"2024-07-26T11:00:12.989961+0000","last_clean":"2024-07-26T11:00:12.989961+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T11:00:12.989961+0000","last_undegraded":"2024-07-26T11:00:12.989961+0000","last_fullsized":"2024-07-26T11:00:12.989961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974704+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T11:00:12.974704+0000","last_peered":"2024-07-26T11:00:12.974704+0000","last_clean":"2024-07-26T11:00:12.974704+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T11:00:12.974704+0000","last_undegraded":"2024-07-26T11:00:12.974704+0000","last_fullsized":"2024-07-26T11:00:12.974704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72068,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":768,"num_read_kb":459,"num_write":177,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":216204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678073,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874303,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1199087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037828,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430286,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593812,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790042,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378730,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":599145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542256,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":599039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:00:49.357 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T11:00:49.357 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T11:00:49.357 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-26T11:00:49.357 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph health --format=json 2024-07-26T11:00:49.861 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:50.348 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:50 smithi144 ceph-mon[32129]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:00:50.348 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:50 smithi144 ceph-mon[32129]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:50 smithi160 ceph-mon[36715]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:00:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:50 smithi160 ceph-mon[36715]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:50 smithi183 ceph-mon[37273]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:00:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:50 smithi183 ceph-mon[37273]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:51 smithi144 ceph-mon[32129]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:51 smithi160 ceph-mon[36715]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:51 smithi183 ceph-mon[37273]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 -- 172.21.15.144:0/1211619166 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 msgr2=0x7f0bb810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/1211619166 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 0x7f0bb810ef50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009ca0 tx=0x7f0bb002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 -- 172.21.15.144:0/1211619166 shutdown_connections 2024-07-26T11:00:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/1211619166 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 0x7f0bb810ef50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/1211619166 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/1211619166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 0x7f0bb81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.053+0000 7f0bbfc00640 1 -- 172.21.15.144:0/1211619166 >> 172.21.15.144:0/1211619166 conn(0x7f0bb8079160 msgr2=0x7f0bb8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:52.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.054+0000 7f0bbfc00640 1 -- 172.21.15.144:0/1211619166 shutdown_connections 2024-07-26T11:00:52.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.054+0000 7f0bbfc00640 1 -- 172.21.15.144:0/1211619166 wait complete. 2024-07-26T11:00:52.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 Processor -- start 2024-07-26T11:00:52.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 -- start start 2024-07-26T11:00:52.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 0x7f0bb8199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:52.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:52.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 0x7f0bb819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb8112380 con 0x7f0bb8102490 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb8112200 con 0x7f0bb8108720 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.055+0000 7f0bbfc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb8112500 con 0x7f0bb8107dd0 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bbd200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 0x7f0bb8199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:43846/0 (socket says 172.21.15.144:43846) 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 -- 172.21.15.144:0/8506476 learned_addr learned my addr 172.21.15.144:0/8506476 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bbd200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 0x7f0bb8199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52298/0 (socket says 172.21.15.144:52298) 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bbdc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 0x7f0bb819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 -- 172.21.15.144:0/8506476 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 msgr2=0x7f0bb819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 0x7f0bb819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 -- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 msgr2=0x7f0bb8199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 0x7f0bb8199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.056+0000 7f0bb7e00640 1 -- 172.21.15.144:0/8506476 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb819fe50 con 0x7f0bb8107dd0 2024-07-26T11:00:52.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.057+0000 7f0bb7e00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819a480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400e9c0 tx=0x7f0ba400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:52.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.057+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba400cd80 con 0x7f0bb8107dd0 2024-07-26T11:00:52.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.057+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba40054d0 con 0x7f0bb8107dd0 2024-07-26T11:00:52.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.057+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb81a0140 con 0x7f0bb8107dd0 2024-07-26T11:00:52.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.057+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4010430 con 0x7f0bb8107dd0 2024-07-26T11:00:52.061 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.057+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb81a68a0 con 0x7f0bb8107dd0 2024-07-26T11:00:52.061 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.058+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb81028a0 con 0x7f0bb8107dd0 2024-07-26T11:00:52.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.061+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f0ba4004fb0 con 0x7f0bb8107dd0 2024-07-26T11:00:52.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.063+0000 7f0bb5600640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0ba00b3c20 0x7f0ba00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:52.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.064+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f0ba40e4690 con 0x7f0bb8107dd0 2024-07-26T11:00:52.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.064+0000 7f0bbd200640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0ba00b3c20 0x7f0ba00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:52.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.064+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0ba40ab7f0 con 0x7f0bb8107dd0 2024-07-26T11:00:52.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.065+0000 7f0bbd200640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0ba00b3c20 0x7f0ba00b6130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0095f0 tx=0x7f0bac005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:52.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.311+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0bb819b010 con 0x7f0bb8107dd0 2024-07-26T11:00:52.314 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.312+0000 7f0bb5600640 1 -- 172.21.15.144:0/8506476 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0bb819b010 con 0x7f0bb8107dd0 2024-07-26T11:00:52.314 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:00:52.314 INFO:teuthology.orchestra.run.smithi144.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T11:00:52.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0ba00b3c20 msgr2=0x7f0ba00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:52.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0ba00b3c20 0x7f0ba00b6130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0095f0 tx=0x7f0bac005f90 comp rx=0 tx=0).stop 2024-07-26T11:00:52.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 msgr2=0x7f0bb819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819a480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400e9c0 tx=0x7f0ba400ee80 comp rx=0 tx=0).stop 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 shutdown_connections 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0ba00b3c20 0x7f0ba00b6130 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0bb8108720 0x7f0bb819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819a480 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 --2- 172.21.15.144:0/8506476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb8102490 0x7f0bb8199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.315+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 >> 172.21.15.144:0/8506476 conn(0x7f0bb8079160 msgr2=0x7f0bb806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.316+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 shutdown_connections 2024-07-26T11:00:52.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:52.316+0000 7f0bbfc00640 1 -- 172.21.15.144:0/8506476 wait complete. 2024-07-26T11:00:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:52 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/8506476' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T11:00:52.703 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:52 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/8506476' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T11:00:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:52 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/8506476' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T11:00:53.162 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-26T11:00:53.163 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:00:53.612 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:53 smithi183 ceph-mon[37273]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:53 smithi144 ceph-mon[32129]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:53 smithi160 ceph-mon[36715]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 -- 172.21.15.144:0/1553288304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc108d20 msgr2=0x7f89dc109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 --2- 172.21.15.144:0/1553288304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc108d20 0x7f89dc109130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009900 tx=0x7f89c802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 -- 172.21.15.144:0/1553288304 shutdown_connections 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 --2- 172.21.15.144:0/1553288304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89dc109670 0x7f89dc10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 --2- 172.21.15.144:0/1553288304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc108d20 0x7f89dc109130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 --2- 172.21.15.144:0/1553288304 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc1033e0 0x7f89dc1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 -- 172.21.15.144:0/1553288304 >> 172.21.15.144:0/1553288304 conn(0x7f89dc0fee70 msgr2=0x7f89dc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 -- 172.21.15.144:0/1553288304 shutdown_connections 2024-07-26T11:00:55.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.841+0000 7f89e2a00640 1 -- 172.21.15.144:0/1553288304 wait complete. 2024-07-26T11:00:55.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.842+0000 7f89e2a00640 1 Processor -- start 2024-07-26T11:00:55.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.842+0000 7f89e2a00640 1 -- start start 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89dc1033e0 0x7f89dc199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc108d20 0x7f89dc19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 0x7f89dc19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc113300 con 0x7f89dc109670 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc113180 con 0x7f89dc108d20 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc113480 con 0x7f89dc1033e0 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 0x7f89dc19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 0x7f89dc19f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52322/0 (socket says 172.21.15.144:52322) 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e2000640 1 -- 172.21.15.144:0/2508201540 learned_addr learned my addr 172.21.15.144:0/2508201540 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:55.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e1600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89dc1033e0 0x7f89dc199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:55.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.843+0000 7f89e0c00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc108d20 0x7f89dc19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:55.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e2000640 1 -- 172.21.15.144:0/2508201540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89dc1033e0 msgr2=0x7f89dc199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e2000640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89dc1033e0 0x7f89dc199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e2000640 1 -- 172.21.15.144:0/2508201540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc108d20 msgr2=0x7f89dc19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e2000640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc108d20 0x7f89dc19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e2000640 1 -- 172.21.15.144:0/2508201540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc19fd20 con 0x7f89dc109670 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e0c00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc108d20 0x7f89dc19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.844+0000 7f89e2000640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 0x7f89dc19f5f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f89d400e970 tx=0x7f89d400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.845+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d400ccc0 con 0x7f89dc109670 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.845+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d400ce60 con 0x7f89dc109670 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.845+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4010430 con 0x7f89dc109670 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.845+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1a0010 con 0x7f89dc109670 2024-07-26T11:00:55.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.845+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc1a6770 con 0x7f89dc109670 2024-07-26T11:00:55.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.846+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc1037f0 con 0x7f89dc109670 2024-07-26T11:00:55.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.850+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f89d4004f70 con 0x7f89dc109670 2024-07-26T11:00:55.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.852+0000 7f89da000640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f89c00b3ca0 0x7f89c00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:55.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.852+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f89d40e45d0 con 0x7f89dc109670 2024-07-26T11:00:55.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.852+0000 7f89e1600640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f89c00b3ca0 0x7f89c00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:55.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.853+0000 7f89e1600640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f89c00b3ca0 0x7f89c00b61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0099a0 tx=0x7f89cc009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:55.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:55.854+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f89d4055600 con 0x7f89dc109670 2024-07-26T11:00:56.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.092+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89dc19aee0 con 0x7f89dc109670 2024-07-26T11:00:56.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.094+0000 7f89da000640 1 -- 172.21.15.144:0/2508201540 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11931 (secure 0 0 0) 0x7f89dc19aee0 con 0x7f89dc109670 2024-07-26T11:00:56.096 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:00:56.096 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":25,"btime":"2024-07-26T10:58:47:222852+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:58:46.244128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T11:00:56.097 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f89c00b3ca0 msgr2=0x7f89c00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f89c00b3ca0 0x7f89c00b61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0099a0 tx=0x7f89cc009500 comp rx=0 tx=0).stop 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 msgr2=0x7f89dc19f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 0x7f89dc19f5f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f89d400e970 tx=0x7f89d400ed70 comp rx=0 tx=0).stop 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 shutdown_connections 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f89c00b3ca0 0x7f89c00b61b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:56.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89dc109670 0x7f89dc19f5f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:56.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89dc108d20 0x7f89dc19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:56.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 --2- 172.21.15.144:0/2508201540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89dc1033e0 0x7f89dc199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:56.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 >> 172.21.15.144:0/2508201540 conn(0x7f89dc0fee70 msgr2=0x7f89dc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:56.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 shutdown_connections 2024-07-26T11:00:56.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:56.097+0000 7f89e2a00640 1 -- 172.21.15.144:0/2508201540 wait complete. 2024-07-26T11:00:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:56 smithi183 ceph-mon[37273]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:00:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:56 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2508201540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:00:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:56 smithi144 ceph-mon[32129]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:00:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:56 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2508201540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:00:56.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:56 smithi160 ceph-mon[36715]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:00:56.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:56 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2508201540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:00:57.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:00:57.543 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:00:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:00:58 smithi183 ceph-mon[37273]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:00:58 smithi144 ceph-mon[32129]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:00:58 smithi160 ceph-mon[36715]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 -- 172.21.15.144:0/3175126933 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d9010b860 msgr2=0x7f2d90112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 --2- 172.21.15.144:0/3175126933 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d9010b860 0x7f2d90112090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009ca0 tx=0x7f2d8802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 -- 172.21.15.144:0/3175126933 shutdown_connections 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 --2- 172.21.15.144:0/3175126933 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d9010b860 0x7f2d90112090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 --2- 172.21.15.144:0/3175126933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010af10 0x7f2d9010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 --2- 172.21.15.144:0/3175126933 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d901055d0 0x7f2d901059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 -- 172.21.15.144:0/3175126933 >> 172.21.15.144:0/3175126933 conn(0x7f2d90101040 msgr2=0x7f2d90103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:00:59.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 -- 172.21.15.144:0/3175126933 shutdown_connections 2024-07-26T11:00:59.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.816+0000 7f2d8f200640 1 -- 172.21.15.144:0/3175126933 wait complete. 2024-07-26T11:00:59.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.818+0000 7f2d8f200640 1 Processor -- start 2024-07-26T11:00:59.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.818+0000 7f2d8f200640 1 -- start start 2024-07-26T11:00:59.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d901055d0 0x7f2d9019bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:59.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d9010af10 0x7f2d9019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:59.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 0x7f2d901a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:59.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2d90115530 con 0x7f2d9010b860 2024-07-26T11:00:59.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2d901153b0 con 0x7f2d901055d0 2024-07-26T11:00:59.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2d901156b0 con 0x7f2d9010af10 2024-07-26T11:00:59.822 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 0x7f2d901a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:59.822 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d9010af10 0x7f2d9019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:59.822 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 0x7f2d901a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46348/0 (socket says 172.21.15.144:46348) 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8de00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d901055d0 0x7f2d9019bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.819+0000 7f2d8e800640 1 -- 172.21.15.144:0/2998563137 learned_addr learned my addr 172.21.15.144:0/2998563137 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8d400640 1 -- 172.21.15.144:0/2998563137 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d901055d0 msgr2=0x7f2d9019bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8d400640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d901055d0 0x7f2d9019bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8d400640 1 -- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 msgr2=0x7f2d901a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8d400640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 0x7f2d901a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8d400640 1 -- 172.21.15.144:0/2998563137 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d901a1e50 con 0x7f2d9010af10 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8e800640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 0x7f2d901a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8de00640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d901055d0 0x7f2d9019bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8d400640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d9010af10 0x7f2d9019c480 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2d9006c270 tx=0x7f2d7c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00cc40 con 0x7f2d9010af10 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d901a2140 con 0x7f2d9010af10 2024-07-26T11:00:59.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d7c00cde0 con 0x7f2d9010af10 2024-07-26T11:00:59.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c018760 con 0x7f2d9010af10 2024-07-26T11:00:59.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.820+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d901a88a0 con 0x7f2d9010af10 2024-07-26T11:00:59.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.822+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d901059e0 con 0x7f2d9010af10 2024-07-26T11:00:59.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.825+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f2d7c010a90 con 0x7f2d9010af10 2024-07-26T11:00:59.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.826+0000 7f2d86a00640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2d740b3ca0 0x7f2d740b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:00:59.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.826+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f2d7c0e4330 con 0x7f2d9010af10 2024-07-26T11:00:59.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.826+0000 7f2d8de00640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2d740b3ca0 0x7f2d740b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:59.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.827+0000 7f2d8de00640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2d740b3ca0 0x7f2d740b61b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009fc0 tx=0x7f2d80005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:00:59.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:00:59.830+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2d7c0aae40 con 0x7f2d9010af10 2024-07-26T11:01:00.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.069+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d9010d6b0 con 0x7f2d9010af10 2024-07-26T11:01:00.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.070+0000 7f2d86a00640 1 -- 172.21.15.144:0/2998563137 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11931 (secure 0 0 0) 0x7f2d7c0af390 con 0x7f2d9010af10 2024-07-26T11:01:00.072 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:00.073 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":25,"btime":"2024-07-26T10:58:47:222852+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:58:46.244128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T11:01:00.073 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T11:01:00.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.073+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2d740b3ca0 msgr2=0x7f2d740b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:00.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2d740b3ca0 0x7f2d740b61b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009fc0 tx=0x7f2d80005f90 comp rx=0 tx=0).stop 2024-07-26T11:01:00.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d9010af10 msgr2=0x7f2d9019c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:00.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d9010af10 0x7f2d9019c480 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2d9006c270 tx=0x7f2d7c00ee50 comp rx=0 tx=0).stop 2024-07-26T11:01:00.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 shutdown_connections 2024-07-26T11:01:00.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2d740b3ca0 0x7f2d740b61b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d9010b860 0x7f2d901a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d9010af10 0x7f2d9019c480 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 --2- 172.21.15.144:0/2998563137 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d901055d0 0x7f2d9019bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 >> 172.21.15.144:0/2998563137 conn(0x7f2d90101040 msgr2=0x7f2d90103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 shutdown_connections 2024-07-26T11:01:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:00.074+0000 7f2d8f200640 1 -- 172.21.15.144:0/2998563137 wait complete. 2024-07-26T11:01:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:00 smithi183 ceph-mon[37273]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:00 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2998563137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:00.466 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:00 smithi144 ceph-mon[32129]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:00.467 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:00 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2998563137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:00 smithi160 ceph-mon[36715]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:00 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2998563137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:00.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:01:01.334 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:02 smithi144 ceph-mon[32129]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:02 smithi160 ceph-mon[36715]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:02 smithi183 ceph-mon[37273]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:03.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 -- 172.21.15.144:0/2538015721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98041041f0 msgr2=0x7f9804104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:03.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 --2- 172.21.15.144:0/2538015721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98041041f0 0x7f9804104600 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009ca0 tx=0x7f97f402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 -- 172.21.15.144:0/2538015721 shutdown_connections 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 --2- 172.21.15.144:0/2538015721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f980410d650 0x7f980410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 --2- 172.21.15.144:0/2538015721 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9804104b40 0x7f980410d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 --2- 172.21.15.144:0/2538015721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98041041f0 0x7f9804104600 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 -- 172.21.15.144:0/2538015721 >> 172.21.15.144:0/2538015721 conn(0x7f98040fdcf0 msgr2=0x7f9804100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 -- 172.21.15.144:0/2538015721 shutdown_connections 2024-07-26T11:01:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.499+0000 7f9803200640 1 -- 172.21.15.144:0/2538015721 wait complete. 2024-07-26T11:01:03.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.500+0000 7f9803200640 1 Processor -- start 2024-07-26T11:01:03.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.500+0000 7f9803200640 1 -- start start 2024-07-26T11:01:03.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9803200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 0x7f980419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9803200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9804104b40 0x7f980419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9803200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980410d650 0x7f98041a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9803200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9804113330 con 0x7f980410d650 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9803200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f98041131b0 con 0x7f98041041f0 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9803200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98041134b0 con 0x7f9804104b40 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9802800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980410d650 0x7f98041a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f97fbe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9804104b40 0x7f980419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9801e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 0x7f980419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f97fbe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9804104b40 0x7f980419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48584/0 (socket says 172.21.15.144:48584) 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9801e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 0x7f980419bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:52796/0 (socket says 172.21.15.144:52796) 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f97fbe00640 1 -- 172.21.15.144:0/2723405975 learned_addr learned my addr 172.21.15.144:0/2723405975 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:03.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.501+0000 7f9802800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980410d650 0x7f98041a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46368/0 (socket says 172.21.15.144:46368) 2024-07-26T11:01:03.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f9801e00640 1 -- 172.21.15.144:0/2723405975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9804104b40 msgr2=0x7f980419c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f9801e00640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9804104b40 0x7f980419c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f9801e00640 1 -- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980410d650 msgr2=0x7f98041a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f9801e00640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980410d650 0x7f98041a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f9801e00640 1 -- 172.21.15.144:0/2723405975 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98041a1e50 con 0x7f98041041f0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f9801e00640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 0x7f980419bee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f97f4000f80 tx=0x7f97f4031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.502+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4003b70 con 0x7f98041041f0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.503+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97f4003d10 con 0x7f98041041f0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.503+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98041a20e0 con 0x7f98041041f0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.503+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f402fa30 con 0x7f98041041f0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.503+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98041a88a0 con 0x7f98041041f0 2024-07-26T11:01:03.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.504+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9804104600 con 0x7f98041041f0 2024-07-26T11:01:03.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.507+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f97f4004580 con 0x7f98041041f0 2024-07-26T11:01:03.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.508+0000 7f97faa00640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f97e80b3ca0 0x7f97e80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:03.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.508+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f97f4105510 con 0x7f98041041f0 2024-07-26T11:01:03.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.508+0000 7f97fbe00640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f97e80b3ca0 0x7f97e80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:03.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.509+0000 7f97fbe00640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f97e80b3ca0 0x7f97e80b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f980410fad0 tx=0x7f97ec005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:03.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.510+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f97f40cc670 con 0x7f98041041f0 2024-07-26T11:01:03.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.706+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98041a8b90 con 0x7f98041041f0 2024-07-26T11:01:03.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.708+0000 7f97faa00640 1 -- 172.21.15.144:0/2723405975 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7f97f40cbf70 con 0x7f98041041f0 2024-07-26T11:01:03.711 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:03.712 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.711+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f97e80b3ca0 msgr2=0x7f97e80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.711+0000 7f9803200640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f97e80b3ca0 0x7f97e80b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f980410fad0 tx=0x7f97ec005f90 comp rx=0 tx=0).stop 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 msgr2=0x7f980419bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 0x7f980419bee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f97f4000f80 tx=0x7f97f4031750 comp rx=0 tx=0).stop 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 shutdown_connections 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f97e80b3ca0 0x7f97e80b61b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980410d650 0x7f98041a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9804104b40 0x7f980419c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 --2- 172.21.15.144:0/2723405975 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98041041f0 0x7f980419bee0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:03.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 >> 172.21.15.144:0/2723405975 conn(0x7f98040fdcf0 msgr2=0x7f980410c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:03.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 shutdown_connections 2024-07-26T11:01:03.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:03.712+0000 7f9803200640 1 -- 172.21.15.144:0/2723405975 wait complete. 2024-07-26T11:01:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:04 smithi144 ceph-mon[32129]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:04 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2723405975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:01:04.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:01:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:04 smithi160 ceph-mon[36715]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:04 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2723405975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:01:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:04 smithi183 ceph-mon[37273]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:04 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2723405975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:01:05.019 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:06.390 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:06 smithi144 ceph-mon[32129]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:06 smithi160 ceph-mon[36715]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:06 smithi183 ceph-mon[37273]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:07.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.282+0000 7f610ee00640 1 -- 172.21.15.144:0/3910380402 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61081033e0 msgr2=0x7f61081037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.282+0000 7f610ee00640 1 --2- 172.21.15.144:0/3910380402 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61081033e0 0x7f61081037f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6100009ca0 tx=0x7f610002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.282+0000 7f610ee00640 1 -- 172.21.15.144:0/3910380402 shutdown_connections 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.283+0000 7f610ee00640 1 --2- 172.21.15.144:0/3910380402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 0x7f610810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.283+0000 7f610ee00640 1 --2- 172.21.15.144:0/3910380402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108108d20 0x7f6108109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.283+0000 7f610ee00640 1 --2- 172.21.15.144:0/3910380402 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61081033e0 0x7f61081037f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.283+0000 7f610ee00640 1 -- 172.21.15.144:0/3910380402 >> 172.21.15.144:0/3910380402 conn(0x7f61080fee70 msgr2=0x7f61081012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.283+0000 7f610ee00640 1 -- 172.21.15.144:0/3910380402 shutdown_connections 2024-07-26T11:01:07.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.283+0000 7f610ee00640 1 -- 172.21.15.144:0/3910380402 wait complete. 2024-07-26T11:01:07.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 Processor -- start 2024-07-26T11:01:07.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 -- start start 2024-07-26T11:01:07.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61081033e0 0x7f610819bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:07.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6108108d20 0x7f610819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:07.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 0x7f61081a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:07.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6108113370 con 0x7f6108109670 2024-07-26T11:01:07.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f61081131f0 con 0x7f6108108d20 2024-07-26T11:01:07.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.284+0000 7f610ee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f61081134f0 con 0x7f61081033e0 2024-07-26T11:01:07.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610da00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61081033e0 0x7f610819bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:07.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610da00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61081033e0 0x7f610819bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48608/0 (socket says 172.21.15.144:48608) 2024-07-26T11:01:07.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610da00640 1 -- 172.21.15.144:0/2513949043 learned_addr learned my addr 172.21.15.144:0/2513949043 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:07.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610e400640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 0x7f61081a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:07.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610d000640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6108108d20 0x7f610819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:07.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610d000640 1 -- 172.21.15.144:0/2513949043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61081033e0 msgr2=0x7f610819bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610d000640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61081033e0 0x7f610819bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610d000640 1 -- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 msgr2=0x7f61081a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610d000640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 0x7f61081a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610d000640 1 -- 172.21.15.144:0/2513949043 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61081a1f30 con 0x7f6108108d20 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.285+0000 7f610e400640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 0x7f61081a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.286+0000 7f610d000640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6108108d20 0x7f610819c560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f610806b3c0 tx=0x7f60f400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.286+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f400cd00 con 0x7f6108108d20 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.286+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61081a2220 con 0x7f6108108d20 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.286+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f40035f0 con 0x7f6108108d20 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.286+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4010690 con 0x7f6108108d20 2024-07-26T11:01:07.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.286+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61081a8980 con 0x7f6108108d20 2024-07-26T11:01:07.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.288+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61081037f0 con 0x7f6108108d20 2024-07-26T11:01:07.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.291+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f60f40037a0 con 0x7f6108108d20 2024-07-26T11:01:07.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.292+0000 7f6106000640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f60ec0b3ca0 0x7f60ec0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:07.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.292+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f60f40e43c0 con 0x7f6108108d20 2024-07-26T11:01:07.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.292+0000 7f610da00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f60ec0b3ca0 0x7f60ec0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:07.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.293+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f60f40ab520 con 0x7f6108108d20 2024-07-26T11:01:07.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.293+0000 7f610da00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f60ec0b3ca0 0x7f60ec0b61b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6100009a60 tx=0x7f6100005d80 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:07.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.530+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f610819d0f0 con 0x7f6108108d20 2024-07-26T11:01:07.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.532+0000 7f6106000640 1 -- 172.21.15.144:0/2513949043 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11931 (secure 0 0 0) 0x7f60f40aae20 con 0x7f6108108d20 2024-07-26T11:01:07.534 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:07.535 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":25,"btime":"2024-07-26T10:58:47:222852+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:58:46.244128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T11:01:07.535 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f60ec0b3ca0 msgr2=0x7f60ec0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f60ec0b3ca0 0x7f60ec0b61b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6100009a60 tx=0x7f6100005d80 comp rx=0 tx=0).stop 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6108108d20 msgr2=0x7f610819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6108108d20 0x7f610819c560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f610806b3c0 tx=0x7f60f400ee50 comp rx=0 tx=0).stop 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 shutdown_connections 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f60ec0b3ca0 0x7f60ec0b61b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6108109670 0x7f61081a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6108108d20 0x7f610819c560 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 --2- 172.21.15.144:0/2513949043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61081033e0 0x7f610819bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:07.537 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 >> 172.21.15.144:0/2513949043 conn(0x7f61080fee70 msgr2=0x7f61081083d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:07.538 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 shutdown_connections 2024-07-26T11:01:07.538 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:07.535+0000 7f610ee00640 1 -- 172.21.15.144:0/2513949043 wait complete. 2024-07-26T11:01:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:08 smithi144 ceph-mon[32129]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:08 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2513949043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:08.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T10:57:25.002914+0000', 'modified': '2024-07-26T10:58:46.244128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24728, 'mds_1': 24715, 'mds_2': 24703}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24677': {'gid': 24677, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6835/405635522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6834', 'nonce': 405635522}, {'type': 'v1', 'addr': '172.21.15.160:6835', 'nonce': 405635522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24691': {'gid': 24691, 'name': 'i', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.183:6839/3458220569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6838', 'nonce': 3458220569}, {'type': 'v1', 'addr': '172.21.15.183:6839', 'nonce': 3458220569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24701': {'gid': 24701, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6837/4027087859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6836', 'nonce': 4027087859}, {'type': 'v1', 'addr': '172.21.15.160:6837', 'nonce': 4027087859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24703': {'gid': 24703, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 27, 'addr': '172.21.15.144:6839/2313694680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6838', 'nonce': 2313694680}, {'type': 'v1', 'addr': '172.21.15.144:6839', 'nonce': 2313694680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24715': {'gid': 24715, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.144:6841/1389845259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6840', 'nonce': 1389845259}, {'type': 'v1', 'addr': '172.21.15.144:6841', 'nonce': 1389845259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24728': {'gid': 24728, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.160:6839/1074049609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6838', 'nonce': 1074049609}, {'type': 'v1', 'addr': '172.21.15.160:6839', 'nonce': 1074049609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24728, 'qdb_cluster': [24728, 24715, 24703]} 2024-07-26T11:01:08.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-07-26T11:01:08.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.g status 2024-07-26T11:01:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:08 smithi160 ceph-mon[36715]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:08 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2513949043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:08 smithi183 ceph-mon[37273]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:08 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2513949043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:08.871 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:10 smithi144 ceph-mon[32129]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:10 smithi160 ceph-mon[36715]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:10 smithi183 ceph-mon[37273]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 -- 172.21.15.144:0/952789232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 msgr2=0x7f02d4112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 --2- 172.21.15.144:0/952789232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d4112090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009ca0 tx=0x7f02cc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 -- 172.21.15.144:0/952789232 shutdown_connections 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 --2- 172.21.15.144:0/952789232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d4112090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 --2- 172.21.15.144:0/952789232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d410af10 0x7f02d410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 --2- 172.21.15.144:0/952789232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d41055d0 0x7f02d41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 -- 172.21.15.144:0/952789232 >> 172.21.15.144:0/952789232 conn(0x7f02d4101040 msgr2=0x7f02d4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 -- 172.21.15.144:0/952789232 shutdown_connections 2024-07-26T11:01:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.196+0000 7f02d3e00640 1 -- 172.21.15.144:0/952789232 wait complete. 2024-07-26T11:01:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 Processor -- start 2024-07-26T11:01:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 -- start start 2024-07-26T11:01:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41055d0 0x7f02d419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 0x7f02d419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d41a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f02d4115580 con 0x7f02d410af10 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f02d4115400 con 0x7f02d410b860 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.198+0000 7f02d3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f02d4115700 con 0x7f02d41055d0 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41055d0 0x7f02d419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 0x7f02d419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41055d0 0x7f02d419bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:38208/0 (socket says 172.21.15.144:38208) 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 0x7f02d419c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60956/0 (socket says 172.21.15.144:60956) 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2a00640 1 -- 172.21.15.144:0/936163603 learned_addr learned my addr 172.21.15.144:0/936163603 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d3400640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d41a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 -- 172.21.15.144:0/936163603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41055d0 msgr2=0x7f02d419bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41055d0 0x7f02d419bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 -- 172.21.15.144:0/936163603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 msgr2=0x7f02d41a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d41a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d2000640 1 -- 172.21.15.144:0/936163603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d41a1e40 con 0x7f02d410af10 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.199+0000 7f02d3400640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d41a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.200+0000 7f02d2000640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 0x7f02d419c470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f02d406c270 tx=0x7f02c000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.200+0000 7f02d8a00640 1 -- 172.21.15.144:0/936163603 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c000cc40 con 0x7f02d410af10 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.200+0000 7f02d8a00640 1 -- 172.21.15.144:0/936163603 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c000cde0 con 0x7f02d410af10 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.200+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d41a2130 con 0x7f02d410af10 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.200+0000 7f02d8a00640 1 -- 172.21.15.144:0/936163603 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0004a00 con 0x7f02d410af10 2024-07-26T11:01:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.200+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d41a8890 con 0x7f02d410af10 2024-07-26T11:01:11.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.203+0000 7f02e0a90740 1 Processor -- start 2024-07-26T11:01:11.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 -- start start 2024-07-26T11:01:11.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02d8a00640 1 -- 172.21.15.144:0/936163603 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f02c0004ba0 con 0x7f02d410af10 2024-07-26T11:01:11.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e321ef60 0x5642e35c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 0x5642e35c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5642e353a950 con 0x5642e31f60a0 2024-07-26T11:01:11.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642e3529e50 con 0x5642e31fe280 2024-07-26T11:01:11.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02e0a90740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642e352de90 con 0x5642e321ef60 2024-07-26T11:01:11.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.204+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 0x5642e35c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 0x5642e35c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60970/0 (socket says 172.21.15.144:60970) 2024-07-26T11:01:11.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 -- 172.21.15.144:0/733292199 learned_addr learned my addr 172.21.15.144:0/733292199 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:11.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2000640 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d3400640 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e321ef60 0x5642e35c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 -- 172.21.15.144:0/733292199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e321ef60 msgr2=0x5642e35c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e321ef60 0x5642e35c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 -- 172.21.15.144:0/733292199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 msgr2=0x5642e35c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d2a00640 1 -- 172.21.15.144:0/733292199 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642e34d10f0 con 0x5642e31f60a0 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.205+0000 7f02d3400640 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e321ef60 0x5642e35c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.206+0000 7f02d2a00640 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 0x5642e35c4ca0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009dc0 tx=0x7f02c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f1e08bebd44891 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:11.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.206+0000 7f02bfe00640 1 -- 172.21.15.144:0/733292199 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4005220 con 0x5642e31f60a0 2024-07-26T11:01:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.206+0000 7f02bfe00640 1 -- 172.21.15.144:0/733292199 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c4004c50 con 0x5642e31f60a0 2024-07-26T11:01:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.206+0000 7f02bfe00640 1 -- 172.21.15.144:0/733292199 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c40197f0 con 0x5642e31f60a0 2024-07-26T11:01:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.206+0000 7f02d8a00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02b80b3ca0 0x7f02b80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02d8a00640 1 -- 172.21.15.144:0/936163603 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f02c00e3fe0 con 0x7f02d410af10 2024-07-26T11:01:11.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02d3400640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02b80b3ca0 0x7f02b80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 -- 172.21.15.144:0/733292199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 msgr2=0x5642e35c4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 0x5642e35c4ca0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009dc0 tx=0x7f02c4011810 comp rx=0 tx=0).stop 2024-07-26T11:01:11.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 -- 172.21.15.144:0/733292199 shutdown_connections 2024-07-26T11:01:11.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31f60a0 0x5642e35c4ca0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 --2- 172.21.15.144:0/733292199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e321ef60 0x5642e35c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 -- 172.21.15.144:0/733292199 >> 172.21.15.144:0/733292199 conn(0x5642e3213350 msgr2=0x5642e35c2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02e0a90740 1 -- 172.21.15.144:0/733292199 shutdown_connections 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.207+0000 7f02d3400640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02b80b3ca0 0x7f02b80b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02cc006010 tx=0x7f02cc005d80 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.208+0000 7f02e0a90740 1 -- 172.21.15.144:0/733292199 wait complete. 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.208+0000 7f02e0a90740 1 -- start start 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.208+0000 7f02e0a90740 1 Processor -- start 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35dc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5642e338d5f0 con 0x5642e321ef60 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642e34c6520 con 0x5642e31fe280 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642e35d2f40 con 0x5642e31f60a0 2024-07-26T11:01:11.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02e0a90740 10 client.0 _ll_register_callbacks cb 0x5642e34d5ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:01:11.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02d3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35dc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02d3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35dc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60982/0 (socket says 172.21.15.144:60982) 2024-07-26T11:01:11.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02d3400640 1 -- 172.21.15.144:0/3195566321 learned_addr learned my addr 172.21.15.144:0/3195566321 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:11.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.209+0000 7f02d2000640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d3400640 1 -- 172.21.15.144:0/3195566321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 msgr2=0x5642e35cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d3400640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d3400640 1 -- 172.21.15.144:0/3195566321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 msgr2=0x5642e35d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d3400640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d3400640 1 -- 172.21.15.144:0/3195566321 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642e328a540 con 0x5642e321ef60 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d2000640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02d3400640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35dc230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009ca0 tx=0x7f02cc02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f94d4dc4a1a3d3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.210+0000 7f02bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02bcc00640 1 -- 172.21.15.144:0/3195566321 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc032eb0 con 0x5642e321ef60 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02bcc00640 1 -- 172.21.15.144:0/3195566321 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f02cc004250 con 0x5642e321ef60 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c4004c50 con 0x5642e321ef60 2024-07-26T11:01:11.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02bcc00640 1 -- 172.21.15.144:0/3195566321 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f02cc004450 con 0x5642e321ef60 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02e0a90740 10 client.15033 fetch_fsmap learned FSMap version 25 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5642e35deda0 con 0x5642e321ef60 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02bcc00640 1 -- 172.21.15.144:0/3195566321 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c4004c50 con 0x5642e321ef60 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.211+0000 7f02bcc00640 1 -- 172.21.15.144:0/3195566321 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f02cc052070 con 0x5642e321ef60 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02e0a90740 10 client.15033 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02e0a90740 10 client.15033 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02e0a90740 20 client.15033 populate_metadata read hostname 'smithi144' 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02e0a90740 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31dd8e0 0x5642e35e28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02e0a90740 4 client.15033 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:01:11.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642e353c8a0 con 0x5642e31dd8e0 2024-07-26T11:01:11.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.212+0000 7f02d2a00640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31dd8e0 0x5642e35e28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.213+0000 7f02d2a00640 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31dd8e0 0x5642e35e28a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6842bdd34a6f8b2e server_cookie=cff39f79470aab6b in_seq=0 out_seq=0 2024-07-26T11:01:11.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.213+0000 7f02bcc00640 10 client.15033 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02bcc00640 1 -- 172.21.15.144:0/3195566321 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5642e353c8a0 con 0x5642e31dd8e0 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02bcc00640 10 client.15033 handle_command_reply: tid=0 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 client.15033 shutdown 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 msgr2=0x5642e35dc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35dc230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009ca0 tx=0x7f02cc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 shutdown_connections 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31dd8e0 0x5642e35e28a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35dc230 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31fe280 0x5642e35d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 --2- 172.21.15.144:0/3195566321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02bcc00640 0 client.15033 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:01:11.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 >> 172.21.15.144:0/3195566321 conn(0x5642e3213350 msgr2=0x5642e35c78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:11.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 shutdown_connections 2024-07-26T11:01:11.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.214+0000 7f02e0a90740 1 -- 172.21.15.144:0/3195566321 wait complete. 2024-07-26T11:01:11.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.215+0000 7f02e0a90740 20 client.15033 trim_cache size 0 max 16384 2024-07-26T11:01:11.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.234+0000 7f02e0a90740 1 Processor -- start 2024-07-26T11:01:11.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.234+0000 7f02e0a90740 1 -- start start 2024-07-26T11:01:11.237 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:11 smithi144 ceph-mon[32129]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:11.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35ca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31fe280 0x5642e35e28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e321ef60 0x5642e35cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02e0a90740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5642e35e0be0 con 0x5642e31fe280 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02e0a90740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642e35e12b0 con 0x5642e321ef60 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02e0a90740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642e35e1980 con 0x5642e31f60a0 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02d2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:38226/0 (socket says 172.21.15.144:38226) 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02d2a00640 1 -- 172.21.15.144:0/565096222 learned_addr learned my addr 172.21.15.144:0/565096222 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02d2000640 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31fe280 0x5642e35e28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.235+0000 7f02d3400640 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e321ef60 0x5642e35cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02d2a00640 1 -- 172.21.15.144:0/565096222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e321ef60 msgr2=0x5642e35cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02d2a00640 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e321ef60 0x5642e35cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02d2a00640 1 -- 172.21.15.144:0/565096222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31fe280 msgr2=0x5642e35e28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02d2a00640 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31fe280 0x5642e35e28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02d2a00640 1 -- 172.21.15.144:0/565096222 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642e35cb7e0 con 0x5642e31f60a0 2024-07-26T11:01:11.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02d2a00640 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35ca880 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02c40057c0 tx=0x7f02c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda14ca13ada4f6c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02bf400640 1 -- 172.21.15.144:0/565096222 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4031070 con 0x5642e31f60a0 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02bf400640 1 -- 172.21.15.144:0/565096222 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c40044f0 con 0x5642e31f60a0 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.236+0000 7f02bf400640 1 -- 172.21.15.144:0/565096222 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c40054b0 con 0x5642e31f60a0 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.237+0000 7f02e0a90740 1 -- 172.21.15.144:0/565096222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 msgr2=0x5642e35ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.237+0000 7f02e0a90740 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35ca880 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02c40057c0 tx=0x7f02c4011810 comp rx=0 tx=0).stop 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 -- 172.21.15.144:0/565096222 shutdown_connections 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e321ef60 0x5642e35cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e31fe280 0x5642e35e28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 --2- 172.21.15.144:0/565096222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31f60a0 0x5642e35ca880 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 -- 172.21.15.144:0/565096222 >> 172.21.15.144:0/565096222 conn(0x5642e3213350 msgr2=0x5642e35ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 -- 172.21.15.144:0/565096222 shutdown_connections 2024-07-26T11:01:11.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 -- 172.21.15.144:0/565096222 wait complete. 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.238+0000 7f02e0a90740 1 -- start start 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 Processor -- start 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31f60a0 0x5642e35eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31fe280 0x5642e35e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35e7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5642e35e2720 con 0x5642e321ef60 2024-07-26T11:01:11.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642e35e2050 con 0x5642e31f60a0 2024-07-26T11:01:11.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642e35e2df0 con 0x5642e31fe280 2024-07-26T11:01:11.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.239+0000 7f02e0a90740 10 client.0 _ll_register_callbacks cb 0x7f02cc031960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31fe280 0x5642e35e6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31fe280 0x5642e35e6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:38242/0 (socket says 172.21.15.144:38242) 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d2000640 1 -- 172.21.15.144:0/2351619428 learned_addr learned my addr 172.21.15.144:0/2351619428 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d2a00640 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31f60a0 0x5642e35eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d3400640 1 -- 172.21.15.144:0/2351619428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31fe280 msgr2=0x5642e35e6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d3400640 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31fe280 0x5642e35e6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d3400640 1 -- 172.21.15.144:0/2351619428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31f60a0 msgr2=0x5642e35eb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d3400640 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31f60a0 0x5642e35eb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.240+0000 7f02d3400640 1 -- 172.21.15.144:0/2351619428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642e35e7660 con 0x5642e321ef60 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.241+0000 7f02d3400640 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35e7070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f02cc005e30 tx=0x7f02cc02f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286a80db2eb9fd0f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.241+0000 7f02bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.241+0000 7f02bcc00640 1 -- 172.21.15.144:0/2351619428 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc032eb0 con 0x5642e321ef60 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.241+0000 7f02bcc00640 1 -- 172.21.15.144:0/2351619428 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f02cc004250 con 0x5642e321ef60 2024-07-26T11:01:11.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.241+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c40044f0 con 0x5642e321ef60 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.241+0000 7f02bcc00640 1 -- 172.21.15.144:0/2351619428 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f02cc06a050 con 0x5642e321ef60 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 10 client.15039 fetch_fsmap learned FSMap version 25 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5642e364a8d0 con 0x5642e321ef60 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02bcc00640 1 -- 172.21.15.144:0/2351619428 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c40044f0 con 0x5642e321ef60 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02bcc00640 1 -- 172.21.15.144:0/2351619428 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f02cc061070 con 0x5642e321ef60 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 10 client.15039 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 10 client.15039 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T11:01:11.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 20 client.15039 populate_metadata read hostname 'smithi144' 2024-07-26T11:01:11.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31b2c90 0x5642e364e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:11.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 4 client.15039 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:01:11.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.242+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "status"}) -- 0x5642e3538340 con 0x5642e31b2c90 2024-07-26T11:01:11.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.243+0000 7f02d2a00640 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31b2c90 0x5642e364e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02d2a00640 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31b2c90 0x5642e364e230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b738ad8cf7341ae3 server_cookie=4539cef6179f02b7 in_seq=0 out_seq=0 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02bcc00640 10 client.15039 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02bcc00640 1 -- 172.21.15.144:0/2351619428 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5642e3538340 con 0x5642e31b2c90 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02bcc00640 10 client.15039 handle_command_reply: tid=0 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 client.15039 shutdown 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 msgr2=0x5642e35e7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35e7070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f02cc005e30 tx=0x7f02cc02f8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 shutdown_connections 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x5642e31b2c90 0x5642e364e230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642e321ef60 0x5642e35e7070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.244+0000 7f02e0a90740 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642e31fe280 0x5642e35e6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.245+0000 7f02e0a90740 1 --2- 172.21.15.144:0/2351619428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642e31f60a0 0x5642e35eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.245+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 >> 172.21.15.144:0/2351619428 conn(0x5642e3213350 msgr2=0x5642e35d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.245+0000 7f02bcc00640 0 client.15039 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.245+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 shutdown_connections 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.245+0000 7f02e0a90740 1 -- 172.21.15.144:0/2351619428 wait complete. 2024-07-26T11:01:11.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.245+0000 7f02e0a90740 20 client.15039 trim_cache size 0 max 16384 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 2, 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24703, 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 153.10467814699999, 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 25, 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 90, 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 253.40705092499999 2024-07-26T11:01:11.249 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:01:11.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02b80b3ca0 msgr2=0x7f02b80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02b80b3ca0 0x7f02b80b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02cc006010 tx=0x7f02cc005d80 comp rx=0 tx=0).stop 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 msgr2=0x7f02d419c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 0x7f02d419c470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f02d406c270 tx=0x7f02c000ee50 comp rx=0 tx=0).stop 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 shutdown_connections 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02b80b3ca0 0x7f02b80b61b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02d410b860 0x7f02d41a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d410af10 0x7f02d419c470 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 --2- 172.21.15.144:0/936163603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41055d0 0x7f02d419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 >> 172.21.15.144:0/936163603 conn(0x7f02d4101040 msgr2=0x7f02d410a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 shutdown_connections 2024-07-26T11:01:11.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:11.246+0000 7f02d3e00640 1 -- 172.21.15.144:0/936163603 wait complete. 2024-07-26T11:01:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:11 smithi160 ceph-mon[36715]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:11 smithi183 ceph-mon[37273]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:12.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.a status 2024-07-26T11:01:12.550 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:14.328 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:14 smithi144 ceph-mon[32129]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:14 smithi183 ceph-mon[37273]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:14 smithi160 ceph-mon[36715]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 -- 172.21.15.144:0/543597245 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c10af10 msgr2=0x7f774c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 --2- 172.21.15.144:0/543597245 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c10af10 0x7f774c10b320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7734009860 tx=0x7f773402a340 comp rx=0 tx=0).stop 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 -- 172.21.15.144:0/543597245 shutdown_connections 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 --2- 172.21.15.144:0/543597245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 0x7f774c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 --2- 172.21.15.144:0/543597245 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c10af10 0x7f774c10b320 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 --2- 172.21.15.144:0/543597245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f774c1055d0 0x7f774c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 -- 172.21.15.144:0/543597245 >> 172.21.15.144:0/543597245 conn(0x7f774c101040 msgr2=0x7f774c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 -- 172.21.15.144:0/543597245 shutdown_connections 2024-07-26T11:01:14.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.882+0000 7f7750a00640 1 -- 172.21.15.144:0/543597245 wait complete. 2024-07-26T11:01:14.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.884+0000 7f7750a00640 1 Processor -- start 2024-07-26T11:01:14.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.884+0000 7f7750a00640 1 -- start start 2024-07-26T11:01:14.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7750a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 0x7f774c19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7750a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f774c10af10 0x7f774c19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7750a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 0x7f774c1a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7750a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f774c1154f0 con 0x7f774c10af10 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7750a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f774c115370 con 0x7f774c1055d0 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7750a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f774c115670 con 0x7f774c10b860 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f7741e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f774c10af10 0x7f774c19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f774a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 0x7f774c19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f774a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 0x7f774c19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36424/0 (socket says 172.21.15.144:36424) 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f774a800640 1 -- 172.21.15.144:0/1877078758 learned_addr learned my addr 172.21.15.144:0/1877078758 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:14.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.885+0000 7f774b200640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 0x7f774c1a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774a800640 1 -- 172.21.15.144:0/1877078758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 msgr2=0x7f774c1a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774a800640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 0x7f774c1a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774a800640 1 -- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f774c10af10 msgr2=0x7f774c19c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774a800640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f774c10af10 0x7f774c19c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774a800640 1 -- 172.21.15.144:0/1877078758 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c1a1e70 con 0x7f774c1055d0 2024-07-26T11:01:14.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774b200640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 0x7f774c1a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:01:14.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f774a800640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 0x7f774c19bf00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f773c00d950 tx=0x7f773c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:14.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f7743e00640 1 -- 172.21.15.144:0/1877078758 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c017030 con 0x7f774c1055d0 2024-07-26T11:01:14.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f7743e00640 1 -- 172.21.15.144:0/1877078758 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f773c004020 con 0x7f774c1055d0 2024-07-26T11:01:14.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774c1a2160 con 0x7f774c1055d0 2024-07-26T11:01:14.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.886+0000 7f7743e00640 1 -- 172.21.15.144:0/1877078758 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0047a0 con 0x7f774c1055d0 2024-07-26T11:01:14.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.887+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774c1a88c0 con 0x7f774c1055d0 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.890+0000 7f775808d740 1 Processor -- start 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.890+0000 7f775808d740 1 -- start start 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f775808d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b22760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f775808d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b22ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f775808d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f775808d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x558a24a98950 con 0x558a247540a0 2024-07-26T11:01:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f775808d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558a24a87e50 con 0x558a2477cf60 2024-07-26T11:01:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f775808d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558a24a8be90 con 0x558a2475c280 2024-07-26T11:01:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f774b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b22760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f774b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b22760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36440/0 (socket says 172.21.15.144:36440) 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f774b200640 1 -- 172.21.15.144:0/3524630959 learned_addr learned my addr 172.21.15.144:0/3524630959 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.891+0000 7f7741e00640 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f774b200640 1 -- 172.21.15.144:0/3524630959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 msgr2=0x558a24b231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f774b200640 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b231e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f774b200640 1 -- 172.21.15.144:0/3524630959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 msgr2=0x558a24b22ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f774b200640 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b22ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f774b200640 1 -- 172.21.15.144:0/3524630959 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a24a2f0f0 con 0x558a2477cf60 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f7743e00640 1 -- 172.21.15.144:0/1877078758 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f773c009a30 con 0x7f774c1055d0 2024-07-26T11:01:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f774b200640 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b22760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f774400f400 tx=0x7f774400f8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a56dcc1f013394d7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:14.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.892+0000 7f773b400640 1 -- 172.21.15.144:0/3524630959 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774401d070 con 0x558a2477cf60 2024-07-26T11:01:14.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.893+0000 7f773b400640 1 -- 172.21.15.144:0/3524630959 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f774400cb50 con 0x558a2477cf60 2024-07-26T11:01:14.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.893+0000 7f773b400640 1 -- 172.21.15.144:0/3524630959 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744018b80 con 0x558a2477cf60 2024-07-26T11:01:14.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.893+0000 7f775808d740 1 -- 172.21.15.144:0/3524630959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 msgr2=0x558a24b22760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.893+0000 7f775808d740 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b22760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f774400f400 tx=0x7f774400f8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 -- 172.21.15.144:0/3524630959 shutdown_connections 2024-07-26T11:01:14.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b231e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b22ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 --2- 172.21.15.144:0/3524630959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b22760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 -- 172.21.15.144:0/3524630959 >> 172.21.15.144:0/3524630959 conn(0x558a24771350 msgr2=0x558a24b20650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f7743e00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77300b3c20 0x7f77300b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 -- 172.21.15.144:0/3524630959 shutdown_connections 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f7743e00640 1 -- 172.21.15.144:0/1877078758 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f773c0e3c30 con 0x7f774c1055d0 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f775808d740 1 -- 172.21.15.144:0/3524630959 wait complete. 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.894+0000 7f7741e00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77300b3c20 0x7f77300b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f775808d740 1 -- start start 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f775808d740 1 Processor -- start 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f7741e00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77300b3c20 0x7f77300b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7734035040 tx=0x7f773402c040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f775808d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b2e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f775808d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b2e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f775808d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b3a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.895+0000 7f775808d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x558a248eb5f0 con 0x558a247540a0 2024-07-26T11:01:14.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f775808d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558a24a24520 con 0x558a2477cf60 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f775808d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558a24b30f40 con 0x558a2475c280 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f775808d740 10 client.0 _ll_register_callbacks cb 0x558a24a33ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f7741e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b2e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b2e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f7741e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b2e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:38276/0 (socket says 172.21.15.144:38276) 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f7741e00640 1 -- 172.21.15.144:0/2944978900 learned_addr learned my addr 172.21.15.144:0/2944978900 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774b200640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b3a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774a800640 1 -- 172.21.15.144:0/2944978900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 msgr2=0x558a24b2e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774a800640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b2e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774a800640 1 -- 172.21.15.144:0/2944978900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 msgr2=0x558a24b3a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774a800640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b3a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774a800640 1 -- 172.21.15.144:0/2944978900 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a247e8540 con 0x558a247540a0 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f774b200640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b3a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:14.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.896+0000 7f7741e00640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b2e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f774a800640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b2e240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x558a24b213e0 tx=0x7f773c0a9f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9596badf5f68952 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f772fe00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f772fe00640 1 -- 172.21.15.144:0/2944978900 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0aec90 con 0x558a247540a0 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f772fe00640 1 -- 172.21.15.144:0/2944978900 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f773c0041d0 con 0x558a247540a0 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f774400cb50 con 0x558a247540a0 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f772fe00640 1 -- 172.21.15.144:0/2944978900 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f773c0b3740 con 0x558a247540a0 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f775808d740 10 client.15048 fetch_fsmap learned FSMap version 25 2024-07-26T11:01:14.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.897+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558a24b3cef0 con 0x558a247540a0 2024-07-26T11:01:14.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f772fe00640 1 -- 172.21.15.144:0/2944978900 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f774400cb50 con 0x558a247540a0 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f772fe00640 1 -- 172.21.15.144:0/2944978900 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f773c0cbcb0 con 0x558a247540a0 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f775808d740 10 client.15048 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f775808d740 10 client.15048 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f775808d740 20 client.15048 populate_metadata read hostname 'smithi144' 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f775808d740 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a2473b8e0 0x558a24b408e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f775808d740 4 client.15048 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a24a9a8a0 con 0x558a2473b8e0 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.898+0000 7f7741e00640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a2473b8e0 0x558a24b408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.899+0000 7f7741e00640 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a2473b8e0 0x558a24b408e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df8db7aba3f79185 server_cookie=211f57a81ceaf3d6 in_seq=0 out_seq=0 2024-07-26T11:01:14.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.899+0000 7f772fe00640 10 client.15048 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f772fe00640 1 -- 172.21.15.144:0/2944978900 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a24a9a8a0 con 0x558a2473b8e0 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f772fe00640 10 client.15048 handle_command_reply: tid=0 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 client.15048 shutdown 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 msgr2=0x558a24b2e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b2e240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x558a24b213e0 tx=0x7f773c0a9f00 comp rx=0 tx=0).stop 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 shutdown_connections 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a2473b8e0 0x558a24b408e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2477cf60 0x558a24b3a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2475c280 0x558a24b2e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 --2- 172.21.15.144:0/2944978900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b2e240 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f772fe00640 0 client.15048 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.900+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 >> 172.21.15.144:0/2944978900 conn(0x558a24771350 msgr2=0x558a24b258f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.901+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 shutdown_connections 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.901+0000 7f775808d740 1 -- 172.21.15.144:0/2944978900 wait complete. 2024-07-26T11:01:14.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.901+0000 7f775808d740 20 client.15048 trim_cache size 0 max 16384 2024-07-26T11:01:14.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 Processor -- start 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 -- start start 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a247540a0 0x558a24b3f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b28880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 0x558a24b3a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x558a24b3ec20 con 0x558a2477cf60 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558a24b3f2f0 con 0x558a2475c280 2024-07-26T11:01:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.921+0000 7f775808d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558a24b3f9c0 con 0x558a247540a0 2024-07-26T11:01:14.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 0x558a24b3a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a247540a0 0x558a24b3f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 0x558a24b3a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:32802/0 (socket says 172.21.15.144:32802) 2024-07-26T11:01:14.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774b200640 1 -- 172.21.15.144:0/2176689496 learned_addr learned my addr 172.21.15.144:0/2176689496 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f7741e00640 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b28880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 -- 172.21.15.144:0/2176689496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 msgr2=0x558a24b28880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b28880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 -- 172.21.15.144:0/2176689496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 msgr2=0x558a24b3a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 0x558a24b3a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 -- 172.21.15.144:0/2176689496 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a24b1b7d0 con 0x558a247540a0 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f7741e00640 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b28880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774b200640 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 0x558a24b3a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.922+0000 7f774a800640 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a247540a0 0x558a24b3f470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f773c0a6710 tx=0x7f773c0a6000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d5184a650734402 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.923+0000 7f773aa00640 1 -- 172.21.15.144:0/2176689496 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c014070 con 0x558a247540a0 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.923+0000 7f773aa00640 1 -- 172.21.15.144:0/2176689496 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f773c0aec90 con 0x558a247540a0 2024-07-26T11:01:14.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.923+0000 7f773aa00640 1 -- 172.21.15.144:0/2176689496 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0a6920 con 0x558a247540a0 2024-07-26T11:01:14.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 -- 172.21.15.144:0/2176689496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a247540a0 msgr2=0x558a24b3f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a247540a0 0x558a24b3f470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f773c0a6710 tx=0x7f773c0a6000 comp rx=0 tx=0).stop 2024-07-26T11:01:14.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 -- 172.21.15.144:0/2176689496 shutdown_connections 2024-07-26T11:01:14.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a2477cf60 0x558a24b3a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b28880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 --2- 172.21.15.144:0/2176689496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a247540a0 0x558a24b3f470 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 -- 172.21.15.144:0/2176689496 >> 172.21.15.144:0/2176689496 conn(0x558a24771350 msgr2=0x558a24b291b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 -- 172.21.15.144:0/2176689496 shutdown_connections 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 -- 172.21.15.144:0/2176689496 wait complete. 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.924+0000 7f775808d740 1 -- start start 2024-07-26T11:01:14.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 Processor -- start 2024-07-26T11:01:14.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b491c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b41ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2477cf60 0x558a24b42020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x558a24b40760 con 0x558a247540a0 2024-07-26T11:01:14.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558a24b40090 con 0x558a2475c280 2024-07-26T11:01:14.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558a24b40e30 con 0x558a2477cf60 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f775808d740 10 client.0 _ll_register_callbacks cb 0x7f773c0aa0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f774a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f7741e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b41ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f774a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:32814/0 (socket says 172.21.15.144:32814) 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f7741e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b41ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36468/0 (socket says 172.21.15.144:36468) 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.925+0000 7f774a800640 1 -- 172.21.15.144:0/1357136539 learned_addr learned my addr 172.21.15.144:0/1357136539 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f774b200640 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2477cf60 0x558a24b42020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f7741e00640 1 -- 172.21.15.144:0/1357136539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2477cf60 msgr2=0x558a24b42020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f7741e00640 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2477cf60 0x558a24b42020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f7741e00640 1 -- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 msgr2=0x558a24b491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f7741e00640 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f7741e00640 1 -- 172.21.15.144:0/1357136539 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a24b42710 con 0x558a2475c280 2024-07-26T11:01:14.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f7741e00640 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b41ae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x558a24b40fe0 tx=0x7f773402a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2929b7073eb82a8b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.926+0000 7f772fe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f772fe00640 1 -- 172.21.15.144:0/1357136539 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734037040 con 0x558a2475c280 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f772fe00640 1 -- 172.21.15.144:0/1357136539 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f7734003970 con 0x558a2475c280 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f773c0aec90 con 0x558a2475c280 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f772fe00640 1 -- 172.21.15.144:0/1357136539 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7734003b70 con 0x558a2475c280 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f775808d740 10 client.25202 fetch_fsmap learned FSMap version 25 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558a24ba89d0 con 0x558a2475c280 2024-07-26T11:01:14.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.927+0000 7f772fe00640 1 -- 172.21.15.144:0/1357136539 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f773c0aec90 con 0x558a2475c280 2024-07-26T11:01:14.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f772fe00640 1 -- 172.21.15.144:0/1357136539 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f7734049040 con 0x558a2475c280 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f775808d740 10 client.25202 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f775808d740 10 client.25202 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f775808d740 20 client.25202 populate_metadata read hostname 'smithi144' 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f775808d740 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a24710c90 0x558a24bac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f775808d740 4 client.25202 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "status"}) -- 0x558a24a96340 con 0x558a24710c90 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.928+0000 7f774a800640 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a24710c90 0x558a24bac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f774a800640 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a24710c90 0x558a24bac1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=28d3db88369851a server_cookie=854ce38560a927b8 in_seq=0 out_seq=0 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f772fe00640 10 client.25202 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f772fe00640 1 -- 172.21.15.144:0/1357136539 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558a24a96340 con 0x558a24710c90 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f772fe00640 10 client.25202 handle_command_reply: tid=0 2024-07-26T11:01:14.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 client.25202 shutdown 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 msgr2=0x558a24b41ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b41ae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x558a24b40fe0 tx=0x7f773402a340 comp rx=0 tx=0).stop 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 shutdown_connections 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558a2477cf60 0x558a24b42020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x558a24710c90 0x558a24bac1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558a2475c280 0x558a24b41ae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 --2- 172.21.15.144:0/1357136539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558a247540a0 0x558a24b491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f772fe00640 0 client.25202 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.929+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 >> 172.21.15.144:0/1357136539 conn(0x558a24771350 msgr2=0x558a24b1fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:14.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.930+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 shutdown_connections 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.930+0000 7f775808d740 1 -- 172.21.15.144:0/1357136539 wait complete. 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.930+0000 7f775808d740 20 client.25202 trim_cache size 0 max 16384 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 1, 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24715, 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 157.799798428, 2024-07-26T11:01:14.935 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 25, 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 90, 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 252.28648085200001 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77300b3c20 msgr2=0x7f77300b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77300b3c20 0x7f77300b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7734035040 tx=0x7f773402c040 comp rx=0 tx=0).stop 2024-07-26T11:01:14.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 msgr2=0x7f774c19bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 0x7f774c19bf00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f773c00d950 tx=0x7f773c00de10 comp rx=0 tx=0).stop 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 shutdown_connections 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77300b3c20 0x7f77300b6130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f774c10b860 0x7f774c1a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f774c10af10 0x7f774c19c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 --2- 172.21.15.144:0/1877078758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f774c1055d0 0x7f774c19bf00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 >> 172.21.15.144:0/1877078758 conn(0x7f774c101040 msgr2=0x7f774c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 shutdown_connections 2024-07-26T11:01:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:14.931+0000 7f7750a00640 1 -- 172.21.15.144:0/1877078758 wait complete. 2024-07-26T11:01:15.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T11:01:16.333 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:16 smithi144 ceph-mon[32129]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:16 smithi160 ceph-mon[36715]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:16 smithi183 ceph-mon[37273]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:18.360 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:18 smithi144 ceph-mon[32129]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:18 smithi160 ceph-mon[36715]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 -- 172.21.15.144:0/4127696527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 msgr2=0x7f82901059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 --2- 172.21.15.144:0/4127696527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f82901059e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8284009860 tx=0x7f828402a340 comp rx=0 tx=0).stop 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 -- 172.21.15.144:0/4127696527 shutdown_connections 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 --2- 172.21.15.144:0/4127696527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829010b860 0x7f8290112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 --2- 172.21.15.144:0/4127696527 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010af10 0x7f829010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 --2- 172.21.15.144:0/4127696527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f82901059e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 -- 172.21.15.144:0/4127696527 >> 172.21.15.144:0/4127696527 conn(0x7f8290101040 msgr2=0x7f8290103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 -- 172.21.15.144:0/4127696527 shutdown_connections 2024-07-26T11:01:18.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.563+0000 7f8296c00640 1 -- 172.21.15.144:0/4127696527 wait complete. 2024-07-26T11:01:18.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.564+0000 7f8296c00640 1 Processor -- start 2024-07-26T11:01:18.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 -- start start 2024-07-26T11:01:18.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f829019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829010af10 0x7f829019c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010b860 0x7f82901a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8290115510 con 0x7f829010af10 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8290115390 con 0x7f829010b860 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8296c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8290115690 con 0x7f82901055d0 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.565+0000 7f8295800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f829019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010b860 0x7f82901a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8295800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f829019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:38322/0 (socket says 172.21.15.144:38322) 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8294e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829010af10 0x7f829019c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8295800640 1 -- 172.21.15.144:0/4070546945 learned_addr learned my addr 172.21.15.144:0/4070546945 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 -- 172.21.15.144:0/4070546945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 msgr2=0x7f829019bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f829019bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 -- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829010af10 msgr2=0x7f829019c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829010af10 0x7f829019c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 -- 172.21.15.144:0/4070546945 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82901a1eb0 con 0x7f829010b860 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.566+0000 7f8296200640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010b860 0x7f82901a1780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f828800efc0 tx=0x7f828800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:18.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.567+0000 7f828e000640 1 -- 172.21.15.144:0/4070546945 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8288003970 con 0x7f829010b860 2024-07-26T11:01:18.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.567+0000 7f828e000640 1 -- 172.21.15.144:0/4070546945 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8288004020 con 0x7f829010b860 2024-07-26T11:01:18.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.567+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82901a21a0 con 0x7f829010b860 2024-07-26T11:01:18.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.567+0000 7f828e000640 1 -- 172.21.15.144:0/4070546945 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828800b940 con 0x7f829010b860 2024-07-26T11:01:18.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.567+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82901a8900 con 0x7f829010b860 2024-07-26T11:01:18.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.570+0000 7f829eebd740 1 Processor -- start 2024-07-26T11:01:18.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f828e000640 1 -- 172.21.15.144:0/4070546945 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f828801d020 con 0x7f829010b860 2024-07-26T11:01:18.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 -- start start 2024-07-26T11:01:18.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397a7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 0x5637397a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393e1280 0x5637397a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56373971d950 con 0x5637393e1280 2024-07-26T11:01:18.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56373970ce50 con 0x5637393d90a0 2024-07-26T11:01:18.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.571+0000 7f829eebd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563739710e90 con 0x563739401f60 2024-07-26T11:01:18.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8294e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 0x5637397a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8294e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393e1280 0x5637397a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 0x5637397a7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36496/0 (socket says 172.21.15.144:36496) 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 -- 172.21.15.144:0/3942969699 learned_addr learned my addr 172.21.15.144:0/3942969699 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 -- 172.21.15.144:0/3942969699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 msgr2=0x5637397a7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397a7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 -- 172.21.15.144:0/3942969699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393e1280 msgr2=0x5637397a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393e1280 0x5637397a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8295800640 1 -- 172.21.15.144:0/3942969699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637396b40f0 con 0x5637393d90a0 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.572+0000 7f8294e00640 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393e1280 0x5637397a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.573+0000 7f828e000640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f82780b3ca0 0x7f82780b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.573+0000 7f8295800640 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 0x5637397a7ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8284000c00 tx=0x7f8284009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27aec9df69d4651c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:18.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.573+0000 7f828e000640 1 -- 172.21.15.144:0/4070546945 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f82880e3f50 con 0x7f829010b860 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.573+0000 7f8294e00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f82780b3ca0 0x7f82780b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.573+0000 7f8282a00640 1 -- 172.21.15.144:0/3942969699 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284037040 con 0x5637393d90a0 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.573+0000 7f8282a00640 1 -- 172.21.15.144:0/3942969699 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8284030450 con 0x5637393d90a0 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.574+0000 7f8282a00640 1 -- 172.21.15.144:0/3942969699 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284003830 con 0x5637393d90a0 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 -- 172.21.15.144:0/3942969699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 msgr2=0x5637397a7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 0x5637397a7ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8284000c00 tx=0x7f8284009490 comp rx=0 tx=0).stop 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f8294e00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f82780b3ca0 0x7f82780b61b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f829010aac0 tx=0x7f827c004730 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 -- 172.21.15.144:0/3942969699 shutdown_connections 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393e1280 0x5637397a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393d90a0 0x5637397a7ca0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 --2- 172.21.15.144:0/3942969699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397a7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 -- 172.21.15.144:0/3942969699 >> 172.21.15.144:0/3942969699 conn(0x5637393f6350 msgr2=0x5637397a5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:18.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 -- 172.21.15.144:0/3942969699 shutdown_connections 2024-07-26T11:01:18.578 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 -- 172.21.15.144:0/3942969699 wait complete. 2024-07-26T11:01:18.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.575+0000 7f829eebd740 1 -- start start 2024-07-26T11:01:18.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 Processor -- start 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5637395705f0 con 0x5637393d90a0 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5637396a9520 con 0x563739401f60 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637397b5ee0 con 0x5637393e1280 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.576+0000 7f829eebd740 10 client.0 _ll_register_callbacks cb 0x5637396b8ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8295800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36502/0 (socket says 172.21.15.144:36502) 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 -- 172.21.15.144:0/3187012520 learned_addr learned my addr 172.21.15.144:0/3187012520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:18.580 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8294e00640 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 -- 172.21.15.144:0/3187012520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 msgr2=0x5637397b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 -- 172.21.15.144:0/3187012520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 msgr2=0x5637397b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 -- 172.21.15.144:0/3187012520 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56373946d540 con 0x563739401f60 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.577+0000 7f8296200640 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82880a6a30 tx=0x7f82880ab180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74f415a435a32f43 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f8277400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f8277400640 1 -- 172.21.15.144:0/3187012520 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82880b3a60 con 0x563739401f60 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f8277400640 1 -- 172.21.15.144:0/3187012520 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f82880041d0 con 0x563739401f60 2024-07-26T11:01:18.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8284030450 con 0x563739401f60 2024-07-26T11:01:18.583 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f8277400640 1 -- 172.21.15.144:0/3187012520 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f82880ca050 con 0x563739401f60 2024-07-26T11:01:18.583 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f829eebd740 10 client.25214 fetch_fsmap learned FSMap version 25 2024-07-26T11:01:18.583 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5637397c1df0 con 0x563739401f60 2024-07-26T11:01:18.583 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.578+0000 7f8277400640 1 -- 172.21.15.144:0/3187012520 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8284030450 con 0x563739401f60 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f8277400640 1 -- 172.21.15.144:0/3187012520 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f82880abce0 con 0x563739401f60 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f829eebd740 10 client.25214 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f829eebd740 10 client.25214 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f829eebd740 20 client.25214 populate_metadata read hostname 'smithi144' 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f829eebd740 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5637393c08e0 0x5637397c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f829eebd740 4 client.25214 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56373971f8a0 con 0x5637393c08e0 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.579+0000 7f8295800640 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5637393c08e0 0x5637397c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.580+0000 7f8295800640 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5637393c08e0 0x5637397c5890 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a04822ab3e9a3807 server_cookie=a21bf35ca99f61b6 in_seq=0 out_seq=0 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.580+0000 7f8277400640 10 client.25214 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f8277400640 1 -- 172.21.15.144:0/3187012520 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56373971f8a0 con 0x5637393c08e0 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f8277400640 10 client.25214 handle_command_reply: tid=0 2024-07-26T11:01:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 client.25214 shutdown 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 msgr2=0x5637397bf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82880a6a30 tx=0x7f82880ab180 comp rx=0 tx=0).stop 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 shutdown_connections 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5637393c08e0 0x5637397c5890 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf230 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 --2- 172.21.15.144:0/3187012520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 >> 172.21.15.144:0/3187012520 conn(0x5637393f6350 msgr2=0x5637397aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f8277400640 0 client.25214 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 shutdown_connections 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.581+0000 7f829eebd740 1 -- 172.21.15.144:0/3187012520 wait complete. 2024-07-26T11:01:18.585 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.582+0000 7f829eebd740 20 client.25214 trim_cache size 0 max 16384 2024-07-26T11:01:18.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.601+0000 7f829eebd740 1 Processor -- start 2024-07-26T11:01:18.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.601+0000 7f829eebd740 1 -- start start 2024-07-26T11:01:18.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f829eebd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f829eebd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f829eebd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f829eebd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5637397c3bd0 con 0x5637393d90a0 2024-07-26T11:01:18.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f829eebd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5637397c42a0 con 0x563739401f60 2024-07-26T11:01:18.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f829eebd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637397c4970 con 0x5637393e1280 2024-07-26T11:01:18.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:32868/0 (socket says 172.21.15.144:32868) 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 -- 172.21.15.144:0/3317540070 learned_addr learned my addr 172.21.15.144:0/3317540070 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8294e00640 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8296200640 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 -- 172.21.15.144:0/3317540070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 msgr2=0x5637397b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 -- 172.21.15.144:0/3317540070 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 msgr2=0x5637397bf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.602+0000 7f8295800640 1 -- 172.21.15.144:0/3317540070 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637397ae7e0 con 0x5637393d90a0 2024-07-26T11:01:18.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.603+0000 7f8295800640 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397ad880 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8284009500 tx=0x7f8284030340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e24d57eefe70a77 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:18.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.603+0000 7f8282000640 1 -- 172.21.15.144:0/3317540070 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284037040 con 0x5637393d90a0 2024-07-26T11:01:18.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.603+0000 7f8282000640 1 -- 172.21.15.144:0/3317540070 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8284030650 con 0x5637393d90a0 2024-07-26T11:01:18.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.603+0000 7f8282000640 1 -- 172.21.15.144:0/3317540070 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828403dcf0 con 0x5637393d90a0 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 -- 172.21.15.144:0/3317540070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 msgr2=0x5637397ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397ad880 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8284009500 tx=0x7f8284030340 comp rx=0 tx=0).stop 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 -- 172.21.15.144:0/3317540070 shutdown_connections 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563739401f60 0x5637397bf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637393e1280 0x5637397b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 --2- 172.21.15.144:0/3317540070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397ad880 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 -- 172.21.15.144:0/3317540070 >> 172.21.15.144:0/3317540070 conn(0x5637393f6350 msgr2=0x5637397b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 -- 172.21.15.144:0/3317540070 shutdown_connections 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.604+0000 7f829eebd740 1 -- 172.21.15.144:0/3317540070 wait complete. 2024-07-26T11:01:18.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.605+0000 7f829eebd740 1 -- start start 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.605+0000 7f829eebd740 1 Processor -- start 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393e1280 0x5637397c7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5637397c5710 con 0x5637393d90a0 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5637397c5040 con 0x5637393e1280 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637397c5de0 con 0x563739401f60 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f829eebd740 10 client.0 _ll_register_callbacks cb 0x7f82880aa550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:32878/0 (socket says 172.21.15.144:32878) 2024-07-26T11:01:18.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 -- 172.21.15.144:0/377150838 learned_addr learned my addr 172.21.15.144:0/377150838 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8294e00640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393e1280 0x5637397c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8296200640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 -- 172.21.15.144:0/377150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 msgr2=0x5637397cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 -- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393e1280 msgr2=0x5637397c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393e1280 0x5637397c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.606+0000 7f8295800640 1 -- 172.21.15.144:0/377150838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637397ce130 con 0x5637393d90a0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f8295800640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397c6c90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f828402a930 tx=0x7f828402aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7ef26de8c1c23f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f8277400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f8277400640 1 -- 172.21.15.144:0/377150838 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284037040 con 0x5637393d90a0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f8277400640 1 -- 172.21.15.144:0/377150838 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f828403d480 con 0x5637393d90a0 2024-07-26T11:01:18.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8284030650 con 0x5637393d90a0 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f8277400640 1 -- 172.21.15.144:0/377150838 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8284067050 con 0x5637393d90a0 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.607+0000 7f829eebd740 10 client.15072 fetch_fsmap learned FSMap version 25 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56373982d520 con 0x5637393d90a0 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f8277400640 1 -- 172.21.15.144:0/377150838 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8284030650 con 0x5637393d90a0 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f8277400640 1 -- 172.21.15.144:0/377150838 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f82840042e0 con 0x5637393d90a0 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 10 client.15072 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:01:18.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 10 client.15072 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 20 client.15072 populate_metadata read hostname 'smithi144' 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563739395c90 0x5637398307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 4 client.15072 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.608+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "status"}) -- 0x56373971b340 con 0x563739395c90 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.609+0000 7f8294e00640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563739395c90 0x5637398307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.609+0000 7f8294e00640 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563739395c90 0x5637398307a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c8d3cc586ed0087 server_cookie=15af38336e68f148 in_seq=0 out_seq=0 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.609+0000 7f8277400640 10 client.15072 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f8277400640 1 -- 172.21.15.144:0/377150838 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56373971b340 con 0x563739395c90 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f8277400640 10 client.15072 handle_command_reply: tid=0 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 client.15072 shutdown 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 msgr2=0x5637397c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397c6c90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f828402a930 tx=0x7f828402aac0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 shutdown_connections 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563739395c90 0x5637398307a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563739401f60 0x5637397cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5637393e1280 0x5637397c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 --2- 172.21.15.144:0/377150838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5637393d90a0 0x5637397c6c90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f8277400640 0 client.15072 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 >> 172.21.15.144:0/377150838 conn(0x5637393f6350 msgr2=0x5637397a3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 shutdown_connections 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 1 -- 172.21.15.144:0/377150838 wait complete. 2024-07-26T11:01:18.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.610+0000 7f829eebd740 20 client.15072 trim_cache size 0 max 16384 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 0, 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24728, 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 233.47550216299999, 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 25, 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 90, 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:01:18.615 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 253.285313933 2024-07-26T11:01:18.616 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:01:18.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.611+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f82780b3ca0 msgr2=0x7f82780b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.611+0000 7f8296c00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f82780b3ca0 0x7f82780b61b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f829010aac0 tx=0x7f827c004730 comp rx=0 tx=0).stop 2024-07-26T11:01:18.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.611+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010b860 msgr2=0x7f82901a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:18.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.611+0000 7f8296c00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010b860 0x7f82901a1780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f828800efc0 tx=0x7f828800c800 comp rx=0 tx=0).stop 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 shutdown_connections 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f82780b3ca0 0x7f82780b61b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f829010b860 0x7f82901a1780 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829010af10 0x7f829019c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 --2- 172.21.15.144:0/4070546945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82901055d0 0x7f829019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 >> 172.21.15.144:0/4070546945 conn(0x7f8290101040 msgr2=0x7f8290103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 shutdown_connections 2024-07-26T11:01:18.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:18.612+0000 7f8296c00640 1 -- 172.21.15.144:0/4070546945 wait complete. 2024-07-26T11:01:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:18 smithi183 ceph-mon[37273]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:19.492 INFO:tasks.cephadm:Setup complete, yielding 2024-07-26T11:01:19.493 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T11:01:19.503 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T11:01:19.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch status' 2024-07-26T11:01:19.967 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:20.435 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:20 smithi144 ceph-mon[32129]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:20 smithi160 ceph-mon[36715]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:20 smithi183 ceph-mon[37273]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:22.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 -- 172.21.15.144:0/947441553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd84410b860 msgr2=0x7fd844112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:22.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 --2- 172.21.15.144:0/947441553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd84410b860 0x7fd844112090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd83c009ca0 tx=0x7fd83c02a8b0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 -- 172.21.15.144:0/947441553 shutdown_connections 2024-07-26T11:01:22.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 --2- 172.21.15.144:0/947441553 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd84410b860 0x7fd844112090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 --2- 172.21.15.144:0/947441553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84410af10 0x7fd84410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 --2- 172.21.15.144:0/947441553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8441055d0 0x7fd8441059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 -- 172.21.15.144:0/947441553 >> 172.21.15.144:0/947441553 conn(0x7fd844101040 msgr2=0x7fd844103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:22.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 -- 172.21.15.144:0/947441553 shutdown_connections 2024-07-26T11:01:22.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.321+0000 7fd84b000640 1 -- 172.21.15.144:0/947441553 wait complete. 2024-07-26T11:01:22.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.322+0000 7fd84b000640 1 Processor -- start 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.322+0000 7fd84b000640 1 -- start start 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84b000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8441055d0 0x7fd84419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84b000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84410af10 0x7fd84419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84b000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd84410b860 0x7fd8441a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84b000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd844115510 con 0x7fd84410b860 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84b000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd844115390 con 0x7fd8441055d0 2024-07-26T11:01:22.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84b000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd844115690 con 0x7fd84410af10 2024-07-26T11:01:22.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84a600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd84410b860 0x7fd8441a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:22.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84a600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd84410b860 0x7fd8441a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35018/0 (socket says 172.21.15.144:35018) 2024-07-26T11:01:22.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd84a600640 1 -- 172.21.15.144:0/1248649680 learned_addr learned my addr 172.21.15.144:0/1248649680 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:22.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd849c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8441055d0 0x7fd84419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:22.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.323+0000 7fd849200640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84410af10 0x7fd84419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd849c00640 1 -- 172.21.15.144:0/1248649680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84410af10 msgr2=0x7fd84419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd849c00640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84410af10 0x7fd84419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd849c00640 1 -- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd84410b860 msgr2=0x7fd8441a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd849c00640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd84410b860 0x7fd8441a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd849c00640 1 -- 172.21.15.144:0/1248649680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8441a1e90 con 0x7fd8441055d0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd849c00640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8441055d0 0x7fd84419bf20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd83800e9f0 tx=0x7fd83800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.324+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83800ccc0 con 0x7fd8441055d0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.325+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8441a2180 con 0x7fd8441055d0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.325+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8380108e0 con 0x7fd8441055d0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.325+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838018880 con 0x7fd8441055d0 2024-07-26T11:01:22.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.325+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8441a88e0 con 0x7fd8441055d0 2024-07-26T11:01:22.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.326+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8441059e0 con 0x7fd8441055d0 2024-07-26T11:01:22.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.328+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fd838010430 con 0x7fd8441055d0 2024-07-26T11:01:22.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.329+0000 7fd842000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd82c0b3ca0 0x7fd82c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:22.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.330+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd8380e43e0 con 0x7fd8441055d0 2024-07-26T11:01:22.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.330+0000 7fd849200640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd82c0b3ca0 0x7fd82c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:22.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.331+0000 7fd849200640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd82c0b3ca0 0x7fd82c0b61b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd8300095f0 tx=0x7fd830005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:22.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.334+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd8380aaef0 con 0x7fd8441055d0 2024-07-26T11:01:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:22 smithi144 ceph-mon[32129]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:22.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.550+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fd844063230 con 0x7fd82c0b3ca0 2024-07-26T11:01:22.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.553+0000 7fd842000640 1 -- 172.21.15.144:0/1248649680 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd844063230 con 0x7fd82c0b3ca0 2024-07-26T11:01:22.555 INFO:teuthology.orchestra.run.smithi144.stdout:Backend: cephadm 2024-07-26T11:01:22.555 INFO:teuthology.orchestra.run.smithi144.stdout:Available: Yes 2024-07-26T11:01:22.555 INFO:teuthology.orchestra.run.smithi144.stdout:Paused: No 2024-07-26T11:01:22.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd82c0b3ca0 msgr2=0x7fd82c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:22.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd82c0b3ca0 0x7fd82c0b61b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd8300095f0 tx=0x7fd830005f90 comp rx=0 tx=0).stop 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8441055d0 msgr2=0x7fd84419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8441055d0 0x7fd84419bf20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd83800e9f0 tx=0x7fd83800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 shutdown_connections 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd82c0b3ca0 0x7fd82c0b61b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd84410b860 0x7fd8441a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84410af10 0x7fd84419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 --2- 172.21.15.144:0/1248649680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8441055d0 0x7fd84419bf20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.556+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 >> 172.21.15.144:0/1248649680 conn(0x7fd844101040 msgr2=0x7fd84410a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.557+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 shutdown_connections 2024-07-26T11:01:22.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:22.557+0000 7fd84b000640 1 -- 172.21.15.144:0/1248649680 wait complete. 2024-07-26T11:01:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:22 smithi160 ceph-mon[36715]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:22 smithi183 ceph-mon[37273]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:23.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-26T11:01:23.971 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:24.418 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:24 smithi144 ceph-mon[32129]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:24.419 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:24 smithi144 ceph-mon[32129]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:24 smithi160 ceph-mon[36715]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:24 smithi160 ceph-mon[36715]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:24 smithi183 ceph-mon[37273]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:24 smithi183 ceph-mon[37273]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:25 smithi144 ceph-mon[32129]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:25 smithi160 ceph-mon[36715]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:25 smithi183 ceph-mon[37273]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:26.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 -- 172.21.15.144:0/590481946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810fc80 msgr2=0x7fb8a81164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:26.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/590481946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810fc80 0x7fb8a81164b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009ca0 tx=0x7fb8a002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 -- 172.21.15.144:0/590481946 shutdown_connections 2024-07-26T11:01:26.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/590481946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810fc80 0x7fb8a81164b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/590481946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a810f330 0x7fb8a810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/590481946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8a81099f0 0x7fb8a8109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 -- 172.21.15.144:0/590481946 >> 172.21.15.144:0/590481946 conn(0x7fb8a8078ff0 msgr2=0x7fb8a8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:26.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 -- 172.21.15.144:0/590481946 shutdown_connections 2024-07-26T11:01:26.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.150+0000 7fb8a7e00640 1 -- 172.21.15.144:0/590481946 wait complete. 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.151+0000 7fb8a7e00640 1 Processor -- start 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.151+0000 7fb8a7e00640 1 -- start start 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 0x7fb8a81a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810f330 0x7fb8a81a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8a810fc80 0x7fb8a81a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a8119940 con 0x7fb8a810fc80 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a81197c0 con 0x7fb8a810f330 2024-07-26T11:01:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a8119ac0 con 0x7fb8a81099f0 2024-07-26T11:01:26.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 0x7fb8a81a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:26.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8a810fc80 0x7fb8a81a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:26.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 0x7fb8a81a0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:44222/0 (socket says 172.21.15.144:44222) 2024-07-26T11:01:26.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb8a6a00640 1 -- 172.21.15.144:0/648700891 learned_addr learned my addr 172.21.15.144:0/648700891 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.152+0000 7fb89e000640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810f330 0x7fb8a81a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a6a00640 1 -- 172.21.15.144:0/648700891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810f330 msgr2=0x7fb8a81a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a6a00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810f330 0x7fb8a81a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a6a00640 1 -- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8a810fc80 msgr2=0x7fb8a81a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a6a00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8a810fc80 0x7fb8a81a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a6a00640 1 -- 172.21.15.144:0/648700891 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a81a6300 con 0x7fb8a81099f0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb89e000640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810f330 0x7fb8a81a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a6a00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 0x7fb8a81a0390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb89800e9f0 tx=0x7fb89800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89800cd80 con 0x7fb8a81099f0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8980108e0 con 0x7fb8a81099f0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898018980 con 0x7fb8a81099f0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a81a65f0 con 0x7fb8a81099f0 2024-07-26T11:01:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.153+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a81acd50 con 0x7fb8a81099f0 2024-07-26T11:01:26.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.155+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a8109e00 con 0x7fb8a81099f0 2024-07-26T11:01:26.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.157+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fb898010430 con 0x7fb8a81099f0 2024-07-26T11:01:26.160 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.159+0000 7fb89fe00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb88c0b3ca0 0x7fb88c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:26.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.159+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fb8980e48a0 con 0x7fb8a81099f0 2024-07-26T11:01:26.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.159+0000 7fb89e000640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb88c0b3ca0 0x7fb88c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:26.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.160+0000 7fb89e000640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb88c0b3ca0 0x7fb88c0b61b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb8a810fa50 tx=0x7fb890005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:26.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.161+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb8980ab3b0 con 0x7fb8a81099f0 2024-07-26T11:01:26.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.367+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fb8a8111ad0 con 0x7fb88c0b3ca0 2024-07-26T11:01:26.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.402+0000 7fb89fe00640 1 -- 172.21.15.144:0/648700891 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fb8a8111ad0 con 0x7fb88c0b3ca0 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.a smithi144 running (4m) 119s ago 4m 84.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c881e9e79670 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.b smithi160 running (4m) 2m ago 4m 96.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa decd0104d935 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.c smithi183 running (4m) 2m ago 4m 81.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 34383d6ad4a1 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.d smithi144 running (4m) 119s ago 4m 80.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 6f685a8b5390 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.e smithi160 running (4m) 2m ago 4m 84.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa dadfa41b89a3 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.f smithi183 running (4m) 2m ago 4m 84.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1f1bd016736e 2024-07-26T11:01:26.405 INFO:teuthology.orchestra.run.smithi144.stdout:mds.g smithi144 running (4m) 119s ago 4m 84.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 833e9cdd0f01 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mds.h smithi160 running (4m) 2m ago 4m 86.1M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 300270b2af95 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mds.i smithi183 running (4m) 2m ago 4m 84.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 73dd7ee98bd6 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mds.j smithi144 running (4m) 119s ago 4m 80.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa f17952baa7f0 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mds.k smithi160 running (4m) 2m ago 4m 82.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa b5a2932745cb 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mds.l smithi183 running (4m) 2m ago 4m 82.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 44e1580f9f8c 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mgr.x smithi144 *:9283,8765 running (13m) 119s ago 13m 586M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d5fcf7416ecb 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mgr.y smithi160 *:8443,8765 running (11m) 2m ago 11m 523M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa f82b659576a3 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mgr.z smithi183 *:8443,8765 running (11m) 2m ago 11m 525M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c45a95c952ac 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mon.a smithi144 running (14m) 119s ago 14m 117M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e91cf4998150 2024-07-26T11:01:26.406 INFO:teuthology.orchestra.run.smithi144.stdout:mon.b smithi160 running (12m) 2m ago 12m 114M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa cfcdf27a13dd 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:mon.c smithi183 running (12m) 2m ago 12m 114M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 0fbd7a966f9e 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.0 smithi144 running (10m) 119s ago 10m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 7ceff469238d 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.1 smithi160 running (9m) 2m ago 9m 122M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4b1b8823df89 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.2 smithi183 running (9m) 2m ago 9m 115M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa bc7a7f2bad79 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.3 smithi144 running (8m) 119s ago 8m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 48248ff0faf4 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.4 smithi160 running (8m) 2m ago 8m 119M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 7ec5bbae30c1 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.5 smithi183 running (8m) 2m ago 8m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa c4088519f829 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.6 smithi144 running (7m) 119s ago 7m 117M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa fa9efde73403 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.7 smithi160 running (7m) 2m ago 7m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 64aa28a47226 2024-07-26T11:01:26.407 INFO:teuthology.orchestra.run.smithi144.stdout:osd.8 smithi183 running (6m) 2m ago 6m 113M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1231598b5dc7 2024-07-26T11:01:26.408 INFO:teuthology.orchestra.run.smithi144.stdout:osd.9 smithi144 running (6m) 119s ago 6m 115M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 0f4b52e62352 2024-07-26T11:01:26.408 INFO:teuthology.orchestra.run.smithi144.stdout:osd.10 smithi160 running (5m) 2m ago 5m 119M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa ab79b706914a 2024-07-26T11:01:26.408 INFO:teuthology.orchestra.run.smithi144.stdout:osd.11 smithi183 running (5m) 2m ago 5m 119M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa baf315524039 2024-07-26T11:01:26.408 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb88c0b3ca0 msgr2=0x7fb88c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb88c0b3ca0 0x7fb88c0b61b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb8a810fa50 tx=0x7fb890005f90 comp rx=0 tx=0).stop 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 msgr2=0x7fb8a81a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 0x7fb8a81a0390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb89800e9f0 tx=0x7fb89800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 shutdown_connections 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fb88c0b3ca0 0x7fb88c0b61b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8a810fc80 0x7fb8a81a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a810f330 0x7fb8a81a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 --2- 172.21.15.144:0/648700891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8a81099f0 0x7fb8a81a0390 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.405+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 >> 172.21.15.144:0/648700891 conn(0x7fb8a8078ff0 msgr2=0x7fb8a8079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:26.409 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.406+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 shutdown_connections 2024-07-26T11:01:26.410 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:26.406+0000 7fb8a7e00640 1 -- 172.21.15.144:0/648700891 wait complete. 2024-07-26T11:01:27.417 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-26T11:01:27.871 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:28.267 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:28 smithi144 ceph-mon[32129]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:28.267 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:28 smithi144 ceph-mon[32129]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:28 smithi160 ceph-mon[36715]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:28 smithi160 ceph-mon[36715]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:28 smithi183 ceph-mon[37273]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:28 smithi183 ceph-mon[37273]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:30.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 -- 172.21.15.144:0/1199564270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20108720 msgr2=0x7fec2010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:30.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 --2- 172.21.15.144:0/1199564270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20108720 0x7fec2010ef50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fec18009ca0 tx=0x7fec1802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 -- 172.21.15.144:0/1199564270 shutdown_connections 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 --2- 172.21.15.144:0/1199564270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20108720 0x7fec2010ef50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 --2- 172.21.15.144:0/1199564270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20107dd0 0x7fec201081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 --2- 172.21.15.144:0/1199564270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20102490 0x7fec201028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 -- 172.21.15.144:0/1199564270 >> 172.21.15.144:0/1199564270 conn(0x7fec20079160 msgr2=0x7fec20079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 -- 172.21.15.144:0/1199564270 shutdown_connections 2024-07-26T11:01:30.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.079+0000 7fec26e00640 1 -- 172.21.15.144:0/1199564270 wait complete. 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.080+0000 7fec26e00640 1 Processor -- start 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 -- start start 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20102490 0x7fec2019c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 0x7fec2019c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20108720 0x7fec201a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fec201123d0 con 0x7fec20102490 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fec20112250 con 0x7fec20107dd0 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.081+0000 7fec26e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fec20112550 con 0x7fec20108720 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 0x7fec2019c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20102490 0x7fec2019c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:30.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 0x7fec2019c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:60624/0 (socket says 172.21.15.144:60624) 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 -- 172.21.15.144:0/3087571629 learned_addr learned my addr 172.21.15.144:0/3087571629 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec24e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20108720 0x7fec201a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 -- 172.21.15.144:0/3087571629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20108720 msgr2=0x7fec201a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20108720 0x7fec201a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 -- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20102490 msgr2=0x7fec2019c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20102490 0x7fec2019c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 -- 172.21.15.144:0/3087571629 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec201a1fe0 con 0x7fec20107dd0 2024-07-26T11:01:30.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec24e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20108720 0x7fec201a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.082+0000 7fec1f400640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 0x7fec2019c610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00e9c0 tx=0x7fec0c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.083+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00cd80 con 0x7fec20107dd0 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.083+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec0c0054d0 con 0x7fec20107dd0 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.083+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c010430 con 0x7fec20107dd0 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.083+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec201a22d0 con 0x7fec20107dd0 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.083+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec201a8a30 con 0x7fec20107dd0 2024-07-26T11:01:30.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.084+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec201028a0 con 0x7fec20107dd0 2024-07-26T11:01:30.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.087+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fec0c004fb0 con 0x7fec20107dd0 2024-07-26T11:01:30.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.089+0000 7fec1cc00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fec040b3c20 0x7fec040b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:30.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.089+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fec0c0e4830 con 0x7fec20107dd0 2024-07-26T11:01:30.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.089+0000 7fec1fe00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fec040b3c20 0x7fec040b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:30.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.089+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fec0c055600 con 0x7fec20107dd0 2024-07-26T11:01:30.092 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.090+0000 7fec1fe00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fec040b3c20 0x7fec040b6130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec100095f0 tx=0x7fec10005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:30.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.307+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fec2006a870 con 0x7fec040b3c20 2024-07-26T11:01:30.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.321+0000 7fec1cc00640 1 -- 172.21.15.144:0/3087571629 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fec2006a870 con 0x7fec040b3c20 2024-07-26T11:01:30.324 INFO:teuthology.orchestra.run.smithi144.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-26T11:01:30.324 INFO:teuthology.orchestra.run.smithi144.stdout:mds.all 12/12 2m ago 5m smithi144=a;smithi144=d;smithi144=g;smithi144=j;smithi160=b;smithi160=e;smithi160=h;smithi160=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-07-26T11:01:30.324 INFO:teuthology.orchestra.run.smithi144.stdout:mgr 3/3 2m ago 11m smithi144=x;smithi160=y;smithi183=z;count:3 2024-07-26T11:01:30.324 INFO:teuthology.orchestra.run.smithi144.stdout:mon 3/3 2m ago 12m smithi144:172.21.15.144=a;smithi160:172.21.15.160=b;smithi183:172.21.15.183=c;count:3 2024-07-26T11:01:30.324 INFO:teuthology.orchestra.run.smithi144.stdout:osd 12 2m ago - 2024-07-26T11:01:30.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fec040b3c20 msgr2=0x7fec040b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:30.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fec040b3c20 0x7fec040b6130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec100095f0 tx=0x7fec10005f90 comp rx=0 tx=0).stop 2024-07-26T11:01:30.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 msgr2=0x7fec2019c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:30.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 0x7fec2019c610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00e9c0 tx=0x7fec0c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:01:30.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 shutdown_connections 2024-07-26T11:01:30.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fec040b3c20 0x7fec040b6130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec20108720 0x7fec201a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fec20107dd0 0x7fec2019c610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 --2- 172.21.15.144:0/3087571629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec20102490 0x7fec2019c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:30.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 >> 172.21.15.144:0/3087571629 conn(0x7fec20079160 msgr2=0x7fec200ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:30.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 shutdown_connections 2024-07-26T11:01:30.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:30.325+0000 7fec26e00640 1 -- 172.21.15.144:0/3087571629 wait complete. 2024-07-26T11:01:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:30 smithi144 ceph-mon[32129]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:01:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:30 smithi160 ceph-mon[36715]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:30.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:01:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:30 smithi183 ceph-mon[37273]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:01:31.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-26T11:01:31.692 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:32 smithi144 ceph-mon[32129]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:32 smithi144 ceph-mon[32129]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:32 smithi160 ceph-mon[36715]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:32 smithi160 ceph-mon[36715]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:32 smithi183 ceph-mon[37273]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:32 smithi183 ceph-mon[37273]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:33 smithi144 ceph-mon[32129]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:33 smithi160 ceph-mon[36715]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:33 smithi183 ceph-mon[37273]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:34.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 -- 172.21.15.144:0/3895664275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 msgr2=0x7f26f0104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:34.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 --2- 172.21.15.144:0/3895664275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0104600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009860 tx=0x7f26e002a340 comp rx=0 tx=0).stop 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 -- 172.21.15.144:0/3895664275 shutdown_connections 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 --2- 172.21.15.144:0/3895664275 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 0x7f26f010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 --2- 172.21.15.144:0/3895664275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f010d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 --2- 172.21.15.144:0/3895664275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0104600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 -- 172.21.15.144:0/3895664275 >> 172.21.15.144:0/3895664275 conn(0x7f26f00fdcf0 msgr2=0x7f26f0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 -- 172.21.15.144:0/3895664275 shutdown_connections 2024-07-26T11:01:34.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.134+0000 7f26f5600640 1 -- 172.21.15.144:0/3895664275 wait complete. 2024-07-26T11:01:34.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.135+0000 7f26f5600640 1 Processor -- start 2024-07-26T11:01:34.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 -- start start 2024-07-26T11:01:34.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 0x7f26f019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01132e0 con 0x7f26f0104b40 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0113160 con 0x7f26f010d650 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.136+0000 7f26f5600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0113460 con 0x7f26f01041f0 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26ef400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26efe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26ef400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50570/0 (socket says 172.21.15.144:50570) 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26ef400640 1 -- 172.21.15.144:0/2879723386 learned_addr learned my addr 172.21.15.144:0/2879723386 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26f4c00640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 0x7f26f019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26efe00640 1 -- 172.21.15.144:0/2879723386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 msgr2=0x7f26f019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26efe00640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 0x7f26f019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26efe00640 1 -- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 msgr2=0x7f26f019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:34.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26efe00640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26efe00640 1 -- 172.21.15.144:0/2879723386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f019fcc0 con 0x7f26f01041f0 2024-07-26T11:01:34.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26f4c00640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 0x7f26f019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:34.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.137+0000 7f26ef400640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:34.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.138+0000 7f26efe00640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0199d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f26e0035040 tx=0x7f26e0032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:34.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.138+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0040020 con 0x7f26f01041f0 2024-07-26T11:01:34.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.138+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26e0030e10 con 0x7f26f01041f0 2024-07-26T11:01:34.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.138+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0037040 con 0x7f26f01041f0 2024-07-26T11:01:34.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.138+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f019ff50 con 0x7f26f01041f0 2024-07-26T11:01:34.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.138+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f01a6710 con 0x7f26f01041f0 2024-07-26T11:01:34.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.139+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f006af90 con 0x7f26f01041f0 2024-07-26T11:01:34.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.142+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f26e0043020 con 0x7f26f01041f0 2024-07-26T11:01:34.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.143+0000 7f26ecc00640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3ca0 0x7f26d40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:34.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.144+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f26e01043d0 con 0x7f26f01041f0 2024-07-26T11:01:34.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.144+0000 7f26ef400640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3ca0 0x7f26d40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:34.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.145+0000 7f26ef400640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3ca0 0x7f26d40b61b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f26f0105cd0 tx=0x7f26dc009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:34.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.145+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f26e004a020 con 0x7f26f01041f0 2024-07-26T11:01:34.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.357+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f26f0106770 con 0x7f26d40b3ca0 2024-07-26T11:01:34.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.359+0000 7f26ecc00640 1 -- 172.21.15.144:0/2879723386 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f26f0106770 con 0x7f26d40b3ca0 2024-07-26T11:01:34.362 INFO:teuthology.orchestra.run.smithi144.stdout:HOST ADDR LABELS STATUS 2024-07-26T11:01:34.363 INFO:teuthology.orchestra.run.smithi144.stdout:smithi144 172.21.15.144 2024-07-26T11:01:34.363 INFO:teuthology.orchestra.run.smithi144.stdout:smithi160 172.21.15.160 2024-07-26T11:01:34.363 INFO:teuthology.orchestra.run.smithi144.stdout:smithi183 172.21.15.183 2024-07-26T11:01:34.363 INFO:teuthology.orchestra.run.smithi144.stdout:3 hosts in cluster 2024-07-26T11:01:34.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3ca0 msgr2=0x7f26d40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:34.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3ca0 0x7f26d40b61b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f26f0105cd0 tx=0x7f26dc009260 comp rx=0 tx=0).stop 2024-07-26T11:01:34.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 msgr2=0x7f26f0199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:34.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0199d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f26e0035040 tx=0x7f26e0032d50 comp rx=0 tx=0).stop 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 shutdown_connections 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f26d40b3ca0 0x7f26d40b61b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f26f010d650 0x7f26f019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26f0104b40 0x7f26f019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 --2- 172.21.15.144:0/2879723386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26f01041f0 0x7f26f0199d50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 >> 172.21.15.144:0/2879723386 conn(0x7f26f00fdcf0 msgr2=0x7f26f0106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 shutdown_connections 2024-07-26T11:01:34.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:34.363+0000 7f26f5600640 1 -- 172.21.15.144:0/2879723386 wait complete. 2024-07-26T11:01:35.323 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-26T11:01:35.787 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:36 smithi144 ceph-mon[32129]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:36 smithi144 ceph-mon[32129]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:36 smithi160 ceph-mon[36715]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:36 smithi160 ceph-mon[36715]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:36 smithi183 ceph-mon[37273]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:36 smithi183 ceph-mon[37273]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:38.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 -- 172.21.15.144:0/639750630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718108ee0 msgr2=0x7f97181092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 --2- 172.21.15.144:0/639750630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718108ee0 0x7f97181092f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9704009900 tx=0x7f970402a410 comp rx=0 tx=0).stop 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 -- 172.21.15.144:0/639750630 shutdown_connections 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 --2- 172.21.15.144:0/639750630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9718109830 0x7f9718110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 --2- 172.21.15.144:0/639750630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718108ee0 0x7f97181092f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 --2- 172.21.15.144:0/639750630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f97181035a0 0x7f97181039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 -- 172.21.15.144:0/639750630 >> 172.21.15.144:0/639750630 conn(0x7f97180ff010 msgr2=0x7f9718101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 -- 172.21.15.144:0/639750630 shutdown_connections 2024-07-26T11:01:38.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.146+0000 7f971e000640 1 -- 172.21.15.144:0/639750630 wait complete. 2024-07-26T11:01:38.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.148+0000 7f971e000640 1 Processor -- start 2024-07-26T11:01:38.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.148+0000 7f971e000640 1 -- start start 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.148+0000 7f971e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 0x7f9718075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.148+0000 7f971e000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9718108ee0 0x7f9718075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f971e000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718109830 0x7f9718075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f971e000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9718113510 con 0x7f9718109830 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f971e000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9718113390 con 0x7f9718108ee0 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f971e000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9718113690 con 0x7f97181035a0 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 0x7f9718075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 0x7f9718075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:36750/0 (socket says 172.21.15.144:36750) 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 -- 172.21.15.144:0/1371833011 learned_addr learned my addr 172.21.15.144:0/1371833011 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717e00640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718109830 0x7f9718075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:38.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9716a00640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9718108ee0 0x7f9718075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:38.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 -- 172.21.15.144:0/1371833011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9718108ee0 msgr2=0x7f9718075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:38.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9718108ee0 0x7f9718075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 -- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718109830 msgr2=0x7f9718075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:38.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718109830 0x7f9718075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.149+0000 7f9717400640 1 -- 172.21.15.144:0/1371833011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97181a81c0 con 0x7f97181035a0 2024-07-26T11:01:38.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.150+0000 7f9717400640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 0x7f9718075140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f970800e9f0 tx=0x7f970800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.153+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970800cd80 con 0x7f97181035a0 2024-07-26T11:01:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.153+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97080108e0 con 0x7f97181035a0 2024-07-26T11:01:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.153+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708018980 con 0x7f97181035a0 2024-07-26T11:01:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.153+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97181a84b0 con 0x7f97181035a0 2024-07-26T11:01:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.154+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97181a8970 con 0x7f97181035a0 2024-07-26T11:01:38.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.155+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97181039b0 con 0x7f97181035a0 2024-07-26T11:01:38.160 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.157+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f9708010430 con 0x7f97181035a0 2024-07-26T11:01:38.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.159+0000 7f970fe00640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96fc0b3ca0 0x7f96fc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:38.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.159+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f97080e4620 con 0x7f97181035a0 2024-07-26T11:01:38.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.159+0000 7f9716a00640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96fc0b3ca0 0x7f96fc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:38.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.160+0000 7f9716a00640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96fc0b3ca0 0x7f96fc0b61b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f970402aa00 tx=0x7f9704005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:38.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.161+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f97080ab130 con 0x7f97181035a0 2024-07-26T11:01:38.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.349+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f9718063230 con 0x7f96fc0b3ca0 2024-07-26T11:01:38.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.353+0000 7f970fe00640 1 -- 172.21.15.144:0/1371833011 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f9718063230 con 0x7f96fc0b3ca0 2024-07-26T11:01:38.356 INFO:teuthology.orchestra.run.smithi144.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-26T11:01:38.356 INFO:teuthology.orchestra.run.smithi144.stdout:smithi144 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050085400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T11:01:38.356 INFO:teuthology.orchestra.run.smithi144.stdout:smithi160 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JY400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T11:01:38.357 INFO:teuthology.orchestra.run.smithi144.stdout:smithi183 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96fc0b3ca0 msgr2=0x7f96fc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96fc0b3ca0 0x7f96fc0b61b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f970402aa00 tx=0x7f9704005a20 comp rx=0 tx=0).stop 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 msgr2=0x7f9718075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 0x7f9718075140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f970800e9f0 tx=0x7f970800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 shutdown_connections 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96fc0b3ca0 0x7f96fc0b61b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9718109830 0x7f9718075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9718108ee0 0x7f9718075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 --2- 172.21.15.144:0/1371833011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97181035a0 0x7f9718075140 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:38.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 >> 172.21.15.144:0/1371833011 conn(0x7f97180ff010 msgr2=0x7f9718101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:38.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 shutdown_connections 2024-07-26T11:01:38.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:38.356+0000 7f971e000640 1 -- 172.21.15.144:0/1371833011 wait complete. 2024-07-26T11:01:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:38 smithi144 ceph-mon[32129]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:38 smithi160 ceph-mon[36715]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:38 smithi183 ceph-mon[37273]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:39.286 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T11:01:39.295 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T11:01:39.296 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-26T11:01:39.766 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:40 smithi144 ceph-mon[32129]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:40 smithi144 ceph-mon[32129]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:40 smithi160 ceph-mon[36715]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:40 smithi160 ceph-mon[36715]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:40 smithi183 ceph-mon[37273]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:40 smithi183 ceph-mon[37273]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 -- 172.21.15.144:0/3661423356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149010d650 msgr2=0x7f149010fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 --2- 172.21.15.144:0/3661423356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149010d650 0x7f149010fb40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1488009ca0 tx=0x7f148802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 -- 172.21.15.144:0/3661423356 shutdown_connections 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 --2- 172.21.15.144:0/3661423356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149010d650 0x7f149010fb40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 --2- 172.21.15.144:0/3661423356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490104b40 0x7f149010d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 --2- 172.21.15.144:0/3661423356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 -- 172.21.15.144:0/3661423356 >> 172.21.15.144:0/3661423356 conn(0x7f14900fdcf0 msgr2=0x7f1490100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 -- 172.21.15.144:0/3661423356 shutdown_connections 2024-07-26T11:01:42.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.021+0000 7f1495600640 1 -- 172.21.15.144:0/3661423356 wait complete. 2024-07-26T11:01:42.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 Processor -- start 2024-07-26T11:01:42.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 -- start start 2024-07-26T11:01:42.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:42.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1490104b40 0x7f149019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:42.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149010d650 0x7f149019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:42.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1490113310 con 0x7f14901041f0 2024-07-26T11:01:42.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1490113190 con 0x7f1490104b40 2024-07-26T11:01:42.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.023+0000 7f1495600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1490113490 con 0x7f149010d650 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53228/0 (socket says 172.21.15.144:53228) 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f1494c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149010d650 0x7f149019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 -- 172.21.15.144:0/1662606904 learned_addr learned my addr 172.21.15.144:0/1662606904 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f1487e00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1490104b40 0x7f149019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 -- 172.21.15.144:0/1662606904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149010d650 msgr2=0x7f149019f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149010d650 0x7f149019f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 -- 172.21.15.144:0/1662606904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1490104b40 msgr2=0x7f149019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1490104b40 0x7f149019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f148fe00640 1 -- 172.21.15.144:0/1662606904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149019fc10 con 0x7f14901041f0 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.024+0000 7f1494c00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149010d650 0x7f149019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:42.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.025+0000 7f148fe00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490199ca0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f148000ea60 tx=0x7f148000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:42.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.025+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148000cde0 con 0x7f14901041f0 2024-07-26T11:01:42.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.025+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149019ff00 con 0x7f14901041f0 2024-07-26T11:01:42.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.025+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14800035f0 con 0x7f14901041f0 2024-07-26T11:01:42.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.025+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480005220 con 0x7f14901041f0 2024-07-26T11:01:42.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.025+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14901a6660 con 0x7f14901041f0 2024-07-26T11:01:42.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.026+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1490104600 con 0x7f14901041f0 2024-07-26T11:01:42.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.029+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f14800037a0 con 0x7f14901041f0 2024-07-26T11:01:42.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.031+0000 7f148d600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14740b3ca0 0x7f14740b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:42.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.031+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f14800e4810 con 0x7f14901041f0 2024-07-26T11:01:42.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.031+0000 7f1487e00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14740b3ca0 0x7f14740b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:42.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.032+0000 7f1487e00640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14740b3ca0 0x7f14740b61b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f14780095f0 tx=0x7f1478005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:42.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.032+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14800ab970 con 0x7f14901041f0 2024-07-26T11:01:42.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.273+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f14901a6a30 con 0x7f14901041f0 2024-07-26T11:01:42.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.274+0000 7f148d600640 1 -- 172.21.15.144:0/1662606904 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3102 (secure 0 0 0) 0x7f14800ab270 con 0x7f14901041f0 2024-07-26T11:01:42.276 INFO:teuthology.orchestra.run.smithi144.stdout:e25 2024-07-26T11:01:42.276 INFO:teuthology.orchestra.run.smithi144.stdout:btime 2024-07-26T10:58:47:222852+0000 2024-07-26T11:01:42.276 INFO:teuthology.orchestra.run.smithi144.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T11:01:42.276 INFO:teuthology.orchestra.run.smithi144.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:legacy client fscid: 1 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:Filesystem 'cephfs' (1) 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:fs_name cephfs 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:epoch 25 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:created 2024-07-26T10:57:25.002914+0000 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:modified 2024-07-26T10:58:46.244128+0000 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:tableserver 0 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:root 0 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:session_timeout 300 2024-07-26T11:01:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:session_autoclose 300 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:max_file_size 1099511627776 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:max_xattr_size 65536 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:required_client_features {} 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:last_failure 0 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:last_failure_osd_epoch 0 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:max_mds 3 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:in 0,1,2 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:up {0=24728,1=24715,2=24703} 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:failed 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:damaged 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:stopped 2024-07-26T11:01:42.278 INFO:teuthology.orchestra.run.smithi144.stdout:data_pools [3,4] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:metadata_pool 2 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:inline_data disabled 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:balancer 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:bal_rank_mask -1 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:standby_count_wanted 0 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:qdb_cluster leader: 24728 members: 24728,24715,24703 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.b{0:24728} state up:active seq 6 addr [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.h{0:24701} state up:standby-replay seq 1 addr [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.a{1:24715} state up:active seq 25 addr [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.i{1:24691} state up:standby-replay seq 1 addr [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.g{2:24703} state up:active seq 27 addr [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.e{2:24677} state up:standby-replay seq 1 addr [v2:172.21.15.160:6834/405635522,v1:172.21.15.160:6835/405635522] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:42.279 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:Standby daemons: 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.l{-1:24622} state up:standby seq 1 addr [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.k{-1:24647} state up:standby seq 1 addr [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.j{-1:24653} state up:standby seq 1 addr [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.f{-1:24667} state up:standby seq 1 addr [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.c{-1:24673} state up:standby seq 1 addr [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.276+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14740b3ca0 msgr2=0x7f14740b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.276+0000 7f1495600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14740b3ca0 0x7f14740b61b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f14780095f0 tx=0x7f1478005f90 comp rx=0 tx=0).stop 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 msgr2=0x7f1490199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490199ca0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f148000ea60 tx=0x7f148000ef20 comp rx=0 tx=0).stop 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 shutdown_connections 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14740b3ca0 0x7f14740b61b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149010d650 0x7f149019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1490104b40 0x7f149019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 --2- 172.21.15.144:0/1662606904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14901041f0 0x7f1490199ca0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:42.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 >> 172.21.15.144:0/1662606904 conn(0x7f14900fdcf0 msgr2=0x7f14901070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:42.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 shutdown_connections 2024-07-26T11:01:42.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:42.277+0000 7f1495600640 1 -- 172.21.15.144:0/1662606904 wait complete. 2024-07-26T11:01:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:42 smithi144 ceph-mon[32129]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:42 smithi160 ceph-mon[36715]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:42 smithi183 ceph-mon[37273]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:43.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-26T11:01:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:43 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1662606904' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T11:01:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:43 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1662606904' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T11:01:43.632 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:43 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1662606904' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T11:01:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:44 smithi144 ceph-mon[32129]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:44.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:44 smithi183 ceph-mon[37273]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:44 smithi160 ceph-mon[36715]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:45.313 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:45 smithi144 ceph-mon[32129]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:45 smithi160 ceph-mon[36715]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:45 smithi183 ceph-mon[37273]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 -- 172.21.15.144:0/450547633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 msgr2=0x7f9658109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 --2- 172.21.15.144:0/450547633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f9658109e00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9640009900 tx=0x7f964002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 -- 172.21.15.144:0/450547633 shutdown_connections 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 --2- 172.21.15.144:0/450547633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 0x7f96581164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 --2- 172.21.15.144:0/450547633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f965810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 --2- 172.21.15.144:0/450547633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f9658109e00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 -- 172.21.15.144:0/450547633 >> 172.21.15.144:0/450547633 conn(0x7f9658078ff0 msgr2=0x7f9658107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:45.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 -- 172.21.15.144:0/450547633 shutdown_connections 2024-07-26T11:01:45.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.892+0000 7f965d000640 1 -- 172.21.15.144:0/450547633 wait complete. 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.893+0000 7f965d000640 1 Processor -- start 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 -- start start 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f96581a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f96581a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 0x7f96581a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9658119940 con 0x7f965810fc80 2024-07-26T11:01:45.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f96581197c0 con 0x7f965810f330 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f965d000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9658119ac0 con 0x7f96581099f0 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f9656a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f96581a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.894+0000 7f9657400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f96581a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f96581a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54416/0 (socket says 172.21.15.144:54416) 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 -- 172.21.15.144:0/2345407996 learned_addr learned my addr 172.21.15.144:0/2345407996 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9657e00640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 0x7f96581a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 -- 172.21.15.144:0/2345407996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 msgr2=0x7f96581a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f96581a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 -- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 msgr2=0x7f96581a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 0x7f96581a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9656a00640 1 -- 172.21.15.144:0/2345407996 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96581a6280 con 0x7f965810f330 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9657400640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f96581a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:45.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.895+0000 7f9657e00640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 0x7f96581a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:45.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.896+0000 7f9656a00640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f96581a08b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f964800e9c0 tx=0x7f964800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:45.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.896+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964800ccc0 con 0x7f965810f330 2024-07-26T11:01:45.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.896+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96581a6570 con 0x7f965810f330 2024-07-26T11:01:45.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.896+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96480054d0 con 0x7f965810f330 2024-07-26T11:01:45.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.896+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648010430 con 0x7f965810f330 2024-07-26T11:01:45.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.896+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96581accd0 con 0x7f965810f330 2024-07-26T11:01:45.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.897+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9658109e00 con 0x7f965810f330 2024-07-26T11:01:45.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.900+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f9648005000 con 0x7f965810f330 2024-07-26T11:01:45.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.901+0000 7f964fe00640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f963c0b3c20 0x7f963c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:45.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.902+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f96480e4660 con 0x7f965810f330 2024-07-26T11:01:45.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.902+0000 7f9657400640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f963c0b3c20 0x7f963c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:45.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.903+0000 7f9657400640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f963c0b3c20 0x7f963c0b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9640031440 tx=0x7f9640032ed0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:45.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:45.903+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9648055600 con 0x7f965810f330 2024-07-26T11:01:46.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.104+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f96581a1440 con 0x7f965810f330 2024-07-26T11:01:46.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.106+0000 7f964fe00640 1 -- 172.21.15.144:0/2345407996 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+6101 (secure 0 0 0) 0x7f96581a1440 con 0x7f965810f330 2024-07-26T11:01:46.109 INFO:teuthology.orchestra.run.smithi144.stdout:epoch 90 2024-07-26T11:01:46.109 INFO:teuthology.orchestra.run.smithi144.stdout:fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 2024-07-26T11:01:46.109 INFO:teuthology.orchestra.run.smithi144.stdout:created 2024-07-26T10:47:18.301633+0000 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:modified 2024-07-26T10:58:45.199165+0000 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:crush_version 27 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:full_ratio 0.95 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:backfillfull_ratio 0.9 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:nearfull_ratio 0.85 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:require_min_compat_client luminous 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:min_compat_client luminous 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:require_osd_release squid 2024-07-26T11:01:46.110 INFO:teuthology.orchestra.run.smithi144.stdout:stretch_mode_enabled false 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:max_osd 12 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:osd.0 up in weight 1 up_from 9 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6802/3520329112,v1:172.21.15.144:6803/3520329112] [v2:172.21.15.144:6804/3520329112,v1:172.21.15.144:6805/3520329112] exists,up a20cb98e-b483-474f-aafe-5145286b6145 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:osd.1 up in weight 1 up_from 14 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6800/3146107361,v1:172.21.15.160:6801/3146107361] [v2:172.21.15.160:6802/3146107361,v1:172.21.15.160:6803/3146107361] exists,up 938ca752-2db7-47b3-aaad-b31ed684607e 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:osd.2 up in weight 1 up_from 19 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/2239917243,v1:172.21.15.183:6801/2239917243] [v2:172.21.15.183:6802/2239917243,v1:172.21.15.183:6803/2239917243] exists,up 91ba96d0-9d5c-4128-98cf-517ac651af19 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:osd.3 up in weight 1 up_from 26 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6810/1981116547,v1:172.21.15.144:6811/1981116547] [v2:172.21.15.144:6812/1981116547,v1:172.21.15.144:6813/1981116547] exists,up e932bb8c-4b48-4cf1-8f02-4c151c98a009 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:osd.4 up in weight 1 up_from 32 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6808/2452808824,v1:172.21.15.160:6809/2452808824] [v2:172.21.15.160:6810/2452808824,v1:172.21.15.160:6811/2452808824] exists,up f6f12400-2f76-4c60-88e2-d06235484603 2024-07-26T11:01:46.114 INFO:teuthology.orchestra.run.smithi144.stdout:osd.5 up in weight 1 up_from 37 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6808/1593211139,v1:172.21.15.183:6809/1593211139] [v2:172.21.15.183:6810/1593211139,v1:172.21.15.183:6811/1593211139] exists,up 5b45f883-af04-42bd-a38e-38e88cc49515 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:osd.6 up in weight 1 up_from 43 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6818/1312925195,v1:172.21.15.144:6819/1312925195] [v2:172.21.15.144:6820/1312925195,v1:172.21.15.144:6821/1312925195] exists,up aedef072-4034-47cb-bab6-11f1fd44d82a 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:osd.7 up in weight 1 up_from 48 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6816/602437452,v1:172.21.15.160:6817/602437452] [v2:172.21.15.160:6818/602437452,v1:172.21.15.160:6819/602437452] exists,up 23dda395-05ca-4eef-8655-74edaefbfe90 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:osd.8 up in weight 1 up_from 54 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6816/2430686973,v1:172.21.15.183:6817/2430686973] [v2:172.21.15.183:6818/2430686973,v1:172.21.15.183:6819/2430686973] exists,up b0afc0c4-8c60-46bf-a684-f418e6d1ec5a 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:osd.9 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6826/2364379216,v1:172.21.15.144:6827/2364379216] [v2:172.21.15.144:6828/2364379216,v1:172.21.15.144:6829/2364379216] exists,up 8f586925-f5c7-495c-a472-0d3e6af7dfdf 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:osd.10 up in weight 1 up_from 65 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6824/949715799,v1:172.21.15.160:6825/949715799] [v2:172.21.15.160:6826/949715799,v1:172.21.15.160:6827/949715799] exists,up fac8da53-9c38-4368-9278-612c04a06498 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:osd.11 up in weight 1 up_from 71 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6824/2003058932,v1:172.21.15.183:6825/2003058932] [v2:172.21.15.183:6826/2003058932,v1:172.21.15.183:6827/2003058932] exists,up 7f00c381-b270-461b-a3be-ae5822a8affb 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 2.4 [2,11] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 2.1b [2,1] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 2.1c [8,11] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 2.1f [8,4] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 3.11 [4,8] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 3.12 [3,9] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 3.13 [4,9] 2024-07-26T11:01:46.115 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 3.1b [5,9] 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 3.31 [4,6] 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 4.5 [9,10] 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 4.6 [5,8] 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 4.c [9,10] 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:pg_upmap_items 4.f [9,8] 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:6801/1820197496 expires 2024-07-27T10:49:42.802348+0000 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/288744986 expires 2024-07-27T10:47:54.806379+0000 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/563407546 expires 2024-07-27T10:49:42.802348+0000 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:6800/1828713044 expires 2024-07-27T10:47:54.806379+0000 2024-07-26T11:01:46.116 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:6801/1828713044 expires 2024-07-27T10:47:54.806379+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/2087175124 expires 2024-07-27T10:47:54.806379+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/3434609332 expires 2024-07-27T10:48:24.009859+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/2275934374 expires 2024-07-27T10:47:54.806379+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/2481813561 expires 2024-07-27T10:49:42.802348+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:6800/2854801065 expires 2024-07-27T10:48:24.009859+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/1824026059 expires 2024-07-27T10:48:24.009859+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:6801/2854801065 expires 2024-07-27T10:48:24.009859+0000 2024-07-26T11:01:46.117 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:6800/1820197496 expires 2024-07-27T10:49:42.802348+0000 2024-07-26T11:01:46.118 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/3586311140 expires 2024-07-27T10:48:24.009859+0000 2024-07-26T11:01:46.118 INFO:teuthology.orchestra.run.smithi144.stdout:blocklist 172.21.15.144:0/3284599906 expires 2024-07-27T10:49:42.802348+0000 2024-07-26T11:01:46.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.108+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f963c0b3c20 msgr2=0x7f963c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:46.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.108+0000 7f965d000640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f963c0b3c20 0x7f963c0b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9640031440 tx=0x7f9640032ed0 comp rx=0 tx=0).stop 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.108+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 msgr2=0x7f96581a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.108+0000 7f965d000640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f96581a08b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f964800e9c0 tx=0x7f964800ee80 comp rx=0 tx=0).stop 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 shutdown_connections 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f963c0b3c20 0x7f963c0b6130 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f965810fc80 0x7f96581a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f965810f330 0x7f96581a08b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 --2- 172.21.15.144:0/2345407996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96581099f0 0x7f96581a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 >> 172.21.15.144:0/2345407996 conn(0x7f9658078ff0 msgr2=0x7f96580796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 shutdown_connections 2024-07-26T11:01:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:46.109+0000 7f965d000640 1 -- 172.21.15.144:0/2345407996 wait complete. 2024-07-26T11:01:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:46 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2345407996' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T11:01:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:46 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2345407996' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T11:01:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:46 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2345407996' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T11:01:47.007 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-26T11:01:47.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:01:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:47 smithi144 ceph-mon[32129]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:47.489 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:47 smithi160 ceph-mon[36715]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:47 smithi183 ceph-mon[37273]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:49.782 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 -- 172.21.15.144:0/4144019631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 msgr2=0x7f54280755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 --2- 172.21.15.144:0/4144019631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 0x7f54280755f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5418009900 tx=0x7f541802a400 comp rx=0 tx=0).stop 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 -- 172.21.15.144:0/4144019631 shutdown_connections 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 --2- 172.21.15.144:0/4144019631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f5428111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 --2- 172.21.15.144:0/4144019631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 --2- 172.21.15.144:0/4144019631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 0x7f54280755f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 -- 172.21.15.144:0/4144019631 >> 172.21.15.144:0/4144019631 conn(0x7f54280fee70 msgr2=0x7f54281012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 -- 172.21.15.144:0/4144019631 shutdown_connections 2024-07-26T11:01:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.780+0000 7f542e200640 1 -- 172.21.15.144:0/4144019631 wait complete. 2024-07-26T11:01:49.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 Processor -- start 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 -- start start 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 0x7f542819bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f54281a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5428115520 con 0x7f542810f940 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f54281153a0 con 0x7f5428075b30 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.782+0000 7f542e200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f54281156a0 con 0x7f54280751e0 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f542d800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f54281a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f542ce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 0x7f542819bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:49.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542819c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:54426/0 (socket says 172.21.15.144:54426) 2024-07-26T11:01:49.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f542d800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f54281a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53264/0 (socket says 172.21.15.144:53264) 2024-07-26T11:01:49.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 -- 172.21.15.144:0/2850846809 learned_addr learned my addr 172.21.15.144:0/2850846809 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 -- 172.21.15.144:0/2850846809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 msgr2=0x7f542819bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 0x7f542819bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 -- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 msgr2=0x7f54281a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f54281a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 -- 172.21.15.144:0/2850846809 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54281a1e60 con 0x7f5428075b30 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f542d800640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f54281a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.783+0000 7f5427e00640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542819c490 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f541000e9c0 tx=0x7f541000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.784+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541000cd80 con 0x7f5428075b30 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.784+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54100035f0 con 0x7f5428075b30 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.784+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54281a2150 con 0x7f5428075b30 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.784+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410010690 con 0x7f5428075b30 2024-07-26T11:01:49.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.784+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54281a88b0 con 0x7f5428075b30 2024-07-26T11:01:49.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.785+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54280755f0 con 0x7f5428075b30 2024-07-26T11:01:49.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.788+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f54100037a0 con 0x7f5428075b30 2024-07-26T11:01:49.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.789+0000 7f5425600640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f540c0b3c20 0x7f540c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:49.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.789+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f54100e45a0 con 0x7f5428075b30 2024-07-26T11:01:49.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.789+0000 7f542ce00640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f540c0b3c20 0x7f540c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:49.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.790+0000 7f542ce00640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f540c0b3c20 0x7f540c0b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5418005ab0 tx=0x7f5418005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:49.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:49.791+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f54100ab700 con 0x7f5428075b30 2024-07-26T11:01:50.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.030+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f542819d020 con 0x7f5428075b30 2024-07-26T11:01:50.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.032+0000 7f5425600640 1 -- 172.21.15.144:0/2850846809 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11931 (secure 0 0 0) 0x7f54100ab000 con 0x7f5428075b30 2024-07-26T11:01:50.034 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:50.035 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":25,"btime":"2024-07-26T10:58:47:222852+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:58:46.244128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T11:01:50.035 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T11:01:50.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f540c0b3c20 msgr2=0x7f540c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:50.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f540c0b3c20 0x7f540c0b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5418005ab0 tx=0x7f5418005a20 comp rx=0 tx=0).stop 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 msgr2=0x7f542819c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542819c490 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f541000e9c0 tx=0x7f541000ee80 comp rx=0 tx=0).stop 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 shutdown_connections 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f540c0b3c20 0x7f540c0b6130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542810f940 0x7f54281a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5428075b30 0x7f542819c490 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 --2- 172.21.15.144:0/2850846809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54280751e0 0x7f542819bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.035+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 >> 172.21.15.144:0/2850846809 conn(0x7f54280fee70 msgr2=0x7f54281094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.036+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 shutdown_connections 2024-07-26T11:01:50.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:50.036+0000 7f542e200640 1 -- 172.21.15.144:0/2850846809 wait complete. 2024-07-26T11:01:50.407 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:50 smithi144 ceph-mon[32129]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:50.407 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:50 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2850846809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:50 smithi160 ceph-mon[36715]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:50 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2850846809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:50 smithi183 ceph-mon[37273]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:50 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2850846809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:50.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:01:51.277 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:52.440 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:52 smithi144 ceph-mon[32129]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:52 smithi160 ceph-mon[36715]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:52 smithi183 ceph-mon[37273]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 -- 172.21.15.144:0/1082148793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0076db0 msgr2=0x7fa8f00751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1082148793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0076db0 0x7fa8f00751e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0009ca0 tx=0x7fa8e002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 -- 172.21.15.144:0/1082148793 shutdown_connections 2024-07-26T11:01:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1082148793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0113cb0 0x7fa8f0116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1082148793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0075800 0x7fa8f0075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1082148793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0076db0 0x7fa8f00751e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.658+0000 7fa8f6800640 1 -- 172.21.15.144:0/1082148793 >> 172.21.15.144:0/1082148793 conn(0x7fa8f00fee70 msgr2=0x7fa8f01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.659+0000 7fa8f6800640 1 -- 172.21.15.144:0/1082148793 shutdown_connections 2024-07-26T11:01:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.659+0000 7fa8f6800640 1 -- 172.21.15.144:0/1082148793 wait complete. 2024-07-26T11:01:53.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.660+0000 7fa8f6800640 1 Processor -- start 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.660+0000 7fa8f6800640 1 -- start start 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f6800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0075800 0x7fa8f01a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0076db0 0x7fa8f01a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f6800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0113cb0 0x7fa8f01a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f6800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01198f0 con 0x7fa8f0076db0 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f6800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0119770 con 0x7fa8f0113cb0 2024-07-26T11:01:53.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f6800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0119a70 con 0x7fa8f0075800 2024-07-26T11:01:53.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f4a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0076db0 0x7fa8f01a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:53.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0075800 0x7fa8f01a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:53.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0075800 0x7fa8f01a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:49342/0 (socket says 172.21.15.144:49342) 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0113cb0 0x7fa8f01a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f5400640 1 -- 172.21.15.144:0/1807489961 learned_addr learned my addr 172.21.15.144:0/1807489961 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.661+0000 7fa8f4a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0076db0 0x7fa8f01a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52762/0 (socket says 172.21.15.144:52762) 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f5e00640 1 -- 172.21.15.144:0/1807489961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0075800 msgr2=0x7fa8f01a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f5e00640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0075800 0x7fa8f01a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f5e00640 1 -- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0076db0 msgr2=0x7fa8f01a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f5e00640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0076db0 0x7fa8f01a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f5e00640 1 -- 172.21.15.144:0/1807489961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f01a6270 con 0x7fa8f0113cb0 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f5e00640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0113cb0 0x7fa8f01a5b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800e970 tx=0x7fa8e800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e800ccc0 con 0x7fa8f0113cb0 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8e80035f0 con 0x7fa8f0113cb0 2024-07-26T11:01:53.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f01a6560 con 0x7fa8f0113cb0 2024-07-26T11:01:53.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.662+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e800ce60 con 0x7fa8f0113cb0 2024-07-26T11:01:53.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.663+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f01accc0 con 0x7fa8f0113cb0 2024-07-26T11:01:53.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.664+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f0075c10 con 0x7fa8f0113cb0 2024-07-26T11:01:53.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.667+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fa8e8004020 con 0x7fa8f0113cb0 2024-07-26T11:01:53.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.668+0000 7fa8ee000640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa8d40b3c20 0x7fa8d40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:53.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.669+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa8e80e4430 con 0x7fa8f0113cb0 2024-07-26T11:01:53.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.669+0000 7fa8f5400640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa8d40b3c20 0x7fa8d40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:53.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.669+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa8e80ab620 con 0x7fa8f0113cb0 2024-07-26T11:01:53.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.670+0000 7fa8f5400640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa8d40b3c20 0x7fa8d40b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0004af0 tx=0x7fa8e0009360 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:53.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.909+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8f01a1430 con 0x7fa8f0113cb0 2024-07-26T11:01:53.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.910+0000 7fa8ee000640 1 -- 172.21.15.144:0/1807489961 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11931 (secure 0 0 0) 0x7fa8f01a1430 con 0x7fa8f0113cb0 2024-07-26T11:01:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:53.913 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 25 2024-07-26T11:01:53.913 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":25,"btime":"2024-07-26T10:58:47:222852+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T10:58:46.244128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T11:01:53.915 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.913+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa8d40b3c20 msgr2=0x7fa8d40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:53.915 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.913+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa8d40b3c20 0x7fa8d40b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0004af0 tx=0x7fa8e0009360 comp rx=0 tx=0).stop 2024-07-26T11:01:53.915 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.913+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0113cb0 msgr2=0x7fa8f01a5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:53.915 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.913+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0113cb0 0x7fa8f01a5b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800e970 tx=0x7fa8e800ee30 comp rx=0 tx=0).stop 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 shutdown_connections 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa8d40b3c20 0x7fa8d40b6130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8f0113cb0 0x7fa8f01a5b40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8f0076db0 0x7fa8f01a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 --2- 172.21.15.144:0/1807489961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8f0075800 0x7fa8f01a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 >> 172.21.15.144:0/1807489961 conn(0x7fa8f00fee70 msgr2=0x7fa8f010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:53.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 shutdown_connections 2024-07-26T11:01:53.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:53.914+0000 7fa8f6800640 1 -- 172.21.15.144:0/1807489961 wait complete. 2024-07-26T11:01:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:54 smithi144 ceph-mon[32129]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:54 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1807489961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:54 smithi160 ceph-mon[36715]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:54 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1807489961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:54 smithi183 ceph-mon[37273]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:01:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:54 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1807489961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:01:54.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:01:55.364 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:55 smithi144 ceph-mon[32129]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:55 smithi160 ceph-mon[36715]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:55 smithi183 ceph-mon[37273]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:01:57.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 -- 172.21.15.144:0/3790401223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410b860 msgr2=0x7f73f4112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:57.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3790401223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410b860 0x7f73f4112090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009ca0 tx=0x7f73ec02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 -- 172.21.15.144:0/3790401223 shutdown_connections 2024-07-26T11:01:57.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3790401223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410b860 0x7f73f4112090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3790401223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410af10 0x7f73f410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3790401223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.516+0000 7f73f8a00640 1 -- 172.21.15.144:0/3790401223 >> 172.21.15.144:0/3790401223 conn(0x7f73f4101040 msgr2=0x7f73f4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:57.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.517+0000 7f73f8a00640 1 -- 172.21.15.144:0/3790401223 shutdown_connections 2024-07-26T11:01:57.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.517+0000 7f73f8a00640 1 -- 172.21.15.144:0/3790401223 wait complete. 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 Processor -- start 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 -- start start 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410af10 0x7f73f419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410b860 0x7f73f41a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4115560 con 0x7f73f41055d0 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41153e0 con 0x7f73f410b860 2024-07-26T11:01:57.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.518+0000 7f73f8a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41156e0 con 0x7f73f410af10 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f419beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52782/0 (socket says 172.21.15.144:52782) 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 -- 172.21.15.144:0/3955992050 learned_addr learned my addr 172.21.15.144:0/3955992050 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73e9e00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410af10 0x7f73f419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f3200640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410b860 0x7f73f41a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 -- 172.21.15.144:0/3955992050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410af10 msgr2=0x7f73f419c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410af10 0x7f73f419c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 -- 172.21.15.144:0/3955992050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410b860 msgr2=0x7f73f41a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410b860 0x7f73f41a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f2800640 1 -- 172.21.15.144:0/3955992050 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f41a1e20 con 0x7f73f41055d0 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73e9e00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410af10 0x7f73f419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:01:57.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.519+0000 7f73f3200640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410b860 0x7f73f41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.520+0000 7f73f2800640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f419beb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f73e400e9f0 tx=0x7f73e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.520+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e400cd80 con 0x7f73f41055d0 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.520+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e40108e0 con 0x7f73f41055d0 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.520+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4018980 con 0x7f73f41055d0 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.520+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f41a2110 con 0x7f73f41055d0 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.520+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f41a8870 con 0x7f73f41055d0 2024-07-26T11:01:57.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.521+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f41059e0 con 0x7f73f41055d0 2024-07-26T11:01:57.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.524+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f73e4010430 con 0x7f73f41055d0 2024-07-26T11:01:57.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.525+0000 7f73ebe00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73d80b3ca0 0x7f73d80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:01:57.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.526+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f73e40e49b0 con 0x7f73f41055d0 2024-07-26T11:01:57.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.526+0000 7f73e9e00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73d80b3ca0 0x7f73d80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:57.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.526+0000 7f73e9e00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73d80b3ca0 0x7f73d80b61b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f73f41048f0 tx=0x7f73dc00b410 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:01:57.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.527+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f73e40abb10 con 0x7f73f41055d0 2024-07-26T11:01:57.726 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.724+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73f402e0c0 con 0x7f73f41055d0 2024-07-26T11:01:57.729 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.728+0000 7f73ebe00640 1 -- 172.21.15.144:0/3955992050 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7f73e40ab410 con 0x7f73f41055d0 2024-07-26T11:01:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:01:57.731 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:01:57.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.730+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73d80b3ca0 msgr2=0x7f73d80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:57.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.730+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73d80b3ca0 0x7f73d80b61b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f73f41048f0 tx=0x7f73dc00b410 comp rx=0 tx=0).stop 2024-07-26T11:01:57.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.730+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 msgr2=0x7f73f419beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:01:57.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.730+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f419beb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f73e400e9f0 tx=0x7f73e400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.733 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 shutdown_connections 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f73d80b3ca0 0x7f73d80b61b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f410b860 0x7f73f41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73f410af10 0x7f73f419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 --2- 172.21.15.144:0/3955992050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73f41055d0 0x7f73f419beb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 >> 172.21.15.144:0/3955992050 conn(0x7f73f4101040 msgr2=0x7f73f410a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 shutdown_connections 2024-07-26T11:01:57.734 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:01:57.731+0000 7f73f8a00640 1 -- 172.21.15.144:0/3955992050 wait complete. 2024-07-26T11:01:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:58 smithi144 ceph-mon[32129]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:01:58 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3955992050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:01:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:58 smithi160 ceph-mon[36715]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:01:58 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3955992050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:01:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:58 smithi183 ceph-mon[37273]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:01:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:01:58 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3955992050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:01:58.769 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T10:57:25.002914+0000', 'modified': '2024-07-26T10:58:46.244128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24728, 'mds_1': 24715, 'mds_2': 24703}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24677': {'gid': 24677, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6835/405635522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6834', 'nonce': 405635522}, {'type': 'v1', 'addr': '172.21.15.160:6835', 'nonce': 405635522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24691': {'gid': 24691, 'name': 'i', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.183:6839/3458220569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6838', 'nonce': 3458220569}, {'type': 'v1', 'addr': '172.21.15.183:6839', 'nonce': 3458220569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24701': {'gid': 24701, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6837/4027087859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6836', 'nonce': 4027087859}, {'type': 'v1', 'addr': '172.21.15.160:6837', 'nonce': 4027087859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24703': {'gid': 24703, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 27, 'addr': '172.21.15.144:6839/2313694680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6838', 'nonce': 2313694680}, {'type': 'v1', 'addr': '172.21.15.144:6839', 'nonce': 2313694680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24715': {'gid': 24715, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.144:6841/1389845259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6840', 'nonce': 1389845259}, {'type': 'v1', 'addr': '172.21.15.144:6841', 'nonce': 1389845259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24728': {'gid': 24728, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.160:6839/1074049609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6838', 'nonce': 1074049609}, {'type': 'v1', 'addr': '172.21.15.160:6839', 'nonce': 1074049609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24728, 'qdb_cluster': [24728, 24715, 24703]} 2024-07-26T11:01:58.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-07-26T11:01:58.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.g status 2024-07-26T11:01:59.249 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:00.441 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:00 smithi144 ceph-mon[32129]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:00 smithi160 ceph-mon[36715]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:00 smithi183 ceph-mon[37273]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:01.460 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:01 smithi144 ceph-mon[32129]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:01 smithi160 ceph-mon[36715]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:01 smithi183 ceph-mon[37273]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 -- 172.21.15.144:0/1924006272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c4104240 msgr2=0x7f17c4104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 --2- 172.21.15.144:0/1924006272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c4104240 0x7f17c4104650 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009900 tx=0x7f17b402a400 comp rx=0 tx=0).stop 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 -- 172.21.15.144:0/1924006272 shutdown_connections 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 --2- 172.21.15.144:0/1924006272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c410d6a0 0x7f17c410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 --2- 172.21.15.144:0/1924006272 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17c4104b90 0x7f17c410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 --2- 172.21.15.144:0/1924006272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c4104240 0x7f17c4104650 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 -- 172.21.15.144:0/1924006272 >> 172.21.15.144:0/1924006272 conn(0x7f17c40fdd60 msgr2=0x7f17c41001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 -- 172.21.15.144:0/1924006272 shutdown_connections 2024-07-26T11:02:01.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.665+0000 7f17ca600640 1 -- 172.21.15.144:0/1924006272 wait complete. 2024-07-26T11:02:01.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.667+0000 7f17ca600640 1 Processor -- start 2024-07-26T11:02:01.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.667+0000 7f17ca600640 1 -- start start 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17ca600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17c4104240 0x7f17c4199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17ca600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c4104b90 0x7f17c419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17ca600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 0x7f17c419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17ca600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4113380 con 0x7f17c4104b90 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17ca600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4113200 con 0x7f17c4104240 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17ca600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4113500 con 0x7f17c410d6a0 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c4104b90 0x7f17c419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 0x7f17c419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 0x7f17c419f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35664/0 (socket says 172.21.15.144:35664) 2024-07-26T11:02:01.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 -- 172.21.15.144:0/3225996098 learned_addr learned my addr 172.21.15.144:0/3225996098 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:01.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9200640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17c4104240 0x7f17c4199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 -- 172.21.15.144:0/3225996098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17c4104240 msgr2=0x7f17c4199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17c4104240 0x7f17c4199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 -- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c4104b90 msgr2=0x7f17c419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c4104b90 0x7f17c419a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.668+0000 7f17c9c00640 1 -- 172.21.15.144:0/3225996098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c419fca0 con 0x7f17c410d6a0 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.669+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 0x7f17c419f570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00e970 tx=0x7f17bc00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.669+0000 7f17c1600640 1 -- 172.21.15.144:0/3225996098 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00ccc0 con 0x7f17c410d6a0 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.669+0000 7f17c1600640 1 -- 172.21.15.144:0/3225996098 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17bc00ce60 con 0x7f17c410d6a0 2024-07-26T11:02:01.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.669+0000 7f17c1600640 1 -- 172.21.15.144:0/3225996098 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc010430 con 0x7f17c410d6a0 2024-07-26T11:02:01.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.669+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c419ff90 con 0x7f17c410d6a0 2024-07-26T11:02:01.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.669+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c41a66f0 con 0x7f17c410d6a0 2024-07-26T11:02:01.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.672+0000 7f17d28ca740 1 Processor -- start 2024-07-26T11:02:01.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.673+0000 7f17c1600640 1 -- 172.21.15.144:0/3225996098 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f17bc004ff0 con 0x7f17c410d6a0 2024-07-26T11:02:01.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.673+0000 7f17d28ca740 1 -- start start 2024-07-26T11:02:01.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7b8d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b8dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7b8e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17d28ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7b03970 con 0x55a1d77c7280 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17d28ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7af2e70 con 0x55a1d77e7f60 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17d28ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7af6eb0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17c9200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b8dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7b8d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17c9200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b8dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35678/0 (socket says 172.21.15.144:35678) 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17c9200640 1 -- 172.21.15.144:0/1059399318 learned_addr learned my addr 172.21.15.144:0/1059399318 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.674+0000 7f17c3e00640 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7b8e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c9200640 1 -- 172.21.15.144:0/1059399318 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 msgr2=0x55a1d7b8d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c9200640 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7b8d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c9200640 1 -- 172.21.15.144:0/1059399318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 msgr2=0x55a1d7b8e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c9200640 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7b8e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c9200640 1 -- 172.21.15.144:0/1059399318 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1d7a9a110 con 0x55a1d77bf0a0 2024-07-26T11:02:01.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c3e00640 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7b8e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c9200640 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b8dcc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009230 tx=0x7f17b4004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46e3d88795ab3cfb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17c1600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f17a80b3ca0 0x7f17a80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17ba000640 1 -- 172.21.15.144:0/1059399318 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4035040 con 0x55a1d77bf0a0 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17ba000640 1 -- 172.21.15.144:0/1059399318 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b40054f0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.675+0000 7f17ba000640 1 -- 172.21.15.144:0/1059399318 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b40035f0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.676+0000 7f17c1600640 1 -- 172.21.15.144:0/3225996098 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f17bc0e44d0 con 0x7f17c410d6a0 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.676+0000 7f17c3e00640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f17a80b3ca0 0x7f17a80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.676+0000 7f17d28ca740 1 -- 172.21.15.144:0/1059399318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 msgr2=0x55a1d7b8dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.676+0000 7f17c3e00640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f17a80b3ca0 0x7f17a80b61b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f17c410f8d0 tx=0x7f17ac004ab0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.676+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b8dcc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009230 tx=0x7f17b4004040 comp rx=0 tx=0).stop 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 -- 172.21.15.144:0/1059399318 shutdown_connections 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7b8e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b8dcc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1059399318 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7b8d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 -- 172.21.15.144:0/1059399318 >> 172.21.15.144:0/1059399318 conn(0x55a1d77dc350 msgr2=0x55a1d7b8b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 -- 172.21.15.144:0/1059399318 shutdown_connections 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 -- 172.21.15.144:0/1059399318 wait complete. 2024-07-26T11:02:01.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.677+0000 7f17d28ca740 1 -- start start 2024-07-26T11:02:01.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 Processor -- start 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b9d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77c7280 0x55a1d7b9d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55a1d79565f0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7a8f540 con 0x55a1d77c7280 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7ba0260 con 0x55a1d77e7f60 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17d28ca740 10 client.0 _ll_register_callbacks cb 0x55a1d7a9ef10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.678+0000 7f17c9200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b9d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b9d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58708/0 (socket says 172.21.15.144:58708) 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 -- 172.21.15.144:0/1549089934 learned_addr learned my addr 172.21.15.144:0/1549089934 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:01.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c3e00640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77c7280 0x55a1d7b9d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9c00640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 -- 172.21.15.144:0/1549089934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77e7f60 msgr2=0x55a1d7ba95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 -- 172.21.15.144:0/1549089934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77c7280 msgr2=0x55a1d7b9d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77c7280 0x55a1d7b9d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 -- 172.21.15.144:0/1549089934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1d7853540 con 0x55a1d77bf0a0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17c9200640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b9d3e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005ab0 tx=0x7f17b4004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccd1c8c68e9eb26 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.679+0000 7f17b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17b2a00640 1 -- 172.21.15.144:0/1549089934 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4035040 con 0x55a1d77bf0a0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17b2a00640 1 -- 172.21.15.144:0/1549089934 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f17b4004ce0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b40054f0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17b2a00640 1 -- 172.21.15.144:0/1549089934 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17b405a050 con 0x55a1d77bf0a0 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17d28ca740 10 client.15144 fetch_fsmap learned FSMap version 25 2024-07-26T11:02:01.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a1d7bac160 con 0x55a1d77bf0a0 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.680+0000 7f17b2a00640 1 -- 172.21.15.144:0/1549089934 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b40054f0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17b2a00640 1 -- 172.21.15.144:0/1549089934 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f17b4055030 con 0x55a1d77bf0a0 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17d28ca740 10 client.15144 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17d28ca740 10 client.15144 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17d28ca740 20 client.15144 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d77a68e0 0x55a1d7baf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17d28ca740 4 client.15144 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:02:01.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1d7b058c0 con 0x55a1d77a68e0 2024-07-26T11:02:01.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.681+0000 7f17c3e00640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d77a68e0 0x55a1d7baf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.682+0000 7f17c3e00640 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d77a68e0 0x55a1d7baf820 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a25c0b74089b325 server_cookie=6ca5c19039f85b50 in_seq=0 out_seq=0 2024-07-26T11:02:01.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.682+0000 7f17b2a00640 10 client.15144 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17b2a00640 1 -- 172.21.15.144:0/1549089934 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a1d7b058c0 con 0x55a1d77a68e0 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17b2a00640 10 client.15144 handle_command_reply: tid=0 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 client.15144 shutdown 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 msgr2=0x55a1d7b9d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b9d3e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005ab0 tx=0x7f17b4004500 comp rx=0 tx=0).stop 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 shutdown_connections 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d77a68e0 0x55a1d7baf820 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77c7280 0x55a1d7b9d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 --2- 172.21.15.144:0/1549089934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7b9d3e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17b2a00640 0 client.15144 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 >> 172.21.15.144:0/1549089934 conn(0x55a1d77dc350 msgr2=0x55a1d7b89c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:01.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 shutdown_connections 2024-07-26T11:02:01.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.683+0000 7f17d28ca740 1 -- 172.21.15.144:0/1549089934 wait complete. 2024-07-26T11:02:01.688 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.684+0000 7f17d28ca740 20 client.15144 trim_cache size 0 max 16384 2024-07-26T11:02:01.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 Processor -- start 2024-07-26T11:02:01.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 -- start start 2024-07-26T11:02:01.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7ba9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7badb60 con 0x55a1d77c7280 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7bae230 con 0x55a1d77e7f60 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.703+0000 7f17d28ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7bae900 con 0x55a1d77bf0a0 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7ba9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7ba9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58710/0 (socket says 172.21.15.144:58710) 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c9200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 -- 172.21.15.144:0/285829081 learned_addr learned my addr 172.21.15.144:0/285829081 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:01.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c9c00640 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 -- 172.21.15.144:0/285829081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 msgr2=0x55a1d7bae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 -- 172.21.15.144:0/285829081 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 msgr2=0x55a1d7ba9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.704+0000 7f17c3e00640 1 -- 172.21.15.144:0/285829081 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1d7b867f0 con 0x55a1d77c7280 2024-07-26T11:02:01.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.705+0000 7f17c3e00640 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7ba9530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f17c406ace0 tx=0x7f17ac012070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3c0abd62dc7f14 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:01.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.705+0000 7f17b9600640 1 -- 172.21.15.144:0/285829081 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac019040 con 0x55a1d77c7280 2024-07-26T11:02:01.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.705+0000 7f17b9600640 1 -- 172.21.15.144:0/285829081 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17ac02c8f0 con 0x55a1d77c7280 2024-07-26T11:02:01.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.705+0000 7f17b9600640 1 -- 172.21.15.144:0/285829081 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac0207a0 con 0x55a1d77c7280 2024-07-26T11:02:01.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 -- 172.21.15.144:0/285829081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 msgr2=0x55a1d7ba9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7ba9530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f17c406ace0 tx=0x7f17ac012070 comp rx=0 tx=0).stop 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 -- 172.21.15.144:0/285829081 shutdown_connections 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7ba9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77c7280 0x55a1d7ba9530 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 --2- 172.21.15.144:0/285829081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 -- 172.21.15.144:0/285829081 >> 172.21.15.144:0/285829081 conn(0x55a1d77dc350 msgr2=0x55a1d7b9d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 -- 172.21.15.144:0/285829081 shutdown_connections 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 -- 172.21.15.144:0/285829081 wait complete. 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.706+0000 7f17d28ca740 1 -- start start 2024-07-26T11:02:01.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 Processor -- start 2024-07-26T11:02:01.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bb0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77c7280 0x55a1d7bb11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7bb5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7baf6a0 con 0x55a1d77bf0a0 2024-07-26T11:02:01.711 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7baefd0 con 0x55a1d77e7f60 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a1d7bafd70 con 0x55a1d77c7280 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17d28ca740 10 client.0 _ll_register_callbacks cb 0x7f17b4009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17c9200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bb0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17c9200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bb0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58716/0 (socket says 172.21.15.144:58716) 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.707+0000 7f17c9200640 1 -- 172.21.15.144:0/3564448602 learned_addr learned my addr 172.21.15.144:0/3564448602 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7bb5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c3e00640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77c7280 0x55a1d7bb11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 -- 172.21.15.144:0/3564448602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77c7280 msgr2=0x55a1d7bb11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77c7280 0x55a1d7bb11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 -- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 msgr2=0x55a1d7bb0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bb0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 -- 172.21.15.144:0/3564448602 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1d7bb6660 con 0x55a1d77e7f60 2024-07-26T11:02:01.712 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9200640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bb0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17c9c00640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7bb5ff0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00e940 tx=0x7f17bc0aa7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c832318d8df55b8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.708+0000 7f17b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17b2a00640 1 -- 172.21.15.144:0/3564448602 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0e42c0 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17ac02c8f0 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17b2a00640 1 -- 172.21.15.144:0/3564448602 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f17bc0b3b50 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17b2a00640 1 -- 172.21.15.144:0/3564448602 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17bc086050 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17d28ca740 10 client.25298 fetch_fsmap learned FSMap version 25 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a1d7bb7ad0 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17b2a00640 1 -- 172.21.15.144:0/3564448602 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17ac02c8f0 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17b2a00640 1 -- 172.21.15.144:0/3564448602 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f17bc0a68a0 con 0x55a1d77e7f60 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.709+0000 7f17d28ca740 10 client.25298 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.710+0000 7f17d28ca740 10 client.25298 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T11:02:01.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.710+0000 7f17d28ca740 20 client.25298 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:01.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.710+0000 7f17d28ca740 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d777bc90 0x55a1d7c1b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:01.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.710+0000 7f17d28ca740 4 client.25298 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:02:01.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.710+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "status"}) -- 0x55a1d7b01360 con 0x55a1d777bc90 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.710+0000 7f17c9200640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d777bc90 0x55a1d7c1b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17c9200640 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d777bc90 0x55a1d7c1b4b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad5efc37b8880d91 server_cookie=98613c09cdd6e0bd in_seq=0 out_seq=0 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17b2a00640 10 client.25298 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17b2a00640 1 -- 172.21.15.144:0/3564448602 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a1d7b01360 con 0x55a1d777bc90 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17b2a00640 10 client.25298 handle_command_reply: tid=0 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 client.25298 shutdown 2024-07-26T11:02:01.715 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 msgr2=0x55a1d7bb5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7bb5ff0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00e940 tx=0x7f17bc0aa7b0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 shutdown_connections 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55a1d777bc90 0x55a1d7c1b4b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a1d77e7f60 0x55a1d7bb5ff0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a1d77c7280 0x55a1d7bb11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 --2- 172.21.15.144:0/3564448602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1d77bf0a0 0x55a1d7bb0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17b2a00640 0 client.25298 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.711+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 >> 172.21.15.144:0/3564448602 conn(0x55a1d77dc350 msgr2=0x55a1d7b8e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.712+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 shutdown_connections 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.712+0000 7f17d28ca740 1 -- 172.21.15.144:0/3564448602 wait complete. 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.712+0000 7f17d28ca740 20 client.25298 trim_cache size 0 max 16384 2024-07-26T11:02:01.716 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 2, 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24703, 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 203.57176714799999, 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 25, 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 90, 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 303.874141619 2024-07-26T11:02:01.717 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:02:01.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f17a80b3ca0 msgr2=0x7f17a80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f17a80b3ca0 0x7f17a80b61b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f17c410f8d0 tx=0x7f17ac004ab0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 msgr2=0x7f17c419f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:01.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 0x7f17c419f570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00e970 tx=0x7f17bc00ed70 comp rx=0 tx=0).stop 2024-07-26T11:02:01.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 shutdown_connections 2024-07-26T11:02:01.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f17a80b3ca0 0x7f17a80b61b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17c410d6a0 0x7f17c419f570 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c4104b90 0x7f17c419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 --2- 172.21.15.144:0/3225996098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17c4104240 0x7f17c4199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:01.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 >> 172.21.15.144:0/3225996098 conn(0x7f17c40fdd60 msgr2=0x7f17c4107130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:01.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 shutdown_connections 2024-07-26T11:02:01.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:01.713+0000 7f17ca600640 1 -- 172.21.15.144:0/3225996098 wait complete. 2024-07-26T11:02:02.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.a status 2024-07-26T11:02:03.072 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:04 smithi144 ceph-mon[32129]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:04 smithi160 ceph-mon[36715]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:04 smithi183 ceph-mon[37273]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:05.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 -- 172.21.15.144:0/2404292757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 msgr2=0x7f1094118860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 --2- 172.21.15.144:0/2404292757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 0x7f1094118860 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f108c009ca0 tx=0x7f108c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 -- 172.21.15.144:0/2404292757 shutdown_connections 2024-07-26T11:02:05.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 --2- 172.21.15.144:0/2404292757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 0x7f1094118860 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 --2- 172.21.15.144:0/2404292757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10941116e0 0x7f1094111af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 --2- 172.21.15.144:0/2404292757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f109410bda0 0x7f109410c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 -- 172.21.15.144:0/2404292757 >> 172.21.15.144:0/2404292757 conn(0x7f10940714b0 msgr2=0x7f1094109df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 -- 172.21.15.144:0/2404292757 shutdown_connections 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.372+0000 7f109b800640 1 -- 172.21.15.144:0/2404292757 wait complete. 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.373+0000 7f109b800640 1 Processor -- start 2024-07-26T11:02:05.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 -- start start 2024-07-26T11:02:05.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 0x7f109406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10941116e0 0x7f109406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 0x7f109406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f109411bd40 con 0x7f1094112030 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f109411bbc0 con 0x7f109410bda0 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.374+0000 7f109b800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f109411bec0 con 0x7f10941116e0 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 0x7f109406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1099800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 0x7f109406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 0x7f109406c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:48114/0 (socket says 172.21.15.144:48114) 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 -- 172.21.15.144:0/3850350330 learned_addr learned my addr 172.21.15.144:0/3850350330 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:05.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1093e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10941116e0 0x7f109406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 -- 172.21.15.144:0/3850350330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10941116e0 msgr2=0x7f109406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10941116e0 0x7f109406ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 -- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 msgr2=0x7f109406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 0x7f109406d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.375+0000 7f1098e00640 1 -- 172.21.15.144:0/3850350330 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10941b0930 con 0x7f109410bda0 2024-07-26T11:02:05.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.376+0000 7f1098e00640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 0x7f109406c8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f108400e9f0 tx=0x7f108400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:05.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.376+0000 7f1091600640 1 -- 172.21.15.144:0/3850350330 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108400cd80 con 0x7f109410bda0 2024-07-26T11:02:05.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.376+0000 7f1091600640 1 -- 172.21.15.144:0/3850350330 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10840108e0 con 0x7f109410bda0 2024-07-26T11:02:05.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.376+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10941b0c20 con 0x7f109410bda0 2024-07-26T11:02:05.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.376+0000 7f1091600640 1 -- 172.21.15.144:0/3850350330 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084018980 con 0x7f109410bda0 2024-07-26T11:02:05.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.376+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10941b1160 con 0x7f109410bda0 2024-07-26T11:02:05.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 Processor -- start 2024-07-26T11:02:05.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 -- start start 2024-07-26T11:02:05.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f1091600640 1 -- 172.21.15.144:0/3850350330 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f1084010430 con 0x7f109410bda0 2024-07-26T11:02:05.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ebe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af00a0 0x55e0f1ebeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af8280 0x55e0f1ebeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1e34970 con 0x55e0f1af00a0 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1e23e70 con 0x55e0f1af8280 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.380+0000 7f10a246d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1e27eb0 con 0x55e0f1b18f60 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1098e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af00a0 0x55e0f1ebeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ebe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ebe570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35734/0 (socket says 172.21.15.144:35734) 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 -- 172.21.15.144:0/4083598452 learned_addr learned my addr 172.21.15.144:0/4083598452 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:05.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1093e00640 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af8280 0x55e0f1ebeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 -- 172.21.15.144:0/4083598452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af8280 msgr2=0x55e0f1ebeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af8280 0x55e0f1ebeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 -- 172.21.15.144:0/4083598452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af00a0 msgr2=0x55e0f1ebeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af00a0 0x55e0f1ebeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.381+0000 7f1099800640 1 -- 172.21.15.144:0/4083598452 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f1dcb110 con 0x55e0f1b18f60 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.382+0000 7f1099800640 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ebe570 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f108c009ca0 tx=0x7f108c009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8276c4f67d4c791 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.382+0000 7f108a000640 1 -- 172.21.15.144:0/4083598452 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c031cf0 con 0x55e0f1b18f60 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.382+0000 7f108a000640 1 -- 172.21.15.144:0/4083598452 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f108c036030 con 0x55e0f1b18f60 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.382+0000 7f108a000640 1 -- 172.21.15.144:0/4083598452 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c004280 con 0x55e0f1b18f60 2024-07-26T11:02:05.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f1091600640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10780b3c20 0x7f10780b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 -- 172.21.15.144:0/4083598452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 msgr2=0x55e0f1ebe570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ebe570 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f108c009ca0 tx=0x7f108c009430 comp rx=0 tx=0).stop 2024-07-26T11:02:05.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f1091600640 1 -- 172.21.15.144:0/3850350330 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f10840e4440 con 0x7f109410bda0 2024-07-26T11:02:05.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f1098e00640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10780b3c20 0x7f10780b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 -- 172.21.15.144:0/4083598452 shutdown_connections 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af8280 0x55e0f1ebeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af00a0 0x55e0f1ebeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 --2- 172.21.15.144:0/4083598452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ebe570 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 -- 172.21.15.144:0/4083598452 >> 172.21.15.144:0/4083598452 conn(0x55e0f1b0d350 msgr2=0x55e0f1ebc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 -- 172.21.15.144:0/4083598452 shutdown_connections 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.383+0000 7f10a246d740 1 -- 172.21.15.144:0/4083598452 wait complete. 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.384+0000 7f1098e00640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10780b3c20 0x7f10780b6130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10840aaee0 tx=0x7f10840e1920 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.384+0000 7f10a246d740 1 -- start start 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.384+0000 7f10a246d740 1 Processor -- start 2024-07-26T11:02:05.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af00a0 0x55e0f1ec9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1eca370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1c875f0 con 0x55e0f1af8280 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1dc0540 con 0x55e0f1b18f60 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1eccb70 con 0x55e0f1af00a0 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f10a246d740 10 client.0 _ll_register_callbacks cb 0x55e0f1dcff10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f1093e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1eca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f1093e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1eca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58758/0 (socket says 172.21.15.144:58758) 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f1099800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af00a0 0x55e0f1ec9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f1093e00640 1 -- 172.21.15.144:0/2134075520 learned_addr learned my addr 172.21.15.144:0/2134075520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.385+0000 7f1099800640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1093e00640 1 -- 172.21.15.144:0/2134075520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af00a0 msgr2=0x55e0f1ec9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1093e00640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af00a0 0x55e0f1ec9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1093e00640 1 -- 172.21.15.144:0/2134075520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1b18f60 msgr2=0x55e0f1ed5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1093e00640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1093e00640 1 -- 172.21.15.144:0/2134075520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f1b84540 con 0x55e0f1af8280 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1099800640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af00a0 0x55e0f1ec9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f1093e00640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1eca370 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f10800126e0 tx=0x7f1080012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ab4b0d0cecb427 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f107ea00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.386+0000 7f107ea00640 1 -- 172.21.15.144:0/2134075520 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080012190 con 0x55e0f1af8280 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f107ea00640 1 -- 172.21.15.144:0/2134075520 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f108000f920 con 0x55e0f1af8280 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f108c036030 con 0x55e0f1af8280 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f107ea00640 1 -- 172.21.15.144:0/2134075520 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f108000fb20 con 0x55e0f1af8280 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f10a246d740 10 client.15162 fetch_fsmap learned FSMap version 25 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e0f1ed8a70 con 0x55e0f1af8280 2024-07-26T11:02:05.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f107ea00640 1 -- 172.21.15.144:0/2134075520 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f108c036030 con 0x55e0f1af8280 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f107ea00640 1 -- 172.21.15.144:0/2134075520 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1080010c90 con 0x55e0f1af8280 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.387+0000 7f10a246d740 10 client.15162 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.388+0000 7f10a246d740 10 client.15162 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.388+0000 7f10a246d740 20 client.15162 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.388+0000 7f10a246d740 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1ad78e0 0x55e0f1edc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.388+0000 7f10a246d740 4 client.15162 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.388+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0f1e368c0 con 0x55e0f1ad78e0 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.388+0000 7f1099800640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1ad78e0 0x55e0f1edc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.389+0000 7f1099800640 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1ad78e0 0x55e0f1edc520 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91067f57bf0d37be server_cookie=3dd73e3b0510f0c9 in_seq=0 out_seq=0 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.389+0000 7f107ea00640 10 client.15162 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.389+0000 7f107ea00640 1 -- 172.21.15.144:0/2134075520 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e0f1e368c0 con 0x55e0f1ad78e0 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.389+0000 7f107ea00640 10 client.15162 handle_command_reply: tid=0 2024-07-26T11:02:05.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.389+0000 7f10a246d740 1 client.15162 shutdown 2024-07-26T11:02:05.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 msgr2=0x55e0f1eca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1eca370 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f10800126e0 tx=0x7f1080012ba0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 shutdown_connections 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1ad78e0 0x55e0f1edc520 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1eca370 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 --2- 172.21.15.144:0/2134075520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af00a0 0x55e0f1ec9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 >> 172.21.15.144:0/2134075520 conn(0x55e0f1b0d350 msgr2=0x55e0f1ec1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f107ea00640 0 client.15162 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 shutdown_connections 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 1 -- 172.21.15.144:0/2134075520 wait complete. 2024-07-26T11:02:05.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.390+0000 7f10a246d740 20 client.15162 trim_cache size 0 max 16384 2024-07-26T11:02:05.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 Processor -- start 2024-07-26T11:02:05.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 -- start start 2024-07-26T11:02:05.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1eda9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.414 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 0x55e0f1edb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1eda860 con 0x55e0f1b18f60 2024-07-26T11:02:05.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1edaf30 con 0x55e0f1af00a0 2024-07-26T11:02:05.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.410+0000 7f10a246d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1edb600 con 0x55e0f1af8280 2024-07-26T11:02:05.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 0x55e0f1edb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.415 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 0x55e0f1edb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35746/0 (socket says 172.21.15.144:35746) 2024-07-26T11:02:05.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 -- 172.21.15.144:0/3412127793 learned_addr learned my addr 172.21.15.144:0/3412127793 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:05.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1099800640 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 -- 172.21.15.144:0/3412127793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 msgr2=0x55e0f1eda9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1099800640 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1eda9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1eda9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 -- 172.21.15.144:0/3412127793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1b18f60 msgr2=0x55e0f1ed5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 -- 172.21.15.144:0/3412127793 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f1ecaa80 con 0x55e0f1af8280 2024-07-26T11:02:05.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1099800640 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1eda9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:02:05.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.411+0000 7f1093e00640 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 0x55e0f1edb0b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f108000fab0 tx=0x7f108000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72220335657b0b48 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:05.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.412+0000 7f1089600640 1 -- 172.21.15.144:0/3412127793 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108000fcc0 con 0x55e0f1af8280 2024-07-26T11:02:05.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.412+0000 7f1089600640 1 -- 172.21.15.144:0/3412127793 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f108000fe60 con 0x55e0f1af8280 2024-07-26T11:02:05.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.412+0000 7f1089600640 1 -- 172.21.15.144:0/3412127793 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108001cdd0 con 0x55e0f1af8280 2024-07-26T11:02:05.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 -- 172.21.15.144:0/3412127793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 msgr2=0x55e0f1edb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 0x55e0f1edb0b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f108000fab0 tx=0x7f108000f790 comp rx=0 tx=0).stop 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 -- 172.21.15.144:0/3412127793 shutdown_connections 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1b18f60 0x55e0f1ed5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1af8280 0x55e0f1edb0b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 --2- 172.21.15.144:0/3412127793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1eda9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 -- 172.21.15.144:0/3412127793 >> 172.21.15.144:0/3412127793 conn(0x55e0f1b0d350 msgr2=0x55e0f1ec8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 -- 172.21.15.144:0/3412127793 shutdown_connections 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 -- 172.21.15.144:0/3412127793 wait complete. 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.413+0000 7f10a246d740 1 -- start start 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 Processor -- start 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1ee4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1ee0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ee0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1edc3a0 con 0x55e0f1af8280 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1edbcd0 con 0x55e0f1af00a0 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e0f1edca70 con 0x55e0f1b18f60 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f10a246d740 10 client.0 _ll_register_callbacks cb 0x7f1080009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f1099800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1ee4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f1093e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1ee0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f1099800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1ee4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:48154/0 (socket says 172.21.15.144:48154) 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.414+0000 7f1099800640 1 -- 172.21.15.144:0/1909503800 learned_addr learned my addr 172.21.15.144:0/1909503800 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ee0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 -- 172.21.15.144:0/1909503800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 msgr2=0x55e0f1ee0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ee0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 -- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 msgr2=0x55e0f1ee0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1ee0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 -- 172.21.15.144:0/1909503800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f1ee1620 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f1099800640 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1ee4f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f108c004af0 tx=0x7f108c057040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e97386df996d1cb4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f107ea00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.415+0000 7f107ea00640 1 -- 172.21.15.144:0/1909503800 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c031660 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f108000fe60 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f107ea00640 1 -- 172.21.15.144:0/1909503800 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f108c04a040 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f107ea00640 1 -- 172.21.15.144:0/1909503800 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f108c031800 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f10a246d740 10 client.25313 fetch_fsmap learned FSMap version 25 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e0f1f44670 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f107ea00640 1 -- 172.21.15.144:0/1909503800 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f108000fe60 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f107ea00640 1 -- 172.21.15.144:0/1909503800 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f108c043070 con 0x55e0f1af00a0 2024-07-26T11:02:05.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.416+0000 7f10a246d740 10 client.25313 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f10a246d740 10 client.25313 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f10a246d740 20 client.25313 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f10a246d740 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1aacc90 0x55e0f1f47cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f10a246d740 4 client.25313 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "status"}) -- 0x55e0f1e32360 con 0x55e0f1aacc90 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f1093e00640 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1aacc90 0x55e0f1f47cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f1093e00640 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1aacc90 0x55e0f1f47cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d9dc11f696da3b2 server_cookie=34e23cc50e7f197d in_seq=0 out_seq=0 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.417+0000 7f107ea00640 10 client.25313 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f107ea00640 1 -- 172.21.15.144:0/1909503800 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e0f1e32360 con 0x55e0f1aacc90 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f107ea00640 10 client.25313 handle_command_reply: tid=0 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 client.25313 shutdown 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 msgr2=0x55e0f1ee4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1ee4f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f108c004af0 tx=0x7f108c057040 comp rx=0 tx=0).stop 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 shutdown_connections 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e0f1b18f60 0x55e0f1ee0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0f1af8280 0x55e0f1ee0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55e0f1aacc90 0x55e0f1f47cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 --2- 172.21.15.144:0/1909503800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e0f1af00a0 0x55e0f1ee4f00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 >> 172.21.15.144:0/1909503800 conn(0x55e0f1b0d350 msgr2=0x55e0f1ec9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f107ea00640 0 client.25313 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.418+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 shutdown_connections 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f10a246d740 1 -- 172.21.15.144:0/1909503800 wait complete. 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f10a246d740 20 client.25313 trim_cache size 0 max 16384 2024-07-26T11:02:05.429 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 1, 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24715, 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 208.28856186900001, 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 25, 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 90, 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:02:05.430 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 302.77524648299999 2024-07-26T11:02:05.431 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:02:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10780b3c20 msgr2=0x7f10780b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f109b800640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10780b3c20 0x7f10780b6130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10840aaee0 tx=0x7f10840e1920 comp rx=0 tx=0).stop 2024-07-26T11:02:05.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 msgr2=0x7f109406c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f109b800640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 0x7f109406c8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f108400e9f0 tx=0x7f108400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.419+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 shutdown_connections 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f10780b3c20 0x7f10780b6130 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1094112030 0x7f109406d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10941116e0 0x7f109406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 --2- 172.21.15.144:0/3850350330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f109410bda0 0x7f109406c8d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 >> 172.21.15.144:0/3850350330 conn(0x7f10940714b0 msgr2=0x7f1094071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 shutdown_connections 2024-07-26T11:02:05.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:05.420+0000 7f109b800640 1 -- 172.21.15.144:0/3850350330 wait complete. 2024-07-26T11:02:06.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T11:02:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:06 smithi144 ceph-mon[32129]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:02:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:06 smithi160 ceph-mon[36715]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:02:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:06 smithi183 ceph-mon[37273]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:02:06.664 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:07.414 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:07 smithi144 ceph-mon[32129]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:07 smithi160 ceph-mon[36715]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:07 smithi183 ceph-mon[37273]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:09.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 -- 172.21.15.144:0/2461970431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b41099f0 msgr2=0x7fd1b4109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 --2- 172.21.15.144:0/2461970431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b41099f0 0x7fd1b4109e00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009900 tx=0x7fd1a802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 -- 172.21.15.144:0/2461970431 shutdown_connections 2024-07-26T11:02:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 --2- 172.21.15.144:0/2461970431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b410fc80 0x7fd1b41164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 --2- 172.21.15.144:0/2461970431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b410f330 0x7fd1b410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 --2- 172.21.15.144:0/2461970431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b41099f0 0x7fd1b4109e00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 -- 172.21.15.144:0/2461970431 >> 172.21.15.144:0/2461970431 conn(0x7fd1b4078ff0 msgr2=0x7fd1b4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 -- 172.21.15.144:0/2461970431 shutdown_connections 2024-07-26T11:02:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.004+0000 7fd1bac00640 1 -- 172.21.15.144:0/2461970431 wait complete. 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.005+0000 7fd1bac00640 1 Processor -- start 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 -- start start 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 0x7fd1b41a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b410f330 0x7fd1b41a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b410fc80 0x7fd1b41a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4119990 con 0x7fd1b410f330 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4119810 con 0x7fd1b41099f0 2024-07-26T11:02:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1bac00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4119b10 con 0x7fd1b410fc80 2024-07-26T11:02:09.009 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1b8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b410f330 0x7fd1b41a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1b9800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 0x7fd1b41a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1b9800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 0x7fd1b41a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:48164/0 (socket says 172.21.15.144:48164) 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1b9800640 1 -- 172.21.15.144:0/1418873945 learned_addr learned my addr 172.21.15.144:0/1418873945 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.006+0000 7fd1b8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b410f330 0x7fd1b41a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58802/0 (socket says 172.21.15.144:58802) 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1ba200640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b410fc80 0x7fd1b41a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b9800640 1 -- 172.21.15.144:0/1418873945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b410fc80 msgr2=0x7fd1b41a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b9800640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b410fc80 0x7fd1b41a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b9800640 1 -- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b410f330 msgr2=0x7fd1b41a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b9800640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b410f330 0x7fd1b41a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b9800640 1 -- 172.21.15.144:0/1418873945 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b41a6290 con 0x7fd1b41099f0 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b9800640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 0x7fd1b41a0320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8005ab0 tx=0x7fd1a8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:09.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.007+0000 7fd1b2000640 1 -- 172.21.15.144:0/1418873945 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a803f070 con 0x7fd1b41099f0 2024-07-26T11:02:09.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.008+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b41a6520 con 0x7fd1b41099f0 2024-07-26T11:02:09.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.008+0000 7fd1b2000640 1 -- 172.21.15.144:0/1418873945 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1a8031b60 con 0x7fd1b41099f0 2024-07-26T11:02:09.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.008+0000 7fd1b2000640 1 -- 172.21.15.144:0/1418873945 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8035030 con 0x7fd1b41099f0 2024-07-26T11:02:09.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.008+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b41acce0 con 0x7fd1b41099f0 2024-07-26T11:02:09.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 Processor -- start 2024-07-26T11:02:09.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 -- start start 2024-07-26T11:02:09.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79d96760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799c80a0 0x559b79d96ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 0x559b79d971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x559b79d0c950 con 0x559b799d0280 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b79cfbe50 con 0x559b799c80a0 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.011+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559b79cffe90 con 0x559b799f0f60 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1b8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 0x559b79d971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79d96760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79d96760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35784/0 (socket says 172.21.15.144:35784) 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1b8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 0x559b79d971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58814/0 (socket says 172.21.15.144:58814) 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 -- 172.21.15.144:0/3847157774 learned_addr learned my addr 172.21.15.144:0/3847157774 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:09.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1b2000640 1 -- 172.21.15.144:0/1418873945 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fd1a8042070 con 0x7fd1b41099f0 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 -- 172.21.15.144:0/3847157774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799c80a0 msgr2=0x559b79d96ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1b9800640 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799c80a0 0x559b79d96ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799c80a0 0x559b79d96ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 -- 172.21.15.144:0/3847157774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 msgr2=0x559b79d971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 0x559b79d971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1ba200640 1 -- 172.21.15.144:0/3847157774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b79ca30f0 con 0x559b799f0f60 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1b9800640 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799c80a0 0x559b79d96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.012+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 0x559b79d971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.013+0000 7fd1ba200640 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79d96760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009fd0 tx=0x7fd1ac0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13450af766922776 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.013+0000 7fd1a6a00640 1 -- 172.21.15.144:0/3847157774 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac004d20 con 0x559b799f0f60 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.013+0000 7fd1a6a00640 1 -- 172.21.15.144:0/3847157774 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1ac004050 con 0x559b799f0f60 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.013+0000 7fd1a6a00640 1 -- 172.21.15.144:0/3847157774 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac005280 con 0x559b799f0f60 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1b2000640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd19c0b3c20 0x7fd19c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/3847157774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 msgr2=0x559b79d96760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79d96760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009fd0 tx=0x7fd1ac0118d0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1b2000640 1 -- 172.21.15.144:0/1418873945 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd1a81047c0 con 0x7fd1b41099f0 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/3847157774 shutdown_connections 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1b9800640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd19c0b3c20 0x7fd19c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799d0280 0x559b79d971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799c80a0 0x559b79d96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/3847157774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79d96760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/3847157774 >> 172.21.15.144:0/3847157774 conn(0x559b799e5350 msgr2=0x559b79d94650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/3847157774 shutdown_connections 2024-07-26T11:02:09.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.014+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/3847157774 wait complete. 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.015+0000 7fd1b9800640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd19c0b3c20 0x7fd19c0b6130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x559b79d953e0 tx=0x7fd1a8031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.015+0000 7fd1c2eb0740 1 -- start start 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.015+0000 7fd1c2eb0740 1 Processor -- start 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79da2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79da2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79dae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x559b79b5f5f0 con 0x559b799c80a0 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b79c98520 con 0x559b799d0280 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559b79da4f40 con 0x559b799f0f60 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1c2eb0740 10 client.0 _ll_register_callbacks cb 0x559b79ca7ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79da2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79da2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58820/0 (socket says 172.21.15.144:58820) 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1ba200640 1 -- 172.21.15.144:0/721501981 learned_addr learned my addr 172.21.15.144:0/721501981 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79da2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1ba200640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79dae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1b8e00640 1 -- 172.21.15.144:0/721501981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 msgr2=0x559b79dae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79dae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1b8e00640 1 -- 172.21.15.144:0/721501981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 msgr2=0x559b79da2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79da2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1b8e00640 1 -- 172.21.15.144:0/721501981 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b79a5c540 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.016+0000 7fd1ba200640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79da2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79da2770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd1b406bfe0 tx=0x7fd1a0012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19578a4b9f13399e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd19b400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd19b400640 1 -- 172.21.15.144:0/721501981 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0004030 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd19b400640 1 -- 172.21.15.144:0/721501981 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd1a00041d0 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1ac004050 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd19b400640 1 -- 172.21.15.144:0/721501981 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd1a0025050 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd1c2eb0740 10 client.25322 fetch_fsmap learned FSMap version 25 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.017+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559b79db0e30 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd19b400640 1 -- 172.21.15.144:0/721501981 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1ac004050 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd19b400640 1 -- 172.21.15.144:0/721501981 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fd1a0003900 con 0x559b799d0280 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1c2eb0740 10 client.25322 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:02:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1c2eb0740 10 client.25322 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:02:09.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1c2eb0740 20 client.25322 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:09.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b799af8e0 0x559b79db48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1c2eb0740 4 client.25322 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:02:09.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b79d0e8a0 con 0x559b799af8e0 2024-07-26T11:02:09.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.018+0000 7fd1ba200640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b799af8e0 0x559b79db48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.019+0000 7fd1ba200640 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b799af8e0 0x559b79db48d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78d0787219e6eb1f server_cookie=98d81c00c4cf039d in_seq=0 out_seq=0 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.019+0000 7fd19b400640 10 client.25322 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd19b400640 1 -- 172.21.15.144:0/721501981 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b79d0e8a0 con 0x559b799af8e0 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd19b400640 10 client.25322 handle_command_reply: tid=0 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 client.25322 shutdown 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 msgr2=0x559b79da2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79da2770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd1b406bfe0 tx=0x7fd1a0012b60 comp rx=0 tx=0).stop 2024-07-26T11:02:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 shutdown_connections 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b799af8e0 0x559b79db48d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79dae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79da2770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/721501981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79da2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd19b400640 0 client.25322 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 >> 172.21.15.144:0/721501981 conn(0x559b799e5350 msgr2=0x559b79d998f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 shutdown_connections 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/721501981 wait complete. 2024-07-26T11:02:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.020+0000 7fd1c2eb0740 20 client.25322 trim_cache size 0 max 16384 2024-07-26T11:02:09.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.039+0000 7fd1c2eb0740 1 Processor -- start 2024-07-26T11:02:09.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.039+0000 7fd1c2eb0740 1 -- start start 2024-07-26T11:02:09.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.039+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 0x559b79db2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.039+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.042 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.039+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799f0f60 0x559b79dae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.042 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x559b79db2c10 con 0x559b799f0f60 2024-07-26T11:02:09.042 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b79db32e0 con 0x559b799d0280 2024-07-26T11:02:09.042 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559b79db39b0 con 0x559b799c80a0 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 0x559b79db2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 0x559b79db2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35800/0 (socket says 172.21.15.144:35800) 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 -- 172.21.15.144:0/598725581 learned_addr learned my addr 172.21.15.144:0/598725581 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1b8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799f0f60 0x559b79dae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 -- 172.21.15.144:0/598725581 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 msgr2=0x559b79db3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 -- 172.21.15.144:0/598725581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799f0f60 msgr2=0x559b79dae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799f0f60 0x559b79dae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.040+0000 7fd1ba200640 1 -- 172.21.15.144:0/598725581 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b79d8f7d0 con 0x559b799c80a0 2024-07-26T11:02:09.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.041+0000 7fd1ba200640 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 0x559b79db2d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00a010 tx=0x7fd1ac003f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f054034575f94b3e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:09.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.041+0000 7fd1a6000640 1 -- 172.21.15.144:0/598725581 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac004970 con 0x559b799c80a0 2024-07-26T11:02:09.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.041+0000 7fd1a6000640 1 -- 172.21.15.144:0/598725581 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1ac005050 con 0x559b799c80a0 2024-07-26T11:02:09.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.041+0000 7fd1a6000640 1 -- 172.21.15.144:0/598725581 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac00f740 con 0x559b799c80a0 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/598725581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 msgr2=0x559b79db2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 0x559b79db2d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00a010 tx=0x7fd1ac003f50 comp rx=0 tx=0).stop 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/598725581 shutdown_connections 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799f0f60 0x559b79dae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/598725581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799c80a0 0x559b79db2d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/598725581 >> 172.21.15.144:0/598725581 conn(0x559b799e5350 msgr2=0x559b79d9d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/598725581 shutdown_connections 2024-07-26T11:02:09.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/598725581 wait complete. 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.042+0000 7fd1c2eb0740 1 -- start start 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 Processor -- start 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79dbd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79db6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x559b79db4750 con 0x559b799c80a0 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b79db4080 con 0x559b799d0280 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559b79db4e20 con 0x559b799f0f60 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1c2eb0740 10 client.0 _ll_register_callbacks cb 0x7fd1a0009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79dbd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.043+0000 7fd1b8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1ba200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79dbd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58834/0 (socket says 172.21.15.144:58834) 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1ba200640 1 -- 172.21.15.144:0/2633362391 learned_addr learned my addr 172.21.15.144:0/2633362391 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:09.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1ba200640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79db6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1b8e00640 1 -- 172.21.15.144:0/2633362391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 msgr2=0x559b79db6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79db6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1b8e00640 1 -- 172.21.15.144:0/2633362391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 msgr2=0x559b79dbd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79dbd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1b8e00640 1 -- 172.21.15.144:0/2633362391 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b79db6890 con 0x559b799d0280 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1ba200640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79dbd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1b8e00640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db5ad0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1a00059f0 tx=0x7fd1a0003960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3726e9d8db1eaff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:09.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd19b400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:02:09.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd19b400640 1 -- 172.21.15.144:0/2633362391 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a002d030 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd19b400640 1 -- 172.21.15.144:0/2633362391 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd1a0018040 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.044+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1ac005050 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd19b400640 1 -- 172.21.15.144:0/2633362391 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd1a0037050 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd1c2eb0740 10 client.25328 fetch_fsmap learned FSMap version 25 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559b79e1cb70 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd19b400640 1 -- 172.21.15.144:0/2633362391 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1ac005050 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd19b400640 1 -- 172.21.15.144:0/2633362391 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fd1a0005550 con 0x559b799d0280 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd1c2eb0740 10 client.25328 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd1c2eb0740 10 client.25328 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd1c2eb0740 20 client.25328 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.045+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b79984c90 0x559b79e204f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.046+0000 7fd1c2eb0740 4 client.25328 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:02:09.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.046+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "status"}) -- 0x559b79d0a340 con 0x559b79984c90 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.046+0000 7fd1ba200640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b79984c90 0x559b79e204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.046+0000 7fd1ba200640 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b79984c90 0x559b79e204f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a20daed5d9e420 server_cookie=acb7ae2ceb3c1e5d in_seq=0 out_seq=0 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.046+0000 7fd19b400640 10 client.25328 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd19b400640 1 -- 172.21.15.144:0/2633362391 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559b79d0a340 con 0x559b79984c90 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd19b400640 10 client.25328 handle_command_reply: tid=0 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 client.25328 shutdown 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 msgr2=0x559b79db5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db5ad0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1a00059f0 tx=0x7fd1a0003960 comp rx=0 tx=0).stop 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 shutdown_connections 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559b79984c90 0x559b79e204f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559b799f0f60 0x559b79db6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b799d0280 0x559b79db5ad0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd19b400640 0 client.25328 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:09.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 --2- 172.21.15.144:0/2633362391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559b799c80a0 0x559b79dbd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 >> 172.21.15.144:0/2633362391 conn(0x559b799e5350 msgr2=0x559b79d9d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 shutdown_connections 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 1 -- 172.21.15.144:0/2633362391 wait complete. 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.047+0000 7fd1c2eb0740 20 client.25328 trim_cache size 0 max 16384 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout: "cluster_fsid": "493f92c6-4b3c-11ef-bcb7-c7b262605968", 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout: "whoami": 0, 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout: "id": 24728, 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout: "want_state": "up:active", 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout: "state": "up:active", 2024-07-26T11:02:09.053 INFO:teuthology.orchestra.run.smithi144.stdout: "fs_name": "cephfs", 2024-07-26T11:02:09.054 INFO:teuthology.orchestra.run.smithi144.stdout: "rank_uptime": 283.91261833999999, 2024-07-26T11:02:09.054 INFO:teuthology.orchestra.run.smithi144.stdout: "mdsmap_epoch": 25, 2024-07-26T11:02:09.054 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch": 90, 2024-07-26T11:02:09.054 INFO:teuthology.orchestra.run.smithi144.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:02:09.054 INFO:teuthology.orchestra.run.smithi144.stdout: "uptime": 303.722430385 2024-07-26T11:02:09.054 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd19c0b3c20 msgr2=0x7fd19c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd19c0b3c20 0x7fd19c0b6130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x559b79d953e0 tx=0x7fd1a8031400 comp rx=0 tx=0).stop 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 msgr2=0x7fd1b41a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 0x7fd1b41a0320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8005ab0 tx=0x7fd1a8003c10 comp rx=0 tx=0).stop 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 shutdown_connections 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd19c0b3c20 0x7fd19c0b6130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b410fc80 0x7fd1b41a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1b410f330 0x7fd1b41a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 --2- 172.21.15.144:0/1418873945 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1b41099f0 0x7fd1b41a0320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 >> 172.21.15.144:0/1418873945 conn(0x7fd1b4078ff0 msgr2=0x7fd1b4107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 shutdown_connections 2024-07-26T11:02:09.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:09.048+0000 7fd1bac00640 1 -- 172.21.15.144:0/1418873945 wait complete. 2024-07-26T11:02:09.996 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T11:02:10.006 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T11:02:10.007 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T11:02:10.007 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T11:02:10.007 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-07-26T11:02:10.008 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:02:10.008 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T11:02:10.009 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-07-26T11:02:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:10 smithi144 ceph-mon[32129]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:10.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 -- 172.21.15.144:0/2092304369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010b7b0 msgr2=0x7f19e0111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:10.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 --2- 172.21.15.144:0/2092304369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010b7b0 0x7f19e0111fe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009be0 tx=0x7f19d802a6c0 comp rx=0 tx=0).stop 2024-07-26T11:02:10.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 -- 172.21.15.144:0/2092304369 shutdown_connections 2024-07-26T11:02:10.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 --2- 172.21.15.144:0/2092304369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010b7b0 0x7f19e0111fe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:10.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 --2- 172.21.15.144:0/2092304369 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e010ae60 0x7f19e010b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:10.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 --2- 172.21.15.144:0/2092304369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e0105530 0x7f19e0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:10.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 -- 172.21.15.144:0/2092304369 >> 172.21.15.144:0/2092304369 conn(0x7f19e0100f90 msgr2=0x7f19e01033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:10.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 -- 172.21.15.144:0/2092304369 shutdown_connections 2024-07-26T11:02:10.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.278+0000 7f19df200640 1 -- 172.21.15.144:0/2092304369 wait complete. 2024-07-26T11:02:10.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 Processor -- start 2024-07-26T11:02:10.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 -- start start 2024-07-26T11:02:10.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 0x7f19e019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010ae60 0x7f19e019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e010b7b0 0x7f19e01a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0115480 con 0x7f19e010b7b0 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0115300 con 0x7f19e010ae60 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.280+0000 7f19df200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0115600 con 0x7f19e0105530 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19de800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e010b7b0 0x7f19e01a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 0x7f19e019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010ae60 0x7f19e019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19de800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e010b7b0 0x7f19e01a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58878/0 (socket says 172.21.15.144:58878) 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 0x7f19e019bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:37566/0 (socket says 172.21.15.144:37566) 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19de800640 1 -- 172.21.15.144:0/3439205222 learned_addr learned my addr 172.21.15.144:0/3439205222 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:10.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19d7e00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010ae60 0x7f19e019c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36762/0 (socket says 172.21.15.144:36762) 2024-07-26T11:02:10.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 -- 172.21.15.144:0/3439205222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010ae60 msgr2=0x7f19e019c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:10.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010ae60 0x7f19e019c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:10.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 -- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e010b7b0 msgr2=0x7f19e01a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:10.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e010b7b0 0x7f19e01a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.281+0000 7f19dde00640 1 -- 172.21.15.144:0/3439205222 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e01a1e30 con 0x7f19e0105530 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.282+0000 7f19dde00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 0x7f19e019bf00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f19d000e9f0 tx=0x7f19d000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.282+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d000ccc0 con 0x7f19e0105530 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.282+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d00108e0 con 0x7f19e0105530 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.282+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0018840 con 0x7f19e0105530 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.282+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e01a2120 con 0x7f19e0105530 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.282+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e01a88b0 con 0x7f19e0105530 2024-07-26T11:02:10.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.283+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e0105940 con 0x7f19e0105530 2024-07-26T11:02:10.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.286+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f19d0010430 con 0x7f19e0105530 2024-07-26T11:02:10.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.287+0000 7f19d6a00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19c40b3ca0 0x7f19c40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:10.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.287+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f19d00e4380 con 0x7f19e0105530 2024-07-26T11:02:10.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.287+0000 7f19d7e00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19c40b3ca0 0x7f19c40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:10.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.288+0000 7f19d7e00640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19c40b3ca0 0x7f19c40b61b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f19e0104690 tx=0x7f19c8009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:10.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.289+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f19d00ab4e0 con 0x7f19e0105530 2024-07-26T11:02:10.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:10.548+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f19e002e0c0 con 0x7f19e0105530 2024-07-26T11:02:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:10 smithi160 ceph-mon[36715]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:10 smithi183 ceph-mon[37273]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:11.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.191+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f19d00a7070 con 0x7f19e0105530 2024-07-26T11:02:11.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.193+0000 7f19d6a00640 1 -- 172.21.15.144:0/3439205222 <== mon.1 v2:172.21.15.183:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 module 'snap_schedule' is already enabled v23) ==== 133+0+0 (secure 0 0 0) 0x7f19d00e5b20 con 0x7f19e0105530 2024-07-26T11:02:11.195 INFO:teuthology.orchestra.run.smithi144.stderr:module 'snap_schedule' is already enabled 2024-07-26T11:02:11.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19c40b3ca0 msgr2=0x7f19c40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19c40b3ca0 0x7f19c40b61b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f19e0104690 tx=0x7f19c8009260 comp rx=0 tx=0).stop 2024-07-26T11:02:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 msgr2=0x7f19e019bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 0x7f19e019bf00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f19d000e9f0 tx=0x7f19d000edf0 comp rx=0 tx=0).stop 2024-07-26T11:02:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 shutdown_connections 2024-07-26T11:02:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19c40b3ca0 0x7f19c40b61b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:11.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19e010b7b0 0x7f19e01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f19e010ae60 0x7f19e019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 --2- 172.21.15.144:0/3439205222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19e0105530 0x7f19e019bf00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 >> 172.21.15.144:0/3439205222 conn(0x7f19e0100f90 msgr2=0x7f19e010a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.197+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 shutdown_connections 2024-07-26T11:02:11.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:11.198+0000 7f19df200640 1 -- 172.21.15.144:0/3439205222 wait complete. 2024-07-26T11:02:11.231 INFO:teuthology.run_tasks:Running task kclient... 2024-07-26T11:02:11.244 INFO:tasks.kclient:Mounting kernel clients... 2024-07-26T11:02:11.245 INFO:tasks.kclient:config is {'client.0': {}} 2024-07-26T11:02:11.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:11.245 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2024-07-26T11:02:11.296 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-07-26T11:02:11.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:11.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:02:11.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:11.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2024-07-26T11:02:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:11 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3439205222' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:11 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:11 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3439205222' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:11 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:11 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3439205222' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:11 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:11.749 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-07-26T11:02:11.749 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T11:02:11.749 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:11.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T11:02:12.195 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:12 smithi144 ceph-mon[32129]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:12 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:02:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:12 smithi144 ceph-mon[32129]: mgrmap e23: x(active, since 12m), standbys: z, y 2024-07-26T11:02:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:12 smithi160 ceph-mon[36715]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:12.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:12 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:02:12.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:12 smithi160 ceph-mon[36715]: mgrmap e23: x(active, since 12m), standbys: z, y 2024-07-26T11:02:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:12 smithi183 ceph-mon[37273]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:12 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:02:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:12 smithi183 ceph-mon[37273]: mgrmap e23: x(active, since 12m), standbys: z, y 2024-07-26T11:02:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:13 smithi144 ceph-mon[32129]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:13 smithi160 ceph-mon[36715]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:13 smithi183 ceph-mon[37273]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:02:14.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 -- 172.21.15.144:0/145860666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 msgr2=0x7f454810b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 --2- 172.21.15.144:0/145860666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454810b530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4530009900 tx=0x7f453002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 -- 172.21.15.144:0/145860666 shutdown_connections 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 --2- 172.21.15.144:0/145860666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f454810ba70 0x7f45481122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 --2- 172.21.15.144:0/145860666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454810b530 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 --2- 172.21.15.144:0/145860666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45481057e0 0x7f4548105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 -- 172.21.15.144:0/145860666 >> 172.21.15.144:0/145860666 conn(0x7f45481011f0 msgr2=0x7f4548103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 -- 172.21.15.144:0/145860666 shutdown_connections 2024-07-26T11:02:14.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.461+0000 7f454d400640 1 -- 172.21.15.144:0/145860666 wait complete. 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.462+0000 7f454d400640 1 Processor -- start 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 -- start start 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 0x7f454819c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f454810ba70 0x7f45481a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4548115730 con 0x7f454810b120 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f45481155b0 con 0x7f454810ba70 2024-07-26T11:02:14.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.463+0000 7f454d400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f45481158b0 con 0x7f45481057e0 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 0x7f454819c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f453fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 0x7f454819c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:37584/0 (socket says 172.21.15.144:37584) 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 -- 172.21.15.144:0/3662840328 learned_addr learned my addr 172.21.15.144:0/3662840328 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4547200640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f454810ba70 0x7f45481a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 -- 172.21.15.144:0/3662840328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f454810ba70 msgr2=0x7f45481a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f454810ba70 0x7f45481a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 -- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 msgr2=0x7f454819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4546800640 1 -- 172.21.15.144:0/3662840328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45481a20d0 con 0x7f45481057e0 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f453fe00640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.464+0000 7f4547200640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f454810ba70 0x7f45481a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:14.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.465+0000 7f4546800640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 0x7f454819c160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f453800e9f0 tx=0x7f453800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:14.468 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.465+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453800ccc0 con 0x7f45481057e0 2024-07-26T11:02:14.468 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.465+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45481a23c0 con 0x7f45481057e0 2024-07-26T11:02:14.468 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.465+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45380108e0 con 0x7f45481057e0 2024-07-26T11:02:14.468 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.465+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538018880 con 0x7f45481057e0 2024-07-26T11:02:14.468 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.465+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45481a8b20 con 0x7f45481057e0 2024-07-26T11:02:14.468 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.466+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4548105bf0 con 0x7f45481057e0 2024-07-26T11:02:14.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.469+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f4538010430 con 0x7f45481057e0 2024-07-26T11:02:14.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.470+0000 7f453f400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f452c0b3c20 0x7f452c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:14.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.470+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f45380e4510 con 0x7f45481057e0 2024-07-26T11:02:14.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.470+0000 7f453fe00640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f452c0b3c20 0x7f452c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:14.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.471+0000 7f453fe00640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f452c0b3c20 0x7f452c0b6130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f453002a9e0 tx=0x7f4530005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:14.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.474+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f45380ab6d0 con 0x7f45481057e0 2024-07-26T11:02:14.713 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.711+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f454802e0c0 con 0x7f45481057e0 2024-07-26T11:02:14.714 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.712+0000 7f453f400640 1 -- 172.21.15.144:0/3662840328 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+88 (secure 0 0 0) 0x7f45380aafd0 con 0x7f45481057e0 2024-07-26T11:02:14.714 INFO:teuthology.orchestra.run.smithi144.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T11:02:14.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f452c0b3c20 msgr2=0x7f452c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:14.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f452c0b3c20 0x7f452c0b6130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f453002a9e0 tx=0x7f4530005a20 comp rx=0 tx=0).stop 2024-07-26T11:02:14.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 msgr2=0x7f454819c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:14.717 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 0x7f454819c160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f453800e9f0 tx=0x7f453800edf0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 shutdown_connections 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f452c0b3c20 0x7f452c0b6130 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f454810ba70 0x7f45481a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f454810b120 0x7f454819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 --2- 172.21.15.144:0/3662840328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45481057e0 0x7f454819c160 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 >> 172.21.15.144:0/3662840328 conn(0x7f45481011f0 msgr2=0x7f45481097c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.715+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 shutdown_connections 2024-07-26T11:02:14.718 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:14.716+0000 7f454d400640 1 -- 172.21.15.144:0/3662840328 wait complete. 2024-07-26T11:02:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:14 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3662840328' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:02:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:14 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3662840328' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:02:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:14 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3662840328' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi144.front.sepia.ceph.com 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T11:02:15.681 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T11:02:15.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:15.682 DEBUG:teuthology.orchestra.run.smithi144:> ip addr 2024-07-26T11:02:15.740 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T11:02:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T11:02:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T11:02:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:02:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:22 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f0 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:23 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f1 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5e brd ff:ff:ff:ff:ff:ff 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5f brd ff:ff:ff:ff:ff:ff 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: inet 172.21.15.144/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T11:02:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft 41723sec preferred_lft 41723sec 2024-07-26T11:02:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2024-07-26T11:02:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:02:15.743 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add name ceph-brx type bridge 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr flush dev ceph-brx 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set ceph-brx up 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T11:02:15.743 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T11:02:15.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:02:15.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:15 smithi144 ceph-mon[32129]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:02:15.977 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:15.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:15.986 DEBUG:teuthology.orchestra.run.smithi144:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T11:02:16.050 INFO:teuthology.orchestra.run.smithi144.stdout:1 2024-07-26T11:02:16.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:16.054 DEBUG:teuthology.orchestra.run.smithi144:> route 2024-07-26T11:02:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:15 smithi160 ceph-mon[36715]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:02:16.099 INFO:teuthology.orchestra.run.smithi144.stdout:Kernel IP routing table 2024-07-26T11:02:16.099 INFO:teuthology.orchestra.run.smithi144.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T11:02:16.099 INFO:teuthology.orchestra.run.smithi144.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T11:02:16.099 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T11:02:16.099 INFO:teuthology.orchestra.run.smithi144.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T11:02:16.100 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:02:16.100 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T11:02:16.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T11:02:16.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T11:02:16.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T11:02:16.100 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T11:02:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:15 smithi183 ceph-mon[37273]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-26T11:02:16.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:02:16.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:16.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:16.354 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2024-07-26T11:02:16.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:16.401 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list-id 2024-07-26T11:02:16.499 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:02:16.499 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T11:02:16.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T11:02:16.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T11:02:16.499 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T11:02:16.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:02:16.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:16.684 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T11:02:16.684 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:02:16.684 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T11:02:16.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T11:02:16.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T11:02:16.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T11:02:16.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T11:02:16.685 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T11:02:16.685 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T11:02:16.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:02:16.944 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:16.954 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:02:16.954 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2024-07-26T11:02:16.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set brx.0 up 2024-07-26T11:02:16.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T11:02:16.954 DEBUG:teuthology.orchestra.run.smithi144:> ') 2024-07-26T11:02:17.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:02:17.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:17.153 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T11:02:17.153 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:17.199 INFO:teuthology.orchestra.run.smithi144.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T11:02:17.199 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T11:02:17.200 DEBUG:teuthology.orchestra.run.smithi144:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:17.291 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-07-26T11:02:17.291 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-07-26T11:02:17.291 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/fe73673a-5e07-410e-a5c2-219a3360d93b 2024-07-26T11:02:17.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:17.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/fe73673a-5e07-410e-a5c2-219a3360d93b /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-07-26T11:02:17.482 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-07-26T11:02:17.483 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-07-26T11:02:18.250 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-07-26T11:02:18.250 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-07-26T11:02:18.250 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.144:3300,172.21.15.160:3300,172.21.15.183:3300" 2024-07-26T11:02:18.250 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: trying mount with new device syntax: 0@493f92c6-4b3c-11ef-bcb7-c7b262605968.cephfs=/volumes/qa/sv_1/fe73673a-5e07-410e-a5c2-219a3360d93b 2024-07-26T11:02:18.250 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.144:3300/172.21.15.160:3300/172.21.15.183:3300" will pass to kernel 2024-07-26T11:02:18.255 INFO:teuthology.orchestra.run.smithi144.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T11:02:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: You just mounted a file system that supports labels which does not 2024-07-26T11:02:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T11:02:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T11:02:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T11:02:18.260 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T11:02:18.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:02:18.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:18.349 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T11:02:18.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:02:18.462 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:18 smithi144 ceph-mon[32129]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:18.471 INFO:teuthology.orchestra.run.smithi144.stdout:client15201 2024-07-26T11:02:18.472 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T11:02:18.472 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T11:02:18.520 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T11:02:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:18 smithi160 ceph-mon[36715]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:18 smithi183 ceph-mon[37273]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:02:18.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:02:18.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:18.683 INFO:teuthology.orchestra.run.smithi144.stdout:client15201 2024-07-26T11:02:18.683 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T11:02:18.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:02:18.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:18.802 INFO:teuthology.orchestra.run.smithi144.stdout:client15201 2024-07-26T11:02:18.802 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968.client15201/status') 2024-07-26T11:02:18.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:02:18.936 INFO:teuthology.orchestra.run.smithi144.stdout:instance: client.15201 (3)192.168.0.1:0/3243981714 2024-07-26T11:02:18.936 INFO:teuthology.orchestra.run.smithi144.stdout:blocklisted: false 2024-07-26T11:02:18.937 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2024-07-26T11:02:18.937 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2024-07-26T11:02:18.937 INFO:teuthology.orchestra.run.smithi144.stderr:70 bytes copied, 7.4477e-05 s, 940 kB/s 2024-07-26T11:02:18.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:02:18.949 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T11:02:18.959 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:02:18.959 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T11:02:18.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-07-26T11:02:19.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 -- 172.21.15.144:0/3484246043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c10d790 msgr2=0x7f200c10fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:19.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 --2- 172.21.15.144:0/3484246043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c10d790 0x7f200c10fc80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2004009860 tx=0x7f200402a340 comp rx=0 tx=0).stop 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 -- 172.21.15.144:0/3484246043 shutdown_connections 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 --2- 172.21.15.144:0/3484246043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c10d790 0x7f200c10fc80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 --2- 172.21.15.144:0/3484246043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104c80 0x7f200c10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 --2- 172.21.15.144:0/3484246043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f200c104330 0x7f200c104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 -- 172.21.15.144:0/3484246043 >> 172.21.15.144:0/3484246043 conn(0x7f200c07a170 msgr2=0x7f200c07a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 -- 172.21.15.144:0/3484246043 shutdown_connections 2024-07-26T11:02:19.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.226+0000 7f2013000640 1 -- 172.21.15.144:0/3484246043 wait complete. 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.227+0000 7f2013000640 1 Processor -- start 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 -- start start 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 0x7f200c19c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c104c80 0x7f200c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f200c10d790 0x7f200c1a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f200c113400 con 0x7f200c10d790 2024-07-26T11:02:19.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f200c113280 con 0x7f200c104330 2024-07-26T11:02:19.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2013000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f200c113580 con 0x7f200c104c80 2024-07-26T11:02:19.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f200be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 0x7f200c19c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:19.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f200be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 0x7f200c19c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36794/0 (socket says 172.21.15.144:36794) 2024-07-26T11:02:19.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f200be00640 1 -- 172.21.15.144:0/418953571 learned_addr learned my addr 172.21.15.144:0/418953571 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.228+0000 7f2011000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f200c10d790 0x7f200c1a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f2003400640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c104c80 0x7f200c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f200be00640 1 -- 172.21.15.144:0/418953571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c104c80 msgr2=0x7f200c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f200be00640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c104c80 0x7f200c19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f200be00640 1 -- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f200c10d790 msgr2=0x7f200c1a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f200be00640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f200c10d790 0x7f200c1a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f200be00640 1 -- 172.21.15.144:0/418953571 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c1a2090 con 0x7f200c104330 2024-07-26T11:02:19.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f200be00640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 0x7f200c19c160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00eab0 tx=0x7f1ffc00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.229+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc00ce40 con 0x7f200c104330 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.230+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200c1a2380 con 0x7f200c104330 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.230+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ffc011920 con 0x7f200c104330 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.230+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc010a30 con 0x7f200c104330 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.230+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200c1a8b10 con 0x7f200c104330 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.231+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200c104740 con 0x7f200c104330 2024-07-26T11:02:19.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.233+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f1ffc010bd0 con 0x7f200c104330 2024-07-26T11:02:19.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.234+0000 7f2009600640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1ff00b3c20 0x7f1ff00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:19.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.235+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f1ffc0e4700 con 0x7f200c104330 2024-07-26T11:02:19.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.235+0000 7f2003400640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1ff00b3c20 0x7f1ff00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:19.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.235+0000 7f2003400640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1ff00b3c20 0x7f1ff00b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40096c0 tx=0x7f1ff4009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:19.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.239+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1ffc016030 con 0x7f200c104330 2024-07-26T11:02:19.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:19.490+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f200c1a8d40 con 0x7f200c104330 2024-07-26T11:02:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:19 smithi160 ceph-mon[36715]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-07-26T11:02:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:19 smithi183 ceph-mon[37273]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-07-26T11:02:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:19 smithi144 ceph-mon[32129]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-07-26T11:02:20.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.238+0000 7f2009600640 1 -- 172.21.15.144:0/418953571 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7f1ffc0ab860 con 0x7f200c104330 2024-07-26T11:02:20.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1ff00b3c20 msgr2=0x7f1ff00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1ff00b3c20 0x7f1ff00b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40096c0 tx=0x7f1ff4009260 comp rx=0 tx=0).stop 2024-07-26T11:02:20.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 msgr2=0x7f200c19c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 0x7f200c19c160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00eab0 tx=0x7f1ffc00ef70 comp rx=0 tx=0).stop 2024-07-26T11:02:20.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 shutdown_connections 2024-07-26T11:02:20.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1ff00b3c20 0x7f1ff00b6130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f200c10d790 0x7f200c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f200c104c80 0x7f200c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 --2- 172.21.15.144:0/418953571 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f200c104330 0x7f200c19c160 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 >> 172.21.15.144:0/418953571 conn(0x7f200c07a170 msgr2=0x7f200c0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:20.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.243+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 shutdown_connections 2024-07-26T11:02:20.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.244+0000 7f2013000640 1 -- 172.21.15.144:0/418953571 wait complete. 2024-07-26T11:02:20.273 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-26T11:02:20.284 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-07-26T11:02:20.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T11:02:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:20 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/418953571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:02:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:20 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 -- 172.21.15.144:0/1357469890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 msgr2=0x7fd4741163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 --2- 172.21.15.144:0/1357469890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 0x7fd4741163f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009ca0 tx=0x7fd46c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 -- 172.21.15.144:0/1357469890 shutdown_connections 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 --2- 172.21.15.144:0/1357469890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 0x7fd4741163f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 --2- 172.21.15.144:0/1357469890 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd47410c030 0x7fd47410c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 --2- 172.21.15.144:0/1357469890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd474109990 0x7fd474109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 -- 172.21.15.144:0/1357469890 >> 172.21.15.144:0/1357469890 conn(0x7fd4740790b0 msgr2=0x7fd474107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 -- 172.21.15.144:0/1357469890 shutdown_connections 2024-07-26T11:02:20.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.571+0000 7fd479a00640 1 -- 172.21.15.144:0/1357469890 wait complete. 2024-07-26T11:02:20.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.572+0000 7fd479a00640 1 Processor -- start 2024-07-26T11:02:20.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.572+0000 7fd479a00640 1 -- start start 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd474109990 0x7fd4741a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 0x7fd4741a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 0x7fd4741a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd474119880 con 0x7fd47410c030 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd474119700 con 0x7fd474109990 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd474119a00 con 0x7fd47410c980 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd46b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 0x7fd4741a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd473e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd474109990 0x7fd4741a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd46b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 0x7fd4741a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45626/0 (socket says 172.21.15.144:45626) 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd46b400640 1 -- 172.21.15.144:0/4228175870 learned_addr learned my addr 172.21.15.144:0/4228175870 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:20.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.573+0000 7fd479000640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 0x7fd4741a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:20.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd46b400640 1 -- 172.21.15.144:0/4228175870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 msgr2=0x7fd4741a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd46b400640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 0x7fd4741a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.576 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd46b400640 1 -- 172.21.15.144:0/4228175870 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd474109990 msgr2=0x7fd4741a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd46b400640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd474109990 0x7fd4741a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd46b400640 1 -- 172.21.15.144:0/4228175870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4741a60f0 con 0x7fd47410c030 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd473e00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd474109990 0x7fd4741a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd46b400640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 0x7fd4741a0760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00e9f0 tx=0x7fd45c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00ccc0 con 0x7fd47410c030 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd45c0054d0 con 0x7fd47410c030 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4741a63e0 con 0x7fd47410c030 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c010430 con 0x7fd47410c030 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.574+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4741acb70 con 0x7fd47410c030 2024-07-26T11:02:20.577 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.575+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd474109da0 con 0x7fd47410c030 2024-07-26T11:02:20.579 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.578+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fd45c0046c0 con 0x7fd47410c030 2024-07-26T11:02:20.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.579+0000 7fd471600640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd4580b3c20 0x7fd4580b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:20.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.579+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd45c0e4680 con 0x7fd47410c030 2024-07-26T11:02:20.581 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.580+0000 7fd473e00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd4580b3c20 0x7fd4580b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:20.582 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.580+0000 7fd473e00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd4580b3c20 0x7fd4580b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd464009fc0 tx=0x7fd464005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:20.583 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.581+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd45c055600 con 0x7fd47410c030 2024-07-26T11:02:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:20 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/418953571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:02:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:20 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:02:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:20 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/418953571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:02:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:20 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:02:20.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.791+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4741a12e0 con 0x7fd47410c030 2024-07-26T11:02:20.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.795+0000 7fd471600640 1 -- 172.21.15.144:0/4228175870 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7fd45c0ab7e0 con 0x7fd47410c030 2024-07-26T11:02:20.798 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:20.798 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:02:20.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd4580b3c20 msgr2=0x7fd4580b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd4580b3c20 0x7fd4580b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd464009fc0 tx=0x7fd464005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:20.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 msgr2=0x7fd4741a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:20.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 0x7fd4741a0760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00e9f0 tx=0x7fd45c00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 shutdown_connections 2024-07-26T11:02:20.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd4580b3c20 0x7fd4580b6130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47410c980 0x7fd4741a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47410c030 0x7fd4741a0760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 --2- 172.21.15.144:0/4228175870 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd474109990 0x7fd4741a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:20.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 >> 172.21.15.144:0/4228175870 conn(0x7fd4740790b0 msgr2=0x7fd47410d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:20.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.798+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 shutdown_connections 2024-07-26T11:02:20.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:20.799+0000 7fd479a00640 1 -- 172.21.15.144:0/4228175870 wait complete. 2024-07-26T11:02:20.827 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T10:52:04.947795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-07-26T10:57:16.258941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-07-26T10:57:21.007430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-07-26T10:57:33.064159+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-07-26T11:02:20.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T11:02:21.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 -- 172.21.15.144:0/2830281567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1010c030 msgr2=0x7f0e1010c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 --2- 172.21.15.144:0/2830281567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1010c030 0x7f0e1010c440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009900 tx=0x7f0dfc02a410 comp rx=0 tx=0).stop 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 -- 172.21.15.144:0/2830281567 shutdown_connections 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 --2- 172.21.15.144:0/2830281567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 0x7f0e101163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 --2- 172.21.15.144:0/2830281567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1010c030 0x7f0e1010c440 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 --2- 172.21.15.144:0/2830281567 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e10109990 0x7f0e10109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 -- 172.21.15.144:0/2830281567 >> 172.21.15.144:0/2830281567 conn(0x7f0e100790b0 msgr2=0x7f0e10107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 -- 172.21.15.144:0/2830281567 shutdown_connections 2024-07-26T11:02:21.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.091+0000 7f0e16800640 1 -- 172.21.15.144:0/2830281567 wait complete. 2024-07-26T11:02:21.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 Processor -- start 2024-07-26T11:02:21.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 -- start start 2024-07-26T11:02:21.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 0x7f0e101a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e1010c030 0x7f0e101a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 0x7f0e101a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10119880 con 0x7f0e1010c980 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10119700 con 0x7f0e1010c030 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.093+0000 7f0e16800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10119a00 con 0x7f0e10109990 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 0x7f0e101a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 0x7f0e101a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e14a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e1010c030 0x7f0e101a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 0x7f0e101a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:52456/0 (socket says 172.21.15.144:52456) 2024-07-26T11:02:21.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 -- 172.21.15.144:0/3902316634 learned_addr learned my addr 172.21.15.144:0/3902316634 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 -- 172.21.15.144:0/3902316634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e1010c030 msgr2=0x7f0e101a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e1010c030 0x7f0e101a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 -- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 msgr2=0x7f0e101a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 0x7f0e101a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 -- 172.21.15.144:0/3902316634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e101a6170 con 0x7f0e10109990 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15e00640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 0x7f0e101a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.094+0000 7f0e15400640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 0x7f0e101a0240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400ea60 tx=0x7f0e0400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:21.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.095+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400cde0 con 0x7f0e10109990 2024-07-26T11:02:21.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.095+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e101a6460 con 0x7f0e10109990 2024-07-26T11:02:21.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.095+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e040035f0 con 0x7f0e10109990 2024-07-26T11:02:21.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.095+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e04010690 con 0x7f0e10109990 2024-07-26T11:02:21.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.095+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e101acbf0 con 0x7f0e10109990 2024-07-26T11:02:21.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.096+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e10109da0 con 0x7f0e10109990 2024-07-26T11:02:21.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.098+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f0e040037a0 con 0x7f0e10109990 2024-07-26T11:02:21.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.100+0000 7f0e0e000640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0df80b3c20 0x7f0df80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.100+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f0e040e4610 con 0x7f0e10109990 2024-07-26T11:02:21.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.100+0000 7f0e14a00640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0df80b3c20 0x7f0df80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.101+0000 7f0e14a00640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0df80b3c20 0x7f0df80b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000c00 tx=0x7f0dfc031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:21.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.101+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0e04055600 con 0x7f0e10109990 2024-07-26T11:02:21.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.316+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0e1002e0c0 con 0x7f0e10109990 2024-07-26T11:02:21.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.317+0000 7f0e0e000640 1 -- 172.21.15.144:0/3902316634 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f0e040ab770 con 0x7f0e10109990 2024-07-26T11:02:21.319 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 1 2024-07-26T11:02:21.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0df80b3c20 msgr2=0x7f0df80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0df80b3c20 0x7f0df80b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000c00 tx=0x7f0dfc031400 comp rx=0 tx=0).stop 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 msgr2=0x7f0e101a0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 0x7f0e101a0240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400ea60 tx=0x7f0e0400ef20 comp rx=0 tx=0).stop 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 shutdown_connections 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f0df80b3c20 0x7f0df80b6130 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e1010c980 0x7f0e101a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e1010c030 0x7f0e101a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 --2- 172.21.15.144:0/3902316634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e10109990 0x7f0e101a0240 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 >> 172.21.15.144:0/3902316634 conn(0x7f0e100790b0 msgr2=0x7f0e1010d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 shutdown_connections 2024-07-26T11:02:21.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.320+0000 7f0e16800640 1 -- 172.21.15.144:0/3902316634 wait complete. 2024-07-26T11:02:21.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-07-26T11:02:21.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:21 smithi144 ceph-mon[32129]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T11:02:21.554 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:21 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:02:21.554 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:21 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/4228175870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:02:21.554 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:21 smithi144 ceph-mon[32129]: pgmap v457: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-07-26T11:02:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:21 smithi160 ceph-mon[36715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T11:02:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:21 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:02:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:21 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/4228175870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:02:21.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:21 smithi160 ceph-mon[36715]: pgmap v457: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 -- 172.21.15.144:0/1319938070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a40109990 msgr2=0x7f1a40109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1319938070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a40109990 0x7f1a40109da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009860 tx=0x7f1a3002a340 comp rx=0 tx=0).stop 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 -- 172.21.15.144:0/1319938070 shutdown_connections 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1319938070 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1a4010c980 0x7f1a401163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1319938070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a4010c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1319938070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a40109990 0x7f1a40109da0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 -- 172.21.15.144:0/1319938070 >> 172.21.15.144:0/1319938070 conn(0x7f1a400790b0 msgr2=0x7f1a40107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:21.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 -- 172.21.15.144:0/1319938070 shutdown_connections 2024-07-26T11:02:21.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.624+0000 7f1a3f200640 1 -- 172.21.15.144:0/1319938070 wait complete. 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.625+0000 7f1a3f200640 1 Processor -- start 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.625+0000 7f1a3f200640 1 -- start start 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1a40109990 0x7f1a401a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a401a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4010c980 0x7f1a401a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40119880 con 0x7f1a4010c030 2024-07-26T11:02:21.628 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40119700 con 0x7f1a40109990 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40119a00 con 0x7f1a4010c980 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a401a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a401a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45666/0 (socket says 172.21.15.144:45666) 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4010c980 0x7f1a401a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 -- 172.21.15.144:0/1959465038 learned_addr learned my addr 172.21.15.144:0/1959465038 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3de00640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1a40109990 0x7f1a401a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 -- 172.21.15.144:0/1959465038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4010c980 msgr2=0x7f1a401a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4010c980 0x7f1a401a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 -- 172.21.15.144:0/1959465038 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1a40109990 msgr2=0x7f1a401a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1a40109990 0x7f1a401a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3d400640 1 -- 172.21.15.144:0/1959465038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a401a6110 con 0x7f1a4010c030 2024-07-26T11:02:21.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.626+0000 7f1a3e800640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4010c980 0x7f1a401a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:21.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.627+0000 7f1a3d400640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a401a0780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00e9a0 tx=0x7f1a2c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:21.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.627+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00cd80 con 0x7f1a4010c030 2024-07-26T11:02:21.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.627+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a401a6400 con 0x7f1a4010c030 2024-07-26T11:02:21.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.627+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a2c0035f0 con 0x7f1a4010c030 2024-07-26T11:02:21.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.627+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c010690 con 0x7f1a4010c030 2024-07-26T11:02:21.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.627+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a401acb90 con 0x7f1a4010c030 2024-07-26T11:02:21.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.629+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a40109da0 con 0x7f1a4010c030 2024-07-26T11:02:21.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.631+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f1a2c003fb0 con 0x7f1a4010c030 2024-07-26T11:02:21.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.632+0000 7f1a36a00640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1a240b3c20 0x7f1a240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:21.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.633+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f1a2c0e4480 con 0x7f1a4010c030 2024-07-26T11:02:21.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.633+0000 7f1a3de00640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1a240b3c20 0x7f1a240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:21 smithi183 ceph-mon[37273]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T11:02:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:21 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:02:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:21 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/4228175870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:02:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:21 smithi183 ceph-mon[37273]: pgmap v457: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-07-26T11:02:21.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.634+0000 7f1a3de00640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1a240b3c20 0x7f1a240b6130 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a3002a930 tx=0x7f1a300058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:21.636 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.634+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1a2c0ab670 con 0x7f1a4010c030 2024-07-26T11:02:21.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.844+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a4002e0c0 con 0x7f1a4010c030 2024-07-26T11:02:21.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.845+0000 7f1a36a00640 1 -- 172.21.15.144:0/1959465038 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f1a2c0aaf70 con 0x7f1a4010c030 2024-07-26T11:02:21.847 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 32 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1a240b3c20 msgr2=0x7f1a240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1a240b3c20 0x7f1a240b6130 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a3002a930 tx=0x7f1a300058d0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 msgr2=0x7f1a401a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a401a0780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00e9a0 tx=0x7f1a2c00ee60 comp rx=0 tx=0).stop 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 shutdown_connections 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1a240b3c20 0x7f1a240b6130 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4010c980 0x7f1a401a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a4010c030 0x7f1a401a0780 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 --2- 172.21.15.144:0/1959465038 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1a40109990 0x7f1a401a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:21.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 >> 172.21.15.144:0/1959465038 conn(0x7f1a400790b0 msgr2=0x7f1a401080d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:21.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 shutdown_connections 2024-07-26T11:02:21.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:21.848+0000 7f1a3f200640 1 -- 172.21.15.144:0/1959465038 wait complete. 2024-07-26T11:02:21.882 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 -- 172.21.15.144:0/3521462243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810b7b0 msgr2=0x7fdce8111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 --2- 172.21.15.144:0/3521462243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810b7b0 0x7fdce8111fe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009ca0 tx=0x7fdce002a8b0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 -- 172.21.15.144:0/3521462243 shutdown_connections 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 --2- 172.21.15.144:0/3521462243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810b7b0 0x7fdce8111fe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 --2- 172.21.15.144:0/3521462243 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce810ae60 0x7fdce810b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 --2- 172.21.15.144:0/3521462243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce8105530 0x7fdce8105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 -- 172.21.15.144:0/3521462243 >> 172.21.15.144:0/3521462243 conn(0x7fdce8100f90 msgr2=0x7fdce81033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 -- 172.21.15.144:0/3521462243 shutdown_connections 2024-07-26T11:02:22.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.155+0000 7fdced800640 1 -- 172.21.15.144:0/3521462243 wait complete. 2024-07-26T11:02:22.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.156+0000 7fdced800640 1 Processor -- start 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.156+0000 7fdced800640 1 -- start start 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdced800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 0x7fdce819beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdced800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810ae60 0x7fdce819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdced800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce810b7b0 0x7fdce81a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdced800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81154b0 con 0x7fdce810ae60 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdced800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8115330 con 0x7fdce8105530 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdced800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8115630 con 0x7fdce810b7b0 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 0x7fdce819beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 0x7fdce819beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47142/0 (socket says 172.21.15.144:47142) 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 -- 172.21.15.144:0/1401008731 learned_addr learned my addr 172.21.15.144:0/1401008731 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdcece00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce810b7b0 0x7fdce81a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdcdfe00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810ae60 0x7fdce819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 -- 172.21.15.144:0/1401008731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce810b7b0 msgr2=0x7fdce81a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce810b7b0 0x7fdce81a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 -- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810ae60 msgr2=0x7fdce819c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810ae60 0x7fdce819c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdce7e00640 1 -- 172.21.15.144:0/1401008731 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce81a1de0 con 0x7fdce8105530 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.157+0000 7fdcece00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce810b7b0 0x7fdce81a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.158+0000 7fdce7e00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 0x7fdce819beb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800ea60 tx=0x7fdcd800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.158+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd800cde0 con 0x7fdce8105530 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.158+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce81a20d0 con 0x7fdce8105530 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.158+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcd80035f0 con 0x7fdce8105530 2024-07-26T11:02:22.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.158+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8010690 con 0x7fdce8105530 2024-07-26T11:02:22.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.158+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce81a8860 con 0x7fdce8105530 2024-07-26T11:02:22.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.159+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce8105940 con 0x7fdce8105530 2024-07-26T11:02:22.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.162+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fdcd80037a0 con 0x7fdce8105530 2024-07-26T11:02:22.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.163+0000 7fdce5600640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdccc0b3ca0 0x7fdccc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.164+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fdcd80e4590 con 0x7fdce8105530 2024-07-26T11:02:22.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.164+0000 7fdcdfe00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdccc0b3ca0 0x7fdccc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.165+0000 7fdcdfe00640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdccc0b3ca0 0x7fdccc0b61b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdce8104690 tx=0x7fdcd0005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:22.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.165+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdcd8055600 con 0x7fdce8105530 2024-07-26T11:02:22.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.372+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdce802e0c0 con 0x7fdce8105530 2024-07-26T11:02:22.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.373+0000 7fdce5600640 1 -- 172.21.15.144:0/1401008731 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fdcd80ab6f0 con 0x7fdce8105530 2024-07-26T11:02:22.375 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 64 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdccc0b3ca0 msgr2=0x7fdccc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdccc0b3ca0 0x7fdccc0b61b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdce8104690 tx=0x7fdcd0005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 msgr2=0x7fdce819beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 0x7fdce819beb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800ea60 tx=0x7fdcd800ef20 comp rx=0 tx=0).stop 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 shutdown_connections 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fdccc0b3ca0 0x7fdccc0b61b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdce810b7b0 0x7fdce81a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce810ae60 0x7fdce819c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 --2- 172.21.15.144:0/1401008731 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdce8105530 0x7fdce819beb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 >> 172.21.15.144:0/1401008731 conn(0x7fdce8100f90 msgr2=0x7fdce810a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:22.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 shutdown_connections 2024-07-26T11:02:22.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.376+0000 7fdced800640 1 -- 172.21.15.144:0/1401008731 wait complete. 2024-07-26T11:02:22.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-07-26T11:02:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:22 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3902316634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:02:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:22 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1959465038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T11:02:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:22 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3902316634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:02:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:22 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1959465038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T11:02:22.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 -- 172.21.15.144:0/1062852453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d010ae60 msgr2=0x7f57d010b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 --2- 172.21.15.144:0/1062852453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d010ae60 0x7f57d010b270 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009900 tx=0x7f57b802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 -- 172.21.15.144:0/1062852453 shutdown_connections 2024-07-26T11:02:22.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 --2- 172.21.15.144:0/1062852453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57d010b7b0 0x7f57d0111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 --2- 172.21.15.144:0/1062852453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d010ae60 0x7f57d010b270 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 --2- 172.21.15.144:0/1062852453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0105530 0x7f57d0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 -- 172.21.15.144:0/1062852453 >> 172.21.15.144:0/1062852453 conn(0x7f57d0100f90 msgr2=0x7f57d01033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:22.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 -- 172.21.15.144:0/1062852453 shutdown_connections 2024-07-26T11:02:22.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.668+0000 7f57cfe00640 1 -- 172.21.15.144:0/1062852453 wait complete. 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.669+0000 7f57cfe00640 1 Processor -- start 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.669+0000 7f57cfe00640 1 -- start start 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d0105530 0x7f57d019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57d010ae60 0x7f57d019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cfe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 0x7f57d01a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cfe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f57d0115440 con 0x7f57d010b7b0 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cfe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f57d01152c0 con 0x7f57d0105530 2024-07-26T11:02:22.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cfe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f57d01155c0 con 0x7f57d010ae60 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cf400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 0x7f57d01a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cf400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 0x7f57d01a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45712/0 (socket says 172.21.15.144:45712) 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cf400640 1 -- 172.21.15.144:0/3995965802 learned_addr learned my addr 172.21.15.144:0/3995965802 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57c6000640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57d010ae60 0x7f57d019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cea00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d0105530 0x7f57d019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cf400640 1 -- 172.21.15.144:0/3995965802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57d010ae60 msgr2=0x7f57d019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cf400640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57d010ae60 0x7f57d019c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.670+0000 7f57cf400640 1 -- 172.21.15.144:0/3995965802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d0105530 msgr2=0x7f57d019be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57cf400640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d0105530 0x7f57d019be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57cf400640 1 -- 172.21.15.144:0/3995965802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d01a1db0 con 0x7f57d010b7b0 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57cea00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d0105530 0x7f57d019be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57cf400640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 0x7f57d01a16b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57c800eb10 tx=0x7f57c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:22.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8004b20 con 0x7f57d010b7b0 2024-07-26T11:02:22.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57c8004cc0 con 0x7f57d010b7b0 2024-07-26T11:02:22.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d01a20a0 con 0x7f57d010b7b0 2024-07-26T11:02:22.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8010690 con 0x7f57d010b7b0 2024-07-26T11:02:22.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.671+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d01a8830 con 0x7f57d010b7b0 2024-07-26T11:02:22.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.672+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d0105940 con 0x7f57d010b7b0 2024-07-26T11:02:22.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.675+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f57c8005000 con 0x7f57d010b7b0 2024-07-26T11:02:22.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.676+0000 7f57c7e00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f57b40b3c20 0x7f57b40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:22.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.677+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f57c80e48d0 con 0x7f57d010b7b0 2024-07-26T11:02:22.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.677+0000 7f57cea00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f57b40b3c20 0x7f57b40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.678+0000 7f57cea00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f57b40b3c20 0x7f57b40b6130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57c00095f0 tx=0x7f57c0005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:22.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.678+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f57c8055600 con 0x7f57d010b7b0 2024-07-26T11:02:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:22 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3902316634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:02:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:22 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1959465038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T11:02:22.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.887+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f57d002e0c0 con 0x7f57d010b7b0 2024-07-26T11:02:22.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.888+0000 7f57c7e00640 1 -- 172.21.15.144:0/3995965802 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f57c80aba30 con 0x7f57d010b7b0 2024-07-26T11:02:22.890 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 32 2024-07-26T11:02:22.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f57b40b3c20 msgr2=0x7f57b40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f57b40b3c20 0x7f57b40b6130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57c00095f0 tx=0x7f57c0005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 msgr2=0x7f57d01a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 0x7f57d01a16b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57c800eb10 tx=0x7f57c800c2d0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 shutdown_connections 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f57b40b3c20 0x7f57b40b6130 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d010b7b0 0x7f57d01a16b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57d010ae60 0x7f57d019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 --2- 172.21.15.144:0/3995965802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f57d0105530 0x7f57d019be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 >> 172.21.15.144:0/3995965802 conn(0x7f57d0100f90 msgr2=0x7f57d01033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 shutdown_connections 2024-07-26T11:02:22.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:22.891+0000 7f57cfe00640 1 -- 172.21.15.144:0/3995965802 wait complete. 2024-07-26T11:02:22.920 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-07-26T11:02:22.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:02:23.372 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:23.518 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:23 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1401008731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T11:02:23.518 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:23 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3995965802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T11:02:23.519 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:23 smithi144 ceph-mon[32129]: pgmap v458: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-07-26T11:02:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:23 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1401008731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T11:02:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:23 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3995965802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T11:02:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:23 smithi160 ceph-mon[36715]: pgmap v458: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-07-26T11:02:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:23 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1401008731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T11:02:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:23 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3995965802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T11:02:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:23 smithi183 ceph-mon[37273]: pgmap v458: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-07-26T11:02:25.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 -- 172.21.15.144:0/1613503554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b810f940 msgr2=0x7fc7b8111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:25.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 --2- 172.21.15.144:0/1613503554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b810f940 0x7fc7b8111d80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009900 tx=0x7fc7b002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:25.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 -- 172.21.15.144:0/1613503554 shutdown_connections 2024-07-26T11:02:25.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 --2- 172.21.15.144:0/1613503554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b810f940 0x7fc7b8111d80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:25.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 --2- 172.21.15.144:0/1613503554 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc7b8075b30 0x7fc7b810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:25.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 --2- 172.21.15.144:0/1613503554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b80755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:25.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 -- 172.21.15.144:0/1613503554 >> 172.21.15.144:0/1613503554 conn(0x7fc7b80fee70 msgr2=0x7fc7b81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:25.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 -- 172.21.15.144:0/1613503554 shutdown_connections 2024-07-26T11:02:25.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.795+0000 7fc7bd200640 1 -- 172.21.15.144:0/1613503554 wait complete. 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.796+0000 7fc7bd200640 1 Processor -- start 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.796+0000 7fc7bd200640 1 -- start start 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7bd200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7bd200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b8075b30 0x7fc7b819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7bd200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc7b810f940 0x7fc7b81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7bd200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81154f0 con 0x7fc7b80751e0 2024-07-26T11:02:25.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7bd200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b8115370 con 0x7fc7b810f940 2024-07-26T11:02:25.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7bd200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b8115670 con 0x7fc7b8075b30 2024-07-26T11:02:25.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:25.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45730/0 (socket says 172.21.15.144:45730) 2024-07-26T11:02:25.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7b7400640 1 -- 172.21.15.144:0/2048189007 learned_addr learned my addr 172.21.15.144:0/2048189007 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:25.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7b6a00640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b8075b30 0x7fc7b819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:25.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.797+0000 7fc7b7e00640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc7b810f940 0x7fc7b81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7b7400640 1 -- 172.21.15.144:0/2048189007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b8075b30 msgr2=0x7fc7b819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7b7400640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b8075b30 0x7fc7b819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7b7400640 1 -- 172.21.15.144:0/2048189007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc7b810f940 msgr2=0x7fc7b81a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7b7400640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc7b810f940 0x7fc7b81a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7b7400640 1 -- 172.21.15.144:0/2048189007 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b81a1eb0 con 0x7fc7b80751e0 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7b7400640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b819bf40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800e9f0 tx=0x7fc7a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800cd80 con 0x7fc7b80751e0 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a80108e0 con 0x7fc7b80751e0 2024-07-26T11:02:25.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.798+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b81a21a0 con 0x7fc7b80751e0 2024-07-26T11:02:25.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.799+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8018980 con 0x7fc7b80751e0 2024-07-26T11:02:25.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.799+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b81a8900 con 0x7fc7b80751e0 2024-07-26T11:02:25.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.800+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b80755f0 con 0x7fc7b80751e0 2024-07-26T11:02:25.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.802+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fc7a8010430 con 0x7fc7b80751e0 2024-07-26T11:02:25.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.804+0000 7fc7afe00640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7980b3c20 0x7fc7980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:25.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.804+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc7a80e4920 con 0x7fc7b80751e0 2024-07-26T11:02:25.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.804+0000 7fc7b6a00640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7980b3c20 0x7fc7980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:25.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.805+0000 7fc7b6a00640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7980b3c20 0x7fc7980b6130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8101650 tx=0x7fc7a0009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:25.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:25.806+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc7a80ab430 con 0x7fc7b80751e0 2024-07-26T11:02:26.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.045+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7b81a8c70 con 0x7fc7b80751e0 2024-07-26T11:02:26.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.046+0000 7fc7afe00640 1 -- 172.21.15.144:0/2048189007 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11931 (secure 0 0 0) 0x7fc7a80af980 con 0x7fc7b80751e0 2024-07-26T11:02:26.048 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:26.049 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":26,"btime":"2024-07-26T11:02:20:221771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T11:02:19.492590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24703,24715,24728]},"id":1}]} 2024-07-26T11:02:26.049 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 26 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.049+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7980b3c20 msgr2=0x7fc7980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.049+0000 7fc7bd200640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7980b3c20 0x7fc7980b6130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8101650 tx=0x7fc7a0009500 comp rx=0 tx=0).stop 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.049+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 msgr2=0x7fc7b819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.049+0000 7fc7bd200640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b819bf40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800e9f0 tx=0x7fc7a800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 shutdown_connections 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc7980b3c20 0x7fc7980b6130 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc7b810f940 0x7fc7b81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7b8075b30 0x7fc7b819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 --2- 172.21.15.144:0/2048189007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b80751e0 0x7fc7b819bf40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:26.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 >> 172.21.15.144:0/2048189007 conn(0x7fc7b80fee70 msgr2=0x7fc7b81094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:26.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 shutdown_connections 2024-07-26T11:02:26.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:26.050+0000 7fc7bd200640 1 -- 172.21.15.144:0/2048189007 wait complete. 2024-07-26T11:02:26.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:26 smithi144 ceph-mon[32129]: pgmap v459: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-07-26T11:02:26.462 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:26 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2048189007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:02:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:26 smithi160 ceph-mon[36715]: pgmap v459: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-07-26T11:02:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:26 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2048189007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:02:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:26 smithi183 ceph-mon[37273]: pgmap v459: 129 pgs: 129 active+clean; 648 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-07-26T11:02:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:26 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2048189007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:02:26.982 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-07-26T11:02:26.982 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-07-26T11:02:26.982 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T11:02:27.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 -- 172.21.15.144:0/1293248047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 msgr2=0x7fd61c0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 --2- 172.21.15.144:0/1293248047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c0755f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009900 tx=0x7fd60c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 -- 172.21.15.144:0/1293248047 shutdown_connections 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 --2- 172.21.15.144:0/1293248047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd61c10f870 0x7fd61c111cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 --2- 172.21.15.144:0/1293248047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 --2- 172.21.15.144:0/1293248047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c0755f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 -- 172.21.15.144:0/1293248047 >> 172.21.15.144:0/1293248047 conn(0x7fd61c0fed80 msgr2=0x7fd61c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 -- 172.21.15.144:0/1293248047 shutdown_connections 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.263+0000 7fd622800640 1 -- 172.21.15.144:0/1293248047 wait complete. 2024-07-26T11:02:27.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.265+0000 7fd622800640 1 Processor -- start 2024-07-26T11:02:27.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.265+0000 7fd622800640 1 -- start start 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd622800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd622800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd622800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd61c10f870 0x7fd61c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd622800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c115480 con 0x7fd61c0751e0 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd622800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c115300 con 0x7fd61c10f870 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd622800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c115600 con 0x7fd61c075b30 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd621400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:52514/0 (socket says 172.21.15.144:52514) 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd621400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c19be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45756/0 (socket says 172.21.15.144:45756) 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 -- 172.21.15.144:0/3646213661 learned_addr learned my addr 172.21.15.144:0/3646213661 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd621e00640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd61c10f870 0x7fd61c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 -- 172.21.15.144:0/3646213661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd61c10f870 msgr2=0x7fd61c1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd61c10f870 0x7fd61c1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 -- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 msgr2=0x7fd61c19be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c19be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.266+0000 7fd620a00640 1 -- 172.21.15.144:0/3646213661 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61c1a1db0 con 0x7fd61c075b30 2024-07-26T11:02:27.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.267+0000 7fd620a00640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c19c420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e9c0 tx=0x7fd60800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:27.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.267+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60800ccc0 con 0x7fd61c075b30 2024-07-26T11:02:27.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.267+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6080054d0 con 0x7fd61c075b30 2024-07-26T11:02:27.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.267+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61c1a20a0 con 0x7fd61c075b30 2024-07-26T11:02:27.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.267+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd608010430 con 0x7fd61c075b30 2024-07-26T11:02:27.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.267+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd61c1a8830 con 0x7fd61c075b30 2024-07-26T11:02:27.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.269+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61c0755f0 con 0x7fd61c075b30 2024-07-26T11:02:27.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.271+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fd608005020 con 0x7fd61c075b30 2024-07-26T11:02:27.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.272+0000 7fd61a000640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd6000b3ca0 0x7fd6000b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.272+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd6080e4410 con 0x7fd61c075b30 2024-07-26T11:02:27.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.273+0000 7fd621400640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd6000b3ca0 0x7fd6000b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.273+0000 7fd621400640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd6000b3ca0 0x7fd6000b61b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd60c031440 tx=0x7fd60c009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:27.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.277+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd6080ab570 con 0x7fd61c075b30 2024-07-26T11:02:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:27 smithi144 ceph-mon[32129]: pgmap v460: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-07-26T11:02:27.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.481+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd61c109a90 con 0x7fd6000b3ca0 2024-07-26T11:02:27.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.501+0000 7fd61a000640 1 -- 172.21.15.144:0/3646213661 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340481 (secure 0 0 0) 0x7fd61c109a90 con 0x7fd6000b3ca0 2024-07-26T11:02:27.508 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:27.511 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T11:02:27.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd6000b3ca0 msgr2=0x7fd6000b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd6000b3ca0 0x7fd6000b61b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd60c031440 tx=0x7fd60c009560 comp rx=0 tx=0).stop 2024-07-26T11:02:27.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 msgr2=0x7fd61c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c19c420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e9c0 tx=0x7fd60800ee80 comp rx=0 tx=0).stop 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 shutdown_connections 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd6000b3ca0 0x7fd6000b61b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd61c10f870 0x7fd61c1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd61c075b30 0x7fd61c19c420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 --2- 172.21.15.144:0/3646213661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61c0751e0 0x7fd61c19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 >> 172.21.15.144:0/3646213661 conn(0x7fd61c0fed80 msgr2=0x7fd61c109380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 shutdown_connections 2024-07-26T11:02:27.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.509+0000 7fd622800640 1 -- 172.21.15.144:0/3646213661 wait complete. 2024-07-26T11:02:27.541 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-07-26T11:02:27.115339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667598,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1452,"num_read_kb":835,"num_write":311,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":752,"ondisk_log_size":752,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380104,"kb_used_data":14312,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561720,"statfs":{"total":1151940427776,"available":1151551201280,"internally_reserved":0,"allocated":14655488,"data_stored":8400752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":47,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005738"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:25.593785+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:59:25.593785+0000","last_peered":"2024-07-26T10:59:25.593785+0000","last_clean":"2024-07-26T10:59:25.593785+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:59:25.593785+0000","last_undegraded":"2024-07-26T10:59:25.593785+0000","last_fullsized":"2024-07-26T10:59:25.593785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969791+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T11:00:12.969791+0000","last_peered":"2024-07-26T11:00:12.969791+0000","last_clean":"2024-07-26T11:00:12.969791+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T11:00:12.969791+0000","last_undegraded":"2024-07-26T11:00:12.969791+0000","last_fullsized":"2024-07-26T11:00:12.969791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:59.988270+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:59:59.988270+0000","last_peered":"2024-07-26T10:59:59.988270+0000","last_clean":"2024-07-26T10:59:59.988270+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:59:59.988270+0000","last_undegraded":"2024-07-26T10:59:59.988270+0000","last_fullsized":"2024-07-26T10:59:59.988270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:20.900477+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:59:20.900477+0000","last_peered":"2024-07-26T10:59:20.900477+0000","last_clean":"2024-07-26T10:59:20.900477+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:59:20.900477+0000","last_undegraded":"2024-07-26T10:59:20.900477+0000","last_fullsized":"2024-07-26T10:59:20.900477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.995989+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T11:00:12.995989+0000","last_peered":"2024-07-26T11:00:12.995989+0000","last_clean":"2024-07-26T11:00:12.995989+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T11:00:12.995989+0000","last_undegraded":"2024-07-26T11:00:12.995989+0000","last_fullsized":"2024-07-26T11:00:12.995989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:01.711358+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T11:00:01.711358+0000","last_peered":"2024-07-26T11:00:01.711358+0000","last_clean":"2024-07-26T11:00:01.711358+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T11:00:01.711358+0000","last_undegraded":"2024-07-26T11:00:01.711358+0000","last_fullsized":"2024-07-26T11:00:01.711358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.235288+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:59:51.235288+0000","last_peered":"2024-07-26T10:59:51.235288+0000","last_clean":"2024-07-26T10:59:51.235288+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:59:51.235288+0000","last_undegraded":"2024-07-26T10:59:51.235288+0000","last_fullsized":"2024-07-26T10:59:51.235288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986654+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T11:00:12.986654+0000","last_peered":"2024-07-26T11:00:12.986654+0000","last_clean":"2024-07-26T11:00:12.986654+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T11:00:12.986654+0000","last_undegraded":"2024-07-26T11:00:12.986654+0000","last_fullsized":"2024-07-26T11:00:12.986654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212336+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212336+0000","last_peered":"2024-07-26T10:58:45.212336+0000","last_clean":"2024-07-26T10:58:45.212336+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212336+0000","last_undegraded":"2024-07-26T10:58:45.212336+0000","last_fullsized":"2024-07-26T10:58:45.212336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395309+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395309+0000","last_peered":"2024-07-26T10:58:45.395309+0000","last_clean":"2024-07-26T10:58:45.395309+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395309+0000","last_undegraded":"2024-07-26T10:58:45.395309+0000","last_fullsized":"2024-07-26T10:58:45.395309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952745+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T11:00:12.952745+0000","last_peered":"2024-07-26T11:00:12.952745+0000","last_clean":"2024-07-26T11:00:12.952745+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T11:00:12.952745+0000","last_undegraded":"2024-07-26T11:00:12.952745+0000","last_fullsized":"2024-07-26T11:00:12.952745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212134+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212134+0000","last_peered":"2024-07-26T10:58:45.212134+0000","last_clean":"2024-07-26T10:58:45.212134+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212134+0000","last_undegraded":"2024-07-26T10:58:45.212134+0000","last_fullsized":"2024-07-26T10:58:45.212134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969821+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T11:00:12.969821+0000","last_peered":"2024-07-26T11:00:12.969821+0000","last_clean":"2024-07-26T11:00:12.969821+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T11:00:12.969821+0000","last_undegraded":"2024-07-26T11:00:12.969821+0000","last_fullsized":"2024-07-26T11:00:12.969821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:36.233647+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:59:36.233647+0000","last_peered":"2024-07-26T10:59:36.233647+0000","last_clean":"2024-07-26T10:59:36.233647+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:59:36.233647+0000","last_undegraded":"2024-07-26T10:59:36.233647+0000","last_fullsized":"2024-07-26T10:59:36.233647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710263+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:59:51.710263+0000","last_peered":"2024-07-26T10:59:51.710263+0000","last_clean":"2024-07-26T10:59:51.710263+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:59:51.710263+0000","last_undegraded":"2024-07-26T10:59:51.710263+0000","last_fullsized":"2024-07-26T10:59:51.710263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.997712+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T11:00:12.997712+0000","last_peered":"2024-07-26T11:00:12.997712+0000","last_clean":"2024-07-26T11:00:12.997712+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T11:00:12.997712+0000","last_undegraded":"2024-07-26T11:00:12.997712+0000","last_fullsized":"2024-07-26T11:00:12.997712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:01:40.000319+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T11:01:40.000319+0000","last_peered":"2024-07-26T11:01:40.000319+0000","last_clean":"2024-07-26T11:01:40.000319+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T11:01:40.000319+0000","last_undegraded":"2024-07-26T11:01:40.000319+0000","last_fullsized":"2024-07-26T11:01:40.000319+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710222+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:59:51.710222+0000","last_peered":"2024-07-26T10:59:51.710222+0000","last_clean":"2024-07-26T10:59:51.710222+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:59:51.710222+0000","last_undegraded":"2024-07-26T10:59:51.710222+0000","last_fullsized":"2024-07-26T10:59:51.710222+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.970711+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T11:00:12.970711+0000","last_peered":"2024-07-26T11:00:12.970711+0000","last_clean":"2024-07-26T11:00:12.970711+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T11:00:12.970711+0000","last_undegraded":"2024-07-26T11:00:12.970711+0000","last_fullsized":"2024-07-26T11:00:12.970711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:30.902021+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:59:30.902021+0000","last_peered":"2024-07-26T10:59:30.902021+0000","last_clean":"2024-07-26T10:59:30.902021+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:59:30.902021+0000","last_undegraded":"2024-07-26T10:59:30.902021+0000","last_fullsized":"2024-07-26T10:59:30.902021+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:01.236531+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T11:00:01.236531+0000","last_peered":"2024-07-26T11:00:01.236531+0000","last_clean":"2024-07-26T11:00:01.236531+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T11:00:01.236531+0000","last_undegraded":"2024-07-26T11:00:01.236531+0000","last_fullsized":"2024-07-26T11:00:01.236531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.935240+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T11:00:12.935240+0000","last_peered":"2024-07-26T11:00:12.935240+0000","last_clean":"2024-07-26T11:00:12.935240+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T11:00:12.935240+0000","last_undegraded":"2024-07-26T11:00:12.935240+0000","last_fullsized":"2024-07-26T11:00:12.935240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:28.660005+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:59:28.660005+0000","last_peered":"2024-07-26T10:59:28.660005+0000","last_clean":"2024-07-26T10:59:28.660005+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:59:28.660005+0000","last_undegraded":"2024-07-26T10:59:28.660005+0000","last_fullsized":"2024-07-26T10:59:28.660005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710276+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:59:51.710276+0000","last_peered":"2024-07-26T10:59:51.710276+0000","last_clean":"2024-07-26T10:59:51.710276+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:59:51.710276+0000","last_undegraded":"2024-07-26T10:59:51.710276+0000","last_fullsized":"2024-07-26T10:59:51.710276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994606+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T11:00:12.994606+0000","last_peered":"2024-07-26T11:00:12.994606+0000","last_clean":"2024-07-26T11:00:12.994606+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T11:00:12.994606+0000","last_undegraded":"2024-07-26T11:00:12.994606+0000","last_fullsized":"2024-07-26T11:00:12.994606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:31.707610+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:59:31.707610+0000","last_peered":"2024-07-26T10:59:31.707610+0000","last_clean":"2024-07-26T10:59:31.707610+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:59:31.707610+0000","last_undegraded":"2024-07-26T10:59:31.707610+0000","last_fullsized":"2024-07-26T10:59:31.707610+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.985686+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:59:39.985686+0000","last_peered":"2024-07-26T10:59:39.985686+0000","last_clean":"2024-07-26T10:59:39.985686+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:59:39.985686+0000","last_undegraded":"2024-07-26T10:59:39.985686+0000","last_fullsized":"2024-07-26T10:59:39.985686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982250+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T11:00:12.982250+0000","last_peered":"2024-07-26T11:00:12.982250+0000","last_clean":"2024-07-26T11:00:12.982250+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T11:00:12.982250+0000","last_undegraded":"2024-07-26T11:00:12.982250+0000","last_fullsized":"2024-07-26T11:00:12.982250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:01:40.617170+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T11:01:40.617170+0000","last_peered":"2024-07-26T11:01:40.617170+0000","last_clean":"2024-07-26T11:01:40.617170+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T11:01:40.617170+0000","last_undegraded":"2024-07-26T11:01:40.617170+0000","last_fullsized":"2024-07-26T11:01:40.617170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:35.903083+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:59:35.903083+0000","last_peered":"2024-07-26T10:59:35.903083+0000","last_clean":"2024-07-26T10:59:35.903083+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:59:35.903083+0000","last_undegraded":"2024-07-26T10:59:35.903083+0000","last_fullsized":"2024-07-26T10:59:35.903083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988263+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.988263+0000","last_peered":"2024-07-26T11:00:12.988263+0000","last_clean":"2024-07-26T11:00:12.988263+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T11:00:12.988263+0000","last_undegraded":"2024-07-26T11:00:12.988263+0000","last_fullsized":"2024-07-26T11:00:12.988263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:26.232295+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:59:26.232295+0000","last_peered":"2024-07-26T10:59:26.232295+0000","last_clean":"2024-07-26T10:59:26.232295+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:59:26.232295+0000","last_undegraded":"2024-07-26T10:59:26.232295+0000","last_fullsized":"2024-07-26T10:59:26.232295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:40.717767+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:59:40.717767+0000","last_peered":"2024-07-26T10:59:40.717767+0000","last_clean":"2024-07-26T10:59:40.717767+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:59:40.717767+0000","last_undegraded":"2024-07-26T10:59:40.717767+0000","last_fullsized":"2024-07-26T10:59:40.717767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980610+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.980610+0000","last_peered":"2024-07-26T11:00:12.980610+0000","last_clean":"2024-07-26T11:00:12.980610+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T11:00:12.980610+0000","last_undegraded":"2024-07-26T11:00:12.980610+0000","last_fullsized":"2024-07-26T11:00:12.980610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:25.593738+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:59:25.593738+0000","last_peered":"2024-07-26T10:59:25.593738+0000","last_clean":"2024-07-26T10:59:25.593738+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:59:25.593738+0000","last_undegraded":"2024-07-26T10:59:25.593738+0000","last_fullsized":"2024-07-26T10:59:25.593738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:41.234222+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:59:41.234222+0000","last_peered":"2024-07-26T10:59:41.234222+0000","last_clean":"2024-07-26T10:59:41.234222+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:59:41.234222+0000","last_undegraded":"2024-07-26T10:59:41.234222+0000","last_fullsized":"2024-07-26T10:59:41.234222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.976216+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T11:00:12.976216+0000","last_peered":"2024-07-26T11:00:12.976216+0000","last_clean":"2024-07-26T11:00:12.976216+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T11:00:12.976216+0000","last_undegraded":"2024-07-26T11:00:12.976216+0000","last_fullsized":"2024-07-26T11:00:12.976216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:22.629190+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:59:22.629190+0000","last_peered":"2024-07-26T10:59:22.629190+0000","last_clean":"2024-07-26T10:59:22.629190+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:59:22.629190+0000","last_undegraded":"2024-07-26T10:59:22.629190+0000","last_fullsized":"2024-07-26T10:59:22.629190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:43.045581+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:59:43.045581+0000","last_peered":"2024-07-26T10:59:43.045581+0000","last_clean":"2024-07-26T10:59:43.045581+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:59:43.045581+0000","last_undegraded":"2024-07-26T10:59:43.045581+0000","last_fullsized":"2024-07-26T10:59:43.045581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950189+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T11:00:12.950189+0000","last_peered":"2024-07-26T11:00:12.950189+0000","last_clean":"2024-07-26T11:00:12.950189+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T11:00:12.950189+0000","last_undegraded":"2024-07-26T11:00:12.950189+0000","last_fullsized":"2024-07-26T11:00:12.950189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":272,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:21.700129+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T11:02:21.700129+0000","last_peered":"2024-07-26T11:02:21.700129+0000","last_clean":"2024-07-26T11:02:21.700129+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T11:02:21.700129+0000","last_undegraded":"2024-07-26T11:02:21.700129+0000","last_fullsized":"2024-07-26T11:02:21.700129+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":221,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:48.046731+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:59:48.046731+0000","last_peered":"2024-07-26T10:59:48.046731+0000","last_clean":"2024-07-26T10:59:48.046731+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:59:48.046731+0000","last_undegraded":"2024-07-26T10:59:48.046731+0000","last_fullsized":"2024-07-26T10:59:48.046731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.983164+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T11:00:12.983164+0000","last_peered":"2024-07-26T11:00:12.983164+0000","last_clean":"2024-07-26T11:00:12.983164+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T11:00:12.983164+0000","last_undegraded":"2024-07-26T11:00:12.983164+0000","last_fullsized":"2024-07-26T11:00:12.983164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:17.628714+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:59:17.628714+0000","last_peered":"2024-07-26T10:59:17.628714+0000","last_clean":"2024-07-26T10:59:17.628714+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:59:17.628714+0000","last_undegraded":"2024-07-26T10:59:17.628714+0000","last_fullsized":"2024-07-26T10:59:17.628714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:46.227347+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:46.227347+0000","last_peered":"2024-07-26T10:58:46.227347+0000","last_clean":"2024-07-26T10:58:46.227347+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:46.227347+0000","last_undegraded":"2024-07-26T10:58:46.227347+0000","last_fullsized":"2024-07-26T10:58:46.227347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.971643+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T11:00:12.971643+0000","last_peered":"2024-07-26T11:00:12.971643+0000","last_clean":"2024-07-26T11:00:12.971643+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T11:00:12.971643+0000","last_undegraded":"2024-07-26T11:00:12.971643+0000","last_fullsized":"2024-07-26T11:00:12.971643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:26.707005+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:59:26.707005+0000","last_peered":"2024-07-26T10:59:26.707005+0000","last_clean":"2024-07-26T10:59:26.707005+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:59:26.707005+0000","last_undegraded":"2024-07-26T10:59:26.707005+0000","last_fullsized":"2024-07-26T10:59:26.707005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210654+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210654+0000","last_peered":"2024-07-26T10:58:45.210654+0000","last_clean":"2024-07-26T10:58:45.210654+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210654+0000","last_undegraded":"2024-07-26T10:58:45.210654+0000","last_fullsized":"2024-07-26T10:58:45.210654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:42.632233+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:59:42.632233+0000","last_peered":"2024-07-26T10:59:42.632233+0000","last_clean":"2024-07-26T10:59:42.632233+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:59:42.632233+0000","last_undegraded":"2024-07-26T10:59:42.632233+0000","last_fullsized":"2024-07-26T10:59:42.632233+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955668+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T11:00:12.955668+0000","last_peered":"2024-07-26T11:00:12.955668+0000","last_clean":"2024-07-26T11:00:12.955668+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T11:00:12.955668+0000","last_undegraded":"2024-07-26T11:00:12.955668+0000","last_fullsized":"2024-07-26T11:00:12.955668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:34.984986+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:59:34.984986+0000","last_peered":"2024-07-26T10:59:34.984986+0000","last_clean":"2024-07-26T10:59:34.984986+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:59:34.984986+0000","last_undegraded":"2024-07-26T10:59:34.984986+0000","last_fullsized":"2024-07-26T10:59:34.984986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:48.662570+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:59:48.662570+0000","last_peered":"2024-07-26T10:59:48.662570+0000","last_clean":"2024-07-26T10:59:48.662570+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:59:48.662570+0000","last_undegraded":"2024-07-26T10:59:48.662570+0000","last_fullsized":"2024-07-26T10:59:48.662570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980806+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T11:00:12.980806+0000","last_peered":"2024-07-26T11:00:12.980806+0000","last_clean":"2024-07-26T11:00:12.980806+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T11:00:12.980806+0000","last_undegraded":"2024-07-26T11:00:12.980806+0000","last_fullsized":"2024-07-26T11:00:12.980806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:34.043903+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:59:34.043903+0000","last_peered":"2024-07-26T10:59:34.043903+0000","last_clean":"2024-07-26T10:59:34.043903+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:59:34.043903+0000","last_undegraded":"2024-07-26T10:59:34.043903+0000","last_fullsized":"2024-07-26T10:59:34.043903+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:53.558798+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:59:53.558798+0000","last_peered":"2024-07-26T10:59:53.558798+0000","last_clean":"2024-07-26T10:59:53.558798+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:59:53.558798+0000","last_undegraded":"2024-07-26T10:59:53.558798+0000","last_fullsized":"2024-07-26T10:59:53.558798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954883+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T11:00:12.954883+0000","last_peered":"2024-07-26T11:00:12.954883+0000","last_clean":"2024-07-26T11:00:12.954883+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T11:00:12.954883+0000","last_undegraded":"2024-07-26T11:00:12.954883+0000","last_fullsized":"2024-07-26T11:00:12.954883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:33.043613+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:59:33.043613+0000","last_peered":"2024-07-26T10:59:33.043613+0000","last_clean":"2024-07-26T10:59:33.043613+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:59:33.043613+0000","last_undegraded":"2024-07-26T10:59:33.043613+0000","last_fullsized":"2024-07-26T10:59:33.043613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:49.045647+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:59:49.045647+0000","last_peered":"2024-07-26T10:59:49.045647+0000","last_clean":"2024-07-26T10:59:49.045647+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:59:49.045647+0000","last_undegraded":"2024-07-26T10:59:49.045647+0000","last_fullsized":"2024-07-26T10:59:49.045647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945946+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T11:00:12.945946+0000","last_peered":"2024-07-26T11:00:12.945946+0000","last_clean":"2024-07-26T11:00:12.945946+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T11:00:12.945946+0000","last_undegraded":"2024-07-26T11:00:12.945946+0000","last_fullsized":"2024-07-26T11:00:12.945946+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:36.233709+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:59:36.233709+0000","last_peered":"2024-07-26T10:59:36.233709+0000","last_clean":"2024-07-26T10:59:36.233709+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:59:36.233709+0000","last_undegraded":"2024-07-26T10:59:36.233709+0000","last_fullsized":"2024-07-26T10:59:36.233709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:49.045694+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:59:49.045694+0000","last_peered":"2024-07-26T10:59:49.045694+0000","last_clean":"2024-07-26T10:59:49.045694+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:59:49.045694+0000","last_undegraded":"2024-07-26T10:59:49.045694+0000","last_fullsized":"2024-07-26T10:59:49.045694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.943041+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T11:00:12.943041+0000","last_peered":"2024-07-26T11:00:12.943041+0000","last_clean":"2024-07-26T11:00:12.943041+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T11:00:12.943041+0000","last_undegraded":"2024-07-26T11:00:12.943041+0000","last_fullsized":"2024-07-26T11:00:12.943041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:41.234191+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:59:41.234191+0000","last_peered":"2024-07-26T10:59:41.234191+0000","last_clean":"2024-07-26T10:59:41.234191+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:59:41.234191+0000","last_undegraded":"2024-07-26T10:59:41.234191+0000","last_fullsized":"2024-07-26T10:59:41.234191+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397349+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397349+0000","last_peered":"2024-07-26T10:58:45.397349+0000","last_clean":"2024-07-26T10:58:45.397349+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397349+0000","last_undegraded":"2024-07-26T10:58:45.397349+0000","last_fullsized":"2024-07-26T10:58:45.397349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.953676+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T11:00:12.953676+0000","last_peered":"2024-07-26T11:00:12.953676+0000","last_clean":"2024-07-26T11:00:12.953676+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T11:00:12.953676+0000","last_undegraded":"2024-07-26T11:00:12.953676+0000","last_fullsized":"2024-07-26T11:00:12.953676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":280,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:25.710137+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T11:02:25.710137+0000","last_peered":"2024-07-26T11:02:25.710137+0000","last_clean":"2024-07-26T11:02:25.710137+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T11:02:25.710137+0000","last_undegraded":"2024-07-26T11:02:25.710137+0000","last_fullsized":"2024-07-26T11:02:25.710137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:00.600535+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T11:00:00.600535+0000","last_peered":"2024-07-26T11:00:00.600535+0000","last_clean":"2024-07-26T11:00:00.600535+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T11:00:00.600535+0000","last_undegraded":"2024-07-26T11:00:00.600535+0000","last_fullsized":"2024-07-26T11:00:00.600535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987945+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T11:00:12.987945+0000","last_peered":"2024-07-26T11:00:12.987945+0000","last_clean":"2024-07-26T11:00:12.987945+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T11:00:12.987945+0000","last_undegraded":"2024-07-26T11:00:12.987945+0000","last_fullsized":"2024-07-26T11:00:12.987945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.044678+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:59:39.044678+0000","last_peered":"2024-07-26T10:59:39.044678+0000","last_clean":"2024-07-26T10:59:39.044678+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:59:39.044678+0000","last_undegraded":"2024-07-26T10:59:39.044678+0000","last_fullsized":"2024-07-26T10:59:39.044678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.719471+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:59:55.719471+0000","last_peered":"2024-07-26T10:59:55.719471+0000","last_clean":"2024-07-26T10:59:55.719471+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:59:55.719471+0000","last_undegraded":"2024-07-26T10:59:55.719471+0000","last_fullsized":"2024-07-26T10:59:55.719471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986504+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T11:00:12.986504+0000","last_peered":"2024-07-26T11:00:12.986504+0000","last_clean":"2024-07-26T11:00:12.986504+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T11:00:12.986504+0000","last_undegraded":"2024-07-26T11:00:12.986504+0000","last_fullsized":"2024-07-26T11:00:12.986504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.985729+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:59:39.985729+0000","last_peered":"2024-07-26T10:59:39.985729+0000","last_clean":"2024-07-26T10:59:39.985729+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:59:39.985729+0000","last_undegraded":"2024-07-26T10:59:39.985729+0000","last_fullsized":"2024-07-26T10:59:39.985729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954572+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T11:00:12.954572+0000","last_peered":"2024-07-26T11:00:12.954572+0000","last_clean":"2024-07-26T11:00:12.954572+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T11:00:12.954572+0000","last_undegraded":"2024-07-26T11:00:12.954572+0000","last_fullsized":"2024-07-26T11:00:12.954572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":545,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:24.542089+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T11:02:24.542089+0000","last_peered":"2024-07-26T11:02:24.542089+0000","last_clean":"2024-07-26T11:02:24.542089+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T11:02:24.542089+0000","last_undegraded":"2024-07-26T11:02:24.542089+0000","last_fullsized":"2024-07-26T11:02:24.542089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72547,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":459,"num_read_kb":301,"num_write":41,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.599761+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:59:55.599761+0000","last_peered":"2024-07-26T10:59:55.599761+0000","last_clean":"2024-07-26T10:59:55.599761+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:59:55.599761+0000","last_undegraded":"2024-07-26T10:59:55.599761+0000","last_fullsized":"2024-07-26T10:59:55.599761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952219+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T11:00:12.952219+0000","last_peered":"2024-07-26T11:00:12.952219+0000","last_clean":"2024-07-26T11:00:12.952219+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T11:00:12.952219+0000","last_undegraded":"2024-07-26T11:00:12.952219+0000","last_fullsized":"2024-07-26T11:00:12.952219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:44.045189+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:59:44.045189+0000","last_peered":"2024-07-26T10:59:44.045189+0000","last_clean":"2024-07-26T10:59:44.045189+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:59:44.045189+0000","last_undegraded":"2024-07-26T10:59:44.045189+0000","last_fullsized":"2024-07-26T10:59:44.045189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:56.710801+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:59:56.710801+0000","last_peered":"2024-07-26T10:59:56.710801+0000","last_clean":"2024-07-26T10:59:56.710801+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:59:56.710801+0000","last_undegraded":"2024-07-26T10:59:56.710801+0000","last_fullsized":"2024-07-26T10:59:56.710801+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988804+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T11:00:12.988804+0000","last_peered":"2024-07-26T11:00:12.988804+0000","last_clean":"2024-07-26T11:00:12.988804+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T11:00:12.988804+0000","last_undegraded":"2024-07-26T11:00:12.988804+0000","last_fullsized":"2024-07-26T11:00:12.988804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:42.632249+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:59:42.632249+0000","last_peered":"2024-07-26T10:59:42.632249+0000","last_clean":"2024-07-26T10:59:42.632249+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:59:42.632249+0000","last_undegraded":"2024-07-26T10:59:42.632249+0000","last_fullsized":"2024-07-26T10:59:42.632249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.599712+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:59:55.599712+0000","last_peered":"2024-07-26T10:59:55.599712+0000","last_clean":"2024-07-26T10:59:55.599712+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:59:55.599712+0000","last_undegraded":"2024-07-26T10:59:55.599712+0000","last_fullsized":"2024-07-26T10:59:55.599712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950532+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T11:00:12.950532+0000","last_peered":"2024-07-26T11:00:12.950532+0000","last_clean":"2024-07-26T11:00:12.950532+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T11:00:12.950532+0000","last_undegraded":"2024-07-26T11:00:12.950532+0000","last_fullsized":"2024-07-26T11:00:12.950532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":273,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:23.540725+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T11:02:23.540725+0000","last_peered":"2024-07-26T11:02:23.540725+0000","last_clean":"2024-07-26T11:02:23.540725+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T11:02:23.540725+0000","last_undegraded":"2024-07-26T11:02:23.540725+0000","last_fullsized":"2024-07-26T11:02:23.540725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":223,"num_read_kb":223,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.907236+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:59:55.907236+0000","last_peered":"2024-07-26T10:59:55.907236+0000","last_clean":"2024-07-26T10:59:55.907236+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:59:55.907236+0000","last_undegraded":"2024-07-26T10:59:55.907236+0000","last_fullsized":"2024-07-26T10:59:55.907236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:38.556347+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:59:38.556347+0000","last_peered":"2024-07-26T10:59:38.556347+0000","last_clean":"2024-07-26T10:59:38.556347+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:59:38.556347+0000","last_undegraded":"2024-07-26T10:59:38.556347+0000","last_fullsized":"2024-07-26T10:59:38.556347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949136+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T11:00:12.949136+0000","last_peered":"2024-07-26T11:00:12.949136+0000","last_clean":"2024-07-26T11:00:12.949136+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T11:00:12.949136+0000","last_undegraded":"2024-07-26T11:00:12.949136+0000","last_fullsized":"2024-07-26T11:00:12.949136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:44.045142+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:59:44.045142+0000","last_peered":"2024-07-26T10:59:44.045142+0000","last_clean":"2024-07-26T10:59:44.045142+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:59:44.045142+0000","last_undegraded":"2024-07-26T10:59:44.045142+0000","last_fullsized":"2024-07-26T10:59:44.045142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:40.717868+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:59:40.717868+0000","last_peered":"2024-07-26T10:59:40.717868+0000","last_clean":"2024-07-26T10:59:40.717868+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:59:40.717868+0000","last_undegraded":"2024-07-26T10:59:40.717868+0000","last_fullsized":"2024-07-26T10:59:40.717868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988737+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T11:00:12.988737+0000","last_peered":"2024-07-26T11:00:12.988737+0000","last_clean":"2024-07-26T11:00:12.988737+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T11:00:12.988737+0000","last_undegraded":"2024-07-26T11:00:12.988737+0000","last_fullsized":"2024-07-26T11:00:12.988737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:43.045394+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:59:43.045394+0000","last_peered":"2024-07-26T10:59:43.045394+0000","last_clean":"2024-07-26T10:59:43.045394+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:59:43.045394+0000","last_undegraded":"2024-07-26T10:59:43.045394+0000","last_fullsized":"2024-07-26T10:59:43.045394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:50.113187+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:59:50.113187+0000","last_peered":"2024-07-26T10:59:50.113187+0000","last_clean":"2024-07-26T10:59:50.113187+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:59:50.113187+0000","last_undegraded":"2024-07-26T10:59:50.113187+0000","last_fullsized":"2024-07-26T10:59:50.113187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949369+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T11:00:12.949369+0000","last_peered":"2024-07-26T11:00:12.949369+0000","last_clean":"2024-07-26T11:00:12.949369+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T11:00:12.949369+0000","last_undegraded":"2024-07-26T11:00:12.949369+0000","last_fullsized":"2024-07-26T11:00:12.949369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205546+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205546+0000","last_peered":"2024-07-26T10:58:45.205546+0000","last_clean":"2024-07-26T10:58:45.205546+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205546+0000","last_undegraded":"2024-07-26T10:58:45.205546+0000","last_fullsized":"2024-07-26T10:58:45.205546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":285,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:22.539964+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T11:02:22.539964+0000","last_peered":"2024-07-26T11:02:22.539964+0000","last_clean":"2024-07-26T11:02:22.539964+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T11:02:22.539964+0000","last_undegraded":"2024-07-26T11:02:22.539964+0000","last_fullsized":"2024-07-26T11:02:22.539964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":929,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":225,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.939415+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T11:00:12.939415+0000","last_peered":"2024-07-26T11:00:12.939415+0000","last_clean":"2024-07-26T11:00:12.939415+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T11:00:12.939415+0000","last_undegraded":"2024-07-26T11:00:12.939415+0000","last_fullsized":"2024-07-26T11:00:12.939415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205755+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205755+0000","last_peered":"2024-07-26T10:58:45.205755+0000","last_clean":"2024-07-26T10:58:45.205755+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205755+0000","last_undegraded":"2024-07-26T10:58:45.205755+0000","last_fullsized":"2024-07-26T10:58:45.205755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942853+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T11:00:12.942853+0000","last_peered":"2024-07-26T11:00:12.942853+0000","last_clean":"2024-07-26T11:00:12.942853+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T11:00:12.942853+0000","last_undegraded":"2024-07-26T11:00:12.942853+0000","last_fullsized":"2024-07-26T11:00:12.942853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946199+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T11:00:12.946199+0000","last_peered":"2024-07-26T11:00:12.946199+0000","last_clean":"2024-07-26T11:00:12.946199+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T11:00:12.946199+0000","last_undegraded":"2024-07-26T11:00:12.946199+0000","last_fullsized":"2024-07-26T11:00:12.946199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.975244+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T11:00:12.975244+0000","last_peered":"2024-07-26T11:00:12.975244+0000","last_clean":"2024-07-26T11:00:12.975244+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T11:00:12.975244+0000","last_undegraded":"2024-07-26T11:00:12.975244+0000","last_fullsized":"2024-07-26T11:00:12.975244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992785+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T11:00:12.992785+0000","last_peered":"2024-07-26T11:00:12.992785+0000","last_clean":"2024-07-26T11:00:12.992785+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T11:00:12.992785+0000","last_undegraded":"2024-07-26T11:00:12.992785+0000","last_fullsized":"2024-07-26T11:00:12.992785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.996976+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T11:00:12.996976+0000","last_peered":"2024-07-26T11:00:12.996976+0000","last_clean":"2024-07-26T11:00:12.996976+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T11:00:12.996976+0000","last_undegraded":"2024-07-26T11:00:12.996976+0000","last_fullsized":"2024-07-26T11:00:12.996976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.940323+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T11:00:12.940323+0000","last_peered":"2024-07-26T11:00:12.940323+0000","last_clean":"2024-07-26T11:00:12.940323+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T11:00:12.940323+0000","last_undegraded":"2024-07-26T11:00:12.940323+0000","last_fullsized":"2024-07-26T11:00:12.940323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942010+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T11:00:12.942010+0000","last_peered":"2024-07-26T11:00:12.942010+0000","last_clean":"2024-07-26T11:00:12.942010+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T11:00:12.942010+0000","last_undegraded":"2024-07-26T11:00:12.942010+0000","last_fullsized":"2024-07-26T11:00:12.942010+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.973405+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T11:00:12.973405+0000","last_peered":"2024-07-26T11:00:12.973405+0000","last_clean":"2024-07-26T11:00:12.973405+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T11:00:12.973405+0000","last_undegraded":"2024-07-26T11:00:12.973405+0000","last_fullsized":"2024-07-26T11:00:12.973405+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967597+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T11:00:12.967597+0000","last_peered":"2024-07-26T11:00:12.967597+0000","last_clean":"2024-07-26T11:00:12.967597+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T11:00:12.967597+0000","last_undegraded":"2024-07-26T11:00:12.967597+0000","last_fullsized":"2024-07-26T11:00:12.967597+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982036+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T11:00:12.982036+0000","last_peered":"2024-07-26T11:00:12.982036+0000","last_clean":"2024-07-26T11:00:12.982036+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T11:00:12.982036+0000","last_undegraded":"2024-07-26T11:00:12.982036+0000","last_fullsized":"2024-07-26T11:00:12.982036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981299+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T11:00:12.981299+0000","last_peered":"2024-07-26T11:00:12.981299+0000","last_clean":"2024-07-26T11:00:12.981299+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T11:00:12.981299+0000","last_undegraded":"2024-07-26T11:00:12.981299+0000","last_fullsized":"2024-07-26T11:00:12.981299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.999210+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T11:00:12.999210+0000","last_peered":"2024-07-26T11:00:12.999210+0000","last_clean":"2024-07-26T11:00:12.999210+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T11:00:12.999210+0000","last_undegraded":"2024-07-26T11:00:12.999210+0000","last_fullsized":"2024-07-26T11:00:12.999210+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946804+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T11:00:12.946804+0000","last_peered":"2024-07-26T11:00:12.946804+0000","last_clean":"2024-07-26T11:00:12.946804+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T11:00:12.946804+0000","last_undegraded":"2024-07-26T11:00:12.946804+0000","last_fullsized":"2024-07-26T11:00:12.946804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947300+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T11:00:12.947300+0000","last_peered":"2024-07-26T11:00:12.947300+0000","last_clean":"2024-07-26T11:00:12.947300+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T11:00:12.947300+0000","last_undegraded":"2024-07-26T11:00:12.947300+0000","last_fullsized":"2024-07-26T11:00:12.947300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945515+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T11:00:12.945515+0000","last_peered":"2024-07-26T11:00:12.945515+0000","last_clean":"2024-07-26T11:00:12.945515+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T11:00:12.945515+0000","last_undegraded":"2024-07-26T11:00:12.945515+0000","last_fullsized":"2024-07-26T11:00:12.945515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974957+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T11:00:12.974957+0000","last_peered":"2024-07-26T11:00:12.974957+0000","last_clean":"2024-07-26T11:00:12.974957+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T11:00:12.974957+0000","last_undegraded":"2024-07-26T11:00:12.974957+0000","last_fullsized":"2024-07-26T11:00:12.974957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992896+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T11:00:12.992896+0000","last_peered":"2024-07-26T11:00:12.992896+0000","last_clean":"2024-07-26T11:00:12.992896+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T11:00:12.992896+0000","last_undegraded":"2024-07-26T11:00:12.992896+0000","last_fullsized":"2024-07-26T11:00:12.992896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992111+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T11:00:12.992111+0000","last_peered":"2024-07-26T11:00:12.992111+0000","last_clean":"2024-07-26T11:00:12.992111+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T11:00:12.992111+0000","last_undegraded":"2024-07-26T11:00:12.992111+0000","last_fullsized":"2024-07-26T11:00:12.992111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994447+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T11:00:12.994447+0000","last_peered":"2024-07-26T11:00:12.994447+0000","last_clean":"2024-07-26T11:00:12.994447+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T11:00:12.994447+0000","last_undegraded":"2024-07-26T11:00:12.994447+0000","last_fullsized":"2024-07-26T11:00:12.994447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988020+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T11:00:12.988020+0000","last_peered":"2024-07-26T11:00:12.988020+0000","last_clean":"2024-07-26T11:00:12.988020+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T11:00:12.988020+0000","last_undegraded":"2024-07-26T11:00:12.988020+0000","last_fullsized":"2024-07-26T11:00:12.988020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951958+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T11:00:12.951958+0000","last_peered":"2024-07-26T11:00:12.951958+0000","last_clean":"2024-07-26T11:00:12.951958+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T11:00:12.951958+0000","last_undegraded":"2024-07-26T11:00:12.951958+0000","last_fullsized":"2024-07-26T11:00:12.951958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955435+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T11:00:12.955435+0000","last_peered":"2024-07-26T11:00:12.955435+0000","last_clean":"2024-07-26T11:00:12.955435+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T11:00:12.955435+0000","last_undegraded":"2024-07-26T11:00:12.955435+0000","last_fullsized":"2024-07-26T11:00:12.955435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.948784+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T11:00:12.948784+0000","last_peered":"2024-07-26T11:00:12.948784+0000","last_clean":"2024-07-26T11:00:12.948784+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T11:00:12.948784+0000","last_undegraded":"2024-07-26T11:00:12.948784+0000","last_fullsized":"2024-07-26T11:00:12.948784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946071+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T11:00:12.946071+0000","last_peered":"2024-07-26T11:00:12.946071+0000","last_clean":"2024-07-26T11:00:12.946071+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T11:00:12.946071+0000","last_undegraded":"2024-07-26T11:00:12.946071+0000","last_fullsized":"2024-07-26T11:00:12.946071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981175+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T11:00:12.981175+0000","last_peered":"2024-07-26T11:00:12.981175+0000","last_clean":"2024-07-26T11:00:12.981175+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T11:00:12.981175+0000","last_undegraded":"2024-07-26T11:00:12.981175+0000","last_fullsized":"2024-07-26T11:00:12.981175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952874+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T11:00:12.952874+0000","last_peered":"2024-07-26T11:00:12.952874+0000","last_clean":"2024-07-26T11:00:12.952874+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T11:00:12.952874+0000","last_undegraded":"2024-07-26T11:00:12.952874+0000","last_fullsized":"2024-07-26T11:00:12.952874+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947037+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T11:00:12.947037+0000","last_peered":"2024-07-26T11:00:12.947037+0000","last_clean":"2024-07-26T11:00:12.947037+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T11:00:12.947037+0000","last_undegraded":"2024-07-26T11:00:12.947037+0000","last_fullsized":"2024-07-26T11:00:12.947037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967730+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T11:00:12.967730+0000","last_peered":"2024-07-26T11:00:12.967730+0000","last_clean":"2024-07-26T11:00:12.967730+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T11:00:12.967730+0000","last_undegraded":"2024-07-26T11:00:12.967730+0000","last_fullsized":"2024-07-26T11:00:12.967730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.953352+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T11:00:12.953352+0000","last_peered":"2024-07-26T11:00:12.953352+0000","last_clean":"2024-07-26T11:00:12.953352+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T11:00:12.953352+0000","last_undegraded":"2024-07-26T11:00:12.953352+0000","last_fullsized":"2024-07-26T11:00:12.953352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989600+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T11:00:12.989600+0000","last_peered":"2024-07-26T11:00:12.989600+0000","last_clean":"2024-07-26T11:00:12.989600+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T11:00:12.989600+0000","last_undegraded":"2024-07-26T11:00:12.989600+0000","last_fullsized":"2024-07-26T11:00:12.989600+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.990190+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T11:00:12.990190+0000","last_peered":"2024-07-26T11:00:12.990190+0000","last_clean":"2024-07-26T11:00:12.990190+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T11:00:12.990190+0000","last_undegraded":"2024-07-26T11:00:12.990190+0000","last_fullsized":"2024-07-26T11:00:12.990190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974796+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T11:00:12.974796+0000","last_peered":"2024-07-26T11:00:12.974796+0000","last_clean":"2024-07-26T11:00:12.974796+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T11:00:12.974796+0000","last_undegraded":"2024-07-26T11:00:12.974796+0000","last_fullsized":"2024-07-26T11:00:12.974796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76996,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":1371,"num_read_kb":766,"num_write":186,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":230988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678093,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874323,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1203708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234078,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430306,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593832,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790062,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953587,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378750,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":603766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542276,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":603967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":73275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:02:27.547 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T11:02:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:27 smithi160 ceph-mon[36715]: pgmap v460: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-07-26T11:02:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:27 smithi183 ceph-mon[37273]: pgmap v460: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-07-26T11:02:27.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.856+0000 7fefed200640 1 -- 172.21.15.144:0/3224802039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104ac0 msgr2=0x7fefe810d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.856+0000 7fefed200640 1 --2- 172.21.15.144:0/3224802039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104ac0 0x7fefe810d000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009860 tx=0x7fefd002a340 comp rx=0 tx=0).stop 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.856+0000 7fefed200640 1 -- 172.21.15.144:0/3224802039 shutdown_connections 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.856+0000 7fefed200640 1 --2- 172.21.15.144:0/3224802039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fefe810d5c0 0x7fefe810fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.856+0000 7fefed200640 1 --2- 172.21.15.144:0/3224802039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104ac0 0x7fefe810d000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.856+0000 7fefed200640 1 --2- 172.21.15.144:0/3224802039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe8104170 0x7fefe8104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.857+0000 7fefed200640 1 -- 172.21.15.144:0/3224802039 >> 172.21.15.144:0/3224802039 conn(0x7fefe80fdc40 msgr2=0x7fefe8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.857+0000 7fefed200640 1 -- 172.21.15.144:0/3224802039 shutdown_connections 2024-07-26T11:02:27.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.857+0000 7fefed200640 1 -- 172.21.15.144:0/3224802039 wait complete. 2024-07-26T11:02:27.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.858+0000 7fefed200640 1 Processor -- start 2024-07-26T11:02:27.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.858+0000 7fefed200640 1 -- start start 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefed200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104170 0x7fefe8101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefed200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fefe8104ac0 0x7fefe81023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefed200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 0x7fefe8102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefed200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113210 con 0x7fefe810d5c0 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefed200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113090 con 0x7fefe8104ac0 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefed200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113390 con 0x7fefe8104170 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 0x7fefe8102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 0x7fefe8102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45786/0 (socket says 172.21.15.144:45786) 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 -- 172.21.15.144:0/916930732 learned_addr learned my addr 172.21.15.144:0/916930732 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7400640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104170 0x7fefe8101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe6a00640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fefe8104ac0 0x7fefe81023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 -- 172.21.15.144:0/916930732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104170 msgr2=0x7fefe8101ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104170 0x7fefe8101ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 -- 172.21.15.144:0/916930732 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fefe8104ac0 msgr2=0x7fefe81023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fefe8104ac0 0x7fefe81023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.859+0000 7fefe7e00640 1 -- 172.21.15.144:0/916930732 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe81a7ee0 con 0x7fefe810d5c0 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefe7400640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104170 0x7fefe8101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:27.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefe7e00640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 0x7fefe8102920 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fefe000eb10 tx=0x7fefe000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:27.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0004b20 con 0x7fefe810d5c0 2024-07-26T11:02:27.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefe0004cc0 con 0x7fefe810d5c0 2024-07-26T11:02:27.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0010670 con 0x7fefe810d5c0 2024-07-26T11:02:27.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe81a81d0 con 0x7fefe810d5c0 2024-07-26T11:02:27.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.860+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe81a86f0 con 0x7fefe810d5c0 2024-07-26T11:02:27.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.861+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe8104580 con 0x7fefe810d5c0 2024-07-26T11:02:27.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.864+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fefe0004e70 con 0x7fefe810d5c0 2024-07-26T11:02:27.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.866+0000 7fefdfe00640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fefc80b3c50 0x7fefc80b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:27.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.866+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fefe00e48a0 con 0x7fefe810d5c0 2024-07-26T11:02:27.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.866+0000 7fefe7400640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fefc80b3c50 0x7fefc80b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:27.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.867+0000 7fefe7400640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fefc80b3c50 0x7fefc80b6160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fefd80096c0 tx=0x7fefd8009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:27.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:27.868+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fefe0055600 con 0x7fefe810d5c0 2024-07-26T11:02:28.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.072+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fefe8063230 con 0x7fefc80b3c50 2024-07-26T11:02:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.088+0000 7fefdfe00640 1 -- 172.21.15.144:0/916930732 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340481 (secure 0 0 0) 0x7fefe8063230 con 0x7fefc80b3c50 2024-07-26T11:02:28.094 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:28.098 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T11:02:28.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fefc80b3c50 msgr2=0x7fefc80b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fefc80b3c50 0x7fefc80b6160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fefd80096c0 tx=0x7fefd8009260 comp rx=0 tx=0).stop 2024-07-26T11:02:28.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 msgr2=0x7fefe8102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 0x7fefe8102920 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fefe000eb10 tx=0x7fefe000c2d0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 shutdown_connections 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fefc80b3c50 0x7fefc80b6160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe810d5c0 0x7fefe8102920 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fefe8104ac0 0x7fefe81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.097+0000 7fefed200640 1 --2- 172.21.15.144:0/916930732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefe8104170 0x7fefe8101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.098+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 >> 172.21.15.144:0/916930732 conn(0x7fefe80fdc40 msgr2=0x7fefe80fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.098+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 shutdown_connections 2024-07-26T11:02:28.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.098+0000 7fefed200640 1 -- 172.21.15.144:0/916930732 wait complete. 2024-07-26T11:02:28.128 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-07-26T11:02:27.115339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667598,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1452,"num_read_kb":835,"num_write":311,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":752,"ondisk_log_size":752,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380104,"kb_used_data":14312,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561720,"statfs":{"total":1151940427776,"available":1151551201280,"internally_reserved":0,"allocated":14655488,"data_stored":8400752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":47,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005738"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:25.593785+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:59:25.593785+0000","last_peered":"2024-07-26T10:59:25.593785+0000","last_clean":"2024-07-26T10:59:25.593785+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:59:25.593785+0000","last_undegraded":"2024-07-26T10:59:25.593785+0000","last_fullsized":"2024-07-26T10:59:25.593785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969791+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T11:00:12.969791+0000","last_peered":"2024-07-26T11:00:12.969791+0000","last_clean":"2024-07-26T11:00:12.969791+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T11:00:12.969791+0000","last_undegraded":"2024-07-26T11:00:12.969791+0000","last_fullsized":"2024-07-26T11:00:12.969791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:59.988270+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:59:59.988270+0000","last_peered":"2024-07-26T10:59:59.988270+0000","last_clean":"2024-07-26T10:59:59.988270+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:59:59.988270+0000","last_undegraded":"2024-07-26T10:59:59.988270+0000","last_fullsized":"2024-07-26T10:59:59.988270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:20.900477+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:59:20.900477+0000","last_peered":"2024-07-26T10:59:20.900477+0000","last_clean":"2024-07-26T10:59:20.900477+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:59:20.900477+0000","last_undegraded":"2024-07-26T10:59:20.900477+0000","last_fullsized":"2024-07-26T10:59:20.900477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.995989+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T11:00:12.995989+0000","last_peered":"2024-07-26T11:00:12.995989+0000","last_clean":"2024-07-26T11:00:12.995989+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T11:00:12.995989+0000","last_undegraded":"2024-07-26T11:00:12.995989+0000","last_fullsized":"2024-07-26T11:00:12.995989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:01.711358+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T11:00:01.711358+0000","last_peered":"2024-07-26T11:00:01.711358+0000","last_clean":"2024-07-26T11:00:01.711358+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T11:00:01.711358+0000","last_undegraded":"2024-07-26T11:00:01.711358+0000","last_fullsized":"2024-07-26T11:00:01.711358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.235288+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:59:51.235288+0000","last_peered":"2024-07-26T10:59:51.235288+0000","last_clean":"2024-07-26T10:59:51.235288+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:59:51.235288+0000","last_undegraded":"2024-07-26T10:59:51.235288+0000","last_fullsized":"2024-07-26T10:59:51.235288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986654+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T11:00:12.986654+0000","last_peered":"2024-07-26T11:00:12.986654+0000","last_clean":"2024-07-26T11:00:12.986654+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T11:00:12.986654+0000","last_undegraded":"2024-07-26T11:00:12.986654+0000","last_fullsized":"2024-07-26T11:00:12.986654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212336+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212336+0000","last_peered":"2024-07-26T10:58:45.212336+0000","last_clean":"2024-07-26T10:58:45.212336+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212336+0000","last_undegraded":"2024-07-26T10:58:45.212336+0000","last_fullsized":"2024-07-26T10:58:45.212336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395309+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395309+0000","last_peered":"2024-07-26T10:58:45.395309+0000","last_clean":"2024-07-26T10:58:45.395309+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395309+0000","last_undegraded":"2024-07-26T10:58:45.395309+0000","last_fullsized":"2024-07-26T10:58:45.395309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952745+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T11:00:12.952745+0000","last_peered":"2024-07-26T11:00:12.952745+0000","last_clean":"2024-07-26T11:00:12.952745+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T11:00:12.952745+0000","last_undegraded":"2024-07-26T11:00:12.952745+0000","last_fullsized":"2024-07-26T11:00:12.952745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212134+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212134+0000","last_peered":"2024-07-26T10:58:45.212134+0000","last_clean":"2024-07-26T10:58:45.212134+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212134+0000","last_undegraded":"2024-07-26T10:58:45.212134+0000","last_fullsized":"2024-07-26T10:58:45.212134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969821+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T11:00:12.969821+0000","last_peered":"2024-07-26T11:00:12.969821+0000","last_clean":"2024-07-26T11:00:12.969821+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T11:00:12.969821+0000","last_undegraded":"2024-07-26T11:00:12.969821+0000","last_fullsized":"2024-07-26T11:00:12.969821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:36.233647+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:59:36.233647+0000","last_peered":"2024-07-26T10:59:36.233647+0000","last_clean":"2024-07-26T10:59:36.233647+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:59:36.233647+0000","last_undegraded":"2024-07-26T10:59:36.233647+0000","last_fullsized":"2024-07-26T10:59:36.233647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710263+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:59:51.710263+0000","last_peered":"2024-07-26T10:59:51.710263+0000","last_clean":"2024-07-26T10:59:51.710263+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:59:51.710263+0000","last_undegraded":"2024-07-26T10:59:51.710263+0000","last_fullsized":"2024-07-26T10:59:51.710263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.997712+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T11:00:12.997712+0000","last_peered":"2024-07-26T11:00:12.997712+0000","last_clean":"2024-07-26T11:00:12.997712+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T11:00:12.997712+0000","last_undegraded":"2024-07-26T11:00:12.997712+0000","last_fullsized":"2024-07-26T11:00:12.997712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:01:40.000319+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T11:01:40.000319+0000","last_peered":"2024-07-26T11:01:40.000319+0000","last_clean":"2024-07-26T11:01:40.000319+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T11:01:40.000319+0000","last_undegraded":"2024-07-26T11:01:40.000319+0000","last_fullsized":"2024-07-26T11:01:40.000319+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710222+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:59:51.710222+0000","last_peered":"2024-07-26T10:59:51.710222+0000","last_clean":"2024-07-26T10:59:51.710222+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:59:51.710222+0000","last_undegraded":"2024-07-26T10:59:51.710222+0000","last_fullsized":"2024-07-26T10:59:51.710222+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.970711+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T11:00:12.970711+0000","last_peered":"2024-07-26T11:00:12.970711+0000","last_clean":"2024-07-26T11:00:12.970711+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T11:00:12.970711+0000","last_undegraded":"2024-07-26T11:00:12.970711+0000","last_fullsized":"2024-07-26T11:00:12.970711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:30.902021+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:59:30.902021+0000","last_peered":"2024-07-26T10:59:30.902021+0000","last_clean":"2024-07-26T10:59:30.902021+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:59:30.902021+0000","last_undegraded":"2024-07-26T10:59:30.902021+0000","last_fullsized":"2024-07-26T10:59:30.902021+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:01.236531+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T11:00:01.236531+0000","last_peered":"2024-07-26T11:00:01.236531+0000","last_clean":"2024-07-26T11:00:01.236531+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T11:00:01.236531+0000","last_undegraded":"2024-07-26T11:00:01.236531+0000","last_fullsized":"2024-07-26T11:00:01.236531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.935240+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T11:00:12.935240+0000","last_peered":"2024-07-26T11:00:12.935240+0000","last_clean":"2024-07-26T11:00:12.935240+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T11:00:12.935240+0000","last_undegraded":"2024-07-26T11:00:12.935240+0000","last_fullsized":"2024-07-26T11:00:12.935240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:28.660005+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:59:28.660005+0000","last_peered":"2024-07-26T10:59:28.660005+0000","last_clean":"2024-07-26T10:59:28.660005+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:59:28.660005+0000","last_undegraded":"2024-07-26T10:59:28.660005+0000","last_fullsized":"2024-07-26T10:59:28.660005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710276+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:59:51.710276+0000","last_peered":"2024-07-26T10:59:51.710276+0000","last_clean":"2024-07-26T10:59:51.710276+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:59:51.710276+0000","last_undegraded":"2024-07-26T10:59:51.710276+0000","last_fullsized":"2024-07-26T10:59:51.710276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994606+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T11:00:12.994606+0000","last_peered":"2024-07-26T11:00:12.994606+0000","last_clean":"2024-07-26T11:00:12.994606+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T11:00:12.994606+0000","last_undegraded":"2024-07-26T11:00:12.994606+0000","last_fullsized":"2024-07-26T11:00:12.994606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:31.707610+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:59:31.707610+0000","last_peered":"2024-07-26T10:59:31.707610+0000","last_clean":"2024-07-26T10:59:31.707610+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:59:31.707610+0000","last_undegraded":"2024-07-26T10:59:31.707610+0000","last_fullsized":"2024-07-26T10:59:31.707610+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.985686+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:59:39.985686+0000","last_peered":"2024-07-26T10:59:39.985686+0000","last_clean":"2024-07-26T10:59:39.985686+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:59:39.985686+0000","last_undegraded":"2024-07-26T10:59:39.985686+0000","last_fullsized":"2024-07-26T10:59:39.985686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982250+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T11:00:12.982250+0000","last_peered":"2024-07-26T11:00:12.982250+0000","last_clean":"2024-07-26T11:00:12.982250+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T11:00:12.982250+0000","last_undegraded":"2024-07-26T11:00:12.982250+0000","last_fullsized":"2024-07-26T11:00:12.982250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:01:40.617170+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T11:01:40.617170+0000","last_peered":"2024-07-26T11:01:40.617170+0000","last_clean":"2024-07-26T11:01:40.617170+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T11:01:40.617170+0000","last_undegraded":"2024-07-26T11:01:40.617170+0000","last_fullsized":"2024-07-26T11:01:40.617170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:35.903083+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:59:35.903083+0000","last_peered":"2024-07-26T10:59:35.903083+0000","last_clean":"2024-07-26T10:59:35.903083+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:59:35.903083+0000","last_undegraded":"2024-07-26T10:59:35.903083+0000","last_fullsized":"2024-07-26T10:59:35.903083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988263+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.988263+0000","last_peered":"2024-07-26T11:00:12.988263+0000","last_clean":"2024-07-26T11:00:12.988263+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T11:00:12.988263+0000","last_undegraded":"2024-07-26T11:00:12.988263+0000","last_fullsized":"2024-07-26T11:00:12.988263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:26.232295+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:59:26.232295+0000","last_peered":"2024-07-26T10:59:26.232295+0000","last_clean":"2024-07-26T10:59:26.232295+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:59:26.232295+0000","last_undegraded":"2024-07-26T10:59:26.232295+0000","last_fullsized":"2024-07-26T10:59:26.232295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:40.717767+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:59:40.717767+0000","last_peered":"2024-07-26T10:59:40.717767+0000","last_clean":"2024-07-26T10:59:40.717767+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:59:40.717767+0000","last_undegraded":"2024-07-26T10:59:40.717767+0000","last_fullsized":"2024-07-26T10:59:40.717767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980610+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.980610+0000","last_peered":"2024-07-26T11:00:12.980610+0000","last_clean":"2024-07-26T11:00:12.980610+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T11:00:12.980610+0000","last_undegraded":"2024-07-26T11:00:12.980610+0000","last_fullsized":"2024-07-26T11:00:12.980610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:25.593738+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:59:25.593738+0000","last_peered":"2024-07-26T10:59:25.593738+0000","last_clean":"2024-07-26T10:59:25.593738+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:59:25.593738+0000","last_undegraded":"2024-07-26T10:59:25.593738+0000","last_fullsized":"2024-07-26T10:59:25.593738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:41.234222+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:59:41.234222+0000","last_peered":"2024-07-26T10:59:41.234222+0000","last_clean":"2024-07-26T10:59:41.234222+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:59:41.234222+0000","last_undegraded":"2024-07-26T10:59:41.234222+0000","last_fullsized":"2024-07-26T10:59:41.234222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.976216+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T11:00:12.976216+0000","last_peered":"2024-07-26T11:00:12.976216+0000","last_clean":"2024-07-26T11:00:12.976216+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T11:00:12.976216+0000","last_undegraded":"2024-07-26T11:00:12.976216+0000","last_fullsized":"2024-07-26T11:00:12.976216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:22.629190+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:59:22.629190+0000","last_peered":"2024-07-26T10:59:22.629190+0000","last_clean":"2024-07-26T10:59:22.629190+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:59:22.629190+0000","last_undegraded":"2024-07-26T10:59:22.629190+0000","last_fullsized":"2024-07-26T10:59:22.629190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:43.045581+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:59:43.045581+0000","last_peered":"2024-07-26T10:59:43.045581+0000","last_clean":"2024-07-26T10:59:43.045581+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:59:43.045581+0000","last_undegraded":"2024-07-26T10:59:43.045581+0000","last_fullsized":"2024-07-26T10:59:43.045581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950189+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T11:00:12.950189+0000","last_peered":"2024-07-26T11:00:12.950189+0000","last_clean":"2024-07-26T11:00:12.950189+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T11:00:12.950189+0000","last_undegraded":"2024-07-26T11:00:12.950189+0000","last_fullsized":"2024-07-26T11:00:12.950189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":272,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:21.700129+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T11:02:21.700129+0000","last_peered":"2024-07-26T11:02:21.700129+0000","last_clean":"2024-07-26T11:02:21.700129+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T11:02:21.700129+0000","last_undegraded":"2024-07-26T11:02:21.700129+0000","last_fullsized":"2024-07-26T11:02:21.700129+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":221,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:48.046731+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:59:48.046731+0000","last_peered":"2024-07-26T10:59:48.046731+0000","last_clean":"2024-07-26T10:59:48.046731+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:59:48.046731+0000","last_undegraded":"2024-07-26T10:59:48.046731+0000","last_fullsized":"2024-07-26T10:59:48.046731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.983164+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T11:00:12.983164+0000","last_peered":"2024-07-26T11:00:12.983164+0000","last_clean":"2024-07-26T11:00:12.983164+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T11:00:12.983164+0000","last_undegraded":"2024-07-26T11:00:12.983164+0000","last_fullsized":"2024-07-26T11:00:12.983164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:17.628714+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:59:17.628714+0000","last_peered":"2024-07-26T10:59:17.628714+0000","last_clean":"2024-07-26T10:59:17.628714+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:59:17.628714+0000","last_undegraded":"2024-07-26T10:59:17.628714+0000","last_fullsized":"2024-07-26T10:59:17.628714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:46.227347+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:46.227347+0000","last_peered":"2024-07-26T10:58:46.227347+0000","last_clean":"2024-07-26T10:58:46.227347+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:46.227347+0000","last_undegraded":"2024-07-26T10:58:46.227347+0000","last_fullsized":"2024-07-26T10:58:46.227347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.971643+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T11:00:12.971643+0000","last_peered":"2024-07-26T11:00:12.971643+0000","last_clean":"2024-07-26T11:00:12.971643+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T11:00:12.971643+0000","last_undegraded":"2024-07-26T11:00:12.971643+0000","last_fullsized":"2024-07-26T11:00:12.971643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:26.707005+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:59:26.707005+0000","last_peered":"2024-07-26T10:59:26.707005+0000","last_clean":"2024-07-26T10:59:26.707005+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:59:26.707005+0000","last_undegraded":"2024-07-26T10:59:26.707005+0000","last_fullsized":"2024-07-26T10:59:26.707005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210654+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210654+0000","last_peered":"2024-07-26T10:58:45.210654+0000","last_clean":"2024-07-26T10:58:45.210654+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210654+0000","last_undegraded":"2024-07-26T10:58:45.210654+0000","last_fullsized":"2024-07-26T10:58:45.210654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:42.632233+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:59:42.632233+0000","last_peered":"2024-07-26T10:59:42.632233+0000","last_clean":"2024-07-26T10:59:42.632233+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:59:42.632233+0000","last_undegraded":"2024-07-26T10:59:42.632233+0000","last_fullsized":"2024-07-26T10:59:42.632233+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955668+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T11:00:12.955668+0000","last_peered":"2024-07-26T11:00:12.955668+0000","last_clean":"2024-07-26T11:00:12.955668+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T11:00:12.955668+0000","last_undegraded":"2024-07-26T11:00:12.955668+0000","last_fullsized":"2024-07-26T11:00:12.955668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:34.984986+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:59:34.984986+0000","last_peered":"2024-07-26T10:59:34.984986+0000","last_clean":"2024-07-26T10:59:34.984986+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:59:34.984986+0000","last_undegraded":"2024-07-26T10:59:34.984986+0000","last_fullsized":"2024-07-26T10:59:34.984986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:48.662570+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:59:48.662570+0000","last_peered":"2024-07-26T10:59:48.662570+0000","last_clean":"2024-07-26T10:59:48.662570+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:59:48.662570+0000","last_undegraded":"2024-07-26T10:59:48.662570+0000","last_fullsized":"2024-07-26T10:59:48.662570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980806+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T11:00:12.980806+0000","last_peered":"2024-07-26T11:00:12.980806+0000","last_clean":"2024-07-26T11:00:12.980806+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T11:00:12.980806+0000","last_undegraded":"2024-07-26T11:00:12.980806+0000","last_fullsized":"2024-07-26T11:00:12.980806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:34.043903+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:59:34.043903+0000","last_peered":"2024-07-26T10:59:34.043903+0000","last_clean":"2024-07-26T10:59:34.043903+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:59:34.043903+0000","last_undegraded":"2024-07-26T10:59:34.043903+0000","last_fullsized":"2024-07-26T10:59:34.043903+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:53.558798+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:59:53.558798+0000","last_peered":"2024-07-26T10:59:53.558798+0000","last_clean":"2024-07-26T10:59:53.558798+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:59:53.558798+0000","last_undegraded":"2024-07-26T10:59:53.558798+0000","last_fullsized":"2024-07-26T10:59:53.558798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954883+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T11:00:12.954883+0000","last_peered":"2024-07-26T11:00:12.954883+0000","last_clean":"2024-07-26T11:00:12.954883+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T11:00:12.954883+0000","last_undegraded":"2024-07-26T11:00:12.954883+0000","last_fullsized":"2024-07-26T11:00:12.954883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:33.043613+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:59:33.043613+0000","last_peered":"2024-07-26T10:59:33.043613+0000","last_clean":"2024-07-26T10:59:33.043613+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:59:33.043613+0000","last_undegraded":"2024-07-26T10:59:33.043613+0000","last_fullsized":"2024-07-26T10:59:33.043613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:49.045647+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:59:49.045647+0000","last_peered":"2024-07-26T10:59:49.045647+0000","last_clean":"2024-07-26T10:59:49.045647+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:59:49.045647+0000","last_undegraded":"2024-07-26T10:59:49.045647+0000","last_fullsized":"2024-07-26T10:59:49.045647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945946+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T11:00:12.945946+0000","last_peered":"2024-07-26T11:00:12.945946+0000","last_clean":"2024-07-26T11:00:12.945946+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T11:00:12.945946+0000","last_undegraded":"2024-07-26T11:00:12.945946+0000","last_fullsized":"2024-07-26T11:00:12.945946+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:36.233709+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:59:36.233709+0000","last_peered":"2024-07-26T10:59:36.233709+0000","last_clean":"2024-07-26T10:59:36.233709+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:59:36.233709+0000","last_undegraded":"2024-07-26T10:59:36.233709+0000","last_fullsized":"2024-07-26T10:59:36.233709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:49.045694+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:59:49.045694+0000","last_peered":"2024-07-26T10:59:49.045694+0000","last_clean":"2024-07-26T10:59:49.045694+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:59:49.045694+0000","last_undegraded":"2024-07-26T10:59:49.045694+0000","last_fullsized":"2024-07-26T10:59:49.045694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.943041+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T11:00:12.943041+0000","last_peered":"2024-07-26T11:00:12.943041+0000","last_clean":"2024-07-26T11:00:12.943041+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T11:00:12.943041+0000","last_undegraded":"2024-07-26T11:00:12.943041+0000","last_fullsized":"2024-07-26T11:00:12.943041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:41.234191+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:59:41.234191+0000","last_peered":"2024-07-26T10:59:41.234191+0000","last_clean":"2024-07-26T10:59:41.234191+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:59:41.234191+0000","last_undegraded":"2024-07-26T10:59:41.234191+0000","last_fullsized":"2024-07-26T10:59:41.234191+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397349+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397349+0000","last_peered":"2024-07-26T10:58:45.397349+0000","last_clean":"2024-07-26T10:58:45.397349+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397349+0000","last_undegraded":"2024-07-26T10:58:45.397349+0000","last_fullsized":"2024-07-26T10:58:45.397349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.953676+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T11:00:12.953676+0000","last_peered":"2024-07-26T11:00:12.953676+0000","last_clean":"2024-07-26T11:00:12.953676+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T11:00:12.953676+0000","last_undegraded":"2024-07-26T11:00:12.953676+0000","last_fullsized":"2024-07-26T11:00:12.953676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":280,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:25.710137+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T11:02:25.710137+0000","last_peered":"2024-07-26T11:02:25.710137+0000","last_clean":"2024-07-26T11:02:25.710137+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T11:02:25.710137+0000","last_undegraded":"2024-07-26T11:02:25.710137+0000","last_fullsized":"2024-07-26T11:02:25.710137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:00.600535+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T11:00:00.600535+0000","last_peered":"2024-07-26T11:00:00.600535+0000","last_clean":"2024-07-26T11:00:00.600535+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T11:00:00.600535+0000","last_undegraded":"2024-07-26T11:00:00.600535+0000","last_fullsized":"2024-07-26T11:00:00.600535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987945+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T11:00:12.987945+0000","last_peered":"2024-07-26T11:00:12.987945+0000","last_clean":"2024-07-26T11:00:12.987945+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T11:00:12.987945+0000","last_undegraded":"2024-07-26T11:00:12.987945+0000","last_fullsized":"2024-07-26T11:00:12.987945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.044678+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:59:39.044678+0000","last_peered":"2024-07-26T10:59:39.044678+0000","last_clean":"2024-07-26T10:59:39.044678+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:59:39.044678+0000","last_undegraded":"2024-07-26T10:59:39.044678+0000","last_fullsized":"2024-07-26T10:59:39.044678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.719471+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:59:55.719471+0000","last_peered":"2024-07-26T10:59:55.719471+0000","last_clean":"2024-07-26T10:59:55.719471+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:59:55.719471+0000","last_undegraded":"2024-07-26T10:59:55.719471+0000","last_fullsized":"2024-07-26T10:59:55.719471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986504+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T11:00:12.986504+0000","last_peered":"2024-07-26T11:00:12.986504+0000","last_clean":"2024-07-26T11:00:12.986504+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T11:00:12.986504+0000","last_undegraded":"2024-07-26T11:00:12.986504+0000","last_fullsized":"2024-07-26T11:00:12.986504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.985729+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:59:39.985729+0000","last_peered":"2024-07-26T10:59:39.985729+0000","last_clean":"2024-07-26T10:59:39.985729+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:59:39.985729+0000","last_undegraded":"2024-07-26T10:59:39.985729+0000","last_fullsized":"2024-07-26T10:59:39.985729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954572+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T11:00:12.954572+0000","last_peered":"2024-07-26T11:00:12.954572+0000","last_clean":"2024-07-26T11:00:12.954572+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T11:00:12.954572+0000","last_undegraded":"2024-07-26T11:00:12.954572+0000","last_fullsized":"2024-07-26T11:00:12.954572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":545,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:24.542089+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T11:02:24.542089+0000","last_peered":"2024-07-26T11:02:24.542089+0000","last_clean":"2024-07-26T11:02:24.542089+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T11:02:24.542089+0000","last_undegraded":"2024-07-26T11:02:24.542089+0000","last_fullsized":"2024-07-26T11:02:24.542089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72547,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":459,"num_read_kb":301,"num_write":41,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.599761+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:59:55.599761+0000","last_peered":"2024-07-26T10:59:55.599761+0000","last_clean":"2024-07-26T10:59:55.599761+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:59:55.599761+0000","last_undegraded":"2024-07-26T10:59:55.599761+0000","last_fullsized":"2024-07-26T10:59:55.599761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952219+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T11:00:12.952219+0000","last_peered":"2024-07-26T11:00:12.952219+0000","last_clean":"2024-07-26T11:00:12.952219+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T11:00:12.952219+0000","last_undegraded":"2024-07-26T11:00:12.952219+0000","last_fullsized":"2024-07-26T11:00:12.952219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:44.045189+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:59:44.045189+0000","last_peered":"2024-07-26T10:59:44.045189+0000","last_clean":"2024-07-26T10:59:44.045189+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:59:44.045189+0000","last_undegraded":"2024-07-26T10:59:44.045189+0000","last_fullsized":"2024-07-26T10:59:44.045189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:56.710801+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:59:56.710801+0000","last_peered":"2024-07-26T10:59:56.710801+0000","last_clean":"2024-07-26T10:59:56.710801+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:59:56.710801+0000","last_undegraded":"2024-07-26T10:59:56.710801+0000","last_fullsized":"2024-07-26T10:59:56.710801+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988804+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T11:00:12.988804+0000","last_peered":"2024-07-26T11:00:12.988804+0000","last_clean":"2024-07-26T11:00:12.988804+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T11:00:12.988804+0000","last_undegraded":"2024-07-26T11:00:12.988804+0000","last_fullsized":"2024-07-26T11:00:12.988804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:42.632249+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:59:42.632249+0000","last_peered":"2024-07-26T10:59:42.632249+0000","last_clean":"2024-07-26T10:59:42.632249+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:59:42.632249+0000","last_undegraded":"2024-07-26T10:59:42.632249+0000","last_fullsized":"2024-07-26T10:59:42.632249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.599712+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:59:55.599712+0000","last_peered":"2024-07-26T10:59:55.599712+0000","last_clean":"2024-07-26T10:59:55.599712+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:59:55.599712+0000","last_undegraded":"2024-07-26T10:59:55.599712+0000","last_fullsized":"2024-07-26T10:59:55.599712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950532+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T11:00:12.950532+0000","last_peered":"2024-07-26T11:00:12.950532+0000","last_clean":"2024-07-26T11:00:12.950532+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T11:00:12.950532+0000","last_undegraded":"2024-07-26T11:00:12.950532+0000","last_fullsized":"2024-07-26T11:00:12.950532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":273,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:23.540725+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T11:02:23.540725+0000","last_peered":"2024-07-26T11:02:23.540725+0000","last_clean":"2024-07-26T11:02:23.540725+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T11:02:23.540725+0000","last_undegraded":"2024-07-26T11:02:23.540725+0000","last_fullsized":"2024-07-26T11:02:23.540725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":223,"num_read_kb":223,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.907236+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:59:55.907236+0000","last_peered":"2024-07-26T10:59:55.907236+0000","last_clean":"2024-07-26T10:59:55.907236+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:59:55.907236+0000","last_undegraded":"2024-07-26T10:59:55.907236+0000","last_fullsized":"2024-07-26T10:59:55.907236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:38.556347+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:59:38.556347+0000","last_peered":"2024-07-26T10:59:38.556347+0000","last_clean":"2024-07-26T10:59:38.556347+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:59:38.556347+0000","last_undegraded":"2024-07-26T10:59:38.556347+0000","last_fullsized":"2024-07-26T10:59:38.556347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949136+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T11:00:12.949136+0000","last_peered":"2024-07-26T11:00:12.949136+0000","last_clean":"2024-07-26T11:00:12.949136+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T11:00:12.949136+0000","last_undegraded":"2024-07-26T11:00:12.949136+0000","last_fullsized":"2024-07-26T11:00:12.949136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:44.045142+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:59:44.045142+0000","last_peered":"2024-07-26T10:59:44.045142+0000","last_clean":"2024-07-26T10:59:44.045142+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:59:44.045142+0000","last_undegraded":"2024-07-26T10:59:44.045142+0000","last_fullsized":"2024-07-26T10:59:44.045142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:40.717868+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:59:40.717868+0000","last_peered":"2024-07-26T10:59:40.717868+0000","last_clean":"2024-07-26T10:59:40.717868+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:59:40.717868+0000","last_undegraded":"2024-07-26T10:59:40.717868+0000","last_fullsized":"2024-07-26T10:59:40.717868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988737+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T11:00:12.988737+0000","last_peered":"2024-07-26T11:00:12.988737+0000","last_clean":"2024-07-26T11:00:12.988737+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T11:00:12.988737+0000","last_undegraded":"2024-07-26T11:00:12.988737+0000","last_fullsized":"2024-07-26T11:00:12.988737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:43.045394+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:59:43.045394+0000","last_peered":"2024-07-26T10:59:43.045394+0000","last_clean":"2024-07-26T10:59:43.045394+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:59:43.045394+0000","last_undegraded":"2024-07-26T10:59:43.045394+0000","last_fullsized":"2024-07-26T10:59:43.045394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:50.113187+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:59:50.113187+0000","last_peered":"2024-07-26T10:59:50.113187+0000","last_clean":"2024-07-26T10:59:50.113187+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:59:50.113187+0000","last_undegraded":"2024-07-26T10:59:50.113187+0000","last_fullsized":"2024-07-26T10:59:50.113187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949369+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T11:00:12.949369+0000","last_peered":"2024-07-26T11:00:12.949369+0000","last_clean":"2024-07-26T11:00:12.949369+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T11:00:12.949369+0000","last_undegraded":"2024-07-26T11:00:12.949369+0000","last_fullsized":"2024-07-26T11:00:12.949369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205546+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205546+0000","last_peered":"2024-07-26T10:58:45.205546+0000","last_clean":"2024-07-26T10:58:45.205546+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205546+0000","last_undegraded":"2024-07-26T10:58:45.205546+0000","last_fullsized":"2024-07-26T10:58:45.205546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":285,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:22.539964+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T11:02:22.539964+0000","last_peered":"2024-07-26T11:02:22.539964+0000","last_clean":"2024-07-26T11:02:22.539964+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T11:02:22.539964+0000","last_undegraded":"2024-07-26T11:02:22.539964+0000","last_fullsized":"2024-07-26T11:02:22.539964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":929,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":225,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.939415+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T11:00:12.939415+0000","last_peered":"2024-07-26T11:00:12.939415+0000","last_clean":"2024-07-26T11:00:12.939415+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T11:00:12.939415+0000","last_undegraded":"2024-07-26T11:00:12.939415+0000","last_fullsized":"2024-07-26T11:00:12.939415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205755+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205755+0000","last_peered":"2024-07-26T10:58:45.205755+0000","last_clean":"2024-07-26T10:58:45.205755+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205755+0000","last_undegraded":"2024-07-26T10:58:45.205755+0000","last_fullsized":"2024-07-26T10:58:45.205755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942853+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T11:00:12.942853+0000","last_peered":"2024-07-26T11:00:12.942853+0000","last_clean":"2024-07-26T11:00:12.942853+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T11:00:12.942853+0000","last_undegraded":"2024-07-26T11:00:12.942853+0000","last_fullsized":"2024-07-26T11:00:12.942853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946199+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T11:00:12.946199+0000","last_peered":"2024-07-26T11:00:12.946199+0000","last_clean":"2024-07-26T11:00:12.946199+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T11:00:12.946199+0000","last_undegraded":"2024-07-26T11:00:12.946199+0000","last_fullsized":"2024-07-26T11:00:12.946199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.975244+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T11:00:12.975244+0000","last_peered":"2024-07-26T11:00:12.975244+0000","last_clean":"2024-07-26T11:00:12.975244+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T11:00:12.975244+0000","last_undegraded":"2024-07-26T11:00:12.975244+0000","last_fullsized":"2024-07-26T11:00:12.975244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992785+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T11:00:12.992785+0000","last_peered":"2024-07-26T11:00:12.992785+0000","last_clean":"2024-07-26T11:00:12.992785+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T11:00:12.992785+0000","last_undegraded":"2024-07-26T11:00:12.992785+0000","last_fullsized":"2024-07-26T11:00:12.992785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.996976+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T11:00:12.996976+0000","last_peered":"2024-07-26T11:00:12.996976+0000","last_clean":"2024-07-26T11:00:12.996976+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T11:00:12.996976+0000","last_undegraded":"2024-07-26T11:00:12.996976+0000","last_fullsized":"2024-07-26T11:00:12.996976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.940323+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T11:00:12.940323+0000","last_peered":"2024-07-26T11:00:12.940323+0000","last_clean":"2024-07-26T11:00:12.940323+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T11:00:12.940323+0000","last_undegraded":"2024-07-26T11:00:12.940323+0000","last_fullsized":"2024-07-26T11:00:12.940323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942010+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T11:00:12.942010+0000","last_peered":"2024-07-26T11:00:12.942010+0000","last_clean":"2024-07-26T11:00:12.942010+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T11:00:12.942010+0000","last_undegraded":"2024-07-26T11:00:12.942010+0000","last_fullsized":"2024-07-26T11:00:12.942010+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.973405+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T11:00:12.973405+0000","last_peered":"2024-07-26T11:00:12.973405+0000","last_clean":"2024-07-26T11:00:12.973405+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T11:00:12.973405+0000","last_undegraded":"2024-07-26T11:00:12.973405+0000","last_fullsized":"2024-07-26T11:00:12.973405+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967597+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T11:00:12.967597+0000","last_peered":"2024-07-26T11:00:12.967597+0000","last_clean":"2024-07-26T11:00:12.967597+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T11:00:12.967597+0000","last_undegraded":"2024-07-26T11:00:12.967597+0000","last_fullsized":"2024-07-26T11:00:12.967597+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982036+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T11:00:12.982036+0000","last_peered":"2024-07-26T11:00:12.982036+0000","last_clean":"2024-07-26T11:00:12.982036+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T11:00:12.982036+0000","last_undegraded":"2024-07-26T11:00:12.982036+0000","last_fullsized":"2024-07-26T11:00:12.982036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981299+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T11:00:12.981299+0000","last_peered":"2024-07-26T11:00:12.981299+0000","last_clean":"2024-07-26T11:00:12.981299+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T11:00:12.981299+0000","last_undegraded":"2024-07-26T11:00:12.981299+0000","last_fullsized":"2024-07-26T11:00:12.981299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.999210+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T11:00:12.999210+0000","last_peered":"2024-07-26T11:00:12.999210+0000","last_clean":"2024-07-26T11:00:12.999210+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T11:00:12.999210+0000","last_undegraded":"2024-07-26T11:00:12.999210+0000","last_fullsized":"2024-07-26T11:00:12.999210+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946804+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T11:00:12.946804+0000","last_peered":"2024-07-26T11:00:12.946804+0000","last_clean":"2024-07-26T11:00:12.946804+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T11:00:12.946804+0000","last_undegraded":"2024-07-26T11:00:12.946804+0000","last_fullsized":"2024-07-26T11:00:12.946804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947300+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T11:00:12.947300+0000","last_peered":"2024-07-26T11:00:12.947300+0000","last_clean":"2024-07-26T11:00:12.947300+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T11:00:12.947300+0000","last_undegraded":"2024-07-26T11:00:12.947300+0000","last_fullsized":"2024-07-26T11:00:12.947300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945515+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T11:00:12.945515+0000","last_peered":"2024-07-26T11:00:12.945515+0000","last_clean":"2024-07-26T11:00:12.945515+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T11:00:12.945515+0000","last_undegraded":"2024-07-26T11:00:12.945515+0000","last_fullsized":"2024-07-26T11:00:12.945515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974957+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T11:00:12.974957+0000","last_peered":"2024-07-26T11:00:12.974957+0000","last_clean":"2024-07-26T11:00:12.974957+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T11:00:12.974957+0000","last_undegraded":"2024-07-26T11:00:12.974957+0000","last_fullsized":"2024-07-26T11:00:12.974957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992896+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T11:00:12.992896+0000","last_peered":"2024-07-26T11:00:12.992896+0000","last_clean":"2024-07-26T11:00:12.992896+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T11:00:12.992896+0000","last_undegraded":"2024-07-26T11:00:12.992896+0000","last_fullsized":"2024-07-26T11:00:12.992896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992111+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T11:00:12.992111+0000","last_peered":"2024-07-26T11:00:12.992111+0000","last_clean":"2024-07-26T11:00:12.992111+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T11:00:12.992111+0000","last_undegraded":"2024-07-26T11:00:12.992111+0000","last_fullsized":"2024-07-26T11:00:12.992111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994447+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T11:00:12.994447+0000","last_peered":"2024-07-26T11:00:12.994447+0000","last_clean":"2024-07-26T11:00:12.994447+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T11:00:12.994447+0000","last_undegraded":"2024-07-26T11:00:12.994447+0000","last_fullsized":"2024-07-26T11:00:12.994447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988020+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T11:00:12.988020+0000","last_peered":"2024-07-26T11:00:12.988020+0000","last_clean":"2024-07-26T11:00:12.988020+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T11:00:12.988020+0000","last_undegraded":"2024-07-26T11:00:12.988020+0000","last_fullsized":"2024-07-26T11:00:12.988020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951958+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T11:00:12.951958+0000","last_peered":"2024-07-26T11:00:12.951958+0000","last_clean":"2024-07-26T11:00:12.951958+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T11:00:12.951958+0000","last_undegraded":"2024-07-26T11:00:12.951958+0000","last_fullsized":"2024-07-26T11:00:12.951958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955435+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T11:00:12.955435+0000","last_peered":"2024-07-26T11:00:12.955435+0000","last_clean":"2024-07-26T11:00:12.955435+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T11:00:12.955435+0000","last_undegraded":"2024-07-26T11:00:12.955435+0000","last_fullsized":"2024-07-26T11:00:12.955435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.948784+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T11:00:12.948784+0000","last_peered":"2024-07-26T11:00:12.948784+0000","last_clean":"2024-07-26T11:00:12.948784+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T11:00:12.948784+0000","last_undegraded":"2024-07-26T11:00:12.948784+0000","last_fullsized":"2024-07-26T11:00:12.948784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946071+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T11:00:12.946071+0000","last_peered":"2024-07-26T11:00:12.946071+0000","last_clean":"2024-07-26T11:00:12.946071+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T11:00:12.946071+0000","last_undegraded":"2024-07-26T11:00:12.946071+0000","last_fullsized":"2024-07-26T11:00:12.946071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981175+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T11:00:12.981175+0000","last_peered":"2024-07-26T11:00:12.981175+0000","last_clean":"2024-07-26T11:00:12.981175+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T11:00:12.981175+0000","last_undegraded":"2024-07-26T11:00:12.981175+0000","last_fullsized":"2024-07-26T11:00:12.981175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952874+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T11:00:12.952874+0000","last_peered":"2024-07-26T11:00:12.952874+0000","last_clean":"2024-07-26T11:00:12.952874+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T11:00:12.952874+0000","last_undegraded":"2024-07-26T11:00:12.952874+0000","last_fullsized":"2024-07-26T11:00:12.952874+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947037+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T11:00:12.947037+0000","last_peered":"2024-07-26T11:00:12.947037+0000","last_clean":"2024-07-26T11:00:12.947037+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T11:00:12.947037+0000","last_undegraded":"2024-07-26T11:00:12.947037+0000","last_fullsized":"2024-07-26T11:00:12.947037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967730+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T11:00:12.967730+0000","last_peered":"2024-07-26T11:00:12.967730+0000","last_clean":"2024-07-26T11:00:12.967730+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T11:00:12.967730+0000","last_undegraded":"2024-07-26T11:00:12.967730+0000","last_fullsized":"2024-07-26T11:00:12.967730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.953352+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T11:00:12.953352+0000","last_peered":"2024-07-26T11:00:12.953352+0000","last_clean":"2024-07-26T11:00:12.953352+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T11:00:12.953352+0000","last_undegraded":"2024-07-26T11:00:12.953352+0000","last_fullsized":"2024-07-26T11:00:12.953352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989600+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T11:00:12.989600+0000","last_peered":"2024-07-26T11:00:12.989600+0000","last_clean":"2024-07-26T11:00:12.989600+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T11:00:12.989600+0000","last_undegraded":"2024-07-26T11:00:12.989600+0000","last_fullsized":"2024-07-26T11:00:12.989600+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.990190+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T11:00:12.990190+0000","last_peered":"2024-07-26T11:00:12.990190+0000","last_clean":"2024-07-26T11:00:12.990190+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T11:00:12.990190+0000","last_undegraded":"2024-07-26T11:00:12.990190+0000","last_fullsized":"2024-07-26T11:00:12.990190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974796+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T11:00:12.974796+0000","last_peered":"2024-07-26T11:00:12.974796+0000","last_clean":"2024-07-26T11:00:12.974796+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T11:00:12.974796+0000","last_undegraded":"2024-07-26T11:00:12.974796+0000","last_fullsized":"2024-07-26T11:00:12.974796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76996,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":1371,"num_read_kb":766,"num_write":186,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":230988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678093,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874323,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1203708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234078,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430306,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593832,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790062,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953587,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378750,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":603766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542276,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":603967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":73275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:02:28.133 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-07-26T11:02:28.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T11:02:28.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 -- 172.21.15.144:0/2021623723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 msgr2=0x7f88d010b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 --2- 172.21.15.144:0/2021623723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 0x7f88d010b270 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009900 tx=0x7f88b802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 -- 172.21.15.144:0/2021623723 shutdown_connections 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 --2- 172.21.15.144:0/2021623723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d010b7b0 0x7f88d0111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 --2- 172.21.15.144:0/2021623723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 0x7f88d010b270 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 --2- 172.21.15.144:0/2021623723 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d0105530 0x7f88d0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.438+0000 7f88cf200640 1 -- 172.21.15.144:0/2021623723 >> 172.21.15.144:0/2021623723 conn(0x7f88d0100f90 msgr2=0x7f88d01033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.439+0000 7f88cf200640 1 -- 172.21.15.144:0/2021623723 shutdown_connections 2024-07-26T11:02:28.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.439+0000 7f88cf200640 1 -- 172.21.15.144:0/2021623723 wait complete. 2024-07-26T11:02:28.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 Processor -- start 2024-07-26T11:02:28.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 -- start start 2024-07-26T11:02:28.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 0x7f88d019bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:28.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 0x7f88d019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d010b7b0 0x7f88d01a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0115440 con 0x7f88d010ae60 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f88d01152c0 con 0x7f88d0105530 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cf200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f88d01155c0 con 0x7f88d010b7b0 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 0x7f88d019bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 0x7f88d019bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47240/0 (socket says 172.21.15.144:47240) 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88cde00640 1 -- 172.21.15.144:0/2235327915 learned_addr learned my addr 172.21.15.144:0/2235327915 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.440+0000 7f88c7e00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 0x7f88d019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88ce800640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d010b7b0 0x7f88d01a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88cde00640 1 -- 172.21.15.144:0/2235327915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d010b7b0 msgr2=0x7f88d01a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88cde00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d010b7b0 0x7f88d01a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88cde00640 1 -- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 msgr2=0x7f88d019c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88cde00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 0x7f88d019c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88cde00640 1 -- 172.21.15.144:0/2235327915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d01a1dd0 con 0x7f88d0105530 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88cde00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 0x7f88d019bea0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f88c000e9f0 tx=0x7f88c000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.441+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c000cd80 con 0x7f88d0105530 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.442+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d01a20c0 con 0x7f88d0105530 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.442+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88c00054d0 con 0x7f88d0105530 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.442+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0010690 con 0x7f88d0105530 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.442+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d01a8850 con 0x7f88d0105530 2024-07-26T11:02:28.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.443+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d0105940 con 0x7f88d0105530 2024-07-26T11:02:28.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.445+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f88c0004e90 con 0x7f88d0105530 2024-07-26T11:02:28.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.446+0000 7f88c6a00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f88b40b3c20 0x7f88b40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:28.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.447+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f88c00e46f0 con 0x7f88d0105530 2024-07-26T11:02:28.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.447+0000 7f88c7e00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f88b40b3c20 0x7f88b40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:28.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.448+0000 7f88c7e00640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f88b40b3c20 0x7f88b40b6130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009230 tx=0x7f88b8031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:28.450 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.448+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f88c00ab850 con 0x7f88d0105530 2024-07-26T11:02:28.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:28 smithi160 ceph-mon[36715]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:28.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:28 smithi160 ceph-mon[36715]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:28 smithi144 ceph-mon[32129]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:28 smithi144 ceph-mon[32129]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:28 smithi183 ceph-mon[37273]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:28 smithi183 ceph-mon[37273]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:28.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.652+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f88d0063230 con 0x7f88b40b3c20 2024-07-26T11:02:28.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.669+0000 7f88c6a00640 1 -- 172.21.15.144:0/2235327915 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340481 (secure 0 0 0) 0x7f88d0063230 con 0x7f88b40b3c20 2024-07-26T11:02:28.677 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:28.679 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-07-26T11:02:28.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f88b40b3c20 msgr2=0x7f88b40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f88b40b3c20 0x7f88b40b6130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009230 tx=0x7f88b8031400 comp rx=0 tx=0).stop 2024-07-26T11:02:28.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 msgr2=0x7f88d019bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:28.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 0x7f88d019bea0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f88c000e9f0 tx=0x7f88c000edf0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 shutdown_connections 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f88b40b3c20 0x7f88b40b6130 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d010b7b0 0x7f88d01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d010ae60 0x7f88d019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 --2- 172.21.15.144:0/2235327915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88d0105530 0x7f88d019bea0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 >> 172.21.15.144:0/2235327915 conn(0x7f88d0100f90 msgr2=0x7f88d01033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.678+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 shutdown_connections 2024-07-26T11:02:28.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:28.679+0000 7f88cf200640 1 -- 172.21.15.144:0/2235327915 wait complete. 2024-07-26T11:02:28.706 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-07-26T11:02:27.115339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667598,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1452,"num_read_kb":835,"num_write":311,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":752,"ondisk_log_size":752,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380104,"kb_used_data":14312,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561720,"statfs":{"total":1151940427776,"available":1151551201280,"internally_reserved":0,"allocated":14655488,"data_stored":8400752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":47,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005738"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:25.593785+0000","last_change":"2024-07-26T10:57:44.915734+0000","last_active":"2024-07-26T10:59:25.593785+0000","last_peered":"2024-07-26T10:59:25.593785+0000","last_clean":"2024-07-26T10:59:25.593785+0000","last_became_active":"2024-07-26T10:57:44.907160+0000","last_became_peered":"2024-07-26T10:57:44.907160+0000","last_unstale":"2024-07-26T10:59:25.593785+0000","last_undegraded":"2024-07-26T10:59:25.593785+0000","last_fullsized":"2024-07-26T10:59:25.593785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:34.904797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969791+0000","last_change":"2024-07-26T10:57:22.108925+0000","last_active":"2024-07-26T11:00:12.969791+0000","last_peered":"2024-07-26T11:00:12.969791+0000","last_clean":"2024-07-26T11:00:12.969791+0000","last_became_active":"2024-07-26T10:57:22.107620+0000","last_became_peered":"2024-07-26T10:57:22.107620+0000","last_unstale":"2024-07-26T11:00:12.969791+0000","last_undegraded":"2024-07-26T11:00:12.969791+0000","last_fullsized":"2024-07-26T11:00:12.969791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:42:38.950691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:59.988270+0000","last_change":"2024-07-26T10:57:35.091946+0000","last_active":"2024-07-26T10:59:59.988270+0000","last_peered":"2024-07-26T10:59:59.988270+0000","last_clean":"2024-07-26T10:59:59.988270+0000","last_became_active":"2024-07-26T10:57:35.091490+0000","last_became_peered":"2024-07-26T10:57:35.091490+0000","last_unstale":"2024-07-26T10:59:59.988270+0000","last_undegraded":"2024-07-26T10:59:59.988270+0000","last_fullsized":"2024-07-26T10:59:59.988270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:45:27.995398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:20.900477+0000","last_change":"2024-07-26T10:57:18.153719+0000","last_active":"2024-07-26T10:59:20.900477+0000","last_peered":"2024-07-26T10:59:20.900477+0000","last_clean":"2024-07-26T10:59:20.900477+0000","last_became_active":"2024-07-26T10:57:18.152437+0000","last_became_peered":"2024-07-26T10:57:18.152437+0000","last_unstale":"2024-07-26T10:59:20.900477+0000","last_undegraded":"2024-07-26T10:59:20.900477+0000","last_fullsized":"2024-07-26T10:59:20.900477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:56.909228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.995989+0000","last_change":"2024-07-26T10:57:22.120702+0000","last_active":"2024-07-26T11:00:12.995989+0000","last_peered":"2024-07-26T11:00:12.995989+0000","last_clean":"2024-07-26T11:00:12.995989+0000","last_became_active":"2024-07-26T10:57:22.120343+0000","last_became_peered":"2024-07-26T10:57:22.120343+0000","last_unstale":"2024-07-26T11:00:12.995989+0000","last_undegraded":"2024-07-26T11:00:12.995989+0000","last_fullsized":"2024-07-26T11:00:12.995989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:42.229622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:01.711358+0000","last_change":"2024-07-26T10:57:35.080653+0000","last_active":"2024-07-26T11:00:01.711358+0000","last_peered":"2024-07-26T11:00:01.711358+0000","last_clean":"2024-07-26T11:00:01.711358+0000","last_became_active":"2024-07-26T10:57:35.080154+0000","last_became_peered":"2024-07-26T10:57:35.080154+0000","last_unstale":"2024-07-26T11:00:01.711358+0000","last_undegraded":"2024-07-26T11:00:01.711358+0000","last_fullsized":"2024-07-26T11:00:01.711358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:56.717497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.235288+0000","last_change":"2024-07-26T10:57:18.055661+0000","last_active":"2024-07-26T10:59:51.235288+0000","last_peered":"2024-07-26T10:59:51.235288+0000","last_clean":"2024-07-26T10:59:51.235288+0000","last_became_active":"2024-07-26T10:57:18.055277+0000","last_became_peered":"2024-07-26T10:57:18.055277+0000","last_unstale":"2024-07-26T10:59:51.235288+0000","last_undegraded":"2024-07-26T10:59:51.235288+0000","last_fullsized":"2024-07-26T10:59:51.235288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:16.418632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986654+0000","last_change":"2024-07-26T10:57:22.129015+0000","last_active":"2024-07-26T11:00:12.986654+0000","last_peered":"2024-07-26T11:00:12.986654+0000","last_clean":"2024-07-26T11:00:12.986654+0000","last_became_active":"2024-07-26T10:57:22.128576+0000","last_became_peered":"2024-07-26T10:57:22.128576+0000","last_unstale":"2024-07-26T11:00:12.986654+0000","last_undegraded":"2024-07-26T11:00:12.986654+0000","last_fullsized":"2024-07-26T11:00:12.986654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:17:19.676320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212336+0000","last_change":"2024-07-26T10:57:35.080823+0000","last_active":"2024-07-26T10:58:45.212336+0000","last_peered":"2024-07-26T10:58:45.212336+0000","last_clean":"2024-07-26T10:58:45.212336+0000","last_became_active":"2024-07-26T10:57:35.079423+0000","last_became_peered":"2024-07-26T10:57:35.079423+0000","last_unstale":"2024-07-26T10:58:45.212336+0000","last_undegraded":"2024-07-26T10:58:45.212336+0000","last_fullsized":"2024-07-26T10:58:45.212336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:01.122193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395309+0000","last_change":"2024-07-26T10:57:44.893728+0000","last_active":"2024-07-26T10:58:45.395309+0000","last_peered":"2024-07-26T10:58:45.395309+0000","last_clean":"2024-07-26T10:58:45.395309+0000","last_became_active":"2024-07-26T10:57:44.888764+0000","last_became_peered":"2024-07-26T10:57:44.888764+0000","last_unstale":"2024-07-26T10:58:45.395309+0000","last_undegraded":"2024-07-26T10:58:45.395309+0000","last_fullsized":"2024-07-26T10:58:45.395309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:40.155037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952745+0000","last_change":"2024-07-26T10:57:22.114066+0000","last_active":"2024-07-26T11:00:12.952745+0000","last_peered":"2024-07-26T11:00:12.952745+0000","last_clean":"2024-07-26T11:00:12.952745+0000","last_became_active":"2024-07-26T10:57:22.113786+0000","last_became_peered":"2024-07-26T10:57:22.113786+0000","last_unstale":"2024-07-26T11:00:12.952745+0000","last_undegraded":"2024-07-26T11:00:12.952745+0000","last_fullsized":"2024-07-26T11:00:12.952745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:30.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.212134+0000","last_change":"2024-07-26T10:57:35.071494+0000","last_active":"2024-07-26T10:58:45.212134+0000","last_peered":"2024-07-26T10:58:45.212134+0000","last_clean":"2024-07-26T10:58:45.212134+0000","last_became_active":"2024-07-26T10:57:35.071039+0000","last_became_peered":"2024-07-26T10:57:35.071039+0000","last_unstale":"2024-07-26T10:58:45.212134+0000","last_undegraded":"2024-07-26T10:58:45.212134+0000","last_fullsized":"2024-07-26T10:58:45.212134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:27.688117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.969821+0000","last_change":"2024-07-26T10:57:22.126431+0000","last_active":"2024-07-26T11:00:12.969821+0000","last_peered":"2024-07-26T11:00:12.969821+0000","last_clean":"2024-07-26T11:00:12.969821+0000","last_became_active":"2024-07-26T10:57:22.126023+0000","last_became_peered":"2024-07-26T10:57:22.126023+0000","last_unstale":"2024-07-26T11:00:12.969821+0000","last_undegraded":"2024-07-26T11:00:12.969821+0000","last_fullsized":"2024-07-26T11:00:12.969821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:33.688774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:36.233647+0000","last_change":"2024-07-26T10:57:18.054859+0000","last_active":"2024-07-26T10:59:36.233647+0000","last_peered":"2024-07-26T10:59:36.233647+0000","last_clean":"2024-07-26T10:59:36.233647+0000","last_became_active":"2024-07-26T10:57:18.054269+0000","last_became_peered":"2024-07-26T10:57:18.054269+0000","last_unstale":"2024-07-26T10:59:36.233647+0000","last_undegraded":"2024-07-26T10:59:36.233647+0000","last_fullsized":"2024-07-26T10:59:36.233647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:58:16.243396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710263+0000","last_change":"2024-07-26T10:57:35.101123+0000","last_active":"2024-07-26T10:59:51.710263+0000","last_peered":"2024-07-26T10:59:51.710263+0000","last_clean":"2024-07-26T10:59:51.710263+0000","last_became_active":"2024-07-26T10:57:35.100569+0000","last_became_peered":"2024-07-26T10:57:35.100569+0000","last_unstale":"2024-07-26T10:59:51.710263+0000","last_undegraded":"2024-07-26T10:59:51.710263+0000","last_fullsized":"2024-07-26T10:59:51.710263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:31:03.772956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.997712+0000","last_change":"2024-07-26T10:57:22.109422+0000","last_active":"2024-07-26T11:00:12.997712+0000","last_peered":"2024-07-26T11:00:12.997712+0000","last_clean":"2024-07-26T11:00:12.997712+0000","last_became_active":"2024-07-26T10:57:22.109030+0000","last_became_peered":"2024-07-26T10:57:22.109030+0000","last_unstale":"2024-07-26T11:00:12.997712+0000","last_undegraded":"2024-07-26T11:00:12.997712+0000","last_fullsized":"2024-07-26T11:00:12.997712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:39:48.234466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:01:40.000319+0000","last_change":"2024-07-26T10:57:18.057206+0000","last_active":"2024-07-26T11:01:40.000319+0000","last_peered":"2024-07-26T11:01:40.000319+0000","last_clean":"2024-07-26T11:01:40.000319+0000","last_became_active":"2024-07-26T10:57:18.056853+0000","last_became_peered":"2024-07-26T10:57:18.056853+0000","last_unstale":"2024-07-26T11:01:40.000319+0000","last_undegraded":"2024-07-26T11:01:40.000319+0000","last_fullsized":"2024-07-26T11:01:40.000319+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:28.852495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710222+0000","last_change":"2024-07-26T10:58:45.398106+0000","last_active":"2024-07-26T10:59:51.710222+0000","last_peered":"2024-07-26T10:59:51.710222+0000","last_clean":"2024-07-26T10:59:51.710222+0000","last_became_active":"2024-07-26T10:58:45.397440+0000","last_became_peered":"2024-07-26T10:58:45.397440+0000","last_unstale":"2024-07-26T10:59:51.710222+0000","last_undegraded":"2024-07-26T10:59:51.710222+0000","last_fullsized":"2024-07-26T10:59:51.710222+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:17.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.970711+0000","last_change":"2024-07-26T10:57:22.099778+0000","last_active":"2024-07-26T11:00:12.970711+0000","last_peered":"2024-07-26T11:00:12.970711+0000","last_clean":"2024-07-26T11:00:12.970711+0000","last_became_active":"2024-07-26T10:57:22.099322+0000","last_became_peered":"2024-07-26T10:57:22.099322+0000","last_unstale":"2024-07-26T11:00:12.970711+0000","last_undegraded":"2024-07-26T11:00:12.970711+0000","last_fullsized":"2024-07-26T11:00:12.970711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:45.455632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:30.902021+0000","last_change":"2024-07-26T10:57:18.056088+0000","last_active":"2024-07-26T10:59:30.902021+0000","last_peered":"2024-07-26T10:59:30.902021+0000","last_clean":"2024-07-26T10:59:30.902021+0000","last_became_active":"2024-07-26T10:57:18.055647+0000","last_became_peered":"2024-07-26T10:57:18.055647+0000","last_unstale":"2024-07-26T10:59:30.902021+0000","last_undegraded":"2024-07-26T10:59:30.902021+0000","last_fullsized":"2024-07-26T10:59:30.902021+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:52.230712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:01.236531+0000","last_change":"2024-07-26T10:57:45.408405+0000","last_active":"2024-07-26T11:00:01.236531+0000","last_peered":"2024-07-26T11:00:01.236531+0000","last_clean":"2024-07-26T11:00:01.236531+0000","last_became_active":"2024-07-26T10:57:45.407791+0000","last_became_peered":"2024-07-26T10:57:45.407791+0000","last_unstale":"2024-07-26T11:00:01.236531+0000","last_undegraded":"2024-07-26T11:00:01.236531+0000","last_fullsized":"2024-07-26T11:00:01.236531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:53.236560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.935240+0000","last_change":"2024-07-26T10:57:22.104514+0000","last_active":"2024-07-26T11:00:12.935240+0000","last_peered":"2024-07-26T11:00:12.935240+0000","last_clean":"2024-07-26T11:00:12.935240+0000","last_became_active":"2024-07-26T10:57:22.104053+0000","last_became_peered":"2024-07-26T10:57:22.104053+0000","last_unstale":"2024-07-26T11:00:12.935240+0000","last_undegraded":"2024-07-26T11:00:12.935240+0000","last_fullsized":"2024-07-26T11:00:12.935240+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:42.837057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:28.660005+0000","last_change":"2024-07-26T10:57:18.152640+0000","last_active":"2024-07-26T10:59:28.660005+0000","last_peered":"2024-07-26T10:59:28.660005+0000","last_clean":"2024-07-26T10:59:28.660005+0000","last_became_active":"2024-07-26T10:57:18.152307+0000","last_became_peered":"2024-07-26T10:57:18.152307+0000","last_unstale":"2024-07-26T10:59:28.660005+0000","last_undegraded":"2024-07-26T10:59:28.660005+0000","last_fullsized":"2024-07-26T10:59:28.660005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:24.376499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:51.710276+0000","last_change":"2024-07-26T10:57:35.101387+0000","last_active":"2024-07-26T10:59:51.710276+0000","last_peered":"2024-07-26T10:59:51.710276+0000","last_clean":"2024-07-26T10:59:51.710276+0000","last_became_active":"2024-07-26T10:57:35.100618+0000","last_became_peered":"2024-07-26T10:57:35.100618+0000","last_unstale":"2024-07-26T10:59:51.710276+0000","last_undegraded":"2024-07-26T10:59:51.710276+0000","last_fullsized":"2024-07-26T10:59:51.710276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:52.277636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994606+0000","last_change":"2024-07-26T10:57:22.109075+0000","last_active":"2024-07-26T11:00:12.994606+0000","last_peered":"2024-07-26T11:00:12.994606+0000","last_clean":"2024-07-26T11:00:12.994606+0000","last_became_active":"2024-07-26T10:57:22.108664+0000","last_became_peered":"2024-07-26T10:57:22.108664+0000","last_unstale":"2024-07-26T11:00:12.994606+0000","last_undegraded":"2024-07-26T11:00:12.994606+0000","last_fullsized":"2024-07-26T11:00:12.994606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:00:50.874607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:31.707610+0000","last_change":"2024-07-26T10:57:18.120765+0000","last_active":"2024-07-26T10:59:31.707610+0000","last_peered":"2024-07-26T10:59:31.707610+0000","last_clean":"2024-07-26T10:59:31.707610+0000","last_became_active":"2024-07-26T10:57:18.120434+0000","last_became_peered":"2024-07-26T10:57:18.120434+0000","last_unstale":"2024-07-26T10:59:31.707610+0000","last_undegraded":"2024-07-26T10:59:31.707610+0000","last_fullsized":"2024-07-26T10:59:31.707610+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:12.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.985686+0000","last_change":"2024-07-26T10:57:35.060922+0000","last_active":"2024-07-26T10:59:39.985686+0000","last_peered":"2024-07-26T10:59:39.985686+0000","last_clean":"2024-07-26T10:59:39.985686+0000","last_became_active":"2024-07-26T10:57:35.060451+0000","last_became_peered":"2024-07-26T10:57:35.060451+0000","last_unstale":"2024-07-26T10:59:39.985686+0000","last_undegraded":"2024-07-26T10:59:39.985686+0000","last_fullsized":"2024-07-26T10:59:39.985686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:26.988695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982250+0000","last_change":"2024-07-26T10:57:22.118382+0000","last_active":"2024-07-26T11:00:12.982250+0000","last_peered":"2024-07-26T11:00:12.982250+0000","last_clean":"2024-07-26T11:00:12.982250+0000","last_became_active":"2024-07-26T10:57:22.117995+0000","last_became_peered":"2024-07-26T10:57:22.117995+0000","last_unstale":"2024-07-26T11:00:12.982250+0000","last_undegraded":"2024-07-26T11:00:12.982250+0000","last_fullsized":"2024-07-26T11:00:12.982250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:11.315954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:01:40.617170+0000","last_change":"2024-07-26T10:57:18.058151+0000","last_active":"2024-07-26T11:01:40.617170+0000","last_peered":"2024-07-26T11:01:40.617170+0000","last_clean":"2024-07-26T11:01:40.617170+0000","last_became_active":"2024-07-26T10:57:18.057732+0000","last_became_peered":"2024-07-26T10:57:18.057732+0000","last_unstale":"2024-07-26T11:01:40.617170+0000","last_undegraded":"2024-07-26T11:01:40.617170+0000","last_fullsized":"2024-07-26T11:01:40.617170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:04:57.042753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:35.903083+0000","last_change":"2024-07-26T10:57:35.100290+0000","last_active":"2024-07-26T10:59:35.903083+0000","last_peered":"2024-07-26T10:59:35.903083+0000","last_clean":"2024-07-26T10:59:35.903083+0000","last_became_active":"2024-07-26T10:57:35.099665+0000","last_became_peered":"2024-07-26T10:57:35.099665+0000","last_unstale":"2024-07-26T10:59:35.903083+0000","last_undegraded":"2024-07-26T10:59:35.903083+0000","last_fullsized":"2024-07-26T10:59:35.903083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:15:25.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988263+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.988263+0000","last_peered":"2024-07-26T11:00:12.988263+0000","last_clean":"2024-07-26T11:00:12.988263+0000","last_became_active":"2024-07-26T10:57:22.126008+0000","last_became_peered":"2024-07-26T10:57:22.126008+0000","last_unstale":"2024-07-26T11:00:12.988263+0000","last_undegraded":"2024-07-26T11:00:12.988263+0000","last_fullsized":"2024-07-26T11:00:12.988263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:18.158166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:26.232295+0000","last_change":"2024-07-26T10:57:18.151358+0000","last_active":"2024-07-26T10:59:26.232295+0000","last_peered":"2024-07-26T10:59:26.232295+0000","last_clean":"2024-07-26T10:59:26.232295+0000","last_became_active":"2024-07-26T10:57:18.150914+0000","last_became_peered":"2024-07-26T10:57:18.150914+0000","last_unstale":"2024-07-26T10:59:26.232295+0000","last_undegraded":"2024-07-26T10:59:26.232295+0000","last_fullsized":"2024-07-26T10:59:26.232295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:29.147659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:40.717767+0000","last_change":"2024-07-26T10:57:35.079887+0000","last_active":"2024-07-26T10:59:40.717767+0000","last_peered":"2024-07-26T10:59:40.717767+0000","last_clean":"2024-07-26T10:59:40.717767+0000","last_became_active":"2024-07-26T10:57:35.079356+0000","last_became_peered":"2024-07-26T10:57:35.079356+0000","last_unstale":"2024-07-26T10:59:40.717767+0000","last_undegraded":"2024-07-26T10:59:40.717767+0000","last_fullsized":"2024-07-26T10:59:40.717767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.758884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980610+0000","last_change":"2024-07-26T10:57:22.126495+0000","last_active":"2024-07-26T11:00:12.980610+0000","last_peered":"2024-07-26T11:00:12.980610+0000","last_clean":"2024-07-26T11:00:12.980610+0000","last_became_active":"2024-07-26T10:57:22.126111+0000","last_became_peered":"2024-07-26T10:57:22.126111+0000","last_unstale":"2024-07-26T11:00:12.980610+0000","last_undegraded":"2024-07-26T11:00:12.980610+0000","last_fullsized":"2024-07-26T11:00:12.980610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:36:29.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:25.593738+0000","last_change":"2024-07-26T10:57:44.910873+0000","last_active":"2024-07-26T10:59:25.593738+0000","last_peered":"2024-07-26T10:59:25.593738+0000","last_clean":"2024-07-26T10:59:25.593738+0000","last_became_active":"2024-07-26T10:57:44.909540+0000","last_became_peered":"2024-07-26T10:57:44.909540+0000","last_unstale":"2024-07-26T10:59:25.593738+0000","last_undegraded":"2024-07-26T10:59:25.593738+0000","last_fullsized":"2024-07-26T10:59:25.593738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:49.425256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:41.234222+0000","last_change":"2024-07-26T10:57:35.100439+0000","last_active":"2024-07-26T10:59:41.234222+0000","last_peered":"2024-07-26T10:59:41.234222+0000","last_clean":"2024-07-26T10:59:41.234222+0000","last_became_active":"2024-07-26T10:57:35.099941+0000","last_became_peered":"2024-07-26T10:57:35.099941+0000","last_unstale":"2024-07-26T10:59:41.234222+0000","last_undegraded":"2024-07-26T10:59:41.234222+0000","last_fullsized":"2024-07-26T10:59:41.234222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:14.907089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.976216+0000","last_change":"2024-07-26T10:57:22.108106+0000","last_active":"2024-07-26T11:00:12.976216+0000","last_peered":"2024-07-26T11:00:12.976216+0000","last_clean":"2024-07-26T11:00:12.976216+0000","last_became_active":"2024-07-26T10:57:22.107611+0000","last_became_peered":"2024-07-26T10:57:22.107611+0000","last_unstale":"2024-07-26T11:00:12.976216+0000","last_undegraded":"2024-07-26T11:00:12.976216+0000","last_fullsized":"2024-07-26T11:00:12.976216+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:32.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:22.629190+0000","last_change":"2024-07-26T10:57:18.050213+0000","last_active":"2024-07-26T10:59:22.629190+0000","last_peered":"2024-07-26T10:59:22.629190+0000","last_clean":"2024-07-26T10:59:22.629190+0000","last_became_active":"2024-07-26T10:57:18.049883+0000","last_became_peered":"2024-07-26T10:57:18.049883+0000","last_unstale":"2024-07-26T10:59:22.629190+0000","last_undegraded":"2024-07-26T10:59:22.629190+0000","last_fullsized":"2024-07-26T10:59:22.629190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:34.981967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:43.045581+0000","last_change":"2024-07-26T10:57:35.082151+0000","last_active":"2024-07-26T10:59:43.045581+0000","last_peered":"2024-07-26T10:59:43.045581+0000","last_clean":"2024-07-26T10:59:43.045581+0000","last_became_active":"2024-07-26T10:57:35.081719+0000","last_became_peered":"2024-07-26T10:57:35.081719+0000","last_unstale":"2024-07-26T10:59:43.045581+0000","last_undegraded":"2024-07-26T10:59:43.045581+0000","last_fullsized":"2024-07-26T10:59:43.045581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:36.117620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950189+0000","last_change":"2024-07-26T10:57:22.105605+0000","last_active":"2024-07-26T11:00:12.950189+0000","last_peered":"2024-07-26T11:00:12.950189+0000","last_clean":"2024-07-26T11:00:12.950189+0000","last_became_active":"2024-07-26T10:57:22.105167+0000","last_became_peered":"2024-07-26T10:57:22.105167+0000","last_unstale":"2024-07-26T11:00:12.950189+0000","last_undegraded":"2024-07-26T11:00:12.950189+0000","last_fullsized":"2024-07-26T11:00:12.950189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:29.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":272,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:21.700129+0000","last_change":"2024-07-26T10:57:18.154925+0000","last_active":"2024-07-26T11:02:21.700129+0000","last_peered":"2024-07-26T11:02:21.700129+0000","last_clean":"2024-07-26T11:02:21.700129+0000","last_became_active":"2024-07-26T10:57:18.153147+0000","last_became_peered":"2024-07-26T10:57:18.153147+0000","last_unstale":"2024-07-26T11:02:21.700129+0000","last_undegraded":"2024-07-26T11:02:21.700129+0000","last_fullsized":"2024-07-26T11:02:21.700129+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:21.421944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":221,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:48.046731+0000","last_change":"2024-07-26T10:57:35.102103+0000","last_active":"2024-07-26T10:59:48.046731+0000","last_peered":"2024-07-26T10:59:48.046731+0000","last_clean":"2024-07-26T10:59:48.046731+0000","last_became_active":"2024-07-26T10:57:35.100789+0000","last_became_peered":"2024-07-26T10:57:35.100789+0000","last_unstale":"2024-07-26T10:59:48.046731+0000","last_undegraded":"2024-07-26T10:59:48.046731+0000","last_fullsized":"2024-07-26T10:59:48.046731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:39.740819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.983164+0000","last_change":"2024-07-26T10:57:22.117999+0000","last_active":"2024-07-26T11:00:12.983164+0000","last_peered":"2024-07-26T11:00:12.983164+0000","last_clean":"2024-07-26T11:00:12.983164+0000","last_became_active":"2024-07-26T10:57:22.116567+0000","last_became_peered":"2024-07-26T10:57:22.116567+0000","last_unstale":"2024-07-26T11:00:12.983164+0000","last_undegraded":"2024-07-26T11:00:12.983164+0000","last_fullsized":"2024-07-26T11:00:12.983164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:25:31.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:17.628714+0000","last_change":"2024-07-26T10:57:18.051189+0000","last_active":"2024-07-26T10:59:17.628714+0000","last_peered":"2024-07-26T10:59:17.628714+0000","last_clean":"2024-07-26T10:59:17.628714+0000","last_became_active":"2024-07-26T10:57:18.050634+0000","last_became_peered":"2024-07-26T10:57:18.050634+0000","last_unstale":"2024-07-26T10:59:17.628714+0000","last_undegraded":"2024-07-26T10:59:17.628714+0000","last_fullsized":"2024-07-26T10:59:17.628714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:53.782054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:46.227347+0000","last_change":"2024-07-26T10:58:45.398874+0000","last_active":"2024-07-26T10:58:46.227347+0000","last_peered":"2024-07-26T10:58:46.227347+0000","last_clean":"2024-07-26T10:58:46.227347+0000","last_became_active":"2024-07-26T10:58:45.398375+0000","last_became_peered":"2024-07-26T10:58:45.398375+0000","last_unstale":"2024-07-26T10:58:46.227347+0000","last_undegraded":"2024-07-26T10:58:46.227347+0000","last_fullsized":"2024-07-26T10:58:46.227347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:26:36.880672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.971643+0000","last_change":"2024-07-26T10:57:22.124009+0000","last_active":"2024-07-26T11:00:12.971643+0000","last_peered":"2024-07-26T11:00:12.971643+0000","last_clean":"2024-07-26T11:00:12.971643+0000","last_became_active":"2024-07-26T10:57:22.121544+0000","last_became_peered":"2024-07-26T10:57:22.121544+0000","last_unstale":"2024-07-26T11:00:12.971643+0000","last_undegraded":"2024-07-26T11:00:12.971643+0000","last_fullsized":"2024-07-26T11:00:12.971643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:33.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:26.707005+0000","last_change":"2024-07-26T10:57:18.049714+0000","last_active":"2024-07-26T10:59:26.707005+0000","last_peered":"2024-07-26T10:59:26.707005+0000","last_clean":"2024-07-26T10:59:26.707005+0000","last_became_active":"2024-07-26T10:57:18.049266+0000","last_became_peered":"2024-07-26T10:57:18.049266+0000","last_unstale":"2024-07-26T10:59:26.707005+0000","last_undegraded":"2024-07-26T10:59:26.707005+0000","last_fullsized":"2024-07-26T10:59:26.707005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:41.599651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.210654+0000","last_change":"2024-07-26T10:55:46.819713+0000","last_active":"2024-07-26T10:58:45.210654+0000","last_peered":"2024-07-26T10:58:45.210654+0000","last_clean":"2024-07-26T10:58:45.210654+0000","last_became_active":"2024-07-26T10:55:46.814897+0000","last_became_peered":"2024-07-26T10:55:46.814897+0000","last_unstale":"2024-07-26T10:58:45.210654+0000","last_undegraded":"2024-07-26T10:58:45.210654+0000","last_fullsized":"2024-07-26T10:58:45.210654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:52:05.049324+0000","last_clean_scrub_stamp":"2024-07-26T10:52:05.049324+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:20:08.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:42.632233+0000","last_change":"2024-07-26T10:58:45.397868+0000","last_active":"2024-07-26T10:59:42.632233+0000","last_peered":"2024-07-26T10:59:42.632233+0000","last_clean":"2024-07-26T10:59:42.632233+0000","last_became_active":"2024-07-26T10:58:45.397232+0000","last_became_peered":"2024-07-26T10:58:45.397232+0000","last_unstale":"2024-07-26T10:59:42.632233+0000","last_undegraded":"2024-07-26T10:59:42.632233+0000","last_fullsized":"2024-07-26T10:59:42.632233+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:45.984154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955668+0000","last_change":"2024-07-26T10:57:22.109686+0000","last_active":"2024-07-26T11:00:12.955668+0000","last_peered":"2024-07-26T11:00:12.955668+0000","last_clean":"2024-07-26T11:00:12.955668+0000","last_became_active":"2024-07-26T10:57:22.109284+0000","last_became_peered":"2024-07-26T10:57:22.109284+0000","last_unstale":"2024-07-26T11:00:12.955668+0000","last_undegraded":"2024-07-26T11:00:12.955668+0000","last_fullsized":"2024-07-26T11:00:12.955668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:54.490988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:34.984986+0000","last_change":"2024-07-26T10:57:18.151247+0000","last_active":"2024-07-26T10:59:34.984986+0000","last_peered":"2024-07-26T10:59:34.984986+0000","last_clean":"2024-07-26T10:59:34.984986+0000","last_became_active":"2024-07-26T10:57:18.150880+0000","last_became_peered":"2024-07-26T10:57:18.150880+0000","last_unstale":"2024-07-26T10:59:34.984986+0000","last_undegraded":"2024-07-26T10:59:34.984986+0000","last_fullsized":"2024-07-26T10:59:34.984986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:05:51.382237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:48.662570+0000","last_change":"2024-07-26T10:57:35.079853+0000","last_active":"2024-07-26T10:59:48.662570+0000","last_peered":"2024-07-26T10:59:48.662570+0000","last_clean":"2024-07-26T10:59:48.662570+0000","last_became_active":"2024-07-26T10:57:35.079378+0000","last_became_peered":"2024-07-26T10:57:35.079378+0000","last_unstale":"2024-07-26T10:59:48.662570+0000","last_undegraded":"2024-07-26T10:59:48.662570+0000","last_fullsized":"2024-07-26T10:59:48.662570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:42:48.396209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.980806+0000","last_change":"2024-07-26T10:57:22.108480+0000","last_active":"2024-07-26T11:00:12.980806+0000","last_peered":"2024-07-26T11:00:12.980806+0000","last_clean":"2024-07-26T11:00:12.980806+0000","last_became_active":"2024-07-26T10:57:22.108084+0000","last_became_peered":"2024-07-26T10:57:22.108084+0000","last_unstale":"2024-07-26T11:00:12.980806+0000","last_undegraded":"2024-07-26T11:00:12.980806+0000","last_fullsized":"2024-07-26T11:00:12.980806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:22.573402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:34.043903+0000","last_change":"2024-07-26T10:57:18.047469+0000","last_active":"2024-07-26T10:59:34.043903+0000","last_peered":"2024-07-26T10:59:34.043903+0000","last_clean":"2024-07-26T10:59:34.043903+0000","last_became_active":"2024-07-26T10:57:18.046830+0000","last_became_peered":"2024-07-26T10:57:18.046830+0000","last_unstale":"2024-07-26T10:59:34.043903+0000","last_undegraded":"2024-07-26T10:59:34.043903+0000","last_fullsized":"2024-07-26T10:59:34.043903+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:30.347496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:53.558798+0000","last_change":"2024-07-26T10:57:35.078541+0000","last_active":"2024-07-26T10:59:53.558798+0000","last_peered":"2024-07-26T10:59:53.558798+0000","last_clean":"2024-07-26T10:59:53.558798+0000","last_became_active":"2024-07-26T10:57:35.077617+0000","last_became_peered":"2024-07-26T10:57:35.077617+0000","last_unstale":"2024-07-26T10:59:53.558798+0000","last_undegraded":"2024-07-26T10:59:53.558798+0000","last_fullsized":"2024-07-26T10:59:53.558798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:42.731301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954883+0000","last_change":"2024-07-26T10:57:22.108932+0000","last_active":"2024-07-26T11:00:12.954883+0000","last_peered":"2024-07-26T11:00:12.954883+0000","last_clean":"2024-07-26T11:00:12.954883+0000","last_became_active":"2024-07-26T10:57:22.107798+0000","last_became_peered":"2024-07-26T10:57:22.107798+0000","last_unstale":"2024-07-26T11:00:12.954883+0000","last_undegraded":"2024-07-26T11:00:12.954883+0000","last_fullsized":"2024-07-26T11:00:12.954883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:47.928442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:33.043613+0000","last_change":"2024-07-26T10:57:18.059374+0000","last_active":"2024-07-26T10:59:33.043613+0000","last_peered":"2024-07-26T10:59:33.043613+0000","last_clean":"2024-07-26T10:59:33.043613+0000","last_became_active":"2024-07-26T10:57:18.058707+0000","last_became_peered":"2024-07-26T10:57:18.058707+0000","last_unstale":"2024-07-26T10:59:33.043613+0000","last_undegraded":"2024-07-26T10:59:33.043613+0000","last_fullsized":"2024-07-26T10:59:33.043613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:05.501290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:49.045647+0000","last_change":"2024-07-26T10:57:35.064974+0000","last_active":"2024-07-26T10:59:49.045647+0000","last_peered":"2024-07-26T10:59:49.045647+0000","last_clean":"2024-07-26T10:59:49.045647+0000","last_became_active":"2024-07-26T10:57:35.064247+0000","last_became_peered":"2024-07-26T10:57:35.064247+0000","last_unstale":"2024-07-26T10:59:49.045647+0000","last_undegraded":"2024-07-26T10:59:49.045647+0000","last_fullsized":"2024-07-26T10:59:49.045647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:49.879769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945946+0000","last_change":"2024-07-26T10:57:22.107873+0000","last_active":"2024-07-26T11:00:12.945946+0000","last_peered":"2024-07-26T11:00:12.945946+0000","last_clean":"2024-07-26T11:00:12.945946+0000","last_became_active":"2024-07-26T10:57:22.107514+0000","last_became_peered":"2024-07-26T10:57:22.107514+0000","last_unstale":"2024-07-26T11:00:12.945946+0000","last_undegraded":"2024-07-26T11:00:12.945946+0000","last_fullsized":"2024-07-26T11:00:12.945946+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:15:26.231601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:36.233709+0000","last_change":"2024-07-26T10:57:18.053630+0000","last_active":"2024-07-26T10:59:36.233709+0000","last_peered":"2024-07-26T10:59:36.233709+0000","last_clean":"2024-07-26T10:59:36.233709+0000","last_became_active":"2024-07-26T10:57:18.053149+0000","last_became_peered":"2024-07-26T10:57:18.053149+0000","last_unstale":"2024-07-26T10:59:36.233709+0000","last_undegraded":"2024-07-26T10:59:36.233709+0000","last_fullsized":"2024-07-26T10:59:36.233709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:51:14.579716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:49.045694+0000","last_change":"2024-07-26T10:57:35.073246+0000","last_active":"2024-07-26T10:59:49.045694+0000","last_peered":"2024-07-26T10:59:49.045694+0000","last_clean":"2024-07-26T10:59:49.045694+0000","last_became_active":"2024-07-26T10:57:35.072730+0000","last_became_peered":"2024-07-26T10:57:35.072730+0000","last_unstale":"2024-07-26T10:59:49.045694+0000","last_undegraded":"2024-07-26T10:59:49.045694+0000","last_fullsized":"2024-07-26T10:59:49.045694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:29.871837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.943041+0000","last_change":"2024-07-26T10:57:44.892016+0000","last_active":"2024-07-26T11:00:12.943041+0000","last_peered":"2024-07-26T11:00:12.943041+0000","last_clean":"2024-07-26T11:00:12.943041+0000","last_became_active":"2024-07-26T10:57:44.891263+0000","last_became_peered":"2024-07-26T10:57:44.891263+0000","last_unstale":"2024-07-26T11:00:12.943041+0000","last_undegraded":"2024-07-26T11:00:12.943041+0000","last_fullsized":"2024-07-26T11:00:12.943041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:20.811842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:41.234191+0000","last_change":"2024-07-26T10:57:18.151389+0000","last_active":"2024-07-26T10:59:41.234191+0000","last_peered":"2024-07-26T10:59:41.234191+0000","last_clean":"2024-07-26T10:59:41.234191+0000","last_became_active":"2024-07-26T10:57:18.150900+0000","last_became_peered":"2024-07-26T10:57:18.150900+0000","last_unstale":"2024-07-26T10:59:41.234191+0000","last_undegraded":"2024-07-26T10:59:41.234191+0000","last_fullsized":"2024-07-26T10:59:41.234191+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:46.265203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.397349+0000","last_change":"2024-07-26T10:57:35.087902+0000","last_active":"2024-07-26T10:58:45.397349+0000","last_peered":"2024-07-26T10:58:45.397349+0000","last_clean":"2024-07-26T10:58:45.397349+0000","last_became_active":"2024-07-26T10:57:35.083496+0000","last_became_peered":"2024-07-26T10:57:35.083496+0000","last_unstale":"2024-07-26T10:58:45.397349+0000","last_undegraded":"2024-07-26T10:58:45.397349+0000","last_fullsized":"2024-07-26T10:58:45.397349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:19.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.953676+0000","last_change":"2024-07-26T10:57:22.109057+0000","last_active":"2024-07-26T11:00:12.953676+0000","last_peered":"2024-07-26T11:00:12.953676+0000","last_clean":"2024-07-26T11:00:12.953676+0000","last_became_active":"2024-07-26T10:57:22.108613+0000","last_became_peered":"2024-07-26T10:57:22.108613+0000","last_unstale":"2024-07-26T11:00:12.953676+0000","last_undegraded":"2024-07-26T11:00:12.953676+0000","last_fullsized":"2024-07-26T11:00:12.953676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:35:56.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":280,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:25.710137+0000","last_change":"2024-07-26T10:57:18.047522+0000","last_active":"2024-07-26T11:02:25.710137+0000","last_peered":"2024-07-26T11:02:25.710137+0000","last_clean":"2024-07-26T11:02:25.710137+0000","last_became_active":"2024-07-26T10:57:18.047141+0000","last_became_peered":"2024-07-26T10:57:18.047141+0000","last_unstale":"2024-07-26T11:02:25.710137+0000","last_undegraded":"2024-07-26T11:02:25.710137+0000","last_fullsized":"2024-07-26T11:02:25.710137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:14.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:00.600535+0000","last_change":"2024-07-26T10:57:35.084107+0000","last_active":"2024-07-26T11:00:00.600535+0000","last_peered":"2024-07-26T11:00:00.600535+0000","last_clean":"2024-07-26T11:00:00.600535+0000","last_became_active":"2024-07-26T10:57:35.081182+0000","last_became_peered":"2024-07-26T10:57:35.081182+0000","last_unstale":"2024-07-26T11:00:00.600535+0000","last_undegraded":"2024-07-26T11:00:00.600535+0000","last_fullsized":"2024-07-26T11:00:00.600535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:17.749063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.987945+0000","last_change":"2024-07-26T10:57:44.900785+0000","last_active":"2024-07-26T11:00:12.987945+0000","last_peered":"2024-07-26T11:00:12.987945+0000","last_clean":"2024-07-26T11:00:12.987945+0000","last_became_active":"2024-07-26T10:57:44.900466+0000","last_became_peered":"2024-07-26T10:57:44.900466+0000","last_unstale":"2024-07-26T11:00:12.987945+0000","last_undegraded":"2024-07-26T11:00:12.987945+0000","last_fullsized":"2024-07-26T11:00:12.987945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:58.792312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.044678+0000","last_change":"2024-07-26T10:57:18.050097+0000","last_active":"2024-07-26T10:59:39.044678+0000","last_peered":"2024-07-26T10:59:39.044678+0000","last_clean":"2024-07-26T10:59:39.044678+0000","last_became_active":"2024-07-26T10:57:18.049572+0000","last_became_peered":"2024-07-26T10:57:18.049572+0000","last_unstale":"2024-07-26T10:59:39.044678+0000","last_undegraded":"2024-07-26T10:59:39.044678+0000","last_fullsized":"2024-07-26T10:59:39.044678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:03:21.066000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.719471+0000","last_change":"2024-07-26T10:57:35.092019+0000","last_active":"2024-07-26T10:59:55.719471+0000","last_peered":"2024-07-26T10:59:55.719471+0000","last_clean":"2024-07-26T10:59:55.719471+0000","last_became_active":"2024-07-26T10:57:35.091478+0000","last_became_peered":"2024-07-26T10:57:35.091478+0000","last_unstale":"2024-07-26T10:59:55.719471+0000","last_undegraded":"2024-07-26T10:59:55.719471+0000","last_fullsized":"2024-07-26T10:59:55.719471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:06:06.017013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.986504+0000","last_change":"2024-07-26T10:57:44.906158+0000","last_active":"2024-07-26T11:00:12.986504+0000","last_peered":"2024-07-26T11:00:12.986504+0000","last_clean":"2024-07-26T11:00:12.986504+0000","last_became_active":"2024-07-26T10:57:44.902235+0000","last_became_peered":"2024-07-26T10:57:44.902235+0000","last_unstale":"2024-07-26T11:00:12.986504+0000","last_undegraded":"2024-07-26T11:00:12.986504+0000","last_fullsized":"2024-07-26T11:00:12.986504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:52.399739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:39.985729+0000","last_change":"2024-07-26T10:57:18.055106+0000","last_active":"2024-07-26T10:59:39.985729+0000","last_peered":"2024-07-26T10:59:39.985729+0000","last_clean":"2024-07-26T10:59:39.985729+0000","last_became_active":"2024-07-26T10:57:18.054405+0000","last_became_peered":"2024-07-26T10:57:18.054405+0000","last_unstale":"2024-07-26T10:59:39.985729+0000","last_undegraded":"2024-07-26T10:59:39.985729+0000","last_fullsized":"2024-07-26T10:59:39.985729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:17.037926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.395529+0000","last_change":"2024-07-26T10:57:35.073396+0000","last_active":"2024-07-26T10:58:45.395529+0000","last_peered":"2024-07-26T10:58:45.395529+0000","last_clean":"2024-07-26T10:58:45.395529+0000","last_became_active":"2024-07-26T10:57:35.072898+0000","last_became_peered":"2024-07-26T10:57:35.072898+0000","last_unstale":"2024-07-26T10:58:45.395529+0000","last_undegraded":"2024-07-26T10:58:45.395529+0000","last_fullsized":"2024-07-26T10:58:45.395529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:55.263627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.954572+0000","last_change":"2024-07-26T10:57:22.115427+0000","last_active":"2024-07-26T11:00:12.954572+0000","last_peered":"2024-07-26T11:00:12.954572+0000","last_clean":"2024-07-26T11:00:12.954572+0000","last_became_active":"2024-07-26T10:57:22.114983+0000","last_became_peered":"2024-07-26T10:57:22.114983+0000","last_unstale":"2024-07-26T11:00:12.954572+0000","last_undegraded":"2024-07-26T11:00:12.954572+0000","last_fullsized":"2024-07-26T11:00:12.954572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:59.221360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":545,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:24.542089+0000","last_change":"2024-07-26T10:57:18.054100+0000","last_active":"2024-07-26T11:02:24.542089+0000","last_peered":"2024-07-26T11:02:24.542089+0000","last_clean":"2024-07-26T11:02:24.542089+0000","last_became_active":"2024-07-26T10:57:18.053183+0000","last_became_peered":"2024-07-26T10:57:18.053183+0000","last_unstale":"2024-07-26T11:02:24.542089+0000","last_undegraded":"2024-07-26T11:02:24.542089+0000","last_fullsized":"2024-07-26T11:02:24.542089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:42.750047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72547,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":459,"num_read_kb":301,"num_write":41,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.599761+0000","last_change":"2024-07-26T10:57:35.075585+0000","last_active":"2024-07-26T10:59:55.599761+0000","last_peered":"2024-07-26T10:59:55.599761+0000","last_clean":"2024-07-26T10:59:55.599761+0000","last_became_active":"2024-07-26T10:57:35.075038+0000","last_became_peered":"2024-07-26T10:57:35.075038+0000","last_unstale":"2024-07-26T10:59:55.599761+0000","last_undegraded":"2024-07-26T10:59:55.599761+0000","last_fullsized":"2024-07-26T10:59:55.599761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:29.555393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952219+0000","last_change":"2024-07-26T10:57:22.109550+0000","last_active":"2024-07-26T11:00:12.952219+0000","last_peered":"2024-07-26T11:00:12.952219+0000","last_clean":"2024-07-26T11:00:12.952219+0000","last_became_active":"2024-07-26T10:57:22.109064+0000","last_became_peered":"2024-07-26T10:57:22.109064+0000","last_unstale":"2024-07-26T11:00:12.952219+0000","last_undegraded":"2024-07-26T11:00:12.952219+0000","last_fullsized":"2024-07-26T11:00:12.952219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:37:41.833021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:44.045189+0000","last_change":"2024-07-26T10:57:18.153308+0000","last_active":"2024-07-26T10:59:44.045189+0000","last_peered":"2024-07-26T10:59:44.045189+0000","last_clean":"2024-07-26T10:59:44.045189+0000","last_became_active":"2024-07-26T10:57:18.152930+0000","last_became_peered":"2024-07-26T10:57:18.152930+0000","last_unstale":"2024-07-26T10:59:44.045189+0000","last_undegraded":"2024-07-26T10:59:44.045189+0000","last_fullsized":"2024-07-26T10:59:44.045189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:56.453701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:56.710801+0000","last_change":"2024-07-26T10:57:35.081895+0000","last_active":"2024-07-26T10:59:56.710801+0000","last_peered":"2024-07-26T10:59:56.710801+0000","last_clean":"2024-07-26T10:59:56.710801+0000","last_became_active":"2024-07-26T10:57:35.081474+0000","last_became_peered":"2024-07-26T10:57:35.081474+0000","last_unstale":"2024-07-26T10:59:56.710801+0000","last_undegraded":"2024-07-26T10:59:56.710801+0000","last_fullsized":"2024-07-26T10:59:56.710801+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:17.737600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988804+0000","last_change":"2024-07-26T10:57:22.111499+0000","last_active":"2024-07-26T11:00:12.988804+0000","last_peered":"2024-07-26T11:00:12.988804+0000","last_clean":"2024-07-26T11:00:12.988804+0000","last_became_active":"2024-07-26T10:57:22.111078+0000","last_became_peered":"2024-07-26T10:57:22.111078+0000","last_unstale":"2024-07-26T11:00:12.988804+0000","last_undegraded":"2024-07-26T11:00:12.988804+0000","last_fullsized":"2024-07-26T11:00:12.988804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:50:27.610699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:42.632249+0000","last_change":"2024-07-26T10:57:18.053916+0000","last_active":"2024-07-26T10:59:42.632249+0000","last_peered":"2024-07-26T10:59:42.632249+0000","last_clean":"2024-07-26T10:59:42.632249+0000","last_became_active":"2024-07-26T10:57:18.053496+0000","last_became_peered":"2024-07-26T10:57:18.053496+0000","last_unstale":"2024-07-26T10:59:42.632249+0000","last_undegraded":"2024-07-26T10:59:42.632249+0000","last_fullsized":"2024-07-26T10:59:42.632249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:50.816598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.599712+0000","last_change":"2024-07-26T10:57:35.088308+0000","last_active":"2024-07-26T10:59:55.599712+0000","last_peered":"2024-07-26T10:59:55.599712+0000","last_clean":"2024-07-26T10:59:55.599712+0000","last_became_active":"2024-07-26T10:57:35.087766+0000","last_became_peered":"2024-07-26T10:57:35.087766+0000","last_unstale":"2024-07-26T10:59:55.599712+0000","last_undegraded":"2024-07-26T10:59:55.599712+0000","last_fullsized":"2024-07-26T10:59:55.599712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:49.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.950532+0000","last_change":"2024-07-26T10:57:22.117162+0000","last_active":"2024-07-26T11:00:12.950532+0000","last_peered":"2024-07-26T11:00:12.950532+0000","last_clean":"2024-07-26T11:00:12.950532+0000","last_became_active":"2024-07-26T10:57:22.116720+0000","last_became_peered":"2024-07-26T10:57:22.116720+0000","last_unstale":"2024-07-26T11:00:12.950532+0000","last_undegraded":"2024-07-26T11:00:12.950532+0000","last_fullsized":"2024-07-26T11:00:12.950532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:06.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":273,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:23.540725+0000","last_change":"2024-07-26T10:57:18.053456+0000","last_active":"2024-07-26T11:02:23.540725+0000","last_peered":"2024-07-26T11:02:23.540725+0000","last_clean":"2024-07-26T11:02:23.540725+0000","last_became_active":"2024-07-26T10:57:18.052907+0000","last_became_peered":"2024-07-26T10:57:18.052907+0000","last_unstale":"2024-07-26T11:02:23.540725+0000","last_undegraded":"2024-07-26T11:02:23.540725+0000","last_fullsized":"2024-07-26T11:02:23.540725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:31.328498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":223,"num_read_kb":223,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:55.907236+0000","last_change":"2024-07-26T10:57:35.079378+0000","last_active":"2024-07-26T10:59:55.907236+0000","last_peered":"2024-07-26T10:59:55.907236+0000","last_clean":"2024-07-26T10:59:55.907236+0000","last_became_active":"2024-07-26T10:57:35.069405+0000","last_became_peered":"2024-07-26T10:57:35.069405+0000","last_unstale":"2024-07-26T10:59:55.907236+0000","last_undegraded":"2024-07-26T10:59:55.907236+0000","last_fullsized":"2024-07-26T10:59:55.907236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:57:38.207634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:38.556347+0000","last_change":"2024-07-26T10:57:35.082324+0000","last_active":"2024-07-26T10:59:38.556347+0000","last_peered":"2024-07-26T10:59:38.556347+0000","last_clean":"2024-07-26T10:59:38.556347+0000","last_became_active":"2024-07-26T10:57:35.079770+0000","last_became_peered":"2024-07-26T10:57:35.079770+0000","last_unstale":"2024-07-26T10:59:38.556347+0000","last_undegraded":"2024-07-26T10:59:38.556347+0000","last_fullsized":"2024-07-26T10:59:38.556347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:18.820608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949136+0000","last_change":"2024-07-26T10:57:22.110271+0000","last_active":"2024-07-26T11:00:12.949136+0000","last_peered":"2024-07-26T11:00:12.949136+0000","last_clean":"2024-07-26T11:00:12.949136+0000","last_became_active":"2024-07-26T10:57:22.109869+0000","last_became_peered":"2024-07-26T10:57:22.109869+0000","last_unstale":"2024-07-26T11:00:12.949136+0000","last_undegraded":"2024-07-26T11:00:12.949136+0000","last_fullsized":"2024-07-26T11:00:12.949136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:33.462810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:44.045142+0000","last_change":"2024-07-26T10:57:18.048117+0000","last_active":"2024-07-26T10:59:44.045142+0000","last_peered":"2024-07-26T10:59:44.045142+0000","last_clean":"2024-07-26T10:59:44.045142+0000","last_became_active":"2024-07-26T10:57:18.047688+0000","last_became_peered":"2024-07-26T10:57:18.047688+0000","last_unstale":"2024-07-26T10:59:44.045142+0000","last_undegraded":"2024-07-26T10:59:44.045142+0000","last_fullsized":"2024-07-26T10:59:44.045142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:55:17.922629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:40.717868+0000","last_change":"2024-07-26T10:57:35.075337+0000","last_active":"2024-07-26T10:59:40.717868+0000","last_peered":"2024-07-26T10:59:40.717868+0000","last_clean":"2024-07-26T10:59:40.717868+0000","last_became_active":"2024-07-26T10:57:35.074936+0000","last_became_peered":"2024-07-26T10:57:35.074936+0000","last_unstale":"2024-07-26T10:59:40.717868+0000","last_undegraded":"2024-07-26T10:59:40.717868+0000","last_fullsized":"2024-07-26T10:59:40.717868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:58.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988737+0000","last_change":"2024-07-26T10:57:22.125501+0000","last_active":"2024-07-26T11:00:12.988737+0000","last_peered":"2024-07-26T11:00:12.988737+0000","last_clean":"2024-07-26T11:00:12.988737+0000","last_became_active":"2024-07-26T10:57:22.124756+0000","last_became_peered":"2024-07-26T10:57:22.124756+0000","last_unstale":"2024-07-26T11:00:12.988737+0000","last_undegraded":"2024-07-26T11:00:12.988737+0000","last_fullsized":"2024-07-26T11:00:12.988737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.133521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:43.045394+0000","last_change":"2024-07-26T10:57:18.053979+0000","last_active":"2024-07-26T10:59:43.045394+0000","last_peered":"2024-07-26T10:59:43.045394+0000","last_clean":"2024-07-26T10:59:43.045394+0000","last_became_active":"2024-07-26T10:57:18.053280+0000","last_became_peered":"2024-07-26T10:57:18.053280+0000","last_unstale":"2024-07-26T10:59:43.045394+0000","last_undegraded":"2024-07-26T10:59:43.045394+0000","last_fullsized":"2024-07-26T10:59:43.045394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:55.048167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:59:50.113187+0000","last_change":"2024-07-26T10:57:18.054009+0000","last_active":"2024-07-26T10:59:50.113187+0000","last_peered":"2024-07-26T10:59:50.113187+0000","last_clean":"2024-07-26T10:59:50.113187+0000","last_became_active":"2024-07-26T10:57:18.053480+0000","last_became_peered":"2024-07-26T10:57:18.053480+0000","last_unstale":"2024-07-26T10:59:50.113187+0000","last_undegraded":"2024-07-26T10:59:50.113187+0000","last_fullsized":"2024-07-26T10:59:50.113187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:22:48.226671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.949369+0000","last_change":"2024-07-26T10:57:44.907486+0000","last_active":"2024-07-26T11:00:12.949369+0000","last_peered":"2024-07-26T11:00:12.949369+0000","last_clean":"2024-07-26T11:00:12.949369+0000","last_became_active":"2024-07-26T10:57:44.906532+0000","last_became_peered":"2024-07-26T10:57:44.906532+0000","last_unstale":"2024-07-26T11:00:12.949369+0000","last_undegraded":"2024-07-26T11:00:12.949369+0000","last_fullsized":"2024-07-26T11:00:12.949369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:29.793682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205546+0000","last_change":"2024-07-26T10:57:35.092084+0000","last_active":"2024-07-26T10:58:45.205546+0000","last_peered":"2024-07-26T10:58:45.205546+0000","last_clean":"2024-07-26T10:58:45.205546+0000","last_became_active":"2024-07-26T10:57:35.091632+0000","last_became_peered":"2024-07-26T10:57:35.091632+0000","last_unstale":"2024-07-26T10:58:45.205546+0000","last_undegraded":"2024-07-26T10:58:45.205546+0000","last_fullsized":"2024-07-26T10:58:45.205546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:25.991692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":285,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:02:22.539964+0000","last_change":"2024-07-26T10:57:44.891931+0000","last_active":"2024-07-26T11:02:22.539964+0000","last_peered":"2024-07-26T11:02:22.539964+0000","last_clean":"2024-07-26T11:02:22.539964+0000","last_became_active":"2024-07-26T10:57:44.891294+0000","last_became_peered":"2024-07-26T10:57:44.891294+0000","last_unstale":"2024-07-26T11:02:22.539964+0000","last_undegraded":"2024-07-26T11:02:22.539964+0000","last_fullsized":"2024-07-26T11:02:22.539964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:17.010153+0000","last_clean_scrub_stamp":"2024-07-26T10:57:17.010153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:17:35.197965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":929,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":225,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.939415+0000","last_change":"2024-07-26T10:57:22.110681+0000","last_active":"2024-07-26T11:00:12.939415+0000","last_peered":"2024-07-26T11:00:12.939415+0000","last_clean":"2024-07-26T11:00:12.939415+0000","last_became_active":"2024-07-26T10:57:22.110179+0000","last_became_peered":"2024-07-26T10:57:22.110179+0000","last_unstale":"2024-07-26T11:00:12.939415+0000","last_undegraded":"2024-07-26T11:00:12.939415+0000","last_fullsized":"2024-07-26T11:00:12.939415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:35.285890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T10:58:45.205755+0000","last_change":"2024-07-26T10:57:35.064384+0000","last_active":"2024-07-26T10:58:45.205755+0000","last_peered":"2024-07-26T10:58:45.205755+0000","last_clean":"2024-07-26T10:58:45.205755+0000","last_became_active":"2024-07-26T10:57:35.063826+0000","last_became_peered":"2024-07-26T10:57:35.063826+0000","last_unstale":"2024-07-26T10:58:45.205755+0000","last_undegraded":"2024-07-26T10:58:45.205755+0000","last_fullsized":"2024-07-26T10:58:45.205755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:34.033417+0000","last_clean_scrub_stamp":"2024-07-26T10:57:34.033417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:01.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942853+0000","last_change":"2024-07-26T10:57:22.106139+0000","last_active":"2024-07-26T11:00:12.942853+0000","last_peered":"2024-07-26T11:00:12.942853+0000","last_clean":"2024-07-26T11:00:12.942853+0000","last_became_active":"2024-07-26T10:57:22.105129+0000","last_became_peered":"2024-07-26T10:57:22.105129+0000","last_unstale":"2024-07-26T11:00:12.942853+0000","last_undegraded":"2024-07-26T11:00:12.942853+0000","last_fullsized":"2024-07-26T11:00:12.942853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:03.729326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946199+0000","last_change":"2024-07-26T10:57:22.107054+0000","last_active":"2024-07-26T11:00:12.946199+0000","last_peered":"2024-07-26T11:00:12.946199+0000","last_clean":"2024-07-26T11:00:12.946199+0000","last_became_active":"2024-07-26T10:57:22.106655+0000","last_became_peered":"2024-07-26T10:57:22.106655+0000","last_unstale":"2024-07-26T11:00:12.946199+0000","last_undegraded":"2024-07-26T11:00:12.946199+0000","last_fullsized":"2024-07-26T11:00:12.946199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:51.587064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.975244+0000","last_change":"2024-07-26T10:57:22.124475+0000","last_active":"2024-07-26T11:00:12.975244+0000","last_peered":"2024-07-26T11:00:12.975244+0000","last_clean":"2024-07-26T11:00:12.975244+0000","last_became_active":"2024-07-26T10:57:22.124075+0000","last_became_peered":"2024-07-26T10:57:22.124075+0000","last_unstale":"2024-07-26T11:00:12.975244+0000","last_undegraded":"2024-07-26T11:00:12.975244+0000","last_fullsized":"2024-07-26T11:00:12.975244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:37.228656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992785+0000","last_change":"2024-07-26T10:57:22.132466+0000","last_active":"2024-07-26T11:00:12.992785+0000","last_peered":"2024-07-26T11:00:12.992785+0000","last_clean":"2024-07-26T11:00:12.992785+0000","last_became_active":"2024-07-26T10:57:22.131994+0000","last_became_peered":"2024-07-26T10:57:22.131994+0000","last_unstale":"2024-07-26T11:00:12.992785+0000","last_undegraded":"2024-07-26T11:00:12.992785+0000","last_fullsized":"2024-07-26T11:00:12.992785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:19.367965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.996976+0000","last_change":"2024-07-26T10:57:22.127648+0000","last_active":"2024-07-26T11:00:12.996976+0000","last_peered":"2024-07-26T11:00:12.996976+0000","last_clean":"2024-07-26T11:00:12.996976+0000","last_became_active":"2024-07-26T10:57:22.126845+0000","last_became_peered":"2024-07-26T10:57:22.126845+0000","last_unstale":"2024-07-26T11:00:12.996976+0000","last_undegraded":"2024-07-26T11:00:12.996976+0000","last_fullsized":"2024-07-26T11:00:12.996976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:07.960307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.940323+0000","last_change":"2024-07-26T10:57:22.105757+0000","last_active":"2024-07-26T11:00:12.940323+0000","last_peered":"2024-07-26T11:00:12.940323+0000","last_clean":"2024-07-26T11:00:12.940323+0000","last_became_active":"2024-07-26T10:57:22.105407+0000","last_became_peered":"2024-07-26T10:57:22.105407+0000","last_unstale":"2024-07-26T11:00:12.940323+0000","last_undegraded":"2024-07-26T11:00:12.940323+0000","last_fullsized":"2024-07-26T11:00:12.940323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:13:23.563739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.942010+0000","last_change":"2024-07-26T10:57:22.105504+0000","last_active":"2024-07-26T11:00:12.942010+0000","last_peered":"2024-07-26T11:00:12.942010+0000","last_clean":"2024-07-26T11:00:12.942010+0000","last_became_active":"2024-07-26T10:57:22.105023+0000","last_became_peered":"2024-07-26T10:57:22.105023+0000","last_unstale":"2024-07-26T11:00:12.942010+0000","last_undegraded":"2024-07-26T11:00:12.942010+0000","last_fullsized":"2024-07-26T11:00:12.942010+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:14:28.266286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.973405+0000","last_change":"2024-07-26T10:57:22.121481+0000","last_active":"2024-07-26T11:00:12.973405+0000","last_peered":"2024-07-26T11:00:12.973405+0000","last_clean":"2024-07-26T11:00:12.973405+0000","last_became_active":"2024-07-26T10:57:22.117621+0000","last_became_peered":"2024-07-26T10:57:22.117621+0000","last_unstale":"2024-07-26T11:00:12.973405+0000","last_undegraded":"2024-07-26T11:00:12.973405+0000","last_fullsized":"2024-07-26T11:00:12.973405+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:35:38.165410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967597+0000","last_change":"2024-07-26T10:57:22.112447+0000","last_active":"2024-07-26T11:00:12.967597+0000","last_peered":"2024-07-26T11:00:12.967597+0000","last_clean":"2024-07-26T11:00:12.967597+0000","last_became_active":"2024-07-26T10:57:22.110665+0000","last_became_peered":"2024-07-26T10:57:22.110665+0000","last_unstale":"2024-07-26T11:00:12.967597+0000","last_undegraded":"2024-07-26T11:00:12.967597+0000","last_fullsized":"2024-07-26T11:00:12.967597+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:49.748312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.982036+0000","last_change":"2024-07-26T10:57:22.111402+0000","last_active":"2024-07-26T11:00:12.982036+0000","last_peered":"2024-07-26T11:00:12.982036+0000","last_clean":"2024-07-26T11:00:12.982036+0000","last_became_active":"2024-07-26T10:57:22.108085+0000","last_became_peered":"2024-07-26T10:57:22.108085+0000","last_unstale":"2024-07-26T11:00:12.982036+0000","last_undegraded":"2024-07-26T11:00:12.982036+0000","last_fullsized":"2024-07-26T11:00:12.982036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:53:33.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981299+0000","last_change":"2024-07-26T10:57:22.124338+0000","last_active":"2024-07-26T11:00:12.981299+0000","last_peered":"2024-07-26T11:00:12.981299+0000","last_clean":"2024-07-26T11:00:12.981299+0000","last_became_active":"2024-07-26T10:57:22.123943+0000","last_became_peered":"2024-07-26T10:57:22.123943+0000","last_unstale":"2024-07-26T11:00:12.981299+0000","last_undegraded":"2024-07-26T11:00:12.981299+0000","last_fullsized":"2024-07-26T11:00:12.981299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:23:20.358002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.999210+0000","last_change":"2024-07-26T10:57:22.120652+0000","last_active":"2024-07-26T11:00:12.999210+0000","last_peered":"2024-07-26T11:00:12.999210+0000","last_clean":"2024-07-26T11:00:12.999210+0000","last_became_active":"2024-07-26T10:57:22.120227+0000","last_became_peered":"2024-07-26T10:57:22.120227+0000","last_unstale":"2024-07-26T11:00:12.999210+0000","last_undegraded":"2024-07-26T11:00:12.999210+0000","last_fullsized":"2024-07-26T11:00:12.999210+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:20.929940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946804+0000","last_change":"2024-07-26T10:57:22.101173+0000","last_active":"2024-07-26T11:00:12.946804+0000","last_peered":"2024-07-26T11:00:12.946804+0000","last_clean":"2024-07-26T11:00:12.946804+0000","last_became_active":"2024-07-26T10:57:22.100826+0000","last_became_peered":"2024-07-26T10:57:22.100826+0000","last_unstale":"2024-07-26T11:00:12.946804+0000","last_undegraded":"2024-07-26T11:00:12.946804+0000","last_fullsized":"2024-07-26T11:00:12.946804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:31.345735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947300+0000","last_change":"2024-07-26T10:57:22.130451+0000","last_active":"2024-07-26T11:00:12.947300+0000","last_peered":"2024-07-26T11:00:12.947300+0000","last_clean":"2024-07-26T11:00:12.947300+0000","last_became_active":"2024-07-26T10:57:22.130040+0000","last_became_peered":"2024-07-26T10:57:22.130040+0000","last_unstale":"2024-07-26T11:00:12.947300+0000","last_undegraded":"2024-07-26T11:00:12.947300+0000","last_fullsized":"2024-07-26T11:00:12.947300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:29:38.509851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.945515+0000","last_change":"2024-07-26T10:57:22.113970+0000","last_active":"2024-07-26T11:00:12.945515+0000","last_peered":"2024-07-26T11:00:12.945515+0000","last_clean":"2024-07-26T11:00:12.945515+0000","last_became_active":"2024-07-26T10:57:22.113280+0000","last_became_peered":"2024-07-26T10:57:22.113280+0000","last_unstale":"2024-07-26T11:00:12.945515+0000","last_undegraded":"2024-07-26T11:00:12.945515+0000","last_fullsized":"2024-07-26T11:00:12.945515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:12.174979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974957+0000","last_change":"2024-07-26T10:57:22.111661+0000","last_active":"2024-07-26T11:00:12.974957+0000","last_peered":"2024-07-26T11:00:12.974957+0000","last_clean":"2024-07-26T11:00:12.974957+0000","last_became_active":"2024-07-26T10:57:22.110651+0000","last_became_peered":"2024-07-26T10:57:22.110651+0000","last_unstale":"2024-07-26T11:00:12.974957+0000","last_undegraded":"2024-07-26T11:00:12.974957+0000","last_fullsized":"2024-07-26T11:00:12.974957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:15.336283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992896+0000","last_change":"2024-07-26T10:57:22.119890+0000","last_active":"2024-07-26T11:00:12.992896+0000","last_peered":"2024-07-26T11:00:12.992896+0000","last_clean":"2024-07-26T11:00:12.992896+0000","last_became_active":"2024-07-26T10:57:22.119504+0000","last_became_peered":"2024-07-26T10:57:22.119504+0000","last_unstale":"2024-07-26T11:00:12.992896+0000","last_undegraded":"2024-07-26T11:00:12.992896+0000","last_fullsized":"2024-07-26T11:00:12.992896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:05.400109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.992111+0000","last_change":"2024-07-26T10:57:45.408490+0000","last_active":"2024-07-26T11:00:12.992111+0000","last_peered":"2024-07-26T11:00:12.992111+0000","last_clean":"2024-07-26T11:00:12.992111+0000","last_became_active":"2024-07-26T10:57:45.407719+0000","last_became_peered":"2024-07-26T10:57:45.407719+0000","last_unstale":"2024-07-26T11:00:12.992111+0000","last_undegraded":"2024-07-26T11:00:12.992111+0000","last_fullsized":"2024-07-26T11:00:12.992111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:10:31.694422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.994447+0000","last_change":"2024-07-26T10:57:22.105990+0000","last_active":"2024-07-26T11:00:12.994447+0000","last_peered":"2024-07-26T11:00:12.994447+0000","last_clean":"2024-07-26T11:00:12.994447+0000","last_became_active":"2024-07-26T10:57:22.103408+0000","last_became_peered":"2024-07-26T10:57:22.103408+0000","last_unstale":"2024-07-26T11:00:12.994447+0000","last_undegraded":"2024-07-26T11:00:12.994447+0000","last_fullsized":"2024-07-26T11:00:12.994447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:24.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.988020+0000","last_change":"2024-07-26T10:57:22.130163+0000","last_active":"2024-07-26T11:00:12.988020+0000","last_peered":"2024-07-26T11:00:12.988020+0000","last_clean":"2024-07-26T11:00:12.988020+0000","last_became_active":"2024-07-26T10:57:22.128469+0000","last_became_peered":"2024-07-26T10:57:22.128469+0000","last_unstale":"2024-07-26T11:00:12.988020+0000","last_undegraded":"2024-07-26T11:00:12.988020+0000","last_fullsized":"2024-07-26T11:00:12.988020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:38:18.688639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.951958+0000","last_change":"2024-07-26T10:57:22.109741+0000","last_active":"2024-07-26T11:00:12.951958+0000","last_peered":"2024-07-26T11:00:12.951958+0000","last_clean":"2024-07-26T11:00:12.951958+0000","last_became_active":"2024-07-26T10:57:22.109361+0000","last_became_peered":"2024-07-26T10:57:22.109361+0000","last_unstale":"2024-07-26T11:00:12.951958+0000","last_undegraded":"2024-07-26T11:00:12.951958+0000","last_fullsized":"2024-07-26T11:00:12.951958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:28.103773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.955435+0000","last_change":"2024-07-26T10:57:22.114166+0000","last_active":"2024-07-26T11:00:12.955435+0000","last_peered":"2024-07-26T11:00:12.955435+0000","last_clean":"2024-07-26T11:00:12.955435+0000","last_became_active":"2024-07-26T10:57:22.113818+0000","last_became_peered":"2024-07-26T10:57:22.113818+0000","last_unstale":"2024-07-26T11:00:12.955435+0000","last_undegraded":"2024-07-26T11:00:12.955435+0000","last_fullsized":"2024-07-26T11:00:12.955435+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:26.163411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.948784+0000","last_change":"2024-07-26T10:57:22.107127+0000","last_active":"2024-07-26T11:00:12.948784+0000","last_peered":"2024-07-26T11:00:12.948784+0000","last_clean":"2024-07-26T11:00:12.948784+0000","last_became_active":"2024-07-26T10:57:22.106471+0000","last_became_peered":"2024-07-26T10:57:22.106471+0000","last_unstale":"2024-07-26T11:00:12.948784+0000","last_undegraded":"2024-07-26T11:00:12.948784+0000","last_fullsized":"2024-07-26T11:00:12.948784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:01:36.605999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.946071+0000","last_change":"2024-07-26T10:57:22.109460+0000","last_active":"2024-07-26T11:00:12.946071+0000","last_peered":"2024-07-26T11:00:12.946071+0000","last_clean":"2024-07-26T11:00:12.946071+0000","last_became_active":"2024-07-26T10:57:22.109048+0000","last_became_peered":"2024-07-26T10:57:22.109048+0000","last_unstale":"2024-07-26T11:00:12.946071+0000","last_undegraded":"2024-07-26T11:00:12.946071+0000","last_fullsized":"2024-07-26T11:00:12.946071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:23.987590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.981175+0000","last_change":"2024-07-26T10:57:22.108603+0000","last_active":"2024-07-26T11:00:12.981175+0000","last_peered":"2024-07-26T11:00:12.981175+0000","last_clean":"2024-07-26T11:00:12.981175+0000","last_became_active":"2024-07-26T10:57:22.108168+0000","last_became_peered":"2024-07-26T10:57:22.108168+0000","last_unstale":"2024-07-26T11:00:12.981175+0000","last_undegraded":"2024-07-26T11:00:12.981175+0000","last_fullsized":"2024-07-26T11:00:12.981175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:04.133612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.952874+0000","last_change":"2024-07-26T10:57:22.116051+0000","last_active":"2024-07-26T11:00:12.952874+0000","last_peered":"2024-07-26T11:00:12.952874+0000","last_clean":"2024-07-26T11:00:12.952874+0000","last_became_active":"2024-07-26T10:57:22.115635+0000","last_became_peered":"2024-07-26T10:57:22.115635+0000","last_unstale":"2024-07-26T11:00:12.952874+0000","last_undegraded":"2024-07-26T11:00:12.952874+0000","last_fullsized":"2024-07-26T11:00:12.952874+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:47:35.028000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.947037+0000","last_change":"2024-07-26T10:57:22.118465+0000","last_active":"2024-07-26T11:00:12.947037+0000","last_peered":"2024-07-26T11:00:12.947037+0000","last_clean":"2024-07-26T11:00:12.947037+0000","last_became_active":"2024-07-26T10:57:22.118057+0000","last_became_peered":"2024-07-26T10:57:22.118057+0000","last_unstale":"2024-07-26T11:00:12.947037+0000","last_undegraded":"2024-07-26T11:00:12.947037+0000","last_fullsized":"2024-07-26T11:00:12.947037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:21:48.733457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.967730+0000","last_change":"2024-07-26T10:57:22.131407+0000","last_active":"2024-07-26T11:00:12.967730+0000","last_peered":"2024-07-26T11:00:12.967730+0000","last_clean":"2024-07-26T11:00:12.967730+0000","last_became_active":"2024-07-26T10:57:22.131079+0000","last_became_peered":"2024-07-26T10:57:22.131079+0000","last_unstale":"2024-07-26T11:00:12.967730+0000","last_undegraded":"2024-07-26T11:00:12.967730+0000","last_fullsized":"2024-07-26T11:00:12.967730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:13:45.656995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.953352+0000","last_change":"2024-07-26T10:57:22.105974+0000","last_active":"2024-07-26T11:00:12.953352+0000","last_peered":"2024-07-26T11:00:12.953352+0000","last_clean":"2024-07-26T11:00:12.953352+0000","last_became_active":"2024-07-26T10:57:22.105202+0000","last_became_peered":"2024-07-26T10:57:22.105202+0000","last_unstale":"2024-07-26T11:00:12.953352+0000","last_undegraded":"2024-07-26T11:00:12.953352+0000","last_fullsized":"2024-07-26T11:00:12.953352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:55:41.321483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.989600+0000","last_change":"2024-07-26T10:57:22.122993+0000","last_active":"2024-07-26T11:00:12.989600+0000","last_peered":"2024-07-26T11:00:12.989600+0000","last_clean":"2024-07-26T11:00:12.989600+0000","last_became_active":"2024-07-26T10:57:22.121597+0000","last_became_peered":"2024-07-26T10:57:22.121597+0000","last_unstale":"2024-07-26T11:00:12.989600+0000","last_undegraded":"2024-07-26T11:00:12.989600+0000","last_fullsized":"2024-07-26T11:00:12.989600+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:55.314941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.990190+0000","last_change":"2024-07-26T10:57:22.102598+0000","last_active":"2024-07-26T11:00:12.990190+0000","last_peered":"2024-07-26T11:00:12.990190+0000","last_clean":"2024-07-26T11:00:12.990190+0000","last_became_active":"2024-07-26T10:57:22.101479+0000","last_became_peered":"2024-07-26T10:57:22.101479+0000","last_unstale":"2024-07-26T11:00:12.990190+0000","last_undegraded":"2024-07-26T11:00:12.990190+0000","last_fullsized":"2024-07-26T11:00:12.990190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:40:42.401539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:00:12.974796+0000","last_change":"2024-07-26T10:57:22.111914+0000","last_active":"2024-07-26T11:00:12.974796+0000","last_peered":"2024-07-26T11:00:12.974796+0000","last_clean":"2024-07-26T11:00:12.974796+0000","last_became_active":"2024-07-26T10:57:22.111107+0000","last_became_peered":"2024-07-26T10:57:22.111107+0000","last_unstale":"2024-07-26T11:00:12.974796+0000","last_undegraded":"2024-07-26T11:00:12.974796+0000","last_fullsized":"2024-07-26T11:00:12.974796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T10:57:21.048334+0000","last_clean_scrub_stamp":"2024-07-26T10:57:21.048334+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:45:32.629875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76996,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":1371,"num_read_kb":766,"num_write":186,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":230988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678093,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":535169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874323,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1203708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":531309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234078,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":534743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430306,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1131682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593832,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":530581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790062,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1126738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953587,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":536345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":530775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378750,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":603766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542276,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":603967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":531969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":73275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:02:28.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:02:29.210 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:29 smithi144 ceph-mon[32129]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:29 smithi144 ceph-mon[32129]: pgmap v461: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-07-26T11:02:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:29 smithi160 ceph-mon[36715]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:29 smithi160 ceph-mon[36715]: pgmap v461: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-07-26T11:02:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:29 smithi183 ceph-mon[37273]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:29 smithi183 ceph-mon[37273]: pgmap v461: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-07-26T11:02:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:31.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.615+0000 7f3a1b200640 1 -- 172.21.15.144:0/2690924659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c10b880 msgr2=0x7f3a1c1120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:31.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.615+0000 7f3a1b200640 1 --2- 172.21.15.144:0/2690924659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c10b880 0x7f3a1c1120b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009ca0 tx=0x7f3a1402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 -- 172.21.15.144:0/2690924659 shutdown_connections 2024-07-26T11:02:31.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 --2- 172.21.15.144:0/2690924659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c10b880 0x7f3a1c1120b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 --2- 172.21.15.144:0/2690924659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 0x7f3a1c10b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 --2- 172.21.15.144:0/2690924659 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 -- 172.21.15.144:0/2690924659 >> 172.21.15.144:0/2690924659 conn(0x7f3a1c101060 msgr2=0x7f3a1c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:31.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 -- 172.21.15.144:0/2690924659 shutdown_connections 2024-07-26T11:02:31.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.616+0000 7f3a1b200640 1 -- 172.21.15.144:0/2690924659 wait complete. 2024-07-26T11:02:31.620 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:31 smithi144 ceph-mon[32129]: pgmap v462: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 852 B/s wr, 7 op/s 2024-07-26T11:02:31.620 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:31.620 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:31.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:02:31.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.618+0000 7f3a1b200640 1 Processor -- start 2024-07-26T11:02:31.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.618+0000 7f3a1b200640 1 -- start start 2024-07-26T11:02:31.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:31.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 0x7f3a1c19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:31.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3a1c10b880 0x7f3a1c1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:31.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c115530 con 0x7f3a1c1055f0 2024-07-26T11:02:31.622 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1153b0 con 0x7f3a1c10b880 2024-07-26T11:02:31.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1156b0 con 0x7f3a1c10af30 2024-07-26T11:02:31.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a19e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:31.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a19e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41478/0 (socket says 172.21.15.144:41478) 2024-07-26T11:02:31.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a19e00640 1 -- 172.21.15.144:0/977200139 learned_addr learned my addr 172.21.15.144:0/977200139 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:31.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a19400640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 0x7f3a1c19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:31.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.619+0000 7f3a1a800640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3a1c10b880 0x7f3a1c1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:31.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19e00640 1 -- 172.21.15.144:0/977200139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 msgr2=0x7f3a1c19c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:31.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19e00640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 0x7f3a1c19c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19e00640 1 -- 172.21.15.144:0/977200139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3a1c10b880 msgr2=0x7f3a1c1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:31.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19e00640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3a1c10b880 0x7f3a1c1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19e00640 1 -- 172.21.15.144:0/977200139 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c1a1dd0 con 0x7f3a1c1055f0 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19400640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 0x7f3a1c19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.620+0000 7f3a19e00640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c19be60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e9f0 tx=0x7f3a0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.621+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00cd80 con 0x7f3a1c1055f0 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.621+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c0108e0 con 0x7f3a1c1055f0 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.621+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c1a20c0 con 0x7f3a1c1055f0 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.621+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c018980 con 0x7f3a1c1055f0 2024-07-26T11:02:31.625 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.621+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c1a8820 con 0x7f3a1c1055f0 2024-07-26T11:02:31.626 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.622+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a1c105a00 con 0x7f3a1c1055f0 2024-07-26T11:02:31.627 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.624+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f3a0c010430 con 0x7f3a1c1055f0 2024-07-26T11:02:31.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.626+0000 7f3a12a00640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3a000b3ca0 0x7f3a000b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:31.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.626+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f3a0c0e4900 con 0x7f3a1c1055f0 2024-07-26T11:02:31.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.626+0000 7f3a19400640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3a000b3ca0 0x7f3a000b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:31.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.627+0000 7f3a19400640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3a000b3ca0 0x7f3a000b61b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c109930 tx=0x7f3a08005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:31.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.628+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3a0c0ab410 con 0x7f3a1c1055f0 2024-07-26T11:02:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:31 smithi183 ceph-mon[37273]: pgmap v462: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 852 B/s wr, 7 op/s 2024-07-26T11:02:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:02:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:31 smithi160 ceph-mon[36715]: pgmap v462: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 852 B/s wr, 7 op/s 2024-07-26T11:02:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:02:31.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.864+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a1c02e0c0 con 0x7f3a1c1055f0 2024-07-26T11:02:31.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.866+0000 7f3a12a00640 1 -- 172.21.15.144:0/977200139 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11931 (secure 0 0 0) 0x7f3a0c0af960 con 0x7f3a1c1055f0 2024-07-26T11:02:31.869 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:31.870 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":26,"btime":"2024-07-26T11:02:20:221771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T11:02:19.492590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24703,24715,24728]},"id":1}]} 2024-07-26T11:02:31.870 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 26 2024-07-26T11:02:31.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.869+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3a000b3ca0 msgr2=0x7f3a000b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:31.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.869+0000 7f3a1b200640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3a000b3ca0 0x7f3a000b61b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c109930 tx=0x7f3a08005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:31.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.869+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 msgr2=0x7f3a1c19be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:31.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.869+0000 7f3a1b200640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c19be60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e9f0 tx=0x7f3a0c00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 shutdown_connections 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3a000b3ca0 0x7f3a000b61b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3a1c10b880 0x7f3a1c1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a1c10af30 0x7f3a1c19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 --2- 172.21.15.144:0/977200139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a1c1055f0 0x7f3a1c19be60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 >> 172.21.15.144:0/977200139 conn(0x7f3a1c101060 msgr2=0x7f3a1c103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 shutdown_connections 2024-07-26T11:02:31.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:31.870+0000 7f3a1b200640 1 -- 172.21.15.144:0/977200139 wait complete. 2024-07-26T11:02:32.616 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:32 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/977200139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:02:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:32 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/977200139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:02:32.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:02:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:32 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/977200139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:02:33.152 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:33 smithi183 ceph-mon[37273]: pgmap v463: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T11:02:33.658 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:33 smithi144 ceph-mon[32129]: pgmap v463: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T11:02:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:33 smithi160 ceph-mon[36715]: pgmap v463: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 -- 172.21.15.144:0/4174748568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 msgr2=0x7fc940105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 --2- 172.21.15.144:0/4174748568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc940105a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc934009900 tx=0x7fc93402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 -- 172.21.15.144:0/4174748568 shutdown_connections 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 --2- 172.21.15.144:0/4174748568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010b880 0x7fc9401120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 --2- 172.21.15.144:0/4174748568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc94010af30 0x7fc94010b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 --2- 172.21.15.144:0/4174748568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc940105a00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 -- 172.21.15.144:0/4174748568 >> 172.21.15.144:0/4174748568 conn(0x7fc940101060 msgr2=0x7fc9401034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 -- 172.21.15.144:0/4174748568 shutdown_connections 2024-07-26T11:02:35.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.503+0000 7fc947200640 1 -- 172.21.15.144:0/4174748568 wait complete. 2024-07-26T11:02:35.508 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:35 smithi144 ceph-mon[32129]: pgmap v464: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T11:02:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.504+0000 7fc947200640 1 Processor -- start 2024-07-26T11:02:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.504+0000 7fc947200640 1 -- start start 2024-07-26T11:02:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc947200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc94019bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc947200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010af30 0x7fc94019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc947200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc94010b880 0x7fc9401a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc947200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc940115580 con 0x7fc9401055f0 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc947200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc940115400 con 0x7fc94010af30 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc947200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc940115700 con 0x7fc94010b880 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc945e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc94019bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc945e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc94019bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41486/0 (socket says 172.21.15.144:41486) 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc945e00640 1 -- 172.21.15.144:0/3307291501 learned_addr learned my addr 172.21.15.144:0/3307291501 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc945400640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010af30 0x7fc94019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.505+0000 7fc946800640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc94010b880 0x7fc9401a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945e00640 1 -- 172.21.15.144:0/3307291501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc94010b880 msgr2=0x7fc9401a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945e00640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc94010b880 0x7fc9401a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945e00640 1 -- 172.21.15.144:0/3307291501 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010af30 msgr2=0x7fc94019c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945e00640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010af30 0x7fc94019c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945e00640 1 -- 172.21.15.144:0/3307291501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9401a1f60 con 0x7fc9401055f0 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945400640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010af30 0x7fc94019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc945e00640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc94019bff0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc934005ab0 tx=0x7fc934003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93403f070 con 0x7fc9401055f0 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9401a21f0 con 0x7fc9401055f0 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc934031b60 con 0x7fc9401055f0 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.506+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934035030 con 0x7fc9401055f0 2024-07-26T11:02:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.507+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9401a89b0 con 0x7fc9401055f0 2024-07-26T11:02:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.508+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc940105a00 con 0x7fc9401055f0 2024-07-26T11:02:35.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.511+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fc934042070 con 0x7fc9401055f0 2024-07-26T11:02:35.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.512+0000 7fc93aa00640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9280b3c20 0x7fc9280b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:35.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.512+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc9341047e0 con 0x7fc9401055f0 2024-07-26T11:02:35.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.512+0000 7fc945400640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9280b3c20 0x7fc9280b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.513+0000 7fc945400640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9280b3c20 0x7fc9280b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc92c0095f0 tx=0x7fc92c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.517+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc9340cb2f0 con 0x7fc9401055f0 2024-07-26T11:02:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:35 smithi160 ceph-mon[36715]: pgmap v464: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T11:02:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:35 smithi183 ceph-mon[37273]: pgmap v464: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-07-26T11:02:35.719 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.716+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc94002e0c0 con 0x7fc9401055f0 2024-07-26T11:02:35.721 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.718+0000 7fc93aa00640 1 -- 172.21.15.144:0/3307291501 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25187 (secure 0 0 0) 0x7fc9340cf840 con 0x7fc9401055f0 2024-07-26T11:02:35.722 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:02:35.722 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":90,"fsid":"493f92c6-4b3c-11ef-bcb7-c7b262605968","created":"2024-07-26T10:47:18.301633+0000","modified":"2024-07-26T10:58:45.199165+0000","last_up_change":"2024-07-26T10:56:10.431188+0000","last_in_change":"2024-07-26T10:55:50.596964+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T10:52:04.947795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T10:57:16.258941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T10:57:21.007430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T10:57:33.064159+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a20cb98e-b483-474f-aafe-5145286b6145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6803","nonce":3520329112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6805","nonce":3520329112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6809","nonce":3520329112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3520329112},{"type":"v1","addr":"172.21.15.144:6807","nonce":3520329112}]},"public_addr":"172.21.15.144:6803/3520329112","cluster_addr":"172.21.15.144:6805/3520329112","heartbeat_back_addr":"172.21.15.144:6809/3520329112","heartbeat_front_addr":"172.21.15.144:6807/3520329112","state":["exists","up"]},{"osd":1,"uuid":"938ca752-2db7-47b3-aaad-b31ed684607e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6801","nonce":3146107361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6803","nonce":3146107361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6807","nonce":3146107361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3146107361},{"type":"v1","addr":"172.21.15.160:6805","nonce":3146107361}]},"public_addr":"172.21.15.160:6801/3146107361","cluster_addr":"172.21.15.160:6803/3146107361","heartbeat_back_addr":"172.21.15.160:6807/3146107361","heartbeat_front_addr":"172.21.15.160:6805/3146107361","state":["exists","up"]},{"osd":2,"uuid":"91ba96d0-9d5c-4128-98cf-517ac651af19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6801","nonce":2239917243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6803","nonce":2239917243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6807","nonce":2239917243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2239917243},{"type":"v1","addr":"172.21.15.183:6805","nonce":2239917243}]},"public_addr":"172.21.15.183:6801/2239917243","cluster_addr":"172.21.15.183:6803/2239917243","heartbeat_back_addr":"172.21.15.183:6807/2239917243","heartbeat_front_addr":"172.21.15.183:6805/2239917243","state":["exists","up"]},{"osd":3,"uuid":"e932bb8c-4b48-4cf1-8f02-4c151c98a009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6811","nonce":1981116547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6813","nonce":1981116547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6817","nonce":1981116547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1981116547},{"type":"v1","addr":"172.21.15.144:6815","nonce":1981116547}]},"public_addr":"172.21.15.144:6811/1981116547","cluster_addr":"172.21.15.144:6813/1981116547","heartbeat_back_addr":"172.21.15.144:6817/1981116547","heartbeat_front_addr":"172.21.15.144:6815/1981116547","state":["exists","up"]},{"osd":4,"uuid":"f6f12400-2f76-4c60-88e2-d06235484603","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6809","nonce":2452808824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6811","nonce":2452808824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6815","nonce":2452808824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2452808824},{"type":"v1","addr":"172.21.15.160:6813","nonce":2452808824}]},"public_addr":"172.21.15.160:6809/2452808824","cluster_addr":"172.21.15.160:6811/2452808824","heartbeat_back_addr":"172.21.15.160:6815/2452808824","heartbeat_front_addr":"172.21.15.160:6813/2452808824","state":["exists","up"]},{"osd":5,"uuid":"5b45f883-af04-42bd-a38e-38e88cc49515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6809","nonce":1593211139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6811","nonce":1593211139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6815","nonce":1593211139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":1593211139},{"type":"v1","addr":"172.21.15.183:6813","nonce":1593211139}]},"public_addr":"172.21.15.183:6809/1593211139","cluster_addr":"172.21.15.183:6811/1593211139","heartbeat_back_addr":"172.21.15.183:6815/1593211139","heartbeat_front_addr":"172.21.15.183:6813/1593211139","state":["exists","up"]},{"osd":6,"uuid":"aedef072-4034-47cb-bab6-11f1fd44d82a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6819","nonce":1312925195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6821","nonce":1312925195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6825","nonce":1312925195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1312925195},{"type":"v1","addr":"172.21.15.144:6823","nonce":1312925195}]},"public_addr":"172.21.15.144:6819/1312925195","cluster_addr":"172.21.15.144:6821/1312925195","heartbeat_back_addr":"172.21.15.144:6825/1312925195","heartbeat_front_addr":"172.21.15.144:6823/1312925195","state":["exists","up"]},{"osd":7,"uuid":"23dda395-05ca-4eef-8655-74edaefbfe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6817","nonce":602437452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6819","nonce":602437452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6823","nonce":602437452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":602437452},{"type":"v1","addr":"172.21.15.160:6821","nonce":602437452}]},"public_addr":"172.21.15.160:6817/602437452","cluster_addr":"172.21.15.160:6819/602437452","heartbeat_back_addr":"172.21.15.160:6823/602437452","heartbeat_front_addr":"172.21.15.160:6821/602437452","state":["exists","up"]},{"osd":8,"uuid":"b0afc0c4-8c60-46bf-a684-f418e6d1ec5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6817","nonce":2430686973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6819","nonce":2430686973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6823","nonce":2430686973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":2430686973},{"type":"v1","addr":"172.21.15.183:6821","nonce":2430686973}]},"public_addr":"172.21.15.183:6817/2430686973","cluster_addr":"172.21.15.183:6819/2430686973","heartbeat_back_addr":"172.21.15.183:6823/2430686973","heartbeat_front_addr":"172.21.15.183:6821/2430686973","state":["exists","up"]},{"osd":9,"uuid":"8f586925-f5c7-495c-a472-0d3e6af7dfdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6827","nonce":2364379216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6829","nonce":2364379216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6833","nonce":2364379216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2364379216},{"type":"v1","addr":"172.21.15.144:6831","nonce":2364379216}]},"public_addr":"172.21.15.144:6827/2364379216","cluster_addr":"172.21.15.144:6829/2364379216","heartbeat_back_addr":"172.21.15.144:6833/2364379216","heartbeat_front_addr":"172.21.15.144:6831/2364379216","state":["exists","up"]},{"osd":10,"uuid":"fac8da53-9c38-4368-9278-612c04a06498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6825","nonce":949715799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6827","nonce":949715799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6831","nonce":949715799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":949715799},{"type":"v1","addr":"172.21.15.160:6829","nonce":949715799}]},"public_addr":"172.21.15.160:6825/949715799","cluster_addr":"172.21.15.160:6827/949715799","heartbeat_back_addr":"172.21.15.160:6831/949715799","heartbeat_front_addr":"172.21.15.160:6829/949715799","state":["exists","up"]},{"osd":11,"uuid":"7f00c381-b270-461b-a3be-ae5822a8affb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6825","nonce":2003058932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6827","nonce":2003058932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6831","nonce":2003058932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2003058932},{"type":"v1","addr":"172.21.15.183:6829","nonce":2003058932}]},"public_addr":"172.21.15.183:6825/2003058932","cluster_addr":"172.21.15.183:6827/2003058932","heartbeat_back_addr":"172.21.15.183:6831/2003058932","heartbeat_front_addr":"172.21.15.183:6829/2003058932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:16.463565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:51:40.708051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:02.363129+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:34.519298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:52:58.813472+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:22.120980+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:53:55.406451+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:20.754877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:54:44.861203+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:18.215675+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:55:43.679547+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T10:56:07.832049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6801/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/288744986":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/563407546":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:6801/1828713044":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2087175124":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/3434609332":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/2275934374":"2024-07-27T10:47:54.806379+0000","172.21.15.144:0/2481813561":"2024-07-27T10:49:42.802348+0000","172.21.15.144:6800/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/1824026059":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6801/2854801065":"2024-07-27T10:48:24.009859+0000","172.21.15.144:6800/1820197496":"2024-07-27T10:49:42.802348+0000","172.21.15.144:0/3586311140":"2024-07-27T10:48:24.009859+0000","172.21.15.144:0/3284599906":"2024-07-27T10:49:42.802348+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:02:35.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9280b3c20 msgr2=0x7fc9280b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:35.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9280b3c20 0x7fc9280b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc92c0095f0 tx=0x7fc92c005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:35.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 msgr2=0x7fc94019bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:35.724 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc94019bff0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc934005ab0 tx=0x7fc934003c10 comp rx=0 tx=0).stop 2024-07-26T11:02:35.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 shutdown_connections 2024-07-26T11:02:35.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc9280b3c20 0x7fc9280b6130 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc94010b880 0x7fc9401a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc94010af30 0x7fc94019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 --2- 172.21.15.144:0/3307291501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9401055f0 0x7fc94019bff0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:35.725 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.721+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 >> 172.21.15.144:0/3307291501 conn(0x7fc940101060 msgr2=0x7fc940103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:35.726 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.722+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 shutdown_connections 2024-07-26T11:02:35.726 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:35.722+0000 7fc947200640 1 -- 172.21.15.144:0/3307291501 wait complete. 2024-07-26T11:02:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:36 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3307291501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:02:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:36 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3307291501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:02:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:36 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3307291501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:02:36.651 DEBUG:tasks.fwd_scrub:Yielding 2024-07-26T11:02:36.651 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T11:02:36.661 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:02:36.661 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T11:02:36.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-07-26T11:02:36.661 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-07-26T11:02:36.662 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:02:36.662 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 -- 172.21.15.144:0/3711088373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c10f8b0 msgr2=0x7f4b6c111d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 --2- 172.21.15.144:0/3711088373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c10f8b0 0x7f4b6c111d10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009ca0 tx=0x7f4b6402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 -- 172.21.15.144:0/3711088373 shutdown_connections 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 --2- 172.21.15.144:0/3711088373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c10f8b0 0x7f4b6c111d10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 --2- 172.21.15.144:0/3711088373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c075b30 0x7f4b6c10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 --2- 172.21.15.144:0/3711088373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 -- 172.21.15.144:0/3711088373 >> 172.21.15.144:0/3711088373 conn(0x7f4b6c0fedc0 msgr2=0x7f4b6c101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:36.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 -- 172.21.15.144:0/3711088373 shutdown_connections 2024-07-26T11:02:36.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.917+0000 7f4b70a00640 1 -- 172.21.15.144:0/3711088373 wait complete. 2024-07-26T11:02:36.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.918+0000 7f4b70a00640 1 Processor -- start 2024-07-26T11:02:36.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 -- start start 2024-07-26T11:02:36.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c075b30 0x7f4b6c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b6c10f8b0 0x7f4b6c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1154b0 con 0x7f4b6c10f8b0 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c115330 con 0x7f4b6c075b30 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b70a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c115630 con 0x7f4b6c0751e0 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b6a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b6a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c19beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56062/0 (socket says 172.21.15.144:56062) 2024-07-26T11:02:36.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.919+0000 7f4b6a800640 1 -- 172.21.15.144:0/1665117105 learned_addr learned my addr 172.21.15.144:0/1665117105 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6b200640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b6c10f8b0 0x7f4b6c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b63e00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c075b30 0x7f4b6c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6a800640 1 -- 172.21.15.144:0/1665117105 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c075b30 msgr2=0x7f4b6c19c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6a800640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c075b30 0x7f4b6c19c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6a800640 1 -- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b6c10f8b0 msgr2=0x7f4b6c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6a800640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b6c10f8b0 0x7f4b6c1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:36.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6a800640 1 -- 172.21.15.144:0/1665117105 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c1a1de0 con 0x7f4b6c0751e0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.920+0000 7f4b6a800640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c19beb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00e9f0 tx=0x7f4b5c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.921+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00cd80 con 0x7f4b6c0751e0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.921+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b5c0108e0 con 0x7f4b6c0751e0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.921+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c1a20d0 con 0x7f4b6c0751e0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.921+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c018980 con 0x7f4b6c0751e0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.921+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1a8860 con 0x7f4b6c0751e0 2024-07-26T11:02:36.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.922+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c0755f0 con 0x7f4b6c0751e0 2024-07-26T11:02:36.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.924+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f4b5c010430 con 0x7f4b6c0751e0 2024-07-26T11:02:36.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.926+0000 7f4b63400640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4b500b3ca0 0x7f4b500b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:36.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.926+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f4b5c0e48a0 con 0x7f4b6c0751e0 2024-07-26T11:02:36.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.926+0000 7f4b63e00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4b500b3ca0 0x7f4b500b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:36.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.927+0000 7f4b63e00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4b500b3ca0 0x7f4b500b61b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0ffd20 tx=0x7f4b54005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:36.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:36.927+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b5c0ab3b0 con 0x7f4b6c0751e0 2024-07-26T11:02:37.141 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:37.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.176+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4b6c109ad0 con 0x7f4b500b3ca0 2024-07-26T11:02:37.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.398+0000 7f4b63400640 1 -- 172.21.15.144:0/1665117105 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4b6c109ad0 con 0x7f4b500b3ca0 2024-07-26T11:02:37.401 INFO:teuthology.orchestra.run.smithi144.stdout:Schedule set for path / 2024-07-26T11:02:37.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4b500b3ca0 msgr2=0x7f4b500b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:37.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4b500b3ca0 0x7f4b500b61b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0ffd20 tx=0x7f4b54005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:37.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 msgr2=0x7f4b6c19beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c19beb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00e9f0 tx=0x7f4b5c00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 shutdown_connections 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4b500b3ca0 0x7f4b500b61b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b6c10f8b0 0x7f4b6c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c075b30 0x7f4b6c19c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 --2- 172.21.15.144:0/1665117105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b6c0751e0 0x7f4b6c19beb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.400+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 >> 172.21.15.144:0/1665117105 conn(0x7f4b6c0fedc0 msgr2=0x7f4b6c1093c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.401+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 shutdown_connections 2024-07-26T11:02:37.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.401+0000 7f4b70a00640 1 -- 172.21.15.144:0/1665117105 wait complete. 2024-07-26T11:02:37.429 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-07-26T11:02:37.488 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:37 smithi144 ceph-mon[32129]: pgmap v465: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:02:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:37 smithi160 ceph-mon[36715]: pgmap v465: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:02:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:37 smithi183 ceph-mon[37273]: pgmap v465: 129 pgs: 129 active+clean; 652 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 -- 172.21.15.144:0/2962292349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04103510 msgr2=0x7ffa04103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 --2- 172.21.15.144:0/2962292349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04103510 0x7ffa04103920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009900 tx=0x7ff9f802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 -- 172.21.15.144:0/2962292349 shutdown_connections 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 --2- 172.21.15.144:0/2962292349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa04109790 0x7ffa0410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 --2- 172.21.15.144:0/2962292349 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa04109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 --2- 172.21.15.144:0/2962292349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04103510 0x7ffa04103920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 -- 172.21.15.144:0/2962292349 >> 172.21.15.144:0/2962292349 conn(0x7ffa040fef50 msgr2=0x7ffa041013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:37.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 -- 172.21.15.144:0/2962292349 shutdown_connections 2024-07-26T11:02:37.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.679+0000 7ffa0c400640 1 -- 172.21.15.144:0/2962292349 wait complete. 2024-07-26T11:02:37.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.680+0000 7ffa0c400640 1 Processor -- start 2024-07-26T11:02:37.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 -- start start 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa04103510 0x7ffa04199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa0419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04109790 0x7ffa0419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041134b0 con 0x7ffa04103510 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04113330 con 0x7ffa04108e40 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa0c400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04113630 con 0x7ffa04109790 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.681+0000 7ffa09000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa0419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa0a400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04109790 0x7ffa0419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa0419a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47820/0 (socket says 172.21.15.144:47820) 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 -- 172.21.15.144:0/3152392141 learned_addr learned my addr 172.21.15.144:0/3152392141 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09a00640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa04103510 0x7ffa04199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:37.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 -- 172.21.15.144:0/3152392141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04109790 msgr2=0x7ffa0419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:37.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04109790 0x7ffa0419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 -- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa04103510 msgr2=0x7ffa04199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:37.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa04103510 0x7ffa04199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:37.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 -- 172.21.15.144:0/3152392141 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0419fe20 con 0x7ffa04108e40 2024-07-26T11:02:37.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa0a400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04109790 0x7ffa0419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.682+0000 7ffa09000640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa0419a490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000e9c0 tx=0x7ff9f000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.683+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f000ccc0 con 0x7ffa04108e40 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.683+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa041a0110 con 0x7ffa04108e40 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.683+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9f00054d0 con 0x7ffa04108e40 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.683+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0010430 con 0x7ffa04108e40 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.683+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a68a0 con 0x7ffa04108e40 2024-07-26T11:02:37.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.684+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa04103920 con 0x7ffa04108e40 2024-07-26T11:02:37.690 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.687+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7ff9f0005020 con 0x7ffa04108e40 2024-07-26T11:02:37.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.690+0000 7ffa02000640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff9ec0b3c20 0x7ff9ec0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:37.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.690+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7ff9f00e44d0 con 0x7ffa04108e40 2024-07-26T11:02:37.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.690+0000 7ffa09a00640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff9ec0b3c20 0x7ff9ec0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:37.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.690+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff9f00ab6d0 con 0x7ffa04108e40 2024-07-26T11:02:37.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.691+0000 7ffa09a00640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff9ec0b3c20 0x7ff9ec0b6130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8005ab0 tx=0x7ff9f8005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:37.925+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7ffa04063230 con 0x7ff9ec0b3c20 2024-07-26T11:02:38.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.043+0000 7ffa02000640 1 -- 172.21.15.144:0/3152392141 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7ffa04063230 con 0x7ff9ec0b3c20 2024-07-26T11:02:38.046 INFO:teuthology.orchestra.run.smithi144.stdout:Retention added to path / 2024-07-26T11:02:38.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff9ec0b3c20 msgr2=0x7ff9ec0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff9ec0b3c20 0x7ff9ec0b6130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8005ab0 tx=0x7ff9f8005a20 comp rx=0 tx=0).stop 2024-07-26T11:02:38.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 msgr2=0x7ffa0419a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa0419a490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000e9c0 tx=0x7ff9f000ee80 comp rx=0 tx=0).stop 2024-07-26T11:02:38.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 shutdown_connections 2024-07-26T11:02:38.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ff9ec0b3c20 0x7ff9ec0b6130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa04109790 0x7ffa0419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffa04108e40 0x7ffa0419a490 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 --2- 172.21.15.144:0/3152392141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa04103510 0x7ffa04199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 >> 172.21.15.144:0/3152392141 conn(0x7ffa040fef50 msgr2=0x7ffa04101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:38.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 shutdown_connections 2024-07-26T11:02:38.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.046+0000 7ffa0c400640 1 -- 172.21.15.144:0/3152392141 wait complete. 2024-07-26T11:02:38.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.321+0000 7f54dce00640 1 -- 172.21.15.144:0/2379510081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d81023e0 msgr2=0x7f54d81027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.321+0000 7f54dce00640 1 --2- 172.21.15.144:0/2379510081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d81023e0 0x7f54d81027f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009900 tx=0x7f54c802a410 comp rx=0 tx=0).stop 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 -- 172.21.15.144:0/2379510081 shutdown_connections 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 --2- 172.21.15.144:0/2379510081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54d8108660 0x7f54d810ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 --2- 172.21.15.144:0/2379510081 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f54d8107d10 0x7f54d8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 --2- 172.21.15.144:0/2379510081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d81023e0 0x7f54d81027f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 -- 172.21.15.144:0/2379510081 >> 172.21.15.144:0/2379510081 conn(0x7f54d807a170 msgr2=0x7f54d807a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:38.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 -- 172.21.15.144:0/2379510081 shutdown_connections 2024-07-26T11:02:38.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.322+0000 7f54dce00640 1 -- 172.21.15.144:0/2379510081 wait complete. 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.323+0000 7f54dce00640 1 Processor -- start 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.323+0000 7f54dce00640 1 -- start start 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54dce00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54d81023e0 0x7f54d819c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54dce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 0x7f54d819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54dce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f54d8108660 0x7f54d81a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54dce00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f54d8113470 con 0x7f54d81023e0 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54dce00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81132f0 con 0x7f54d8108660 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54dce00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81135f0 con 0x7f54d8107d10 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 0x7f54d819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54d5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54d81023e0 0x7f54d819c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 0x7f54d819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56116/0 (socket says 172.21.15.144:56116) 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 -- 172.21.15.144:0/2332714386 learned_addr learned my addr 172.21.15.144:0/2332714386 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54d6800640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f54d8108660 0x7f54d81a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 -- 172.21.15.144:0/2332714386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f54d8108660 msgr2=0x7f54d81a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f54d8108660 0x7f54d81a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 -- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54d81023e0 msgr2=0x7f54d819c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54d81023e0 0x7f54d819c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.324+0000 7f54cfe00640 1 -- 172.21.15.144:0/2332714386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d81a2020 con 0x7f54d8107d10 2024-07-26T11:02:38.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.325+0000 7f54cfe00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 0x7f54d819c690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e9c0 tx=0x7f54c000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:38.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.325+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c000ccc0 con 0x7f54d8107d10 2024-07-26T11:02:38.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.325+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c00054d0 con 0x7f54d8107d10 2024-07-26T11:02:38.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.325+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d81a2310 con 0x7f54d8107d10 2024-07-26T11:02:38.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.325+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0010430 con 0x7f54d8107d10 2024-07-26T11:02:38.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.325+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d81a8aa0 con 0x7f54d8107d10 2024-07-26T11:02:38.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.326+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d81027f0 con 0x7f54d8107d10 2024-07-26T11:02:38.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.328+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f54c0005020 con 0x7f54d8107d10 2024-07-26T11:02:38.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.330+0000 7f54cea00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f54bc0b3d30 0x7f54bc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.330+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f54c00e4510 con 0x7f54d8107d10 2024-07-26T11:02:38.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.330+0000 7f54d5e00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f54bc0b3d30 0x7f54bc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.331+0000 7f54d5e00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f54bc0b3d30 0x7f54bc0b6240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54c8005ab0 tx=0x7f54c8005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:38.336 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:38 smithi144 ceph-mon[32129]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:38.336 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:38 smithi144 ceph-mon[32129]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:38.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.334+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f54c00ab570 con 0x7f54d8107d10 2024-07-26T11:02:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:38 smithi160 ceph-mon[36715]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:38 smithi160 ceph-mon[36715]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:38.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.570+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f54d8106230 con 0x7f54bc0b3d30 2024-07-26T11:02:38.584 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.581+0000 7f54cea00640 1 -- 172.21.15.144:0/2332714386 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f54d8106230 con 0x7f54bc0b3d30 2024-07-26T11:02:38.584 INFO:teuthology.orchestra.run.smithi144.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-07-26T00:00:00", "created": "2024-07-26T11:02:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f54bc0b3d30 msgr2=0x7f54bc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f54bc0b3d30 0x7f54bc0b6240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54c8005ab0 tx=0x7f54c8005a20 comp rx=0 tx=0).stop 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 msgr2=0x7f54d819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 0x7f54d819c690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e9c0 tx=0x7f54c000ee80 comp rx=0 tx=0).stop 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 shutdown_connections 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f54bc0b3d30 0x7f54bc0b6240 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f54d8108660 0x7f54d81a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.587 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54d8107d10 0x7f54d819c690 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 --2- 172.21.15.144:0/2332714386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54d81023e0 0x7f54d819c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 >> 172.21.15.144:0/2332714386 conn(0x7f54d807a170 msgr2=0x7f54d8105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:38.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 shutdown_connections 2024-07-26T11:02:38.588 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.584+0000 7f54dce00640 1 -- 172.21.15.144:0/2332714386 wait complete. 2024-07-26T11:02:38.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-07-26T11:02:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:38 smithi183 ceph-mon[37273]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:38 smithi183 ceph-mon[37273]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.863+0000 7f98c2600640 1 -- 172.21.15.144:0/3080993827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc102220 msgr2=0x7f98bc102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.863+0000 7f98c2600640 1 --2- 172.21.15.144:0/3080993827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc102220 0x7f98bc102630 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009ca0 tx=0x7f98b402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 -- 172.21.15.144:0/3080993827 shutdown_connections 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 --2- 172.21.15.144:0/3080993827 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98bc1084a0 0x7f98bc10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 --2- 172.21.15.144:0/3080993827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc107b50 0x7f98bc107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 --2- 172.21.15.144:0/3080993827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc102220 0x7f98bc102630 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 -- 172.21.15.144:0/3080993827 >> 172.21.15.144:0/3080993827 conn(0x7f98bc0fdc60 msgr2=0x7f98bc1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 -- 172.21.15.144:0/3080993827 shutdown_connections 2024-07-26T11:02:38.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.864+0000 7f98c2600640 1 -- 172.21.15.144:0/3080993827 wait complete. 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.865+0000 7f98c2600640 1 Processor -- start 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.865+0000 7f98c2600640 1 -- start start 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.865+0000 7f98c2600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 0x7f98bc10f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.865+0000 7f98c2600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98bc107b50 0x7f98bc10fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.865+0000 7f98c2600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc1084a0 0x7f98bc110230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c2600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc113280 con 0x7f98bc1084a0 2024-07-26T11:02:38.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c2600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc113100 con 0x7f98bc107b50 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c2600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc113400 con 0x7f98bc102220 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 0x7f98bc10f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc1084a0 0x7f98bc110230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 0x7f98bc10f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56144/0 (socket says 172.21.15.144:56144) 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 -- 172.21.15.144:0/90207646 learned_addr learned my addr 172.21.15.144:0/90207646 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98bbe00640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98bc107b50 0x7f98bc10fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 -- 172.21.15.144:0/90207646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98bc107b50 msgr2=0x7f98bc10fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98bc107b50 0x7f98bc10fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 -- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc1084a0 msgr2=0x7f98bc110230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc1084a0 0x7f98bc110230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 -- 172.21.15.144:0/90207646 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98bc1a7f90 con 0x7f98bc102220 2024-07-26T11:02:38.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.866+0000 7f98c1200640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 0x7f98bc10f7b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98b4033040 tx=0x7f98b40316f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:38.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.867+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4003b70 con 0x7f98bc102220 2024-07-26T11:02:38.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.867+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98bc1a8220 con 0x7f98bc102220 2024-07-26T11:02:38.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.867+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b40318e0 con 0x7f98bc102220 2024-07-26T11:02:38.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.867+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b402f950 con 0x7f98bc102220 2024-07-26T11:02:38.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.867+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98bc1a8770 con 0x7f98bc102220 2024-07-26T11:02:38.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.868+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98bc102630 con 0x7f98bc102220 2024-07-26T11:02:38.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.871+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f98b4004b40 con 0x7f98bc102220 2024-07-26T11:02:38.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.872+0000 7f98b9600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98a00b3ce0 0x7f98a00b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:38.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.872+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f98b4105700 con 0x7f98bc102220 2024-07-26T11:02:38.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.872+0000 7f98bbe00640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98a00b3ce0 0x7f98a00b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:38.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.873+0000 7f98bbe00640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98a00b3ce0 0x7f98a00b61f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f98bc1076b0 tx=0x7f98a4005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:38.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:38.874+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f98b40cc810 con 0x7f98bc102220 2024-07-26T11:02:39.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.121+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 --> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f98bc063230 con 0x7f98a00b3ce0 2024-07-26T11:02:39.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.135+0000 7f98b9600640 1 -- 172.21.15.144:0/90207646 <== mgr.24109 v2:172.21.15.144:6800/2154879590 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f98bc063230 con 0x7f98a00b3ce0 2024-07-26T11:02:39.138 INFO:teuthology.orchestra.run.smithi144.stdout:/ 1m 6m3h 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98a00b3ce0 msgr2=0x7f98a00b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98a00b3ce0 0x7f98a00b61f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f98bc1076b0 tx=0x7f98a4005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 msgr2=0x7f98bc10f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 0x7f98bc10f7b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98b4033040 tx=0x7f98b40316f0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 shutdown_connections 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98a00b3ce0 0x7f98a00b61f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98bc1084a0 0x7f98bc110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98bc107b50 0x7f98bc10fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 --2- 172.21.15.144:0/90207646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98bc102220 0x7f98bc10f7b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 >> 172.21.15.144:0/90207646 conn(0x7f98bc0fdc60 msgr2=0x7f98bc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 shutdown_connections 2024-07-26T11:02:39.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.138+0000 7f98c2600640 1 -- 172.21.15.144:0/90207646 wait complete. 2024-07-26T11:02:39.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-07-26T11:02:39.243 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-07-26T11:02:39.254 INFO:teuthology.run_tasks:Running task background_exec... 2024-07-26T11:02:39.263 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi144.front.sepia.ceph.com 2024-07-26T11:02:39.263 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-07-26T11:02:39.268 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T11:02:39.280 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T11:02:39.280 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T11:02:39.281 INFO:tasks.workunit:timeout=6h 2024-07-26T11:02:39.281 INFO:tasks.workunit:cleanup=True 2024-07-26T11:02:39.281 DEBUG:teuthology.orchestra.run.smithi144:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6bh/107d Inode: 1099511628283 Links: 2 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-07-26 10:59:02.731219484 +0000 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-07-26 10:59:02.731219484 +0000 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-07-26 11:02:18.336210279 +0000 2024-07-26T11:02:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-07-26 10:59:02.731219484 +0000 2024-07-26T11:02:39.324 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-26T11:02:39.325 DEBUG:teuthology.orchestra.run.smithi144:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.330+0000 7fcbac000640 1 -- 172.21.15.144:0/2074798423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba410b0f0 msgr2=0x7fcba410b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.330+0000 7fcbac000640 1 --2- 172.21.15.144:0/2074798423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba410b0f0 0x7fcba410b500 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009900 tx=0x7fcb9002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 -- 172.21.15.144:0/2074798423 shutdown_connections 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 --2- 172.21.15.144:0/2074798423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 0x7fcba4112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 --2- 172.21.15.144:0/2074798423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba410b0f0 0x7fcba410b500 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 --2- 172.21.15.144:0/2074798423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba41057b0 0x7fcba4105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 -- 172.21.15.144:0/2074798423 >> 172.21.15.144:0/2074798423 conn(0x7fcba4101240 msgr2=0x7fcba4103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 -- 172.21.15.144:0/2074798423 shutdown_connections 2024-07-26T11:02:39.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.331+0000 7fcbac000640 1 -- 172.21.15.144:0/2074798423 wait complete. 2024-07-26T11:02:39.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.332+0000 7fcbac000640 1 Processor -- start 2024-07-26T11:02:39.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.332+0000 7fcbac000640 1 -- start start 2024-07-26T11:02:39.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbac000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba41057b0 0x7fcba419c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbac000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba410b0f0 0x7fcba419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbac000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 0x7fcba41a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbac000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41156d0 con 0x7fcba41057b0 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbac000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcba4115550 con 0x7fcba410ba40 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbac000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcba4115850 con 0x7fcba410b0f0 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba41057b0 0x7fcba419c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba41057b0 0x7fcba419c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41628/0 (socket says 172.21.15.144:41628) 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcba9600640 1 -- 172.21.15.144:0/3364209254 learned_addr learned my addr 172.21.15.144:0/3364209254 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcba8c00640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba410b0f0 0x7fcba419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.333+0000 7fcbaa000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 0x7fcba41a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.334+0000 7fcbaa000640 1 -- 172.21.15.144:0/3364209254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba410b0f0 msgr2=0x7fcba419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.334+0000 7fcbaa000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba410b0f0 0x7fcba419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.334+0000 7fcbaa000640 1 -- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba41057b0 msgr2=0x7fcba419c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.334+0000 7fcbaa000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba41057b0 0x7fcba419c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.334+0000 7fcbaa000640 1 -- 172.21.15.144:0/3364209254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba41a2070 con 0x7fcba410ba40 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.334+0000 7fcbaa000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 0x7fcba41a1940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009fd0 tx=0x7fcb9c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.335+0000 7fcba2000640 1 -- 172.21.15.144:0/3364209254 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c00ce20 con 0x7fcba410ba40 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.335+0000 7fcba2000640 1 -- 172.21.15.144:0/3364209254 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb9c005020 con 0x7fcba410ba40 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.335+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba41a2360 con 0x7fcba410ba40 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.335+0000 7fcba2000640 1 -- 172.21.15.144:0/3364209254 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c010690 con 0x7fcba410ba40 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.335+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba41a8ac0 con 0x7fcba410ba40 2024-07-26T11:02:39.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.338+0000 7fcbb2c55740 1 Processor -- start 2024-07-26T11:02:39.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.340+0000 7fcba2000640 1 -- 172.21.15.144:0/3364209254 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7fcb9c010830 con 0x7fcba410ba40 2024-07-26T11:02:39.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.340+0000 7fcbb2c55740 1 -- start start 2024-07-26T11:02:39.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 0x55b991fd3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c06490 0x55b991fd42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fd4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b991f4abe0 con 0x55b991c2f350 2024-07-26T11:02:39.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b991f3a0e0 con 0x55b991c06490 2024-07-26T11:02:39.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b991f3e120 con 0x55b991c0e670 2024-07-26T11:02:39.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcba8c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 0x55b991fd3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcba8c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 0x55b991fd3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41640/0 (socket says 172.21.15.144:41640) 2024-07-26T11:02:39.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcba8c00640 1 -- 172.21.15.144:0/2577720943 learned_addr learned my addr 172.21.15.144:0/2577720943 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:39.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.341+0000 7fcba9600640 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c06490 0x55b991fd42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fd4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba8c00640 1 -- 172.21.15.144:0/2577720943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 msgr2=0x55b991fd4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fd4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba8c00640 1 -- 172.21.15.144:0/2577720943 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c06490 msgr2=0x55b991fd42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c06490 0x55b991fd42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba8c00640 1 -- 172.21.15.144:0/2577720943 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b991c9a930 con 0x55b991c2f350 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.342+0000 7fcba2000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcb8c0b3d30 0x7fcb8c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.343+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 0x55b991fd3d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb90005940 tx=0x7fcb900058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3851926f0583de27 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:39.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.343+0000 7fcba2000640 1 -- 172.21.15.144:0/3364209254 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fcb9c0e4980 con 0x7fcba410ba40 2024-07-26T11:02:39.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.343+0000 7fcba9600640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcb8c0b3d30 0x7fcb8c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.343+0000 7fcb96600640 1 -- 172.21.15.144:0/2577720943 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90035070 con 0x55b991c2f350 2024-07-26T11:02:39.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.343+0000 7fcb96600640 1 -- 172.21.15.144:0/2577720943 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb900318c0 con 0x55b991c2f350 2024-07-26T11:02:39.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.343+0000 7fcb96600640 1 -- 172.21.15.144:0/2577720943 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb900046c0 con 0x55b991c2f350 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2577720943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 msgr2=0x55b991fd3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 0x55b991fd3d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb90005940 tx=0x7fcb900058b0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcba9600640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcb8c0b3d30 0x7fcb8c0b6240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55b991fd2a00 tx=0x7fcb98005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2577720943 shutdown_connections 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fd4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c06490 0x55b991fd42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2577720943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c2f350 0x55b991fd3d80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.344+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2577720943 >> 172.21.15.144:0/2577720943 conn(0x55b991c23740 msgr2=0x55b991fd1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.345+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2577720943 shutdown_connections 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.345+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2577720943 wait complete. 2024-07-26T11:02:39.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.345+0000 7fcbb2c55740 1 -- start start 2024-07-26T11:02:39.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.345+0000 7fcbb2c55740 1 Processor -- start 2024-07-26T11:02:39.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c06490 0x55b991fdf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fdfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991bedcd0 0x55b991feb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b991d9d890 con 0x55b991c06490 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b991ed6320 con 0x55b991bedcd0 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b991fe2500 con 0x55b991c0e670 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcbb2c55740 10 client.0 _ll_register_callbacks cb 0x55b991f02000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c06490 0x55b991fdf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcba8c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fdfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c06490 0x55b991fdf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41656/0 (socket says 172.21.15.144:41656) 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.346+0000 7fcba9600640 1 -- 172.21.15.144:0/184523328 learned_addr learned my addr 172.21.15.144:0/184523328 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:39.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcbaa000640 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991bedcd0 0x55b991feb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcba8c00640 1 -- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991bedcd0 msgr2=0x55b991feb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcba8c00640 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991bedcd0 0x55b991feb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcba8c00640 1 -- 172.21.15.144:0/184523328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c06490 msgr2=0x55b991fdf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcba8c00640 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c06490 0x55b991fdf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcba8c00640 1 -- 172.21.15.144:0/184523328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b991fe02d0 con 0x55b991c0e670 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcba8c00640 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fdfa30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcb9003f000 tx=0x7fcb90005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7162a39ac03d585b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:39.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcb8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:02:39.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcb8aa00640 1 -- 172.21.15.144:0/184523328 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90035050 con 0x55b991c0e670 2024-07-26T11:02:39.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.347+0000 7fcb8aa00640 1 -- 172.21.15.144:0/184523328 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fcb900035f0 con 0x55b991c0e670 2024-07-26T11:02:39.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.348+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb900318c0 con 0x55b991c0e670 2024-07-26T11:02:39.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.348+0000 7fcb8aa00640 1 -- 172.21.15.144:0/184523328 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb90055910 con 0x55b991c0e670 2024-07-26T11:02:39.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.348+0000 7fcbb2c55740 10 client.25450 fetch_fsmap learned FSMap version 26 2024-07-26T11:02:39.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.348+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b991fee460 con 0x55b991c0e670 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.348+0000 7fcb8aa00640 1 -- 172.21.15.144:0/184523328 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb900318c0 con 0x55b991c0e670 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.348+0000 7fcb8aa00640 1 -- 172.21.15.144:0/184523328 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcb900550d0 con 0x55b991c0e670 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcbb2c55740 10 client.25450 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcbb2c55740 10 client.25450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcbb2c55740 20 client.25450 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b991bc3080 0x55b991ff17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcbb2c55740 4 client.25450 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b991f4cb30 con 0x55b991bc3080 2024-07-26T11:02:39.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.349+0000 7fcba9600640 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b991bc3080 0x55b991ff17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.350+0000 7fcba9600640 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b991bc3080 0x55b991ff17c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de39bd12c874d4ee server_cookie=3f4b4359cf394a3d in_seq=0 out_seq=0 2024-07-26T11:02:39.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.350+0000 7fcb8aa00640 10 client.25450 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcb8aa00640 1 -- 172.21.15.144:0/184523328 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b991f4cb30 con 0x55b991bc3080 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcb8aa00640 10 client.25450 handle_command_reply: tid=0 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 client.25450 shutdown 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 msgr2=0x55b991fdfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fdfa30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcb9003f000 tx=0x7fcb90005fd0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 shutdown_connections 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b991bc3080 0x55b991ff17c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991bedcd0 0x55b991feb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c0e670 0x55b991fdfa30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/184523328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991c06490 0x55b991fdf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 >> 172.21.15.144:0/184523328 conn(0x55b991c23740 msgr2=0x55b991fd6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcb8aa00640 0 client.25450 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:39.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 shutdown_connections 2024-07-26T11:02:39.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 1 -- 172.21.15.144:0/184523328 wait complete. 2024-07-26T11:02:39.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.351+0000 7fcbb2c55740 20 client.25450 trim_cache size 0 max 16384 2024-07-26T11:02:39.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.370+0000 7fcbb2c55740 1 Processor -- start 2024-07-26T11:02:39.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.370+0000 7fcbb2c55740 1 -- start start 2024-07-26T11:02:39.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991fdaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991fe13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991fd54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b5230 con 0x55b991bedcd0 2024-07-26T11:02:39.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b2b00 con 0x55b991c0e670 2024-07-26T11:02:39.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b0950 con 0x55b991c06490 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991fdaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991fdaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41662/0 (socket says 172.21.15.144:41662) 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcba9600640 1 -- 172.21.15.144:0/2565661331 learned_addr learned my addr 172.21.15.144:0/2565661331 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991fe13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.371+0000 7fcbaa000640 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991fd54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcba8c00640 1 -- 172.21.15.144:0/2565661331 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 msgr2=0x55b991fd54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991fd54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcba8c00640 1 -- 172.21.15.144:0/2565661331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 msgr2=0x55b991fdaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991fdaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcba8c00640 1 -- 172.21.15.144:0/2565661331 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b991fd6be0 con 0x55b991c06490 2024-07-26T11:02:39.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcba8c00640 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991fe13d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb90004000 tx=0x7fcb9002aa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97b890574d3b778 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:39.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcb95c00640 1 -- 172.21.15.144:0/2565661331 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90035040 con 0x55b991c06490 2024-07-26T11:02:39.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcb95c00640 1 -- 172.21.15.144:0/2565661331 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb90032d40 con 0x55b991c06490 2024-07-26T11:02:39.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.372+0000 7fcb95c00640 1 -- 172.21.15.144:0/2565661331 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90031a40 con 0x55b991c06490 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2565661331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 msgr2=0x55b991fe13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991fe13d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb90004000 tx=0x7fcb9002aa70 comp rx=0 tx=0).stop 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2565661331 shutdown_connections 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991fd54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991fe13d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/2565661331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991fdaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.373+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2565661331 >> 172.21.15.144:0/2565661331 conn(0x55b991c23740 msgr2=0x55b991fd4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.374+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2565661331 shutdown_connections 2024-07-26T11:02:39.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.374+0000 7fcbb2c55740 1 -- 172.21.15.144:0/2565661331 wait complete. 2024-07-26T11:02:39.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.374+0000 7fcbb2c55740 1 -- start start 2024-07-26T11:02:39.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.374+0000 7fcbb2c55740 1 Processor -- start 2024-07-26T11:02:39.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991ff2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991ff2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991ff3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b0fc0 con 0x55b991bedcd0 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b2430 con 0x55b991c0e670 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b1690 con 0x55b991c06490 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcbb2c55740 10 client.0 _ll_register_callbacks cb 0x7fcb98009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991ff2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcba9600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991ff2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41678/0 (socket says 172.21.15.144:41678) 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcba9600640 1 -- 172.21.15.144:0/989110604 learned_addr learned my addr 172.21.15.144:0/989110604 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.375+0000 7fcba8c00640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991ff2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcbaa000640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991ff3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcba8c00640 1 -- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 msgr2=0x55b991ff3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcba8c00640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991ff3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcba8c00640 1 -- 172.21.15.144:0/989110604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 msgr2=0x55b991ff2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:39.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcba8c00640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991ff2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcba8c00640 1 -- 172.21.15.144:0/989110604 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b992061520 con 0x55b991c06490 2024-07-26T11:02:39.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcbaa000640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991ff3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:39.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcba8c00640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991ff2bc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb90031440 tx=0x7fcb900558a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d3121c1947b6e45 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:39.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.376+0000 7fcb8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:02:39.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcb8aa00640 1 -- 172.21.15.144:0/989110604 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90062020 con 0x55b991c06490 2024-07-26T11:02:39.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb90032d40 con 0x55b991c06490 2024-07-26T11:02:39.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcb8aa00640 1 -- 172.21.15.144:0/989110604 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fcb900317c0 con 0x55b991c06490 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcb8aa00640 1 -- 172.21.15.144:0/989110604 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb90035040 con 0x55b991c06490 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcbb2c55740 10 client.25456 fetch_fsmap learned FSMap version 26 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b992062740 con 0x55b991c06490 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.377+0000 7fcb8aa00640 1 -- 172.21.15.144:0/989110604 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb90032d40 con 0x55b991c06490 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcb8aa00640 1 -- 172.21.15.144:0/989110604 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcb9005c020 con 0x55b991c06490 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcbb2c55740 10 client.25456 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcbb2c55740 10 client.25456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcbb2c55740 20 client.25456 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b992062b00 0x55b992064f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcbb2c55740 4 client.25456 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b991f485d0 con 0x55b992062b00 2024-07-26T11:02:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.378+0000 7fcba9600640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b992062b00 0x55b992064f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:39.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.379+0000 7fcb8aa00640 10 client.25456 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:39.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.379+0000 7fcba9600640 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b992062b00 0x55b992064f20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d7db7979b65062 server_cookie=eded81b2d009dcef in_seq=0 out_seq=0 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcb8aa00640 1 -- 172.21.15.144:0/989110604 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b991f485d0 con 0x55b992062b00 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcb8aa00640 10 client.25456 handle_command_reply: tid=0 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 client.25456 shutdown 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 msgr2=0x55b991ff2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991ff2bc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb90031440 tx=0x7fcb900558a0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 shutdown_connections 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b992062b00 0x55b992064f20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b991c0e670 0x55b991ff3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b991c06490 0x55b991ff2bc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 --2- 172.21.15.144:0/989110604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b991bedcd0 0x55b991ff2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcb8aa00640 0 client.25456 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.380+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 >> 172.21.15.144:0/989110604 conn(0x55b991c23740 msgr2=0x55b991fe1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.381+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 shutdown_connections 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.381+0000 7fcbb2c55740 1 -- 172.21.15.144:0/989110604 wait complete. 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.381+0000 7fcbb2c55740 20 client.25456 trim_cache size 0 max 16384 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stdout: "return_code": 0, 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stdout: "scrub_tag": "8c28678f-8ab5-4362-bc32-b404a53230bc", 2024-07-26T11:02:39.385 INFO:teuthology.orchestra.run.smithi144.stdout: "mode": "asynchronous" 2024-07-26T11:02:39.386 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:02:39.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcb8c0b3d30 msgr2=0x7fcb8c0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcb8c0b3d30 0x7fcb8c0b6240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55b991fd2a00 tx=0x7fcb98005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:39.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 msgr2=0x7fcba41a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 0x7fcba41a1940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009fd0 tx=0x7fcb9c00ef10 comp rx=0 tx=0).stop 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 shutdown_connections 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcb8c0b3d30 0x7fcb8c0b6240 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcba410ba40 0x7fcba41a1940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba410b0f0 0x7fcba419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 --2- 172.21.15.144:0/3364209254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcba41057b0 0x7fcba419c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.382+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 >> 172.21.15.144:0/3364209254 conn(0x7fcba4101240 msgr2=0x7fcba4103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.383+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 shutdown_connections 2024-07-26T11:02:39.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:39.383+0000 7fcbac000640 1 -- 172.21.15.144:0/3364209254 wait complete. 2024-07-26T11:02:39.414 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T11:02:39.494 INFO:tasks.workunit.client.0.smithi144.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T11:02:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:39 smithi160 ceph-mon[36715]: mgrmap e24: x(active, since 12m), standbys: z, y 2024-07-26T11:02:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:39 smithi160 ceph-mon[36715]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:39 smithi160 ceph-mon[36715]: pgmap v466: 129 pgs: 129 active+clean; 1.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 73 KiB/s wr, 14 op/s 2024-07-26T11:02:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:39 smithi160 ceph-mon[36715]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:39 smithi183 ceph-mon[37273]: mgrmap e24: x(active, since 12m), standbys: z, y 2024-07-26T11:02:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:39 smithi183 ceph-mon[37273]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:39 smithi183 ceph-mon[37273]: pgmap v466: 129 pgs: 129 active+clean; 1.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 73 KiB/s wr, 14 op/s 2024-07-26T11:02:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:39 smithi183 ceph-mon[37273]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:39 smithi144 ceph-mon[32129]: mgrmap e24: x(active, since 12m), standbys: z, y 2024-07-26T11:02:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:39 smithi144 ceph-mon[32129]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:39 smithi144 ceph-mon[32129]: pgmap v466: 129 pgs: 129 active+clean; 1.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 73 KiB/s wr, 14 op/s 2024-07-26T11:02:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:39 smithi144 ceph-mon[32129]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:40.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:02:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:40 smithi144 ceph-mon[32129]: scrub queued for path: / 2024-07-26T11:02:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:40 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:02:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:40 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:02:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:40 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:02:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:40 smithi160 ceph-mon[36715]: scrub queued for path: / 2024-07-26T11:02:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:40 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:02:40.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:40 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:02:40.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:40 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:02:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:40 smithi183 ceph-mon[37273]: scrub queued for path: / 2024-07-26T11:02:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:40 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:02:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:40 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:02:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:40 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:02:40.914 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:41 smithi144 ceph-mon[32129]: pgmap v467: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:41 smithi160 ceph-mon[36715]: pgmap v467: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:41 smithi183 ceph-mon[37273]: pgmap v467: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:43.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 -- 172.21.15.144:0/2239621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b010fc80 msgr2=0x7f05b01164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 --2- 172.21.15.144:0/2239621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b010fc80 0x7f05b01164b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009ca0 tx=0x7f05a802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 -- 172.21.15.144:0/2239621106 shutdown_connections 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 --2- 172.21.15.144:0/2239621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b010fc80 0x7f05b01164b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 --2- 172.21.15.144:0/2239621106 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010f330 0x7f05b010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 --2- 172.21.15.144:0/2239621106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b01099f0 0x7f05b0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 -- 172.21.15.144:0/2239621106 >> 172.21.15.144:0/2239621106 conn(0x7f05b0078ff0 msgr2=0x7f05b0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 -- 172.21.15.144:0/2239621106 shutdown_connections 2024-07-26T11:02:43.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.343+0000 7f05b4a00640 1 -- 172.21.15.144:0/2239621106 wait complete. 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 Processor -- start 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 -- start start 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b01099f0 0x7f05b01a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b010f330 0x7f05b01a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010fc80 0x7f05b01a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f05b0119990 con 0x7f05b01099f0 2024-07-26T11:02:43.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f05b0119810 con 0x7f05b010fc80 2024-07-26T11:02:43.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.345+0000 7f05b4a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f05b0119b10 con 0x7f05b010f330 2024-07-26T11:02:43.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b01099f0 0x7f05b01a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b01099f0 0x7f05b01a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:34784/0 (socket says 172.21.15.144:34784) 2024-07-26T11:02:43.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05af400640 1 -- 172.21.15.144:0/224818798 learned_addr learned my addr 172.21.15.144:0/224818798 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05afe00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010fc80 0x7f05b01a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05aea00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b010f330 0x7f05b01a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05afe00640 1 -- 172.21.15.144:0/224818798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b010f330 msgr2=0x7f05b01a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05afe00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b010f330 0x7f05b01a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05afe00640 1 -- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b01099f0 msgr2=0x7f05b01a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05afe00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b01099f0 0x7f05b01a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05afe00640 1 -- 172.21.15.144:0/224818798 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b01a61d0 con 0x7f05b010fc80 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.346+0000 7f05aea00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b010f330 0x7f05b01a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:43.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.347+0000 7f05afe00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010fc80 0x7f05b01a5aa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05a8005e30 tx=0x7f05a80317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:43.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.347+0000 7f05a7e00640 1 -- 172.21.15.144:0/224818798 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8003b70 con 0x7f05b010fc80 2024-07-26T11:02:43.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.347+0000 7f05a7e00640 1 -- 172.21.15.144:0/224818798 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05a8003d10 con 0x7f05b010fc80 2024-07-26T11:02:43.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.347+0000 7f05a7e00640 1 -- 172.21.15.144:0/224818798 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a802f970 con 0x7f05b010fc80 2024-07-26T11:02:43.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.347+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b01a6460 con 0x7f05b010fc80 2024-07-26T11:02:43.350 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.348+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b01acc20 con 0x7f05b010fc80 2024-07-26T11:02:43.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.351+0000 7f05bcc4a740 1 Processor -- start 2024-07-26T11:02:43.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.352+0000 7f05a7e00640 1 -- 172.21.15.144:0/224818798 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f05a802fb10 con 0x7f05b010fc80 2024-07-26T11:02:43.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 -- start start 2024-07-26T11:02:43.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014ff070 0x555a018a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018a4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 0x555a018a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x555a0181ab90 con 0x555a014d61b0 2024-07-26T11:02:43.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555a0180a090 con 0x555a014ff070 2024-07-26T11:02:43.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555a0180e0d0 con 0x555a014de390 2024-07-26T11:02:43.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05aea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 0x555a018a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05aea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 0x555a018a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58798/0 (socket says 172.21.15.144:58798) 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05aea00640 1 -- 172.21.15.144:0/158057180 learned_addr learned my addr 172.21.15.144:0/158057180 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.353+0000 7f05aea00640 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014ff070 0x555a018a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05aea00640 1 -- 172.21.15.144:0/158057180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014ff070 msgr2=0x555a018a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05aea00640 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014ff070 0x555a018a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05aea00640 1 -- 172.21.15.144:0/158057180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 msgr2=0x555a018a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05aea00640 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018a4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05aea00640 1 -- 172.21.15.144:0/158057180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0156a650 con 0x555a014de390 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05a7e00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f05940b3cb0 0x7f05940b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05aea00640 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 0x555a018a4a50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f059c011b60 tx=0x7f059c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54aeb279dfd562ba server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.354+0000 7f05a7e00640 1 -- 172.21.15.144:0/224818798 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f05a81053c0 con 0x7f05b010fc80 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.355+0000 7f05af400640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f05940b3cb0 0x7f05940b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.355+0000 7f059be00640 1 -- 172.21.15.144:0/158057180 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c00fc50 con 0x555a014de390 2024-07-26T11:02:43.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.355+0000 7f059be00640 1 -- 172.21.15.144:0/158057180 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f059c00fdf0 con 0x555a014de390 2024-07-26T11:02:43.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.355+0000 7f059be00640 1 -- 172.21.15.144:0/158057180 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c003880 con 0x555a014de390 2024-07-26T11:02:43.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 -- 172.21.15.144:0/158057180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 msgr2=0x555a018a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 0x555a018a4a50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f059c011b60 tx=0x7f059c005850 comp rx=0 tx=0).stop 2024-07-26T11:02:43.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05af400640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f05940b3cb0 0x7f05940b61c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x555a018a2c50 tx=0x7f05a0005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 -- 172.21.15.144:0/158057180 shutdown_connections 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014de390 0x555a018a4a50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018a4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/158057180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014ff070 0x555a018a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 -- 172.21.15.144:0/158057180 >> 172.21.15.144:0/158057180 conn(0x555a014f3460 msgr2=0x555a018a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 -- 172.21.15.144:0/158057180 shutdown_connections 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.356+0000 7f05bcc4a740 1 -- 172.21.15.144:0/158057180 wait complete. 2024-07-26T11:02:43.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.357+0000 7f05bcc4a740 1 -- start start 2024-07-26T11:02:43.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.357+0000 7f05bcc4a740 1 Processor -- start 2024-07-26T11:02:43.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x555a0166d700 con 0x555a014d61b0 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555a017a6660 con 0x555a014de390 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555a018b27b0 con 0x555a014ff070 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05bcc4a740 10 client.0 _ll_register_callbacks cb 0x555a017b6110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05aea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58814/0 (socket says 172.21.15.144:58814) 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05afe00640 1 -- 172.21.15.144:0/4097243589 learned_addr learned my addr 172.21.15.144:0/4097243589 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.358+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018af5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:34790/0 (socket says 172.21.15.144:34790) 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05aea00640 1 -- 172.21.15.144:0/4097243589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 msgr2=0x555a018b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05aea00640 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05aea00640 1 -- 172.21.15.144:0/4097243589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 msgr2=0x555a018af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05aea00640 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05aea00640 1 -- 172.21.15.144:0/4097243589 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a017bd800 con 0x555a014de390 2024-07-26T11:02:43.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05aea00640 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018afb20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f059c009250 tx=0x7f059c0053d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdebc8d18def594d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:43.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f0598c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:02:43.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f0598c00640 1 -- 172.21.15.144:0/4097243589 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c00fc50 con 0x555a014de390 2024-07-26T11:02:43.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f059c00fdf0 con 0x555a014de390 2024-07-26T11:02:43.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.359+0000 7f0598c00640 1 -- 172.21.15.144:0/4097243589 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f059c011270 con 0x555a014de390 2024-07-26T11:02:43.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f0598c00640 1 -- 172.21.15.144:0/4097243589 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f059c030050 con 0x555a014de390 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 10 client.25472 fetch_fsmap learned FSMap version 26 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555a018be410 con 0x555a014de390 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f0598c00640 1 -- 172.21.15.144:0/4097243589 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f059c00fdf0 con 0x555a014de390 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f0598c00640 1 -- 172.21.15.144:0/4097243589 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f059c01f5d0 con 0x555a014de390 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 10 client.25472 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 10 client.25472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 20 client.25472 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a014bd9f0 0x555a018c1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 4 client.25472 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.360+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a0181cae0 con 0x555a014bd9f0 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.361+0000 7f05af400640 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a014bd9f0 0x555a018c1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.362+0000 7f0598c00640 10 client.25472 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:43.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.362+0000 7f05af400640 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a014bd9f0 0x555a018c1770 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f407c917062ab219 server_cookie=a025f78538a54ce8 in_seq=0 out_seq=0 2024-07-26T11:02:43.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f0598c00640 1 -- 172.21.15.144:0/4097243589 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555a0181cae0 con 0x555a014bd9f0 2024-07-26T11:02:43.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f0598c00640 10 client.25472 handle_command_reply: tid=0 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 client.25472 shutdown 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 msgr2=0x555a018afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018afb20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f059c009250 tx=0x7f059c0053d0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 shutdown_connections 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a014bd9f0 0x555a018c1770 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018afb20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f0598c00640 0 client.25472 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/4097243589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 >> 172.21.15.144:0/4097243589 conn(0x555a014f3460 msgr2=0x555a018a7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 shutdown_connections 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 1 -- 172.21.15.144:0/4097243589 wait complete. 2024-07-26T11:02:43.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.363+0000 7f05bcc4a740 20 client.25472 trim_cache size 0 max 16384 2024-07-26T11:02:43.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.382+0000 7f05bcc4a740 1 Processor -- start 2024-07-26T11:02:43.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.382+0000 7f05bcc4a740 1 -- start start 2024-07-26T11:02:43.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.382+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.382+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.382+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018c7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.382+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01b9410 con 0x555a014d61b0 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01b6ce0 con 0x555a014de390 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01b4b30 con 0x555a014ff070 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58828/0 (socket says 172.21.15.144:58828) 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 -- 172.21.15.144:0/189395488 learned_addr learned my addr 172.21.15.144:0/189395488 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:43.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05aea00640 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 -- 172.21.15.144:0/189395488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 msgr2=0x555a018c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 -- 172.21.15.144:0/189395488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 msgr2=0x555a018b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.383+0000 7f05afe00640 1 -- 172.21.15.144:0/189395488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a018a62c0 con 0x555a014ff070 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.384+0000 7f05afe00640 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018c7340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05a80c9040 tx=0x7f05a80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3be4e779729ed65b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.384+0000 7f059b400640 1 -- 172.21.15.144:0/189395488 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a80e6070 con 0x555a014ff070 2024-07-26T11:02:43.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.384+0000 7f059b400640 1 -- 172.21.15.144:0/189395488 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05a8036040 con 0x555a014ff070 2024-07-26T11:02:43.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.384+0000 7f059b400640 1 -- 172.21.15.144:0/189395488 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8105170 con 0x555a014ff070 2024-07-26T11:02:43.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 -- 172.21.15.144:0/189395488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 msgr2=0x555a018c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018c7340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05a80c9040 tx=0x7f05a80095c0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 -- 172.21.15.144:0/189395488 shutdown_connections 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018c7340 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/189395488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 -- 172.21.15.144:0/189395488 >> 172.21.15.144:0/189395488 conn(0x555a014f3460 msgr2=0x555a018a59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 -- 172.21.15.144:0/189395488 shutdown_connections 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 -- 172.21.15.144:0/189395488 wait complete. 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.385+0000 7f05bcc4a740 1 -- start start 2024-07-26T11:02:43.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.386+0000 7f05bcc4a740 1 Processor -- start 2024-07-26T11:02:43.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.386+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.386+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05bcc4a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01b51a0 con 0x555a014d61b0 2024-07-26T11:02:43.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01b6610 con 0x555a014de390 2024-07-26T11:02:43.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05bcc4a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01b5870 con 0x555a014ff070 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05bcc4a740 10 client.0 _ll_register_callbacks cb 0x7f059c011a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:34804/0 (socket says 172.21.15.144:34804) 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 -- 172.21.15.144:0/2001621520 learned_addr learned my addr 172.21.15.144:0/2001621520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05afe00640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05aea00640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 -- 172.21.15.144:0/2001621520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 msgr2=0x555a018b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 -- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 msgr2=0x555a018b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:43.390 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.387+0000 7f05af400640 1 -- 172.21.15.144:0/2001621520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a018b3ee0 con 0x555a014d61b0 2024-07-26T11:02:43.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f05af400640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b7f30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009cb0 tx=0x7f05a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac839f7538ad594 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f0598c00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f0598c00640 1 -- 172.21.15.144:0/2001621520 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a002b5b0 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f0598c00640 1 -- 172.21.15.144:0/2001621520 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f05a002b750 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a8036040 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f0598c00640 1 -- 172.21.15.144:0/2001621520 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f05a002b950 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f05bcc4a740 10 client.15339 fetch_fsmap learned FSMap version 26 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.388+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555a01929ba0 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f0598c00640 1 -- 172.21.15.144:0/2001621520 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05a8036040 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f0598c00640 1 -- 172.21.15.144:0/2001621520 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f05a00323e0 con 0x555a014d61b0 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05bcc4a740 10 client.15339 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:02:43.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05bcc4a740 10 client.15339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05bcc4a740 20 client.15339 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a01492da0 0x555a0192c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05bcc4a740 4 client.15339 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555a018a1360 con 0x555a01492da0 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.389+0000 7f05aea00640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a01492da0 0x555a0192c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05aea00640 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a01492da0 0x555a0192c390 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15bf9a14d53ba108 server_cookie=ce70274c908ae5db in_seq=0 out_seq=0 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f0598c00640 10 client.15339 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f0598c00640 1 -- 172.21.15.144:0/2001621520 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555a018a1360 con 0x555a01492da0 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f0598c00640 10 client.15339 handle_command_reply: tid=0 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 client.15339 shutdown 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 msgr2=0x555a018b7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b7f30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009cb0 tx=0x7f05a0017000 comp rx=0 tx=0).stop 2024-07-26T11:02:43.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 shutdown_connections 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a01492da0 0x555a0192c390 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a014ff070 0x555a018b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a014de390 0x555a018b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 --2- 172.21.15.144:0/2001621520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a014d61b0 0x555a018b7f30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.390+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 >> 172.21.15.144:0/2001621520 conn(0x555a014f3460 msgr2=0x555a018afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.391+0000 7f0598c00640 0 client.15339 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.391+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 shutdown_connections 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.391+0000 7f05bcc4a740 1 -- 172.21.15.144:0/2001621520 wait complete. 2024-07-26T11:02:43.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.391+0000 7f05bcc4a740 20 client.15339 trim_cache size 0 max 16384 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: "8c28678f-8ab5-4362-bc32-b404a53230bc": { 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "8c28678f-8ab5-4362-bc32-b404a53230bc", 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:02:43.396 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f05940b3cb0 msgr2=0x7f05940b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f05940b3cb0 0x7f05940b61c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x555a018a2c50 tx=0x7f05a0005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010fc80 msgr2=0x7f05b01a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010fc80 0x7f05b01a5aa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05a8005e30 tx=0x7f05a80317c0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 shutdown_connections 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f05940b3cb0 0x7f05940b61c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05b010fc80 0x7f05b01a5aa0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b010f330 0x7f05b01a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 --2- 172.21.15.144:0/224818798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05b01099f0 0x7f05b01a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:43.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 >> 172.21.15.144:0/224818798 conn(0x7f05b0078ff0 msgr2=0x7f05b00795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:43.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 shutdown_connections 2024-07-26T11:02:43.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:02:43.392+0000 7f05b4a00640 1 -- 172.21.15.144:0/224818798 wait complete. 2024-07-26T11:02:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:43 smithi160 ceph-mon[36715]: pgmap v468: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:43 smithi183 ceph-mon[37273]: pgmap v468: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:43 smithi144 ceph-mon[32129]: pgmap v468: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:44.319 INFO:tasks.cephfs.filesystem:scrub status for tag:8c28678f-8ab5-4362-bc32-b404a53230bc - {'path': '/', 'tag': '8c28678f-8ab5-4362-bc32-b404a53230bc', 'options': 'recursive,force'} 2024-07-26T11:02:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:45 smithi160 ceph-mon[36715]: pgmap v469: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 52 op/s 2024-07-26T11:02:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:45 smithi183 ceph-mon[37273]: pgmap v469: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 52 op/s 2024-07-26T11:02:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:45 smithi144 ceph-mon[32129]: pgmap v469: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 52 op/s 2024-07-26T11:02:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:47 smithi160 ceph-mon[36715]: pgmap v470: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 54 op/s 2024-07-26T11:02:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:47 smithi183 ceph-mon[37273]: pgmap v470: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 54 op/s 2024-07-26T11:02:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:47 smithi144 ceph-mon[32129]: pgmap v470: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 54 op/s 2024-07-26T11:02:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:49 smithi160 ceph-mon[36715]: pgmap v471: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:49 smithi183 ceph-mon[37273]: pgmap v471: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:49 smithi144 ceph-mon[32129]: pgmap v471: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T11:02:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:50 smithi144 ceph-mon[32129]: scrub summary: idle 2024-07-26T11:02:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:50 smithi160 ceph-mon[36715]: scrub summary: idle 2024-07-26T11:02:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:50 smithi183 ceph-mon[37273]: scrub summary: idle 2024-07-26T11:02:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:51 smithi144 ceph-mon[32129]: pgmap v472: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 102 KiB/s wr, 45 op/s 2024-07-26T11:02:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:51 smithi160 ceph-mon[36715]: pgmap v472: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 102 KiB/s wr, 45 op/s 2024-07-26T11:02:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:51 smithi183 ceph-mon[37273]: pgmap v472: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 102 KiB/s wr, 45 op/s 2024-07-26T11:02:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 -- 172.21.15.144:0/2279321397 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 msgr2=0x7fd2f010b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/2279321397 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f010b460 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009860 tx=0x7fd2d802a340 comp rx=0 tx=0).stop 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 -- 172.21.15.144:0/2279321397 shutdown_connections 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/2279321397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f010b9a0 0x7fd2f01121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/2279321397 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f010b460 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/2279321397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f0105720 0x7fd2f0105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.576+0000 7fd2f6e00640 1 -- 172.21.15.144:0/2279321397 >> 172.21.15.144:0/2279321397 conn(0x7fd2f0101120 msgr2=0x7fd2f0103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.577+0000 7fd2f6e00640 1 -- 172.21.15.144:0/2279321397 shutdown_connections 2024-07-26T11:02:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.577+0000 7fd2f6e00640 1 -- 172.21.15.144:0/2279321397 wait complete. 2024-07-26T11:02:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 Processor -- start 2024-07-26T11:02:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 -- start start 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f0105720 0x7fd2f019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f010b9a0 0x7fd2f01a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f0115630 con 0x7fd2f010b9a0 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01154b0 con 0x7fd2f010b050 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f6e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01157b0 con 0x7fd2f0105720 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.578+0000 7fd2f4e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f010b9a0 0x7fd2f01a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2efe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f0105720 0x7fd2f019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2efe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f0105720 0x7fd2f019c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55922/0 (socket says 172.21.15.144:55922) 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2efe00640 1 -- 172.21.15.144:0/267931986 learned_addr learned my addr 172.21.15.144:0/267931986 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2e7400640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2f4e00640 1 -- 172.21.15.144:0/267931986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f0105720 msgr2=0x7fd2f019c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2f4e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f0105720 0x7fd2f019c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2f4e00640 1 -- 172.21.15.144:0/267931986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 msgr2=0x7fd2f019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2f4e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2f4e00640 1 -- 172.21.15.144:0/267931986 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f01a1fe0 con 0x7fd2f010b9a0 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2e7400640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.579+0000 7fd2f4e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f010b9a0 0x7fd2f01a18e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009fd0 tx=0x7fd2e800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.580+0000 7fd2ed600640 1 -- 172.21.15.144:0/267931986 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e800ce20 con 0x7fd2f010b9a0 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.580+0000 7fd2ed600640 1 -- 172.21.15.144:0/267931986 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e8005020 con 0x7fd2f010b9a0 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.580+0000 7fd2ed600640 1 -- 172.21.15.144:0/267931986 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8010690 con 0x7fd2f010b9a0 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.580+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f01a22d0 con 0x7fd2f010b9a0 2024-07-26T11:02:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.580+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f01a8a60 con 0x7fd2f010b9a0 2024-07-26T11:02:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.583+0000 7fd2fde6c740 1 Processor -- start 2024-07-26T11:02:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.583+0000 7fd2ed600640 1 -- 172.21.15.144:0/267931986 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7fd2e80054d0 con 0x7fd2f010b9a0 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 -- start start 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587efba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587efbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587efc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x559587e70270 con 0x559587b374d0 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559587e5f770 con 0x559587b581b0 2024-07-26T11:02:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559587e637b0 con 0x559587b2f2f0 2024-07-26T11:02:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2e7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587efba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2e7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587efba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:50218/0 (socket says 172.21.15.144:50218) 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2e7400640 1 -- 172.21.15.144:0/3245771824 learned_addr learned my addr 172.21.15.144:0/3245771824 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.584+0000 7fd2e7400640 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587efc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587efbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 -- 172.21.15.144:0/3245771824 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 msgr2=0x559587efba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587efba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 -- 172.21.15.144:0/3245771824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 msgr2=0x559587efc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587efc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 -- 172.21.15.144:0/3245771824 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559587bc3790 con 0x559587b2f2f0 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2ed600640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd2d40b3cb0 0x7fd2d40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.585+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587efbf80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0009cd0 tx=0x7fd2e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3400b1b5e3243940 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.586+0000 7fd2ed600640 1 -- 172.21.15.144:0/267931986 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd2e80e4990 con 0x7fd2f010b9a0 2024-07-26T11:02:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.586+0000 7fd2e7400640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd2d40b3cb0 0x7fd2d40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.586+0000 7fd2e5600640 1 -- 172.21.15.144:0/3245771824 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0004230 con 0x559587b2f2f0 2024-07-26T11:02:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.586+0000 7fd2e5600640 1 -- 172.21.15.144:0/3245771824 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e00043d0 con 0x559587b2f2f0 2024-07-26T11:02:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.586+0000 7fd2e5600640 1 -- 172.21.15.144:0/3245771824 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0019710 con 0x559587b2f2f0 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3245771824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 msgr2=0x559587efbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2e7400640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd2d40b3cb0 0x7fd2d40b61c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80095f0 tx=0x7fd2d8042040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587efbf80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0009cd0 tx=0x7fd2e0011810 comp rx=0 tx=0).stop 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3245771824 shutdown_connections 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587efc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587efbf80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3245771824 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587efba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3245771824 >> 172.21.15.144:0/3245771824 conn(0x559587b4c300 msgr2=0x559587ef9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3245771824 shutdown_connections 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.587+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3245771824 wait complete. 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.588+0000 7fd2fde6c740 1 -- start start 2024-07-26T11:02:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.588+0000 7fd2fde6c740 1 Processor -- start 2024-07-26T11:02:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f07600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b374d0 0x559587f07b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f134e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x559587d99ab0 con 0x559587b581b0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559587dfcd50 con 0x559587b374d0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559587f0a1c0 con 0x559587b2f2f0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2fde6c740 10 client.0 _ll_register_callbacks cb 0x559587b6fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f07600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f07600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55940/0 (socket says 172.21.15.144:55940) 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 -- 172.21.15.144:0/620681757 learned_addr learned my addr 172.21.15.144:0/620681757 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2f4e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2e7400640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b374d0 0x559587f07b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 -- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b374d0 msgr2=0x559587f07b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b374d0 0x559587f07b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 -- 172.21.15.144:0/620681757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 msgr2=0x559587f134e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f134e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2efe00640 1 -- 172.21.15.144:0/620681757 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559587c3c620 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.589+0000 7fd2e7400640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b374d0 0x559587f07b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2efe00640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f07600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0003e40 tx=0x7fd2e0005e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b18d9cbd4a270a98 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2de000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2de000640 1 -- 172.21.15.144:0/620681757 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0027070 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2de000640 1 -- 172.21.15.144:0/620681757 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd2e0003800 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e00043d0 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2de000640 1 -- 172.21.15.144:0/620681757 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd2e0033050 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2fde6c740 10 client.25486 fetch_fsmap learned FSMap version 26 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.590+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559587f16200 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2de000640 1 -- 172.21.15.144:0/620681757 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e00043d0 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2de000640 1 -- 172.21.15.144:0/620681757 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd2e0018070 con 0x559587b2f2f0 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2fde6c740 10 client.25486 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:02:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2fde6c740 10 client.25486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:02:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2fde6c740 20 client.25486 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587b16d80 0x559587f195c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2fde6c740 4 client.25486 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:02:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.591+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559587e721c0 con 0x559587b16d80 2024-07-26T11:02:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.592+0000 7fd2e7400640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587b16d80 0x559587f195c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.592+0000 7fd2e7400640 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587b16d80 0x559587f195c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deabf53a5c35474e server_cookie=5ebaeead8a2d7011 in_seq=0 out_seq=0 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.592+0000 7fd2de000640 10 client.25486 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.593+0000 7fd2de000640 1 -- 172.21.15.144:0/620681757 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559587e721c0 con 0x559587b16d80 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.593+0000 7fd2de000640 10 client.25486 handle_command_reply: tid=0 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.593+0000 7fd2fde6c740 1 client.25486 shutdown 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.593+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 msgr2=0x559587f07600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f07600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0003e40 tx=0x7fd2e0005e70 comp rx=0 tx=0).stop 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 shutdown_connections 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587b16d80 0x559587f195c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b374d0 0x559587f07b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2de000640 0 client.25486 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/620681757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f07600 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 >> 172.21.15.144:0/620681757 conn(0x559587b4c300 msgr2=0x559587ef80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 shutdown_connections 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 1 -- 172.21.15.144:0/620681757 wait complete. 2024-07-26T11:02:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.594+0000 7fd2fde6c740 20 client.25486 trim_cache size 0 max 16384 2024-07-26T11:02:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.610+0000 7fd2fde6c740 1 Processor -- start 2024-07-26T11:02:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.610+0000 7fd2fde6c740 1 -- start start 2024-07-26T11:02:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 0x559587ef8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b374d0 0x559587ef9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01b5320 con 0x559587b581b0 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01b2bf0 con 0x559587b2f2f0 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01b0a40 con 0x559587b374d0 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2f4e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2efe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 0x559587ef8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2efe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 0x559587ef8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:50232/0 (socket says 172.21.15.144:50232) 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2efe00640 1 -- 172.21.15.144:0/3006669981 learned_addr learned my addr 172.21.15.144:0/3006669981 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.611+0000 7fd2e7400640 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b374d0 0x559587ef9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2efe00640 1 -- 172.21.15.144:0/3006669981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b374d0 msgr2=0x559587ef9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b374d0 0x559587ef9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2efe00640 1 -- 172.21.15.144:0/3006669981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 msgr2=0x559587f078d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f078d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2efe00640 1 -- 172.21.15.144:0/3006669981 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559587ef5350 con 0x559587b2f2f0 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 0x559587ef8720 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0019f50 tx=0x7fd2e0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b5bff39fc36ab4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2e4c00640 1 -- 172.21.15.144:0/3006669981 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0027070 con 0x559587b2f2f0 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2e4c00640 1 -- 172.21.15.144:0/3006669981 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e0011d10 con 0x559587b2f2f0 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.612+0000 7fd2e4c00640 1 -- 172.21.15.144:0/3006669981 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e001a8b0 con 0x559587b2f2f0 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3006669981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 msgr2=0x559587ef8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 0x559587ef8720 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0019f50 tx=0x7fd2e0009260 comp rx=0 tx=0).stop 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3006669981 shutdown_connections 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b581b0 0x559587f078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b374d0 0x559587ef9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3006669981 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b2f2f0 0x559587ef8720 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3006669981 >> 172.21.15.144:0/3006669981 conn(0x559587b4c300 msgr2=0x559587ef9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3006669981 shutdown_connections 2024-07-26T11:02:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.613+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3006669981 wait complete. 2024-07-26T11:02:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.614+0000 7fd2fde6c740 1 -- start start 2024-07-26T11:02:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.614+0000 7fd2fde6c740 1 Processor -- start 2024-07-26T11:02:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f0a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587f0a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587f0ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01b10b0 con 0x559587b374d0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01b2520 con 0x559587b581b0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01b1780 con 0x559587b2f2f0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2fde6c740 10 client.0 _ll_register_callbacks cb 0x7fd2e80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587f0a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587f0a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46858/0 (socket says 172.21.15.144:46858) 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 -- 172.21.15.144:0/3040795803 learned_addr learned my addr 172.21.15.144:0/3040795803 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f0a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2f4e00640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587f0ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 -- 172.21.15.144:0/3040795803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 msgr2=0x559587f0a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f0a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 -- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 msgr2=0x559587f0ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587f0ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.615+0000 7fd2e7400640 1 -- 172.21.15.144:0/3040795803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559587f88440 con 0x559587b374d0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2e7400640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587f0a940 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005940 tx=0x7fd2d802a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1090ed90f376318f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2de000640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2de000640 1 -- 172.21.15.144:0/3040795803 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8037040 con 0x559587b374d0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e0011d10 con 0x559587b374d0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2de000640 1 -- 172.21.15.144:0/3040795803 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd2d80329f0 con 0x559587b374d0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2de000640 1 -- 172.21.15.144:0/3040795803 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd2d80308e0 con 0x559587b374d0 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2fde6c740 10 client.15357 fetch_fsmap learned FSMap version 26 2024-07-26T11:02:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559587f89670 con 0x559587b374d0 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.616+0000 7fd2de000640 1 -- 172.21.15.144:0/3040795803 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e0011d10 con 0x559587b374d0 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2de000640 1 -- 172.21.15.144:0/3040795803 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd2d8049070 con 0x559587b374d0 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2fde6c740 10 client.15357 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2fde6c740 10 client.15357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2fde6c740 20 client.15357 populate_metadata read hostname 'smithi144' 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587aec1c0 0x559587f8be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2fde6c740 4 client.15357 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559587e6dc60 con 0x559587aec1c0 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.617+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587aec1c0 0x559587f8be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.618+0000 7fd2efe00640 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587aec1c0 0x559587f8be30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0c736738a761dc server_cookie=8c36b55d06a15f40 in_seq=0 out_seq=0 2024-07-26T11:02:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.618+0000 7fd2de000640 10 client.15357 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2de000640 1 -- 172.21.15.144:0/3040795803 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559587e6dc60 con 0x559587aec1c0 2024-07-26T11:02:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2de000640 10 client.15357 handle_command_reply: tid=0 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 client.15357 shutdown 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 msgr2=0x559587f0a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587f0a940 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005940 tx=0x7fd2d802a340 comp rx=0 tx=0).stop 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 shutdown_connections 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x559587aec1c0 0x559587f8be30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559587b581b0 0x559587f0ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x559587b374d0 0x559587f0a940 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 --2- 172.21.15.144:0/3040795803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559587b2f2f0 0x559587f0a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2de000640 0 client.15357 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.632+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 >> 172.21.15.144:0/3040795803 conn(0x559587b4c300 msgr2=0x559587f088a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 shutdown_connections 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2fde6c740 1 -- 172.21.15.144:0/3040795803 wait complete. 2024-07-26T11:02:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2fde6c740 20 client.15357 trim_cache size 0 max 16384 2024-07-26T11:02:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:02:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:02:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:02:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd2d40b3cb0 msgr2=0x7fd2d40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd2d40b3cb0 0x7fd2d40b61c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80095f0 tx=0x7fd2d8042040 comp rx=0 tx=0).stop 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f010b9a0 msgr2=0x7fd2f01a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.633+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f010b9a0 0x7fd2f01a18e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009fd0 tx=0x7fd2e800ef10 comp rx=0 tx=0).stop 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 shutdown_connections 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd2d40b3cb0 0x7fd2d40b61c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f010b9a0 0x7fd2f01a18e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd2f010b050 0x7fd2f019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 --2- 172.21.15.144:0/267931986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f0105720 0x7fd2f019c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 >> 172.21.15.144:0/267931986 conn(0x7fd2f0101120 msgr2=0x7fd2f010a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 shutdown_connections 2024-07-26T11:02:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:02:52.634+0000 7fd2f6e00640 1 -- 172.21.15.144:0/267931986 wait complete. 2024-07-26T11:02:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:53 smithi144 ceph-mon[32129]: pgmap v473: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:02:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:53 smithi160 ceph-mon[36715]: pgmap v473: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:02:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:53 smithi183 ceph-mon[37273]: pgmap v473: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:02:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:55 smithi160 ceph-mon[36715]: pgmap v474: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 7 op/s 2024-07-26T11:02:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:55 smithi183 ceph-mon[37273]: pgmap v474: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 7 op/s 2024-07-26T11:02:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:55 smithi144 ceph-mon[32129]: pgmap v474: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 7 op/s 2024-07-26T11:02:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:57 smithi160 ceph-mon[36715]: pgmap v475: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T11:02:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:57 smithi183 ceph-mon[37273]: pgmap v475: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T11:02:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:57 smithi144 ceph-mon[32129]: pgmap v475: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T11:02:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:02:59 smithi160 ceph-mon[36715]: pgmap v476: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T11:02:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:02:59 smithi183 ceph-mon[37273]: pgmap v476: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T11:02:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:02:59 smithi144 ceph-mon[32129]: pgmap v476: 129 pgs: 129 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T11:03:00.470 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 11:03:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T11:03:00.070+0000 7f4a4ba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:03:00.470 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 11:03:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T11:03:00.070+0000 7f4a4ba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:03:00.470 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 11:03:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T11:03:00.070+0000 7f4a4ba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:03:00.471 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 11:03:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T11:03:00.070+0000 7f4a4ba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:03:00.471 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Jul 26 11:03:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mgr-x[32366]: 2024-07-26T11:03:00.070+0000 7f4a4ba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:03:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:01 smithi160 ceph-mon[36715]: pgmap v477: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-07-26T11:03:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:01 smithi183 ceph-mon[37273]: pgmap v477: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-07-26T11:03:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:01 smithi144 ceph-mon[32129]: pgmap v477: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-07-26T11:03:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:03 smithi160 ceph-mon[36715]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T11:03:03.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:03 smithi160 ceph-mon[36715]: pgmap v478: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 23 op/s 2024-07-26T11:03:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:03 smithi183 ceph-mon[37273]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T11:03:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:03 smithi183 ceph-mon[37273]: pgmap v478: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 23 op/s 2024-07-26T11:03:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:03 smithi144 ceph-mon[32129]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T11:03:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:03 smithi144 ceph-mon[32129]: pgmap v478: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 23 op/s 2024-07-26T11:03:05.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:05 smithi160 ceph-mon[36715]: pgmap v479: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-07-26T11:03:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:05 smithi183 ceph-mon[37273]: pgmap v479: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-07-26T11:03:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:05 smithi144 ceph-mon[32129]: pgmap v479: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-07-26T11:03:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 -- 172.21.15.144:0/2461902669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c102220 msgr2=0x7f3e1c102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 --2- 172.21.15.144:0/2461902669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c102220 0x7f3e1c102630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009860 tx=0x7f3e0c02a340 comp rx=0 tx=0).stop 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 -- 172.21.15.144:0/2461902669 shutdown_connections 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 --2- 172.21.15.144:0/2461902669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e1c1084a0 0x7f3e1c10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 --2- 172.21.15.144:0/2461902669 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 --2- 172.21.15.144:0/2461902669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c102220 0x7f3e1c102630 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 -- 172.21.15.144:0/2461902669 >> 172.21.15.144:0/2461902669 conn(0x7f3e1c0fdc40 msgr2=0x7f3e1c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 -- 172.21.15.144:0/2461902669 shutdown_connections 2024-07-26T11:03:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.853+0000 7f3e1be00640 1 -- 172.21.15.144:0/2461902669 wait complete. 2024-07-26T11:03:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.854+0000 7f3e1be00640 1 Processor -- start 2024-07-26T11:03:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.854+0000 7f3e1be00640 1 -- start start 2024-07-26T11:03:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e1c102220 0x7f3e1c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c1084a0 0x7f3e1c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1be00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c112170 con 0x7f3e1c107b50 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1be00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c111ff0 con 0x7f3e1c1084a0 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1122f0 con 0x7f3e1c102220 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1a000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1a000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45774/0 (socket says 172.21.15.144:45774) 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1a000640 1 -- 172.21.15.144:0/4037991775 learned_addr learned my addr 172.21.15.144:0/4037991775 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1b400640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c1084a0 0x7f3e1c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.855+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e1c102220 0x7f3e1c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1a000640 1 -- 172.21.15.144:0/4037991775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e1c102220 msgr2=0x7f3e1c19beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1a000640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e1c102220 0x7f3e1c19beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1a000640 1 -- 172.21.15.144:0/4037991775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c1084a0 msgr2=0x7f3e1c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1a000640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c1084a0 0x7f3e1c1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1a000640 1 -- 172.21.15.144:0/4037991775 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c1a1de0 con 0x7f3e1c107b50 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1b400640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c1084a0 0x7f3e1c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1a000640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c19c450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e080058c0 tx=0x7f3e0800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e20a00640 1 -- 172.21.15.144:0/4037991775 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0800cd30 con 0x7f3e1c107b50 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e20a00640 1 -- 172.21.15.144:0/4037991775 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e080035f0 con 0x7f3e1c107b50 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c1a20d0 con 0x7f3e1c107b50 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e20a00640 1 -- 172.21.15.144:0/4037991775 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080044e0 con 0x7f3e1c107b50 2024-07-26T11:03:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.856+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c1a8860 con 0x7f3e1c107b50 2024-07-26T11:03:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.861+0000 7f3e20a00640 1 -- 172.21.15.144:0/4037991775 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3e08003aa0 con 0x7f3e1c107b50 2024-07-26T11:03:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.861+0000 7f3e28fe4740 1 Processor -- start 2024-07-26T11:03:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.861+0000 7f3e28fe4740 1 -- start start 2024-07-26T11:03:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3825c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3826160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa38266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55cfa379a270 con 0x55cfa34614d0 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55cfa3789770 con 0x55cfa34592f0 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cfa378d7b0 con 0x55cfa34821b0 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e1b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3825c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e1b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3825c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:47602/0 (socket says 172.21.15.144:47602) 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e1b400640 1 -- 172.21.15.144:0/1935464408 learned_addr learned my addr 172.21.15.144:0/1935464408 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3826160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e1a000640 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa38266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.862+0000 7f3e20a00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3e000b3dc0 0x7f3e000b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 -- 172.21.15.144:0/1935464408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 msgr2=0x55cfa3826160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3826160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 -- 172.21.15.144:0/1935464408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 msgr2=0x55cfa38266a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa38266a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 -- 172.21.15.144:0/1935464408 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa34ed790 con 0x55cfa34821b0 2024-07-26T11:03:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e20a00640 1 -- 172.21.15.144:0/4037991775 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f3e080e45a0 con 0x7f3e1c107b50 2024-07-26T11:03:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3e000b3dc0 0x7f3e000b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.863+0000 7f3e1b400640 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3825c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3e14011810 tx=0x7f3e14011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ec88ca8c5f4ee25 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.864+0000 7f3e07e00640 1 -- 172.21.15.144:0/1935464408 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14010570 con 0x55cfa34821b0 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.864+0000 7f3e07e00640 1 -- 172.21.15.144:0/1935464408 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e14010bc0 con 0x55cfa34821b0 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.864+0000 7f3e07e00640 1 -- 172.21.15.144:0/1935464408 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1401ca80 con 0x55cfa34821b0 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1935464408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 msgr2=0x55cfa3825c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3825c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3e14011810 tx=0x7f3e14011cd0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e1b400640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3e000b3dc0 0x7f3e000b62d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3e14011490 tx=0x7f3e14021000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1935464408 shutdown_connections 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa38266a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3826160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1935464408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3825c20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1935464408 >> 172.21.15.144:0/1935464408 conn(0x55cfa3476300 msgr2=0x55cfa3823b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1935464408 shutdown_connections 2024-07-26T11:03:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.865+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1935464408 wait complete. 2024-07-26T11:03:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 -- start start 2024-07-26T11:03:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 Processor -- start 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 0x55cfa38317a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3831ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34821b0 0x55cfa383d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55cfa36c3ab0 con 0x55cfa34614d0 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55cfa3726d50 con 0x55cfa34821b0 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cfa38343a0 con 0x55cfa34592f0 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.866+0000 7f3e28fe4740 10 client.0 _ll_register_callbacks cb 0x55cfa3499a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1a000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3831ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 0x55cfa38317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 0x55cfa38317a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:47616/0 (socket says 172.21.15.144:47616) 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1a000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3831ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45792/0 (socket says 172.21.15.144:45792) 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 -- 172.21.15.144:0/150812014 learned_addr learned my addr 172.21.15.144:0/150812014 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1b400640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34821b0 0x55cfa383d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 -- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34821b0 msgr2=0x55cfa383d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34821b0 0x55cfa383d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 -- 172.21.15.144:0/150812014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 msgr2=0x55cfa3831ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3831ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1aa00640 1 -- 172.21.15.144:0/150812014 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa3566620 con 0x55cfa34592f0 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.867+0000 7f3e1b400640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34821b0 0x55cfa383d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 0x55cfa38317a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02a860 tx=0x7f3e0c030d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7325c3839c36188 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e04c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e04c00640 1 -- 172.21.15.144:0/150812014 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c042070 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e14010bc0 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e04c00640 1 -- 172.21.15.144:0/150812014 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f3e0c004240 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e04c00640 1 -- 172.21.15.144:0/150812014 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e0c04e050 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e28fe4740 10 client.25507 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cfa3840450 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.868+0000 7f3e04c00640 1 -- 172.21.15.144:0/150812014 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e14010bc0 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e04c00640 1 -- 172.21.15.144:0/150812014 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3e0c003b10 con 0x55cfa34592f0 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e28fe4740 10 client.25507 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e28fe4740 10 client.25507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e28fe4740 20 client.25507 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa3440d80 0x55cfa38439f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e28fe4740 4 client.25507 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfa379c1c0 con 0x55cfa3440d80 2024-07-26T11:03:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.869+0000 7f3e1a000640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa3440d80 0x55cfa38439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.870+0000 7f3e1a000640 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa3440d80 0x55cfa38439f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5616c0ed874eac0e server_cookie=d07404ba9ff833b8 in_seq=0 out_seq=0 2024-07-26T11:03:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.870+0000 7f3e04c00640 10 client.25507 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e04c00640 1 -- 172.21.15.144:0/150812014 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cfa379c1c0 con 0x55cfa3440d80 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e04c00640 10 client.25507 handle_command_reply: tid=0 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 client.25507 shutdown 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 msgr2=0x55cfa38317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 0x55cfa38317a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02a860 tx=0x7f3e0c030d80 comp rx=0 tx=0).stop 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 shutdown_connections 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa3440d80 0x55cfa38439f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34821b0 0x55cfa383d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e04c00640 0 client.25507 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3831ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/150812014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34592f0 0x55cfa38317a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 >> 172.21.15.144:0/150812014 conn(0x55cfa3476300 msgr2=0x55cfa3826c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 shutdown_connections 2024-07-26T11:03:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 1 -- 172.21.15.144:0/150812014 wait complete. 2024-07-26T11:03:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.871+0000 7f3e28fe4740 20 client.25507 trim_cache size 0 max 16384 2024-07-26T11:03:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.888+0000 7f3e28fe4740 1 Processor -- start 2024-07-26T11:03:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 -- start start 2024-07-26T11:03:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa382be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3833580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3831a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1b5070 con 0x55cfa34614d0 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1b2940 con 0x55cfa34592f0 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1b0790 con 0x55cfa34821b0 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e1aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa382be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e1a000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3833580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e1aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa382be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:34684/0 (socket says 172.21.15.144:34684) 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.889+0000 7f3e1aa00640 1 -- 172.21.15.144:0/1350560976 learned_addr learned my addr 172.21.15.144:0/1350560976 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1b400640 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3831a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1aa00640 1 -- 172.21.15.144:0/1350560976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 msgr2=0x55cfa3831a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3831a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1aa00640 1 -- 172.21.15.144:0/1350560976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 msgr2=0x55cfa3833580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3833580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1aa00640 1 -- 172.21.15.144:0/1350560976 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa381dd50 con 0x55cfa34592f0 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa382be20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02a930 tx=0x7f3e0c02a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=487bdce1d5d68ea1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e07400640 1 -- 172.21.15.144:0/1350560976 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c042040 con 0x55cfa34592f0 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e07400640 1 -- 172.21.15.144:0/1350560976 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e0c005d30 con 0x55cfa34592f0 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.890+0000 7f3e07400640 1 -- 172.21.15.144:0/1350560976 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c037040 con 0x55cfa34592f0 2024-07-26T11:03:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.891+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1350560976 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 msgr2=0x55cfa382be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.891+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa382be20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02a930 tx=0x7f3e0c02a340 comp rx=0 tx=0).stop 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1350560976 shutdown_connections 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa3831a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3833580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/1350560976 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa382be20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1350560976 >> 172.21.15.144:0/1350560976 conn(0x55cfa3476300 msgr2=0x55cfa382b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1350560976 shutdown_connections 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 -- 172.21.15.144:0/1350560976 wait complete. 2024-07-26T11:03:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 -- start start 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.892+0000 7f3e28fe4740 1 Processor -- start 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3839b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3834fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa38354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1b0e00 con 0x55cfa34614d0 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1b2270 con 0x55cfa34592f0 2024-07-26T11:03:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1b14d0 con 0x55cfa34821b0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e28fe4740 10 client.0 _ll_register_callbacks cb 0x7f3e140093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e1aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3839b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e1aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3839b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:34692/0 (socket says 172.21.15.144:34692) 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e1aa00640 1 -- 172.21.15.144:0/4087322592 learned_addr learned my addr 172.21.15.144:0/4087322592 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.893+0000 7f3e1a000640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3834fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1b400640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa38354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1aa00640 1 -- 172.21.15.144:0/4087322592 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 msgr2=0x55cfa38354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa38354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1aa00640 1 -- 172.21.15.144:0/4087322592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 msgr2=0x55cfa3834fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3834fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1aa00640 1 -- 172.21.15.144:0/4087322592 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa3835a60 con 0x55cfa34592f0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1b400640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa38354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e1aa00640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3839b10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c005c40 tx=0x7f3e0c03e630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b5f94e212d1b12d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e04c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e04c00640 1 -- 172.21.15.144:0/4087322592 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c042070 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e0c005d30 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.894+0000 7f3e04c00640 1 -- 172.21.15.144:0/4087322592 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f3e0c0041b0 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e04c00640 1 -- 172.21.15.144:0/4087322592 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e0c060050 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 10 client.25520 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cfa38abeb0 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e04c00640 1 -- 172.21.15.144:0/4087322592 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e0c005d30 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e04c00640 1 -- 172.21.15.144:0/4087322592 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3e0c003660 con 0x55cfa34592f0 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 10 client.25520 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 10 client.25520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 20 client.25520 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa34161c0 0x55cfa38ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 4 client.25520 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:03:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.895+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cfa3797c60 con 0x55cfa34161c0 2024-07-26T11:03:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.896+0000 7f3e1a000640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa34161c0 0x55cfa38ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.896+0000 7f3e1a000640 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa34161c0 0x55cfa38ae540 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378d4ac0dc191b12 server_cookie=77ef533ca2c809e7 in_seq=0 out_seq=0 2024-07-26T11:03:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.896+0000 7f3e04c00640 10 client.25520 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e04c00640 1 -- 172.21.15.144:0/4087322592 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cfa3797c60 con 0x55cfa34161c0 2024-07-26T11:03:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e04c00640 10 client.25520 handle_command_reply: tid=0 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 client.25520 shutdown 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 msgr2=0x55cfa3839b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3839b10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c005c40 tx=0x7f3e0c03e630 comp rx=0 tx=0).stop 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 shutdown_connections 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cfa34161c0 0x55cfa38ae540 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfa34821b0 0x55cfa38354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cfa34614d0 0x55cfa3834fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 --2- 172.21.15.144:0/4087322592 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cfa34592f0 0x55cfa3839b10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e04c00640 0 client.25520 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 >> 172.21.15.144:0/4087322592 conn(0x55cfa3476300 msgr2=0x55cfa3830a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 shutdown_connections 2024-07-26T11:03:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.904+0000 7f3e28fe4740 1 -- 172.21.15.144:0/4087322592 wait complete. 2024-07-26T11:03:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.905+0000 7f3e28fe4740 20 client.25520 trim_cache size 0 max 16384 2024-07-26T11:03:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:03:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:03:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:03:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.905+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3e000b3dc0 msgr2=0x7f3e000b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.905+0000 7f3e1be00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3e000b3dc0 0x7f3e000b62d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3e14011490 tx=0x7f3e14021000 comp rx=0 tx=0).stop 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.905+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 msgr2=0x7f3e1c19c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.905+0000 7f3e1be00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c19c450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e080058c0 tx=0x7f3e0800ee50 comp rx=0 tx=0).stop 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 shutdown_connections 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3e000b3dc0 0x7f3e000b62d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3e1c1084a0 0x7f3e1c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e1c107b50 0x7f3e1c19c450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 --2- 172.21.15.144:0/4037991775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e1c102220 0x7f3e1c19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 >> 172.21.15.144:0/4037991775 conn(0x7f3e1c0fdc40 msgr2=0x7f3e1c1071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 shutdown_connections 2024-07-26T11:03:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:05.906+0000 7f3e1be00640 1 -- 172.21.15.144:0/4037991775 wait complete. 2024-07-26T11:03:07.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:07 smithi144 ceph-mon[32129]: pgmap v480: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-07-26T11:03:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:07 smithi160 ceph-mon[36715]: pgmap v480: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-07-26T11:03:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:07 smithi183 ceph-mon[37273]: pgmap v480: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-07-26T11:03:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:09 smithi160 ceph-mon[36715]: pgmap v481: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-07-26T11:03:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:09 smithi183 ceph-mon[37273]: pgmap v481: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-07-26T11:03:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:09 smithi144 ceph-mon[32129]: pgmap v481: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-07-26T11:03:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:11 smithi160 ceph-mon[36715]: pgmap v482: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-07-26T11:03:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:11 smithi183 ceph-mon[37273]: pgmap v482: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-07-26T11:03:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:11 smithi144 ceph-mon[32129]: pgmap v482: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-07-26T11:03:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:13 smithi160 ceph-mon[36715]: pgmap v483: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-07-26T11:03:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:13 smithi183 ceph-mon[37273]: pgmap v483: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-07-26T11:03:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:13 smithi144 ceph-mon[32129]: pgmap v483: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-07-26T11:03:14.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:03:14.875 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:15.359 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:15 smithi144 ceph-mon[32129]: pgmap v484: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-07-26T11:03:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:15 smithi160 ceph-mon[36715]: pgmap v484: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-07-26T11:03:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:15 smithi183 ceph-mon[37273]: pgmap v484: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 -- 172.21.15.144:0/1056837075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 msgr2=0x7f8f04109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 --2- 172.21.15.144:0/1056837075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 0x7f8f04109130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009900 tx=0x7f8ef002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 -- 172.21.15.144:0/1056837075 shutdown_connections 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 --2- 172.21.15.144:0/1056837075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 --2- 172.21.15.144:0/1056837075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 0x7f8f04109130 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 --2- 172.21.15.144:0/1056837075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f041033e0 0x7f8f041037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.303+0000 7f8f09200640 1 -- 172.21.15.144:0/1056837075 >> 172.21.15.144:0/1056837075 conn(0x7f8f040fee30 msgr2=0x7f8f04101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:17.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.304+0000 7f8f09200640 1 -- 172.21.15.144:0/1056837075 shutdown_connections 2024-07-26T11:03:17.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.304+0000 7f8f09200640 1 -- 172.21.15.144:0/1056837075 wait complete. 2024-07-26T11:03:17.307 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:17 smithi144 ceph-mon[32129]: pgmap v485: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-07-26T11:03:17.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.305+0000 7f8f09200640 1 Processor -- start 2024-07-26T11:03:17.307 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 -- start start 2024-07-26T11:03:17.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f041033e0 0x7f8f04074870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 0x7f8f0406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0406e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04113350 con 0x7f8f041033e0 2024-07-26T11:03:17.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041131d0 con 0x7f8f04109670 2024-07-26T11:03:17.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.306+0000 7f8f09200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041134d0 con 0x7f8f04108d20 2024-07-26T11:03:17.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0406e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f02a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 0x7f8f0406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f041033e0 0x7f8f04074870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0406e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:57414/0 (socket says 172.21.15.144:57414) 2024-07-26T11:03:17.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 -- 172.21.15.144:0/111284440 learned_addr learned my addr 172.21.15.144:0/111284440 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:17.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 -- 172.21.15.144:0/111284440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 msgr2=0x7f8f0406db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 0x7f8f0406db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 -- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f041033e0 msgr2=0x7f8f04074870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f041033e0 0x7f8f04074870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.307+0000 7f8f03e00640 1 -- 172.21.15.144:0/111284440 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0406e8c0 con 0x7f8f04109670 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.308+0000 7f8f03e00640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0406e0a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00eb10 tx=0x7f8efc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.308+0000 7f8efbe00640 1 -- 172.21.15.144:0/111284440 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc004b20 con 0x7f8f04109670 2024-07-26T11:03:17.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.308+0000 7f8efbe00640 1 -- 172.21.15.144:0/111284440 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8efc004cc0 con 0x7f8f04109670 2024-07-26T11:03:17.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.308+0000 7f8efbe00640 1 -- 172.21.15.144:0/111284440 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc005580 con 0x7f8f04109670 2024-07-26T11:03:17.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.308+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f04073540 con 0x7f8f04109670 2024-07-26T11:03:17.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.308+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f04073a00 con 0x7f8f04109670 2024-07-26T11:03:17.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.311+0000 7f8f114cc740 1 Processor -- start 2024-07-26T11:03:17.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.314+0000 7f8efbe00640 1 -- 172.21.15.144:0/111284440 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8efc005720 con 0x7f8f04109670 2024-07-26T11:03:17.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.314+0000 7f8f114cc740 1 -- start start 2024-07-26T11:03:17.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec65070 0x55b54f00a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f00a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec44390 0x55b54f00aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f114cc740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b54ef80bc0 con 0x55b54ec44390 2024-07-26T11:03:17.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f114cc740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b54ef700c0 con 0x55b54ec65070 2024-07-26T11:03:17.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f114cc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b54ef74100 con 0x55b54ec3c1b0 2024-07-26T11:03:17.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f02a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec65070 0x55b54f00a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f02a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec65070 0x55b54f00a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:57418/0 (socket says 172.21.15.144:57418) 2024-07-26T11:03:17.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.315+0000 7f8f02a00640 1 -- 172.21.15.144:0/166592531 learned_addr learned my addr 172.21.15.144:0/166592531 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:17.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f03400640 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f00a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f02a00640 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec44390 0x55b54f00aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f03400640 1 -- 172.21.15.144:0/166592531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec65070 msgr2=0x55b54f00a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f03400640 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec65070 0x55b54f00a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f03400640 1 -- 172.21.15.144:0/166592531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec44390 msgr2=0x55b54f00aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f03400640 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec44390 0x55b54f00aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f03400640 1 -- 172.21.15.144:0/166592531 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54ecd0650 con 0x55b54ec3c1b0 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8efbe00640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8ee80b3dc0 0x7f8ee80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.316+0000 7f8f02a00640 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec44390 0x55b54f00aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:17.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.317+0000 7f8efbe00640 1 -- 172.21.15.144:0/111284440 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8efc0e87d0 con 0x7f8f04109670 2024-07-26T11:03:17.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.317+0000 7f8f03400640 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f00a540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009c20 tx=0x7f8ef4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=209e247a5a642e9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:17.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.317+0000 7f8ef8c00640 1 -- 172.21.15.144:0/166592531 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4009dd0 con 0x55b54ec3c1b0 2024-07-26T11:03:17.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.318+0000 7f8ef8c00640 1 -- 172.21.15.144:0/166592531 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef4005020 con 0x55b54ec3c1b0 2024-07-26T11:03:17.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.318+0000 7f8ef8c00640 1 -- 172.21.15.144:0/166592531 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4004b40 con 0x55b54ec3c1b0 2024-07-26T11:03:17.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.318+0000 7f8f02a00640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8ee80b3dc0 0x7f8ee80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 -- 172.21.15.144:0/166592531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 msgr2=0x55b54f00a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f00a540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009c20 tx=0x7f8ef4011810 comp rx=0 tx=0).stop 2024-07-26T11:03:17.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 -- 172.21.15.144:0/166592531 shutdown_connections 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec44390 0x55b54f00aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f00a540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 --2- 172.21.15.144:0/166592531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec65070 0x55b54f00a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 -- 172.21.15.144:0/166592531 >> 172.21.15.144:0/166592531 conn(0x55b54ec59460 msgr2=0x55b54f007ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.319+0000 7f8f114cc740 1 -- 172.21.15.144:0/166592531 shutdown_connections 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.320+0000 7f8f02a00640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8ee80b3dc0 0x7f8ee80b62d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0006010 tx=0x7f8ef0031400 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.320+0000 7f8f114cc740 1 -- 172.21.15.144:0/166592531 wait complete. 2024-07-26T11:03:17.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.320+0000 7f8f114cc740 1 -- start start 2024-07-26T11:03:17.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 Processor -- start 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec3c1b0 0x55b54f015580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec44390 0x55b54f015ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f016000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b54edd3700 con 0x55b54ec65070 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b54ef0c690 con 0x55b54ec3c1b0 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b54f0187e0 con 0x55b54ec44390 2024-07-26T11:03:17.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.321+0000 7f8f114cc740 10 client.0 _ll_register_callbacks cb 0x55b54ef1c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:17.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.322+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f016000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.322+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f016000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54314/0 (socket says 172.21.15.144:54314) 2024-07-26T11:03:17.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.322+0000 7f8f03e00640 1 -- 172.21.15.144:0/2708176947 learned_addr learned my addr 172.21.15.144:0/2708176947 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:17.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.322+0000 7f8f02a00640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec44390 0x55b54f015ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.322+0000 7f8f03400640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec3c1b0 0x55b54f015580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.323+0000 7f8f03e00640 1 -- 172.21.15.144:0/2708176947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec44390 msgr2=0x55b54f015ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.323+0000 7f8f03e00640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec44390 0x55b54f015ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.323+0000 7f8f03e00640 1 -- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec3c1b0 msgr2=0x55b54f015580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.323+0000 7f8f03e00640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec3c1b0 0x55b54f015580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.323+0000 7f8f03e00640 1 -- 172.21.15.144:0/2708176947 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54ef23830 con 0x55b54ec65070 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.324+0000 7f8f03e00640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f016000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0ab1a0 tx=0x7f8efc0ae950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5704aee60cdba057 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:17.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.324+0000 7f8eed600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:17.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.324+0000 7f8eed600640 1 -- 172.21.15.144:0/2708176947 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0af820 con 0x55b54ec65070 2024-07-26T11:03:17.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.324+0000 7f8eed600640 1 -- 172.21.15.144:0/2708176947 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8efc0b7d40 con 0x55b54ec65070 2024-07-26T11:03:17.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.324+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ef4005020 con 0x55b54ec65070 2024-07-26T11:03:17.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.325+0000 7f8eed600640 1 -- 172.21.15.144:0/2708176947 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8efc0e8590 con 0x55b54ec65070 2024-07-26T11:03:17.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.325+0000 7f8f114cc740 10 client.15393 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.325+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b54f024440 con 0x55b54ec65070 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.325+0000 7f8eed600640 1 -- 172.21.15.144:0/2708176947 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef4005020 con 0x55b54ec65070 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8eed600640 1 -- 172.21.15.144:0/2708176947 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8efc0aaa90 con 0x55b54ec65070 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f114cc740 10 client.15393 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f114cc740 10 client.15393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f114cc740 20 client.15393 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ec239f0 0x55b54f0279e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f114cc740 4 client.15393 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b54ef82b10 con 0x55b54ec239f0 2024-07-26T11:03:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.326+0000 7f8f03400640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ec239f0 0x55b54f0279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.327+0000 7f8eed600640 10 client.15393 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.327+0000 7f8f03400640 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ec239f0 0x55b54f0279e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=308c5db0dc855d43 server_cookie=22d05360aae32267 in_seq=0 out_seq=0 2024-07-26T11:03:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.328+0000 7f8eed600640 1 -- 172.21.15.144:0/2708176947 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b54ef82b10 con 0x55b54ec239f0 2024-07-26T11:03:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.328+0000 7f8eed600640 10 client.15393 handle_command_reply: tid=0 2024-07-26T11:03:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.328+0000 7f8f114cc740 1 client.15393 shutdown 2024-07-26T11:03:17.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 msgr2=0x55b54f016000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f016000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0ab1a0 tx=0x7f8efc0ae950 comp rx=0 tx=0).stop 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 shutdown_connections 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ec239f0 0x55b54f0279e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8eed600640 0 client.15393 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f016000 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec44390 0x55b54f015ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2708176947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec3c1b0 0x55b54f015580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 >> 172.21.15.144:0/2708176947 conn(0x55b54ec59460 msgr2=0x55b54f00d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.329+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 shutdown_connections 2024-07-26T11:03:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.330+0000 7f8f114cc740 1 -- 172.21.15.144:0/2708176947 wait complete. 2024-07-26T11:03:17.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.330+0000 7f8f114cc740 20 client.15393 trim_cache size 0 max 16384 2024-07-26T11:03:17.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.364+0000 7f8f114cc740 1 Processor -- start 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.364+0000 7f8f114cc740 1 -- start start 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 0x55b54f017460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f006530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec65070 0x55b54f00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f114cc740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041b9510 con 0x55b54ec3c1b0 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f114cc740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041b6de0 con 0x55b54ec44390 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f114cc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041b4c30 con 0x55b54ec65070 2024-07-26T11:03:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f03400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 0x55b54f017460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f03400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 0x55b54f017460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54320/0 (socket says 172.21.15.144:54320) 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f03400640 1 -- 172.21.15.144:0/2748584075 learned_addr learned my addr 172.21.15.144:0/2748584075 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f03e00640 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec65070 0x55b54f00b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.365+0000 7f8f02a00640 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f006530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8f03400640 1 -- 172.21.15.144:0/2748584075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec65070 msgr2=0x55b54f00b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8f03400640 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec65070 0x55b54f00b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8f03400640 1 -- 172.21.15.144:0/2748584075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 msgr2=0x55b54f006530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8f03400640 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f006530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8f03400640 1 -- 172.21.15.144:0/2748584075 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54f00c2f0 con 0x55b54ec3c1b0 2024-07-26T11:03:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8f03400640 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 0x55b54f017460 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4011d10 tx=0x7f8ef4005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d17ab1bc56892a7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.366+0000 7f8eefe00640 1 -- 172.21.15.144:0/2748584075 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4005490 con 0x55b54ec3c1b0 2024-07-26T11:03:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.367+0000 7f8eefe00640 1 -- 172.21.15.144:0/2748584075 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef402f020 con 0x55b54ec3c1b0 2024-07-26T11:03:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.367+0000 7f8eefe00640 1 -- 172.21.15.144:0/2748584075 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef402bc60 con 0x55b54ec3c1b0 2024-07-26T11:03:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.367+0000 7f8f114cc740 1 -- 172.21.15.144:0/2748584075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 msgr2=0x55b54f017460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.367+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 0x55b54f017460 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4011d10 tx=0x7f8ef4005d30 comp rx=0 tx=0).stop 2024-07-26T11:03:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 -- 172.21.15.144:0/2748584075 shutdown_connections 2024-07-26T11:03:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec65070 0x55b54f00b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f006530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2748584075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec3c1b0 0x55b54f017460 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 -- 172.21.15.144:0/2748584075 >> 172.21.15.144:0/2748584075 conn(0x55b54ec59460 msgr2=0x55b54f00b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 -- 172.21.15.144:0/2748584075 shutdown_connections 2024-07-26T11:03:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 -- 172.21.15.144:0/2748584075 wait complete. 2024-07-26T11:03:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.368+0000 7f8f114cc740 1 -- start start 2024-07-26T11:03:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 Processor -- start 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f01b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f01be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f01c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041b52a0 con 0x55b54ec65070 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041b6710 con 0x55b54ec44390 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041b5970 con 0x55b54ec3c1b0 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f114cc740 10 client.0 _ll_register_callbacks cb 0x7f8efc0aefc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.369+0000 7f8f03400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f01b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f01be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f03400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f01b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57894/0 (socket says 172.21.15.144:57894) 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f03400640 1 -- 172.21.15.144:0/2751517368 learned_addr learned my addr 172.21.15.144:0/2751517368 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f01be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:57448/0 (socket says 172.21.15.144:57448) 2024-07-26T11:03:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f03e00640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f01c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 -- 172.21.15.144:0/2751517368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 msgr2=0x55b54f01b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f01b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 -- 172.21.15.144:0/2751517368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 msgr2=0x55b54f01c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f01c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f02a00640 1 -- 172.21.15.144:0/2751517368 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54f08f180 con 0x55b54ec44390 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.370+0000 7f8f03e00640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f01c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.371+0000 7f8f02a00640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f01be10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00057c0 tx=0x7f8ef0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4545c2b4b8cdf354 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.371+0000 7f8eed600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.371+0000 7f8eed600640 1 -- 172.21.15.144:0/2751517368 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0035040 con 0x55b54ec44390 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.371+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ef402f020 con 0x55b54ec44390 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.371+0000 7f8eed600640 1 -- 172.21.15.144:0/2751517368 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8ef0031e00 con 0x55b54ec44390 2024-07-26T11:03:17.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.372+0000 7f8eed600640 1 -- 172.21.15.144:0/2751517368 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8ef0003e30 con 0x55b54ec44390 2024-07-26T11:03:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.372+0000 7f8f114cc740 10 client.25538 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.372+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b54f098c00 con 0x55b54ec44390 2024-07-26T11:03:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.372+0000 7f8eed600640 1 -- 172.21.15.144:0/2751517368 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef402f020 con 0x55b54ec44390 2024-07-26T11:03:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8eed600640 1 -- 172.21.15.144:0/2751517368 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8ef0053020 con 0x55b54ec44390 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f114cc740 10 client.25538 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f114cc740 10 client.25538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f114cc740 20 client.25538 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ebf8da0 0x55b54f09b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f114cc740 4 client.25538 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b54ef7e5b0 con 0x55b54ebf8da0 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.373+0000 7f8f03400640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ebf8da0 0x55b54f09b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8f03400640 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ebf8da0 0x55b54f09b320 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=915b4befbc422405 server_cookie=3f9ce3072b873b38 in_seq=0 out_seq=0 2024-07-26T11:03:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8eed600640 10 client.25538 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8eed600640 1 -- 172.21.15.144:0/2751517368 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b54ef7e5b0 con 0x55b54ebf8da0 2024-07-26T11:03:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8eed600640 10 client.25538 handle_command_reply: tid=0 2024-07-26T11:03:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8f114cc740 1 client.25538 shutdown 2024-07-26T11:03:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 msgr2=0x55b54f01be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.374+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f01be10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00057c0 tx=0x7f8ef0000bc0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 shutdown_connections 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b54ebf8da0 0x55b54f09b320 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b54ec65070 0x55b54f01c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b54ec44390 0x55b54f01be10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 --2- 172.21.15.144:0/2751517368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b54ec3c1b0 0x55b54f01b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 >> 172.21.15.144:0/2751517368 conn(0x55b54ec59460 msgr2=0x55b54f0171c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8eed600640 0 client.25538 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 shutdown_connections 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 1 -- 172.21.15.144:0/2751517368 wait complete. 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.375+0000 7f8f114cc740 20 client.25538 trim_cache size 0 max 16384 2024-07-26T11:03:17.378 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:03:17.379 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "no active scrubs running", 2024-07-26T11:03:17.379 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": {} 2024-07-26T11:03:17.379 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:03:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8ee80b3dc0 msgr2=0x7f8ee80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8ee80b3dc0 0x7f8ee80b62d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0006010 tx=0x7f8ef0031400 comp rx=0 tx=0).stop 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 msgr2=0x7f8f0406e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0406e0a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00eb10 tx=0x7f8efc00c2d0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 shutdown_connections 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8ee80b3dc0 0x7f8ee80b62d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8f04109670 0x7f8f0406e0a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f04108d20 0x7f8f0406db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 --2- 172.21.15.144:0/111284440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f041033e0 0x7f8f04074870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 >> 172.21.15.144:0/111284440 conn(0x7f8f040fee30 msgr2=0x7f8f04101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.376+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 shutdown_connections 2024-07-26T11:03:17.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:17.377+0000 7f8f09200640 1 -- 172.21.15.144:0/111284440 wait complete. 2024-07-26T11:03:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:17 smithi160 ceph-mon[36715]: pgmap v485: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-07-26T11:03:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:17 smithi183 ceph-mon[37273]: pgmap v485: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-07-26T11:03:18.345 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:03:18.345 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:03:18.784 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.141+0000 7fa36f200640 1 -- 172.21.15.144:0/2716934094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3700751e0 msgr2=0x7fa3700755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.141+0000 7fa36f200640 1 --2- 172.21.15.144:0/2716934094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3700751e0 0x7fa3700755f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa360009900 tx=0x7fa36002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 -- 172.21.15.144:0/2716934094 shutdown_connections 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 --2- 172.21.15.144:0/2716934094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa37010f890 0x7fa370111cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 --2- 172.21.15.144:0/2716934094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa370075b30 0x7fa37010f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 --2- 172.21.15.144:0/2716934094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3700751e0 0x7fa3700755f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 -- 172.21.15.144:0/2716934094 >> 172.21.15.144:0/2716934094 conn(0x7fa3700fed60 msgr2=0x7fa3701011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 -- 172.21.15.144:0/2716934094 shutdown_connections 2024-07-26T11:03:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.142+0000 7fa36f200640 1 -- 172.21.15.144:0/2716934094 wait complete. 2024-07-26T11:03:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.144+0000 7fa36f200640 1 Processor -- start 2024-07-26T11:03:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.145+0000 7fa36f200640 1 -- start start 2024-07-26T11:03:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.145+0000 7fa36f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 0x7fa37019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa370075b30 0x7fa37019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa37010f890 0x7fa3701a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701154d0 con 0x7fa3700751e0 2024-07-26T11:03:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa370115350 con 0x7fa370075b30 2024-07-26T11:03:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa370115650 con 0x7fa37010f890 2024-07-26T11:03:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 0x7fa37019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 0x7fa37019be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54342/0 (socket says 172.21.15.144:54342) 2024-07-26T11:03:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36de00640 1 -- 172.21.15.144:0/4130657896 learned_addr learned my addr 172.21.15.144:0/4130657896 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36e800640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa37010f890 0x7fa3701a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.146+0000 7fa36d400640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa370075b30 0x7fa37019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36d400640 1 -- 172.21.15.144:0/4130657896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa37010f890 msgr2=0x7fa3701a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36d400640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa37010f890 0x7fa3701a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36d400640 1 -- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 msgr2=0x7fa37019be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36d400640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 0x7fa37019be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36d400640 1 -- 172.21.15.144:0/4130657896 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3701a1da0 con 0x7fa370075b30 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36de00640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 0x7fa37019be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa36d400640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa370075b30 0x7fa37019c410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa370077200 tx=0x7fa35c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.147+0000 7fa366a00640 1 -- 172.21.15.144:0/4130657896 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c00cd00 con 0x7fa370075b30 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.148+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3701a2090 con 0x7fa370075b30 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.148+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3701a8820 con 0x7fa370075b30 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.148+0000 7fa366a00640 1 -- 172.21.15.144:0/4130657896 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa35c0035f0 con 0x7fa370075b30 2024-07-26T11:03:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.148+0000 7fa366a00640 1 -- 172.21.15.144:0/4130657896 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c010690 con 0x7fa370075b30 2024-07-26T11:03:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.153+0000 7fa37ba2b740 1 Processor -- start 2024-07-26T11:03:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.154+0000 7fa37ba2b740 1 -- start start 2024-07-26T11:03:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.154+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 0x55d883340c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.154+0000 7fa366a00640 1 -- 172.21.15.144:0/4130657896 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa35c010830 con 0x7fa370075b30 2024-07-26T11:03:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.154+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f742f0 0x55d883341180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d8833416c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55d8832b5270 con 0x55d882f9d1b0 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d8832a4770 con 0x55d882f7c4d0 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d8832a87b0 con 0x55d882f742f0 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 0x55d883340c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36e800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 0x55d883340c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54356/0 (socket says 172.21.15.144:54356) 2024-07-26T11:03:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36e800640 1 -- 172.21.15.144:0/681091700 learned_addr learned my addr 172.21.15.144:0/681091700 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36de00640 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f742f0 0x55d883341180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36d400640 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d8833416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36de00640 1 -- 172.21.15.144:0/681091700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 msgr2=0x55d8833416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36de00640 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d8833416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.155+0000 7fa36de00640 1 -- 172.21.15.144:0/681091700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 msgr2=0x55d883340c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.156+0000 7fa36de00640 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 0x55d883340c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.156+0000 7fa36de00640 1 -- 172.21.15.144:0/681091700 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d883008790 con 0x55d882f742f0 2024-07-26T11:03:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.156+0000 7fa36e800640 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 0x55d883340c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.156+0000 7fa36de00640 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f742f0 0x55d883341180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa360031440 tx=0x7fa3600044b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae2ef55be123c074 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.157+0000 7fa35b000640 1 -- 172.21.15.144:0/681091700 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360035040 con 0x55d882f742f0 2024-07-26T11:03:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.157+0000 7fa35b000640 1 -- 172.21.15.144:0/681091700 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3600047a0 con 0x55d882f742f0 2024-07-26T11:03:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.157+0000 7fa35b000640 1 -- 172.21.15.144:0/681091700 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3600039c0 con 0x55d882f742f0 2024-07-26T11:03:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.157+0000 7fa366a00640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3540b3dc0 0x7fa3540b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.158+0000 7fa366a00640 1 -- 172.21.15.144:0/4130657896 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa35c0e4530 con 0x7fa370075b30 2024-07-26T11:03:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.158+0000 7fa36e800640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3540b3dc0 0x7fa3540b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.158+0000 7fa37ba2b740 1 -- 172.21.15.144:0/681091700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f742f0 msgr2=0x55d883341180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.158+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f742f0 0x55d883341180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa360031440 tx=0x7fa3600044b0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa36e800640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3540b3dc0 0x7fa3540b62d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa368011890 tx=0x7fa368009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 -- 172.21.15.144:0/681091700 shutdown_connections 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d8833416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f742f0 0x55d883341180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/681091700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f9d1b0 0x55d883340c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 -- 172.21.15.144:0/681091700 >> 172.21.15.144:0/681091700 conn(0x55d882f91300 msgr2=0x55d88333eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 -- 172.21.15.144:0/681091700 shutdown_connections 2024-07-26T11:03:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.159+0000 7fa37ba2b740 1 -- 172.21.15.144:0/681091700 wait complete. 2024-07-26T11:03:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.160+0000 7fa37ba2b740 1 -- start start 2024-07-26T11:03:19.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.160+0000 7fa37ba2b740 1 Processor -- start 2024-07-26T11:03:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 0x55d88334c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d88334cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d8833586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55d8831deab0 con 0x55d882f742f0 2024-07-26T11:03:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d883241d50 con 0x55d882f9d1b0 2024-07-26T11:03:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d88334f360 con 0x55d882f7c4d0 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa37ba2b740 10 client.0 _ll_register_callbacks cb 0x55d882fb4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.161+0000 7fa36de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 0x55d88334c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 0x55d88334c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54358/0 (socket says 172.21.15.144:54358) 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 -- 172.21.15.144:0/1432924768 learned_addr learned my addr 172.21.15.144:0/1432924768 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36d400640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d88334cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36e800640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d8833586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 -- 172.21.15.144:0/1432924768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 msgr2=0x55d88334cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d88334cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 -- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 msgr2=0x55d8833586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d8833586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.162+0000 7fa36de00640 1 -- 172.21.15.144:0/1432924768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d883081620 con 0x55d882f742f0 2024-07-26T11:03:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.163+0000 7fa36de00640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 0x55d88334c800 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa360005ee0 tx=0x7fa360003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78070c954d7be714 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.163+0000 7fa34f400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.163+0000 7fa34f400640 1 -- 172.21.15.144:0/1432924768 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360035070 con 0x55d882f742f0 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.163+0000 7fa34f400640 1 -- 172.21.15.144:0/1432924768 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa3600314a0 con 0x55d882f742f0 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.163+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3600047a0 con 0x55d882f742f0 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.164+0000 7fa34f400640 1 -- 172.21.15.144:0/1432924768 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa3600316a0 con 0x55d882f742f0 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.164+0000 7fa37ba2b740 10 client.15411 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.164+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d88335b3d0 con 0x55d882f742f0 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.164+0000 7fa34f400640 1 -- 172.21.15.144:0/1432924768 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3600047a0 con 0x55d882f742f0 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.164+0000 7fa34f400640 1 -- 172.21.15.144:0/1432924768 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa36003c9f0 con 0x55d882f742f0 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa37ba2b740 10 client.15411 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa37ba2b740 10 client.15411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa37ba2b740 20 client.15411 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d882f5bd80 0x55d88335e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa37ba2b740 4 client.15411 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8832b71c0 con 0x55d882f5bd80 2024-07-26T11:03:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.165+0000 7fa36d400640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d882f5bd80 0x55d88335e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.166+0000 7fa36d400640 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d882f5bd80 0x55d88335e730 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b4d15bfb66a9bb1 server_cookie=3faa9aacb478019d in_seq=0 out_seq=0 2024-07-26T11:03:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.166+0000 7fa34f400640 10 client.15411 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa34f400640 1 -- 172.21.15.144:0/1432924768 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d8832b71c0 con 0x55d882f5bd80 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa34f400640 10 client.15411 handle_command_reply: tid=0 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 client.15411 shutdown 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 msgr2=0x55d88334c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 0x55d88334c800 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa360005ee0 tx=0x7fa360003a40 comp rx=0 tx=0).stop 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 shutdown_connections 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d882f5bd80 0x55d88335e730 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d8833586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d88334cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/1432924768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f742f0 0x55d88334c800 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 >> 172.21.15.144:0/1432924768 conn(0x55d882f91300 msgr2=0x55d883341c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.167+0000 7fa34f400640 0 client.15411 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.168+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 shutdown_connections 2024-07-26T11:03:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.168+0000 7fa37ba2b740 1 -- 172.21.15.144:0/1432924768 wait complete. 2024-07-26T11:03:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.168+0000 7fa37ba2b740 20 client.15411 trim_cache size 0 max 16384 2024-07-26T11:03:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 Processor -- start 2024-07-26T11:03:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 -- start start 2024-07-26T11:03:19.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d883346d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d88334e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88333e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b5080 con 0x55d882f311c0 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b2950 con 0x55d882f9d1b0 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.199+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b07a0 con 0x55d882f7c4d0 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d883346d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d883346d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57948/0 (socket says 172.21.15.144:57948) 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 -- 172.21.15.144:0/489318381 learned_addr learned my addr 172.21.15.144:0/489318381 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36e800640 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88333e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36d400640 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d88334e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 -- 172.21.15.144:0/489318381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 msgr2=0x55d88334e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d88334e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 -- 172.21.15.144:0/489318381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 msgr2=0x55d88333e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88333e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36de00640 1 -- 172.21.15.144:0/489318381 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d88334ed30 con 0x55d882f7c4d0 2024-07-26T11:03:19.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.200+0000 7fa36e800640 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88333e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:19.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.201+0000 7fa36de00640 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d883346d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa360009800 tx=0x7fa3600044b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c133efcd6cbfa2c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.201+0000 7fa35a600640 1 -- 172.21.15.144:0/489318381 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360035070 con 0x55d882f7c4d0 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.201+0000 7fa35a600640 1 -- 172.21.15.144:0/489318381 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa360005850 con 0x55d882f7c4d0 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.201+0000 7fa35a600640 1 -- 172.21.15.144:0/489318381 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360005590 con 0x55d882f7c4d0 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.202+0000 7fa37ba2b740 1 -- 172.21.15.144:0/489318381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 msgr2=0x55d883346d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.202+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d883346d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa360009800 tx=0x7fa3600044b0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 -- 172.21.15.144:0/489318381 shutdown_connections 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88333e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f9d1b0 0x55d88334e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/489318381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f7c4d0 0x55d883346d80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 -- 172.21.15.144:0/489318381 >> 172.21.15.144:0/489318381 conn(0x55d882f91300 msgr2=0x55d88334be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 -- 172.21.15.144:0/489318381 shutdown_connections 2024-07-26T11:03:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 -- 172.21.15.144:0/489318381 wait complete. 2024-07-26T11:03:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.203+0000 7fa37ba2b740 1 -- start start 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 Processor -- start 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88335ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d88335ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f9d1b0 0x55d8833cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b0e10 con 0x55d882f311c0 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b2280 con 0x55d882f7c4d0 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b14e0 con 0x55d882f9d1b0 2024-07-26T11:03:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.204+0000 7fa37ba2b740 10 client.0 _ll_register_callbacks cb 0x7fa368009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88335ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88335ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54376/0 (socket says 172.21.15.144:54376) 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 -- 172.21.15.144:0/377638666 learned_addr learned my addr 172.21.15.144:0/377638666 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36e800640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f9d1b0 0x55d8833cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36d400640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d88335ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 -- 172.21.15.144:0/377638666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f9d1b0 msgr2=0x55d8833cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f9d1b0 0x55d8833cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 -- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 msgr2=0x55d88335ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d88335ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36de00640 1 -- 172.21.15.144:0/377638666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8833cd9c0 con 0x55d882f311c0 2024-07-26T11:03:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.205+0000 7fa36d400640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d88335ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.206+0000 7fa36de00640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88335ea50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa360003fd0 tx=0x7fa360005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324572b611c70179 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.206+0000 7fa34f400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.206+0000 7fa34f400640 1 -- 172.21.15.144:0/377638666 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360035040 con 0x55d882f311c0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.206+0000 7fa34f400640 1 -- 172.21.15.144:0/377638666 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa3600314a0 con 0x55d882f311c0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.206+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa360005850 con 0x55d882f311c0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa34f400640 1 -- 172.21.15.144:0/377638666 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa360067050 con 0x55d882f311c0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa37ba2b740 10 client.15417 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d8833ceaf0 con 0x55d882f311c0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa34f400640 1 -- 172.21.15.144:0/377638666 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa360005850 con 0x55d882f311c0 2024-07-26T11:03:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa34f400640 1 -- 172.21.15.144:0/377638666 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa3600047a0 con 0x55d882f311c0 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa37ba2b740 10 client.15417 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.207+0000 7fa37ba2b740 10 client.15417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa37ba2b740 20 client.15417 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d8833cefb0 0x55d8833d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa37ba2b740 4 client.15417 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8832b2c60 con 0x55d8833cefb0 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa36d400640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d8833cefb0 0x55d8833d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa36d400640 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d8833cefb0 0x55d8833d13d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c5ab3cb1c577af server_cookie=1a07556246d07253 in_seq=0 out_seq=0 2024-07-26T11:03:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.208+0000 7fa34f400640 10 client.15417 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.217+0000 7fa34f400640 1 -- 172.21.15.144:0/377638666 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d8832b2c60 con 0x55d8833cefb0 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.217+0000 7fa34f400640 10 client.15417 handle_command_reply: tid=0 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.217+0000 7fa37ba2b740 1 client.15417 shutdown 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.217+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 msgr2=0x55d88335ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.217+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88335ea50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa360003fd0 tx=0x7fa360005fd0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 shutdown_connections 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55d8833cefb0 0x55d8833d13d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d882f9d1b0 0x55d8833cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d882f7c4d0 0x55d88335ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 --2- 172.21.15.144:0/377638666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d882f311c0 0x55d88335ea50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 >> 172.21.15.144:0/377638666 conn(0x55d882f91300 msgr2=0x55d88334e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa34f400640 0 client.15417 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 shutdown_connections 2024-07-26T11:03:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 1 -- 172.21.15.144:0/377638666 wait complete. 2024-07-26T11:03:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.218+0000 7fa37ba2b740 20 client.15417 trim_cache size 0 max 16384 2024-07-26T11:03:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:03:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:03:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:03:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3540b3dc0 msgr2=0x7fa3540b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3540b3dc0 0x7fa3540b62d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa368011890 tx=0x7fa368009500 comp rx=0 tx=0).stop 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa370075b30 msgr2=0x7fa37019c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa370075b30 0x7fa37019c410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa370077200 tx=0x7fa35c00ee50 comp rx=0 tx=0).stop 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 shutdown_connections 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa3540b3dc0 0x7fa3540b62d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa37010f890 0x7fa3701a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa370075b30 0x7fa37019c410 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 --2- 172.21.15.144:0/4130657896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3700751e0 0x7fa37019be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 >> 172.21.15.144:0/4130657896 conn(0x7fa3700fed60 msgr2=0x7fa3701008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.219+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 shutdown_connections 2024-07-26T11:03:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:19.220+0000 7fa36f200640 1 -- 172.21.15.144:0/4130657896 wait complete. 2024-07-26T11:03:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:19 smithi160 ceph-mon[36715]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:03:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:19 smithi183 ceph-mon[37273]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:03:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:19 smithi144 ceph-mon[32129]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 -- 172.21.15.144:0/2978072067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 msgr2=0x7f7800075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 --2- 172.21.15.144:0/2978072067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f7800075c10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009900 tx=0x7f77ec02a400 comp rx=0 tx=0).stop 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 -- 172.21.15.144:0/2978072067 shutdown_connections 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 --2- 172.21.15.144:0/2978072067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800113cb0 0x7f7800116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 --2- 172.21.15.144:0/2978072067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f7800075c10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 --2- 172.21.15.144:0/2978072067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7800076db0 0x7f78000751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 -- 172.21.15.144:0/2978072067 >> 172.21.15.144:0/2978072067 conn(0x7f78000fee70 msgr2=0x7f78001012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:20.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 -- 172.21.15.144:0/2978072067 shutdown_connections 2024-07-26T11:03:20.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.985+0000 7f7805400640 1 -- 172.21.15.144:0/2978072067 wait complete. 2024-07-26T11:03:20.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 Processor -- start 2024-07-26T11:03:20.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 -- start start 2024-07-26T11:03:20.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f78001a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 0x7f78001a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7800113cb0 0x7f78001a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f78001198b0 con 0x7f7800075800 2024-07-26T11:03:20.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7800119730 con 0x7f7800076db0 2024-07-26T11:03:20.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.987+0000 7f7805400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7800119a30 con 0x7f7800113cb0 2024-07-26T11:03:20.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.988+0000 7f77ff400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 0x7f78001a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.988+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f78001a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.988+0000 7f77ff400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 0x7f78001a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42454/0 (socket says 172.21.15.144:42454) 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.988+0000 7f77ff400640 1 -- 172.21.15.144:0/894053980 learned_addr learned my addr 172.21.15.144:0/894053980 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.988+0000 7f7804a00640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7800113cb0 0x7f78001a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ffe00640 1 -- 172.21.15.144:0/894053980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7800113cb0 msgr2=0x7f78001a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ffe00640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7800113cb0 0x7f78001a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ffe00640 1 -- 172.21.15.144:0/894053980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 msgr2=0x7f78001a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ffe00640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 0x7f78001a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ffe00640 1 -- 172.21.15.144:0/894053980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78001a62d0 con 0x7f7800075800 2024-07-26T11:03:20.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ff400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 0x7f78001a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:20.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.989+0000 7f77ffe00640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f78001a0360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f77f000e9f0 tx=0x7f77f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:20.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.990+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f000ccc0 con 0x7f7800075800 2024-07-26T11:03:20.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.990+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77f00035f0 con 0x7f7800075800 2024-07-26T11:03:20.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.990+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f000ce60 con 0x7f7800075800 2024-07-26T11:03:20.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.990+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78001a65c0 con 0x7f7800075800 2024-07-26T11:03:20.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.990+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78001acd20 con 0x7f7800075800 2024-07-26T11:03:20.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.992+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7800075c10 con 0x7f7800075800 2024-07-26T11:03:20.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.994+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f77f00037a0 con 0x7f7800075800 2024-07-26T11:03:20.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.997+0000 7f77fcc00640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77e40b3dc0 0x7f77e40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.997+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f77f00e4640 con 0x7f7800075800 2024-07-26T11:03:20.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.997+0000 7f77ff400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77e40b3dc0 0x7f77e40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:21.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:20.998+0000 7f77ff400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77e40b3dc0 0x7f77e40b62d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7800115be0 tx=0x7f77ec009680 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:21.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.004+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f77f00ab580 con 0x7f7800075800 2024-07-26T11:03:21.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.238+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78001a1490 con 0x7f7800075800 2024-07-26T11:03:21.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.239+0000 7f77fcc00640 1 -- 172.21.15.144:0/894053980 <== mon.0 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11931 (secure 0 0 0) 0x7f77f00aae80 con 0x7f7800075800 2024-07-26T11:03:21.242 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:03:21.242 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":26,"btime":"2024-07-26T11:02:20:221771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T11:02:19.492590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24703,24715,24728]},"id":1}]} 2024-07-26T11:03:21.242 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 26 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77e40b3dc0 msgr2=0x7f77e40b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77e40b3dc0 0x7f77e40b62d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7800115be0 tx=0x7f77ec009680 comp rx=0 tx=0).stop 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 msgr2=0x7f78001a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f78001a0360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f77f000e9f0 tx=0x7f77f000edf0 comp rx=0 tx=0).stop 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 shutdown_connections 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f77e40b3dc0 0x7f77e40b62d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7800113cb0 0x7f78001a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:21.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7800076db0 0x7f78001a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 --2- 172.21.15.144:0/894053980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7800075800 0x7f78001a0360 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.243+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 >> 172.21.15.144:0/894053980 conn(0x7f78000fee70 msgr2=0x7f780010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.244+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 shutdown_connections 2024-07-26T11:03:21.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:21.244+0000 7f7805400640 1 -- 172.21.15.144:0/894053980 wait complete. 2024-07-26T11:03:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:21 smithi183 ceph-mon[37273]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-07-26T11:03:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:21 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/894053980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:03:21.663 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:21 smithi144 ceph-mon[32129]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-07-26T11:03:21.663 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:21 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/894053980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:03:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:21 smithi160 ceph-mon[36715]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-07-26T11:03:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:21 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/894053980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:03:22.182 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:03:22.648 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:23 smithi144 ceph-mon[32129]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-07-26T11:03:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:23 smithi160 ceph-mon[36715]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-07-26T11:03:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:23 smithi183 ceph-mon[37273]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-07-26T11:03:25.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 -- 172.21.15.144:0/4190160956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 msgr2=0x7f223c1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 --2- 172.21.15.144:0/4190160956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 0x7f223c1037f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f222c009900 tx=0x7f222c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 -- 172.21.15.144:0/4190160956 shutdown_connections 2024-07-26T11:03:25.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 --2- 172.21.15.144:0/4190160956 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 --2- 172.21.15.144:0/4190160956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 0x7f223c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 --2- 172.21.15.144:0/4190160956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 0x7f223c1037f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 -- 172.21.15.144:0/4190160956 >> 172.21.15.144:0/4190160956 conn(0x7f223c0fee30 msgr2=0x7f223c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.128+0000 7f2241400640 1 -- 172.21.15.144:0/4190160956 shutdown_connections 2024-07-26T11:03:25.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.129+0000 7f2241400640 1 -- 172.21.15.144:0/4190160956 wait complete. 2024-07-26T11:03:25.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.130+0000 7f2241400640 1 Processor -- start 2024-07-26T11:03:25.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 -- start start 2024-07-26T11:03:25.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 0x7f223c19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 0x7f223c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f223c113370 con 0x7f223c108d20 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1131f0 con 0x7f223c109670 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.131+0000 7f2241400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1134f0 con 0x7f223c1033e0 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f223be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 0x7f223c19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c1a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42470/0 (socket says 172.21.15.144:42470) 2024-07-26T11:03:25.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 -- 172.21.15.144:0/4022985514 learned_addr learned my addr 172.21.15.144:0/4022985514 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:25.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f223b400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 0x7f223c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 -- 172.21.15.144:0/4022985514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 msgr2=0x7f223c19bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 0x7f223c19bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 -- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 msgr2=0x7f223c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 0x7f223c19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f2240a00640 1 -- 172.21.15.144:0/4022985514 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c1a1eb0 con 0x7f223c109670 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.132+0000 7f223b400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 0x7f223c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.133+0000 7f2240a00640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c1a1780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f223400efd0 tx=0x7f223400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.133+0000 7f2238c00640 1 -- 172.21.15.144:0/4022985514 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234003b70 con 0x7f223c109670 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.133+0000 7f2238c00640 1 -- 172.21.15.144:0/4022985514 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2234003d10 con 0x7f223c109670 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.133+0000 7f2238c00640 1 -- 172.21.15.144:0/4022985514 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340108a0 con 0x7f223c109670 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.133+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223c1a21a0 con 0x7f223c109670 2024-07-26T11:03:25.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.133+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223c1a8900 con 0x7f223c109670 2024-07-26T11:03:25.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.138+0000 7f22497f8740 1 Processor -- start 2024-07-26T11:03:25.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.139+0000 7f2238c00640 1 -- 172.21.15.144:0/4022985514 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f2234004020 con 0x7f223c109670 2024-07-26T11:03:25.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 -- start start 2024-07-26T11:03:25.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e882bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e845d390 0x55f3e882c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f3e8799bf0 con 0x55f3e847e070 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f3e87890f0 con 0x55f3e845d390 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.140+0000 7f22497f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f3e878d130 con 0x55f3e84551b0 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.141+0000 7f223be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e882bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.141+0000 7f223be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e882bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:47784/0 (socket says 172.21.15.144:47784) 2024-07-26T11:03:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.141+0000 7f223be00640 1 -- 172.21.15.144:0/3071862895 learned_addr learned my addr 172.21.15.144:0/3071862895 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.141+0000 7f223b400640 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.141+0000 7f223b400640 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e845d390 0x55f3e882c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f223b400640 1 -- 172.21.15.144:0/3071862895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 msgr2=0x55f3e882bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f223b400640 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e882bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f223b400640 1 -- 172.21.15.144:0/3071862895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 msgr2=0x55f3e882b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f223b400640 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f223b400640 1 -- 172.21.15.144:0/3071862895 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e84e9650 con 0x55f3e845d390 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f223b400640 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e845d390 0x55f3e882c2c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f223c06b130 tx=0x7f2228004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52d09d60b10b49a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.142+0000 7f2238c00640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22200b3e40 0x7f22200b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.143+0000 7f2238c00640 1 -- 172.21.15.144:0/4022985514 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f22340e4830 con 0x7f223c109670 2024-07-26T11:03:25.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.143+0000 7f2231200640 1 -- 172.21.15.144:0/3071862895 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228013050 con 0x55f3e845d390 2024-07-26T11:03:25.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.143+0000 7f2231200640 1 -- 172.21.15.144:0/3071862895 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22280131f0 con 0x55f3e845d390 2024-07-26T11:03:25.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.143+0000 7f2231200640 1 -- 172.21.15.144:0/3071862895 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222800ce20 con 0x55f3e845d390 2024-07-26T11:03:25.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.144+0000 7f223be00640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22200b3e40 0x7f22200b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.144+0000 7f22497f8740 1 -- 172.21.15.144:0/3071862895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e845d390 msgr2=0x55f3e882c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.144+0000 7f22497f8740 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e845d390 0x55f3e882c2c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f223c06b130 tx=0x7f2228004730 comp rx=0 tx=0).stop 2024-07-26T11:03:25.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.145+0000 7f22497f8740 1 -- 172.21.15.144:0/3071862895 shutdown_connections 2024-07-26T11:03:25.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.145+0000 7f22497f8740 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e845d390 0x55f3e882c2c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.145+0000 7f22497f8740 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e882bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.145+0000 7f22497f8740 1 --2- 172.21.15.144:0/3071862895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.145+0000 7f22497f8740 1 -- 172.21.15.144:0/3071862895 >> 172.21.15.144:0/3071862895 conn(0x55f3e8472460 msgr2=0x55f3e8829730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.145+0000 7f223be00640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22200b3e40 0x7f22200b6350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f222c0096c0 tx=0x7f222c005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.146+0000 7f22497f8740 1 -- 172.21.15.144:0/3071862895 shutdown_connections 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.146+0000 7f22497f8740 1 -- 172.21.15.144:0/3071862895 wait complete. 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.146+0000 7f22497f8740 1 -- start start 2024-07-26T11:03:25.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.147+0000 7f22497f8740 1 Processor -- start 2024-07-26T11:03:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e84551b0 0x55f3e8836ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 0x55f3e88373e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e847e070 0x55f3e8837920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f3e85ec700 con 0x55f3e845d390 2024-07-26T11:03:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f3e87256c0 con 0x55f3e84551b0 2024-07-26T11:03:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f3e883a020 con 0x55f3e847e070 2024-07-26T11:03:25.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f22497f8740 10 client.0 _ll_register_callbacks cb 0x55f3e8735170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:25.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f223be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e84551b0 0x55f3e8836ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f223b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 0x55f3e88373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f2240a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e847e070 0x55f3e8837920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f223b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 0x55f3e88373e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49800/0 (socket says 172.21.15.144:49800) 2024-07-26T11:03:25.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.148+0000 7f223b400640 1 -- 172.21.15.144:0/2633255764 learned_addr learned my addr 172.21.15.144:0/2633255764 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.149+0000 7f223b400640 1 -- 172.21.15.144:0/2633255764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e847e070 msgr2=0x55f3e8837920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.149+0000 7f223b400640 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e847e070 0x55f3e8837920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.149+0000 7f223b400640 1 -- 172.21.15.144:0/2633255764 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e84551b0 msgr2=0x55f3e8836ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.149+0000 7f223b400640 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e84551b0 0x55f3e8836ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.149+0000 7f223b400640 1 -- 172.21.15.144:0/2633255764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e873c860 con 0x55f3e845d390 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.150+0000 7f223b400640 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 0x55f3e88373e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2228009c60 tx=0x7f222800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe29a5fec808494 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.150+0000 7f2225600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.150+0000 7f2225600640 1 -- 172.21.15.144:0/2633255764 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228025070 con 0x55f3e845d390 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.150+0000 7f2225600640 1 -- 172.21.15.144:0/2633255764 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f2228021830 con 0x55f3e845d390 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.150+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22280131f0 con 0x55f3e845d390 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.150+0000 7f2225600640 1 -- 172.21.15.144:0/2633255764 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2228031050 con 0x55f3e845d390 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.151+0000 7f22497f8740 10 client.15435 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.151+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f3e8845d00 con 0x55f3e845d390 2024-07-26T11:03:25.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.151+0000 7f2225600640 1 -- 172.21.15.144:0/2633255764 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22280131f0 con 0x55f3e845d390 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.151+0000 7f2225600640 1 -- 172.21.15.144:0/2633255764 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f22280107f0 con 0x55f3e845d390 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f22497f8740 10 client.15435 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f22497f8740 10 client.15435 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f22497f8740 20 client.15435 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f22497f8740 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e843c9f0 0x55f3e88490c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f22497f8740 4 client.15435 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3e879bb40 con 0x55f3e843c9f0 2024-07-26T11:03:25.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.152+0000 7f223be00640 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e843c9f0 0x55f3e88490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.153+0000 7f223be00640 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e843c9f0 0x55f3e88490c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4ef2570dd49aaba0 server_cookie=56a516b70659955f in_seq=0 out_seq=0 2024-07-26T11:03:25.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.153+0000 7f2225600640 10 client.15435 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:25.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.155+0000 7f2225600640 1 -- 172.21.15.144:0/2633255764 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f3e879bb40 con 0x55f3e843c9f0 2024-07-26T11:03:25.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.155+0000 7f2225600640 10 client.15435 handle_command_reply: tid=0 2024-07-26T11:03:25.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.155+0000 7f22497f8740 1 client.15435 shutdown 2024-07-26T11:03:25.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.155+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 msgr2=0x55f3e88373e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.155+0000 7f22497f8740 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 0x55f3e88373e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2228009c60 tx=0x7f222800cdd0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 shutdown_connections 2024-07-26T11:03:25.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e843c9f0 0x55f3e88490c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e847e070 0x55f3e8837920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e845d390 0x55f3e88373e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 --2- 172.21.15.144:0/2633255764 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e84551b0 0x55f3e8836ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 >> 172.21.15.144:0/2633255764 conn(0x55f3e8472460 msgr2=0x55f3e882e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f2225600640 0 client.15435 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:25.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 shutdown_connections 2024-07-26T11:03:25.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 1 -- 172.21.15.144:0/2633255764 wait complete. 2024-07-26T11:03:25.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.156+0000 7f22497f8740 20 client.15435 trim_cache size 0 max 16384 2024-07-26T11:03:25.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 Processor -- start 2024-07-26T11:03:25.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 -- start start 2024-07-26T11:03:25.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e8837860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e8411da0 0x55f3e8843190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1b0c50 con 0x55f3e847e070 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1ae520 con 0x55f3e8411da0 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.171+0000 7f22497f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1ac3b0 con 0x55f3e84551b0 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49808/0 (socket says 172.21.15.144:49808) 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 -- 172.21.15.144:0/1541623160 learned_addr learned my addr 172.21.15.144:0/1541623160 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223be00640 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e8837860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f2240a00640 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e8411da0 0x55f3e8843190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 -- 172.21.15.144:0/1541623160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 msgr2=0x55f3e8837860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e8837860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 -- 172.21.15.144:0/1541623160 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e8411da0 msgr2=0x55f3e8843190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e8411da0 0x55f3e8843190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.172+0000 7f223b400640 1 -- 172.21.15.144:0/1541623160 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e88353a0 con 0x55f3e847e070 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.173+0000 7f223b400640 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882cf80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2228010bf0 tx=0x7f2228009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c23359286c1d05 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.173+0000 7f2227e00640 1 -- 172.21.15.144:0/1541623160 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222802e070 con 0x55f3e847e070 2024-07-26T11:03:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.173+0000 7f2227e00640 1 -- 172.21.15.144:0/1541623160 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22280099b0 con 0x55f3e847e070 2024-07-26T11:03:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.173+0000 7f2227e00640 1 -- 172.21.15.144:0/1541623160 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228025040 con 0x55f3e847e070 2024-07-26T11:03:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 -- 172.21.15.144:0/1541623160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 msgr2=0x55f3e882cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882cf80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2228010bf0 tx=0x7f2228009610 comp rx=0 tx=0).stop 2024-07-26T11:03:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 -- 172.21.15.144:0/1541623160 shutdown_connections 2024-07-26T11:03:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e8411da0 0x55f3e8843190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e847e070 0x55f3e882cf80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 --2- 172.21.15.144:0/1541623160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e8837860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 -- 172.21.15.144:0/1541623160 >> 172.21.15.144:0/1541623160 conn(0x55f3e8472460 msgr2=0x55f3e882c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 -- 172.21.15.144:0/1541623160 shutdown_connections 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.174+0000 7f22497f8740 1 -- 172.21.15.144:0/1541623160 wait complete. 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.175+0000 7f22497f8740 1 -- start start 2024-07-26T11:03:25.177 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.175+0000 7f22497f8740 1 Processor -- start 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e8411da0 0x55f3e883a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e883acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 0x55f3e88af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1aca00 con 0x55f3e8411da0 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1ade50 con 0x55f3e847e070 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1ad0b0 con 0x55f3e84551b0 2024-07-26T11:03:25.178 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f22497f8740 10 client.0 _ll_register_callbacks cb 0x7f22340093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:25.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f2240a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 0x55f3e88af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f2240a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 0x55f3e88af680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42488/0 (socket says 172.21.15.144:42488) 2024-07-26T11:03:25.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f2240a00640 1 -- 172.21.15.144:0/4158925821 learned_addr learned my addr 172.21.15.144:0/4158925821 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:25.179 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.176+0000 7f223be00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e8411da0 0x55f3e883a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2240a00640 1 -- 172.21.15.144:0/4158925821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 msgr2=0x55f3e883acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2240a00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e883acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2240a00640 1 -- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e8411da0 msgr2=0x55f3e883a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2240a00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e8411da0 0x55f3e883a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2240a00640 1 -- 172.21.15.144:0/4158925821 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e88afbc0 con 0x55f3e847e070 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f223be00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e8411da0 0x55f3e883a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2240a00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 0x55f3e88af680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f22340e1c10 tx=0x7f2234004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e7af0f2d8fef51 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.180 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.177+0000 7f2225600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f2225600640 1 -- 172.21.15.144:0/4158925821 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340c4070 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f2225600640 1 -- 172.21.15.144:0/4158925821 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f22340e43f0 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22280099b0 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f2225600640 1 -- 172.21.15.144:0/4158925821 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f22340e2040 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f22497f8740 10 client.25577 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f3e88b0d50 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.178+0000 7f2225600640 1 -- 172.21.15.144:0/4158925821 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22280099b0 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f2225600640 1 -- 172.21.15.144:0/4158925821 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f223401f020 con 0x55f3e847e070 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f22497f8740 10 client.25577 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:25.181 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f22497f8740 10 client.25577 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:03:25.182 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f22497f8740 20 client.25577 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:25.182 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f22497f8740 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e88b14d0 0x55f3e88b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.182 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f22497f8740 4 client.25577 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:03:25.182 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3e87975e0 con 0x55f3e88b14d0 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.179+0000 7f223be00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e88b14d0 0x55f3e88b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.180+0000 7f223be00640 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e88b14d0 0x55f3e88b38f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=360196eeecb2a8c8 server_cookie=7cebb4dfb9b2b122 in_seq=0 out_seq=0 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.180+0000 7f2225600640 10 client.25577 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.180+0000 7f2225600640 1 -- 172.21.15.144:0/4158925821 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3e87975e0 con 0x55f3e88b14d0 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.180+0000 7f2225600640 10 client.25577 handle_command_reply: tid=0 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.180+0000 7f22497f8740 1 client.25577 shutdown 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 msgr2=0x55f3e88af680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.183 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 0x55f3e88af680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f22340e1c10 tx=0x7f2234004790 comp rx=0 tx=0).stop 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 shutdown_connections 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55f3e88b14d0 0x55f3e88b38f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3e847e070 0x55f3e88af680 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3e84551b0 0x55f3e883acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 --2- 172.21.15.144:0/4158925821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3e8411da0 0x55f3e883a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 >> 172.21.15.144:0/4158925821 conn(0x55f3e8472460 msgr2=0x55f3e8837600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f2225600640 0 client.25577 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 shutdown_connections 2024-07-26T11:03:25.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.181+0000 7f22497f8740 1 -- 172.21.15.144:0/4158925821 wait complete. 2024-07-26T11:03:25.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.182+0000 7f22497f8740 20 client.25577 trim_cache size 0 max 16384 2024-07-26T11:03:25.185 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:03:25.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.182+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22200b3e40 msgr2=0x7f22200b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.182+0000 7f2241400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22200b3e40 0x7f22200b6350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f222c0096c0 tx=0x7f222c005a20 comp rx=0 tx=0).stop 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.182+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 msgr2=0x7f223c1a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.182+0000 7f2241400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c1a1780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f223400efd0 tx=0x7f223400c6a0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 shutdown_connections 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f22200b3e40 0x7f22200b6350 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f223c109670 0x7f223c1a1780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f223c108d20 0x7f223c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 --2- 172.21.15.144:0/4022985514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f223c1033e0 0x7f223c19bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 >> 172.21.15.144:0/4022985514 conn(0x7f223c0fee30 msgr2=0x7f223c108480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 shutdown_connections 2024-07-26T11:03:25.186 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:25.183+0000 7f2241400640 1 -- 172.21.15.144:0/4022985514 wait complete. 2024-07-26T11:03:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:25 smithi144 ceph-mon[32129]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-07-26T11:03:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:25 smithi160 ceph-mon[36715]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-07-26T11:03:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:25 smithi183 ceph-mon[37273]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-07-26T11:03:26.102 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:03:26.511 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:27.474 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:27 smithi144 ceph-mon[32129]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-07-26T11:03:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:27 smithi160 ceph-mon[36715]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-07-26T11:03:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:27 smithi183 ceph-mon[37273]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-07-26T11:03:28.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 -- 172.21.15.144:0/808084813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef64108d20 msgr2=0x7fef64109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 --2- 172.21.15.144:0/808084813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef64108d20 0x7fef64109130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fef50009860 tx=0x7fef5002a340 comp rx=0 tx=0).stop 2024-07-26T11:03:28.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 -- 172.21.15.144:0/808084813 shutdown_connections 2024-07-26T11:03:28.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 --2- 172.21.15.144:0/808084813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef6410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 --2- 172.21.15.144:0/808084813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef64108d20 0x7fef64109130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 --2- 172.21.15.144:0/808084813 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef641033e0 0x7fef641037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 -- 172.21.15.144:0/808084813 >> 172.21.15.144:0/808084813 conn(0x7fef640fee30 msgr2=0x7fef64101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.852+0000 7fef6a400640 1 -- 172.21.15.144:0/808084813 shutdown_connections 2024-07-26T11:03:28.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.853+0000 7fef6a400640 1 -- 172.21.15.144:0/808084813 wait complete. 2024-07-26T11:03:28.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.855+0000 7fef6a400640 1 Processor -- start 2024-07-26T11:03:28.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.855+0000 7fef6a400640 1 -- start start 2024-07-26T11:03:28.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef6a400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef641033e0 0x7fef641a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef6a400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef64108d20 0x7fef641a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef6a400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef641ac040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef6a400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fef641132e0 con 0x7fef64109670 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef6a400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fef64113160 con 0x7fef64108d20 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef6a400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fef64113460 con 0x7fef641033e0 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef69000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef641033e0 0x7fef641a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef69a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef641ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef69a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef641ac040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49844/0 (socket says 172.21.15.144:49844) 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.856+0000 7fef69a00640 1 -- 172.21.15.144:0/2338753564 learned_addr learned my addr 172.21.15.144:0/2338753564 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef63e00640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef64108d20 0x7fef641a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef69000640 1 -- 172.21.15.144:0/2338753564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef64108d20 msgr2=0x7fef641a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef69000640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef64108d20 0x7fef641a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef69000640 1 -- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 msgr2=0x7fef641ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef69000640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef641ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef69000640 1 -- 172.21.15.144:0/2338753564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef641ac580 con 0x7fef641033e0 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.857+0000 7fef69a00640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef641ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.858+0000 7fef69000640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef641033e0 0x7fef641a0fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef5800e790 tx=0x7fef5800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.858+0000 7fef61600640 1 -- 172.21.15.144:0/2338753564 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5800eea0 con 0x7fef641033e0 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.858+0000 7fef61600640 1 -- 172.21.15.144:0/2338753564 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef5800ccc0 con 0x7fef641033e0 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.858+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef641ac750 con 0x7fef641033e0 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.858+0000 7fef61600640 1 -- 172.21.15.144:0/2338753564 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef58018680 con 0x7fef641033e0 2024-07-26T11:03:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.859+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef641acc50 con 0x7fef641033e0 2024-07-26T11:03:28.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.863+0000 7fef727c1740 1 Processor -- start 2024-07-26T11:03:28.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.863+0000 7fef727c1740 1 -- start start 2024-07-26T11:03:28.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.864+0000 7fef727c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564297515070 0x5642978ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.864+0000 7fef727c1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974ec1b0 0x5642978ba550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.864+0000 7fef727c1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978baa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.864+0000 7fef727c1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564297830bd0 con 0x5642974ec1b0 2024-07-26T11:03:28.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.864+0000 7fef727c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642978200d0 con 0x564297515070 2024-07-26T11:03:28.866 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.864+0000 7fef727c1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564297824110 con 0x5642974f4390 2024-07-26T11:03:28.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978baa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:47830/0 (socket says 172.21.15.144:47830) 2024-07-26T11:03:28.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 -- 172.21.15.144:0/3277616181 learned_addr learned my addr 172.21.15.144:0/3277616181 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:28.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef69a00640 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564297515070 0x5642978ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 -- 172.21.15.144:0/3277616181 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564297515070 msgr2=0x5642978ba010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564297515070 0x5642978ba010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 -- 172.21.15.144:0/3277616181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974ec1b0 msgr2=0x5642978ba550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974ec1b0 0x5642978ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef63e00640 1 -- 172.21.15.144:0/3277616181 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564297580650 con 0x5642974f4390 2024-07-26T11:03:28.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.865+0000 7fef61600640 1 -- 172.21.15.144:0/2338753564 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fef58010430 con 0x7fef641033e0 2024-07-26T11:03:28.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.866+0000 7fef63e00640 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978baa90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fef50009830 tx=0x7fef50005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee6b853af0457ac0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.866+0000 7fef56000640 1 -- 172.21.15.144:0/3277616181 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50041020 con 0x5642974f4390 2024-07-26T11:03:28.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.866+0000 7fef56000640 1 -- 172.21.15.144:0/3277616181 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef50030450 con 0x5642974f4390 2024-07-26T11:03:28.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.867+0000 7fef56000640 1 -- 172.21.15.144:0/3277616181 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50037040 con 0x5642974f4390 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 -- 172.21.15.144:0/3277616181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 msgr2=0x5642978baa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978baa90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fef50009830 tx=0x7fef50005e00 comp rx=0 tx=0).stop 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 -- 172.21.15.144:0/3277616181 shutdown_connections 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978baa90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974ec1b0 0x5642978ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 --2- 172.21.15.144:0/3277616181 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564297515070 0x5642978ba010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef727c1740 1 -- 172.21.15.144:0/3277616181 >> 172.21.15.144:0/3277616181 conn(0x564297509460 msgr2=0x5642978b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.868+0000 7fef61600640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef4c0b3e40 0x7fef4c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.869+0000 7fef61600640 1 -- 172.21.15.144:0/2338753564 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fef580e4560 con 0x7fef641033e0 2024-07-26T11:03:28.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.869+0000 7fef727c1740 1 -- 172.21.15.144:0/3277616181 shutdown_connections 2024-07-26T11:03:28.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.869+0000 7fef727c1740 1 -- 172.21.15.144:0/3277616181 wait complete. 2024-07-26T11:03:28.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.869+0000 7fef63e00640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef4c0b3e40 0x7fef4c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.870+0000 7fef727c1740 1 -- start start 2024-07-26T11:03:28.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.870+0000 7fef727c1740 1 Processor -- start 2024-07-26T11:03:28.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef63e00640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef4c0b3e40 0x7fef4c0b6350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fef5002a930 tx=0x7fef5002a340 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974ec1b0 0x5642978c5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974f4390 0x5642978c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564297683700 con 0x564297515070 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642977bc6a0 con 0x5642974f4390 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642978c8790 con 0x5642974ec1b0 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.871+0000 7fef727c1740 10 client.0 _ll_register_callbacks cb 0x5642977cc150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef63e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974f4390 0x5642978c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef63e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974f4390 0x5642978c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42518/0 (socket says 172.21.15.144:42518) 2024-07-26T11:03:28.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef63e00640 1 -- 172.21.15.144:0/2500746426 learned_addr learned my addr 172.21.15.144:0/2500746426 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:28.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef69a00640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.875 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef69000640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974ec1b0 0x5642978c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef69000640 1 -- 172.21.15.144:0/2500746426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974f4390 msgr2=0x5642978c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef69000640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974f4390 0x5642978c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.872+0000 7fef69000640 1 -- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 msgr2=0x5642978c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.873+0000 7fef69000640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.873+0000 7fef69000640 1 -- 172.21.15.144:0/2500746426 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642977d3840 con 0x5642974ec1b0 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.873+0000 7fef69a00640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.873+0000 7fef69000640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974ec1b0 0x5642978c5630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fef580e18e0 tx=0x7fef580ab3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbad55e173100562 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.873+0000 7fef4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.873+0000 7fef4aa00640 1 -- 172.21.15.144:0/2500746426 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580aa510 con 0x5642974ec1b0 2024-07-26T11:03:28.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.874+0000 7fef4aa00640 1 -- 172.21.15.144:0/2500746426 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fef58010720 con 0x5642974ec1b0 2024-07-26T11:03:28.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.874+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef50030450 con 0x5642974ec1b0 2024-07-26T11:03:28.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.874+0000 7fef4aa00640 1 -- 172.21.15.144:0/2500746426 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef580aa8b0 con 0x5642974ec1b0 2024-07-26T11:03:28.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.874+0000 7fef727c1740 10 client.25579 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.874+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5642978d43a0 con 0x5642974ec1b0 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef4aa00640 1 -- 172.21.15.144:0/2500746426 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef50030450 con 0x5642974ec1b0 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef4aa00640 1 -- 172.21.15.144:0/2500746426 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fef580a9a70 con 0x5642974ec1b0 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef727c1740 10 client.25579 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef727c1740 10 client.25579 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef727c1740 20 client.25579 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef727c1740 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974d39f0 0x5642978d78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef727c1740 4 client.25579 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:28.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.875+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564297832b20 con 0x5642974d39f0 2024-07-26T11:03:28.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.876+0000 7fef63e00640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974d39f0 0x5642978d78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.877+0000 7fef63e00640 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974d39f0 0x5642978d78c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=753051151976ef85 server_cookie=f428742347238958 in_seq=0 out_seq=0 2024-07-26T11:03:28.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.877+0000 7fef4aa00640 10 client.25579 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:28.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.878+0000 7fef4aa00640 1 -- 172.21.15.144:0/2500746426 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564297832b20 con 0x5642974d39f0 2024-07-26T11:03:28.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.878+0000 7fef4aa00640 10 client.25579 handle_command_reply: tid=0 2024-07-26T11:03:28.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.878+0000 7fef727c1740 1 client.25579 shutdown 2024-07-26T11:03:28.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974ec1b0 msgr2=0x5642978c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974ec1b0 0x5642978c5630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fef580e18e0 tx=0x7fef580ab3c0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 shutdown_connections 2024-07-26T11:03:28.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974d39f0 0x5642978d78c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974f4390 0x5642978c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 --2- 172.21.15.144:0/2500746426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974ec1b0 0x5642978c5630 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef4aa00640 0 client.25579 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 >> 172.21.15.144:0/2500746426 conn(0x564297509460 msgr2=0x5642978bd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 shutdown_connections 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.879+0000 7fef727c1740 1 -- 172.21.15.144:0/2500746426 wait complete. 2024-07-26T11:03:28.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.880+0000 7fef727c1740 20 client.25579 trim_cache size 0 max 16384 2024-07-26T11:03:28.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.906+0000 7fef727c1740 1 Processor -- start 2024-07-26T11:03:28.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 -- start start 2024-07-26T11:03:28.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978dd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978dd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fef641b9500 con 0x564297515070 2024-07-26T11:03:28.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fef641b6dd0 con 0x5642974ec1b0 2024-07-26T11:03:28.909 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef727c1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fef641b4c20 con 0x5642974f4390 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.907+0000 7fef69000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42530/0 (socket says 172.21.15.144:42530) 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 -- 172.21.15.144:0/2684513390 learned_addr learned my addr 172.21.15.144:0/2684513390 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69a00640 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978dd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef63e00640 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978dd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 -- 172.21.15.144:0/2684513390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 msgr2=0x5642978dd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978dd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.911 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 -- 172.21.15.144:0/2684513390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 msgr2=0x5642978dd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978dd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 -- 172.21.15.144:0/2684513390 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642978bc300 con 0x5642974ec1b0 2024-07-26T11:03:28.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.908+0000 7fef69000640 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c73f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef580a6e20 tx=0x7fef580aaca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10981feaa4511c8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.909+0000 7fef55600640 1 -- 172.21.15.144:0/2684513390 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580aa510 con 0x5642974ec1b0 2024-07-26T11:03:28.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.909+0000 7fef55600640 1 -- 172.21.15.144:0/2684513390 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef580ab4a0 con 0x5642974ec1b0 2024-07-26T11:03:28.912 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.909+0000 7fef55600640 1 -- 172.21.15.144:0/2684513390 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580a6710 con 0x5642974ec1b0 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 -- 172.21.15.144:0/2684513390 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 msgr2=0x5642978c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c73f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef580a6e20 tx=0x7fef580aaca0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 -- 172.21.15.144:0/2684513390 shutdown_connections 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564297515070 0x5642978dd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642974f4390 0x5642978dd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 --2- 172.21.15.144:0/2684513390 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c73f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.910+0000 7fef727c1740 1 -- 172.21.15.144:0/2684513390 >> 172.21.15.144:0/2684513390 conn(0x564297509460 msgr2=0x5642978c0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.911+0000 7fef727c1740 1 -- 172.21.15.144:0/2684513390 shutdown_connections 2024-07-26T11:03:28.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.911+0000 7fef727c1740 1 -- 172.21.15.144:0/2684513390 wait complete. 2024-07-26T11:03:28.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.911+0000 7fef727c1740 1 -- start start 2024-07-26T11:03:28.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.912+0000 7fef727c1740 1 Processor -- start 2024-07-26T11:03:28.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.912+0000 7fef727c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.915 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.912+0000 7fef727c1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974f4390 0x5642978c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef727c1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564297515070 0x5642978cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef727c1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fef641b5290 con 0x5642974f4390 2024-07-26T11:03:28.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef727c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fef641b6700 con 0x5642974ec1b0 2024-07-26T11:03:28.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef727c1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fef641b5960 con 0x564297515070 2024-07-26T11:03:28.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef69000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef727c1740 10 client.0 _ll_register_callbacks cb 0x7fef580a6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef69000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:42534/0 (socket says 172.21.15.144:42534) 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef69000640 1 -- 172.21.15.144:0/879086140 learned_addr learned my addr 172.21.15.144:0/879086140 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef63e00640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974f4390 0x5642978c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.913+0000 7fef69a00640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564297515070 0x5642978cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.914+0000 7fef63e00640 1 -- 172.21.15.144:0/879086140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564297515070 msgr2=0x5642978cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.914+0000 7fef63e00640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564297515070 0x5642978cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.914+0000 7fef63e00640 1 -- 172.21.15.144:0/879086140 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 msgr2=0x5642978c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.914+0000 7fef63e00640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.914+0000 7fef63e00640 1 -- 172.21.15.144:0/879086140 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642978cdd50 con 0x5642974f4390 2024-07-26T11:03:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.914+0000 7fef69a00640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564297515070 0x5642978cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:03:28.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.915+0000 7fef63e00640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974f4390 0x5642978c97f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5642978d1b20 tx=0x7fef50005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450335a81763c7f3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.915+0000 7fef4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.915+0000 7fef4aa00640 1 -- 172.21.15.144:0/879086140 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50041040 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.915+0000 7fef4aa00640 1 -- 172.21.15.144:0/879086140 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fef500046c0 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.915+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef580ab4a0 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef4aa00640 1 -- 172.21.15.144:0/879086140 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef5006a050 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 10 client.15456 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56429793f9e0 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef4aa00640 1 -- 172.21.15.144:0/879086140 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef580ab4a0 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef4aa00640 1 -- 172.21.15.144:0/879086140 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fef5003d7d0 con 0x5642974f4390 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 10 client.15456 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 10 client.15456 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-07-26T11:03:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 20 client.15456 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974a8da0 0x564297941cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 4 client.15456 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:03:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.916+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56429782e5c0 con 0x5642974a8da0 2024-07-26T11:03:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.917+0000 7fef69000640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974a8da0 0x564297941cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.918+0000 7fef4aa00640 10 client.15456 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.918+0000 7fef69000640 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974a8da0 0x564297941cf0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=711dc0aa6c32ad0f server_cookie=4621b569059479e3 in_seq=0 out_seq=0 2024-07-26T11:03:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef4aa00640 1 -- 172.21.15.144:0/879086140 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56429782e5c0 con 0x5642974a8da0 2024-07-26T11:03:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef4aa00640 10 client.15456 handle_command_reply: tid=0 2024-07-26T11:03:28.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 client.15456 shutdown 2024-07-26T11:03:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974f4390 msgr2=0x5642978c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974f4390 0x5642978c97f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5642978d1b20 tx=0x7fef50005e00 comp rx=0 tx=0).stop 2024-07-26T11:03:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 shutdown_connections 2024-07-26T11:03:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564297515070 0x5642978cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642974f4390 0x5642978c97f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x5642974a8da0 0x564297941cf0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.919+0000 7fef727c1740 1 --2- 172.21.15.144:0/879086140 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642974ec1b0 0x5642978c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.920+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 >> 172.21.15.144:0/879086140 conn(0x564297509460 msgr2=0x5642978c0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.920+0000 7fef4aa00640 0 client.15456 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.920+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 shutdown_connections 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.920+0000 7fef727c1740 1 -- 172.21.15.144:0/879086140 wait complete. 2024-07-26T11:03:28.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.921+0000 7fef727c1740 20 client.15456 trim_cache size 0 max 16384 2024-07-26T11:03:28.924 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:03:28.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef4c0b3e40 msgr2=0x7fef4c0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef4c0b3e40 0x7fef4c0b6350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fef5002a930 tx=0x7fef5002a340 comp rx=0 tx=0).stop 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef641033e0 msgr2=0x7fef641a0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef641033e0 0x7fef641a0fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef5800e790 tx=0x7fef5800ec50 comp rx=0 tx=0).stop 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 shutdown_connections 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fef4c0b3e40 0x7fef4c0b6350 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef64109670 0x7fef641ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fef64108d20 0x7fef641a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 --2- 172.21.15.144:0/2338753564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef641033e0 0x7fef641a0fc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 >> 172.21.15.144:0/2338753564 conn(0x7fef640fee30 msgr2=0x7fef640ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.922+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 shutdown_connections 2024-07-26T11:03:28.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:28.923+0000 7fef6a400640 1 -- 172.21.15.144:0/2338753564 wait complete. 2024-07-26T11:03:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:29 smithi160 ceph-mon[36715]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-07-26T11:03:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:29 smithi183 ceph-mon[37273]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-07-26T11:03:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:29 smithi144 ceph-mon[32129]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-07-26T11:03:29.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:03:30.253 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:31 smithi160 ceph-mon[36715]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-07-26T11:03:31.628 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:31 smithi144 ceph-mon[32129]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-07-26T11:03:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:31 smithi183 ceph-mon[37273]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-07-26T11:03:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 -- 172.21.15.144:0/981994319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac103500 msgr2=0x7f8fac103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/981994319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac103500 0x7f8fac103910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009900 tx=0x7f8f9c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 -- 172.21.15.144:0/981994319 shutdown_connections 2024-07-26T11:03:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/981994319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac109780 0x7f8fac10ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/981994319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac108e30 0x7f8fac109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/981994319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac103500 0x7f8fac103910 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.542+0000 7f8fb2c00640 1 -- 172.21.15.144:0/981994319 >> 172.21.15.144:0/981994319 conn(0x7f8fac0fef40 msgr2=0x7f8fac101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.543+0000 7f8fb2c00640 1 -- 172.21.15.144:0/981994319 shutdown_connections 2024-07-26T11:03:32.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.543+0000 7f8fb2c00640 1 -- 172.21.15.144:0/981994319 wait complete. 2024-07-26T11:03:32.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.544+0000 7f8fb2c00640 1 Processor -- start 2024-07-26T11:03:32.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 -- start start 2024-07-26T11:03:32.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 0x7f8fac074950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac108e30 0x7f8fac06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac109780 0x7f8fac06f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac06d110 con 0x7f8fac109780 2024-07-26T11:03:32.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac06cf90 con 0x7f8fac103500 2024-07-26T11:03:32.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.545+0000 7f8fb2c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac06d290 con 0x7f8fac108e30 2024-07-26T11:03:32.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.546+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 0x7f8fac074950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.546+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac109780 0x7f8fac06f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.546+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 0x7f8fac074950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49374/0 (socket says 172.21.15.144:49374) 2024-07-26T11:03:32.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.546+0000 7f8fabe00640 1 -- 172.21.15.144:0/625929241 learned_addr learned my addr 172.21.15.144:0/625929241 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.546+0000 7f8fabe00640 1 -- 172.21.15.144:0/625929241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac108e30 msgr2=0x7f8fac06ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.546+0000 7f8fabe00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac108e30 0x7f8fac06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.547+0000 7f8fabe00640 1 -- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac109780 msgr2=0x7f8fac06f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.547+0000 7f8fabe00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac109780 0x7f8fac06f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.547+0000 7f8fabe00640 1 -- 172.21.15.144:0/625929241 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac1ae8d0 con 0x7f8fac103500 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.547+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac109780 0x7f8fac06f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.547+0000 7f8fabe00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 0x7f8fac074950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c005ab0 tx=0x7f8f9c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.547+0000 7f8fa8c00640 1 -- 172.21.15.144:0/625929241 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c03f070 con 0x7f8fac103500 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.548+0000 7f8fa8c00640 1 -- 172.21.15.144:0/625929241 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f9c031b60 con 0x7f8fac103500 2024-07-26T11:03:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.548+0000 7f8fa8c00640 1 -- 172.21.15.144:0/625929241 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c035030 con 0x7f8fac103500 2024-07-26T11:03:32.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.548+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac075020 con 0x7f8fac103500 2024-07-26T11:03:32.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.548+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac0754c0 con 0x7f8fac103500 2024-07-26T11:03:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.553+0000 7f8fb9be4740 1 Processor -- start 2024-07-26T11:03:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.553+0000 7f8fb9be4740 1 -- start start 2024-07-26T11:03:32.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.554+0000 7f8fa8c00640 1 -- 172.21.15.144:0/625929241 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8f9c042070 con 0x7f8fac103500 2024-07-26T11:03:32.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.554+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55845c725270 con 0x55845c3ec4d0 2024-07-26T11:03:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55845c714770 con 0x55845c3e42f0 2024-07-26T11:03:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55845c7187b0 con 0x55845c40d1b0 2024-07-26T11:03:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39300/0 (socket says 172.21.15.144:39300) 2024-07-26T11:03:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.555+0000 7f8fb0c00640 1 -- 172.21.15.144:0/1016335418 learned_addr learned my addr 172.21.15.144:0/1016335418 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fab400640 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fb0c00640 1 -- 172.21.15.144:0/1016335418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 msgr2=0x55845c7b0fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7b0fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fb0c00640 1 -- 172.21.15.144:0/1016335418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 msgr2=0x55845c7b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fb0c00640 1 -- 172.21.15.144:0/1016335418 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845c478790 con 0x55845c40d1b0 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fab400640 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.556+0000 7f8fa8c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8f900b3e40 0x7f8f900b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.557+0000 7f8fa8c00640 1 -- 172.21.15.144:0/625929241 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8f9c104ad0 con 0x7f8fac103500 2024-07-26T11:03:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.557+0000 7f8fabe00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8f900b3e40 0x7f8f900b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.557+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7b0a60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4019c20 tx=0x7f8fa40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9d3497cd5b07db server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.558+0000 7f8fa1200640 1 -- 172.21.15.144:0/1016335418 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4004d20 con 0x55845c40d1b0 2024-07-26T11:03:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.558+0000 7f8fa1200640 1 -- 172.21.15.144:0/1016335418 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4004050 con 0x55845c40d1b0 2024-07-26T11:03:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.558+0000 7f8fa1200640 1 -- 172.21.15.144:0/1016335418 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4004d20 con 0x55845c40d1b0 2024-07-26T11:03:32.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fb9be4740 1 -- 172.21.15.144:0/1016335418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 msgr2=0x55845c7b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fabe00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8f900b3e40 0x7f8f900b6350 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55845c7af6e0 tx=0x7f8f9c009260 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7b0a60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4019c20 tx=0x7f8fa40107a0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fb9be4740 1 -- 172.21.15.144:0/1016335418 shutdown_connections 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.559+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/1016335418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7b0a60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.560+0000 7f8fb9be4740 1 -- 172.21.15.144:0/1016335418 >> 172.21.15.144:0/1016335418 conn(0x55845c401300 msgr2=0x55845c7ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.560+0000 7f8fb9be4740 1 -- 172.21.15.144:0/1016335418 shutdown_connections 2024-07-26T11:03:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.560+0000 7f8fb9be4740 1 -- 172.21.15.144:0/1016335418 wait complete. 2024-07-26T11:03:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.561+0000 7f8fb9be4740 1 -- start start 2024-07-26T11:03:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.561+0000 7f8fb9be4740 1 Processor -- start 2024-07-26T11:03:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c3e42f0 0x55845c7bc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 0x55845c7bcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c40d1b0 0x55845c7c8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55845c64eab0 con 0x55845c40d1b0 2024-07-26T11:03:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55845c6b1d50 con 0x55845c3ec4d0 2024-07-26T11:03:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55845c7bf1e0 con 0x55845c3e42f0 2024-07-26T11:03:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fb9be4740 10 client.0 _ll_register_callbacks cb 0x55845c424a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 0x55845c7bcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.562+0000 7f8fab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 0x55845c7bcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49402/0 (socket says 172.21.15.144:49402) 2024-07-26T11:03:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.563+0000 7f8fab400640 1 -- 172.21.15.144:0/561776982 learned_addr learned my addr 172.21.15.144:0/561776982 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.563+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c40d1b0 0x55845c7c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.563+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c3e42f0 0x55845c7bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8fab400640 1 -- 172.21.15.144:0/561776982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c3e42f0 msgr2=0x55845c7bc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8fab400640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c3e42f0 0x55845c7bc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8fab400640 1 -- 172.21.15.144:0/561776982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c40d1b0 msgr2=0x55845c7c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8fab400640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c40d1b0 0x55845c7c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8fab400640 1 -- 172.21.15.144:0/561776982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845c4f1620 con 0x55845c3ec4d0 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8fab400640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 0x55845c7bcb80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8f980103c0 tx=0x7f8f98010880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d16fb378b32539f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.564+0000 7f8f95600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.565+0000 7f8f95600640 1 -- 172.21.15.144:0/561776982 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98012920 con 0x55845c3ec4d0 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.565+0000 7f8f95600640 1 -- 172.21.15.144:0/561776982 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8f98012ac0 con 0x55845c3ec4d0 2024-07-26T11:03:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.565+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa4004050 con 0x55845c3ec4d0 2024-07-26T11:03:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.566+0000 7f8f95600640 1 -- 172.21.15.144:0/561776982 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8f98012cc0 con 0x55845c3ec4d0 2024-07-26T11:03:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.566+0000 7f8fb9be4740 10 client.25604 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.566+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55845c7cb1c0 con 0x55845c3ec4d0 2024-07-26T11:03:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.566+0000 7f8f95600640 1 -- 172.21.15.144:0/561776982 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4004050 con 0x55845c3ec4d0 2024-07-26T11:03:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.568+0000 7f8f95600640 1 -- 172.21.15.144:0/561776982 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8f9800f040 con 0x55845c3ec4d0 2024-07-26T11:03:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.568+0000 7f8fb9be4740 10 client.25604 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.568+0000 7f8fb9be4740 10 client.25604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.568+0000 7f8fb9be4740 20 client.25604 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.568+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3cbd80 0x55845c7ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.569+0000 7f8fb9be4740 4 client.25604 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.569+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55845c7271c0 con 0x55845c3cbd80 2024-07-26T11:03:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.569+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3cbd80 0x55845c7ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.570+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3cbd80 0x55845c7ce750 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70adee4b1338fe81 server_cookie=c5b56ed77e2dac15 in_seq=0 out_seq=0 2024-07-26T11:03:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.570+0000 7f8f95600640 10 client.25604 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.571+0000 7f8f95600640 1 -- 172.21.15.144:0/561776982 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55845c7271c0 con 0x55845c3cbd80 2024-07-26T11:03:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.571+0000 7f8f95600640 10 client.25604 handle_command_reply: tid=0 2024-07-26T11:03:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.571+0000 7f8fb9be4740 1 client.25604 shutdown 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.571+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 msgr2=0x55845c7bcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.571+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 0x55845c7bcb80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8f980103c0 tx=0x7f8f98010880 comp rx=0 tx=0).stop 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 shutdown_connections 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3cbd80 0x55845c7ce750 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c40d1b0 0x55845c7c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3ec4d0 0x55845c7bcb80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/561776982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c3e42f0 0x55845c7bc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 >> 172.21.15.144:0/561776982 conn(0x55845c401300 msgr2=0x55845c7b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8f95600640 0 client.25604 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 shutdown_connections 2024-07-26T11:03:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.572+0000 7f8fb9be4740 1 -- 172.21.15.144:0/561776982 wait complete. 2024-07-26T11:03:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.573+0000 7f8fb9be4740 20 client.25604 trim_cache size 0 max 16384 2024-07-26T11:03:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.605+0000 7f8fb9be4740 1 Processor -- start 2024-07-26T11:03:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.605+0000 7f8fb9be4740 1 -- start start 2024-07-26T11:03:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.606+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7be060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.606+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7d3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.606+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7be060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.607+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7be060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49410/0 (socket says 172.21.15.144:49410) 2024-07-26T11:03:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.607+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.607+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b9540 con 0x55845c3ec4d0 2024-07-26T11:03:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.607+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b6e10 con 0x55845c3e42f0 2024-07-26T11:03:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.607+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b4d10 con 0x55845c40d1b0 2024-07-26T11:03:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.607+0000 7f8fb0c00640 1 -- 172.21.15.144:0/3619402189 learned_addr learned my addr 172.21.15.144:0/3619402189 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.608+0000 7f8fab400640 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7d3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.608+0000 7f8fb0c00640 1 -- 172.21.15.144:0/3619402189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 msgr2=0x55845c7d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.608+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.608+0000 7f8fb0c00640 1 -- 172.21.15.144:0/3619402189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 msgr2=0x55845c7d3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.608+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7d3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.608+0000 7f8fb0c00640 1 -- 172.21.15.144:0/3619402189 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845c7aa370 con 0x55845c3e42f0 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.609+0000 7f8fab400640 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7d3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:03:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.609+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7be060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4004910 tx=0x7f8fa40048a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac52678bfad7780 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.609+0000 7f8f97e00640 1 -- 172.21.15.144:0/3619402189 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4026040 con 0x55845c3e42f0 2024-07-26T11:03:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.610+0000 7f8f97e00640 1 -- 172.21.15.144:0/3619402189 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4027040 con 0x55845c3e42f0 2024-07-26T11:03:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.610+0000 7f8f97e00640 1 -- 172.21.15.144:0/3619402189 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4004d20 con 0x55845c3e42f0 2024-07-26T11:03:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 -- 172.21.15.144:0/3619402189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 msgr2=0x55845c7be060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7be060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4004910 tx=0x7f8fa40048a0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 -- 172.21.15.144:0/3619402189 shutdown_connections 2024-07-26T11:03:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c7d4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7d3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/3619402189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7be060 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 -- 172.21.15.144:0/3619402189 >> 172.21.15.144:0/3619402189 conn(0x55845c401300 msgr2=0x55845c7bd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.611+0000 7f8fb9be4740 1 -- 172.21.15.144:0/3619402189 shutdown_connections 2024-07-26T11:03:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.612+0000 7f8fb9be4740 1 -- 172.21.15.144:0/3619402189 wait complete. 2024-07-26T11:03:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.612+0000 7f8fb9be4740 1 -- start start 2024-07-26T11:03:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 Processor -- start 2024-07-26T11:03:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c82eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b52d0 con 0x55845c3ec4d0 2024-07-26T11:03:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b6740 con 0x55845c3e42f0 2024-07-26T11:03:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b59a0 con 0x55845c40d1b0 2024-07-26T11:03:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.613+0000 7f8fb9be4740 10 client.0 _ll_register_callbacks cb 0x7f8f980093d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.614+0000 7f8fab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.614+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.614+0000 7f8fab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7c2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39102/0 (socket says 172.21.15.144:39102) 2024-07-26T11:03:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.614+0000 7f8fb0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49420/0 (socket says 172.21.15.144:49420) 2024-07-26T11:03:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.614+0000 7f8fab400640 1 -- 172.21.15.144:0/4092969795 learned_addr learned my addr 172.21.15.144:0/4092969795 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c82eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fab400640 1 -- 172.21.15.144:0/4092969795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 msgr2=0x55845c82eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fab400640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c82eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fab400640 1 -- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 msgr2=0x55845c7c1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fab400640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7c1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fab400640 1 -- 172.21.15.144:0/4092969795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845c82f520 con 0x55845c3ec4d0 2024-07-26T11:03:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.615+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.616+0000 7f8fab400640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7c2210 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8f98005940 tx=0x7f8f98025e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520fae24bea6a078 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.616+0000 7f8f95600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.617+0000 7f8f95600640 1 -- 172.21.15.144:0/4092969795 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9802a070 con 0x55845c3ec4d0 2024-07-26T11:03:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.617+0000 7f8f95600640 1 -- 172.21.15.144:0/4092969795 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f8f98012920 con 0x55845c3ec4d0 2024-07-26T11:03:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.617+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa4027040 con 0x55845c3ec4d0 2024-07-26T11:03:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.618+0000 7f8f95600640 1 -- 172.21.15.144:0/4092969795 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8f98004040 con 0x55845c3ec4d0 2024-07-26T11:03:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.618+0000 7f8fb9be4740 10 client.15468 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.618+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55845c8367f0 con 0x55845c3ec4d0 2024-07-26T11:03:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.618+0000 7f8f95600640 1 -- 172.21.15.144:0/4092969795 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4027040 con 0x55845c3ec4d0 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8f95600640 1 -- 172.21.15.144:0/4092969795 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8f98003b60 con 0x55845c3ec4d0 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8fb9be4740 10 client.15468 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8fb9be4740 10 client.15468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8fb9be4740 20 client.15468 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3a11c0 0x55845c838f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8fb9be4740 4 client.15468 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:03:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.620+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55845c722c60 con 0x55845c3a11c0 2024-07-26T11:03:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.622+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3a11c0 0x55845c838f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.623+0000 7f8fb0c00640 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3a11c0 0x55845c838f60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=170a60fa78a09132 server_cookie=24a666e3bec29cc in_seq=0 out_seq=0 2024-07-26T11:03:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.623+0000 7f8f95600640 10 client.15468 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.627+0000 7f8f95600640 1 -- 172.21.15.144:0/4092969795 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55845c722c60 con 0x55845c3a11c0 2024-07-26T11:03:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.627+0000 7f8f95600640 10 client.15468 handle_command_reply: tid=0 2024-07-26T11:03:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.627+0000 7f8fb9be4740 1 client.15468 shutdown 2024-07-26T11:03:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 msgr2=0x55845c7c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7c2210 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8f98005940 tx=0x7f8f98025e30 comp rx=0 tx=0).stop 2024-07-26T11:03:32.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 shutdown_connections 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55845c3a11c0 0x55845c838f60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55845c40d1b0 0x55845c82eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8f95600640 0 client.15468 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55845c3ec4d0 0x55845c7c2210 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 --2- 172.21.15.144:0/4092969795 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55845c3e42f0 0x55845c7c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.629+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 >> 172.21.15.144:0/4092969795 conn(0x55845c401300 msgr2=0x55845c7bd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.630+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 shutdown_connections 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.630+0000 7f8fb9be4740 1 -- 172.21.15.144:0/4092969795 wait complete. 2024-07-26T11:03:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.630+0000 7f8fb9be4740 20 client.15468 trim_cache size 0 max 16384 2024-07-26T11:03:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:03:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:03:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:03:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:03:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8f900b3e40 msgr2=0x7f8f900b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8f900b3e40 0x7f8f900b6350 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55845c7af6e0 tx=0x7f8f9c009260 comp rx=0 tx=0).stop 2024-07-26T11:03:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 msgr2=0x7f8fac074950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 0x7f8fac074950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c005ab0 tx=0x7f8f9c003c10 comp rx=0 tx=0).stop 2024-07-26T11:03:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 shutdown_connections 2024-07-26T11:03:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f8f900b3e40 0x7f8f900b6350 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac109780 0x7f8fac06f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fac108e30 0x7f8fac06ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 --2- 172.21.15.144:0/625929241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8fac103500 0x7f8fac074950 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.631+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 >> 172.21.15.144:0/625929241 conn(0x7f8fac0fef40 msgr2=0x7f8fac101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.635+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 shutdown_connections 2024-07-26T11:03:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:32.635+0000 7f8fb2c00640 1 -- 172.21.15.144:0/625929241 wait complete. 2024-07-26T11:03:32.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.675+0000 7faaed000640 1 -- 172.21.15.144:0/1793782267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 msgr2=0x7faae80755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.679 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:32.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.675+0000 7faaed000640 1 --2- 172.21.15.144:0/1793782267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae80755f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faad8009900 tx=0x7faad802a400 comp rx=0 tx=0).stop 2024-07-26T11:03:32.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.676+0000 7faaed000640 1 -- 172.21.15.144:0/1793782267 shutdown_connections 2024-07-26T11:03:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.676+0000 7faaed000640 1 --2- 172.21.15.144:0/1793782267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faae810f940 0x7faae8111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.676+0000 7faaed000640 1 --2- 172.21.15.144:0/1793782267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 0x7faae810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.676+0000 7faaed000640 1 --2- 172.21.15.144:0/1793782267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae80755f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.676+0000 7faaed000640 1 -- 172.21.15.144:0/1793782267 >> 172.21.15.144:0/1793782267 conn(0x7faae80fee70 msgr2=0x7faae81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.677+0000 7faaed000640 1 -- 172.21.15.144:0/1793782267 shutdown_connections 2024-07-26T11:03:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.677+0000 7faaed000640 1 -- 172.21.15.144:0/1793782267 wait complete. 2024-07-26T11:03:32.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.679+0000 7faaed000640 1 Processor -- start 2024-07-26T11:03:32.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.679+0000 7faaed000640 1 -- start start 2024-07-26T11:03:32.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faaed000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae810e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faaed000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 0x7faae810e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faaed000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faae810f940 0x7faae81a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faaed000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7faae8104040 con 0x7faae8075b30 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faaed000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faae8103ec0 con 0x7faae80751e0 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faaed000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faae81041c0 con 0x7faae810f940 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faae7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae810e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faae7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae810e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49426/0 (socket says 172.21.15.144:49426) 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.680+0000 7faae7400640 1 -- 172.21.15.144:0/3301661960 learned_addr learned my addr 172.21.15.144:0/3301661960 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.681+0000 7faae7e00640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faae810f940 0x7faae81a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.681+0000 7faae6a00640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 0x7faae810e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.681+0000 7faae7400640 1 -- 172.21.15.144:0/3301661960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faae810f940 msgr2=0x7faae81a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.681+0000 7faae7400640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faae810f940 0x7faae81a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.681+0000 7faae7400640 1 -- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 msgr2=0x7faae810e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.682+0000 7faae7400640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 0x7faae810e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.682+0000 7faae7400640 1 -- 172.21.15.144:0/3301661960 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae81a5160 con 0x7faae80751e0 2024-07-26T11:03:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.682+0000 7faae6a00640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 0x7faae810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.682+0000 7faae7400640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae810e2c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faad8005ab0 tx=0x7faad8003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.683+0000 7faadfe00640 1 -- 172.21.15.144:0/3301661960 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad803f070 con 0x7faae80751e0 2024-07-26T11:03:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.683+0000 7faadfe00640 1 -- 172.21.15.144:0/3301661960 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad8031d60 con 0x7faae80751e0 2024-07-26T11:03:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.683+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae81a53f0 con 0x7faae80751e0 2024-07-26T11:03:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.683+0000 7faadfe00640 1 -- 172.21.15.144:0/3301661960 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8035030 con 0x7faae80751e0 2024-07-26T11:03:32.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.683+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae81acce0 con 0x7faae80751e0 2024-07-26T11:03:32.691 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.688+0000 7faaf53e7740 1 Processor -- start 2024-07-26T11:03:32.691 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.688+0000 7faaf53e7740 1 -- start start 2024-07-26T11:03:32.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.689+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eecfb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.692 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.689+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee92d1b0 0x5596eecfb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.689+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eecfbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.689+0000 7faaf53e7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5596eec71bd0 con 0x5596ee935390 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.689+0000 7faaf53e7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5596eec610d0 con 0x5596ee92d1b0 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.689+0000 7faaf53e7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5596eec65110 con 0x5596ee956070 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.690+0000 7faae7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eecfb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.690+0000 7faae7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eecfb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39352/0 (socket says 172.21.15.144:39352) 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.690+0000 7faae7e00640 1 -- 172.21.15.144:0/3266360657 learned_addr learned my addr 172.21.15.144:0/3266360657 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.693 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.690+0000 7faae6a00640 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eecfbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.691+0000 7faae6a00640 1 -- 172.21.15.144:0/3266360657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 msgr2=0x5596eecfb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.691+0000 7faae6a00640 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eecfb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.694 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.691+0000 7faae6a00640 1 -- 172.21.15.144:0/3266360657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee92d1b0 msgr2=0x5596eecfb740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:03:32.695 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.691+0000 7faae6a00640 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee92d1b0 0x5596eecfb740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.695 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.691+0000 7faae6a00640 1 -- 172.21.15.144:0/3266360657 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596ee9c1650 con 0x5596ee935390 2024-07-26T11:03:32.695 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.691+0000 7faae6a00640 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eecfbc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faae806b130 tx=0x7faad40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6c3701ed71b04 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.695 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.692+0000 7faadcc00640 1 -- 172.21.15.144:0/3266360657 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4012450 con 0x5596ee935390 2024-07-26T11:03:32.695 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.692+0000 7faadcc00640 1 -- 172.21.15.144:0/3266360657 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad4012aa0 con 0x5596ee935390 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.693+0000 7faadfe00640 1 -- 172.21.15.144:0/3301661960 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7faad8042070 con 0x7faae80751e0 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.693+0000 7faaf53e7740 1 -- 172.21.15.144:0/3266360657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 msgr2=0x5596eecfbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faadcc00640 1 -- 172.21.15.144:0/3266360657 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4012450 con 0x5596ee935390 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eecfbc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faae806b130 tx=0x7faad40117b0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 -- 172.21.15.144:0/3266360657 shutdown_connections 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eecfbc80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.697 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee92d1b0 0x5596eecfb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.698 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3266360657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eecfb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.698 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 -- 172.21.15.144:0/3266360657 >> 172.21.15.144:0/3266360657 conn(0x5596ee94a460 msgr2=0x5596eecf90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.698 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.694+0000 7faaf53e7740 1 -- 172.21.15.144:0/3266360657 shutdown_connections 2024-07-26T11:03:32.698 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.695+0000 7faaf53e7740 1 -- 172.21.15.144:0/3266360657 wait complete. 2024-07-26T11:03:32.698 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.695+0000 7faaf53e7740 1 -- start start 2024-07-26T11:03:32.699 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.696+0000 7faaf53e7740 1 Processor -- start 2024-07-26T11:03:32.700 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.696+0000 7faadfe00640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faacc0b3d00 0x7faacc0b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.700 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.696+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee92d1b0 0x5596eed06720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.700 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.696+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed06c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eed071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faaf53e7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5596eeac4700 con 0x5596ee92d1b0 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faaf53e7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5596eebfd6a0 con 0x5596ee935390 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faaf53e7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5596eed09a30 con 0x5596ee956070 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faadfe00640 1 -- 172.21.15.144:0/3301661960 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7faad8104b30 con 0x7faae80751e0 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faae6a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed06c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faae6a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed06c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49428/0 (socket says 172.21.15.144:49428) 2024-07-26T11:03:32.701 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.697+0000 7faae6a00640 1 -- 172.21.15.144:0/3500783768 learned_addr learned my addr 172.21.15.144:0/3500783768 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.702 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.698+0000 7faaf53e7740 10 client.0 _ll_register_callbacks cb 0x5596eec0d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:32.702 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.698+0000 7faae7400640 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee92d1b0 0x5596eed06720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.702 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.698+0000 7faae7e00640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faacc0b3d00 0x7faacc0b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.702 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.699+0000 7faae7400640 1 -- 172.21.15.144:0/3500783768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 msgr2=0x5596eed071a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.699+0000 7faae7400640 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eed071a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.699+0000 7faae7400640 1 -- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 msgr2=0x5596eed06c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.699+0000 7faae7400640 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed06c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.699+0000 7faae7400640 1 -- 172.21.15.144:0/3500783768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596eec14840 con 0x5596ee92d1b0 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.699+0000 7faae7e00640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faacc0b3d00 0x7faacc0b6210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faae0005fd0 tx=0x7faae00098a0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.700+0000 7faae7400640 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee92d1b0 0x5596eed06720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faad8101eb0 tx=0x7faad80c78b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b104a5f88d076f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.703 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.700+0000 7faad1600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:32.704 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.700+0000 7faad1600640 1 -- 172.21.15.144:0/3500783768 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad80cae80 con 0x5596ee92d1b0 2024-07-26T11:03:32.704 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.700+0000 7faad1600640 1 -- 172.21.15.144:0/3500783768 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7faad8102130 con 0x5596ee92d1b0 2024-07-26T11:03:32.704 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.701+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faad4012aa0 con 0x5596ee92d1b0 2024-07-26T11:03:32.704 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.701+0000 7faad1600640 1 -- 172.21.15.144:0/3500783768 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faad80cba70 con 0x5596ee92d1b0 2024-07-26T11:03:32.704 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.701+0000 7faaf53e7740 10 client.15480 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:32.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.701+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5596eed15610 con 0x5596ee92d1b0 2024-07-26T11:03:32.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.702+0000 7faad1600640 1 -- 172.21.15.144:0/3500783768 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad4012aa0 con 0x5596ee92d1b0 2024-07-26T11:03:32.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.702+0000 7faad1600640 1 -- 172.21.15.144:0/3500783768 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faad80d3f90 con 0x5596ee92d1b0 2024-07-26T11:03:32.705 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.702+0000 7faaf53e7740 10 client.15480 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:32.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.702+0000 7faaf53e7740 10 client.15480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:32.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.703+0000 7faaf53e7740 20 client.15480 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:32.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.703+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596ee9149f0 0x5596eed18b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.703+0000 7faaf53e7740 4 client.15480 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:32.706 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.703+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596eec73b20 con 0x5596ee9149f0 2024-07-26T11:03:32.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.703+0000 7faae6a00640 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596ee9149f0 0x5596eed18b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.704+0000 7faae6a00640 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596ee9149f0 0x5596eed18b20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdbf32ad5036d230 server_cookie=189ba7f2c3f04023 in_seq=0 out_seq=0 2024-07-26T11:03:32.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.704+0000 7faad1600640 10 client.15480 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.707 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.705+0000 7faad1600640 1 -- 172.21.15.144:0/3500783768 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5596eec73b20 con 0x5596ee9149f0 2024-07-26T11:03:32.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.705+0000 7faad1600640 10 client.15480 handle_command_reply: tid=0 2024-07-26T11:03:32.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.705+0000 7faaf53e7740 1 client.15480 shutdown 2024-07-26T11:03:32.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee92d1b0 msgr2=0x5596eed06720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.708 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee92d1b0 0x5596eed06720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faad8101eb0 tx=0x7faad80c78b0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 shutdown_connections 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596ee9149f0 0x5596eed18b20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee956070 0x5596eed071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed06c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3500783768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee92d1b0 0x5596eed06720 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 >> 172.21.15.144:0/3500783768 conn(0x5596ee94a460 msgr2=0x5596eecfe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faad1600640 0 client.15480 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 shutdown_connections 2024-07-26T11:03:32.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.706+0000 7faaf53e7740 1 -- 172.21.15.144:0/3500783768 wait complete. 2024-07-26T11:03:32.710 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.707+0000 7faaf53e7740 20 client.15480 trim_cache size 0 max 16384 2024-07-26T11:03:32.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.733+0000 7faaf53e7740 1 Processor -- start 2024-07-26T11:03:32.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 -- start start 2024-07-26T11:03:32.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed08600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee956070 0x5596eecf7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eecfc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7faae81b9510 con 0x5596ee956070 2024-07-26T11:03:32.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faae81b6de0 con 0x5596ee935390 2024-07-26T11:03:32.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.734+0000 7faaf53e7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faae81b4c30 con 0x5596ee8e9da0 2024-07-26T11:03:32.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed08600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed08600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49442/0 (socket says 172.21.15.144:49442) 2024-07-26T11:03:32.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 -- 172.21.15.144:0/3051897503 learned_addr learned my addr 172.21.15.144:0/3051897503 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7e00640 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eecfc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae6a00640 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee956070 0x5596eecf7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 -- 172.21.15.144:0/3051897503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 msgr2=0x5596eecfc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eecfc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 -- 172.21.15.144:0/3051897503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee956070 msgr2=0x5596eecf7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.735+0000 7faae7400640 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee956070 0x5596eecf7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.736+0000 7faae7400640 1 -- 172.21.15.144:0/3051897503 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596eecfd4f0 con 0x5596ee935390 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.736+0000 7faae6a00640 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee956070 0x5596eecf7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.736+0000 7faae7400640 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed08600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5596eed08f70 tx=0x7faad803b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4959aa7416637f0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.736+0000 7faad3e00640 1 -- 172.21.15.144:0/3051897503 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad80cae80 con 0x5596ee935390 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.736+0000 7faad3e00640 1 -- 172.21.15.144:0/3051897503 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad80cf970 con 0x5596ee935390 2024-07-26T11:03:32.741 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.736+0000 7faad3e00640 1 -- 172.21.15.144:0/3051897503 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad80ca5d0 con 0x5596ee935390 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 -- 172.21.15.144:0/3051897503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 msgr2=0x5596eed08600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed08600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5596eed08f70 tx=0x7faad803b890 comp rx=0 tx=0).stop 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 -- 172.21.15.144:0/3051897503 shutdown_connections 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eecfc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee956070 0x5596eecf7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 --2- 172.21.15.144:0/3051897503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee935390 0x5596eed08600 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 -- 172.21.15.144:0/3051897503 >> 172.21.15.144:0/3051897503 conn(0x5596ee94a460 msgr2=0x5596eecfc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 -- 172.21.15.144:0/3051897503 shutdown_connections 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.738+0000 7faaf53e7740 1 -- 172.21.15.144:0/3051897503 wait complete. 2024-07-26T11:03:32.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.739+0000 7faaf53e7740 1 -- start start 2024-07-26T11:03:32.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.739+0000 7faaf53e7740 1 Processor -- start 2024-07-26T11:03:32.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eed0f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eed0c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee956070 0x5596eed0cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7faae81b52a0 con 0x5596ee935390 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faae81b6710 con 0x5596ee956070 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faae81b5970 con 0x5596ee8e9da0 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faaf53e7740 10 client.0 _ll_register_callbacks cb 0x7faad8003ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faae7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eed0f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faae7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eed0f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39366/0 (socket says 172.21.15.144:39366) 2024-07-26T11:03:32.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.740+0000 7faae7400640 1 -- 172.21.15.144:0/2229326606 learned_addr learned my addr 172.21.15.144:0/2229326606 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.741+0000 7faae7e00640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee956070 0x5596eed0cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.741+0000 7faae6a00640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eed0c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.741+0000 7faae7e00640 1 -- 172.21.15.144:0/2229326606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 msgr2=0x5596eed0f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.741+0000 7faae7e00640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eed0f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.741+0000 7faae7e00640 1 -- 172.21.15.144:0/2229326606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 msgr2=0x5596eed0c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.742+0000 7faae7e00640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eed0c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.742+0000 7faae7e00640 1 -- 172.21.15.144:0/2229326606 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596eed0d530 con 0x5596ee956070 2024-07-26T11:03:32.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.742+0000 7faae6a00640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eed0c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:32.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.742+0000 7faae7e00640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee956070 0x5596eed0cdc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faae0009e10 tx=0x7faae0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc87d2e840336368 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:32.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.742+0000 7faad1600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:32.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.743+0000 7faad1600640 1 -- 172.21.15.144:0/2229326606 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae002d060 con 0x5596ee956070 2024-07-26T11:03:32.746 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.743+0000 7faad1600640 1 -- 172.21.15.144:0/2229326606 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7faae0020980 con 0x5596ee956070 2024-07-26T11:03:32.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.743+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faad80cf970 con 0x5596ee956070 2024-07-26T11:03:32.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.743+0000 7faad1600640 1 -- 172.21.15.144:0/2229326606 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faae0034050 con 0x5596ee956070 2024-07-26T11:03:32.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.743+0000 7faaf53e7740 10 client.25625 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:32.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.743+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5596eed811e0 con 0x5596ee956070 2024-07-26T11:03:32.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faad1600640 1 -- 172.21.15.144:0/2229326606 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad80cf970 con 0x5596ee956070 2024-07-26T11:03:32.747 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faad1600640 1 -- 172.21.15.144:0/2229326606 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faae0027030 con 0x5596ee956070 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faaf53e7740 10 client.25625 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faaf53e7740 10 client.25625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faaf53e7740 20 client.25625 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faaf53e7740 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596eed81990 0x5596eed83db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faaf53e7740 4 client.25625 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.744+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596eec6f5c0 con 0x5596eed81990 2024-07-26T11:03:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.745+0000 7faae7400640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596eed81990 0x5596eed83db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:32.749 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faae7400640 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596eed81990 0x5596eed83db0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ddab7baed3f2383 server_cookie=627185200bf18879 in_seq=0 out_seq=0 2024-07-26T11:03:32.749 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faad1600640 10 client.25625 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.749 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faad1600640 1 -- 172.21.15.144:0/2229326606 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596eec6f5c0 con 0x5596eed81990 2024-07-26T11:03:32.749 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faad1600640 10 client.25625 handle_command_reply: tid=0 2024-07-26T11:03:32.749 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faaf53e7740 1 client.25625 shutdown 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee956070 msgr2=0x5596eed0cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.746+0000 7faaf53e7740 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee956070 0x5596eed0cdc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faae0009e10 tx=0x7faae0004a40 comp rx=0 tx=0).stop 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 shutdown_connections 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faaf53e7740 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5596eed81990 0x5596eed83db0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faaf53e7740 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5596ee956070 0x5596eed0cdc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faaf53e7740 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5596ee935390 0x5596eed0c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.750 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faaf53e7740 1 --2- 172.21.15.144:0/2229326606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596ee8e9da0 0x5596eed0f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 >> 172.21.15.144:0/2229326606 conn(0x5596ee94a460 msgr2=0x5596eecfcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.747+0000 7faad1600640 0 client.25625 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:32.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.748+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 shutdown_connections 2024-07-26T11:03:32.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.748+0000 7faaf53e7740 1 -- 172.21.15.144:0/2229326606 wait complete. 2024-07-26T11:03:32.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.748+0000 7faaf53e7740 20 client.25625 trim_cache size 0 max 16384 2024-07-26T11:03:32.752 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:03:32.752 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faacc0b3d00 msgr2=0x7faacc0b6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.752 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faacc0b3d00 0x7faacc0b6210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faae0005fd0 tx=0x7faae00098a0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.752 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 msgr2=0x7faae810e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:32.752 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae810e2c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faad8005ab0 tx=0x7faad8003c80 comp rx=0 tx=0).stop 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 shutdown_connections 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faacc0b3d00 0x7faacc0b6210 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faae810f940 0x7faae81a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faae8075b30 0x7faae810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 --2- 172.21.15.144:0/3301661960 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faae80751e0 0x7faae810e2c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.749+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 >> 172.21.15.144:0/3301661960 conn(0x7faae80fee70 msgr2=0x7faae8100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.750+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 shutdown_connections 2024-07-26T11:03:32.753 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:32.750+0000 7faaed000640 1 -- 172.21.15.144:0/3301661960 wait complete. 2024-07-26T11:03:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:03:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:33 smithi144 ceph-mon[32129]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-07-26T11:03:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:03:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:33 smithi160 ceph-mon[36715]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-07-26T11:03:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:03:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:33 smithi183 ceph-mon[37273]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-07-26T11:03:34.628 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:03:34.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:03:35.281 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:35 smithi160 ceph-mon[36715]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:03:35.596 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:35 smithi144 ceph-mon[32129]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:03:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:35 smithi183 ceph-mon[37273]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:03:37.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 -- 172.21.15.144:0/328266065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b34108d20 msgr2=0x7f5b34109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 --2- 172.21.15.144:0/328266065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b34108d20 0x7f5b34109130 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009900 tx=0x7f5b2002a410 comp rx=0 tx=0).stop 2024-07-26T11:03:37.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 -- 172.21.15.144:0/328266065 shutdown_connections 2024-07-26T11:03:37.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 --2- 172.21.15.144:0/328266065 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b34109670 0x7f5b3410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 --2- 172.21.15.144:0/328266065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b34108d20 0x7f5b34109130 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 --2- 172.21.15.144:0/328266065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b341033e0 0x7f5b341037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.474+0000 7f5b33e00640 1 -- 172.21.15.144:0/328266065 >> 172.21.15.144:0/328266065 conn(0x7f5b340fee70 msgr2=0x7f5b341012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:37.478 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:37 smithi144 ceph-mon[32129]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-07-26T11:03:37.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.476+0000 7f5b33e00640 1 -- 172.21.15.144:0/328266065 shutdown_connections 2024-07-26T11:03:37.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.476+0000 7f5b33e00640 1 -- 172.21.15.144:0/328266065 wait complete. 2024-07-26T11:03:37.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.477+0000 7f5b33e00640 1 Processor -- start 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 -- start start 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 0x7f5b34199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b34108d20 0x7f5b3419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b34109670 0x7f5b3419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5b34113300 con 0x7f5b34109670 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5b34113180 con 0x7f5b34108d20 2024-07-26T11:03:37.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.478+0000 7f5b33e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b34113480 con 0x7f5b341033e0 2024-07-26T11:03:37.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.479+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 0x7f5b34199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.479+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 0x7f5b34199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39382/0 (socket says 172.21.15.144:39382) 2024-07-26T11:03:37.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.479+0000 7f5b32a00640 1 -- 172.21.15.144:0/497241699 learned_addr learned my addr 172.21.15.144:0/497241699 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:37.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.479+0000 7f5b32000640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b34108d20 0x7f5b3419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.479+0000 7f5b33400640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b34109670 0x7f5b3419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b32a00640 1 -- 172.21.15.144:0/497241699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b34108d20 msgr2=0x7f5b3419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b32a00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b34108d20 0x7f5b3419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b32a00640 1 -- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b34109670 msgr2=0x7f5b3419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b32a00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b34109670 0x7f5b3419f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b32a00640 1 -- 172.21.15.144:0/497241699 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3419fcc0 con 0x7f5b341033e0 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b33400640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b34109670 0x7f5b3419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b32a00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 0x7f5b34199d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e9f0 tx=0x7f5b2400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:37.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b2b400640 1 -- 172.21.15.144:0/497241699 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2400cd80 con 0x7f5b341033e0 2024-07-26T11:03:37.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b2b400640 1 -- 172.21.15.144:0/497241699 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b240108e0 con 0x7f5b341033e0 2024-07-26T11:03:37.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.480+0000 7f5b2b400640 1 -- 172.21.15.144:0/497241699 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24018860 con 0x7f5b341033e0 2024-07-26T11:03:37.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.481+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3419ffb0 con 0x7f5b341033e0 2024-07-26T11:03:37.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.481+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b341a6710 con 0x7f5b341033e0 2024-07-26T11:03:37.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.486+0000 7f5b405f4740 1 Processor -- start 2024-07-26T11:03:37.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.486+0000 7f5b405f4740 1 -- start start 2024-07-26T11:03:37.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b2b400640 1 -- 172.21.15.144:0/497241699 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5b24010430 con 0x7f5b341033e0 2024-07-26T11:03:37.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222102f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222103490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x5572221039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b405f4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x557222079be0 con 0x557221d3d670 2024-07-26T11:03:37.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b405f4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5572220690e0 con 0x557221d5e350 2024-07-26T11:03:37.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.487+0000 7f5b405f4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55722206d120 con 0x557221d35490 2024-07-26T11:03:37.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.488+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222103490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.488+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222103490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39390/0 (socket says 172.21.15.144:39390) 2024-07-26T11:03:37.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.488+0000 7f5b32a00640 1 -- 172.21.15.144:0/559482634 learned_addr learned my addr 172.21.15.144:0/559482634 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:37.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.488+0000 7f5b32000640 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x5572221039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.488+0000 7f5b33400640 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222102f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b32a00640 1 -- 172.21.15.144:0/559482634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 msgr2=0x557222102f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b32a00640 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222102f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b32a00640 1 -- 172.21.15.144:0/559482634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 msgr2=0x5572221039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b32a00640 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x5572221039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b32a00640 1 -- 172.21.15.144:0/559482634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557221dc9930 con 0x557221d35490 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b32000640 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x5572221039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.489+0000 7f5b2b400640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b180b3dc0 0x7f5b180b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.490+0000 7f5b33400640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b180b3dc0 0x7f5b180b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.490+0000 7f5b2b400640 1 -- 172.21.15.144:0/497241699 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f5b240e4650 con 0x7f5b341033e0 2024-07-26T11:03:37.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.490+0000 7f5b32a00640 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222103490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5b240e1910 tx=0x7f5b240eacb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324c84ee3224edb0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:37.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.490+0000 7f5b1fe00640 1 -- 172.21.15.144:0/559482634 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240a6e00 con 0x557221d35490 2024-07-26T11:03:37.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.490+0000 7f5b1fe00640 1 -- 172.21.15.144:0/559482634 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b24010720 con 0x557221d35490 2024-07-26T11:03:37.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.490+0000 7f5b1fe00640 1 -- 172.21.15.144:0/559482634 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240aa6b0 con 0x557221d35490 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 -- 172.21.15.144:0/559482634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 msgr2=0x557222103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222103490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5b240e1910 tx=0x7f5b240eacb0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b33400640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b180b3dc0 0x7f5b180b62d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c009fd0 tx=0x7f5b2c009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 -- 172.21.15.144:0/559482634 shutdown_connections 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x5572221039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222103490 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 --2- 172.21.15.144:0/559482634 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 -- 172.21.15.144:0/559482634 >> 172.21.15.144:0/559482634 conn(0x557221d52740 msgr2=0x557222100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:37.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.492+0000 7f5b405f4740 1 -- 172.21.15.144:0/559482634 shutdown_connections 2024-07-26T11:03:37.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.493+0000 7f5b405f4740 1 -- 172.21.15.144:0/559482634 wait complete. 2024-07-26T11:03:37.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.493+0000 7f5b405f4740 1 -- start start 2024-07-26T11:03:37.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.494+0000 7f5b405f4740 1 Processor -- start 2024-07-26T11:03:37.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.494+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d35490 0x5572221151b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.494+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x557222112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 0x5572221131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b405f4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x557221ecc890 con 0x557221d3d670 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b405f4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557222005320 con 0x557221d35490 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b405f4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557222115ae0 con 0x557221d5e350 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b405f4740 10 client.0 _ll_register_callbacks cb 0x557222031000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d35490 0x5572221151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b33400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 0x5572221131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b33400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 0x5572221131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39392/0 (socket says 172.21.15.144:39392) 2024-07-26T11:03:37.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d35490 0x5572221151b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49488/0 (socket says 172.21.15.144:49488) 2024-07-26T11:03:37.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b33400640 1 -- 172.21.15.144:0/2513609281 learned_addr learned my addr 172.21.15.144:0/2513609281 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:37.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.495+0000 7f5b32000640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x557222112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.496+0000 7f5b33400640 1 -- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d35490 msgr2=0x5572221151b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.496+0000 7f5b33400640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d35490 0x5572221151b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.496+0000 7f5b33400640 1 -- 172.21.15.144:0/2513609281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 msgr2=0x557222112ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.496+0000 7f5b33400640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x557222112ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.496+0000 7f5b33400640 1 -- 172.21.15.144:0/2513609281 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722210c2b0 con 0x557221d5e350 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.496+0000 7f5b32000640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x557222112ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.497+0000 7f5b33400640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 0x5572221131c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c003e20 tx=0x7f5b2c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c68364f71ea8f083 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:37.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.497+0000 7f5b1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.497+0000 7f5b1cc00640 1 -- 172.21.15.144:0/2513609281 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c004550 con 0x557221d5e350 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.497+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55722207bb30 con 0x557221d5e350 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.497+0000 7f5b1cc00640 1 -- 172.21.15.144:0/2513609281 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f5b2c018030 con 0x557221d5e350 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.498+0000 7f5b1cc00640 1 -- 172.21.15.144:0/2513609281 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5b2c0046f0 con 0x557221d5e350 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.498+0000 7f5b405f4740 10 client.25618 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.498+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557222125f20 con 0x557221d5e350 2024-07-26T11:03:37.501 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.498+0000 7f5b1cc00640 1 -- 172.21.15.144:0/2513609281 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55722207bb30 con 0x557221d5e350 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b1cc00640 1 -- 172.21.15.144:0/2513609281 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5b2c028420 con 0x557221d5e350 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b405f4740 10 client.25618 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b405f4740 10 client.25618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b405f4740 20 client.25618 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b405f4740 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221d1ccd0 0x557222129340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b405f4740 4 client.25618 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:37.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572220775d0 con 0x557221d1ccd0 2024-07-26T11:03:37.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.499+0000 7f5b32a00640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221d1ccd0 0x557222129340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.500+0000 7f5b32a00640 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221d1ccd0 0x557222129340 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198cf5b2847c498 server_cookie=9db8ef4fb9ed8a74 in_seq=0 out_seq=0 2024-07-26T11:03:37.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.500+0000 7f5b1cc00640 10 client.25618 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:37.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.501+0000 7f5b1cc00640 1 -- 172.21.15.144:0/2513609281 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f5b2408be80 con 0x557221d1ccd0 2024-07-26T11:03:37.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.501+0000 7f5b1cc00640 10 client.25618 handle_command_reply: tid=0 2024-07-26T11:03:37.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.501+0000 7f5b405f4740 1 client.25618 shutdown 2024-07-26T11:03:37.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 msgr2=0x5572221131c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 0x5572221131c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c003e20 tx=0x7f5b2c004ab0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 shutdown_connections 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221d1ccd0 0x557222129340 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d5e350 0x5572221131c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d3d670 0x557222112ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 --2- 172.21.15.144:0/2513609281 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d35490 0x5572221151b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b1cc00640 0 client.25618 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:37.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.502+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 >> 172.21.15.144:0/2513609281 conn(0x557221d52740 msgr2=0x557222104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:37.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.503+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 shutdown_connections 2024-07-26T11:03:37.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.503+0000 7f5b405f4740 1 -- 172.21.15.144:0/2513609281 wait complete. 2024-07-26T11:03:37.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.503+0000 7f5b405f4740 20 client.25618 trim_cache size 0 max 16384 2024-07-26T11:03:37.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.538+0000 7f5b405f4740 1 Processor -- start 2024-07-26T11:03:37.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.538+0000 7f5b405f4740 1 -- start start 2024-07-26T11:03:37.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222104690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d3d670 0x55722212ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d5e350 0x55722212efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b405f4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341b0d60 con 0x557221d5e350 2024-07-26T11:03:37.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b405f4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341ae630 con 0x557221d3d670 2024-07-26T11:03:37.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b405f4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341ac480 con 0x557221d35490 2024-07-26T11:03:37.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.543 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222104690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39402/0 (socket says 172.21.15.144:39402) 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.539+0000 7f5b32a00640 1 -- 172.21.15.144:0/185494463 learned_addr learned my addr 172.21.15.144:0/185494463 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32a00640 1 -- 172.21.15.144:0/185494463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d3d670 msgr2=0x55722212ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32000640 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d3d670 0x55722212ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32a00640 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d3d670 0x55722212ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32a00640 1 -- 172.21.15.144:0/185494463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d5e350 msgr2=0x55722212efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32a00640 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d5e350 0x55722212efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32a00640 1 -- 172.21.15.144:0/185494463 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572221296b0 con 0x557221d35490 2024-07-26T11:03:37.544 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b32a00640 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222104690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5b240ec670 tx=0x7f5b24010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e746dbaa1cf2c6b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b1f400640 1 -- 172.21.15.144:0/185494463 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240aa890 con 0x557221d35490 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.540+0000 7f5b1f400640 1 -- 172.21.15.144:0/185494463 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5572220775d0 con 0x557221d35490 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.541+0000 7f5b1f400640 1 -- 172.21.15.144:0/185494463 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240aad10 con 0x557221d35490 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 -- 172.21.15.144:0/185494463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 msgr2=0x557222104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222104690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5b240ec670 tx=0x7f5b24010ca0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 -- 172.21.15.144:0/185494463 shutdown_connections 2024-07-26T11:03:37.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d5e350 0x55722212efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d3d670 0x55722212ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 --2- 172.21.15.144:0/185494463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d35490 0x557222104690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 -- 172.21.15.144:0/185494463 >> 172.21.15.144:0/185494463 conn(0x557221d52740 msgr2=0x557222103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:37.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 -- 172.21.15.144:0/185494463 shutdown_connections 2024-07-26T11:03:37.546 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.542+0000 7f5b405f4740 1 -- 172.21.15.144:0/185494463 wait complete. 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.543+0000 7f5b405f4740 1 -- start start 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.543+0000 7f5b405f4740 1 Processor -- start 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 0x55722211fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d3d670 0x557222120310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222190a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341acaf0 con 0x557221d35490 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341adf60 con 0x557221d5e350 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341ad1c0 con 0x557221d3d670 2024-07-26T11:03:37.547 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b405f4740 10 client.0 _ll_register_callbacks cb 0x7f5b2c012070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:37.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 0x55722211fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b32a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 0x55722211fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39212/0 (socket says 172.21.15.144:39212) 2024-07-26T11:03:37.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b32a00640 1 -- 172.21.15.144:0/1556720242 learned_addr learned my addr 172.21.15.144:0/1556720242 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:37.548 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.544+0000 7f5b32000640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d3d670 0x557222120310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.545+0000 7f5b33400640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222190a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.545+0000 7f5b32a00640 1 -- 172.21.15.144:0/1556720242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d3d670 msgr2=0x557222120310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.545+0000 7f5b32a00640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d3d670 0x557222120310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.545+0000 7f5b32a00640 1 -- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 msgr2=0x557222190a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:37.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.545+0000 7f5b32a00640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222190a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.549 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.545+0000 7f5b32a00640 1 -- 172.21.15.144:0/1556720242 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557222108f70 con 0x557221d35490 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.546+0000 7f5b32a00640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 0x55722211fdd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b240a81b0 tx=0x7f5b24009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207ca3152b7b87b1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.546+0000 7f5b1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.546+0000 7f5b1cc00640 1 -- 172.21.15.144:0/1556720242 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240c4070 con 0x557221d35490 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.546+0000 7f5b1cc00640 1 -- 172.21.15.144:0/1556720242 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f5b240ea380 con 0x557221d35490 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.546+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5572220775d0 con 0x557221d35490 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.546+0000 7f5b1cc00640 1 -- 172.21.15.144:0/1556720242 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5b240aa890 con 0x557221d35490 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 10 client.15498 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:37.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5572221920c0 con 0x557221d35490 2024-07-26T11:03:37.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b1cc00640 1 -- 172.21.15.144:0/1556720242 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5572220775d0 con 0x557221d35490 2024-07-26T11:03:37.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b1cc00640 1 -- 172.21.15.144:0/1556720242 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5b24089080 con 0x557221d35490 2024-07-26T11:03:37.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 10 client.15498 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:37.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 20 client.15498 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221cf2080 0x5572221942a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 4 client.15498 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.547+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557222074d10 con 0x557221cf2080 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.548+0000 7f5b32000640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221cf2080 0x5572221942a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.548+0000 7f5b32000640 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221cf2080 0x5572221942a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a89ade55fca393 server_cookie=51d743e5cff9ad08 in_seq=0 out_seq=0 2024-07-26T11:03:37.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.548+0000 7f5b1cc00640 10 client.15498 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:37.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.549+0000 7f5b1cc00640 1 -- 172.21.15.144:0/1556720242 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557222074d10 con 0x557221cf2080 2024-07-26T11:03:37.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.549+0000 7f5b1cc00640 10 client.15498 handle_command_reply: tid=0 2024-07-26T11:03:37.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.549+0000 7f5b405f4740 1 client.15498 shutdown 2024-07-26T11:03:37.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 msgr2=0x55722211fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 0x55722211fdd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b240a81b0 tx=0x7f5b24009700 comp rx=0 tx=0).stop 2024-07-26T11:03:37.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 shutdown_connections 2024-07-26T11:03:37.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x557221cf2080 0x5572221942a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557221d5e350 0x557222190a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557221d3d670 0x557222120310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 --2- 172.21.15.144:0/1556720242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x557221d35490 0x55722211fdd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.550+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 >> 172.21.15.144:0/1556720242 conn(0x557221d52740 msgr2=0x55722210de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:37.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.551+0000 7f5b1cc00640 0 client.15498 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:37.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.551+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 shutdown_connections 2024-07-26T11:03:37.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.551+0000 7f5b405f4740 1 -- 172.21.15.144:0/1556720242 wait complete. 2024-07-26T11:03:37.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.551+0000 7f5b405f4740 20 client.15498 trim_cache size 0 max 16384 2024-07-26T11:03:37.555 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:03:37.555 INFO:teuthology.orchestra.run.smithi144.stdout: "return_code": 0, 2024-07-26T11:03:37.556 INFO:teuthology.orchestra.run.smithi144.stdout: "scrub_tag": "3222b8ce-c1b4-41a4-83c6-6cbeee6cf3dc", 2024-07-26T11:03:37.556 INFO:teuthology.orchestra.run.smithi144.stdout: "mode": "asynchronous" 2024-07-26T11:03:37.556 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:03:37.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b180b3dc0 msgr2=0x7f5b180b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b180b3dc0 0x7f5b180b62d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c009fd0 tx=0x7f5b2c009500 comp rx=0 tx=0).stop 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 msgr2=0x7f5b34199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 0x7f5b34199d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e9f0 tx=0x7f5b2400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 shutdown_connections 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5b180b3dc0 0x7f5b180b62d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b34109670 0x7f5b3419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b34108d20 0x7f5b3419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 --2- 172.21.15.144:0/497241699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b341033e0 0x7f5b34199d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 >> 172.21.15.144:0/497241699 conn(0x7f5b340fee70 msgr2=0x7f5b34101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 shutdown_connections 2024-07-26T11:03:37.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:37.553+0000 7f5b33e00640 1 -- 172.21.15.144:0/497241699 wait complete. 2024-07-26T11:03:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:37 smithi160 ceph-mon[36715]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-07-26T11:03:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:37 smithi183 ceph-mon[37273]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-07-26T11:03:38.480 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:03:38.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:38 smithi144 ceph-mon[32129]: scrub queued for path: / 2024-07-26T11:03:38.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:38 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:03:38.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:38 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:03:38.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:38 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:03:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:38 smithi160 ceph-mon[36715]: scrub queued for path: / 2024-07-26T11:03:38.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:38 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:03:38.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:38 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:03:38.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:38 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:03:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:38 smithi183 ceph-mon[37273]: scrub queued for path: / 2024-07-26T11:03:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:38 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:03:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:38 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:03:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:38 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:03:38.957 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:39 smithi144 ceph-mon[32129]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:03:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:39 smithi160 ceph-mon[36715]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:03:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:39 smithi183 ceph-mon[37273]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-07-26T11:03:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:40 smithi144 ceph-mon[32129]: scrub summary: idle 2024-07-26T11:03:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:40 smithi160 ceph-mon[36715]: scrub summary: idle 2024-07-26T11:03:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:40 smithi183 ceph-mon[37273]: scrub summary: idle 2024-07-26T11:03:41.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 -- 172.21.15.144:0/2443882150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124108530 msgr2=0x7fc12410ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 --2- 172.21.15.144:0/2443882150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124108530 0x7fc12410ed60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009ca0 tx=0x7fc11c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 -- 172.21.15.144:0/2443882150 shutdown_connections 2024-07-26T11:03:41.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 --2- 172.21.15.144:0/2443882150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124108530 0x7fc12410ed60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 --2- 172.21.15.144:0/2443882150 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc124107be0 0x7fc124107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 --2- 172.21.15.144:0/2443882150 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc1241026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 -- 172.21.15.144:0/2443882150 >> 172.21.15.144:0/2443882150 conn(0x7fc1240fdcf0 msgr2=0x7fc124100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:41.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 -- 172.21.15.144:0/2443882150 shutdown_connections 2024-07-26T11:03:41.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.315+0000 7fc12a800640 1 -- 172.21.15.144:0/2443882150 wait complete. 2024-07-26T11:03:41.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.316+0000 7fc12a800640 1 Processor -- start 2024-07-26T11:03:41.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 -- start start 2024-07-26T11:03:41.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc12419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124107be0 0x7fc12419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc124108530 0x7fc1241a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc124112200 con 0x7fc124107be0 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc124112080 con 0x7fc1241022a0 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc12a800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc124112380 con 0x7fc124108530 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc129400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc12419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc128a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124107be0 0x7fc12419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc129400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc12419bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:37548/0 (socket says 172.21.15.144:37548) 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc129400640 1 -- 172.21.15.144:0/2342098406 learned_addr learned my addr 172.21.15.144:0/2342098406 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc128a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124107be0 0x7fc12419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60430/0 (socket says 172.21.15.144:60430) 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.317+0000 7fc129e00640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc124108530 0x7fc1241a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.318+0000 7fc129400640 1 -- 172.21.15.144:0/2342098406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc124108530 msgr2=0x7fc1241a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.318+0000 7fc129400640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc124108530 0x7fc1241a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.318+0000 7fc129400640 1 -- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124107be0 msgr2=0x7fc12419c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.318+0000 7fc129400640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124107be0 0x7fc12419c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.318+0000 7fc129400640 1 -- 172.21.15.144:0/2342098406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1241a1e50 con 0x7fc1241022a0 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.318+0000 7fc129400640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc12419bee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc11800e9f0 tx=0x7fc11800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.319+0000 7fc122000640 1 -- 172.21.15.144:0/2342098406 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11800cd80 con 0x7fc1241022a0 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.319+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1241a2140 con 0x7fc1241022a0 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.319+0000 7fc122000640 1 -- 172.21.15.144:0/2342098406 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1180108e0 con 0x7fc1241022a0 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.319+0000 7fc122000640 1 -- 172.21.15.144:0/2342098406 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118018980 con 0x7fc1241022a0 2024-07-26T11:03:41.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.319+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1241a88a0 con 0x7fc1241022a0 2024-07-26T11:03:41.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.322+0000 7fc132be7740 1 Processor -- start 2024-07-26T11:03:41.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.322+0000 7fc132be7740 1 -- start start 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc122000640 1 -- 172.21.15.144:0/2342098406 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fc118010430 con 0x7fc1241022a0 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc132be7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e938efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc132be7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e938f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc132be7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e938fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc132be7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5640e9305b90 con 0x5640e8fea070 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc132be7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5640e92f5090 con 0x5640e8fc11b0 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc132be7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5640e92f90d0 con 0x5640e8fc9390 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc128a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e938fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc128a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e938fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42006/0 (socket says 172.21.15.144:42006) 2024-07-26T11:03:41.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc128a00640 1 -- 172.21.15.144:0/3370749698 learned_addr learned my addr 172.21.15.144:0/3370749698 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:41.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc129e00640 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e938efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.323+0000 7fc129400640 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e938f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc128a00640 1 -- 172.21.15.144:0/3370749698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 msgr2=0x5640e938f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc128a00640 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e938f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc128a00640 1 -- 172.21.15.144:0/3370749698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 msgr2=0x5640e938efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc128a00640 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e938efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc128a00640 1 -- 172.21.15.144:0/3370749698 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e9055650 con 0x5640e8fc9390 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc128a00640 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e938fa50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc110009c20 tx=0x7fc110011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7156e61dbae19b2d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc116600640 1 -- 172.21.15.144:0/3370749698 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110004230 con 0x5640e8fc9390 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc116600640 1 -- 172.21.15.144:0/3370749698 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1100043d0 con 0x5640e8fc9390 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.324+0000 7fc116600640 1 -- 172.21.15.144:0/3370749698 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110019640 con 0x5640e8fc9390 2024-07-26T11:03:41.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.325+0000 7fc122000640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc10c0b3dc0 0x7fc10c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.325+0000 7fc122000640 1 -- 172.21.15.144:0/2342098406 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc1180e4670 con 0x7fc1241022a0 2024-07-26T11:03:41.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.325+0000 7fc129e00640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc10c0b3dc0 0x7fc10c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.325+0000 7fc132be7740 1 -- 172.21.15.144:0/3370749698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 msgr2=0x5640e938fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.325+0000 7fc132be7740 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e938fa50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc110009c20 tx=0x7fc110011810 comp rx=0 tx=0).stop 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 -- 172.21.15.144:0/3370749698 shutdown_connections 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e938fa50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e938f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 --2- 172.21.15.144:0/3370749698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e938efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 -- 172.21.15.144:0/3370749698 >> 172.21.15.144:0/3370749698 conn(0x5640e8fde460 msgr2=0x5640e938cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 -- 172.21.15.144:0/3370749698 shutdown_connections 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc132be7740 1 -- 172.21.15.144:0/3370749698 wait complete. 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.326+0000 7fc129e00640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc10c0b3dc0 0x7fc10c0b62d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02f500 tx=0x7fc11c02f310 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 -- start start 2024-07-26T11:03:41.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 Processor -- start 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e939a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e939aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e939af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5640e9158700 con 0x5640e8fea070 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5640e9291660 con 0x5640e8fc11b0 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.327+0000 7fc132be7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5640e939d7b0 con 0x5640e8fc9390 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc132be7740 10 client.0 _ll_register_callbacks cb 0x5640e92a1110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e939aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc129e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e939af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e939aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42016/0 (socket says 172.21.15.144:42016) 2024-07-26T11:03:41.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 -- 172.21.15.144:0/2758319067 learned_addr learned my addr 172.21.15.144:0/2758319067 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc129400640 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e939a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 -- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 msgr2=0x5640e939a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e939a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 -- 172.21.15.144:0/2758319067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 msgr2=0x5640e939af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e939af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.328+0000 7fc128a00640 1 -- 172.21.15.144:0/2758319067 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e92a8800 con 0x5640e8fc9390 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc128a00640 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e939aa30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc110005890 tx=0x7fc1100052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576f8a845a213816 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc10aa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc10aa00640 1 -- 172.21.15.144:0/2758319067 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110028020 con 0x5640e8fc9390 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc10aa00640 1 -- 172.21.15.144:0/2758319067 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc1100035f0 con 0x5640e8fc9390 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1100043d0 con 0x5640e8fc9390 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc10aa00640 1 -- 172.21.15.144:0/2758319067 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc110033050 con 0x5640e8fc9390 2024-07-26T11:03:41.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc132be7740 10 client.25636 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.329+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5640e93a9300 con 0x5640e8fc9390 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc10aa00640 1 -- 172.21.15.144:0/2758319067 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1100043d0 con 0x5640e8fc9390 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc10aa00640 1 -- 172.21.15.144:0/2758319067 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc110018070 con 0x5640e8fc9390 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc132be7740 10 client.25636 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc132be7740 10 client.25636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc132be7740 20 client.25636 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc132be7740 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8fa89f0 0x5640e93ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc132be7740 4 client.25636 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:41.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640e9307ae0 con 0x5640e8fa89f0 2024-07-26T11:03:41.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.330+0000 7fc129400640 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8fa89f0 0x5640e93ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.331+0000 7fc10aa00640 10 client.25636 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.331+0000 7fc129400640 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8fa89f0 0x5640e93ac710 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d708a1db96fee server_cookie=89c752d953055698 in_seq=0 out_seq=0 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc10aa00640 1 -- 172.21.15.144:0/2758319067 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5640e9307ae0 con 0x5640e8fa89f0 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc10aa00640 10 client.25636 handle_command_reply: tid=0 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 client.25636 shutdown 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 msgr2=0x5640e939aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e939aa30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc110005890 tx=0x7fc1100052f0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 shutdown_connections 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8fa89f0 0x5640e93ac710 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e939af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc9390 0x5640e939aa30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 --2- 172.21.15.144:0/2758319067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc11b0 0x5640e939a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc10aa00640 0 client.25636 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 >> 172.21.15.144:0/2758319067 conn(0x5640e8fde460 msgr2=0x5640e93921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 shutdown_connections 2024-07-26T11:03:41.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.332+0000 7fc132be7740 1 -- 172.21.15.144:0/2758319067 wait complete. 2024-07-26T11:03:41.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.333+0000 7fc132be7740 20 client.25636 trim_cache size 0 max 16384 2024-07-26T11:03:41.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.351+0000 7fc132be7740 1 Processor -- start 2024-07-26T11:03:41.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.351+0000 7fc132be7740 1 -- start start 2024-07-26T11:03:41.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc132be7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 0x5640e93909c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc132be7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e938b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc132be7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e93a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc132be7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc1241b5080 con 0x5640e8fea070 2024-07-26T11:03:41.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc132be7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc1241b2950 con 0x5640e8fc9390 2024-07-26T11:03:41.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc132be7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc1241b07a0 con 0x5640e8fc11b0 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc129400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 0x5640e93909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc129400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 0x5640e93909c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42022/0 (socket says 172.21.15.144:42022) 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc129400640 1 -- 172.21.15.144:0/3825066139 learned_addr learned my addr 172.21.15.144:0/3825066139 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc128a00640 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e938b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.352+0000 7fc129e00640 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e93a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc129400640 1 -- 172.21.15.144:0/3825066139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 msgr2=0x5640e938b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc129400640 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e938b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc129400640 1 -- 172.21.15.144:0/3825066139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 msgr2=0x5640e93a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc129400640 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e93a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc129400640 1 -- 172.21.15.144:0/3825066139 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e939d340 con 0x5640e8fc11b0 2024-07-26T11:03:41.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc129400640 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 0x5640e93909c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc1180aaef0 tx=0x7fc1180ea530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e7c31737d361f76 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:41.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc115c00640 1 -- 172.21.15.144:0/3825066139 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180aaa70 con 0x5640e8fc11b0 2024-07-26T11:03:41.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc115c00640 1 -- 172.21.15.144:0/3825066139 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1180aaf60 con 0x5640e8fc11b0 2024-07-26T11:03:41.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.353+0000 7fc115c00640 1 -- 172.21.15.144:0/3825066139 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180e1c70 con 0x5640e8fc11b0 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 -- 172.21.15.144:0/3825066139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 msgr2=0x5640e93909c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 0x5640e93909c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc1180aaef0 tx=0x7fc1180ea530 comp rx=0 tx=0).stop 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 -- 172.21.15.144:0/3825066139 shutdown_connections 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fea070 0x5640e93a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e938b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 --2- 172.21.15.144:0/3825066139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fc11b0 0x5640e93909c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 -- 172.21.15.144:0/3825066139 >> 172.21.15.144:0/3825066139 conn(0x5640e8fde460 msgr2=0x5640e93920d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 -- 172.21.15.144:0/3825066139 shutdown_connections 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.354+0000 7fc132be7740 1 -- 172.21.15.144:0/3825066139 wait complete. 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.355+0000 7fc132be7740 1 -- start start 2024-07-26T11:03:41.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.355+0000 7fc132be7740 1 Processor -- start 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.355+0000 7fc132be7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 0x5640e939de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc132be7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e939e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc132be7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fea070 0x5640e9412db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc132be7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc1241b0e10 con 0x5640e8fc11b0 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc132be7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc1241b2280 con 0x5640e8fc9390 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc132be7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc1241b14e0 con 0x5640e8fea070 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc132be7740 10 client.0 _ll_register_callbacks cb 0x7fc110009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 0x5640e939de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 0x5640e939de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60472/0 (socket says 172.21.15.144:60472) 2024-07-26T11:03:41.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 -- 172.21.15.144:0/2907661641 learned_addr learned my addr 172.21.15.144:0/2907661641 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129e00640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fea070 0x5640e9412db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc128a00640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e939e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 -- 172.21.15.144:0/2907661641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fea070 msgr2=0x5640e9412db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fea070 0x5640e9412db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 -- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 msgr2=0x5640e939e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e939e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129400640 1 -- 172.21.15.144:0/2907661641 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e94132f0 con 0x5640e8fc11b0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.356+0000 7fc129e00640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fea070 0x5640e9412db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc129400640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 0x5640e939de20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc1180aac60 tx=0x7fc118000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1a35a4c90ac56d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc10aa00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc10aa00640 1 -- 172.21.15.144:0/2907661641 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180aaa70 con 0x5640e8fc11b0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc10aa00640 1 -- 172.21.15.144:0/2907661641 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fc1180a71f0 con 0x5640e8fc11b0 2024-07-26T11:03:41.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1180aaf60 con 0x5640e8fc11b0 2024-07-26T11:03:41.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc10aa00640 1 -- 172.21.15.144:0/2907661641 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc1180af4b0 con 0x5640e8fc11b0 2024-07-26T11:03:41.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc132be7740 10 client.15516 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:41.362 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.357+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5640e94143d0 con 0x5640e8fc11b0 2024-07-26T11:03:41.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc10aa00640 1 -- 172.21.15.144:0/2907661641 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1180aaf60 con 0x5640e8fc11b0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc10aa00640 1 -- 172.21.15.144:0/2907661641 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc1180c0080 con 0x5640e8fc11b0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc132be7740 10 client.15516 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc132be7740 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc132be7740 20 client.15516 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc132be7740 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8f7dda0 0x5640e9416b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc132be7740 4 client.15516 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5640e9303580 con 0x5640e8f7dda0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.358+0000 7fc128a00640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8f7dda0 0x5640e9416b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc128a00640 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8f7dda0 0x5640e9416b10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b081cfad72faf1 server_cookie=5fee0c760b31a6c1 in_seq=0 out_seq=0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc10aa00640 10 client.15516 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc10aa00640 1 -- 172.21.15.144:0/2907661641 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5640e9303580 con 0x5640e8f7dda0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc10aa00640 10 client.15516 handle_command_reply: tid=0 2024-07-26T11:03:41.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 client.15516 shutdown 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 msgr2=0x5640e939de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 0x5640e939de20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc1180aac60 tx=0x7fc118000bc0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 shutdown_connections 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x5640e8f7dda0 0x5640e9416b10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640e8fea070 0x5640e9412db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5640e8fc9390 0x5640e939e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 --2- 172.21.15.144:0/2907661641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5640e8fc11b0 0x5640e939de20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 >> 172.21.15.144:0/2907661641 conn(0x5640e8fde460 msgr2=0x5640e9392a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.359+0000 7fc10aa00640 0 client.15516 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.360+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 shutdown_connections 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.360+0000 7fc132be7740 1 -- 172.21.15.144:0/2907661641 wait complete. 2024-07-26T11:03:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.360+0000 7fc132be7740 20 client.15516 trim_cache size 0 max 16384 2024-07-26T11:03:41.366 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:03:41.366 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "no active scrubs running", 2024-07-26T11:03:41.366 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": {} 2024-07-26T11:03:41.366 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:03:41.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc10c0b3dc0 msgr2=0x7fc10c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc10c0b3dc0 0x7fc10c0b62d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc11c02f500 tx=0x7fc11c02f310 comp rx=0 tx=0).stop 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 msgr2=0x7fc12419bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc12419bee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc11800e9f0 tx=0x7fc11800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 shutdown_connections 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc10c0b3dc0 0x7fc10c0b62d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc124108530 0x7fc1241a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc124107be0 0x7fc12419c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 --2- 172.21.15.144:0/2342098406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1241022a0 0x7fc12419bee0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 >> 172.21.15.144:0/2342098406 conn(0x7fc1240fdcf0 msgr2=0x7fc1241071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 shutdown_connections 2024-07-26T11:03:41.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:41.361+0000 7fc12a800640 1 -- 172.21.15.144:0/2342098406 wait complete. 2024-07-26T11:03:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:41 smithi144 ceph-mon[32129]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:03:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:41 smithi160 ceph-mon[36715]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:03:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:41 smithi183 ceph-mon[37273]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:03:42.581 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:03:42.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:03:43.028 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:43 smithi144 ceph-mon[32129]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:03:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:43 smithi160 ceph-mon[36715]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:03:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:43 smithi183 ceph-mon[37273]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:03:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:45 smithi160 ceph-mon[36715]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-26T11:03:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:45 smithi183 ceph-mon[37273]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-26T11:03:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:45 smithi144 ceph-mon[32129]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-26T11:03:45.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 -- 172.21.15.144:0/2689669869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 msgr2=0x7f08d4108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 --2- 172.21.15.144:0/2689669869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 0x7f08d4108120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009900 tx=0x7f08c002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 -- 172.21.15.144:0/2689669869 shutdown_connections 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 --2- 172.21.15.144:0/2689669869 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 0x7f08d410ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 --2- 172.21.15.144:0/2689669869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 0x7f08d4108120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 --2- 172.21.15.144:0/2689669869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d41027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.866+0000 7f08db800640 1 -- 172.21.15.144:0/2689669869 >> 172.21.15.144:0/2689669869 conn(0x7f08d407a170 msgr2=0x7f08d407a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.867+0000 7f08db800640 1 -- 172.21.15.144:0/2689669869 shutdown_connections 2024-07-26T11:03:45.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.867+0000 7f08db800640 1 -- 172.21.15.144:0/2689669869 wait complete. 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 Processor -- start 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 -- start start 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d4199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 0x7f08d419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 0x7f08d419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f08d4112340 con 0x7f08d41023e0 2024-07-26T11:03:45.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41121c0 con 0x7f08d4108660 2024-07-26T11:03:45.871 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.868+0000 7f08db800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41124c0 con 0x7f08d4107d10 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d4199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d4199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60500/0 (socket says 172.21.15.144:60500) 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d8e00640 1 -- 172.21.15.144:0/4169546799 learned_addr learned my addr 172.21.15.144:0/4169546799 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 0x7f08d419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d3e00640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 0x7f08d419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 -- 172.21.15.144:0/4169546799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 msgr2=0x7f08d419a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 0x7f08d419a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 -- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 msgr2=0x7f08d4199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d4199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 -- 172.21.15.144:0/4169546799 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d419fe00 con 0x7f08d4108660 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d8e00640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d4199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.869+0000 7f08d9800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 0x7f08d419f700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009fd0 tx=0x7f08cc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.870+0000 7f08d1600640 1 -- 172.21.15.144:0/4169546799 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00ce20 con 0x7f08d4108660 2024-07-26T11:03:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.870+0000 7f08d1600640 1 -- 172.21.15.144:0/4169546799 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08cc005020 con 0x7f08d4108660 2024-07-26T11:03:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.870+0000 7f08d1600640 1 -- 172.21.15.144:0/4169546799 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc010690 con 0x7f08d4108660 2024-07-26T11:03:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.870+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d41a00f0 con 0x7f08d4108660 2024-07-26T11:03:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.870+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d41a6880 con 0x7f08d4108660 2024-07-26T11:03:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.873+0000 7f08e2781740 1 Processor -- start 2024-07-26T11:03:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08d1600640 1 -- 172.21.15.144:0/4169546799 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f08cc0054d0 con 0x7f08d4108660 2024-07-26T11:03:45.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 -- start start 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b24338ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24338efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b24338f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b243303270 con 0x55b242feb1b0 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b2432f2770 con 0x55b242fc22f0 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.874+0000 7f08e2781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b2432f67b0 con 0x55b242fca4d0 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b24338ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24338efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24338efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:37626/0 (socket says 172.21.15.144:37626) 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 -- 172.21.15.144:0/208689607 learned_addr learned my addr 172.21.15.144:0/208689607 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d3e00640 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b24338f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 -- 172.21.15.144:0/208689607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 msgr2=0x55b24338f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b24338f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 -- 172.21.15.144:0/208689607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 msgr2=0x55b24338ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b24338ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d8e00640 1 -- 172.21.15.144:0/208689607 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b243056790 con 0x55b242fc22f0 2024-07-26T11:03:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.875+0000 7f08d3e00640 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b24338ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08d8e00640 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24338efa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08c8011650 tx=0x7f08c8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebefd946ef87e7ac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08d1600640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f08bc0b3d40 0x7f08bc0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08c6000640 1 -- 172.21.15.144:0/208689607 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c801e070 con 0x55b242fc22f0 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08c6000640 1 -- 172.21.15.144:0/208689607 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c800fb70 con 0x55b242fc22f0 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08c6000640 1 -- 172.21.15.144:0/208689607 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8019c90 con 0x55b242fc22f0 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08d1600640 1 -- 172.21.15.144:0/4169546799 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f08cc0e4a30 con 0x7f08d4108660 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.876+0000 7f08d3e00640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f08bc0b3d40 0x7f08bc0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 -- 172.21.15.144:0/208689607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 msgr2=0x55b24338efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24338efa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08c8011650 tx=0x7f08c8009f90 comp rx=0 tx=0).stop 2024-07-26T11:03:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08d3e00640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f08bc0b3d40 0x7f08bc0b6250 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f08c002a9e0 tx=0x7f08c0000bc0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 -- 172.21.15.144:0/208689607 shutdown_connections 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b24338f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24338efa0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 --2- 172.21.15.144:0/208689607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b24338ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 -- 172.21.15.144:0/208689607 >> 172.21.15.144:0/208689607 conn(0x55b242fdf300 msgr2=0x55b24338c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 -- 172.21.15.144:0/208689607 shutdown_connections 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.877+0000 7f08e2781740 1 -- 172.21.15.144:0/208689607 wait complete. 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 -- start start 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 Processor -- start 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 0x55b24339e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b24339ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242feb1b0 0x55b2433aa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b24322cab0 con 0x55b242fca4d0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b24328fd50 con 0x55b242feb1b0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b2433a15c0 con 0x55b242fc22f0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.878+0000 7f08e2781740 10 client.0 _ll_register_callbacks cb 0x55b243002a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b24339ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 0x55b24339e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 0x55b24339e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42066/0 (socket says 172.21.15.144:42066) 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b24339ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60512/0 (socket says 172.21.15.144:60512) 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 -- 172.21.15.144:0/1960911716 learned_addr learned my addr 172.21.15.144:0/1960911716 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d9800640 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242feb1b0 0x55b2433aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 -- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242feb1b0 msgr2=0x55b2433aa950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242feb1b0 0x55b2433aa950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 -- 172.21.15.144:0/1960911716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 msgr2=0x55b24339ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b24339ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 -- 172.21.15.144:0/1960911716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2430cf620 con 0x55b242fc22f0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 0x55b24339e9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f08c80097f0 tx=0x7f08c8009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbe102c8f37b7d9a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.879+0000 7f08baa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08baa00640 1 -- 172.21.15.144:0/1960911716 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8027060 con 0x55b242fc22f0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08baa00640 1 -- 172.21.15.144:0/1960911716 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f08c800f7c0 con 0x55b242fc22f0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c800fb70 con 0x55b242fc22f0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08baa00640 1 -- 172.21.15.144:0/1960911716 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08c801e040 con 0x55b242fc22f0 2024-07-26T11:03:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08e2781740 10 client.25654 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b2433ad600 con 0x55b242fc22f0 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.880+0000 7f08baa00640 1 -- 172.21.15.144:0/1960911716 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c800fb70 con 0x55b242fc22f0 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08baa00640 1 -- 172.21.15.144:0/1960911716 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f08c8021030 con 0x55b242fc22f0 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08e2781740 10 client.25654 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08e2781740 10 client.25654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08e2781740 20 client.25654 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08e2781740 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242fa9d80 0x55b2433b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08e2781740 4 client.25654 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2433051c0 con 0x55b242fa9d80 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.881+0000 7f08d3e00640 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242fa9d80 0x55b2433b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.882+0000 7f08baa00640 10 client.25654 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.882+0000 7f08d3e00640 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242fa9d80 0x55b2433b0a10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58e924da8141d875 server_cookie=f9f8b1f284f65a24 in_seq=0 out_seq=0 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.882+0000 7f08baa00640 1 -- 172.21.15.144:0/1960911716 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b2433051c0 con 0x55b242fa9d80 2024-07-26T11:03:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.882+0000 7f08baa00640 10 client.25654 handle_command_reply: tid=0 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 client.25654 shutdown 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 msgr2=0x55b24339e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 0x55b24339e9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f08c80097f0 tx=0x7f08c8009ef0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 shutdown_connections 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242fa9d80 0x55b2433b0a10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242feb1b0 0x55b2433aa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b24339ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 --2- 172.21.15.144:0/1960911716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fc22f0 0x55b24339e9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 >> 172.21.15.144:0/1960911716 conn(0x55b242fdf300 msgr2=0x55b243391bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08baa00640 0 client.25654 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 shutdown_connections 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 1 -- 172.21.15.144:0/1960911716 wait complete. 2024-07-26T11:03:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.883+0000 7f08e2781740 20 client.25654 trim_cache size 0 max 16384 2024-07-26T11:03:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 Processor -- start 2024-07-26T11:03:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 -- start start 2024-07-26T11:03:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24339ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b2433901a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b2433aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41b0f20 con 0x55b242feb1b0 2024-07-26T11:03:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ae7f0 con 0x55b242fc22f0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08e2781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ac640 con 0x55b242fca4d0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b2433901a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08d9800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b2433aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b2433901a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42074/0 (socket says 172.21.15.144:42074) 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.900+0000 7f08d3e00640 1 -- 172.21.15.144:0/309612251 learned_addr learned my addr 172.21.15.144:0/309612251 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d8e00640 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24339ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d3e00640 1 -- 172.21.15.144:0/309612251 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 msgr2=0x55b24339ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d3e00640 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24339ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d3e00640 1 -- 172.21.15.144:0/309612251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 msgr2=0x55b2433aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d3e00640 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b2433aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d3e00640 1 -- 172.21.15.144:0/309612251 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b242fa9fb0 con 0x55b242fca4d0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d9800640 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b2433aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08d3e00640 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b2433901a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f08c0005f70 tx=0x7f08c003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e2b5435c4a01556 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.901+0000 7f08c5600640 1 -- 172.21.15.144:0/309612251 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c005b070 con 0x55b242fca4d0 2024-07-26T11:03:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.902+0000 7f08c5600640 1 -- 172.21.15.144:0/309612251 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c0055af0 con 0x55b242fca4d0 2024-07-26T11:03:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.902+0000 7f08c5600640 1 -- 172.21.15.144:0/309612251 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0035040 con 0x55b242fca4d0 2024-07-26T11:03:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 -- 172.21.15.144:0/309612251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 msgr2=0x55b2433901a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b2433901a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f08c0005f70 tx=0x7f08c003e000 comp rx=0 tx=0).stop 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 -- 172.21.15.144:0/309612251 shutdown_connections 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242feb1b0 0x55b2433aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242fca4d0 0x55b2433901a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 --2- 172.21.15.144:0/309612251 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b24339ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 -- 172.21.15.144:0/309612251 >> 172.21.15.144:0/309612251 conn(0x55b242fdf300 msgr2=0x55b24338fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 -- 172.21.15.144:0/309612251 shutdown_connections 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.903+0000 7f08e2781740 1 -- 172.21.15.144:0/309612251 wait complete. 2024-07-26T11:03:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.904+0000 7f08e2781740 1 -- start start 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.904+0000 7f08e2781740 1 Processor -- start 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b2433a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b2433a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242feb1b0 0x55b243417000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41accb0 con 0x55b242fca4d0 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ae120 con 0x55b242fc22f0 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f08d41ad380 con 0x55b242feb1b0 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08e2781740 10 client.0 _ll_register_callbacks cb 0x7f08c801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b2433a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08d3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b2433a25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60526/0 (socket says 172.21.15.144:60526) 2024-07-26T11:03:45.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08d3e00640 1 -- 172.21.15.144:0/1520974714 learned_addr learned my addr 172.21.15.144:0/1520974714 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b2433a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.905+0000 7f08d9800640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242feb1b0 0x55b243417000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d8e00640 1 -- 172.21.15.144:0/1520974714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242feb1b0 msgr2=0x55b243417000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242feb1b0 0x55b243417000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d8e00640 1 -- 172.21.15.144:0/1520974714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 msgr2=0x55b2433a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b2433a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d8e00640 1 -- 172.21.15.144:0/1520974714 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b243417570 con 0x55b242fc22f0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d3e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b2433a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08d8e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b2433a20b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08c8011650 tx=0x7f08c8009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348cbb03346192b7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08baa00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08baa00640 1 -- 172.21.15.144:0/1520974714 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8027070 con 0x55b242fc22f0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08baa00640 1 -- 172.21.15.144:0/1520974714 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f08c800fad0 con 0x55b242fc22f0 2024-07-26T11:03:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.906+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c0055af0 con 0x55b242fc22f0 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08baa00640 1 -- 172.21.15.144:0/1520974714 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08c8005bf0 con 0x55b242fc22f0 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 10 client.25676 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b243418800 con 0x55b242fc22f0 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08baa00640 1 -- 172.21.15.144:0/1520974714 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c0055af0 con 0x55b242fc22f0 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08baa00640 1 -- 172.21.15.144:0/1520974714 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f08c8021050 con 0x55b242fc22f0 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 10 client.25676 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 10 client.25676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 20 client.25676 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242f7f1c0 0x55b24341aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 4 client.25676 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:03:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.907+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b243300c60 con 0x55b242f7f1c0 2024-07-26T11:03:45.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.908+0000 7f08d3e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242f7f1c0 0x55b24341aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.908+0000 7f08d3e00640 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242f7f1c0 0x55b24341aef0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8bfadb85d505b1e server_cookie=708ab3cc821b5a4c in_seq=0 out_seq=0 2024-07-26T11:03:45.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.908+0000 7f08baa00640 10 client.25676 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08baa00640 1 -- 172.21.15.144:0/1520974714 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b243300c60 con 0x55b242f7f1c0 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08baa00640 10 client.25676 handle_command_reply: tid=0 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 client.25676 shutdown 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 msgr2=0x55b2433a20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b2433a20b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08c8011650 tx=0x7f08c8009fd0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 shutdown_connections 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b242f7f1c0 0x55b24341aef0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b242feb1b0 0x55b243417000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b242fca4d0 0x55b2433a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08baa00640 0 client.25676 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 --2- 172.21.15.144:0/1520974714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b242fc22f0 0x55b2433a20b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 >> 172.21.15.144:0/1520974714 conn(0x55b242fdf300 msgr2=0x55b24338fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 shutdown_connections 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 1 -- 172.21.15.144:0/1520974714 wait complete. 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.915+0000 7f08e2781740 20 client.25676 trim_cache size 0 max 16384 2024-07-26T11:03:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:03:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:03:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:03:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f08bc0b3d40 msgr2=0x7f08bc0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f08bc0b3d40 0x7f08bc0b6250 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f08c002a9e0 tx=0x7f08c0000bc0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 msgr2=0x7f08d419f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 0x7f08d419f700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009fd0 tx=0x7f08cc00ef10 comp rx=0 tx=0).stop 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 shutdown_connections 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f08bc0b3d40 0x7f08bc0b6250 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08d4108660 0x7f08d419f700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08d4107d10 0x7f08d419a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 --2- 172.21.15.144:0/4169546799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08d41023e0 0x7f08d4199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 >> 172.21.15.144:0/4169546799 conn(0x7f08d407a170 msgr2=0x7f08d4101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.916+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 shutdown_connections 2024-07-26T11:03:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:45.917+0000 7f08db800640 1 -- 172.21.15.144:0/4169546799 wait complete. 2024-07-26T11:03:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:47 smithi144 ceph-mon[32129]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:03:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:47 smithi160 ceph-mon[36715]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:03:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:47 smithi183 ceph-mon[37273]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:03:48.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 -- 172.21.15.144:0/193411602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41020a0 msgr2=0x7f51f41024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:48.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 --2- 172.21.15.144:0/193411602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41020a0 0x7f51f41024b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009900 tx=0x7f51e402a480 comp rx=0 tx=0).stop 2024-07-26T11:03:48.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 -- 172.21.15.144:0/193411602 shutdown_connections 2024-07-26T11:03:48.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 --2- 172.21.15.144:0/193411602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51f410fc50 0x7f51f4112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 --2- 172.21.15.144:0/193411602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41029f0 0x7f51f410f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 --2- 172.21.15.144:0/193411602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41020a0 0x7f51f41024b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 -- 172.21.15.144:0/193411602 >> 172.21.15.144:0/193411602 conn(0x7f51f4079170 msgr2=0x7f51f4079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:48.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 -- 172.21.15.144:0/193411602 shutdown_connections 2024-07-26T11:03:48.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.063+0000 7f51faa00640 1 -- 172.21.15.144:0/193411602 wait complete. 2024-07-26T11:03:48.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.064+0000 7f51faa00640 1 Processor -- start 2024-07-26T11:03:48.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.064+0000 7f51faa00640 1 -- start start 2024-07-26T11:03:48.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51faa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 0x7f51f419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:48.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51faa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41029f0 0x7f51f419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:48.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51faa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51f410fc50 0x7f51f41a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51faa00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4115670 con 0x7f51f41029f0 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51faa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41154f0 con 0x7f51f41020a0 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51faa00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41157f0 con 0x7f51f410fc50 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51f3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 0x7f51f419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51f3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 0x7f51f419c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:37658/0 (socket says 172.21.15.144:37658) 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41029f0 0x7f51f419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51f3e00640 1 -- 172.21.15.144:0/2340461282 learned_addr learned my addr 172.21.15.144:0/2340461282 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.065+0000 7f51f8a00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51f410fc50 0x7f51f41a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f3e00640 1 -- 172.21.15.144:0/2340461282 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51f410fc50 msgr2=0x7f51f41a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:48.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f3e00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51f410fc50 0x7f51f41a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f3e00640 1 -- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41029f0 msgr2=0x7f51f419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:48.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f3e00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41029f0 0x7f51f419c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f3e00640 1 -- 172.21.15.144:0/2340461282 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f41a2030 con 0x7f51f41020a0 2024-07-26T11:03:48.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f3e00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 0x7f51f419c0c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51e4005940 tx=0x7f51e40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:48.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e403f030 con 0x7f51f41020a0 2024-07-26T11:03:48.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.066+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f41a22c0 con 0x7f51f41020a0 2024-07-26T11:03:48.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.067+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e4003cb0 con 0x7f51f41020a0 2024-07-26T11:03:48.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.067+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4035070 con 0x7f51f41020a0 2024-07-26T11:03:48.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.067+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f41a8a80 con 0x7f51f41020a0 2024-07-26T11:03:48.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.068+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f406bbf0 con 0x7f51f41020a0 2024-07-26T11:03:48.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.070+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f51e4042030 con 0x7f51f41020a0 2024-07-26T11:03:48.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.072+0000 7f51f1600640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f51d80b3d40 0x7f51d80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:48.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.072+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f51e4104e00 con 0x7f51f41020a0 2024-07-26T11:03:48.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.072+0000 7f51ebe00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f51d80b3d40 0x7f51d80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:48.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.073+0000 7f51ebe00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f51d80b3d40 0x7f51d80b6250 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f51f4103b20 tx=0x7f51dc00b410 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:48.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.074+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f51e40cb6f0 con 0x7f51f41020a0 2024-07-26T11:03:48.170 INFO:tasks.workunit.client.0.smithi144.stderr:Updating files: 59% (7540/12697) Updating files: 60% (7619/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 97% (12350/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-26T11:03:48.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.318+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51f41024b0 con 0x7f51f41020a0 2024-07-26T11:03:48.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.319+0000 7f51f1600640 1 -- 172.21.15.144:0/2340461282 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11931 (secure 0 0 0) 0x7f51e40cfc40 con 0x7f51f41020a0 2024-07-26T11:03:48.321 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:03:48.322 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":26,"btime":"2024-07-26T11:02:20:221771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T11:02:19.492590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24703,24715,24728]},"id":1}]} 2024-07-26T11:03:48.322 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 26 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f51d80b3d40 msgr2=0x7f51d80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f51d80b3d40 0x7f51d80b6250 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f51f4103b20 tx=0x7f51dc00b410 comp rx=0 tx=0).stop 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 msgr2=0x7f51f419c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 0x7f51f419c0c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51e4005940 tx=0x7f51e40042f0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 shutdown_connections 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f51d80b3d40 0x7f51d80b6250 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51f410fc50 0x7f51f41a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51f41029f0 0x7f51f419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 --2- 172.21.15.144:0/2340461282 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51f41020a0 0x7f51f419c0c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:48.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 >> 172.21.15.144:0/2340461282 conn(0x7f51f4079170 msgr2=0x7f51f41086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:48.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 shutdown_connections 2024-07-26T11:03:48.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:48.322+0000 7f51faa00640 1 -- 172.21.15.144:0/2340461282 wait complete. 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr: git switch -c 2024-07-26T11:03:48.583 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr:Or undo this operation with: 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr: git switch - 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-07-26T11:03:48.584 INFO:tasks.workunit.client.0.smithi144.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-26T11:03:48.592 DEBUG:teuthology.orchestra.run.smithi144:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T11:03:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:48 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/2340461282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:03:48.708 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:48 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/2340461282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:03:48.711 INFO:tasks.workunit.client.0.smithi144.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T11:03:48.718 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T11:03:48.719 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T11:03:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:48 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/2340461282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:03:49.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:03:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:49 smithi183 ceph-mon[37273]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:03:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:49 smithi144 ceph-mon[32129]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:03:49.767 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:49 smithi160 ceph-mon[36715]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:03:50.448 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T11:03:50.563 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T11:03:50.643 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T11:03:50.647 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T11:03:50.647 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T11:03:50.718 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T11:03:50.727 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-07-26T11:03:50.728 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T11:03:50.771 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-07-26T11:03:50.772 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-07-26T11:03:50.772 DEBUG:teuthology.orchestra.run.smithi144:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-07-26T11:03:50.884 INFO:tasks.workunit.client.0.smithi144.stderr:+ mkdir -p fsstress 2024-07-26T11:03:50.888 INFO:tasks.workunit.client.0.smithi144.stderr:+ pushd fsstress 2024-07-26T11:03:50.889 INFO:tasks.workunit.client.0.smithi144.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-07-26T11:03:50.889 INFO:tasks.workunit.client.0.smithi144.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-07-26T11:03:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:51 smithi144 ceph-mon[32129]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 7 op/s 2024-07-26T11:03:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:51 smithi160 ceph-mon[36715]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 7 op/s 2024-07-26T11:03:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:51 smithi183 ceph-mon[37273]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 7 op/s 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 -- 172.21.15.144:0/902500947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c410d650 msgr2=0x7fa4c410fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/902500947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c410d650 0x7fa4c410fb40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009900 tx=0x7fa4bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 -- 172.21.15.144:0/902500947 shutdown_connections 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/902500947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c410d650 0x7fa4c410fb40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/902500947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c4104b40 0x7fa4c410d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/902500947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4c41041f0 0x7fa4c4104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 -- 172.21.15.144:0/902500947 >> 172.21.15.144:0/902500947 conn(0x7fa4c40fdcf0 msgr2=0x7fa4c4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:52.470 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.467+0000 7fa4c3e00640 1 -- 172.21.15.144:0/902500947 shutdown_connections 2024-07-26T11:03:52.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.468+0000 7fa4c3e00640 1 -- 172.21.15.144:0/902500947 wait complete. 2024-07-26T11:03:52.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 Processor -- start 2024-07-26T11:03:52.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 -- start start 2024-07-26T11:03:52.471 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c41041f0 0x7fa4c419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 0x7fa4c419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4c410d650 0x7fa4c41a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41132c0 con 0x7fa4c410d650 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4113140 con 0x7fa4c4104b40 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.469+0000 7fa4c3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4113440 con 0x7fa4c41041f0 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4c410d650 0x7fa4c41a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 0x7fa4c419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 0x7fa4c419c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59074/0 (socket says 172.21.15.144:59074) 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 -- 172.21.15.144:0/608176790 learned_addr learned my addr 172.21.15.144:0/608176790 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:52.472 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c41041f0 0x7fa4c419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 -- 172.21.15.144:0/608176790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c41041f0 msgr2=0x7fa4c419bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c41041f0 0x7fa4c419bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 -- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4c410d650 msgr2=0x7fa4c41a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4c410d650 0x7fa4c41a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 -- 172.21.15.144:0/608176790 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c41a1e40 con 0x7fa4c4104b40 2024-07-26T11:03:52.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c41041f0 0x7fa4c419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:52.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.470+0000 7fa4c2000640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 0x7fa4c419c470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e970 tx=0x7fa4ac00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:52.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.471+0000 7fa4bb400640 1 -- 172.21.15.144:0/608176790 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac00cd80 con 0x7fa4c4104b40 2024-07-26T11:03:52.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.471+0000 7fa4bb400640 1 -- 172.21.15.144:0/608176790 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4ac0035f0 con 0x7fa4c4104b40 2024-07-26T11:03:52.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.471+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c41a2130 con 0x7fa4c4104b40 2024-07-26T11:03:52.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.471+0000 7fa4bb400640 1 -- 172.21.15.144:0/608176790 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac010690 con 0x7fa4c4104b40 2024-07-26T11:03:52.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.471+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c41a8890 con 0x7fa4c4104b40 2024-07-26T11:03:52.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.475+0000 7fa4d0644740 1 Processor -- start 2024-07-26T11:03:52.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.475+0000 7fa4d0644740 1 -- start start 2024-07-26T11:03:52.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 0x55e7d22e8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22e8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4d0644740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e7d225ebd0 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4d0644740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e7d224e0d0 con 0x55e7d1f22390 2024-07-26T11:03:52.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4d0644740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e7d2252110 con 0x55e7d1f43070 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4c3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 0x55e7d22e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4c3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 0x55e7d22e8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48318/0 (socket says 172.21.15.144:48318) 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4c3400640 1 -- 172.21.15.144:0/496640131 learned_addr learned my addr 172.21.15.144:0/496640131 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.476+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4c3400640 1 -- 172.21.15.144:0/496640131 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 msgr2=0x55e7d22e8a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4c3400640 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22e8a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4c3400640 1 -- 172.21.15.144:0/496640131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 msgr2=0x55e7d22e8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4c3400640 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4c3400640 1 -- 172.21.15.144:0/496640131 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7d1fae650 con 0x55e7d1f43070 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4bb400640 1 -- 172.21.15.144:0/608176790 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa4ac003aa0 con 0x7fa4c4104b40 2024-07-26T11:03:52.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4c3400640 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 0x55e7d22e8010 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009900 tx=0x7fa4bc032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d203b6799b765388 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:52.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4b3e00640 1 -- 172.21.15.144:0/496640131 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc040070 con 0x55e7d1f43070 2024-07-26T11:03:52.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4b3e00640 1 -- 172.21.15.144:0/496640131 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4bc004030 con 0x55e7d1f43070 2024-07-26T11:03:52.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.477+0000 7fa4b3e00640 1 -- 172.21.15.144:0/496640131 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc035040 con 0x55e7d1f43070 2024-07-26T11:03:52.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 -- 172.21.15.144:0/496640131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 msgr2=0x55e7d22e8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 0x55e7d22e8010 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009900 tx=0x7fa4bc032b20 comp rx=0 tx=0).stop 2024-07-26T11:03:52.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 -- 172.21.15.144:0/496640131 shutdown_connections 2024-07-26T11:03:52.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22e8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 --2- 172.21.15.144:0/496640131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f43070 0x55e7d22e8010 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 -- 172.21.15.144:0/496640131 >> 172.21.15.144:0/496640131 conn(0x55e7d1f37460 msgr2=0x55e7d22e5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 -- 172.21.15.144:0/496640131 shutdown_connections 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4d0644740 1 -- 172.21.15.144:0/496640131 wait complete. 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4bb400640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa4a80b3cd0 0x7fa4a80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.479+0000 7fa4bb400640 1 -- 172.21.15.144:0/608176790 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa4ac0e4650 con 0x7fa4c4104b40 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.480+0000 7fa4d0644740 1 -- start start 2024-07-26T11:03:52.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.480+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa4a80b3cd0 0x7fa4a80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.480+0000 7fa4d0644740 1 Processor -- start 2024-07-26T11:03:52.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa4a80b3cd0 0x7fa4a80b61e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x55e7d22e6c90 tx=0x7fa4b4009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:52.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d22f4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e7d20b1700 con 0x55e7d1f43070 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e7d21ea6a0 con 0x55e7d1f22390 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e7d22f6790 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4d0644740 10 client.0 _ll_register_callbacks cb 0x55e7d21fa150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4c2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4c2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48320/0 (socket says 172.21.15.144:48320) 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4c2a00640 1 -- 172.21.15.144:0/904545674 learned_addr learned my addr 172.21.15.144:0/904545674 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4c3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d22f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.481+0000 7fa4c2000640 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4c2a00640 1 -- 172.21.15.144:0/904545674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 msgr2=0x55e7d22f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4c2a00640 1 -- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 msgr2=0x55e7d22f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d22f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4c2a00640 1 -- 172.21.15.144:0/904545674 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7d2201840 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f35b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40094d0 tx=0x7fa4b4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c85f442aab8fe68 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.482+0000 7fa4b0c00640 1 -- 172.21.15.144:0/904545674 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b402c060 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.483+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4bc004030 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.483+0000 7fa4b0c00640 1 -- 172.21.15.144:0/904545674 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa4b4017030 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.483+0000 7fa4b0c00640 1 -- 172.21.15.144:0/904545674 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa4b4033050 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.483+0000 7fa4d0644740 10 client.25678 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.483+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7d23023e0 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.483+0000 7fa4b0c00640 1 -- 172.21.15.144:0/904545674 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4bc004030 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4b0c00640 1 -- 172.21.15.144:0/904545674 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa4b401c990 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4d0644740 10 client.25678 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4d0644740 10 client.25678 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4d0644740 20 client.25678 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4d0644740 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1f019f0 0x55e7d23058f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4d0644740 4 client.25678 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7d2260b20 con 0x55e7d1f019f0 2024-07-26T11:03:52.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.484+0000 7fa4c2000640 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1f019f0 0x55e7d23058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.485+0000 7fa4c2000640 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1f019f0 0x55e7d23058f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa1d9f4175126020 server_cookie=dda20b4f7ca4674f in_seq=0 out_seq=0 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.485+0000 7fa4b0c00640 10 client.25678 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.485+0000 7fa4b0c00640 1 -- 172.21.15.144:0/904545674 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e7d2260b20 con 0x55e7d1f019f0 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.485+0000 7fa4b0c00640 10 client.25678 handle_command_reply: tid=0 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.485+0000 7fa4d0644740 1 client.25678 shutdown 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 msgr2=0x55e7d22f35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f35b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40094d0 tx=0x7fa4b4004ab0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 shutdown_connections 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1f019f0 0x55e7d23058f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d22f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 --2- 172.21.15.144:0/904545674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f35b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4b0c00640 0 client.25678 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:52.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 >> 172.21.15.144:0/904545674 conn(0x55e7d1f37460 msgr2=0x55e7d22eb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:52.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 shutdown_connections 2024-07-26T11:03:52.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 1 -- 172.21.15.144:0/904545674 wait complete. 2024-07-26T11:03:52.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.486+0000 7fa4d0644740 20 client.25678 trim_cache size 0 max 16384 2024-07-26T11:03:52.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 Processor -- start 2024-07-26T11:03:52.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 -- start start 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f22390 0x55e7d22e9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f43070 0x55e7d230b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b5150 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b2a20 con 0x55e7d1f43070 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4d0644740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b0870 con 0x55e7d1f22390 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4c2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4c2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37608/0 (socket says 172.21.15.144:37608) 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.501+0000 7fa4c2a00640 1 -- 172.21.15.144:0/1904474979 learned_addr learned my addr 172.21.15.144:0/1904474979 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:52.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c3400640 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f43070 0x55e7d230b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2000640 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f22390 0x55e7d22e9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2a00640 1 -- 172.21.15.144:0/1904474979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f22390 msgr2=0x55e7d22e9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f22390 0x55e7d22e9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2a00640 1 -- 172.21.15.144:0/1904474979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f43070 msgr2=0x55e7d230b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f43070 0x55e7d230b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2a00640 1 -- 172.21.15.144:0/1904474979 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7d22f66d0 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.502+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e4f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4b401cde0 tx=0x7fa4b401f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b134695b0b186c9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.503+0000 7fa4b3400640 1 -- 172.21.15.144:0/1904474979 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4011a10 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.503+0000 7fa4b3400640 1 -- 172.21.15.144:0/1904474979 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b401c8a0 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.503+0000 7fa4b3400640 1 -- 172.21.15.144:0/1904474979 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4037700 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 -- 172.21.15.144:0/1904474979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 msgr2=0x55e7d22e4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e4f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4b401cde0 tx=0x7fa4b401f850 comp rx=0 tx=0).stop 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 -- 172.21.15.144:0/1904474979 shutdown_connections 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f43070 0x55e7d230b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f22390 0x55e7d22e9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 --2- 172.21.15.144:0/1904474979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22e4f00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 -- 172.21.15.144:0/1904474979 >> 172.21.15.144:0/1904474979 conn(0x55e7d1f37460 msgr2=0x55e7d22e9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 -- 172.21.15.144:0/1904474979 shutdown_connections 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.504+0000 7fa4d0644740 1 -- 172.21.15.144:0/1904474979 wait complete. 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.505+0000 7fa4d0644740 1 -- start start 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.505+0000 7fa4d0644740 1 Processor -- start 2024-07-26T11:03:52.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d236c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b0ee0 con 0x55e7d1f43070 2024-07-26T11:03:52.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b2350 con 0x55e7d1f22390 2024-07-26T11:03:52.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41b15b0 con 0x55e7d1f1a1b0 2024-07-26T11:03:52.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4d0644740 10 client.0 _ll_register_callbacks cb 0x7fa4b401f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4c2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4c3400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d236c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4c2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48338/0 (socket says 172.21.15.144:48338) 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4c2a00640 1 -- 172.21.15.144:0/2331792650 learned_addr learned my addr 172.21.15.144:0/2331792650 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.506+0000 7fa4c2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2000640 1 -- 172.21.15.144:0/2331792650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 msgr2=0x55e7d22f91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2000640 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2000640 1 -- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 msgr2=0x55e7d236c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2000640 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d236c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2000640 1 -- 172.21.15.144:0/2331792650 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7d236c730 con 0x55e7d1f22390 2024-07-26T11:03:52.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:52.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4c2000640 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f97c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0e19d0 tx=0x7fa4ac0aa900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da4ad0296228d8e6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:52.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4b0c00640 1 -- 172.21.15.144:0/2331792650 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0af490 con 0x55e7d1f22390 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4b0c00640 1 -- 172.21.15.144:0/2331792650 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fa4ac0a7120 con 0x55e7d1f22390 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.507+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4b401c8a0 con 0x55e7d1f22390 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4b0c00640 1 -- 172.21.15.144:0/2331792650 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa4ac0b3f40 con 0x55e7d1f22390 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 10 client.25697 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7d236da10 con 0x55e7d1f22390 2024-07-26T11:03:52.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4b0c00640 1 -- 172.21.15.144:0/2331792650 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b401c8a0 con 0x55e7d1f22390 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4b0c00640 1 -- 172.21.15.144:0/2331792650 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa4ac0a6800 con 0x55e7d1f22390 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 10 client.25697 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 10 client.25697 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 20 client.25697 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1ed6da0 0x55e7d23701d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 4 client.25697 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.508+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7d225c5c0 con 0x55e7d1ed6da0 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.509+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1ed6da0 0x55e7d23701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.509+0000 7fa4c2a00640 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1ed6da0 0x55e7d23701d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=749ff93012c357bf server_cookie=35a3d43456f2eeff in_seq=0 out_seq=0 2024-07-26T11:03:52.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.509+0000 7fa4b0c00640 10 client.25697 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:52.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4b0c00640 1 -- 172.21.15.144:0/2331792650 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7d225c5c0 con 0x55e7d1ed6da0 2024-07-26T11:03:52.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4b0c00640 10 client.25697 handle_command_reply: tid=0 2024-07-26T11:03:52.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 client.25697 shutdown 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 msgr2=0x55e7d22f97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f97c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0e19d0 tx=0x7fa4ac0aa900 comp rx=0 tx=0).stop 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 shutdown_connections 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e7d1ed6da0 0x55e7d23701d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e7d1f43070 0x55e7d236c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e7d1f22390 0x55e7d22f97c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 --2- 172.21.15.144:0/2331792650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e7d1f1a1b0 0x55e7d22f91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 >> 172.21.15.144:0/2331792650 conn(0x55e7d1f37460 msgr2=0x55e7d22e4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4b0c00640 0 client.25697 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 shutdown_connections 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 1 -- 172.21.15.144:0/2331792650 wait complete. 2024-07-26T11:03:52.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.510+0000 7fa4d0644740 20 client.25697 trim_cache size 0 max 16384 2024-07-26T11:03:52.515 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:03:52.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa4a80b3cd0 msgr2=0x7fa4a80b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa4a80b3cd0 0x7fa4a80b61e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x55e7d22e6c90 tx=0x7fa4b4009500 comp rx=0 tx=0).stop 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 msgr2=0x7fa4c419c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 0x7fa4c419c470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00e970 tx=0x7fa4ac00ee30 comp rx=0 tx=0).stop 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 shutdown_connections 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fa4a80b3cd0 0x7fa4a80b61e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4c410d650 0x7fa4c41a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c4104b40 0x7fa4c419c470 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 --2- 172.21.15.144:0/608176790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4c41041f0 0x7fa4c419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.511+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 >> 172.21.15.144:0/608176790 conn(0x7fa4c40fdcf0 msgr2=0x7fa4c4101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.512+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 shutdown_connections 2024-07-26T11:03:52.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:52.512+0000 7fa4c3e00640 1 -- 172.21.15.144:0/608176790 wait complete. 2024-07-26T11:03:52.554 INFO:tasks.workunit.client.0.smithi144.stderr:+ tar xzf ltp-full.tgz 2024-07-26T11:03:53.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:03:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:53 smithi160 ceph-mon[36715]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-07-26T11:03:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:53 smithi183 ceph-mon[37273]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-07-26T11:03:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:53 smithi144 ceph-mon[32129]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-07-26T11:03:53.874 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:55 smithi160 ceph-mon[36715]: pgmap v504: 129 pgs: 129 active+clean; 3.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2024-07-26T11:03:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:55 smithi183 ceph-mon[37273]: pgmap v504: 129 pgs: 129 active+clean; 3.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2024-07-26T11:03:55.648 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:55 smithi144 ceph-mon[32129]: pgmap v504: 129 pgs: 129 active+clean; 3.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2024-07-26T11:03:56.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.254+0000 7f43e5a00640 1 -- 172.21.15.144:0/2086208895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 msgr2=0x7f43e010b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.254+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2086208895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e010b530 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009900 tx=0x7f43cc02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 -- 172.21.15.144:0/2086208895 shutdown_connections 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2086208895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43e010ba70 0x7f43e01122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2086208895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e010b530 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2086208895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43e01057e0 0x7f43e0105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 -- 172.21.15.144:0/2086208895 >> 172.21.15.144:0/2086208895 conn(0x7f43e01011f0 msgr2=0x7f43e0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 -- 172.21.15.144:0/2086208895 shutdown_connections 2024-07-26T11:03:56.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.255+0000 7f43e5a00640 1 -- 172.21.15.144:0/2086208895 wait complete. 2024-07-26T11:03:56.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.256+0000 7f43e5a00640 1 Processor -- start 2024-07-26T11:03:56.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.256+0000 7f43e5a00640 1 -- start start 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43e5a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43e01057e0 0x7f43e019c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43e5a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43e5a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43e010ba70 0x7f43e01a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43e5a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f43e0115730 con 0x7f43e01057e0 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43e5a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01155b0 con 0x7f43e010b120 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43e5a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01158b0 con 0x7f43e010ba70 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43de000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43de000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59130/0 (socket says 172.21.15.144:59130) 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43de000640 1 -- 172.21.15.144:0/2015498039 learned_addr learned my addr 172.21.15.144:0/2015498039 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43dea00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43e01057e0 0x7f43e019c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.257+0000 7f43df400640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43e010ba70 0x7f43e01a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43de000640 1 -- 172.21.15.144:0/2015498039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43e010ba70 msgr2=0x7f43e01a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43de000640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43e010ba70 0x7f43e01a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43de000640 1 -- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43e01057e0 msgr2=0x7f43e019c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43de000640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43e01057e0 0x7f43e019c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43de000640 1 -- 172.21.15.144:0/2015498039 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e01a2130 con 0x7f43e010b120 2024-07-26T11:03:56.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43de000640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e019c760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43cc005ab0 tx=0x7f43cc003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:56.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.258+0000 7f43d7400640 1 -- 172.21.15.144:0/2015498039 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc03f070 con 0x7f43e010b120 2024-07-26T11:03:56.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.259+0000 7f43d7400640 1 -- 172.21.15.144:0/2015498039 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43cc031b50 con 0x7f43e010b120 2024-07-26T11:03:56.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.259+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e01a23c0 con 0x7f43e010b120 2024-07-26T11:03:56.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.259+0000 7f43d7400640 1 -- 172.21.15.144:0/2015498039 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc035030 con 0x7f43e010b120 2024-07-26T11:03:56.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.259+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e01a8b80 con 0x7f43e010b120 2024-07-26T11:03:56.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.262+0000 7f43ec6d2740 1 Processor -- start 2024-07-26T11:03:56.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.262+0000 7f43ec6d2740 1 -- start start 2024-07-26T11:03:56.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60551f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60551f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d60551faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55d605495bd0 con 0x55d6051511b0 2024-07-26T11:03:56.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d6054850d0 con 0x55d60517a070 2024-07-26T11:03:56.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d605489110 con 0x55d605159390 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60551f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60551f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59144/0 (socket says 172.21.15.144:59144) 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 -- 172.21.15.144:0/3553820598 learned_addr learned my addr 172.21.15.144:0/3553820598 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43dea00640 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60551f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 -- 172.21.15.144:0/3553820598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 msgr2=0x55d60551faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d60551faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 -- 172.21.15.144:0/3553820598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 msgr2=0x55d60551f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60551f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.263+0000 7f43df400640 1 -- 172.21.15.144:0/3553820598 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6051e5650 con 0x55d60517a070 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.264+0000 7f43d7400640 1 -- 172.21.15.144:0/2015498039 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f43cc042070 con 0x7f43e010b120 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.264+0000 7f43df400640 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60551f020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009f50 tx=0x7f43d80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=721499c184a433b2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.264+0000 7f43cbe00640 1 -- 172.21.15.144:0/3553820598 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8004b20 con 0x55d60517a070 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.264+0000 7f43cbe00640 1 -- 172.21.15.144:0/3553820598 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d8003e20 con 0x55d60517a070 2024-07-26T11:03:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.264+0000 7f43cbe00640 1 -- 172.21.15.144:0/3553820598 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8004840 con 0x55d60517a070 2024-07-26T11:03:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3553820598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 msgr2=0x55d60551f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60551f020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009f50 tx=0x7f43d80118d0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3553820598 shutdown_connections 2024-07-26T11:03:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d60551faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60551f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3553820598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60551f020 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.265+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3553820598 >> 172.21.15.144:0/3553820598 conn(0x55d60516e460 msgr2=0x55d60551cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.266+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3553820598 shutdown_connections 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.266+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3553820598 wait complete. 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.266+0000 7f43d7400640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43c40b3cd0 0x7f43c40b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.266+0000 7f43d7400640 1 -- 172.21.15.144:0/2015498039 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f43cc104630 con 0x7f43e010b120 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.266+0000 7f43dea00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43c40b3cd0 0x7f43c40b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.266+0000 7f43ec6d2740 1 -- start start 2024-07-26T11:03:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 Processor -- start 2024-07-26T11:03:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43dea00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43c40b3cd0 0x7f43c40b61e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55d60551dca0 tx=0x7f43d000b390 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 0x55d60552ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d60517a070 0x55d60552b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55d6052e8700 con 0x55d60517a070 2024-07-26T11:03:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d6054216a0 con 0x55d605159390 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d60552d800 con 0x55d6051511b0 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.267+0000 7f43ec6d2740 10 client.0 _ll_register_callbacks cb 0x55d605431150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 0x55d60552ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 0x55d60552ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59160/0 (socket says 172.21.15.144:59160) 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 -- 172.21.15.144:0/3047456327 learned_addr learned my addr 172.21.15.144:0/3047456327 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43dea00640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43df400640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d60517a070 0x55d60552b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 -- 172.21.15.144:0/3047456327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 msgr2=0x55d60552a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 -- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d60517a070 msgr2=0x55d60552b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d60517a070 0x55d60552b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 -- 172.21.15.144:0/3047456327 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d605438840 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43dea00640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43de000640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 0x55d60552ab80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f43cc031440 tx=0x7f43cc0ca750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcba7f733d348cb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.268+0000 7f43c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43c8c00640 1 -- 172.21.15.144:0/3047456327 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc049070 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43c8c00640 1 -- 172.21.15.144:0/3047456327 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f43cc101c30 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43d8003e20 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43c8c00640 1 -- 172.21.15.144:0/3047456327 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f43cc0cf470 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43ec6d2740 10 client.25709 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d6055394a0 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.269+0000 7f43c8c00640 1 -- 172.21.15.144:0/3047456327 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d8003e20 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43c8c00640 1 -- 172.21.15.144:0/3047456327 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f43cc04c060 con 0x55d605159390 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43ec6d2740 10 client.25709 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43ec6d2740 10 client.25709 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-07-26T11:03:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43ec6d2740 20 client.25709 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d6051389f0 0x55d60553c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43ec6d2740 4 client.25709 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d605497b20 con 0x55d6051389f0 2024-07-26T11:03:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.270+0000 7f43dea00640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d6051389f0 0x55d60553c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.271+0000 7f43dea00640 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d6051389f0 0x55d60553c7d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82c2507c3b5d0983 server_cookie=1cf196a87dee258c in_seq=0 out_seq=0 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.271+0000 7f43c8c00640 10 client.25709 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43c8c00640 1 -- 172.21.15.144:0/3047456327 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d605497b20 con 0x55d6051389f0 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43c8c00640 10 client.25709 handle_command_reply: tid=0 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 client.25709 shutdown 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 msgr2=0x55d60552ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 0x55d60552ab80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f43cc031440 tx=0x7f43cc0ca750 comp rx=0 tx=0).stop 2024-07-26T11:03:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 shutdown_connections 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d60517a070 0x55d60552b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d6051389f0 0x55d60553c7d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d605159390 0x55d60552ab80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/3047456327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43c8c00640 0 client.25709 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 >> 172.21.15.144:0/3047456327 conn(0x55d60516e460 msgr2=0x55d6055221b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 shutdown_connections 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.272+0000 7f43ec6d2740 1 -- 172.21.15.144:0/3047456327 wait complete. 2024-07-26T11:03:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.273+0000 7f43ec6d2740 20 client.25709 trim_cache size 0 max 16384 2024-07-26T11:03:56.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.289+0000 7f43ec6d2740 1 Processor -- start 2024-07-26T11:03:56.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.289+0000 7f43ec6d2740 1 -- start start 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.289+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60552c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d605536070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d6055365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01b52e0 con 0x55d6051511b0 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01b2bb0 con 0x55d60517a070 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01b0a00 con 0x55d605159390 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43dea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60552c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43dea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60552c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37662/0 (socket says 172.21.15.144:37662) 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43dea00640 1 -- 172.21.15.144:0/653899339 learned_addr learned my addr 172.21.15.144:0/653899339 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:56.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43df400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d6055365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43de000640 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d605536070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43df400640 1 -- 172.21.15.144:0/653899339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 msgr2=0x55d605536070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43df400640 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d605536070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43df400640 1 -- 172.21.15.144:0/653899339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 msgr2=0x55d60552c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43df400640 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60552c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43df400640 1 -- 172.21.15.144:0/653899339 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d605520cf0 con 0x55d60517a070 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.290+0000 7f43dea00640 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60552c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.291+0000 7f43df400640 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d6055365b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f43d8012fd0 tx=0x7f43d8004a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e9ba6b2df4e2691 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.291+0000 7f43cb400640 1 -- 172.21.15.144:0/653899339 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8028020 con 0x55d60517a070 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.291+0000 7f43cb400640 1 -- 172.21.15.144:0/653899339 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d800fab0 con 0x55d60517a070 2024-07-26T11:03:56.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.291+0000 7f43cb400640 1 -- 172.21.15.144:0/653899339 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8024b60 con 0x55d60517a070 2024-07-26T11:03:56.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 -- 172.21.15.144:0/653899339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 msgr2=0x55d6055365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d6055365b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f43d8012fd0 tx=0x7f43d8004a30 comp rx=0 tx=0).stop 2024-07-26T11:03:56.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 -- 172.21.15.144:0/653899339 shutdown_connections 2024-07-26T11:03:56.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d6055365b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d605159390 0x55d605536070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/653899339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6051511b0 0x55d60552c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 -- 172.21.15.144:0/653899339 >> 172.21.15.144:0/653899339 conn(0x55d60516e460 msgr2=0x55d605535960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 -- 172.21.15.144:0/653899339 shutdown_connections 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.292+0000 7f43ec6d2740 1 -- 172.21.15.144:0/653899339 wait complete. 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.293+0000 7f43ec6d2740 1 -- start start 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.293+0000 7f43ec6d2740 1 Processor -- start 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.293+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d605159390 0x55d60552e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.296 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43ec6d2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60559d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01b1070 con 0x55d605159390 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01b24e0 con 0x55d60517a070 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43ec6d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43e01b1740 con 0x55d6051511b0 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43ec6d2740 10 client.0 _ll_register_callbacks cb 0x7f43cc0cad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43dea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43df400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60559d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43dea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48376/0 (socket says 172.21.15.144:48376) 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43dea00640 1 -- 172.21.15.144:0/894546594 learned_addr learned my addr 172.21.15.144:0/894546594 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43de000640 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d605159390 0x55d60552e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43df400640 1 -- 172.21.15.144:0/894546594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 msgr2=0x55d60552e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43df400640 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43df400640 1 -- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d605159390 msgr2=0x55d60552e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43df400640 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d605159390 0x55d60552e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.294+0000 7f43df400640 1 -- 172.21.15.144:0/894546594 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d60559d670 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.295+0000 7f43df400640 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60559d000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f43d801b040 tx=0x7f43d80246a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3125712a5de67b8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.295+0000 7f43c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.295+0000 7f43c8c00640 1 -- 172.21.15.144:0/894546594 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8033020 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.295+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43d800fab0 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.295+0000 7f43c8c00640 1 -- 172.21.15.144:0/894546594 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f43d800f810 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.295+0000 7f43c8c00640 1 -- 172.21.15.144:0/894546594 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f43d803e050 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 10 client.25715 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d6055a49b0 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43c8c00640 1 -- 172.21.15.144:0/894546594 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d800fab0 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43c8c00640 1 -- 172.21.15.144:0/894546594 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f43d802d830 con 0x55d60517a070 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 10 client.25715 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 10 client.25715 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 20 client.25715 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:56.300 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d60510dda0 0x55d6055a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:56.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 4 client.25715 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:03:56.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.296+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6054935c0 con 0x55d60510dda0 2024-07-26T11:03:56.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.297+0000 7f43dea00640 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d60510dda0 0x55d6055a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.297+0000 7f43dea00640 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d60510dda0 0x55d6055a6ec0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c7bf51fea44fc97 server_cookie=2affabad42c2dd1 in_seq=0 out_seq=0 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.297+0000 7f43c8c00640 10 client.25715 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43c8c00640 1 -- 172.21.15.144:0/894546594 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6054935c0 con 0x55d60510dda0 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43c8c00640 10 client.25715 handle_command_reply: tid=0 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 client.25715 shutdown 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 msgr2=0x55d60559d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60559d000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f43d801b040 tx=0x7f43d80246a0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 shutdown_connections 2024-07-26T11:03:56.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55d60510dda0 0x55d6055a6ec0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d60517a070 0x55d60559d000 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d605159390 0x55d60552e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 --2- 172.21.15.144:0/894546594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6051511b0 0x55d60552e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43c8c00640 0 client.25715 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 >> 172.21.15.144:0/894546594 conn(0x55d60516e460 msgr2=0x55d60552c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 shutdown_connections 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 1 -- 172.21.15.144:0/894546594 wait complete. 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.298+0000 7f43ec6d2740 20 client.25715 trim_cache size 0 max 16384 2024-07-26T11:03:56.303 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43c40b3cd0 msgr2=0x7f43c40b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43c40b3cd0 0x7f43c40b61e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55d60551dca0 tx=0x7f43d000b390 comp rx=0 tx=0).stop 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 msgr2=0x7f43e019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e019c760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43cc005ab0 tx=0x7f43cc003b90 comp rx=0 tx=0).stop 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 shutdown_connections 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43c40b3cd0 0x7f43c40b61e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43e010ba70 0x7f43e01a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43e010b120 0x7f43e019c760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 --2- 172.21.15.144:0/2015498039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43e01057e0 0x7f43e019c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.299+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 >> 172.21.15.144:0/2015498039 conn(0x7f43e01011f0 msgr2=0x7f43e0109700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:56.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.300+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 shutdown_connections 2024-07-26T11:03:56.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:03:56.300+0000 7f43e5a00640 1 -- 172.21.15.144:0/2015498039 wait complete. 2024-07-26T11:03:57.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:03:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:57 smithi144 ceph-mon[32129]: pgmap v505: 129 pgs: 129 active+clean; 3.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 232 KiB/s wr, 15 op/s 2024-07-26T11:03:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:57 smithi160 ceph-mon[36715]: pgmap v505: 129 pgs: 129 active+clean; 3.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 232 KiB/s wr, 15 op/s 2024-07-26T11:03:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:57 smithi183 ceph-mon[37273]: pgmap v505: 129 pgs: 129 active+clean; 3.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 232 KiB/s wr, 15 op/s 2024-07-26T11:03:57.650 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 -- 172.21.15.144:0/3920765410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c1095e0 msgr2=0x7fae1c10fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 --2- 172.21.15.144:0/3920765410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c1095e0 0x7fae1c10fe10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fae10009900 tx=0x7fae1002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 -- 172.21.15.144:0/3920765410 shutdown_connections 2024-07-26T11:03:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 --2- 172.21.15.144:0/3920765410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c1095e0 0x7fae1c10fe10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 --2- 172.21.15.144:0/3920765410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae1c108c90 0x7fae1c1090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 --2- 172.21.15.144:0/3920765410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae1c103360 0x7fae1c103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 -- 172.21.15.144:0/3920765410 >> 172.21.15.144:0/3920765410 conn(0x7fae1c0fedc0 msgr2=0x7fae1c101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 -- 172.21.15.144:0/3920765410 shutdown_connections 2024-07-26T11:03:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.144+0000 7fae22c00640 1 -- 172.21.15.144:0/3920765410 wait complete. 2024-07-26T11:03:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.145+0000 7fae22c00640 1 Processor -- start 2024-07-26T11:03:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.145+0000 7fae22c00640 1 -- start start 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 0x7fae1c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae1c108c90 0x7fae1c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae1c1095e0 0x7fae1c1a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c113300 con 0x7fae1c1095e0 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c113180 con 0x7fae1c103360 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c113480 con 0x7fae1c108c90 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 0x7fae1c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 0x7fae1c19bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59206/0 (socket says 172.21.15.144:59206) 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 -- 172.21.15.144:0/3451812793 learned_addr learned my addr 172.21.15.144:0/3451812793 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae22200640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae1c1095e0 0x7fae1c1a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae20e00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae1c108c90 0x7fae1c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 -- 172.21.15.144:0/3451812793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae1c108c90 msgr2=0x7fae1c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae1c108c90 0x7fae1c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 -- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae1c1095e0 msgr2=0x7fae1c1a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:59.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae1c1095e0 0x7fae1c1a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.146+0000 7fae21800640 1 -- 172.21.15.144:0/3451812793 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c1a1e90 con 0x7fae1c103360 2024-07-26T11:03:59.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.147+0000 7fae21800640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 0x7fae1c19bf60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fae1400eb90 tx=0x7fae1400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:59.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.147+0000 7fae1a000640 1 -- 172.21.15.144:0/3451812793 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14004d20 con 0x7fae1c103360 2024-07-26T11:03:59.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.147+0000 7fae1a000640 1 -- 172.21.15.144:0/3451812793 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae1400ee90 con 0x7fae1c103360 2024-07-26T11:03:59.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.147+0000 7fae1a000640 1 -- 172.21.15.144:0/3451812793 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14010670 con 0x7fae1c103360 2024-07-26T11:03:59.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.147+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c1a2180 con 0x7fae1c103360 2024-07-26T11:03:59.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.147+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c1a8910 con 0x7fae1c103360 2024-07-26T11:03:59.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 Processor -- start 2024-07-26T11:03:59.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae1a000640 1 -- 172.21.15.144:0/3451812793 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fae14005020 con 0x7fae1c103360 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 -- start start 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6fa061b0 0x555a6fda9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdaa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdaa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x555a6fd1e270 con 0x555a6f9e54d0 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555a6fd0d770 con 0x555a6f9dd2f0 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.151+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555a6fd117b0 con 0x555a6fa061b0 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae21800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdaa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae22200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6fa061b0 0x555a6fda9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae20e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdaa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae22200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6fa061b0 0x555a6fda9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48402/0 (socket says 172.21.15.144:48402) 2024-07-26T11:03:59.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae22200640 1 -- 172.21.15.144:0/1506952560 learned_addr learned my addr 172.21.15.144:0/1506952560 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:59.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae21800640 1 -- 172.21.15.144:0/1506952560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6fa061b0 msgr2=0x555a6fda9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae21800640 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6fa061b0 0x555a6fda9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae21800640 1 -- 172.21.15.144:0/1506952560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 msgr2=0x555a6fdaa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae21800640 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdaa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.152+0000 7fae21800640 1 -- 172.21.15.144:0/1506952560 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a6fa71790 con 0x555a6f9dd2f0 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae21800640 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdaa160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fae140e1e60 tx=0x7fae140e8bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a1bf3e119e2a92f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae1a000640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae040b3dc0 0x7fae040b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae0aa00640 1 -- 172.21.15.144:0/1506952560 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae140ed7b0 con 0x555a6f9dd2f0 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae0aa00640 1 -- 172.21.15.144:0/1506952560 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae140f6070 con 0x555a6f9dd2f0 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae0aa00640 1 -- 172.21.15.144:0/1506952560 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae140a79a0 con 0x555a6f9dd2f0 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae22200640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae040b3dc0 0x7fae040b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.153+0000 7fae1a000640 1 -- 172.21.15.144:0/3451812793 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fae140e4ad0 con 0x7fae1c103360 2024-07-26T11:03:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 -- 172.21.15.144:0/1506952560 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 msgr2=0x555a6fdaa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdaa160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fae140e1e60 tx=0x7fae140e8bb0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae22200640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae040b3dc0 0x7fae040b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fae100096c0 tx=0x7fae10009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 -- 172.21.15.144:0/1506952560 shutdown_connections 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdaa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdaa160 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/1506952560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6fa061b0 0x555a6fda9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 -- 172.21.15.144:0/1506952560 >> 172.21.15.144:0/1506952560 conn(0x555a6f9fa300 msgr2=0x555a6fda7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 -- 172.21.15.144:0/1506952560 shutdown_connections 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.154+0000 7fae2b3c6740 1 -- 172.21.15.144:0/1506952560 wait complete. 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.155+0000 7fae2b3c6740 1 -- start start 2024-07-26T11:03:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.155+0000 7fae2b3c6740 1 Processor -- start 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.155+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdb5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.155+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6fa061b0 0x555a6fdc1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x555a6fc47ab0 con 0x555a6fa061b0 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555a6fcaad50 con 0x555a6f9dd2f0 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555a6fdb8340 con 0x555a6f9e54d0 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae2b3c6740 10 client.0 _ll_register_callbacks cb 0x555a6fa1da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdb5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdb5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59232/0 (socket says 172.21.15.144:59232) 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 -- 172.21.15.144:0/949094518 learned_addr learned my addr 172.21.15.144:0/949094518 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae22200640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6fa061b0 0x555a6fdc1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae20e00640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 -- 172.21.15.144:0/949094518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 msgr2=0x555a6fdb5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 -- 172.21.15.144:0/949094518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6fa061b0 msgr2=0x555a6fdc1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6fa061b0 0x555a6fdc1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 -- 172.21.15.144:0/949094518 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a6faea620 con 0x555a6f9dd2f0 2024-07-26T11:03:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fae21800640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdb5780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae140a85c0 tx=0x7fae140f1860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9284798a24bcef0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.156+0000 7fadff400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fadff400640 1 -- 172.21.15.144:0/949094518 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae140aaca0 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae140f6070 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fadff400640 1 -- 172.21.15.144:0/949094518 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fae140b3b70 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fadff400640 1 -- 172.21.15.144:0/949094518 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae140aae90 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 10 client.25727 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555a6fdc4300 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fadff400640 1 -- 172.21.15.144:0/949094518 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae140f6070 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fadff400640 1 -- 172.21.15.144:0/949094518 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fae140e2110 con 0x555a6f9dd2f0 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 10 client.25727 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 10 client.25727 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 20 client.25727 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f9c4d80 0x555a6fdc7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 4 client.25727 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:03:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.157+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a6fd201c0 con 0x555a6f9c4d80 2024-07-26T11:03:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.158+0000 7fae20e00640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f9c4d80 0x555a6fdc7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.158+0000 7fae20e00640 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f9c4d80 0x555a6fdc7660 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a832664a68ed2fc server_cookie=d464039292c06a32 in_seq=0 out_seq=0 2024-07-26T11:03:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.158+0000 7fadff400640 10 client.25727 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fadff400640 1 -- 172.21.15.144:0/949094518 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555a6fd201c0 con 0x555a6f9c4d80 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fadff400640 10 client.25727 handle_command_reply: tid=0 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 client.25727 shutdown 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 msgr2=0x555a6fdb5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdb5780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae140a85c0 tx=0x7fae140f1860 comp rx=0 tx=0).stop 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 shutdown_connections 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f9c4d80 0x555a6fdc7660 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6fa061b0 0x555a6fdc1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/949094518 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6f9dd2f0 0x555a6fdb5780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fadff400640 0 client.25727 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 >> 172.21.15.144:0/949094518 conn(0x555a6f9fa300 msgr2=0x555a6fdaac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 shutdown_connections 2024-07-26T11:03:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 1 -- 172.21.15.144:0/949094518 wait complete. 2024-07-26T11:03:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.160+0000 7fae2b3c6740 20 client.25727 trim_cache size 0 max 16384 2024-07-26T11:03:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 Processor -- start 2024-07-26T11:03:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 -- start start 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9dd2f0 0x555a6fdafcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fdca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1b5260 con 0x555a6f9dd2f0 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1b2b30 con 0x555a6fa061b0 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.180+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1b0980 con 0x555a6f9e54d0 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fdca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fdca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59242/0 (socket says 172.21.15.144:59242) 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae21800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9dd2f0 0x555a6fdafcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 -- 172.21.15.144:0/3244582679 learned_addr learned my addr 172.21.15.144:0/3244582679 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae20e00640 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 -- 172.21.15.144:0/3244582679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 msgr2=0x555a6fdb7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 -- 172.21.15.144:0/3244582679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9dd2f0 msgr2=0x555a6fdafcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9dd2f0 0x555a6fdafcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 -- 172.21.15.144:0/3244582679 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a6fdcd840 con 0x555a6fa061b0 2024-07-26T11:03:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae20e00640 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.181+0000 7fae22200640 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fdca690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fae10009900 tx=0x7fae1003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e6a4c2a0fb4d92c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.182+0000 7fae0a000640 1 -- 172.21.15.144:0/3244582679 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10035040 con 0x555a6fa061b0 2024-07-26T11:03:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.182+0000 7fae0a000640 1 -- 172.21.15.144:0/3244582679 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae10004e30 con 0x555a6fa061b0 2024-07-26T11:03:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.182+0000 7fae0a000640 1 -- 172.21.15.144:0/3244582679 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10032710 con 0x555a6fa061b0 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3244582679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 msgr2=0x555a6fdca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fdca690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fae10009900 tx=0x7fae1003e000 comp rx=0 tx=0).stop 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3244582679 shutdown_connections 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fdca690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9e54d0 0x555a6fdb7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3244582679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9dd2f0 0x555a6fdafcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3244582679 >> 172.21.15.144:0/3244582679 conn(0x555a6f9fa300 msgr2=0x555a6fdc0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3244582679 shutdown_connections 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3244582679 wait complete. 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.183+0000 7fae2b3c6740 1 -- start start 2024-07-26T11:03:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 Processor -- start 2024-07-26T11:03:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9dd2f0 0x555a6fdc99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdc9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fe2dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1b0ff0 con 0x555a6f9e54d0 2024-07-26T11:03:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1b2460 con 0x555a6fa061b0 2024-07-26T11:03:59.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1b16c0 con 0x555a6f9dd2f0 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae2b3c6740 10 client.0 _ll_register_callbacks cb 0x7fae0c00d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae20e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdc9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae20e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdc9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37726/0 (socket says 172.21.15.144:37726) 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae20e00640 1 -- 172.21.15.144:0/3123936900 learned_addr learned my addr 172.21.15.144:0/3123936900 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.184+0000 7fae22200640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fe2dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae21800640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9dd2f0 0x555a6fdc99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae22200640 1 -- 172.21.15.144:0/3123936900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9dd2f0 msgr2=0x555a6fdc99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae22200640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9dd2f0 0x555a6fdc99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae22200640 1 -- 172.21.15.144:0/3123936900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 msgr2=0x555a6fdc9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae22200640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdc9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae22200640 1 -- 172.21.15.144:0/3123936900 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a6fe2e0f0 con 0x555a6fa061b0 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae20e00640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdc9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fae22200640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fe2dbb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae1002a910 tx=0x7fae10003c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eabcdb19feb9666f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fadff400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.185+0000 7fadff400640 1 -- 172.21.15.144:0/3123936900 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10060060 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae10004e30 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fadff400640 1 -- 172.21.15.144:0/3123936900 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fae10004b90 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fadff400640 1 -- 172.21.15.144:0/3123936900 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae10067050 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 10 client.25733 fetch_fsmap learned FSMap version 26 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555a6fe2f280 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fadff400640 1 -- 172.21.15.144:0/3123936900 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae10004e30 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fadff400640 1 -- 172.21.15.144:0/3123936900 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fae1005a030 con 0x555a6fa061b0 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 10 client.25733 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 20 client.25733 populate_metadata read hostname 'smithi144' 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f99a1c0 0x555a6fe319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 4 client.25733 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:03:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.186+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a6fd1bc60 con 0x555a6f99a1c0 2024-07-26T11:03:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.187+0000 7fae21800640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f99a1c0 0x555a6fe319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.187+0000 7fadff400640 10 client.25733 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:03:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:03:59.187+0000 7fae21800640 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f99a1c0 0x555a6fe319d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e64c68e9abe5c3b server_cookie=fcf91de21a516099 in_seq=0 out_seq=0 2024-07-26T11:03:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:03:59 smithi144 ceph-mon[32129]: pgmap v506: 129 pgs: 129 active+clean; 12 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 956 KiB/s wr, 66 op/s 2024-07-26T11:03:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:03:59 smithi160 ceph-mon[36715]: pgmap v506: 129 pgs: 129 active+clean; 12 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 956 KiB/s wr, 66 op/s 2024-07-26T11:03:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:03:59 smithi183 ceph-mon[37273]: pgmap v506: 129 pgs: 129 active+clean; 12 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 956 KiB/s wr, 66 op/s 2024-07-26T11:04:00.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 -- 172.21.15.144:0/4229572893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee0109df0 msgr2=0x7f5ee010a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/4229572893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee0109df0 0x7f5ee010a200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009900 tx=0x7f5ed002a410 comp rx=0 tx=0).stop 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 -- 172.21.15.144:0/4229572893 shutdown_connections 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/4229572893 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ee0118330 0x7f5ee011a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/4229572893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee010a740 0x7f5ee0117df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/4229572893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee0109df0 0x7f5ee010a200 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.209+0000 7f5ee4a00640 1 -- 172.21.15.144:0/4229572893 >> 172.21.15.144:0/4229572893 conn(0x7f5ee00708f0 msgr2=0x7f5ee0070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.210+0000 7f5ee4a00640 1 -- 172.21.15.144:0/4229572893 shutdown_connections 2024-07-26T11:04:00.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.210+0000 7f5ee4a00640 1 -- 172.21.15.144:0/4229572893 wait complete. 2024-07-26T11:04:00.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.212+0000 7f5ee4a00640 1 Processor -- start 2024-07-26T11:04:00.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.213+0000 7f5ee4a00640 1 -- start start 2024-07-26T11:04:00.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.214+0000 7f5ee4a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee0109df0 0x7f5ee0116e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.215+0000 7f5ee4a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 0x7f5ee01173f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.215+0000 7f5ee4a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ee0118330 0x7f5ee01ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.215+0000 7f5edd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 0x7f5ee01173f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.217 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.215+0000 7f5edd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 0x7f5ee01173f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55678/0 (socket says 172.21.15.144:55678) 2024-07-26T11:04:00.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.216+0000 7f5ee4a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee010eba0 con 0x7f5ee0109df0 2024-07-26T11:04:00.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.216+0000 7f5ee4a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee010ea20 con 0x7f5ee0118330 2024-07-26T11:04:00.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.216+0000 7f5ee4a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee010ed20 con 0x7f5ee010a740 2024-07-26T11:04:00.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.217+0000 7f5edd400640 1 -- 172.21.15.144:0/1111969662 learned_addr learned my addr 172.21.15.144:0/1111969662 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:00.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.217+0000 7f5ede800640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ee0118330 0x7f5ee01ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.217+0000 7f5edde00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee0109df0 0x7f5ee0116e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.220 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.218+0000 7f5edd400640 1 -- 172.21.15.144:0/1111969662 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ee0118330 msgr2=0x7f5ee01ac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.220 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.218+0000 7f5edd400640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ee0118330 0x7f5ee01ac460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.220 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.218+0000 7f5edd400640 1 -- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee0109df0 msgr2=0x7f5ee0116e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.221 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.218+0000 7f5edd400640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee0109df0 0x7f5ee0116e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.221 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.218+0000 7f5edd400640 1 -- 172.21.15.144:0/1111969662 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee01acb90 con 0x7f5ee010a740 2024-07-26T11:04:00.221 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.218+0000 7f5edde00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee0109df0 0x7f5ee0116e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:04:00.221 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.219+0000 7f5edd400640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 0x7f5ee01173f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006fd0 tx=0x7f5ecc00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:00.221 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.219+0000 7f5ed6a00640 1 -- 172.21.15.144:0/1111969662 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011830 con 0x7f5ee010a740 2024-07-26T11:04:00.222 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.219+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee01ace80 con 0x7f5ee010a740 2024-07-26T11:04:00.222 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.220+0000 7f5ed6a00640 1 -- 172.21.15.144:0/1111969662 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ecc016070 con 0x7f5ee010a740 2024-07-26T11:04:00.222 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.220+0000 7f5ed6a00640 1 -- 172.21.15.144:0/1111969662 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc010e00 con 0x7f5ee010a740 2024-07-26T11:04:00.222 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.220+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee01b57d0 con 0x7f5ee010a740 2024-07-26T11:04:00.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.225+0000 7f5eeb6c9740 1 Processor -- start 2024-07-26T11:04:00.227 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.225+0000 7f5eeb6c9740 1 -- start start 2024-07-26T11:04:00.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.226+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927da4070 0x564928148e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d7b1b0 0x564928149370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 0x5649281498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5649280bfbd0 con 0x564927da4070 2024-07-26T11:04:00.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5649280af0d0 con 0x564927d7b1b0 2024-07-26T11:04:00.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5649280b3110 con 0x564927d83390 2024-07-26T11:04:00.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5ede800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927da4070 0x564928148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5ede800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927da4070 0x564928148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33800/0 (socket says 172.21.15.144:33800) 2024-07-26T11:04:00.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5ede800640 1 -- 172.21.15.144:0/2371679025 learned_addr learned my addr 172.21.15.144:0/2371679025 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:00.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.227+0000 7f5edde00640 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d7b1b0 0x564928149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5edde00640 1 -- 172.21.15.144:0/2371679025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 msgr2=0x5649281498b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:04:00.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5edde00640 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 0x5649281498b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5edde00640 1 -- 172.21.15.144:0/2371679025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927da4070 msgr2=0x564928148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:00.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5edde00640 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927da4070 0x564928148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5edde00640 1 -- 172.21.15.144:0/2371679025 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564927e0f650 con 0x564927d7b1b0 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5ed6a00640 1 -- 172.21.15.144:0/1111969662 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5ecc01b480 con 0x7f5ee010a740 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.228+0000 7f5edde00640 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d7b1b0 0x564928149370 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0005ab0 tx=0x7f5ed0004110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f550e5c14f1a9a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.229+0000 7f5ecb000640 1 -- 172.21.15.144:0/2371679025 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed003f070 con 0x564927d7b1b0 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.229+0000 7f5ecb000640 1 -- 172.21.15.144:0/2371679025 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed00054f0 con 0x564927d7b1b0 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.229+0000 7f5ecb000640 1 -- 172.21.15.144:0/2371679025 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0035030 con 0x564927d7b1b0 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.230+0000 7f5ed6a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5ec40b3df0 0x7f5ec40b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.230+0000 7f5ed6a00640 1 -- 172.21.15.144:0/1111969662 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f5ecc0e6170 con 0x7f5ee010a740 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2371679025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d7b1b0 msgr2=0x564928149370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d7b1b0 0x564928149370 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0005ab0 tx=0x7f5ed0004110 comp rx=0 tx=0).stop 2024-07-26T11:04:00.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2371679025 shutdown_connections 2024-07-26T11:04:00.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 0x5649281498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d7b1b0 0x564928149370 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0005ab0 tx=0x7f5ed0004110 comp rx=0 tx=0).stop 2024-07-26T11:04:00.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2371679025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927da4070 0x564928148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.231+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2371679025 >> 172.21.15.144:0/2371679025 conn(0x564927d98460 msgr2=0x564928146d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.232+0000 7f5ede800640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5ec40b3df0 0x7f5ec40b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.233+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2371679025 shutdown_connections 2024-07-26T11:04:00.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.233+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2371679025 wait complete. 2024-07-26T11:04:00.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.234+0000 7f5eeb6c9740 1 -- start start 2024-07-26T11:04:00.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.234+0000 7f5eeb6c9740 1 Processor -- start 2024-07-26T11:04:00.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.234+0000 7f5ede800640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5ec40b3df0 0x7f5ec40b6300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8011630 tx=0x7f5ed8009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:00.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 0x564928154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 0x564928154a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x5649281608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564927f12700 con 0x564927d629f0 2024-07-26T11:04:00.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56492804b6a0 con 0x564927da4070 2024-07-26T11:04:00.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5649281575b0 con 0x564927d83390 2024-07-26T11:04:00.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5edd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 0x564928154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 0x564928154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51226/0 (socket says 172.21.15.144:51226) 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 -- 172.21.15.144:0/3257021276 learned_addr learned my addr 172.21.15.144:0/3257021276 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5ede800640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x5649281608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.235+0000 7f5eeb6c9740 10 client.0 _ll_register_callbacks cb 0x56492805b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 -- 172.21.15.144:0/3257021276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 msgr2=0x564928154530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 0x564928154530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 -- 172.21.15.144:0/3257021276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 msgr2=0x5649281608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x5649281608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 -- 172.21.15.144:0/3257021276 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564928062840 con 0x564927da4070 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5ede800640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x5649281608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5edd400640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 0x564928154a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009fd0 tx=0x7f5ecc0047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bdbc95585915ea1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.236+0000 7f5ebf400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.237+0000 7f5ebf400640 1 -- 172.21.15.144:0/3257021276 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc0e7060 con 0x564927da4070 2024-07-26T11:04:00.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.237+0000 7f5ebf400640 1 -- 172.21.15.144:0/3257021276 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f5ecc0e06a0 con 0x564927da4070 2024-07-26T11:04:00.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.237+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ed00054f0 con 0x564927da4070 2024-07-26T11:04:00.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.238+0000 7f5ebf400640 1 -- 172.21.15.144:0/3257021276 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ecc0c2020 con 0x564927da4070 2024-07-26T11:04:00.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.238+0000 7f5eeb6c9740 10 client.25745 fetch_fsmap learned FSMap version 26 2024-07-26T11:04:00.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.238+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5649281743e0 con 0x564927da4070 2024-07-26T11:04:00.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.239+0000 7f5ebf400640 1 -- 172.21.15.144:0/3257021276 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed00054f0 con 0x564927da4070 2024-07-26T11:04:00.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.239+0000 7f5ebf400640 1 -- 172.21.15.144:0/3257021276 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5ecc0a5c80 con 0x564927da4070 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.242+0000 7f5eeb6c9740 10 client.25745 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.242+0000 7f5eeb6c9740 10 client.25745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.242+0000 7f5eeb6c9740 20 client.25745 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.243+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927d37da0 0x564928177910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.243+0000 7f5eeb6c9740 4 client.25745 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.243+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649280c1b20 con 0x564927d37da0 2024-07-26T11:04:00.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.243+0000 7f5edde00640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927d37da0 0x564928177910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.244+0000 7f5edde00640 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927d37da0 0x564928177910 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f836a46035ecb623 server_cookie=23025ca1d4161264 in_seq=0 out_seq=0 2024-07-26T11:04:00.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.244+0000 7f5ebf400640 10 client.25745 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:00.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.245+0000 7f5ebf400640 1 -- 172.21.15.144:0/3257021276 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649280c1b20 con 0x564927d37da0 2024-07-26T11:04:00.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.245+0000 7f5ebf400640 10 client.25745 handle_command_reply: tid=0 2024-07-26T11:04:00.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.245+0000 7f5eeb6c9740 1 client.25745 shutdown 2024-07-26T11:04:00.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.246+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 msgr2=0x564928154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.246+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 0x564928154a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009fd0 tx=0x7f5ecc0047d0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.246+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 shutdown_connections 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927d37da0 0x564928177910 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x5649281608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5ebf400640 0 client.25745 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927da4070 0x564928154a70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/3257021276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d83390 0x564928154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 >> 172.21.15.144:0/3257021276 conn(0x564927d98460 msgr2=0x564928149e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.247+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 shutdown_connections 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.248+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/3257021276 wait complete. 2024-07-26T11:04:00.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.248+0000 7f5eeb6c9740 20 client.25745 trim_cache size 0 max 16384 2024-07-26T11:04:00.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.279+0000 7f5eeb6c9740 1 Processor -- start 2024-07-26T11:04:00.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.279+0000 7f5eeb6c9740 1 -- start start 2024-07-26T11:04:00.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.280+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d37da0 0x56492814fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.280+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x564928156160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.280+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928160690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.280+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee01b99f0 con 0x564927d629f0 2024-07-26T11:04:00.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.280+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee01b72c0 con 0x564927d83390 2024-07-26T11:04:00.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.280+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee0117f80 con 0x564927d37da0 2024-07-26T11:04:00.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.281+0000 7f5ede800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.281+0000 7f5ede800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51232/0 (socket says 172.21.15.144:51232) 2024-07-26T11:04:00.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.281+0000 7f5ede800640 1 -- 172.21.15.144:0/2881158259 learned_addr learned my addr 172.21.15.144:0/2881158259 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:00.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.281+0000 7f5edd400640 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x564928156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.282+0000 7f5ede800640 1 -- 172.21.15.144:0/2881158259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d37da0 msgr2=0x56492814fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:00.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.282+0000 7f5ede800640 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d37da0 0x56492814fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.282+0000 7f5ede800640 1 -- 172.21.15.144:0/2881158259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 msgr2=0x564928156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:00.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.282+0000 7f5ede800640 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x564928156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.282+0000 7f5ede800640 1 -- 172.21.15.144:0/2881158259 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492814b350 con 0x564927d83390 2024-07-26T11:04:00.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.283+0000 7f5ede800640 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928160690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009420 tx=0x7f5ed8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43fb490d3d06947 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:00.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.284+0000 7f5eca600640 1 -- 172.21.15.144:0/2881158259 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed80045c0 con 0x564927d83390 2024-07-26T11:04:00.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.284+0000 7f5eca600640 1 -- 172.21.15.144:0/2881158259 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed8004760 con 0x564927d83390 2024-07-26T11:04:00.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.284+0000 7f5eca600640 1 -- 172.21.15.144:0/2881158259 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed801e590 con 0x564927d83390 2024-07-26T11:04:00.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.285+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2881158259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 msgr2=0x564928160690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.285+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928160690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009420 tx=0x7f5ed8016000 comp rx=0 tx=0).stop 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2881158259 shutdown_connections 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928160690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d629f0 0x564928156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/2881158259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d37da0 0x56492814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2881158259 >> 172.21.15.144:0/2881158259 conn(0x564927d98460 msgr2=0x564928155a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2881158259 shutdown_connections 2024-07-26T11:04:00.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/2881158259 wait complete. 2024-07-26T11:04:00.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.286+0000 7f5eeb6c9740 1 -- start start 2024-07-26T11:04:00.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.287+0000 7f5eeb6c9740 1 Processor -- start 2024-07-26T11:04:00.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.288+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d37da0 0x5649281d8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.288+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 0x5649281d89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.288+0000 7f5eeb6c9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee0125330 con 0x564927d37da0 2024-07-26T11:04:00.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee01b6bf0 con 0x564927d83390 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5eeb6c9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee01b5e50 con 0x564927d629f0 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5eeb6c9740 10 client.0 _ll_register_callbacks cb 0x7f5ecc0a9340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5edd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 0x5649281d89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5edd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 0x5649281d89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55696/0 (socket says 172.21.15.144:55696) 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5edde00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d37da0 0x5649281d8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5edd400640 1 -- 172.21.15.144:0/1580427314 learned_addr learned my addr 172.21.15.144:0/1580427314 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:00.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.289+0000 7f5ede800640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.290+0000 7f5edde00640 1 -- 172.21.15.144:0/1580427314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 msgr2=0x5649281d89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:00.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.290+0000 7f5edde00640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 0x5649281d89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.290+0000 7f5edde00640 1 -- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 msgr2=0x564928157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:00.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.290+0000 7f5edde00640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.290+0000 7f5edde00640 1 -- 172.21.15.144:0/1580427314 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564928158660 con 0x564927d37da0 2024-07-26T11:04:00.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.290+0000 7f5edd400640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 0x5649281d89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:00.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.291+0000 7f5edde00640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d37da0 0x5649281d8410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x56492814bb30 tx=0x7f5ed002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbd8ec738ddd9ed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:00.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.291+0000 7f5ebf400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:04:00.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.293+0000 7f5ebf400640 1 -- 172.21.15.144:0/1580427314 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0035030 con 0x564927d37da0 2024-07-26T11:04:00.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.293+0000 7f5ebf400640 1 -- 172.21.15.144:0/1580427314 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f5ed00035f0 con 0x564927d37da0 2024-07-26T11:04:00.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.293+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ed8004760 con 0x564927d37da0 2024-07-26T11:04:00.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.296+0000 7f5ebf400640 1 -- 172.21.15.144:0/1580427314 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ed005a050 con 0x564927d37da0 2024-07-26T11:04:00.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.296+0000 7f5eeb6c9740 10 client.15606 fetch_fsmap learned FSMap version 26 2024-07-26T11:04:00.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.296+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5649281dfc20 con 0x564927d37da0 2024-07-26T11:04:00.298 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5ebf400640 1 -- 172.21.15.144:0/1580427314 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed8004760 con 0x564927d37da0 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5ebf400640 1 -- 172.21.15.144:0/1580427314 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5ed0004e00 con 0x564927d37da0 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5eeb6c9740 10 client.15606 fetch_fsmap finished waiting for FSMap version 26 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5eeb6c9740 10 client.15606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5eeb6c9740 20 client.15606 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927da4070 0x5649281e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5eeb6c9740 4 client.15606 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.297+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649280bd5c0 con 0x564927da4070 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.298+0000 7f5edd400640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927da4070 0x5649281e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.299+0000 7f5edd400640 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927da4070 0x5649281e1ec0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7132b5ad2ba1c75d server_cookie=85db35db5deef726 in_seq=0 out_seq=0 2024-07-26T11:04:00.301 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.299+0000 7f5ebf400640 10 client.15606 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:00.302 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5ebf400640 1 -- 172.21.15.144:0/1580427314 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649280bd5c0 con 0x564927da4070 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5ebf400640 10 client.15606 handle_command_reply: tid=0 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 client.15606 shutdown 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d37da0 msgr2=0x5649281d8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d37da0 0x5649281d8410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x56492814bb30 tx=0x7f5ed002a410 comp rx=0 tx=0).stop 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 shutdown_connections 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564927da4070 0x5649281e1ec0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564927d83390 0x564928157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564927d629f0 0x5649281d89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 --2- 172.21.15.144:0/1580427314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564927d37da0 0x5649281d8410 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5ebf400640 0 client.15606 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:00.303 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.300+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 >> 172.21.15.144:0/1580427314 conn(0x564927d98460 msgr2=0x564928156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.301+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 shutdown_connections 2024-07-26T11:04:00.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.301+0000 7f5eeb6c9740 1 -- 172.21.15.144:0/1580427314 wait complete. 2024-07-26T11:04:00.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.302+0000 7f5eeb6c9740 20 client.15606 trim_cache size 0 max 16384 2024-07-26T11:04:00.304 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:04:00.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5ec40b3df0 msgr2=0x7f5ec40b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5ec40b3df0 0x7f5ec40b6300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8011630 tx=0x7f5ed8009500 comp rx=0 tx=0).stop 2024-07-26T11:04:00.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 msgr2=0x7f5ee01173f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 0x7f5ee01173f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006fd0 tx=0x7f5ecc00b8e0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 shutdown_connections 2024-07-26T11:04:00.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f5ec40b3df0 0x7f5ec40b6300 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.305 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ee0118330 0x7f5ee01ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee010a740 0x7f5ee01173f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 --2- 172.21.15.144:0/1111969662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ee0109df0 0x7f5ee0116e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.303+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 >> 172.21.15.144:0/1111969662 conn(0x7f5ee00708f0 msgr2=0x7f5ee0118880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.304+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 shutdown_connections 2024-07-26T11:04:00.306 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:00.304+0000 7f5ee4a00640 1 -- 172.21.15.144:0/1111969662 wait complete. 2024-07-26T11:04:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.658+0000 7fadff400640 1 -- 172.21.15.144:0/3123936900 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a6fd1bc60 con 0x555a6f99a1c0 2024-07-26T11:04:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.658+0000 7fadff400640 10 client.25733 handle_command_reply: tid=0 2024-07-26T11:04:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.658+0000 7fae2b3c6740 1 client.25733 shutdown 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 msgr2=0x555a6fe2dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fe2dbb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae1002a910 tx=0x7fae10003c40 comp rx=0 tx=0).stop 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 shutdown_connections 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x555a6f99a1c0 0x555a6fe319d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555a6fa061b0 0x555a6fe2dbb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555a6f9e54d0 0x555a6fdc9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 --2- 172.21.15.144:0/3123936900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a6f9dd2f0 0x555a6fdc99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 >> 172.21.15.144:0/3123936900 conn(0x555a6f9fa300 msgr2=0x555a6fdb7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fadff400640 0 client.25733 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 shutdown_connections 2024-07-26T11:04:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 1 -- 172.21.15.144:0/3123936900 wait complete. 2024-07-26T11:04:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.659+0000 7fae2b3c6740 20 client.25733 trim_cache size 0 max 16384 2024-07-26T11:04:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:04:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:04:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:04:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:04:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae040b3dc0 msgr2=0x7fae040b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae040b3dc0 0x7fae040b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fae100096c0 tx=0x7fae10009560 comp rx=0 tx=0).stop 2024-07-26T11:04:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 msgr2=0x7fae1c19bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 0x7fae1c19bf60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fae1400eb90 tx=0x7fae1400c6a0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 shutdown_connections 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fae040b3dc0 0x7fae040b62d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae1c1095e0 0x7fae1c1a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae1c108c90 0x7fae1c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 --2- 172.21.15.144:0/3451812793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fae1c103360 0x7fae1c19bf60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.660+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 >> 172.21.15.144:0/3451812793 conn(0x7fae1c0fedc0 msgr2=0x7fae1c106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.661+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 shutdown_connections 2024-07-26T11:04:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:00.661+0000 7fae22c00640 1 -- 172.21.15.144:0/3451812793 wait complete. 2024-07-26T11:04:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:01 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:04:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:01 smithi144 ceph-mon[32129]: pgmap v507: 129 pgs: 129 active+clean; 20 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.7 MiB/s wr, 381 op/s 2024-07-26T11:04:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:01 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:04:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:01 smithi160 ceph-mon[36715]: pgmap v507: 129 pgs: 129 active+clean; 20 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.7 MiB/s wr, 381 op/s 2024-07-26T11:04:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:01 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:04:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:01 smithi183 ceph-mon[37273]: pgmap v507: 129 pgs: 129 active+clean; 20 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.7 MiB/s wr, 381 op/s 2024-07-26T11:04:02.272 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:04:02.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:04:02.750 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:04:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:03 smithi144 ceph-mon[32129]: pgmap v508: 129 pgs: 129 active+clean; 29 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.6 MiB/s wr, 739 op/s 2024-07-26T11:04:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:03 smithi160 ceph-mon[36715]: pgmap v508: 129 pgs: 129 active+clean; 29 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.6 MiB/s wr, 739 op/s 2024-07-26T11:04:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:03 smithi183 ceph-mon[37273]: pgmap v508: 129 pgs: 129 active+clean; 29 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.6 MiB/s wr, 739 op/s 2024-07-26T11:04:05.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 -- 172.21.15.144:0/1900997112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c41027c0 msgr2=0x7fe5c410f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1900997112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c41027c0 0x7fe5c410f4e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009900 tx=0x7fe5b002a480 comp rx=0 tx=0).stop 2024-07-26T11:04:05.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 -- 172.21.15.144:0/1900997112 shutdown_connections 2024-07-26T11:04:05.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1900997112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c410fa20 0x7fe5c4111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1900997112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c41027c0 0x7fe5c410f4e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1900997112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c4101e70 0x7fe5c4102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.027+0000 7fe5ca600640 1 -- 172.21.15.144:0/1900997112 >> 172.21.15.144:0/1900997112 conn(0x7fe5c40fdcf0 msgr2=0x7fe5c4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:05.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.028+0000 7fe5ca600640 1 -- 172.21.15.144:0/1900997112 shutdown_connections 2024-07-26T11:04:05.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.028+0000 7fe5ca600640 1 -- 172.21.15.144:0/1900997112 wait complete. 2024-07-26T11:04:05.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.029+0000 7fe5ca600640 1 Processor -- start 2024-07-26T11:04:05.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 -- start start 2024-07-26T11:04:05.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 0x7fe5c419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c41027c0 0x7fe5c419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c410fa20 0x7fe5c41a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41154c0 con 0x7fe5c410fa20 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4115340 con 0x7fe5c4101e70 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5ca600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4115640 con 0x7fe5c41027c0 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5c9200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 0x7fe5c419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5c9200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 0x7fe5c419bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51256/0 (socket says 172.21.15.144:51256) 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.030+0000 7fe5c9200640 1 -- 172.21.15.144:0/1338349544 learned_addr learned my addr 172.21.15.144:0/1338349544 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c410fa20 0x7fe5c41a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c41027c0 0x7fe5c419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9200640 1 -- 172.21.15.144:0/1338349544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c41027c0 msgr2=0x7fe5c419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9200640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c41027c0 0x7fe5c419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9200640 1 -- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c410fa20 msgr2=0x7fe5c41a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9200640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c410fa20 0x7fe5c41a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9200640 1 -- 172.21.15.144:0/1338349544 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c41a1ee0 con 0x7fe5c4101e70 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c41027c0 0x7fe5c419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c9200640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 0x7fe5c419bf70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800e9f0 tx=0x7fe5b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.031+0000 7fe5c1600640 1 -- 172.21.15.144:0/1338349544 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b800cd80 con 0x7fe5c4101e70 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.032+0000 7fe5c1600640 1 -- 172.21.15.144:0/1338349544 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5b80108e0 con 0x7fe5c4101e70 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.032+0000 7fe5c1600640 1 -- 172.21.15.144:0/1338349544 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8018980 con 0x7fe5c4101e70 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.032+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c41a21d0 con 0x7fe5c4101e70 2024-07-26T11:04:05.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.032+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c41a8930 con 0x7fe5c4101e70 2024-07-26T11:04:05.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.035+0000 7fe5c1600640 1 -- 172.21.15.144:0/1338349544 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fe5b8010430 con 0x7fe5c4101e70 2024-07-26T11:04:05.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 Processor -- start 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 -- start start 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695e1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695e24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695e29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55ce69558bf0 con 0x55ce69214490 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ce695480f0 con 0x55ce6923d350 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5d2848740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ce6954c130 con 0x55ce6921c670 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.036+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695e1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695e1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51268/0 (socket says 172.21.15.144:51268) 2024-07-26T11:04:05.039 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 -- 172.21.15.144:0/4229703828 learned_addr learned my addr 172.21.15.144:0/4229703828 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:05.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695e29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9200640 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695e24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 -- 172.21.15.144:0/4229703828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 msgr2=0x55ce695e29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695e29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 -- 172.21.15.144:0/4229703828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 msgr2=0x55ce695e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695e24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.037+0000 7fe5c9c00640 1 -- 172.21.15.144:0/4229703828 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce692a8930 con 0x55ce6923d350 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5c1600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe5ac0b3dc0 0x7fe5ac0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695e1f60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009f20 tx=0x7fe5bc00f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795c852aece54429 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5c9200640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe5ac0b3dc0 0x7fe5ac0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5c1600640 1 -- 172.21.15.144:0/1338349544 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fe5b80e47d0 con 0x7fe5c4101e70 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5b6000640 1 -- 172.21.15.144:0/4229703828 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc015070 con 0x55ce6923d350 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5b6000640 1 -- 172.21.15.144:0/4229703828 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5bc01ab10 con 0x55ce6923d350 2024-07-26T11:04:05.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.038+0000 7fe5b6000640 1 -- 172.21.15.144:0/4229703828 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc011c20 con 0x55ce6923d350 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 -- 172.21.15.144:0/4229703828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 msgr2=0x55ce695e1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695e1f60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009f20 tx=0x7fe5bc00f7f0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5c9200640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe5ac0b3dc0 0x7fe5ac0b62d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55ce695e0be0 tx=0x7fe5b80ea530 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 -- 172.21.15.144:0/4229703828 shutdown_connections 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695e29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695e24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4229703828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695e1f60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 -- 172.21.15.144:0/4229703828 >> 172.21.15.144:0/4229703828 conn(0x55ce69231740 msgr2=0x55ce695dfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:05.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 -- 172.21.15.144:0/4229703828 shutdown_connections 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.039+0000 7fe5d2848740 1 -- 172.21.15.144:0/4229703828 wait complete. 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.040+0000 7fe5d2848740 1 -- start start 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.040+0000 7fe5d2848740 1 Processor -- start 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce69214490 0x55ce695ed710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695edc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6923d350 0x55ce695f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55ce693ab890 con 0x55ce6923d350 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ce694e4330 con 0x55ce69214490 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ce695f0740 con 0x55ce6921c670 2024-07-26T11:04:05.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5d2848740 10 client.0 _ll_register_callbacks cb 0x55ce69510010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce69214490 0x55ce695ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce69214490 0x55ce695ed710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51280/0 (socket says 172.21.15.144:51280) 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695edc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55736/0 (socket says 172.21.15.144:55736) 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5c9c00640 1 -- 172.21.15.144:0/4116587126 learned_addr learned my addr 172.21.15.144:0/4116587126 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.041+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6923d350 0x55ce695f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c3e00640 1 -- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce69214490 msgr2=0x55ce695ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce69214490 0x55ce695ed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c3e00640 1 -- 172.21.15.144:0/4116587126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6923d350 msgr2=0x55ce695f9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6923d350 0x55ce695f9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c3e00640 1 -- 172.21.15.144:0/4116587126 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce695ee540 con 0x55ce6921c670 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6923d350 0x55ce695f9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695edc50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5b002aa70 tx=0x7fe5b0005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4174d93fb71eec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.042+0000 7fe5aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.043+0000 7fe5aaa00640 1 -- 172.21.15.144:0/4116587126 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0035040 con 0x55ce6921c670 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.043+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5bc01ab10 con 0x55ce6921c670 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.043+0000 7fe5aaa00640 1 -- 172.21.15.144:0/4116587126 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fe5b00048c0 con 0x55ce6921c670 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.043+0000 7fe5aaa00640 1 -- 172.21.15.144:0/4116587126 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5b0004ac0 con 0x55ce6921c670 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.043+0000 7fe5d2848740 10 client.25738 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.043+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce695fc720 con 0x55ce6921c670 2024-07-26T11:04:05.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5aaa00640 1 -- 172.21.15.144:0/4116587126 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5bc01ab10 con 0x55ce6921c670 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5aaa00640 1 -- 172.21.15.144:0/4116587126 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe5b00314a0 con 0x55ce6921c670 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5d2848740 10 client.25738 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5d2848740 10 client.25738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5d2848740 20 client.25738 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691fbcd0 0x55ce695ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5d2848740 4 client.25738 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce6955ab40 con 0x55ce691fbcd0 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.044+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691fbcd0 0x55ce695ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.045+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691fbcd0 0x55ce695ffbe0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d5487711bbf4f95 server_cookie=6b10a1bb759368c4 in_seq=0 out_seq=0 2024-07-26T11:04:05.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.045+0000 7fe5aaa00640 10 client.25738 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:05.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.046+0000 7fe5aaa00640 1 -- 172.21.15.144:0/4116587126 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ce6955ab40 con 0x55ce691fbcd0 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.046+0000 7fe5aaa00640 10 client.25738 handle_command_reply: tid=0 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.046+0000 7fe5d2848740 1 client.25738 shutdown 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.046+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 msgr2=0x55ce695edc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.046+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695edc50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5b002aa70 tx=0x7fe5b0005270 comp rx=0 tx=0).stop 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 shutdown_connections 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691fbcd0 0x55ce695ffbe0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6923d350 0x55ce695f9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695edc50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 --2- 172.21.15.144:0/4116587126 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce69214490 0x55ce695ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5aaa00640 0 client.25738 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:05.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 >> 172.21.15.144:0/4116587126 conn(0x55ce69231740 msgr2=0x55ce695e5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:05.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 shutdown_connections 2024-07-26T11:04:05.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 1 -- 172.21.15.144:0/4116587126 wait complete. 2024-07-26T11:04:05.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.047+0000 7fe5d2848740 20 client.25738 trim_cache size 0 max 16384 2024-07-26T11:04:05.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 Processor -- start 2024-07-26T11:04:05.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 -- start start 2024-07-26T11:04:05.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 0x55ce695e8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6921c670 0x55ce695e3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695f9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41b5220 con 0x55ce6921c670 2024-07-26T11:04:05.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41b2af0 con 0x55ce6923d350 2024-07-26T11:04:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.069+0000 7fe5d2848740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41b0940 con 0x55ce69214490 2024-07-26T11:04:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.070+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 0x55ce695e8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.070+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 0x55ce695e8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55746/0 (socket says 172.21.15.144:55746) 2024-07-26T11:04:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.070+0000 7fe5c9c00640 1 -- 172.21.15.144:0/3426247224 learned_addr learned my addr 172.21.15.144:0/3426247224 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.070+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.070+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6921c670 0x55ce695e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5c9c00640 1 -- 172.21.15.144:0/3426247224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 msgr2=0x55ce695f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5c9c00640 1 -- 172.21.15.144:0/3426247224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6921c670 msgr2=0x55ce695e3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6921c670 0x55ce695e3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5c9c00640 1 -- 172.21.15.144:0/3426247224 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce695df560 con 0x55ce69214490 2024-07-26T11:04:05.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 0x55ce695e8ee0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009f20 tx=0x7fe5bc009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaae461a04435460 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:05.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5b5600640 1 -- 172.21.15.144:0/3426247224 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc015040 con 0x55ce69214490 2024-07-26T11:04:05.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5b5600640 1 -- 172.21.15.144:0/3426247224 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5bc02d030 con 0x55ce69214490 2024-07-26T11:04:05.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.071+0000 7fe5b5600640 1 -- 172.21.15.144:0/3426247224 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc02bba0 con 0x55ce69214490 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.072+0000 7fe5d2848740 1 -- 172.21.15.144:0/3426247224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 msgr2=0x55ce695e8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.072+0000 7fe5d2848740 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 0x55ce695e8ee0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009f20 tx=0x7fe5bc009820 comp rx=0 tx=0).stop 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 -- 172.21.15.144:0/3426247224 shutdown_connections 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce695f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce6921c670 0x55ce695e3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 --2- 172.21.15.144:0/3426247224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce69214490 0x55ce695e8ee0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 -- 172.21.15.144:0/3426247224 >> 172.21.15.144:0/3426247224 conn(0x55ce69231740 msgr2=0x55ce695dee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 -- 172.21.15.144:0/3426247224 shutdown_connections 2024-07-26T11:04:05.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 -- 172.21.15.144:0/3426247224 wait complete. 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.073+0000 7fe5d2848740 1 -- start start 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 Processor -- start 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce69661380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41b0fb0 con 0x55ce69214490 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41b2420 con 0x55ce6923d350 2024-07-26T11:04:05.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c41b1680 con 0x55ce6921c670 2024-07-26T11:04:05.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5d2848740 10 client.0 _ll_register_callbacks cb 0x7fe5b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:05.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.074+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c9c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695f20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33874/0 (socket says 172.21.15.144:33874) 2024-07-26T11:04:05.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c9c00640 1 -- 172.21.15.144:0/116619021 learned_addr learned my addr 172.21.15.144:0/116619021 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:05.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce69661380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 -- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 msgr2=0x55ce69661380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce69661380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 -- 172.21.15.144:0/116619021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 msgr2=0x55ce695f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 -- 172.21.15.144:0/116619021 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce696619f0 con 0x55ce6921c670 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce69661380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5c3e00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695f26a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00318a0 tx=0x7fe5b0004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74133c49659f196 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.075+0000 7fe5aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5aaa00640 1 -- 172.21.15.144:0/116619021 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0035040 con 0x55ce6921c670 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5bc02d030 con 0x55ce6921c670 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5aaa00640 1 -- 172.21.15.144:0/116619021 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fe5b00048c0 con 0x55ce6921c670 2024-07-26T11:04:05.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5aaa00640 1 -- 172.21.15.144:0/116619021 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5b005f050 con 0x55ce6921c670 2024-07-26T11:04:05.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5d2848740 10 client.25744 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:05.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce69668db0 con 0x55ce6921c670 2024-07-26T11:04:05.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.076+0000 7fe5aaa00640 1 -- 172.21.15.144:0/116619021 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5bc02d030 con 0x55ce6921c670 2024-07-26T11:04:05.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5aaa00640 1 -- 172.21.15.144:0/116619021 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe5b005a030 con 0x55ce6921c670 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5d2848740 10 client.25744 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5d2848740 10 client.25744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5d2848740 20 client.25744 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5d2848740 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691d1080 0x55ce6966b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5d2848740 4 client.25744 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ce695565e0 con 0x55ce691d1080 2024-07-26T11:04:05.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.077+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691d1080 0x55ce6966b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:05.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.078+0000 7fe5c9c00640 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691d1080 0x55ce6966b620 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fd4c3d813bce2d server_cookie=dd7cddb16b8d5d7d in_seq=0 out_seq=0 2024-07-26T11:04:05.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.078+0000 7fe5aaa00640 10 client.25744 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:05.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.079+0000 7fe5aaa00640 1 -- 172.21.15.144:0/116619021 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ce695565e0 con 0x55ce691d1080 2024-07-26T11:04:05.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.079+0000 7fe5aaa00640 10 client.25744 handle_command_reply: tid=0 2024-07-26T11:04:05.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.079+0000 7fe5d2848740 1 client.25744 shutdown 2024-07-26T11:04:05.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.079+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 msgr2=0x55ce695f26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.079+0000 7fe5d2848740 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695f26a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00318a0 tx=0x7fe5b0004610 comp rx=0 tx=0).stop 2024-07-26T11:04:05.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 shutdown_connections 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55ce691d1080 0x55ce6966b620 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ce6923d350 0x55ce69661380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5aaa00640 0 client.25744 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ce6921c670 0x55ce695f26a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 --2- 172.21.15.144:0/116619021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ce69214490 0x55ce695f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 >> 172.21.15.144:0/116619021 conn(0x55ce69231740 msgr2=0x55ce695dee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 shutdown_connections 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 1 -- 172.21.15.144:0/116619021 wait complete. 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.080+0000 7fe5d2848740 20 client.25744 trim_cache size 0 max 16384 2024-07-26T11:04:05.084 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:04:05.085 INFO:teuthology.orchestra.run.smithi144.stdout: "return_code": 0, 2024-07-26T11:04:05.085 INFO:teuthology.orchestra.run.smithi144.stdout: "scrub_tag": "1fa4d932-34d7-4b05-ae42-7bfe1a3d3b32", 2024-07-26T11:04:05.085 INFO:teuthology.orchestra.run.smithi144.stdout: "mode": "asynchronous" 2024-07-26T11:04:05.085 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:04:05.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe5ac0b3dc0 msgr2=0x7fe5ac0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe5ac0b3dc0 0x7fe5ac0b62d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55ce695e0be0 tx=0x7fe5b80ea530 comp rx=0 tx=0).stop 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 msgr2=0x7fe5c419bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 0x7fe5c419bf70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800e9f0 tx=0x7fe5b800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 shutdown_connections 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fe5ac0b3dc0 0x7fe5ac0b62d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c410fa20 0x7fe5c41a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5c41027c0 0x7fe5c419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 --2- 172.21.15.144:0/1338349544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe5c4101e70 0x7fe5c419bf70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 >> 172.21.15.144:0/1338349544 conn(0x7fe5c40fdcf0 msgr2=0x7fe5c4111c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 shutdown_connections 2024-07-26T11:04:05.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:05.081+0000 7fe5ca600640 1 -- 172.21.15.144:0/1338349544 wait complete. 2024-07-26T11:04:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:05 smithi144 ceph-mon[32129]: pgmap v509: 129 pgs: 129 active+clean; 21 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 943 op/s 2024-07-26T11:04:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:05 smithi160 ceph-mon[36715]: pgmap v509: 129 pgs: 129 active+clean; 21 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 943 op/s 2024-07-26T11:04:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:05 smithi183 ceph-mon[37273]: pgmap v509: 129 pgs: 129 active+clean; 21 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 943 op/s 2024-07-26T11:04:06.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub queued for path: / 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:04:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:06 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub queued for path: / 2024-07-26T11:04:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:04:06.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:04:06.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:04:06.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:06 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.635 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:04:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub queued for path: / 2024-07-26T11:04:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:04:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:04:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:04:06.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:06 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.307 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:07 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:04:07.307 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:07 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:04:07.307 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:07 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:07 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:07 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:07 smithi144 ceph-mon[32129]: pgmap v510: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 1.04k op/s 2024-07-26T11:04:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:07 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:04:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:07 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:04:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:07 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:07 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:07 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:07 smithi160 ceph-mon[36715]: pgmap v510: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 1.04k op/s 2024-07-26T11:04:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:07 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:04:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:07 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:04:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:07 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:07 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:07 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:04:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:07 smithi183 ceph-mon[37273]: pgmap v510: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 1.04k op/s 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 -- 172.21.15.144:0/3418133783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0077cc0 msgr2=0x7feec0113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 --2- 172.21.15.144:0/3418133783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0077cc0 0x7feec0113a40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feeac009900 tx=0x7feeac02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 -- 172.21.15.144:0/3418133783 shutdown_connections 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 --2- 172.21.15.144:0/3418133783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0113f80 0x7feec01163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 --2- 172.21.15.144:0/3418133783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0077cc0 0x7feec0113a40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 --2- 172.21.15.144:0/3418133783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 0x7feec0077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 -- 172.21.15.144:0/3418133783 >> 172.21.15.144:0/3418133783 conn(0x7feec0101010 msgr2=0x7feec0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 -- 172.21.15.144:0/3418133783 shutdown_connections 2024-07-26T11:04:08.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.986+0000 7feebf200640 1 -- 172.21.15.144:0/3418133783 wait complete. 2024-07-26T11:04:08.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 Processor -- start 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 -- start start 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 0x7feec01a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0077cc0 0x7feec01a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0113f80 0x7feec01a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7feec01198e0 con 0x7feec0077cc0 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feec0119760 con 0x7feec0113f80 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.988+0000 7feebf200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feec0119a60 con 0x7feec0077370 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebd400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0077cc0 0x7feec01a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 0x7feec01a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebd400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0077cc0 0x7feec01a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33896/0 (socket says 172.21.15.144:33896) 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebd400640 1 -- 172.21.15.144:0/782972998 learned_addr learned my addr 172.21.15.144:0/782972998 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:08.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebe800640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0113f80 0x7feec01a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:08.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebde00640 1 -- 172.21.15.144:0/782972998 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0113f80 msgr2=0x7feec01a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:08.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebde00640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0113f80 0x7feec01a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebde00640 1 -- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0077cc0 msgr2=0x7feec01a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:08.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebde00640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0077cc0 0x7feec01a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebde00640 1 -- 172.21.15.144:0/782972998 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec01a62a0 con 0x7feec0077370 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.989+0000 7feebe800640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0113f80 0x7feec01a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.990+0000 7feebde00640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 0x7feec01a0330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feeb000e9f0 tx=0x7feeb000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.990+0000 7feeb6a00640 1 -- 172.21.15.144:0/782972998 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb000ccc0 con 0x7feec0077370 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.990+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec01a6590 con 0x7feec0077370 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.990+0000 7feeb6a00640 1 -- 172.21.15.144:0/782972998 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feeb00108e0 con 0x7feec0077370 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.990+0000 7feeb6a00640 1 -- 172.21.15.144:0/782972998 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0018840 con 0x7feec0077370 2024-07-26T11:04:08.993 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.990+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec01accf0 con 0x7feec0077370 2024-07-26T11:04:08.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.993+0000 7feecbe48740 1 Processor -- start 2024-07-26T11:04:08.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 -- start start 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feeb6a00640 1 -- 172.21.15.144:0/782972998 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7feeb0010430 con 0x7feec0077370 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 0x563bdfe40000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe40540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa7a390 0x563bdfe40a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563bdfdb6bc0 con 0x563bdfa7a390 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563bdfda60c0 con 0x563bdfa721b0 2024-07-26T11:04:08.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.994+0000 7feecbe48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563bdfdaa100 con 0x563bdfa9b070 2024-07-26T11:04:08.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 0x563bdfe40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:08.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe40540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:08.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 0x563bdfe40000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55784/0 (socket says 172.21.15.144:55784) 2024-07-26T11:04:08.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 -- 172.21.15.144:0/3472899590 learned_addr learned my addr 172.21.15.144:0/3472899590 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebd400640 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa7a390 0x563bdfe40a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 -- 172.21.15.144:0/3472899590 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 msgr2=0x563bdfe40540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe40540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 -- 172.21.15.144:0/3472899590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa7a390 msgr2=0x563bdfe40a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa7a390 0x563bdfe40a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 -- 172.21.15.144:0/3472899590 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdfb06650 con 0x563bdfa9b070 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.995+0000 7feebe800640 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 0x563bdfe40000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feeb8011810 tx=0x7feeb8011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d909feaf5a3adc59 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.996+0000 7feeab400640 1 -- 172.21.15.144:0/3472899590 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8011310 con 0x563bdfa9b070 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.996+0000 7feeab400640 1 -- 172.21.15.144:0/3472899590 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feeb800f9a0 con 0x563bdfa9b070 2024-07-26T11:04:08.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.996+0000 7feeab400640 1 -- 172.21.15.144:0/3472899590 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8005650 con 0x563bdfa9b070 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.996+0000 7feeb6a00640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7feea40b3d40 0x7feea40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.996+0000 7feeb6a00640 1 -- 172.21.15.144:0/782972998 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7feeb00e4610 con 0x7feec0077370 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.996+0000 7feebde00640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7feea40b3d40 0x7feea40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 -- 172.21.15.144:0/3472899590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 msgr2=0x563bdfe40000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 0x563bdfe40000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feeb8011810 tx=0x7feeb8011cd0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 -- 172.21.15.144:0/3472899590 shutdown_connections 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa7a390 0x563bdfe40a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe40540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 --2- 172.21.15.144:0/3472899590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa9b070 0x563bdfe40000 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 -- 172.21.15.144:0/3472899590 >> 172.21.15.144:0/3472899590 conn(0x563bdfa8f460 msgr2=0x563bdfe3def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:09.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feebde00640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7feea40b3d40 0x7feea40b6250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x563bdfe3ec80 tx=0x7feeb00ea530 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:09.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 -- 172.21.15.144:0/3472899590 shutdown_connections 2024-07-26T11:04:09.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.997+0000 7feecbe48740 1 -- 172.21.15.144:0/3472899590 wait complete. 2024-07-26T11:04:09.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.998+0000 7feecbe48740 1 -- start start 2024-07-26T11:04:09.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.998+0000 7feecbe48740 1 Processor -- start 2024-07-26T11:04:09.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 0x563bdfe4b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa7a390 0x563bdfe4ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe4bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563bdfc09700 con 0x563bdfa9b070 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563bdfd42690 con 0x563bdfa7a390 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563bdfe4e7e0 con 0x563bdfa721b0 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feecbe48740 10 client.0 _ll_register_callbacks cb 0x563bdfd52140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 0x563bdfe4b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 0x563bdfe4b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55796/0 (socket says 172.21.15.144:55796) 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 -- 172.21.15.144:0/3453381810 learned_addr learned my addr 172.21.15.144:0/3453381810 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebd400640 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa7a390 0x563bdfe4ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 -- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa7a390 msgr2=0x563bdfe4ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa7a390 0x563bdfe4ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 -- 172.21.15.144:0/3453381810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 msgr2=0x563bdfe4bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe4bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:08.999+0000 7feebe800640 1 -- 172.21.15.144:0/3453381810 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdfd59830 con 0x563bdfa721b0 2024-07-26T11:04:09.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7feebe800640 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 0x563bdfe4b520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feeb80051e0 tx=0x7feeb8005210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3875812a7310d2ff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7fee9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7fee9fe00640 1 -- 172.21.15.144:0/3453381810 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb801e040 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7fee9fe00640 1 -- 172.21.15.144:0/3453381810 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7feeb8004c40 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb800f9a0 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7fee9fe00640 1 -- 172.21.15.144:0/3453381810 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feeb8033050 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7feecbe48740 10 client.25756 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.000+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563bdfe5a390 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7fee9fe00640 1 -- 172.21.15.144:0/3453381810 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feeb800f9a0 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7fee9fe00640 1 -- 172.21.15.144:0/3453381810 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7feeb8024040 con 0x563bdfa721b0 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7feecbe48740 10 client.25756 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7feecbe48740 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7feecbe48740 20 client.25756 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7feecbe48740 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa599f0 0x563bdfe5d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.005 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7feecbe48740 4 client.25756 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:09.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.001+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bdfdb8b10 con 0x563bdfa599f0 2024-07-26T11:04:09.006 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.002+0000 7feebd400640 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa599f0 0x563bdfe5d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.002+0000 7feebd400640 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa599f0 0x563bdfe5d7a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d18655ca4edac72 server_cookie=f89f8791808075b8 in_seq=0 out_seq=0 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.002+0000 7fee9fe00640 10 client.25756 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7fee9fe00640 1 -- 172.21.15.144:0/3453381810 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563bdfdb8b10 con 0x563bdfa599f0 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7fee9fe00640 10 client.25756 handle_command_reply: tid=0 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 client.25756 shutdown 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 msgr2=0x563bdfe4b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 0x563bdfe4b520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feeb80051e0 tx=0x7feeb8005210 comp rx=0 tx=0).stop 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 shutdown_connections 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa599f0 0x563bdfe5d7a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe4bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa7a390 0x563bdfe4ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.007 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7fee9fe00640 0 client.25756 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 --2- 172.21.15.144:0/3453381810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa721b0 0x563bdfe4b520 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.003+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 >> 172.21.15.144:0/3453381810 conn(0x563bdfa8f460 msgr2=0x563bdfe43190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.004+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 shutdown_connections 2024-07-26T11:04:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.004+0000 7feecbe48740 1 -- 172.21.15.144:0/3453381810 wait complete. 2024-07-26T11:04:09.008 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.004+0000 7feecbe48740 20 client.25756 trim_cache size 0 max 16384 2024-07-26T11:04:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.022+0000 7feecbe48740 1 Processor -- start 2024-07-26T11:04:09.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.022+0000 7feecbe48740 1 -- start start 2024-07-26T11:04:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feecbe48740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe4be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feecbe48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe3c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feecbe48740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe41740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feecbe48740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7feec01b95f0 con 0x563bdfa9b070 2024-07-26T11:04:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feecbe48740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feec01b6ec0 con 0x563bdfa721b0 2024-07-26T11:04:09.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feecbe48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feec01b4d10 con 0x563bdfa7a390 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe41740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe4be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe3c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe4be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51354/0 (socket says 172.21.15.144:51354) 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 -- 172.21.15.144:0/3594640861 learned_addr learned my addr 172.21.15.144:0/3594640861 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 -- 172.21.15.144:0/3594640861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 msgr2=0x563bdfe3c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe3c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 -- 172.21.15.144:0/3594640861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 msgr2=0x563bdfe41740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe41740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.023+0000 7feebe800640 1 -- 172.21.15.144:0/3594640861 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdfe422f0 con 0x563bdfa721b0 2024-07-26T11:04:09.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.024+0000 7feebe800640 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe4be50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009b80 tx=0x7feeb8000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20f8f4a0ed5a0f0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:09.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.024+0000 7feeaaa00640 1 -- 172.21.15.144:0/3594640861 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb800fe60 con 0x563bdfa721b0 2024-07-26T11:04:09.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.024+0000 7feeaaa00640 1 -- 172.21.15.144:0/3594640861 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feeb802a070 con 0x563bdfa721b0 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.024+0000 7feeaaa00640 1 -- 172.21.15.144:0/3594640861 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8010c30 con 0x563bdfa721b0 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 -- 172.21.15.144:0/3594640861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 msgr2=0x563bdfe4be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe4be50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009b80 tx=0x7feeb8000f40 comp rx=0 tx=0).stop 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 -- 172.21.15.144:0/3594640861 shutdown_connections 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa9b070 0x563bdfe41740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe3c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 --2- 172.21.15.144:0/3594640861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa721b0 0x563bdfe4be50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 -- 172.21.15.144:0/3594640861 >> 172.21.15.144:0/3594640861 conn(0x563bdfa8f460 msgr2=0x563bdfe41030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 -- 172.21.15.144:0/3594640861 shutdown_connections 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.025+0000 7feecbe48740 1 -- 172.21.15.144:0/3594640861 wait complete. 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 -- start start 2024-07-26T11:04:09.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 Processor -- start 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa721b0 0x563bdfe51290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe51870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa9b070 0x563bdfebdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7feec01b5380 con 0x563bdfa721b0 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feec01b67f0 con 0x563bdfa9b070 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.026+0000 7feecbe48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feec01b5a50 con 0x563bdfa7a390 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feecbe48740 10 client.0 _ll_register_callbacks cb 0x7feeb80116e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebe800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa721b0 0x563bdfe51290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe51870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe51870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:55826/0 (socket says 172.21.15.144:55826) 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 -- 172.21.15.144:0/1898531848 learned_addr learned my addr 172.21.15.144:0/1898531848 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebe800640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa9b070 0x563bdfebdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 -- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa9b070 msgr2=0x563bdfebdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa9b070 0x563bdfebdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 -- 172.21.15.144:0/1898531848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa721b0 msgr2=0x563bdfe51290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa721b0 0x563bdfe51290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebd400640 1 -- 172.21.15.144:0/1898531848 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdfebe590 con 0x563bdfa7a390 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.027+0000 7feebe800640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa9b070 0x563bdfebdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7feebd400640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe51870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feeac005940 tx=0x7feeac02aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=305b50d4b4c2cfd3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7fee9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7fee9fe00640 1 -- 172.21.15.144:0/1898531848 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac04a070 con 0x563bdfa7a390 2024-07-26T11:04:09.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7fee9fe00640 1 -- 172.21.15.144:0/1898531848 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7feeac0316a0 con 0x563bdfa7a390 2024-07-26T11:04:09.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb802a070 con 0x563bdfa7a390 2024-07-26T11:04:09.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7fee9fe00640 1 -- 172.21.15.144:0/1898531848 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feeac035040 con 0x563bdfa7a390 2024-07-26T11:04:09.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7feecbe48740 10 client.25762 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:09.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.028+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563bdfec5950 con 0x563bdfa7a390 2024-07-26T11:04:09.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7fee9fe00640 1 -- 172.21.15.144:0/1898531848 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feeb802a070 con 0x563bdfa7a390 2024-07-26T11:04:09.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7fee9fe00640 1 -- 172.21.15.144:0/1898531848 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7feeac03d570 con 0x563bdfa7a390 2024-07-26T11:04:09.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feecbe48740 10 client.25762 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feecbe48740 10 client.25762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feecbe48740 20 client.25762 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feecbe48740 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa2eda0 0x563bdfec81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feecbe48740 4 client.25762 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563bdfdb45b0 con 0x563bdfa2eda0 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.029+0000 7feebe800640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa2eda0 0x563bdfec81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feebe800640 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa2eda0 0x563bdfec81a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56e2aaa61ad9a5b4 server_cookie=b720cc7bb82158e1 in_seq=0 out_seq=0 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7fee9fe00640 10 client.25762 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7fee9fe00640 1 -- 172.21.15.144:0/1898531848 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563bdfdb45b0 con 0x563bdfa2eda0 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7fee9fe00640 10 client.25762 handle_command_reply: tid=0 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 client.25762 shutdown 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 msgr2=0x563bdfe51870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:09.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe51870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feeac005940 tx=0x7feeac02aba0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 shutdown_connections 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563bdfa2eda0 0x563bdfec81a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563bdfa9b070 0x563bdfebdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bdfa7a390 0x563bdfe51870 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 --2- 172.21.15.144:0/1898531848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563bdfa721b0 0x563bdfe51290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7fee9fe00640 0 client.25762 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.030+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 >> 172.21.15.144:0/1898531848 conn(0x563bdfa8f460 msgr2=0x563bdfe419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.031+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 shutdown_connections 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.031+0000 7feecbe48740 1 -- 172.21.15.144:0/1898531848 wait complete. 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.031+0000 7feecbe48740 20 client.25762 trim_cache size 0 max 16384 2024-07-26T11:04:09.035 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: "1fa4d932-34d7-4b05-ae42-7bfe1a3d3b32": { 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1fa4d932-34d7-4b05-ae42-7bfe1a3d3b32", 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:04:09.036 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.031+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7feea40b3d40 msgr2=0x7feea40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.031+0000 7feebf200640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7feea40b3d40 0x7feea40b6250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x563bdfe3ec80 tx=0x7feeb00ea530 comp rx=0 tx=0).stop 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 msgr2=0x7feec01a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 0x7feec01a0330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feeb000e9f0 tx=0x7feeb000edf0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 shutdown_connections 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7feea40b3d40 0x7feea40b6250 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feec0113f80 0x7feec01a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feec0077cc0 0x7feec01a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 --2- 172.21.15.144:0/782972998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feec0077370 0x7feec01a0330 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:09.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 >> 172.21.15.144:0/782972998 conn(0x7feec0101010 msgr2=0x7feec0103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:09.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 shutdown_connections 2024-07-26T11:04:09.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:09.032+0000 7feebf200640 1 -- 172.21.15.144:0/782972998 wait complete. 2024-07-26T11:04:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:09 smithi160 ceph-mon[36715]: pgmap v511: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 1.11k op/s 2024-07-26T11:04:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:09 smithi183 ceph-mon[37273]: pgmap v511: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 1.11k op/s 2024-07-26T11:04:09.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:09 smithi144 ceph-mon[32129]: pgmap v511: 129 pgs: 129 active+clean; 28 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 1.11k op/s 2024-07-26T11:04:09.920 INFO:tasks.cephfs.filesystem:scrub status for tag:1fa4d932-34d7-4b05-ae42-7bfe1a3d3b32 - {'path': '/', 'tag': '1fa4d932-34d7-4b05-ae42-7bfe1a3d3b32', 'options': 'recursive,force'} 2024-07-26T11:04:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:11 smithi160 ceph-mon[36715]: pgmap v512: 129 pgs: 129 active+clean; 28 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 1.14k op/s 2024-07-26T11:04:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:11 smithi183 ceph-mon[37273]: pgmap v512: 129 pgs: 129 active+clean; 28 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 1.14k op/s 2024-07-26T11:04:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:11 smithi144 ceph-mon[32129]: pgmap v512: 129 pgs: 129 active+clean; 28 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 1.14k op/s 2024-07-26T11:04:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:13 smithi160 ceph-mon[36715]: pgmap v513: 129 pgs: 129 active+clean; 28 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 833 op/s 2024-07-26T11:04:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:13 smithi183 ceph-mon[37273]: pgmap v513: 129 pgs: 129 active+clean; 28 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 833 op/s 2024-07-26T11:04:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:13 smithi144 ceph-mon[32129]: pgmap v513: 129 pgs: 129 active+clean; 28 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 833 op/s 2024-07-26T11:04:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 -- 172.21.15.144:0/3449837494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 msgr2=0x7fd3ac112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 --2- 172.21.15.144:0/3449837494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac112000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009be0 tx=0x7fd3a402a6d0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 -- 172.21.15.144:0/3449837494 shutdown_connections 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 --2- 172.21.15.144:0/3449837494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac112000 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 --2- 172.21.15.144:0/3449837494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ae80 0x7fd3ac10b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 --2- 172.21.15.144:0/3449837494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 -- 172.21.15.144:0/3449837494 >> 172.21.15.144:0/3449837494 conn(0x7fd3ac100f90 msgr2=0x7fd3ac1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.881+0000 7fd3abe00640 1 -- 172.21.15.144:0/3449837494 shutdown_connections 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.882+0000 7fd3abe00640 1 -- 172.21.15.144:0/3449837494 wait complete. 2024-07-26T11:04:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.883+0000 7fd3abe00640 1 Processor -- start 2024-07-26T11:04:13.885 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.883+0000 7fd3abe00640 1 -- start start 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.883+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ae80 0x7fd3ac19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3abe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac115480 con 0x7fd3ac105550 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3abe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac115300 con 0x7fd3ac10b7d0 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3abe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac115600 con 0x7fd3ac10ae80 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac19be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38520/0 (socket says 172.21.15.144:38520) 2024-07-26T11:04:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 -- 172.21.15.144:0/2329355878 learned_addr learned my addr 172.21.15.144:0/2329355878 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aa000640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ae80 0x7fd3ac19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 -- 172.21.15.144:0/2329355878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ae80 msgr2=0x7fd3ac19c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ae80 0x7fd3ac19c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 -- 172.21.15.144:0/2329355878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 msgr2=0x7fd3ac1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.884+0000 7fd3aaa00640 1 -- 172.21.15.144:0/2329355878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac1a1db0 con 0x7fd3ac105550 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3ab400640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac19be80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00e9c0 tx=0x7fd39c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3a3400640 1 -- 172.21.15.144:0/2329355878 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c00ccc0 con 0x7fd3ac105550 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac1a20a0 con 0x7fd3ac105550 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3a3400640 1 -- 172.21.15.144:0/2329355878 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd39c0054d0 con 0x7fd3ac105550 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3a3400640 1 -- 172.21.15.144:0/2329355878 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c010430 con 0x7fd3ac105550 2024-07-26T11:04:13.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.885+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac1a8830 con 0x7fd3ac105550 2024-07-26T11:04:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.889+0000 7fd3b8dc7740 1 Processor -- start 2024-07-26T11:04:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 -- start start 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1825c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1826160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 0x55f3f18266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f3f179a270 con 0x55f3f14821b0 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f3f1789770 con 0x55f3f14614d0 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f3f178d7b0 con 0x55f3f14592f0 2024-07-26T11:04:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.890+0000 7fd3a3400640 1 -- 172.21.15.144:0/2329355878 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd39c004f80 con 0x7fd3ac105550 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1825c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 0x55f3f18266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 0x55f3f18266a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47508/0 (socket says 172.21.15.144:47508) 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1825c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38526/0 (socket says 172.21.15.144:38526) 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 -- 172.21.15.144:0/1419650091 learned_addr learned my addr 172.21.15.144:0/1419650091 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1826160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 -- 172.21.15.144:0/1419650091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 msgr2=0x55f3f1826160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1826160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 -- 172.21.15.144:0/1419650091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 msgr2=0x55f3f1825c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1825c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.891+0000 7fd3aa000640 1 -- 172.21.15.144:0/1419650091 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f14ed790 con 0x55f3f14614d0 2024-07-26T11:04:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.892+0000 7fd3aa000640 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 0x55f3f18266a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd394009c50 tx=0x7fd394011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b67ec6a7a0523a3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.892+0000 7fd39be00640 1 -- 172.21.15.144:0/1419650091 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394004030 con 0x55f3f14614d0 2024-07-26T11:04:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.892+0000 7fd39be00640 1 -- 172.21.15.144:0/1419650091 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd394005020 con 0x55f3f14614d0 2024-07-26T11:04:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.892+0000 7fd39be00640 1 -- 172.21.15.144:0/1419650091 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394004d80 con 0x55f3f14614d0 2024-07-26T11:04:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.892+0000 7fd3a3400640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3900b3d40 0x7fd3900b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3a3400640 1 -- 172.21.15.144:0/2329355878 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd39c0e4780 con 0x7fd3ac105550 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3ab400640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3900b3d40 0x7fd3900b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1419650091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 msgr2=0x55f3f18266a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 0x55f3f18266a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd394009c50 tx=0x7fd394011810 comp rx=0 tx=0).stop 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1419650091 shutdown_connections 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14614d0 0x55f3f18266a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1826160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1419650091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1825c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1419650091 >> 172.21.15.144:0/1419650091 conn(0x55f3f1476300 msgr2=0x55f3f1823b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1419650091 shutdown_connections 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.893+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1419650091 wait complete. 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.894+0000 7fd3ab400640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3900b3d40 0x7fd3900b6250 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009370 tx=0x7fd3a4004a40 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.894+0000 7fd3b8dc7740 1 -- start start 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.894+0000 7fd3b8dc7740 1 Processor -- start 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1831d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f183d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f3f16c3ab0 con 0x55f3f14821b0 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f3f1726d50 con 0x55f3f14592f0 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f3f18343a0 con 0x55f3f14614d0 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3b8dc7740 10 client.0 _ll_register_callbacks cb 0x55f3f1499a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1831d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f183d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1831d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57238/0 (socket says 172.21.15.144:57238) 2024-07-26T11:04:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 -- 172.21.15.144:0/1891141339 learned_addr learned my addr 172.21.15.144:0/1891141339 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 -- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 msgr2=0x55f3f18317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18317f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 -- 172.21.15.144:0/1891141339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 msgr2=0x55f3f183d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f183d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aa000640 1 -- 172.21.15.144:0/1891141339 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f1566620 con 0x55f3f14614d0 2024-07-26T11:04:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.895+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd3aa000640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1831d30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd3940057c0 tx=0x7fd394005660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=298529ac24d1bd9b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd398c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd398c00640 1 -- 172.21.15.144:0/1891141339 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394026070 con 0x55f3f14614d0 2024-07-26T11:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd394005020 con 0x55f3f14614d0 2024-07-26T11:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd398c00640 1 -- 172.21.15.144:0/1891141339 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd394004a80 con 0x55f3f14614d0 2024-07-26T11:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd398c00640 1 -- 172.21.15.144:0/1891141339 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd394031050 con 0x55f3f14614d0 2024-07-26T11:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd3b8dc7740 10 client.25774 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3f1840410 con 0x55f3f14614d0 2024-07-26T11:04:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.896+0000 7fd398c00640 1 -- 172.21.15.144:0/1891141339 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd394005020 con 0x55f3f14614d0 2024-07-26T11:04:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd398c00640 1 -- 172.21.15.144:0/1891141339 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd394018070 con 0x55f3f14614d0 2024-07-26T11:04:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3b8dc7740 10 client.25774 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3b8dc7740 10 client.25774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3b8dc7740 20 client.25774 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f1440d80 0x55f3f1843770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3b8dc7740 4 client.25774 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3f179c1c0 con 0x55f3f1440d80 2024-07-26T11:04:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.897+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f1440d80 0x55f3f1843770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.898+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f1440d80 0x55f3f1843770 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b09779e293056b server_cookie=be12ba0a869ff6b1 in_seq=0 out_seq=0 2024-07-26T11:04:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.898+0000 7fd398c00640 10 client.25774 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:13.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd398c00640 1 -- 172.21.15.144:0/1891141339 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f3f179c1c0 con 0x55f3f1440d80 2024-07-26T11:04:13.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd398c00640 10 client.25774 handle_command_reply: tid=0 2024-07-26T11:04:13.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 client.25774 shutdown 2024-07-26T11:04:13.908 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 msgr2=0x55f3f1831d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:13.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1831d30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd3940057c0 tx=0x7fd394005660 comp rx=0 tx=0).stop 2024-07-26T11:04:13.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 shutdown_connections 2024-07-26T11:04:13.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f1440d80 0x55f3f1843770 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f183d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1831d30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/1891141339 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd398c00640 0 client.25774 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 >> 172.21.15.144:0/1891141339 conn(0x55f3f1476300 msgr2=0x55f3f1826c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 shutdown_connections 2024-07-26T11:04:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.899+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/1891141339 wait complete. 2024-07-26T11:04:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.900+0000 7fd3b8dc7740 20 client.25774 trim_cache size 0 max 16384 2024-07-26T11:04:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 Processor -- start 2024-07-26T11:04:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 -- start start 2024-07-26T11:04:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1827360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14614d0 0x55f3f183d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 0x55f3f183d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac1b5170 con 0x55f3f14614d0 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac1b2a40 con 0x55f3f14821b0 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.917+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac1b0890 con 0x55f3f14592f0 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 0x55f3f183d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14614d0 0x55f3f183d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 0x55f3f183d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47520/0 (socket says 172.21.15.144:47520) 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 -- 172.21.15.144:0/3183009500 learned_addr learned my addr 172.21.15.144:0/3183009500 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1827360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 -- 172.21.15.144:0/3183009500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 msgr2=0x55f3f1827360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1827360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 -- 172.21.15.144:0/3183009500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14614d0 msgr2=0x55f3f183d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14614d0 0x55f3f183d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 -- 172.21.15.144:0/3183009500 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f1843bd0 con 0x55f3f14821b0 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.918+0000 7fd3ab400640 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 0x55f3f183d9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009be0 tx=0x7fd3a4040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8df9092b79e82da server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.919+0000 7fd39b400640 1 -- 172.21.15.144:0/3183009500 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4036040 con 0x55f3f14821b0 2024-07-26T11:04:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.919+0000 7fd39b400640 1 -- 172.21.15.144:0/3183009500 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3a4050070 con 0x55f3f14821b0 2024-07-26T11:04:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.919+0000 7fd39b400640 1 -- 172.21.15.144:0/3183009500 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4055be0 con 0x55f3f14821b0 2024-07-26T11:04:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.919+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/3183009500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 msgr2=0x55f3f183d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.919+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 0x55f3f183d9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009be0 tx=0x7fd3a4040000 comp rx=0 tx=0).stop 2024-07-26T11:04:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/3183009500 shutdown_connections 2024-07-26T11:04:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14821b0 0x55f3f183d9f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14614d0 0x55f3f183d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/3183009500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14592f0 0x55f3f1827360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/3183009500 >> 172.21.15.144:0/3183009500 conn(0x55f3f1476300 msgr2=0x55f3f1826c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/3183009500 shutdown_connections 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/3183009500 wait complete. 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 -- start start 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.920+0000 7fd3b8dc7740 1 Processor -- start 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18345e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1834b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1835060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac1b0f00 con 0x55f3f14821b0 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac1b2370 con 0x55f3f14592f0 2024-07-26T11:04:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac1b15d0 con 0x55f3f14614d0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3b8dc7740 10 client.0 _ll_register_callbacks cb 0x7fd39c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3ab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1835060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1834b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1834b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57260/0 (socket says 172.21.15.144:57260) 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 -- 172.21.15.144:0/2761966709 learned_addr learned my addr 172.21.15.144:0/2761966709 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 -- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 msgr2=0x55f3f18345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 -- 172.21.15.144:0/2761966709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 msgr2=0x55f3f1835060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1835060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.921+0000 7fd3aa000640 1 -- 172.21.15.144:0/2761966709 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f18b2610 con 0x55f3f14614d0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd3aa000640 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1834b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3940095f0 tx=0x7fd394019620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99ec056444fe65fb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd398c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd398c00640 1 -- 172.21.15.144:0/2761966709 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39403a060 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3a4050070 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd398c00640 1 -- 172.21.15.144:0/2761966709 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd394004250 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd398c00640 1 -- 172.21.15.144:0/2761966709 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd394041050 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd3b8dc7740 10 client.25780 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.922+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3f18b3880 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd398c00640 1 -- 172.21.15.144:0/2761966709 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3a4050070 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd398c00640 1 -- 172.21.15.144:0/2761966709 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd394018070 con 0x55f3f14614d0 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3b8dc7740 10 client.25780 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3b8dc7740 10 client.25780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3b8dc7740 20 client.25780 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f14161c0 0x55f3f18b6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3b8dc7740 4 client.25780 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:04:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f3f1797c60 con 0x55f3f14161c0 2024-07-26T11:04:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.923+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f14161c0 0x55f3f18b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.924+0000 7fd3aaa00640 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f14161c0 0x55f3f18b6010 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=857ba0eed918e4e7 server_cookie=453aa8cb0755f94a in_seq=0 out_seq=0 2024-07-26T11:04:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:13.924+0000 7fd398c00640 10 client.25780 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.924+0000 7fd398c00640 1 -- 172.21.15.144:0/2761966709 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f3f1797c60 con 0x55f3f14161c0 2024-07-26T11:04:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.924+0000 7fd398c00640 10 client.25780 handle_command_reply: tid=0 2024-07-26T11:04:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.924+0000 7fd3b8dc7740 1 client.25780 shutdown 2024-07-26T11:04:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.924+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 msgr2=0x55f3f1834b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.924+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1834b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3940095f0 tx=0x7fd394019620 comp rx=0 tx=0).stop 2024-07-26T11:04:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 shutdown_connections 2024-07-26T11:04:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f3f14161c0 0x55f3f18b6010 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f3f14821b0 0x55f3f1835060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd398c00640 0 client.25780 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3f14614d0 0x55f3f1834b20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 --2- 172.21.15.144:0/2761966709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f3f14592f0 0x55f3f18345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 >> 172.21.15.144:0/2761966709 conn(0x55f3f1476300 msgr2=0x55f3f1832940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 shutdown_connections 2024-07-26T11:04:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 1 -- 172.21.15.144:0/2761966709 wait complete. 2024-07-26T11:04:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.925+0000 7fd3b8dc7740 20 client.25780 trim_cache size 0 max 16384 2024-07-26T11:04:14.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:04:14.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:04:14.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:04:14.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:04:14.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3900b3d40 msgr2=0x7fd3900b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:14.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3900b3d40 0x7fd3900b6250 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009370 tx=0x7fd3a4004a40 comp rx=0 tx=0).stop 2024-07-26T11:04:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 msgr2=0x7fd3ac19be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac19be80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00e9c0 tx=0x7fd39c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:04:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 shutdown_connections 2024-07-26T11:04:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3900b3d40 0x7fd3900b6250 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd3ac10b7d0 0x7fd3ac1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ae80 0x7fd3ac19c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 --2- 172.21.15.144:0/2329355878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3ac105550 0x7fd3ac19be80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 >> 172.21.15.144:0/2329355878 conn(0x7fd3ac100f90 msgr2=0x7fd3ac1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 shutdown_connections 2024-07-26T11:04:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:14.926+0000 7fd3abe00640 1 -- 172.21.15.144:0/2329355878 wait complete. 2024-07-26T11:04:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:15 smithi160 ceph-mon[36715]: pgmap v514: 129 pgs: 129 active+clean; 32 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 570 op/s 2024-07-26T11:04:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:15 smithi183 ceph-mon[37273]: pgmap v514: 129 pgs: 129 active+clean; 32 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 570 op/s 2024-07-26T11:04:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:15 smithi144 ceph-mon[32129]: pgmap v514: 129 pgs: 129 active+clean; 32 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 570 op/s 2024-07-26T11:04:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:16 smithi160 ceph-mon[36715]: scrub summary: idle 2024-07-26T11:04:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:16 smithi160 ceph-mon[36715]: scrub summary: idle 2024-07-26T11:04:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:16 smithi160 ceph-mon[36715]: scrub summary: idle 2024-07-26T11:04:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:16 smithi183 ceph-mon[37273]: scrub summary: idle 2024-07-26T11:04:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:16 smithi183 ceph-mon[37273]: scrub summary: idle 2024-07-26T11:04:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:16 smithi183 ceph-mon[37273]: scrub summary: idle 2024-07-26T11:04:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:16 smithi144 ceph-mon[32129]: scrub summary: idle 2024-07-26T11:04:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:16 smithi144 ceph-mon[32129]: scrub summary: idle 2024-07-26T11:04:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:16 smithi144 ceph-mon[32129]: scrub summary: idle 2024-07-26T11:04:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:17 smithi160 ceph-mon[36715]: pgmap v515: 129 pgs: 129 active+clean; 24 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 694 op/s 2024-07-26T11:04:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:17 smithi183 ceph-mon[37273]: pgmap v515: 129 pgs: 129 active+clean; 24 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 694 op/s 2024-07-26T11:04:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:17 smithi144 ceph-mon[32129]: pgmap v515: 129 pgs: 129 active+clean; 24 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 694 op/s 2024-07-26T11:04:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:19 smithi160 ceph-mon[36715]: pgmap v516: 129 pgs: 129 active+clean; 28 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 984 KiB/s wr, 879 op/s 2024-07-26T11:04:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:19 smithi183 ceph-mon[37273]: pgmap v516: 129 pgs: 129 active+clean; 28 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 984 KiB/s wr, 879 op/s 2024-07-26T11:04:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:19 smithi144 ceph-mon[32129]: pgmap v516: 129 pgs: 129 active+clean; 28 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 984 KiB/s wr, 879 op/s 2024-07-26T11:04:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:21 smithi160 ceph-mon[36715]: pgmap v517: 129 pgs: 129 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 1.8 MiB/s wr, 876 op/s 2024-07-26T11:04:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:21 smithi183 ceph-mon[37273]: pgmap v517: 129 pgs: 129 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 1.8 MiB/s wr, 876 op/s 2024-07-26T11:04:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:21 smithi144 ceph-mon[32129]: pgmap v517: 129 pgs: 129 active+clean; 39 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 1.8 MiB/s wr, 876 op/s 2024-07-26T11:04:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:23 smithi160 ceph-mon[36715]: pgmap v518: 129 pgs: 129 active+clean; 49 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 867 op/s 2024-07-26T11:04:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:23 smithi183 ceph-mon[37273]: pgmap v518: 129 pgs: 129 active+clean; 49 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 867 op/s 2024-07-26T11:04:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:23 smithi144 ceph-mon[32129]: pgmap v518: 129 pgs: 129 active+clean; 49 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 867 op/s 2024-07-26T11:04:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:25 smithi160 ceph-mon[36715]: pgmap v519: 129 pgs: 129 active+clean; 57 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 946 op/s 2024-07-26T11:04:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:25 smithi183 ceph-mon[37273]: pgmap v519: 129 pgs: 129 active+clean; 57 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 946 op/s 2024-07-26T11:04:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:25 smithi144 ceph-mon[32129]: pgmap v519: 129 pgs: 129 active+clean; 57 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 946 op/s 2024-07-26T11:04:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:27 smithi160 ceph-mon[36715]: pgmap v520: 129 pgs: 129 active+clean; 73 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 1.01k op/s 2024-07-26T11:04:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:27 smithi183 ceph-mon[37273]: pgmap v520: 129 pgs: 129 active+clean; 73 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 1.01k op/s 2024-07-26T11:04:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:27 smithi144 ceph-mon[32129]: pgmap v520: 129 pgs: 129 active+clean; 73 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 1.01k op/s 2024-07-26T11:04:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.178+0000 7f744b200640 1 -- 172.21.15.144:0/3029545987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 msgr2=0x7f744c10fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.178+0000 7f744b200640 1 --2- 172.21.15.144:0/3029545987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c10fdf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7444009ca0 tx=0x7f744402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.179+0000 7f744b200640 1 -- 172.21.15.144:0/3029545987 shutdown_connections 2024-07-26T11:04:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.179+0000 7f744b200640 1 --2- 172.21.15.144:0/3029545987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c10fdf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.179+0000 7f744b200640 1 --2- 172.21.15.144:0/3029545987 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f744c108c70 0x7f744c109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.179+0000 7f744b200640 1 --2- 172.21.15.144:0/3029545987 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c103340 0x7f744c103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.179+0000 7f744b200640 1 -- 172.21.15.144:0/3029545987 >> 172.21.15.144:0/3029545987 conn(0x7f744c0fed60 msgr2=0x7f744c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.180+0000 7f744b200640 1 -- 172.21.15.144:0/3029545987 shutdown_connections 2024-07-26T11:04:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.180+0000 7f744b200640 1 -- 172.21.15.144:0/3029545987 wait complete. 2024-07-26T11:04:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.181+0000 7f744b200640 1 Processor -- start 2024-07-26T11:04:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.181+0000 7f744b200640 1 -- start start 2024-07-26T11:04:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f744c103340 0x7f744c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c108c70 0x7f744c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f744c113290 con 0x7f744c1095c0 2024-07-26T11:04:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f744c113110 con 0x7f744c108c70 2024-07-26T11:04:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f744c113410 con 0x7f744c103340 2024-07-26T11:04:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f7449400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c108c70 0x7f744c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41058/0 (socket says 172.21.15.144:41058) 2024-07-26T11:04:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f744a800640 1 -- 172.21.15.144:0/2888341842 learned_addr learned my addr 172.21.15.144:0/2888341842 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.182+0000 7f7449e00640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f744c103340 0x7f744c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.183+0000 7f7449400640 1 -- 172.21.15.144:0/2888341842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f744c103340 msgr2=0x7f744c075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.183+0000 7f7449400640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f744c103340 0x7f744c075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.183+0000 7f7449400640 1 -- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 msgr2=0x7f744c075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.183+0000 7f7449400640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.183+0000 7f7449400640 1 -- 172.21.15.144:0/2888341842 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c1a7f20 con 0x7f744c108c70 2024-07-26T11:04:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.183+0000 7f7449400640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c108c70 0x7f744c075680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f744c06c0c0 tx=0x7f743800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.184+0000 7f7442a00640 1 -- 172.21.15.144:0/2888341842 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743800cc40 con 0x7f744c108c70 2024-07-26T11:04:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.184+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744c1a8210 con 0x7f744c108c70 2024-07-26T11:04:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.184+0000 7f7442a00640 1 -- 172.21.15.144:0/2888341842 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f743800cde0 con 0x7f744c108c70 2024-07-26T11:04:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.184+0000 7f7442a00640 1 -- 172.21.15.144:0/2888341842 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438004a00 con 0x7f744c108c70 2024-07-26T11:04:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.184+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744c1a8710 con 0x7f744c108c70 2024-07-26T11:04:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.187+0000 7f7457bc9740 1 Processor -- start 2024-07-26T11:04:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 -- start start 2024-07-26T11:04:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034cb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55da0343f270 con 0x55da030fe2f0 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55da0342e770 con 0x55da031064d0 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7457bc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55da034327b0 con 0x55da031271b0 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.188+0000 7f7449e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f7449e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41060/0 (socket says 172.21.15.144:41060) 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f744a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f7449e00640 1 -- 172.21.15.144:0/486613503 learned_addr learned my addr 172.21.15.144:0/486613503 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f7442a00640 1 -- 172.21.15.144:0/2888341842 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7438004ba0 con 0x7f744c108c70 2024-07-26T11:04:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f744a800640 1 -- 172.21.15.144:0/486613503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 msgr2=0x55da034cb6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:04:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f744a800640 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034cb6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f744a800640 1 -- 172.21.15.144:0/486613503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 msgr2=0x55da034cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f744a800640 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.189+0000 7f744a800640 1 -- 172.21.15.144:0/486613503 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da03192790 con 0x55da031271b0 2024-07-26T11:04:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.190+0000 7f744a800640 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034cac20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7444006010 tx=0x7f744402f420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e25a5436a737f592 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.190+0000 7f7437000640 1 -- 172.21.15.144:0/486613503 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444031cf0 con 0x55da031271b0 2024-07-26T11:04:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.190+0000 7f7437000640 1 -- 172.21.15.144:0/486613503 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7444036070 con 0x55da031271b0 2024-07-26T11:04:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.190+0000 7f7437000640 1 -- 172.21.15.144:0/486613503 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444004280 con 0x55da031271b0 2024-07-26T11:04:28.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 -- 172.21.15.144:0/486613503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 msgr2=0x55da034cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034cac20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7444006010 tx=0x7f744402f420 comp rx=0 tx=0).stop 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 -- 172.21.15.144:0/486613503 shutdown_connections 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 --2- 172.21.15.144:0/486613503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034cac20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 -- 172.21.15.144:0/486613503 >> 172.21.15.144:0/486613503 conn(0x55da0311b300 msgr2=0x55da034c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.191+0000 7f7457bc9740 1 -- 172.21.15.144:0/486613503 shutdown_connections 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.192+0000 7f7457bc9740 1 -- 172.21.15.144:0/486613503 wait complete. 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.192+0000 7f7442a00640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74300b3c80 0x7f74300b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.192+0000 7f7442a00640 1 -- 172.21.15.144:0/2888341842 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f74380e42e0 con 0x7f744c108c70 2024-07-26T11:04:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.192+0000 7f7449e00640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74300b3c80 0x7f74300b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.192+0000 7f7457bc9740 1 -- start start 2024-07-26T11:04:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.193+0000 7f7457bc9740 1 Processor -- start 2024-07-26T11:04:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.193+0000 7f7449e00640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74300b3c80 0x7f74300b6190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f743c009530 tx=0x7f743c005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.193+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 0x55da034d68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031064d0 0x55da034d6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da031271b0 0x55da034dc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7457bc9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55da03368ab0 con 0x55da031271b0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7457bc9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55da033cbd50 con 0x55da030fe2f0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7457bc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55da034d91c0 con 0x55da031064d0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 0x55da034d68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 0x55da034d68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:33828/0 (socket says 172.21.15.144:33828) 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7457bc9740 10 client.0 _ll_register_callbacks cb 0x55da0313ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 -- 172.21.15.144:0/1923929711 learned_addr learned my addr 172.21.15.144:0/1923929711 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449400640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031064d0 0x55da034d6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f744a800640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da031271b0 0x55da034dc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 -- 172.21.15.144:0/1923929711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031064d0 msgr2=0x55da034d6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031064d0 0x55da034d6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 -- 172.21.15.144:0/1923929711 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da031271b0 msgr2=0x55da034dc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da031271b0 0x55da034dc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f7449e00640 1 -- 172.21.15.144:0/1923929711 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0320b620 con 0x55da030fe2f0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.194+0000 7f744a800640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da031271b0 0x55da034dc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.195+0000 7f7449e00640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 0x55da034d68d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f743c0045f0 tx=0x7f743c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=161ec98a2608383b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.195+0000 7f742b400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.195+0000 7f742b400640 1 -- 172.21.15.144:0/1923929711 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c009e20 con 0x55da030fe2f0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.195+0000 7f742b400640 1 -- 172.21.15.144:0/1923929711 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f743c01f640 con 0x55da030fe2f0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.195+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7444036070 con 0x55da030fe2f0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.195+0000 7f742b400640 1 -- 172.21.15.144:0/1923929711 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f743c01f840 con 0x55da030fe2f0 2024-07-26T11:04:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f7457bc9740 10 client.25814 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da034e5310 con 0x55da030fe2f0 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f742b400640 1 -- 172.21.15.144:0/1923929711 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7444036070 con 0x55da030fe2f0 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f742b400640 1 -- 172.21.15.144:0/1923929711 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f743c027070 con 0x55da030fe2f0 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f7457bc9740 10 client.25814 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f7457bc9740 10 client.25814 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f7457bc9740 20 client.25814 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.196+0000 7f7457bc9740 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030e5d80 0x55da034e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.197+0000 7f7457bc9740 4 client.25814 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.197+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da034411c0 con 0x55da030e5d80 2024-07-26T11:04:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.197+0000 7f7449400640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030e5d80 0x55da034e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.197+0000 7f7449400640 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030e5d80 0x55da034e88e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858543714a0aa62a server_cookie=d6fd27b1b080fc78 in_seq=0 out_seq=0 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.197+0000 7f742b400640 10 client.25814 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.198+0000 7f742b400640 1 -- 172.21.15.144:0/1923929711 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55da034411c0 con 0x55da030e5d80 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.198+0000 7f742b400640 10 client.25814 handle_command_reply: tid=0 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.198+0000 7f7457bc9740 1 client.25814 shutdown 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 msgr2=0x55da034d68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 0x55da034d68d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f743c0045f0 tx=0x7f743c004730 comp rx=0 tx=0).stop 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 shutdown_connections 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030e5d80 0x55da034e88e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da031271b0 0x55da034dc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031064d0 0x55da034d6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 --2- 172.21.15.144:0/1923929711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da030fe2f0 0x55da034d68d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 >> 172.21.15.144:0/1923929711 conn(0x55da0311b300 msgr2=0x55da034cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f742b400640 0 client.25814 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 shutdown_connections 2024-07-26T11:04:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 1 -- 172.21.15.144:0/1923929711 wait complete. 2024-07-26T11:04:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.200+0000 7f7457bc9740 20 client.25814 trim_cache size 0 max 16384 2024-07-26T11:04:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.232+0000 7f7457bc9740 1 Processor -- start 2024-07-26T11:04:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 -- start start 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034d5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034e3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034dc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1b5060 con 0x55da030fe2f0 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1b2930 con 0x55da031064d0 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7457bc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1b0780 con 0x55da031271b0 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.233+0000 7f7449e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034d5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034d5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41086/0 (socket says 172.21.15.144:41086) 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034e3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449e00640 1 -- 172.21.15.144:0/242997558 learned_addr learned my addr 172.21.15.144:0/242997558 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f744a800640 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034dc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 -- 172.21.15.144:0/242997558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 msgr2=0x55da034dc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034dc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 -- 172.21.15.144:0/242997558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 msgr2=0x55da034d5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034d5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 -- 172.21.15.144:0/242997558 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da034d6ba0 con 0x55da031064d0 2024-07-26T11:04:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.234+0000 7f7449400640 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034e3280 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f74380e1630 tx=0x7f7438092040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe72fa39634b9842 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.235+0000 7f7436600640 1 -- 172.21.15.144:0/242997558 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743801c040 con 0x55da031064d0 2024-07-26T11:04:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.235+0000 7f7436600640 1 -- 172.21.15.144:0/242997558 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74380abe50 con 0x55da031064d0 2024-07-26T11:04:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.235+0000 7f7436600640 1 -- 172.21.15.144:0/242997558 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74380af120 con 0x55da031064d0 2024-07-26T11:04:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 -- 172.21.15.144:0/242997558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 msgr2=0x55da034e3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034e3280 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f74380e1630 tx=0x7f7438092040 comp rx=0 tx=0).stop 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 -- 172.21.15.144:0/242997558 shutdown_connections 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034dc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034e3280 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 --2- 172.21.15.144:0/242997558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034d5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 -- 172.21.15.144:0/242997558 >> 172.21.15.144:0/242997558 conn(0x55da0311b300 msgr2=0x55da034c8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 -- 172.21.15.144:0/242997558 shutdown_connections 2024-07-26T11:04:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.236+0000 7f7457bc9740 1 -- 172.21.15.144:0/242997558 wait complete. 2024-07-26T11:04:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.237+0000 7f7457bc9740 1 -- start start 2024-07-26T11:04:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.237+0000 7f7457bc9740 1 Processor -- start 2024-07-26T11:04:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034dda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1b0df0 con 0x55da030fe2f0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1b2260 con 0x55da031064d0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1b14c0 con 0x55da031271b0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7449e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034dda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7449400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7449e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034dda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41090/0 (socket says 172.21.15.144:41090) 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7449e00640 1 -- 172.21.15.144:0/750740854 learned_addr learned my addr 172.21.15.144:0/750740854 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f744a800640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.238+0000 7f7457bc9740 10 client.0 _ll_register_callbacks cb 0x7f743c009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449400640 1 -- 172.21.15.144:0/750740854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 msgr2=0x55da034de4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449400640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034de4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449400640 1 -- 172.21.15.144:0/750740854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 msgr2=0x55da034dda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449400640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034dda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449400640 1 -- 172.21.15.144:0/750740854 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0354f2a0 con 0x55da031064d0 2024-07-26T11:04:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449e00640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f7449400640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034ddfa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f74380e1630 tx=0x7f7438005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c1dee40119355a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f742b400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.239+0000 7f742b400640 1 -- 172.21.15.144:0/750740854 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743801c040 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74380abe50 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f742b400640 1 -- 172.21.15.144:0/750740854 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f74380a7110 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f742b400640 1 -- 172.21.15.144:0/750740854 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74380c5440 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 10 client.25820 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da03550530 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f742b400640 1 -- 172.21.15.144:0/750740854 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74380abe50 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f742b400640 1 -- 172.21.15.144:0/750740854 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f74380920b0 con 0x55da031064d0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 10 client.25820 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 10 client.25820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 20 client.25820 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030bb1c0 0x55da03552ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 4 client.25820 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.240+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da034d2250 con 0x55da030bb1c0 2024-07-26T11:04:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.241+0000 7f7449e00640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030bb1c0 0x55da03552ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.241+0000 7f7449e00640 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030bb1c0 0x55da03552ca0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d58d62bf9469f565 server_cookie=4c82fa2723df995a in_seq=0 out_seq=0 2024-07-26T11:04:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.241+0000 7f742b400640 10 client.25820 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.855+0000 7f742b400640 1 -- 172.21.15.144:0/750740854 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da034d2250 con 0x55da030bb1c0 2024-07-26T11:04:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.855+0000 7f742b400640 10 client.25820 handle_command_reply: tid=0 2024-07-26T11:04:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.855+0000 7f7457bc9740 1 client.25820 shutdown 2024-07-26T11:04:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.856+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 msgr2=0x55da034ddfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.856+0000 7f7457bc9740 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034ddfa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f74380e1630 tx=0x7f7438005250 comp rx=0 tx=0).stop 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 shutdown_connections 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f7457bc9740 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55da030bb1c0 0x55da03552ca0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f7457bc9740 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da031271b0 0x55da034de4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f7457bc9740 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55da031064d0 0x55da034ddfa0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f7457bc9740 1 --2- 172.21.15.144:0/750740854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55da030fe2f0 0x55da034dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 >> 172.21.15.144:0/750740854 conn(0x55da0311b300 msgr2=0x55da034cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.857+0000 7f742b400640 0 client.25820 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.858+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 shutdown_connections 2024-07-26T11:04:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.858+0000 7f7457bc9740 1 -- 172.21.15.144:0/750740854 wait complete. 2024-07-26T11:04:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.858+0000 7f7457bc9740 20 client.25820 trim_cache size 0 max 16384 2024-07-26T11:04:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:04:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:04:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:04:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:04:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74300b3c80 msgr2=0x7f74300b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74300b3c80 0x7f74300b6190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f743c009530 tx=0x7f743c005f90 comp rx=0 tx=0).stop 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c108c70 msgr2=0x7f744c075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c108c70 0x7f744c075680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f744c06c0c0 tx=0x7f743800ee50 comp rx=0 tx=0).stop 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 shutdown_connections 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74300b3c80 0x7f74300b6190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f744c1095c0 0x7f744c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f744c108c70 0x7f744c075680 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 --2- 172.21.15.144:0/2888341842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f744c103340 0x7f744c075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 >> 172.21.15.144:0/2888341842 conn(0x7f744c0fed60 msgr2=0x7f744c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 shutdown_connections 2024-07-26T11:04:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:28.859+0000 7f744b200640 1 -- 172.21.15.144:0/2888341842 wait complete. 2024-07-26T11:04:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:29 smithi160 ceph-mon[36715]: pgmap v521: 129 pgs: 129 active+clean; 75 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 783 op/s 2024-07-26T11:04:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:29 smithi183 ceph-mon[37273]: pgmap v521: 129 pgs: 129 active+clean; 75 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 783 op/s 2024-07-26T11:04:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:29 smithi144 ceph-mon[32129]: pgmap v521: 129 pgs: 129 active+clean; 75 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 783 op/s 2024-07-26T11:04:31.041 INFO:tasks.workunit.client.0.smithi144.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-07-26T11:04:31.046 INFO:tasks.workunit.client.0.smithi144.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-07-26T11:04:31.046 INFO:tasks.workunit.client.0.smithi144.stderr:+ make 2024-07-26T11:04:31.138 INFO:tasks.workunit.client.0.smithi144.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-07-26T11:04:31.548 INFO:tasks.workunit.client.0.smithi144.stderr:++ readlink -f fsstress 2024-07-26T11:04:31.550 INFO:tasks.workunit.client.0.smithi144.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-07-26T11:04:31.550 INFO:tasks.workunit.client.0.smithi144.stderr:+ popd 2024-07-26T11:04:31.551 INFO:tasks.workunit.client.0.smithi144.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-07-26T11:04:31.551 INFO:tasks.workunit.client.0.smithi144.stderr:+ popd 2024-07-26T11:04:31.551 INFO:tasks.workunit.client.0.smithi144.stdout:~/cephtest/mnt.0/client.0/tmp 2024-07-26T11:04:31.552 INFO:tasks.workunit.client.0.smithi144.stderr:++ mktemp -d -p . 2024-07-26T11:04:31.556 INFO:tasks.workunit.client.0.smithi144.stderr:+ T=./tmp.Q0elH8bjWH 2024-07-26T11:04:31.557 INFO:tasks.workunit.client.0.smithi144.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.Q0elH8bjWH -l 1 -n 1000 -p 10 -v 2024-07-26T11:04:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:31 smithi160 ceph-mon[36715]: pgmap v522: 129 pgs: 129 active+clean; 69 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 599 op/s 2024-07-26T11:04:31.560 INFO:tasks.workunit.client.0.smithi144.stdout:seed = 1722484805 2024-07-26T11:04:31.565 INFO:tasks.workunit.client.0.smithi144.stdout:0/0: write - no filename 2024-07-26T11:04:31.565 INFO:tasks.workunit.client.0.smithi144.stdout:0/1: dwrite - no filename 2024-07-26T11:04:31.565 INFO:tasks.workunit.client.0.smithi144.stdout:0/2: fdatasync - no filename 2024-07-26T11:04:31.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/0: write - no filename 2024-07-26T11:04:31.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/1: write - no filename 2024-07-26T11:04:31.566 INFO:tasks.workunit.client.0.smithi144.stdout:0/3: mkdir d0 0 2024-07-26T11:04:31.566 INFO:tasks.workunit.client.0.smithi144.stdout:0/4: dwrite - no filename 2024-07-26T11:04:31.566 INFO:tasks.workunit.client.0.smithi144.stdout:1/0: chown . 767 1 2024-07-26T11:04:31.567 INFO:tasks.workunit.client.0.smithi144.stdout:6/2: creat f0 x:0 0 0 2024-07-26T11:04:31.567 INFO:tasks.workunit.client.0.smithi144.stdout:6/3: write f0 [985099,4020] 0 2024-07-26T11:04:31.567 INFO:tasks.workunit.client.0.smithi144.stdout:0/5: symlink d0/l1 0 2024-07-26T11:04:31.568 INFO:tasks.workunit.client.0.smithi144.stdout:0/6: read - no filename 2024-07-26T11:04:31.568 INFO:tasks.workunit.client.0.smithi144.stdout:2/0: write - no filename 2024-07-26T11:04:31.568 INFO:tasks.workunit.client.0.smithi144.stdout:2/1: write - no filename 2024-07-26T11:04:31.569 INFO:tasks.workunit.client.0.smithi144.stdout:2/2: dwrite - no filename 2024-07-26T11:04:31.569 INFO:tasks.workunit.client.0.smithi144.stdout:2/3: chown . 82 1 2024-07-26T11:04:31.569 INFO:tasks.workunit.client.0.smithi144.stdout:1/1: creat f0 x:0 0 0 2024-07-26T11:04:31.569 INFO:tasks.workunit.client.0.smithi144.stdout:6/4: creat f1 x:0 0 0 2024-07-26T11:04:31.569 INFO:tasks.workunit.client.0.smithi144.stdout:6/5: stat f1 0 2024-07-26T11:04:31.571 INFO:tasks.workunit.client.0.smithi144.stdout:0/7: rename d0/l1 to d0/l2 0 2024-07-26T11:04:31.571 INFO:tasks.workunit.client.0.smithi144.stdout:3/0: write - no filename 2024-07-26T11:04:31.571 INFO:tasks.workunit.client.0.smithi144.stdout:2/4: mkdir d0 0 2024-07-26T11:04:31.572 INFO:tasks.workunit.client.0.smithi144.stdout:6/6: read f0 [694740,107548] 0 2024-07-26T11:04:31.572 INFO:tasks.workunit.client.0.smithi144.stdout:1/2: unlink f0 0 2024-07-26T11:04:31.573 INFO:tasks.workunit.client.0.smithi144.stdout:0/8: mknod d0/c3 0 2024-07-26T11:04:31.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/0: truncate - no filename 2024-07-26T11:04:31.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/1: dread - no filename 2024-07-26T11:04:31.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/2: unlink - no file 2024-07-26T11:04:31.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/3: unlink - no file 2024-07-26T11:04:31.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/4: unlink - no file 2024-07-26T11:04:31.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/5: readlink - no filename 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/6: chown . 25 1 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/7: getdents . 0 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/8: dread - no filename 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/9: truncate - no filename 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/10: dread - no filename 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:3/1: symlink l0 0 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:3/2: dwrite - no filename 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:2/5: creat d0/f1 x:0 0 0 2024-07-26T11:04:31.575 INFO:tasks.workunit.client.0.smithi144.stdout:2/6: chown d0/f1 2599303 1 2024-07-26T11:04:31.576 INFO:tasks.workunit.client.0.smithi144.stdout:6/7: link f0 f2 0 2024-07-26T11:04:31.576 INFO:tasks.workunit.client.0.smithi144.stdout:6/8: readlink - no filename 2024-07-26T11:04:31.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/3: creat f1 x:0 0 0 2024-07-26T11:04:31.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/4: dread - f1 zero size 2024-07-26T11:04:31.577 INFO:tasks.workunit.client.0.smithi144.stdout:0/9: creat d0/f4 x:0 0 0 2024-07-26T11:04:31.577 INFO:tasks.workunit.client.0.smithi144.stdout:0/10: write d0/f4 [64267,69864] 0 2024-07-26T11:04:31.578 INFO:tasks.workunit.client.0.smithi144.stdout:4/11: creat f0 x:0 0 0 2024-07-26T11:04:31.578 INFO:tasks.workunit.client.0.smithi144.stdout:4/12: rmdir - no directory 2024-07-26T11:04:31.578 INFO:tasks.workunit.client.0.smithi144.stdout:4/13: chown f0 56 1 2024-07-26T11:04:31.579 INFO:tasks.workunit.client.0.smithi144.stdout:3/3: mkdir d1 0 2024-07-26T11:04:31.579 INFO:tasks.workunit.client.0.smithi144.stdout:2/7: mkdir d0/d2 0 2024-07-26T11:04:31.580 INFO:tasks.workunit.client.0.smithi144.stdout:1/5: symlink l2 0 2024-07-26T11:04:31.580 INFO:tasks.workunit.client.0.smithi144.stdout:1/6: write f1 [997796,71523] 0 2024-07-26T11:04:31.581 INFO:tasks.workunit.client.0.smithi144.stdout:0/11: creat d0/f5 x:0 0 0 2024-07-26T11:04:31.582 INFO:tasks.workunit.client.0.smithi144.stdout:1/7: write f1 [743817,116321] 0 2024-07-26T11:04:31.585 INFO:tasks.workunit.client.0.smithi144.stdout:5/0: creat f0 x:0 0 0 2024-07-26T11:04:31.585 INFO:tasks.workunit.client.0.smithi144.stdout:7/0: unlink - no file 2024-07-26T11:04:31.585 INFO:tasks.workunit.client.0.smithi144.stdout:7/1: chown . 1070797 1 2024-07-26T11:04:31.585 INFO:tasks.workunit.client.0.smithi144.stdout:7/2: rename - no filename 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:3/4: symlink d1/l2 0 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:7/3: stat - no entries 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:7/4: dread - no filename 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:3/5: write - no filename 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:3/6: dwrite - no filename 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:3/7: truncate - no filename 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:5/1: truncate f0 564335 0 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:4/14: creat f1 x:0 0 0 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:2/8: symlink d0/d2/l3 0 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:4/15: dread - f0 zero size 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:4/16: read - f0 zero size 2024-07-26T11:04:31.586 INFO:tasks.workunit.client.0.smithi144.stdout:4/17: stat f0 0 2024-07-26T11:04:31.587 INFO:tasks.workunit.client.0.smithi144.stdout:4/18: fdatasync f0 0 2024-07-26T11:04:31.587 INFO:tasks.workunit.client.0.smithi144.stdout:4/19: dread - f0 zero size 2024-07-26T11:04:31.587 INFO:tasks.workunit.client.0.smithi144.stdout:4/20: truncate f1 506228 0 2024-07-26T11:04:31.587 INFO:tasks.workunit.client.0.smithi144.stdout:4/21: chown f0 210740703 1 2024-07-26T11:04:31.587 INFO:tasks.workunit.client.0.smithi144.stdout:1/8: mkdir d3 0 2024-07-26T11:04:31.587 INFO:tasks.workunit.client.0.smithi144.stdout:1/9: getdents d3 0 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:7/5: mkdir d0 0 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:8/0: stat - no entries 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:8/1: fsync - no filename 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:8/2: chown . 1 1 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:8/3: getdents . 0 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:8/4: dwrite - no filename 2024-07-26T11:04:31.588 INFO:tasks.workunit.client.0.smithi144.stdout:8/5: link - no file 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:8/6: dwrite - no filename 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:8/7: chown . 137 1 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:8/8: dwrite - no filename 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:8/9: unlink - no file 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:3/8: creat d1/f3 x:0 0 0 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:2/9: creat d0/d2/f4 x:0 0 0 2024-07-26T11:04:31.589 INFO:tasks.workunit.client.0.smithi144.stdout:4/22: mknod c2 0 2024-07-26T11:04:31.590 INFO:tasks.workunit.client.0.smithi144.stdout:7/6: symlink d0/l1 0 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:8/10: creat f0 x:0 0 0 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:2/10: symlink d0/d2/l5 0 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:9/0: symlink l0 0 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:9/1: stat l0 0 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:9/2: dwrite - no filename 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:9/3: dwrite - no filename 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:7/7: creat d0/f2 x:0 0 0 2024-07-26T11:04:31.594 INFO:tasks.workunit.client.0.smithi144.stdout:7/8: chown d0 29864 1 2024-07-26T11:04:31.595 INFO:tasks.workunit.client.0.smithi144.stdout:8/11: rename f0 to f1 0 2024-07-26T11:04:31.595 INFO:tasks.workunit.client.0.smithi144.stdout:8/12: rmdir - no directory 2024-07-26T11:04:31.595 INFO:tasks.workunit.client.0.smithi144.stdout:8/13: chown f1 2888 1 2024-07-26T11:04:31.596 INFO:tasks.workunit.client.0.smithi144.stdout:2/11: mkdir d0/d6 0 2024-07-26T11:04:31.596 INFO:tasks.workunit.client.0.smithi144.stdout:9/4: symlink l1 0 2024-07-26T11:04:31.596 INFO:tasks.workunit.client.0.smithi144.stdout:9/5: write - no filename 2024-07-26T11:04:31.597 INFO:tasks.workunit.client.0.smithi144.stdout:9/6: dread - no filename 2024-07-26T11:04:31.597 INFO:tasks.workunit.client.0.smithi144.stdout:9/7: dread - no filename 2024-07-26T11:04:31.597 INFO:tasks.workunit.client.0.smithi144.stdout:9/8: chown l0 178266 1 2024-07-26T11:04:31.597 INFO:tasks.workunit.client.0.smithi144.stdout:9/9: write - no filename 2024-07-26T11:04:31.597 INFO:tasks.workunit.client.0.smithi144.stdout:7/9: creat d0/f3 x:0 0 0 2024-07-26T11:04:31.599 INFO:tasks.workunit.client.0.smithi144.stdout:2/12: mknod d0/c7 0 2024-07-26T11:04:31.601 INFO:tasks.workunit.client.0.smithi144.stdout:9/10: creat f2 x:0 0 0 2024-07-26T11:04:31.603 INFO:tasks.workunit.client.0.smithi144.stdout:7/10: unlink d0/l1 0 2024-07-26T11:04:31.609 INFO:tasks.workunit.client.0.smithi144.stdout:2/13: creat d0/d6/f8 x:0 0 0 2024-07-26T11:04:31.609 INFO:tasks.workunit.client.0.smithi144.stdout:2/14: chown d0/c7 1873 1 2024-07-26T11:04:31.609 INFO:tasks.workunit.client.0.smithi144.stdout:2/15: truncate d0/d2/f4 384885 0 2024-07-26T11:04:31.609 INFO:tasks.workunit.client.0.smithi144.stdout:9/11: creat f3 x:0 0 0 2024-07-26T11:04:31.609 INFO:tasks.workunit.client.0.smithi144.stdout:9/12: creat f4 x:0 0 0 2024-07-26T11:04:31.610 INFO:tasks.workunit.client.0.smithi144.stdout:9/13: symlink l5 0 2024-07-26T11:04:31.611 INFO:tasks.workunit.client.0.smithi144.stdout:9/14: rename f2 to f6 0 2024-07-26T11:04:31.615 INFO:tasks.workunit.client.0.smithi144.stdout:9/15: rename l0 to l7 0 2024-07-26T11:04:31.615 INFO:tasks.workunit.client.0.smithi144.stdout:9/16: chown f3 51694 1 2024-07-26T11:04:31.615 INFO:tasks.workunit.client.0.smithi144.stdout:9/17: chown l7 601782 1 2024-07-26T11:04:31.615 INFO:tasks.workunit.client.0.smithi144.stdout:9/18: dread - f6 zero size 2024-07-26T11:04:31.617 INFO:tasks.workunit.client.0.smithi144.stdout:4/23: fsync f0 0 2024-07-26T11:04:31.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/24: write f1 [891169,57114] 0 2024-07-26T11:04:31.627 INFO:tasks.workunit.client.0.smithi144.stdout:9/19: symlink l8 0 2024-07-26T11:04:31.634 INFO:tasks.workunit.client.0.smithi144.stdout:8/14: rename f1 to f2 0 2024-07-26T11:04:31.634 INFO:tasks.workunit.client.0.smithi144.stdout:8/15: dread - f2 zero size 2024-07-26T11:04:31.634 INFO:tasks.workunit.client.0.smithi144.stdout:8/16: dread - f2 zero size 2024-07-26T11:04:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:31 smithi183 ceph-mon[37273]: pgmap v522: 129 pgs: 129 active+clean; 69 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 599 op/s 2024-07-26T11:04:31.635 INFO:tasks.workunit.client.0.smithi144.stdout:9/20: fsync f6 0 2024-07-26T11:04:31.645 INFO:tasks.workunit.client.0.smithi144.stdout:9/21: unlink f3 0 2024-07-26T11:04:31.649 INFO:tasks.workunit.client.0.smithi144.stdout:5/2: dwrite f0 [0,4194304] 0 2024-07-26T11:04:31.653 INFO:tasks.workunit.client.0.smithi144.stdout:5/3: mkdir d1 0 2024-07-26T11:04:31.655 INFO:tasks.workunit.client.0.smithi144.stdout:5/4: mknod d1/c2 0 2024-07-26T11:04:31.660 INFO:tasks.workunit.client.0.smithi144.stdout:0/12: dwrite d0/f5 [0,4194304] 0 2024-07-26T11:04:31.689 INFO:tasks.workunit.client.0.smithi144.stdout:1/10: dread f1 [0,4194304] 0 2024-07-26T11:04:31.723 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:31 smithi144 ceph-mon[32129]: pgmap v522: 129 pgs: 129 active+clean; 69 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 599 op/s 2024-07-26T11:04:31.797 INFO:tasks.workunit.client.0.smithi144.stdout:1/11: dread f1 [0,4194304] 0 2024-07-26T11:04:31.820 INFO:tasks.workunit.client.0.smithi144.stdout:5/5: dread f0 [0,4194304] 0 2024-07-26T11:04:31.822 INFO:tasks.workunit.client.0.smithi144.stdout:5/6: rmdir d1 39 2024-07-26T11:04:31.825 INFO:tasks.workunit.client.0.smithi144.stdout:5/7: symlink d1/l3 0 2024-07-26T11:04:31.827 INFO:tasks.workunit.client.0.smithi144.stdout:0/13: write d0/f5 [2998697,31652] 0 2024-07-26T11:04:31.830 INFO:tasks.workunit.client.0.smithi144.stdout:5/8: mkdir d1/d4 0 2024-07-26T11:04:31.832 INFO:tasks.workunit.client.0.smithi144.stdout:5/9: mkdir d1/d5 0 2024-07-26T11:04:31.834 INFO:tasks.workunit.client.0.smithi144.stdout:5/10: symlink d1/l6 0 2024-07-26T11:04:31.838 INFO:tasks.workunit.client.0.smithi144.stdout:6/9: dwrite f0 [0,4194304] 0 2024-07-26T11:04:31.838 INFO:tasks.workunit.client.0.smithi144.stdout:6/10: dread - f1 zero size 2024-07-26T11:04:31.838 INFO:tasks.workunit.client.0.smithi144.stdout:6/11: rmdir - no directory 2024-07-26T11:04:31.840 INFO:tasks.workunit.client.0.smithi144.stdout:6/12: fsync f0 0 2024-07-26T11:04:31.910 INFO:tasks.workunit.client.0.smithi144.stdout:8/17: dwrite f2 [0,4194304] 0 2024-07-26T11:04:31.914 INFO:tasks.workunit.client.0.smithi144.stdout:8/18: creat f3 x:0 0 0 2024-07-26T11:04:31.914 INFO:tasks.workunit.client.0.smithi144.stdout:8/19: write f3 [234148,31495] 0 2024-07-26T11:04:31.917 INFO:tasks.workunit.client.0.smithi144.stdout:8/20: creat f4 x:0 0 0 2024-07-26T11:04:31.921 INFO:tasks.workunit.client.0.smithi144.stdout:8/21: creat f5 x:0 0 0 2024-07-26T11:04:31.925 INFO:tasks.workunit.client.0.smithi144.stdout:8/22: rename f3 to f6 0 2024-07-26T11:04:31.925 INFO:tasks.workunit.client.0.smithi144.stdout:8/23: write f6 [257020,109557] 0 2024-07-26T11:04:31.927 INFO:tasks.workunit.client.0.smithi144.stdout:8/24: mkdir d7 0 2024-07-26T11:04:31.927 INFO:tasks.workunit.client.0.smithi144.stdout:8/25: truncate f5 929965 0 2024-07-26T11:04:31.930 INFO:tasks.workunit.client.0.smithi144.stdout:8/26: rmdir d7 0 2024-07-26T11:04:31.930 INFO:tasks.workunit.client.0.smithi144.stdout:8/27: fsync f2 0 2024-07-26T11:04:31.934 INFO:tasks.workunit.client.0.smithi144.stdout:8/28: unlink f2 0 2024-07-26T11:04:31.934 INFO:tasks.workunit.client.0.smithi144.stdout:8/29: stat f6 0 2024-07-26T11:04:31.935 INFO:tasks.workunit.client.0.smithi144.stdout:8/30: truncate f6 890730 0 2024-07-26T11:04:31.980 INFO:tasks.workunit.client.0.smithi144.stdout:3/9: dwrite d1/f3 [0,4194304] 0 2024-07-26T11:04:32.050 INFO:tasks.workunit.client.0.smithi144.stdout:1/12: dread f1 [0,4194304] 0 2024-07-26T11:04:32.050 INFO:tasks.workunit.client.0.smithi144.stdout:1/13: write f1 [1619996,111845] 0 2024-07-26T11:04:32.126 INFO:tasks.workunit.client.0.smithi144.stdout:9/22: dwrite f4 [0,4194304] 0 2024-07-26T11:04:32.131 INFO:tasks.workunit.client.0.smithi144.stdout:9/23: getdents . 0 2024-07-26T11:04:32.131 INFO:tasks.workunit.client.0.smithi144.stdout:9/24: chown l8 1226 1 2024-07-26T11:04:32.133 INFO:tasks.workunit.client.0.smithi144.stdout:9/25: mkdir d9 0 2024-07-26T11:04:32.166 INFO:tasks.workunit.client.0.smithi144.stdout:9/26: truncate f4 3521622 0 2024-07-26T11:04:32.169 INFO:tasks.workunit.client.0.smithi144.stdout:9/27: fdatasync f6 0 2024-07-26T11:04:32.173 INFO:tasks.workunit.client.0.smithi144.stdout:9/28: creat d9/fa x:0 0 0 2024-07-26T11:04:32.185 INFO:tasks.workunit.client.0.smithi144.stdout:9/29: rename l1 to d9/lb 0 2024-07-26T11:04:32.187 INFO:tasks.workunit.client.0.smithi144.stdout:9/30: creat d9/fc x:0 0 0 2024-07-26T11:04:32.191 INFO:tasks.workunit.client.0.smithi144.stdout:9/31: link l5 d9/ld 0 2024-07-26T11:04:32.191 INFO:tasks.workunit.client.0.smithi144.stdout:9/32: chown d9 1782 1 2024-07-26T11:04:32.191 INFO:tasks.workunit.client.0.smithi144.stdout:9/33: dread - d9/fc zero size 2024-07-26T11:04:32.193 INFO:tasks.workunit.client.0.smithi144.stdout:9/34: mkdir d9/de 0 2024-07-26T11:04:32.194 INFO:tasks.workunit.client.0.smithi144.stdout:9/35: stat d9/de 0 2024-07-26T11:04:32.198 INFO:tasks.workunit.client.0.smithi144.stdout:9/36: creat d9/de/ff x:0 0 0 2024-07-26T11:04:32.198 INFO:tasks.workunit.client.0.smithi144.stdout:9/37: chown d9/lb 515 1 2024-07-26T11:04:32.206 INFO:tasks.workunit.client.0.smithi144.stdout:9/38: unlink d9/ld 0 2024-07-26T11:04:32.227 INFO:tasks.workunit.client.0.smithi144.stdout:7/11: dwrite d0/f2 [0,4194304] 0 2024-07-26T11:04:32.257 INFO:tasks.workunit.client.0.smithi144.stdout:7/12: mknod d0/c4 0 2024-07-26T11:04:32.272 INFO:tasks.workunit.client.0.smithi144.stdout:5/11: dread f0 [0,4194304] 0 2024-07-26T11:04:32.272 INFO:tasks.workunit.client.0.smithi144.stdout:5/12: fdatasync f0 0 2024-07-26T11:04:32.276 INFO:tasks.workunit.client.0.smithi144.stdout:7/13: symlink d0/l5 0 2024-07-26T11:04:32.276 INFO:tasks.workunit.client.0.smithi144.stdout:7/14: fsync d0/f2 0 2024-07-26T11:04:32.300 INFO:tasks.workunit.client.0.smithi144.stdout:1/14: dwrite f1 [0,4194304] 0 2024-07-26T11:04:32.300 INFO:tasks.workunit.client.0.smithi144.stdout:1/15: getdents d3 0 2024-07-26T11:04:32.300 INFO:tasks.workunit.client.0.smithi144.stdout:1/16: chown f1 481512 1 2024-07-26T11:04:32.302 INFO:tasks.workunit.client.0.smithi144.stdout:5/13: mknod d1/d5/c7 0 2024-07-26T11:04:32.304 INFO:tasks.workunit.client.0.smithi144.stdout:7/15: mkdir d0/d6 0 2024-07-26T11:04:32.327 INFO:tasks.workunit.client.0.smithi144.stdout:1/17: creat d3/f4 x:0 0 0 2024-07-26T11:04:32.327 INFO:tasks.workunit.client.0.smithi144.stdout:5/14: symlink d1/l8 0 2024-07-26T11:04:32.327 INFO:tasks.workunit.client.0.smithi144.stdout:7/16: creat d0/f7 x:0 0 0 2024-07-26T11:04:32.372 INFO:tasks.workunit.client.0.smithi144.stdout:1/18: dread f1 [0,4194304] 0 2024-07-26T11:04:32.378 INFO:tasks.workunit.client.0.smithi144.stdout:1/19: rmdir d3 39 2024-07-26T11:04:32.387 INFO:tasks.workunit.client.0.smithi144.stdout:1/20: symlink d3/l5 0 2024-07-26T11:04:32.397 INFO:tasks.workunit.client.0.smithi144.stdout:7/17: getdents d0 0 2024-07-26T11:04:32.397 INFO:tasks.workunit.client.0.smithi144.stdout:7/18: truncate d0/f3 1040695 0 2024-07-26T11:04:32.448 INFO:tasks.workunit.client.0.smithi144.stdout:1/21: getdents d3 0 2024-07-26T11:04:32.460 INFO:tasks.workunit.client.0.smithi144.stdout:1/22: mkdir d3/d6 0 2024-07-26T11:04:32.535 INFO:tasks.workunit.client.0.smithi144.stdout:1/23: dwrite f1 [0,4194304] 0 2024-07-26T11:04:32.535 INFO:tasks.workunit.client.0.smithi144.stdout:1/24: dread - d3/f4 zero size 2024-07-26T11:04:32.556 INFO:tasks.workunit.client.0.smithi144.stdout:5/15: dwrite f0 [0,4194304] 0 2024-07-26T11:04:32.560 INFO:tasks.workunit.client.0.smithi144.stdout:5/16: rename d1/d4 to d1/d9 0 2024-07-26T11:04:32.560 INFO:tasks.workunit.client.0.smithi144.stdout:5/17: stat d1/d9 0 2024-07-26T11:04:32.593 INFO:tasks.workunit.client.0.smithi144.stdout:7/19: dwrite d0/f3 [0,4194304] 0 2024-07-26T11:04:32.604 INFO:tasks.workunit.client.0.smithi144.stdout:7/20: creat d0/d6/f8 x:0 0 0 2024-07-26T11:04:32.619 INFO:tasks.workunit.client.0.smithi144.stdout:5/18: dwrite f0 [0,4194304] 0 2024-07-26T11:04:32.644 INFO:tasks.workunit.client.0.smithi144.stdout:1/25: dwrite f1 [0,4194304] 0 2024-07-26T11:04:32.644 INFO:tasks.workunit.client.0.smithi144.stdout:1/26: write d3/f4 [526686,31543] 0 2024-07-26T11:04:32.648 INFO:tasks.workunit.client.0.smithi144.stdout:7/21: write d0/f3 [2839599,41429] 0 2024-07-26T11:04:32.649 INFO:tasks.workunit.client.0.smithi144.stdout:1/27: fsync d3/f4 0 2024-07-26T11:04:32.649 INFO:tasks.workunit.client.0.smithi144.stdout:1/28: write d3/f4 [1549002,115478] 0 2024-07-26T11:04:32.658 INFO:tasks.workunit.client.0.smithi144.stdout:7/22: mkdir d0/d6/d9 0 2024-07-26T11:04:32.669 INFO:tasks.workunit.client.0.smithi144.stdout:5/19: write f0 [1472486,84265] 0 2024-07-26T11:04:32.670 INFO:tasks.workunit.client.0.smithi144.stdout:1/29: mknod d3/c7 0 2024-07-26T11:04:32.674 INFO:tasks.workunit.client.0.smithi144.stdout:7/23: rmdir d0 39 2024-07-26T11:04:32.674 INFO:tasks.workunit.client.0.smithi144.stdout:7/24: dread - d0/d6/f8 zero size 2024-07-26T11:04:32.674 INFO:tasks.workunit.client.0.smithi144.stdout:7/25: rename d0 to d0/d6/d9/da 22 2024-07-26T11:04:32.684 INFO:tasks.workunit.client.0.smithi144.stdout:5/20: link d1/d5/c7 d1/d5/ca 0 2024-07-26T11:04:32.685 INFO:tasks.workunit.client.0.smithi144.stdout:5/21: stat d1/l3 0 2024-07-26T11:04:32.685 INFO:tasks.workunit.client.0.smithi144.stdout:5/22: read f0 [15818,77118] 0 2024-07-26T11:04:32.685 INFO:tasks.workunit.client.0.smithi144.stdout:1/30: symlink d3/d6/l8 0 2024-07-26T11:04:32.685 INFO:tasks.workunit.client.0.smithi144.stdout:1/31: chown d3/d6/l8 190509 1 2024-07-26T11:04:32.686 INFO:tasks.workunit.client.0.smithi144.stdout:7/26: creat d0/fb x:0 0 0 2024-07-26T11:04:32.686 INFO:tasks.workunit.client.0.smithi144.stdout:7/27: chown d0/l5 175823290 1 2024-07-26T11:04:32.690 INFO:tasks.workunit.client.0.smithi144.stdout:5/23: link d1/c2 d1/d9/cb 0 2024-07-26T11:04:32.690 INFO:tasks.workunit.client.0.smithi144.stdout:5/24: read f0 [4005074,115283] 0 2024-07-26T11:04:32.695 INFO:tasks.workunit.client.0.smithi144.stdout:1/32: unlink d3/d6/l8 0 2024-07-26T11:04:32.697 INFO:tasks.workunit.client.0.smithi144.stdout:7/28: creat d0/d6/d9/fc x:0 0 0 2024-07-26T11:04:32.704 INFO:tasks.workunit.client.0.smithi144.stdout:1/33: mknod d3/c9 0 2024-07-26T11:04:32.704 INFO:tasks.workunit.client.0.smithi144.stdout:1/34: fsync f1 0 2024-07-26T11:04:32.705 INFO:tasks.workunit.client.0.smithi144.stdout:7/29: creat d0/fd x:0 0 0 2024-07-26T11:04:32.705 INFO:tasks.workunit.client.0.smithi144.stdout:7/30: stat d0/l5 0 2024-07-26T11:04:32.705 INFO:tasks.workunit.client.0.smithi144.stdout:5/25: rename d1/l3 to d1/lc 0 2024-07-26T11:04:32.705 INFO:tasks.workunit.client.0.smithi144.stdout:5/26: chown d1/lc 60 1 2024-07-26T11:04:32.706 INFO:tasks.workunit.client.0.smithi144.stdout:5/27: chown d1/lc 64 1 2024-07-26T11:04:32.713 INFO:tasks.workunit.client.0.smithi144.stdout:7/31: creat d0/d6/d9/fe x:0 0 0 2024-07-26T11:04:32.716 INFO:tasks.workunit.client.0.smithi144.stdout:5/28: unlink f0 0 2024-07-26T11:04:32.727 INFO:tasks.workunit.client.0.smithi144.stdout:7/32: symlink d0/lf 0 2024-07-26T11:04:32.729 INFO:tasks.workunit.client.0.smithi144.stdout:6/13: dwrite f0 [0,4194304] 0 2024-07-26T11:04:32.732 INFO:tasks.workunit.client.0.smithi144.stdout:5/29: mknod d1/cd 0 2024-07-26T11:04:32.741 INFO:tasks.workunit.client.0.smithi144.stdout:7/33: rmdir d0/d6 39 2024-07-26T11:04:32.741 INFO:tasks.workunit.client.0.smithi144.stdout:7/34: readlink d0/l5 0 2024-07-26T11:04:32.741 INFO:tasks.workunit.client.0.smithi144.stdout:7/35: chown d0/f2 17 1 2024-07-26T11:04:32.749 INFO:tasks.workunit.client.0.smithi144.stdout:6/14: symlink l3 0 2024-07-26T11:04:32.749 INFO:tasks.workunit.client.0.smithi144.stdout:5/30: symlink d1/d9/le 0 2024-07-26T11:04:32.749 INFO:tasks.workunit.client.0.smithi144.stdout:5/31: dwrite - no filename 2024-07-26T11:04:32.749 INFO:tasks.workunit.client.0.smithi144.stdout:7/36: rmdir d0 39 2024-07-26T11:04:32.752 INFO:tasks.workunit.client.0.smithi144.stdout:5/32: rename d1/cd to d1/cf 0 2024-07-26T11:04:32.752 INFO:tasks.workunit.client.0.smithi144.stdout:5/33: truncate - no filename 2024-07-26T11:04:32.752 INFO:tasks.workunit.client.0.smithi144.stdout:5/34: chown d1/cf 169951 1 2024-07-26T11:04:32.753 INFO:tasks.workunit.client.0.smithi144.stdout:5/35: creat d1/d9/f10 x:0 0 0 2024-07-26T11:04:32.753 INFO:tasks.workunit.client.0.smithi144.stdout:5/36: dread - d1/d9/f10 zero size 2024-07-26T11:04:32.766 INFO:tasks.workunit.client.0.smithi144.stdout:1/35: dread f1 [0,4194304] 0 2024-07-26T11:04:32.768 INFO:tasks.workunit.client.0.smithi144.stdout:1/36: mkdir d3/d6/da 0 2024-07-26T11:04:32.773 INFO:tasks.workunit.client.0.smithi144.stdout:9/39: dwrite f6 [0,4194304] 0 2024-07-26T11:04:32.774 INFO:tasks.workunit.client.0.smithi144.stdout:9/40: write d9/fa [196784,129128] 0 2024-07-26T11:04:32.774 INFO:tasks.workunit.client.0.smithi144.stdout:9/41: write d9/fc [364319,88325] 0 2024-07-26T11:04:32.779 INFO:tasks.workunit.client.0.smithi144.stdout:9/42: creat d9/de/f10 x:0 0 0 2024-07-26T11:04:32.781 INFO:tasks.workunit.client.0.smithi144.stdout:9/43: mknod d9/c11 0 2024-07-26T11:04:32.781 INFO:tasks.workunit.client.0.smithi144.stdout:9/44: symlink d9/de/l12 0 2024-07-26T11:04:32.817 INFO:tasks.workunit.client.0.smithi144.stdout:1/37: dwrite d3/f4 [0,4194304] 0 2024-07-26T11:04:32.823 INFO:tasks.workunit.client.0.smithi144.stdout:1/38: unlink d3/l5 0 2024-07-26T11:04:32.823 INFO:tasks.workunit.client.0.smithi144.stdout:1/39: chown d3/c9 2 1 2024-07-26T11:04:32.856 INFO:tasks.workunit.client.0.smithi144.stdout:1/40: write f1 [3528430,110795] 0 2024-07-26T11:04:32.870 INFO:tasks.workunit.client.0.smithi144.stdout:9/45: dwrite f6 [4194304,4194304] 0 2024-07-26T11:04:32.872 INFO:tasks.workunit.client.0.smithi144.stdout:9/46: mknod d9/de/c13 0 2024-07-26T11:04:32.872 INFO:tasks.workunit.client.0.smithi144.stdout:9/47: chown l7 695661 1 2024-07-26T11:04:32.874 INFO:tasks.workunit.client.0.smithi144.stdout:9/48: mkdir d9/de/d14 0 2024-07-26T11:04:32.874 INFO:tasks.workunit.client.0.smithi144.stdout:9/49: dread - d9/de/f10 zero size 2024-07-26T11:04:32.877 INFO:tasks.workunit.client.0.smithi144.stdout:4/25: dread f1 [0,4194304] 0 2024-07-26T11:04:32.897 INFO:tasks.workunit.client.0.smithi144.stdout:9/50: fdatasync d9/fc 0 2024-07-26T11:04:32.897 INFO:tasks.workunit.client.0.smithi144.stdout:9/51: chown d9/de/l12 1937951 1 2024-07-26T11:04:32.947 INFO:tasks.workunit.client.0.smithi144.stdout:1/41: dread f1 [0,4194304] 0 2024-07-26T11:04:32.948 INFO:tasks.workunit.client.0.smithi144.stdout:1/42: getdents d3 0 2024-07-26T11:04:32.948 INFO:tasks.workunit.client.0.smithi144.stdout:1/43: rename d3/d6/da to d3/d6/da/db 22 2024-07-26T11:04:32.952 INFO:tasks.workunit.client.0.smithi144.stdout:1/44: creat d3/fc x:0 0 0 2024-07-26T11:04:32.974 INFO:tasks.workunit.client.0.smithi144.stdout:4/26: dwrite f0 [0,4194304] 0 2024-07-26T11:04:33.006 INFO:tasks.workunit.client.0.smithi144.stdout:1/45: getdents d3 0 2024-07-26T11:04:33.045 INFO:tasks.workunit.client.0.smithi144.stdout:4/27: dwrite f1 [0,4194304] 0 2024-07-26T11:04:33.053 INFO:tasks.workunit.client.0.smithi144.stdout:4/28: creat f3 x:0 0 0 2024-07-26T11:04:33.053 INFO:tasks.workunit.client.0.smithi144.stdout:4/29: write f3 [1026783,42255] 0 2024-07-26T11:04:33.076 INFO:tasks.workunit.client.0.smithi144.stdout:6/15: dread f2 [0,4194304] 0 2024-07-26T11:04:33.076 INFO:tasks.workunit.client.0.smithi144.stdout:6/16: fsync f2 0 2024-07-26T11:04:33.080 INFO:tasks.workunit.client.0.smithi144.stdout:8/31: dwrite f4 [0,4194304] 0 2024-07-26T11:04:33.095 INFO:tasks.workunit.client.0.smithi144.stdout:6/17: creat f4 x:0 0 0 2024-07-26T11:04:33.095 INFO:tasks.workunit.client.0.smithi144.stdout:6/18: rmdir - no directory 2024-07-26T11:04:33.104 INFO:tasks.workunit.client.0.smithi144.stdout:8/32: truncate f4 2216340 0 2024-07-26T11:04:33.117 INFO:tasks.workunit.client.0.smithi144.stdout:0/14: dwrite d0/f4 [0,4194304] 0 2024-07-26T11:04:33.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/33: dread f6 [0,4194304] 0 2024-07-26T11:04:33.121 INFO:tasks.workunit.client.0.smithi144.stdout:0/15: rename d0/f5 to d0/f6 0 2024-07-26T11:04:33.123 INFO:tasks.workunit.client.0.smithi144.stdout:0/16: creat d0/f7 x:0 0 0 2024-07-26T11:04:33.123 INFO:tasks.workunit.client.0.smithi144.stdout:0/17: write d0/f7 [370015,12501] 0 2024-07-26T11:04:33.126 INFO:tasks.workunit.client.0.smithi144.stdout:4/30: dwrite f1 [0,4194304] 0 2024-07-26T11:04:33.143 INFO:tasks.workunit.client.0.smithi144.stdout:4/31: link f1 f4 0 2024-07-26T11:04:33.144 INFO:tasks.workunit.client.0.smithi144.stdout:6/19: dread f0 [0,4194304] 0 2024-07-26T11:04:33.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/20: unlink f1 0 2024-07-26T11:04:33.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/21: chown f2 5258758 1 2024-07-26T11:04:33.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/22: chown l3 63 1 2024-07-26T11:04:33.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/23: fsync f4 0 2024-07-26T11:04:33.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/24: write f4 [79835,20836] 0 2024-07-26T11:04:33.147 INFO:tasks.workunit.client.0.smithi144.stdout:8/34: dread f6 [0,4194304] 0 2024-07-26T11:04:33.147 INFO:tasks.workunit.client.0.smithi144.stdout:8/35: rmdir - no directory 2024-07-26T11:04:33.149 INFO:tasks.workunit.client.0.smithi144.stdout:6/25: mknod c5 0 2024-07-26T11:04:33.161 INFO:tasks.workunit.client.0.smithi144.stdout:8/36: creat f8 x:0 0 0 2024-07-26T11:04:33.163 INFO:tasks.workunit.client.0.smithi144.stdout:8/37: mknod c9 0 2024-07-26T11:04:33.163 INFO:tasks.workunit.client.0.smithi144.stdout:8/38: write f6 [1554563,47787] 0 2024-07-26T11:04:33.165 INFO:tasks.workunit.client.0.smithi144.stdout:8/39: mknod ca 0 2024-07-26T11:04:33.165 INFO:tasks.workunit.client.0.smithi144.stdout:8/40: dread - f8 zero size 2024-07-26T11:04:33.168 INFO:tasks.workunit.client.0.smithi144.stdout:8/41: mknod cb 0 2024-07-26T11:04:33.192 INFO:tasks.workunit.client.0.smithi144.stdout:8/42: read f4 [1071171,26549] 0 2024-07-26T11:04:33.195 INFO:tasks.workunit.client.0.smithi144.stdout:8/43: mknod cc 0 2024-07-26T11:04:33.195 INFO:tasks.workunit.client.0.smithi144.stdout:6/26: dread f2 [0,4194304] 0 2024-07-26T11:04:33.197 INFO:tasks.workunit.client.0.smithi144.stdout:6/27: rename f4 to f6 0 2024-07-26T11:04:33.200 INFO:tasks.workunit.client.0.smithi144.stdout:6/28: rename l3 to l7 0 2024-07-26T11:04:33.213 INFO:tasks.workunit.client.0.smithi144.stdout:0/18: dwrite d0/f6 [0,4194304] 0 2024-07-26T11:04:33.226 INFO:tasks.workunit.client.0.smithi144.stdout:4/32: dread f1 [0,4194304] 0 2024-07-26T11:04:33.254 INFO:tasks.workunit.client.0.smithi144.stdout:5/37: dwrite d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:33.262 INFO:tasks.workunit.client.0.smithi144.stdout:4/33: dread f1 [0,4194304] 0 2024-07-26T11:04:33.271 INFO:tasks.workunit.client.0.smithi144.stdout:5/38: creat d1/d9/f11 x:0 0 0 2024-07-26T11:04:33.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/34: rename f4 to f5 0 2024-07-26T11:04:33.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/35: stat f0 0 2024-07-26T11:04:33.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/36: fdatasync f0 0 2024-07-26T11:04:33.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/37: chown f1 3416 1 2024-07-26T11:04:33.278 INFO:tasks.workunit.client.0.smithi144.stdout:0/19: dwrite d0/f7 [0,4194304] 0 2024-07-26T11:04:33.279 INFO:tasks.workunit.client.0.smithi144.stdout:0/20: chown d0 65514 1 2024-07-26T11:04:33.280 INFO:tasks.workunit.client.0.smithi144.stdout:5/39: mknod d1/d9/c12 0 2024-07-26T11:04:33.280 INFO:tasks.workunit.client.0.smithi144.stdout:5/40: stat d1 0 2024-07-26T11:04:33.285 INFO:tasks.workunit.client.0.smithi144.stdout:4/38: creat f6 x:0 0 0 2024-07-26T11:04:33.289 INFO:tasks.workunit.client.0.smithi144.stdout:0/21: getdents d0 0 2024-07-26T11:04:33.289 INFO:tasks.workunit.client.0.smithi144.stdout:0/22: chown d0/l2 32 1 2024-07-26T11:04:33.291 INFO:tasks.workunit.client.0.smithi144.stdout:4/39: truncate f0 2158994 0 2024-07-26T11:04:33.291 INFO:tasks.workunit.client.0.smithi144.stdout:4/40: fdatasync f1 0 2024-07-26T11:04:33.293 INFO:tasks.workunit.client.0.smithi144.stdout:0/23: creat d0/f8 x:0 0 0 2024-07-26T11:04:33.344 INFO:tasks.workunit.client.0.smithi144.stdout:0/24: dwrite d0/f7 [0,4194304] 0 2024-07-26T11:04:33.344 INFO:tasks.workunit.client.0.smithi144.stdout:0/25: fdatasync d0/f8 0 2024-07-26T11:04:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:33.390 INFO:tasks.workunit.client.0.smithi144.stdout:0/26: dwrite d0/f6 [0,4194304] 0 2024-07-26T11:04:33.392 INFO:tasks.workunit.client.0.smithi144.stdout:0/27: mknod d0/c9 0 2024-07-26T11:04:33.414 INFO:tasks.workunit.client.0.smithi144.stdout:0/28: dread d0/f4 [0,4194304] 0 2024-07-26T11:04:33.415 INFO:tasks.workunit.client.0.smithi144.stdout:0/29: symlink d0/la 0 2024-07-26T11:04:33.417 INFO:tasks.workunit.client.0.smithi144.stdout:0/30: link d0/f7 d0/fb 0 2024-07-26T11:04:33.438 INFO:tasks.workunit.client.0.smithi144.stdout:0/31: dread d0/f7 [0,4194304] 0 2024-07-26T11:04:33.442 INFO:tasks.workunit.client.0.smithi144.stdout:0/32: getdents d0 0 2024-07-26T11:04:33.443 INFO:tasks.workunit.client.0.smithi144.stdout:0/33: symlink d0/lc 0 2024-07-26T11:04:33.443 INFO:tasks.workunit.client.0.smithi144.stdout:0/34: write d0/f8 [432803,91360] 0 2024-07-26T11:04:33.445 INFO:tasks.workunit.client.0.smithi144.stdout:0/35: creat d0/fd x:0 0 0 2024-07-26T11:04:33.445 INFO:tasks.workunit.client.0.smithi144.stdout:0/36: truncate d0/f7 5145333 0 2024-07-26T11:04:33.465 INFO:tasks.workunit.client.0.smithi144.stdout:0/37: write d0/fb [1229085,34527] 0 2024-07-26T11:04:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:33.491 INFO:tasks.workunit.client.0.smithi144.stdout:0/38: dread d0/fb [0,4194304] 0 2024-07-26T11:04:33.549 INFO:tasks.workunit.client.0.smithi144.stdout:0/39: dwrite d0/f4 [0,4194304] 0 2024-07-26T11:04:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:33.565 INFO:tasks.workunit.client.0.smithi144.stdout:0/40: dread d0/f8 [0,4194304] 0 2024-07-26T11:04:33.567 INFO:tasks.workunit.client.0.smithi144.stdout:0/41: mknod d0/ce 0 2024-07-26T11:04:33.567 INFO:tasks.workunit.client.0.smithi144.stdout:0/42: truncate d0/fd 75184 0 2024-07-26T11:04:33.612 INFO:tasks.workunit.client.0.smithi144.stdout:0/43: dwrite d0/fb [4194304,4194304] 0 2024-07-26T11:04:33.612 INFO:tasks.workunit.client.0.smithi144.stdout:0/44: truncate d0/f8 1310041 0 2024-07-26T11:04:33.646 INFO:tasks.workunit.client.0.smithi144.stdout:0/45: dwrite d0/f4 [0,4194304] 0 2024-07-26T11:04:33.646 INFO:tasks.workunit.client.0.smithi144.stdout:0/46: write d0/fb [3857326,108795] 0 2024-07-26T11:04:33.646 INFO:tasks.workunit.client.0.smithi144.stdout:0/47: fdatasync d0/f4 0 2024-07-26T11:04:33.690 INFO:tasks.workunit.client.0.smithi144.stdout:0/48: dwrite d0/fd [0,4194304] 0 2024-07-26T11:04:33.691 INFO:tasks.workunit.client.0.smithi144.stdout:0/49: stat d0/lc 0 2024-07-26T11:04:33.691 INFO:tasks.workunit.client.0.smithi144.stdout:0/50: chown d0/lc 172 1 2024-07-26T11:04:33.692 INFO:tasks.workunit.client.0.smithi144.stdout:0/51: rename d0/fd to d0/ff 0 2024-07-26T11:04:33.692 INFO:tasks.workunit.client.0.smithi144.stdout:0/52: write d0/f8 [448708,28486] 0 2024-07-26T11:04:33.693 INFO:tasks.workunit.client.0.smithi144.stdout:0/53: creat d0/f10 x:0 0 0 2024-07-26T11:04:33.694 INFO:tasks.workunit.client.0.smithi144.stdout:0/54: symlink d0/l11 0 2024-07-26T11:04:33.695 INFO:tasks.workunit.client.0.smithi144.stdout:0/55: fdatasync d0/f10 0 2024-07-26T11:04:33.695 INFO:tasks.workunit.client.0.smithi144.stdout:0/56: fsync d0/f4 0 2024-07-26T11:04:33.721 INFO:tasks.workunit.client.0.smithi144.stdout:0/57: dread d0/f7 [0,4194304] 0 2024-07-26T11:04:33.721 INFO:tasks.workunit.client.0.smithi144.stdout:0/58: fdatasync d0/f10 0 2024-07-26T11:04:33.721 INFO:tasks.workunit.client.0.smithi144.stdout:0/59: chown d0/c9 1537 1 2024-07-26T11:04:33.721 INFO:tasks.workunit.client.0.smithi144.stdout:0/60: stat d0/f7 0 2024-07-26T11:04:33.747 INFO:tasks.workunit.client.0.smithi144.stdout:0/61: dread d0/ff [0,4194304] 0 2024-07-26T11:04:33.747 INFO:tasks.workunit.client.0.smithi144.stdout:0/62: fsync d0/ff 0 2024-07-26T11:04:33.747 INFO:tasks.workunit.client.0.smithi144.stdout:0/63: write d0/f10 [296950,76310] 0 2024-07-26T11:04:33.748 INFO:tasks.workunit.client.0.smithi144.stdout:0/64: rename d0/lc to d0/l12 0 2024-07-26T11:04:33.749 INFO:tasks.workunit.client.0.smithi144.stdout:0/65: mknod d0/c13 0 2024-07-26T11:04:33.750 INFO:tasks.workunit.client.0.smithi144.stdout:0/66: truncate d0/f6 2860917 0 2024-07-26T11:04:33.751 INFO:tasks.workunit.client.0.smithi144.stdout:0/67: stat d0/l2 0 2024-07-26T11:04:33.752 INFO:tasks.workunit.client.0.smithi144.stdout:0/68: mkdir d0/d14 0 2024-07-26T11:04:33.752 INFO:tasks.workunit.client.0.smithi144.stdout:0/69: readlink d0/l2 0 2024-07-26T11:04:33.803 INFO:tasks.workunit.client.0.smithi144.stdout:0/70: dwrite d0/f10 [0,4194304] 0 2024-07-26T11:04:33.803 INFO:tasks.workunit.client.0.smithi144.stdout:0/71: fsync d0/fb 0 2024-07-26T11:04:33.803 INFO:tasks.workunit.client.0.smithi144.stdout:0/72: chown d0 220 1 2024-07-26T11:04:33.863 INFO:tasks.workunit.client.0.smithi144.stdout:0/73: dwrite d0/ff [0,4194304] 0 2024-07-26T11:04:33.888 INFO:tasks.workunit.client.0.smithi144.stdout:0/74: write d0/f10 [2265823,24998] 0 2024-07-26T11:04:33.888 INFO:tasks.workunit.client.0.smithi144.stdout:0/75: chown d0/l12 1 1 2024-07-26T11:04:33.888 INFO:tasks.workunit.client.0.smithi144.stdout:0/76: stat d0/l11 0 2024-07-26T11:04:33.891 INFO:tasks.workunit.client.0.smithi144.stdout:0/77: rename d0/c3 to d0/c15 0 2024-07-26T11:04:33.891 INFO:tasks.workunit.client.0.smithi144.stdout:0/78: chown d0/la 724194574 1 2024-07-26T11:04:33.892 INFO:tasks.workunit.client.0.smithi144.stdout:0/79: link d0/la d0/l16 0 2024-07-26T11:04:33.893 INFO:tasks.workunit.client.0.smithi144.stdout:0/80: symlink d0/l17 0 2024-07-26T11:04:33.909 INFO:tasks.workunit.client.0.smithi144.stdout:0/81: dread d0/f6 [0,4194304] 0 2024-07-26T11:04:33.925 INFO:tasks.workunit.client.0.smithi144.stdout:0/82: dread d0/f6 [0,4194304] 0 2024-07-26T11:04:33.926 INFO:tasks.workunit.client.0.smithi144.stdout:0/83: creat d0/f18 x:0 0 0 2024-07-26T11:04:33.928 INFO:tasks.workunit.client.0.smithi144.stdout:0/84: rename d0/c13 to d0/d14/c19 0 2024-07-26T11:04:33.936 INFO:tasks.workunit.client.0.smithi144.stdout:0/85: truncate d0/f8 1067997 0 2024-07-26T11:04:33.936 INFO:tasks.workunit.client.0.smithi144.stdout:0/86: chown d0/la 2 1 2024-07-26T11:04:33.937 INFO:tasks.workunit.client.0.smithi144.stdout:0/87: fdatasync d0/f4 0 2024-07-26T11:04:33.937 INFO:tasks.workunit.client.0.smithi144.stdout:0/88: chown d0 350224375 1 2024-07-26T11:04:33.937 INFO:tasks.workunit.client.0.smithi144.stdout:0/89: rmdir d0 39 2024-07-26T11:04:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:34 smithi183 ceph-mon[37273]: pgmap v523: 129 pgs: 129 active+clean; 71 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 635 op/s 2024-07-26T11:04:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:34 smithi144 ceph-mon[32129]: pgmap v523: 129 pgs: 129 active+clean; 71 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 635 op/s 2024-07-26T11:04:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:34 smithi160 ceph-mon[36715]: pgmap v523: 129 pgs: 129 active+clean; 71 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 635 op/s 2024-07-26T11:04:35.353 INFO:tasks.workunit.client.0.smithi144.stdout:8/44: sync 2024-07-26T11:04:35.353 INFO:tasks.workunit.client.0.smithi144.stdout:7/37: sync 2024-07-26T11:04:35.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/16: sync 2024-07-26T11:04:35.353 INFO:tasks.workunit.client.0.smithi144.stdout:3/10: sync 2024-07-26T11:04:35.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/17: dread - d0/f1 zero size 2024-07-26T11:04:35.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/18: fsync d0/d6/f8 0 2024-07-26T11:04:35.358 INFO:tasks.workunit.client.0.smithi144.stdout:6/29: sync 2024-07-26T11:04:35.358 INFO:tasks.workunit.client.0.smithi144.stdout:9/52: sync 2024-07-26T11:04:35.358 INFO:tasks.workunit.client.0.smithi144.stdout:1/46: sync 2024-07-26T11:04:35.358 INFO:tasks.workunit.client.0.smithi144.stdout:1/47: fsync d3/fc 0 2024-07-26T11:04:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:04:35.406 INFO:tasks.workunit.client.0.smithi144.stdout:7/38: dread d0/f2 [0,4194304] 0 2024-07-26T11:04:35.411 INFO:tasks.workunit.client.0.smithi144.stdout:7/39: mkdir d0/d10 0 2024-07-26T11:04:35.411 INFO:tasks.workunit.client.0.smithi144.stdout:7/40: write d0/d6/d9/fc [363330,12029] 0 2024-07-26T11:04:35.412 INFO:tasks.workunit.client.0.smithi144.stdout:3/11: read d1/f3 [1684829,47916] 0 2024-07-26T11:04:35.412 INFO:tasks.workunit.client.0.smithi144.stdout:3/12: write d1/f3 [4586240,8841] 0 2024-07-26T11:04:35.418 INFO:tasks.workunit.client.0.smithi144.stdout:7/41: unlink d0/lf 0 2024-07-26T11:04:35.421 INFO:tasks.workunit.client.0.smithi144.stdout:1/48: dwrite d3/fc [0,4194304] 0 2024-07-26T11:04:35.422 INFO:tasks.workunit.client.0.smithi144.stdout:7/42: creat d0/f11 x:0 0 0 2024-07-26T11:04:35.424 INFO:tasks.workunit.client.0.smithi144.stdout:1/49: creat d3/d6/fd x:0 0 0 2024-07-26T11:04:35.425 INFO:tasks.workunit.client.0.smithi144.stdout:1/50: truncate d3/f4 1737828 0 2024-07-26T11:04:35.427 INFO:tasks.workunit.client.0.smithi144.stdout:1/51: symlink d3/d6/le 0 2024-07-26T11:04:35.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/53: dwrite d9/fa [0,4194304] 0 2024-07-26T11:04:35.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/54: dread - d9/de/ff zero size 2024-07-26T11:04:35.455 INFO:tasks.workunit.client.0.smithi144.stdout:7/43: dread d0/f3 [0,4194304] 0 2024-07-26T11:04:35.455 INFO:tasks.workunit.client.0.smithi144.stdout:7/44: write d0/f11 [322004,13052] 0 2024-07-26T11:04:35.458 INFO:tasks.workunit.client.0.smithi144.stdout:6/30: dwrite f2 [0,4194304] 0 2024-07-26T11:04:35.458 INFO:tasks.workunit.client.0.smithi144.stdout:7/45: mkdir d0/d6/d9/d12 0 2024-07-26T11:04:35.461 INFO:tasks.workunit.client.0.smithi144.stdout:9/55: write d9/fa [4739614,22583] 0 2024-07-26T11:04:35.463 INFO:tasks.workunit.client.0.smithi144.stdout:7/46: creat d0/d6/d9/f13 x:0 0 0 2024-07-26T11:04:35.466 INFO:tasks.workunit.client.0.smithi144.stdout:6/31: truncate f2 1110164 0 2024-07-26T11:04:35.467 INFO:tasks.workunit.client.0.smithi144.stdout:9/56: unlink d9/lb 0 2024-07-26T11:04:35.467 INFO:tasks.workunit.client.0.smithi144.stdout:9/57: chown d9/fc 16424 1 2024-07-26T11:04:35.467 INFO:tasks.workunit.client.0.smithi144.stdout:9/58: write f4 [4234806,31384] 0 2024-07-26T11:04:35.467 INFO:tasks.workunit.client.0.smithi144.stdout:7/47: mknod d0/d6/c14 0 2024-07-26T11:04:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:04:35.471 INFO:tasks.workunit.client.0.smithi144.stdout:9/59: creat d9/de/f15 x:0 0 0 2024-07-26T11:04:35.471 INFO:tasks.workunit.client.0.smithi144.stdout:9/60: chown d9/de/ff 16059 1 2024-07-26T11:04:35.475 INFO:tasks.workunit.client.0.smithi144.stdout:1/52: dread d3/fc [0,4194304] 0 2024-07-26T11:04:35.475 INFO:tasks.workunit.client.0.smithi144.stdout:7/48: symlink d0/l15 0 2024-07-26T11:04:35.476 INFO:tasks.workunit.client.0.smithi144.stdout:3/13: dwrite d1/f3 [0,4194304] 0 2024-07-26T11:04:35.476 INFO:tasks.workunit.client.0.smithi144.stdout:9/61: rename l5 to d9/de/d14/l16 0 2024-07-26T11:04:35.478 INFO:tasks.workunit.client.0.smithi144.stdout:7/49: mknod d0/d10/c16 0 2024-07-26T11:04:35.479 INFO:tasks.workunit.client.0.smithi144.stdout:9/62: mknod d9/c17 0 2024-07-26T11:04:35.480 INFO:tasks.workunit.client.0.smithi144.stdout:7/50: creat d0/d6/f17 x:0 0 0 2024-07-26T11:04:35.481 INFO:tasks.workunit.client.0.smithi144.stdout:9/63: link f6 d9/f18 0 2024-07-26T11:04:35.482 INFO:tasks.workunit.client.0.smithi144.stdout:7/51: mknod d0/c18 0 2024-07-26T11:04:35.483 INFO:tasks.workunit.client.0.smithi144.stdout:7/52: write d0/fd [125194,38189] 0 2024-07-26T11:04:35.483 INFO:tasks.workunit.client.0.smithi144.stdout:7/53: chown d0/d10 140298 1 2024-07-26T11:04:35.487 INFO:tasks.workunit.client.0.smithi144.stdout:9/64: fdatasync f4 0 2024-07-26T11:04:35.487 INFO:tasks.workunit.client.0.smithi144.stdout:9/65: chown f4 7 1 2024-07-26T11:04:35.487 INFO:tasks.workunit.client.0.smithi144.stdout:9/66: write d9/de/f10 [575675,45982] 0 2024-07-26T11:04:35.487 INFO:tasks.workunit.client.0.smithi144.stdout:6/32: truncate f2 659412 0 2024-07-26T11:04:35.489 INFO:tasks.workunit.client.0.smithi144.stdout:1/53: dread d3/f4 [0,4194304] 0 2024-07-26T11:04:35.489 INFO:tasks.workunit.client.0.smithi144.stdout:1/54: fdatasync d3/fc 0 2024-07-26T11:04:35.492 INFO:tasks.workunit.client.0.smithi144.stdout:7/54: dread d0/f11 [0,4194304] 0 2024-07-26T11:04:35.493 INFO:tasks.workunit.client.0.smithi144.stdout:9/67: rename l7 to d9/de/l19 0 2024-07-26T11:04:35.496 INFO:tasks.workunit.client.0.smithi144.stdout:1/55: link d3/c7 d3/d6/da/cf 0 2024-07-26T11:04:35.496 INFO:tasks.workunit.client.0.smithi144.stdout:7/55: mknod d0/c19 0 2024-07-26T11:04:35.497 INFO:tasks.workunit.client.0.smithi144.stdout:6/33: mkdir d8 0 2024-07-26T11:04:35.497 INFO:tasks.workunit.client.0.smithi144.stdout:7/56: mknod d0/d6/d9/c1a 0 2024-07-26T11:04:35.499 INFO:tasks.workunit.client.0.smithi144.stdout:7/57: rename d0/c18 to d0/d6/c1b 0 2024-07-26T11:04:35.501 INFO:tasks.workunit.client.0.smithi144.stdout:7/58: symlink d0/l1c 0 2024-07-26T11:04:35.502 INFO:tasks.workunit.client.0.smithi144.stdout:7/59: creat d0/d6/d9/d12/f1d x:0 0 0 2024-07-26T11:04:35.503 INFO:tasks.workunit.client.0.smithi144.stdout:7/60: creat d0/d10/f1e x:0 0 0 2024-07-26T11:04:35.504 INFO:tasks.workunit.client.0.smithi144.stdout:7/61: mknod d0/d6/d9/c1f 0 2024-07-26T11:04:35.506 INFO:tasks.workunit.client.0.smithi144.stdout:7/62: mknod d0/d6/c20 0 2024-07-26T11:04:35.506 INFO:tasks.workunit.client.0.smithi144.stdout:7/63: fdatasync d0/fb 0 2024-07-26T11:04:35.506 INFO:tasks.workunit.client.0.smithi144.stdout:7/64: truncate d0/d6/d9/fe 152138 0 2024-07-26T11:04:35.507 INFO:tasks.workunit.client.0.smithi144.stdout:7/65: symlink d0/d6/d9/d12/l21 0 2024-07-26T11:04:35.510 INFO:tasks.workunit.client.0.smithi144.stdout:7/66: chown d0/d6/d9/fe 24057 1 2024-07-26T11:04:35.510 INFO:tasks.workunit.client.0.smithi144.stdout:7/67: read - d0/d10/f1e zero size 2024-07-26T11:04:35.510 INFO:tasks.workunit.client.0.smithi144.stdout:7/68: readlink d0/d6/d9/d12/l21 0 2024-07-26T11:04:35.513 INFO:tasks.workunit.client.0.smithi144.stdout:6/34: dread f6 [0,4194304] 0 2024-07-26T11:04:35.517 INFO:tasks.workunit.client.0.smithi144.stdout:5/41: sync 2024-07-26T11:04:35.526 INFO:tasks.workunit.client.0.smithi144.stdout:6/35: read f0 [420762,1122] 0 2024-07-26T11:04:35.526 INFO:tasks.workunit.client.0.smithi144.stdout:6/36: getdents d8 0 2024-07-26T11:04:35.526 INFO:tasks.workunit.client.0.smithi144.stdout:6/37: getdents d8 0 2024-07-26T11:04:35.527 INFO:tasks.workunit.client.0.smithi144.stdout:6/38: creat d8/f9 x:0 0 0 2024-07-26T11:04:35.528 INFO:tasks.workunit.client.0.smithi144.stdout:6/39: symlink d8/la 0 2024-07-26T11:04:35.542 INFO:tasks.workunit.client.0.smithi144.stdout:3/14: dwrite d1/f3 [0,4194304] 0 2024-07-26T11:04:35.556 INFO:tasks.workunit.client.0.smithi144.stdout:3/15: creat d1/f4 x:0 0 0 2024-07-26T11:04:35.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/42: write d1/d9/f10 [3167800,95004] 0 2024-07-26T11:04:35.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/43: chown d1/lc 353 1 2024-07-26T11:04:35.666 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:35.667 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:35.667 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/44: rename d1 to d1/d13 22 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:3/16: mkdir d1/d5 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/45: mkdir d1/d14 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/46: write d1/d9/f11 [533427,129602] 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:1/56: read d3/fc [2429647,90560] 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:9/68: dwrite d9/de/ff [0,4194304] 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:9/69: creat d9/f1a x:0 0 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:9/70: symlink d9/de/d14/l1b 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:7/69: dwrite d0/d6/d9/d12/f1d [0,4194304] 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:9/71: symlink d9/de/d14/l1c 0 2024-07-26T11:04:35.670 INFO:tasks.workunit.client.0.smithi144.stdout:9/72: dread - d9/de/f15 zero size 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:9/73: getdents d9 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:9/74: link f6 d9/f1d 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:9/75: mknod d9/de/c1e 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:3/17: write d1/f3 [3129650,35717] 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:3/18: chown d1/l2 0 1 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/40: dwrite f2 [0,4194304] 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/41: mkdir d8/db 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/42: creat d8/fc x:0 0 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/43: readlink l7 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/44: truncate f0 4825953 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/45: rename f0 to d8/fd 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/46: symlink d8/db/le 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/47: read f2 [4597763,120545] 0 2024-07-26T11:04:35.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/48: write d8/f9 [437091,8720] 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:6/49: chown c5 2530 1 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:6/50: chown d8/la 131601 1 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:6/51: fdatasync f2 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:9/76: write f4 [4026089,5783] 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:9/77: write d9/fc [1184164,1520] 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:9/78: write d9/fc [1296343,122048] 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:7/70: dwrite d0/d6/d9/d12/f1d [0,4194304] 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:9/79: rename d9/de/d14/l16 to d9/de/l1f 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:8/45: sync 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:2/19: sync 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:0/90: sync 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:0/91: fdatasync d0/f6 0 2024-07-26T11:04:35.672 INFO:tasks.workunit.client.0.smithi144.stdout:4/41: sync 2024-07-26T11:04:35.673 INFO:tasks.workunit.client.0.smithi144.stdout:4/42: truncate f0 2945132 0 2024-07-26T11:04:35.673 INFO:tasks.workunit.client.0.smithi144.stdout:0/92: mknod d0/c1a 0 2024-07-26T11:04:35.673 INFO:tasks.workunit.client.0.smithi144.stdout:4/43: symlink l7 0 2024-07-26T11:04:35.673 INFO:tasks.workunit.client.0.smithi144.stdout:4/44: rename c2 to c8 0 2024-07-26T11:04:35.673 INFO:tasks.workunit.client.0.smithi144.stdout:0/93: rmdir d0/d14 39 2024-07-26T11:04:35.678 INFO:tasks.workunit.client.0.smithi144.stdout:1/57: dwrite d3/fc [4194304,4194304] 0 2024-07-26T11:04:35.679 INFO:tasks.workunit.client.0.smithi144.stdout:1/58: chown d3/f4 0 1 2024-07-26T11:04:35.696 INFO:tasks.workunit.client.0.smithi144.stdout:8/46: dwrite f5 [0,4194304] 0 2024-07-26T11:04:35.696 INFO:tasks.workunit.client.0.smithi144.stdout:3/19: write d1/f3 [4547655,102035] 0 2024-07-26T11:04:35.697 INFO:tasks.workunit.client.0.smithi144.stdout:5/47: dwrite d1/d9/f11 [0,4194304] 0 2024-07-26T11:04:35.701 INFO:tasks.workunit.client.0.smithi144.stdout:3/20: creat d1/d5/f6 x:0 0 0 2024-07-26T11:04:35.701 INFO:tasks.workunit.client.0.smithi144.stdout:3/21: chown d1 0 1 2024-07-26T11:04:35.701 INFO:tasks.workunit.client.0.smithi144.stdout:3/22: fsync d1/f4 0 2024-07-26T11:04:35.701 INFO:tasks.workunit.client.0.smithi144.stdout:3/23: write d1/f3 [2538541,24174] 0 2024-07-26T11:04:35.701 INFO:tasks.workunit.client.0.smithi144.stdout:3/24: truncate d1/f4 307116 0 2024-07-26T11:04:35.701 INFO:tasks.workunit.client.0.smithi144.stdout:3/25: rename d1 to d1/d5/d7 22 2024-07-26T11:04:35.702 INFO:tasks.workunit.client.0.smithi144.stdout:6/52: dwrite d8/fd [0,4194304] 0 2024-07-26T11:04:35.708 INFO:tasks.workunit.client.0.smithi144.stdout:6/53: mknod d8/cf 0 2024-07-26T11:04:35.714 INFO:tasks.workunit.client.0.smithi144.stdout:9/80: dwrite d9/de/f10 [0,4194304] 0 2024-07-26T11:04:35.717 INFO:tasks.workunit.client.0.smithi144.stdout:1/59: dread f1 [0,4194304] 0 2024-07-26T11:04:35.718 INFO:tasks.workunit.client.0.smithi144.stdout:7/71: dwrite d0/d6/d9/f13 [0,4194304] 0 2024-07-26T11:04:35.726 INFO:tasks.workunit.client.0.smithi144.stdout:2/20: dwrite d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:35.726 INFO:tasks.workunit.client.0.smithi144.stdout:2/21: write d0/f1 [959284,43986] 0 2024-07-26T11:04:35.733 INFO:tasks.workunit.client.0.smithi144.stdout:0/94: dwrite d0/f7 [0,4194304] 0 2024-07-26T11:04:35.733 INFO:tasks.workunit.client.0.smithi144.stdout:0/95: write d0/f4 [4578437,108473] 0 2024-07-26T11:04:35.733 INFO:tasks.workunit.client.0.smithi144.stdout:8/47: dread f4 [0,4194304] 0 2024-07-26T11:04:35.741 INFO:tasks.workunit.client.0.smithi144.stdout:9/81: unlink f6 0 2024-07-26T11:04:35.744 INFO:tasks.workunit.client.0.smithi144.stdout:7/72: rename d0/d6/d9/fe to d0/d10/f22 0 2024-07-26T11:04:35.744 INFO:tasks.workunit.client.0.smithi144.stdout:7/73: truncate d0/d10/f1e 647924 0 2024-07-26T11:04:35.746 INFO:tasks.workunit.client.0.smithi144.stdout:0/96: truncate d0/f10 599783 0 2024-07-26T11:04:35.747 INFO:tasks.workunit.client.0.smithi144.stdout:1/60: truncate d3/f4 478984 0 2024-07-26T11:04:35.749 INFO:tasks.workunit.client.0.smithi144.stdout:0/97: unlink d0/f4 0 2024-07-26T11:04:35.750 INFO:tasks.workunit.client.0.smithi144.stdout:1/61: creat d3/f10 x:0 0 0 2024-07-26T11:04:35.750 INFO:tasks.workunit.client.0.smithi144.stdout:1/62: fdatasync f1 0 2024-07-26T11:04:35.750 INFO:tasks.workunit.client.0.smithi144.stdout:0/98: creat d0/d14/f1b x:0 0 0 2024-07-26T11:04:35.750 INFO:tasks.workunit.client.0.smithi144.stdout:9/82: rename d9/f18 to d9/de/f20 0 2024-07-26T11:04:35.750 INFO:tasks.workunit.client.0.smithi144.stdout:9/83: fsync d9/de/ff 0 2024-07-26T11:04:35.751 INFO:tasks.workunit.client.0.smithi144.stdout:0/99: unlink d0/ff 0 2024-07-26T11:04:35.752 INFO:tasks.workunit.client.0.smithi144.stdout:0/100: dread - d0/f18 zero size 2024-07-26T11:04:35.752 INFO:tasks.workunit.client.0.smithi144.stdout:9/84: creat d9/de/f21 x:0 0 0 2024-07-26T11:04:35.755 INFO:tasks.workunit.client.0.smithi144.stdout:9/85: read - d9/de/f21 zero size 2024-07-26T11:04:35.755 INFO:tasks.workunit.client.0.smithi144.stdout:0/101: symlink d0/d14/l1c 0 2024-07-26T11:04:35.757 INFO:tasks.workunit.client.0.smithi144.stdout:0/102: unlink d0/l12 0 2024-07-26T11:04:35.757 INFO:tasks.workunit.client.0.smithi144.stdout:4/45: dwrite f1 [0,4194304] 0 2024-07-26T11:04:35.759 INFO:tasks.workunit.client.0.smithi144.stdout:0/103: mknod d0/d14/c1d 0 2024-07-26T11:04:35.782 INFO:tasks.workunit.client.0.smithi144.stdout:1/63: read d3/f4 [239231,9094] 0 2024-07-26T11:04:35.791 INFO:tasks.workunit.client.0.smithi144.stdout:5/48: dread d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:35.792 INFO:tasks.workunit.client.0.smithi144.stdout:5/49: rmdir d1 39 2024-07-26T11:04:35.792 INFO:tasks.workunit.client.0.smithi144.stdout:5/50: truncate d1/d9/f10 4625436 0 2024-07-26T11:04:35.796 INFO:tasks.workunit.client.0.smithi144.stdout:5/51: mknod d1/d14/c15 0 2024-07-26T11:04:35.796 INFO:tasks.workunit.client.0.smithi144.stdout:1/64: dread d3/f4 [0,4194304] 0 2024-07-26T11:04:35.797 INFO:tasks.workunit.client.0.smithi144.stdout:3/26: dwrite d1/d5/f6 [0,4194304] 0 2024-07-26T11:04:35.799 INFO:tasks.workunit.client.0.smithi144.stdout:1/65: creat d3/d6/f11 x:0 0 0 2024-07-26T11:04:35.800 INFO:tasks.workunit.client.0.smithi144.stdout:8/48: dwrite f5 [0,4194304] 0 2024-07-26T11:04:35.802 INFO:tasks.workunit.client.0.smithi144.stdout:1/66: creat d3/d6/da/f12 x:0 0 0 2024-07-26T11:04:35.803 INFO:tasks.workunit.client.0.smithi144.stdout:1/67: chown d3/fc 6465 1 2024-07-26T11:04:35.803 INFO:tasks.workunit.client.0.smithi144.stdout:1/68: read - d3/f10 zero size 2024-07-26T11:04:35.803 INFO:tasks.workunit.client.0.smithi144.stdout:1/69: chown f1 161 1 2024-07-26T11:04:35.803 INFO:tasks.workunit.client.0.smithi144.stdout:1/70: write d3/d6/f11 [888238,61037] 0 2024-07-26T11:04:35.804 INFO:tasks.workunit.client.0.smithi144.stdout:1/71: mkdir d3/d6/da/d13 0 2024-07-26T11:04:35.805 INFO:tasks.workunit.client.0.smithi144.stdout:1/72: mknod d3/d6/da/d13/c14 0 2024-07-26T11:04:35.806 INFO:tasks.workunit.client.0.smithi144.stdout:1/73: write d3/d6/fd [831343,105336] 0 2024-07-26T11:04:35.808 INFO:tasks.workunit.client.0.smithi144.stdout:2/22: dread d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:35.818 INFO:tasks.workunit.client.0.smithi144.stdout:6/54: dwrite d8/f9 [0,4194304] 0 2024-07-26T11:04:35.818 INFO:tasks.workunit.client.0.smithi144.stdout:1/74: symlink d3/d6/l15 0 2024-07-26T11:04:35.818 INFO:tasks.workunit.client.0.smithi144.stdout:6/55: write f6 [785181,27714] 0 2024-07-26T11:04:35.819 INFO:tasks.workunit.client.0.smithi144.stdout:2/23: mkdir d0/d6/d9 0 2024-07-26T11:04:35.819 INFO:tasks.workunit.client.0.smithi144.stdout:2/24: fdatasync d0/f1 0 2024-07-26T11:04:35.820 INFO:tasks.workunit.client.0.smithi144.stdout:1/75: mknod d3/d6/da/d13/c16 0 2024-07-26T11:04:35.821 INFO:tasks.workunit.client.0.smithi144.stdout:1/76: dread - d3/f10 zero size 2024-07-26T11:04:35.821 INFO:tasks.workunit.client.0.smithi144.stdout:2/25: creat d0/fa x:0 0 0 2024-07-26T11:04:35.822 INFO:tasks.workunit.client.0.smithi144.stdout:2/26: symlink d0/lb 0 2024-07-26T11:04:35.822 INFO:tasks.workunit.client.0.smithi144.stdout:8/49: write f5 [772195,17532] 0 2024-07-26T11:04:35.822 INFO:tasks.workunit.client.0.smithi144.stdout:8/50: read f5 [1235568,125626] 0 2024-07-26T11:04:35.823 INFO:tasks.workunit.client.0.smithi144.stdout:2/27: unlink d0/d6/f8 0 2024-07-26T11:04:35.824 INFO:tasks.workunit.client.0.smithi144.stdout:8/51: creat fd x:0 0 0 2024-07-26T11:04:35.826 INFO:tasks.workunit.client.0.smithi144.stdout:7/74: dwrite d0/f3 [0,4194304] 0 2024-07-26T11:04:35.826 INFO:tasks.workunit.client.0.smithi144.stdout:2/28: mknod d0/cc 0 2024-07-26T11:04:35.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/46: dwrite f6 [0,4194304] 0 2024-07-26T11:04:35.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/47: fdatasync f3 0 2024-07-26T11:04:35.827 INFO:tasks.workunit.client.0.smithi144.stdout:8/52: creat fe x:0 0 0 2024-07-26T11:04:35.830 INFO:tasks.workunit.client.0.smithi144.stdout:2/29: link d0/c7 d0/d6/d9/cd 0 2024-07-26T11:04:35.830 INFO:tasks.workunit.client.0.smithi144.stdout:7/75: symlink d0/l23 0 2024-07-26T11:04:35.833 INFO:tasks.workunit.client.0.smithi144.stdout:7/76: truncate d0/d6/f8 143498 0 2024-07-26T11:04:35.833 INFO:tasks.workunit.client.0.smithi144.stdout:7/77: write d0/d6/f8 [760463,28947] 0 2024-07-26T11:04:35.833 INFO:tasks.workunit.client.0.smithi144.stdout:7/78: readlink d0/l15 0 2024-07-26T11:04:35.833 INFO:tasks.workunit.client.0.smithi144.stdout:4/48: mkdir d9 0 2024-07-26T11:04:35.833 INFO:tasks.workunit.client.0.smithi144.stdout:4/49: getdents d9 0 2024-07-26T11:04:35.836 INFO:tasks.workunit.client.0.smithi144.stdout:2/30: link d0/d6/d9/cd d0/d6/ce 0 2024-07-26T11:04:35.837 INFO:tasks.workunit.client.0.smithi144.stdout:7/79: mknod d0/d10/c24 0 2024-07-26T11:04:35.839 INFO:tasks.workunit.client.0.smithi144.stdout:4/50: mkdir d9/da 0 2024-07-26T11:04:35.839 INFO:tasks.workunit.client.0.smithi144.stdout:2/31: truncate d0/d2/f4 2296038 0 2024-07-26T11:04:35.840 INFO:tasks.workunit.client.0.smithi144.stdout:5/52: dwrite d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:35.842 INFO:tasks.workunit.client.0.smithi144.stdout:7/80: mkdir d0/d6/d9/d25 0 2024-07-26T11:04:35.842 INFO:tasks.workunit.client.0.smithi144.stdout:4/51: rmdir d9 39 2024-07-26T11:04:35.842 INFO:tasks.workunit.client.0.smithi144.stdout:5/53: rmdir d1/d9 39 2024-07-26T11:04:35.844 INFO:tasks.workunit.client.0.smithi144.stdout:4/52: link f6 d9/fb 0 2024-07-26T11:04:35.846 INFO:tasks.workunit.client.0.smithi144.stdout:5/54: link d1/d9/f10 d1/d14/f16 0 2024-07-26T11:04:35.848 INFO:tasks.workunit.client.0.smithi144.stdout:4/53: creat d9/da/fc x:0 0 0 2024-07-26T11:04:35.849 INFO:tasks.workunit.client.0.smithi144.stdout:3/27: dwrite d1/d5/f6 [0,4194304] 0 2024-07-26T11:04:35.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/81: write d0/d10/f1e [506961,106804] 0 2024-07-26T11:04:35.851 INFO:tasks.workunit.client.0.smithi144.stdout:5/55: mknod d1/d9/c17 0 2024-07-26T11:04:35.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/82: chown d0/d6/c20 256869416 1 2024-07-26T11:04:35.851 INFO:tasks.workunit.client.0.smithi144.stdout:1/77: dread f1 [0,4194304] 0 2024-07-26T11:04:35.852 INFO:tasks.workunit.client.0.smithi144.stdout:8/53: getdents . 0 2024-07-26T11:04:35.852 INFO:tasks.workunit.client.0.smithi144.stdout:3/28: getdents d1/d5 0 2024-07-26T11:04:35.852 INFO:tasks.workunit.client.0.smithi144.stdout:3/29: write d1/f4 [639240,10316] 0 2024-07-26T11:04:35.854 INFO:tasks.workunit.client.0.smithi144.stdout:5/56: mkdir d1/d18 0 2024-07-26T11:04:35.854 INFO:tasks.workunit.client.0.smithi144.stdout:5/57: readlink d1/lc 0 2024-07-26T11:04:35.858 INFO:tasks.workunit.client.0.smithi144.stdout:1/78: symlink d3/d6/da/l17 0 2024-07-26T11:04:35.858 INFO:tasks.workunit.client.0.smithi144.stdout:8/54: mkdir df 0 2024-07-26T11:04:35.858 INFO:tasks.workunit.client.0.smithi144.stdout:3/30: symlink d1/l8 0 2024-07-26T11:04:35.858 INFO:tasks.workunit.client.0.smithi144.stdout:3/31: truncate d1/f4 1478725 0 2024-07-26T11:04:35.867 INFO:tasks.workunit.client.0.smithi144.stdout:7/83: getdents d0/d6/d9 0 2024-07-26T11:04:35.867 INFO:tasks.workunit.client.0.smithi144.stdout:5/58: creat d1/d9/f19 x:0 0 0 2024-07-26T11:04:35.867 INFO:tasks.workunit.client.0.smithi144.stdout:5/59: readlink d1/l8 0 2024-07-26T11:04:35.867 INFO:tasks.workunit.client.0.smithi144.stdout:8/55: rmdir df 0 2024-07-26T11:04:35.867 INFO:tasks.workunit.client.0.smithi144.stdout:1/79: creat d3/d6/f18 x:0 0 0 2024-07-26T11:04:35.868 INFO:tasks.workunit.client.0.smithi144.stdout:6/56: dread d8/f9 [0,4194304] 0 2024-07-26T11:04:35.868 INFO:tasks.workunit.client.0.smithi144.stdout:6/57: readlink d8/la 0 2024-07-26T11:04:35.868 INFO:tasks.workunit.client.0.smithi144.stdout:6/58: chown d8/cf 3 1 2024-07-26T11:04:35.869 INFO:tasks.workunit.client.0.smithi144.stdout:3/32: unlink d1/f4 0 2024-07-26T11:04:35.869 INFO:tasks.workunit.client.0.smithi144.stdout:3/33: fsync d1/d5/f6 0 2024-07-26T11:04:35.873 INFO:tasks.workunit.client.0.smithi144.stdout:7/84: creat d0/d10/f26 x:0 0 0 2024-07-26T11:04:35.873 INFO:tasks.workunit.client.0.smithi144.stdout:7/85: chown d0/d10/f1e 136148 1 2024-07-26T11:04:35.873 INFO:tasks.workunit.client.0.smithi144.stdout:8/56: link ca c10 0 2024-07-26T11:04:35.873 INFO:tasks.workunit.client.0.smithi144.stdout:8/57: fdatasync f8 0 2024-07-26T11:04:35.873 INFO:tasks.workunit.client.0.smithi144.stdout:8/58: dread - fd zero size 2024-07-26T11:04:35.875 INFO:tasks.workunit.client.0.smithi144.stdout:8/59: rename fd to f11 0 2024-07-26T11:04:35.875 INFO:tasks.workunit.client.0.smithi144.stdout:8/60: write f11 [51134,116901] 0 2024-07-26T11:04:35.881 INFO:tasks.workunit.client.0.smithi144.stdout:2/32: read d0/d2/f4 [509251,57242] 0 2024-07-26T11:04:35.881 INFO:tasks.workunit.client.0.smithi144.stdout:2/33: dread - d0/fa zero size 2024-07-26T11:04:35.881 INFO:tasks.workunit.client.0.smithi144.stdout:8/61: getdents . 0 2024-07-26T11:04:35.881 INFO:tasks.workunit.client.0.smithi144.stdout:8/62: rmdir - no directory 2024-07-26T11:04:35.883 INFO:tasks.workunit.client.0.smithi144.stdout:8/63: mknod c12 0 2024-07-26T11:04:35.884 INFO:tasks.workunit.client.0.smithi144.stdout:8/64: symlink l13 0 2024-07-26T11:04:35.885 INFO:tasks.workunit.client.0.smithi144.stdout:8/65: write f11 [644119,99561] 0 2024-07-26T11:04:35.885 INFO:tasks.workunit.client.0.smithi144.stdout:8/66: chown f5 42671 1 2024-07-26T11:04:35.885 INFO:tasks.workunit.client.0.smithi144.stdout:8/67: readlink l13 0 2024-07-26T11:04:35.887 INFO:tasks.workunit.client.0.smithi144.stdout:8/68: symlink l14 0 2024-07-26T11:04:35.889 INFO:tasks.workunit.client.0.smithi144.stdout:8/69: creat f15 x:0 0 0 2024-07-26T11:04:35.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/70: creat f16 x:0 0 0 2024-07-26T11:04:35.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/71: dread - f8 zero size 2024-07-26T11:04:35.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/72: write f16 [986479,57777] 0 2024-07-26T11:04:35.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/73: truncate f15 818617 0 2024-07-26T11:04:35.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/74: write f5 [656120,16140] 0 2024-07-26T11:04:35.897 INFO:tasks.workunit.client.0.smithi144.stdout:6/59: write f6 [654892,13916] 0 2024-07-26T11:04:35.899 INFO:tasks.workunit.client.0.smithi144.stdout:6/60: link f2 d8/f10 0 2024-07-26T11:04:35.899 INFO:tasks.workunit.client.0.smithi144.stdout:6/61: chown l7 201635703 1 2024-07-26T11:04:35.902 INFO:tasks.workunit.client.0.smithi144.stdout:5/60: dread d1/d14/f16 [0,4194304] 0 2024-07-26T11:04:35.925 INFO:tasks.workunit.client.0.smithi144.stdout:4/54: dwrite f0 [0,4194304] 0 2024-07-26T11:04:35.926 INFO:tasks.workunit.client.0.smithi144.stdout:4/55: creat d9/da/fd x:0 0 0 2024-07-26T11:04:35.926 INFO:tasks.workunit.client.0.smithi144.stdout:4/56: chown d9 643005 1 2024-07-26T11:04:35.926 INFO:tasks.workunit.client.0.smithi144.stdout:4/57: fdatasync f5 0 2024-07-26T11:04:35.927 INFO:tasks.workunit.client.0.smithi144.stdout:7/86: dwrite d0/d10/f22 [0,4194304] 0 2024-07-26T11:04:35.927 INFO:tasks.workunit.client.0.smithi144.stdout:7/87: chown d0/d6/c14 26 1 2024-07-26T11:04:35.928 INFO:tasks.workunit.client.0.smithi144.stdout:7/88: creat d0/d6/d9/f27 x:0 0 0 2024-07-26T11:04:35.928 INFO:tasks.workunit.client.0.smithi144.stdout:7/89: write d0/d10/f1e [513057,37875] 0 2024-07-26T11:04:35.931 INFO:tasks.workunit.client.0.smithi144.stdout:7/90: mknod d0/d10/c28 0 2024-07-26T11:04:35.943 INFO:tasks.workunit.client.0.smithi144.stdout:1/80: dwrite d3/fc [8388608,4194304] 0 2024-07-26T11:04:35.945 INFO:tasks.workunit.client.0.smithi144.stdout:1/81: mkdir d3/d6/da/d13/d19 0 2024-07-26T11:04:35.947 INFO:tasks.workunit.client.0.smithi144.stdout:1/82: link d3/d6/l15 d3/d6/da/l1a 0 2024-07-26T11:04:35.949 INFO:tasks.workunit.client.0.smithi144.stdout:3/34: dwrite d1/d5/f6 [4194304,4194304] 0 2024-07-26T11:04:35.954 INFO:tasks.workunit.client.0.smithi144.stdout:3/35: link l0 d1/l9 0 2024-07-26T11:04:35.962 INFO:tasks.workunit.client.0.smithi144.stdout:8/75: dwrite f6 [0,4194304] 0 2024-07-26T11:04:35.966 INFO:tasks.workunit.client.0.smithi144.stdout:8/76: creat f17 x:0 0 0 2024-07-26T11:04:35.967 INFO:tasks.workunit.client.0.smithi144.stdout:8/77: write fe [844618,66887] 0 2024-07-26T11:04:35.968 INFO:tasks.workunit.client.0.smithi144.stdout:1/83: dread d3/d6/fd [0,4194304] 0 2024-07-26T11:04:35.972 INFO:tasks.workunit.client.0.smithi144.stdout:6/62: dwrite f6 [0,4194304] 0 2024-07-26T11:04:35.974 INFO:tasks.workunit.client.0.smithi144.stdout:1/84: creat d3/d6/f1b x:0 0 0 2024-07-26T11:04:35.975 INFO:tasks.workunit.client.0.smithi144.stdout:6/63: mknod d8/db/c11 0 2024-07-26T11:04:35.976 INFO:tasks.workunit.client.0.smithi144.stdout:1/85: symlink d3/d6/da/d13/l1c 0 2024-07-26T11:04:35.976 INFO:tasks.workunit.client.0.smithi144.stdout:1/86: fsync d3/d6/da/f12 0 2024-07-26T11:04:35.977 INFO:tasks.workunit.client.0.smithi144.stdout:1/87: fsync d3/fc 0 2024-07-26T11:04:35.977 INFO:tasks.workunit.client.0.smithi144.stdout:1/88: chown d3/d6/da/d13/d19 28387 1 2024-07-26T11:04:35.977 INFO:tasks.workunit.client.0.smithi144.stdout:6/64: creat d8/db/f12 x:0 0 0 2024-07-26T11:04:35.978 INFO:tasks.workunit.client.0.smithi144.stdout:8/78: getdents . 0 2024-07-26T11:04:35.979 INFO:tasks.workunit.client.0.smithi144.stdout:5/61: write d1/d9/f11 [1764210,73799] 0 2024-07-26T11:04:35.979 INFO:tasks.workunit.client.0.smithi144.stdout:2/34: dwrite d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:35.980 INFO:tasks.workunit.client.0.smithi144.stdout:6/65: creat d8/db/f13 x:0 0 0 2024-07-26T11:04:35.981 INFO:tasks.workunit.client.0.smithi144.stdout:8/79: mknod c18 0 2024-07-26T11:04:35.981 INFO:tasks.workunit.client.0.smithi144.stdout:8/80: chown c18 0 1 2024-07-26T11:04:35.985 INFO:tasks.workunit.client.0.smithi144.stdout:6/66: symlink d8/l14 0 2024-07-26T11:04:35.988 INFO:tasks.workunit.client.0.smithi144.stdout:6/67: mkdir d8/d15 0 2024-07-26T11:04:35.989 INFO:tasks.workunit.client.0.smithi144.stdout:8/81: rename ca to c19 0 2024-07-26T11:04:35.993 INFO:tasks.workunit.client.0.smithi144.stdout:7/91: dwrite d0/f7 [0,4194304] 0 2024-07-26T11:04:35.996 INFO:tasks.workunit.client.0.smithi144.stdout:7/92: unlink d0/fb 0 2024-07-26T11:04:36.009 INFO:tasks.workunit.client.0.smithi144.stdout:1/89: dwrite d3/d6/f18 [0,4194304] 0 2024-07-26T11:04:36.012 INFO:tasks.workunit.client.0.smithi144.stdout:1/90: symlink d3/d6/da/l1d 0 2024-07-26T11:04:36.015 INFO:tasks.workunit.client.0.smithi144.stdout:6/68: dread d8/f10 [0,4194304] 0 2024-07-26T11:04:36.017 INFO:tasks.workunit.client.0.smithi144.stdout:6/69: creat d8/f16 x:0 0 0 2024-07-26T11:04:36.018 INFO:tasks.workunit.client.0.smithi144.stdout:6/70: dread - d8/f16 zero size 2024-07-26T11:04:36.018 INFO:tasks.workunit.client.0.smithi144.stdout:5/62: dwrite d1/d14/f16 [4194304,4194304] 0 2024-07-26T11:04:36.019 INFO:tasks.workunit.client.0.smithi144.stdout:6/71: rmdir d8/db 39 2024-07-26T11:04:36.028 INFO:tasks.workunit.client.0.smithi144.stdout:7/93: dread d0/f7 [0,4194304] 0 2024-07-26T11:04:36.030 INFO:tasks.workunit.client.0.smithi144.stdout:6/72: rename d8/db/f12 to d8/f17 0 2024-07-26T11:04:36.032 INFO:tasks.workunit.client.0.smithi144.stdout:7/94: mknod d0/c29 0 2024-07-26T11:04:36.034 INFO:tasks.workunit.client.0.smithi144.stdout:7/95: link d0/d6/f8 d0/d10/f2a 0 2024-07-26T11:04:36.038 INFO:tasks.workunit.client.0.smithi144.stdout:2/35: dwrite d0/d2/f4 [4194304,4194304] 0 2024-07-26T11:04:36.038 INFO:tasks.workunit.client.0.smithi144.stdout:6/73: getdents d8/db 0 2024-07-26T11:04:36.044 INFO:tasks.workunit.client.0.smithi144.stdout:2/36: rmdir d0 39 2024-07-26T11:04:36.044 INFO:tasks.workunit.client.0.smithi144.stdout:8/82: dwrite f15 [0,4194304] 0 2024-07-26T11:04:36.045 INFO:tasks.workunit.client.0.smithi144.stdout:6/74: truncate f6 3435999 0 2024-07-26T11:04:36.045 INFO:tasks.workunit.client.0.smithi144.stdout:6/75: getdents d8/d15 0 2024-07-26T11:04:36.046 INFO:tasks.workunit.client.0.smithi144.stdout:8/83: readlink l13 0 2024-07-26T11:04:36.048 INFO:tasks.workunit.client.0.smithi144.stdout:8/84: fsync f11 0 2024-07-26T11:04:36.050 INFO:tasks.workunit.client.0.smithi144.stdout:8/85: mkdir d1a 0 2024-07-26T11:04:36.052 INFO:tasks.workunit.client.0.smithi144.stdout:8/86: creat d1a/f1b x:0 0 0 2024-07-26T11:04:36.053 INFO:tasks.workunit.client.0.smithi144.stdout:8/87: chown c10 1584192 1 2024-07-26T11:04:36.055 INFO:tasks.workunit.client.0.smithi144.stdout:8/88: mkdir d1a/d1c 0 2024-07-26T11:04:36.056 INFO:tasks.workunit.client.0.smithi144.stdout:8/89: mkdir d1a/d1c/d1d 0 2024-07-26T11:04:36.057 INFO:tasks.workunit.client.0.smithi144.stdout:8/90: truncate d1a/f1b 1013906 0 2024-07-26T11:04:36.064 INFO:tasks.workunit.client.0.smithi144.stdout:1/91: dwrite d3/fc [8388608,4194304] 0 2024-07-26T11:04:36.065 INFO:tasks.workunit.client.0.smithi144.stdout:1/92: read - d3/d6/da/f12 zero size 2024-07-26T11:04:36.065 INFO:tasks.workunit.client.0.smithi144.stdout:6/76: write d8/f9 [1708672,15763] 0 2024-07-26T11:04:36.065 INFO:tasks.workunit.client.0.smithi144.stdout:6/77: chown d8/fd 16188 1 2024-07-26T11:04:36.065 INFO:tasks.workunit.client.0.smithi144.stdout:6/78: write d8/f10 [5366011,91343] 0 2024-07-26T11:04:36.065 INFO:tasks.workunit.client.0.smithi144.stdout:1/93: creat d3/f1e x:0 0 0 2024-07-26T11:04:36.066 INFO:tasks.workunit.client.0.smithi144.stdout:6/79: mknod d8/d15/c18 0 2024-07-26T11:04:36.066 INFO:tasks.workunit.client.0.smithi144.stdout:6/80: write d8/fc [591662,47491] 0 2024-07-26T11:04:36.077 INFO:tasks.workunit.client.0.smithi144.stdout:6/81: truncate f2 3861094 0 2024-07-26T11:04:36.079 INFO:tasks.workunit.client.0.smithi144.stdout:6/82: rename d8/la to d8/d15/l19 0 2024-07-26T11:04:36.080 INFO:tasks.workunit.client.0.smithi144.stdout:5/63: dwrite d1/d9/f11 [0,4194304] 0 2024-07-26T11:04:36.082 INFO:tasks.workunit.client.0.smithi144.stdout:5/64: mknod d1/d9/c1a 0 2024-07-26T11:04:36.082 INFO:tasks.workunit.client.0.smithi144.stdout:2/37: dread d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:36.087 INFO:tasks.workunit.client.0.smithi144.stdout:5/65: mkdir d1/d1b 0 2024-07-26T11:04:36.087 INFO:tasks.workunit.client.0.smithi144.stdout:5/66: fdatasync d1/d14/f16 0 2024-07-26T11:04:36.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/67: link d1/d9/c12 d1/d5/c1c 0 2024-07-26T11:04:36.091 INFO:tasks.workunit.client.0.smithi144.stdout:5/68: creat d1/d18/f1d x:0 0 0 2024-07-26T11:04:36.095 INFO:tasks.workunit.client.0.smithi144.stdout:9/86: sync 2024-07-26T11:04:36.095 INFO:tasks.workunit.client.0.smithi144.stdout:7/96: dwrite d0/d6/f8 [0,4194304] 0 2024-07-26T11:04:36.098 INFO:tasks.workunit.client.0.smithi144.stdout:9/87: unlink d9/de/f10 0 2024-07-26T11:04:36.104 INFO:tasks.workunit.client.0.smithi144.stdout:3/36: sync 2024-07-26T11:04:36.104 INFO:tasks.workunit.client.0.smithi144.stdout:0/104: sync 2024-07-26T11:04:36.104 INFO:tasks.workunit.client.0.smithi144.stdout:7/97: truncate d0/d10/f2a 291751 0 2024-07-26T11:04:36.104 INFO:tasks.workunit.client.0.smithi144.stdout:0/105: mkdir d0/d14/d1e 0 2024-07-26T11:04:36.104 INFO:tasks.workunit.client.0.smithi144.stdout:0/106: dread - d0/d14/f1b zero size 2024-07-26T11:04:36.105 INFO:tasks.workunit.client.0.smithi144.stdout:7/98: creat d0/d6/f2b x:0 0 0 2024-07-26T11:04:36.106 INFO:tasks.workunit.client.0.smithi144.stdout:7/99: symlink d0/d6/d9/d12/l2c 0 2024-07-26T11:04:36.106 INFO:tasks.workunit.client.0.smithi144.stdout:0/107: mknod d0/d14/c1f 0 2024-07-26T11:04:36.107 INFO:tasks.workunit.client.0.smithi144.stdout:7/100: creat d0/f2d x:0 0 0 2024-07-26T11:04:36.107 INFO:tasks.workunit.client.0.smithi144.stdout:0/108: symlink d0/d14/d1e/l20 0 2024-07-26T11:04:36.109 INFO:tasks.workunit.client.0.smithi144.stdout:2/38: dread d0/f1 [0,4194304] 0 2024-07-26T11:04:36.109 INFO:tasks.workunit.client.0.smithi144.stdout:2/39: fdatasync d0/f1 0 2024-07-26T11:04:36.111 INFO:tasks.workunit.client.0.smithi144.stdout:0/109: creat d0/d14/d1e/f21 x:0 0 0 2024-07-26T11:04:36.120 INFO:tasks.workunit.client.0.smithi144.stdout:0/110: symlink d0/d14/l22 0 2024-07-26T11:04:36.121 INFO:tasks.workunit.client.0.smithi144.stdout:4/58: sync 2024-07-26T11:04:36.125 INFO:tasks.workunit.client.0.smithi144.stdout:0/111: rename d0/la to d0/d14/d1e/l23 0 2024-07-26T11:04:36.157 INFO:tasks.workunit.client.0.smithi144.stdout:5/69: dwrite d1/d9/f10 [4194304,4194304] 0 2024-07-26T11:04:36.158 INFO:tasks.workunit.client.0.smithi144.stdout:1/94: dwrite d3/d6/f11 [0,4194304] 0 2024-07-26T11:04:36.160 INFO:tasks.workunit.client.0.smithi144.stdout:5/70: symlink d1/d9/l1e 0 2024-07-26T11:04:36.161 INFO:tasks.workunit.client.0.smithi144.stdout:1/95: rmdir d3 39 2024-07-26T11:04:36.162 INFO:tasks.workunit.client.0.smithi144.stdout:1/96: symlink d3/l1f 0 2024-07-26T11:04:36.163 INFO:tasks.workunit.client.0.smithi144.stdout:1/97: truncate d3/f10 702683 0 2024-07-26T11:04:36.167 INFO:tasks.workunit.client.0.smithi144.stdout:1/98: symlink d3/l20 0 2024-07-26T11:04:36.167 INFO:tasks.workunit.client.0.smithi144.stdout:1/99: truncate f1 4197905 0 2024-07-26T11:04:36.179 INFO:tasks.workunit.client.0.smithi144.stdout:3/37: dwrite d1/d5/f6 [4194304,4194304] 0 2024-07-26T11:04:36.179 INFO:tasks.workunit.client.0.smithi144.stdout:7/101: dwrite d0/f2 [0,4194304] 0 2024-07-26T11:04:36.179 INFO:tasks.workunit.client.0.smithi144.stdout:0/112: dwrite d0/f18 [0,4194304] 0 2024-07-26T11:04:36.181 INFO:tasks.workunit.client.0.smithi144.stdout:3/38: rename d1/f3 to d1/d5/fa 0 2024-07-26T11:04:36.182 INFO:tasks.workunit.client.0.smithi144.stdout:4/59: write f6 [3973701,106425] 0 2024-07-26T11:04:36.184 INFO:tasks.workunit.client.0.smithi144.stdout:9/88: dwrite f4 [0,4194304] 0 2024-07-26T11:04:36.184 INFO:tasks.workunit.client.0.smithi144.stdout:7/102: link d0/d6/d9/d12/f1d d0/d6/f2e 0 2024-07-26T11:04:36.198 INFO:tasks.workunit.client.0.smithi144.stdout:8/91: dwrite fe [0,4194304] 0 2024-07-26T11:04:36.201 INFO:tasks.workunit.client.0.smithi144.stdout:9/89: truncate d9/de/f20 4486872 0 2024-07-26T11:04:36.205 INFO:tasks.workunit.client.0.smithi144.stdout:8/92: chown c12 1244 1 2024-07-26T11:04:36.207 INFO:tasks.workunit.client.0.smithi144.stdout:9/90: creat d9/de/d14/f22 x:0 0 0 2024-07-26T11:04:36.208 INFO:tasks.workunit.client.0.smithi144.stdout:8/93: stat f5 0 2024-07-26T11:04:36.208 INFO:tasks.workunit.client.0.smithi144.stdout:9/91: write d9/de/f21 [489081,33095] 0 2024-07-26T11:04:36.209 INFO:tasks.workunit.client.0.smithi144.stdout:8/94: mknod d1a/d1c/c1e 0 2024-07-26T11:04:36.210 INFO:tasks.workunit.client.0.smithi144.stdout:8/95: dread - f17 zero size 2024-07-26T11:04:36.211 INFO:tasks.workunit.client.0.smithi144.stdout:8/96: write f8 [207423,91334] 0 2024-07-26T11:04:36.212 INFO:tasks.workunit.client.0.smithi144.stdout:1/100: dwrite d3/d6/f18 [0,4194304] 0 2024-07-26T11:04:36.212 INFO:tasks.workunit.client.0.smithi144.stdout:1/101: write d3/d6/fd [1459683,73827] 0 2024-07-26T11:04:36.212 INFO:tasks.workunit.client.0.smithi144.stdout:1/102: write d3/f10 [1079037,46697] 0 2024-07-26T11:04:36.214 INFO:tasks.workunit.client.0.smithi144.stdout:7/103: write d0/d6/d9/f13 [3793471,4667] 0 2024-07-26T11:04:36.214 INFO:tasks.workunit.client.0.smithi144.stdout:7/104: write d0/d10/f1e [1195797,78076] 0 2024-07-26T11:04:36.215 INFO:tasks.workunit.client.0.smithi144.stdout:5/71: dwrite d1/d14/f16 [4194304,4194304] 0 2024-07-26T11:04:36.222 INFO:tasks.workunit.client.0.smithi144.stdout:7/105: mkdir d0/d6/d9/d25/d2f 0 2024-07-26T11:04:36.222 INFO:tasks.workunit.client.0.smithi144.stdout:1/103: mkdir d3/d6/da/d21 0 2024-07-26T11:04:36.222 INFO:tasks.workunit.client.0.smithi144.stdout:8/97: link f16 d1a/d1c/f1f 0 2024-07-26T11:04:36.224 INFO:tasks.workunit.client.0.smithi144.stdout:8/98: rename f16 to d1a/f20 0 2024-07-26T11:04:36.227 INFO:tasks.workunit.client.0.smithi144.stdout:8/99: read f8 [173144,121425] 0 2024-07-26T11:04:36.237 INFO:tasks.workunit.client.0.smithi144.stdout:3/39: write d1/d5/f6 [2322349,115068] 0 2024-07-26T11:04:36.237 INFO:tasks.workunit.client.0.smithi144.stdout:3/40: chown d1/d5/f6 260182193 1 2024-07-26T11:04:36.247 INFO:tasks.workunit.client.0.smithi144.stdout:4/60: dwrite d9/fb [0,4194304] 0 2024-07-26T11:04:36.247 INFO:tasks.workunit.client.0.smithi144.stdout:4/61: readlink l7 0 2024-07-26T11:04:36.249 INFO:tasks.workunit.client.0.smithi144.stdout:0/113: dwrite d0/f7 [4194304,4194304] 0 2024-07-26T11:04:36.250 INFO:tasks.workunit.client.0.smithi144.stdout:5/72: dread d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:36.253 INFO:tasks.workunit.client.0.smithi144.stdout:1/104: write d3/d6/f18 [3101690,91692] 0 2024-07-26T11:04:36.253 INFO:tasks.workunit.client.0.smithi144.stdout:0/114: creat d0/d14/d1e/f24 x:0 0 0 2024-07-26T11:04:36.254 INFO:tasks.workunit.client.0.smithi144.stdout:0/115: mkdir d0/d25 0 2024-07-26T11:04:36.255 INFO:tasks.workunit.client.0.smithi144.stdout:0/116: rename d0/d14/d1e/l20 to d0/d14/d1e/l26 0 2024-07-26T11:04:36.256 INFO:tasks.workunit.client.0.smithi144.stdout:5/73: rename d1/d9/c1a to d1/d9/c1f 0 2024-07-26T11:04:36.256 INFO:tasks.workunit.client.0.smithi144.stdout:5/74: fdatasync d1/d9/f10 0 2024-07-26T11:04:36.259 INFO:tasks.workunit.client.0.smithi144.stdout:5/75: fsync d1/d14/f16 0 2024-07-26T11:04:36.271 INFO:tasks.workunit.client.0.smithi144.stdout:1/105: dread d3/f4 [0,4194304] 0 2024-07-26T11:04:36.273 INFO:tasks.workunit.client.0.smithi144.stdout:8/100: dread f15 [0,4194304] 0 2024-07-26T11:04:36.273 INFO:tasks.workunit.client.0.smithi144.stdout:8/101: getdents d1a/d1c/d1d 0 2024-07-26T11:04:36.273 INFO:tasks.workunit.client.0.smithi144.stdout:8/102: dread - f17 zero size 2024-07-26T11:04:36.273 INFO:tasks.workunit.client.0.smithi144.stdout:8/103: chown d1a/d1c/c1e 4 1 2024-07-26T11:04:36.276 INFO:tasks.workunit.client.0.smithi144.stdout:8/104: creat d1a/f21 x:0 0 0 2024-07-26T11:04:36.276 INFO:tasks.workunit.client.0.smithi144.stdout:8/105: fdatasync f15 0 2024-07-26T11:04:36.279 INFO:tasks.workunit.client.0.smithi144.stdout:8/106: rmdir d1a/d1c/d1d 0 2024-07-26T11:04:36.280 INFO:tasks.workunit.client.0.smithi144.stdout:7/106: dwrite d0/d6/f8 [0,4194304] 0 2024-07-26T11:04:36.281 INFO:tasks.workunit.client.0.smithi144.stdout:7/107: creat d0/d10/f30 x:0 0 0 2024-07-26T11:04:36.283 INFO:tasks.workunit.client.0.smithi144.stdout:7/108: rename d0/d10/c28 to d0/d6/c31 0 2024-07-26T11:04:36.283 INFO:tasks.workunit.client.0.smithi144.stdout:8/107: fsync d1a/f21 0 2024-07-26T11:04:36.285 INFO:tasks.workunit.client.0.smithi144.stdout:8/108: creat d1a/d1c/f22 x:0 0 0 2024-07-26T11:04:36.286 INFO:tasks.workunit.client.0.smithi144.stdout:8/109: mknod d1a/c23 0 2024-07-26T11:04:36.287 INFO:tasks.workunit.client.0.smithi144.stdout:8/110: creat d1a/d1c/f24 x:0 0 0 2024-07-26T11:04:36.288 INFO:tasks.workunit.client.0.smithi144.stdout:8/111: write f11 [863493,3578] 0 2024-07-26T11:04:36.304 INFO:tasks.workunit.client.0.smithi144.stdout:3/41: dwrite d1/d5/f6 [4194304,4194304] 0 2024-07-26T11:04:36.307 INFO:tasks.workunit.client.0.smithi144.stdout:3/42: creat d1/d5/fb x:0 0 0 2024-07-26T11:04:36.311 INFO:tasks.workunit.client.0.smithi144.stdout:3/43: link d1/l8 d1/lc 0 2024-07-26T11:04:36.311 INFO:tasks.workunit.client.0.smithi144.stdout:3/44: read d1/d5/fa [573899,110986] 0 2024-07-26T11:04:36.311 INFO:tasks.workunit.client.0.smithi144.stdout:3/45: chown d1 1219801 1 2024-07-26T11:04:36.311 INFO:tasks.workunit.client.0.smithi144.stdout:3/46: write d1/d5/f6 [2892049,1882] 0 2024-07-26T11:04:36.321 INFO:tasks.workunit.client.0.smithi144.stdout:5/76: dwrite d1/d9/f19 [0,4194304] 0 2024-07-26T11:04:36.321 INFO:tasks.workunit.client.0.smithi144.stdout:5/77: chown d1/d5/c7 2346 1 2024-07-26T11:04:36.321 INFO:tasks.workunit.client.0.smithi144.stdout:5/78: chown d1/d5/c7 778 1 2024-07-26T11:04:36.322 INFO:tasks.workunit.client.0.smithi144.stdout:4/62: dwrite f1 [0,4194304] 0 2024-07-26T11:04:36.322 INFO:tasks.workunit.client.0.smithi144.stdout:4/63: stat f1 0 2024-07-26T11:04:36.323 INFO:tasks.workunit.client.0.smithi144.stdout:0/117: dwrite d0/f10 [0,4194304] 0 2024-07-26T11:04:36.323 INFO:tasks.workunit.client.0.smithi144.stdout:0/118: chown d0/c15 120044057 1 2024-07-26T11:04:36.323 INFO:tasks.workunit.client.0.smithi144.stdout:0/119: dread - d0/d14/d1e/f21 zero size 2024-07-26T11:04:36.324 INFO:tasks.workunit.client.0.smithi144.stdout:5/79: symlink d1/d5/l20 0 2024-07-26T11:04:36.326 INFO:tasks.workunit.client.0.smithi144.stdout:0/120: getdents d0/d14 0 2024-07-26T11:04:36.327 INFO:tasks.workunit.client.0.smithi144.stdout:5/80: stat d1/d5/ca 0 2024-07-26T11:04:36.329 INFO:tasks.workunit.client.0.smithi144.stdout:1/106: dwrite d3/f4 [0,4194304] 0 2024-07-26T11:04:36.329 INFO:tasks.workunit.client.0.smithi144.stdout:5/81: rename d1/d9/f11 to d1/d1b/f21 0 2024-07-26T11:04:36.329 INFO:tasks.workunit.client.0.smithi144.stdout:5/82: truncate d1/d1b/f21 4307006 0 2024-07-26T11:04:36.331 INFO:tasks.workunit.client.0.smithi144.stdout:3/47: truncate d1/d5/f6 5747385 0 2024-07-26T11:04:36.331 INFO:tasks.workunit.client.0.smithi144.stdout:1/107: mkdir d3/d6/da/d13/d22 0 2024-07-26T11:04:36.333 INFO:tasks.workunit.client.0.smithi144.stdout:3/48: rename d1/l8 to d1/ld 0 2024-07-26T11:04:36.333 INFO:tasks.workunit.client.0.smithi144.stdout:3/49: write d1/d5/fa [1248131,96527] 0 2024-07-26T11:04:36.333 INFO:tasks.workunit.client.0.smithi144.stdout:1/108: creat d3/d6/f23 x:0 0 0 2024-07-26T11:04:36.337 INFO:tasks.workunit.client.0.smithi144.stdout:3/50: unlink d1/d5/fb 0 2024-07-26T11:04:36.342 INFO:tasks.workunit.client.0.smithi144.stdout:3/51: link d1/ld d1/le 0 2024-07-26T11:04:36.350 INFO:tasks.workunit.client.0.smithi144.stdout:5/83: read d1/d9/f19 [1423007,39688] 0 2024-07-26T11:04:36.350 INFO:tasks.workunit.client.0.smithi144.stdout:6/83: sync 2024-07-26T11:04:36.351 INFO:tasks.workunit.client.0.smithi144.stdout:6/84: truncate f2 2126195 0 2024-07-26T11:04:36.351 INFO:tasks.workunit.client.0.smithi144.stdout:6/85: dread - d8/f16 zero size 2024-07-26T11:04:36.356 INFO:tasks.workunit.client.0.smithi144.stdout:4/64: dread f5 [0,4194304] 0 2024-07-26T11:04:36.357 INFO:tasks.workunit.client.0.smithi144.stdout:7/109: dwrite d0/d6/d9/fc [0,4194304] 0 2024-07-26T11:04:36.357 INFO:tasks.workunit.client.0.smithi144.stdout:7/110: chown d0/d10/f1e 31 1 2024-07-26T11:04:36.363 INFO:tasks.workunit.client.0.smithi144.stdout:0/121: dread d0/fb [4194304,4194304] 0 2024-07-26T11:04:36.367 INFO:tasks.workunit.client.0.smithi144.stdout:7/111: symlink d0/d6/d9/d25/d2f/l32 0 2024-07-26T11:04:36.367 INFO:tasks.workunit.client.0.smithi144.stdout:4/65: creat d9/fe x:0 0 0 2024-07-26T11:04:36.370 INFO:tasks.workunit.client.0.smithi144.stdout:0/122: rename d0/d14/f1b to d0/f27 0 2024-07-26T11:04:36.370 INFO:tasks.workunit.client.0.smithi144.stdout:0/123: dread - d0/f27 zero size 2024-07-26T11:04:36.373 INFO:tasks.workunit.client.0.smithi144.stdout:7/112: link d0/f2d d0/d6/d9/d25/d2f/f33 0 2024-07-26T11:04:36.373 INFO:tasks.workunit.client.0.smithi144.stdout:4/66: symlink d9/lf 0 2024-07-26T11:04:36.373 INFO:tasks.workunit.client.0.smithi144.stdout:0/124: creat d0/f28 x:0 0 0 2024-07-26T11:04:36.373 INFO:tasks.workunit.client.0.smithi144.stdout:4/67: mknod d9/c10 0 2024-07-26T11:04:36.374 INFO:tasks.workunit.client.0.smithi144.stdout:0/125: mkdir d0/d14/d29 0 2024-07-26T11:04:36.376 INFO:tasks.workunit.client.0.smithi144.stdout:4/68: creat d9/f11 x:0 0 0 2024-07-26T11:04:36.376 INFO:tasks.workunit.client.0.smithi144.stdout:4/69: write d9/fe [707763,58962] 0 2024-07-26T11:04:36.377 INFO:tasks.workunit.client.0.smithi144.stdout:4/70: rename d9/da/fd to d9/f12 0 2024-07-26T11:04:36.379 INFO:tasks.workunit.client.0.smithi144.stdout:4/71: write d9/fe [483821,95060] 0 2024-07-26T11:04:36.403 INFO:tasks.workunit.client.0.smithi144.stdout:0/126: write d0/f18 [3370963,39859] 0 2024-07-26T11:04:36.407 INFO:tasks.workunit.client.0.smithi144.stdout:1/109: dwrite d3/d6/f1b [0,4194304] 0 2024-07-26T11:04:36.407 INFO:tasks.workunit.client.0.smithi144.stdout:0/127: link d0/d14/d1e/l23 d0/d14/l2a 0 2024-07-26T11:04:36.407 INFO:tasks.workunit.client.0.smithi144.stdout:5/84: dwrite d1/d18/f1d [0,4194304] 0 2024-07-26T11:04:36.420 INFO:tasks.workunit.client.0.smithi144.stdout:6/86: dwrite f6 [4194304,4194304] 0 2024-07-26T11:04:36.422 INFO:tasks.workunit.client.0.smithi144.stdout:6/87: creat d8/f1a x:0 0 0 2024-07-26T11:04:36.423 INFO:tasks.workunit.client.0.smithi144.stdout:6/88: link d8/fc d8/db/f1b 0 2024-07-26T11:04:36.435 INFO:tasks.workunit.client.0.smithi144.stdout:1/110: dread d3/fc [0,4194304] 0 2024-07-26T11:04:36.435 INFO:tasks.workunit.client.0.smithi144.stdout:1/111: creat d3/d6/da/d21/f24 x:0 0 0 2024-07-26T11:04:36.435 INFO:tasks.workunit.client.0.smithi144.stdout:1/112: symlink d3/l25 0 2024-07-26T11:04:36.437 INFO:tasks.workunit.client.0.smithi144.stdout:5/85: read d1/d14/f16 [5519937,77900] 0 2024-07-26T11:04:36.439 INFO:tasks.workunit.client.0.smithi144.stdout:5/86: symlink d1/d5/l22 0 2024-07-26T11:04:36.439 INFO:tasks.workunit.client.0.smithi144.stdout:5/87: chown d1/d5/c1c 487984 1 2024-07-26T11:04:36.439 INFO:tasks.workunit.client.0.smithi144.stdout:4/72: write f5 [120216,24982] 0 2024-07-26T11:04:36.440 INFO:tasks.workunit.client.0.smithi144.stdout:7/113: dwrite d0/d6/d9/fc [0,4194304] 0 2024-07-26T11:04:36.444 INFO:tasks.workunit.client.0.smithi144.stdout:0/128: dread d0/fb [0,4194304] 0 2024-07-26T11:04:36.444 INFO:tasks.workunit.client.0.smithi144.stdout:0/129: readlink d0/l16 0 2024-07-26T11:04:36.450 INFO:tasks.workunit.client.0.smithi144.stdout:5/88: symlink d1/d9/l23 0 2024-07-26T11:04:36.450 INFO:tasks.workunit.client.0.smithi144.stdout:5/89: fsync d1/d9/f19 0 2024-07-26T11:04:36.452 INFO:tasks.workunit.client.0.smithi144.stdout:4/73: symlink d9/l13 0 2024-07-26T11:04:36.452 INFO:tasks.workunit.client.0.smithi144.stdout:4/74: readlink l7 0 2024-07-26T11:04:36.452 INFO:tasks.workunit.client.0.smithi144.stdout:4/75: stat d9/c10 0 2024-07-26T11:04:36.452 INFO:tasks.workunit.client.0.smithi144.stdout:4/76: chown f6 132 1 2024-07-26T11:04:36.452 INFO:tasks.workunit.client.0.smithi144.stdout:7/114: mknod d0/d10/c34 0 2024-07-26T11:04:36.453 INFO:tasks.workunit.client.0.smithi144.stdout:0/130: rename d0/ce to d0/d14/c2b 0 2024-07-26T11:04:36.454 INFO:tasks.workunit.client.0.smithi144.stdout:7/115: creat d0/d6/d9/d25/d2f/f35 x:0 0 0 2024-07-26T11:04:36.455 INFO:tasks.workunit.client.0.smithi144.stdout:0/131: creat d0/d14/f2c x:0 0 0 2024-07-26T11:04:36.456 INFO:tasks.workunit.client.0.smithi144.stdout:7/116: symlink d0/d6/d9/l36 0 2024-07-26T11:04:36.457 INFO:tasks.workunit.client.0.smithi144.stdout:0/132: unlink d0/fb 0 2024-07-26T11:04:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:36 smithi144 ceph-mon[32129]: pgmap v524: 129 pgs: 129 active+clean; 110 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 699 op/s 2024-07-26T11:04:36.491 INFO:tasks.workunit.client.0.smithi144.stdout:1/113: dwrite d3/d6/f1b [4194304,4194304] 0 2024-07-26T11:04:36.491 INFO:tasks.workunit.client.0.smithi144.stdout:5/90: dread d1/d18/f1d [0,4194304] 0 2024-07-26T11:04:36.491 INFO:tasks.workunit.client.0.smithi144.stdout:6/89: dwrite f6 [0,4194304] 0 2024-07-26T11:04:36.492 INFO:tasks.workunit.client.0.smithi144.stdout:6/90: chown d8/fc 11751365 1 2024-07-26T11:04:36.493 INFO:tasks.workunit.client.0.smithi144.stdout:5/91: truncate d1/d18/f1d 4121465 0 2024-07-26T11:04:36.493 INFO:tasks.workunit.client.0.smithi144.stdout:5/92: fsync d1/d9/f10 0 2024-07-26T11:04:36.494 INFO:tasks.workunit.client.0.smithi144.stdout:1/114: creat d3/d6/da/d13/f26 x:0 0 0 2024-07-26T11:04:36.495 INFO:tasks.workunit.client.0.smithi144.stdout:5/93: mkdir d1/d18/d24 0 2024-07-26T11:04:36.495 INFO:tasks.workunit.client.0.smithi144.stdout:1/115: mkdir d3/d6/da/d13/d19/d27 0 2024-07-26T11:04:36.496 INFO:tasks.workunit.client.0.smithi144.stdout:5/94: stat d1/d9/cb 0 2024-07-26T11:04:36.496 INFO:tasks.workunit.client.0.smithi144.stdout:5/95: write d1/d14/f16 [5473335,104927] 0 2024-07-26T11:04:36.501 INFO:tasks.workunit.client.0.smithi144.stdout:5/96: link d1/d9/le d1/d14/l25 0 2024-07-26T11:04:36.501 INFO:tasks.workunit.client.0.smithi144.stdout:2/40: sync 2024-07-26T11:04:36.501 INFO:tasks.workunit.client.0.smithi144.stdout:1/116: read d3/d6/fd [1216623,52571] 0 2024-07-26T11:04:36.502 INFO:tasks.workunit.client.0.smithi144.stdout:1/117: readlink d3/d6/da/l17 0 2024-07-26T11:04:36.502 INFO:tasks.workunit.client.0.smithi144.stdout:1/118: fdatasync d3/d6/f1b 0 2024-07-26T11:04:36.503 INFO:tasks.workunit.client.0.smithi144.stdout:1/119: getdents d3/d6/da 0 2024-07-26T11:04:36.503 INFO:tasks.workunit.client.0.smithi144.stdout:1/120: fdatasync d3/fc 0 2024-07-26T11:04:36.503 INFO:tasks.workunit.client.0.smithi144.stdout:1/121: fdatasync f1 0 2024-07-26T11:04:36.503 INFO:tasks.workunit.client.0.smithi144.stdout:1/122: stat d3 0 2024-07-26T11:04:36.504 INFO:tasks.workunit.client.0.smithi144.stdout:1/123: creat d3/d6/da/d13/d19/d27/f28 x:0 0 0 2024-07-26T11:04:36.526 INFO:tasks.workunit.client.0.smithi144.stdout:2/41: write d0/d2/f4 [2544728,100780] 0 2024-07-26T11:04:36.526 INFO:tasks.workunit.client.0.smithi144.stdout:2/42: chown d0 2229617 1 2024-07-26T11:04:36.529 INFO:tasks.workunit.client.0.smithi144.stdout:0/133: dwrite d0/f18 [0,4194304] 0 2024-07-26T11:04:36.531 INFO:tasks.workunit.client.0.smithi144.stdout:0/134: creat d0/d14/d29/f2d x:0 0 0 2024-07-26T11:04:36.532 INFO:tasks.workunit.client.0.smithi144.stdout:0/135: chown d0/c15 32312542 1 2024-07-26T11:04:36.532 INFO:tasks.workunit.client.0.smithi144.stdout:0/136: stat d0/d14/l22 0 2024-07-26T11:04:36.532 INFO:tasks.workunit.client.0.smithi144.stdout:2/43: truncate d0/d2/f4 6154192 0 2024-07-26T11:04:36.532 INFO:tasks.workunit.client.0.smithi144.stdout:2/44: truncate d0/fa 773500 0 2024-07-26T11:04:36.534 INFO:tasks.workunit.client.0.smithi144.stdout:4/77: dwrite d9/fb [0,4194304] 0 2024-07-26T11:04:36.534 INFO:tasks.workunit.client.0.smithi144.stdout:4/78: write d9/da/fc [338692,9929] 0 2024-07-26T11:04:36.534 INFO:tasks.workunit.client.0.smithi144.stdout:5/97: dread d1/d1b/f21 [0,4194304] 0 2024-07-26T11:04:36.535 INFO:tasks.workunit.client.0.smithi144.stdout:4/79: mkdir d9/da/d14 0 2024-07-26T11:04:36.535 INFO:tasks.workunit.client.0.smithi144.stdout:5/98: rename d1/lc to d1/l26 0 2024-07-26T11:04:36.537 INFO:tasks.workunit.client.0.smithi144.stdout:4/80: creat d9/f15 x:0 0 0 2024-07-26T11:04:36.537 INFO:tasks.workunit.client.0.smithi144.stdout:4/81: fsync f5 0 2024-07-26T11:04:36.542 INFO:tasks.workunit.client.0.smithi144.stdout:4/82: mknod d9/da/c16 0 2024-07-26T11:04:36.543 INFO:tasks.workunit.client.0.smithi144.stdout:4/83: read d9/da/fc [325756,16409] 0 2024-07-26T11:04:36.545 INFO:tasks.workunit.client.0.smithi144.stdout:4/84: truncate d9/f15 24414 0 2024-07-26T11:04:36.554 INFO:tasks.workunit.client.0.smithi144.stdout:4/85: symlink d9/da/l17 0 2024-07-26T11:04:36.554 INFO:tasks.workunit.client.0.smithi144.stdout:1/124: dread d3/d6/f11 [0,4194304] 0 2024-07-26T11:04:36.555 INFO:tasks.workunit.client.0.smithi144.stdout:4/86: mknod d9/c18 0 2024-07-26T11:04:36.557 INFO:tasks.workunit.client.0.smithi144.stdout:6/91: dwrite d8/f16 [0,4194304] 0 2024-07-26T11:04:36.557 INFO:tasks.workunit.client.0.smithi144.stdout:4/87: rmdir d9/da/d14 0 2024-07-26T11:04:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:36 smithi160 ceph-mon[36715]: pgmap v524: 129 pgs: 129 active+clean; 110 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 699 op/s 2024-07-26T11:04:36.559 INFO:tasks.workunit.client.0.smithi144.stdout:5/99: write d1/d1b/f21 [2450358,53065] 0 2024-07-26T11:04:36.562 INFO:tasks.workunit.client.0.smithi144.stdout:6/92: symlink d8/l1c 0 2024-07-26T11:04:36.564 INFO:tasks.workunit.client.0.smithi144.stdout:5/100: mkdir d1/d18/d24/d27 0 2024-07-26T11:04:36.564 INFO:tasks.workunit.client.0.smithi144.stdout:5/101: write d1/d9/f10 [8519203,73014] 0 2024-07-26T11:04:36.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/93: creat d8/f1d x:0 0 0 2024-07-26T11:04:36.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/94: fdatasync f2 0 2024-07-26T11:04:36.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/95: fdatasync d8/f10 0 2024-07-26T11:04:36.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/96: rename d8/db to d8/db/d1e 22 2024-07-26T11:04:36.566 INFO:tasks.workunit.client.0.smithi144.stdout:5/102: symlink d1/d14/l28 0 2024-07-26T11:04:36.576 INFO:tasks.workunit.client.0.smithi144.stdout:6/97: mknod d8/d15/c1f 0 2024-07-26T11:04:36.576 INFO:tasks.workunit.client.0.smithi144.stdout:5/103: symlink d1/d14/l29 0 2024-07-26T11:04:36.579 INFO:tasks.workunit.client.0.smithi144.stdout:2/45: dwrite d0/f1 [0,4194304] 0 2024-07-26T11:04:36.593 INFO:tasks.workunit.client.0.smithi144.stdout:5/104: rmdir d1/d5 39 2024-07-26T11:04:36.593 INFO:tasks.workunit.client.0.smithi144.stdout:1/125: write d3/f4 [415378,78815] 0 2024-07-26T11:04:36.594 INFO:tasks.workunit.client.0.smithi144.stdout:1/126: write f1 [4201065,67647] 0 2024-07-26T11:04:36.594 INFO:tasks.workunit.client.0.smithi144.stdout:6/98: getdents d8 0 2024-07-26T11:04:36.594 INFO:tasks.workunit.client.0.smithi144.stdout:1/127: chown d3/l1f 2 1 2024-07-26T11:04:36.596 INFO:tasks.workunit.client.0.smithi144.stdout:2/46: rename d0/d2/l3 to d0/d2/lf 0 2024-07-26T11:04:36.612 INFO:tasks.workunit.client.0.smithi144.stdout:6/99: dread d8/f10 [0,4194304] 0 2024-07-26T11:04:36.620 INFO:tasks.workunit.client.0.smithi144.stdout:4/88: dwrite d9/da/fc [0,4194304] 0 2024-07-26T11:04:36.620 INFO:tasks.workunit.client.0.smithi144.stdout:4/89: rename d9/da to d9/da/d19 22 2024-07-26T11:04:36.625 INFO:tasks.workunit.client.0.smithi144.stdout:1/128: write d3/d6/f1b [8275560,100294] 0 2024-07-26T11:04:36.626 INFO:tasks.workunit.client.0.smithi144.stdout:1/129: creat d3/d6/da/d13/f29 x:0 0 0 2024-07-26T11:04:36.627 INFO:tasks.workunit.client.0.smithi144.stdout:9/92: sync 2024-07-26T11:04:36.628 INFO:tasks.workunit.client.0.smithi144.stdout:1/130: mknod d3/c2a 0 2024-07-26T11:04:36.628 INFO:tasks.workunit.client.0.smithi144.stdout:1/131: chown d3/d6/f18 245671 1 2024-07-26T11:04:36.628 INFO:tasks.workunit.client.0.smithi144.stdout:1/132: fsync d3/d6/f23 0 2024-07-26T11:04:36.628 INFO:tasks.workunit.client.0.smithi144.stdout:1/133: dread - d3/d6/da/d13/f29 zero size 2024-07-26T11:04:36.628 INFO:tasks.workunit.client.0.smithi144.stdout:1/134: fsync d3/f4 0 2024-07-26T11:04:36.629 INFO:tasks.workunit.client.0.smithi144.stdout:9/93: getdents d9/de 0 2024-07-26T11:04:36.630 INFO:tasks.workunit.client.0.smithi144.stdout:1/135: symlink d3/d6/da/d21/l2b 0 2024-07-26T11:04:36.630 INFO:tasks.workunit.client.0.smithi144.stdout:1/136: fdatasync d3/d6/f18 0 2024-07-26T11:04:36.630 INFO:tasks.workunit.client.0.smithi144.stdout:1/137: chown d3/l25 0 1 2024-07-26T11:04:36.633 INFO:tasks.workunit.client.0.smithi144.stdout:2/47: dread d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:36.633 INFO:tasks.workunit.client.0.smithi144.stdout:2/48: rename d0/d6/d9/cd to d0/d6/c10 0 2024-07-26T11:04:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:36 smithi183 ceph-mon[37273]: pgmap v524: 129 pgs: 129 active+clean; 110 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 699 op/s 2024-07-26T11:04:36.650 INFO:tasks.workunit.client.0.smithi144.stdout:5/105: dwrite d1/d18/f1d [0,4194304] 0 2024-07-26T11:04:36.650 INFO:tasks.workunit.client.0.smithi144.stdout:5/106: chown d1/d9/c1f 166698948 1 2024-07-26T11:04:36.676 INFO:tasks.workunit.client.0.smithi144.stdout:1/138: dwrite d3/d6/f11 [0,4194304] 0 2024-07-26T11:04:36.676 INFO:tasks.workunit.client.0.smithi144.stdout:1/139: fdatasync d3/f10 0 2024-07-26T11:04:36.679 INFO:tasks.workunit.client.0.smithi144.stdout:2/49: dwrite d0/fa [0,4194304] 0 2024-07-26T11:04:36.680 INFO:tasks.workunit.client.0.smithi144.stdout:6/100: dwrite d8/f16 [4194304,4194304] 0 2024-07-26T11:04:36.684 INFO:tasks.workunit.client.0.smithi144.stdout:2/50: creat d0/d2/f11 x:0 0 0 2024-07-26T11:04:36.690 INFO:tasks.workunit.client.0.smithi144.stdout:2/51: getdents d0/d2 0 2024-07-26T11:04:36.695 INFO:tasks.workunit.client.0.smithi144.stdout:5/107: dwrite d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:36.696 INFO:tasks.workunit.client.0.smithi144.stdout:5/108: getdents d1/d14 0 2024-07-26T11:04:36.700 INFO:tasks.workunit.client.0.smithi144.stdout:6/101: dread d8/f9 [0,4194304] 0 2024-07-26T11:04:36.700 INFO:tasks.workunit.client.0.smithi144.stdout:6/102: readlink d8/db/le 0 2024-07-26T11:04:36.700 INFO:tasks.workunit.client.0.smithi144.stdout:6/103: chown d8/l14 1006246 1 2024-07-26T11:04:36.701 INFO:tasks.workunit.client.0.smithi144.stdout:6/104: symlink d8/l20 0 2024-07-26T11:04:36.701 INFO:tasks.workunit.client.0.smithi144.stdout:6/105: fsync d8/f17 0 2024-07-26T11:04:36.715 INFO:tasks.workunit.client.0.smithi144.stdout:1/140: dwrite d3/d6/f18 [0,4194304] 0 2024-07-26T11:04:36.715 INFO:tasks.workunit.client.0.smithi144.stdout:1/141: stat d3/d6/da/l17 0 2024-07-26T11:04:36.715 INFO:tasks.workunit.client.0.smithi144.stdout:1/142: chown d3/l20 255452 1 2024-07-26T11:04:36.715 INFO:tasks.workunit.client.0.smithi144.stdout:1/143: fsync f1 0 2024-07-26T11:04:36.715 INFO:tasks.workunit.client.0.smithi144.stdout:1/144: rename d3/d6/da to d3/d6/da/d13/d19/d2c 22 2024-07-26T11:04:36.716 INFO:tasks.workunit.client.0.smithi144.stdout:1/145: read - d3/d6/da/d13/f26 zero size 2024-07-26T11:04:36.716 INFO:tasks.workunit.client.0.smithi144.stdout:1/146: readlink d3/d6/le 0 2024-07-26T11:04:36.716 INFO:tasks.workunit.client.0.smithi144.stdout:1/147: chown f1 97171 1 2024-07-26T11:04:36.717 INFO:tasks.workunit.client.0.smithi144.stdout:1/148: creat d3/f2d x:0 0 0 2024-07-26T11:04:36.719 INFO:tasks.workunit.client.0.smithi144.stdout:1/149: unlink d3/l25 0 2024-07-26T11:04:36.720 INFO:tasks.workunit.client.0.smithi144.stdout:1/150: symlink d3/d6/da/l2e 0 2024-07-26T11:04:36.721 INFO:tasks.workunit.client.0.smithi144.stdout:3/52: sync 2024-07-26T11:04:36.721 INFO:tasks.workunit.client.0.smithi144.stdout:8/112: sync 2024-07-26T11:04:36.721 INFO:tasks.workunit.client.0.smithi144.stdout:7/117: sync 2024-07-26T11:04:36.725 INFO:tasks.workunit.client.0.smithi144.stdout:8/113: symlink d1a/d1c/l25 0 2024-07-26T11:04:36.725 INFO:tasks.workunit.client.0.smithi144.stdout:8/114: chown l13 1214366 1 2024-07-26T11:04:36.725 INFO:tasks.workunit.client.0.smithi144.stdout:8/115: fsync f17 0 2024-07-26T11:04:36.727 INFO:tasks.workunit.client.0.smithi144.stdout:5/109: write d1/d14/f16 [3464981,57239] 0 2024-07-26T11:04:36.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/110: creat d1/d18/d24/f2a x:0 0 0 2024-07-26T11:04:36.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/111: write d1/d1b/f21 [5068972,82262] 0 2024-07-26T11:04:36.732 INFO:tasks.workunit.client.0.smithi144.stdout:5/112: chown d1/d9/l1e 7 1 2024-07-26T11:04:36.737 INFO:tasks.workunit.client.0.smithi144.stdout:5/113: truncate d1/d1b/f21 1281057 0 2024-07-26T11:04:36.741 INFO:tasks.workunit.client.0.smithi144.stdout:3/53: dread d1/d5/fa [0,4194304] 0 2024-07-26T11:04:36.741 INFO:tasks.workunit.client.0.smithi144.stdout:5/114: truncate d1/d1b/f21 1467712 0 2024-07-26T11:04:36.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/54: creat d1/ff x:0 0 0 2024-07-26T11:04:36.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/55: chown d1/le 41389 1 2024-07-26T11:04:36.743 INFO:tasks.workunit.client.0.smithi144.stdout:3/56: symlink d1/l10 0 2024-07-26T11:04:36.743 INFO:tasks.workunit.client.0.smithi144.stdout:3/57: chown d1/l10 10537 1 2024-07-26T11:04:36.744 INFO:tasks.workunit.client.0.smithi144.stdout:3/58: symlink d1/l11 0 2024-07-26T11:04:36.746 INFO:tasks.workunit.client.0.smithi144.stdout:3/59: link d1/le d1/l12 0 2024-07-26T11:04:36.746 INFO:tasks.workunit.client.0.smithi144.stdout:3/60: chown d1/l10 1694406 1 2024-07-26T11:04:36.751 INFO:tasks.workunit.client.0.smithi144.stdout:8/116: dread d1a/f20 [0,4194304] 0 2024-07-26T11:04:36.751 INFO:tasks.workunit.client.0.smithi144.stdout:6/106: dwrite f2 [0,4194304] 0 2024-07-26T11:04:36.753 INFO:tasks.workunit.client.0.smithi144.stdout:6/107: creat d8/d15/f21 x:0 0 0 2024-07-26T11:04:36.753 INFO:tasks.workunit.client.0.smithi144.stdout:6/108: readlink l7 0 2024-07-26T11:04:36.753 INFO:tasks.workunit.client.0.smithi144.stdout:6/109: chown d8/l1c 2 1 2024-07-26T11:04:36.754 INFO:tasks.workunit.client.0.smithi144.stdout:8/117: link cb d1a/d1c/c26 0 2024-07-26T11:04:36.756 INFO:tasks.workunit.client.0.smithi144.stdout:6/110: symlink d8/l22 0 2024-07-26T11:04:36.756 INFO:tasks.workunit.client.0.smithi144.stdout:7/118: dread d0/d6/d9/d12/f1d [0,4194304] 0 2024-07-26T11:04:36.757 INFO:tasks.workunit.client.0.smithi144.stdout:6/111: mknod d8/c23 0 2024-07-26T11:04:36.759 INFO:tasks.workunit.client.0.smithi144.stdout:6/112: symlink d8/d15/l24 0 2024-07-26T11:04:36.765 INFO:tasks.workunit.client.0.smithi144.stdout:7/119: dread d0/fd [0,4194304] 0 2024-07-26T11:04:36.766 INFO:tasks.workunit.client.0.smithi144.stdout:7/120: symlink d0/d10/l37 0 2024-07-26T11:04:36.788 INFO:tasks.workunit.client.0.smithi144.stdout:6/113: write d8/f10 [951939,110902] 0 2024-07-26T11:04:36.788 INFO:tasks.workunit.client.0.smithi144.stdout:6/114: chown d8/db/c11 7387 1 2024-07-26T11:04:36.796 INFO:tasks.workunit.client.0.smithi144.stdout:3/61: dwrite d1/d5/fa [4194304,4194304] 0 2024-07-26T11:04:36.798 INFO:tasks.workunit.client.0.smithi144.stdout:3/62: link d1/l9 d1/d5/l13 0 2024-07-26T11:04:36.811 INFO:tasks.workunit.client.0.smithi144.stdout:8/118: dwrite d1a/f21 [0,4194304] 0 2024-07-26T11:04:36.811 INFO:tasks.workunit.client.0.smithi144.stdout:5/115: dwrite d1/d1b/f21 [0,4194304] 0 2024-07-26T11:04:36.813 INFO:tasks.workunit.client.0.smithi144.stdout:8/119: rename f17 to d1a/d1c/f27 0 2024-07-26T11:04:36.813 INFO:tasks.workunit.client.0.smithi144.stdout:5/116: getdents d1/d18 0 2024-07-26T11:04:36.820 INFO:tasks.workunit.client.0.smithi144.stdout:7/121: dwrite d0/d10/f22 [0,4194304] 0 2024-07-26T11:04:36.820 INFO:tasks.workunit.client.0.smithi144.stdout:7/122: chown d0/d6/d9/d12 14529848 1 2024-07-26T11:04:36.820 INFO:tasks.workunit.client.0.smithi144.stdout:7/123: dread - d0/d10/f26 zero size 2024-07-26T11:04:36.823 INFO:tasks.workunit.client.0.smithi144.stdout:7/124: symlink d0/d6/d9/d25/d2f/l38 0 2024-07-26T11:04:36.823 INFO:tasks.workunit.client.0.smithi144.stdout:7/125: write d0/d6/d9/f27 [654449,122720] 0 2024-07-26T11:04:36.826 INFO:tasks.workunit.client.0.smithi144.stdout:0/137: sync 2024-07-26T11:04:36.826 INFO:tasks.workunit.client.0.smithi144.stdout:0/138: fdatasync d0/d14/d1e/f24 0 2024-07-26T11:04:36.826 INFO:tasks.workunit.client.0.smithi144.stdout:0/139: fdatasync d0/f6 0 2024-07-26T11:04:36.826 INFO:tasks.workunit.client.0.smithi144.stdout:0/140: chown d0/d14/l2a 198399 1 2024-07-26T11:04:36.842 INFO:tasks.workunit.client.0.smithi144.stdout:7/126: write d0/f7 [1596323,50834] 0 2024-07-26T11:04:36.842 INFO:tasks.workunit.client.0.smithi144.stdout:7/127: dread - d0/d6/d9/d25/d2f/f35 zero size 2024-07-26T11:04:36.843 INFO:tasks.workunit.client.0.smithi144.stdout:3/63: dwrite d1/ff [0,4194304] 0 2024-07-26T11:04:36.845 INFO:tasks.workunit.client.0.smithi144.stdout:3/64: creat d1/f14 x:0 0 0 2024-07-26T11:04:36.851 INFO:tasks.workunit.client.0.smithi144.stdout:3/65: getdents d1 0 2024-07-26T11:04:36.851 INFO:tasks.workunit.client.0.smithi144.stdout:3/66: chown d1/ld 28388 1 2024-07-26T11:04:36.851 INFO:tasks.workunit.client.0.smithi144.stdout:0/141: write d0/f18 [1356377,69548] 0 2024-07-26T11:04:36.851 INFO:tasks.workunit.client.0.smithi144.stdout:0/142: dread - d0/f28 zero size 2024-07-26T11:04:36.851 INFO:tasks.workunit.client.0.smithi144.stdout:0/143: truncate d0/f27 1046101 0 2024-07-26T11:04:36.862 INFO:tasks.workunit.client.0.smithi144.stdout:0/144: rename d0/d14/l1c to d0/l2e 0 2024-07-26T11:04:36.862 INFO:tasks.workunit.client.0.smithi144.stdout:0/145: fsync d0/f28 0 2024-07-26T11:04:36.864 INFO:tasks.workunit.client.0.smithi144.stdout:0/146: truncate d0/f8 187149 0 2024-07-26T11:04:36.865 INFO:tasks.workunit.client.0.smithi144.stdout:0/147: mknod d0/d25/c2f 0 2024-07-26T11:04:36.866 INFO:tasks.workunit.client.0.smithi144.stdout:8/120: dwrite fe [0,4194304] 0 2024-07-26T11:04:36.868 INFO:tasks.workunit.client.0.smithi144.stdout:0/148: mkdir d0/d14/d1e/d30 0 2024-07-26T11:04:36.868 INFO:tasks.workunit.client.0.smithi144.stdout:0/149: fdatasync d0/d14/d1e/f21 0 2024-07-26T11:04:36.869 INFO:tasks.workunit.client.0.smithi144.stdout:5/117: dwrite d1/d18/d24/f2a [0,4194304] 0 2024-07-26T11:04:36.871 INFO:tasks.workunit.client.0.smithi144.stdout:8/121: mkdir d1a/d1c/d28 0 2024-07-26T11:04:36.872 INFO:tasks.workunit.client.0.smithi144.stdout:0/150: symlink d0/d14/d29/l31 0 2024-07-26T11:04:36.874 INFO:tasks.workunit.client.0.smithi144.stdout:5/118: mknod d1/d18/c2b 0 2024-07-26T11:04:36.875 INFO:tasks.workunit.client.0.smithi144.stdout:0/151: creat d0/f32 x:0 0 0 2024-07-26T11:04:36.875 INFO:tasks.workunit.client.0.smithi144.stdout:0/152: dread - d0/d14/d29/f2d zero size 2024-07-26T11:04:36.880 INFO:tasks.workunit.client.0.smithi144.stdout:3/67: dread d1/ff [0,4194304] 0 2024-07-26T11:04:36.880 INFO:tasks.workunit.client.0.smithi144.stdout:0/153: getdents d0/d25 0 2024-07-26T11:04:36.882 INFO:tasks.workunit.client.0.smithi144.stdout:3/68: truncate d1/d5/fa 487149 0 2024-07-26T11:04:36.887 INFO:tasks.workunit.client.0.smithi144.stdout:3/69: creat d1/d5/f15 x:0 0 0 2024-07-26T11:04:36.889 INFO:tasks.workunit.client.0.smithi144.stdout:8/122: dread d1a/f20 [0,4194304] 0 2024-07-26T11:04:36.890 INFO:tasks.workunit.client.0.smithi144.stdout:8/123: mknod d1a/d1c/d28/c29 0 2024-07-26T11:04:36.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/124: mkdir d1a/d1c/d2a 0 2024-07-26T11:04:36.892 INFO:tasks.workunit.client.0.smithi144.stdout:8/125: creat d1a/f2b x:0 0 0 2024-07-26T11:04:36.899 INFO:tasks.workunit.client.0.smithi144.stdout:7/128: dwrite d0/d10/f2a [0,4194304] 0 2024-07-26T11:04:36.900 INFO:tasks.workunit.client.0.smithi144.stdout:5/119: dread d1/d1b/f21 [0,4194304] 0 2024-07-26T11:04:36.902 INFO:tasks.workunit.client.0.smithi144.stdout:7/129: creat d0/d6/d9/d25/d2f/f39 x:0 0 0 2024-07-26T11:04:36.903 INFO:tasks.workunit.client.0.smithi144.stdout:5/120: truncate d1/d9/f19 2318797 0 2024-07-26T11:04:36.903 INFO:tasks.workunit.client.0.smithi144.stdout:7/130: rmdir d0/d6/d9/d12 39 2024-07-26T11:04:36.907 INFO:tasks.workunit.client.0.smithi144.stdout:5/121: mknod d1/d14/c2c 0 2024-07-26T11:04:36.907 INFO:tasks.workunit.client.0.smithi144.stdout:5/122: chown d1/d18/d24/f2a 7990459 1 2024-07-26T11:04:36.909 INFO:tasks.workunit.client.0.smithi144.stdout:8/126: dread f11 [0,4194304] 0 2024-07-26T11:04:36.911 INFO:tasks.workunit.client.0.smithi144.stdout:8/127: mkdir d1a/d1c/d28/d2c 0 2024-07-26T11:04:36.911 INFO:tasks.workunit.client.0.smithi144.stdout:1/151: sync 2024-07-26T11:04:36.915 INFO:tasks.workunit.client.0.smithi144.stdout:6/115: sync 2024-07-26T11:04:36.915 INFO:tasks.workunit.client.0.smithi144.stdout:2/52: sync 2024-07-26T11:04:36.915 INFO:tasks.workunit.client.0.smithi144.stdout:9/94: sync 2024-07-26T11:04:36.915 INFO:tasks.workunit.client.0.smithi144.stdout:4/90: sync 2024-07-26T11:04:36.915 INFO:tasks.workunit.client.0.smithi144.stdout:1/152: creat d3/d6/da/f2f x:0 0 0 2024-07-26T11:04:36.916 INFO:tasks.workunit.client.0.smithi144.stdout:8/128: getdents d1a/d1c 0 2024-07-26T11:04:36.916 INFO:tasks.workunit.client.0.smithi144.stdout:6/116: creat d8/db/f25 x:0 0 0 2024-07-26T11:04:36.916 INFO:tasks.workunit.client.0.smithi144.stdout:6/117: fdatasync d8/fd 0 2024-07-26T11:04:36.917 INFO:tasks.workunit.client.0.smithi144.stdout:9/95: mkdir d9/de/d14/d23 0 2024-07-26T11:04:36.917 INFO:tasks.workunit.client.0.smithi144.stdout:9/96: readlink d9/de/l1f 0 2024-07-26T11:04:36.918 INFO:tasks.workunit.client.0.smithi144.stdout:4/91: rmdir d9 39 2024-07-26T11:04:36.919 INFO:tasks.workunit.client.0.smithi144.stdout:6/118: truncate d8/f10 3396421 0 2024-07-26T11:04:36.919 INFO:tasks.workunit.client.0.smithi144.stdout:4/92: creat d9/da/f1a x:0 0 0 2024-07-26T11:04:36.922 INFO:tasks.workunit.client.0.smithi144.stdout:9/97: read d9/de/f21 [205888,103849] 0 2024-07-26T11:04:36.931 INFO:tasks.workunit.client.0.smithi144.stdout:6/119: fsync d8/f10 0 2024-07-26T11:04:36.944 INFO:tasks.workunit.client.0.smithi144.stdout:3/70: dwrite d1/f14 [0,4194304] 0 2024-07-26T11:04:36.944 INFO:tasks.workunit.client.0.smithi144.stdout:3/71: chown d1/d5/f15 140208959 1 2024-07-26T11:04:36.944 INFO:tasks.workunit.client.0.smithi144.stdout:3/72: getdents d1 0 2024-07-26T11:04:36.944 INFO:tasks.workunit.client.0.smithi144.stdout:6/120: mkdir d8/d15/d26 0 2024-07-26T11:04:36.947 INFO:tasks.workunit.client.0.smithi144.stdout:3/73: symlink d1/l16 0 2024-07-26T11:04:36.947 INFO:tasks.workunit.client.0.smithi144.stdout:6/121: creat d8/db/f27 x:0 0 0 2024-07-26T11:04:36.948 INFO:tasks.workunit.client.0.smithi144.stdout:6/122: write d8/f1d [526285,43235] 0 2024-07-26T11:04:36.950 INFO:tasks.workunit.client.0.smithi144.stdout:6/123: link d8/db/f25 d8/d15/d26/f28 0 2024-07-26T11:04:36.950 INFO:tasks.workunit.client.0.smithi144.stdout:6/124: stat f2 0 2024-07-26T11:04:36.950 INFO:tasks.workunit.client.0.smithi144.stdout:6/125: fsync d8/f1a 0 2024-07-26T11:04:36.955 INFO:tasks.workunit.client.0.smithi144.stdout:6/126: creat d8/d15/d26/f29 x:0 0 0 2024-07-26T11:04:36.955 INFO:tasks.workunit.client.0.smithi144.stdout:6/127: read f2 [1226214,65249] 0 2024-07-26T11:04:36.957 INFO:tasks.workunit.client.0.smithi144.stdout:6/128: link d8/f1d d8/db/f2a 0 2024-07-26T11:04:36.957 INFO:tasks.workunit.client.0.smithi144.stdout:6/129: truncate d8/d15/d26/f28 960595 0 2024-07-26T11:04:36.957 INFO:tasks.workunit.client.0.smithi144.stdout:6/130: truncate d8/d15/f21 53946 0 2024-07-26T11:04:36.959 INFO:tasks.workunit.client.0.smithi144.stdout:6/131: mknod d8/d15/c2b 0 2024-07-26T11:04:36.961 INFO:tasks.workunit.client.0.smithi144.stdout:6/132: creat d8/db/f2c x:0 0 0 2024-07-26T11:04:36.964 INFO:tasks.workunit.client.0.smithi144.stdout:2/53: write d0/fa [2036094,60531] 0 2024-07-26T11:04:36.964 INFO:tasks.workunit.client.0.smithi144.stdout:2/54: chown d0/d6/c10 78165 1 2024-07-26T11:04:36.964 INFO:tasks.workunit.client.0.smithi144.stdout:7/131: dwrite d0/f7 [4194304,4194304] 0 2024-07-26T11:04:36.968 INFO:tasks.workunit.client.0.smithi144.stdout:2/55: unlink d0/d2/f11 0 2024-07-26T11:04:36.969 INFO:tasks.workunit.client.0.smithi144.stdout:2/56: mknod d0/d2/c12 0 2024-07-26T11:04:36.971 INFO:tasks.workunit.client.0.smithi144.stdout:2/57: symlink d0/l13 0 2024-07-26T11:04:36.973 INFO:tasks.workunit.client.0.smithi144.stdout:1/153: dread d3/d6/f1b [0,4194304] 0 2024-07-26T11:04:36.973 INFO:tasks.workunit.client.0.smithi144.stdout:0/154: dwrite d0/f27 [0,4194304] 0 2024-07-26T11:04:36.974 INFO:tasks.workunit.client.0.smithi144.stdout:6/133: dread d8/db/f25 [0,4194304] 0 2024-07-26T11:04:36.978 INFO:tasks.workunit.client.0.smithi144.stdout:0/155: getdents d0/d14/d29 0 2024-07-26T11:04:36.978 INFO:tasks.workunit.client.0.smithi144.stdout:6/134: creat d8/f2d x:0 0 0 2024-07-26T11:04:36.979 INFO:tasks.workunit.client.0.smithi144.stdout:9/98: read d9/f1d [1778999,35531] 0 2024-07-26T11:04:36.981 INFO:tasks.workunit.client.0.smithi144.stdout:9/99: creat d9/de/d14/f24 x:0 0 0 2024-07-26T11:04:36.981 INFO:tasks.workunit.client.0.smithi144.stdout:9/100: truncate d9/f1d 4392544 0 2024-07-26T11:04:36.986 INFO:tasks.workunit.client.0.smithi144.stdout:3/74: write d1/d5/f6 [5600067,25450] 0 2024-07-26T11:04:36.986 INFO:tasks.workunit.client.0.smithi144.stdout:3/75: chown d1/lc 97767 1 2024-07-26T11:04:36.987 INFO:tasks.workunit.client.0.smithi144.stdout:3/76: chown d1/l11 2499544 1 2024-07-26T11:04:36.987 INFO:tasks.workunit.client.0.smithi144.stdout:3/77: creat d1/f17 x:0 0 0 2024-07-26T11:04:37.003 INFO:tasks.workunit.client.0.smithi144.stdout:4/93: dwrite d9/fe [0,4194304] 0 2024-07-26T11:04:37.014 INFO:tasks.workunit.client.0.smithi144.stdout:5/123: dwrite d1/d1b/f21 [0,4194304] 0 2024-07-26T11:04:37.018 INFO:tasks.workunit.client.0.smithi144.stdout:1/154: dread f1 [0,4194304] 0 2024-07-26T11:04:37.018 INFO:tasks.workunit.client.0.smithi144.stdout:5/124: creat d1/d5/f2d x:0 0 0 2024-07-26T11:04:37.019 INFO:tasks.workunit.client.0.smithi144.stdout:6/135: dwrite d8/db/f25 [0,4194304] 0 2024-07-26T11:04:37.022 INFO:tasks.workunit.client.0.smithi144.stdout:6/136: write d8/fd [2972841,45426] 0 2024-07-26T11:04:37.022 INFO:tasks.workunit.client.0.smithi144.stdout:6/137: readlink d8/db/le 0 2024-07-26T11:04:37.048 INFO:tasks.workunit.client.0.smithi144.stdout:6/138: read d8/f16 [4955350,13933] 0 2024-07-26T11:04:37.048 INFO:tasks.workunit.client.0.smithi144.stdout:6/139: fsync d8/f17 0 2024-07-26T11:04:37.051 INFO:tasks.workunit.client.0.smithi144.stdout:6/140: creat d8/d15/d26/f2e x:0 0 0 2024-07-26T11:04:37.051 INFO:tasks.workunit.client.0.smithi144.stdout:6/141: chown d8/c23 11724 1 2024-07-26T11:04:37.052 INFO:tasks.workunit.client.0.smithi144.stdout:4/94: dread f1 [0,4194304] 0 2024-07-26T11:04:37.053 INFO:tasks.workunit.client.0.smithi144.stdout:4/95: write f3 [1778035,125485] 0 2024-07-26T11:04:37.053 INFO:tasks.workunit.client.0.smithi144.stdout:4/96: dread - d9/f12 zero size 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:3/78: dwrite d1/d5/fa [0,4194304] 0 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:6/142: rmdir d8/db 39 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:3/79: chown l0 523961 1 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:6/143: chown d8/d15/c2b 1 1 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:6/144: readlink d8/db/le 0 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:6/145: dread - d8/f17 zero size 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:3/80: truncate d1/d5/fa 4209113 0 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:3/81: readlink d1/l10 0 2024-07-26T11:04:37.056 INFO:tasks.workunit.client.0.smithi144.stdout:3/82: fdatasync d1/d5/fa 0 2024-07-26T11:04:37.057 INFO:tasks.workunit.client.0.smithi144.stdout:6/146: symlink d8/d15/l2f 0 2024-07-26T11:04:37.057 INFO:tasks.workunit.client.0.smithi144.stdout:5/125: dread d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:37.057 INFO:tasks.workunit.client.0.smithi144.stdout:5/126: readlink d1/d14/l29 0 2024-07-26T11:04:37.057 INFO:tasks.workunit.client.0.smithi144.stdout:0/156: dwrite d0/d14/d1e/f21 [0,4194304] 0 2024-07-26T11:04:37.064 INFO:tasks.workunit.client.0.smithi144.stdout:6/147: creat d8/d15/d26/f30 x:0 0 0 2024-07-26T11:04:37.067 INFO:tasks.workunit.client.0.smithi144.stdout:1/155: dwrite d3/f1e [0,4194304] 0 2024-07-26T11:04:37.082 INFO:tasks.workunit.client.0.smithi144.stdout:1/156: creat d3/d6/da/d13/d22/f30 x:0 0 0 2024-07-26T11:04:37.083 INFO:tasks.workunit.client.0.smithi144.stdout:1/157: symlink d3/d6/da/l31 0 2024-07-26T11:04:37.085 INFO:tasks.workunit.client.0.smithi144.stdout:5/127: dread d1/d14/f16 [0,4194304] 0 2024-07-26T11:04:37.085 INFO:tasks.workunit.client.0.smithi144.stdout:5/128: fdatasync d1/d1b/f21 0 2024-07-26T11:04:37.086 INFO:tasks.workunit.client.0.smithi144.stdout:1/158: unlink d3/d6/da/d13/c16 0 2024-07-26T11:04:37.089 INFO:tasks.workunit.client.0.smithi144.stdout:3/83: dread d1/f14 [0,4194304] 0 2024-07-26T11:04:37.089 INFO:tasks.workunit.client.0.smithi144.stdout:3/84: read - d1/d5/f15 zero size 2024-07-26T11:04:37.089 INFO:tasks.workunit.client.0.smithi144.stdout:9/101: dwrite d9/de/f20 [0,4194304] 0 2024-07-26T11:04:37.089 INFO:tasks.workunit.client.0.smithi144.stdout:9/102: fdatasync d9/de/d14/f24 0 2024-07-26T11:04:37.089 INFO:tasks.workunit.client.0.smithi144.stdout:5/129: creat d1/d5/f2e x:0 0 0 2024-07-26T11:04:37.091 INFO:tasks.workunit.client.0.smithi144.stdout:5/130: creat d1/d18/d24/d27/f2f x:0 0 0 2024-07-26T11:04:37.096 INFO:tasks.workunit.client.0.smithi144.stdout:5/131: fsync d1/d5/f2e 0 2024-07-26T11:04:37.102 INFO:tasks.workunit.client.0.smithi144.stdout:2/58: dwrite d0/fa [0,4194304] 0 2024-07-26T11:04:37.104 INFO:tasks.workunit.client.0.smithi144.stdout:5/132: rename d1/d14/l29 to d1/d9/l30 0 2024-07-26T11:04:37.106 INFO:tasks.workunit.client.0.smithi144.stdout:4/97: write d9/fe [2467802,92137] 0 2024-07-26T11:04:37.108 INFO:tasks.workunit.client.0.smithi144.stdout:5/133: getdents d1/d18/d24/d27 0 2024-07-26T11:04:37.108 INFO:tasks.workunit.client.0.smithi144.stdout:4/98: mknod d9/c1b 0 2024-07-26T11:04:37.108 INFO:tasks.workunit.client.0.smithi144.stdout:4/99: rename d9/da to d9/da/d1c 22 2024-07-26T11:04:37.112 INFO:tasks.workunit.client.0.smithi144.stdout:5/134: symlink d1/l31 0 2024-07-26T11:04:37.117 INFO:tasks.workunit.client.0.smithi144.stdout:9/103: dread f4 [0,4194304] 0 2024-07-26T11:04:37.117 INFO:tasks.workunit.client.0.smithi144.stdout:5/135: mknod d1/c32 0 2024-07-26T11:04:37.118 INFO:tasks.workunit.client.0.smithi144.stdout:5/136: truncate d1/d9/f19 957034 0 2024-07-26T11:04:37.119 INFO:tasks.workunit.client.0.smithi144.stdout:5/137: read - d1/d5/f2d zero size 2024-07-26T11:04:37.120 INFO:tasks.workunit.client.0.smithi144.stdout:6/148: dread d8/d15/d26/f28 [0,4194304] 0 2024-07-26T11:04:37.121 INFO:tasks.workunit.client.0.smithi144.stdout:5/138: symlink d1/d9/l33 0 2024-07-26T11:04:37.122 INFO:tasks.workunit.client.0.smithi144.stdout:5/139: dread - d1/d5/f2d zero size 2024-07-26T11:04:37.122 INFO:tasks.workunit.client.0.smithi144.stdout:6/149: symlink d8/db/l31 0 2024-07-26T11:04:37.123 INFO:tasks.workunit.client.0.smithi144.stdout:0/157: dwrite d0/d14/d1e/f21 [0,4194304] 0 2024-07-26T11:04:37.125 INFO:tasks.workunit.client.0.smithi144.stdout:6/150: creat d8/db/f32 x:0 0 0 2024-07-26T11:04:37.125 INFO:tasks.workunit.client.0.smithi144.stdout:6/151: write d8/f1d [1176481,110635] 0 2024-07-26T11:04:37.125 INFO:tasks.workunit.client.0.smithi144.stdout:6/152: fsync d8/db/f25 0 2024-07-26T11:04:37.126 INFO:tasks.workunit.client.0.smithi144.stdout:0/158: mknod d0/d14/d1e/c33 0 2024-07-26T11:04:37.127 INFO:tasks.workunit.client.0.smithi144.stdout:6/153: getdents d8/d15/d26 0 2024-07-26T11:04:37.128 INFO:tasks.workunit.client.0.smithi144.stdout:0/159: mknod d0/c34 0 2024-07-26T11:04:37.129 INFO:tasks.workunit.client.0.smithi144.stdout:6/154: rename d8/fc to d8/f33 0 2024-07-26T11:04:37.134 INFO:tasks.workunit.client.0.smithi144.stdout:0/160: symlink d0/d14/l35 0 2024-07-26T11:04:37.137 INFO:tasks.workunit.client.0.smithi144.stdout:8/129: sync 2024-07-26T11:04:37.142 INFO:tasks.workunit.client.0.smithi144.stdout:8/130: getdents d1a 0 2024-07-26T11:04:37.142 INFO:tasks.workunit.client.0.smithi144.stdout:8/131: write f8 [815664,16398] 0 2024-07-26T11:04:37.142 INFO:tasks.workunit.client.0.smithi144.stdout:1/159: dwrite d3/d6/fd [0,4194304] 0 2024-07-26T11:04:37.146 INFO:tasks.workunit.client.0.smithi144.stdout:1/160: rename d3/d6/f1b to d3/d6/f32 0 2024-07-26T11:04:37.146 INFO:tasks.workunit.client.0.smithi144.stdout:8/132: creat d1a/d1c/f2d x:0 0 0 2024-07-26T11:04:37.149 INFO:tasks.workunit.client.0.smithi144.stdout:1/161: symlink d3/d6/da/d13/d19/l33 0 2024-07-26T11:04:37.150 INFO:tasks.workunit.client.0.smithi144.stdout:1/162: unlink d3/f4 0 2024-07-26T11:04:37.151 INFO:tasks.workunit.client.0.smithi144.stdout:8/133: creat d1a/d1c/d28/f2e x:0 0 0 2024-07-26T11:04:37.151 INFO:tasks.workunit.client.0.smithi144.stdout:8/134: readlink d1a/d1c/l25 0 2024-07-26T11:04:37.151 INFO:tasks.workunit.client.0.smithi144.stdout:8/135: fdatasync d1a/f21 0 2024-07-26T11:04:37.153 INFO:tasks.workunit.client.0.smithi144.stdout:1/163: unlink d3/d6/da/l17 0 2024-07-26T11:04:37.153 INFO:tasks.workunit.client.0.smithi144.stdout:8/136: mkdir d1a/d1c/d2f 0 2024-07-26T11:04:37.156 INFO:tasks.workunit.client.0.smithi144.stdout:1/164: symlink d3/d6/da/d13/d19/l34 0 2024-07-26T11:04:37.156 INFO:tasks.workunit.client.0.smithi144.stdout:1/165: rename d3 to d3/d6/d35 22 2024-07-26T11:04:37.156 INFO:tasks.workunit.client.0.smithi144.stdout:1/166: fsync d3/d6/da/d13/f26 0 2024-07-26T11:04:37.156 INFO:tasks.workunit.client.0.smithi144.stdout:8/137: rmdir d1a/d1c 39 2024-07-26T11:04:37.159 INFO:tasks.workunit.client.0.smithi144.stdout:8/138: read f4 [1259388,102055] 0 2024-07-26T11:04:37.159 INFO:tasks.workunit.client.0.smithi144.stdout:1/167: mkdir d3/d6/da/d13/d19/d27/d36 0 2024-07-26T11:04:37.160 INFO:tasks.workunit.client.0.smithi144.stdout:1/168: dread - d3/d6/da/d21/f24 zero size 2024-07-26T11:04:37.160 INFO:tasks.workunit.client.0.smithi144.stdout:1/169: fdatasync d3/d6/da/d21/f24 0 2024-07-26T11:04:37.165 INFO:tasks.workunit.client.0.smithi144.stdout:1/170: write d3/d6/da/d21/f24 [401775,45944] 0 2024-07-26T11:04:37.165 INFO:tasks.workunit.client.0.smithi144.stdout:1/171: dread - d3/d6/da/d13/f26 zero size 2024-07-26T11:04:37.166 INFO:tasks.workunit.client.0.smithi144.stdout:2/59: dwrite d0/f1 [0,4194304] 0 2024-07-26T11:04:37.166 INFO:tasks.workunit.client.0.smithi144.stdout:2/60: chown d0/l13 159606932 1 2024-07-26T11:04:37.166 INFO:tasks.workunit.client.0.smithi144.stdout:5/140: dread d1/d9/f19 [0,4194304] 0 2024-07-26T11:04:37.166 INFO:tasks.workunit.client.0.smithi144.stdout:9/104: dwrite f4 [4194304,4194304] 0 2024-07-26T11:04:37.168 INFO:tasks.workunit.client.0.smithi144.stdout:2/61: rename d0/fa to d0/f14 0 2024-07-26T11:04:37.168 INFO:tasks.workunit.client.0.smithi144.stdout:8/139: mkdir d1a/d1c/d30 0 2024-07-26T11:04:37.169 INFO:tasks.workunit.client.0.smithi144.stdout:3/85: dwrite d1/d5/f6 [4194304,4194304] 0 2024-07-26T11:04:37.170 INFO:tasks.workunit.client.0.smithi144.stdout:9/105: rename d9/de/l1f to d9/l25 0 2024-07-26T11:04:37.170 INFO:tasks.workunit.client.0.smithi144.stdout:9/106: write d9/de/d14/f24 [749991,2617] 0 2024-07-26T11:04:37.178 INFO:tasks.workunit.client.0.smithi144.stdout:3/86: mknod d1/c18 0 2024-07-26T11:04:37.187 INFO:tasks.workunit.client.0.smithi144.stdout:9/107: rename d9/de/c13 to d9/c26 0 2024-07-26T11:04:37.197 INFO:tasks.workunit.client.0.smithi144.stdout:4/100: dwrite d9/da/fc [0,4194304] 0 2024-07-26T11:04:37.200 INFO:tasks.workunit.client.0.smithi144.stdout:0/161: dwrite d0/f10 [0,4194304] 0 2024-07-26T11:04:37.200 INFO:tasks.workunit.client.0.smithi144.stdout:2/62: fsync d0/f14 0 2024-07-26T11:04:37.201 INFO:tasks.workunit.client.0.smithi144.stdout:2/63: mkdir d0/d15 0 2024-07-26T11:04:37.202 INFO:tasks.workunit.client.0.smithi144.stdout:2/64: fsync d0/d2/f4 0 2024-07-26T11:04:37.202 INFO:tasks.workunit.client.0.smithi144.stdout:0/162: creat d0/d14/d1e/f36 x:0 0 0 2024-07-26T11:04:37.203 INFO:tasks.workunit.client.0.smithi144.stdout:2/65: symlink d0/d15/l16 0 2024-07-26T11:04:37.203 INFO:tasks.workunit.client.0.smithi144.stdout:2/66: chown d0/d6/ce 4416295 1 2024-07-26T11:04:37.203 INFO:tasks.workunit.client.0.smithi144.stdout:0/163: mknod d0/d14/d1e/d30/c37 0 2024-07-26T11:04:37.205 INFO:tasks.workunit.client.0.smithi144.stdout:2/67: creat d0/d15/f17 x:0 0 0 2024-07-26T11:04:37.206 INFO:tasks.workunit.client.0.smithi144.stdout:0/164: rename d0/c1a to d0/d25/c38 0 2024-07-26T11:04:37.207 INFO:tasks.workunit.client.0.smithi144.stdout:7/132: sync 2024-07-26T11:04:37.207 INFO:tasks.workunit.client.0.smithi144.stdout:7/133: dread - d0/d6/d9/d25/d2f/f33 zero size 2024-07-26T11:04:37.209 INFO:tasks.workunit.client.0.smithi144.stdout:2/68: mknod d0/d2/c18 0 2024-07-26T11:04:37.209 INFO:tasks.workunit.client.0.smithi144.stdout:0/165: creat d0/d14/d1e/d30/f39 x:0 0 0 2024-07-26T11:04:37.210 INFO:tasks.workunit.client.0.smithi144.stdout:0/166: mknod d0/c3a 0 2024-07-26T11:04:37.211 INFO:tasks.workunit.client.0.smithi144.stdout:6/155: dwrite d8/f33 [0,4194304] 0 2024-07-26T11:04:37.216 INFO:tasks.workunit.client.0.smithi144.stdout:6/156: creat d8/db/f34 x:0 0 0 2024-07-26T11:04:37.220 INFO:tasks.workunit.client.0.smithi144.stdout:5/141: dwrite d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:37.221 INFO:tasks.workunit.client.0.smithi144.stdout:5/142: read d1/d9/f19 [54830,35383] 0 2024-07-26T11:04:37.223 INFO:tasks.workunit.client.0.smithi144.stdout:8/140: dwrite d1a/d1c/f2d [0,4194304] 0 2024-07-26T11:04:37.228 INFO:tasks.workunit.client.0.smithi144.stdout:6/157: creat d8/d15/f35 x:0 0 0 2024-07-26T11:04:37.228 INFO:tasks.workunit.client.0.smithi144.stdout:4/101: dread f0 [0,4194304] 0 2024-07-26T11:04:37.228 INFO:tasks.workunit.client.0.smithi144.stdout:8/141: write d1a/f20 [920831,13871] 0 2024-07-26T11:04:37.240 INFO:tasks.workunit.client.0.smithi144.stdout:5/143: getdents d1/d1b 0 2024-07-26T11:04:37.240 INFO:tasks.workunit.client.0.smithi144.stdout:4/102: symlink d9/da/l1d 0 2024-07-26T11:04:37.240 INFO:tasks.workunit.client.0.smithi144.stdout:5/144: rename d1 to d1/d1b/d34 22 2024-07-26T11:04:37.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/158: mknod d8/db/c36 0 2024-07-26T11:04:37.244 INFO:tasks.workunit.client.0.smithi144.stdout:4/103: symlink d9/da/l1e 0 2024-07-26T11:04:37.244 INFO:tasks.workunit.client.0.smithi144.stdout:4/104: chown d9/c18 65233 1 2024-07-26T11:04:37.247 INFO:tasks.workunit.client.0.smithi144.stdout:5/145: mkdir d1/d18/d24/d35 0 2024-07-26T11:04:37.252 INFO:tasks.workunit.client.0.smithi144.stdout:5/146: rmdir d1 39 2024-07-26T11:04:37.253 INFO:tasks.workunit.client.0.smithi144.stdout:5/147: read - d1/d5/f2d zero size 2024-07-26T11:04:37.253 INFO:tasks.workunit.client.0.smithi144.stdout:2/69: dwrite d0/f1 [0,4194304] 0 2024-07-26T11:04:37.253 INFO:tasks.workunit.client.0.smithi144.stdout:2/70: mknod d0/d15/c19 0 2024-07-26T11:04:37.253 INFO:tasks.workunit.client.0.smithi144.stdout:2/71: chown d0/d2 239176 1 2024-07-26T11:04:37.253 INFO:tasks.workunit.client.0.smithi144.stdout:2/72: chown d0/d15/l16 241211 1 2024-07-26T11:04:37.254 INFO:tasks.workunit.client.0.smithi144.stdout:2/73: rmdir d0/d15 39 2024-07-26T11:04:37.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/74: mknod d0/d6/c1a 0 2024-07-26T11:04:37.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/75: dread - d0/d15/f17 zero size 2024-07-26T11:04:37.258 INFO:tasks.workunit.client.0.smithi144.stdout:2/76: unlink d0/d2/c12 0 2024-07-26T11:04:37.269 INFO:tasks.workunit.client.0.smithi144.stdout:0/167: dwrite d0/f6 [0,4194304] 0 2024-07-26T11:04:37.271 INFO:tasks.workunit.client.0.smithi144.stdout:0/168: unlink d0/d14/l22 0 2024-07-26T11:04:37.272 INFO:tasks.workunit.client.0.smithi144.stdout:0/169: chown d0/d14/d29/l31 181096016 1 2024-07-26T11:04:37.272 INFO:tasks.workunit.client.0.smithi144.stdout:0/170: creat d0/d14/d1e/d30/f3b x:0 0 0 2024-07-26T11:04:37.274 INFO:tasks.workunit.client.0.smithi144.stdout:0/171: rename d0/f6 to d0/d14/d1e/f3c 0 2024-07-26T11:04:37.275 INFO:tasks.workunit.client.0.smithi144.stdout:0/172: creat d0/d25/f3d x:0 0 0 2024-07-26T11:04:37.276 INFO:tasks.workunit.client.0.smithi144.stdout:0/173: mkdir d0/d3e 0 2024-07-26T11:04:37.281 INFO:tasks.workunit.client.0.smithi144.stdout:8/142: dwrite d1a/d1c/f22 [0,4194304] 0 2024-07-26T11:04:37.283 INFO:tasks.workunit.client.0.smithi144.stdout:8/143: write d1a/d1c/f1f [637691,120194] 0 2024-07-26T11:04:37.289 INFO:tasks.workunit.client.0.smithi144.stdout:6/159: dread f2 [0,4194304] 0 2024-07-26T11:04:37.291 INFO:tasks.workunit.client.0.smithi144.stdout:6/160: symlink d8/d15/l37 0 2024-07-26T11:04:37.291 INFO:tasks.workunit.client.0.smithi144.stdout:6/161: stat d8/db/c36 0 2024-07-26T11:04:37.304 INFO:tasks.workunit.client.0.smithi144.stdout:4/105: dwrite d9/fb [0,4194304] 0 2024-07-26T11:04:37.307 INFO:tasks.workunit.client.0.smithi144.stdout:5/148: dwrite d1/d5/f2d [0,4194304] 0 2024-07-26T11:04:37.307 INFO:tasks.workunit.client.0.smithi144.stdout:5/149: write d1/d18/d24/d27/f2f [984600,81167] 0 2024-07-26T11:04:37.307 INFO:tasks.workunit.client.0.smithi144.stdout:5/150: fsync d1/d9/f10 0 2024-07-26T11:04:37.315 INFO:tasks.workunit.client.0.smithi144.stdout:0/174: dread d0/f18 [0,4194304] 0 2024-07-26T11:04:37.316 INFO:tasks.workunit.client.0.smithi144.stdout:0/175: creat d0/d14/d1e/f3f x:0 0 0 2024-07-26T11:04:37.318 INFO:tasks.workunit.client.0.smithi144.stdout:2/77: dwrite d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:37.318 INFO:tasks.workunit.client.0.smithi144.stdout:2/78: fdatasync d0/f1 0 2024-07-26T11:04:37.319 INFO:tasks.workunit.client.0.smithi144.stdout:8/144: write fe [418302,18922] 0 2024-07-26T11:04:37.323 INFO:tasks.workunit.client.0.smithi144.stdout:2/79: truncate d0/d2/f4 640006 0 2024-07-26T11:04:37.323 INFO:tasks.workunit.client.0.smithi144.stdout:2/80: chown d0/d6/d9 320 1 2024-07-26T11:04:37.324 INFO:tasks.workunit.client.0.smithi144.stdout:2/81: getdents d0/d6/d9 0 2024-07-26T11:04:37.327 INFO:tasks.workunit.client.0.smithi144.stdout:2/82: mknod d0/c1b 0 2024-07-26T11:04:37.327 INFO:tasks.workunit.client.0.smithi144.stdout:4/106: dread f1 [0,4194304] 0 2024-07-26T11:04:37.337 INFO:tasks.workunit.client.0.smithi144.stdout:4/107: rename d9/da/fc to d9/f1f 0 2024-07-26T11:04:37.337 INFO:tasks.workunit.client.0.smithi144.stdout:4/108: write d9/da/f1a [810774,11791] 0 2024-07-26T11:04:37.359 INFO:tasks.workunit.client.0.smithi144.stdout:4/109: dread f1 [0,4194304] 0 2024-07-26T11:04:37.368 INFO:tasks.workunit.client.0.smithi144.stdout:5/151: dwrite d1/d18/d24/d27/f2f [0,4194304] 0 2024-07-26T11:04:37.388 INFO:tasks.workunit.client.0.smithi144.stdout:0/176: dwrite d0/d14/d1e/f36 [0,4194304] 0 2024-07-26T11:04:37.390 INFO:tasks.workunit.client.0.smithi144.stdout:0/177: symlink d0/d14/d1e/l40 0 2024-07-26T11:04:37.390 INFO:tasks.workunit.client.0.smithi144.stdout:0/178: truncate d0/d14/d1e/d30/f39 686003 0 2024-07-26T11:04:37.391 INFO:tasks.workunit.client.0.smithi144.stdout:0/179: creat d0/d14/f41 x:0 0 0 2024-07-26T11:04:37.392 INFO:tasks.workunit.client.0.smithi144.stdout:2/83: dwrite d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:37.392 INFO:tasks.workunit.client.0.smithi144.stdout:2/84: fsync d0/f1 0 2024-07-26T11:04:37.392 INFO:tasks.workunit.client.0.smithi144.stdout:2/85: chown d0/d2/f4 467 1 2024-07-26T11:04:37.392 INFO:tasks.workunit.client.0.smithi144.stdout:2/86: write d0/d15/f17 [762671,45668] 0 2024-07-26T11:04:37.393 INFO:tasks.workunit.client.0.smithi144.stdout:0/180: creat d0/d14/d29/f42 x:0 0 0 2024-07-26T11:04:37.393 INFO:tasks.workunit.client.0.smithi144.stdout:8/145: dwrite f4 [0,4194304] 0 2024-07-26T11:04:37.393 INFO:tasks.workunit.client.0.smithi144.stdout:2/87: write d0/d2/f4 [4511888,94131] 0 2024-07-26T11:04:37.393 INFO:tasks.workunit.client.0.smithi144.stdout:8/146: chown d1a/d1c/f22 53629579 1 2024-07-26T11:04:37.394 INFO:tasks.workunit.client.0.smithi144.stdout:8/147: chown l14 256346293 1 2024-07-26T11:04:37.395 INFO:tasks.workunit.client.0.smithi144.stdout:8/148: write d1a/f1b [1911959,56259] 0 2024-07-26T11:04:37.395 INFO:tasks.workunit.client.0.smithi144.stdout:0/181: creat d0/d3e/f43 x:0 0 0 2024-07-26T11:04:37.395 INFO:tasks.workunit.client.0.smithi144.stdout:0/182: fdatasync d0/f27 0 2024-07-26T11:04:37.402 INFO:tasks.workunit.client.0.smithi144.stdout:5/152: write d1/d18/d24/d27/f2f [907663,64131] 0 2024-07-26T11:04:37.402 INFO:tasks.workunit.client.0.smithi144.stdout:5/153: stat d1/d1b/f21 0 2024-07-26T11:04:37.402 INFO:tasks.workunit.client.0.smithi144.stdout:5/154: fsync d1/d5/f2e 0 2024-07-26T11:04:37.403 INFO:tasks.workunit.client.0.smithi144.stdout:5/155: symlink d1/d1b/l36 0 2024-07-26T11:04:37.403 INFO:tasks.workunit.client.0.smithi144.stdout:4/110: dwrite f3 [0,4194304] 0 2024-07-26T11:04:37.403 INFO:tasks.workunit.client.0.smithi144.stdout:4/111: fdatasync f6 0 2024-07-26T11:04:37.404 INFO:tasks.workunit.client.0.smithi144.stdout:4/112: readlink d9/da/l1e 0 2024-07-26T11:04:37.406 INFO:tasks.workunit.client.0.smithi144.stdout:4/113: mknod d9/c20 0 2024-07-26T11:04:37.407 INFO:tasks.workunit.client.0.smithi144.stdout:4/114: mknod d9/da/c21 0 2024-07-26T11:04:37.418 INFO:tasks.workunit.client.0.smithi144.stdout:4/115: getdents d9/da 0 2024-07-26T11:04:37.418 INFO:tasks.workunit.client.0.smithi144.stdout:4/116: stat d9/f1f 0 2024-07-26T11:04:37.418 INFO:tasks.workunit.client.0.smithi144.stdout:4/117: write d9/f15 [779634,16915] 0 2024-07-26T11:04:37.422 INFO:tasks.workunit.client.0.smithi144.stdout:1/172: sync 2024-07-26T11:04:37.423 INFO:tasks.workunit.client.0.smithi144.stdout:1/173: mkdir d3/d6/da/d13/d19/d27/d36/d37 0 2024-07-26T11:04:37.424 INFO:tasks.workunit.client.0.smithi144.stdout:1/174: link d3/d6/da/d21/l2b d3/d6/l38 0 2024-07-26T11:04:37.425 INFO:tasks.workunit.client.0.smithi144.stdout:1/175: stat d3/d6/da/d21/l2b 0 2024-07-26T11:04:37.448 INFO:tasks.workunit.client.0.smithi144.stdout:1/176: rename d3/fc to d3/f39 0 2024-07-26T11:04:37.448 INFO:tasks.workunit.client.0.smithi144.stdout:1/177: read - d3/d6/da/d13/f26 zero size 2024-07-26T11:04:37.448 INFO:tasks.workunit.client.0.smithi144.stdout:1/178: write d3/d6/da/d13/f26 [98463,103358] 0 2024-07-26T11:04:37.465 INFO:tasks.workunit.client.0.smithi144.stdout:2/88: dwrite d0/f14 [0,4194304] 0 2024-07-26T11:04:37.473 INFO:tasks.workunit.client.0.smithi144.stdout:2/89: rename d0/lb to d0/d6/d9/l1c 0 2024-07-26T11:04:37.482 INFO:tasks.workunit.client.0.smithi144.stdout:5/156: dwrite d1/d18/f1d [0,4194304] 0 2024-07-26T11:04:37.482 INFO:tasks.workunit.client.0.smithi144.stdout:4/118: write f0 [672194,48305] 0 2024-07-26T11:04:37.484 INFO:tasks.workunit.client.0.smithi144.stdout:5/157: mkdir d1/d18/d24/d27/d37 0 2024-07-26T11:04:37.486 INFO:tasks.workunit.client.0.smithi144.stdout:8/149: dwrite f5 [0,4194304] 0 2024-07-26T11:04:37.486 INFO:tasks.workunit.client.0.smithi144.stdout:0/183: dwrite d0/d14/d1e/f24 [0,4194304] 0 2024-07-26T11:04:37.491 INFO:tasks.workunit.client.0.smithi144.stdout:0/184: symlink d0/l44 0 2024-07-26T11:04:37.491 INFO:tasks.workunit.client.0.smithi144.stdout:0/185: dread - d0/d25/f3d zero size 2024-07-26T11:04:37.491 INFO:tasks.workunit.client.0.smithi144.stdout:0/186: dread - d0/d14/d29/f2d zero size 2024-07-26T11:04:37.493 INFO:tasks.workunit.client.0.smithi144.stdout:0/187: mkdir d0/d3e/d45 0 2024-07-26T11:04:37.497 INFO:tasks.workunit.client.0.smithi144.stdout:0/188: creat d0/d3e/d45/f46 x:0 0 0 2024-07-26T11:04:37.497 INFO:tasks.workunit.client.0.smithi144.stdout:1/179: dread d3/f1e [0,4194304] 0 2024-07-26T11:04:37.500 INFO:tasks.workunit.client.0.smithi144.stdout:0/189: rename d0/d14/l2a to d0/d25/l47 0 2024-07-26T11:04:37.500 INFO:tasks.workunit.client.0.smithi144.stdout:0/190: dread - d0/f32 zero size 2024-07-26T11:04:37.500 INFO:tasks.workunit.client.0.smithi144.stdout:3/87: sync 2024-07-26T11:04:37.500 INFO:tasks.workunit.client.0.smithi144.stdout:6/162: sync 2024-07-26T11:04:37.500 INFO:tasks.workunit.client.0.smithi144.stdout:9/108: sync 2024-07-26T11:04:37.503 INFO:tasks.workunit.client.0.smithi144.stdout:7/134: sync 2024-07-26T11:04:37.503 INFO:tasks.workunit.client.0.smithi144.stdout:2/90: dread d0/f14 [0,4194304] 0 2024-07-26T11:04:37.503 INFO:tasks.workunit.client.0.smithi144.stdout:2/91: chown d0/cc 14 1 2024-07-26T11:04:37.506 INFO:tasks.workunit.client.0.smithi144.stdout:6/163: truncate f2 384042 0 2024-07-26T11:04:37.506 INFO:tasks.workunit.client.0.smithi144.stdout:9/109: readlink l8 0 2024-07-26T11:04:37.506 INFO:tasks.workunit.client.0.smithi144.stdout:3/88: getdents d1/d5 0 2024-07-26T11:04:37.507 INFO:tasks.workunit.client.0.smithi144.stdout:3/89: truncate d1/f17 74557 0 2024-07-26T11:04:37.508 INFO:tasks.workunit.client.0.smithi144.stdout:3/90: link d1/c18 d1/c19 0 2024-07-26T11:04:37.508 INFO:tasks.workunit.client.0.smithi144.stdout:3/91: write d1/d5/f15 [654091,73441] 0 2024-07-26T11:04:37.523 INFO:tasks.workunit.client.0.smithi144.stdout:5/158: dread d1/d18/d24/d27/f2f [0,4194304] 0 2024-07-26T11:04:37.526 INFO:tasks.workunit.client.0.smithi144.stdout:1/180: dread f1 [0,4194304] 0 2024-07-26T11:04:37.531 INFO:tasks.workunit.client.0.smithi144.stdout:5/159: rename d1/l6 to d1/d18/d24/d27/d37/l38 0 2024-07-26T11:04:37.548 INFO:tasks.workunit.client.0.smithi144.stdout:7/135: dread d0/d10/f2a [0,4194304] 0 2024-07-26T11:04:37.555 INFO:tasks.workunit.client.0.smithi144.stdout:7/136: getdents d0/d6/d9/d25/d2f 0 2024-07-26T11:04:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:37 smithi160 ceph-mon[36715]: pgmap v525: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 25 MiB/s wr, 755 op/s 2024-07-26T11:04:37.562 INFO:tasks.workunit.client.0.smithi144.stdout:4/119: dread f1 [0,4194304] 0 2024-07-26T11:04:37.564 INFO:tasks.workunit.client.0.smithi144.stdout:4/120: fdatasync d9/f1f 0 2024-07-26T11:04:37.569 INFO:tasks.workunit.client.0.smithi144.stdout:9/110: dwrite d9/de/d14/f24 [0,4194304] 0 2024-07-26T11:04:37.570 INFO:tasks.workunit.client.0.smithi144.stdout:2/92: dread d0/f1 [0,4194304] 0 2024-07-26T11:04:37.583 INFO:tasks.workunit.client.0.smithi144.stdout:2/93: creat d0/d6/f1d x:0 0 0 2024-07-26T11:04:37.584 INFO:tasks.workunit.client.0.smithi144.stdout:2/94: creat d0/f1e x:0 0 0 2024-07-26T11:04:37.585 INFO:tasks.workunit.client.0.smithi144.stdout:2/95: creat d0/f1f x:0 0 0 2024-07-26T11:04:37.588 INFO:tasks.workunit.client.0.smithi144.stdout:2/96: mkdir d0/d15/d20 0 2024-07-26T11:04:37.588 INFO:tasks.workunit.client.0.smithi144.stdout:2/97: fdatasync d0/d6/f1d 0 2024-07-26T11:04:37.589 INFO:tasks.workunit.client.0.smithi144.stdout:3/92: dwrite d1/d5/fa [0,4194304] 0 2024-07-26T11:04:37.589 INFO:tasks.workunit.client.0.smithi144.stdout:2/98: mknod d0/c21 0 2024-07-26T11:04:37.597 INFO:tasks.workunit.client.0.smithi144.stdout:1/181: dwrite d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:37.598 INFO:tasks.workunit.client.0.smithi144.stdout:1/182: mkdir d3/d6/da/d13/d19/d3a 0 2024-07-26T11:04:37.598 INFO:tasks.workunit.client.0.smithi144.stdout:1/183: readlink d3/d6/da/l1d 0 2024-07-26T11:04:37.600 INFO:tasks.workunit.client.0.smithi144.stdout:6/164: dwrite d8/db/f2c [0,4194304] 0 2024-07-26T11:04:37.600 INFO:tasks.workunit.client.0.smithi144.stdout:1/184: creat d3/f3b x:0 0 0 2024-07-26T11:04:37.604 INFO:tasks.workunit.client.0.smithi144.stdout:5/160: dread d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:37.604 INFO:tasks.workunit.client.0.smithi144.stdout:5/161: fsync d1/d5/f2e 0 2024-07-26T11:04:37.604 INFO:tasks.workunit.client.0.smithi144.stdout:7/137: dwrite d0/f2d [0,4194304] 0 2024-07-26T11:04:37.629 INFO:tasks.workunit.client.0.smithi144.stdout:6/165: creat d8/d15/f38 x:0 0 0 2024-07-26T11:04:37.629 INFO:tasks.workunit.client.0.smithi144.stdout:6/166: readlink d8/l14 0 2024-07-26T11:04:37.630 INFO:tasks.workunit.client.0.smithi144.stdout:6/167: getdents d8/db 0 2024-07-26T11:04:37.631 INFO:tasks.workunit.client.0.smithi144.stdout:4/121: dwrite f3 [0,4194304] 0 2024-07-26T11:04:37.633 INFO:tasks.workunit.client.0.smithi144.stdout:3/93: dwrite d1/d5/f15 [0,4194304] 0 2024-07-26T11:04:37.633 INFO:tasks.workunit.client.0.smithi144.stdout:3/94: chown d1/d5 10 1 2024-07-26T11:04:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:37 smithi183 ceph-mon[37273]: pgmap v525: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 25 MiB/s wr, 755 op/s 2024-07-26T11:04:37.652 INFO:tasks.workunit.client.0.smithi144.stdout:4/122: dread f5 [0,4194304] 0 2024-07-26T11:04:37.654 INFO:tasks.workunit.client.0.smithi144.stdout:9/111: dwrite d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:37.655 INFO:tasks.workunit.client.0.smithi144.stdout:4/123: creat d9/f22 x:0 0 0 2024-07-26T11:04:37.656 INFO:tasks.workunit.client.0.smithi144.stdout:9/112: truncate d9/de/f20 3882169 0 2024-07-26T11:04:37.657 INFO:tasks.workunit.client.0.smithi144.stdout:9/113: unlink d9/de/d14/l1c 0 2024-07-26T11:04:37.658 INFO:tasks.workunit.client.0.smithi144.stdout:9/114: readlink d9/de/l12 0 2024-07-26T11:04:37.664 INFO:tasks.workunit.client.0.smithi144.stdout:5/162: dread d1/d18/d24/d27/f2f [0,4194304] 0 2024-07-26T11:04:37.664 INFO:tasks.workunit.client.0.smithi144.stdout:5/163: stat d1/d1b/l36 0 2024-07-26T11:04:37.665 INFO:tasks.workunit.client.0.smithi144.stdout:2/99: dwrite d0/f1f [0,4194304] 0 2024-07-26T11:04:37.667 INFO:tasks.workunit.client.0.smithi144.stdout:5/164: creat d1/f39 x:0 0 0 2024-07-26T11:04:37.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/165: mknod d1/d18/d24/d27/d37/c3a 0 2024-07-26T11:04:37.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/166: write d1/d9/f19 [348231,110822] 0 2024-07-26T11:04:37.673 INFO:tasks.workunit.client.0.smithi144.stdout:2/100: write d0/d15/f17 [575318,3438] 0 2024-07-26T11:04:37.673 INFO:tasks.workunit.client.0.smithi144.stdout:5/167: creat d1/d18/f3b x:0 0 0 2024-07-26T11:04:37.674 INFO:tasks.workunit.client.0.smithi144.stdout:5/168: symlink d1/d18/d24/d27/d37/l3c 0 2024-07-26T11:04:37.676 INFO:tasks.workunit.client.0.smithi144.stdout:5/169: creat d1/d18/d24/d35/f3d x:0 0 0 2024-07-26T11:04:37.678 INFO:tasks.workunit.client.0.smithi144.stdout:5/170: creat d1/d14/f3e x:0 0 0 2024-07-26T11:04:37.679 INFO:tasks.workunit.client.0.smithi144.stdout:5/171: rmdir d1/d14 39 2024-07-26T11:04:37.684 INFO:tasks.workunit.client.0.smithi144.stdout:5/172: rename d1/d18/d24/d27/d37 to d1/d5/d3f 0 2024-07-26T11:04:37.685 INFO:tasks.workunit.client.0.smithi144.stdout:5/173: symlink d1/d18/l40 0 2024-07-26T11:04:37.695 INFO:tasks.workunit.client.0.smithi144.stdout:2/101: dread d0/d2/f4 [0,4194304] 0 2024-07-26T11:04:37.697 INFO:tasks.workunit.client.0.smithi144.stdout:2/102: truncate d0/d2/f4 1090251 0 2024-07-26T11:04:37.705 INFO:tasks.workunit.client.0.smithi144.stdout:5/174: dread d1/d5/f2d [0,4194304] 0 2024-07-26T11:04:37.706 INFO:tasks.workunit.client.0.smithi144.stdout:5/175: creat d1/d5/d3f/f41 x:0 0 0 2024-07-26T11:04:37.706 INFO:tasks.workunit.client.0.smithi144.stdout:5/176: write d1/d14/f16 [9519109,41256] 0 2024-07-26T11:04:37.713 INFO:tasks.workunit.client.0.smithi144.stdout:6/168: dwrite f2 [0,4194304] 0 2024-07-26T11:04:37.715 INFO:tasks.workunit.client.0.smithi144.stdout:6/169: unlink d8/d15/l37 0 2024-07-26T11:04:37.716 INFO:tasks.workunit.client.0.smithi144.stdout:6/170: creat d8/d15/f39 x:0 0 0 2024-07-26T11:04:37.717 INFO:tasks.workunit.client.0.smithi144.stdout:9/115: dread d9/f1d [0,4194304] 0 2024-07-26T11:04:37.717 INFO:tasks.workunit.client.0.smithi144.stdout:3/95: dwrite d1/f14 [0,4194304] 0 2024-07-26T11:04:37.717 INFO:tasks.workunit.client.0.smithi144.stdout:3/96: readlink d1/l12 0 2024-07-26T11:04:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:37 smithi144 ceph-mon[32129]: pgmap v525: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 25 MiB/s wr, 755 op/s 2024-07-26T11:04:37.721 INFO:tasks.workunit.client.0.smithi144.stdout:6/171: mknod d8/db/c3a 0 2024-07-26T11:04:37.722 INFO:tasks.workunit.client.0.smithi144.stdout:2/103: write d0/f14 [2516820,35897] 0 2024-07-26T11:04:37.722 INFO:tasks.workunit.client.0.smithi144.stdout:9/116: read - d9/f1a zero size 2024-07-26T11:04:37.723 INFO:tasks.workunit.client.0.smithi144.stdout:3/97: creat d1/f1a x:0 0 0 2024-07-26T11:04:37.723 INFO:tasks.workunit.client.0.smithi144.stdout:3/98: fdatasync d1/d5/fa 0 2024-07-26T11:04:37.724 INFO:tasks.workunit.client.0.smithi144.stdout:6/172: symlink d8/d15/d26/l3b 0 2024-07-26T11:04:37.725 INFO:tasks.workunit.client.0.smithi144.stdout:2/104: unlink d0/d2/c18 0 2024-07-26T11:04:37.726 INFO:tasks.workunit.client.0.smithi144.stdout:2/105: readlink d0/d2/l5 0 2024-07-26T11:04:37.726 INFO:tasks.workunit.client.0.smithi144.stdout:2/106: write d0/d2/f4 [2099131,77885] 0 2024-07-26T11:04:37.728 INFO:tasks.workunit.client.0.smithi144.stdout:9/117: rename d9/f1d to d9/de/d14/d23/f27 0 2024-07-26T11:04:37.729 INFO:tasks.workunit.client.0.smithi144.stdout:3/99: creat d1/f1b x:0 0 0 2024-07-26T11:04:37.729 INFO:tasks.workunit.client.0.smithi144.stdout:5/177: dread d1/d9/f19 [0,4194304] 0 2024-07-26T11:04:37.729 INFO:tasks.workunit.client.0.smithi144.stdout:5/178: read d1/d14/f16 [6380229,61338] 0 2024-07-26T11:04:37.729 INFO:tasks.workunit.client.0.smithi144.stdout:5/179: readlink d1/d18/l40 0 2024-07-26T11:04:37.730 INFO:tasks.workunit.client.0.smithi144.stdout:9/118: rename d9/de/l19 to d9/de/d14/d23/l28 0 2024-07-26T11:04:37.731 INFO:tasks.workunit.client.0.smithi144.stdout:3/100: creat d1/d5/f1c x:0 0 0 2024-07-26T11:04:37.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/180: symlink d1/d9/l42 0 2024-07-26T11:04:37.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/181: fsync d1/d18/f1d 0 2024-07-26T11:04:37.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/182: write d1/d18/d24/d27/f2f [4782872,92121] 0 2024-07-26T11:04:37.732 INFO:tasks.workunit.client.0.smithi144.stdout:3/101: mkdir d1/d1d 0 2024-07-26T11:04:37.732 INFO:tasks.workunit.client.0.smithi144.stdout:3/102: write d1/f1a [585820,60779] 0 2024-07-26T11:04:37.774 INFO:tasks.workunit.client.0.smithi144.stdout:3/103: dwrite d1/d5/fa [0,4194304] 0 2024-07-26T11:04:37.779 INFO:tasks.workunit.client.0.smithi144.stdout:5/183: write d1/d1b/f21 [910931,22404] 0 2024-07-26T11:04:37.786 INFO:tasks.workunit.client.0.smithi144.stdout:9/119: write d9/de/d14/f22 [814103,3455] 0 2024-07-26T11:04:37.801 INFO:tasks.workunit.client.0.smithi144.stdout:3/104: write d1/d5/f15 [4018068,108941] 0 2024-07-26T11:04:37.801 INFO:tasks.workunit.client.0.smithi144.stdout:6/173: dwrite f2 [0,4194304] 0 2024-07-26T11:04:37.801 INFO:tasks.workunit.client.0.smithi144.stdout:6/174: stat d8/f17 0 2024-07-26T11:04:37.804 INFO:tasks.workunit.client.0.smithi144.stdout:5/184: mknod d1/d18/d24/d27/c43 0 2024-07-26T11:04:37.804 INFO:tasks.workunit.client.0.smithi144.stdout:9/120: creat d9/f29 x:0 0 0 2024-07-26T11:04:37.804 INFO:tasks.workunit.client.0.smithi144.stdout:5/185: dread - d1/d18/d24/d35/f3d zero size 2024-07-26T11:04:37.810 INFO:tasks.workunit.client.0.smithi144.stdout:6/175: mknod d8/d15/d26/c3c 0 2024-07-26T11:04:37.810 INFO:tasks.workunit.client.0.smithi144.stdout:6/176: stat d8/d15/d26/f30 0 2024-07-26T11:04:37.811 INFO:tasks.workunit.client.0.smithi144.stdout:6/177: mknod d8/d15/d26/c3d 0 2024-07-26T11:04:37.821 INFO:tasks.workunit.client.0.smithi144.stdout:2/107: dwrite d0/f14 [0,4194304] 0 2024-07-26T11:04:37.824 INFO:tasks.workunit.client.0.smithi144.stdout:2/108: getdents d0/d2 0 2024-07-26T11:04:37.825 INFO:tasks.workunit.client.0.smithi144.stdout:2/109: truncate d0/f1f 2414646 0 2024-07-26T11:04:37.825 INFO:tasks.workunit.client.0.smithi144.stdout:2/110: fsync d0/f1e 0 2024-07-26T11:04:37.827 INFO:tasks.workunit.client.0.smithi144.stdout:0/191: sync 2024-07-26T11:04:37.827 INFO:tasks.workunit.client.0.smithi144.stdout:8/150: sync 2024-07-26T11:04:37.827 INFO:tasks.workunit.client.0.smithi144.stdout:0/192: mknod d0/d14/c48 0 2024-07-26T11:04:37.828 INFO:tasks.workunit.client.0.smithi144.stdout:8/151: truncate d1a/d1c/f2d 228190 0 2024-07-26T11:04:37.829 INFO:tasks.workunit.client.0.smithi144.stdout:0/193: stat d0/f7 0 2024-07-26T11:04:37.829 INFO:tasks.workunit.client.0.smithi144.stdout:0/194: write d0/d25/f3d [854553,15501] 0 2024-07-26T11:04:37.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/195: mknod d0/d14/d1e/c49 0 2024-07-26T11:04:37.842 INFO:tasks.workunit.client.0.smithi144.stdout:2/111: write d0/d2/f4 [1873495,97339] 0 2024-07-26T11:04:37.842 INFO:tasks.workunit.client.0.smithi144.stdout:2/112: chown d0/d2/lf 58254 1 2024-07-26T11:04:37.850 INFO:tasks.workunit.client.0.smithi144.stdout:6/178: read d8/f33 [3653602,76355] 0 2024-07-26T11:04:37.851 INFO:tasks.workunit.client.0.smithi144.stdout:6/179: readlink d8/l20 0 2024-07-26T11:04:37.853 INFO:tasks.workunit.client.0.smithi144.stdout:3/105: dwrite d1/f14 [0,4194304] 0 2024-07-26T11:04:37.855 INFO:tasks.workunit.client.0.smithi144.stdout:6/180: rename d8/d15/d26/f29 to d8/db/f3e 0 2024-07-26T11:04:37.858 INFO:tasks.workunit.client.0.smithi144.stdout:3/106: rmdir d1/d1d 0 2024-07-26T11:04:37.860 INFO:tasks.workunit.client.0.smithi144.stdout:6/181: link d8/f10 d8/d15/d26/f3f 0 2024-07-26T11:04:37.860 INFO:tasks.workunit.client.0.smithi144.stdout:6/182: stat c5 0 2024-07-26T11:04:37.860 INFO:tasks.workunit.client.0.smithi144.stdout:3/107: mknod d1/d5/c1e 0 2024-07-26T11:04:37.860 INFO:tasks.workunit.client.0.smithi144.stdout:3/108: stat d1/f1b 0 2024-07-26T11:04:37.861 INFO:tasks.workunit.client.0.smithi144.stdout:6/183: symlink d8/d15/d26/l40 0 2024-07-26T11:04:37.861 INFO:tasks.workunit.client.0.smithi144.stdout:6/184: chown d8/f9 24 1 2024-07-26T11:04:37.861 INFO:tasks.workunit.client.0.smithi144.stdout:3/109: unlink d1/d5/c1e 0 2024-07-26T11:04:37.863 INFO:tasks.workunit.client.0.smithi144.stdout:9/121: dread d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:37.865 INFO:tasks.workunit.client.0.smithi144.stdout:6/185: link d8/db/f1b d8/d15/d26/f41 0 2024-07-26T11:04:37.865 INFO:tasks.workunit.client.0.smithi144.stdout:6/186: stat d8/d15/d26/f28 0 2024-07-26T11:04:37.866 INFO:tasks.workunit.client.0.smithi144.stdout:6/187: chown d8/f1d 169 1 2024-07-26T11:04:37.866 INFO:tasks.workunit.client.0.smithi144.stdout:6/188: chown d8/db/f2c 234748 1 2024-07-26T11:04:37.866 INFO:tasks.workunit.client.0.smithi144.stdout:5/186: dwrite d1/d5/f2d [0,4194304] 0 2024-07-26T11:04:37.866 INFO:tasks.workunit.client.0.smithi144.stdout:5/187: write d1/d14/f3e [524219,64066] 0 2024-07-26T11:04:37.867 INFO:tasks.workunit.client.0.smithi144.stdout:6/189: mkdir d8/d42 0 2024-07-26T11:04:37.871 INFO:tasks.workunit.client.0.smithi144.stdout:5/188: rmdir d1/d18 39 2024-07-26T11:04:37.871 INFO:tasks.workunit.client.0.smithi144.stdout:6/190: rename d8/db/f2a to d8/d42/f43 0 2024-07-26T11:04:37.879 INFO:tasks.workunit.client.0.smithi144.stdout:7/138: sync 2024-07-26T11:04:37.879 INFO:tasks.workunit.client.0.smithi144.stdout:1/185: sync 2024-07-26T11:04:37.879 INFO:tasks.workunit.client.0.smithi144.stdout:4/124: sync 2024-07-26T11:04:37.882 INFO:tasks.workunit.client.0.smithi144.stdout:1/186: truncate f1 705735 0 2024-07-26T11:04:37.882 INFO:tasks.workunit.client.0.smithi144.stdout:1/187: dread - d3/d6/da/d13/f29 zero size 2024-07-26T11:04:37.888 INFO:tasks.workunit.client.0.smithi144.stdout:2/113: dwrite d0/d6/f1d [0,4194304] 0 2024-07-26T11:04:37.892 INFO:tasks.workunit.client.0.smithi144.stdout:7/139: symlink d0/d6/d9/d25/d2f/l3a 0 2024-07-26T11:04:37.892 INFO:tasks.workunit.client.0.smithi144.stdout:7/140: chown d0/d10 850 1 2024-07-26T11:04:37.894 INFO:tasks.workunit.client.0.smithi144.stdout:9/122: write f4 [1690836,60187] 0 2024-07-26T11:04:37.895 INFO:tasks.workunit.client.0.smithi144.stdout:9/123: write d9/de/d14/f22 [751599,43636] 0 2024-07-26T11:04:37.895 INFO:tasks.workunit.client.0.smithi144.stdout:7/141: mknod d0/d6/d9/d25/d2f/c3b 0 2024-07-26T11:04:37.895 INFO:tasks.workunit.client.0.smithi144.stdout:7/142: write d0/d6/d9/d25/d2f/f39 [285320,27149] 0 2024-07-26T11:04:37.895 INFO:tasks.workunit.client.0.smithi144.stdout:7/143: chown d0/d10/f2a 36 1 2024-07-26T11:04:37.896 INFO:tasks.workunit.client.0.smithi144.stdout:7/144: truncate d0/d6/d9/d25/d2f/f39 616940 0 2024-07-26T11:04:37.896 INFO:tasks.workunit.client.0.smithi144.stdout:7/145: chown d0/d6/d9/d12/l21 4 1 2024-07-26T11:04:37.896 INFO:tasks.workunit.client.0.smithi144.stdout:2/114: truncate d0/d2/f4 674261 0 2024-07-26T11:04:37.896 INFO:tasks.workunit.client.0.smithi144.stdout:2/115: stat d0/l13 0 2024-07-26T11:04:37.896 INFO:tasks.workunit.client.0.smithi144.stdout:0/196: dwrite d0/d14/d1e/f3f [0,4194304] 0 2024-07-26T11:04:37.896 INFO:tasks.workunit.client.0.smithi144.stdout:0/197: dread - d0/d14/f2c zero size 2024-07-26T11:04:37.899 INFO:tasks.workunit.client.0.smithi144.stdout:2/116: creat d0/f22 x:0 0 0 2024-07-26T11:04:37.900 INFO:tasks.workunit.client.0.smithi144.stdout:0/198: creat d0/d3e/d45/f4a x:0 0 0 2024-07-26T11:04:37.903 INFO:tasks.workunit.client.0.smithi144.stdout:2/117: mknod d0/c23 0 2024-07-26T11:04:37.907 INFO:tasks.workunit.client.0.smithi144.stdout:0/199: mkdir d0/d14/d1e/d30/d4b 0 2024-07-26T11:04:37.919 INFO:tasks.workunit.client.0.smithi144.stdout:9/124: truncate f4 3620334 0 2024-07-26T11:04:37.919 INFO:tasks.workunit.client.0.smithi144.stdout:3/110: dwrite d1/f17 [0,4194304] 0 2024-07-26T11:04:37.920 INFO:tasks.workunit.client.0.smithi144.stdout:3/111: write d1/d5/f1c [416447,100476] 0 2024-07-26T11:04:37.921 INFO:tasks.workunit.client.0.smithi144.stdout:9/125: rename l8 to d9/de/d14/d23/l2a 0 2024-07-26T11:04:37.921 INFO:tasks.workunit.client.0.smithi144.stdout:9/126: dread - d9/f29 zero size 2024-07-26T11:04:37.925 INFO:tasks.workunit.client.0.smithi144.stdout:9/127: mknod d9/de/d14/d23/c2b 0 2024-07-26T11:04:37.925 INFO:tasks.workunit.client.0.smithi144.stdout:3/112: unlink d1/l12 0 2024-07-26T11:04:37.927 INFO:tasks.workunit.client.0.smithi144.stdout:9/128: rename d9/de/d14/d23/f27 to d9/de/d14/f2c 0 2024-07-26T11:04:37.928 INFO:tasks.workunit.client.0.smithi144.stdout:3/113: symlink d1/d5/l1f 0 2024-07-26T11:04:37.928 INFO:tasks.workunit.client.0.smithi144.stdout:3/114: fsync d1/f17 0 2024-07-26T11:04:37.931 INFO:tasks.workunit.client.0.smithi144.stdout:3/115: creat d1/f20 x:0 0 0 2024-07-26T11:04:37.933 INFO:tasks.workunit.client.0.smithi144.stdout:9/129: mkdir d9/d2d 0 2024-07-26T11:04:37.933 INFO:tasks.workunit.client.0.smithi144.stdout:9/130: readlink d9/de/l12 0 2024-07-26T11:04:37.937 INFO:tasks.workunit.client.0.smithi144.stdout:5/189: dwrite d1/d14/f3e [0,4194304] 0 2024-07-26T11:04:37.942 INFO:tasks.workunit.client.0.smithi144.stdout:5/190: rename d1/d5/d3f to d1/d9/d44 0 2024-07-26T11:04:37.945 INFO:tasks.workunit.client.0.smithi144.stdout:3/116: getdents d1 0 2024-07-26T11:04:37.945 INFO:tasks.workunit.client.0.smithi144.stdout:3/117: fsync d1/ff 0 2024-07-26T11:04:37.945 INFO:tasks.workunit.client.0.smithi144.stdout:9/131: getdents d9 0 2024-07-26T11:04:37.947 INFO:tasks.workunit.client.0.smithi144.stdout:5/191: mkdir d1/d9/d44/d45 0 2024-07-26T11:04:37.948 INFO:tasks.workunit.client.0.smithi144.stdout:9/132: rmdir d9/d2d 0 2024-07-26T11:04:37.948 INFO:tasks.workunit.client.0.smithi144.stdout:9/133: stat d9/l25 0 2024-07-26T11:04:37.948 INFO:tasks.workunit.client.0.smithi144.stdout:6/191: dwrite d8/db/f2c [0,4194304] 0 2024-07-26T11:04:37.949 INFO:tasks.workunit.client.0.smithi144.stdout:5/192: creat d1/d18/d24/f46 x:0 0 0 2024-07-26T11:04:37.949 INFO:tasks.workunit.client.0.smithi144.stdout:9/134: rmdir d9/de/d14 39 2024-07-26T11:04:37.951 INFO:tasks.workunit.client.0.smithi144.stdout:9/135: symlink d9/de/d14/l2e 0 2024-07-26T11:04:37.952 INFO:tasks.workunit.client.0.smithi144.stdout:9/136: creat d9/de/f2f x:0 0 0 2024-07-26T11:04:37.954 INFO:tasks.workunit.client.0.smithi144.stdout:9/137: link d9/de/f21 d9/de/f30 0 2024-07-26T11:04:37.955 INFO:tasks.workunit.client.0.smithi144.stdout:7/146: dwrite d0/d6/d9/fc [0,4194304] 0 2024-07-26T11:04:37.956 INFO:tasks.workunit.client.0.smithi144.stdout:7/147: rmdir d0/d6/d9/d12 39 2024-07-26T11:04:37.957 INFO:tasks.workunit.client.0.smithi144.stdout:7/148: getdents d0/d6 0 2024-07-26T11:04:37.957 INFO:tasks.workunit.client.0.smithi144.stdout:7/149: dread - d0/d6/f2b zero size 2024-07-26T11:04:37.958 INFO:tasks.workunit.client.0.smithi144.stdout:7/150: mknod d0/d6/d9/d12/c3c 0 2024-07-26T11:04:37.962 INFO:tasks.workunit.client.0.smithi144.stdout:5/193: getdents d1/d9 0 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/194: mknod d1/c47 0 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/195: write d1/d5/f2e [833133,3978] 0 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/196: dread - d1/d18/f3b zero size 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/197: truncate d1/d18/d24/f46 601378 0 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/198: readlink d1/d9/l1e 0 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/199: chown d1/d5/f2d 1003682262 1 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/200: write d1/d18/d24/d27/f2f [5803255,34615] 0 2024-07-26T11:04:37.964 INFO:tasks.workunit.client.0.smithi144.stdout:5/201: getdents d1/d9 0 2024-07-26T11:04:37.965 INFO:tasks.workunit.client.0.smithi144.stdout:5/202: creat d1/d18/f48 x:0 0 0 2024-07-26T11:04:37.969 INFO:tasks.workunit.client.0.smithi144.stdout:3/118: write d1/ff [1706843,38643] 0 2024-07-26T11:04:37.972 INFO:tasks.workunit.client.0.smithi144.stdout:9/138: fsync d9/de/f30 0 2024-07-26T11:04:37.972 INFO:tasks.workunit.client.0.smithi144.stdout:9/139: stat d9/de/f20 0 2024-07-26T11:04:37.973 INFO:tasks.workunit.client.0.smithi144.stdout:6/192: write d8/db/f25 [506091,104518] 0 2024-07-26T11:04:37.973 INFO:tasks.workunit.client.0.smithi144.stdout:3/119: symlink d1/d5/l21 0 2024-07-26T11:04:37.975 INFO:tasks.workunit.client.0.smithi144.stdout:6/193: truncate d8/db/f2c 2880973 0 2024-07-26T11:04:37.975 INFO:tasks.workunit.client.0.smithi144.stdout:6/194: truncate d8/d15/f35 392498 0 2024-07-26T11:04:37.977 INFO:tasks.workunit.client.0.smithi144.stdout:6/195: creat d8/db/f44 x:0 0 0 2024-07-26T11:04:37.984 INFO:tasks.workunit.client.0.smithi144.stdout:6/196: creat d8/f45 x:0 0 0 2024-07-26T11:04:37.990 INFO:tasks.workunit.client.0.smithi144.stdout:0/200: dwrite d0/d3e/f43 [0,4194304] 0 2024-07-26T11:04:37.991 INFO:tasks.workunit.client.0.smithi144.stdout:5/203: dread d1/d14/f16 [4194304,4194304] 0 2024-07-26T11:04:37.995 INFO:tasks.workunit.client.0.smithi144.stdout:0/201: rmdir d0/d14/d29 39 2024-07-26T11:04:37.995 INFO:tasks.workunit.client.0.smithi144.stdout:0/202: chown d0/c9 704250 1 2024-07-26T11:04:37.998 INFO:tasks.workunit.client.0.smithi144.stdout:5/204: truncate d1/d14/f3e 2484099 0 2024-07-26T11:04:38.020 INFO:tasks.workunit.client.0.smithi144.stdout:5/205: dread d1/d1b/f21 [0,4194304] 0 2024-07-26T11:04:38.020 INFO:tasks.workunit.client.0.smithi144.stdout:5/206: chown d1/c32 55 1 2024-07-26T11:04:38.022 INFO:tasks.workunit.client.0.smithi144.stdout:5/207: creat d1/d5/f49 x:0 0 0 2024-07-26T11:04:38.025 INFO:tasks.workunit.client.0.smithi144.stdout:9/140: dwrite f4 [0,4194304] 0 2024-07-26T11:04:38.025 INFO:tasks.workunit.client.0.smithi144.stdout:9/141: write d9/de/f30 [163483,63463] 0 2024-07-26T11:04:38.025 INFO:tasks.workunit.client.0.smithi144.stdout:9/142: chown d9/de/ff 10579 1 2024-07-26T11:04:38.027 INFO:tasks.workunit.client.0.smithi144.stdout:9/143: creat d9/de/d14/d23/f31 x:0 0 0 2024-07-26T11:04:38.027 INFO:tasks.workunit.client.0.smithi144.stdout:9/144: read - d9/de/d14/d23/f31 zero size 2024-07-26T11:04:38.027 INFO:tasks.workunit.client.0.smithi144.stdout:9/145: dread - d9/de/d14/d23/f31 zero size 2024-07-26T11:04:38.027 INFO:tasks.workunit.client.0.smithi144.stdout:0/203: dread d0/d14/d1e/f3c [0,4194304] 0 2024-07-26T11:04:38.027 INFO:tasks.workunit.client.0.smithi144.stdout:9/146: symlink d9/de/l32 0 2024-07-26T11:04:38.074 INFO:tasks.workunit.client.0.smithi144.stdout:7/151: dwrite d0/d6/d9/d25/d2f/f35 [0,4194304] 0 2024-07-26T11:04:38.074 INFO:tasks.workunit.client.0.smithi144.stdout:7/152: write d0/d6/f2b [328938,73348] 0 2024-07-26T11:04:38.078 INFO:tasks.workunit.client.0.smithi144.stdout:0/204: dwrite d0/d14/f41 [0,4194304] 0 2024-07-26T11:04:38.082 INFO:tasks.workunit.client.0.smithi144.stdout:7/153: mknod d0/c3d 0 2024-07-26T11:04:38.085 INFO:tasks.workunit.client.0.smithi144.stdout:7/154: creat d0/d10/f3e x:0 0 0 2024-07-26T11:04:38.087 INFO:tasks.workunit.client.0.smithi144.stdout:7/155: mkdir d0/d6/d9/d3f 0 2024-07-26T11:04:38.087 INFO:tasks.workunit.client.0.smithi144.stdout:5/208: write d1/d18/d24/f2a [3703906,83556] 0 2024-07-26T11:04:38.087 INFO:tasks.workunit.client.0.smithi144.stdout:5/209: chown d1/d18/f1d 31500991 1 2024-07-26T11:04:38.088 INFO:tasks.workunit.client.0.smithi144.stdout:7/156: mknod d0/d10/c40 0 2024-07-26T11:04:38.090 INFO:tasks.workunit.client.0.smithi144.stdout:9/147: dwrite d9/de/d14/d23/f31 [0,4194304] 0 2024-07-26T11:04:38.090 INFO:tasks.workunit.client.0.smithi144.stdout:9/148: readlink d9/de/d14/d23/l28 0 2024-07-26T11:04:38.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/210: symlink d1/d9/d44/d45/l4a 0 2024-07-26T11:04:38.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/211: chown d1/c32 124638967 1 2024-07-26T11:04:38.093 INFO:tasks.workunit.client.0.smithi144.stdout:9/149: getdents d9/de 0 2024-07-26T11:04:38.093 INFO:tasks.workunit.client.0.smithi144.stdout:0/205: dread d0/f8 [0,4194304] 0 2024-07-26T11:04:38.093 INFO:tasks.workunit.client.0.smithi144.stdout:6/197: write f6 [5021973,44330] 0 2024-07-26T11:04:38.096 INFO:tasks.workunit.client.0.smithi144.stdout:6/198: symlink d8/d15/d26/l46 0 2024-07-26T11:04:38.097 INFO:tasks.workunit.client.0.smithi144.stdout:6/199: getdents d8/d42 0 2024-07-26T11:04:38.097 INFO:tasks.workunit.client.0.smithi144.stdout:6/200: stat d8 0 2024-07-26T11:04:38.113 INFO:tasks.workunit.client.0.smithi144.stdout:9/150: write d9/de/d14/f2c [2528399,129043] 0 2024-07-26T11:04:38.113 INFO:tasks.workunit.client.0.smithi144.stdout:9/151: write d9/f1a [796192,28981] 0 2024-07-26T11:04:38.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/152: unlink d9/f1a 0 2024-07-26T11:04:38.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/153: write d9/de/f20 [281439,17454] 0 2024-07-26T11:04:38.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/154: readlink d9/de/d14/l1b 0 2024-07-26T11:04:38.116 INFO:tasks.workunit.client.0.smithi144.stdout:6/201: dread d8/d15/d26/f28 [0,4194304] 0 2024-07-26T11:04:38.117 INFO:tasks.workunit.client.0.smithi144.stdout:6/202: readlink d8/l20 0 2024-07-26T11:04:38.117 INFO:tasks.workunit.client.0.smithi144.stdout:9/155: symlink d9/de/l33 0 2024-07-26T11:04:38.118 INFO:tasks.workunit.client.0.smithi144.stdout:9/156: chown d9/de/d14/d23/f31 14 1 2024-07-26T11:04:38.118 INFO:tasks.workunit.client.0.smithi144.stdout:9/157: dread - d9/f29 zero size 2024-07-26T11:04:38.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/203: creat d8/d42/f47 x:0 0 0 2024-07-26T11:04:38.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/204: dread - d8/f45 zero size 2024-07-26T11:04:38.119 INFO:tasks.workunit.client.0.smithi144.stdout:9/158: link d9/de/d14/d23/f31 d9/de/d14/f34 0 2024-07-26T11:04:38.121 INFO:tasks.workunit.client.0.smithi144.stdout:6/205: rename l7 to d8/d42/l48 0 2024-07-26T11:04:38.122 INFO:tasks.workunit.client.0.smithi144.stdout:9/159: creat d9/de/d14/f35 x:0 0 0 2024-07-26T11:04:38.122 INFO:tasks.workunit.client.0.smithi144.stdout:6/206: rmdir d8/d42 39 2024-07-26T11:04:38.123 INFO:tasks.workunit.client.0.smithi144.stdout:6/207: rename d8/d15/d26/l40 to d8/d42/l49 0 2024-07-26T11:04:38.124 INFO:tasks.workunit.client.0.smithi144.stdout:6/208: readlink d8/d15/l19 0 2024-07-26T11:04:38.125 INFO:tasks.workunit.client.0.smithi144.stdout:8/152: sync 2024-07-26T11:04:38.125 INFO:tasks.workunit.client.0.smithi144.stdout:8/153: chown d1a/d1c/f22 5 1 2024-07-26T11:04:38.125 INFO:tasks.workunit.client.0.smithi144.stdout:8/154: write d1a/d1c/f27 [686238,36754] 0 2024-07-26T11:04:38.126 INFO:tasks.workunit.client.0.smithi144.stdout:8/155: rmdir d1a/d1c/d2a 0 2024-07-26T11:04:38.127 INFO:tasks.workunit.client.0.smithi144.stdout:8/156: dread - d1a/d1c/d28/f2e zero size 2024-07-26T11:04:38.127 INFO:tasks.workunit.client.0.smithi144.stdout:8/157: chown fe 105071747 1 2024-07-26T11:04:38.127 INFO:tasks.workunit.client.0.smithi144.stdout:8/158: write d1a/d1c/d28/f2e [607973,66587] 0 2024-07-26T11:04:38.128 INFO:tasks.workunit.client.0.smithi144.stdout:8/159: creat d1a/f31 x:0 0 0 2024-07-26T11:04:38.149 INFO:tasks.workunit.client.0.smithi144.stdout:0/206: dwrite d0/d14/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:04:38.149 INFO:tasks.workunit.client.0.smithi144.stdout:9/160: write f4 [2445272,64391] 0 2024-07-26T11:04:38.151 INFO:tasks.workunit.client.0.smithi144.stdout:6/209: write f2 [708358,38626] 0 2024-07-26T11:04:38.155 INFO:tasks.workunit.client.0.smithi144.stdout:0/207: unlink d0/d14/c19 0 2024-07-26T11:04:38.155 INFO:tasks.workunit.client.0.smithi144.stdout:0/208: dread - d0/d14/f2c zero size 2024-07-26T11:04:38.155 INFO:tasks.workunit.client.0.smithi144.stdout:9/161: symlink d9/l36 0 2024-07-26T11:04:38.155 INFO:tasks.workunit.client.0.smithi144.stdout:9/162: fdatasync d9/de/d14/f2c 0 2024-07-26T11:04:38.158 INFO:tasks.workunit.client.0.smithi144.stdout:6/210: unlink d8/d15/d26/c3c 0 2024-07-26T11:04:38.158 INFO:tasks.workunit.client.0.smithi144.stdout:9/163: mknod d9/c37 0 2024-07-26T11:04:38.161 INFO:tasks.workunit.client.0.smithi144.stdout:6/211: link d8/db/c3a d8/db/c4a 0 2024-07-26T11:04:38.161 INFO:tasks.workunit.client.0.smithi144.stdout:6/212: fdatasync d8/f17 0 2024-07-26T11:04:38.163 INFO:tasks.workunit.client.0.smithi144.stdout:8/160: write d1a/d1c/f22 [1400456,49045] 0 2024-07-26T11:04:38.163 INFO:tasks.workunit.client.0.smithi144.stdout:6/213: mkdir d8/d15/d4b 0 2024-07-26T11:04:38.164 INFO:tasks.workunit.client.0.smithi144.stdout:8/161: mkdir d1a/d32 0 2024-07-26T11:04:38.164 INFO:tasks.workunit.client.0.smithi144.stdout:8/162: chown f8 1535153 1 2024-07-26T11:04:38.164 INFO:tasks.workunit.client.0.smithi144.stdout:8/163: write d1a/d1c/f1f [1004750,55609] 0 2024-07-26T11:04:38.168 INFO:tasks.workunit.client.0.smithi144.stdout:6/214: creat d8/d15/d26/f4c x:0 0 0 2024-07-26T11:04:38.168 INFO:tasks.workunit.client.0.smithi144.stdout:8/164: symlink d1a/d1c/d28/d2c/l33 0 2024-07-26T11:04:38.169 INFO:tasks.workunit.client.0.smithi144.stdout:6/215: mknod d8/db/c4d 0 2024-07-26T11:04:38.170 INFO:tasks.workunit.client.0.smithi144.stdout:6/216: write d8/d15/d26/f4c [43312,92928] 0 2024-07-26T11:04:38.171 INFO:tasks.workunit.client.0.smithi144.stdout:8/165: mknod d1a/d1c/d2f/c34 0 2024-07-26T11:04:38.173 INFO:tasks.workunit.client.0.smithi144.stdout:8/166: mknod d1a/d1c/d2f/c35 0 2024-07-26T11:04:38.173 INFO:tasks.workunit.client.0.smithi144.stdout:6/217: fsync d8/f10 0 2024-07-26T11:04:38.175 INFO:tasks.workunit.client.0.smithi144.stdout:8/167: creat d1a/d1c/d28/d2c/f36 x:0 0 0 2024-07-26T11:04:38.176 INFO:tasks.workunit.client.0.smithi144.stdout:6/218: symlink d8/d15/d26/l4e 0 2024-07-26T11:04:38.176 INFO:tasks.workunit.client.0.smithi144.stdout:9/164: write d9/fa [1295164,52554] 0 2024-07-26T11:04:38.177 INFO:tasks.workunit.client.0.smithi144.stdout:8/168: rename f8 to d1a/d1c/d30/f37 0 2024-07-26T11:04:38.182 INFO:tasks.workunit.client.0.smithi144.stdout:6/219: rename d8/d15/d4b to d8/d4f 0 2024-07-26T11:04:38.182 INFO:tasks.workunit.client.0.smithi144.stdout:6/220: readlink d8/d15/l24 0 2024-07-26T11:04:38.182 INFO:tasks.workunit.client.0.smithi144.stdout:8/169: mkdir d1a/d1c/d28/d2c/d38 0 2024-07-26T11:04:38.186 INFO:tasks.workunit.client.0.smithi144.stdout:6/221: getdents d8/db 0 2024-07-26T11:04:38.187 INFO:tasks.workunit.client.0.smithi144.stdout:6/222: read - d8/d15/f38 zero size 2024-07-26T11:04:38.187 INFO:tasks.workunit.client.0.smithi144.stdout:6/223: write d8/d15/f21 [723024,69692] 0 2024-07-26T11:04:38.187 INFO:tasks.workunit.client.0.smithi144.stdout:6/224: dread - d8/f1a zero size 2024-07-26T11:04:38.187 INFO:tasks.workunit.client.0.smithi144.stdout:6/225: write d8/db/f32 [554445,82627] 0 2024-07-26T11:04:38.190 INFO:tasks.workunit.client.0.smithi144.stdout:6/226: mkdir d8/d50 0 2024-07-26T11:04:38.198 INFO:tasks.workunit.client.0.smithi144.stdout:0/209: dwrite d0/f7 [4194304,4194304] 0 2024-07-26T11:04:38.199 INFO:tasks.workunit.client.0.smithi144.stdout:0/210: chown d0/d14/c1f 10846222 1 2024-07-26T11:04:38.199 INFO:tasks.workunit.client.0.smithi144.stdout:0/211: chown d0/d14/d29/f42 19769710 1 2024-07-26T11:04:38.219 INFO:tasks.workunit.client.0.smithi144.stdout:8/170: dwrite d1a/d1c/d30/f37 [0,4194304] 0 2024-07-26T11:04:38.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/171: truncate f4 1212049 0 2024-07-26T11:04:38.340 INFO:tasks.workunit.client.0.smithi144.stdout:4/125: sync 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:1/188: sync 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:7/157: sync 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:2/118: sync 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:3/120: sync 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:2/119: stat d0/cc 0 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:4/126: truncate d9/f22 522096 0 2024-07-26T11:04:38.341 INFO:tasks.workunit.client.0.smithi144.stdout:4/127: truncate d9/f12 638355 0 2024-07-26T11:04:38.342 INFO:tasks.workunit.client.0.smithi144.stdout:2/120: mkdir d0/d24 0 2024-07-26T11:04:38.342 INFO:tasks.workunit.client.0.smithi144.stdout:2/121: readlink d0/d2/l5 0 2024-07-26T11:04:38.343 INFO:tasks.workunit.client.0.smithi144.stdout:3/121: creat d1/f22 x:0 0 0 2024-07-26T11:04:38.344 INFO:tasks.workunit.client.0.smithi144.stdout:3/122: link d1/d5/l1f d1/l23 0 2024-07-26T11:04:38.344 INFO:tasks.workunit.client.0.smithi144.stdout:3/123: dread - d1/f22 zero size 2024-07-26T11:04:38.346 INFO:tasks.workunit.client.0.smithi144.stdout:3/124: truncate d1/d5/fa 2490596 0 2024-07-26T11:04:38.346 INFO:tasks.workunit.client.0.smithi144.stdout:3/125: write d1/ff [3013923,118427] 0 2024-07-26T11:04:38.387 INFO:tasks.workunit.client.0.smithi144.stdout:3/126: dread d1/f14 [0,4194304] 0 2024-07-26T11:04:38.387 INFO:tasks.workunit.client.0.smithi144.stdout:3/127: stat d1/d5/l13 0 2024-07-26T11:04:38.387 INFO:tasks.workunit.client.0.smithi144.stdout:3/128: chown d1/d5 1 1 2024-07-26T11:04:38.388 INFO:tasks.workunit.client.0.smithi144.stdout:3/129: write d1/ff [5102278,128833] 0 2024-07-26T11:04:38.390 INFO:tasks.workunit.client.0.smithi144.stdout:7/158: read d0/f2 [1193590,26716] 0 2024-07-26T11:04:38.394 INFO:tasks.workunit.client.0.smithi144.stdout:3/130: creat d1/d5/f24 x:0 0 0 2024-07-26T11:04:38.394 INFO:tasks.workunit.client.0.smithi144.stdout:7/159: creat d0/d6/d9/d25/f41 x:0 0 0 2024-07-26T11:04:38.396 INFO:tasks.workunit.client.0.smithi144.stdout:1/189: dread d3/f1e [0,4194304] 0 2024-07-26T11:04:38.396 INFO:tasks.workunit.client.0.smithi144.stdout:1/190: write d3/f10 [1591974,112215] 0 2024-07-26T11:04:38.401 INFO:tasks.workunit.client.0.smithi144.stdout:1/191: rmdir d3/d6/da/d13 39 2024-07-26T11:04:38.416 INFO:tasks.workunit.client.0.smithi144.stdout:3/131: write d1/f14 [126470,7775] 0 2024-07-26T11:04:38.418 INFO:tasks.workunit.client.0.smithi144.stdout:3/132: mkdir d1/d5/d25 0 2024-07-26T11:04:38.446 INFO:tasks.workunit.client.0.smithi144.stdout:7/160: dwrite d0/d6/d9/d25/f41 [0,4194304] 0 2024-07-26T11:04:38.459 INFO:tasks.workunit.client.0.smithi144.stdout:1/192: dwrite d3/d6/f11 [4194304,4194304] 0 2024-07-26T11:04:38.459 INFO:tasks.workunit.client.0.smithi144.stdout:1/193: getdents d3/d6/da/d13/d19/d27/d36/d37 0 2024-07-26T11:04:38.459 INFO:tasks.workunit.client.0.smithi144.stdout:1/194: fsync d3/d6/f23 0 2024-07-26T11:04:38.460 INFO:tasks.workunit.client.0.smithi144.stdout:7/161: dread d0/d6/f2b [0,4194304] 0 2024-07-26T11:04:38.464 INFO:tasks.workunit.client.0.smithi144.stdout:7/162: creat d0/d6/d9/d25/f42 x:0 0 0 2024-07-26T11:04:38.464 INFO:tasks.workunit.client.0.smithi144.stdout:7/163: write d0/d6/f17 [720646,34485] 0 2024-07-26T11:04:38.465 INFO:tasks.workunit.client.0.smithi144.stdout:7/164: symlink d0/d6/d9/d25/l43 0 2024-07-26T11:04:38.465 INFO:tasks.workunit.client.0.smithi144.stdout:7/165: chown d0/c4 6 1 2024-07-26T11:04:38.466 INFO:tasks.workunit.client.0.smithi144.stdout:5/212: sync 2024-07-26T11:04:38.466 INFO:tasks.workunit.client.0.smithi144.stdout:9/165: sync 2024-07-26T11:04:38.466 INFO:tasks.workunit.client.0.smithi144.stdout:5/213: fdatasync d1/d18/d24/d35/f3d 0 2024-07-26T11:04:38.466 INFO:tasks.workunit.client.0.smithi144.stdout:0/212: sync 2024-07-26T11:04:38.466 INFO:tasks.workunit.client.0.smithi144.stdout:9/166: write d9/fa [5651603,128888] 0 2024-07-26T11:04:38.467 INFO:tasks.workunit.client.0.smithi144.stdout:7/166: mknod d0/d6/d9/d3f/c44 0 2024-07-26T11:04:38.467 INFO:tasks.workunit.client.0.smithi144.stdout:0/213: mkdir d0/d3e/d45/d4c 0 2024-07-26T11:04:38.468 INFO:tasks.workunit.client.0.smithi144.stdout:6/227: sync 2024-07-26T11:04:38.468 INFO:tasks.workunit.client.0.smithi144.stdout:8/172: sync 2024-07-26T11:04:38.468 INFO:tasks.workunit.client.0.smithi144.stdout:1/195: dread d3/d6/da/d13/f26 [0,4194304] 0 2024-07-26T11:04:38.470 INFO:tasks.workunit.client.0.smithi144.stdout:9/167: mknod d9/c38 0 2024-07-26T11:04:38.471 INFO:tasks.workunit.client.0.smithi144.stdout:9/168: readlink d9/de/d14/d23/l28 0 2024-07-26T11:04:38.471 INFO:tasks.workunit.client.0.smithi144.stdout:7/167: rename d0/d10 to d0/d6/d9/d45 0 2024-07-26T11:04:38.472 INFO:tasks.workunit.client.0.smithi144.stdout:0/214: creat d0/d3e/d45/f4d x:0 0 0 2024-07-26T11:04:38.474 INFO:tasks.workunit.client.0.smithi144.stdout:1/196: creat d3/d6/da/d13/d22/f3c x:0 0 0 2024-07-26T11:04:38.474 INFO:tasks.workunit.client.0.smithi144.stdout:1/197: chown l2 67081 1 2024-07-26T11:04:38.474 INFO:tasks.workunit.client.0.smithi144.stdout:9/169: mkdir d9/de/d14/d23/d39 0 2024-07-26T11:04:38.501 INFO:tasks.workunit.client.0.smithi144.stdout:1/198: dread d3/d6/fd [0,4194304] 0 2024-07-26T11:04:38.501 INFO:tasks.workunit.client.0.smithi144.stdout:1/199: chown d3/c7 3460 1 2024-07-26T11:04:38.501 INFO:tasks.workunit.client.0.smithi144.stdout:1/200: readlink d3/d6/l38 0 2024-07-26T11:04:38.509 INFO:tasks.workunit.client.0.smithi144.stdout:9/170: write d9/de/ff [2125650,75741] 0 2024-07-26T11:04:38.511 INFO:tasks.workunit.client.0.smithi144.stdout:9/171: creat d9/de/d14/d23/f3a x:0 0 0 2024-07-26T11:04:38.512 INFO:tasks.workunit.client.0.smithi144.stdout:9/172: rmdir d9/de/d14 39 2024-07-26T11:04:38.514 INFO:tasks.workunit.client.0.smithi144.stdout:9/173: mkdir d9/d3b 0 2024-07-26T11:04:38.521 INFO:tasks.workunit.client.0.smithi144.stdout:4/128: sync 2024-07-26T11:04:38.521 INFO:tasks.workunit.client.0.smithi144.stdout:8/173: dwrite d1a/d1c/d28/d2c/f36 [0,4194304] 0 2024-07-26T11:04:38.528 INFO:tasks.workunit.client.0.smithi144.stdout:9/174: dread d9/fa [4194304,4194304] 0 2024-07-26T11:04:38.528 INFO:tasks.workunit.client.0.smithi144.stdout:9/175: chown d9/d3b 111909 1 2024-07-26T11:04:38.531 INFO:tasks.workunit.client.0.smithi144.stdout:0/215: write d0/f27 [332509,1720] 0 2024-07-26T11:04:38.532 INFO:tasks.workunit.client.0.smithi144.stdout:4/129: creat d9/da/f23 x:0 0 0 2024-07-26T11:04:38.532 INFO:tasks.workunit.client.0.smithi144.stdout:4/130: chown d9/fe 37094 1 2024-07-26T11:04:38.533 INFO:tasks.workunit.client.0.smithi144.stdout:1/201: dread d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:38.533 INFO:tasks.workunit.client.0.smithi144.stdout:1/202: chown l2 75422686 1 2024-07-26T11:04:38.535 INFO:tasks.workunit.client.0.smithi144.stdout:9/176: symlink d9/d3b/l3c 0 2024-07-26T11:04:38.537 INFO:tasks.workunit.client.0.smithi144.stdout:4/131: rename c8 to d9/c24 0 2024-07-26T11:04:38.538 INFO:tasks.workunit.client.0.smithi144.stdout:9/177: link d9/de/d14/f34 d9/de/f3d 0 2024-07-26T11:04:38.540 INFO:tasks.workunit.client.0.smithi144.stdout:4/132: rename d9/fb to d9/da/f25 0 2024-07-26T11:04:38.540 INFO:tasks.workunit.client.0.smithi144.stdout:9/178: mknod d9/de/d14/d23/c3e 0 2024-07-26T11:04:38.546 INFO:tasks.workunit.client.0.smithi144.stdout:9/179: mkdir d9/de/d14/d3f 0 2024-07-26T11:04:38.547 INFO:tasks.workunit.client.0.smithi144.stdout:1/203: dread f1 [0,4194304] 0 2024-07-26T11:04:38.549 INFO:tasks.workunit.client.0.smithi144.stdout:1/204: getdents d3/d6/da/d13/d19 0 2024-07-26T11:04:38.556 INFO:tasks.workunit.client.0.smithi144.stdout:9/180: write d9/fc [337415,101117] 0 2024-07-26T11:04:38.560 INFO:tasks.workunit.client.0.smithi144.stdout:9/181: symlink d9/de/d14/d3f/l40 0 2024-07-26T11:04:38.568 INFO:tasks.workunit.client.0.smithi144.stdout:6/228: dwrite d8/d15/d26/f41 [0,4194304] 0 2024-07-26T11:04:38.568 INFO:tasks.workunit.client.0.smithi144.stdout:6/229: read - d8/d15/d26/f2e zero size 2024-07-26T11:04:38.568 INFO:tasks.workunit.client.0.smithi144.stdout:6/230: truncate d8/d15/d26/f2e 738594 0 2024-07-26T11:04:38.569 INFO:tasks.workunit.client.0.smithi144.stdout:6/231: write d8/f2d [128380,79995] 0 2024-07-26T11:04:38.569 INFO:tasks.workunit.client.0.smithi144.stdout:6/232: write d8/d15/f35 [1047249,17871] 0 2024-07-26T11:04:38.571 INFO:tasks.workunit.client.0.smithi144.stdout:9/182: dread d9/de/f30 [0,4194304] 0 2024-07-26T11:04:38.571 INFO:tasks.workunit.client.0.smithi144.stdout:6/233: rmdir d8/d15/d26 39 2024-07-26T11:04:38.571 INFO:tasks.workunit.client.0.smithi144.stdout:6/234: fdatasync d8/fd 0 2024-07-26T11:04:38.573 INFO:tasks.workunit.client.0.smithi144.stdout:0/216: dread d0/f7 [4194304,4194304] 0 2024-07-26T11:04:38.574 INFO:tasks.workunit.client.0.smithi144.stdout:9/183: truncate d9/fa 2857362 0 2024-07-26T11:04:38.574 INFO:tasks.workunit.client.0.smithi144.stdout:6/235: creat d8/d4f/f51 x:0 0 0 2024-07-26T11:04:38.575 INFO:tasks.workunit.client.0.smithi144.stdout:0/217: truncate d0/d3e/f43 239189 0 2024-07-26T11:04:38.580 INFO:tasks.workunit.client.0.smithi144.stdout:9/184: truncate d9/de/d14/f2c 430066 0 2024-07-26T11:04:38.580 INFO:tasks.workunit.client.0.smithi144.stdout:6/236: mkdir d8/d4f/d52 0 2024-07-26T11:04:38.580 INFO:tasks.workunit.client.0.smithi144.stdout:0/218: symlink d0/d14/d1e/d30/l4e 0 2024-07-26T11:04:38.580 INFO:tasks.workunit.client.0.smithi144.stdout:9/185: creat d9/d3b/f41 x:0 0 0 2024-07-26T11:04:38.581 INFO:tasks.workunit.client.0.smithi144.stdout:2/122: sync 2024-07-26T11:04:38.581 INFO:tasks.workunit.client.0.smithi144.stdout:2/123: write d0/d6/f1d [4813599,15220] 0 2024-07-26T11:04:38.581 INFO:tasks.workunit.client.0.smithi144.stdout:0/219: symlink d0/d14/d1e/d30/d4b/l4f 0 2024-07-26T11:04:38.581 INFO:tasks.workunit.client.0.smithi144.stdout:9/186: mknod d9/d3b/c42 0 2024-07-26T11:04:38.582 INFO:tasks.workunit.client.0.smithi144.stdout:8/174: dwrite fe [0,4194304] 0 2024-07-26T11:04:38.584 INFO:tasks.workunit.client.0.smithi144.stdout:4/133: dwrite d9/da/f1a [0,4194304] 0 2024-07-26T11:04:38.590 INFO:tasks.workunit.client.0.smithi144.stdout:2/124: unlink d0/d6/d9/l1c 0 2024-07-26T11:04:38.590 INFO:tasks.workunit.client.0.smithi144.stdout:2/125: getdents d0/d24 0 2024-07-26T11:04:38.590 INFO:tasks.workunit.client.0.smithi144.stdout:0/220: creat d0/d3e/d45/f50 x:0 0 0 2024-07-26T11:04:38.596 INFO:tasks.workunit.client.0.smithi144.stdout:9/187: symlink d9/de/d14/d3f/l43 0 2024-07-26T11:04:38.597 INFO:tasks.workunit.client.0.smithi144.stdout:9/188: chown d9/de/d14/d23/d39 2089195484 1 2024-07-26T11:04:38.601 INFO:tasks.workunit.client.0.smithi144.stdout:4/134: mkdir d9/d26 0 2024-07-26T11:04:38.601 INFO:tasks.workunit.client.0.smithi144.stdout:1/205: dwrite d3/d6/f18 [4194304,4194304] 0 2024-07-26T11:04:38.601 INFO:tasks.workunit.client.0.smithi144.stdout:1/206: fsync d3/f3b 0 2024-07-26T11:04:38.601 INFO:tasks.workunit.client.0.smithi144.stdout:4/135: truncate d9/da/f25 4263789 0 2024-07-26T11:04:38.602 INFO:tasks.workunit.client.0.smithi144.stdout:2/126: link d0/f1e d0/d6/d9/f25 0 2024-07-26T11:04:38.603 INFO:tasks.workunit.client.0.smithi144.stdout:2/127: truncate d0/f22 954188 0 2024-07-26T11:04:38.603 INFO:tasks.workunit.client.0.smithi144.stdout:0/221: rmdir d0/d3e/d45 39 2024-07-26T11:04:38.603 INFO:tasks.workunit.client.0.smithi144.stdout:0/222: fdatasync d0/f32 0 2024-07-26T11:04:38.606 INFO:tasks.workunit.client.0.smithi144.stdout:1/207: mkdir d3/d3d 0 2024-07-26T11:04:38.606 INFO:tasks.workunit.client.0.smithi144.stdout:1/208: fdatasync d3/d6/da/d13/d22/f30 0 2024-07-26T11:04:38.607 INFO:tasks.workunit.client.0.smithi144.stdout:4/136: symlink d9/d26/l27 0 2024-07-26T11:04:38.607 INFO:tasks.workunit.client.0.smithi144.stdout:4/137: chown f1 19940 1 2024-07-26T11:04:38.608 INFO:tasks.workunit.client.0.smithi144.stdout:4/138: creat d9/d26/f28 x:0 0 0 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:4/139: fsync d9/f22 0 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:7/168: sync 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:3/133: sync 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:4/140: creat d9/da/f29 x:0 0 0 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:4/141: dread - d9/da/f29 zero size 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:5/214: sync 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:9/189: symlink d9/de/d14/d23/d39/l44 0 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:7/169: mknod d0/d6/d9/d45/c46 0 2024-07-26T11:04:38.612 INFO:tasks.workunit.client.0.smithi144.stdout:4/142: creat d9/d26/f2a x:0 0 0 2024-07-26T11:04:38.613 INFO:tasks.workunit.client.0.smithi144.stdout:7/170: truncate d0/f7 3322355 0 2024-07-26T11:04:38.614 INFO:tasks.workunit.client.0.smithi144.stdout:4/143: creat d9/d26/f2b x:0 0 0 2024-07-26T11:04:38.614 INFO:tasks.workunit.client.0.smithi144.stdout:4/144: chown d9/c20 8632568 1 2024-07-26T11:04:38.615 INFO:tasks.workunit.client.0.smithi144.stdout:4/145: mknod d9/c2c 0 2024-07-26T11:04:38.617 INFO:tasks.workunit.client.0.smithi144.stdout:4/146: rename d9/fe to d9/da/f2d 0 2024-07-26T11:04:38.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/147: symlink d9/l2e 0 2024-07-26T11:04:38.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/148: dread - d9/da/f29 zero size 2024-07-26T11:04:38.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/149: chown f1 3947320 1 2024-07-26T11:04:38.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/150: readlink l7 0 2024-07-26T11:04:38.620 INFO:tasks.workunit.client.0.smithi144.stdout:4/151: rmdir d9/da 39 2024-07-26T11:04:38.620 INFO:tasks.workunit.client.0.smithi144.stdout:4/152: chown d9/c20 31 1 2024-07-26T11:04:38.620 INFO:tasks.workunit.client.0.smithi144.stdout:4/153: fdatasync d9/da/f29 0 2024-07-26T11:04:38.621 INFO:tasks.workunit.client.0.smithi144.stdout:4/154: creat d9/d26/f2f x:0 0 0 2024-07-26T11:04:38.622 INFO:tasks.workunit.client.0.smithi144.stdout:4/155: creat d9/d26/f30 x:0 0 0 2024-07-26T11:04:38.623 INFO:tasks.workunit.client.0.smithi144.stdout:4/156: link d9/f12 d9/d26/f31 0 2024-07-26T11:04:38.623 INFO:tasks.workunit.client.0.smithi144.stdout:4/157: chown d9/da/c16 4955 1 2024-07-26T11:04:38.625 INFO:tasks.workunit.client.0.smithi144.stdout:4/158: creat d9/da/f32 x:0 0 0 2024-07-26T11:04:38.625 INFO:tasks.workunit.client.0.smithi144.stdout:4/159: dread - d9/f11 zero size 2024-07-26T11:04:38.641 INFO:tasks.workunit.client.0.smithi144.stdout:1/209: write d3/d6/f18 [2240671,63200] 0 2024-07-26T11:04:38.641 INFO:tasks.workunit.client.0.smithi144.stdout:6/237: dwrite d8/d15/d26/f3f [0,4194304] 0 2024-07-26T11:04:38.641 INFO:tasks.workunit.client.0.smithi144.stdout:7/171: write d0/f7 [971784,18530] 0 2024-07-26T11:04:38.647 INFO:tasks.workunit.client.0.smithi144.stdout:6/238: creat d8/f53 x:0 0 0 2024-07-26T11:04:38.647 INFO:tasks.workunit.client.0.smithi144.stdout:6/239: getdents d8/d50 0 2024-07-26T11:04:38.655 INFO:tasks.workunit.client.0.smithi144.stdout:2/128: read d0/f1 [2547039,1588] 0 2024-07-26T11:04:38.657 INFO:tasks.workunit.client.0.smithi144.stdout:2/129: unlink d0/c7 0 2024-07-26T11:04:38.659 INFO:tasks.workunit.client.0.smithi144.stdout:2/130: creat d0/d6/d9/f26 x:0 0 0 2024-07-26T11:04:38.662 INFO:tasks.workunit.client.0.smithi144.stdout:2/131: truncate d0/f22 591895 0 2024-07-26T11:04:38.664 INFO:tasks.workunit.client.0.smithi144.stdout:2/132: mkdir d0/d2/d27 0 2024-07-26T11:04:38.669 INFO:tasks.workunit.client.0.smithi144.stdout:7/172: write d0/d6/f2b [129653,87053] 0 2024-07-26T11:04:38.669 INFO:tasks.workunit.client.0.smithi144.stdout:9/190: dread d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:38.669 INFO:tasks.workunit.client.0.smithi144.stdout:9/191: chown d9/d3b/c42 5550 1 2024-07-26T11:04:38.669 INFO:tasks.workunit.client.0.smithi144.stdout:9/192: read - d9/de/f15 zero size 2024-07-26T11:04:38.670 INFO:tasks.workunit.client.0.smithi144.stdout:8/175: dwrite d1a/d1c/f2d [0,4194304] 0 2024-07-26T11:04:38.672 INFO:tasks.workunit.client.0.smithi144.stdout:3/134: dwrite d1/f22 [0,4194304] 0 2024-07-26T11:04:38.673 INFO:tasks.workunit.client.0.smithi144.stdout:7/173: mkdir d0/d6/d9/d25/d2f/d47 0 2024-07-26T11:04:38.673 INFO:tasks.workunit.client.0.smithi144.stdout:9/193: symlink d9/de/d14/d23/l45 0 2024-07-26T11:04:38.673 INFO:tasks.workunit.client.0.smithi144.stdout:2/133: rename d0/d2 to d0/d15/d20/d28 0 2024-07-26T11:04:38.676 INFO:tasks.workunit.client.0.smithi144.stdout:3/135: mkdir d1/d5/d25/d26 0 2024-07-26T11:04:38.678 INFO:tasks.workunit.client.0.smithi144.stdout:7/174: creat d0/d6/d9/d25/d2f/f48 x:0 0 0 2024-07-26T11:04:38.678 INFO:tasks.workunit.client.0.smithi144.stdout:9/194: creat d9/de/f46 x:0 0 0 2024-07-26T11:04:38.678 INFO:tasks.workunit.client.0.smithi144.stdout:9/195: write d9/d3b/f41 [602588,20596] 0 2024-07-26T11:04:38.678 INFO:tasks.workunit.client.0.smithi144.stdout:7/175: write d0/f7 [4266235,8282] 0 2024-07-26T11:04:38.679 INFO:tasks.workunit.client.0.smithi144.stdout:9/196: readlink d9/d3b/l3c 0 2024-07-26T11:04:38.681 INFO:tasks.workunit.client.0.smithi144.stdout:2/134: symlink d0/d15/l29 0 2024-07-26T11:04:38.692 INFO:tasks.workunit.client.0.smithi144.stdout:9/197: dread d9/fc [0,4194304] 0 2024-07-26T11:04:38.692 INFO:tasks.workunit.client.0.smithi144.stdout:9/198: readlink d9/de/d14/d23/d39/l44 0 2024-07-26T11:04:38.694 INFO:tasks.workunit.client.0.smithi144.stdout:9/199: link d9/de/d14/f22 d9/de/d14/d23/f47 0 2024-07-26T11:04:38.694 INFO:tasks.workunit.client.0.smithi144.stdout:9/200: write d9/fa [3226581,50411] 0 2024-07-26T11:04:38.694 INFO:tasks.workunit.client.0.smithi144.stdout:9/201: dread - d9/de/d14/f35 zero size 2024-07-26T11:04:38.705 INFO:tasks.workunit.client.0.smithi144.stdout:5/215: dwrite d1/d18/f3b [0,4194304] 0 2024-07-26T11:04:38.705 INFO:tasks.workunit.client.0.smithi144.stdout:1/210: dwrite d3/f1e [0,4194304] 0 2024-07-26T11:04:38.706 INFO:tasks.workunit.client.0.smithi144.stdout:2/135: dread d0/f1f [0,4194304] 0 2024-07-26T11:04:38.706 INFO:tasks.workunit.client.0.smithi144.stdout:2/136: fsync d0/d15/f17 0 2024-07-26T11:04:38.709 INFO:tasks.workunit.client.0.smithi144.stdout:2/137: unlink d0/d6/d9/f25 0 2024-07-26T11:04:38.715 INFO:tasks.workunit.client.0.smithi144.stdout:7/176: dread d0/f2 [0,4194304] 0 2024-07-26T11:04:38.715 INFO:tasks.workunit.client.0.smithi144.stdout:7/177: fsync d0/d6/f2b 0 2024-07-26T11:04:38.715 INFO:tasks.workunit.client.0.smithi144.stdout:7/178: fsync d0/d6/d9/d25/f42 0 2024-07-26T11:04:38.715 INFO:tasks.workunit.client.0.smithi144.stdout:7/179: truncate d0/d6/d9/d45/f30 300306 0 2024-07-26T11:04:38.726 INFO:tasks.workunit.client.0.smithi144.stdout:9/202: dread d9/fa [0,4194304] 0 2024-07-26T11:04:38.728 INFO:tasks.workunit.client.0.smithi144.stdout:9/203: mknod d9/de/d14/d23/d39/c48 0 2024-07-26T11:04:38.729 INFO:tasks.workunit.client.0.smithi144.stdout:9/204: symlink d9/l49 0 2024-07-26T11:04:38.729 INFO:tasks.workunit.client.0.smithi144.stdout:9/205: write d9/de/ff [1862719,113171] 0 2024-07-26T11:04:38.731 INFO:tasks.workunit.client.0.smithi144.stdout:9/206: unlink d9/c37 0 2024-07-26T11:04:38.734 INFO:tasks.workunit.client.0.smithi144.stdout:8/176: dread d1a/d1c/f22 [0,4194304] 0 2024-07-26T11:04:38.734 INFO:tasks.workunit.client.0.smithi144.stdout:1/211: write d3/f39 [10422426,100773] 0 2024-07-26T11:04:38.740 INFO:tasks.workunit.client.0.smithi144.stdout:9/207: getdents d9 0 2024-07-26T11:04:38.749 INFO:tasks.workunit.client.0.smithi144.stdout:5/216: dwrite d1/d18/f3b [0,4194304] 0 2024-07-26T11:04:38.750 INFO:tasks.workunit.client.0.smithi144.stdout:7/180: dread d0/d6/d9/d45/f22 [0,4194304] 0 2024-07-26T11:04:38.750 INFO:tasks.workunit.client.0.smithi144.stdout:7/181: chown d0/l5 134068362 1 2024-07-26T11:04:38.750 INFO:tasks.workunit.client.0.smithi144.stdout:7/182: chown d0/d6/d9/d12/c3c 37 1 2024-07-26T11:04:38.751 INFO:tasks.workunit.client.0.smithi144.stdout:9/208: rmdir d9/de 39 2024-07-26T11:04:38.751 INFO:tasks.workunit.client.0.smithi144.stdout:1/212: mkdir d3/d6/da/d13/d19/d3a/d3e 0 2024-07-26T11:04:38.754 INFO:tasks.workunit.client.0.smithi144.stdout:7/183: creat d0/f49 x:0 0 0 2024-07-26T11:04:38.754 INFO:tasks.workunit.client.0.smithi144.stdout:1/213: rmdir d3/d6 39 2024-07-26T11:04:38.757 INFO:tasks.workunit.client.0.smithi144.stdout:8/177: read f15 [3139845,1521] 0 2024-07-26T11:04:38.758 INFO:tasks.workunit.client.0.smithi144.stdout:7/184: creat d0/d6/d9/d25/d2f/d47/f4a x:0 0 0 2024-07-26T11:04:38.759 INFO:tasks.workunit.client.0.smithi144.stdout:4/160: sync 2024-07-26T11:04:38.760 INFO:tasks.workunit.client.0.smithi144.stdout:8/178: truncate f5 2235482 0 2024-07-26T11:04:38.761 INFO:tasks.workunit.client.0.smithi144.stdout:7/185: link d0/d6/d9/d45/f22 d0/f4b 0 2024-07-26T11:04:38.763 INFO:tasks.workunit.client.0.smithi144.stdout:4/161: creat d9/f33 x:0 0 0 2024-07-26T11:04:38.764 INFO:tasks.workunit.client.0.smithi144.stdout:7/186: creat d0/f4c x:0 0 0 2024-07-26T11:04:38.764 INFO:tasks.workunit.client.0.smithi144.stdout:7/187: chown d0/d6/d9/c1f 40734 1 2024-07-26T11:04:38.764 INFO:tasks.workunit.client.0.smithi144.stdout:7/188: write d0/f4c [729846,35888] 0 2024-07-26T11:04:38.764 INFO:tasks.workunit.client.0.smithi144.stdout:8/179: rmdir d1a/d1c/d2f 39 2024-07-26T11:04:38.765 INFO:tasks.workunit.client.0.smithi144.stdout:7/189: mknod d0/d6/d9/d25/c4d 0 2024-07-26T11:04:38.766 INFO:tasks.workunit.client.0.smithi144.stdout:8/180: link d1a/f20 d1a/d1c/d30/f39 0 2024-07-26T11:04:38.768 INFO:tasks.workunit.client.0.smithi144.stdout:8/181: read d1a/f20 [688666,89773] 0 2024-07-26T11:04:38.770 INFO:tasks.workunit.client.0.smithi144.stdout:8/182: mknod d1a/d1c/d2f/c3a 0 2024-07-26T11:04:38.770 INFO:tasks.workunit.client.0.smithi144.stdout:8/183: symlink d1a/d1c/l3b 0 2024-07-26T11:04:38.772 INFO:tasks.workunit.client.0.smithi144.stdout:8/184: unlink f5 0 2024-07-26T11:04:38.774 INFO:tasks.workunit.client.0.smithi144.stdout:8/185: truncate d1a/d1c/d30/f39 538077 0 2024-07-26T11:04:38.775 INFO:tasks.workunit.client.0.smithi144.stdout:8/186: mkdir d1a/d1c/d2f/d3c 0 2024-07-26T11:04:38.775 INFO:tasks.workunit.client.0.smithi144.stdout:8/187: truncate d1a/d1c/f27 1213330 0 2024-07-26T11:04:38.776 INFO:tasks.workunit.client.0.smithi144.stdout:8/188: symlink d1a/d1c/d28/d2c/l3d 0 2024-07-26T11:04:38.776 INFO:tasks.workunit.client.0.smithi144.stdout:8/189: write d1a/d1c/d28/f2e [1473489,79922] 0 2024-07-26T11:04:38.779 INFO:tasks.workunit.client.0.smithi144.stdout:8/190: mknod d1a/d1c/d2f/c3e 0 2024-07-26T11:04:38.779 INFO:tasks.workunit.client.0.smithi144.stdout:8/191: dread - d1a/f2b zero size 2024-07-26T11:04:38.779 INFO:tasks.workunit.client.0.smithi144.stdout:8/192: write d1a/f2b [562190,33869] 0 2024-07-26T11:04:38.795 INFO:tasks.workunit.client.0.smithi144.stdout:8/193: read d1a/d1c/d28/f2e [268870,60484] 0 2024-07-26T11:04:38.797 INFO:tasks.workunit.client.0.smithi144.stdout:8/194: creat d1a/d1c/d28/f3f x:0 0 0 2024-07-26T11:04:38.798 INFO:tasks.workunit.client.0.smithi144.stdout:5/217: dwrite d1/d18/f3b [0,4194304] 0 2024-07-26T11:04:38.798 INFO:tasks.workunit.client.0.smithi144.stdout:8/195: creat d1a/d1c/d30/f40 x:0 0 0 2024-07-26T11:04:38.799 INFO:tasks.workunit.client.0.smithi144.stdout:8/196: creat d1a/d1c/f41 x:0 0 0 2024-07-26T11:04:38.799 INFO:tasks.workunit.client.0.smithi144.stdout:5/218: mknod d1/d14/c4b 0 2024-07-26T11:04:38.800 INFO:tasks.workunit.client.0.smithi144.stdout:8/197: mknod d1a/d32/c42 0 2024-07-26T11:04:38.801 INFO:tasks.workunit.client.0.smithi144.stdout:5/219: symlink d1/d18/d24/d27/l4c 0 2024-07-26T11:04:38.801 INFO:tasks.workunit.client.0.smithi144.stdout:5/220: readlink d1/d5/l22 0 2024-07-26T11:04:38.802 INFO:tasks.workunit.client.0.smithi144.stdout:8/198: mkdir d1a/d1c/d30/d43 0 2024-07-26T11:04:38.803 INFO:tasks.workunit.client.0.smithi144.stdout:9/209: dwrite d9/fa [0,4194304] 0 2024-07-26T11:04:38.803 INFO:tasks.workunit.client.0.smithi144.stdout:9/210: write f4 [77379,63985] 0 2024-07-26T11:04:38.804 INFO:tasks.workunit.client.0.smithi144.stdout:9/211: write d9/de/d14/f22 [2184571,62061] 0 2024-07-26T11:04:38.804 INFO:tasks.workunit.client.0.smithi144.stdout:9/212: write d9/de/f21 [572693,49089] 0 2024-07-26T11:04:38.804 INFO:tasks.workunit.client.0.smithi144.stdout:8/199: mknod d1a/d1c/d30/c44 0 2024-07-26T11:04:38.811 INFO:tasks.workunit.client.0.smithi144.stdout:9/213: fsync d9/de/d14/f22 0 2024-07-26T11:04:38.811 INFO:tasks.workunit.client.0.smithi144.stdout:9/214: chown d9/de/d14/d23/f31 2 1 2024-07-26T11:04:38.814 INFO:tasks.workunit.client.0.smithi144.stdout:9/215: link d9/de/f20 d9/de/d14/f4a 0 2024-07-26T11:04:38.815 INFO:tasks.workunit.client.0.smithi144.stdout:5/221: rename d1/d14 to d1/d1b/d4d 0 2024-07-26T11:04:38.816 INFO:tasks.workunit.client.0.smithi144.stdout:9/216: unlink d9/de/d14/f34 0 2024-07-26T11:04:38.818 INFO:tasks.workunit.client.0.smithi144.stdout:5/222: link d1/d18/f3b d1/d9/d44/f4e 0 2024-07-26T11:04:38.818 INFO:tasks.workunit.client.0.smithi144.stdout:5/223: truncate d1/d5/f2e 849002 0 2024-07-26T11:04:38.818 INFO:tasks.workunit.client.0.smithi144.stdout:5/224: chown d1/d9/l33 62883 1 2024-07-26T11:04:38.818 INFO:tasks.workunit.client.0.smithi144.stdout:5/225: fdatasync d1/d5/f49 0 2024-07-26T11:04:38.819 INFO:tasks.workunit.client.0.smithi144.stdout:9/217: truncate d9/de/d14/d23/f31 4332952 0 2024-07-26T11:04:38.819 INFO:tasks.workunit.client.0.smithi144.stdout:9/218: chown d9/de/d14/d23/c3e 0 1 2024-07-26T11:04:38.820 INFO:tasks.workunit.client.0.smithi144.stdout:5/226: creat d1/d18/d24/d27/f4f x:0 0 0 2024-07-26T11:04:38.821 INFO:tasks.workunit.client.0.smithi144.stdout:7/190: dwrite d0/f2 [0,4194304] 0 2024-07-26T11:04:38.821 INFO:tasks.workunit.client.0.smithi144.stdout:7/191: chown d0/d6/d9/d45/c34 16219765 1 2024-07-26T11:04:38.821 INFO:tasks.workunit.client.0.smithi144.stdout:7/192: chown d0/fd 37481 1 2024-07-26T11:04:38.822 INFO:tasks.workunit.client.0.smithi144.stdout:5/227: getdents d1/d1b/d4d 0 2024-07-26T11:04:38.825 INFO:tasks.workunit.client.0.smithi144.stdout:5/228: mknod d1/d1b/c50 0 2024-07-26T11:04:38.829 INFO:tasks.workunit.client.0.smithi144.stdout:5/229: creat d1/d18/f51 x:0 0 0 2024-07-26T11:04:38.829 INFO:tasks.workunit.client.0.smithi144.stdout:8/200: dread d1a/f21 [0,4194304] 0 2024-07-26T11:04:38.830 INFO:tasks.workunit.client.0.smithi144.stdout:8/201: chown d1a/d1c/d28/d2c 219703 1 2024-07-26T11:04:38.831 INFO:tasks.workunit.client.0.smithi144.stdout:5/230: rename d1/d9/c1f to d1/d18/d24/d27/c52 0 2024-07-26T11:04:38.833 INFO:tasks.workunit.client.0.smithi144.stdout:9/219: truncate d9/fa 1918331 0 2024-07-26T11:04:38.835 INFO:tasks.workunit.client.0.smithi144.stdout:5/231: rmdir d1/d18/d24 39 2024-07-26T11:04:38.837 INFO:tasks.workunit.client.0.smithi144.stdout:9/220: rename d9/d3b/f41 to d9/de/f4b 0 2024-07-26T11:04:38.837 INFO:tasks.workunit.client.0.smithi144.stdout:9/221: dread - d9/de/d14/d23/f3a zero size 2024-07-26T11:04:38.838 INFO:tasks.workunit.client.0.smithi144.stdout:5/232: mknod d1/d5/c53 0 2024-07-26T11:04:38.843 INFO:tasks.workunit.client.0.smithi144.stdout:5/233: mknod d1/d1b/c54 0 2024-07-26T11:04:38.843 INFO:tasks.workunit.client.0.smithi144.stdout:9/222: getdents d9/d3b 0 2024-07-26T11:04:38.848 INFO:tasks.workunit.client.0.smithi144.stdout:8/202: write d1a/f21 [3505580,45124] 0 2024-07-26T11:04:38.859 INFO:tasks.workunit.client.0.smithi144.stdout:8/203: mknod d1a/d1c/d2f/d3c/c45 0 2024-07-26T11:04:38.860 INFO:tasks.workunit.client.0.smithi144.stdout:8/204: symlink d1a/d1c/l46 0 2024-07-26T11:04:38.862 INFO:tasks.workunit.client.0.smithi144.stdout:8/205: rename d1a/d1c/d2f/c34 to d1a/d1c/d30/d43/c47 0 2024-07-26T11:04:38.862 INFO:tasks.workunit.client.0.smithi144.stdout:8/206: chown l13 4618924 1 2024-07-26T11:04:38.863 INFO:tasks.workunit.client.0.smithi144.stdout:8/207: link d1a/d1c/d28/d2c/f36 d1a/d32/f48 0 2024-07-26T11:04:38.864 INFO:tasks.workunit.client.0.smithi144.stdout:8/208: stat l13 0 2024-07-26T11:04:38.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/234: dwrite d1/d1b/f21 [0,4194304] 0 2024-07-26T11:04:38.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/235: chown d1/l26 252121894 1 2024-07-26T11:04:38.893 INFO:tasks.workunit.client.0.smithi144.stdout:8/209: read fe [2287652,20713] 0 2024-07-26T11:04:38.896 INFO:tasks.workunit.client.0.smithi144.stdout:9/223: dwrite d9/de/d14/f4a [0,4194304] 0 2024-07-26T11:04:38.898 INFO:tasks.workunit.client.0.smithi144.stdout:9/224: symlink d9/de/l4c 0 2024-07-26T11:04:38.899 INFO:tasks.workunit.client.0.smithi144.stdout:9/225: creat d9/de/f4d x:0 0 0 2024-07-26T11:04:38.900 INFO:tasks.workunit.client.0.smithi144.stdout:9/226: chown d9/de/d14/d23/c2b 312651027 1 2024-07-26T11:04:38.900 INFO:tasks.workunit.client.0.smithi144.stdout:9/227: chown d9/de/d14/d23/c2b 5320 1 2024-07-26T11:04:38.901 INFO:tasks.workunit.client.0.smithi144.stdout:9/228: mkdir d9/de/d4e 0 2024-07-26T11:04:38.902 INFO:tasks.workunit.client.0.smithi144.stdout:5/236: read d1/d9/d44/f4e [1017054,86889] 0 2024-07-26T11:04:38.931 INFO:tasks.workunit.client.0.smithi144.stdout:8/210: dwrite d1a/d1c/d28/d2c/f36 [0,4194304] 0 2024-07-26T11:04:38.931 INFO:tasks.workunit.client.0.smithi144.stdout:8/211: write d1a/d1c/d30/f40 [821082,32167] 0 2024-07-26T11:04:38.931 INFO:tasks.workunit.client.0.smithi144.stdout:8/212: chown f6 89 1 2024-07-26T11:04:38.933 INFO:tasks.workunit.client.0.smithi144.stdout:8/213: mkdir d1a/d32/d49 0 2024-07-26T11:04:38.934 INFO:tasks.workunit.client.0.smithi144.stdout:8/214: creat d1a/d32/d49/f4a x:0 0 0 2024-07-26T11:04:38.943 INFO:tasks.workunit.client.0.smithi144.stdout:5/237: dwrite d1/d9/f10 [4194304,4194304] 0 2024-07-26T11:04:38.943 INFO:tasks.workunit.client.0.smithi144.stdout:5/238: dread - d1/d18/d24/d35/f3d zero size 2024-07-26T11:04:38.943 INFO:tasks.workunit.client.0.smithi144.stdout:8/215: truncate d1a/d32/f48 576454 0 2024-07-26T11:04:38.947 INFO:tasks.workunit.client.0.smithi144.stdout:0/223: sync 2024-07-26T11:04:38.960 INFO:tasks.workunit.client.0.smithi144.stdout:5/239: dread d1/d9/d44/f4e [0,4194304] 0 2024-07-26T11:04:38.961 INFO:tasks.workunit.client.0.smithi144.stdout:5/240: readlink d1/l26 0 2024-07-26T11:04:38.975 INFO:tasks.workunit.client.0.smithi144.stdout:0/224: unlink d0/d3e/d45/f4d 0 2024-07-26T11:04:38.975 INFO:tasks.workunit.client.0.smithi144.stdout:0/225: truncate d0/f8 383333 0 2024-07-26T11:04:38.976 INFO:tasks.workunit.client.0.smithi144.stdout:5/241: symlink d1/l55 0 2024-07-26T11:04:38.977 INFO:tasks.workunit.client.0.smithi144.stdout:8/216: dwrite d1a/d1c/f24 [0,4194304] 0 2024-07-26T11:04:38.978 INFO:tasks.workunit.client.0.smithi144.stdout:0/226: rename d0/d25/f3d to d0/d14/f51 0 2024-07-26T11:04:38.978 INFO:tasks.workunit.client.0.smithi144.stdout:5/242: creat d1/d9/d44/d45/f56 x:0 0 0 2024-07-26T11:04:38.979 INFO:tasks.workunit.client.0.smithi144.stdout:8/217: mknod d1a/d1c/d30/d43/c4b 0 2024-07-26T11:04:38.980 INFO:tasks.workunit.client.0.smithi144.stdout:0/227: mknod d0/d3e/d45/c52 0 2024-07-26T11:04:38.981 INFO:tasks.workunit.client.0.smithi144.stdout:5/243: creat d1/d9/d44/d45/f57 x:0 0 0 2024-07-26T11:04:38.983 INFO:tasks.workunit.client.0.smithi144.stdout:5/244: truncate d1/d18/d24/d27/f2f 2250729 0 2024-07-26T11:04:38.984 INFO:tasks.workunit.client.0.smithi144.stdout:5/245: symlink d1/d18/d24/l58 0 2024-07-26T11:04:38.989 INFO:tasks.workunit.client.0.smithi144.stdout:8/218: dread d1a/d1c/d28/d2c/f36 [0,4194304] 0 2024-07-26T11:04:38.992 INFO:tasks.workunit.client.0.smithi144.stdout:8/219: mkdir d1a/d1c/d30/d43/d4c 0 2024-07-26T11:04:38.999 INFO:tasks.workunit.client.0.smithi144.stdout:3/136: sync 2024-07-26T11:04:38.999 INFO:tasks.workunit.client.0.smithi144.stdout:6/240: sync 2024-07-26T11:04:39.002 INFO:tasks.workunit.client.0.smithi144.stdout:0/228: write d0/d14/d1e/f3c [1506571,123833] 0 2024-07-26T11:04:39.007 INFO:tasks.workunit.client.0.smithi144.stdout:5/246: write d1/d9/f10 [863152,23528] 0 2024-07-26T11:04:39.007 INFO:tasks.workunit.client.0.smithi144.stdout:3/137: creat d1/d5/d25/d26/f27 x:0 0 0 2024-07-26T11:04:39.007 INFO:tasks.workunit.client.0.smithi144.stdout:6/241: symlink d8/d15/l54 0 2024-07-26T11:04:39.008 INFO:tasks.workunit.client.0.smithi144.stdout:0/229: rename d0/d3e/d45/c52 to d0/d3e/d45/c53 0 2024-07-26T11:04:39.008 INFO:tasks.workunit.client.0.smithi144.stdout:0/230: write d0/d3e/d45/f4a [432843,46922] 0 2024-07-26T11:04:39.009 INFO:tasks.workunit.client.0.smithi144.stdout:5/247: mknod d1/d1b/d4d/c59 0 2024-07-26T11:04:39.009 INFO:tasks.workunit.client.0.smithi144.stdout:5/248: dread - d1/d18/d24/d27/f4f zero size 2024-07-26T11:04:39.009 INFO:tasks.workunit.client.0.smithi144.stdout:0/231: symlink d0/l54 0 2024-07-26T11:04:39.011 INFO:tasks.workunit.client.0.smithi144.stdout:0/232: creat d0/d3e/d45/f55 x:0 0 0 2024-07-26T11:04:39.012 INFO:tasks.workunit.client.0.smithi144.stdout:0/233: creat d0/d14/d29/f56 x:0 0 0 2024-07-26T11:04:39.012 INFO:tasks.workunit.client.0.smithi144.stdout:0/234: read - d0/d3e/d45/f46 zero size 2024-07-26T11:04:39.013 INFO:tasks.workunit.client.0.smithi144.stdout:0/235: mknod d0/d3e/d45/c57 0 2024-07-26T11:04:39.014 INFO:tasks.workunit.client.0.smithi144.stdout:0/236: creat d0/d14/d1e/d30/d4b/f58 x:0 0 0 2024-07-26T11:04:39.018 INFO:tasks.workunit.client.0.smithi144.stdout:0/237: mknod d0/d14/d1e/d30/d4b/c59 0 2024-07-26T11:04:39.036 INFO:tasks.workunit.client.0.smithi144.stdout:3/138: dread d1/ff [0,4194304] 0 2024-07-26T11:04:39.050 INFO:tasks.workunit.client.0.smithi144.stdout:0/238: read d0/f10 [2135034,38236] 0 2024-07-26T11:04:39.052 INFO:tasks.workunit.client.0.smithi144.stdout:0/239: mknod d0/d14/d1e/d30/c5a 0 2024-07-26T11:04:39.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/240: mknod d0/d25/c5b 0 2024-07-26T11:04:39.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/241: chown d0/d14/d1e/f24 11 1 2024-07-26T11:04:39.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/242: write d0/d14/d1e/f3c [4751908,24305] 0 2024-07-26T11:04:39.054 INFO:tasks.workunit.client.0.smithi144.stdout:0/243: mkdir d0/d14/d1e/d5c 0 2024-07-26T11:04:39.055 INFO:tasks.workunit.client.0.smithi144.stdout:6/242: write d8/db/f1b [4055752,46548] 0 2024-07-26T11:04:39.057 INFO:tasks.workunit.client.0.smithi144.stdout:6/243: creat d8/db/f55 x:0 0 0 2024-07-26T11:04:39.058 INFO:tasks.workunit.client.0.smithi144.stdout:6/244: creat d8/f56 x:0 0 0 2024-07-26T11:04:39.059 INFO:tasks.workunit.client.0.smithi144.stdout:6/245: mkdir d8/d15/d57 0 2024-07-26T11:04:39.070 INFO:tasks.workunit.client.0.smithi144.stdout:5/249: dwrite d1/d18/d24/d27/f4f [0,4194304] 0 2024-07-26T11:04:39.085 INFO:tasks.workunit.client.0.smithi144.stdout:0/244: dread d0/f18 [0,4194304] 0 2024-07-26T11:04:39.087 INFO:tasks.workunit.client.0.smithi144.stdout:3/139: dwrite d1/f1a [0,4194304] 0 2024-07-26T11:04:39.087 INFO:tasks.workunit.client.0.smithi144.stdout:3/140: dread - d1/f1b zero size 2024-07-26T11:04:39.087 INFO:tasks.workunit.client.0.smithi144.stdout:3/141: read - d1/d5/d25/d26/f27 zero size 2024-07-26T11:04:39.087 INFO:tasks.workunit.client.0.smithi144.stdout:5/250: dread d1/d18/d24/d27/f2f [0,4194304] 0 2024-07-26T11:04:39.089 INFO:tasks.workunit.client.0.smithi144.stdout:3/142: mkdir d1/d28 0 2024-07-26T11:04:39.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/251: creat d1/d9/f5a x:0 0 0 2024-07-26T11:04:39.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/252: readlink d1/l26 0 2024-07-26T11:04:39.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/253: dread - d1/d9/d44/d45/f56 zero size 2024-07-26T11:04:39.090 INFO:tasks.workunit.client.0.smithi144.stdout:5/254: write d1/d9/d44/d45/f56 [494091,12910] 0 2024-07-26T11:04:39.101 INFO:tasks.workunit.client.0.smithi144.stdout:5/255: rename d1/d1b to d1/d9/d5b 0 2024-07-26T11:04:39.104 INFO:tasks.workunit.client.0.smithi144.stdout:5/256: creat d1/d5/f5c x:0 0 0 2024-07-26T11:04:39.106 INFO:tasks.workunit.client.0.smithi144.stdout:5/257: symlink d1/d9/d5b/l5d 0 2024-07-26T11:04:39.107 INFO:tasks.workunit.client.0.smithi144.stdout:4/162: sync 2024-07-26T11:04:39.108 INFO:tasks.workunit.client.0.smithi144.stdout:4/163: dread - d9/d26/f30 zero size 2024-07-26T11:04:39.112 INFO:tasks.workunit.client.0.smithi144.stdout:3/143: dread d1/f17 [0,4194304] 0 2024-07-26T11:04:39.114 INFO:tasks.workunit.client.0.smithi144.stdout:3/144: unlink d1/d5/l13 0 2024-07-26T11:04:39.127 INFO:tasks.workunit.client.0.smithi144.stdout:0/245: dwrite d0/d3e/f43 [0,4194304] 0 2024-07-26T11:04:39.127 INFO:tasks.workunit.client.0.smithi144.stdout:0/246: stat d0/f28 0 2024-07-26T11:04:39.128 INFO:tasks.workunit.client.0.smithi144.stdout:4/164: read f3 [2831896,76052] 0 2024-07-26T11:04:39.128 INFO:tasks.workunit.client.0.smithi144.stdout:0/247: mknod d0/d3e/d45/d4c/c5d 0 2024-07-26T11:04:39.130 INFO:tasks.workunit.client.0.smithi144.stdout:5/258: write d1/d9/f10 [6296939,105799] 0 2024-07-26T11:04:39.130 INFO:tasks.workunit.client.0.smithi144.stdout:5/259: readlink d1/d9/d5b/d4d/l25 0 2024-07-26T11:04:39.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/165: creat d9/f34 x:0 0 0 2024-07-26T11:04:39.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/166: readlink d9/d26/l27 0 2024-07-26T11:04:39.132 INFO:tasks.workunit.client.0.smithi144.stdout:0/248: rename d0/d14 to d0/d25/d5e 0 2024-07-26T11:04:39.134 INFO:tasks.workunit.client.0.smithi144.stdout:5/260: creat d1/f5e x:0 0 0 2024-07-26T11:04:39.134 INFO:tasks.workunit.client.0.smithi144.stdout:5/261: read d1/d9/d5b/d4d/f16 [2331035,111500] 0 2024-07-26T11:04:39.134 INFO:tasks.workunit.client.0.smithi144.stdout:3/145: dread d1/ff [4194304,4194304] 0 2024-07-26T11:04:39.134 INFO:tasks.workunit.client.0.smithi144.stdout:5/262: fsync d1/d18/f48 0 2024-07-26T11:04:39.134 INFO:tasks.workunit.client.0.smithi144.stdout:3/146: dread - d1/f20 zero size 2024-07-26T11:04:39.135 INFO:tasks.workunit.client.0.smithi144.stdout:5/263: dread - d1/d9/d44/d45/f57 zero size 2024-07-26T11:04:39.135 INFO:tasks.workunit.client.0.smithi144.stdout:3/147: write d1/f17 [4458851,108908] 0 2024-07-26T11:04:39.135 INFO:tasks.workunit.client.0.smithi144.stdout:0/249: mkdir d0/d3e/d5f 0 2024-07-26T11:04:39.135 INFO:tasks.workunit.client.0.smithi144.stdout:0/250: dread - d0/d25/d5e/d29/f2d zero size 2024-07-26T11:04:39.137 INFO:tasks.workunit.client.0.smithi144.stdout:5/264: rename d1/d9/l42 to d1/d18/d24/l5f 0 2024-07-26T11:04:39.137 INFO:tasks.workunit.client.0.smithi144.stdout:5/265: fdatasync d1/d18/d24/d27/f4f 0 2024-07-26T11:04:39.140 INFO:tasks.workunit.client.0.smithi144.stdout:5/266: getdents d1/d18/d24/d35 0 2024-07-26T11:04:39.141 INFO:tasks.workunit.client.0.smithi144.stdout:2/138: sync 2024-07-26T11:04:39.141 INFO:tasks.workunit.client.0.smithi144.stdout:2/139: chown d0/d15 28 1 2024-07-26T11:04:39.147 INFO:tasks.workunit.client.0.smithi144.stdout:1/214: sync 2024-07-26T11:04:39.148 INFO:tasks.workunit.client.0.smithi144.stdout:7/193: sync 2024-07-26T11:04:39.148 INFO:tasks.workunit.client.0.smithi144.stdout:8/220: sync 2024-07-26T11:04:39.148 INFO:tasks.workunit.client.0.smithi144.stdout:9/229: sync 2024-07-26T11:04:39.148 INFO:tasks.workunit.client.0.smithi144.stdout:9/230: chown d9/de/d14/d23/d39/c48 68 1 2024-07-26T11:04:39.149 INFO:tasks.workunit.client.0.smithi144.stdout:5/267: creat d1/d18/f60 x:0 0 0 2024-07-26T11:04:39.150 INFO:tasks.workunit.client.0.smithi144.stdout:2/140: symlink d0/d15/d20/l2a 0 2024-07-26T11:04:39.151 INFO:tasks.workunit.client.0.smithi144.stdout:7/194: mknod d0/c4e 0 2024-07-26T11:04:39.151 INFO:tasks.workunit.client.0.smithi144.stdout:7/195: fsync d0/d6/f2b 0 2024-07-26T11:04:39.152 INFO:tasks.workunit.client.0.smithi144.stdout:4/167: read d9/f1f [553888,46713] 0 2024-07-26T11:04:39.152 INFO:tasks.workunit.client.0.smithi144.stdout:4/168: dread - d9/f34 zero size 2024-07-26T11:04:39.152 INFO:tasks.workunit.client.0.smithi144.stdout:4/169: stat d9/lf 0 2024-07-26T11:04:39.152 INFO:tasks.workunit.client.0.smithi144.stdout:1/215: symlink d3/d6/da/d13/d19/d3a/l3f 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:9/231: symlink d9/de/d14/d23/l4f 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:5/268: mknod d1/d18/d24/d35/c61 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:8/221: creat d1a/d1c/d28/d2c/f4d x:0 0 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:2/141: mkdir d0/d24/d2b 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:4/170: creat d9/da/f35 x:0 0 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:7/196: creat d0/d6/d9/d25/d2f/d47/f4f x:0 0 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:9/232: write d9/de/f4b [1422945,131030] 0 2024-07-26T11:04:39.165 INFO:tasks.workunit.client.0.smithi144.stdout:9/233: readlink d9/de/d14/l1b 0 2024-07-26T11:04:39.166 INFO:tasks.workunit.client.0.smithi144.stdout:1/216: stat d3/d6/da/d21/l2b 0 2024-07-26T11:04:39.166 INFO:tasks.workunit.client.0.smithi144.stdout:1/217: fdatasync d3/d6/da/d13/d19/d27/f28 0 2024-07-26T11:04:39.166 INFO:tasks.workunit.client.0.smithi144.stdout:0/251: dread d0/f27 [0,4194304] 0 2024-07-26T11:04:39.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/269: rename d1/d9/d44/d45/f57 to d1/d9/d44/f62 0 2024-07-26T11:04:39.170 INFO:tasks.workunit.client.0.smithi144.stdout:2/142: symlink d0/d15/d20/l2c 0 2024-07-26T11:04:39.170 INFO:tasks.workunit.client.0.smithi144.stdout:4/171: symlink d9/d26/l36 0 2024-07-26T11:04:39.170 INFO:tasks.workunit.client.0.smithi144.stdout:2/143: chown d0/d15/d20/d28/f4 1 1 2024-07-26T11:04:39.170 INFO:tasks.workunit.client.0.smithi144.stdout:2/144: chown d0 513869 1 2024-07-26T11:04:39.170 INFO:tasks.workunit.client.0.smithi144.stdout:2/145: truncate d0/d6/d9/f26 771508 0 2024-07-26T11:04:39.170 INFO:tasks.workunit.client.0.smithi144.stdout:1/218: creat d3/d6/da/d13/d22/f40 x:0 0 0 2024-07-26T11:04:39.173 INFO:tasks.workunit.client.0.smithi144.stdout:0/252: symlink d0/d25/d5e/d1e/l60 0 2024-07-26T11:04:39.174 INFO:tasks.workunit.client.0.smithi144.stdout:4/172: mknod d9/c37 0 2024-07-26T11:04:39.175 INFO:tasks.workunit.client.0.smithi144.stdout:4/173: write d9/da/f32 [804449,93274] 0 2024-07-26T11:04:39.175 INFO:tasks.workunit.client.0.smithi144.stdout:4/174: stat d9/da/f35 0 2024-07-26T11:04:39.176 INFO:tasks.workunit.client.0.smithi144.stdout:7/197: rename d0/d6/d9/d25/d2f/l38 to d0/d6/d9/l50 0 2024-07-26T11:04:39.178 INFO:tasks.workunit.client.0.smithi144.stdout:4/175: write f5 [4400736,110805] 0 2024-07-26T11:04:39.179 INFO:tasks.workunit.client.0.smithi144.stdout:4/176: fsync d9/d26/f2f 0 2024-07-26T11:04:39.180 INFO:tasks.workunit.client.0.smithi144.stdout:2/146: rename d0/d15/d20/l2c to d0/d15/d20/d28/l2d 0 2024-07-26T11:04:39.181 INFO:tasks.workunit.client.0.smithi144.stdout:7/198: mknod d0/d6/d9/d3f/c51 0 2024-07-26T11:04:39.183 INFO:tasks.workunit.client.0.smithi144.stdout:3/148: dwrite d1/d5/d25/d26/f27 [0,4194304] 0 2024-07-26T11:04:39.183 INFO:tasks.workunit.client.0.smithi144.stdout:3/149: fdatasync d1/d5/d25/d26/f27 0 2024-07-26T11:04:39.183 INFO:tasks.workunit.client.0.smithi144.stdout:1/219: write d3/f10 [287655,34500] 0 2024-07-26T11:04:39.184 INFO:tasks.workunit.client.0.smithi144.stdout:2/147: mkdir d0/d15/d20/d28/d2e 0 2024-07-26T11:04:39.189 INFO:tasks.workunit.client.0.smithi144.stdout:2/148: mknod d0/d24/d2b/c2f 0 2024-07-26T11:04:39.204 INFO:tasks.workunit.client.0.smithi144.stdout:5/270: dread d1/d18/f3b [0,4194304] 0 2024-07-26T11:04:39.207 INFO:tasks.workunit.client.0.smithi144.stdout:4/177: read f6 [1956054,83069] 0 2024-07-26T11:04:39.209 INFO:tasks.workunit.client.0.smithi144.stdout:4/178: mknod d9/c38 0 2024-07-26T11:04:39.209 INFO:tasks.workunit.client.0.smithi144.stdout:4/179: chown d9/c10 25118218 1 2024-07-26T11:04:39.211 INFO:tasks.workunit.client.0.smithi144.stdout:4/180: creat d9/da/f39 x:0 0 0 2024-07-26T11:04:39.211 INFO:tasks.workunit.client.0.smithi144.stdout:4/181: readlink d9/l13 0 2024-07-26T11:04:39.211 INFO:tasks.workunit.client.0.smithi144.stdout:4/182: truncate d9/d26/f31 1119062 0 2024-07-26T11:04:39.213 INFO:tasks.workunit.client.0.smithi144.stdout:4/183: mknod d9/c3a 0 2024-07-26T11:04:39.215 INFO:tasks.workunit.client.0.smithi144.stdout:4/184: rename f0 to d9/da/f3b 0 2024-07-26T11:04:39.215 INFO:tasks.workunit.client.0.smithi144.stdout:4/185: readlink d9/da/l17 0 2024-07-26T11:04:39.216 INFO:tasks.workunit.client.0.smithi144.stdout:1/220: dread d3/f1e [0,4194304] 0 2024-07-26T11:04:39.216 INFO:tasks.workunit.client.0.smithi144.stdout:2/149: write d0/d6/f1d [824495,99051] 0 2024-07-26T11:04:39.216 INFO:tasks.workunit.client.0.smithi144.stdout:4/186: mkdir d9/d26/d3c 0 2024-07-26T11:04:39.217 INFO:tasks.workunit.client.0.smithi144.stdout:1/221: symlink d3/d6/da/l41 0 2024-07-26T11:04:39.217 INFO:tasks.workunit.client.0.smithi144.stdout:1/222: write d3/d6/da/d13/d22/f3c [350796,53293] 0 2024-07-26T11:04:39.221 INFO:tasks.workunit.client.0.smithi144.stdout:3/150: dread d1/d5/fa [0,4194304] 0 2024-07-26T11:04:39.225 INFO:tasks.workunit.client.0.smithi144.stdout:3/151: read d1/d5/f1c [278208,63989] 0 2024-07-26T11:04:39.226 INFO:tasks.workunit.client.0.smithi144.stdout:4/187: dread d9/f22 [0,4194304] 0 2024-07-26T11:04:39.227 INFO:tasks.workunit.client.0.smithi144.stdout:4/188: read - d9/d26/f2b zero size 2024-07-26T11:04:39.239 INFO:tasks.workunit.client.0.smithi144.stdout:4/189: mknod d9/d26/c3d 0 2024-07-26T11:04:39.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/246: sync 2024-07-26T11:04:39.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/247: truncate d8/d15/d26/f41 4517777 0 2024-07-26T11:04:39.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/248: chown d8/db/le 186284664 1 2024-07-26T11:04:39.240 INFO:tasks.workunit.client.0.smithi144.stdout:3/152: rmdir d1/d5/d25/d26 39 2024-07-26T11:04:39.241 INFO:tasks.workunit.client.0.smithi144.stdout:4/190: mknod d9/c3e 0 2024-07-26T11:04:39.242 INFO:tasks.workunit.client.0.smithi144.stdout:6/249: read d8/f2d [90196,85595] 0 2024-07-26T11:04:39.242 INFO:tasks.workunit.client.0.smithi144.stdout:4/191: creat d9/d26/f3f x:0 0 0 2024-07-26T11:04:39.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/250: mkdir d8/d4f/d52/d58 0 2024-07-26T11:04:39.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/251: chown d8/f53 40 1 2024-07-26T11:04:39.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/252: write d8/d15/d26/f28 [3693931,11628] 0 2024-07-26T11:04:39.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/253: write d8/f1d [2140711,15575] 0 2024-07-26T11:04:39.245 INFO:tasks.workunit.client.0.smithi144.stdout:4/192: unlink f5 0 2024-07-26T11:04:39.246 INFO:tasks.workunit.client.0.smithi144.stdout:7/199: dwrite d0/d6/d9/d45/f2a [0,4194304] 0 2024-07-26T11:04:39.250 INFO:tasks.workunit.client.0.smithi144.stdout:6/254: link d8/l22 d8/d50/l59 0 2024-07-26T11:04:39.250 INFO:tasks.workunit.client.0.smithi144.stdout:4/193: rename d9/da/f29 to d9/d26/d3c/f40 0 2024-07-26T11:04:39.250 INFO:tasks.workunit.client.0.smithi144.stdout:7/200: unlink d0/d6/d9/d45/c16 0 2024-07-26T11:04:39.250 INFO:tasks.workunit.client.0.smithi144.stdout:7/201: write d0/d6/d9/d45/f30 [1125591,45268] 0 2024-07-26T11:04:39.251 INFO:tasks.workunit.client.0.smithi144.stdout:0/253: dwrite d0/d25/d5e/f51 [0,4194304] 0 2024-07-26T11:04:39.257 INFO:tasks.workunit.client.0.smithi144.stdout:7/202: unlink d0/d6/c1b 0 2024-07-26T11:04:39.258 INFO:tasks.workunit.client.0.smithi144.stdout:0/254: symlink d0/d25/d5e/d1e/d30/l61 0 2024-07-26T11:04:39.258 INFO:tasks.workunit.client.0.smithi144.stdout:0/255: write d0/d25/d5e/d29/f42 [875473,75267] 0 2024-07-26T11:04:39.258 INFO:tasks.workunit.client.0.smithi144.stdout:0/256: dread - d0/d3e/d45/f55 zero size 2024-07-26T11:04:39.259 INFO:tasks.workunit.client.0.smithi144.stdout:0/257: write d0/f10 [2273761,71736] 0 2024-07-26T11:04:39.259 INFO:tasks.workunit.client.0.smithi144.stdout:0/258: dread - d0/d25/d5e/d29/f56 zero size 2024-07-26T11:04:39.259 INFO:tasks.workunit.client.0.smithi144.stdout:0/259: chown d0/d25/d5e/d1e/d30/l61 168641443 1 2024-07-26T11:04:39.269 INFO:tasks.workunit.client.0.smithi144.stdout:3/153: dread d1/f17 [0,4194304] 0 2024-07-26T11:04:39.269 INFO:tasks.workunit.client.0.smithi144.stdout:3/154: stat d1/c18 0 2024-07-26T11:04:39.269 INFO:tasks.workunit.client.0.smithi144.stdout:7/203: rename d0/d6/d9/f27 to d0/d6/f52 0 2024-07-26T11:04:39.269 INFO:tasks.workunit.client.0.smithi144.stdout:3/155: rename d1 to d1/d28/d29 22 2024-07-26T11:04:39.269 INFO:tasks.workunit.client.0.smithi144.stdout:3/156: write d1/d5/f15 [1998502,92020] 0 2024-07-26T11:04:39.270 INFO:tasks.workunit.client.0.smithi144.stdout:7/204: mkdir d0/d6/d9/d25/d2f/d53 0 2024-07-26T11:04:39.276 INFO:tasks.workunit.client.0.smithi144.stdout:7/205: truncate d0/d6/d9/d45/f1e 430403 0 2024-07-26T11:04:39.276 INFO:tasks.workunit.client.0.smithi144.stdout:4/194: dread d9/da/f3b [0,4194304] 0 2024-07-26T11:04:39.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/150: dwrite d0/f22 [0,4194304] 0 2024-07-26T11:04:39.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/151: chown d0/d24/d2b/c2f 12 1 2024-07-26T11:04:39.276 INFO:tasks.workunit.client.0.smithi144.stdout:4/195: link d9/f22 d9/f41 0 2024-07-26T11:04:39.276 INFO:tasks.workunit.client.0.smithi144.stdout:4/196: read - d9/d26/f30 zero size 2024-07-26T11:04:39.277 INFO:tasks.workunit.client.0.smithi144.stdout:1/223: dwrite d3/f10 [0,4194304] 0 2024-07-26T11:04:39.282 INFO:tasks.workunit.client.0.smithi144.stdout:6/255: dread f2 [0,4194304] 0 2024-07-26T11:04:39.282 INFO:tasks.workunit.client.0.smithi144.stdout:6/256: write d8/f2d [692179,46647] 0 2024-07-26T11:04:39.287 INFO:tasks.workunit.client.0.smithi144.stdout:1/224: rename d3/d6/da/d13/d19/l33 to d3/l42 0 2024-07-26T11:04:39.287 INFO:tasks.workunit.client.0.smithi144.stdout:1/225: fdatasync d3/d6/f23 0 2024-07-26T11:04:39.287 INFO:tasks.workunit.client.0.smithi144.stdout:0/260: write d0/d25/d5e/d1e/f21 [3402136,67373] 0 2024-07-26T11:04:39.289 INFO:tasks.workunit.client.0.smithi144.stdout:8/222: sync 2024-07-26T11:04:39.289 INFO:tasks.workunit.client.0.smithi144.stdout:8/223: stat d1a/d1c/d28/d2c/d38 0 2024-07-26T11:04:39.292 INFO:tasks.workunit.client.0.smithi144.stdout:1/226: mknod d3/d6/da/d13/d19/d3a/c43 0 2024-07-26T11:04:39.292 INFO:tasks.workunit.client.0.smithi144.stdout:0/261: getdents d0 0 2024-07-26T11:04:39.292 INFO:tasks.workunit.client.0.smithi144.stdout:0/262: dread - d0/d25/d5e/f2c zero size 2024-07-26T11:04:39.292 INFO:tasks.workunit.client.0.smithi144.stdout:8/224: mkdir d1a/d1c/d30/d4e 0 2024-07-26T11:04:39.292 INFO:tasks.workunit.client.0.smithi144.stdout:8/225: fsync d1a/d1c/d30/f40 0 2024-07-26T11:04:39.293 INFO:tasks.workunit.client.0.smithi144.stdout:0/263: truncate d0/d25/d5e/d1e/f3c 3873138 0 2024-07-26T11:04:39.295 INFO:tasks.workunit.client.0.smithi144.stdout:8/226: mkdir d1a/d1c/d30/d43/d4f 0 2024-07-26T11:04:39.299 INFO:tasks.workunit.client.0.smithi144.stdout:0/264: rename d0/c9 to d0/d25/d5e/c62 0 2024-07-26T11:04:39.299 INFO:tasks.workunit.client.0.smithi144.stdout:8/227: mkdir d1a/d32/d50 0 2024-07-26T11:04:39.299 INFO:tasks.workunit.client.0.smithi144.stdout:8/228: write d1a/f20 [1317020,47283] 0 2024-07-26T11:04:39.299 INFO:tasks.workunit.client.0.smithi144.stdout:8/229: chown d1a 30186328 1 2024-07-26T11:04:39.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/230: creat d1a/d1c/d30/d43/d4f/f51 x:0 0 0 2024-07-26T11:04:39.302 INFO:tasks.workunit.client.0.smithi144.stdout:8/231: unlink d1a/d1c/d30/d43/d4f/f51 0 2024-07-26T11:04:39.303 INFO:tasks.workunit.client.0.smithi144.stdout:3/157: write d1/f22 [1252945,92029] 0 2024-07-26T11:04:39.303 INFO:tasks.workunit.client.0.smithi144.stdout:3/158: fdatasync d1/d5/f6 0 2024-07-26T11:04:39.304 INFO:tasks.workunit.client.0.smithi144.stdout:3/159: mknod d1/d5/d25/c2a 0 2024-07-26T11:04:39.304 INFO:tasks.workunit.client.0.smithi144.stdout:3/160: write d1/f14 [1336817,80296] 0 2024-07-26T11:04:39.305 INFO:tasks.workunit.client.0.smithi144.stdout:4/197: dread f3 [0,4194304] 0 2024-07-26T11:04:39.306 INFO:tasks.workunit.client.0.smithi144.stdout:3/161: creat d1/d5/d25/d26/f2b x:0 0 0 2024-07-26T11:04:39.306 INFO:tasks.workunit.client.0.smithi144.stdout:3/162: chown d1/c19 131758033 1 2024-07-26T11:04:39.307 INFO:tasks.workunit.client.0.smithi144.stdout:3/163: chown d1/d5/d25/d26 68860826 1 2024-07-26T11:04:39.307 INFO:tasks.workunit.client.0.smithi144.stdout:3/164: dread - d1/f20 zero size 2024-07-26T11:04:39.313 INFO:tasks.workunit.client.0.smithi144.stdout:8/232: write d1a/d1c/d28/f2e [695549,76585] 0 2024-07-26T11:04:39.331 INFO:tasks.workunit.client.0.smithi144.stdout:7/206: dread d0/d6/d9/d25/d2f/f35 [0,4194304] 0 2024-07-26T11:04:39.332 INFO:tasks.workunit.client.0.smithi144.stdout:3/165: fsync d1/d5/d25/d26/f2b 0 2024-07-26T11:04:39.335 INFO:tasks.workunit.client.0.smithi144.stdout:3/166: rmdir d1/d5/d25/d26 39 2024-07-26T11:04:39.338 INFO:tasks.workunit.client.0.smithi144.stdout:7/207: creat d0/d6/d9/d25/d2f/f54 x:0 0 0 2024-07-26T11:04:39.340 INFO:tasks.workunit.client.0.smithi144.stdout:7/208: unlink d0/f3 0 2024-07-26T11:04:39.342 INFO:tasks.workunit.client.0.smithi144.stdout:7/209: mkdir d0/d6/d55 0 2024-07-26T11:04:39.346 INFO:tasks.workunit.client.0.smithi144.stdout:8/233: fdatasync d1a/d1c/f1f 0 2024-07-26T11:04:39.346 INFO:tasks.workunit.client.0.smithi144.stdout:8/234: stat d1a/f20 0 2024-07-26T11:04:39.346 INFO:tasks.workunit.client.0.smithi144.stdout:8/235: write d1a/d32/f48 [1620299,127064] 0 2024-07-26T11:04:39.346 INFO:tasks.workunit.client.0.smithi144.stdout:6/257: dread d8/f10 [0,4194304] 0 2024-07-26T11:04:39.346 INFO:tasks.workunit.client.0.smithi144.stdout:2/152: dwrite d0/f1f [0,4194304] 0 2024-07-26T11:04:39.348 INFO:tasks.workunit.client.0.smithi144.stdout:7/210: rename d0/d6/f2b to d0/d6/f56 0 2024-07-26T11:04:39.348 INFO:tasks.workunit.client.0.smithi144.stdout:7/211: chown d0/d6/d9/d45 2 1 2024-07-26T11:04:39.348 INFO:tasks.workunit.client.0.smithi144.stdout:7/212: write d0/d6/d9/d45/f3e [495015,102515] 0 2024-07-26T11:04:39.348 INFO:tasks.workunit.client.0.smithi144.stdout:7/213: write d0/f7 [319932,115541] 0 2024-07-26T11:04:39.348 INFO:tasks.workunit.client.0.smithi144.stdout:1/227: dwrite d3/d6/da/d21/f24 [0,4194304] 0 2024-07-26T11:04:39.348 INFO:tasks.workunit.client.0.smithi144.stdout:8/236: mknod d1a/d32/d50/c52 0 2024-07-26T11:04:39.349 INFO:tasks.workunit.client.0.smithi144.stdout:6/258: link f6 d8/d4f/d52/f5a 0 2024-07-26T11:04:39.349 INFO:tasks.workunit.client.0.smithi144.stdout:6/259: fdatasync d8/fd 0 2024-07-26T11:04:39.349 INFO:tasks.workunit.client.0.smithi144.stdout:6/260: read - d8/d15/f39 zero size 2024-07-26T11:04:39.349 INFO:tasks.workunit.client.0.smithi144.stdout:6/261: stat d8/db/c36 0 2024-07-26T11:04:39.349 INFO:tasks.workunit.client.0.smithi144.stdout:2/153: creat d0/d15/d20/f30 x:0 0 0 2024-07-26T11:04:39.350 INFO:tasks.workunit.client.0.smithi144.stdout:2/154: read - d0/d15/d20/f30 zero size 2024-07-26T11:04:39.350 INFO:tasks.workunit.client.0.smithi144.stdout:2/155: write d0/d15/f17 [1510783,81889] 0 2024-07-26T11:04:39.352 INFO:tasks.workunit.client.0.smithi144.stdout:1/228: unlink d3/f10 0 2024-07-26T11:04:39.352 INFO:tasks.workunit.client.0.smithi144.stdout:8/237: creat d1a/d1c/d2f/d3c/f53 x:0 0 0 2024-07-26T11:04:39.353 INFO:tasks.workunit.client.0.smithi144.stdout:6/262: link d8/d15/c18 d8/d15/d57/c5b 0 2024-07-26T11:04:39.353 INFO:tasks.workunit.client.0.smithi144.stdout:6/263: chown d8/db/f32 543061 1 2024-07-26T11:04:39.357 INFO:tasks.workunit.client.0.smithi144.stdout:1/229: symlink d3/l44 0 2024-07-26T11:04:39.357 INFO:tasks.workunit.client.0.smithi144.stdout:1/230: chown d3/d6/da/d13/d19/d3a 0 1 2024-07-26T11:04:39.358 INFO:tasks.workunit.client.0.smithi144.stdout:1/231: stat d3/d6 0 2024-07-26T11:04:39.361 INFO:tasks.workunit.client.0.smithi144.stdout:6/264: mkdir d8/d4f/d5c 0 2024-07-26T11:04:39.364 INFO:tasks.workunit.client.0.smithi144.stdout:0/265: dwrite d0/d25/d5e/d1e/f3c [0,4194304] 0 2024-07-26T11:04:39.364 INFO:tasks.workunit.client.0.smithi144.stdout:0/266: fdatasync d0/d25/d5e/f2c 0 2024-07-26T11:04:39.364 INFO:tasks.workunit.client.0.smithi144.stdout:5/271: sync 2024-07-26T11:04:39.364 INFO:tasks.workunit.client.0.smithi144.stdout:5/272: stat d1/d18/f51 0 2024-07-26T11:04:39.364 INFO:tasks.workunit.client.0.smithi144.stdout:9/234: sync 2024-07-26T11:04:39.367 INFO:tasks.workunit.client.0.smithi144.stdout:5/273: creat d1/d9/d5b/d4d/f63 x:0 0 0 2024-07-26T11:04:39.369 INFO:tasks.workunit.client.0.smithi144.stdout:6/265: link d8/cf d8/d15/d57/c5d 0 2024-07-26T11:04:39.369 INFO:tasks.workunit.client.0.smithi144.stdout:0/267: symlink d0/d25/l63 0 2024-07-26T11:04:39.381 INFO:tasks.workunit.client.0.smithi144.stdout:8/238: write f6 [181860,122461] 0 2024-07-26T11:04:39.383 INFO:tasks.workunit.client.0.smithi144.stdout:9/235: creat d9/de/d14/d23/f50 x:0 0 0 2024-07-26T11:04:39.384 INFO:tasks.workunit.client.0.smithi144.stdout:4/198: dwrite d9/da/f35 [0,4194304] 0 2024-07-26T11:04:39.389 INFO:tasks.workunit.client.0.smithi144.stdout:6/266: creat d8/d15/f5e x:0 0 0 2024-07-26T11:04:39.389 INFO:tasks.workunit.client.0.smithi144.stdout:9/236: rmdir d9/de/d14/d3f 39 2024-07-26T11:04:39.389 INFO:tasks.workunit.client.0.smithi144.stdout:9/237: readlink d9/de/d14/d3f/l43 0 2024-07-26T11:04:39.389 INFO:tasks.workunit.client.0.smithi144.stdout:7/214: write d0/f2d [141109,119639] 0 2024-07-26T11:04:39.390 INFO:tasks.workunit.client.0.smithi144.stdout:0/268: getdents d0/d25/d5e/d1e/d30/d4b 0 2024-07-26T11:04:39.390 INFO:tasks.workunit.client.0.smithi144.stdout:8/239: link d1a/d1c/d28/f3f d1a/d1c/d30/d43/d4c/f54 0 2024-07-26T11:04:39.390 INFO:tasks.workunit.client.0.smithi144.stdout:4/199: creat d9/d26/d3c/f42 x:0 0 0 2024-07-26T11:04:39.391 INFO:tasks.workunit.client.0.smithi144.stdout:6/267: mkdir d8/d15/d57/d5f 0 2024-07-26T11:04:39.393 INFO:tasks.workunit.client.0.smithi144.stdout:9/238: getdents d9/de/d14 0 2024-07-26T11:04:39.395 INFO:tasks.workunit.client.0.smithi144.stdout:0/269: creat d0/d3e/d45/d4c/f64 x:0 0 0 2024-07-26T11:04:39.395 INFO:tasks.workunit.client.0.smithi144.stdout:0/270: write d0/d25/d5e/d1e/f3f [4587773,3119] 0 2024-07-26T11:04:39.395 INFO:tasks.workunit.client.0.smithi144.stdout:8/240: mkdir d1a/d1c/d30/d43/d4c/d55 0 2024-07-26T11:04:39.395 INFO:tasks.workunit.client.0.smithi144.stdout:8/241: stat d1a/d1c/d2f 0 2024-07-26T11:04:39.395 INFO:tasks.workunit.client.0.smithi144.stdout:8/242: chown d1a/d32/d49 117522382 1 2024-07-26T11:04:39.395 INFO:tasks.workunit.client.0.smithi144.stdout:8/243: truncate d1a/d1c/f22 4941210 0 2024-07-26T11:04:39.396 INFO:tasks.workunit.client.0.smithi144.stdout:7/215: mkdir d0/d6/d9/d45/d57 0 2024-07-26T11:04:39.398 INFO:tasks.workunit.client.0.smithi144.stdout:7/216: dread - d0/d6/d9/d45/f26 zero size 2024-07-26T11:04:39.398 INFO:tasks.workunit.client.0.smithi144.stdout:3/167: dwrite d1/d5/f6 [4194304,4194304] 0 2024-07-26T11:04:39.399 INFO:tasks.workunit.client.0.smithi144.stdout:3/168: write d1/f14 [3619685,67692] 0 2024-07-26T11:04:39.399 INFO:tasks.workunit.client.0.smithi144.stdout:4/200: creat d9/d26/f43 x:0 0 0 2024-07-26T11:04:39.399 INFO:tasks.workunit.client.0.smithi144.stdout:4/201: dread - d9/f34 zero size 2024-07-26T11:04:39.400 INFO:tasks.workunit.client.0.smithi144.stdout:4/202: chown d9/da 0 1 2024-07-26T11:04:39.402 INFO:tasks.workunit.client.0.smithi144.stdout:4/203: stat d9/da/f25 0 2024-07-26T11:04:39.403 INFO:tasks.workunit.client.0.smithi144.stdout:8/244: getdents d1a/d1c/d2f 0 2024-07-26T11:04:39.406 INFO:tasks.workunit.client.0.smithi144.stdout:3/169: rename d1/ld to d1/d28/l2c 0 2024-07-26T11:04:39.409 INFO:tasks.workunit.client.0.smithi144.stdout:8/245: symlink d1a/d1c/d30/d43/d4f/l56 0 2024-07-26T11:04:39.409 INFO:tasks.workunit.client.0.smithi144.stdout:8/246: truncate d1a/d32/d49/f4a 186098 0 2024-07-26T11:04:39.422 INFO:tasks.workunit.client.0.smithi144.stdout:2/156: dwrite d0/d6/f1d [0,4194304] 0 2024-07-26T11:04:39.424 INFO:tasks.workunit.client.0.smithi144.stdout:4/204: rename d9/d26/f43 to d9/d26/d3c/f44 0 2024-07-26T11:04:39.445 INFO:tasks.workunit.client.0.smithi144.stdout:2/157: read d0/d6/d9/f26 [463469,5309] 0 2024-07-26T11:04:39.446 INFO:tasks.workunit.client.0.smithi144.stdout:2/158: read d0/d6/d9/f26 [709949,30586] 0 2024-07-26T11:04:39.455 INFO:tasks.workunit.client.0.smithi144.stdout:1/232: dwrite d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:39.455 INFO:tasks.workunit.client.0.smithi144.stdout:1/233: readlink d3/d6/da/d13/d19/l34 0 2024-07-26T11:04:39.455 INFO:tasks.workunit.client.0.smithi144.stdout:1/234: getdents d3/d3d 0 2024-07-26T11:04:39.457 INFO:tasks.workunit.client.0.smithi144.stdout:1/235: creat d3/d6/da/d13/d19/f45 x:0 0 0 2024-07-26T11:04:39.457 INFO:tasks.workunit.client.0.smithi144.stdout:1/236: fdatasync d3/f39 0 2024-07-26T11:04:39.457 INFO:tasks.workunit.client.0.smithi144.stdout:1/237: chown d3/d6/da/d13/d22/f30 46 1 2024-07-26T11:04:39.458 INFO:tasks.workunit.client.0.smithi144.stdout:1/238: rmdir d3/d6/da/d13/d19/d27 39 2024-07-26T11:04:39.460 INFO:tasks.workunit.client.0.smithi144.stdout:1/239: symlink d3/d3d/l46 0 2024-07-26T11:04:39.460 INFO:tasks.workunit.client.0.smithi144.stdout:1/240: dread - d3/f3b zero size 2024-07-26T11:04:39.470 INFO:tasks.workunit.client.0.smithi144.stdout:7/217: dwrite d0/d6/f52 [0,4194304] 0 2024-07-26T11:04:39.473 INFO:tasks.workunit.client.0.smithi144.stdout:3/170: dwrite d1/ff [0,4194304] 0 2024-07-26T11:04:39.477 INFO:tasks.workunit.client.0.smithi144.stdout:3/171: creat d1/d5/d25/d26/f2d x:0 0 0 2024-07-26T11:04:39.477 INFO:tasks.workunit.client.0.smithi144.stdout:7/218: creat d0/d6/d9/d3f/f58 x:0 0 0 2024-07-26T11:04:39.477 INFO:tasks.workunit.client.0.smithi144.stdout:7/219: chown d0/d6/c20 52647 1 2024-07-26T11:04:39.481 INFO:tasks.workunit.client.0.smithi144.stdout:3/172: creat d1/d5/d25/d26/f2e x:0 0 0 2024-07-26T11:04:39.482 INFO:tasks.workunit.client.0.smithi144.stdout:0/271: dwrite d0/f32 [0,4194304] 0 2024-07-26T11:04:39.482 INFO:tasks.workunit.client.0.smithi144.stdout:2/159: dread d0/f14 [0,4194304] 0 2024-07-26T11:04:39.482 INFO:tasks.workunit.client.0.smithi144.stdout:7/220: rename d0/fd to d0/d6/d9/d12/f59 0 2024-07-26T11:04:39.482 INFO:tasks.workunit.client.0.smithi144.stdout:2/160: chown d0/d6/f1d 1529991013 1 2024-07-26T11:04:39.482 INFO:tasks.workunit.client.0.smithi144.stdout:7/221: chown d0/d6/d55 90682 1 2024-07-26T11:04:39.483 INFO:tasks.workunit.client.0.smithi144.stdout:0/272: fsync d0/f7 0 2024-07-26T11:04:39.483 INFO:tasks.workunit.client.0.smithi144.stdout:0/273: chown d0/d25/d5e/d1e/d30/f3b 8873256 1 2024-07-26T11:04:39.484 INFO:tasks.workunit.client.0.smithi144.stdout:0/274: write d0/d3e/d45/f55 [688798,71006] 0 2024-07-26T11:04:39.487 INFO:tasks.workunit.client.0.smithi144.stdout:7/222: mkdir d0/d6/d9/d25/d2f/d53/d5a 0 2024-07-26T11:04:39.490 INFO:tasks.workunit.client.0.smithi144.stdout:7/223: rmdir d0/d6/d9/d25/d2f/d53/d5a 0 2024-07-26T11:04:39.491 INFO:tasks.workunit.client.0.smithi144.stdout:7/224: symlink d0/d6/d9/d25/d2f/d47/l5b 0 2024-07-26T11:04:39.493 INFO:tasks.workunit.client.0.smithi144.stdout:7/225: rename d0/c3d to d0/d6/d9/d25/d2f/d47/c5c 0 2024-07-26T11:04:39.495 INFO:tasks.workunit.client.0.smithi144.stdout:4/205: dwrite d9/d26/f3f [0,4194304] 0 2024-07-26T11:04:39.496 INFO:tasks.workunit.client.0.smithi144.stdout:9/239: dwrite d9/de/d14/d23/f47 [0,4194304] 0 2024-07-26T11:04:39.503 INFO:tasks.workunit.client.0.smithi144.stdout:7/226: creat d0/d6/d9/d12/f5d x:0 0 0 2024-07-26T11:04:39.503 INFO:tasks.workunit.client.0.smithi144.stdout:7/227: dread - d0/d6/d9/d25/d2f/f48 zero size 2024-07-26T11:04:39.503 INFO:tasks.workunit.client.0.smithi144.stdout:7/228: dread - d0/d6/d9/d25/f42 zero size 2024-07-26T11:04:39.503 INFO:tasks.workunit.client.0.smithi144.stdout:7/229: readlink d0/l1c 0 2024-07-26T11:04:39.506 INFO:tasks.workunit.client.0.smithi144.stdout:4/206: creat d9/f45 x:0 0 0 2024-07-26T11:04:39.512 INFO:tasks.workunit.client.0.smithi144.stdout:1/241: dwrite d3/d6/da/d13/d19/d27/f28 [0,4194304] 0 2024-07-26T11:04:39.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/207: mknod d9/da/c46 0 2024-07-26T11:04:39.522 INFO:tasks.workunit.client.0.smithi144.stdout:9/240: creat d9/de/f51 x:0 0 0 2024-07-26T11:04:39.522 INFO:tasks.workunit.client.0.smithi144.stdout:2/161: write d0/f14 [2521172,88103] 0 2024-07-26T11:04:39.523 INFO:tasks.workunit.client.0.smithi144.stdout:7/230: getdents d0/d6/d9/d12 0 2024-07-26T11:04:39.527 INFO:tasks.workunit.client.0.smithi144.stdout:4/208: rename d9/f34 to d9/d26/f47 0 2024-07-26T11:04:39.528 INFO:tasks.workunit.client.0.smithi144.stdout:9/241: rmdir d9/de/d14/d23 39 2024-07-26T11:04:39.528 INFO:tasks.workunit.client.0.smithi144.stdout:9/242: getdents d9/de/d4e 0 2024-07-26T11:04:39.529 INFO:tasks.workunit.client.0.smithi144.stdout:2/162: rename d0/d15/f17 to d0/d15/f31 0 2024-07-26T11:04:39.529 INFO:tasks.workunit.client.0.smithi144.stdout:2/163: read d0/f14 [1938561,98202] 0 2024-07-26T11:04:39.530 INFO:tasks.workunit.client.0.smithi144.stdout:2/164: write d0/d6/f1d [5767857,96095] 0 2024-07-26T11:04:39.530 INFO:tasks.workunit.client.0.smithi144.stdout:7/231: mkdir d0/d6/d9/d25/d2f/d47/d5e 0 2024-07-26T11:04:39.531 INFO:tasks.workunit.client.0.smithi144.stdout:9/243: creat d9/de/f52 x:0 0 0 2024-07-26T11:04:39.532 INFO:tasks.workunit.client.0.smithi144.stdout:2/165: mknod d0/d15/d20/d28/d2e/c32 0 2024-07-26T11:04:39.535 INFO:tasks.workunit.client.0.smithi144.stdout:9/244: creat d9/de/d14/d3f/f53 x:0 0 0 2024-07-26T11:04:39.535 INFO:tasks.workunit.client.0.smithi144.stdout:7/232: mknod d0/d6/d9/d25/d2f/c5f 0 2024-07-26T11:04:39.535 INFO:tasks.workunit.client.0.smithi144.stdout:2/166: truncate d0/d6/d9/f26 215468 0 2024-07-26T11:04:39.535 INFO:tasks.workunit.client.0.smithi144.stdout:7/233: write d0/d6/f17 [1794242,97885] 0 2024-07-26T11:04:39.535 INFO:tasks.workunit.client.0.smithi144.stdout:9/245: symlink d9/de/l54 0 2024-07-26T11:04:39.536 INFO:tasks.workunit.client.0.smithi144.stdout:2/167: creat d0/d6/d9/f33 x:0 0 0 2024-07-26T11:04:39.536 INFO:tasks.workunit.client.0.smithi144.stdout:2/168: stat d0/d15/d20/d28/lf 0 2024-07-26T11:04:39.544 INFO:tasks.workunit.client.0.smithi144.stdout:9/246: mknod d9/de/d14/d23/c55 0 2024-07-26T11:04:39.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/169: mkdir d0/d6/d34 0 2024-07-26T11:04:39.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/170: fdatasync d0/d6/d9/f26 0 2024-07-26T11:04:39.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/171: rmdir d0/d24 39 2024-07-26T11:04:39.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/172: chown d0/d15/d20/d28/lf 671 1 2024-07-26T11:04:39.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/173: readlink d0/d15/d20/d28/lf 0 2024-07-26T11:04:39.551 INFO:tasks.workunit.client.0.smithi144.stdout:0/275: dwrite d0/d3e/f43 [0,4194304] 0 2024-07-26T11:04:39.552 INFO:tasks.workunit.client.0.smithi144.stdout:0/276: rmdir d0/d25/d5e 39 2024-07-26T11:04:39.554 INFO:tasks.workunit.client.0.smithi144.stdout:4/209: dread d9/da/f2d [0,4194304] 0 2024-07-26T11:04:39.555 INFO:tasks.workunit.client.0.smithi144.stdout:1/242: dread d3/d6/f11 [4194304,4194304] 0 2024-07-26T11:04:39.555 INFO:tasks.workunit.client.0.smithi144.stdout:3/173: dwrite d1/d5/f6 [0,4194304] 0 2024-07-26T11:04:39.556 INFO:tasks.workunit.client.0.smithi144.stdout:4/210: mkdir d9/d26/d48 0 2024-07-26T11:04:39.557 INFO:tasks.workunit.client.0.smithi144.stdout:1/243: creat d3/d6/f47 x:0 0 0 2024-07-26T11:04:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:39 smithi160 ceph-mon[36715]: pgmap v526: 129 pgs: 129 active+clean; 332 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 40 MiB/s wr, 615 op/s 2024-07-26T11:04:39.563 INFO:tasks.workunit.client.0.smithi144.stdout:4/211: link d9/da/c21 d9/da/c49 0 2024-07-26T11:04:39.563 INFO:tasks.workunit.client.0.smithi144.stdout:4/212: dread - d9/d26/f47 zero size 2024-07-26T11:04:39.563 INFO:tasks.workunit.client.0.smithi144.stdout:4/213: stat d9/d26 0 2024-07-26T11:04:39.563 INFO:tasks.workunit.client.0.smithi144.stdout:4/214: stat d9/da/f2d 0 2024-07-26T11:04:39.563 INFO:tasks.workunit.client.0.smithi144.stdout:6/268: sync 2024-07-26T11:04:39.563 INFO:tasks.workunit.client.0.smithi144.stdout:5/274: sync 2024-07-26T11:04:39.572 INFO:tasks.workunit.client.0.smithi144.stdout:2/174: read d0/f22 [2113052,77151] 0 2024-07-26T11:04:39.572 INFO:tasks.workunit.client.0.smithi144.stdout:2/175: read d0/f1 [654956,94336] 0 2024-07-26T11:04:39.572 INFO:tasks.workunit.client.0.smithi144.stdout:8/247: sync 2024-07-26T11:04:39.577 INFO:tasks.workunit.client.0.smithi144.stdout:6/269: link d8/d15/c2b d8/d15/d57/d5f/c60 0 2024-07-26T11:04:39.577 INFO:tasks.workunit.client.0.smithi144.stdout:5/275: rename d1/d9/d5b/l36 to d1/d18/l64 0 2024-07-26T11:04:39.577 INFO:tasks.workunit.client.0.smithi144.stdout:5/276: chown d1/d9/d44/d45/l4a 1453000 1 2024-07-26T11:04:39.577 INFO:tasks.workunit.client.0.smithi144.stdout:4/215: unlink d9/da/f35 0 2024-07-26T11:04:39.581 INFO:tasks.workunit.client.0.smithi144.stdout:9/247: dread d9/de/d14/f24 [0,4194304] 0 2024-07-26T11:04:39.581 INFO:tasks.workunit.client.0.smithi144.stdout:8/248: symlink d1a/d1c/d2f/d3c/l57 0 2024-07-26T11:04:39.581 INFO:tasks.workunit.client.0.smithi144.stdout:2/176: getdents d0/d15/d20/d28/d2e 0 2024-07-26T11:04:39.582 INFO:tasks.workunit.client.0.smithi144.stdout:9/248: write d9/de/f46 [307178,29043] 0 2024-07-26T11:04:39.582 INFO:tasks.workunit.client.0.smithi144.stdout:4/216: mknod d9/da/c4a 0 2024-07-26T11:04:39.582 INFO:tasks.workunit.client.0.smithi144.stdout:2/177: write d0/d6/d9/f26 [712292,9299] 0 2024-07-26T11:04:39.582 INFO:tasks.workunit.client.0.smithi144.stdout:8/249: truncate d1a/d1c/d30/f37 3021433 0 2024-07-26T11:04:39.583 INFO:tasks.workunit.client.0.smithi144.stdout:8/250: truncate f11 1261453 0 2024-07-26T11:04:39.583 INFO:tasks.workunit.client.0.smithi144.stdout:8/251: stat d1a/d1c/d2f/c35 0 2024-07-26T11:04:39.583 INFO:tasks.workunit.client.0.smithi144.stdout:8/252: write d1a/f31 [882088,25353] 0 2024-07-26T11:04:39.583 INFO:tasks.workunit.client.0.smithi144.stdout:9/249: rmdir d9/de/d14/d23/d39 39 2024-07-26T11:04:39.583 INFO:tasks.workunit.client.0.smithi144.stdout:9/250: fsync f4 0 2024-07-26T11:04:39.585 INFO:tasks.workunit.client.0.smithi144.stdout:6/270: dread d8/d15/d26/f4c [0,4194304] 0 2024-07-26T11:04:39.585 INFO:tasks.workunit.client.0.smithi144.stdout:6/271: chown d8/d15/d26 3928022 1 2024-07-26T11:04:39.588 INFO:tasks.workunit.client.0.smithi144.stdout:2/178: creat d0/d15/d20/f35 x:0 0 0 2024-07-26T11:04:39.588 INFO:tasks.workunit.client.0.smithi144.stdout:6/272: link d8/d15/f39 d8/d15/f61 0 2024-07-26T11:04:39.589 INFO:tasks.workunit.client.0.smithi144.stdout:6/273: write d8/d15/d26/f4c [629239,121661] 0 2024-07-26T11:04:39.589 INFO:tasks.workunit.client.0.smithi144.stdout:6/274: fsync d8/f56 0 2024-07-26T11:04:39.589 INFO:tasks.workunit.client.0.smithi144.stdout:9/251: link d9/de/d14/d23/f3a d9/d3b/f56 0 2024-07-26T11:04:39.590 INFO:tasks.workunit.client.0.smithi144.stdout:6/275: creat d8/d4f/f62 x:0 0 0 2024-07-26T11:04:39.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/277: dread d1/d18/d24/d27/f2f [0,4194304] 0 2024-07-26T11:04:39.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/278: dread - d1/d18/f60 zero size 2024-07-26T11:04:39.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/279: chown d1/d18/f48 1 1 2024-07-26T11:04:39.592 INFO:tasks.workunit.client.0.smithi144.stdout:9/252: symlink d9/de/d14/d23/d39/l57 0 2024-07-26T11:04:39.592 INFO:tasks.workunit.client.0.smithi144.stdout:6/276: rename d8/d15/f35 to d8/d4f/f63 0 2024-07-26T11:04:39.594 INFO:tasks.workunit.client.0.smithi144.stdout:2/179: rename d0/d6/ce to d0/d6/c36 0 2024-07-26T11:04:39.595 INFO:tasks.workunit.client.0.smithi144.stdout:5/280: symlink d1/d18/l65 0 2024-07-26T11:04:39.596 INFO:tasks.workunit.client.0.smithi144.stdout:8/253: dread f4 [0,4194304] 0 2024-07-26T11:04:39.597 INFO:tasks.workunit.client.0.smithi144.stdout:6/277: unlink d8/d15/d26/f30 0 2024-07-26T11:04:39.601 INFO:tasks.workunit.client.0.smithi144.stdout:7/234: dwrite d0/d6/d9/d45/f30 [0,4194304] 0 2024-07-26T11:04:39.601 INFO:tasks.workunit.client.0.smithi144.stdout:2/180: truncate d0/f22 376336 0 2024-07-26T11:04:39.604 INFO:tasks.workunit.client.0.smithi144.stdout:5/281: unlink d1/l8 0 2024-07-26T11:04:39.604 INFO:tasks.workunit.client.0.smithi144.stdout:8/254: mkdir d1a/d1c/d28/d2c/d38/d58 0 2024-07-26T11:04:39.606 INFO:tasks.workunit.client.0.smithi144.stdout:6/278: mknod d8/c64 0 2024-07-26T11:04:39.607 INFO:tasks.workunit.client.0.smithi144.stdout:6/279: dread - d8/d42/f47 zero size 2024-07-26T11:04:39.609 INFO:tasks.workunit.client.0.smithi144.stdout:5/282: mknod d1/d9/d44/d45/c66 0 2024-07-26T11:04:39.612 INFO:tasks.workunit.client.0.smithi144.stdout:8/255: rmdir d1a/d1c 39 2024-07-26T11:04:39.612 INFO:tasks.workunit.client.0.smithi144.stdout:6/280: symlink d8/d15/d57/l65 0 2024-07-26T11:04:39.614 INFO:tasks.workunit.client.0.smithi144.stdout:5/283: creat d1/d18/f67 x:0 0 0 2024-07-26T11:04:39.614 INFO:tasks.workunit.client.0.smithi144.stdout:5/284: write d1/f5e [763251,78115] 0 2024-07-26T11:04:39.614 INFO:tasks.workunit.client.0.smithi144.stdout:5/285: write d1/d18/d24/d27/f2f [2961340,62481] 0 2024-07-26T11:04:39.614 INFO:tasks.workunit.client.0.smithi144.stdout:5/286: symlink d1/d18/d24/d27/l68 0 2024-07-26T11:04:39.615 INFO:tasks.workunit.client.0.smithi144.stdout:5/287: stat d1/d9/c12 0 2024-07-26T11:04:39.616 INFO:tasks.workunit.client.0.smithi144.stdout:5/288: link d1/d5/f2e d1/d9/d5b/f69 0 2024-07-26T11:04:39.619 INFO:tasks.workunit.client.0.smithi144.stdout:1/244: dread d3/d6/da/d21/f24 [0,4194304] 0 2024-07-26T11:04:39.619 INFO:tasks.workunit.client.0.smithi144.stdout:1/245: readlink d3/d3d/l46 0 2024-07-26T11:04:39.621 INFO:tasks.workunit.client.0.smithi144.stdout:1/246: creat d3/d6/da/f48 x:0 0 0 2024-07-26T11:04:39.621 INFO:tasks.workunit.client.0.smithi144.stdout:1/247: write d3/d6/da/d13/d22/f30 [962210,107100] 0 2024-07-26T11:04:39.624 INFO:tasks.workunit.client.0.smithi144.stdout:1/248: creat d3/d6/da/d13/d19/d3a/d3e/f49 x:0 0 0 2024-07-26T11:04:39.624 INFO:tasks.workunit.client.0.smithi144.stdout:1/249: chown d3/f1e 19 1 2024-07-26T11:04:39.626 INFO:tasks.workunit.client.0.smithi144.stdout:9/253: write d9/de/d14/f2c [1454235,63945] 0 2024-07-26T11:04:39.626 INFO:tasks.workunit.client.0.smithi144.stdout:9/254: fdatasync d9/de/d14/f35 0 2024-07-26T11:04:39.628 INFO:tasks.workunit.client.0.smithi144.stdout:9/255: symlink d9/de/d14/d23/l58 0 2024-07-26T11:04:39.630 INFO:tasks.workunit.client.0.smithi144.stdout:9/256: symlink d9/de/d4e/l59 0 2024-07-26T11:04:39.633 INFO:tasks.workunit.client.0.smithi144.stdout:0/277: dwrite d0/d3e/d45/f4a [0,4194304] 0 2024-07-26T11:04:39.633 INFO:tasks.workunit.client.0.smithi144.stdout:0/278: getdents d0/d3e/d5f 0 2024-07-26T11:04:39.633 INFO:tasks.workunit.client.0.smithi144.stdout:0/279: dread - d0/d25/d5e/f2c zero size 2024-07-26T11:04:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:39 smithi183 ceph-mon[37273]: pgmap v526: 129 pgs: 129 active+clean; 332 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 40 MiB/s wr, 615 op/s 2024-07-26T11:04:39.635 INFO:tasks.workunit.client.0.smithi144.stdout:4/217: dread f1 [0,4194304] 0 2024-07-26T11:04:39.638 INFO:tasks.workunit.client.0.smithi144.stdout:0/280: mkdir d0/d25/d5e/d29/d65 0 2024-07-26T11:04:39.638 INFO:tasks.workunit.client.0.smithi144.stdout:0/281: write d0/f8 [723986,9936] 0 2024-07-26T11:04:39.638 INFO:tasks.workunit.client.0.smithi144.stdout:0/282: fsync d0/d25/d5e/f51 0 2024-07-26T11:04:39.641 INFO:tasks.workunit.client.0.smithi144.stdout:4/218: creat d9/d26/d3c/f4b x:0 0 0 2024-07-26T11:04:39.641 INFO:tasks.workunit.client.0.smithi144.stdout:4/219: getdents d9/d26/d48 0 2024-07-26T11:04:39.641 INFO:tasks.workunit.client.0.smithi144.stdout:4/220: chown d9/d26/f2f 193888 1 2024-07-26T11:04:39.641 INFO:tasks.workunit.client.0.smithi144.stdout:0/283: rmdir d0/d25/d5e/d1e 39 2024-07-26T11:04:39.641 INFO:tasks.workunit.client.0.smithi144.stdout:0/284: readlink d0/d25/d5e/d29/l31 0 2024-07-26T11:04:39.644 INFO:tasks.workunit.client.0.smithi144.stdout:4/221: rename d9/da/f32 to d9/d26/f4c 0 2024-07-26T11:04:39.646 INFO:tasks.workunit.client.0.smithi144.stdout:4/222: creat d9/d26/d48/f4d x:0 0 0 2024-07-26T11:04:39.648 INFO:tasks.workunit.client.0.smithi144.stdout:4/223: creat d9/f4e x:0 0 0 2024-07-26T11:04:39.648 INFO:tasks.workunit.client.0.smithi144.stdout:4/224: fdatasync d9/d26/f47 0 2024-07-26T11:04:39.648 INFO:tasks.workunit.client.0.smithi144.stdout:4/225: read - d9/d26/f2f zero size 2024-07-26T11:04:39.650 INFO:tasks.workunit.client.0.smithi144.stdout:4/226: rename d9/da/l17 to d9/d26/l4f 0 2024-07-26T11:04:39.651 INFO:tasks.workunit.client.0.smithi144.stdout:4/227: mkdir d9/d26/d48/d50 0 2024-07-26T11:04:39.657 INFO:tasks.workunit.client.0.smithi144.stdout:5/289: dread d1/d18/d24/f46 [0,4194304] 0 2024-07-26T11:04:39.657 INFO:tasks.workunit.client.0.smithi144.stdout:5/290: stat d1/d18/f51 0 2024-07-26T11:04:39.658 INFO:tasks.workunit.client.0.smithi144.stdout:5/291: mknod d1/d18/c6a 0 2024-07-26T11:04:39.664 INFO:tasks.workunit.client.0.smithi144.stdout:5/292: mknod d1/d18/c6b 0 2024-07-26T11:04:39.664 INFO:tasks.workunit.client.0.smithi144.stdout:5/293: write d1/d18/d24/f2a [2361234,110749] 0 2024-07-26T11:04:39.664 INFO:tasks.workunit.client.0.smithi144.stdout:5/294: dread - d1/d5/f49 zero size 2024-07-26T11:04:39.664 INFO:tasks.workunit.client.0.smithi144.stdout:6/281: dwrite d8/db/f55 [0,4194304] 0 2024-07-26T11:04:39.666 INFO:tasks.workunit.client.0.smithi144.stdout:0/285: dread d0/d3e/d45/f55 [0,4194304] 0 2024-07-26T11:04:39.666 INFO:tasks.workunit.client.0.smithi144.stdout:6/282: rename d8/d15/c1f to d8/d4f/d52/d58/c66 0 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:0/286: write d0/d25/d5e/d1e/f36 [4673974,410] 0 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:7/235: dread d0/f2d [0,4194304] 0 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:0/287: stat d0/d25/d5e/d29/f2d 0 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:0/288: dread - d0/d25/d5e/f2c zero size 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:7/236: chown d0/c19 1863 1 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:4/228: dread d9/f12 [0,4194304] 0 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:6/283: creat d8/f67 x:0 0 0 2024-07-26T11:04:39.667 INFO:tasks.workunit.client.0.smithi144.stdout:4/229: chown d9/lf 381 1 2024-07-26T11:04:39.668 INFO:tasks.workunit.client.0.smithi144.stdout:4/230: readlink l7 0 2024-07-26T11:04:39.669 INFO:tasks.workunit.client.0.smithi144.stdout:7/237: link d0/d6/d9/d25/d2f/d47/f4f d0/d6/d9/d12/f60 0 2024-07-26T11:04:39.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/284: link d8/d15/d57/c5d d8/d50/c68 0 2024-07-26T11:04:39.671 INFO:tasks.workunit.client.0.smithi144.stdout:6/285: chown d8/d15/l54 49 1 2024-07-26T11:04:39.673 INFO:tasks.workunit.client.0.smithi144.stdout:4/231: creat d9/f51 x:0 0 0 2024-07-26T11:04:39.675 INFO:tasks.workunit.client.0.smithi144.stdout:7/238: truncate d0/d6/d9/d25/d2f/f39 500242 0 2024-07-26T11:04:39.678 INFO:tasks.workunit.client.0.smithi144.stdout:6/286: symlink d8/d15/d57/d5f/l69 0 2024-07-26T11:04:39.678 INFO:tasks.workunit.client.0.smithi144.stdout:6/287: chown d8 3540 1 2024-07-26T11:04:39.682 INFO:tasks.workunit.client.0.smithi144.stdout:4/232: unlink d9/c3a 0 2024-07-26T11:04:39.682 INFO:tasks.workunit.client.0.smithi144.stdout:7/239: link d0/d6/c31 d0/d6/d55/c61 0 2024-07-26T11:04:39.683 INFO:tasks.workunit.client.0.smithi144.stdout:2/181: dwrite d0/d15/d20/f30 [0,4194304] 0 2024-07-26T11:04:39.683 INFO:tasks.workunit.client.0.smithi144.stdout:6/288: mknod d8/d4f/c6a 0 2024-07-26T11:04:39.686 INFO:tasks.workunit.client.0.smithi144.stdout:1/250: dwrite d3/d6/f47 [0,4194304] 0 2024-07-26T11:04:39.693 INFO:tasks.workunit.client.0.smithi144.stdout:7/240: symlink d0/d6/d9/d25/d2f/d53/l62 0 2024-07-26T11:04:39.693 INFO:tasks.workunit.client.0.smithi144.stdout:6/289: getdents d8/d15/d26 0 2024-07-26T11:04:39.693 INFO:tasks.workunit.client.0.smithi144.stdout:9/257: dwrite d9/de/f21 [0,4194304] 0 2024-07-26T11:04:39.693 INFO:tasks.workunit.client.0.smithi144.stdout:9/258: fdatasync d9/de/f4d 0 2024-07-26T11:04:39.693 INFO:tasks.workunit.client.0.smithi144.stdout:2/182: symlink d0/d15/d20/d28/d27/l37 0 2024-07-26T11:04:39.694 INFO:tasks.workunit.client.0.smithi144.stdout:2/183: write d0/d15/d20/d28/f4 [393486,101555] 0 2024-07-26T11:04:39.694 INFO:tasks.workunit.client.0.smithi144.stdout:1/251: truncate d3/d6/da/d13/d19/d27/f28 584566 0 2024-07-26T11:04:39.694 INFO:tasks.workunit.client.0.smithi144.stdout:1/252: write d3/d6/f32 [4695188,104555] 0 2024-07-26T11:04:39.696 INFO:tasks.workunit.client.0.smithi144.stdout:0/289: read d0/d25/d5e/d1e/f24 [1186557,101500] 0 2024-07-26T11:04:39.697 INFO:tasks.workunit.client.0.smithi144.stdout:6/290: creat d8/d15/f6b x:0 0 0 2024-07-26T11:04:39.697 INFO:tasks.workunit.client.0.smithi144.stdout:6/291: write d8/f53 [280046,66519] 0 2024-07-26T11:04:39.698 INFO:tasks.workunit.client.0.smithi144.stdout:9/259: mknod d9/de/d14/c5a 0 2024-07-26T11:04:39.698 INFO:tasks.workunit.client.0.smithi144.stdout:7/241: creat d0/d6/d9/d25/d2f/d47/d5e/f63 x:0 0 0 2024-07-26T11:04:39.699 INFO:tasks.workunit.client.0.smithi144.stdout:2/184: symlink d0/d15/l38 0 2024-07-26T11:04:39.702 INFO:tasks.workunit.client.0.smithi144.stdout:1/253: unlink f1 0 2024-07-26T11:04:39.702 INFO:tasks.workunit.client.0.smithi144.stdout:1/254: chown d3/d6 3245 1 2024-07-26T11:04:39.704 INFO:tasks.workunit.client.0.smithi144.stdout:0/290: unlink d0/d25/d5e/f51 0 2024-07-26T11:04:39.704 INFO:tasks.workunit.client.0.smithi144.stdout:0/291: chown d0/d25/d5e/f41 376154 1 2024-07-26T11:04:39.710 INFO:tasks.workunit.client.0.smithi144.stdout:7/242: unlink d0/c4e 0 2024-07-26T11:04:39.710 INFO:tasks.workunit.client.0.smithi144.stdout:7/243: chown d0/d6/d9/d45/f1e 0 1 2024-07-26T11:04:39.712 INFO:tasks.workunit.client.0.smithi144.stdout:1/255: creat d3/d6/da/f4a x:0 0 0 2024-07-26T11:04:39.713 INFO:tasks.workunit.client.0.smithi144.stdout:0/292: creat d0/d3e/d45/d4c/f66 x:0 0 0 2024-07-26T11:04:39.713 INFO:tasks.workunit.client.0.smithi144.stdout:0/293: write d0/d25/d5e/d1e/f24 [1746401,55265] 0 2024-07-26T11:04:39.713 INFO:tasks.workunit.client.0.smithi144.stdout:0/294: stat d0/f8 0 2024-07-26T11:04:39.713 INFO:tasks.workunit.client.0.smithi144.stdout:0/295: chown d0/d25/d5e/d29/f42 13832 1 2024-07-26T11:04:39.713 INFO:tasks.workunit.client.0.smithi144.stdout:2/185: fdatasync d0/d15/d20/d28/f4 0 2024-07-26T11:04:39.713 INFO:tasks.workunit.client.0.smithi144.stdout:2/186: chown d0/d6/d9 161381 1 2024-07-26T11:04:39.715 INFO:tasks.workunit.client.0.smithi144.stdout:7/244: mkdir d0/d6/d9/d64 0 2024-07-26T11:04:39.715 INFO:tasks.workunit.client.0.smithi144.stdout:1/256: link d3/f1e d3/d6/da/d13/d19/d3a/f4b 0 2024-07-26T11:04:39.717 INFO:tasks.workunit.client.0.smithi144.stdout:0/296: mkdir d0/d25/d5e/d1e/d30/d67 0 2024-07-26T11:04:39.719 INFO:tasks.workunit.client.0.smithi144.stdout:2/187: unlink d0/c23 0 2024-07-26T11:04:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:39 smithi144 ceph-mon[32129]: pgmap v526: 129 pgs: 129 active+clean; 332 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 40 MiB/s wr, 615 op/s 2024-07-26T11:04:39.720 INFO:tasks.workunit.client.0.smithi144.stdout:4/233: dread d9/f1f [0,4194304] 0 2024-07-26T11:04:39.720 INFO:tasks.workunit.client.0.smithi144.stdout:5/295: dwrite d1/d9/d5b/d4d/f3e [0,4194304] 0 2024-07-26T11:04:39.720 INFO:tasks.workunit.client.0.smithi144.stdout:5/296: dread - d1/f39 zero size 2024-07-26T11:04:39.720 INFO:tasks.workunit.client.0.smithi144.stdout:5/297: dread - d1/d18/f48 zero size 2024-07-26T11:04:39.721 INFO:tasks.workunit.client.0.smithi144.stdout:1/257: mknod d3/d6/da/d21/c4c 0 2024-07-26T11:04:39.723 INFO:tasks.workunit.client.0.smithi144.stdout:0/297: rename d0/d25/d5e/d29/f56 to d0/d25/d5e/f68 0 2024-07-26T11:04:39.724 INFO:tasks.workunit.client.0.smithi144.stdout:2/188: truncate d0/d15/d20/f30 3852583 0 2024-07-26T11:04:39.727 INFO:tasks.workunit.client.0.smithi144.stdout:4/234: creat d9/da/f52 x:0 0 0 2024-07-26T11:04:39.727 INFO:tasks.workunit.client.0.smithi144.stdout:5/298: creat d1/d9/d5b/d4d/f6c x:0 0 0 2024-07-26T11:04:39.737 INFO:tasks.workunit.client.0.smithi144.stdout:4/235: symlink d9/d26/d48/d50/l53 0 2024-07-26T11:04:39.737 INFO:tasks.workunit.client.0.smithi144.stdout:2/189: write d0/f1e [24467,59353] 0 2024-07-26T11:04:39.737 INFO:tasks.workunit.client.0.smithi144.stdout:4/236: write d9/f45 [129773,127614] 0 2024-07-26T11:04:39.737 INFO:tasks.workunit.client.0.smithi144.stdout:4/237: dread - d9/f4e zero size 2024-07-26T11:04:39.738 INFO:tasks.workunit.client.0.smithi144.stdout:2/190: rename d0/d15/d20/d28/f4 to d0/d6/f39 0 2024-07-26T11:04:39.739 INFO:tasks.workunit.client.0.smithi144.stdout:4/238: truncate d9/da/f3b 2200348 0 2024-07-26T11:04:39.739 INFO:tasks.workunit.client.0.smithi144.stdout:4/239: readlink l7 0 2024-07-26T11:04:39.740 INFO:tasks.workunit.client.0.smithi144.stdout:4/240: mknod d9/d26/d3c/c54 0 2024-07-26T11:04:39.742 INFO:tasks.workunit.client.0.smithi144.stdout:4/241: symlink d9/d26/d3c/l55 0 2024-07-26T11:04:39.747 INFO:tasks.workunit.client.0.smithi144.stdout:0/298: truncate d0/f8 425682 0 2024-07-26T11:04:39.751 INFO:tasks.workunit.client.0.smithi144.stdout:5/299: read d1/d18/d24/f46 [2123,83302] 0 2024-07-26T11:04:39.752 INFO:tasks.workunit.client.0.smithi144.stdout:0/299: creat d0/d3e/f69 x:0 0 0 2024-07-26T11:04:39.754 INFO:tasks.workunit.client.0.smithi144.stdout:0/300: mkdir d0/d25/d5e/d1e/d30/d6a 0 2024-07-26T11:04:39.755 INFO:tasks.workunit.client.0.smithi144.stdout:0/301: write d0/f28 [714640,105182] 0 2024-07-26T11:04:39.756 INFO:tasks.workunit.client.0.smithi144.stdout:0/302: mknod d0/d25/d5e/d29/c6b 0 2024-07-26T11:04:39.760 INFO:tasks.workunit.client.0.smithi144.stdout:2/191: dread d0/d6/d9/f26 [0,4194304] 0 2024-07-26T11:04:39.760 INFO:tasks.workunit.client.0.smithi144.stdout:2/192: fsync d0/d15/f31 0 2024-07-26T11:04:39.761 INFO:tasks.workunit.client.0.smithi144.stdout:2/193: truncate d0/f14 2498522 0 2024-07-26T11:04:39.761 INFO:tasks.workunit.client.0.smithi144.stdout:0/303: mkdir d0/d25/d5e/d1e/d6c 0 2024-07-26T11:04:39.761 INFO:tasks.workunit.client.0.smithi144.stdout:2/194: chown d0/d15/d20/d28/lf 827609 1 2024-07-26T11:04:39.761 INFO:tasks.workunit.client.0.smithi144.stdout:2/195: chown d0/d15/d20/d28 54774771 1 2024-07-26T11:04:39.763 INFO:tasks.workunit.client.0.smithi144.stdout:9/260: dwrite d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:39.767 INFO:tasks.workunit.client.0.smithi144.stdout:9/261: rename d9/de/d14/f4a to d9/de/d14/d23/d39/f5b 0 2024-07-26T11:04:39.768 INFO:tasks.workunit.client.0.smithi144.stdout:9/262: truncate d9/de/f4b 1780580 0 2024-07-26T11:04:39.768 INFO:tasks.workunit.client.0.smithi144.stdout:6/292: dwrite d8/d42/f47 [0,4194304] 0 2024-07-26T11:04:39.769 INFO:tasks.workunit.client.0.smithi144.stdout:6/293: chown d8/d15/f39 7601061 1 2024-07-26T11:04:39.771 INFO:tasks.workunit.client.0.smithi144.stdout:9/263: unlink d9/c38 0 2024-07-26T11:04:39.799 INFO:tasks.workunit.client.0.smithi144.stdout:2/196: truncate d0/f14 3542300 0 2024-07-26T11:04:39.800 INFO:tasks.workunit.client.0.smithi144.stdout:1/258: dwrite d3/d6/da/d13/d22/f30 [0,4194304] 0 2024-07-26T11:04:39.807 INFO:tasks.workunit.client.0.smithi144.stdout:5/300: dread d1/d18/d24/f2a [0,4194304] 0 2024-07-26T11:04:39.807 INFO:tasks.workunit.client.0.smithi144.stdout:5/301: fdatasync d1/d9/d44/f41 0 2024-07-26T11:04:39.809 INFO:tasks.workunit.client.0.smithi144.stdout:5/302: creat d1/d5/f6d x:0 0 0 2024-07-26T11:04:39.809 INFO:tasks.workunit.client.0.smithi144.stdout:5/303: chown d1/d18/f1d 67074633 1 2024-07-26T11:04:39.810 INFO:tasks.workunit.client.0.smithi144.stdout:5/304: creat d1/f6e x:0 0 0 2024-07-26T11:04:39.812 INFO:tasks.workunit.client.0.smithi144.stdout:5/305: chown d1/d9/d5b/d4d/l25 769443 1 2024-07-26T11:04:39.813 INFO:tasks.workunit.client.0.smithi144.stdout:4/242: dwrite d9/f45 [0,4194304] 0 2024-07-26T11:04:39.816 INFO:tasks.workunit.client.0.smithi144.stdout:0/304: dread d0/f10 [0,4194304] 0 2024-07-26T11:04:39.816 INFO:tasks.workunit.client.0.smithi144.stdout:7/245: dwrite d0/d6/d9/d45/f2a [0,4194304] 0 2024-07-26T11:04:39.818 INFO:tasks.workunit.client.0.smithi144.stdout:5/306: creat d1/d5/f6f x:0 0 0 2024-07-26T11:04:39.821 INFO:tasks.workunit.client.0.smithi144.stdout:4/243: creat d9/d26/d48/f56 x:0 0 0 2024-07-26T11:04:39.821 INFO:tasks.workunit.client.0.smithi144.stdout:4/244: truncate d9/da/f25 4488647 0 2024-07-26T11:04:39.826 INFO:tasks.workunit.client.0.smithi144.stdout:8/256: sync 2024-07-26T11:04:39.827 INFO:tasks.workunit.client.0.smithi144.stdout:8/257: chown d1a/d1c/f22 102 1 2024-07-26T11:04:39.836 INFO:tasks.workunit.client.0.smithi144.stdout:3/174: sync 2024-07-26T11:04:39.836 INFO:tasks.workunit.client.0.smithi144.stdout:4/245: mkdir d9/d57 0 2024-07-26T11:04:39.836 INFO:tasks.workunit.client.0.smithi144.stdout:3/175: dread - d1/d5/d25/d26/f2b zero size 2024-07-26T11:04:39.850 INFO:tasks.workunit.client.0.smithi144.stdout:1/259: dread d3/d6/da/d13/d19/d3a/f4b [0,4194304] 0 2024-07-26T11:04:39.850 INFO:tasks.workunit.client.0.smithi144.stdout:1/260: stat d3/d6/da/d13/c14 0 2024-07-26T11:04:39.877 INFO:tasks.workunit.client.0.smithi144.stdout:9/264: dwrite d9/de/f30 [0,4194304] 0 2024-07-26T11:04:39.878 INFO:tasks.workunit.client.0.smithi144.stdout:7/246: dwrite d0/f11 [0,4194304] 0 2024-07-26T11:04:39.884 INFO:tasks.workunit.client.0.smithi144.stdout:3/176: dwrite d1/d5/d25/d26/f2e [0,4194304] 0 2024-07-26T11:04:39.887 INFO:tasks.workunit.client.0.smithi144.stdout:3/177: symlink d1/d5/d25/l2f 0 2024-07-26T11:04:39.887 INFO:tasks.workunit.client.0.smithi144.stdout:3/178: dread - d1/d5/d25/d26/f2d zero size 2024-07-26T11:04:39.887 INFO:tasks.workunit.client.0.smithi144.stdout:3/179: chown d1/f1a 7571 1 2024-07-26T11:04:39.888 INFO:tasks.workunit.client.0.smithi144.stdout:3/180: rmdir d1 39 2024-07-26T11:04:39.888 INFO:tasks.workunit.client.0.smithi144.stdout:3/181: chown d1/d5/d25/d26/f2b 42 1 2024-07-26T11:04:39.888 INFO:tasks.workunit.client.0.smithi144.stdout:3/182: stat d1/d28/l2c 0 2024-07-26T11:04:39.910 INFO:tasks.workunit.client.0.smithi144.stdout:2/197: dwrite d0/f1 [0,4194304] 0 2024-07-26T11:04:39.910 INFO:tasks.workunit.client.0.smithi144.stdout:2/198: fsync d0/f1f 0 2024-07-26T11:04:39.911 INFO:tasks.workunit.client.0.smithi144.stdout:5/307: dwrite d1/d18/f1d [0,4194304] 0 2024-07-26T11:04:39.913 INFO:tasks.workunit.client.0.smithi144.stdout:5/308: creat d1/d18/d24/d27/f70 x:0 0 0 2024-07-26T11:04:39.915 INFO:tasks.workunit.client.0.smithi144.stdout:5/309: mkdir d1/d18/d24/d27/d71 0 2024-07-26T11:04:39.916 INFO:tasks.workunit.client.0.smithi144.stdout:7/247: dread d0/d6/d9/f13 [0,4194304] 0 2024-07-26T11:04:39.917 INFO:tasks.workunit.client.0.smithi144.stdout:0/305: dwrite d0/f32 [0,4194304] 0 2024-07-26T11:04:39.918 INFO:tasks.workunit.client.0.smithi144.stdout:7/248: creat d0/d6/d9/d64/f65 x:0 0 0 2024-07-26T11:04:39.919 INFO:tasks.workunit.client.0.smithi144.stdout:0/306: creat d0/d25/d5e/d29/f6d x:0 0 0 2024-07-26T11:04:39.920 INFO:tasks.workunit.client.0.smithi144.stdout:7/249: mknod d0/d6/c66 0 2024-07-26T11:04:39.921 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:04:39.922 INFO:tasks.workunit.client.0.smithi144.stdout:1/261: dwrite d3/d6/da/f2f [0,4194304] 0 2024-07-26T11:04:39.924 INFO:tasks.workunit.client.0.smithi144.stdout:8/258: dwrite d1a/d1c/d30/d43/d4c/f54 [0,4194304] 0 2024-07-26T11:04:39.924 INFO:tasks.workunit.client.0.smithi144.stdout:4/246: dread d9/da/f1a [0,4194304] 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:5/310: dread d1/f5e [0,4194304] 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:5/311: write d1/d18/f51 [38694,29980] 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:5/312: truncate d1/d18/d24/d35/f3d 9767 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:5/313: chown d1 6038991 1 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:1/262: rename d3/d6/da/d13/d19/d3a/d3e/f49 to d3/d6/da/d21/f4d 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:1/263: fsync d3/f39 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:2/199: truncate d0/f1 3783741 0 2024-07-26T11:04:39.932 INFO:tasks.workunit.client.0.smithi144.stdout:2/200: dread - d0/d15/d20/f35 zero size 2024-07-26T11:04:39.933 INFO:tasks.workunit.client.0.smithi144.stdout:4/247: creat d9/d26/f58 x:0 0 0 2024-07-26T11:04:39.933 INFO:tasks.workunit.client.0.smithi144.stdout:5/314: rmdir d1/d9/d5b/d4d 39 2024-07-26T11:04:39.933 INFO:tasks.workunit.client.0.smithi144.stdout:2/201: rmdir d0/d6 39 2024-07-26T11:04:39.934 INFO:tasks.workunit.client.0.smithi144.stdout:2/202: stat d0/d15/d20/d28/d27 0 2024-07-26T11:04:39.934 INFO:tasks.workunit.client.0.smithi144.stdout:2/203: fsync d0/d6/f39 0 2024-07-26T11:04:39.934 INFO:tasks.workunit.client.0.smithi144.stdout:1/264: mknod d3/d6/da/d13/d19/d27/d36/d37/c4e 0 2024-07-26T11:04:39.934 INFO:tasks.workunit.client.0.smithi144.stdout:4/248: unlink d9/f33 0 2024-07-26T11:04:39.934 INFO:tasks.workunit.client.0.smithi144.stdout:4/249: fsync d9/d26/d3c/f42 0 2024-07-26T11:04:39.935 INFO:tasks.workunit.client.0.smithi144.stdout:2/204: rename d0/l13 to d0/d6/d34/l3a 0 2024-07-26T11:04:39.937 INFO:tasks.workunit.client.0.smithi144.stdout:2/205: getdents d0/d15/d20/d28/d2e 0 2024-07-26T11:04:39.937 INFO:tasks.workunit.client.0.smithi144.stdout:1/265: creat d3/f4f x:0 0 0 2024-07-26T11:04:39.937 INFO:tasks.workunit.client.0.smithi144.stdout:1/266: chown d3/l20 45588 1 2024-07-26T11:04:39.945 INFO:tasks.workunit.client.0.smithi144.stdout:3/183: dwrite d1/d5/d25/d26/f27 [0,4194304] 0 2024-07-26T11:04:39.952 INFO:tasks.workunit.client.0.smithi144.stdout:1/267: creat d3/d6/da/d13/d19/d3a/d3e/f50 x:0 0 0 2024-07-26T11:04:39.953 INFO:tasks.workunit.client.0.smithi144.stdout:3/184: mkdir d1/d5/d25/d26/d30 0 2024-07-26T11:04:39.953 INFO:tasks.workunit.client.0.smithi144.stdout:1/268: dread - d3/d6/da/f48 zero size 2024-07-26T11:04:39.953 INFO:tasks.workunit.client.0.smithi144.stdout:3/185: truncate d1/d5/d25/d26/f2d 702994 0 2024-07-26T11:04:39.954 INFO:tasks.workunit.client.0.smithi144.stdout:1/269: creat d3/d6/da/d21/f51 x:0 0 0 2024-07-26T11:04:39.954 INFO:tasks.workunit.client.0.smithi144.stdout:1/270: stat d3/d6/da/d13/l1c 0 2024-07-26T11:04:39.955 INFO:tasks.workunit.client.0.smithi144.stdout:1/271: symlink d3/d6/da/d13/d19/d3a/l52 0 2024-07-26T11:04:39.955 INFO:tasks.workunit.client.0.smithi144.stdout:0/307: dread d0/f7 [0,4194304] 0 2024-07-26T11:04:39.956 INFO:tasks.workunit.client.0.smithi144.stdout:0/308: chown d0/d25/d5e/d1e/c33 1055 1 2024-07-26T11:04:39.961 INFO:tasks.workunit.client.0.smithi144.stdout:1/272: mkdir d3/d53 0 2024-07-26T11:04:39.961 INFO:tasks.workunit.client.0.smithi144.stdout:1/273: chown d3/d6/da/d21/l2b 335 1 2024-07-26T11:04:39.961 INFO:tasks.workunit.client.0.smithi144.stdout:0/309: read d0/d25/d5e/d1e/f21 [1134640,26087] 0 2024-07-26T11:04:39.961 INFO:tasks.workunit.client.0.smithi144.stdout:1/274: fsync d3/d6/da/f12 0 2024-07-26T11:04:39.961 INFO:tasks.workunit.client.0.smithi144.stdout:9/265: dwrite d9/de/d14/d23/f31 [0,4194304] 0 2024-07-26T11:04:39.961 INFO:tasks.workunit.client.0.smithi144.stdout:9/266: write d9/de/d14/f35 [317292,70356] 0 2024-07-26T11:04:39.979 INFO:tasks.workunit.client.0.smithi144.stdout:6/294: sync 2024-07-26T11:04:39.982 INFO:tasks.workunit.client.0.smithi144.stdout:6/295: symlink d8/d15/d57/l6c 0 2024-07-26T11:04:40.010 INFO:tasks.workunit.client.0.smithi144.stdout:5/315: dwrite d1/d5/f2d [4194304,4194304] 0 2024-07-26T11:04:40.012 INFO:tasks.workunit.client.0.smithi144.stdout:5/316: creat d1/d9/f72 x:0 0 0 2024-07-26T11:04:40.012 INFO:tasks.workunit.client.0.smithi144.stdout:5/317: fdatasync d1/d9/d44/f4e 0 2024-07-26T11:04:40.014 INFO:tasks.workunit.client.0.smithi144.stdout:5/318: mknod d1/d9/d5b/d4d/c73 0 2024-07-26T11:04:40.018 INFO:tasks.workunit.client.0.smithi144.stdout:5/319: symlink d1/d18/l74 0 2024-07-26T11:04:40.019 INFO:tasks.workunit.client.0.smithi144.stdout:3/186: dwrite d1/d5/d25/d26/f2d [0,4194304] 0 2024-07-26T11:04:40.019 INFO:tasks.workunit.client.0.smithi144.stdout:3/187: mknod d1/d5/d25/d26/c31 0 2024-07-26T11:04:40.023 INFO:tasks.workunit.client.0.smithi144.stdout:6/296: dread d8/db/f1b [0,4194304] 0 2024-07-26T11:04:40.025 INFO:tasks.workunit.client.0.smithi144.stdout:6/297: symlink d8/d42/l6d 0 2024-07-26T11:04:40.026 INFO:tasks.workunit.client.0.smithi144.stdout:6/298: dread - d8/db/f3e zero size 2024-07-26T11:04:40.027 INFO:tasks.workunit.client.0.smithi144.stdout:6/299: mkdir d8/d15/d57/d5f/d6e 0 2024-07-26T11:04:40.032 INFO:tasks.workunit.client.0.smithi144.stdout:2/206: dwrite d0/d6/d9/f26 [0,4194304] 0 2024-07-26T11:04:40.032 INFO:tasks.workunit.client.0.smithi144.stdout:1/275: dread d3/d6/f32 [0,4194304] 0 2024-07-26T11:04:40.032 INFO:tasks.workunit.client.0.smithi144.stdout:1/276: stat d3/c9 0 2024-07-26T11:04:40.032 INFO:tasks.workunit.client.0.smithi144.stdout:4/250: dwrite d9/f12 [0,4194304] 0 2024-07-26T11:04:40.040 INFO:tasks.workunit.client.0.smithi144.stdout:9/267: dread d9/de/d14/d23/f31 [0,4194304] 0 2024-07-26T11:04:40.040 INFO:tasks.workunit.client.0.smithi144.stdout:5/320: dread d1/d18/d24/d27/f4f [0,4194304] 0 2024-07-26T11:04:40.040 INFO:tasks.workunit.client.0.smithi144.stdout:2/207: getdents d0/d6/d34 0 2024-07-26T11:04:40.042 INFO:tasks.workunit.client.0.smithi144.stdout:8/259: dwrite d1a/d1c/d30/d43/d4c/f54 [0,4194304] 0 2024-07-26T11:04:40.042 INFO:tasks.workunit.client.0.smithi144.stdout:8/260: write d1a/d1c/d28/f3f [4939612,68239] 0 2024-07-26T11:04:40.043 INFO:tasks.workunit.client.0.smithi144.stdout:4/251: rename d9/f45 to d9/d26/d48/d50/f59 0 2024-07-26T11:04:40.043 INFO:tasks.workunit.client.0.smithi144.stdout:8/261: write d1a/f1b [2935962,126992] 0 2024-07-26T11:04:40.045 INFO:tasks.workunit.client.0.smithi144.stdout:2/208: getdents d0 0 2024-07-26T11:04:40.046 INFO:tasks.workunit.client.0.smithi144.stdout:5/321: link d1/d9/l23 d1/d18/d24/l75 0 2024-07-26T11:04:40.046 INFO:tasks.workunit.client.0.smithi144.stdout:5/322: fsync d1/d18/d24/d27/f4f 0 2024-07-26T11:04:40.050 INFO:tasks.workunit.client.0.smithi144.stdout:8/262: rename d1a/d1c/l25 to d1a/d1c/d2f/l59 0 2024-07-26T11:04:40.050 INFO:tasks.workunit.client.0.smithi144.stdout:5/323: creat d1/d18/d24/f76 x:0 0 0 2024-07-26T11:04:40.050 INFO:tasks.workunit.client.0.smithi144.stdout:5/324: chown d1/d9/d5b/d4d/l28 1293446 1 2024-07-26T11:04:40.051 INFO:tasks.workunit.client.0.smithi144.stdout:5/325: fdatasync d1/d18/d24/d35/f3d 0 2024-07-26T11:04:40.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/310: dwrite d0/d25/d5e/d1e/f3c [0,4194304] 0 2024-07-26T11:04:40.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/311: fsync d0/f7 0 2024-07-26T11:04:40.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/312: chown d0/d3e/f69 0 1 2024-07-26T11:04:40.055 INFO:tasks.workunit.client.0.smithi144.stdout:4/252: dread d9/f41 [0,4194304] 0 2024-07-26T11:04:40.055 INFO:tasks.workunit.client.0.smithi144.stdout:4/253: fdatasync d9/d26/f28 0 2024-07-26T11:04:40.055 INFO:tasks.workunit.client.0.smithi144.stdout:0/313: creat d0/d25/d5e/d29/f6e x:0 0 0 2024-07-26T11:04:40.055 INFO:tasks.workunit.client.0.smithi144.stdout:5/326: write d1/d9/d44/d45/f56 [39943,77757] 0 2024-07-26T11:04:40.055 INFO:tasks.workunit.client.0.smithi144.stdout:5/327: chown d1/d18/d24 9055646 1 2024-07-26T11:04:40.060 INFO:tasks.workunit.client.0.smithi144.stdout:5/328: creat d1/d18/d24/d35/f77 x:0 0 0 2024-07-26T11:04:40.060 INFO:tasks.workunit.client.0.smithi144.stdout:0/314: truncate d0/d25/d5e/d1e/f24 625057 0 2024-07-26T11:04:40.064 INFO:tasks.workunit.client.0.smithi144.stdout:0/315: mkdir d0/d25/d5e/d1e/d30/d6a/d6f 0 2024-07-26T11:04:40.064 INFO:tasks.workunit.client.0.smithi144.stdout:0/316: write d0/f18 [2678690,116703] 0 2024-07-26T11:04:40.071 INFO:tasks.workunit.client.0.smithi144.stdout:1/277: dread d3/d6/da/d21/f24 [0,4194304] 0 2024-07-26T11:04:40.116 INFO:tasks.workunit.client.0.smithi144.stdout:2/209: dread d0/f1e [0,4194304] 0 2024-07-26T11:04:40.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/300: dwrite d8/db/f2c [0,4194304] 0 2024-07-26T11:04:40.118 INFO:tasks.workunit.client.0.smithi144.stdout:2/210: mknod d0/d6/d34/c3b 0 2024-07-26T11:04:40.118 INFO:tasks.workunit.client.0.smithi144.stdout:5/329: dwrite d1/f39 [0,4194304] 0 2024-07-26T11:04:40.123 INFO:tasks.workunit.client.0.smithi144.stdout:2/211: mknod d0/d6/c3c 0 2024-07-26T11:04:40.128 INFO:tasks.workunit.client.0.smithi144.stdout:2/212: creat d0/f3d x:0 0 0 2024-07-26T11:04:40.144 INFO:tasks.workunit.client.0.smithi144.stdout:0/317: dwrite d0/d25/d5e/d1e/f21 [0,4194304] 0 2024-07-26T11:04:40.146 INFO:tasks.workunit.client.0.smithi144.stdout:0/318: mknod d0/d3e/d45/d4c/c70 0 2024-07-26T11:04:40.146 INFO:tasks.workunit.client.0.smithi144.stdout:0/319: dread - d0/d3e/d45/f46 zero size 2024-07-26T11:04:40.146 INFO:tasks.workunit.client.0.smithi144.stdout:0/320: chown d0/d3e/d45/f4a 607743613 1 2024-07-26T11:04:40.146 INFO:tasks.workunit.client.0.smithi144.stdout:0/321: dread - d0/d25/d5e/d1e/d30/d4b/f58 zero size 2024-07-26T11:04:40.151 INFO:tasks.workunit.client.0.smithi144.stdout:5/330: write d1/d5/f2d [7307995,23074] 0 2024-07-26T11:04:40.151 INFO:tasks.workunit.client.0.smithi144.stdout:5/331: truncate d1/d5/f2e 1039170 0 2024-07-26T11:04:40.151 INFO:tasks.workunit.client.0.smithi144.stdout:5/332: chown d1/d9/d5b/c50 5444 1 2024-07-26T11:04:40.154 INFO:tasks.workunit.client.0.smithi144.stdout:5/333: read d1/d18/d24/f2a [1031645,77740] 0 2024-07-26T11:04:40.154 INFO:tasks.workunit.client.0.smithi144.stdout:2/213: dread d0/d15/f31 [0,4194304] 0 2024-07-26T11:04:40.154 INFO:tasks.workunit.client.0.smithi144.stdout:2/214: fdatasync d0/d15/f31 0 2024-07-26T11:04:40.154 INFO:tasks.workunit.client.0.smithi144.stdout:2/215: write d0/f1e [1040375,96257] 0 2024-07-26T11:04:40.154 INFO:tasks.workunit.client.0.smithi144.stdout:2/216: write d0/f22 [298062,74189] 0 2024-07-26T11:04:40.155 INFO:tasks.workunit.client.0.smithi144.stdout:8/263: dwrite d1a/f20 [0,4194304] 0 2024-07-26T11:04:40.155 INFO:tasks.workunit.client.0.smithi144.stdout:8/264: write f11 [677850,43436] 0 2024-07-26T11:04:40.157 INFO:tasks.workunit.client.0.smithi144.stdout:2/217: rename d0/f22 to d0/f3e 0 2024-07-26T11:04:40.157 INFO:tasks.workunit.client.0.smithi144.stdout:5/334: link d1/d9/d5b/d4d/f16 d1/d5/f78 0 2024-07-26T11:04:40.157 INFO:tasks.workunit.client.0.smithi144.stdout:5/335: read - d1/d9/f5a zero size 2024-07-26T11:04:40.159 INFO:tasks.workunit.client.0.smithi144.stdout:1/278: dwrite d3/d6/f32 [4194304,4194304] 0 2024-07-26T11:04:40.159 INFO:tasks.workunit.client.0.smithi144.stdout:5/336: creat d1/d9/d5b/d4d/f79 x:0 0 0 2024-07-26T11:04:40.159 INFO:tasks.workunit.client.0.smithi144.stdout:9/268: dwrite d9/de/d14/f2c [0,4194304] 0 2024-07-26T11:04:40.162 INFO:tasks.workunit.client.0.smithi144.stdout:1/279: creat d3/d53/f54 x:0 0 0 2024-07-26T11:04:40.163 INFO:tasks.workunit.client.0.smithi144.stdout:1/280: link d3/d6/da/d21/f51 d3/d6/da/d13/d19/d27/d36/f55 0 2024-07-26T11:04:40.167 INFO:tasks.workunit.client.0.smithi144.stdout:2/218: truncate d0/f1e 193390 0 2024-07-26T11:04:40.169 INFO:tasks.workunit.client.0.smithi144.stdout:2/219: mknod d0/d6/c3f 0 2024-07-26T11:04:40.171 INFO:tasks.workunit.client.0.smithi144.stdout:2/220: creat d0/d15/f40 x:0 0 0 2024-07-26T11:04:40.175 INFO:tasks.workunit.client.0.smithi144.stdout:2/221: rename d0/d15/d20/f30 to d0/d6/f41 0 2024-07-26T11:04:40.180 INFO:tasks.workunit.client.0.smithi144.stdout:6/301: dwrite d8/d15/f21 [0,4194304] 0 2024-07-26T11:04:40.195 INFO:tasks.workunit.client.0.smithi144.stdout:1/281: dread d3/d6/da/d13/d19/d3a/f4b [0,4194304] 0 2024-07-26T11:04:40.195 INFO:tasks.workunit.client.0.smithi144.stdout:9/269: dread d9/de/d14/f24 [0,4194304] 0 2024-07-26T11:04:40.195 INFO:tasks.workunit.client.0.smithi144.stdout:9/270: chown d9/de/f2f 156176002 1 2024-07-26T11:04:40.195 INFO:tasks.workunit.client.0.smithi144.stdout:9/271: truncate d9/de/f3d 2024461 0 2024-07-26T11:04:40.197 INFO:tasks.workunit.client.0.smithi144.stdout:9/272: rename d9/de/l54 to d9/de/d14/d3f/l5c 0 2024-07-26T11:04:40.197 INFO:tasks.workunit.client.0.smithi144.stdout:6/302: mknod d8/d4f/d52/d58/c6f 0 2024-07-26T11:04:40.198 INFO:tasks.workunit.client.0.smithi144.stdout:6/303: unlink d8/f17 0 2024-07-26T11:04:40.199 INFO:tasks.workunit.client.0.smithi144.stdout:6/304: rmdir d8/d15 39 2024-07-26T11:04:40.199 INFO:tasks.workunit.client.0.smithi144.stdout:6/305: truncate d8/f9 4533652 0 2024-07-26T11:04:40.199 INFO:tasks.workunit.client.0.smithi144.stdout:6/306: write d8/d42/f43 [2684618,78359] 0 2024-07-26T11:04:40.200 INFO:tasks.workunit.client.0.smithi144.stdout:6/307: mknod d8/d15/d26/c70 0 2024-07-26T11:04:40.204 INFO:tasks.workunit.client.0.smithi144.stdout:8/265: dread d1a/d1c/d30/f39 [0,4194304] 0 2024-07-26T11:04:40.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/188: sync 2024-07-26T11:04:40.207 INFO:tasks.workunit.client.0.smithi144.stdout:5/337: write d1/d9/d5b/f21 [763667,48220] 0 2024-07-26T11:04:40.207 INFO:tasks.workunit.client.0.smithi144.stdout:5/338: fdatasync d1/d18/f51 0 2024-07-26T11:04:40.207 INFO:tasks.workunit.client.0.smithi144.stdout:7/250: sync 2024-07-26T11:04:40.207 INFO:tasks.workunit.client.0.smithi144.stdout:7/251: stat d0/d6/d9/d25/d2f/d47/l5b 0 2024-07-26T11:04:40.207 INFO:tasks.workunit.client.0.smithi144.stdout:7/252: chown d0/d6/d9/d25/d2f 0 1 2024-07-26T11:04:40.207 INFO:tasks.workunit.client.0.smithi144.stdout:2/222: write d0/d6/f1d [243933,98696] 0 2024-07-26T11:04:40.209 INFO:tasks.workunit.client.0.smithi144.stdout:5/339: rename d1/d9/d44/l38 to d1/l7a 0 2024-07-26T11:04:40.209 INFO:tasks.workunit.client.0.smithi144.stdout:7/253: mkdir d0/d6/d9/d25/d67 0 2024-07-26T11:04:40.212 INFO:tasks.workunit.client.0.smithi144.stdout:5/340: mknod d1/d9/d5b/c7b 0 2024-07-26T11:04:40.212 INFO:tasks.workunit.client.0.smithi144.stdout:5/341: write d1/d5/f78 [1981129,46257] 0 2024-07-26T11:04:40.216 INFO:tasks.workunit.client.0.smithi144.stdout:7/254: mkdir d0/d6/d9/d25/d2f/d68 0 2024-07-26T11:04:40.217 INFO:tasks.workunit.client.0.smithi144.stdout:5/342: getdents d1/d18 0 2024-07-26T11:04:40.217 INFO:tasks.workunit.client.0.smithi144.stdout:5/343: symlink d1/d18/d24/d27/d71/l7c 0 2024-07-26T11:04:40.217 INFO:tasks.workunit.client.0.smithi144.stdout:5/344: truncate d1/d9/f19 1714099 0 2024-07-26T11:04:40.222 INFO:tasks.workunit.client.0.smithi144.stdout:5/345: unlink d1/d18/f1d 0 2024-07-26T11:04:40.222 INFO:tasks.workunit.client.0.smithi144.stdout:5/346: truncate d1/d18/d24/d27/f4f 4821890 0 2024-07-26T11:04:40.222 INFO:tasks.workunit.client.0.smithi144.stdout:5/347: chown d1/d18/d24/l75 1391253 1 2024-07-26T11:04:40.222 INFO:tasks.workunit.client.0.smithi144.stdout:8/266: dread f4 [0,4194304] 0 2024-07-26T11:04:40.222 INFO:tasks.workunit.client.0.smithi144.stdout:5/348: creat d1/d9/d5b/f7d x:0 0 0 2024-07-26T11:04:40.228 INFO:tasks.workunit.client.0.smithi144.stdout:1/282: dread d3/f39 [4194304,4194304] 0 2024-07-26T11:04:40.229 INFO:tasks.workunit.client.0.smithi144.stdout:1/283: creat d3/d6/da/f56 x:0 0 0 2024-07-26T11:04:40.231 INFO:tasks.workunit.client.0.smithi144.stdout:1/284: symlink d3/d6/da/d13/d19/d3a/d3e/l57 0 2024-07-26T11:04:40.231 INFO:tasks.workunit.client.0.smithi144.stdout:1/285: write d3/d6/da/d21/f4d [99885,36766] 0 2024-07-26T11:04:40.237 INFO:tasks.workunit.client.0.smithi144.stdout:1/286: rmdir d3/d6/da/d13/d19/d3a/d3e 39 2024-07-26T11:04:40.255 INFO:tasks.workunit.client.0.smithi144.stdout:0/322: dwrite d0/d25/d5e/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:04:40.255 INFO:tasks.workunit.client.0.smithi144.stdout:9/273: dread d9/de/d14/f2c [0,4194304] 0 2024-07-26T11:04:40.262 INFO:tasks.workunit.client.0.smithi144.stdout:9/274: mkdir d9/d3b/d5d 0 2024-07-26T11:04:40.262 INFO:tasks.workunit.client.0.smithi144.stdout:9/275: chown d9/de/f21 3618730 1 2024-07-26T11:04:40.265 INFO:tasks.workunit.client.0.smithi144.stdout:3/189: read d1/ff [157731,19001] 0 2024-07-26T11:04:40.272 INFO:tasks.workunit.client.0.smithi144.stdout:4/254: sync 2024-07-26T11:04:40.273 INFO:tasks.workunit.client.0.smithi144.stdout:4/255: mknod d9/d57/c5a 0 2024-07-26T11:04:40.273 INFO:tasks.workunit.client.0.smithi144.stdout:4/256: chown d9/c10 8314011 1 2024-07-26T11:04:40.273 INFO:tasks.workunit.client.0.smithi144.stdout:4/257: write d9/f51 [264312,107549] 0 2024-07-26T11:04:40.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/258: unlink d9/d26/d3c/f4b 0 2024-07-26T11:04:40.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/259: write d9/f51 [1293383,64012] 0 2024-07-26T11:04:40.276 INFO:tasks.workunit.client.0.smithi144.stdout:4/260: unlink d9/d26/f3f 0 2024-07-26T11:04:40.280 INFO:tasks.workunit.client.0.smithi144.stdout:8/267: dwrite d1a/d1c/d28/d2c/f4d [0,4194304] 0 2024-07-26T11:04:40.285 INFO:tasks.workunit.client.0.smithi144.stdout:8/268: rmdir d1a/d32 39 2024-07-26T11:04:40.289 INFO:tasks.workunit.client.0.smithi144.stdout:3/190: dread d1/d5/d25/d26/f27 [0,4194304] 0 2024-07-26T11:04:40.290 INFO:tasks.workunit.client.0.smithi144.stdout:3/191: truncate d1/d5/f6 1550183 0 2024-07-26T11:04:40.291 INFO:tasks.workunit.client.0.smithi144.stdout:3/192: truncate d1/d5/f24 313380 0 2024-07-26T11:04:40.292 INFO:tasks.workunit.client.0.smithi144.stdout:3/193: stat d1/d5/l1f 0 2024-07-26T11:04:40.308 INFO:tasks.workunit.client.0.smithi144.stdout:8/269: dread d1a/f20 [0,4194304] 0 2024-07-26T11:04:40.308 INFO:tasks.workunit.client.0.smithi144.stdout:8/270: fsync d1a/d1c/f24 0 2024-07-26T11:04:40.310 INFO:tasks.workunit.client.0.smithi144.stdout:8/271: rename d1a/d1c/f27 to d1a/d1c/d28/d2c/d38/f5a 0 2024-07-26T11:04:40.314 INFO:tasks.workunit.client.0.smithi144.stdout:9/276: write d9/fa [1434259,31323] 0 2024-07-26T11:04:40.319 INFO:tasks.workunit.client.0.smithi144.stdout:1/287: dwrite d3/d6/f18 [0,4194304] 0 2024-07-26T11:04:40.320 INFO:tasks.workunit.client.0.smithi144.stdout:1/288: write d3/d6/da/d13/f26 [407194,63993] 0 2024-07-26T11:04:40.320 INFO:tasks.workunit.client.0.smithi144.stdout:9/277: mkdir d9/de/d14/d3f/d5e 0 2024-07-26T11:04:40.322 INFO:tasks.workunit.client.0.smithi144.stdout:9/278: rename d9/c26 to d9/de/d14/d23/d39/c5f 0 2024-07-26T11:04:40.322 INFO:tasks.workunit.client.0.smithi144.stdout:9/279: chown d9/de/l4c 54 1 2024-07-26T11:04:40.324 INFO:tasks.workunit.client.0.smithi144.stdout:9/280: symlink d9/de/d14/d3f/l60 0 2024-07-26T11:04:40.327 INFO:tasks.workunit.client.0.smithi144.stdout:0/323: dwrite d0/d3e/f69 [0,4194304] 0 2024-07-26T11:04:40.327 INFO:tasks.workunit.client.0.smithi144.stdout:0/324: dread - d0/d3e/d45/f50 zero size 2024-07-26T11:04:40.327 INFO:tasks.workunit.client.0.smithi144.stdout:0/325: dread - d0/d25/d5e/d29/f6e zero size 2024-07-26T11:04:40.327 INFO:tasks.workunit.client.0.smithi144.stdout:6/308: dwrite d8/d4f/f51 [0,4194304] 0 2024-07-26T11:04:40.327 INFO:tasks.workunit.client.0.smithi144.stdout:6/309: chown d8/f1d 95743597 1 2024-07-26T11:04:40.327 INFO:tasks.workunit.client.0.smithi144.stdout:7/255: dwrite d0/f2d [4194304,4194304] 0 2024-07-26T11:04:40.339 INFO:tasks.workunit.client.0.smithi144.stdout:6/310: truncate d8/db/f55 729666 0 2024-07-26T11:04:40.339 INFO:tasks.workunit.client.0.smithi144.stdout:9/281: creat d9/de/d14/d3f/f61 x:0 0 0 2024-07-26T11:04:40.339 INFO:tasks.workunit.client.0.smithi144.stdout:9/282: write d9/de/f4b [2050881,123055] 0 2024-07-26T11:04:40.339 INFO:tasks.workunit.client.0.smithi144.stdout:9/283: dread - d9/d3b/f56 zero size 2024-07-26T11:04:40.340 INFO:tasks.workunit.client.0.smithi144.stdout:9/284: truncate d9/de/f15 65926 0 2024-07-26T11:04:40.343 INFO:tasks.workunit.client.0.smithi144.stdout:2/223: dread d0/d6/f1d [0,4194304] 0 2024-07-26T11:04:40.354 INFO:tasks.workunit.client.0.smithi144.stdout:7/256: rename d0/f7 to d0/d6/d9/d25/d2f/d47/d5e/f69 0 2024-07-26T11:04:40.354 INFO:tasks.workunit.client.0.smithi144.stdout:7/257: creat d0/d6/d55/f6a x:0 0 0 2024-07-26T11:04:40.354 INFO:tasks.workunit.client.0.smithi144.stdout:9/285: creat d9/de/d14/d3f/d5e/f62 x:0 0 0 2024-07-26T11:04:40.357 INFO:tasks.workunit.client.0.smithi144.stdout:7/258: readlink d0/d6/d9/l50 0 2024-07-26T11:04:40.357 INFO:tasks.workunit.client.0.smithi144.stdout:7/259: chown d0/d6/d9/d25/d2f/d47/l5b 1 1 2024-07-26T11:04:40.373 INFO:tasks.workunit.client.0.smithi144.stdout:3/194: dwrite d1/d5/f6 [0,4194304] 0 2024-07-26T11:04:40.374 INFO:tasks.workunit.client.0.smithi144.stdout:3/195: write d1/ff [3363628,98161] 0 2024-07-26T11:04:40.381 INFO:tasks.workunit.client.0.smithi144.stdout:8/272: write d1a/d1c/d30/f40 [320211,119344] 0 2024-07-26T11:04:40.381 INFO:tasks.workunit.client.0.smithi144.stdout:3/196: creat d1/f32 x:0 0 0 2024-07-26T11:04:40.381 INFO:tasks.workunit.client.0.smithi144.stdout:3/197: stat d1/d5/d25/d26 0 2024-07-26T11:04:40.398 INFO:tasks.workunit.client.0.smithi144.stdout:0/326: dwrite d0/f10 [0,4194304] 0 2024-07-26T11:04:40.400 INFO:tasks.workunit.client.0.smithi144.stdout:0/327: creat d0/d3e/f71 x:0 0 0 2024-07-26T11:04:40.407 INFO:tasks.workunit.client.0.smithi144.stdout:3/198: dread d1/d5/f24 [0,4194304] 0 2024-07-26T11:04:40.416 INFO:tasks.workunit.client.0.smithi144.stdout:6/311: dread d8/f9 [0,4194304] 0 2024-07-26T11:04:40.416 INFO:tasks.workunit.client.0.smithi144.stdout:6/312: chown d8/d15/d26/f2e 18650 1 2024-07-26T11:04:40.420 INFO:tasks.workunit.client.0.smithi144.stdout:9/286: dwrite d9/de/f21 [0,4194304] 0 2024-07-26T11:04:40.420 INFO:tasks.workunit.client.0.smithi144.stdout:1/289: dwrite d3/d6/da/d13/d19/d3a/d3e/f50 [0,4194304] 0 2024-07-26T11:04:40.423 INFO:tasks.workunit.client.0.smithi144.stdout:6/313: mknod d8/d4f/c71 0 2024-07-26T11:04:40.424 INFO:tasks.workunit.client.0.smithi144.stdout:1/290: truncate d3/d6/da/f2f 3629433 0 2024-07-26T11:04:40.424 INFO:tasks.workunit.client.0.smithi144.stdout:1/291: write d3/d6/da/d21/f51 [135954,91957] 0 2024-07-26T11:04:40.424 INFO:tasks.workunit.client.0.smithi144.stdout:1/292: stat d3/d6/da/d13/d19/d3a/d3e 0 2024-07-26T11:04:40.424 INFO:tasks.workunit.client.0.smithi144.stdout:1/293: truncate d3/d6/da/d13/d22/f40 50028 0 2024-07-26T11:04:40.425 INFO:tasks.workunit.client.0.smithi144.stdout:6/314: creat d8/d4f/f72 x:0 0 0 2024-07-26T11:04:40.434 INFO:tasks.workunit.client.0.smithi144.stdout:2/224: write d0/d6/d9/f26 [2656944,13633] 0 2024-07-26T11:04:40.438 INFO:tasks.workunit.client.0.smithi144.stdout:5/349: dwrite d1/d9/d5b/d4d/f63 [0,4194304] 0 2024-07-26T11:04:40.438 INFO:tasks.workunit.client.0.smithi144.stdout:5/350: chown d1/d9/l23 6 1 2024-07-26T11:04:40.440 INFO:tasks.workunit.client.0.smithi144.stdout:5/351: rename d1/d9/l23 to d1/d9/d5b/l7e 0 2024-07-26T11:04:40.448 INFO:tasks.workunit.client.0.smithi144.stdout:5/352: creat d1/d5/f7f x:0 0 0 2024-07-26T11:04:40.465 INFO:tasks.workunit.client.0.smithi144.stdout:4/261: dwrite d9/f1f [0,4194304] 0 2024-07-26T11:04:40.465 INFO:tasks.workunit.client.0.smithi144.stdout:4/262: dread - d9/da/f52 zero size 2024-07-26T11:04:40.468 INFO:tasks.workunit.client.0.smithi144.stdout:1/294: write d3/d6/da/f2f [3714354,39276] 0 2024-07-26T11:04:40.468 INFO:tasks.workunit.client.0.smithi144.stdout:1/295: dread - d3/f4f zero size 2024-07-26T11:04:40.468 INFO:tasks.workunit.client.0.smithi144.stdout:1/296: chown d3/d6 8168 1 2024-07-26T11:04:40.493 INFO:tasks.workunit.client.0.smithi144.stdout:4/263: creat d9/da/f5b x:0 0 0 2024-07-26T11:04:40.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/297: mkdir d3/d3d/d58 0 2024-07-26T11:04:40.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/298: stat d3/f2d 0 2024-07-26T11:04:40.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/299: chown d3/d6/da/d13/d19/d3a/f4b 144724013 1 2024-07-26T11:04:40.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/300: readlink d3/d6/l15 0 2024-07-26T11:04:40.496 INFO:tasks.workunit.client.0.smithi144.stdout:1/301: mknod d3/d6/da/d13/d22/c59 0 2024-07-26T11:04:40.497 INFO:tasks.workunit.client.0.smithi144.stdout:8/273: dwrite d1a/f1b [0,4194304] 0 2024-07-26T11:04:40.500 INFO:tasks.workunit.client.0.smithi144.stdout:1/302: rename d3/d6/da/d13/f29 to d3/d6/da/d13/d19/d27/f5a 0 2024-07-26T11:04:40.510 INFO:tasks.workunit.client.0.smithi144.stdout:7/260: dwrite d0/d6/d9/d12/f5d [0,4194304] 0 2024-07-26T11:04:40.516 INFO:tasks.workunit.client.0.smithi144.stdout:4/264: write d9/d26/f31 [4112314,124095] 0 2024-07-26T11:04:40.516 INFO:tasks.workunit.client.0.smithi144.stdout:1/303: getdents d3/d6/da/d13/d22 0 2024-07-26T11:04:40.516 INFO:tasks.workunit.client.0.smithi144.stdout:1/304: fdatasync d3/f3b 0 2024-07-26T11:04:40.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/265: getdents d9/d26/d48 0 2024-07-26T11:04:40.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/266: chown d9/d26/d3c 576621 1 2024-07-26T11:04:40.522 INFO:tasks.workunit.client.0.smithi144.stdout:8/274: dread d1a/f31 [0,4194304] 0 2024-07-26T11:04:40.523 INFO:tasks.workunit.client.0.smithi144.stdout:7/261: creat d0/d6/d9/d64/f6b x:0 0 0 2024-07-26T11:04:40.524 INFO:tasks.workunit.client.0.smithi144.stdout:2/225: dwrite d0/f3e [0,4194304] 0 2024-07-26T11:04:40.525 INFO:tasks.workunit.client.0.smithi144.stdout:1/305: mkdir d3/d6/da/d5b 0 2024-07-26T11:04:40.526 INFO:tasks.workunit.client.0.smithi144.stdout:4/267: mknod d9/d26/d48/d50/c5c 0 2024-07-26T11:04:40.527 INFO:tasks.workunit.client.0.smithi144.stdout:2/226: rmdir d0/d24/d2b 39 2024-07-26T11:04:40.528 INFO:tasks.workunit.client.0.smithi144.stdout:1/306: mkdir d3/d6/da/d21/d5c 0 2024-07-26T11:04:40.528 INFO:tasks.workunit.client.0.smithi144.stdout:1/307: truncate d3/d6/da/f56 707634 0 2024-07-26T11:04:40.528 INFO:tasks.workunit.client.0.smithi144.stdout:0/328: dwrite d0/f10 [0,4194304] 0 2024-07-26T11:04:40.529 INFO:tasks.workunit.client.0.smithi144.stdout:4/268: link d9/d26/d3c/c54 d9/da/c5d 0 2024-07-26T11:04:40.529 INFO:tasks.workunit.client.0.smithi144.stdout:6/315: dread d8/d4f/f63 [0,4194304] 0 2024-07-26T11:04:40.529 INFO:tasks.workunit.client.0.smithi144.stdout:4/269: fdatasync d9/da/f39 0 2024-07-26T11:04:40.529 INFO:tasks.workunit.client.0.smithi144.stdout:4/270: dread - d9/d26/f28 zero size 2024-07-26T11:04:40.529 INFO:tasks.workunit.client.0.smithi144.stdout:8/275: link l13 d1a/l5b 0 2024-07-26T11:04:40.530 INFO:tasks.workunit.client.0.smithi144.stdout:2/227: symlink d0/d15/l42 0 2024-07-26T11:04:40.543 INFO:tasks.workunit.client.0.smithi144.stdout:1/308: mkdir d3/d6/da/d13/d22/d5d 0 2024-07-26T11:04:40.543 INFO:tasks.workunit.client.0.smithi144.stdout:1/309: chown d3/d6/f47 1006005 1 2024-07-26T11:04:40.543 INFO:tasks.workunit.client.0.smithi144.stdout:8/276: mknod d1a/d1c/d28/c5c 0 2024-07-26T11:04:40.544 INFO:tasks.workunit.client.0.smithi144.stdout:4/271: mknod d9/da/c5e 0 2024-07-26T11:04:40.544 INFO:tasks.workunit.client.0.smithi144.stdout:4/272: read - d9/d26/d48/f4d zero size 2024-07-26T11:04:40.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/228: unlink d0/d6/c36 0 2024-07-26T11:04:40.544 INFO:tasks.workunit.client.0.smithi144.stdout:5/353: dwrite d1/d18/d24/d35/f3d [0,4194304] 0 2024-07-26T11:04:40.544 INFO:tasks.workunit.client.0.smithi144.stdout:4/273: write d9/d26/f4c [1196231,109758] 0 2024-07-26T11:04:40.544 INFO:tasks.workunit.client.0.smithi144.stdout:4/274: fsync d9/da/f52 0 2024-07-26T11:04:40.552 INFO:tasks.workunit.client.0.smithi144.stdout:9/287: dwrite d9/de/f4b [0,4194304] 0 2024-07-26T11:04:40.552 INFO:tasks.workunit.client.0.smithi144.stdout:9/288: chown d9/de/d14/d23/f47 41 1 2024-07-26T11:04:40.556 INFO:tasks.workunit.client.0.smithi144.stdout:8/277: link c19 d1a/d32/d50/c5d 0 2024-07-26T11:04:40.558 INFO:tasks.workunit.client.0.smithi144.stdout:4/275: getdents d9 0 2024-07-26T11:04:40.559 INFO:tasks.workunit.client.0.smithi144.stdout:5/354: creat d1/d18/d24/d27/d71/f80 x:0 0 0 2024-07-26T11:04:40.559 INFO:tasks.workunit.client.0.smithi144.stdout:4/276: write f1 [4103838,19691] 0 2024-07-26T11:04:40.560 INFO:tasks.workunit.client.0.smithi144.stdout:2/229: truncate d0/d6/f39 584252 0 2024-07-26T11:04:40.560 INFO:tasks.workunit.client.0.smithi144.stdout:4/277: truncate d9/d26/d3c/f42 364008 0 2024-07-26T11:04:40.565 INFO:tasks.workunit.client.0.smithi144.stdout:9/289: symlink d9/de/d14/d23/d39/l63 0 2024-07-26T11:04:40.565 INFO:tasks.workunit.client.0.smithi144.stdout:5/355: rmdir d1/d9 39 2024-07-26T11:04:40.565 INFO:tasks.workunit.client.0.smithi144.stdout:5/356: dread - d1/d5/f6d zero size 2024-07-26T11:04:40.566 INFO:tasks.workunit.client.0.smithi144.stdout:2/230: creat d0/d6/d34/f43 x:0 0 0 2024-07-26T11:04:40.566 INFO:tasks.workunit.client.0.smithi144.stdout:2/231: fsync d0/f3d 0 2024-07-26T11:04:40.566 INFO:tasks.workunit.client.0.smithi144.stdout:6/316: write d8/d4f/d52/f5a [2689456,100504] 0 2024-07-26T11:04:40.567 INFO:tasks.workunit.client.0.smithi144.stdout:4/278: mknod d9/d57/c5f 0 2024-07-26T11:04:40.568 INFO:tasks.workunit.client.0.smithi144.stdout:4/279: write f1 [4433042,8566] 0 2024-07-26T11:04:40.568 INFO:tasks.workunit.client.0.smithi144.stdout:5/357: unlink d1/d9/d5b/d4d/c59 0 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:4/280: mkdir d9/d26/d48/d50/d60 0 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:5/358: mkdir d1/d9/d44/d45/d81 0 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:6/317: rename d8/db/f1b to d8/d4f/d5c/f73 0 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:6/318: chown d8/d42/f43 41522 1 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:5/359: truncate d1/d9/d5b/f7d 355197 0 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:6/319: dread - d8/db/f27 zero size 2024-07-26T11:04:40.579 INFO:tasks.workunit.client.0.smithi144.stdout:6/320: chown d8/d15/c18 4439686 1 2024-07-26T11:04:40.580 INFO:tasks.workunit.client.0.smithi144.stdout:2/232: mkdir d0/d6/d9/d44 0 2024-07-26T11:04:40.580 INFO:tasks.workunit.client.0.smithi144.stdout:2/233: readlink d0/d15/d20/l2a 0 2024-07-26T11:04:40.586 INFO:tasks.workunit.client.0.smithi144.stdout:2/234: truncate d0/d6/d9/f26 1403884 0 2024-07-26T11:04:40.586 INFO:tasks.workunit.client.0.smithi144.stdout:6/321: rmdir d8/d15/d57/d5f 39 2024-07-26T11:04:40.590 INFO:tasks.workunit.client.0.smithi144.stdout:4/281: rename d9/l13 to d9/d26/d48/d50/l61 0 2024-07-26T11:04:40.590 INFO:tasks.workunit.client.0.smithi144.stdout:4/282: write d9/d26/d3c/f42 [924394,97014] 0 2024-07-26T11:04:40.590 INFO:tasks.workunit.client.0.smithi144.stdout:4/283: fdatasync d9/d26/f30 0 2024-07-26T11:04:40.590 INFO:tasks.workunit.client.0.smithi144.stdout:7/262: dwrite d0/d6/d9/d12/f5d [0,4194304] 0 2024-07-26T11:04:40.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/360: link d1/d18/f60 d1/d18/d24/f82 0 2024-07-26T11:04:40.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/361: dread - d1/d9/f5a zero size 2024-07-26T11:04:40.591 INFO:tasks.workunit.client.0.smithi144.stdout:6/322: mkdir d8/d4f/d74 0 2024-07-26T11:04:40.591 INFO:tasks.workunit.client.0.smithi144.stdout:6/323: truncate d8/f53 451195 0 2024-07-26T11:04:40.592 INFO:tasks.workunit.client.0.smithi144.stdout:2/235: link d0/d15/d20/l2a d0/d24/d2b/l45 0 2024-07-26T11:04:40.593 INFO:tasks.workunit.client.0.smithi144.stdout:7/263: mkdir d0/d6/d9/d12/d6c 0 2024-07-26T11:04:40.597 INFO:tasks.workunit.client.0.smithi144.stdout:5/362: mknod d1/c83 0 2024-07-26T11:04:40.599 INFO:tasks.workunit.client.0.smithi144.stdout:0/329: dwrite d0/d3e/f71 [0,4194304] 0 2024-07-26T11:04:40.601 INFO:tasks.workunit.client.0.smithi144.stdout:1/310: dwrite d3/d6/da/f2f [0,4194304] 0 2024-07-26T11:04:40.601 INFO:tasks.workunit.client.0.smithi144.stdout:6/324: mknod d8/c75 0 2024-07-26T11:04:40.601 INFO:tasks.workunit.client.0.smithi144.stdout:2/236: mkdir d0/d24/d2b/d46 0 2024-07-26T11:04:40.602 INFO:tasks.workunit.client.0.smithi144.stdout:7/264: rename d0/d6/d9/d45/f22 to d0/d6/d9/d12/f6d 0 2024-07-26T11:04:40.608 INFO:tasks.workunit.client.0.smithi144.stdout:1/311: unlink d3/d6/l38 0 2024-07-26T11:04:40.608 INFO:tasks.workunit.client.0.smithi144.stdout:1/312: fsync d3/d6/f47 0 2024-07-26T11:04:40.608 INFO:tasks.workunit.client.0.smithi144.stdout:0/330: mknod d0/d25/d5e/d1e/d30/d6a/d6f/c72 0 2024-07-26T11:04:40.610 INFO:tasks.workunit.client.0.smithi144.stdout:2/237: creat d0/d15/d20/f47 x:0 0 0 2024-07-26T11:04:40.612 INFO:tasks.workunit.client.0.smithi144.stdout:5/363: rename d1/d9/f72 to d1/f84 0 2024-07-26T11:04:40.615 INFO:tasks.workunit.client.0.smithi144.stdout:0/331: creat d0/d3e/d45/f73 x:0 0 0 2024-07-26T11:04:40.615 INFO:tasks.workunit.client.0.smithi144.stdout:1/313: creat d3/d6/da/d21/d5c/f5e x:0 0 0 2024-07-26T11:04:40.616 INFO:tasks.workunit.client.0.smithi144.stdout:5/364: mkdir d1/d9/d44/d85 0 2024-07-26T11:04:40.616 INFO:tasks.workunit.client.0.smithi144.stdout:2/238: fdatasync d0/d6/f41 0 2024-07-26T11:04:40.616 INFO:tasks.workunit.client.0.smithi144.stdout:5/365: stat d1/d9/d5b/d4d/l28 0 2024-07-26T11:04:40.617 INFO:tasks.workunit.client.0.smithi144.stdout:1/314: mknod d3/d6/da/d13/d19/d3a/d3e/c5f 0 2024-07-26T11:04:40.621 INFO:tasks.workunit.client.0.smithi144.stdout:5/366: mkdir d1/d9/d5b/d86 0 2024-07-26T11:04:40.621 INFO:tasks.workunit.client.0.smithi144.stdout:5/367: chown d1/c2 149 1 2024-07-26T11:04:40.623 INFO:tasks.workunit.client.0.smithi144.stdout:5/368: mkdir d1/d9/d44/d87 0 2024-07-26T11:04:40.623 INFO:tasks.workunit.client.0.smithi144.stdout:5/369: rename d1/d18/d24 to d1/d18/d24/d88 22 2024-07-26T11:04:40.623 INFO:tasks.workunit.client.0.smithi144.stdout:5/370: write d1/d5/f7f [734959,70075] 0 2024-07-26T11:04:40.625 INFO:tasks.workunit.client.0.smithi144.stdout:1/315: dread d3/d6/da/d21/f4d [0,4194304] 0 2024-07-26T11:04:40.625 INFO:tasks.workunit.client.0.smithi144.stdout:1/316: fdatasync d3/f3b 0 2024-07-26T11:04:40.630 INFO:tasks.workunit.client.0.smithi144.stdout:1/317: mknod d3/d6/da/d13/c60 0 2024-07-26T11:04:40.633 INFO:tasks.workunit.client.0.smithi144.stdout:1/318: write d3/d6/da/d21/d5c/f5e [794202,31965] 0 2024-07-26T11:04:40.633 INFO:tasks.workunit.client.0.smithi144.stdout:1/319: dread - d3/f4f zero size 2024-07-26T11:04:40.644 INFO:tasks.workunit.client.0.smithi144.stdout:8/278: dwrite d1a/d1c/f1f [0,4194304] 0 2024-07-26T11:04:40.645 INFO:tasks.workunit.client.0.smithi144.stdout:8/279: creat d1a/d1c/d30/f5e x:0 0 0 2024-07-26T11:04:40.645 INFO:tasks.workunit.client.0.smithi144.stdout:8/280: fdatasync d1a/d1c/d28/f2e 0 2024-07-26T11:04:40.650 INFO:tasks.workunit.client.0.smithi144.stdout:8/281: creat d1a/d1c/d28/d2c/d38/f5f x:0 0 0 2024-07-26T11:04:40.650 INFO:tasks.workunit.client.0.smithi144.stdout:8/282: mknod d1a/d1c/d30/c60 0 2024-07-26T11:04:40.676 INFO:tasks.workunit.client.0.smithi144.stdout:5/371: write d1/d9/d5b/d4d/f3e [2991913,32665] 0 2024-07-26T11:04:40.676 INFO:tasks.workunit.client.0.smithi144.stdout:6/325: dwrite d8/fd [4194304,4194304] 0 2024-07-26T11:04:40.701 INFO:tasks.workunit.client.0.smithi144.stdout:6/326: dread d8/db/f32 [0,4194304] 0 2024-07-26T11:04:40.701 INFO:tasks.workunit.client.0.smithi144.stdout:2/239: dread d0/f1 [0,4194304] 0 2024-07-26T11:04:40.702 INFO:tasks.workunit.client.0.smithi144.stdout:9/290: dwrite d9/de/d14/d23/f31 [0,4194304] 0 2024-07-26T11:04:40.703 INFO:tasks.workunit.client.0.smithi144.stdout:6/327: truncate d8/d4f/f51 3510794 0 2024-07-26T11:04:40.704 INFO:tasks.workunit.client.0.smithi144.stdout:2/240: mkdir d0/d24/d48 0 2024-07-26T11:04:40.704 INFO:tasks.workunit.client.0.smithi144.stdout:2/241: chown d0/d24/d2b/d46 1921321 1 2024-07-26T11:04:40.708 INFO:tasks.workunit.client.0.smithi144.stdout:9/291: link d9/de/f21 d9/d3b/f64 0 2024-07-26T11:04:40.708 INFO:tasks.workunit.client.0.smithi144.stdout:9/292: chown d9/de/d14/l1b 98168677 1 2024-07-26T11:04:40.708 INFO:tasks.workunit.client.0.smithi144.stdout:7/265: dwrite d0/d6/d9/d12/f6d [0,4194304] 0 2024-07-26T11:04:40.708 INFO:tasks.workunit.client.0.smithi144.stdout:7/266: write d0/d6/f56 [743312,70520] 0 2024-07-26T11:04:40.709 INFO:tasks.workunit.client.0.smithi144.stdout:6/328: mkdir d8/d4f/d52/d76 0 2024-07-26T11:04:40.709 INFO:tasks.workunit.client.0.smithi144.stdout:2/242: link d0/c1b d0/d15/d20/c49 0 2024-07-26T11:04:40.711 INFO:tasks.workunit.client.0.smithi144.stdout:8/283: dwrite d1a/d1c/d28/d2c/d38/f5f [0,4194304] 0 2024-07-26T11:04:40.711 INFO:tasks.workunit.client.0.smithi144.stdout:2/243: write d0/d6/d9/f26 [162603,103633] 0 2024-07-26T11:04:40.713 INFO:tasks.workunit.client.0.smithi144.stdout:1/320: dwrite d3/d6/da/d13/d19/d3a/f4b [0,4194304] 0 2024-07-26T11:04:40.714 INFO:tasks.workunit.client.0.smithi144.stdout:6/329: unlink d8/d15/l54 0 2024-07-26T11:04:40.714 INFO:tasks.workunit.client.0.smithi144.stdout:7/267: creat d0/d6/d9/d3f/f6e x:0 0 0 2024-07-26T11:04:40.717 INFO:tasks.workunit.client.0.smithi144.stdout:0/332: dwrite d0/d25/d5e/d1e/f36 [4194304,4194304] 0 2024-07-26T11:04:40.717 INFO:tasks.workunit.client.0.smithi144.stdout:0/333: write d0/d3e/d45/f55 [773407,113677] 0 2024-07-26T11:04:40.717 INFO:tasks.workunit.client.0.smithi144.stdout:0/334: write d0/f8 [505413,38053] 0 2024-07-26T11:04:40.717 INFO:tasks.workunit.client.0.smithi144.stdout:8/284: mknod d1a/d32/d50/c61 0 2024-07-26T11:04:40.718 INFO:tasks.workunit.client.0.smithi144.stdout:2/244: mknod d0/d15/d20/d28/c4a 0 2024-07-26T11:04:40.718 INFO:tasks.workunit.client.0.smithi144.stdout:6/330: mkdir d8/d4f/d5c/d77 0 2024-07-26T11:04:40.718 INFO:tasks.workunit.client.0.smithi144.stdout:6/331: stat d8/d15/f39 0 2024-07-26T11:04:40.719 INFO:tasks.workunit.client.0.smithi144.stdout:7/268: mknod d0/d6/d9/d45/c6f 0 2024-07-26T11:04:40.719 INFO:tasks.workunit.client.0.smithi144.stdout:7/269: chown d0/f4b 87387 1 2024-07-26T11:04:40.719 INFO:tasks.workunit.client.0.smithi144.stdout:7/270: dread - d0/d6/d9/d3f/f58 zero size 2024-07-26T11:04:40.719 INFO:tasks.workunit.client.0.smithi144.stdout:0/335: symlink d0/d3e/d45/d4c/l74 0 2024-07-26T11:04:40.720 INFO:tasks.workunit.client.0.smithi144.stdout:8/285: creat d1a/d32/f62 x:0 0 0 2024-07-26T11:04:40.722 INFO:tasks.workunit.client.0.smithi144.stdout:6/332: mknod d8/d4f/d52/d58/c78 0 2024-07-26T11:04:40.722 INFO:tasks.workunit.client.0.smithi144.stdout:7/271: creat d0/d6/d9/d25/d2f/f70 x:0 0 0 2024-07-26T11:04:40.727 INFO:tasks.workunit.client.0.smithi144.stdout:6/333: rmdir d8/d4f 39 2024-07-26T11:04:40.738 INFO:tasks.workunit.client.0.smithi144.stdout:2/245: mknod d0/d6/c4b 0 2024-07-26T11:04:40.744 INFO:tasks.workunit.client.0.smithi144.stdout:8/286: mknod d1a/d1c/d2f/c63 0 2024-07-26T11:04:40.744 INFO:tasks.workunit.client.0.smithi144.stdout:8/287: fsync d1a/d32/d49/f4a 0 2024-07-26T11:04:40.749 INFO:tasks.workunit.client.0.smithi144.stdout:2/246: symlink d0/d15/d20/d28/d2e/l4c 0 2024-07-26T11:04:40.749 INFO:tasks.workunit.client.0.smithi144.stdout:3/199: sync 2024-07-26T11:04:40.749 INFO:tasks.workunit.client.0.smithi144.stdout:2/247: unlink d0/d6/c4b 0 2024-07-26T11:04:40.749 INFO:tasks.workunit.client.0.smithi144.stdout:8/288: unlink c18 0 2024-07-26T11:04:40.751 INFO:tasks.workunit.client.0.smithi144.stdout:8/289: creat d1a/d32/f64 x:0 0 0 2024-07-26T11:04:40.751 INFO:tasks.workunit.client.0.smithi144.stdout:8/290: fsync f11 0 2024-07-26T11:04:40.754 INFO:tasks.workunit.client.0.smithi144.stdout:9/293: truncate d9/de/d14/d23/f31 307398 0 2024-07-26T11:04:40.760 INFO:tasks.workunit.client.0.smithi144.stdout:2/248: creat d0/d15/d20/d28/f4d x:0 0 0 2024-07-26T11:04:40.777 INFO:tasks.workunit.client.0.smithi144.stdout:2/249: dread d0/d6/d9/f26 [0,4194304] 0 2024-07-26T11:04:40.780 INFO:tasks.workunit.client.0.smithi144.stdout:2/250: rename d0/d24/d2b/d46 to d0/d15/d20/d28/d2e/d4e 0 2024-07-26T11:04:40.780 INFO:tasks.workunit.client.0.smithi144.stdout:2/251: fdatasync d0/d6/d9/f33 0 2024-07-26T11:04:40.785 INFO:tasks.workunit.client.0.smithi144.stdout:5/372: dwrite d1/d18/d24/d27/d71/f80 [0,4194304] 0 2024-07-26T11:04:40.789 INFO:tasks.workunit.client.0.smithi144.stdout:5/373: mknod d1/d18/c89 0 2024-07-26T11:04:40.792 INFO:tasks.workunit.client.0.smithi144.stdout:6/334: dwrite d8/db/f44 [0,4194304] 0 2024-07-26T11:04:40.792 INFO:tasks.workunit.client.0.smithi144.stdout:5/374: creat d1/d9/d44/f8a x:0 0 0 2024-07-26T11:04:40.796 INFO:tasks.workunit.client.0.smithi144.stdout:5/375: creat d1/d9/f8b x:0 0 0 2024-07-26T11:04:40.796 INFO:tasks.workunit.client.0.smithi144.stdout:6/335: creat d8/d50/f79 x:0 0 0 2024-07-26T11:04:40.796 INFO:tasks.workunit.client.0.smithi144.stdout:6/336: chown d8/d15/d57/l65 0 1 2024-07-26T11:04:40.797 INFO:tasks.workunit.client.0.smithi144.stdout:6/337: creat d8/d15/d26/f7a x:0 0 0 2024-07-26T11:04:40.800 INFO:tasks.workunit.client.0.smithi144.stdout:6/338: creat d8/d4f/d5c/f7b x:0 0 0 2024-07-26T11:04:40.800 INFO:tasks.workunit.client.0.smithi144.stdout:6/339: chown d8/db/l31 0 1 2024-07-26T11:04:40.833 INFO:tasks.workunit.client.0.smithi144.stdout:5/376: write d1/d18/d24/d27/d71/f80 [4083200,54091] 0 2024-07-26T11:04:40.837 INFO:tasks.workunit.client.0.smithi144.stdout:1/321: dwrite d3/d6/da/d13/d19/f45 [0,4194304] 0 2024-07-26T11:04:40.846 INFO:tasks.workunit.client.0.smithi144.stdout:9/294: dwrite d9/de/f4d [0,4194304] 0 2024-07-26T11:04:40.847 INFO:tasks.workunit.client.0.smithi144.stdout:9/295: write d9/fc [314736,29101] 0 2024-07-26T11:04:40.848 INFO:tasks.workunit.client.0.smithi144.stdout:3/200: dwrite d1/d5/d25/d26/f2d [0,4194304] 0 2024-07-26T11:04:40.848 INFO:tasks.workunit.client.0.smithi144.stdout:7/272: dwrite d0/d6/f17 [0,4194304] 0 2024-07-26T11:04:40.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/273: stat d0/d6/d9/d25/d2f/c5f 0 2024-07-26T11:04:40.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/274: fdatasync d0/d6/d9/d25/f42 0 2024-07-26T11:04:40.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/275: chown d0/d6/d9/d25/d2f/d47 224 1 2024-07-26T11:04:40.851 INFO:tasks.workunit.client.0.smithi144.stdout:8/291: dread d1a/d1c/d28/f3f [0,4194304] 0 2024-07-26T11:04:40.851 INFO:tasks.workunit.client.0.smithi144.stdout:9/296: rmdir d9/de 39 2024-07-26T11:04:40.853 INFO:tasks.workunit.client.0.smithi144.stdout:8/292: mknod d1a/d1c/d30/d4e/c65 0 2024-07-26T11:04:40.853 INFO:tasks.workunit.client.0.smithi144.stdout:8/293: stat d1a/d1c/d2f 0 2024-07-26T11:04:40.858 INFO:tasks.workunit.client.0.smithi144.stdout:2/252: dwrite d0/d6/f39 [0,4194304] 0 2024-07-26T11:04:40.858 INFO:tasks.workunit.client.0.smithi144.stdout:9/297: unlink d9/de/f2f 0 2024-07-26T11:04:40.859 INFO:tasks.workunit.client.0.smithi144.stdout:9/298: chown d9/de/d14/d3f 4 1 2024-07-26T11:04:40.859 INFO:tasks.workunit.client.0.smithi144.stdout:9/299: chown d9/de/d14/d23/f3a 19583 1 2024-07-26T11:04:40.861 INFO:tasks.workunit.client.0.smithi144.stdout:8/294: mkdir d1a/d32/d66 0 2024-07-26T11:04:40.862 INFO:tasks.workunit.client.0.smithi144.stdout:2/253: symlink d0/d6/d9/d44/l4f 0 2024-07-26T11:04:40.862 INFO:tasks.workunit.client.0.smithi144.stdout:2/254: fdatasync d0/d15/f40 0 2024-07-26T11:04:40.870 INFO:tasks.workunit.client.0.smithi144.stdout:2/255: mknod d0/d15/d20/d28/c50 0 2024-07-26T11:04:40.870 INFO:tasks.workunit.client.0.smithi144.stdout:2/256: stat d0/d15/d20/d28/d2e/l4c 0 2024-07-26T11:04:40.870 INFO:tasks.workunit.client.0.smithi144.stdout:8/295: rename l14 to d1a/d1c/l67 0 2024-07-26T11:04:40.870 INFO:tasks.workunit.client.0.smithi144.stdout:8/296: chown d1a/d1c/d30/f5e 27936384 1 2024-07-26T11:04:40.873 INFO:tasks.workunit.client.0.smithi144.stdout:6/340: dwrite d8/d42/f43 [0,4194304] 0 2024-07-26T11:04:40.874 INFO:tasks.workunit.client.0.smithi144.stdout:3/201: write d1/d5/d25/d26/f2e [2679443,88472] 0 2024-07-26T11:04:40.874 INFO:tasks.workunit.client.0.smithi144.stdout:4/284: sync 2024-07-26T11:04:40.876 INFO:tasks.workunit.client.0.smithi144.stdout:2/257: dread - d0/d6/d34/f43 zero size 2024-07-26T11:04:40.876 INFO:tasks.workunit.client.0.smithi144.stdout:2/258: read - d0/d6/d34/f43 zero size 2024-07-26T11:04:40.876 INFO:tasks.workunit.client.0.smithi144.stdout:6/341: creat d8/d42/f7c x:0 0 0 2024-07-26T11:04:40.877 INFO:tasks.workunit.client.0.smithi144.stdout:3/202: creat d1/d5/d25/d26/f33 x:0 0 0 2024-07-26T11:04:40.878 INFO:tasks.workunit.client.0.smithi144.stdout:3/203: readlink d1/l23 0 2024-07-26T11:04:40.878 INFO:tasks.workunit.client.0.smithi144.stdout:3/204: stat d1/d5/f24 0 2024-07-26T11:04:40.879 INFO:tasks.workunit.client.0.smithi144.stdout:4/285: creat d9/d26/f62 x:0 0 0 2024-07-26T11:04:40.880 INFO:tasks.workunit.client.0.smithi144.stdout:6/342: creat d8/f7d x:0 0 0 2024-07-26T11:04:40.882 INFO:tasks.workunit.client.0.smithi144.stdout:2/259: link d0/d15/d20/d28/d27/l37 d0/d15/d20/d28/d2e/l51 0 2024-07-26T11:04:40.883 INFO:tasks.workunit.client.0.smithi144.stdout:3/205: creat d1/f34 x:0 0 0 2024-07-26T11:04:40.883 INFO:tasks.workunit.client.0.smithi144.stdout:6/343: rmdir d8/d4f/d52/d76 0 2024-07-26T11:04:40.883 INFO:tasks.workunit.client.0.smithi144.stdout:2/260: getdents d0/d6 0 2024-07-26T11:04:40.883 INFO:tasks.workunit.client.0.smithi144.stdout:2/261: fsync d0/d15/d20/f35 0 2024-07-26T11:04:40.888 INFO:tasks.workunit.client.0.smithi144.stdout:6/344: link d8/f45 d8/d4f/f7e 0 2024-07-26T11:04:40.888 INFO:tasks.workunit.client.0.smithi144.stdout:3/206: creat d1/d5/f35 x:0 0 0 2024-07-26T11:04:40.888 INFO:tasks.workunit.client.0.smithi144.stdout:3/207: read - d1/f32 zero size 2024-07-26T11:04:40.888 INFO:tasks.workunit.client.0.smithi144.stdout:2/262: unlink d0/d6/f41 0 2024-07-26T11:04:40.889 INFO:tasks.workunit.client.0.smithi144.stdout:3/208: mknod d1/d5/d25/c36 0 2024-07-26T11:04:40.889 INFO:tasks.workunit.client.0.smithi144.stdout:3/209: readlink d1/l10 0 2024-07-26T11:04:40.890 INFO:tasks.workunit.client.0.smithi144.stdout:2/263: link d0/d6/d9/f26 d0/d24/d2b/f52 0 2024-07-26T11:04:40.891 INFO:tasks.workunit.client.0.smithi144.stdout:3/210: mkdir d1/d5/d25/d26/d37 0 2024-07-26T11:04:40.894 INFO:tasks.workunit.client.0.smithi144.stdout:2/264: symlink d0/d15/d20/d28/d2e/d4e/l53 0 2024-07-26T11:04:40.897 INFO:tasks.workunit.client.0.smithi144.stdout:2/265: mknod d0/d24/d48/c54 0 2024-07-26T11:04:40.907 INFO:tasks.workunit.client.0.smithi144.stdout:1/322: dread d3/d6/f32 [0,4194304] 0 2024-07-26T11:04:40.913 INFO:tasks.workunit.client.0.smithi144.stdout:5/377: dwrite d1/d5/f5c [0,4194304] 0 2024-07-26T11:04:40.922 INFO:tasks.workunit.client.0.smithi144.stdout:1/323: symlink d3/d6/l61 0 2024-07-26T11:04:40.922 INFO:tasks.workunit.client.0.smithi144.stdout:1/324: fsync d3/d53/f54 0 2024-07-26T11:04:40.925 INFO:tasks.workunit.client.0.smithi144.stdout:1/325: mkdir d3/d6/da/d21/d5c/d62 0 2024-07-26T11:04:40.928 INFO:tasks.workunit.client.0.smithi144.stdout:1/326: mknod d3/d6/da/d21/c63 0 2024-07-26T11:04:40.932 INFO:tasks.workunit.client.0.smithi144.stdout:1/327: unlink d3/l1f 0 2024-07-26T11:04:40.938 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:04:40.939 INFO:tasks.workunit.client.0.smithi144.stdout:1/328: write d3/d6/da/d21/d5c/f5e [153406,19353] 0 2024-07-26T11:04:40.941 INFO:tasks.workunit.client.0.smithi144.stdout:1/329: truncate d3/d6/f11 2925828 0 2024-07-26T11:04:40.943 INFO:tasks.workunit.client.0.smithi144.stdout:1/330: mknod d3/d6/da/d13/d19/d3a/c64 0 2024-07-26T11:04:40.943 INFO:tasks.workunit.client.0.smithi144.stdout:1/331: chown d3/d6/da/f4a 1321762 1 2024-07-26T11:04:40.945 INFO:tasks.workunit.client.0.smithi144.stdout:6/345: dwrite d8/f53 [0,4194304] 0 2024-07-26T11:04:40.945 INFO:tasks.workunit.client.0.smithi144.stdout:6/346: write d8/d4f/d52/f5a [6667158,31044] 0 2024-07-26T11:04:40.949 INFO:tasks.workunit.client.0.smithi144.stdout:6/347: symlink d8/d15/d26/l7f 0 2024-07-26T11:04:40.950 INFO:tasks.workunit.client.0.smithi144.stdout:6/348: symlink d8/d15/d57/d5f/d6e/l80 0 2024-07-26T11:04:40.951 INFO:tasks.workunit.client.0.smithi144.stdout:6/349: stat d8/d15/d57/c5d 0 2024-07-26T11:04:40.951 INFO:tasks.workunit.client.0.smithi144.stdout:6/350: dread - d8/f67 zero size 2024-07-26T11:04:40.952 INFO:tasks.workunit.client.0.smithi144.stdout:6/351: creat d8/db/f81 x:0 0 0 2024-07-26T11:04:40.957 INFO:tasks.workunit.client.0.smithi144.stdout:6/352: rename d8/d42/f43 to d8/f82 0 2024-07-26T11:04:40.957 INFO:tasks.workunit.client.0.smithi144.stdout:6/353: chown d8/db/f25 6 1 2024-07-26T11:04:40.957 INFO:tasks.workunit.client.0.smithi144.stdout:9/300: write d9/de/f21 [3097975,7412] 0 2024-07-26T11:04:40.957 INFO:tasks.workunit.client.0.smithi144.stdout:9/301: stat d9/de/ff 0 2024-07-26T11:04:40.957 INFO:tasks.workunit.client.0.smithi144.stdout:7/276: dwrite d0/d6/d9/d25/d2f/d47/f4f [0,4194304] 0 2024-07-26T11:04:40.966 INFO:tasks.workunit.client.0.smithi144.stdout:5/378: write d1/d5/f5c [406763,142] 0 2024-07-26T11:04:40.966 INFO:tasks.workunit.client.0.smithi144.stdout:5/379: dread - d1/d9/d44/f62 zero size 2024-07-26T11:04:40.966 INFO:tasks.workunit.client.0.smithi144.stdout:0/336: sync 2024-07-26T11:04:40.966 INFO:tasks.workunit.client.0.smithi144.stdout:0/337: read d0/d25/d5e/d1e/f24 [451872,104497] 0 2024-07-26T11:04:40.966 INFO:tasks.workunit.client.0.smithi144.stdout:5/380: stat d1/cf 0 2024-07-26T11:04:40.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/211: dwrite d1/f14 [0,4194304] 0 2024-07-26T11:04:40.969 INFO:tasks.workunit.client.0.smithi144.stdout:3/212: chown d1/lc 9692 1 2024-07-26T11:04:40.969 INFO:tasks.workunit.client.0.smithi144.stdout:3/213: stat d1/d5/d25/d26/f2d 0 2024-07-26T11:04:40.969 INFO:tasks.workunit.client.0.smithi144.stdout:0/338: creat d0/d25/d5e/d1e/d30/d4b/f75 x:0 0 0 2024-07-26T11:04:40.969 INFO:tasks.workunit.client.0.smithi144.stdout:4/286: dwrite d9/d26/f62 [0,4194304] 0 2024-07-26T11:04:40.970 INFO:tasks.workunit.client.0.smithi144.stdout:4/287: truncate d9/d26/f30 968479 0 2024-07-26T11:04:40.973 INFO:tasks.workunit.client.0.smithi144.stdout:0/339: symlink d0/d3e/d5f/l76 0 2024-07-26T11:04:40.973 INFO:tasks.workunit.client.0.smithi144.stdout:0/340: fsync d0/d25/d5e/d1e/f3f 0 2024-07-26T11:04:40.975 INFO:tasks.workunit.client.0.smithi144.stdout:0/341: mknod d0/d3e/c77 0 2024-07-26T11:04:40.980 INFO:tasks.workunit.client.0.smithi144.stdout:1/332: read d3/d6/da/f12 [279405,92546] 0 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:1/333: chown d3/d6/l15 0 1 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:0/342: creat d0/f78 x:0 0 0 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:1/334: symlink d3/d6/da/d13/d22/l65 0 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:1/335: chown d3/d6/f23 93 1 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:1/336: dread - d3/f2d zero size 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:0/343: creat d0/d3e/d45/d4c/f79 x:0 0 0 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:1/337: symlink d3/d6/da/d13/d19/d27/d36/d37/l66 0 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:0/344: rmdir d0/d3e 39 2024-07-26T11:04:40.981 INFO:tasks.workunit.client.0.smithi144.stdout:0/345: write d0/d3e/d45/f50 [472604,119506] 0 2024-07-26T11:04:40.982 INFO:tasks.workunit.client.0.smithi144.stdout:0/346: link d0/c15 d0/d25/d5e/d1e/d30/d67/c7a 0 2024-07-26T11:04:40.985 INFO:tasks.workunit.client.0.smithi144.stdout:9/302: write d9/de/d14/d23/d39/f5b [2295877,7403] 0 2024-07-26T11:04:40.993 INFO:tasks.workunit.client.0.smithi144.stdout:7/277: write d0/f2d [4716144,114841] 0 2024-07-26T11:04:40.995 INFO:tasks.workunit.client.0.smithi144.stdout:9/303: symlink d9/de/d4e/l65 0 2024-07-26T11:04:41.000 INFO:tasks.workunit.client.0.smithi144.stdout:7/278: mkdir d0/d71 0 2024-07-26T11:04:41.000 INFO:tasks.workunit.client.0.smithi144.stdout:9/304: rmdir d9/de/d14/d23 39 2024-07-26T11:04:41.000 INFO:tasks.workunit.client.0.smithi144.stdout:4/288: write d9/da/f1a [2218901,102806] 0 2024-07-26T11:04:41.015 INFO:tasks.workunit.client.0.smithi144.stdout:6/354: dread d8/d15/d26/f3f [0,4194304] 0 2024-07-26T11:04:41.016 INFO:tasks.workunit.client.0.smithi144.stdout:0/347: dread d0/d25/d5e/f41 [0,4194304] 0 2024-07-26T11:04:41.016 INFO:tasks.workunit.client.0.smithi144.stdout:0/348: fdatasync d0/f78 0 2024-07-26T11:04:41.016 INFO:tasks.workunit.client.0.smithi144.stdout:0/349: chown d0/d25/c2f 184315 1 2024-07-26T11:04:41.016 INFO:tasks.workunit.client.0.smithi144.stdout:0/350: dread - d0/d25/d5e/f2c zero size 2024-07-26T11:04:41.031 INFO:tasks.workunit.client.0.smithi144.stdout:5/381: dread d1/d9/f19 [0,4194304] 0 2024-07-26T11:04:41.031 INFO:tasks.workunit.client.0.smithi144.stdout:5/382: getdents d1/d9/d44/d85 0 2024-07-26T11:04:41.037 INFO:tasks.workunit.client.0.smithi144.stdout:1/338: dwrite d3/d6/da/d13/d19/d27/f28 [0,4194304] 0 2024-07-26T11:04:41.039 INFO:tasks.workunit.client.0.smithi144.stdout:1/339: mkdir d3/d6/da/d21/d67 0 2024-07-26T11:04:41.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/351: write d0/d3e/f43 [3128937,7426] 0 2024-07-26T11:04:41.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/352: dread - d0/d25/d5e/d1e/d30/d4b/f58 zero size 2024-07-26T11:04:41.049 INFO:tasks.workunit.client.0.smithi144.stdout:4/289: dwrite f3 [0,4194304] 0 2024-07-26T11:04:41.049 INFO:tasks.workunit.client.0.smithi144.stdout:4/290: dread - d9/da/f39 zero size 2024-07-26T11:04:41.052 INFO:tasks.workunit.client.0.smithi144.stdout:0/353: creat d0/d25/f7b x:0 0 0 2024-07-26T11:04:41.052 INFO:tasks.workunit.client.0.smithi144.stdout:0/354: readlink d0/d3e/d45/d4c/l74 0 2024-07-26T11:04:41.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/355: dread - d0/d3e/d45/d4c/f64 zero size 2024-07-26T11:04:41.054 INFO:tasks.workunit.client.0.smithi144.stdout:4/291: unlink d9/da/f1a 0 2024-07-26T11:04:41.054 INFO:tasks.workunit.client.0.smithi144.stdout:4/292: dread - d9/d26/f2f zero size 2024-07-26T11:04:41.061 INFO:tasks.workunit.client.0.smithi144.stdout:3/214: dwrite d1/d5/d25/d26/f2b [0,4194304] 0 2024-07-26T11:04:41.065 INFO:tasks.workunit.client.0.smithi144.stdout:1/340: write d3/f1e [3942488,99290] 0 2024-07-26T11:04:41.065 INFO:tasks.workunit.client.0.smithi144.stdout:3/215: getdents d1 0 2024-07-26T11:04:41.080 INFO:tasks.workunit.client.0.smithi144.stdout:4/293: dread d9/da/f2d [0,4194304] 0 2024-07-26T11:04:41.081 INFO:tasks.workunit.client.0.smithi144.stdout:6/355: dwrite d8/d15/f38 [0,4194304] 0 2024-07-26T11:04:41.081 INFO:tasks.workunit.client.0.smithi144.stdout:6/356: chown d8/db/c11 18 1 2024-07-26T11:04:41.082 INFO:tasks.workunit.client.0.smithi144.stdout:7/279: dwrite d0/f49 [0,4194304] 0 2024-07-26T11:04:41.082 INFO:tasks.workunit.client.0.smithi144.stdout:7/280: chown d0/d6/d9/d64/f6b 129845484 1 2024-07-26T11:04:41.082 INFO:tasks.workunit.client.0.smithi144.stdout:4/294: rmdir d9 39 2024-07-26T11:04:41.082 INFO:tasks.workunit.client.0.smithi144.stdout:4/295: truncate d9/d26/f2b 199670 0 2024-07-26T11:04:41.084 INFO:tasks.workunit.client.0.smithi144.stdout:9/305: dwrite d9/d3b/f56 [0,4194304] 0 2024-07-26T11:04:41.090 INFO:tasks.workunit.client.0.smithi144.stdout:6/357: link d8/db/c11 d8/d4f/d74/c83 0 2024-07-26T11:04:41.094 INFO:tasks.workunit.client.0.smithi144.stdout:7/281: mknod d0/d6/d9/d12/c72 0 2024-07-26T11:04:41.094 INFO:tasks.workunit.client.0.smithi144.stdout:4/296: unlink d9/d26/f2a 0 2024-07-26T11:04:41.096 INFO:tasks.workunit.client.0.smithi144.stdout:1/341: getdents d3/d6/da/d21 0 2024-07-26T11:04:41.100 INFO:tasks.workunit.client.0.smithi144.stdout:5/383: dwrite d1/d9/f19 [0,4194304] 0 2024-07-26T11:04:41.108 INFO:tasks.workunit.client.0.smithi144.stdout:9/306: getdents d9/de/d4e 0 2024-07-26T11:04:41.109 INFO:tasks.workunit.client.0.smithi144.stdout:9/307: write d9/de/f21 [1126506,33957] 0 2024-07-26T11:04:41.110 INFO:tasks.workunit.client.0.smithi144.stdout:6/358: rmdir d8/d4f/d5c/d77 0 2024-07-26T11:04:41.111 INFO:tasks.workunit.client.0.smithi144.stdout:7/282: getdents d0/d6 0 2024-07-26T11:04:41.112 INFO:tasks.workunit.client.0.smithi144.stdout:5/384: dread d1/f5e [0,4194304] 0 2024-07-26T11:04:41.112 INFO:tasks.workunit.client.0.smithi144.stdout:4/297: creat d9/f63 x:0 0 0 2024-07-26T11:04:41.114 INFO:tasks.workunit.client.0.smithi144.stdout:1/342: mkdir d3/d6/da/d21/d5c/d62/d68 0 2024-07-26T11:04:41.114 INFO:tasks.workunit.client.0.smithi144.stdout:1/343: fsync d3/f4f 0 2024-07-26T11:04:41.114 INFO:tasks.workunit.client.0.smithi144.stdout:9/308: creat d9/de/d14/d23/d39/f66 x:0 0 0 2024-07-26T11:04:41.116 INFO:tasks.workunit.client.0.smithi144.stdout:6/359: creat d8/d15/d57/d5f/f84 x:0 0 0 2024-07-26T11:04:41.116 INFO:tasks.workunit.client.0.smithi144.stdout:6/360: fdatasync d8/d4f/f63 0 2024-07-26T11:04:41.117 INFO:tasks.workunit.client.0.smithi144.stdout:3/216: dwrite d1/f32 [0,4194304] 0 2024-07-26T11:04:41.118 INFO:tasks.workunit.client.0.smithi144.stdout:5/385: link d1/d5/f6f d1/d9/d5b/f8c 0 2024-07-26T11:04:41.119 INFO:tasks.workunit.client.0.smithi144.stdout:5/386: dread - d1/d9/d5b/d4d/f79 zero size 2024-07-26T11:04:41.120 INFO:tasks.workunit.client.0.smithi144.stdout:4/298: rename d9/da/f25 to d9/d26/d48/d50/f64 0 2024-07-26T11:04:41.120 INFO:tasks.workunit.client.0.smithi144.stdout:9/309: symlink d9/l67 0 2024-07-26T11:04:41.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/297: sync 2024-07-26T11:04:41.121 INFO:tasks.workunit.client.0.smithi144.stdout:2/266: sync 2024-07-26T11:04:41.125 INFO:tasks.workunit.client.0.smithi144.stdout:3/217: symlink d1/d5/d25/d26/d37/l38 0 2024-07-26T11:04:41.126 INFO:tasks.workunit.client.0.smithi144.stdout:5/387: rename d1/d9/c12 to d1/d18/d24/c8d 0 2024-07-26T11:04:41.127 INFO:tasks.workunit.client.0.smithi144.stdout:4/299: unlink d9/d26/d48/d50/c5c 0 2024-07-26T11:04:41.128 INFO:tasks.workunit.client.0.smithi144.stdout:6/361: creat d8/d50/f85 x:0 0 0 2024-07-26T11:04:41.130 INFO:tasks.workunit.client.0.smithi144.stdout:2/267: symlink d0/d6/d34/l55 0 2024-07-26T11:04:41.131 INFO:tasks.workunit.client.0.smithi144.stdout:2/268: write d0/d6/f1d [537333,34325] 0 2024-07-26T11:04:41.132 INFO:tasks.workunit.client.0.smithi144.stdout:6/362: rename d8/db/f44 to d8/f86 0 2024-07-26T11:04:41.134 INFO:tasks.workunit.client.0.smithi144.stdout:6/363: unlink d8/db/f55 0 2024-07-26T11:04:41.134 INFO:tasks.workunit.client.0.smithi144.stdout:6/364: truncate d8/db/f34 759988 0 2024-07-26T11:04:41.135 INFO:tasks.workunit.client.0.smithi144.stdout:6/365: symlink d8/d4f/d74/l87 0 2024-07-26T11:04:41.136 INFO:tasks.workunit.client.0.smithi144.stdout:6/366: chown d8/d42/l6d 7 1 2024-07-26T11:04:41.136 INFO:tasks.workunit.client.0.smithi144.stdout:6/367: chown d8/d4f/d74 508662 1 2024-07-26T11:04:41.136 INFO:tasks.workunit.client.0.smithi144.stdout:6/368: dread - d8/d4f/f72 zero size 2024-07-26T11:04:41.136 INFO:tasks.workunit.client.0.smithi144.stdout:6/369: fsync f2 0 2024-07-26T11:04:41.140 INFO:tasks.workunit.client.0.smithi144.stdout:6/370: mkdir d8/d15/d57/d5f/d88 0 2024-07-26T11:04:41.141 INFO:tasks.workunit.client.0.smithi144.stdout:6/371: mknod d8/d15/d57/c89 0 2024-07-26T11:04:41.143 INFO:tasks.workunit.client.0.smithi144.stdout:6/372: creat d8/d15/d57/d5f/d88/f8a x:0 0 0 2024-07-26T11:04:41.145 INFO:tasks.workunit.client.0.smithi144.stdout:6/373: rmdir d8/d42 39 2024-07-26T11:04:41.145 INFO:tasks.workunit.client.0.smithi144.stdout:6/374: write d8/f33 [2971155,37959] 0 2024-07-26T11:04:41.145 INFO:tasks.workunit.client.0.smithi144.stdout:6/375: dread - d8/f56 zero size 2024-07-26T11:04:41.147 INFO:tasks.workunit.client.0.smithi144.stdout:6/376: unlink d8/db/f34 0 2024-07-26T11:04:41.148 INFO:tasks.workunit.client.0.smithi144.stdout:6/377: mknod d8/d42/c8b 0 2024-07-26T11:04:41.150 INFO:tasks.workunit.client.0.smithi144.stdout:7/283: read d0/d6/d9/d12/f1d [3670064,84556] 0 2024-07-26T11:04:41.151 INFO:tasks.workunit.client.0.smithi144.stdout:7/284: fsync d0/d6/d9/d3f/f58 0 2024-07-26T11:04:41.151 INFO:tasks.workunit.client.0.smithi144.stdout:7/285: fsync d0/d6/d9/d64/f6b 0 2024-07-26T11:04:41.151 INFO:tasks.workunit.client.0.smithi144.stdout:6/378: link d8/d15/d57/d5f/c60 d8/d15/c8c 0 2024-07-26T11:04:41.152 INFO:tasks.workunit.client.0.smithi144.stdout:7/286: symlink d0/d6/d9/d64/l73 0 2024-07-26T11:04:41.152 INFO:tasks.workunit.client.0.smithi144.stdout:6/379: getdents d8/d4f 0 2024-07-26T11:04:41.154 INFO:tasks.workunit.client.0.smithi144.stdout:7/287: symlink d0/d6/d9/d25/d67/l74 0 2024-07-26T11:04:41.155 INFO:tasks.workunit.client.0.smithi144.stdout:6/380: creat d8/d4f/d52/f8d x:0 0 0 2024-07-26T11:04:41.155 INFO:tasks.workunit.client.0.smithi144.stdout:6/381: write d8/d4f/f51 [4057521,63535] 0 2024-07-26T11:04:41.158 INFO:tasks.workunit.client.0.smithi144.stdout:7/288: symlink d0/d6/d9/d25/d2f/d53/l75 0 2024-07-26T11:04:41.160 INFO:tasks.workunit.client.0.smithi144.stdout:7/289: symlink d0/d6/d9/d25/d2f/d68/l76 0 2024-07-26T11:04:41.160 INFO:tasks.workunit.client.0.smithi144.stdout:7/290: write d0/d6/f8 [4700098,75000] 0 2024-07-26T11:04:41.168 INFO:tasks.workunit.client.0.smithi144.stdout:8/298: dread d1a/d1c/f22 [0,4194304] 0 2024-07-26T11:04:41.169 INFO:tasks.workunit.client.0.smithi144.stdout:1/344: dwrite d3/d6/da/f48 [0,4194304] 0 2024-07-26T11:04:41.169 INFO:tasks.workunit.client.0.smithi144.stdout:1/345: fdatasync d3/d6/f32 0 2024-07-26T11:04:41.169 INFO:tasks.workunit.client.0.smithi144.stdout:1/346: dread - d3/f2d zero size 2024-07-26T11:04:41.176 INFO:tasks.workunit.client.0.smithi144.stdout:3/218: dwrite d1/d5/f24 [0,4194304] 0 2024-07-26T11:04:41.181 INFO:tasks.workunit.client.0.smithi144.stdout:1/347: rename d3/d6/da/d13/d19/d27 to d3/d6/da/d13/d19/d3a/d69 0 2024-07-26T11:04:41.182 INFO:tasks.workunit.client.0.smithi144.stdout:9/310: dwrite d9/de/d14/f2c [0,4194304] 0 2024-07-26T11:04:41.189 INFO:tasks.workunit.client.0.smithi144.stdout:3/219: truncate d1/d5/d25/d26/f2b 2894445 0 2024-07-26T11:04:41.192 INFO:tasks.workunit.client.0.smithi144.stdout:1/348: mknod d3/d3d/d58/c6a 0 2024-07-26T11:04:41.193 INFO:tasks.workunit.client.0.smithi144.stdout:9/311: mknod d9/d3b/d5d/c68 0 2024-07-26T11:04:41.195 INFO:tasks.workunit.client.0.smithi144.stdout:3/220: getdents d1/d5 0 2024-07-26T11:04:41.200 INFO:tasks.workunit.client.0.smithi144.stdout:6/382: write d8/d15/f38 [1684092,19599] 0 2024-07-26T11:04:41.200 INFO:tasks.workunit.client.0.smithi144.stdout:5/388: dwrite d1/d9/d44/f41 [0,4194304] 0 2024-07-26T11:04:41.200 INFO:tasks.workunit.client.0.smithi144.stdout:3/221: rmdir d1 39 2024-07-26T11:04:41.201 INFO:tasks.workunit.client.0.smithi144.stdout:9/312: truncate f4 2460641 0 2024-07-26T11:04:41.201 INFO:tasks.workunit.client.0.smithi144.stdout:9/313: fsync d9/de/d14/d23/f47 0 2024-07-26T11:04:41.201 INFO:tasks.workunit.client.0.smithi144.stdout:9/314: dread - d9/de/f51 zero size 2024-07-26T11:04:41.201 INFO:tasks.workunit.client.0.smithi144.stdout:1/349: symlink d3/d6/da/d13/d22/d5d/l6b 0 2024-07-26T11:04:41.202 INFO:tasks.workunit.client.0.smithi144.stdout:1/350: chown d3/f2d 801 1 2024-07-26T11:04:41.202 INFO:tasks.workunit.client.0.smithi144.stdout:4/300: dwrite d9/f4e [0,4194304] 0 2024-07-26T11:04:41.202 INFO:tasks.workunit.client.0.smithi144.stdout:4/301: write d9/d26/d3c/f40 [580757,85910] 0 2024-07-26T11:04:41.202 INFO:tasks.workunit.client.0.smithi144.stdout:4/302: read - d9/f11 zero size 2024-07-26T11:04:41.204 INFO:tasks.workunit.client.0.smithi144.stdout:7/291: dwrite d0/f11 [0,4194304] 0 2024-07-26T11:04:41.205 INFO:tasks.workunit.client.0.smithi144.stdout:6/383: rename d8/fd to d8/d15/d57/d5f/d6e/f8e 0 2024-07-26T11:04:41.205 INFO:tasks.workunit.client.0.smithi144.stdout:9/315: write f4 [732150,120548] 0 2024-07-26T11:04:41.206 INFO:tasks.workunit.client.0.smithi144.stdout:5/389: symlink d1/d9/l8e 0 2024-07-26T11:04:41.207 INFO:tasks.workunit.client.0.smithi144.stdout:5/390: chown d1/d9/cb 2 1 2024-07-26T11:04:41.207 INFO:tasks.workunit.client.0.smithi144.stdout:5/391: chown d1/d18/d24/d27/c43 167275570 1 2024-07-26T11:04:41.208 INFO:tasks.workunit.client.0.smithi144.stdout:3/222: creat d1/f39 x:0 0 0 2024-07-26T11:04:41.210 INFO:tasks.workunit.client.0.smithi144.stdout:7/292: creat d0/d6/d9/d45/d57/f77 x:0 0 0 2024-07-26T11:04:41.214 INFO:tasks.workunit.client.0.smithi144.stdout:9/316: getdents d9/de/d14/d3f/d5e 0 2024-07-26T11:04:41.214 INFO:tasks.workunit.client.0.smithi144.stdout:9/317: chown d9/de/d14/f2c 179 1 2024-07-26T11:04:41.214 INFO:tasks.workunit.client.0.smithi144.stdout:9/318: readlink d9/l25 0 2024-07-26T11:04:41.214 INFO:tasks.workunit.client.0.smithi144.stdout:9/319: readlink d9/d3b/l3c 0 2024-07-26T11:04:41.219 INFO:tasks.workunit.client.0.smithi144.stdout:3/223: creat d1/d5/f3a x:0 0 0 2024-07-26T11:04:41.220 INFO:tasks.workunit.client.0.smithi144.stdout:7/293: rename d0/d6/d9/d25/d2f/c3b to d0/d6/d9/d12/c78 0 2024-07-26T11:04:41.223 INFO:tasks.workunit.client.0.smithi144.stdout:8/299: dwrite d1a/d1c/d30/f5e [0,4194304] 0 2024-07-26T11:04:41.223 INFO:tasks.workunit.client.0.smithi144.stdout:8/300: fdatasync f11 0 2024-07-26T11:04:41.225 INFO:tasks.workunit.client.0.smithi144.stdout:4/303: dread d9/d26/f31 [0,4194304] 0 2024-07-26T11:04:41.228 INFO:tasks.workunit.client.0.smithi144.stdout:5/392: dread d1/d5/f2e [0,4194304] 0 2024-07-26T11:04:41.232 INFO:tasks.workunit.client.0.smithi144.stdout:1/351: read d3/d6/da/d13/d22/f30 [2621830,69333] 0 2024-07-26T11:04:41.236 INFO:tasks.workunit.client.0.smithi144.stdout:8/301: rename d1a/d1c/l46 to d1a/d1c/d30/d4e/l68 0 2024-07-26T11:04:41.237 INFO:tasks.workunit.client.0.smithi144.stdout:4/304: unlink d9/d26/d48/f56 0 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:1/352: unlink d3/d6/da/d13/d22/d5d/l6b 0 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:4/305: mknod d9/d26/d3c/c65 0 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:4/306: chown d9/d26/f47 18 1 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:4/307: getdents d9 0 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:4/308: unlink d9/c3e 0 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:4/309: write d9/d26/f2f [47111,127596] 0 2024-07-26T11:04:41.243 INFO:tasks.workunit.client.0.smithi144.stdout:4/310: read - d9/da/f52 zero size 2024-07-26T11:04:41.245 INFO:tasks.workunit.client.0.smithi144.stdout:4/311: creat d9/d26/d48/f66 x:0 0 0 2024-07-26T11:04:41.246 INFO:tasks.workunit.client.0.smithi144.stdout:4/312: dread - d9/d26/f28 zero size 2024-07-26T11:04:41.246 INFO:tasks.workunit.client.0.smithi144.stdout:4/313: write d9/f11 [140615,49624] 0 2024-07-26T11:04:41.246 INFO:tasks.workunit.client.0.smithi144.stdout:4/314: fdatasync d9/f51 0 2024-07-26T11:04:41.248 INFO:tasks.workunit.client.0.smithi144.stdout:4/315: rename d9/d26/c3d to d9/d26/d48/d50/c67 0 2024-07-26T11:04:41.270 INFO:tasks.workunit.client.0.smithi144.stdout:3/224: dwrite d1/d5/f24 [0,4194304] 0 2024-07-26T11:04:41.272 INFO:tasks.workunit.client.0.smithi144.stdout:3/225: creat d1/d28/f3b x:0 0 0 2024-07-26T11:04:41.275 INFO:tasks.workunit.client.0.smithi144.stdout:6/384: dwrite d8/f16 [4194304,4194304] 0 2024-07-26T11:04:41.275 INFO:tasks.workunit.client.0.smithi144.stdout:6/385: write d8/d50/f79 [757323,64196] 0 2024-07-26T11:04:41.275 INFO:tasks.workunit.client.0.smithi144.stdout:6/386: fsync d8/d4f/f72 0 2024-07-26T11:04:41.286 INFO:tasks.workunit.client.0.smithi144.stdout:5/393: dwrite d1/d18/d24/d35/f3d [0,4194304] 0 2024-07-26T11:04:41.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/302: dwrite d1a/d1c/d28/d2c/d38/f5a [0,4194304] 0 2024-07-26T11:04:41.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/303: write d1a/d1c/d30/f40 [1243202,26565] 0 2024-07-26T11:04:41.312 INFO:tasks.workunit.client.0.smithi144.stdout:5/394: dread d1/d5/f2d [0,4194304] 0 2024-07-26T11:04:41.313 INFO:tasks.workunit.client.0.smithi144.stdout:1/353: dread d3/d6/f32 [4194304,4194304] 0 2024-07-26T11:04:41.313 INFO:tasks.workunit.client.0.smithi144.stdout:1/354: truncate d3/d6/da/f4a 1014568 0 2024-07-26T11:04:41.313 INFO:tasks.workunit.client.0.smithi144.stdout:5/395: creat d1/f8f x:0 0 0 2024-07-26T11:04:41.314 INFO:tasks.workunit.client.0.smithi144.stdout:5/396: dread - d1/d18/d24/d35/f77 zero size 2024-07-26T11:04:41.314 INFO:tasks.workunit.client.0.smithi144.stdout:5/397: readlink d1/d9/d5b/l5d 0 2024-07-26T11:04:41.314 INFO:tasks.workunit.client.0.smithi144.stdout:1/355: getdents d3/d53 0 2024-07-26T11:04:41.316 INFO:tasks.workunit.client.0.smithi144.stdout:4/316: dread d9/d26/f4c [0,4194304] 0 2024-07-26T11:04:41.317 INFO:tasks.workunit.client.0.smithi144.stdout:5/398: unlink d1/d18/f67 0 2024-07-26T11:04:41.317 INFO:tasks.workunit.client.0.smithi144.stdout:5/399: stat d1/d5/f5c 0 2024-07-26T11:04:41.319 INFO:tasks.workunit.client.0.smithi144.stdout:4/317: rmdir d9/d26/d3c 39 2024-07-26T11:04:41.324 INFO:tasks.workunit.client.0.smithi144.stdout:6/387: dwrite d8/d15/f21 [0,4194304] 0 2024-07-26T11:04:41.324 INFO:tasks.workunit.client.0.smithi144.stdout:6/388: write d8/db/f25 [4597824,6950] 0 2024-07-26T11:04:41.336 INFO:tasks.workunit.client.0.smithi144.stdout:6/389: unlink d8/cf 0 2024-07-26T11:04:41.347 INFO:tasks.workunit.client.0.smithi144.stdout:4/318: write d9/f4e [1762328,118912] 0 2024-07-26T11:04:41.350 INFO:tasks.workunit.client.0.smithi144.stdout:4/319: link d9/l2e d9/d26/l68 0 2024-07-26T11:04:41.353 INFO:tasks.workunit.client.0.smithi144.stdout:8/304: dwrite d1a/d1c/f24 [0,4194304] 0 2024-07-26T11:04:41.361 INFO:tasks.workunit.client.0.smithi144.stdout:1/356: dwrite d3/d6/fd [0,4194304] 0 2024-07-26T11:04:41.364 INFO:tasks.workunit.client.0.smithi144.stdout:1/357: symlink d3/d53/l6c 0 2024-07-26T11:04:41.383 INFO:tasks.workunit.client.0.smithi144.stdout:8/305: dread d1a/d1c/d30/f39 [0,4194304] 0 2024-07-26T11:04:41.383 INFO:tasks.workunit.client.0.smithi144.stdout:5/400: dwrite d1/d9/d5b/d4d/f6c [0,4194304] 0 2024-07-26T11:04:41.383 INFO:tasks.workunit.client.0.smithi144.stdout:5/401: readlink d1/d9/d44/l3c 0 2024-07-26T11:04:41.390 INFO:tasks.workunit.client.0.smithi144.stdout:1/358: write d3/d6/da/d13/d19/d3a/d69/f28 [3839063,25867] 0 2024-07-26T11:04:41.394 INFO:tasks.workunit.client.0.smithi144.stdout:6/390: dwrite d8/d4f/f62 [0,4194304] 0 2024-07-26T11:04:41.394 INFO:tasks.workunit.client.0.smithi144.stdout:1/359: read d3/d6/da/d21/f4d [72793,69190] 0 2024-07-26T11:04:41.394 INFO:tasks.workunit.client.0.smithi144.stdout:1/360: fsync d3/f3b 0 2024-07-26T11:04:41.395 INFO:tasks.workunit.client.0.smithi144.stdout:1/361: write d3/d6/da/d21/f4d [811710,87294] 0 2024-07-26T11:04:41.395 INFO:tasks.workunit.client.0.smithi144.stdout:1/362: chown d3/d6/da/cf 44672 1 2024-07-26T11:04:41.396 INFO:tasks.workunit.client.0.smithi144.stdout:1/363: readlink d3/d6/da/d13/d19/d3a/d3e/l57 0 2024-07-26T11:04:41.419 INFO:tasks.workunit.client.0.smithi144.stdout:6/391: dread d8/d4f/f62 [0,4194304] 0 2024-07-26T11:04:41.419 INFO:tasks.workunit.client.0.smithi144.stdout:6/392: chown d8/f33 0 1 2024-07-26T11:04:41.419 INFO:tasks.workunit.client.0.smithi144.stdout:6/393: fsync d8/f82 0 2024-07-26T11:04:41.419 INFO:tasks.workunit.client.0.smithi144.stdout:6/394: readlink d8/d15/l2f 0 2024-07-26T11:04:41.419 INFO:tasks.workunit.client.0.smithi144.stdout:6/395: dread - d8/d4f/f72 zero size 2024-07-26T11:04:41.428 INFO:tasks.workunit.client.0.smithi144.stdout:5/402: dwrite d1/d9/d44/f62 [0,4194304] 0 2024-07-26T11:04:41.429 INFO:tasks.workunit.client.0.smithi144.stdout:5/403: dread - d1/d5/f6f zero size 2024-07-26T11:04:41.435 INFO:tasks.workunit.client.0.smithi144.stdout:5/404: creat d1/d18/f90 x:0 0 0 2024-07-26T11:04:41.435 INFO:tasks.workunit.client.0.smithi144.stdout:5/405: dread - d1/d18/d24/f82 zero size 2024-07-26T11:04:41.437 INFO:tasks.workunit.client.0.smithi144.stdout:5/406: link d1/d9/f10 d1/d9/d44/d45/d81/f91 0 2024-07-26T11:04:41.437 INFO:tasks.workunit.client.0.smithi144.stdout:5/407: readlink d1/d9/le 0 2024-07-26T11:04:41.439 INFO:tasks.workunit.client.0.smithi144.stdout:5/408: getdents d1/d18/d24/d27 0 2024-07-26T11:04:41.442 INFO:tasks.workunit.client.0.smithi144.stdout:8/306: dwrite d1a/d1c/d28/d2c/d38/f5f [0,4194304] 0 2024-07-26T11:04:41.445 INFO:tasks.workunit.client.0.smithi144.stdout:1/364: dwrite d3/d6/f47 [0,4194304] 0 2024-07-26T11:04:41.447 INFO:tasks.workunit.client.0.smithi144.stdout:8/307: truncate d1a/d1c/d30/f39 2452340 0 2024-07-26T11:04:41.447 INFO:tasks.workunit.client.0.smithi144.stdout:8/308: rename d1a to d1a/d32/d66/d69 22 2024-07-26T11:04:41.449 INFO:tasks.workunit.client.0.smithi144.stdout:1/365: symlink d3/d53/l6d 0 2024-07-26T11:04:41.450 INFO:tasks.workunit.client.0.smithi144.stdout:1/366: getdents d3/d6/da/d13 0 2024-07-26T11:04:41.458 INFO:tasks.workunit.client.0.smithi144.stdout:6/396: dwrite d8/f86 [0,4194304] 0 2024-07-26T11:04:41.460 INFO:tasks.workunit.client.0.smithi144.stdout:6/397: truncate d8/d4f/f62 1878869 0 2024-07-26T11:04:41.460 INFO:tasks.workunit.client.0.smithi144.stdout:6/398: chown d8/db/l31 76751299 1 2024-07-26T11:04:41.462 INFO:tasks.workunit.client.0.smithi144.stdout:6/399: getdents d8/db 0 2024-07-26T11:04:41.463 INFO:tasks.workunit.client.0.smithi144.stdout:6/400: mknod d8/d4f/d52/d58/c8f 0 2024-07-26T11:04:41.476 INFO:tasks.workunit.client.0.smithi144.stdout:1/367: dread d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:41.476 INFO:tasks.workunit.client.0.smithi144.stdout:1/368: chown d3/d3d/l46 3872 1 2024-07-26T11:04:41.477 INFO:tasks.workunit.client.0.smithi144.stdout:5/409: dwrite d1/d18/f90 [0,4194304] 0 2024-07-26T11:04:41.477 INFO:tasks.workunit.client.0.smithi144.stdout:8/309: dread fe [0,4194304] 0 2024-07-26T11:04:41.479 INFO:tasks.workunit.client.0.smithi144.stdout:5/410: symlink d1/d18/d24/d35/l92 0 2024-07-26T11:04:41.490 INFO:tasks.workunit.client.0.smithi144.stdout:6/401: dread d8/db/f25 [0,4194304] 0 2024-07-26T11:04:41.529 INFO:tasks.workunit.client.0.smithi144.stdout:0/356: sync 2024-07-26T11:04:41.529 INFO:tasks.workunit.client.0.smithi144.stdout:2/269: sync 2024-07-26T11:04:41.529 INFO:tasks.workunit.client.0.smithi144.stdout:2/270: write d0/d15/f40 [697409,92491] 0 2024-07-26T11:04:41.530 INFO:tasks.workunit.client.0.smithi144.stdout:2/271: symlink d0/d15/d20/d28/d2e/d4e/l56 0 2024-07-26T11:04:41.530 INFO:tasks.workunit.client.0.smithi144.stdout:2/272: read d0/f14 [156742,124534] 0 2024-07-26T11:04:41.533 INFO:tasks.workunit.client.0.smithi144.stdout:2/273: rename d0/d15/d20/d28/c50 to d0/d24/d48/c57 0 2024-07-26T11:04:41.536 INFO:tasks.workunit.client.0.smithi144.stdout:2/274: unlink d0/d15/d20/d28/f4d 0 2024-07-26T11:04:41.538 INFO:tasks.workunit.client.0.smithi144.stdout:2/275: unlink d0/f3e 0 2024-07-26T11:04:41.546 INFO:tasks.workunit.client.0.smithi144.stdout:1/369: dwrite d3/d6/da/f48 [0,4194304] 0 2024-07-26T11:04:41.554 INFO:tasks.workunit.client.0.smithi144.stdout:6/402: dwrite d8/d15/f21 [4194304,4194304] 0 2024-07-26T11:04:41.554 INFO:tasks.workunit.client.0.smithi144.stdout:8/310: dwrite d1a/d1c/f24 [0,4194304] 0 2024-07-26T11:04:41.556 INFO:tasks.workunit.client.0.smithi144.stdout:8/311: symlink d1a/d32/d50/l6a 0 2024-07-26T11:04:41.556 INFO:tasks.workunit.client.0.smithi144.stdout:8/312: readlink d1a/d1c/l3b 0 2024-07-26T11:04:41.556 INFO:tasks.workunit.client.0.smithi144.stdout:8/313: rmdir d1a/d1c/d28 39 2024-07-26T11:04:41.559 INFO:tasks.workunit.client.0.smithi144.stdout:6/403: dread d8/db/f32 [0,4194304] 0 2024-07-26T11:04:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:41 smithi160 ceph-mon[36715]: pgmap v527: 129 pgs: 129 active+clean; 514 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s rd, 73 MiB/s wr, 562 op/s 2024-07-26T11:04:41.561 INFO:tasks.workunit.client.0.smithi144.stdout:2/276: dread d0/d15/f40 [0,4194304] 0 2024-07-26T11:04:41.562 INFO:tasks.workunit.client.0.smithi144.stdout:2/277: write d0/d15/f31 [1749472,129463] 0 2024-07-26T11:04:41.562 INFO:tasks.workunit.client.0.smithi144.stdout:2/278: write d0/d6/d9/f26 [1638166,34775] 0 2024-07-26T11:04:41.562 INFO:tasks.workunit.client.0.smithi144.stdout:2/279: write d0/d6/d9/f26 [862013,77580] 0 2024-07-26T11:04:41.562 INFO:tasks.workunit.client.0.smithi144.stdout:6/404: rename d8/d15/d57/d5f/l69 to d8/d4f/d52/l90 0 2024-07-26T11:04:41.564 INFO:tasks.workunit.client.0.smithi144.stdout:6/405: mknod d8/d15/d57/c91 0 2024-07-26T11:04:41.564 INFO:tasks.workunit.client.0.smithi144.stdout:6/406: stat d8/db/c4a 0 2024-07-26T11:04:41.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/407: mkdir d8/d50/d92 0 2024-07-26T11:04:41.565 INFO:tasks.workunit.client.0.smithi144.stdout:6/408: read - d8/d42/f7c zero size 2024-07-26T11:04:41.567 INFO:tasks.workunit.client.0.smithi144.stdout:1/370: dread d3/d6/da/d21/f24 [0,4194304] 0 2024-07-26T11:04:41.613 INFO:tasks.workunit.client.0.smithi144.stdout:8/314: dwrite d1a/f1b [0,4194304] 0 2024-07-26T11:04:41.613 INFO:tasks.workunit.client.0.smithi144.stdout:8/315: fsync f6 0 2024-07-26T11:04:41.618 INFO:tasks.workunit.client.0.smithi144.stdout:2/280: dwrite d0/d6/d9/f26 [0,4194304] 0 2024-07-26T11:04:41.620 INFO:tasks.workunit.client.0.smithi144.stdout:2/281: rmdir d0/d6/d34 39 2024-07-26T11:04:41.620 INFO:tasks.workunit.client.0.smithi144.stdout:2/282: readlink d0/d15/d20/d28/d2e/l4c 0 2024-07-26T11:04:41.620 INFO:tasks.workunit.client.0.smithi144.stdout:2/283: stat d0/d6/d34/f43 0 2024-07-26T11:04:41.620 INFO:tasks.workunit.client.0.smithi144.stdout:2/284: chown d0/d15 1115077221 1 2024-07-26T11:04:41.620 INFO:tasks.workunit.client.0.smithi144.stdout:1/371: dwrite d3/f1e [0,4194304] 0 2024-07-26T11:04:41.623 INFO:tasks.workunit.client.0.smithi144.stdout:6/409: dwrite d8/d4f/d5c/f7b [0,4194304] 0 2024-07-26T11:04:41.623 INFO:tasks.workunit.client.0.smithi144.stdout:6/410: fdatasync d8/d15/d26/f28 0 2024-07-26T11:04:41.634 INFO:tasks.workunit.client.0.smithi144.stdout:2/285: mknod d0/c58 0 2024-07-26T11:04:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:41 smithi183 ceph-mon[37273]: pgmap v527: 129 pgs: 129 active+clean; 514 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s rd, 73 MiB/s wr, 562 op/s 2024-07-26T11:04:41.647 INFO:tasks.workunit.client.0.smithi144.stdout:6/411: link d8/d4f/c6a d8/d4f/d52/c93 0 2024-07-26T11:04:41.647 INFO:tasks.workunit.client.0.smithi144.stdout:6/412: truncate d8/f2d 1436952 0 2024-07-26T11:04:41.648 INFO:tasks.workunit.client.0.smithi144.stdout:6/413: chown d8/d15/d26/f2e 1 1 2024-07-26T11:04:41.648 INFO:tasks.workunit.client.0.smithi144.stdout:6/414: readlink d8/l22 0 2024-07-26T11:04:41.648 INFO:tasks.workunit.client.0.smithi144.stdout:6/415: dread - d8/d15/f5e zero size 2024-07-26T11:04:41.648 INFO:tasks.workunit.client.0.smithi144.stdout:6/416: write d8/f7d [127267,67077] 0 2024-07-26T11:04:41.648 INFO:tasks.workunit.client.0.smithi144.stdout:6/417: dread - d8/d15/d26/f7a zero size 2024-07-26T11:04:41.648 INFO:tasks.workunit.client.0.smithi144.stdout:1/372: dread d3/d6/da/d13/d19/d3a/d69/f28 [0,4194304] 0 2024-07-26T11:04:41.650 INFO:tasks.workunit.client.0.smithi144.stdout:2/286: dread d0/f1e [0,4194304] 0 2024-07-26T11:04:41.650 INFO:tasks.workunit.client.0.smithi144.stdout:2/287: truncate d0/d15/d20/f47 251581 0 2024-07-26T11:04:41.650 INFO:tasks.workunit.client.0.smithi144.stdout:2/288: chown d0/d6/c3c 6 1 2024-07-26T11:04:41.652 INFO:tasks.workunit.client.0.smithi144.stdout:6/418: link d8/d4f/d52/f5a d8/d15/f94 0 2024-07-26T11:04:41.654 INFO:tasks.workunit.client.0.smithi144.stdout:1/373: rmdir d3/d6/da/d13/d22 39 2024-07-26T11:04:41.654 INFO:tasks.workunit.client.0.smithi144.stdout:1/374: readlink d3/d6/le 0 2024-07-26T11:04:41.659 INFO:tasks.workunit.client.0.smithi144.stdout:8/316: dwrite d1a/d1c/d28/d2c/f36 [0,4194304] 0 2024-07-26T11:04:41.661 INFO:tasks.workunit.client.0.smithi144.stdout:8/317: mknod d1a/d1c/d30/d43/d4f/c6b 0 2024-07-26T11:04:41.663 INFO:tasks.workunit.client.0.smithi144.stdout:8/318: rename d1a/d1c/d2f to d1a/d32/d6c 0 2024-07-26T11:04:41.666 INFO:tasks.workunit.client.0.smithi144.stdout:8/319: mknod d1a/d32/d6c/c6d 0 2024-07-26T11:04:41.666 INFO:tasks.workunit.client.0.smithi144.stdout:2/289: write d0/f1 [1520445,54479] 0 2024-07-26T11:04:41.666 INFO:tasks.workunit.client.0.smithi144.stdout:2/290: chown d0/d15/c19 24331511 1 2024-07-26T11:04:41.666 INFO:tasks.workunit.client.0.smithi144.stdout:2/291: readlink d0/d6/d34/l55 0 2024-07-26T11:04:41.668 INFO:tasks.workunit.client.0.smithi144.stdout:8/320: creat d1a/d1c/d28/d2c/d38/d58/f6e x:0 0 0 2024-07-26T11:04:41.668 INFO:tasks.workunit.client.0.smithi144.stdout:8/321: chown d1a/d32/d49/f4a 423830 1 2024-07-26T11:04:41.668 INFO:tasks.workunit.client.0.smithi144.stdout:2/292: creat d0/d24/d2b/f59 x:0 0 0 2024-07-26T11:04:41.669 INFO:tasks.workunit.client.0.smithi144.stdout:8/322: mknod d1a/d32/d6c/c6f 0 2024-07-26T11:04:41.670 INFO:tasks.workunit.client.0.smithi144.stdout:2/293: rename d0/d24/d2b/f52 to d0/d15/d20/d28/d2e/d4e/f5a 0 2024-07-26T11:04:41.671 INFO:tasks.workunit.client.0.smithi144.stdout:8/323: creat d1a/d1c/f70 x:0 0 0 2024-07-26T11:04:41.675 INFO:tasks.workunit.client.0.smithi144.stdout:6/419: dread d8/f53 [0,4194304] 0 2024-07-26T11:04:41.678 INFO:tasks.workunit.client.0.smithi144.stdout:6/420: rename d8/db/f81 to d8/d15/d57/d5f/d88/f95 0 2024-07-26T11:04:41.679 INFO:tasks.workunit.client.0.smithi144.stdout:6/421: fsync d8/d4f/d52/f8d 0 2024-07-26T11:04:41.680 INFO:tasks.workunit.client.0.smithi144.stdout:6/422: creat d8/f96 x:0 0 0 2024-07-26T11:04:41.681 INFO:tasks.workunit.client.0.smithi144.stdout:6/423: creat d8/d4f/d52/d58/f97 x:0 0 0 2024-07-26T11:04:41.682 INFO:tasks.workunit.client.0.smithi144.stdout:6/424: mkdir d8/d50/d98 0 2024-07-26T11:04:41.682 INFO:tasks.workunit.client.0.smithi144.stdout:6/425: fsync d8/f33 0 2024-07-26T11:04:41.682 INFO:tasks.workunit.client.0.smithi144.stdout:6/426: write d8/d42/f7c [22807,46396] 0 2024-07-26T11:04:41.684 INFO:tasks.workunit.client.0.smithi144.stdout:6/427: unlink d8/d15/d26/l46 0 2024-07-26T11:04:41.684 INFO:tasks.workunit.client.0.smithi144.stdout:6/428: write d8/d15/d26/f4c [685971,68912] 0 2024-07-26T11:04:41.684 INFO:tasks.workunit.client.0.smithi144.stdout:6/429: chown d8/f9 1777 1 2024-07-26T11:04:41.685 INFO:tasks.workunit.client.0.smithi144.stdout:6/430: creat d8/d15/d57/d5f/d6e/f99 x:0 0 0 2024-07-26T11:04:41.685 INFO:tasks.workunit.client.0.smithi144.stdout:6/431: dread - d8/f67 zero size 2024-07-26T11:04:41.685 INFO:tasks.workunit.client.0.smithi144.stdout:6/432: fdatasync d8/d15/d57/d5f/d6e/f8e 0 2024-07-26T11:04:41.687 INFO:tasks.workunit.client.0.smithi144.stdout:6/433: mkdir d8/d4f/d52/d9a 0 2024-07-26T11:04:41.687 INFO:tasks.workunit.client.0.smithi144.stdout:8/324: dread f15 [0,4194304] 0 2024-07-26T11:04:41.689 INFO:tasks.workunit.client.0.smithi144.stdout:8/325: creat d1a/d32/f71 x:0 0 0 2024-07-26T11:04:41.692 INFO:tasks.workunit.client.0.smithi144.stdout:6/434: getdents d8/d15 0 2024-07-26T11:04:41.693 INFO:tasks.workunit.client.0.smithi144.stdout:8/326: getdents d1a/d1c/d30/d43/d4f 0 2024-07-26T11:04:41.693 INFO:tasks.workunit.client.0.smithi144.stdout:8/327: fdatasync d1a/d1c/d30/d43/d4c/f54 0 2024-07-26T11:04:41.694 INFO:tasks.workunit.client.0.smithi144.stdout:6/435: creat d8/d50/d98/f9b x:0 0 0 2024-07-26T11:04:41.694 INFO:tasks.workunit.client.0.smithi144.stdout:6/436: read - d8/f67 zero size 2024-07-26T11:04:41.695 INFO:tasks.workunit.client.0.smithi144.stdout:6/437: link d8/d4f/d5c/f7b d8/d4f/d74/f9c 0 2024-07-26T11:04:41.699 INFO:tasks.workunit.client.0.smithi144.stdout:2/294: dread d0/d6/f39 [0,4194304] 0 2024-07-26T11:04:41.699 INFO:tasks.workunit.client.0.smithi144.stdout:1/375: dwrite d3/d6/da/f48 [0,4194304] 0 2024-07-26T11:04:41.700 INFO:tasks.workunit.client.0.smithi144.stdout:1/376: write d3/d6/da/f56 [210313,85660] 0 2024-07-26T11:04:41.717 INFO:tasks.workunit.client.0.smithi144.stdout:8/328: read d1a/d1c/f24 [3497661,32581] 0 2024-07-26T11:04:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:41 smithi144 ceph-mon[32129]: pgmap v527: 129 pgs: 129 active+clean; 514 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s rd, 73 MiB/s wr, 562 op/s 2024-07-26T11:04:41.721 INFO:tasks.workunit.client.0.smithi144.stdout:8/329: rename d1a/d1c/l67 to d1a/d1c/d28/d2c/d38/d58/l72 0 2024-07-26T11:04:41.721 INFO:tasks.workunit.client.0.smithi144.stdout:2/295: dread d0/d6/f1d [0,4194304] 0 2024-07-26T11:04:41.723 INFO:tasks.workunit.client.0.smithi144.stdout:2/296: symlink d0/d15/d20/l5b 0 2024-07-26T11:04:41.726 INFO:tasks.workunit.client.0.smithi144.stdout:2/297: creat d0/d15/d20/d28/f5c x:0 0 0 2024-07-26T11:04:41.728 INFO:tasks.workunit.client.0.smithi144.stdout:8/330: write d1a/f2b [229618,49360] 0 2024-07-26T11:04:41.728 INFO:tasks.workunit.client.0.smithi144.stdout:8/331: truncate d1a/d32/f62 845791 0 2024-07-26T11:04:41.728 INFO:tasks.workunit.client.0.smithi144.stdout:2/298: mkdir d0/d15/d20/d28/d2e/d5d 0 2024-07-26T11:04:41.729 INFO:tasks.workunit.client.0.smithi144.stdout:2/299: rmdir d0/d6 39 2024-07-26T11:04:41.729 INFO:tasks.workunit.client.0.smithi144.stdout:2/300: dread - d0/f3d zero size 2024-07-26T11:04:41.729 INFO:tasks.workunit.client.0.smithi144.stdout:2/301: fdatasync d0/f3d 0 2024-07-26T11:04:41.731 INFO:tasks.workunit.client.0.smithi144.stdout:2/302: unlink d0/f1f 0 2024-07-26T11:04:41.740 INFO:tasks.workunit.client.0.smithi144.stdout:6/438: dwrite d8/d42/f7c [0,4194304] 0 2024-07-26T11:04:41.751 INFO:tasks.workunit.client.0.smithi144.stdout:2/303: dread d0/f14 [0,4194304] 0 2024-07-26T11:04:41.753 INFO:tasks.workunit.client.0.smithi144.stdout:2/304: rename d0/d15/d20/d28/d2e/l51 to d0/d15/d20/d28/d27/l5e 0 2024-07-26T11:04:41.753 INFO:tasks.workunit.client.0.smithi144.stdout:2/305: write d0/d15/f40 [700984,101704] 0 2024-07-26T11:04:41.754 INFO:tasks.workunit.client.0.smithi144.stdout:8/332: read d1a/d1c/d28/f3f [2578403,24721] 0 2024-07-26T11:04:41.755 INFO:tasks.workunit.client.0.smithi144.stdout:2/306: unlink d0/d15/l29 0 2024-07-26T11:04:41.763 INFO:tasks.workunit.client.0.smithi144.stdout:1/377: dwrite d3/f4f [0,4194304] 0 2024-07-26T11:04:41.764 INFO:tasks.workunit.client.0.smithi144.stdout:1/378: mkdir d3/d6/da/d21/d5c/d62/d6e 0 2024-07-26T11:04:41.777 INFO:tasks.workunit.client.0.smithi144.stdout:2/307: read d0/d6/f39 [3311952,37029] 0 2024-07-26T11:04:41.777 INFO:tasks.workunit.client.0.smithi144.stdout:2/308: chown d0/d24/d48/c57 31882853 1 2024-07-26T11:04:41.782 INFO:tasks.workunit.client.0.smithi144.stdout:8/333: dread f15 [0,4194304] 0 2024-07-26T11:04:41.782 INFO:tasks.workunit.client.0.smithi144.stdout:8/334: chown d1a/d1c/d30/d43/d4c/d55 6527403 1 2024-07-26T11:04:41.784 INFO:tasks.workunit.client.0.smithi144.stdout:8/335: rename d1a/d1c/d28 to d1a/d32/d49/d73 0 2024-07-26T11:04:41.787 INFO:tasks.workunit.client.0.smithi144.stdout:6/439: dwrite d8/db/f13 [0,4194304] 0 2024-07-26T11:04:41.787 INFO:tasks.workunit.client.0.smithi144.stdout:6/440: fsync d8/d4f/f62 0 2024-07-26T11:04:41.787 INFO:tasks.workunit.client.0.smithi144.stdout:6/441: write d8/d15/d57/d5f/d88/f8a [928098,13854] 0 2024-07-26T11:04:41.788 INFO:tasks.workunit.client.0.smithi144.stdout:2/309: dread d0/f1e [0,4194304] 0 2024-07-26T11:04:41.789 INFO:tasks.workunit.client.0.smithi144.stdout:6/442: unlink d8/d4f/d52/d58/c66 0 2024-07-26T11:04:41.790 INFO:tasks.workunit.client.0.smithi144.stdout:2/310: mkdir d0/d15/d20/d5f 0 2024-07-26T11:04:41.791 INFO:tasks.workunit.client.0.smithi144.stdout:6/443: symlink d8/d4f/d52/d9a/l9d 0 2024-07-26T11:04:41.791 INFO:tasks.workunit.client.0.smithi144.stdout:6/444: write d8/d4f/d5c/f73 [2139548,125016] 0 2024-07-26T11:04:41.792 INFO:tasks.workunit.client.0.smithi144.stdout:2/311: symlink d0/d15/d20/d5f/l60 0 2024-07-26T11:04:41.793 INFO:tasks.workunit.client.0.smithi144.stdout:6/445: getdents d8/d4f/d52 0 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:3/226: sync 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:5/411: sync 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:7/294: sync 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:5/412: chown d1/l31 4 1 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:7/295: fdatasync d0/d6/d9/d12/f6d 0 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:4/320: sync 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:9/320: sync 2024-07-26T11:04:41.794 INFO:tasks.workunit.client.0.smithi144.stdout:9/321: readlink d9/de/d14/l1b 0 2024-07-26T11:04:41.800 INFO:tasks.workunit.client.0.smithi144.stdout:0/357: sync 2024-07-26T11:04:41.800 INFO:tasks.workunit.client.0.smithi144.stdout:3/227: symlink d1/d5/d25/d26/d30/l3c 0 2024-07-26T11:04:41.800 INFO:tasks.workunit.client.0.smithi144.stdout:4/321: rename d9/d26/d3c/f42 to d9/da/f69 0 2024-07-26T11:04:41.824 INFO:tasks.workunit.client.0.smithi144.stdout:5/413: dread d1/d18/d24/f2a [0,4194304] 0 2024-07-26T11:04:41.824 INFO:tasks.workunit.client.0.smithi144.stdout:5/414: chown d1/d18/d24/d27/c52 1960 1 2024-07-26T11:04:41.844 INFO:tasks.workunit.client.0.smithi144.stdout:2/312: dwrite d0/d6/d9/f33 [0,4194304] 0 2024-07-26T11:04:41.844 INFO:tasks.workunit.client.0.smithi144.stdout:2/313: chown d0/d15/c19 287557061 1 2024-07-26T11:04:41.845 INFO:tasks.workunit.client.0.smithi144.stdout:8/336: dread d1a/f1b [0,4194304] 0 2024-07-26T11:04:41.847 INFO:tasks.workunit.client.0.smithi144.stdout:2/314: mkdir d0/d15/d20/d28/d2e/d5d/d61 0 2024-07-26T11:04:41.847 INFO:tasks.workunit.client.0.smithi144.stdout:8/337: mkdir d1a/d1c/d30/d43/d4c/d55/d74 0 2024-07-26T11:04:41.849 INFO:tasks.workunit.client.0.smithi144.stdout:2/315: creat d0/d24/f62 x:0 0 0 2024-07-26T11:04:41.850 INFO:tasks.workunit.client.0.smithi144.stdout:8/338: rename d1a/d32/f64 to d1a/d1c/d30/d43/d4c/d55/d74/f75 0 2024-07-26T11:04:41.855 INFO:tasks.workunit.client.0.smithi144.stdout:6/446: dwrite d8/f96 [0,4194304] 0 2024-07-26T11:04:41.855 INFO:tasks.workunit.client.0.smithi144.stdout:6/447: chown d8/db/le 1773 1 2024-07-26T11:04:41.855 INFO:tasks.workunit.client.0.smithi144.stdout:4/322: dread d9/d26/f2f [0,4194304] 0 2024-07-26T11:04:41.858 INFO:tasks.workunit.client.0.smithi144.stdout:4/323: symlink d9/d26/d48/l6a 0 2024-07-26T11:04:41.858 INFO:tasks.workunit.client.0.smithi144.stdout:8/339: dread d1a/d1c/f22 [4194304,4194304] 0 2024-07-26T11:04:41.861 INFO:tasks.workunit.client.0.smithi144.stdout:6/448: mknod d8/d15/d57/c9e 0 2024-07-26T11:04:41.861 INFO:tasks.workunit.client.0.smithi144.stdout:6/449: readlink d8/d4f/d52/l90 0 2024-07-26T11:04:41.861 INFO:tasks.workunit.client.0.smithi144.stdout:4/324: creat d9/d26/d48/f6b x:0 0 0 2024-07-26T11:04:41.863 INFO:tasks.workunit.client.0.smithi144.stdout:4/325: creat d9/d26/d48/d50/f6c x:0 0 0 2024-07-26T11:04:41.863 INFO:tasks.workunit.client.0.smithi144.stdout:4/326: stat d9/f4e 0 2024-07-26T11:04:41.865 INFO:tasks.workunit.client.0.smithi144.stdout:4/327: symlink d9/d26/d48/l6d 0 2024-07-26T11:04:41.870 INFO:tasks.workunit.client.0.smithi144.stdout:7/296: dwrite d0/d6/d9/d25/d2f/f33 [0,4194304] 0 2024-07-26T11:04:41.870 INFO:tasks.workunit.client.0.smithi144.stdout:8/340: dread d1a/d32/f62 [0,4194304] 0 2024-07-26T11:04:41.871 INFO:tasks.workunit.client.0.smithi144.stdout:0/358: dwrite d0/d25/d5e/f41 [4194304,4194304] 0 2024-07-26T11:04:41.871 INFO:tasks.workunit.client.0.smithi144.stdout:0/359: chown d0/d3e/d45/f50 1897641250 1 2024-07-26T11:04:41.872 INFO:tasks.workunit.client.0.smithi144.stdout:8/341: symlink d1a/d32/d6c/d3c/l76 0 2024-07-26T11:04:41.877 INFO:tasks.workunit.client.0.smithi144.stdout:8/342: creat d1a/d1c/d30/d43/d4c/f77 x:0 0 0 2024-07-26T11:04:41.877 INFO:tasks.workunit.client.0.smithi144.stdout:8/343: write d1a/d1c/f41 [882816,128826] 0 2024-07-26T11:04:41.877 INFO:tasks.workunit.client.0.smithi144.stdout:8/344: stat d1a/d32/d49/d73/d2c/d38/f5f 0 2024-07-26T11:04:41.898 INFO:tasks.workunit.client.0.smithi144.stdout:3/228: dwrite d1/ff [4194304,4194304] 0 2024-07-26T11:04:41.898 INFO:tasks.workunit.client.0.smithi144.stdout:3/229: stat d1/d5/f6 0 2024-07-26T11:04:41.900 INFO:tasks.workunit.client.0.smithi144.stdout:0/360: write d0/d25/d5e/d1e/d30/f3b [3443332,107108] 0 2024-07-26T11:04:41.900 INFO:tasks.workunit.client.0.smithi144.stdout:2/316: dwrite d0/f1e [0,4194304] 0 2024-07-26T11:04:41.900 INFO:tasks.workunit.client.0.smithi144.stdout:0/361: write d0/d3e/d45/f73 [169028,118877] 0 2024-07-26T11:04:41.901 INFO:tasks.workunit.client.0.smithi144.stdout:0/362: write d0/d3e/d45/f46 [858509,122659] 0 2024-07-26T11:04:41.901 INFO:tasks.workunit.client.0.smithi144.stdout:0/363: stat d0/d25/d5e/d1e/d30/d4b/c59 0 2024-07-26T11:04:41.903 INFO:tasks.workunit.client.0.smithi144.stdout:2/317: unlink d0/d15/d20/d28/d2e/d4e/l53 0 2024-07-26T11:04:41.904 INFO:tasks.workunit.client.0.smithi144.stdout:0/364: rename d0/d25/d5e/c48 to d0/d3e/c7c 0 2024-07-26T11:04:41.905 INFO:tasks.workunit.client.0.smithi144.stdout:2/318: rmdir d0/d6/d9 39 2024-07-26T11:04:41.906 INFO:tasks.workunit.client.0.smithi144.stdout:0/365: getdents d0/d25/d5e/d29 0 2024-07-26T11:04:41.910 INFO:tasks.workunit.client.0.smithi144.stdout:2/319: mkdir d0/d24/d48/d63 0 2024-07-26T11:04:41.911 INFO:tasks.workunit.client.0.smithi144.stdout:8/345: read d1a/f20 [1199715,54179] 0 2024-07-26T11:04:41.911 INFO:tasks.workunit.client.0.smithi144.stdout:0/366: getdents d0/d3e/d45 0 2024-07-26T11:04:41.916 INFO:tasks.workunit.client.0.smithi144.stdout:8/346: mkdir d1a/d1c/d30/d4e/d78 0 2024-07-26T11:04:41.917 INFO:tasks.workunit.client.0.smithi144.stdout:8/347: creat d1a/d32/d49/d73/f79 x:0 0 0 2024-07-26T11:04:41.917 INFO:tasks.workunit.client.0.smithi144.stdout:8/348: fsync d1a/d1c/f24 0 2024-07-26T11:04:41.917 INFO:tasks.workunit.client.0.smithi144.stdout:8/349: rmdir d1a/d32/d6c/d3c 39 2024-07-26T11:04:41.920 INFO:tasks.workunit.client.0.smithi144.stdout:8/350: link c9 d1a/d1c/d30/d4e/d78/c7a 0 2024-07-26T11:04:41.921 INFO:tasks.workunit.client.0.smithi144.stdout:3/230: write d1/d5/d25/d26/f27 [818536,130243] 0 2024-07-26T11:04:41.921 INFO:tasks.workunit.client.0.smithi144.stdout:6/450: dwrite d8/db/f32 [0,4194304] 0 2024-07-26T11:04:41.922 INFO:tasks.workunit.client.0.smithi144.stdout:6/451: chown d8/d15/f5e 0 1 2024-07-26T11:04:41.922 INFO:tasks.workunit.client.0.smithi144.stdout:6/452: dread - d8/d15/f61 zero size 2024-07-26T11:04:41.924 INFO:tasks.workunit.client.0.smithi144.stdout:8/351: write d1a/d32/f62 [117547,37788] 0 2024-07-26T11:04:41.924 INFO:tasks.workunit.client.0.smithi144.stdout:8/352: dread - d1a/d32/d49/d73/d2c/d38/d58/f6e zero size 2024-07-26T11:04:41.924 INFO:tasks.workunit.client.0.smithi144.stdout:2/320: dread d0/d15/f40 [0,4194304] 0 2024-07-26T11:04:41.933 INFO:tasks.workunit.client.0.smithi144.stdout:3/231: mkdir d1/d5/d25/d3d 0 2024-07-26T11:04:41.934 INFO:tasks.workunit.client.0.smithi144.stdout:6/453: creat d8/d50/d98/f9f x:0 0 0 2024-07-26T11:04:41.934 INFO:tasks.workunit.client.0.smithi144.stdout:6/454: fsync d8/f45 0 2024-07-26T11:04:41.935 INFO:tasks.workunit.client.0.smithi144.stdout:4/328: dwrite d9/da/f69 [0,4194304] 0 2024-07-26T11:04:41.935 INFO:tasks.workunit.client.0.smithi144.stdout:4/329: chown d9/f12 303737 1 2024-07-26T11:04:41.936 INFO:tasks.workunit.client.0.smithi144.stdout:4/330: dread - d9/da/f23 zero size 2024-07-26T11:04:41.936 INFO:tasks.workunit.client.0.smithi144.stdout:8/353: rename d1a/d1c/d30 to d1a/d32/d49/d73/d7b 0 2024-07-26T11:04:41.936 INFO:tasks.workunit.client.0.smithi144.stdout:8/354: write f6 [3654944,125375] 0 2024-07-26T11:04:41.936 INFO:tasks.workunit.client.0.smithi144.stdout:2/321: mknod d0/d15/d20/c64 0 2024-07-26T11:04:41.938 INFO:tasks.workunit.client.0.smithi144.stdout:4/331: creat d9/d26/d48/d50/d60/f6e x:0 0 0 2024-07-26T11:04:41.939 INFO:tasks.workunit.client.0.smithi144.stdout:2/322: unlink d0/d15/l38 0 2024-07-26T11:04:41.941 INFO:tasks.workunit.client.0.smithi144.stdout:2/323: rename d0/d15/d20/d28/lf to d0/d24/d48/d63/l65 0 2024-07-26T11:04:41.941 INFO:tasks.workunit.client.0.smithi144.stdout:2/324: fdatasync d0/d15/f40 0 2024-07-26T11:04:41.944 INFO:tasks.workunit.client.0.smithi144.stdout:2/325: fsync d0/f1e 0 2024-07-26T11:04:41.947 INFO:tasks.workunit.client.0.smithi144.stdout:6/455: read d8/d4f/f62 [1421776,30702] 0 2024-07-26T11:04:41.954 INFO:tasks.workunit.client.0.smithi144.stdout:2/326: dread d0/d15/d20/f47 [0,4194304] 0 2024-07-26T11:04:41.954 INFO:tasks.workunit.client.0.smithi144.stdout:2/327: dread - d0/d15/d20/d28/f5c zero size 2024-07-26T11:04:41.961 INFO:tasks.workunit.client.0.smithi144.stdout:8/355: dread d1a/d32/d49/d73/d7b/d43/d4c/f54 [0,4194304] 0 2024-07-26T11:04:41.970 INFO:tasks.workunit.client.0.smithi144.stdout:8/356: symlink d1a/d32/d49/l7c 0 2024-07-26T11:04:41.972 INFO:tasks.workunit.client.0.smithi144.stdout:8/357: symlink d1a/d32/d49/d73/d7b/d43/d4c/d55/d74/l7d 0 2024-07-26T11:04:41.975 INFO:tasks.workunit.client.0.smithi144.stdout:8/358: link d1a/d32/d50/l6a d1a/d32/d49/d73/d2c/d38/d58/l7e 0 2024-07-26T11:04:41.976 INFO:tasks.workunit.client.0.smithi144.stdout:8/359: truncate f11 757330 0 2024-07-26T11:04:41.976 INFO:tasks.workunit.client.0.smithi144.stdout:7/297: dwrite d0/f4b [0,4194304] 0 2024-07-26T11:04:41.976 INFO:tasks.workunit.client.0.smithi144.stdout:7/298: fdatasync d0/d6/d9/d12/f59 0 2024-07-26T11:04:41.976 INFO:tasks.workunit.client.0.smithi144.stdout:6/456: dread d8/f53 [0,4194304] 0 2024-07-26T11:04:41.976 INFO:tasks.workunit.client.0.smithi144.stdout:6/457: readlink d8/d4f/d52/d9a/l9d 0 2024-07-26T11:04:41.976 INFO:tasks.workunit.client.0.smithi144.stdout:0/367: dwrite d0/d25/d5e/d1e/d30/d4b/f58 [0,4194304] 0 2024-07-26T11:04:41.983 INFO:tasks.workunit.client.0.smithi144.stdout:8/360: creat d1a/d32/d6c/f7f x:0 0 0 2024-07-26T11:04:41.984 INFO:tasks.workunit.client.0.smithi144.stdout:6/458: link d8/d15/d57/d5f/d6e/f99 d8/d4f/d5c/fa0 0 2024-07-26T11:04:41.986 INFO:tasks.workunit.client.0.smithi144.stdout:6/459: link d8/f86 d8/db/fa1 0 2024-07-26T11:04:41.986 INFO:tasks.workunit.client.0.smithi144.stdout:6/460: fdatasync d8/d15/f39 0 2024-07-26T11:04:41.986 INFO:tasks.workunit.client.0.smithi144.stdout:6/461: stat d8/db/fa1 0 2024-07-26T11:04:41.987 INFO:tasks.workunit.client.0.smithi144.stdout:2/328: read d0/d15/d20/d28/d2e/d4e/f5a [2590477,3441] 0 2024-07-26T11:04:41.987 INFO:tasks.workunit.client.0.smithi144.stdout:6/462: mknod d8/d15/d57/d5f/d88/ca2 0 2024-07-26T11:04:41.989 INFO:tasks.workunit.client.0.smithi144.stdout:2/329: symlink d0/d24/d48/l66 0 2024-07-26T11:04:41.989 INFO:tasks.workunit.client.0.smithi144.stdout:2/330: write d0/d24/d2b/f59 [235678,66932] 0 2024-07-26T11:04:41.991 INFO:tasks.workunit.client.0.smithi144.stdout:6/463: rename d8/d15/d57/d5f/d6e/f99 to d8/d4f/d52/d9a/fa3 0 2024-07-26T11:04:41.993 INFO:tasks.workunit.client.0.smithi144.stdout:2/331: creat d0/d15/f67 x:0 0 0 2024-07-26T11:04:41.996 INFO:tasks.workunit.client.0.smithi144.stdout:2/332: mkdir d0/d24/d2b/d68 0 2024-07-26T11:04:41.996 INFO:tasks.workunit.client.0.smithi144.stdout:3/232: dwrite d1/d5/d25/d26/f2d [0,4194304] 0 2024-07-26T11:04:41.997 INFO:tasks.workunit.client.0.smithi144.stdout:2/333: write d0/d15/d20/f47 [130604,30558] 0 2024-07-26T11:04:41.998 INFO:tasks.workunit.client.0.smithi144.stdout:2/334: rename d0/d6/c3c to d0/d15/c69 0 2024-07-26T11:04:42.000 INFO:tasks.workunit.client.0.smithi144.stdout:2/335: link d0/d15/d20/d28/f5c d0/d24/d2b/d68/f6a 0 2024-07-26T11:04:42.000 INFO:tasks.workunit.client.0.smithi144.stdout:7/299: write d0/d6/d9/d12/f5d [429646,89476] 0 2024-07-26T11:04:42.001 INFO:tasks.workunit.client.0.smithi144.stdout:7/300: creat d0/d6/d9/d25/f79 x:0 0 0 2024-07-26T11:04:42.003 INFO:tasks.workunit.client.0.smithi144.stdout:7/301: mkdir d0/d6/d9/d25/d67/d7a 0 2024-07-26T11:04:42.005 INFO:tasks.workunit.client.0.smithi144.stdout:7/302: creat d0/d6/d9/d45/d57/f7b x:0 0 0 2024-07-26T11:04:42.005 INFO:tasks.workunit.client.0.smithi144.stdout:0/368: dread d0/d25/d5e/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:04:42.005 INFO:tasks.workunit.client.0.smithi144.stdout:0/369: stat d0/d3e/d45/f73 0 2024-07-26T11:04:42.006 INFO:tasks.workunit.client.0.smithi144.stdout:7/303: rename d0/d6/d9/d45/c6f to d0/d6/d9/d45/d57/c7c 0 2024-07-26T11:04:42.008 INFO:tasks.workunit.client.0.smithi144.stdout:7/304: rename d0/d6/d9/d45/l37 to d0/d6/d9/d45/l7d 0 2024-07-26T11:04:42.008 INFO:tasks.workunit.client.0.smithi144.stdout:7/305: chown d0/d6/d9/d45/d57/c7c 24 1 2024-07-26T11:04:42.008 INFO:tasks.workunit.client.0.smithi144.stdout:7/306: readlink d0/d6/d9/d25/d2f/l32 0 2024-07-26T11:04:42.009 INFO:tasks.workunit.client.0.smithi144.stdout:7/307: creat d0/d6/d9/d45/d57/f7e x:0 0 0 2024-07-26T11:04:42.036 INFO:tasks.workunit.client.0.smithi144.stdout:8/361: dwrite d1a/f2b [0,4194304] 0 2024-07-26T11:04:42.047 INFO:tasks.workunit.client.0.smithi144.stdout:0/370: write d0/d3e/f69 [58863,38252] 0 2024-07-26T11:04:42.047 INFO:tasks.workunit.client.0.smithi144.stdout:0/371: write d0/d3e/d45/d4c/f66 [100519,63875] 0 2024-07-26T11:04:42.048 INFO:tasks.workunit.client.0.smithi144.stdout:2/336: dwrite d0/d24/d2b/d68/f6a [0,4194304] 0 2024-07-26T11:04:42.049 INFO:tasks.workunit.client.0.smithi144.stdout:0/372: creat d0/d3e/f7d x:0 0 0 2024-07-26T11:04:42.049 INFO:tasks.workunit.client.0.smithi144.stdout:0/373: chown d0/d3e/f71 0 1 2024-07-26T11:04:42.055 INFO:tasks.workunit.client.0.smithi144.stdout:2/337: mkdir d0/d15/d20/d28/d2e/d5d/d6b 0 2024-07-26T11:04:42.057 INFO:tasks.workunit.client.0.smithi144.stdout:0/374: mknod d0/d25/c7e 0 2024-07-26T11:04:42.057 INFO:tasks.workunit.client.0.smithi144.stdout:0/375: getdents d0/d25/d5e/d1e/d5c 0 2024-07-26T11:04:42.057 INFO:tasks.workunit.client.0.smithi144.stdout:7/308: dwrite d0/d6/d55/f6a [0,4194304] 0 2024-07-26T11:04:42.064 INFO:tasks.workunit.client.0.smithi144.stdout:0/376: dread d0/d3e/d45/f73 [0,4194304] 0 2024-07-26T11:04:42.064 INFO:tasks.workunit.client.0.smithi144.stdout:2/338: dread d0/d15/f40 [0,4194304] 0 2024-07-26T11:04:42.066 INFO:tasks.workunit.client.0.smithi144.stdout:0/377: symlink d0/d25/d5e/d1e/d30/d6a/d6f/l7f 0 2024-07-26T11:04:42.066 INFO:tasks.workunit.client.0.smithi144.stdout:0/378: truncate d0/d3e/d45/d4c/f79 684739 0 2024-07-26T11:04:42.067 INFO:tasks.workunit.client.0.smithi144.stdout:7/309: dread d0/d6/f56 [0,4194304] 0 2024-07-26T11:04:42.067 INFO:tasks.workunit.client.0.smithi144.stdout:7/310: fsync d0/d6/d9/d64/f65 0 2024-07-26T11:04:42.067 INFO:tasks.workunit.client.0.smithi144.stdout:2/339: mkdir d0/d6/d34/d6c 0 2024-07-26T11:04:42.067 INFO:tasks.workunit.client.0.smithi144.stdout:2/340: write d0/d6/d34/f43 [725624,34846] 0 2024-07-26T11:04:42.067 INFO:tasks.workunit.client.0.smithi144.stdout:2/341: write d0/d6/d9/f26 [1608673,21709] 0 2024-07-26T11:04:42.068 INFO:tasks.workunit.client.0.smithi144.stdout:0/379: symlink d0/d3e/d5f/l80 0 2024-07-26T11:04:42.068 INFO:tasks.workunit.client.0.smithi144.stdout:7/311: creat d0/d6/d9/d12/f7f x:0 0 0 2024-07-26T11:04:42.069 INFO:tasks.workunit.client.0.smithi144.stdout:2/342: rename d0/d15/d20 to d0/d24/d48/d6d 0 2024-07-26T11:04:42.070 INFO:tasks.workunit.client.0.smithi144.stdout:7/312: mknod d0/d6/d55/c80 0 2024-07-26T11:04:42.072 INFO:tasks.workunit.client.0.smithi144.stdout:0/380: rename d0/c3a to d0/d3e/c81 0 2024-07-26T11:04:42.074 INFO:tasks.workunit.client.0.smithi144.stdout:2/343: rename d0/d15/f67 to d0/d24/d2b/f6e 0 2024-07-26T11:04:42.074 INFO:tasks.workunit.client.0.smithi144.stdout:2/344: stat d0/d24/d48/d6d/l5b 0 2024-07-26T11:04:42.074 INFO:tasks.workunit.client.0.smithi144.stdout:2/345: write d0/d6/d9/f33 [4599052,44178] 0 2024-07-26T11:04:42.075 INFO:tasks.workunit.client.0.smithi144.stdout:0/381: creat d0/d25/d5e/d29/f82 x:0 0 0 2024-07-26T11:04:42.079 INFO:tasks.workunit.client.0.smithi144.stdout:8/362: dwrite d1a/d32/d49/d73/d2c/d38/d58/f6e [0,4194304] 0 2024-07-26T11:04:42.080 INFO:tasks.workunit.client.0.smithi144.stdout:8/363: write d1a/d32/d49/d73/f79 [285355,113290] 0 2024-07-26T11:04:42.080 INFO:tasks.workunit.client.0.smithi144.stdout:8/364: chown d1a/d32/d49/d73/d7b/d43/d4f/l56 0 1 2024-07-26T11:04:42.083 INFO:tasks.workunit.client.0.smithi144.stdout:8/365: link d1a/d1c/f2d d1a/d32/d49/d73/f80 0 2024-07-26T11:04:42.084 INFO:tasks.workunit.client.0.smithi144.stdout:8/366: rename d1a/d32/d49/d73/d7b/c44 to d1a/d32/d66/c81 0 2024-07-26T11:04:42.086 INFO:tasks.workunit.client.0.smithi144.stdout:8/367: mkdir d1a/d32/d6c/d82 0 2024-07-26T11:04:42.097 INFO:tasks.workunit.client.0.smithi144.stdout:7/313: write d0/d6/d9/d12/f6d [3785394,94337] 0 2024-07-26T11:04:42.098 INFO:tasks.workunit.client.0.smithi144.stdout:2/346: dread d0/d6/d9/f26 [0,4194304] 0 2024-07-26T11:04:42.098 INFO:tasks.workunit.client.0.smithi144.stdout:2/347: stat d0 0 2024-07-26T11:04:42.099 INFO:tasks.workunit.client.0.smithi144.stdout:7/314: symlink d0/d6/d9/d45/d57/l81 0 2024-07-26T11:04:42.100 INFO:tasks.workunit.client.0.smithi144.stdout:2/348: truncate d0/f14 1000563 0 2024-07-26T11:04:42.102 INFO:tasks.workunit.client.0.smithi144.stdout:2/349: rename d0/d24/d48/c57 to d0/d24/d2b/d68/c6f 0 2024-07-26T11:04:42.103 INFO:tasks.workunit.client.0.smithi144.stdout:2/350: mknod d0/d24/d48/d6d/d28/d2e/d5d/d61/c70 0 2024-07-26T11:04:42.116 INFO:tasks.workunit.client.0.smithi144.stdout:2/351: dread d0/d15/f40 [0,4194304] 0 2024-07-26T11:04:42.125 INFO:tasks.workunit.client.0.smithi144.stdout:0/382: dwrite d0/f28 [0,4194304] 0 2024-07-26T11:04:42.125 INFO:tasks.workunit.client.0.smithi144.stdout:0/383: readlink d0/d3e/d5f/l80 0 2024-07-26T11:04:42.129 INFO:tasks.workunit.client.0.smithi144.stdout:0/384: rename d0/d3e/d45/d4c/f66 to d0/d25/d5e/d1e/d30/d67/f83 0 2024-07-26T11:04:42.137 INFO:tasks.workunit.client.0.smithi144.stdout:2/352: dread d0/d24/d48/d6d/d28/d2e/d4e/f5a [0,4194304] 0 2024-07-26T11:04:42.150 INFO:tasks.workunit.client.0.smithi144.stdout:7/315: dread d0/f49 [0,4194304] 0 2024-07-26T11:04:42.153 INFO:tasks.workunit.client.0.smithi144.stdout:8/368: dwrite d1a/d32/d49/d73/d7b/f39 [0,4194304] 0 2024-07-26T11:04:42.153 INFO:tasks.workunit.client.0.smithi144.stdout:7/316: link d0/c29 d0/d6/d9/d25/d2f/c82 0 2024-07-26T11:04:42.153 INFO:tasks.workunit.client.0.smithi144.stdout:7/317: write d0/d6/d9/f13 [774592,23268] 0 2024-07-26T11:04:42.155 INFO:tasks.workunit.client.0.smithi144.stdout:7/318: creat d0/d6/d9/d3f/f83 x:0 0 0 2024-07-26T11:04:42.155 INFO:tasks.workunit.client.0.smithi144.stdout:7/319: read - d0/d6/d9/d64/f65 zero size 2024-07-26T11:04:42.158 INFO:tasks.workunit.client.0.smithi144.stdout:7/320: creat d0/d71/f84 x:0 0 0 2024-07-26T11:04:42.166 INFO:tasks.workunit.client.0.smithi144.stdout:7/321: rename d0/d6/d9/d25 to d0/d6/d85 0 2024-07-26T11:04:42.173 INFO:tasks.workunit.client.0.smithi144.stdout:7/322: rename d0/d6/d85/c4d to d0/d6/d9/c86 0 2024-07-26T11:04:42.174 INFO:tasks.workunit.client.0.smithi144.stdout:7/323: creat d0/d6/d9/d3f/f87 x:0 0 0 2024-07-26T11:04:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.188+0000 7fac6d200640 1 -- 172.21.15.144:0/3754879903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 msgr2=0x7fac6810f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.188+0000 7fac6d200640 1 --2- 172.21.15.144:0/3754879903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6810f510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fac54009900 tx=0x7fac5402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.192 INFO:tasks.workunit.client.0.smithi144.stdout:0/385: dwrite d0/d25/d5e/d1e/d30/d4b/f58 [0,4194304] 0 2024-07-26T11:04:42.192 INFO:tasks.workunit.client.0.smithi144.stdout:2/353: dwrite d0/f1 [0,4194304] 0 2024-07-26T11:04:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 -- 172.21.15.144:0/3754879903 shutdown_connections 2024-07-26T11:04:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 --2- 172.21.15.144:0/3754879903 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac6810fa50 0x7fac68111eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 --2- 172.21.15.144:0/3754879903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6810f510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 --2- 172.21.15.144:0/3754879903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac680751e0 0x7fac680755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 -- 172.21.15.144:0/3754879903 >> 172.21.15.144:0/3754879903 conn(0x7fac680fef40 msgr2=0x7fac68101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 -- 172.21.15.144:0/3754879903 shutdown_connections 2024-07-26T11:04:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.189+0000 7fac6d200640 1 -- 172.21.15.144:0/3754879903 wait complete. 2024-07-26T11:04:42.195 INFO:tasks.workunit.client.0.smithi144.stdout:0/386: mkdir d0/d25/d5e/d1e/d30/d67/d84 0 2024-07-26T11:04:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.193+0000 7fac6d200640 1 Processor -- start 2024-07-26T11:04:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.193+0000 7fac6d200640 1 -- start start 2024-07-26T11:04:42.196 INFO:tasks.workunit.client.0.smithi144.stdout:2/354: mknod d0/d24/c71 0 2024-07-26T11:04:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac6d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac680751e0 0x7fac6819c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac6d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac6d200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6810fa50 0x7fac681a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac6d200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fac68115620 con 0x7fac6810fa50 2024-07-26T11:04:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac6d200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fac681154a0 con 0x7fac680751e0 2024-07-26T11:04:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac6d200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fac681157a0 con 0x7fac68075b30 2024-07-26T11:04:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.194+0000 7fac65400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6819c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:41312/0 (socket says 172.21.15.144:41312) 2024-07-26T11:04:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 -- 172.21.15.144:0/3957181534 learned_addr learned my addr 172.21.15.144:0/3957181534 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac66800640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6810fa50 0x7fac681a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65e00640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac680751e0 0x7fac6819c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 -- 172.21.15.144:0/3957181534 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac680751e0 msgr2=0x7fac6819c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac680751e0 0x7fac6819c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 -- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6810fa50 msgr2=0x7fac681a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6810fa50 0x7fac681a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.195+0000 7fac65400640 1 -- 172.21.15.144:0/3957181534 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac681a1f90 con 0x7fac68075b30 2024-07-26T11:04:42.199 INFO:tasks.workunit.client.0.smithi144.stdout:0/387: rename d0/d25/d5e/d29/c6b to d0/d25/d5e/d1e/c85 0 2024-07-26T11:04:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.196+0000 7fac65400640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6819c600 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fac5402a910 tx=0x7fac54031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.196+0000 7fac5ea00640 1 -- 172.21.15.144:0/3957181534 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5403f030 con 0x7fac68075b30 2024-07-26T11:04:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.196+0000 7fac5ea00640 1 -- 172.21.15.144:0/3957181534 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac54004490 con 0x7fac68075b30 2024-07-26T11:04:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.196+0000 7fac5ea00640 1 -- 172.21.15.144:0/3957181534 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54035070 con 0x7fac68075b30 2024-07-26T11:04:42.200 INFO:tasks.workunit.client.0.smithi144.stdout:2/355: creat d0/d24/d48/d6d/d28/f72 x:0 0 0 2024-07-26T11:04:42.200 INFO:tasks.workunit.client.0.smithi144.stdout:2/356: chown d0/d24/d48/d6d/d28/d2e 12020 1 2024-07-26T11:04:42.201 INFO:tasks.workunit.client.0.smithi144.stdout:2/357: write d0/d6/d9/f26 [3387061,107314] 0 2024-07-26T11:04:42.201 INFO:tasks.workunit.client.0.smithi144.stdout:2/358: truncate d0/d24/d48/d6d/f47 1236982 0 2024-07-26T11:04:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.197+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac681a2220 con 0x7fac68075b30 2024-07-26T11:04:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.197+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac681a26c0 con 0x7fac68075b30 2024-07-26T11:04:42.201 INFO:tasks.workunit.client.0.smithi144.stdout:2/359: creat d0/d24/d48/d6d/d28/d2e/d5d/d6b/f73 x:0 0 0 2024-07-26T11:04:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.202+0000 7fac7420f740 1 Processor -- start 2024-07-26T11:04:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 -- start start 2024-07-26T11:04:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a3201b0 0x56181a6c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2ff4d0 0x56181a6c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56181a638270 con 0x56181a3201b0 2024-07-26T11:04:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56181a627770 con 0x56181a2f72f0 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac7420f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56181a62b7b0 con 0x56181a2ff4d0 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac65e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac65e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51870/0 (socket says 172.21.15.144:51870) 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.203+0000 7fac65e00640 1 -- 172.21.15.144:0/3278957483 learned_addr learned my addr 172.21.15.144:0/3278957483 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.204+0000 7fac65e00640 1 -- 172.21.15.144:0/3278957483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2ff4d0 msgr2=0x56181a6c44c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.204+0000 7fac65e00640 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2ff4d0 0x56181a6c44c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.204+0000 7fac65e00640 1 -- 172.21.15.144:0/3278957483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a3201b0 msgr2=0x56181a6c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.204+0000 7fac65e00640 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a3201b0 0x56181a6c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.204+0000 7fac65e00640 1 -- 172.21.15.144:0/3278957483 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56181a38b790 con 0x56181a2f72f0 2024-07-26T11:04:42.208 INFO:tasks.workunit.client.0.smithi144.stdout:1/379: sync 2024-07-26T11:04:42.210 INFO:tasks.workunit.client.0.smithi144.stdout:8/369: dwrite d1a/d32/d49/d73/f3f [4194304,4194304] 0 2024-07-26T11:04:42.210 INFO:tasks.workunit.client.0.smithi144.stdout:1/380: creat d3/d6/da/d13/d19/d3a/d3e/f6f x:0 0 0 2024-07-26T11:04:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.209+0000 7fac65e00640 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6c3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fac58009cb0 tx=0x7fac580117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86cefae5e23a4ca9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.209+0000 7fac53400640 1 -- 172.21.15.144:0/3278957483 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5801b470 con 0x56181a2f72f0 2024-07-26T11:04:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.209+0000 7fac53400640 1 -- 172.21.15.144:0/3278957483 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac58016070 con 0x56181a2f72f0 2024-07-26T11:04:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.209+0000 7fac53400640 1 -- 172.21.15.144:0/3278957483 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58012aa0 con 0x56181a2f72f0 2024-07-26T11:04:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.210+0000 7fac5ea00640 1 -- 172.21.15.144:0/3957181534 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fac54042030 con 0x7fac68075b30 2024-07-26T11:04:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.211+0000 7fac7420f740 1 -- 172.21.15.144:0/3278957483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 msgr2=0x56181a6c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.211+0000 7fac7420f740 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6c3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fac58009cb0 tx=0x7fac580117b0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.215 INFO:tasks.workunit.client.0.smithi144.stdout:7/324: write d0/f2 [3707218,78605] 0 2024-07-26T11:04:42.215 INFO:tasks.workunit.client.0.smithi144.stdout:7/325: readlink d0/d6/d85/d67/l74 0 2024-07-26T11:04:42.215 INFO:tasks.workunit.client.0.smithi144.stdout:7/326: stat d0/d6/d85/f79 0 2024-07-26T11:04:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac5ea00640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fac4c0b3d40 0x7fac4c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac5ea00640 1 -- 172.21.15.144:0/3957181534 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fac54104620 con 0x7fac68075b30 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac7420f740 1 -- 172.21.15.144:0/3278957483 shutdown_connections 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac7420f740 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2ff4d0 0x56181a6c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac7420f740 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6c3f80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac7420f740 1 --2- 172.21.15.144:0/3278957483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a3201b0 0x56181a6c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac7420f740 1 -- 172.21.15.144:0/3278957483 >> 172.21.15.144:0/3278957483 conn(0x56181a314300 msgr2=0x56181a6c1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.212+0000 7fac66800640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fac4c0b3d40 0x7fac4c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.213+0000 7fac7420f740 1 -- 172.21.15.144:0/3278957483 shutdown_connections 2024-07-26T11:04:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.213+0000 7fac7420f740 1 -- 172.21.15.144:0/3278957483 wait complete. 2024-07-26T11:04:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.213+0000 7fac7420f740 1 -- start start 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac66800640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fac4c0b3d40 0x7fac4c0b6250 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fac60005fd0 tx=0x7fac60004a40 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 Processor -- start 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 0x56181a6cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6d54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56181a561ab0 con 0x56181a2ff4d0 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56181a5c4d50 con 0x56181a2f72f0 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56181a6d21c0 con 0x56181a3201b0 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.214+0000 7fac7420f740 10 client.0 _ll_register_callbacks cb 0x56181a337a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51884/0 (socket says 172.21.15.144:51884) 2024-07-26T11:04:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 -- 172.21.15.144:0/3880110018 learned_addr learned my addr 172.21.15.144:0/3880110018 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac66800640 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6d54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 -- 172.21.15.144:0/3880110018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 msgr2=0x56181a6d54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6d54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 -- 172.21.15.144:0/3880110018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 msgr2=0x56181a6cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 0x56181a6cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 -- 172.21.15.144:0/3880110018 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56181a404620 con 0x56181a2f72f0 2024-07-26T11:04:42.220 INFO:tasks.workunit.client.0.smithi144.stdout:1/381: mkdir d3/d6/da/d70 0 2024-07-26T11:04:42.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/370: getdents d1a/d32/d49/d73/d7b/d43 0 2024-07-26T11:04:42.220 INFO:tasks.workunit.client.0.smithi144.stdout:9/322: sync 2024-07-26T11:04:42.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/371: chown f6 114 1 2024-07-26T11:04:42.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/372: read - d1a/d32/d49/d73/d7b/d43/d4c/d55/d74/f75 zero size 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.215+0000 7fac65e00640 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6cf730 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fac580125a0 tx=0x7fac58005660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e58ef05a80745a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.216+0000 7fac47e00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.216+0000 7fac47e00640 1 -- 172.21.15.144:0/3880110018 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58029060 con 0x56181a2f72f0 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.216+0000 7fac47e00640 1 -- 172.21.15.144:0/3880110018 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fac580129b0 con 0x56181a2f72f0 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.217+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac58016070 con 0x56181a2f72f0 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.217+0000 7fac47e00640 1 -- 172.21.15.144:0/3880110018 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac58012bb0 con 0x56181a2f72f0 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.218+0000 7fac7420f740 10 client.25832 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.218+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56181a6de330 con 0x56181a2f72f0 2024-07-26T11:04:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac47e00640 1 -- 172.21.15.144:0/3880110018 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac58016070 con 0x56181a2f72f0 2024-07-26T11:04:42.222 INFO:tasks.workunit.client.0.smithi144.stdout:7/327: link d0/d6/d85/d2f/f48 d0/d6/d85/d2f/d47/d5e/f88 0 2024-07-26T11:04:42.222 INFO:tasks.workunit.client.0.smithi144.stdout:9/323: symlink d9/l69 0 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac47e00640 1 -- 172.21.15.144:0/3880110018 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fac5801b470 con 0x56181a2f72f0 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac7420f740 10 client.25832 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac7420f740 10 client.25832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac7420f740 20 client.25832 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac7420f740 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a2ded80 0x56181a6e17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac7420f740 4 client.25832 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.219+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56181a63a1c0 con 0x56181a2ded80 2024-07-26T11:04:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.220+0000 7fac65400640 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a2ded80 0x56181a6e17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.220+0000 7fac47e00640 10 client.25832 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.220+0000 7fac65400640 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a2ded80 0x56181a6e17a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316985eb4808ee8f server_cookie=c8389eed59b4955f in_seq=0 out_seq=0 2024-07-26T11:04:42.224 INFO:tasks.workunit.client.0.smithi144.stdout:7/328: creat d0/d6/d85/d2f/f89 x:0 0 0 2024-07-26T11:04:42.224 INFO:tasks.workunit.client.0.smithi144.stdout:7/329: read d0/d6/d9/f13 [3321426,43049] 0 2024-07-26T11:04:42.224 INFO:tasks.workunit.client.0.smithi144.stdout:7/330: readlink d0/d6/d85/d2f/l3a 0 2024-07-26T11:04:42.225 INFO:tasks.workunit.client.0.smithi144.stdout:7/331: dread - d0/d6/d9/d45/f26 zero size 2024-07-26T11:04:42.225 INFO:tasks.workunit.client.0.smithi144.stdout:5/415: sync 2024-07-26T11:04:42.226 INFO:tasks.workunit.client.0.smithi144.stdout:9/324: mkdir d9/de/d4e/d6a 0 2024-07-26T11:04:42.226 INFO:tasks.workunit.client.0.smithi144.stdout:9/325: truncate d9/fc 2211473 0 2024-07-26T11:04:42.226 INFO:tasks.workunit.client.0.smithi144.stdout:9/326: chown d9/de/f46 1212 1 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.223+0000 7fac47e00640 1 -- 172.21.15.144:0/3880110018 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56181a63a1c0 con 0x56181a2ded80 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.223+0000 7fac47e00640 10 client.25832 handle_command_reply: tid=0 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.223+0000 7fac7420f740 1 client.25832 shutdown 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.223+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 msgr2=0x56181a6cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.223+0000 7fac7420f740 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6cf730 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fac580125a0 tx=0x7fac58005660 comp rx=0 tx=0).stop 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 shutdown_connections 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a2ded80 0x56181a6e17a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6d54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 0x56181a6cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 --2- 172.21.15.144:0/3880110018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6cf730 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 >> 172.21.15.144:0/3880110018 conn(0x56181a314300 msgr2=0x56181a6c00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac47e00640 0 client.25832 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 shutdown_connections 2024-07-26T11:04:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 1 -- 172.21.15.144:0/3880110018 wait complete. 2024-07-26T11:04:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.224+0000 7fac7420f740 20 client.25832 trim_cache size 0 max 16384 2024-07-26T11:04:42.228 INFO:tasks.workunit.client.0.smithi144.stdout:9/327: creat d9/d3b/d5d/f6b x:0 0 0 2024-07-26T11:04:42.228 INFO:tasks.workunit.client.0.smithi144.stdout:9/328: creat d9/de/d14/d3f/d5e/f6c x:0 0 0 2024-07-26T11:04:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.252+0000 7fac7420f740 1 Processor -- start 2024-07-26T11:04:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.252+0000 7fac7420f740 1 -- start start 2024-07-26T11:04:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.253+0000 7fac7420f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6d5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.253+0000 7fac7420f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 0x56181a6dc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.253+0000 7fac7420f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.254+0000 7fac7420f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fac681b96e0 con 0x56181a2ff4d0 2024-07-26T11:04:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.254+0000 7fac7420f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fac681b6fb0 con 0x56181a2f72f0 2024-07-26T11:04:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.254+0000 7fac7420f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fac681b4e80 con 0x56181a3201b0 2024-07-26T11:04:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6c00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:41340/0 (socket says 172.21.15.144:41340) 2024-07-26T11:04:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 -- 172.21.15.144:0/219596232 learned_addr learned my addr 172.21.15.144:0/219596232 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 -- 172.21.15.144:0/219596232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 msgr2=0x56181a6d5420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:04:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6d5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 -- 172.21.15.144:0/219596232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 msgr2=0x56181a6dc2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:04:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 0x56181a6dc2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.255+0000 7fac66800640 1 -- 172.21.15.144:0/219596232 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56181a2defb0 con 0x56181a3201b0 2024-07-26T11:04:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.256+0000 7fac66800640 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6c00b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fac60004080 tx=0x7fac60014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=924260ac3eaa98ef server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.256+0000 7fac52a00640 1 -- 172.21.15.144:0/219596232 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6001d560 con 0x56181a3201b0 2024-07-26T11:04:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.256+0000 7fac52a00640 1 -- 172.21.15.144:0/219596232 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac60011e50 con 0x56181a3201b0 2024-07-26T11:04:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.258+0000 7fac7420f740 1 -- 172.21.15.144:0/219596232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 msgr2=0x56181a6c00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.258+0000 7fac7420f740 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6c00b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fac60004080 tx=0x7fac60014000 comp rx=0 tx=0).stop 2024-07-26T11:04:42.261 INFO:tasks.workunit.client.0.smithi144.stdout:5/416: read d1/d9/d5b/d4d/f63 [3028959,126065] 0 2024-07-26T11:04:42.261 INFO:tasks.workunit.client.0.smithi144.stdout:2/360: dwrite d0/d24/d2b/f59 [0,4194304] 0 2024-07-26T11:04:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.261+0000 7fac7420f740 1 -- 172.21.15.144:0/219596232 shutdown_connections 2024-07-26T11:04:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.261+0000 7fac7420f740 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a3201b0 0x56181a6c00b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.261+0000 7fac7420f740 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2ff4d0 0x56181a6dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.261+0000 7fac7420f740 1 --2- 172.21.15.144:0/219596232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a2f72f0 0x56181a6d5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.261+0000 7fac7420f740 1 -- 172.21.15.144:0/219596232 >> 172.21.15.144:0/219596232 conn(0x56181a314300 msgr2=0x56181a6cdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:42.264 INFO:tasks.workunit.client.0.smithi144.stdout:2/361: mknod d0/d24/d2b/d68/c74 0 2024-07-26T11:04:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.262+0000 7fac7420f740 1 -- 172.21.15.144:0/219596232 shutdown_connections 2024-07-26T11:04:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.262+0000 7fac7420f740 1 -- 172.21.15.144:0/219596232 wait complete. 2024-07-26T11:04:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.263+0000 7fac7420f740 1 -- start start 2024-07-26T11:04:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.263+0000 7fac7420f740 1 Processor -- start 2024-07-26T11:04:42.266 INFO:tasks.workunit.client.0.smithi144.stdout:9/329: dread d9/de/ff [0,4194304] 0 2024-07-26T11:04:42.267 INFO:tasks.workunit.client.0.smithi144.stdout:9/330: fsync d9/d3b/f56 0 2024-07-26T11:04:42.267 INFO:tasks.workunit.client.0.smithi144.stdout:9/331: symlink d9/de/d14/d3f/d5e/l6d 0 2024-07-26T11:04:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.264+0000 7fac7420f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 0x56181a6d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.264+0000 7fac7420f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a3201b0 0x56181a6d6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac7420f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2b41c0 0x56181a747d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac65e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 0x56181a6d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac65e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 0x56181a6d6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:41344/0 (socket says 172.21.15.144:41344) 2024-07-26T11:04:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fac681b54d0 con 0x56181a2b41c0 2024-07-26T11:04:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fac681b68e0 con 0x56181a3201b0 2024-07-26T11:04:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fac681b5b40 con 0x56181a2f72f0 2024-07-26T11:04:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac7420f740 10 client.0 _ll_register_callbacks cb 0x7fac58009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac65400640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a3201b0 0x56181a6d6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac65e00640 1 -- 172.21.15.144:0/2313101978 learned_addr learned my addr 172.21.15.144:0/2313101978 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.265+0000 7fac66800640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2b41c0 0x56181a747d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65e00640 1 -- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a3201b0 msgr2=0x56181a6d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65e00640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a3201b0 0x56181a6d6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65e00640 1 -- 172.21.15.144:0/2313101978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2b41c0 msgr2=0x56181a747d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65e00640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2b41c0 0x56181a747d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65e00640 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56181a6d77b0 con 0x56181a2f72f0 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65400640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a3201b0 0x56181a6d6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac65e00640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 0x56181a6d6a80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fac58006010 tx=0x7fac5801b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26697cca155a1b17 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac47e00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac47e00640 1 -- 172.21.15.144:0/2313101978 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58029040 con 0x56181a2f72f0 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.266+0000 7fac47e00640 1 -- 172.21.15.144:0/2313101978 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fac58021040 con 0x56181a2f72f0 2024-07-26T11:04:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.267+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac60011e50 con 0x56181a2f72f0 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.267+0000 7fac47e00640 1 -- 172.21.15.144:0/2313101978 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac580125d0 con 0x56181a2f72f0 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.267+0000 7fac7420f740 10 client.25813 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.267+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56181a749260 con 0x56181a2f72f0 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.267+0000 7fac47e00640 1 -- 172.21.15.144:0/2313101978 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac60011e50 con 0x56181a2f72f0 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac47e00640 1 -- 172.21.15.144:0/2313101978 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fac58003cb0 con 0x56181a2f72f0 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac7420f740 10 client.25813 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac7420f740 10 client.25813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac7420f740 20 client.25813 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac7420f740 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a749990 0x56181a74bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac7420f740 4 client.25813 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:04:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56181a635c60 con 0x56181a749990 2024-07-26T11:04:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.268+0000 7fac65400640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a749990 0x56181a74bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.269+0000 7fac65400640 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a749990 0x56181a74bdb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42d8c074bd248a57 server_cookie=f91119a3bf142706 in_seq=0 out_seq=0 2024-07-26T11:04:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.269+0000 7fac47e00640 10 client.25813 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:42.274 INFO:tasks.workunit.client.0.smithi144.stdout:0/388: dwrite d0/d25/d5e/d29/f6e [0,4194304] 0 2024-07-26T11:04:42.275 INFO:tasks.workunit.client.0.smithi144.stdout:0/389: fsync d0/d3e/d45/f73 0 2024-07-26T11:04:42.275 INFO:tasks.workunit.client.0.smithi144.stdout:0/390: truncate d0/d3e/d45/d4c/f79 1731655 0 2024-07-26T11:04:42.277 INFO:tasks.workunit.client.0.smithi144.stdout:9/332: creat d9/de/f6e x:0 0 0 2024-07-26T11:04:42.277 INFO:tasks.workunit.client.0.smithi144.stdout:9/333: readlink d9/d3b/l3c 0 2024-07-26T11:04:42.278 INFO:tasks.workunit.client.0.smithi144.stdout:1/382: dread d3/d6/fd [0,4194304] 0 2024-07-26T11:04:42.278 INFO:tasks.workunit.client.0.smithi144.stdout:2/362: dread d0/d6/d34/f43 [0,4194304] 0 2024-07-26T11:04:42.279 INFO:tasks.workunit.client.0.smithi144.stdout:0/391: creat d0/d25/d5e/d29/f86 x:0 0 0 2024-07-26T11:04:42.279 INFO:tasks.workunit.client.0.smithi144.stdout:1/383: rmdir d3/d6/da/d21/d5c 39 2024-07-26T11:04:42.280 INFO:tasks.workunit.client.0.smithi144.stdout:2/363: creat d0/d6/d34/d6c/f75 x:0 0 0 2024-07-26T11:04:42.280 INFO:tasks.workunit.client.0.smithi144.stdout:8/373: dwrite f15 [0,4194304] 0 2024-07-26T11:04:42.292 INFO:tasks.workunit.client.0.smithi144.stdout:0/392: rename d0/d25/d5e/d1e/f3f to d0/d25/d5e/d1e/d30/f87 0 2024-07-26T11:04:42.293 INFO:tasks.workunit.client.0.smithi144.stdout:0/393: chown d0/d25/d5e/c2b 384597461 1 2024-07-26T11:04:42.294 INFO:tasks.workunit.client.0.smithi144.stdout:1/384: creat d3/d6/da/d21/d67/f71 x:0 0 0 2024-07-26T11:04:42.296 INFO:tasks.workunit.client.0.smithi144.stdout:5/417: dread d1/d9/d5b/d4d/f3e [0,4194304] 0 2024-07-26T11:04:42.298 INFO:tasks.workunit.client.0.smithi144.stdout:2/364: unlink d0/d24/d48/d6d/l2a 0 2024-07-26T11:04:42.299 INFO:tasks.workunit.client.0.smithi144.stdout:1/385: truncate d3/d6/f18 691252 0 2024-07-26T11:04:42.300 INFO:tasks.workunit.client.0.smithi144.stdout:2/365: mknod d0/d6/c76 0 2024-07-26T11:04:42.300 INFO:tasks.workunit.client.0.smithi144.stdout:4/332: sync 2024-07-26T11:04:42.300 INFO:tasks.workunit.client.0.smithi144.stdout:6/464: sync 2024-07-26T11:04:42.301 INFO:tasks.workunit.client.0.smithi144.stdout:1/386: mknod d3/d3d/c72 0 2024-07-26T11:04:42.301 INFO:tasks.workunit.client.0.smithi144.stdout:7/332: dwrite d0/d6/f56 [0,4194304] 0 2024-07-26T11:04:42.304 INFO:tasks.workunit.client.0.smithi144.stdout:3/233: sync 2024-07-26T11:04:42.304 INFO:tasks.workunit.client.0.smithi144.stdout:2/366: truncate d0/d24/d48/d6d/d28/d2e/d4e/f5a 3853837 0 2024-07-26T11:04:42.304 INFO:tasks.workunit.client.0.smithi144.stdout:4/333: getdents d9/d26/d48/d50/d60 0 2024-07-26T11:04:42.304 INFO:tasks.workunit.client.0.smithi144.stdout:4/334: write d9/da/f39 [323535,52041] 0 2024-07-26T11:04:42.307 INFO:tasks.workunit.client.0.smithi144.stdout:6/465: write d8/f2d [518235,46979] 0 2024-07-26T11:04:42.307 INFO:tasks.workunit.client.0.smithi144.stdout:7/333: creat d0/d6/d85/d2f/f8a x:0 0 0 2024-07-26T11:04:42.308 INFO:tasks.workunit.client.0.smithi144.stdout:4/335: mkdir d9/d26/d3c/d6f 0 2024-07-26T11:04:42.308 INFO:tasks.workunit.client.0.smithi144.stdout:4/336: write d9/f4e [3945167,112292] 0 2024-07-26T11:04:42.309 INFO:tasks.workunit.client.0.smithi144.stdout:9/334: write d9/de/f4b [2959548,7728] 0 2024-07-26T11:04:42.309 INFO:tasks.workunit.client.0.smithi144.stdout:9/335: write d9/de/d14/d23/f31 [849794,40616] 0 2024-07-26T11:04:42.309 INFO:tasks.workunit.client.0.smithi144.stdout:6/466: mknod d8/d15/d57/d5f/d88/ca4 0 2024-07-26T11:04:42.309 INFO:tasks.workunit.client.0.smithi144.stdout:1/387: rename d3/d6/le to d3/d6/da/d13/l73 0 2024-07-26T11:04:42.322 INFO:tasks.workunit.client.0.smithi144.stdout:4/337: mknod d9/d26/d3c/c70 0 2024-07-26T11:04:42.329 INFO:tasks.workunit.client.0.smithi144.stdout:7/334: dread d0/d6/f2e [0,4194304] 0 2024-07-26T11:04:42.351 INFO:tasks.workunit.client.0.smithi144.stdout:8/374: dwrite d1a/d32/f48 [0,4194304] 0 2024-07-26T11:04:42.354 INFO:tasks.workunit.client.0.smithi144.stdout:1/388: write d3/d6/f11 [2729953,49664] 0 2024-07-26T11:04:42.354 INFO:tasks.workunit.client.0.smithi144.stdout:1/389: truncate d3/d6/f32 8401071 0 2024-07-26T11:04:42.358 INFO:tasks.workunit.client.0.smithi144.stdout:5/418: dwrite d1/d9/d5b/d4d/f79 [0,4194304] 0 2024-07-26T11:04:42.358 INFO:tasks.workunit.client.0.smithi144.stdout:5/419: stat d1/d5/f49 0 2024-07-26T11:04:42.360 INFO:tasks.workunit.client.0.smithi144.stdout:9/336: dwrite d9/de/f46 [0,4194304] 0 2024-07-26T11:04:42.370 INFO:tasks.workunit.client.0.smithi144.stdout:3/234: write d1/f14 [2530420,47653] 0 2024-07-26T11:04:42.371 INFO:tasks.workunit.client.0.smithi144.stdout:0/394: dwrite d0/d25/d5e/d1e/d30/d4b/f58 [0,4194304] 0 2024-07-26T11:04:42.383 INFO:tasks.workunit.client.0.smithi144.stdout:9/337: read d9/de/d14/f22 [977943,97520] 0 2024-07-26T11:04:42.388 INFO:tasks.workunit.client.0.smithi144.stdout:9/338: write d9/de/f3d [325539,92781] 0 2024-07-26T11:04:42.388 INFO:tasks.workunit.client.0.smithi144.stdout:9/339: readlink d9/l36 0 2024-07-26T11:04:42.391 INFO:tasks.workunit.client.0.smithi144.stdout:8/375: dwrite d1a/d32/d6c/d3c/f53 [0,4194304] 0 2024-07-26T11:04:42.391 INFO:tasks.workunit.client.0.smithi144.stdout:8/376: stat d1a/d32/d49/l7c 0 2024-07-26T11:04:42.455 INFO:tasks.workunit.client.0.smithi144.stdout:6/467: link d8/d15/l24 d8/d4f/d52/d58/la5 0 2024-07-26T11:04:42.455 INFO:tasks.workunit.client.0.smithi144.stdout:6/468: getdents d8/d15 0 2024-07-26T11:04:42.461 INFO:tasks.workunit.client.0.smithi144.stdout:7/335: creat d0/d6/f8b x:0 0 0 2024-07-26T11:04:42.497 INFO:tasks.workunit.client.0.smithi144.stdout:1/390: link d3/l42 d3/d6/da/d13/d19/d3a/d69/d36/d37/l74 0 2024-07-26T11:04:42.506 INFO:tasks.workunit.client.0.smithi144.stdout:3/235: mknod d1/d5/d25/d3d/c3e 0 2024-07-26T11:04:42.509 INFO:tasks.workunit.client.0.smithi144.stdout:0/395: symlink d0/d3e/d45/d4c/l88 0 2024-07-26T11:04:42.509 INFO:tasks.workunit.client.0.smithi144.stdout:9/340: symlink d9/l6f 0 2024-07-26T11:04:42.509 INFO:tasks.workunit.client.0.smithi144.stdout:9/341: write d9/de/d14/d23/d39/f66 [382447,1175] 0 2024-07-26T11:04:42.519 INFO:tasks.workunit.client.0.smithi144.stdout:6/469: chown d8/d15/d57/l6c 895 1 2024-07-26T11:04:42.520 INFO:tasks.workunit.client.0.smithi144.stdout:7/336: rmdir d0 39 2024-07-26T11:04:42.531 INFO:tasks.workunit.client.0.smithi144.stdout:0/396: creat d0/d3e/d5f/f89 x:0 0 0 2024-07-26T11:04:42.531 INFO:tasks.workunit.client.0.smithi144.stdout:0/397: fdatasync d0/f8 0 2024-07-26T11:04:42.538 INFO:tasks.workunit.client.0.smithi144.stdout:7/337: truncate d0/f11 1650248 0 2024-07-26T11:04:42.538 INFO:tasks.workunit.client.0.smithi144.stdout:7/338: chown d0/d6/d9/d12/f59 9 1 2024-07-26T11:04:42.632 INFO:tasks.workunit.client.0.smithi144.stdout:0/398: dread d0/d25/d5e/d1e/d30/d67/f83 [0,4194304] 0 2024-07-26T11:04:42.637 INFO:tasks.workunit.client.0.smithi144.stdout:4/338: rmdir d9/d26/d48/d50/d60 39 2024-07-26T11:04:42.638 INFO:tasks.workunit.client.0.smithi144.stdout:4/339: write d9/d26/f2b [1196744,124347] 0 2024-07-26T11:04:42.641 INFO:tasks.workunit.client.0.smithi144.stdout:0/399: unlink d0/d25/d5e/d1e/c49 0 2024-07-26T11:04:42.643 INFO:tasks.workunit.client.0.smithi144.stdout:0/400: stat d0/d25/d5e/d1e/l26 0 2024-07-26T11:04:42.734 INFO:tasks.workunit.client.0.smithi144.stdout:6/470: dread d8/f16 [0,4194304] 0 2024-07-26T11:04:42.734 INFO:tasks.workunit.client.0.smithi144.stdout:6/471: chown d8/d15/d26/f41 11095 1 2024-07-26T11:04:42.808 INFO:tasks.workunit.client.0.smithi144.stdout:5/420: symlink d1/d18/d24/l93 0 2024-07-26T11:04:42.809 INFO:tasks.workunit.client.0.smithi144.stdout:5/421: stat d1/d18/l64 0 2024-07-26T11:04:42.809 INFO:tasks.workunit.client.0.smithi144.stdout:5/422: chown d1/d5/f2e 7 1 2024-07-26T11:04:42.809 INFO:tasks.workunit.client.0.smithi144.stdout:5/423: chown d1/d9/cb 1008166670 1 2024-07-26T11:04:42.823 INFO:tasks.workunit.client.0.smithi144.stdout:4/340: creat d9/d26/d48/f71 x:0 0 0 2024-07-26T11:04:42.824 INFO:tasks.workunit.client.0.smithi144.stdout:5/424: dread d1/f5e [0,4194304] 0 2024-07-26T11:04:42.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/341: creat d9/d57/f72 x:0 0 0 2024-07-26T11:04:42.829 INFO:tasks.workunit.client.0.smithi144.stdout:5/425: link d1/l31 d1/d9/d44/d85/l94 0 2024-07-26T11:04:42.829 INFO:tasks.workunit.client.0.smithi144.stdout:5/426: dread - d1/f8f zero size 2024-07-26T11:04:42.832 INFO:tasks.workunit.client.0.smithi144.stdout:5/427: symlink d1/d9/d44/l95 0 2024-07-26T11:04:42.832 INFO:tasks.workunit.client.0.smithi144.stdout:2/367: rename d0/d15 to d0/d24/d48/d6d/d28/d2e/d77 0 2024-07-26T11:04:42.838 INFO:tasks.workunit.client.0.smithi144.stdout:1/391: mkdir d3/d6/d75 0 2024-07-26T11:04:42.840 INFO:tasks.workunit.client.0.smithi144.stdout:9/342: rename d9/de/d14/d23/f31 to d9/d3b/d5d/f70 0 2024-07-26T11:04:42.840 INFO:tasks.workunit.client.0.smithi144.stdout:9/343: dread - d9/de/d14/d3f/f53 zero size 2024-07-26T11:04:42.840 INFO:tasks.workunit.client.0.smithi144.stdout:9/344: write d9/de/d14/d23/d39/f66 [1016033,125588] 0 2024-07-26T11:04:42.841 INFO:tasks.workunit.client.0.smithi144.stdout:1/392: mknod d3/d6/da/d21/c76 0 2024-07-26T11:04:42.842 INFO:tasks.workunit.client.0.smithi144.stdout:9/345: getdents d9/de/d4e 0 2024-07-26T11:04:42.842 INFO:tasks.workunit.client.0.smithi144.stdout:9/346: chown d9/de/d14/d3f/d5e 3162554 1 2024-07-26T11:04:42.844 INFO:tasks.workunit.client.0.smithi144.stdout:6/472: rename d8/d4f/d52/d9a to d8/db/da6 0 2024-07-26T11:04:42.846 INFO:tasks.workunit.client.0.smithi144.stdout:9/347: creat d9/d3b/f71 x:0 0 0 2024-07-26T11:04:42.846 INFO:tasks.workunit.client.0.smithi144.stdout:9/348: chown d9/de/f52 313 1 2024-07-26T11:04:42.846 INFO:tasks.workunit.client.0.smithi144.stdout:9/349: truncate d9/d3b/f71 610365 0 2024-07-26T11:04:42.847 INFO:tasks.workunit.client.0.smithi144.stdout:4/342: rename l7 to d9/d26/d3c/l73 0 2024-07-26T11:04:42.848 INFO:tasks.workunit.client.0.smithi144.stdout:6/473: mkdir d8/d4f/d5c/da7 0 2024-07-26T11:04:42.851 INFO:tasks.workunit.client.0.smithi144.stdout:4/343: symlink d9/d57/l74 0 2024-07-26T11:04:42.851 INFO:tasks.workunit.client.0.smithi144.stdout:4/344: getdents d9/d26/d3c/d6f 0 2024-07-26T11:04:42.851 INFO:tasks.workunit.client.0.smithi144.stdout:4/345: rename d9 to d9/d26/d75 22 2024-07-26T11:04:42.852 INFO:tasks.workunit.client.0.smithi144.stdout:9/350: link d9/l6f d9/de/d14/l72 0 2024-07-26T11:04:42.854 INFO:tasks.workunit.client.0.smithi144.stdout:7/339: dwrite d0/d6/f52 [0,4194304] 0 2024-07-26T11:04:42.854 INFO:tasks.workunit.client.0.smithi144.stdout:4/346: creat d9/f76 x:0 0 0 2024-07-26T11:04:42.857 INFO:tasks.workunit.client.0.smithi144.stdout:5/428: dread d1/d9/d5b/d4d/f16 [4194304,4194304] 0 2024-07-26T11:04:42.860 INFO:tasks.workunit.client.0.smithi144.stdout:9/351: unlink d9/de/d14/d23/c3e 0 2024-07-26T11:04:42.860 INFO:tasks.workunit.client.0.smithi144.stdout:9/352: readlink d9/d3b/l3c 0 2024-07-26T11:04:42.860 INFO:tasks.workunit.client.0.smithi144.stdout:7/340: symlink d0/d6/d9/d12/l8c 0 2024-07-26T11:04:42.860 INFO:tasks.workunit.client.0.smithi144.stdout:3/236: symlink d1/d5/d25/l3f 0 2024-07-26T11:04:42.861 INFO:tasks.workunit.client.0.smithi144.stdout:8/377: rmdir d1a/d32/d66 39 2024-07-26T11:04:42.861 INFO:tasks.workunit.client.0.smithi144.stdout:5/429: creat d1/f96 x:0 0 0 2024-07-26T11:04:42.861 INFO:tasks.workunit.client.0.smithi144.stdout:3/237: creat d1/d5/d25/d3d/f40 x:0 0 0 2024-07-26T11:04:42.861 INFO:tasks.workunit.client.0.smithi144.stdout:3/238: write d1/d5/f35 [292021,64977] 0 2024-07-26T11:04:42.863 INFO:tasks.workunit.client.0.smithi144.stdout:9/353: symlink d9/d3b/d5d/l73 0 2024-07-26T11:04:42.864 INFO:tasks.workunit.client.0.smithi144.stdout:5/430: creat d1/f97 x:0 0 0 2024-07-26T11:04:42.865 INFO:tasks.workunit.client.0.smithi144.stdout:3/239: creat d1/d5/d25/d26/d37/f41 x:0 0 0 2024-07-26T11:04:42.869 INFO:tasks.workunit.client.0.smithi144.stdout:5/431: mkdir d1/d9/d44/d45/d98 0 2024-07-26T11:04:42.869 INFO:tasks.workunit.client.0.smithi144.stdout:9/354: link d9/c11 d9/de/d14/d3f/d5e/c74 0 2024-07-26T11:04:42.869 INFO:tasks.workunit.client.0.smithi144.stdout:5/432: stat d1/d9/d44/d45/c66 0 2024-07-26T11:04:42.873 INFO:tasks.workunit.client.0.smithi144.stdout:9/355: rename d9/de/d14/c5a to d9/de/d14/d3f/c75 0 2024-07-26T11:04:42.875 INFO:tasks.workunit.client.0.smithi144.stdout:9/356: fsync d9/de/d14/d23/f47 0 2024-07-26T11:04:42.875 INFO:tasks.workunit.client.0.smithi144.stdout:9/357: write d9/de/f6e [820389,39885] 0 2024-07-26T11:04:42.875 INFO:tasks.workunit.client.0.smithi144.stdout:5/433: write d1/d9/d44/d45/f56 [403361,126005] 0 2024-07-26T11:04:42.875 INFO:tasks.workunit.client.0.smithi144.stdout:5/434: unlink d1/f6e 0 2024-07-26T11:04:42.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/435: rename d1/d5/f6f to d1/d18/d24/d27/d71/f99 0 2024-07-26T11:04:42.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/436: truncate d1/f8f 277867 0 2024-07-26T11:04:42.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/437: write d1/d18/d24/d35/f77 [880074,42269] 0 2024-07-26T11:04:42.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/438: chown d1/d18/d24/l93 6 1 2024-07-26T11:04:42.880 INFO:tasks.workunit.client.0.smithi144.stdout:5/439: write d1/d9/f5a [1006560,126132] 0 2024-07-26T11:04:42.881 INFO:tasks.workunit.client.0.smithi144.stdout:5/440: symlink d1/d18/d24/d27/l9a 0 2024-07-26T11:04:42.888 INFO:tasks.workunit.client.0.smithi144.stdout:5/441: link d1/d18/d24/f46 d1/d18/d24/d35/f9b 0 2024-07-26T11:04:42.930 INFO:tasks.workunit.client.0.smithi144.stdout:3/240: dwrite d1/d5/f24 [0,4194304] 0 2024-07-26T11:04:42.931 INFO:tasks.workunit.client.0.smithi144.stdout:3/241: chown d1/d5/d25/d26/f33 5762 1 2024-07-26T11:04:42.935 INFO:tasks.workunit.client.0.smithi144.stdout:3/242: mkdir d1/d5/d25/d3d/d42 0 2024-07-26T11:04:42.937 INFO:tasks.workunit.client.0.smithi144.stdout:3/243: link d1/d5/d25/d26/f2e d1/d5/d25/d26/f43 0 2024-07-26T11:04:42.939 INFO:tasks.workunit.client.0.smithi144.stdout:3/244: mkdir d1/d5/d25/d3d/d42/d44 0 2024-07-26T11:04:42.939 INFO:tasks.workunit.client.0.smithi144.stdout:3/245: fsync d1/f34 0 2024-07-26T11:04:42.941 INFO:tasks.workunit.client.0.smithi144.stdout:3/246: symlink d1/d5/d25/l45 0 2024-07-26T11:04:42.941 INFO:tasks.workunit.client.0.smithi144.stdout:5/442: dread d1/d9/d44/d45/f56 [0,4194304] 0 2024-07-26T11:04:42.941 INFO:tasks.workunit.client.0.smithi144.stdout:5/443: stat d1/d9/d5b/f8c 0 2024-07-26T11:04:42.942 INFO:tasks.workunit.client.0.smithi144.stdout:3/247: mkdir d1/d5/d25/d26/d37/d46 0 2024-07-26T11:04:42.945 INFO:tasks.workunit.client.0.smithi144.stdout:1/393: fsync d3/d6/f11 0 2024-07-26T11:04:42.946 INFO:tasks.workunit.client.0.smithi144.stdout:5/444: creat d1/d18/d24/d35/f9c x:0 0 0 2024-07-26T11:04:42.946 INFO:tasks.workunit.client.0.smithi144.stdout:1/394: creat d3/d6/da/d21/d5c/d62/f77 x:0 0 0 2024-07-26T11:04:42.970 INFO:tasks.workunit.client.0.smithi144.stdout:1/395: read d3/f39 [6976488,57254] 0 2024-07-26T11:04:42.972 INFO:tasks.workunit.client.0.smithi144.stdout:1/396: mkdir d3/d6/da/d78 0 2024-07-26T11:04:42.972 INFO:tasks.workunit.client.0.smithi144.stdout:1/397: chown d3/d53/l6c 1 1 2024-07-26T11:04:42.984 INFO:tasks.workunit.client.0.smithi144.stdout:4/347: dread d9/d26/d3c/f40 [0,4194304] 0 2024-07-26T11:04:42.984 INFO:tasks.workunit.client.0.smithi144.stdout:4/348: write d9/da/f3b [1981582,22758] 0 2024-07-26T11:04:42.984 INFO:tasks.workunit.client.0.smithi144.stdout:4/349: chown d9/d26/f4c 589 1 2024-07-26T11:04:42.984 INFO:tasks.workunit.client.0.smithi144.stdout:4/350: creat d9/d57/f77 x:0 0 0 2024-07-26T11:04:42.987 INFO:tasks.workunit.client.0.smithi144.stdout:1/398: creat d3/d6/da/d13/d19/d3a/f79 x:0 0 0 2024-07-26T11:04:42.988 INFO:tasks.workunit.client.0.smithi144.stdout:5/445: dread d1/d9/d5b/f69 [0,4194304] 0 2024-07-26T11:04:42.988 INFO:tasks.workunit.client.0.smithi144.stdout:8/378: dread d1a/f2b [0,4194304] 0 2024-07-26T11:04:42.991 INFO:tasks.workunit.client.0.smithi144.stdout:1/399: symlink d3/d6/da/d78/l7a 0 2024-07-26T11:04:42.991 INFO:tasks.workunit.client.0.smithi144.stdout:9/358: dwrite d9/de/d14/f2c [0,4194304] 0 2024-07-26T11:04:42.992 INFO:tasks.workunit.client.0.smithi144.stdout:9/359: dread - d9/de/d14/d3f/f61 zero size 2024-07-26T11:04:42.992 INFO:tasks.workunit.client.0.smithi144.stdout:5/446: link d1/f96 d1/d9/d44/f9d 0 2024-07-26T11:04:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.998+0000 7fac47e00640 1 -- 172.21.15.144:0/2313101978 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56181a635c60 con 0x56181a749990 2024-07-26T11:04:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:42.998+0000 7fac47e00640 10 client.25813 handle_command_reply: tid=0 2024-07-26T11:04:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.001+0000 7fac7420f740 1 client.25813 shutdown 2024-07-26T11:04:43.007 INFO:tasks.workunit.client.0.smithi144.stdout:6/474: dread d8/f33 [0,4194304] 0 2024-07-26T11:04:43.007 INFO:tasks.workunit.client.0.smithi144.stdout:0/401: dread d0/d3e/d45/f4a [0,4194304] 0 2024-07-26T11:04:43.007 INFO:tasks.workunit.client.0.smithi144.stdout:7/341: dread d0/d6/d9/d45/f3e [0,4194304] 0 2024-07-26T11:04:43.007 INFO:tasks.workunit.client.0.smithi144.stdout:7/342: getdents d0/d6/d55 0 2024-07-26T11:04:43.007 INFO:tasks.workunit.client.0.smithi144.stdout:7/343: fsync d0/d6/d9/d3f/f83 0 2024-07-26T11:04:43.013 INFO:tasks.workunit.client.0.smithi144.stdout:2/368: dread d0/f1 [0,4194304] 0 2024-07-26T11:04:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.018+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 msgr2=0x56181a6d6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.018+0000 7fac7420f740 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 0x56181a6d6a80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fac58006010 tx=0x7fac5801b720 comp rx=0 tx=0).stop 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 shutdown_connections 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x56181a749990 0x56181a74bdb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56181a2b41c0 0x56181a747d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56181a3201b0 0x56181a6d6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 --2- 172.21.15.144:0/2313101978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56181a2f72f0 0x56181a6d6a80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 >> 172.21.15.144:0/2313101978 conn(0x56181a314300 msgr2=0x56181a6c00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac47e00640 0 client.25813 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 shutdown_connections 2024-07-26T11:04:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.024+0000 7fac7420f740 1 -- 172.21.15.144:0/2313101978 wait complete. 2024-07-26T11:04:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.027+0000 7fac7420f740 20 client.25813 trim_cache size 0 max 16384 2024-07-26T11:04:43.030 INFO:tasks.workunit.client.0.smithi144.stdout:8/379: read d1a/d32/d49/d73/d7b/f39 [532750,123974] 0 2024-07-26T11:04:43.041 INFO:tasks.workunit.client.0.smithi144.stdout:3/248: dwrite d1/f34 [0,4194304] 0 2024-07-26T11:04:43.041 INFO:tasks.workunit.client.0.smithi144.stdout:7/344: creat d0/d6/d85/d2f/d68/f8d x:0 0 0 2024-07-26T11:04:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:04:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:04:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:04:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fac4c0b3d40 msgr2=0x7fac4c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fac4c0b3d40 0x7fac4c0b6250 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fac60005fd0 tx=0x7fac60004a40 comp rx=0 tx=0).stop 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 msgr2=0x7fac6819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6819c600 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fac5402a910 tx=0x7fac54031a20 comp rx=0 tx=0).stop 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 shutdown_connections 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fac4c0b3d40 0x7fac4c0b6250 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6810fa50 0x7fac681a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac68075b30 0x7fac6819c600 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 --2- 172.21.15.144:0/3957181534 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac680751e0 0x7fac6819c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.042+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 >> 172.21.15.144:0/3957181534 conn(0x7fac680fef40 msgr2=0x7fac681095d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:43.047 INFO:tasks.workunit.client.0.smithi144.stdout:8/380: creat d1a/d32/d49/d73/d2c/d38/d58/f83 x:0 0 0 2024-07-26T11:04:43.047 INFO:tasks.workunit.client.0.smithi144.stdout:2/369: fdatasync d0/d24/d2b/d68/f6a 0 2024-07-26T11:04:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.044+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 shutdown_connections 2024-07-26T11:04:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:43.044+0000 7fac6d200640 1 -- 172.21.15.144:0/3957181534 wait complete. 2024-07-26T11:04:43.047 INFO:tasks.workunit.client.0.smithi144.stdout:2/370: link d0/d24/d48/d63/l65 d0/d24/d48/d6d/d28/d2e/l78 0 2024-07-26T11:04:43.059 INFO:tasks.workunit.client.0.smithi144.stdout:1/400: read d3/f1e [1965339,62036] 0 2024-07-26T11:04:43.059 INFO:tasks.workunit.client.0.smithi144.stdout:1/401: write d3/d6/da/d13/d19/d3a/d69/f28 [4202310,46537] 0 2024-07-26T11:04:43.059 INFO:tasks.workunit.client.0.smithi144.stdout:1/402: write d3/f3b [336194,44492] 0 2024-07-26T11:04:43.059 INFO:tasks.workunit.client.0.smithi144.stdout:1/403: chown d3/d6/da/d13/f26 329424169 1 2024-07-26T11:04:43.060 INFO:tasks.workunit.client.0.smithi144.stdout:5/447: dread d1/d9/d5b/d4d/f79 [0,4194304] 0 2024-07-26T11:04:43.060 INFO:tasks.workunit.client.0.smithi144.stdout:5/448: chown d1/d18/d24/c8d 35404 1 2024-07-26T11:04:43.060 INFO:tasks.workunit.client.0.smithi144.stdout:4/351: write d9/f1f [3639355,82105] 0 2024-07-26T11:04:43.061 INFO:tasks.workunit.client.0.smithi144.stdout:4/352: write d9/d26/d48/f6b [813041,17673] 0 2024-07-26T11:04:43.066 INFO:tasks.workunit.client.0.smithi144.stdout:5/449: creat d1/d9/d44/d45/d81/f9e x:0 0 0 2024-07-26T11:04:43.066 INFO:tasks.workunit.client.0.smithi144.stdout:4/353: rename d9/c37 to d9/c78 0 2024-07-26T11:04:43.066 INFO:tasks.workunit.client.0.smithi144.stdout:4/354: mknod d9/d26/d3c/d6f/c79 0 2024-07-26T11:04:43.066 INFO:tasks.workunit.client.0.smithi144.stdout:4/355: write d9/da/f23 [40745,77161] 0 2024-07-26T11:04:43.066 INFO:tasks.workunit.client.0.smithi144.stdout:4/356: rmdir d9/d57 39 2024-07-26T11:04:43.078 INFO:tasks.workunit.client.0.smithi144.stdout:9/360: dread d9/de/d14/d23/d39/f66 [0,4194304] 0 2024-07-26T11:04:43.082 INFO:tasks.workunit.client.0.smithi144.stdout:8/381: dread d1a/d32/d49/d73/d7b/f37 [0,4194304] 0 2024-07-26T11:04:43.094 INFO:tasks.workunit.client.0.smithi144.stdout:8/382: mkdir d1a/d32/d49/d73/d2c/d38/d84 0 2024-07-26T11:04:43.103 INFO:tasks.workunit.client.0.smithi144.stdout:8/383: truncate d1a/d1c/f24 1862483 0 2024-07-26T11:04:43.103 INFO:tasks.workunit.client.0.smithi144.stdout:8/384: fdatasync d1a/f2b 0 2024-07-26T11:04:43.105 INFO:tasks.workunit.client.0.smithi144.stdout:8/385: getdents d1a/d32/d49/d73/d7b/d4e/d78 0 2024-07-26T11:04:43.107 INFO:tasks.workunit.client.0.smithi144.stdout:5/450: dread d1/d18/d24/d35/f9b [0,4194304] 0 2024-07-26T11:04:43.107 INFO:tasks.workunit.client.0.smithi144.stdout:5/451: fdatasync d1/f8f 0 2024-07-26T11:04:43.108 INFO:tasks.workunit.client.0.smithi144.stdout:3/249: write d1/d5/f6 [1430734,61002] 0 2024-07-26T11:04:43.108 INFO:tasks.workunit.client.0.smithi144.stdout:8/386: truncate d1a/d32/d49/d73/d7b/f40 405233 0 2024-07-26T11:04:43.108 INFO:tasks.workunit.client.0.smithi144.stdout:8/387: stat d1a/f21 0 2024-07-26T11:04:43.110 INFO:tasks.workunit.client.0.smithi144.stdout:5/452: rename d1/f84 to d1/d18/d24/d35/f9f 0 2024-07-26T11:04:43.110 INFO:tasks.workunit.client.0.smithi144.stdout:5/453: stat d1/d5/f5c 0 2024-07-26T11:04:43.111 INFO:tasks.workunit.client.0.smithi144.stdout:7/345: dwrite d0/d6/d9/d12/f59 [0,4194304] 0 2024-07-26T11:04:43.113 INFO:tasks.workunit.client.0.smithi144.stdout:3/250: creat d1/d5/d25/d26/f47 x:0 0 0 2024-07-26T11:04:43.113 INFO:tasks.workunit.client.0.smithi144.stdout:1/404: dread d3/d6/da/d21/d5c/f5e [0,4194304] 0 2024-07-26T11:04:43.114 INFO:tasks.workunit.client.0.smithi144.stdout:1/405: truncate d3/f1e 4268873 0 2024-07-26T11:04:43.114 INFO:tasks.workunit.client.0.smithi144.stdout:1/406: truncate d3/d6/da/d21/d67/f71 72400 0 2024-07-26T11:04:43.115 INFO:tasks.workunit.client.0.smithi144.stdout:6/475: dwrite d8/d15/d57/d5f/d6e/f8e [8388608,4194304] 0 2024-07-26T11:04:43.116 INFO:tasks.workunit.client.0.smithi144.stdout:8/388: rename d1a/d1c/c26 to d1a/d32/c85 0 2024-07-26T11:04:43.118 INFO:tasks.workunit.client.0.smithi144.stdout:7/346: creat d0/d6/d9/d45/f8e x:0 0 0 2024-07-26T11:04:43.119 INFO:tasks.workunit.client.0.smithi144.stdout:7/347: readlink d0/d6/d9/d45/l7d 0 2024-07-26T11:04:43.119 INFO:tasks.workunit.client.0.smithi144.stdout:7/348: truncate d0/d6/d9/d3f/f6e 588142 0 2024-07-26T11:04:43.120 INFO:tasks.workunit.client.0.smithi144.stdout:4/357: dwrite d9/f41 [0,4194304] 0 2024-07-26T11:04:43.120 INFO:tasks.workunit.client.0.smithi144.stdout:4/358: read - d9/d57/f77 zero size 2024-07-26T11:04:43.120 INFO:tasks.workunit.client.0.smithi144.stdout:3/251: mkdir d1/d5/d25/d3d/d42/d44/d48 0 2024-07-26T11:04:43.122 INFO:tasks.workunit.client.0.smithi144.stdout:1/407: creat d3/d6/da/d21/d5c/d62/d68/f7b x:0 0 0 2024-07-26T11:04:43.127 INFO:tasks.workunit.client.0.smithi144.stdout:6/476: creat d8/db/fa8 x:0 0 0 2024-07-26T11:04:43.127 INFO:tasks.workunit.client.0.smithi144.stdout:6/477: stat d8/d4f/f62 0 2024-07-26T11:04:43.128 INFO:tasks.workunit.client.0.smithi144.stdout:7/349: rmdir d0/d6/d85/d67/d7a 0 2024-07-26T11:04:43.129 INFO:tasks.workunit.client.0.smithi144.stdout:7/350: write d0/d6/d9/f13 [3879604,75983] 0 2024-07-26T11:04:43.131 INFO:tasks.workunit.client.0.smithi144.stdout:4/359: creat d9/d26/d3c/f7a x:0 0 0 2024-07-26T11:04:43.132 INFO:tasks.workunit.client.0.smithi144.stdout:6/478: rmdir d8/d4f/d52/d58 39 2024-07-26T11:04:43.134 INFO:tasks.workunit.client.0.smithi144.stdout:7/351: truncate d0/f2d 5546723 0 2024-07-26T11:04:43.135 INFO:tasks.workunit.client.0.smithi144.stdout:7/352: rmdir d0 39 2024-07-26T11:04:43.135 INFO:tasks.workunit.client.0.smithi144.stdout:7/353: write d0/d6/d9/f13 [2152435,107907] 0 2024-07-26T11:04:43.136 INFO:tasks.workunit.client.0.smithi144.stdout:7/354: creat d0/d6/d85/d2f/d53/f8f x:0 0 0 2024-07-26T11:04:43.136 INFO:tasks.workunit.client.0.smithi144.stdout:4/360: dread d9/d26/f30 [0,4194304] 0 2024-07-26T11:04:43.137 INFO:tasks.workunit.client.0.smithi144.stdout:7/355: rename d0/d6/d85/d2f/f48 to d0/d6/d9/d45/f90 0 2024-07-26T11:04:43.138 INFO:tasks.workunit.client.0.smithi144.stdout:4/361: mkdir d9/d7b 0 2024-07-26T11:04:43.162 INFO:tasks.workunit.client.0.smithi144.stdout:6/479: write d8/d4f/f63 [998893,104946] 0 2024-07-26T11:04:43.167 INFO:tasks.workunit.client.0.smithi144.stdout:6/480: dread d8/d15/d26/f2e [0,4194304] 0 2024-07-26T11:04:43.179 INFO:tasks.workunit.client.0.smithi144.stdout:1/408: dwrite d3/f2d [0,4194304] 0 2024-07-26T11:04:43.181 INFO:tasks.workunit.client.0.smithi144.stdout:7/356: link d0/f4b d0/d6/d85/d2f/d47/f91 0 2024-07-26T11:04:43.181 INFO:tasks.workunit.client.0.smithi144.stdout:7/357: read - d0/d6/d9/d45/d57/f7b zero size 2024-07-26T11:04:43.182 INFO:tasks.workunit.client.0.smithi144.stdout:2/371: dwrite d0/f14 [0,4194304] 0 2024-07-26T11:04:43.182 INFO:tasks.workunit.client.0.smithi144.stdout:2/372: fdatasync d0/d6/d34/f43 0 2024-07-26T11:04:43.182 INFO:tasks.workunit.client.0.smithi144.stdout:2/373: truncate d0/d24/d48/d6d/d28/d2e/d77/f40 1784914 0 2024-07-26T11:04:43.183 INFO:tasks.workunit.client.0.smithi144.stdout:8/389: dread d1a/d32/d49/d73/d7b/f40 [0,4194304] 0 2024-07-26T11:04:43.183 INFO:tasks.workunit.client.0.smithi144.stdout:8/390: write fe [653751,120437] 0 2024-07-26T11:04:43.184 INFO:tasks.workunit.client.0.smithi144.stdout:4/362: rename d9/d26/d3c/f44 to d9/d26/d3c/d6f/f7c 0 2024-07-26T11:04:43.184 INFO:tasks.workunit.client.0.smithi144.stdout:4/363: chown d9/d26/d48/l6d 5146679 1 2024-07-26T11:04:43.186 INFO:tasks.workunit.client.0.smithi144.stdout:1/409: truncate d3/d6/fd 1418978 0 2024-07-26T11:04:43.191 INFO:tasks.workunit.client.0.smithi144.stdout:8/391: unlink d1a/d32/d49/d73/d7b/d43/d4c/d55/d74/f75 0 2024-07-26T11:04:43.193 INFO:tasks.workunit.client.0.smithi144.stdout:2/374: rename d0/d24/d48/d6d/d28/d2e/l4c to d0/d24/d48/d63/l79 0 2024-07-26T11:04:43.194 INFO:tasks.workunit.client.0.smithi144.stdout:4/364: creat d9/d26/f7d x:0 0 0 2024-07-26T11:04:43.196 INFO:tasks.workunit.client.0.smithi144.stdout:6/481: write d8/d42/f47 [3879536,30274] 0 2024-07-26T11:04:43.197 INFO:tasks.workunit.client.0.smithi144.stdout:8/392: truncate d1a/d32/d49/d73/d2c/d38/f5a 4143836 0 2024-07-26T11:04:43.197 INFO:tasks.workunit.client.0.smithi144.stdout:8/393: chown d1a/d32/d49/d73/d2c/f36 47789 1 2024-07-26T11:04:43.198 INFO:tasks.workunit.client.0.smithi144.stdout:4/365: rename d9/c24 to d9/d26/d48/c7e 0 2024-07-26T11:04:43.198 INFO:tasks.workunit.client.0.smithi144.stdout:4/366: fsync d9/d57/f77 0 2024-07-26T11:04:43.200 INFO:tasks.workunit.client.0.smithi144.stdout:6/482: truncate d8/d4f/f62 753793 0 2024-07-26T11:04:43.201 INFO:tasks.workunit.client.0.smithi144.stdout:8/394: link d1a/d32/d49/d73/d7b/d4e/c65 d1a/d1c/c86 0 2024-07-26T11:04:43.201 INFO:tasks.workunit.client.0.smithi144.stdout:8/395: write d1a/d32/d49/d73/d2c/f4d [4541197,35098] 0 2024-07-26T11:04:43.201 INFO:tasks.workunit.client.0.smithi144.stdout:1/410: fsync d3/d6/fd 0 2024-07-26T11:04:43.202 INFO:tasks.workunit.client.0.smithi144.stdout:6/483: creat d8/d42/fa9 x:0 0 0 2024-07-26T11:04:43.203 INFO:tasks.workunit.client.0.smithi144.stdout:8/396: truncate d1a/d32/d49/d73/f80 193015 0 2024-07-26T11:04:43.203 INFO:tasks.workunit.client.0.smithi144.stdout:7/358: write d0/d6/d85/f41 [2135073,4452] 0 2024-07-26T11:04:43.206 INFO:tasks.workunit.client.0.smithi144.stdout:1/411: rename d3/d6/f32 to d3/d6/da/d21/d5c/d62/f7c 0 2024-07-26T11:04:43.209 INFO:tasks.workunit.client.0.smithi144.stdout:8/397: symlink d1a/d32/d49/d73/d7b/d43/d4c/l87 0 2024-07-26T11:04:43.210 INFO:tasks.workunit.client.0.smithi144.stdout:4/367: dread d9/f15 [0,4194304] 0 2024-07-26T11:04:43.210 INFO:tasks.workunit.client.0.smithi144.stdout:4/368: fdatasync d9/d26/d48/d50/f6c 0 2024-07-26T11:04:43.210 INFO:tasks.workunit.client.0.smithi144.stdout:4/369: mkdir d9/d26/d3c/d6f/d7f 0 2024-07-26T11:04:43.210 INFO:tasks.workunit.client.0.smithi144.stdout:4/370: stat d9/d26/d48/d50 0 2024-07-26T11:04:43.210 INFO:tasks.workunit.client.0.smithi144.stdout:4/371: rmdir d9/d26/d48 39 2024-07-26T11:04:43.211 INFO:tasks.workunit.client.0.smithi144.stdout:4/372: symlink d9/d26/d3c/d6f/l80 0 2024-07-26T11:04:43.213 INFO:tasks.workunit.client.0.smithi144.stdout:4/373: creat d9/d26/d3c/d6f/f81 x:0 0 0 2024-07-26T11:04:43.215 INFO:tasks.workunit.client.0.smithi144.stdout:4/374: truncate f3 2151275 0 2024-07-26T11:04:43.217 INFO:tasks.workunit.client.0.smithi144.stdout:4/375: rename d9/lf to d9/l82 0 2024-07-26T11:04:43.217 INFO:tasks.workunit.client.0.smithi144.stdout:4/376: dread - d9/d26/d48/f4d zero size 2024-07-26T11:04:43.228 INFO:tasks.workunit.client.0.smithi144.stdout:4/377: rmdir d9 39 2024-07-26T11:04:43.228 INFO:tasks.workunit.client.0.smithi144.stdout:4/378: readlink d9/d26/l4f 0 2024-07-26T11:04:43.228 INFO:tasks.workunit.client.0.smithi144.stdout:2/375: dread d0/d24/d48/d6d/d28/f5c [0,4194304] 0 2024-07-26T11:04:43.228 INFO:tasks.workunit.client.0.smithi144.stdout:2/376: rmdir d0/d24/d48/d6d/d28/d2e/d5d 39 2024-07-26T11:04:43.229 INFO:tasks.workunit.client.0.smithi144.stdout:2/377: unlink d0/c58 0 2024-07-26T11:04:43.229 INFO:tasks.workunit.client.0.smithi144.stdout:2/378: truncate d0/d24/d2b/d68/f6a 4875679 0 2024-07-26T11:04:43.229 INFO:tasks.workunit.client.0.smithi144.stdout:2/379: stat d0/d6/c3f 0 2024-07-26T11:04:43.231 INFO:tasks.workunit.client.0.smithi144.stdout:2/380: creat d0/d24/d48/d6d/d28/d2e/d5d/f7a x:0 0 0 2024-07-26T11:04:43.236 INFO:tasks.workunit.client.0.smithi144.stdout:7/359: dread d0/d6/f17 [0,4194304] 0 2024-07-26T11:04:43.236 INFO:tasks.workunit.client.0.smithi144.stdout:8/398: dread d1a/d32/d49/d73/d2c/d38/d58/f6e [0,4194304] 0 2024-07-26T11:04:43.236 INFO:tasks.workunit.client.0.smithi144.stdout:7/360: dread - d0/d6/d9/d3f/f87 zero size 2024-07-26T11:04:43.236 INFO:tasks.workunit.client.0.smithi144.stdout:8/399: fsync d1a/d32/d49/d73/d2c/d38/d58/f83 0 2024-07-26T11:04:43.236 INFO:tasks.workunit.client.0.smithi144.stdout:1/412: write d3/d6/f47 [2624145,93877] 0 2024-07-26T11:04:43.238 INFO:tasks.workunit.client.0.smithi144.stdout:7/361: creat d0/d6/d85/f92 x:0 0 0 2024-07-26T11:04:43.238 INFO:tasks.workunit.client.0.smithi144.stdout:1/413: creat d3/d3d/d58/f7d x:0 0 0 2024-07-26T11:04:43.248 INFO:tasks.workunit.client.0.smithi144.stdout:7/362: mknod d0/d6/d9/d12/c93 0 2024-07-26T11:04:43.249 INFO:tasks.workunit.client.0.smithi144.stdout:7/363: getdents d0/d6/d9/d3f 0 2024-07-26T11:04:43.249 INFO:tasks.workunit.client.0.smithi144.stdout:7/364: fsync d0/d6/d85/d2f/d68/f8d 0 2024-07-26T11:04:43.250 INFO:tasks.workunit.client.0.smithi144.stdout:7/365: symlink d0/d6/d85/d67/l94 0 2024-07-26T11:04:43.251 INFO:tasks.workunit.client.0.smithi144.stdout:7/366: fsync d0/d6/d85/f42 0 2024-07-26T11:04:43.263 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:43 smithi144 ceph-mon[32129]: pgmap v528: 129 pgs: 129 active+clean; 640 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s rd, 98 MiB/s wr, 558 op/s 2024-07-26T11:04:43.268 INFO:tasks.workunit.client.0.smithi144.stdout:1/414: dread d3/d6/da/d21/f24 [0,4194304] 0 2024-07-26T11:04:43.268 INFO:tasks.workunit.client.0.smithi144.stdout:1/415: readlink d3/d6/da/d13/d19/d3a/d69/d36/d37/l74 0 2024-07-26T11:04:43.282 INFO:tasks.workunit.client.0.smithi144.stdout:2/381: dwrite d0/f3d [0,4194304] 0 2024-07-26T11:04:43.286 INFO:tasks.workunit.client.0.smithi144.stdout:2/382: mkdir d0/d24/d7b 0 2024-07-26T11:04:43.289 INFO:tasks.workunit.client.0.smithi144.stdout:6/484: dwrite d8/d15/f21 [0,4194304] 0 2024-07-26T11:04:43.292 INFO:tasks.workunit.client.0.smithi144.stdout:1/416: dread d3/d6/da/d21/d5c/d62/f7c [4194304,4194304] 0 2024-07-26T11:04:43.294 INFO:tasks.workunit.client.0.smithi144.stdout:8/400: dwrite d1a/d32/d49/d73/d2c/d38/f5f [0,4194304] 0 2024-07-26T11:04:43.295 INFO:tasks.workunit.client.0.smithi144.stdout:1/417: truncate d3/d6/da/d13/d19/d3a/d3e/f50 3153273 0 2024-07-26T11:04:43.297 INFO:tasks.workunit.client.0.smithi144.stdout:8/401: rename d1a/f31 to d1a/d32/d49/d73/d2c/f88 0 2024-07-26T11:04:43.298 INFO:tasks.workunit.client.0.smithi144.stdout:1/418: creat d3/d6/da/d78/f7e x:0 0 0 2024-07-26T11:04:43.299 INFO:tasks.workunit.client.0.smithi144.stdout:8/402: symlink d1a/d32/d6c/l89 0 2024-07-26T11:04:43.301 INFO:tasks.workunit.client.0.smithi144.stdout:1/419: creat d3/d6/da/d13/d22/f7f x:0 0 0 2024-07-26T11:04:43.301 INFO:tasks.workunit.client.0.smithi144.stdout:1/420: chown d3/d6/f18 9536 1 2024-07-26T11:04:43.304 INFO:tasks.workunit.client.0.smithi144.stdout:8/403: rmdir d1a/d32/d6c 39 2024-07-26T11:04:43.304 INFO:tasks.workunit.client.0.smithi144.stdout:1/421: creat d3/d6/da/d5b/f80 x:0 0 0 2024-07-26T11:04:43.311 INFO:tasks.workunit.client.0.smithi144.stdout:1/422: dread d3/f3b [0,4194304] 0 2024-07-26T11:04:43.315 INFO:tasks.workunit.client.0.smithi144.stdout:1/423: creat d3/d6/da/d13/d22/f81 x:0 0 0 2024-07-26T11:04:43.318 INFO:tasks.workunit.client.0.smithi144.stdout:1/424: truncate d3/d6/f11 2747867 0 2024-07-26T11:04:43.318 INFO:tasks.workunit.client.0.smithi144.stdout:1/425: fsync d3/d6/da/d21/f24 0 2024-07-26T11:04:43.318 INFO:tasks.workunit.client.0.smithi144.stdout:1/426: readlink d3/d6/da/l31 0 2024-07-26T11:04:43.320 INFO:tasks.workunit.client.0.smithi144.stdout:1/427: creat d3/d6/da/d13/f82 x:0 0 0 2024-07-26T11:04:43.321 INFO:tasks.workunit.client.0.smithi144.stdout:6/485: write d8/d42/f7c [1681089,114442] 0 2024-07-26T11:04:43.321 INFO:tasks.workunit.client.0.smithi144.stdout:6/486: truncate d8/d4f/f7e 85458 0 2024-07-26T11:04:43.321 INFO:tasks.workunit.client.0.smithi144.stdout:6/487: stat d8/f16 0 2024-07-26T11:04:43.334 INFO:tasks.workunit.client.0.smithi144.stdout:6/488: dread d8/f45 [0,4194304] 0 2024-07-26T11:04:43.334 INFO:tasks.workunit.client.0.smithi144.stdout:6/489: readlink d8/l20 0 2024-07-26T11:04:43.338 INFO:tasks.workunit.client.0.smithi144.stdout:4/379: dwrite d9/d26/d48/f71 [0,4194304] 0 2024-07-26T11:04:43.351 INFO:tasks.workunit.client.0.smithi144.stdout:2/383: dwrite d0/f14 [0,4194304] 0 2024-07-26T11:04:43.352 INFO:tasks.workunit.client.0.smithi144.stdout:4/380: dread d9/d26/d48/f6b [0,4194304] 0 2024-07-26T11:04:43.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/384: creat d0/d24/d48/d6d/d28/d2e/d4e/f7c x:0 0 0 2024-07-26T11:04:43.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/385: readlink d0/d24/d48/d6d/d28/d2e/l78 0 2024-07-26T11:04:43.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/386: fsync d0/d24/d48/d6d/f47 0 2024-07-26T11:04:43.354 INFO:tasks.workunit.client.0.smithi144.stdout:7/367: dwrite d0/d6/d9/f13 [0,4194304] 0 2024-07-26T11:04:43.357 INFO:tasks.workunit.client.0.smithi144.stdout:2/387: symlink d0/d24/d48/d6d/d5f/l7d 0 2024-07-26T11:04:43.360 INFO:tasks.workunit.client.0.smithi144.stdout:2/388: rename d0/d6/f39 to d0/d24/d2b/f7e 0 2024-07-26T11:04:43.373 INFO:tasks.workunit.client.0.smithi144.stdout:1/428: dread d3/d6/da/d13/d19/f45 [0,4194304] 0 2024-07-26T11:04:43.373 INFO:tasks.workunit.client.0.smithi144.stdout:1/429: fdatasync d3/d6/da/d5b/f80 0 2024-07-26T11:04:43.379 INFO:tasks.workunit.client.0.smithi144.stdout:1/430: mknod d3/d6/da/d21/d5c/c83 0 2024-07-26T11:04:43.380 INFO:tasks.workunit.client.0.smithi144.stdout:1/431: symlink d3/d6/d75/l84 0 2024-07-26T11:04:43.380 INFO:tasks.workunit.client.0.smithi144.stdout:1/432: write d3/d6/f23 [545312,1132] 0 2024-07-26T11:04:43.381 INFO:tasks.workunit.client.0.smithi144.stdout:8/404: dwrite d1a/d32/d49/d73/f80 [0,4194304] 0 2024-07-26T11:04:43.383 INFO:tasks.workunit.client.0.smithi144.stdout:8/405: creat d1a/d32/d49/d73/d2c/f8a x:0 0 0 2024-07-26T11:04:43.383 INFO:tasks.workunit.client.0.smithi144.stdout:8/406: chown d1a/d32/d49/d73/d2c/f8a 3867017 1 2024-07-26T11:04:43.394 INFO:tasks.workunit.client.0.smithi144.stdout:7/368: dwrite d0/f4c [0,4194304] 0 2024-07-26T11:04:43.399 INFO:tasks.workunit.client.0.smithi144.stdout:4/381: dwrite d9/d26/d3c/d6f/f7c [0,4194304] 0 2024-07-26T11:04:43.399 INFO:tasks.workunit.client.0.smithi144.stdout:4/382: dread - d9/d26/f47 zero size 2024-07-26T11:04:43.403 INFO:tasks.workunit.client.0.smithi144.stdout:7/369: read d0/f11 [106875,24619] 0 2024-07-26T11:04:43.403 INFO:tasks.workunit.client.0.smithi144.stdout:4/383: rename f3 to d9/d26/f83 0 2024-07-26T11:04:43.403 INFO:tasks.workunit.client.0.smithi144.stdout:4/384: write d9/d26/f30 [1416520,38597] 0 2024-07-26T11:04:43.404 INFO:tasks.workunit.client.0.smithi144.stdout:7/370: unlink d0/d6/d9/d12/c93 0 2024-07-26T11:04:43.405 INFO:tasks.workunit.client.0.smithi144.stdout:7/371: mkdir d0/d71/d95 0 2024-07-26T11:04:43.407 INFO:tasks.workunit.client.0.smithi144.stdout:1/433: dread d3/d6/da/d13/d19/d3a/f4b [0,4194304] 0 2024-07-26T11:04:43.407 INFO:tasks.workunit.client.0.smithi144.stdout:2/389: dwrite d0/d24/d2b/f6e [0,4194304] 0 2024-07-26T11:04:43.407 INFO:tasks.workunit.client.0.smithi144.stdout:2/390: readlink d0/d24/d48/d6d/d28/d27/l5e 0 2024-07-26T11:04:43.407 INFO:tasks.workunit.client.0.smithi144.stdout:7/372: link d0/d6/d9/d3f/f58 d0/d6/d85/f96 0 2024-07-26T11:04:43.407 INFO:tasks.workunit.client.0.smithi144.stdout:7/373: write d0/d6/d9/d45/f90 [339834,68695] 0 2024-07-26T11:04:43.407 INFO:tasks.workunit.client.0.smithi144.stdout:7/374: stat d0/d6/d85/d2f/f70 0 2024-07-26T11:04:43.408 INFO:tasks.workunit.client.0.smithi144.stdout:7/375: chown d0/d6/d9/d45/d57/f7b 11441 1 2024-07-26T11:04:43.411 INFO:tasks.workunit.client.0.smithi144.stdout:1/434: mknod d3/d6/da/d21/d5c/d62/d6e/c85 0 2024-07-26T11:04:43.413 INFO:tasks.workunit.client.0.smithi144.stdout:1/435: symlink d3/d6/da/d13/d19/d3a/d69/l86 0 2024-07-26T11:04:43.413 INFO:tasks.workunit.client.0.smithi144.stdout:1/436: fdatasync d3/d6/da/f48 0 2024-07-26T11:04:43.414 INFO:tasks.workunit.client.0.smithi144.stdout:1/437: creat d3/d53/f87 x:0 0 0 2024-07-26T11:04:43.424 INFO:tasks.workunit.client.0.smithi144.stdout:4/385: dread d9/d26/d48/d50/f64 [0,4194304] 0 2024-07-26T11:04:43.427 INFO:tasks.workunit.client.0.smithi144.stdout:7/376: write d0/f49 [4175096,98139] 0 2024-07-26T11:04:43.429 INFO:tasks.workunit.client.0.smithi144.stdout:7/377: creat d0/d6/d9/d64/f97 x:0 0 0 2024-07-26T11:04:43.429 INFO:tasks.workunit.client.0.smithi144.stdout:7/378: mkdir d0/d6/d9/d98 0 2024-07-26T11:04:43.430 INFO:tasks.workunit.client.0.smithi144.stdout:7/379: creat d0/d71/f99 x:0 0 0 2024-07-26T11:04:43.431 INFO:tasks.workunit.client.0.smithi144.stdout:7/380: dread - d0/d6/d85/f42 zero size 2024-07-26T11:04:43.431 INFO:tasks.workunit.client.0.smithi144.stdout:7/381: chown d0/d6/d9/d12/c78 45 1 2024-07-26T11:04:43.464 INFO:tasks.workunit.client.0.smithi144.stdout:4/386: dwrite d9/d26/d3c/d6f/f7c [0,4194304] 0 2024-07-26T11:04:43.464 INFO:tasks.workunit.client.0.smithi144.stdout:4/387: chown d9/d26/d3c/d6f/f81 15 1 2024-07-26T11:04:43.468 INFO:tasks.workunit.client.0.smithi144.stdout:4/388: unlink d9/l82 0 2024-07-26T11:04:43.468 INFO:tasks.workunit.client.0.smithi144.stdout:4/389: truncate d9/f76 322467 0 2024-07-26T11:04:43.472 INFO:tasks.workunit.client.0.smithi144.stdout:4/390: rmdir d9/d26/d3c/d6f/d7f 0 2024-07-26T11:04:43.473 INFO:tasks.workunit.client.0.smithi144.stdout:2/391: dwrite d0/d24/d2b/f59 [0,4194304] 0 2024-07-26T11:04:43.473 INFO:tasks.workunit.client.0.smithi144.stdout:2/392: write d0/d24/d48/d6d/d28/d2e/d4e/f7c [660027,24437] 0 2024-07-26T11:04:43.473 INFO:tasks.workunit.client.0.smithi144.stdout:2/393: fsync d0/d24/d48/d6d/f47 0 2024-07-26T11:04:43.474 INFO:tasks.workunit.client.0.smithi144.stdout:2/394: mknod d0/d24/d48/d6d/d28/d27/c7f 0 2024-07-26T11:04:43.487 INFO:tasks.workunit.client.0.smithi144.stdout:4/391: write d9/d26/f83 [1674689,67535] 0 2024-07-26T11:04:43.487 INFO:tasks.workunit.client.0.smithi144.stdout:4/392: stat d9/d26/f2f 0 2024-07-26T11:04:43.488 INFO:tasks.workunit.client.0.smithi144.stdout:2/395: dread d0/d24/d48/d6d/d28/d2e/d77/f40 [0,4194304] 0 2024-07-26T11:04:43.488 INFO:tasks.workunit.client.0.smithi144.stdout:2/396: fsync d0/d24/d2b/f7e 0 2024-07-26T11:04:43.488 INFO:tasks.workunit.client.0.smithi144.stdout:4/393: getdents d9/d26/d48/d50 0 2024-07-26T11:04:43.490 INFO:tasks.workunit.client.0.smithi144.stdout:2/397: truncate d0/d6/d9/f26 168431 0 2024-07-26T11:04:43.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/438: dwrite d3/d6/da/d13/f82 [0,4194304] 0 2024-07-26T11:04:43.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/439: chown d3/d6/fd 21 1 2024-07-26T11:04:43.493 INFO:tasks.workunit.client.0.smithi144.stdout:7/382: dwrite d0/d6/d85/f41 [0,4194304] 0 2024-07-26T11:04:43.496 INFO:tasks.workunit.client.0.smithi144.stdout:1/440: mkdir d3/d6/d75/d88 0 2024-07-26T11:04:43.496 INFO:tasks.workunit.client.0.smithi144.stdout:1/441: fdatasync d3/d6/da/d13/d19/d3a/f4b 0 2024-07-26T11:04:43.496 INFO:tasks.workunit.client.0.smithi144.stdout:1/442: read d3/d6/da/d13/d22/f30 [3871656,96878] 0 2024-07-26T11:04:43.497 INFO:tasks.workunit.client.0.smithi144.stdout:7/383: mknod d0/d6/d85/d2f/d68/c9a 0 2024-07-26T11:04:43.508 INFO:tasks.workunit.client.0.smithi144.stdout:1/443: rmdir d3/d6/da/d21 39 2024-07-26T11:04:43.508 INFO:tasks.workunit.client.0.smithi144.stdout:1/444: chown d3/d6/da/d13/d19/d3a/d69/d36/d37/l66 47 1 2024-07-26T11:04:43.508 INFO:tasks.workunit.client.0.smithi144.stdout:1/445: chown d3/d6/da/d21/d67 65 1 2024-07-26T11:04:43.508 INFO:tasks.workunit.client.0.smithi144.stdout:1/446: truncate d3/f4f 4279559 0 2024-07-26T11:04:43.538 INFO:tasks.workunit.client.0.smithi144.stdout:2/398: dwrite d0/f14 [0,4194304] 0 2024-07-26T11:04:43.538 INFO:tasks.workunit.client.0.smithi144.stdout:2/399: write d0/d24/d48/d6d/d28/d2e/d5d/f7a [148989,111869] 0 2024-07-26T11:04:43.540 INFO:tasks.workunit.client.0.smithi144.stdout:4/394: dwrite d9/d26/d48/f6b [0,4194304] 0 2024-07-26T11:04:43.541 INFO:tasks.workunit.client.0.smithi144.stdout:2/400: unlink d0/d24/d2b/f6e 0 2024-07-26T11:04:43.541 INFO:tasks.workunit.client.0.smithi144.stdout:2/401: write d0/d24/d48/d6d/f35 [403987,1316] 0 2024-07-26T11:04:43.541 INFO:tasks.workunit.client.0.smithi144.stdout:2/402: write d0/d6/d34/d6c/f75 [201351,19216] 0 2024-07-26T11:04:43.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/403: symlink d0/d24/d48/d6d/l80 0 2024-07-26T11:04:43.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/404: write d0/f1 [5127723,73998] 0 2024-07-26T11:04:43.548 INFO:tasks.workunit.client.0.smithi144.stdout:2/405: rename d0/d24/d48/d6d/d28/c4a to d0/d24/d48/d6d/d28/d2e/d5d/d6b/c81 0 2024-07-26T11:04:43.551 INFO:tasks.workunit.client.0.smithi144.stdout:1/447: dwrite d3/d6/da/d13/d19/d3a/f79 [0,4194304] 0 2024-07-26T11:04:43.551 INFO:tasks.workunit.client.0.smithi144.stdout:1/448: fdatasync d3/d6/da/f4a 0 2024-07-26T11:04:43.553 INFO:tasks.workunit.client.0.smithi144.stdout:1/449: symlink d3/d6/da/d13/d19/d3a/l89 0 2024-07-26T11:04:43.557 INFO:tasks.workunit.client.0.smithi144.stdout:1/450: truncate d3/d6/da/d21/d5c/f5e 532844 0 2024-07-26T11:04:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:43 smithi160 ceph-mon[36715]: pgmap v528: 129 pgs: 129 active+clean; 640 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s rd, 98 MiB/s wr, 558 op/s 2024-07-26T11:04:43.559 INFO:tasks.workunit.client.0.smithi144.stdout:1/451: creat d3/d6/da/d13/d22/f8a x:0 0 0 2024-07-26T11:04:43.562 INFO:tasks.workunit.client.0.smithi144.stdout:1/452: rename d3/l42 to d3/d3d/l8b 0 2024-07-26T11:04:43.562 INFO:tasks.workunit.client.0.smithi144.stdout:1/453: fsync d3/d6/da/d13/d22/f30 0 2024-07-26T11:04:43.562 INFO:tasks.workunit.client.0.smithi144.stdout:1/454: chown d3/c9 7 1 2024-07-26T11:04:43.562 INFO:tasks.workunit.client.0.smithi144.stdout:1/455: fdatasync d3/d6/da/d13/d22/f7f 0 2024-07-26T11:04:43.564 INFO:tasks.workunit.client.0.smithi144.stdout:1/456: mknod d3/d6/da/d21/d67/c8c 0 2024-07-26T11:04:43.564 INFO:tasks.workunit.client.0.smithi144.stdout:1/457: truncate d3/d6/da/d13/d22/f7f 711639 0 2024-07-26T11:04:43.566 INFO:tasks.workunit.client.0.smithi144.stdout:1/458: truncate d3/d6/f47 2832887 0 2024-07-26T11:04:43.566 INFO:tasks.workunit.client.0.smithi144.stdout:1/459: write d3/f1e [3109403,27389] 0 2024-07-26T11:04:43.585 INFO:tasks.workunit.client.0.smithi144.stdout:7/384: dwrite d0/f49 [0,4194304] 0 2024-07-26T11:04:43.585 INFO:tasks.workunit.client.0.smithi144.stdout:7/385: truncate d0/d6/d85/f92 397198 0 2024-07-26T11:04:43.586 INFO:tasks.workunit.client.0.smithi144.stdout:7/386: creat d0/d6/d85/d67/f9b x:0 0 0 2024-07-26T11:04:43.587 INFO:tasks.workunit.client.0.smithi144.stdout:7/387: truncate d0/d6/d9/d64/f97 606465 0 2024-07-26T11:04:43.587 INFO:tasks.workunit.client.0.smithi144.stdout:7/388: stat d0/d71 0 2024-07-26T11:04:43.598 INFO:tasks.workunit.client.0.smithi144.stdout:2/406: dwrite d0/d24/d2b/d68/f6a [0,4194304] 0 2024-07-26T11:04:43.607 INFO:tasks.workunit.client.0.smithi144.stdout:1/460: dwrite d3/d6/f47 [0,4194304] 0 2024-07-26T11:04:43.607 INFO:tasks.workunit.client.0.smithi144.stdout:1/461: write d3/d6/da/d13/d19/d3a/d69/f28 [2427425,66550] 0 2024-07-26T11:04:43.609 INFO:tasks.workunit.client.0.smithi144.stdout:7/389: dread d0/d6/d9/fc [0,4194304] 0 2024-07-26T11:04:43.609 INFO:tasks.workunit.client.0.smithi144.stdout:7/390: truncate d0/d71/f84 1012721 0 2024-07-26T11:04:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:43 smithi183 ceph-mon[37273]: pgmap v528: 129 pgs: 129 active+clean; 640 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s rd, 98 MiB/s wr, 558 op/s 2024-07-26T11:04:43.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/407: dwrite d0/d6/d34/f43 [0,4194304] 0 2024-07-26T11:04:43.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/408: write d0/d6/d9/f26 [825515,48818] 0 2024-07-26T11:04:43.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/409: write d0/d24/d48/d6d/d28/d2e/d77/f31 [797356,12343] 0 2024-07-26T11:04:43.646 INFO:tasks.workunit.client.0.smithi144.stdout:2/410: rename d0/d24/d48/d6d/d28/d2e/d4e to d0/d24/d2b/d68/d82 0 2024-07-26T11:04:43.647 INFO:tasks.workunit.client.0.smithi144.stdout:2/411: mknod d0/d24/c83 0 2024-07-26T11:04:43.648 INFO:tasks.workunit.client.0.smithi144.stdout:2/412: write d0/d24/d48/d6d/d28/d2e/d5d/f7a [982504,56114] 0 2024-07-26T11:04:43.654 INFO:tasks.workunit.client.0.smithi144.stdout:2/413: write d0/d24/d48/d6d/d28/d2e/d5d/f7a [854664,37700] 0 2024-07-26T11:04:43.655 INFO:tasks.workunit.client.0.smithi144.stdout:2/414: mkdir d0/d24/d48/d6d/d28/d2e/d77/d84 0 2024-07-26T11:04:43.656 INFO:tasks.workunit.client.0.smithi144.stdout:2/415: creat d0/d24/d48/d6d/d5f/f85 x:0 0 0 2024-07-26T11:04:43.656 INFO:tasks.workunit.client.0.smithi144.stdout:2/416: chown d0/d24/d2b/d68/d82/l56 50692 1 2024-07-26T11:04:43.660 INFO:tasks.workunit.client.0.smithi144.stdout:1/462: dwrite d3/d6/f23 [0,4194304] 0 2024-07-26T11:04:43.661 INFO:tasks.workunit.client.0.smithi144.stdout:7/391: dwrite d0/f4c [0,4194304] 0 2024-07-26T11:04:43.661 INFO:tasks.workunit.client.0.smithi144.stdout:7/392: chown d0/d6/d85/f96 9054903 1 2024-07-26T11:04:43.663 INFO:tasks.workunit.client.0.smithi144.stdout:1/463: creat d3/d6/da/d5b/f8d x:0 0 0 2024-07-26T11:04:43.664 INFO:tasks.workunit.client.0.smithi144.stdout:1/464: symlink d3/d3d/d58/l8e 0 2024-07-26T11:04:43.664 INFO:tasks.workunit.client.0.smithi144.stdout:1/465: write d3/d6/da/d13/d19/d3a/d69/f5a [386808,69642] 0 2024-07-26T11:04:43.666 INFO:tasks.workunit.client.0.smithi144.stdout:7/393: dread d0/d6/d85/f92 [0,4194304] 0 2024-07-26T11:04:43.666 INFO:tasks.workunit.client.0.smithi144.stdout:7/394: fdatasync d0/d6/f8 0 2024-07-26T11:04:43.669 INFO:tasks.workunit.client.0.smithi144.stdout:1/466: rename d3/d6/f11 to d3/d6/da/d21/d5c/d62/d6e/f8f 0 2024-07-26T11:04:43.677 INFO:tasks.workunit.client.0.smithi144.stdout:2/417: read d0/d24/d48/d6d/d28/f5c [3320957,124838] 0 2024-07-26T11:04:43.698 INFO:tasks.workunit.client.0.smithi144.stdout:1/467: dread d3/d6/da/f48 [0,4194304] 0 2024-07-26T11:04:43.709 INFO:tasks.workunit.client.0.smithi144.stdout:7/395: dwrite d0/d6/d85/f42 [0,4194304] 0 2024-07-26T11:04:43.721 INFO:tasks.workunit.client.0.smithi144.stdout:2/418: dwrite d0/d6/d34/f43 [0,4194304] 0 2024-07-26T11:04:43.733 INFO:tasks.workunit.client.0.smithi144.stdout:2/419: dread d0/d24/d48/d6d/f35 [0,4194304] 0 2024-07-26T11:04:43.733 INFO:tasks.workunit.client.0.smithi144.stdout:2/420: getdents d0/d24/d48/d6d/d28/d2e/d77/d84 0 2024-07-26T11:04:43.733 INFO:tasks.workunit.client.0.smithi144.stdout:2/421: read d0/d6/f1d [1052158,55795] 0 2024-07-26T11:04:43.734 INFO:tasks.workunit.client.0.smithi144.stdout:2/422: creat d0/d6/d34/d6c/f86 x:0 0 0 2024-07-26T11:04:43.735 INFO:tasks.workunit.client.0.smithi144.stdout:2/423: rmdir d0/d6/d34 39 2024-07-26T11:04:43.735 INFO:tasks.workunit.client.0.smithi144.stdout:2/424: chown d0/d6/c3f 1 1 2024-07-26T11:04:43.736 INFO:tasks.workunit.client.0.smithi144.stdout:2/425: link d0/d24/d48/d6d/d28/d2e/d77/f31 d0/d24/d7b/f87 0 2024-07-26T11:04:43.736 INFO:tasks.workunit.client.0.smithi144.stdout:2/426: write d0/d24/d2b/d68/d82/f5a [1167428,78210] 0 2024-07-26T11:04:43.753 INFO:tasks.workunit.client.0.smithi144.stdout:7/396: dwrite d0/d6/f52 [0,4194304] 0 2024-07-26T11:04:43.753 INFO:tasks.workunit.client.0.smithi144.stdout:7/397: write d0/d6/d9/d45/f90 [489595,75319] 0 2024-07-26T11:04:43.755 INFO:tasks.workunit.client.0.smithi144.stdout:1/468: dwrite d3/d6/f47 [0,4194304] 0 2024-07-26T11:04:43.789 INFO:tasks.workunit.client.0.smithi144.stdout:2/427: dwrite d0/d6/d34/f43 [0,4194304] 0 2024-07-26T11:04:43.792 INFO:tasks.workunit.client.0.smithi144.stdout:1/469: dwrite d3/d6/da/d13/d19/d3a/f79 [0,4194304] 0 2024-07-26T11:04:43.794 INFO:tasks.workunit.client.0.smithi144.stdout:1/470: rename d3/d6/da/f56 to d3/d6/da/d70/f90 0 2024-07-26T11:04:43.796 INFO:tasks.workunit.client.0.smithi144.stdout:1/471: mknod d3/d6/da/d13/d19/d3a/d69/d36/d37/c91 0 2024-07-26T11:04:43.802 INFO:tasks.workunit.client.0.smithi144.stdout:7/398: dwrite d0/d6/d9/d3f/f58 [0,4194304] 0 2024-07-26T11:04:43.805 INFO:tasks.workunit.client.0.smithi144.stdout:7/399: creat d0/d6/d9/d12/d6c/f9c x:0 0 0 2024-07-26T11:04:43.809 INFO:tasks.workunit.client.0.smithi144.stdout:7/400: link d0/d6/d9/d45/f3e d0/d6/d85/d67/f9d 0 2024-07-26T11:04:43.810 INFO:tasks.workunit.client.0.smithi144.stdout:7/401: rename d0/l23 to d0/d6/d85/l9e 0 2024-07-26T11:04:43.812 INFO:tasks.workunit.client.0.smithi144.stdout:7/402: creat d0/d6/d85/d67/f9f x:0 0 0 2024-07-26T11:04:43.813 INFO:tasks.workunit.client.0.smithi144.stdout:7/403: creat d0/d6/d85/d2f/d47/fa0 x:0 0 0 2024-07-26T11:04:43.815 INFO:tasks.workunit.client.0.smithi144.stdout:2/428: write d0/f1e [975437,92308] 0 2024-07-26T11:04:43.815 INFO:tasks.workunit.client.0.smithi144.stdout:2/429: fdatasync d0/d24/d48/d6d/d5f/f85 0 2024-07-26T11:04:43.822 INFO:tasks.workunit.client.0.smithi144.stdout:2/430: dread d0/d24/d48/d6d/f35 [0,4194304] 0 2024-07-26T11:04:43.836 INFO:tasks.workunit.client.0.smithi144.stdout:1/472: dwrite d3/d3d/d58/f7d [0,4194304] 0 2024-07-26T11:04:43.837 INFO:tasks.workunit.client.0.smithi144.stdout:1/473: read d3/d6/da/d13/d22/f40 [33220,111828] 0 2024-07-26T11:04:43.839 INFO:tasks.workunit.client.0.smithi144.stdout:1/474: rename d3/f1e to d3/d6/da/d21/d5c/d62/d68/f92 0 2024-07-26T11:04:43.839 INFO:tasks.workunit.client.0.smithi144.stdout:1/475: stat d3/d6/da/l31 0 2024-07-26T11:04:43.843 INFO:tasks.workunit.client.0.smithi144.stdout:2/431: dread d0/d6/f1d [0,4194304] 0 2024-07-26T11:04:43.848 INFO:tasks.workunit.client.0.smithi144.stdout:2/432: truncate d0/d6/d9/f26 919817 0 2024-07-26T11:04:43.849 INFO:tasks.workunit.client.0.smithi144.stdout:2/433: mknod d0/d24/d2b/d68/d82/c88 0 2024-07-26T11:04:43.851 INFO:tasks.workunit.client.0.smithi144.stdout:2/434: creat d0/d6/f89 x:0 0 0 2024-07-26T11:04:43.851 INFO:tasks.workunit.client.0.smithi144.stdout:2/435: chown d0/d24/d2b/d68/d82/l56 246588870 1 2024-07-26T11:04:43.851 INFO:tasks.workunit.client.0.smithi144.stdout:2/436: rmdir d0/d24 39 2024-07-26T11:04:43.852 INFO:tasks.workunit.client.0.smithi144.stdout:2/437: write d0/d24/d48/d6d/d28/d2e/d5d/f7a [830565,78679] 0 2024-07-26T11:04:43.860 INFO:tasks.workunit.client.0.smithi144.stdout:1/476: dread d3/d6/fd [0,4194304] 0 2024-07-26T11:04:43.861 INFO:tasks.workunit.client.0.smithi144.stdout:1/477: creat d3/d6/d75/f93 x:0 0 0 2024-07-26T11:04:43.865 INFO:tasks.workunit.client.0.smithi144.stdout:1/478: read d3/d6/da/d21/d67/f71 [10043,63971] 0 2024-07-26T11:04:43.964 INFO:tasks.workunit.client.0.smithi144.stdout:7/404: dwrite d0/d6/d85/d2f/f39 [0,4194304] 0 2024-07-26T11:04:43.965 INFO:tasks.workunit.client.0.smithi144.stdout:7/405: mkdir d0/d71/da1 0 2024-07-26T11:04:43.965 INFO:tasks.workunit.client.0.smithi144.stdout:7/406: chown d0/d6/d85/d67/f9f 124 1 2024-07-26T11:04:43.972 INFO:tasks.workunit.client.0.smithi144.stdout:2/438: truncate d0/d24/d2b/d68/d82/f5a 461872 0 2024-07-26T11:04:43.973 INFO:tasks.workunit.client.0.smithi144.stdout:2/439: write d0/f1 [5500392,121049] 0 2024-07-26T11:04:43.985 INFO:tasks.workunit.client.0.smithi144.stdout:2/440: symlink d0/d24/d48/d6d/d5f/l8a 0 2024-07-26T11:04:43.999 INFO:tasks.workunit.client.0.smithi144.stdout:7/407: write d0/d6/d85/f41 [3079234,33166] 0 2024-07-26T11:04:44.000 INFO:tasks.workunit.client.0.smithi144.stdout:7/408: mkdir d0/d6/d85/d2f/da2 0 2024-07-26T11:04:44.000 INFO:tasks.workunit.client.0.smithi144.stdout:7/409: chown d0/d6/d85/d2f/d47 0 1 2024-07-26T11:04:44.002 INFO:tasks.workunit.client.0.smithi144.stdout:7/410: rename d0/d6/d9/d3f/f87 to d0/d6/d9/d45/fa3 0 2024-07-26T11:04:44.002 INFO:tasks.workunit.client.0.smithi144.stdout:7/411: dread - d0/d6/d9/d3f/f83 zero size 2024-07-26T11:04:44.002 INFO:tasks.workunit.client.0.smithi144.stdout:7/412: chown d0/d6/d9/d45/d57/c7c 0 1 2024-07-26T11:04:44.002 INFO:tasks.workunit.client.0.smithi144.stdout:7/413: dread - d0/d6/d85/d2f/d47/d5e/f63 zero size 2024-07-26T11:04:44.003 INFO:tasks.workunit.client.0.smithi144.stdout:7/414: creat d0/d6/d9/d45/d57/fa4 x:0 0 0 2024-07-26T11:04:44.004 INFO:tasks.workunit.client.0.smithi144.stdout:7/415: creat d0/d6/d85/d2f/d47/d5e/fa5 x:0 0 0 2024-07-26T11:04:44.004 INFO:tasks.workunit.client.0.smithi144.stdout:7/416: chown d0/d6/d85/d67/f9d 22 1 2024-07-26T11:04:44.005 INFO:tasks.workunit.client.0.smithi144.stdout:7/417: truncate d0/d71/f99 792579 0 2024-07-26T11:04:44.060 INFO:tasks.workunit.client.0.smithi144.stdout:7/418: dread d0/d6/f2e [0,4194304] 0 2024-07-26T11:04:44.065 INFO:tasks.workunit.client.0.smithi144.stdout:2/441: dwrite d0/d6/f1d [4194304,4194304] 0 2024-07-26T11:04:44.132 INFO:tasks.workunit.client.0.smithi144.stdout:2/442: dwrite d0/d24/d48/d6d/d28/d2e/d5d/d6b/f73 [0,4194304] 0 2024-07-26T11:04:44.138 INFO:tasks.workunit.client.0.smithi144.stdout:2/443: read d0/d24/d2b/d68/d82/f5a [228031,82986] 0 2024-07-26T11:04:44.139 INFO:tasks.workunit.client.0.smithi144.stdout:2/444: mkdir d0/d24/d48/d6d/d28/d27/d8b 0 2024-07-26T11:04:44.139 INFO:tasks.workunit.client.0.smithi144.stdout:2/445: fsync d0/d24/d48/d6d/f47 0 2024-07-26T11:04:44.198 INFO:tasks.workunit.client.0.smithi144.stdout:2/446: dwrite d0/d24/d48/d6d/d5f/f85 [0,4194304] 0 2024-07-26T11:04:44.198 INFO:tasks.workunit.client.0.smithi144.stdout:2/447: chown d0/d6/f1d 4672426 1 2024-07-26T11:04:44.216 INFO:tasks.workunit.client.0.smithi144.stdout:8/407: sync 2024-07-26T11:04:44.216 INFO:tasks.workunit.client.0.smithi144.stdout:3/252: sync 2024-07-26T11:04:44.216 INFO:tasks.workunit.client.0.smithi144.stdout:4/395: sync 2024-07-26T11:04:44.216 INFO:tasks.workunit.client.0.smithi144.stdout:9/361: sync 2024-07-26T11:04:44.216 INFO:tasks.workunit.client.0.smithi144.stdout:1/479: sync 2024-07-26T11:04:44.216 INFO:tasks.workunit.client.0.smithi144.stdout:7/419: sync 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:0/402: sync 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:5/454: sync 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:6/490: sync 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:6/491: stat d8/d15/d57/d5f/d6e/f8e 0 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:5/455: stat d1/d18/d24/d27/c52 0 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:6/492: write d8/d42/f7c [1211821,92534] 0 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:5/456: write d1/d9/d44/d45/f56 [690750,59651] 0 2024-07-26T11:04:44.217 INFO:tasks.workunit.client.0.smithi144.stdout:4/396: symlink d9/d26/d48/d50/l84 0 2024-07-26T11:04:44.219 INFO:tasks.workunit.client.0.smithi144.stdout:9/362: mkdir d9/d3b/d76 0 2024-07-26T11:04:44.220 INFO:tasks.workunit.client.0.smithi144.stdout:9/363: readlink d9/de/d14/d3f/l40 0 2024-07-26T11:04:44.220 INFO:tasks.workunit.client.0.smithi144.stdout:3/253: creat d1/d5/d25/d26/d37/f49 x:0 0 0 2024-07-26T11:04:44.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/408: creat d1a/d32/d49/d73/d7b/f8b x:0 0 0 2024-07-26T11:04:44.228 INFO:tasks.workunit.client.0.smithi144.stdout:6/493: unlink d8/db/f3e 0 2024-07-26T11:04:44.229 INFO:tasks.workunit.client.0.smithi144.stdout:7/420: creat d0/d6/d85/d2f/da2/fa6 x:0 0 0 2024-07-26T11:04:44.230 INFO:tasks.workunit.client.0.smithi144.stdout:9/364: symlink d9/l77 0 2024-07-26T11:04:44.230 INFO:tasks.workunit.client.0.smithi144.stdout:9/365: fdatasync d9/d3b/f71 0 2024-07-26T11:04:44.231 INFO:tasks.workunit.client.0.smithi144.stdout:7/421: rename d0/d6/d85/d2f/d53/f8f to d0/d6/d85/d67/fa7 0 2024-07-26T11:04:44.232 INFO:tasks.workunit.client.0.smithi144.stdout:7/422: truncate d0/d6/f8b 863826 0 2024-07-26T11:04:44.234 INFO:tasks.workunit.client.0.smithi144.stdout:7/423: rmdir d0/d6/d85/d2f/d68 39 2024-07-26T11:04:44.234 INFO:tasks.workunit.client.0.smithi144.stdout:7/424: stat d0/d6/d9/d45/f26 0 2024-07-26T11:04:44.237 INFO:tasks.workunit.client.0.smithi144.stdout:7/425: rmdir d0/d6/d9/d12/d6c 39 2024-07-26T11:04:44.242 INFO:tasks.workunit.client.0.smithi144.stdout:7/426: creat d0/d6/d85/d2f/d53/fa8 x:0 0 0 2024-07-26T11:04:44.242 INFO:tasks.workunit.client.0.smithi144.stdout:7/427: chown d0/d6/d85/d2f/d47/d5e/f88 93064130 1 2024-07-26T11:04:44.242 INFO:tasks.workunit.client.0.smithi144.stdout:7/428: write d0/d6/d85/d2f/d47/d5e/f88 [396208,98922] 0 2024-07-26T11:04:44.251 INFO:tasks.workunit.client.0.smithi144.stdout:1/480: write d3/f2d [456062,687] 0 2024-07-26T11:04:44.262 INFO:tasks.workunit.client.0.smithi144.stdout:2/448: write d0/d24/d48/d6d/d5f/f85 [2467276,30471] 0 2024-07-26T11:04:44.262 INFO:tasks.workunit.client.0.smithi144.stdout:8/409: dread d1a/f2b [0,4194304] 0 2024-07-26T11:04:44.264 INFO:tasks.workunit.client.0.smithi144.stdout:8/410: rmdir d1a/d32/d66 39 2024-07-26T11:04:44.265 INFO:tasks.workunit.client.0.smithi144.stdout:2/449: rename d0/d6/c10 to d0/d24/d2b/c8c 0 2024-07-26T11:04:44.269 INFO:tasks.workunit.client.0.smithi144.stdout:8/411: mkdir d1a/d32/d8c 0 2024-07-26T11:04:44.269 INFO:tasks.workunit.client.0.smithi144.stdout:2/450: rmdir d0/d24 39 2024-07-26T11:04:44.271 INFO:tasks.workunit.client.0.smithi144.stdout:2/451: read d0/d6/d9/f26 [336199,103516] 0 2024-07-26T11:04:44.271 INFO:tasks.workunit.client.0.smithi144.stdout:2/452: fdatasync d0/d6/d9/f33 0 2024-07-26T11:04:44.273 INFO:tasks.workunit.client.0.smithi144.stdout:2/453: creat d0/d24/d2b/f8d x:0 0 0 2024-07-26T11:04:44.274 INFO:tasks.workunit.client.0.smithi144.stdout:9/366: dread d9/d3b/f64 [0,4194304] 0 2024-07-26T11:04:44.274 INFO:tasks.workunit.client.0.smithi144.stdout:9/367: readlink d9/de/l4c 0 2024-07-26T11:04:44.274 INFO:tasks.workunit.client.0.smithi144.stdout:9/368: fsync d9/fc 0 2024-07-26T11:04:44.274 INFO:tasks.workunit.client.0.smithi144.stdout:2/454: readlink d0/d24/d2b/l45 0 2024-07-26T11:04:44.274 INFO:tasks.workunit.client.0.smithi144.stdout:2/455: fsync d0/d6/f89 0 2024-07-26T11:04:44.275 INFO:tasks.workunit.client.0.smithi144.stdout:9/369: getdents d9/de/d14/d23 0 2024-07-26T11:04:44.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/456: creat d0/d24/d2b/f8e x:0 0 0 2024-07-26T11:04:44.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/457: truncate d0/d24/d2b/f8d 101015 0 2024-07-26T11:04:44.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/458: write d0/f1e [2969628,100225] 0 2024-07-26T11:04:44.277 INFO:tasks.workunit.client.0.smithi144.stdout:2/459: write d0/d24/d48/d6d/d28/d2e/d5d/f7a [1439098,106764] 0 2024-07-26T11:04:44.278 INFO:tasks.workunit.client.0.smithi144.stdout:2/460: creat d0/d6/d34/f8f x:0 0 0 2024-07-26T11:04:44.278 INFO:tasks.workunit.client.0.smithi144.stdout:2/461: chown d0/d6/d9/f26 454 1 2024-07-26T11:04:44.294 INFO:tasks.workunit.client.0.smithi144.stdout:7/429: write d0/d6/d9/d45/f30 [1240475,69221] 0 2024-07-26T11:04:44.296 INFO:tasks.workunit.client.0.smithi144.stdout:3/254: dwrite d1/d5/f15 [0,4194304] 0 2024-07-26T11:04:44.298 INFO:tasks.workunit.client.0.smithi144.stdout:7/430: symlink d0/d71/da1/la9 0 2024-07-26T11:04:44.299 INFO:tasks.workunit.client.0.smithi144.stdout:6/494: dwrite d8/f96 [0,4194304] 0 2024-07-26T11:04:44.300 INFO:tasks.workunit.client.0.smithi144.stdout:7/431: symlink d0/d6/d55/laa 0 2024-07-26T11:04:44.304 INFO:tasks.workunit.client.0.smithi144.stdout:6/495: mkdir d8/d4f/daa 0 2024-07-26T11:04:44.304 INFO:tasks.workunit.client.0.smithi144.stdout:6/496: read d8/d15/d26/f41 [473170,126206] 0 2024-07-26T11:04:44.304 INFO:tasks.workunit.client.0.smithi144.stdout:6/497: fdatasync d8/f82 0 2024-07-26T11:04:44.306 INFO:tasks.workunit.client.0.smithi144.stdout:9/370: dread d9/de/f21 [0,4194304] 0 2024-07-26T11:04:44.306 INFO:tasks.workunit.client.0.smithi144.stdout:9/371: chown d9/de/ff 0 1 2024-07-26T11:04:44.308 INFO:tasks.workunit.client.0.smithi144.stdout:6/498: unlink d8/db/f32 0 2024-07-26T11:04:44.309 INFO:tasks.workunit.client.0.smithi144.stdout:9/372: creat d9/d3b/f78 x:0 0 0 2024-07-26T11:04:44.312 INFO:tasks.workunit.client.0.smithi144.stdout:9/373: mkdir d9/de/d14/d79 0 2024-07-26T11:04:44.315 INFO:tasks.workunit.client.0.smithi144.stdout:7/432: dread d0/d6/d85/d2f/d47/d5e/f88 [0,4194304] 0 2024-07-26T11:04:44.315 INFO:tasks.workunit.client.0.smithi144.stdout:7/433: dread - d0/d6/d85/d67/fa7 zero size 2024-07-26T11:04:44.318 INFO:tasks.workunit.client.0.smithi144.stdout:7/434: symlink d0/d6/d9/d45/d57/lab 0 2024-07-26T11:04:44.318 INFO:tasks.workunit.client.0.smithi144.stdout:7/435: dread - d0/d6/d9/d64/f65 zero size 2024-07-26T11:04:44.329 INFO:tasks.workunit.client.0.smithi144.stdout:5/457: dwrite d1/d9/d44/d45/d81/f91 [4194304,4194304] 0 2024-07-26T11:04:44.329 INFO:tasks.workunit.client.0.smithi144.stdout:5/458: write d1/d9/d44/f4e [1966734,113877] 0 2024-07-26T11:04:44.340 INFO:tasks.workunit.client.0.smithi144.stdout:2/462: dwrite d0/d24/d48/d6d/f47 [0,4194304] 0 2024-07-26T11:04:44.351 INFO:tasks.workunit.client.0.smithi144.stdout:3/255: dread d1/f14 [0,4194304] 0 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:3/256: readlink d1/d28/l2c 0 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:1/481: write d3/d6/da/d13/d22/f7f [231248,124601] 0 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:3/257: getdents d1/d5/d25/d3d 0 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:3/258: readlink d1/l9 0 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:3/259: dread - d1/f20 zero size 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:2/463: link d0/d24/d48/d6d/d28/d27/l5e d0/d24/l90 0 2024-07-26T11:04:44.352 INFO:tasks.workunit.client.0.smithi144.stdout:1/482: mknod d3/d6/da/d5b/c94 0 2024-07-26T11:04:44.353 INFO:tasks.workunit.client.0.smithi144.stdout:3/260: creat d1/d5/d25/f4a x:0 0 0 2024-07-26T11:04:44.357 INFO:tasks.workunit.client.0.smithi144.stdout:1/483: creat d3/d6/da/d21/d67/f95 x:0 0 0 2024-07-26T11:04:44.357 INFO:tasks.workunit.client.0.smithi144.stdout:1/484: dread - d3/d6/da/d21/d67/f95 zero size 2024-07-26T11:04:44.357 INFO:tasks.workunit.client.0.smithi144.stdout:2/464: creat d0/f91 x:0 0 0 2024-07-26T11:04:44.358 INFO:tasks.workunit.client.0.smithi144.stdout:2/465: symlink d0/d6/d9/d44/l92 0 2024-07-26T11:04:44.358 INFO:tasks.workunit.client.0.smithi144.stdout:2/466: write d0/d24/d2b/d68/d82/f5a [434079,90657] 0 2024-07-26T11:04:44.359 INFO:tasks.workunit.client.0.smithi144.stdout:7/436: dread d0/d6/d9/d3f/f58 [0,4194304] 0 2024-07-26T11:04:44.359 INFO:tasks.workunit.client.0.smithi144.stdout:7/437: write d0/d6/d85/d2f/f8a [766946,104428] 0 2024-07-26T11:04:44.359 INFO:tasks.workunit.client.0.smithi144.stdout:7/438: dread - d0/d6/d85/d2f/d47/d5e/fa5 zero size 2024-07-26T11:04:44.366 INFO:tasks.workunit.client.0.smithi144.stdout:7/439: truncate d0/d6/d9/d45/f30 2062813 0 2024-07-26T11:04:44.368 INFO:tasks.workunit.client.0.smithi144.stdout:7/440: link d0/d6/d85/d67/l94 d0/d6/d85/d2f/lac 0 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:8/412: write d1a/d32/d49/d73/d2c/d38/d58/f6e [2458061,101009] 0 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:1/485: dread d3/d6/da/d21/d5c/f5e [0,4194304] 0 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:7/441: rename d0/d6/d9/c1a to d0/d71/cad 0 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:0/403: dwrite d0/d25/d5e/d1e/f24 [0,4194304] 0 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:0/404: chown d0/d3e/d45/d4c/c5d 61040143 1 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:0/405: fsync d0/f27 0 2024-07-26T11:04:44.370 INFO:tasks.workunit.client.0.smithi144.stdout:8/413: mknod d1a/d32/d49/d73/d2c/c8d 0 2024-07-26T11:04:44.372 INFO:tasks.workunit.client.0.smithi144.stdout:1/486: creat d3/d6/da/d13/d19/d3a/d69/d36/f96 x:0 0 0 2024-07-26T11:04:44.372 INFO:tasks.workunit.client.0.smithi144.stdout:2/467: write d0/f1 [5017754,88253] 0 2024-07-26T11:04:44.373 INFO:tasks.workunit.client.0.smithi144.stdout:6/499: dwrite d8/f16 [4194304,4194304] 0 2024-07-26T11:04:44.373 INFO:tasks.workunit.client.0.smithi144.stdout:6/500: truncate d8/d4f/d5c/f7b 4879689 0 2024-07-26T11:04:44.373 INFO:tasks.workunit.client.0.smithi144.stdout:0/406: rename d0/d25/d5e/d29/f42 to d0/d25/d5e/d1e/d30/d6a/f8a 0 2024-07-26T11:04:44.373 INFO:tasks.workunit.client.0.smithi144.stdout:0/407: write d0/d25/d5e/d29/f2d [786964,139] 0 2024-07-26T11:04:44.377 INFO:tasks.workunit.client.0.smithi144.stdout:8/414: rmdir d1a/d32/d49/d73/d7b/d43/d4f 39 2024-07-26T11:04:44.377 INFO:tasks.workunit.client.0.smithi144.stdout:1/487: mkdir d3/d6/da/d97 0 2024-07-26T11:04:44.379 INFO:tasks.workunit.client.0.smithi144.stdout:2/468: rename d0/d24/d48/d6d/d28/d27/l37 to d0/d24/d2b/d68/l93 0 2024-07-26T11:04:44.379 INFO:tasks.workunit.client.0.smithi144.stdout:2/469: fdatasync d0/d6/d34/d6c/f75 0 2024-07-26T11:04:44.379 INFO:tasks.workunit.client.0.smithi144.stdout:2/470: stat d0/d24/d48/c54 0 2024-07-26T11:04:44.379 INFO:tasks.workunit.client.0.smithi144.stdout:2/471: fdatasync d0/f3d 0 2024-07-26T11:04:44.380 INFO:tasks.workunit.client.0.smithi144.stdout:2/472: chown d0/d6/d34/f8f 4 1 2024-07-26T11:04:44.380 INFO:tasks.workunit.client.0.smithi144.stdout:8/415: truncate d1a/d32/d49/d73/d7b/f5e 2308633 0 2024-07-26T11:04:44.383 INFO:tasks.workunit.client.0.smithi144.stdout:9/374: write d9/de/d14/f2c [767536,105495] 0 2024-07-26T11:04:44.383 INFO:tasks.workunit.client.0.smithi144.stdout:2/473: getdents d0/d24/d2b/d68/d82 0 2024-07-26T11:04:44.385 INFO:tasks.workunit.client.0.smithi144.stdout:6/501: mknod d8/d4f/d5c/cab 0 2024-07-26T11:04:44.387 INFO:tasks.workunit.client.0.smithi144.stdout:2/474: link d0/d24/d2b/d68/d82/l56 d0/d24/d7b/l94 0 2024-07-26T11:04:44.390 INFO:tasks.workunit.client.0.smithi144.stdout:3/261: dread d1/f34 [0,4194304] 0 2024-07-26T11:04:44.390 INFO:tasks.workunit.client.0.smithi144.stdout:3/262: chown d1/d28 0 1 2024-07-26T11:04:44.390 INFO:tasks.workunit.client.0.smithi144.stdout:3/263: stat d1/d5/d25/d26/d37/f49 0 2024-07-26T11:04:44.390 INFO:tasks.workunit.client.0.smithi144.stdout:3/264: write d1/d5/d25/d26/f2d [4622675,106700] 0 2024-07-26T11:04:44.390 INFO:tasks.workunit.client.0.smithi144.stdout:3/265: chown d1/f34 2637 1 2024-07-26T11:04:44.390 INFO:tasks.workunit.client.0.smithi144.stdout:3/266: fsync d1/f1a 0 2024-07-26T11:04:44.396 INFO:tasks.workunit.client.0.smithi144.stdout:3/267: creat d1/d5/d25/d26/d37/d46/f4b x:0 0 0 2024-07-26T11:04:44.404 INFO:tasks.workunit.client.0.smithi144.stdout:3/268: getdents d1/d5/d25/d26/d30 0 2024-07-26T11:04:44.404 INFO:tasks.workunit.client.0.smithi144.stdout:3/269: stat d1/d5/d25/d3d/c3e 0 2024-07-26T11:04:44.404 INFO:tasks.workunit.client.0.smithi144.stdout:3/270: chown d1/d5/d25/d26/c31 193815792 1 2024-07-26T11:04:44.407 INFO:tasks.workunit.client.0.smithi144.stdout:3/271: symlink d1/l4c 0 2024-07-26T11:04:44.408 INFO:tasks.workunit.client.0.smithi144.stdout:3/272: mkdir d1/d5/d25/d26/d30/d4d 0 2024-07-26T11:04:44.409 INFO:tasks.workunit.client.0.smithi144.stdout:3/273: rmdir d1/d5/d25/d26 39 2024-07-26T11:04:44.409 INFO:tasks.workunit.client.0.smithi144.stdout:3/274: chown d1/d28/f3b 987814 1 2024-07-26T11:04:44.410 INFO:tasks.workunit.client.0.smithi144.stdout:3/275: getdents d1/d5/d25/d3d 0 2024-07-26T11:04:44.410 INFO:tasks.workunit.client.0.smithi144.stdout:3/276: chown d1/d5/d25/d26/f2b 0 1 2024-07-26T11:04:44.411 INFO:tasks.workunit.client.0.smithi144.stdout:3/277: creat d1/d5/d25/d3d/d42/d44/f4e x:0 0 0 2024-07-26T11:04:44.411 INFO:tasks.workunit.client.0.smithi144.stdout:3/278: chown d1/d5/d25/d26/f27 12 1 2024-07-26T11:04:44.411 INFO:tasks.workunit.client.0.smithi144.stdout:3/279: chown d1/d5/d25/d26/c31 899773176 1 2024-07-26T11:04:44.412 INFO:tasks.workunit.client.0.smithi144.stdout:3/280: creat d1/f4f x:0 0 0 2024-07-26T11:04:44.412 INFO:tasks.workunit.client.0.smithi144.stdout:3/281: read - d1/f1b zero size 2024-07-26T11:04:44.414 INFO:tasks.workunit.client.0.smithi144.stdout:3/282: creat d1/f50 x:0 0 0 2024-07-26T11:04:44.415 INFO:tasks.workunit.client.0.smithi144.stdout:3/283: mknod d1/c51 0 2024-07-26T11:04:44.416 INFO:tasks.workunit.client.0.smithi144.stdout:3/284: getdents d1/d5/d25/d26/d37/d46 0 2024-07-26T11:04:44.417 INFO:tasks.workunit.client.0.smithi144.stdout:3/285: creat d1/d5/d25/d3d/f52 x:0 0 0 2024-07-26T11:04:44.418 INFO:tasks.workunit.client.0.smithi144.stdout:3/286: rmdir d1/d5/d25/d26/d30 39 2024-07-26T11:04:44.419 INFO:tasks.workunit.client.0.smithi144.stdout:3/287: creat d1/d5/d25/d3d/f53 x:0 0 0 2024-07-26T11:04:44.421 INFO:tasks.workunit.client.0.smithi144.stdout:0/408: dwrite d0/d25/d5e/d29/f6e [0,4194304] 0 2024-07-26T11:04:44.422 INFO:tasks.workunit.client.0.smithi144.stdout:9/375: dread d9/de/d14/f24 [0,4194304] 0 2024-07-26T11:04:44.423 INFO:tasks.workunit.client.0.smithi144.stdout:0/409: mknod d0/d25/d5e/d29/d65/c8b 0 2024-07-26T11:04:44.423 INFO:tasks.workunit.client.0.smithi144.stdout:0/410: write d0/f27 [2897252,67243] 0 2024-07-26T11:04:44.423 INFO:tasks.workunit.client.0.smithi144.stdout:0/411: chown d0/d25/d5e/l35 2118828202 1 2024-07-26T11:04:44.426 INFO:tasks.workunit.client.0.smithi144.stdout:5/459: dwrite d1/f8f [0,4194304] 0 2024-07-26T11:04:44.426 INFO:tasks.workunit.client.0.smithi144.stdout:0/412: unlink d0/d25/d5e/d1e/d30/c37 0 2024-07-26T11:04:44.431 INFO:tasks.workunit.client.0.smithi144.stdout:3/288: rename d1/d5/d25 to d1/d28/d54 0 2024-07-26T11:04:44.432 INFO:tasks.workunit.client.0.smithi144.stdout:5/460: rmdir d1/d18/d24/d27/d71 39 2024-07-26T11:04:44.432 INFO:tasks.workunit.client.0.smithi144.stdout:0/413: symlink d0/d25/d5e/d1e/d6c/l8c 0 2024-07-26T11:04:44.433 INFO:tasks.workunit.client.0.smithi144.stdout:3/289: unlink d1/l2 0 2024-07-26T11:04:44.433 INFO:tasks.workunit.client.0.smithi144.stdout:3/290: write d1/d5/f6 [3560220,28381] 0 2024-07-26T11:04:44.433 INFO:tasks.workunit.client.0.smithi144.stdout:3/291: dread - d1/d28/d54/d26/d37/d46/f4b zero size 2024-07-26T11:04:44.433 INFO:tasks.workunit.client.0.smithi144.stdout:3/292: readlink d1/d5/l21 0 2024-07-26T11:04:44.441 INFO:tasks.workunit.client.0.smithi144.stdout:6/502: write d8/f53 [1038912,97812] 0 2024-07-26T11:04:44.441 INFO:tasks.workunit.client.0.smithi144.stdout:5/461: creat d1/d18/d24/fa0 x:0 0 0 2024-07-26T11:04:44.441 INFO:tasks.workunit.client.0.smithi144.stdout:2/475: dread d0/d24/d48/d6d/d28/d2e/d77/f40 [0,4194304] 0 2024-07-26T11:04:44.441 INFO:tasks.workunit.client.0.smithi144.stdout:0/414: unlink d0/d3e/d45/f73 0 2024-07-26T11:04:44.441 INFO:tasks.workunit.client.0.smithi144.stdout:0/415: dread - d0/d25/f7b zero size 2024-07-26T11:04:44.482 INFO:tasks.workunit.client.0.smithi144.stdout:1/488: dwrite d3/d6/da/f4a [0,4194304] 0 2024-07-26T11:04:44.486 INFO:tasks.workunit.client.0.smithi144.stdout:1/489: unlink d3/d6/da/d13/f82 0 2024-07-26T11:04:44.488 INFO:tasks.workunit.client.0.smithi144.stdout:2/476: dread d0/d24/d2b/d68/d82/f5a [0,4194304] 0 2024-07-26T11:04:44.489 INFO:tasks.workunit.client.0.smithi144.stdout:1/490: symlink d3/d6/da/d21/d5c/d62/l98 0 2024-07-26T11:04:44.489 INFO:tasks.workunit.client.0.smithi144.stdout:1/491: chown l2 12 1 2024-07-26T11:04:44.489 INFO:tasks.workunit.client.0.smithi144.stdout:1/492: write d3/d6/da/d13/d22/f81 [492864,119] 0 2024-07-26T11:04:44.490 INFO:tasks.workunit.client.0.smithi144.stdout:2/477: creat d0/d6/f95 x:0 0 0 2024-07-26T11:04:44.490 INFO:tasks.workunit.client.0.smithi144.stdout:2/478: chown d0/d24/d48/d6d/f47 459044013 1 2024-07-26T11:04:44.490 INFO:tasks.workunit.client.0.smithi144.stdout:2/479: write d0/d24/d48/d6d/d5f/f85 [2580506,64048] 0 2024-07-26T11:04:44.490 INFO:tasks.workunit.client.0.smithi144.stdout:2/480: fdatasync d0/d24/d48/d6d/d28/f5c 0 2024-07-26T11:04:44.490 INFO:tasks.workunit.client.0.smithi144.stdout:2/481: write d0/d6/d9/f26 [1091627,51270] 0 2024-07-26T11:04:44.492 INFO:tasks.workunit.client.0.smithi144.stdout:8/416: dwrite fe [0,4194304] 0 2024-07-26T11:04:44.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/493: creat d3/d6/da/d78/f99 x:0 0 0 2024-07-26T11:04:44.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/494: chown d3/d53 90054995 1 2024-07-26T11:04:44.494 INFO:tasks.workunit.client.0.smithi144.stdout:1/495: getdents d3/d6/da/d13/d19/d3a/d69/d36/d37 0 2024-07-26T11:04:44.498 INFO:tasks.workunit.client.0.smithi144.stdout:7/442: dwrite d0/d6/d85/d2f/f70 [0,4194304] 0 2024-07-26T11:04:44.499 INFO:tasks.workunit.client.0.smithi144.stdout:7/443: write d0/d6/d9/d12/f6d [1881959,80984] 0 2024-07-26T11:04:44.506 INFO:tasks.workunit.client.0.smithi144.stdout:3/293: dwrite d1/d5/f15 [0,4194304] 0 2024-07-26T11:04:44.506 INFO:tasks.workunit.client.0.smithi144.stdout:3/294: truncate d1/d28/d54/d3d/d42/d44/f4e 717174 0 2024-07-26T11:04:44.510 INFO:tasks.workunit.client.0.smithi144.stdout:7/444: creat d0/d6/d85/d2f/d47/fae x:0 0 0 2024-07-26T11:04:44.510 INFO:tasks.workunit.client.0.smithi144.stdout:7/445: dread - d0/d6/d9/d64/f6b zero size 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:1/496: dread d3/d6/f23 [0,4194304] 0 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/397: sync 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/398: dread - d9/d26/f7d zero size 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/399: chown d9/f22 529 1 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/400: stat d9/da/c16 0 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/401: dread - d9/f63 zero size 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/402: truncate d9/f76 551073 0 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/403: chown d9/d26/l36 4140846 1 2024-07-26T11:04:44.522 INFO:tasks.workunit.client.0.smithi144.stdout:4/404: read d9/d26/f83 [850474,62826] 0 2024-07-26T11:04:44.547 INFO:tasks.workunit.client.0.smithi144.stdout:2/482: dwrite d0/d24/d48/d6d/f35 [0,4194304] 0 2024-07-26T11:04:44.549 INFO:tasks.workunit.client.0.smithi144.stdout:2/483: read d0/d24/d2b/f8d [34508,87340] 0 2024-07-26T11:04:44.549 INFO:tasks.workunit.client.0.smithi144.stdout:2/484: truncate d0/d24/d7b/f87 2449650 0 2024-07-26T11:04:44.549 INFO:tasks.workunit.client.0.smithi144.stdout:2/485: write d0/d6/f1d [2460768,10179] 0 2024-07-26T11:04:44.549 INFO:tasks.workunit.client.0.smithi144.stdout:2/486: write d0/d24/d48/d6d/d5f/f85 [1006998,5160] 0 2024-07-26T11:04:44.550 INFO:tasks.workunit.client.0.smithi144.stdout:9/376: dwrite d9/de/d14/f2c [0,4194304] 0 2024-07-26T11:04:44.550 INFO:tasks.workunit.client.0.smithi144.stdout:2/487: creat d0/d24/d48/d6d/f96 x:0 0 0 2024-07-26T11:04:44.550 INFO:tasks.workunit.client.0.smithi144.stdout:2/488: write d0/d24/d2b/f8e [550478,70167] 0 2024-07-26T11:04:44.552 INFO:tasks.workunit.client.0.smithi144.stdout:9/377: mknod d9/de/d14/d3f/d5e/c7a 0 2024-07-26T11:04:44.552 INFO:tasks.workunit.client.0.smithi144.stdout:9/378: chown d9/d3b/d5d/c68 110208 1 2024-07-26T11:04:44.558 INFO:tasks.workunit.client.0.smithi144.stdout:8/417: dwrite d1a/f1b [0,4194304] 0 2024-07-26T11:04:44.567 INFO:tasks.workunit.client.0.smithi144.stdout:8/418: write d1a/d1c/f41 [777272,103386] 0 2024-07-26T11:04:44.567 INFO:tasks.workunit.client.0.smithi144.stdout:8/419: readlink d1a/d32/d49/d73/d2c/d38/d58/l7e 0 2024-07-26T11:04:44.568 INFO:tasks.workunit.client.0.smithi144.stdout:7/446: dwrite d0/d6/d9/d45/d57/f77 [0,4194304] 0 2024-07-26T11:04:44.571 INFO:tasks.workunit.client.0.smithi144.stdout:8/420: symlink d1a/d32/d49/d73/d7b/d4e/l8e 0 2024-07-26T11:04:44.575 INFO:tasks.workunit.client.0.smithi144.stdout:8/421: rename d1a/d32/d6c/c3e to d1a/d32/d6c/d3c/c8f 0 2024-07-26T11:04:44.582 INFO:tasks.workunit.client.0.smithi144.stdout:8/422: dread d1a/d32/f62 [0,4194304] 0 2024-07-26T11:04:44.584 INFO:tasks.workunit.client.0.smithi144.stdout:4/405: dwrite d9/da/f39 [0,4194304] 0 2024-07-26T11:04:44.586 INFO:tasks.workunit.client.0.smithi144.stdout:4/406: creat d9/d26/d3c/d6f/f85 x:0 0 0 2024-07-26T11:04:44.587 INFO:tasks.workunit.client.0.smithi144.stdout:4/407: rmdir d9/d26 39 2024-07-26T11:04:44.589 INFO:tasks.workunit.client.0.smithi144.stdout:4/408: rename d9/d26/f47 to d9/d26/d3c/d6f/f86 0 2024-07-26T11:04:44.605 INFO:tasks.workunit.client.0.smithi144.stdout:9/379: dwrite d9/de/f6e [0,4194304] 0 2024-07-26T11:04:44.605 INFO:tasks.workunit.client.0.smithi144.stdout:4/409: dread d9/f15 [0,4194304] 0 2024-07-26T11:04:44.606 INFO:tasks.workunit.client.0.smithi144.stdout:4/410: rename d9/da/c5e to d9/d26/d48/c87 0 2024-07-26T11:04:44.608 INFO:tasks.workunit.client.0.smithi144.stdout:4/411: creat d9/d26/d48/d50/d60/f88 x:0 0 0 2024-07-26T11:04:44.608 INFO:tasks.workunit.client.0.smithi144.stdout:4/412: chown d9/d26/d3c/c65 3 1 2024-07-26T11:04:44.609 INFO:tasks.workunit.client.0.smithi144.stdout:4/413: mknod d9/d26/d3c/d6f/c89 0 2024-07-26T11:04:44.623 INFO:tasks.workunit.client.0.smithi144.stdout:7/447: dwrite d0/d6/d9/d12/f6d [4194304,4194304] 0 2024-07-26T11:04:44.624 INFO:tasks.workunit.client.0.smithi144.stdout:7/448: stat d0/d6/d85/f41 0 2024-07-26T11:04:44.624 INFO:tasks.workunit.client.0.smithi144.stdout:7/449: write d0/d6/d9/d45/f30 [3061488,26518] 0 2024-07-26T11:04:44.625 INFO:tasks.workunit.client.0.smithi144.stdout:7/450: getdents d0/d6/d9/d64 0 2024-07-26T11:04:44.626 INFO:tasks.workunit.client.0.smithi144.stdout:7/451: getdents d0/d6/d85/d2f/d68 0 2024-07-26T11:04:44.627 INFO:tasks.workunit.client.0.smithi144.stdout:7/452: creat d0/d71/da1/faf x:0 0 0 2024-07-26T11:04:44.638 INFO:tasks.workunit.client.0.smithi144.stdout:4/414: dread d9/da/f39 [0,4194304] 0 2024-07-26T11:04:44.647 INFO:tasks.workunit.client.0.smithi144.stdout:7/453: write d0/d6/d85/f42 [3214195,84460] 0 2024-07-26T11:04:44.649 INFO:tasks.workunit.client.0.smithi144.stdout:4/415: dread d9/d26/d3c/f40 [0,4194304] 0 2024-07-26T11:04:44.649 INFO:tasks.workunit.client.0.smithi144.stdout:4/416: fsync d9/d26/f58 0 2024-07-26T11:04:44.650 INFO:tasks.workunit.client.0.smithi144.stdout:4/417: creat d9/d57/f8a x:0 0 0 2024-07-26T11:04:44.651 INFO:tasks.workunit.client.0.smithi144.stdout:4/418: symlink d9/d26/l8b 0 2024-07-26T11:04:44.652 INFO:tasks.workunit.client.0.smithi144.stdout:4/419: unlink d9/d26/d48/f6b 0 2024-07-26T11:04:44.653 INFO:tasks.workunit.client.0.smithi144.stdout:4/420: mkdir d9/d26/d3c/d6f/d8c 0 2024-07-26T11:04:44.654 INFO:tasks.workunit.client.0.smithi144.stdout:4/421: rmdir d9/d26/d3c/d6f 39 2024-07-26T11:04:44.657 INFO:tasks.workunit.client.0.smithi144.stdout:7/454: dread d0/f11 [0,4194304] 0 2024-07-26T11:04:44.657 INFO:tasks.workunit.client.0.smithi144.stdout:7/455: dread - d0/d6/d85/d2f/f54 zero size 2024-07-26T11:04:44.657 INFO:tasks.workunit.client.0.smithi144.stdout:7/456: dread - d0/d6/d85/d2f/d47/fae zero size 2024-07-26T11:04:44.658 INFO:tasks.workunit.client.0.smithi144.stdout:7/457: rmdir d0/d6/d9/d12/d6c 39 2024-07-26T11:04:44.697 INFO:tasks.workunit.client.0.smithi144.stdout:4/422: dwrite d9/f1f [0,4194304] 0 2024-07-26T11:04:44.697 INFO:tasks.workunit.client.0.smithi144.stdout:4/423: write d9/f63 [15538,16036] 0 2024-07-26T11:04:44.697 INFO:tasks.workunit.client.0.smithi144.stdout:7/458: dread d0/d6/d9/d12/f59 [0,4194304] 0 2024-07-26T11:04:44.697 INFO:tasks.workunit.client.0.smithi144.stdout:7/459: chown d0/d6/d85/d67 2 1 2024-07-26T11:04:44.699 INFO:tasks.workunit.client.0.smithi144.stdout:4/424: link d9/d26/f58 d9/f8d 0 2024-07-26T11:04:44.699 INFO:tasks.workunit.client.0.smithi144.stdout:4/425: stat d9/d26/d48/d50/c67 0 2024-07-26T11:04:44.700 INFO:tasks.workunit.client.0.smithi144.stdout:7/460: symlink d0/d6/d85/d2f/da2/lb0 0 2024-07-26T11:04:44.700 INFO:tasks.workunit.client.0.smithi144.stdout:4/426: creat d9/d57/f8e x:0 0 0 2024-07-26T11:04:44.702 INFO:tasks.workunit.client.0.smithi144.stdout:4/427: unlink d9/da/c49 0 2024-07-26T11:04:44.703 INFO:tasks.workunit.client.0.smithi144.stdout:4/428: unlink d9/d26/f4c 0 2024-07-26T11:04:44.704 INFO:tasks.workunit.client.0.smithi144.stdout:4/429: creat d9/d26/d48/f8f x:0 0 0 2024-07-26T11:04:44.705 INFO:tasks.workunit.client.0.smithi144.stdout:4/430: symlink d9/d57/l90 0 2024-07-26T11:04:44.706 INFO:tasks.workunit.client.0.smithi144.stdout:4/431: getdents d9/d26/d48/d50 0 2024-07-26T11:04:44.707 INFO:tasks.workunit.client.0.smithi144.stdout:4/432: symlink d9/d57/l91 0 2024-07-26T11:04:44.708 INFO:tasks.workunit.client.0.smithi144.stdout:4/433: creat d9/d26/f92 x:0 0 0 2024-07-26T11:04:44.709 INFO:tasks.workunit.client.0.smithi144.stdout:4/434: symlink d9/l93 0 2024-07-26T11:04:44.711 INFO:tasks.workunit.client.0.smithi144.stdout:4/435: symlink d9/d26/d48/d50/d60/l94 0 2024-07-26T11:04:44.711 INFO:tasks.workunit.client.0.smithi144.stdout:4/436: write d9/d57/f8a [999688,77531] 0 2024-07-26T11:04:44.711 INFO:tasks.workunit.client.0.smithi144.stdout:4/437: write d9/d26/f92 [909599,123813] 0 2024-07-26T11:04:44.716 INFO:tasks.workunit.client.0.smithi144.stdout:4/438: creat d9/d26/d3c/d6f/d8c/f95 x:0 0 0 2024-07-26T11:04:44.716 INFO:tasks.workunit.client.0.smithi144.stdout:4/439: chown d9/d26/d48/c87 1 1 2024-07-26T11:04:44.730 INFO:tasks.workunit.client.0.smithi144.stdout:4/440: dread d9/f63 [0,4194304] 0 2024-07-26T11:04:44.732 INFO:tasks.workunit.client.0.smithi144.stdout:4/441: creat d9/d26/d3c/d6f/d8c/f96 x:0 0 0 2024-07-26T11:04:44.748 INFO:tasks.workunit.client.0.smithi144.stdout:7/461: dwrite d0/d6/d85/d2f/d47/d5e/f69 [0,4194304] 0 2024-07-26T11:04:44.754 INFO:tasks.workunit.client.0.smithi144.stdout:7/462: unlink d0/l15 0 2024-07-26T11:04:44.767 INFO:tasks.workunit.client.0.smithi144.stdout:7/463: dread d0/d6/d9/d45/f3e [0,4194304] 0 2024-07-26T11:04:44.768 INFO:tasks.workunit.client.0.smithi144.stdout:7/464: truncate d0/d6/d9/d64/f65 249825 0 2024-07-26T11:04:44.769 INFO:tasks.workunit.client.0.smithi144.stdout:7/465: creat d0/d6/d9/d12/fb1 x:0 0 0 2024-07-26T11:04:44.769 INFO:tasks.workunit.client.0.smithi144.stdout:7/466: chown d0/d6/d9/f13 43872230 1 2024-07-26T11:04:44.779 INFO:tasks.workunit.client.0.smithi144.stdout:4/442: dwrite d9/da/f23 [0,4194304] 0 2024-07-26T11:04:44.791 INFO:tasks.workunit.client.0.smithi144.stdout:4/443: dread d9/d26/f92 [0,4194304] 0 2024-07-26T11:04:44.792 INFO:tasks.workunit.client.0.smithi144.stdout:4/444: creat d9/d26/f97 x:0 0 0 2024-07-26T11:04:44.792 INFO:tasks.workunit.client.0.smithi144.stdout:4/445: getdents d9/d26/d48/d50 0 2024-07-26T11:04:44.793 INFO:tasks.workunit.client.0.smithi144.stdout:4/446: mkdir d9/d26/d48/d98 0 2024-07-26T11:04:44.793 INFO:tasks.workunit.client.0.smithi144.stdout:4/447: chown d9/d26/d3c/d6f/l80 37692 1 2024-07-26T11:04:44.795 INFO:tasks.workunit.client.0.smithi144.stdout:4/448: rename d9/d57/f8a to d9/d26/f99 0 2024-07-26T11:04:44.795 INFO:tasks.workunit.client.0.smithi144.stdout:4/449: dread - d9/d26/d3c/f7a zero size 2024-07-26T11:04:44.818 INFO:tasks.workunit.client.0.smithi144.stdout:4/450: read d9/d26/d48/d50/f59 [2052597,67681] 0 2024-07-26T11:04:44.818 INFO:tasks.workunit.client.0.smithi144.stdout:4/451: truncate d9/d26/f28 676521 0 2024-07-26T11:04:44.820 INFO:tasks.workunit.client.0.smithi144.stdout:4/452: getdents d9/d26/d3c 0 2024-07-26T11:04:44.820 INFO:tasks.workunit.client.0.smithi144.stdout:4/453: chown d9/da/c46 2 1 2024-07-26T11:04:44.820 INFO:tasks.workunit.client.0.smithi144.stdout:4/454: write d9/da/f39 [4949406,81711] 0 2024-07-26T11:04:44.820 INFO:tasks.workunit.client.0.smithi144.stdout:4/455: truncate d9/da/f52 412981 0 2024-07-26T11:04:44.822 INFO:tasks.workunit.client.0.smithi144.stdout:4/456: mknod d9/da/c9a 0 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:8/423: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:1/497: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:5/462: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:6/503: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:0/416: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:9/380: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:3/295: sync 2024-07-26T11:04:44.999 INFO:tasks.workunit.client.0.smithi144.stdout:2/489: sync 2024-07-26T11:04:45.003 INFO:tasks.workunit.client.0.smithi144.stdout:8/424: truncate fe 1175871 0 2024-07-26T11:04:45.003 INFO:tasks.workunit.client.0.smithi144.stdout:1/498: mkdir d3/d6/da/d13/d22/d5d/d9a 0 2024-07-26T11:04:45.003 INFO:tasks.workunit.client.0.smithi144.stdout:5/463: mkdir d1/d9/d44/da1 0 2024-07-26T11:04:45.003 INFO:tasks.workunit.client.0.smithi144.stdout:5/464: dread - d1/d18/d24/f82 zero size 2024-07-26T11:04:45.003 INFO:tasks.workunit.client.0.smithi144.stdout:1/499: write d3/f39 [6813157,106199] 0 2024-07-26T11:04:45.007 INFO:tasks.workunit.client.0.smithi144.stdout:9/381: rmdir d9/de 39 2024-07-26T11:04:45.007 INFO:tasks.workunit.client.0.smithi144.stdout:8/425: rename f4 to d1a/d32/d49/d73/d7b/d4e/d78/f90 0 2024-07-26T11:04:45.008 INFO:tasks.workunit.client.0.smithi144.stdout:0/417: creat d0/d3e/d5f/f8d x:0 0 0 2024-07-26T11:04:45.011 INFO:tasks.workunit.client.0.smithi144.stdout:0/418: truncate d0/d3e/d5f/f89 631630 0 2024-07-26T11:04:45.011 INFO:tasks.workunit.client.0.smithi144.stdout:2/490: symlink d0/d24/d2b/d68/d82/l97 0 2024-07-26T11:04:45.016 INFO:tasks.workunit.client.0.smithi144.stdout:9/382: symlink d9/d3b/d76/l7b 0 2024-07-26T11:04:45.037 INFO:tasks.workunit.client.0.smithi144.stdout:0/419: dread d0/d3e/d45/f55 [0,4194304] 0 2024-07-26T11:04:45.037 INFO:tasks.workunit.client.0.smithi144.stdout:0/420: truncate d0/d25/d5e/d1e/d30/f39 936196 0 2024-07-26T11:04:45.041 INFO:tasks.workunit.client.0.smithi144.stdout:6/504: write d8/f16 [3730173,101430] 0 2024-07-26T11:04:45.041 INFO:tasks.workunit.client.0.smithi144.stdout:6/505: fdatasync d8/f67 0 2024-07-26T11:04:45.044 INFO:tasks.workunit.client.0.smithi144.stdout:6/506: readlink d8/d15/d57/l65 0 2024-07-26T11:04:45.044 INFO:tasks.workunit.client.0.smithi144.stdout:6/507: dread - d8/d50/d98/f9f zero size 2024-07-26T11:04:45.046 INFO:tasks.workunit.client.0.smithi144.stdout:6/508: unlink d8/d15/d57/d5f/d6e/l80 0 2024-07-26T11:04:45.049 INFO:tasks.workunit.client.0.smithi144.stdout:9/383: read d9/de/d14/d23/d39/f5b [1538640,124495] 0 2024-07-26T11:04:45.049 INFO:tasks.workunit.client.0.smithi144.stdout:1/500: write d3/d6/da/d13/d19/d3a/f79 [1701730,95880] 0 2024-07-26T11:04:45.050 INFO:tasks.workunit.client.0.smithi144.stdout:9/384: link d9/de/d14/f22 d9/d3b/f7c 0 2024-07-26T11:04:45.051 INFO:tasks.workunit.client.0.smithi144.stdout:1/501: creat d3/f9b x:0 0 0 2024-07-26T11:04:45.052 INFO:tasks.workunit.client.0.smithi144.stdout:9/385: mknod d9/d3b/c7d 0 2024-07-26T11:04:45.052 INFO:tasks.workunit.client.0.smithi144.stdout:9/386: fsync d9/d3b/d5d/f70 0 2024-07-26T11:04:45.052 INFO:tasks.workunit.client.0.smithi144.stdout:9/387: fsync d9/d3b/f56 0 2024-07-26T11:04:45.076 INFO:tasks.workunit.client.0.smithi144.stdout:8/426: dread d1a/d1c/f2d [0,4194304] 0 2024-07-26T11:04:45.078 INFO:tasks.workunit.client.0.smithi144.stdout:8/427: mknod d1a/d32/d49/d73/d2c/c91 0 2024-07-26T11:04:45.078 INFO:tasks.workunit.client.0.smithi144.stdout:8/428: write d1a/d1c/f22 [3435933,30352] 0 2024-07-26T11:04:45.078 INFO:tasks.workunit.client.0.smithi144.stdout:8/429: stat d1a/d32/d49/d73/d7b/d43/c47 0 2024-07-26T11:04:45.081 INFO:tasks.workunit.client.0.smithi144.stdout:8/430: rename d1a/d32/d49/d73/d2c/d38/d58/l72 to d1a/l92 0 2024-07-26T11:04:45.082 INFO:tasks.workunit.client.0.smithi144.stdout:8/431: mknod d1a/d32/d49/d73/c93 0 2024-07-26T11:04:45.089 INFO:tasks.workunit.client.0.smithi144.stdout:2/491: dwrite d0/f91 [0,4194304] 0 2024-07-26T11:04:45.090 INFO:tasks.workunit.client.0.smithi144.stdout:3/296: dwrite d1/d28/d54/d3d/f40 [0,4194304] 0 2024-07-26T11:04:45.093 INFO:tasks.workunit.client.0.smithi144.stdout:2/492: rename d0/d24/f62 to d0/d24/d2b/d68/d82/f98 0 2024-07-26T11:04:45.095 INFO:tasks.workunit.client.0.smithi144.stdout:1/502: dread d3/d6/da/d13/d19/d3a/d3e/f50 [0,4194304] 0 2024-07-26T11:04:45.095 INFO:tasks.workunit.client.0.smithi144.stdout:1/503: chown d3/d6/da/d21/c63 0 1 2024-07-26T11:04:45.095 INFO:tasks.workunit.client.0.smithi144.stdout:1/504: write d3/d6/da/f4a [4670778,29451] 0 2024-07-26T11:04:45.097 INFO:tasks.workunit.client.0.smithi144.stdout:1/505: write d3/d6/da/d78/f99 [363956,130201] 0 2024-07-26T11:04:45.097 INFO:tasks.workunit.client.0.smithi144.stdout:9/388: write d9/de/d14/f24 [3615459,78051] 0 2024-07-26T11:04:45.100 INFO:tasks.workunit.client.0.smithi144.stdout:2/493: creat d0/d24/d48/d6d/d28/d2e/d5d/d6b/f99 x:0 0 0 2024-07-26T11:04:45.104 INFO:tasks.workunit.client.0.smithi144.stdout:7/467: sync 2024-07-26T11:04:45.104 INFO:tasks.workunit.client.0.smithi144.stdout:7/468: truncate d0/d6/d85/d2f/d47/f4a 1029078 0 2024-07-26T11:04:45.105 INFO:tasks.workunit.client.0.smithi144.stdout:9/389: truncate d9/de/f46 853066 0 2024-07-26T11:04:45.109 INFO:tasks.workunit.client.0.smithi144.stdout:4/457: sync 2024-07-26T11:04:45.110 INFO:tasks.workunit.client.0.smithi144.stdout:7/469: creat d0/d6/d85/fb2 x:0 0 0 2024-07-26T11:04:45.110 INFO:tasks.workunit.client.0.smithi144.stdout:9/390: getdents d9/de/d14 0 2024-07-26T11:04:45.114 INFO:tasks.workunit.client.0.smithi144.stdout:9/391: mkdir d9/d3b/d7e 0 2024-07-26T11:04:45.115 INFO:tasks.workunit.client.0.smithi144.stdout:9/392: fsync d9/de/f3d 0 2024-07-26T11:04:45.116 INFO:tasks.workunit.client.0.smithi144.stdout:5/465: dwrite d1/f8f [0,4194304] 0 2024-07-26T11:04:45.116 INFO:tasks.workunit.client.0.smithi144.stdout:5/466: chown d1/d5/c1c 270637514 1 2024-07-26T11:04:45.119 INFO:tasks.workunit.client.0.smithi144.stdout:9/393: mkdir d9/de/d14/d23/d39/d7f 0 2024-07-26T11:04:45.121 INFO:tasks.workunit.client.0.smithi144.stdout:4/458: dread d9/f76 [0,4194304] 0 2024-07-26T11:04:45.121 INFO:tasks.workunit.client.0.smithi144.stdout:4/459: write d9/d26/d48/d50/f6c [755822,106259] 0 2024-07-26T11:04:45.123 INFO:tasks.workunit.client.0.smithi144.stdout:4/460: mknod d9/d26/d48/d98/c9b 0 2024-07-26T11:04:45.123 INFO:tasks.workunit.client.0.smithi144.stdout:4/461: getdents d9/d26/d3c 0 2024-07-26T11:04:45.124 INFO:tasks.workunit.client.0.smithi144.stdout:4/462: mknod d9/d26/d48/d50/c9c 0 2024-07-26T11:04:45.126 INFO:tasks.workunit.client.0.smithi144.stdout:9/394: mknod d9/de/d14/d23/c80 0 2024-07-26T11:04:45.126 INFO:tasks.workunit.client.0.smithi144.stdout:9/395: write d9/de/f20 [1513809,51654] 0 2024-07-26T11:04:45.126 INFO:tasks.workunit.client.0.smithi144.stdout:9/396: fsync d9/d3b/f7c 0 2024-07-26T11:04:45.129 INFO:tasks.workunit.client.0.smithi144.stdout:4/463: unlink d9/d26/d48/c7e 0 2024-07-26T11:04:45.129 INFO:tasks.workunit.client.0.smithi144.stdout:4/464: chown d9/d26/d3c/c70 37 1 2024-07-26T11:04:45.129 INFO:tasks.workunit.client.0.smithi144.stdout:4/465: chown d9/d26/f83 29276093 1 2024-07-26T11:04:45.129 INFO:tasks.workunit.client.0.smithi144.stdout:9/397: mkdir d9/d3b/d81 0 2024-07-26T11:04:45.129 INFO:tasks.workunit.client.0.smithi144.stdout:4/466: truncate d9/d26/d3c/d6f/d8c/f95 344573 0 2024-07-26T11:04:45.129 INFO:tasks.workunit.client.0.smithi144.stdout:4/467: chown d9/d26/d3c/d6f/f85 0 1 2024-07-26T11:04:45.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/468: mknod d9/d26/d3c/d6f/c9d 0 2024-07-26T11:04:45.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/469: mknod d9/da/c9e 0 2024-07-26T11:04:45.152 INFO:tasks.workunit.client.0.smithi144.stdout:2/494: dwrite d0/d24/d2b/f59 [4194304,4194304] 0 2024-07-26T11:04:45.165 INFO:tasks.workunit.client.0.smithi144.stdout:1/506: dwrite d3/f9b [0,4194304] 0 2024-07-26T11:04:45.165 INFO:tasks.workunit.client.0.smithi144.stdout:1/507: fsync d3/d6/da/d21/d67/f71 0 2024-07-26T11:04:45.166 INFO:tasks.workunit.client.0.smithi144.stdout:1/508: mkdir d3/d6/d9c 0 2024-07-26T11:04:45.169 INFO:tasks.workunit.client.0.smithi144.stdout:6/509: dwrite d8/d4f/d52/d58/f97 [0,4194304] 0 2024-07-26T11:04:45.171 INFO:tasks.workunit.client.0.smithi144.stdout:8/432: dwrite d1a/d32/d49/d73/f2e [0,4194304] 0 2024-07-26T11:04:45.175 INFO:tasks.workunit.client.0.smithi144.stdout:8/433: truncate d1a/d32/d49/d73/d7b/d4e/d78/f90 972739 0 2024-07-26T11:04:45.177 INFO:tasks.workunit.client.0.smithi144.stdout:8/434: creat d1a/d32/d49/d73/d2c/f94 x:0 0 0 2024-07-26T11:04:45.180 INFO:tasks.workunit.client.0.smithi144.stdout:8/435: rename f15 to d1a/f95 0 2024-07-26T11:04:45.186 INFO:tasks.workunit.client.0.smithi144.stdout:9/398: dwrite d9/de/d14/d23/f50 [0,4194304] 0 2024-07-26T11:04:45.186 INFO:tasks.workunit.client.0.smithi144.stdout:9/399: chown d9/de/f46 9 1 2024-07-26T11:04:45.186 INFO:tasks.workunit.client.0.smithi144.stdout:9/400: fdatasync d9/de/f4d 0 2024-07-26T11:04:45.192 INFO:tasks.workunit.client.0.smithi144.stdout:3/297: dwrite d1/f22 [0,4194304] 0 2024-07-26T11:04:45.192 INFO:tasks.workunit.client.0.smithi144.stdout:9/401: rmdir d9/d3b/d7e 0 2024-07-26T11:04:45.192 INFO:tasks.workunit.client.0.smithi144.stdout:3/298: dread - d1/d28/d54/d3d/f53 zero size 2024-07-26T11:04:45.195 INFO:tasks.workunit.client.0.smithi144.stdout:3/299: rmdir d1/d28/d54/d26 39 2024-07-26T11:04:45.195 INFO:tasks.workunit.client.0.smithi144.stdout:3/300: fdatasync d1/f32 0 2024-07-26T11:04:45.196 INFO:tasks.workunit.client.0.smithi144.stdout:9/402: creat d9/de/d14/d79/f82 x:0 0 0 2024-07-26T11:04:45.200 INFO:tasks.workunit.client.0.smithi144.stdout:9/403: rmdir d9/d3b/d5d 39 2024-07-26T11:04:45.200 INFO:tasks.workunit.client.0.smithi144.stdout:9/404: write d9/d3b/f78 [202679,105255] 0 2024-07-26T11:04:45.200 INFO:tasks.workunit.client.0.smithi144.stdout:1/509: dread d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:45.200 INFO:tasks.workunit.client.0.smithi144.stdout:1/510: write d3/d3d/d58/f7d [5163820,98153] 0 2024-07-26T11:04:45.201 INFO:tasks.workunit.client.0.smithi144.stdout:5/467: dwrite d1/d5/f7f [0,4194304] 0 2024-07-26T11:04:45.201 INFO:tasks.workunit.client.0.smithi144.stdout:5/468: write d1/d18/f48 [211153,38349] 0 2024-07-26T11:04:45.202 INFO:tasks.workunit.client.0.smithi144.stdout:4/470: dwrite d9/d26/d48/f66 [0,4194304] 0 2024-07-26T11:04:45.203 INFO:tasks.workunit.client.0.smithi144.stdout:4/471: write d9/d26/f31 [3337041,125346] 0 2024-07-26T11:04:45.204 INFO:tasks.workunit.client.0.smithi144.stdout:7/470: dwrite d0/d6/d85/d67/f9d [0,4194304] 0 2024-07-26T11:04:45.205 INFO:tasks.workunit.client.0.smithi144.stdout:9/405: mknod d9/de/d14/d3f/d5e/c83 0 2024-07-26T11:04:45.205 INFO:tasks.workunit.client.0.smithi144.stdout:9/406: write d9/de/d14/d23/d39/f66 [1417616,64232] 0 2024-07-26T11:04:45.206 INFO:tasks.workunit.client.0.smithi144.stdout:9/407: fsync d9/de/f46 0 2024-07-26T11:04:45.206 INFO:tasks.workunit.client.0.smithi144.stdout:9/408: fdatasync d9/de/d14/f2c 0 2024-07-26T11:04:45.206 INFO:tasks.workunit.client.0.smithi144.stdout:1/511: creat d3/d6/da/d78/f9d x:0 0 0 2024-07-26T11:04:45.209 INFO:tasks.workunit.client.0.smithi144.stdout:4/472: symlink d9/d57/l9f 0 2024-07-26T11:04:45.214 INFO:tasks.workunit.client.0.smithi144.stdout:9/409: creat d9/de/d14/d23/d39/d7f/f84 x:0 0 0 2024-07-26T11:04:45.215 INFO:tasks.workunit.client.0.smithi144.stdout:1/512: rename d3/d6/da/d13/d19/d3a/l52 to d3/d6/da/d21/d67/l9e 0 2024-07-26T11:04:45.215 INFO:tasks.workunit.client.0.smithi144.stdout:1/513: chown d3/d6/l61 1106 1 2024-07-26T11:04:45.230 INFO:tasks.workunit.client.0.smithi144.stdout:3/301: write d1/d5/f15 [393897,80725] 0 2024-07-26T11:04:45.230 INFO:tasks.workunit.client.0.smithi144.stdout:3/302: chown d1/d28/d54/c36 96277790 1 2024-07-26T11:04:45.234 INFO:tasks.workunit.client.0.smithi144.stdout:3/303: truncate d1/d5/f24 4887708 0 2024-07-26T11:04:45.234 INFO:tasks.workunit.client.0.smithi144.stdout:3/304: truncate d1/d28/d54/d26/d37/f41 142147 0 2024-07-26T11:04:45.236 INFO:tasks.workunit.client.0.smithi144.stdout:3/305: creat d1/d28/d54/d26/d30/d4d/f55 x:0 0 0 2024-07-26T11:04:45.237 INFO:tasks.workunit.client.0.smithi144.stdout:2/495: dwrite d0/d24/d48/d6d/d28/d2e/d5d/d6b/f99 [0,4194304] 0 2024-07-26T11:04:45.238 INFO:tasks.workunit.client.0.smithi144.stdout:8/436: dread d1a/d32/d49/d73/f80 [0,4194304] 0 2024-07-26T11:04:45.243 INFO:tasks.workunit.client.0.smithi144.stdout:3/306: fdatasync d1/d5/f15 0 2024-07-26T11:04:45.244 INFO:tasks.workunit.client.0.smithi144.stdout:8/437: mkdir d1a/d32/d66/d96 0 2024-07-26T11:04:45.245 INFO:tasks.workunit.client.0.smithi144.stdout:6/510: dwrite d8/f2d [0,4194304] 0 2024-07-26T11:04:45.247 INFO:tasks.workunit.client.0.smithi144.stdout:3/307: rename d1/d28/d54/d26/d37/f49 to d1/d28/d54/d3d/d42/d44/d48/f56 0 2024-07-26T11:04:45.255 INFO:tasks.workunit.client.0.smithi144.stdout:3/308: truncate d1/f17 1946331 0 2024-07-26T11:04:45.255 INFO:tasks.workunit.client.0.smithi144.stdout:3/309: stat d1/d28/d54/d26/f2e 0 2024-07-26T11:04:45.259 INFO:tasks.workunit.client.0.smithi144.stdout:1/514: write d3/d6/da/f4a [2962299,122523] 0 2024-07-26T11:04:45.259 INFO:tasks.workunit.client.0.smithi144.stdout:1/515: chown d3/d6/da/d21/d5c 54573573 1 2024-07-26T11:04:45.260 INFO:tasks.workunit.client.0.smithi144.stdout:1/516: getdents d3/d6/da/d21/d5c/d62/d6e 0 2024-07-26T11:04:45.262 INFO:tasks.workunit.client.0.smithi144.stdout:5/469: dwrite d1/d5/f2d [4194304,4194304] 0 2024-07-26T11:04:45.265 INFO:tasks.workunit.client.0.smithi144.stdout:5/470: creat d1/d9/d44/d45/fa2 x:0 0 0 2024-07-26T11:04:45.268 INFO:tasks.workunit.client.0.smithi144.stdout:3/310: dread d1/d5/f1c [0,4194304] 0 2024-07-26T11:04:45.271 INFO:tasks.workunit.client.0.smithi144.stdout:3/311: mknod d1/d28/d54/d26/d37/d46/c57 0 2024-07-26T11:04:45.280 INFO:tasks.workunit.client.0.smithi144.stdout:4/473: dwrite d9/d26/f2b [0,4194304] 0 2024-07-26T11:04:45.280 INFO:tasks.workunit.client.0.smithi144.stdout:8/438: dread d1a/d32/d49/d73/d2c/f36 [0,4194304] 0 2024-07-26T11:04:45.288 INFO:tasks.workunit.client.0.smithi144.stdout:6/511: write d8/db/f13 [1721046,83220] 0 2024-07-26T11:04:45.307 INFO:tasks.workunit.client.0.smithi144.stdout:8/439: creat d1a/d1c/f97 x:0 0 0 2024-07-26T11:04:45.310 INFO:tasks.workunit.client.0.smithi144.stdout:6/512: creat d8/d4f/daa/fac x:0 0 0 2024-07-26T11:04:45.314 INFO:tasks.workunit.client.0.smithi144.stdout:8/440: rename d1a/c23 to d1a/d32/d8c/c98 0 2024-07-26T11:04:45.314 INFO:tasks.workunit.client.0.smithi144.stdout:8/441: stat d1a/d32/d49/d73/d7b/d43/d4c/d55/d74 0 2024-07-26T11:04:45.314 INFO:tasks.workunit.client.0.smithi144.stdout:6/513: getdents d8/d4f 0 2024-07-26T11:04:45.315 INFO:tasks.workunit.client.0.smithi144.stdout:6/514: write f2 [13070102,77628] 0 2024-07-26T11:04:45.317 INFO:tasks.workunit.client.0.smithi144.stdout:6/515: mkdir d8/d42/dad 0 2024-07-26T11:04:45.322 INFO:tasks.workunit.client.0.smithi144.stdout:4/474: dread d9/f76 [0,4194304] 0 2024-07-26T11:04:45.322 INFO:tasks.workunit.client.0.smithi144.stdout:4/475: dread - d9/d26/f97 zero size 2024-07-26T11:04:45.322 INFO:tasks.workunit.client.0.smithi144.stdout:6/516: link d8/d4f/d52/c93 d8/d42/cae 0 2024-07-26T11:04:45.322 INFO:tasks.workunit.client.0.smithi144.stdout:4/476: creat d9/d26/d48/fa0 x:0 0 0 2024-07-26T11:04:45.322 INFO:tasks.workunit.client.0.smithi144.stdout:6/517: creat d8/d4f/d52/d58/faf x:0 0 0 2024-07-26T11:04:45.325 INFO:tasks.workunit.client.0.smithi144.stdout:1/517: dwrite d3/d6/da/d21/f51 [0,4194304] 0 2024-07-26T11:04:45.325 INFO:tasks.workunit.client.0.smithi144.stdout:4/477: mkdir d9/da/da1 0 2024-07-26T11:04:45.326 INFO:tasks.workunit.client.0.smithi144.stdout:6/518: creat d8/d50/d92/fb0 x:0 0 0 2024-07-26T11:04:45.326 INFO:tasks.workunit.client.0.smithi144.stdout:6/519: truncate d8/d50/d92/fb0 539382 0 2024-07-26T11:04:45.326 INFO:tasks.workunit.client.0.smithi144.stdout:6/520: chown d8/db/c11 212 1 2024-07-26T11:04:45.326 INFO:tasks.workunit.client.0.smithi144.stdout:6/521: fsync d8/d15/d26/f4c 0 2024-07-26T11:04:45.327 INFO:tasks.workunit.client.0.smithi144.stdout:9/410: dwrite d9/de/f4b [0,4194304] 0 2024-07-26T11:04:45.333 INFO:tasks.workunit.client.0.smithi144.stdout:1/518: mkdir d3/d6/d9c/d9f 0 2024-07-26T11:04:45.333 INFO:tasks.workunit.client.0.smithi144.stdout:6/522: link d8/d15/f94 d8/d4f/d74/fb1 0 2024-07-26T11:04:45.333 INFO:tasks.workunit.client.0.smithi144.stdout:3/312: dwrite d1/f50 [0,4194304] 0 2024-07-26T11:04:45.333 INFO:tasks.workunit.client.0.smithi144.stdout:3/313: fdatasync d1/f50 0 2024-07-26T11:04:45.335 INFO:tasks.workunit.client.0.smithi144.stdout:6/523: mknod d8/db/cb2 0 2024-07-26T11:04:45.354 INFO:tasks.workunit.client.0.smithi144.stdout:9/411: write d9/de/d14/d23/f50 [3112752,115520] 0 2024-07-26T11:04:45.355 INFO:tasks.workunit.client.0.smithi144.stdout:1/519: dread d3/d6/f18 [0,4194304] 0 2024-07-26T11:04:45.355 INFO:tasks.workunit.client.0.smithi144.stdout:1/520: read d3/d6/da/f4a [2570131,25781] 0 2024-07-26T11:04:45.355 INFO:tasks.workunit.client.0.smithi144.stdout:1/521: truncate d3/d6/da/d21/d67/f71 570125 0 2024-07-26T11:04:45.355 INFO:tasks.workunit.client.0.smithi144.stdout:1/522: readlink d3/d6/l61 0 2024-07-26T11:04:45.356 INFO:tasks.workunit.client.0.smithi144.stdout:9/412: truncate d9/de/f20 1416026 0 2024-07-26T11:04:45.356 INFO:tasks.workunit.client.0.smithi144.stdout:9/413: chown d9/fc 45841 1 2024-07-26T11:04:45.356 INFO:tasks.workunit.client.0.smithi144.stdout:1/523: symlink d3/d3d/la0 0 2024-07-26T11:04:45.359 INFO:tasks.workunit.client.0.smithi144.stdout:9/414: symlink d9/de/d4e/d6a/l85 0 2024-07-26T11:04:45.359 INFO:tasks.workunit.client.0.smithi144.stdout:9/415: fsync d9/d3b/f7c 0 2024-07-26T11:04:45.359 INFO:tasks.workunit.client.0.smithi144.stdout:9/416: chown d9/de/f3d 745861905 1 2024-07-26T11:04:45.368 INFO:tasks.workunit.client.0.smithi144.stdout:4/478: dwrite f1 [4194304,4194304] 0 2024-07-26T11:04:45.368 INFO:tasks.workunit.client.0.smithi144.stdout:4/479: write d9/d26/d48/fa0 [686215,42468] 0 2024-07-26T11:04:45.368 INFO:tasks.workunit.client.0.smithi144.stdout:4/480: truncate d9/d26/f2f 762390 0 2024-07-26T11:04:45.368 INFO:tasks.workunit.client.0.smithi144.stdout:4/481: write d9/d26/d48/d50/f59 [3170580,48535] 0 2024-07-26T11:04:45.373 INFO:tasks.workunit.client.0.smithi144.stdout:6/524: write d8/d15/d26/f3f [3941663,50578] 0 2024-07-26T11:04:45.373 INFO:tasks.workunit.client.0.smithi144.stdout:9/417: mknod d9/de/d14/d79/c86 0 2024-07-26T11:04:45.373 INFO:tasks.workunit.client.0.smithi144.stdout:4/482: symlink d9/d7b/la2 0 2024-07-26T11:04:45.373 INFO:tasks.workunit.client.0.smithi144.stdout:4/483: write d9/d57/f72 [761251,86095] 0 2024-07-26T11:04:45.373 INFO:tasks.workunit.client.0.smithi144.stdout:4/484: stat d9/f8d 0 2024-07-26T11:04:45.373 INFO:tasks.workunit.client.0.smithi144.stdout:9/418: symlink d9/de/d14/d3f/d5e/l87 0 2024-07-26T11:04:45.376 INFO:tasks.workunit.client.0.smithi144.stdout:3/314: dread d1/ff [0,4194304] 0 2024-07-26T11:04:45.376 INFO:tasks.workunit.client.0.smithi144.stdout:3/315: write d1/f1b [959581,50726] 0 2024-07-26T11:04:45.377 INFO:tasks.workunit.client.0.smithi144.stdout:4/485: mknod d9/d26/d3c/d6f/ca3 0 2024-07-26T11:04:45.378 INFO:tasks.workunit.client.0.smithi144.stdout:9/419: rmdir d9/de/d14/d3f 39 2024-07-26T11:04:45.378 INFO:tasks.workunit.client.0.smithi144.stdout:4/486: fsync d9/d26/f31 0 2024-07-26T11:04:45.382 INFO:tasks.workunit.client.0.smithi144.stdout:3/316: dread d1/d28/d54/d3d/d42/d44/f4e [0,4194304] 0 2024-07-26T11:04:45.384 INFO:tasks.workunit.client.0.smithi144.stdout:1/524: dread d3/f4f [0,4194304] 0 2024-07-26T11:04:45.384 INFO:tasks.workunit.client.0.smithi144.stdout:8/442: dwrite d1a/d32/d49/d73/d2c/d38/d58/f83 [0,4194304] 0 2024-07-26T11:04:45.384 INFO:tasks.workunit.client.0.smithi144.stdout:9/420: mknod d9/de/d14/d23/d39/c88 0 2024-07-26T11:04:45.386 INFO:tasks.workunit.client.0.smithi144.stdout:4/487: rename d9/d26/f31 to d9/d57/fa4 0 2024-07-26T11:04:45.387 INFO:tasks.workunit.client.0.smithi144.stdout:3/317: creat d1/d28/d54/d26/d37/f58 x:0 0 0 2024-07-26T11:04:45.390 INFO:tasks.workunit.client.0.smithi144.stdout:1/525: creat d3/d6/da/d13/d19/d3a/d69/d36/d37/fa1 x:0 0 0 2024-07-26T11:04:45.393 INFO:tasks.workunit.client.0.smithi144.stdout:9/421: link d9/de/d14/d3f/l40 d9/d3b/d76/l89 0 2024-07-26T11:04:45.393 INFO:tasks.workunit.client.0.smithi144.stdout:9/422: write d9/de/d14/d23/f50 [2481968,44073] 0 2024-07-26T11:04:45.397 INFO:tasks.workunit.client.0.smithi144.stdout:4/488: rename d9/d26/d3c/c65 to d9/ca5 0 2024-07-26T11:04:45.397 INFO:tasks.workunit.client.0.smithi144.stdout:3/318: mknod d1/d28/c59 0 2024-07-26T11:04:45.403 INFO:tasks.workunit.client.0.smithi144.stdout:1/526: creat d3/d6/da/d70/fa2 x:0 0 0 2024-07-26T11:04:45.404 INFO:tasks.workunit.client.0.smithi144.stdout:3/319: rmdir d1/d28 39 2024-07-26T11:04:45.404 INFO:tasks.workunit.client.0.smithi144.stdout:4/489: mknod d9/d26/d48/d50/ca6 0 2024-07-26T11:04:45.405 INFO:tasks.workunit.client.0.smithi144.stdout:9/423: creat d9/de/d4e/d6a/f8a x:0 0 0 2024-07-26T11:04:45.405 INFO:tasks.workunit.client.0.smithi144.stdout:9/424: write d9/d3b/d5d/f6b [172383,88571] 0 2024-07-26T11:04:45.405 INFO:tasks.workunit.client.0.smithi144.stdout:4/490: write d9/f76 [746144,68313] 0 2024-07-26T11:04:45.405 INFO:tasks.workunit.client.0.smithi144.stdout:4/491: read - d9/d57/f77 zero size 2024-07-26T11:04:45.405 INFO:tasks.workunit.client.0.smithi144.stdout:8/443: dread d1a/f95 [0,4194304] 0 2024-07-26T11:04:45.405 INFO:tasks.workunit.client.0.smithi144.stdout:1/527: mknod d3/d6/da/d78/ca3 0 2024-07-26T11:04:45.407 INFO:tasks.workunit.client.0.smithi144.stdout:3/320: link d1/d28/c59 d1/d28/d54/d26/c5a 0 2024-07-26T11:04:45.409 INFO:tasks.workunit.client.0.smithi144.stdout:9/425: unlink d9/de/d14/d3f/c75 0 2024-07-26T11:04:45.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/492: unlink d9/da/f23 0 2024-07-26T11:04:45.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/493: truncate d9/d26/d48/d50/d60/f88 704145 0 2024-07-26T11:04:45.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/494: write d9/d57/f8e [825516,27972] 0 2024-07-26T11:04:45.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/495: stat d9/d26/d48/f4d 0 2024-07-26T11:04:45.412 INFO:tasks.workunit.client.0.smithi144.stdout:1/528: link d3/d6/da/d13/d22/f30 d3/d6/fa4 0 2024-07-26T11:04:45.414 INFO:tasks.workunit.client.0.smithi144.stdout:3/321: mkdir d1/d28/d54/d5b 0 2024-07-26T11:04:45.414 INFO:tasks.workunit.client.0.smithi144.stdout:3/322: stat d1/d5/fa 0 2024-07-26T11:04:45.414 INFO:tasks.workunit.client.0.smithi144.stdout:3/323: chown d1/d28 120076157 1 2024-07-26T11:04:45.415 INFO:tasks.workunit.client.0.smithi144.stdout:9/426: creat d9/d3b/d5d/f8b x:0 0 0 2024-07-26T11:04:45.415 INFO:tasks.workunit.client.0.smithi144.stdout:9/427: chown d9/d3b/f7c 25 1 2024-07-26T11:04:45.415 INFO:tasks.workunit.client.0.smithi144.stdout:9/428: chown d9/de/d14/d23/d39 52676750 1 2024-07-26T11:04:45.416 INFO:tasks.workunit.client.0.smithi144.stdout:4/496: creat d9/d26/d48/d50/fa7 x:0 0 0 2024-07-26T11:04:45.422 INFO:tasks.workunit.client.0.smithi144.stdout:9/429: link d9/de/d4e/d6a/l85 d9/de/d14/d3f/d5e/l8c 0 2024-07-26T11:04:45.426 INFO:tasks.workunit.client.0.smithi144.stdout:9/430: rename d9/d3b/d5d/l73 to d9/d3b/d5d/l8d 0 2024-07-26T11:04:45.432 INFO:tasks.workunit.client.0.smithi144.stdout:4/497: dread d9/d26/f99 [0,4194304] 0 2024-07-26T11:04:45.440 INFO:tasks.workunit.client.0.smithi144.stdout:1/529: dread d3/d6/fa4 [0,4194304] 0 2024-07-26T11:04:45.440 INFO:tasks.workunit.client.0.smithi144.stdout:1/530: write d3/d6/da/d13/d19/d3a/d3e/f6f [527135,4126] 0 2024-07-26T11:04:45.440 INFO:tasks.workunit.client.0.smithi144.stdout:1/531: write d3/d6/da/d21/f4d [1233507,109353] 0 2024-07-26T11:04:45.452 INFO:tasks.workunit.client.0.smithi144.stdout:6/525: dwrite d8/db/f2c [0,4194304] 0 2024-07-26T11:04:45.452 INFO:tasks.workunit.client.0.smithi144.stdout:6/526: chown d8/d4f/d52/d58/c8f 250 1 2024-07-26T11:04:45.453 INFO:tasks.workunit.client.0.smithi144.stdout:6/527: mkdir d8/d50/d92/db3 0 2024-07-26T11:04:45.453 INFO:tasks.workunit.client.0.smithi144.stdout:6/528: dread - d8/d50/f85 zero size 2024-07-26T11:04:45.453 INFO:tasks.workunit.client.0.smithi144.stdout:6/529: fdatasync d8/d15/d26/f4c 0 2024-07-26T11:04:45.453 INFO:tasks.workunit.client.0.smithi144.stdout:6/530: readlink d8/db/le 0 2024-07-26T11:04:45.457 INFO:tasks.workunit.client.0.smithi144.stdout:6/531: rename d8/f10 to d8/d50/d98/fb4 0 2024-07-26T11:04:45.460 INFO:tasks.workunit.client.0.smithi144.stdout:6/532: creat d8/d4f/d74/fb5 x:0 0 0 2024-07-26T11:04:45.466 INFO:tasks.workunit.client.0.smithi144.stdout:6/533: rename d8/d15/l19 to d8/d42/lb6 0 2024-07-26T11:04:45.466 INFO:tasks.workunit.client.0.smithi144.stdout:6/534: chown d8/f67 15 1 2024-07-26T11:04:45.469 INFO:tasks.workunit.client.0.smithi144.stdout:8/444: dwrite d1a/d32/d49/d73/d2c/d38/d58/f6e [0,4194304] 0 2024-07-26T11:04:45.471 INFO:tasks.workunit.client.0.smithi144.stdout:8/445: rename d1a/d32/d49/f4a to d1a/d1c/f99 0 2024-07-26T11:04:45.510 INFO:tasks.workunit.client.0.smithi144.stdout:9/431: dread d9/de/d14/d23/f47 [0,4194304] 0 2024-07-26T11:04:45.510 INFO:tasks.workunit.client.0.smithi144.stdout:9/432: write d9/de/d14/d3f/d5e/f62 [653771,68082] 0 2024-07-26T11:04:45.516 INFO:tasks.workunit.client.0.smithi144.stdout:6/535: dwrite d8/d15/d26/f28 [0,4194304] 0 2024-07-26T11:04:45.516 INFO:tasks.workunit.client.0.smithi144.stdout:1/532: dwrite d3/d6/da/d21/d67/f95 [0,4194304] 0 2024-07-26T11:04:45.519 INFO:tasks.workunit.client.0.smithi144.stdout:1/533: symlink d3/d6/da/d13/d19/d3a/la5 0 2024-07-26T11:04:45.522 INFO:tasks.workunit.client.0.smithi144.stdout:1/534: link d3/d6/da/d21/d67/f71 d3/d53/fa6 0 2024-07-26T11:04:45.522 INFO:tasks.workunit.client.0.smithi144.stdout:1/535: stat d3/d6/da/d13/d19/d3a/d3e 0 2024-07-26T11:04:45.534 INFO:tasks.workunit.client.0.smithi144.stdout:1/536: getdents d3/d6/da/d78 0 2024-07-26T11:04:45.535 INFO:tasks.workunit.client.0.smithi144.stdout:1/537: write d3/d6/da/d13/d19/f45 [4788996,92534] 0 2024-07-26T11:04:45.537 INFO:tasks.workunit.client.0.smithi144.stdout:1/538: link d3/d6/da/d13/d19/d3a/d3e/f6f d3/d6/d9c/d9f/fa7 0 2024-07-26T11:04:45.540 INFO:tasks.workunit.client.0.smithi144.stdout:1/539: unlink d3/d6/da/d5b/f80 0 2024-07-26T11:04:45.550 INFO:tasks.workunit.client.0.smithi144.stdout:4/498: dwrite d9/d26/f62 [0,4194304] 0 2024-07-26T11:04:45.550 INFO:tasks.workunit.client.0.smithi144.stdout:8/446: dwrite d1a/d1c/f41 [0,4194304] 0 2024-07-26T11:04:45.551 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:45 smithi144 ceph-mon[32129]: pgmap v529: 129 pgs: 129 active+clean; 784 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s rd, 122 MiB/s wr, 637 op/s 2024-07-26T11:04:45.553 INFO:tasks.workunit.client.0.smithi144.stdout:8/447: rmdir d1a/d1c 39 2024-07-26T11:04:45.557 INFO:tasks.workunit.client.0.smithi144.stdout:6/536: dwrite d8/f53 [0,4194304] 0 2024-07-26T11:04:45.558 INFO:tasks.workunit.client.0.smithi144.stdout:6/537: readlink d8/l1c 0 2024-07-26T11:04:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:45 smithi160 ceph-mon[36715]: pgmap v529: 129 pgs: 129 active+clean; 784 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s rd, 122 MiB/s wr, 637 op/s 2024-07-26T11:04:45.562 INFO:tasks.workunit.client.0.smithi144.stdout:8/448: mkdir d1a/d32/d49/d73/d7b/d4e/d9a 0 2024-07-26T11:04:45.562 INFO:tasks.workunit.client.0.smithi144.stdout:8/449: write d1a/d1c/f22 [736564,54027] 0 2024-07-26T11:04:45.572 INFO:tasks.workunit.client.0.smithi144.stdout:0/421: sync 2024-07-26T11:04:45.578 INFO:tasks.workunit.client.0.smithi144.stdout:1/540: dread d3/d6/da/d13/d22/f30 [0,4194304] 0 2024-07-26T11:04:45.579 INFO:tasks.workunit.client.0.smithi144.stdout:1/541: truncate d3/d6/f23 4065024 0 2024-07-26T11:04:45.580 INFO:tasks.workunit.client.0.smithi144.stdout:1/542: symlink d3/d6/da/d21/la8 0 2024-07-26T11:04:45.585 INFO:tasks.workunit.client.0.smithi144.stdout:1/543: fsync d3/d6/da/d13/d22/f7f 0 2024-07-26T11:04:45.585 INFO:tasks.workunit.client.0.smithi144.stdout:1/544: dread - d3/d6/da/d78/f7e zero size 2024-07-26T11:04:45.585 INFO:tasks.workunit.client.0.smithi144.stdout:1/545: stat d3/d6/d9c/d9f 0 2024-07-26T11:04:45.588 INFO:tasks.workunit.client.0.smithi144.stdout:9/433: dwrite d9/de/d14/d23/f47 [0,4194304] 0 2024-07-26T11:04:45.589 INFO:tasks.workunit.client.0.smithi144.stdout:9/434: chown d9/de/f15 694 1 2024-07-26T11:04:45.596 INFO:tasks.workunit.client.0.smithi144.stdout:4/499: dwrite d9/d26/d3c/f7a [0,4194304] 0 2024-07-26T11:04:45.596 INFO:tasks.workunit.client.0.smithi144.stdout:4/500: write d9/d26/d3c/d6f/f7c [4515700,57651] 0 2024-07-26T11:04:45.596 INFO:tasks.workunit.client.0.smithi144.stdout:4/501: dread - d9/d26/d48/f4d zero size 2024-07-26T11:04:45.615 INFO:tasks.workunit.client.0.smithi144.stdout:2/496: sync 2024-07-26T11:04:45.615 INFO:tasks.workunit.client.0.smithi144.stdout:5/471: sync 2024-07-26T11:04:45.615 INFO:tasks.workunit.client.0.smithi144.stdout:3/324: sync 2024-07-26T11:04:45.615 INFO:tasks.workunit.client.0.smithi144.stdout:3/325: rmdir d1/d28/d54/d3d/d42 39 2024-07-26T11:04:45.617 INFO:tasks.workunit.client.0.smithi144.stdout:2/497: rename d0/d24/d2b/d68/d82/c88 to d0/d24/d48/d6d/c9a 0 2024-07-26T11:04:45.617 INFO:tasks.workunit.client.0.smithi144.stdout:2/498: chown d0/d24/d48/d6d/d28/f5c 76148 1 2024-07-26T11:04:45.618 INFO:tasks.workunit.client.0.smithi144.stdout:5/472: creat d1/d18/d24/fa3 x:0 0 0 2024-07-26T11:04:45.618 INFO:tasks.workunit.client.0.smithi144.stdout:5/473: truncate d1/d9/f5a 1256844 0 2024-07-26T11:04:45.619 INFO:tasks.workunit.client.0.smithi144.stdout:3/326: mknod d1/d28/c5c 0 2024-07-26T11:04:45.619 INFO:tasks.workunit.client.0.smithi144.stdout:2/499: truncate d0/d6/d9/f33 632844 0 2024-07-26T11:04:45.619 INFO:tasks.workunit.client.0.smithi144.stdout:3/327: chown d1/f1a 45309555 1 2024-07-26T11:04:45.621 INFO:tasks.workunit.client.0.smithi144.stdout:2/500: symlink d0/d24/d48/d6d/d28/l9b 0 2024-07-26T11:04:45.621 INFO:tasks.workunit.client.0.smithi144.stdout:2/501: fsync d0/d24/d48/d6d/d28/f72 0 2024-07-26T11:04:45.627 INFO:tasks.workunit.client.0.smithi144.stdout:6/538: dwrite d8/db/f2c [4194304,4194304] 0 2024-07-26T11:04:45.631 INFO:tasks.workunit.client.0.smithi144.stdout:7/471: sync 2024-07-26T11:04:45.633 INFO:tasks.workunit.client.0.smithi144.stdout:7/472: creat d0/d6/d9/fb3 x:0 0 0 2024-07-26T11:04:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:45 smithi183 ceph-mon[37273]: pgmap v529: 129 pgs: 129 active+clean; 784 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s rd, 122 MiB/s wr, 637 op/s 2024-07-26T11:04:45.636 INFO:tasks.workunit.client.0.smithi144.stdout:7/473: rmdir d0/d6/d9/d45 39 2024-07-26T11:04:45.636 INFO:tasks.workunit.client.0.smithi144.stdout:7/474: chown d0/d6/d9/d3f/c44 217 1 2024-07-26T11:04:45.637 INFO:tasks.workunit.client.0.smithi144.stdout:0/422: dwrite d0/d3e/d45/d4c/f79 [0,4194304] 0 2024-07-26T11:04:45.637 INFO:tasks.workunit.client.0.smithi144.stdout:7/475: mknod d0/cb4 0 2024-07-26T11:04:45.640 INFO:tasks.workunit.client.0.smithi144.stdout:0/423: link d0/d25/d5e/c1f d0/d25/d5e/d1e/d30/d4b/c8e 0 2024-07-26T11:04:45.640 INFO:tasks.workunit.client.0.smithi144.stdout:0/424: chown d0/d3e/d5f/f8d 7956396 1 2024-07-26T11:04:45.640 INFO:tasks.workunit.client.0.smithi144.stdout:7/476: truncate d0/d6/d85/d2f/f70 1742475 0 2024-07-26T11:04:45.641 INFO:tasks.workunit.client.0.smithi144.stdout:7/477: dread - d0/d6/d85/d2f/d47/fae zero size 2024-07-26T11:04:45.641 INFO:tasks.workunit.client.0.smithi144.stdout:7/478: stat d0/d6/d85/d2f/d47/f4a 0 2024-07-26T11:04:45.641 INFO:tasks.workunit.client.0.smithi144.stdout:0/425: rename d0/d3e/d5f/f89 to d0/d3e/d45/d4c/f8f 0 2024-07-26T11:04:45.642 INFO:tasks.workunit.client.0.smithi144.stdout:7/479: mkdir d0/d71/da1/db5 0 2024-07-26T11:04:45.643 INFO:tasks.workunit.client.0.smithi144.stdout:0/426: creat d0/d25/d5e/d1e/d30/d67/f90 x:0 0 0 2024-07-26T11:04:45.643 INFO:tasks.workunit.client.0.smithi144.stdout:0/427: stat d0/d25/d5e/d1e/d30/d6a/f8a 0 2024-07-26T11:04:45.644 INFO:tasks.workunit.client.0.smithi144.stdout:7/480: unlink d0/d6/d85/d67/l94 0 2024-07-26T11:04:45.644 INFO:tasks.workunit.client.0.smithi144.stdout:7/481: chown d0/d6/d85/fb2 62943 1 2024-07-26T11:04:45.644 INFO:tasks.workunit.client.0.smithi144.stdout:7/482: write d0/d6/d9/d45/f8e [857399,48791] 0 2024-07-26T11:04:45.645 INFO:tasks.workunit.client.0.smithi144.stdout:5/474: read d1/d18/f90 [238369,39733] 0 2024-07-26T11:04:45.646 INFO:tasks.workunit.client.0.smithi144.stdout:5/475: mkdir d1/d18/da4 0 2024-07-26T11:04:45.646 INFO:tasks.workunit.client.0.smithi144.stdout:5/476: write d1/d9/d44/d45/f56 [1473931,79292] 0 2024-07-26T11:04:45.646 INFO:tasks.workunit.client.0.smithi144.stdout:5/477: chown d1/d5/f78 84962956 1 2024-07-26T11:04:45.647 INFO:tasks.workunit.client.0.smithi144.stdout:5/478: chown d1/d9/d44/f9d 0 1 2024-07-26T11:04:45.648 INFO:tasks.workunit.client.0.smithi144.stdout:5/479: link d1/d18/d24/f46 d1/d9/d5b/d4d/fa5 0 2024-07-26T11:04:45.648 INFO:tasks.workunit.client.0.smithi144.stdout:5/480: chown d1/d18/d24/d27/l4c 2600318 1 2024-07-26T11:04:45.660 INFO:tasks.workunit.client.0.smithi144.stdout:5/481: symlink d1/d9/d44/da1/la6 0 2024-07-26T11:04:45.660 INFO:tasks.workunit.client.0.smithi144.stdout:5/482: fsync d1/d18/d24/f76 0 2024-07-26T11:04:45.660 INFO:tasks.workunit.client.0.smithi144.stdout:5/483: chown d1/d5/c1c 52 1 2024-07-26T11:04:45.661 INFO:tasks.workunit.client.0.smithi144.stdout:6/539: write d8/db/f25 [2415469,124581] 0 2024-07-26T11:04:45.665 INFO:tasks.workunit.client.0.smithi144.stdout:6/540: creat d8/d15/d57/d5f/d6e/fb7 x:0 0 0 2024-07-26T11:04:45.665 INFO:tasks.workunit.client.0.smithi144.stdout:6/541: fdatasync d8/f16 0 2024-07-26T11:04:45.667 INFO:tasks.workunit.client.0.smithi144.stdout:2/502: write d0/d24/d48/d6d/d28/d2e/d5d/d6b/f99 [3355132,87945] 0 2024-07-26T11:04:45.669 INFO:tasks.workunit.client.0.smithi144.stdout:6/542: rename d8/d50/d92/fb0 to d8/db/fb8 0 2024-07-26T11:04:45.669 INFO:tasks.workunit.client.0.smithi144.stdout:6/543: chown d8/f53 1319492 1 2024-07-26T11:04:45.672 INFO:tasks.workunit.client.0.smithi144.stdout:2/503: unlink d0/d24/d48/d6d/d28/d2e/d77/f31 0 2024-07-26T11:04:45.672 INFO:tasks.workunit.client.0.smithi144.stdout:3/328: read d1/d28/d54/d26/f2d [2773363,2187] 0 2024-07-26T11:04:45.673 INFO:tasks.workunit.client.0.smithi144.stdout:1/546: dwrite d3/f2d [0,4194304] 0 2024-07-26T11:04:45.674 INFO:tasks.workunit.client.0.smithi144.stdout:6/544: rename d8/d15/l2f to d8/d4f/daa/lb9 0 2024-07-26T11:04:45.674 INFO:tasks.workunit.client.0.smithi144.stdout:6/545: write d8/db/da6/fa3 [794252,3257] 0 2024-07-26T11:04:45.676 INFO:tasks.workunit.client.0.smithi144.stdout:2/504: symlink d0/d24/d2b/d68/l9c 0 2024-07-26T11:04:45.678 INFO:tasks.workunit.client.0.smithi144.stdout:3/329: rmdir d1/d28/d54/d5b 0 2024-07-26T11:04:45.680 INFO:tasks.workunit.client.0.smithi144.stdout:6/546: rename d8/db/c3a to d8/d50/d92/cba 0 2024-07-26T11:04:45.680 INFO:tasks.workunit.client.0.smithi144.stdout:7/483: write d0/d6/d9/fc [3240421,118775] 0 2024-07-26T11:04:45.680 INFO:tasks.workunit.client.0.smithi144.stdout:7/484: write d0/d6/d9/d12/f7f [1043172,39802] 0 2024-07-26T11:04:45.680 INFO:tasks.workunit.client.0.smithi144.stdout:7/485: chown d0/d6/c66 149766153 1 2024-07-26T11:04:45.683 INFO:tasks.workunit.client.0.smithi144.stdout:2/505: unlink d0/d24/c83 0 2024-07-26T11:04:45.683 INFO:tasks.workunit.client.0.smithi144.stdout:2/506: chown d0/f14 58 1 2024-07-26T11:04:45.694 INFO:tasks.workunit.client.0.smithi144.stdout:3/330: creat d1/d28/d54/d26/d30/f5d x:0 0 0 2024-07-26T11:04:45.703 INFO:tasks.workunit.client.0.smithi144.stdout:5/484: read d1/d9/d5b/d4d/f16 [7663093,37144] 0 2024-07-26T11:04:45.703 INFO:tasks.workunit.client.0.smithi144.stdout:5/485: readlink d1/d5/l20 0 2024-07-26T11:04:45.711 INFO:tasks.workunit.client.0.smithi144.stdout:7/486: write d0/f2d [3513065,115763] 0 2024-07-26T11:04:45.713 INFO:tasks.workunit.client.0.smithi144.stdout:7/487: mkdir d0/d6/d85/d2f/d53/db6 0 2024-07-26T11:04:45.715 INFO:tasks.workunit.client.0.smithi144.stdout:2/507: write d0/d24/d48/d6d/f35 [1036348,4052] 0 2024-07-26T11:04:45.721 INFO:tasks.workunit.client.0.smithi144.stdout:3/331: write d1/f32 [2238340,90625] 0 2024-07-26T11:04:45.725 INFO:tasks.workunit.client.0.smithi144.stdout:0/428: dwrite d0/d25/d5e/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:04:45.725 INFO:tasks.workunit.client.0.smithi144.stdout:0/429: read - d0/d3e/f7d zero size 2024-07-26T11:04:45.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/486: dread d1/d9/d5b/d4d/f16 [4194304,4194304] 0 2024-07-26T11:04:45.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/487: getdents d1/d9/d5b/d86 0 2024-07-26T11:04:45.731 INFO:tasks.workunit.client.0.smithi144.stdout:5/488: dread - d1/d9/d44/f8a zero size 2024-07-26T11:04:45.736 INFO:tasks.workunit.client.0.smithi144.stdout:5/489: getdents d1/d9/d44/da1 0 2024-07-26T11:04:45.745 INFO:tasks.workunit.client.0.smithi144.stdout:6/547: dread d8/d15/d26/f28 [0,4194304] 0 2024-07-26T11:04:45.752 INFO:tasks.workunit.client.0.smithi144.stdout:1/547: dwrite d3/d6/da/d13/f26 [0,4194304] 0 2024-07-26T11:04:45.755 INFO:tasks.workunit.client.0.smithi144.stdout:1/548: unlink d3/d6/da/d13/d19/d3a/d69/d36/d37/c4e 0 2024-07-26T11:04:45.755 INFO:tasks.workunit.client.0.smithi144.stdout:1/549: stat d3/d6/da/d21/l2b 0 2024-07-26T11:04:45.767 INFO:tasks.workunit.client.0.smithi144.stdout:8/450: sync 2024-07-26T11:04:45.767 INFO:tasks.workunit.client.0.smithi144.stdout:7/488: dwrite d0/d6/d85/f42 [0,4194304] 0 2024-07-26T11:04:45.767 INFO:tasks.workunit.client.0.smithi144.stdout:4/502: sync 2024-07-26T11:04:45.767 INFO:tasks.workunit.client.0.smithi144.stdout:4/503: fsync d9/d26/d48/d50/d60/f88 0 2024-07-26T11:04:45.767 INFO:tasks.workunit.client.0.smithi144.stdout:8/451: readlink d1a/d32/d49/d73/d2c/l3d 0 2024-07-26T11:04:45.767 INFO:tasks.workunit.client.0.smithi144.stdout:9/435: sync 2024-07-26T11:04:45.768 INFO:tasks.workunit.client.0.smithi144.stdout:4/504: symlink d9/d26/d48/d50/d60/la8 0 2024-07-26T11:04:45.768 INFO:tasks.workunit.client.0.smithi144.stdout:9/436: symlink d9/de/d14/d79/l8e 0 2024-07-26T11:04:45.770 INFO:tasks.workunit.client.0.smithi144.stdout:9/437: truncate d9/de/d14/d3f/f53 591948 0 2024-07-26T11:04:45.770 INFO:tasks.workunit.client.0.smithi144.stdout:9/438: rmdir d9/de/d14/d3f 39 2024-07-26T11:04:45.770 INFO:tasks.workunit.client.0.smithi144.stdout:4/505: mknod d9/d26/ca9 0 2024-07-26T11:04:45.770 INFO:tasks.workunit.client.0.smithi144.stdout:4/506: fdatasync d9/f12 0 2024-07-26T11:04:45.770 INFO:tasks.workunit.client.0.smithi144.stdout:9/439: symlink d9/de/d14/d23/l8f 0 2024-07-26T11:04:45.771 INFO:tasks.workunit.client.0.smithi144.stdout:9/440: write d9/d3b/d5d/f8b [976968,116324] 0 2024-07-26T11:04:45.771 INFO:tasks.workunit.client.0.smithi144.stdout:4/507: creat d9/d26/d48/faa x:0 0 0 2024-07-26T11:04:45.772 INFO:tasks.workunit.client.0.smithi144.stdout:9/441: mknod d9/d3b/d5d/c90 0 2024-07-26T11:04:45.772 INFO:tasks.workunit.client.0.smithi144.stdout:4/508: creat d9/d26/fab x:0 0 0 2024-07-26T11:04:45.797 INFO:tasks.workunit.client.0.smithi144.stdout:2/508: dwrite d0/d24/d7b/f87 [0,4194304] 0 2024-07-26T11:04:45.802 INFO:tasks.workunit.client.0.smithi144.stdout:2/509: creat d0/d24/d2b/d68/f9d x:0 0 0 2024-07-26T11:04:45.802 INFO:tasks.workunit.client.0.smithi144.stdout:2/510: dread - d0/d24/d48/d6d/d28/f72 zero size 2024-07-26T11:04:45.805 INFO:tasks.workunit.client.0.smithi144.stdout:2/511: unlink d0/d6/d9/d44/l4f 0 2024-07-26T11:04:45.809 INFO:tasks.workunit.client.0.smithi144.stdout:6/548: dwrite d8/d4f/d52/d58/f97 [4194304,4194304] 0 2024-07-26T11:04:45.810 INFO:tasks.workunit.client.0.smithi144.stdout:6/549: fsync d8/d50/f79 0 2024-07-26T11:04:45.810 INFO:tasks.workunit.client.0.smithi144.stdout:7/489: dread d0/d6/d55/f6a [0,4194304] 0 2024-07-26T11:04:45.811 INFO:tasks.workunit.client.0.smithi144.stdout:0/430: dwrite d0/d25/d5e/f41 [0,4194304] 0 2024-07-26T11:04:45.819 INFO:tasks.workunit.client.0.smithi144.stdout:2/512: mknod d0/d24/d48/d6d/d28/d2e/d77/c9e 0 2024-07-26T11:04:45.819 INFO:tasks.workunit.client.0.smithi144.stdout:1/550: write d3/d6/da/d21/d5c/d62/f7c [2151490,113012] 0 2024-07-26T11:04:45.821 INFO:tasks.workunit.client.0.smithi144.stdout:5/490: dwrite d1/d5/f78 [4194304,4194304] 0 2024-07-26T11:04:45.821 INFO:tasks.workunit.client.0.smithi144.stdout:6/550: symlink d8/d42/lbb 0 2024-07-26T11:04:45.821 INFO:tasks.workunit.client.0.smithi144.stdout:6/551: read d8/d4f/d52/f5a [4306920,122801] 0 2024-07-26T11:04:45.821 INFO:tasks.workunit.client.0.smithi144.stdout:6/552: chown d8/l14 4019 1 2024-07-26T11:04:45.822 INFO:tasks.workunit.client.0.smithi144.stdout:0/431: mkdir d0/d25/d5e/d1e/d91 0 2024-07-26T11:04:45.823 INFO:tasks.workunit.client.0.smithi144.stdout:2/513: creat d0/d24/d2b/f9f x:0 0 0 2024-07-26T11:04:45.823 INFO:tasks.workunit.client.0.smithi144.stdout:2/514: readlink d0/d24/d48/d6d/d28/d27/l5e 0 2024-07-26T11:04:45.824 INFO:tasks.workunit.client.0.smithi144.stdout:6/553: link d8/d15/d26/l4e d8/d4f/d5c/lbc 0 2024-07-26T11:04:45.825 INFO:tasks.workunit.client.0.smithi144.stdout:9/442: write d9/de/d14/f22 [1127488,15962] 0 2024-07-26T11:04:45.826 INFO:tasks.workunit.client.0.smithi144.stdout:7/490: creat d0/d6/d85/d2f/d47/fb7 x:0 0 0 2024-07-26T11:04:45.826 INFO:tasks.workunit.client.0.smithi144.stdout:7/491: write d0/d6/f56 [4533723,102731] 0 2024-07-26T11:04:45.826 INFO:tasks.workunit.client.0.smithi144.stdout:7/492: dread - d0/d6/d85/d2f/d53/fa8 zero size 2024-07-26T11:04:45.827 INFO:tasks.workunit.client.0.smithi144.stdout:4/509: dwrite d9/f4e [0,4194304] 0 2024-07-26T11:04:45.829 INFO:tasks.workunit.client.0.smithi144.stdout:2/515: creat d0/d24/d48/d6d/fa0 x:0 0 0 2024-07-26T11:04:45.830 INFO:tasks.workunit.client.0.smithi144.stdout:9/443: creat d9/de/d14/d79/f91 x:0 0 0 2024-07-26T11:04:45.830 INFO:tasks.workunit.client.0.smithi144.stdout:6/554: creat d8/d4f/d52/fbd x:0 0 0 2024-07-26T11:04:45.839 INFO:tasks.workunit.client.0.smithi144.stdout:2/516: creat d0/d24/d48/fa1 x:0 0 0 2024-07-26T11:04:45.840 INFO:tasks.workunit.client.0.smithi144.stdout:7/493: rmdir d0/d71 39 2024-07-26T11:04:45.840 INFO:tasks.workunit.client.0.smithi144.stdout:7/494: read - d0/d6/d85/d67/f9f zero size 2024-07-26T11:04:45.840 INFO:tasks.workunit.client.0.smithi144.stdout:7/495: readlink d0/d6/d9/d12/l2c 0 2024-07-26T11:04:45.840 INFO:tasks.workunit.client.0.smithi144.stdout:7/496: write d0/d6/d85/d2f/d47/d5e/fa5 [531958,106159] 0 2024-07-26T11:04:45.844 INFO:tasks.workunit.client.0.smithi144.stdout:2/517: unlink d0/f14 0 2024-07-26T11:04:45.845 INFO:tasks.workunit.client.0.smithi144.stdout:2/518: creat d0/d6/d9/d44/fa2 x:0 0 0 2024-07-26T11:04:45.847 INFO:tasks.workunit.client.0.smithi144.stdout:6/555: rename d8/d4f to d8/db/dbe 0 2024-07-26T11:04:45.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/491: write d1/d18/d24/d27/f4f [2426601,117328] 0 2024-07-26T11:04:45.851 INFO:tasks.workunit.client.0.smithi144.stdout:2/519: read d0/d24/d2b/d68/d82/f7c [623396,98463] 0 2024-07-26T11:04:45.858 INFO:tasks.workunit.client.0.smithi144.stdout:1/551: dread d3/d6/da/d21/d5c/d62/f7c [4194304,4194304] 0 2024-07-26T11:04:45.858 INFO:tasks.workunit.client.0.smithi144.stdout:1/552: chown d3/c9 406435570 1 2024-07-26T11:04:45.858 INFO:tasks.workunit.client.0.smithi144.stdout:1/553: chown d3/d6/da/d13/d22/c59 0 1 2024-07-26T11:04:45.862 INFO:tasks.workunit.client.0.smithi144.stdout:1/554: mkdir d3/d6/d75/d88/da9 0 2024-07-26T11:04:45.862 INFO:tasks.workunit.client.0.smithi144.stdout:1/555: write d3/d53/f54 [498539,23398] 0 2024-07-26T11:04:45.862 INFO:tasks.workunit.client.0.smithi144.stdout:1/556: chown d3/d6/da/d21/c76 11092406 1 2024-07-26T11:04:45.863 INFO:tasks.workunit.client.0.smithi144.stdout:1/557: unlink d3/d6/da/d21/d67/l9e 0 2024-07-26T11:04:45.863 INFO:tasks.workunit.client.0.smithi144.stdout:1/558: chown d3/d6/f18 3579 1 2024-07-26T11:04:45.865 INFO:tasks.workunit.client.0.smithi144.stdout:1/559: rename d3/d6/da/l1d to d3/d6/da/laa 0 2024-07-26T11:04:45.868 INFO:tasks.workunit.client.0.smithi144.stdout:1/560: rmdir d3 39 2024-07-26T11:04:45.904 INFO:tasks.workunit.client.0.smithi144.stdout:0/432: dwrite d0/d25/d5e/d29/f2d [0,4194304] 0 2024-07-26T11:04:45.904 INFO:tasks.workunit.client.0.smithi144.stdout:5/492: dwrite d1/d18/d24/d35/f3d [0,4194304] 0 2024-07-26T11:04:45.904 INFO:tasks.workunit.client.0.smithi144.stdout:0/433: unlink d0/c34 0 2024-07-26T11:04:45.904 INFO:tasks.workunit.client.0.smithi144.stdout:0/434: stat d0/d25/d5e 0 2024-07-26T11:04:45.905 INFO:tasks.workunit.client.0.smithi144.stdout:5/493: link d1/d18/d24/d27/l9a d1/d9/d44/la7 0 2024-07-26T11:04:45.905 INFO:tasks.workunit.client.0.smithi144.stdout:0/435: mkdir d0/d25/d5e/d29/d92 0 2024-07-26T11:04:45.934 INFO:tasks.workunit.client.0.smithi144.stdout:9/444: dwrite d9/de/f51 [0,4194304] 0 2024-07-26T11:04:45.934 INFO:tasks.workunit.client.0.smithi144.stdout:9/445: rmdir d9/de/d14/d23/d39 39 2024-07-26T11:04:45.934 INFO:tasks.workunit.client.0.smithi144.stdout:2/520: dwrite d0/d24/d2b/f9f [0,4194304] 0 2024-07-26T11:04:45.934 INFO:tasks.workunit.client.0.smithi144.stdout:1/561: dwrite d3/d6/da/d21/f4d [0,4194304] 0 2024-07-26T11:04:45.939 INFO:tasks.workunit.client.0.smithi144.stdout:6/556: dwrite d8/d15/d26/f41 [0,4194304] 0 2024-07-26T11:04:45.941 INFO:tasks.workunit.client.0.smithi144.stdout:1/562: dread d3/d6/da/d21/d67/f71 [0,4194304] 0 2024-07-26T11:04:45.941 INFO:tasks.workunit.client.0.smithi144.stdout:2/521: mkdir d0/d24/d48/d63/da3 0 2024-07-26T11:04:45.943 INFO:tasks.workunit.client.0.smithi144.stdout:1/563: mknod d3/d6/da/d70/cab 0 2024-07-26T11:04:45.943 INFO:tasks.workunit.client.0.smithi144.stdout:9/446: write d9/de/d14/d23/d39/f66 [436723,88041] 0 2024-07-26T11:04:45.944 INFO:tasks.workunit.client.0.smithi144.stdout:2/522: rename d0/d24/d48/d6d/c64 to d0/d24/d48/d6d/d28/d2e/d77/ca4 0 2024-07-26T11:04:45.945 INFO:tasks.workunit.client.0.smithi144.stdout:2/523: dread - d0/d24/d48/d6d/fa0 zero size 2024-07-26T11:04:45.946 INFO:tasks.workunit.client.0.smithi144.stdout:1/564: mkdir d3/d6/d75/d88/da9/dac 0 2024-07-26T11:04:45.946 INFO:tasks.workunit.client.0.smithi144.stdout:1/565: write d3/d53/f87 [564693,123409] 0 2024-07-26T11:04:45.946 INFO:tasks.workunit.client.0.smithi144.stdout:1/566: dread - d3/d6/da/d21/d5c/d62/f77 zero size 2024-07-26T11:04:45.947 INFO:tasks.workunit.client.0.smithi144.stdout:9/447: creat d9/de/d14/d23/d39/d7f/f92 x:0 0 0 2024-07-26T11:04:45.948 INFO:tasks.workunit.client.0.smithi144.stdout:1/567: mknod d3/d6/da/cad 0 2024-07-26T11:04:45.971 INFO:tasks.workunit.client.0.smithi144.stdout:6/557: read d8/d15/f21 [7893806,21703] 0 2024-07-26T11:04:45.972 INFO:tasks.workunit.client.0.smithi144.stdout:0/436: dwrite d0/d3e/d45/f46 [0,4194304] 0 2024-07-26T11:04:45.973 INFO:tasks.workunit.client.0.smithi144.stdout:6/558: rmdir d8/db/dbe 39 2024-07-26T11:04:45.975 INFO:tasks.workunit.client.0.smithi144.stdout:6/559: mkdir d8/db/dbe/d52/d58/dbf 0 2024-07-26T11:04:45.977 INFO:tasks.workunit.client.0.smithi144.stdout:9/448: dread d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:45.978 INFO:tasks.workunit.client.0.smithi144.stdout:6/560: rename d8/d15/d26/l4e to d8/d15/d26/lc0 0 2024-07-26T11:04:45.979 INFO:tasks.workunit.client.0.smithi144.stdout:6/561: getdents d8/db/dbe/d52 0 2024-07-26T11:04:45.981 INFO:tasks.workunit.client.0.smithi144.stdout:2/524: dwrite d0/d24/d2b/d68/d82/f7c [0,4194304] 0 2024-07-26T11:04:45.981 INFO:tasks.workunit.client.0.smithi144.stdout:6/562: symlink d8/d50/d92/lc1 0 2024-07-26T11:04:45.981 INFO:tasks.workunit.client.0.smithi144.stdout:6/563: fdatasync d8/f45 0 2024-07-26T11:04:45.986 INFO:tasks.workunit.client.0.smithi144.stdout:2/525: symlink d0/d24/d7b/la5 0 2024-07-26T11:04:45.994 INFO:tasks.workunit.client.0.smithi144.stdout:2/526: mknod d0/ca6 0 2024-07-26T11:04:46.000 INFO:tasks.workunit.client.0.smithi144.stdout:1/568: dwrite d3/d6/f47 [4194304,4194304] 0 2024-07-26T11:04:46.001 INFO:tasks.workunit.client.0.smithi144.stdout:6/564: dread d8/db/dbe/d52/f5a [0,4194304] 0 2024-07-26T11:04:46.005 INFO:tasks.workunit.client.0.smithi144.stdout:6/565: creat d8/db/da6/fc2 x:0 0 0 2024-07-26T11:04:46.007 INFO:tasks.workunit.client.0.smithi144.stdout:6/566: creat d8/d15/d57/d5f/fc3 x:0 0 0 2024-07-26T11:04:46.015 INFO:tasks.workunit.client.0.smithi144.stdout:9/449: dwrite d9/de/ff [0,4194304] 0 2024-07-26T11:04:46.016 INFO:tasks.workunit.client.0.smithi144.stdout:9/450: read - d9/f29 zero size 2024-07-26T11:04:46.016 INFO:tasks.workunit.client.0.smithi144.stdout:9/451: chown d9/de/d14/d23/d39/c5f 1465231 1 2024-07-26T11:04:46.018 INFO:tasks.workunit.client.0.smithi144.stdout:1/569: dread d3/f9b [0,4194304] 0 2024-07-26T11:04:46.028 INFO:tasks.workunit.client.0.smithi144.stdout:3/332: sync 2024-07-26T11:04:46.028 INFO:tasks.workunit.client.0.smithi144.stdout:3/333: write d1/f20 [849259,19126] 0 2024-07-26T11:04:46.030 INFO:tasks.workunit.client.0.smithi144.stdout:1/570: creat d3/d3d/fae x:0 0 0 2024-07-26T11:04:46.032 INFO:tasks.workunit.client.0.smithi144.stdout:3/334: mkdir d1/d5/d5e 0 2024-07-26T11:04:46.034 INFO:tasks.workunit.client.0.smithi144.stdout:1/571: rename d3/f4f to d3/d6/da/d13/d22/d5d/faf 0 2024-07-26T11:04:46.034 INFO:tasks.workunit.client.0.smithi144.stdout:1/572: chown d3/d6/da/d5b/f8d 28583 1 2024-07-26T11:04:46.036 INFO:tasks.workunit.client.0.smithi144.stdout:1/573: mknod d3/d6/d9c/cb0 0 2024-07-26T11:04:46.036 INFO:tasks.workunit.client.0.smithi144.stdout:1/574: fdatasync d3/d6/da/d13/d19/d3a/d69/f5a 0 2024-07-26T11:04:46.039 INFO:tasks.workunit.client.0.smithi144.stdout:1/575: rename l2 to d3/lb1 0 2024-07-26T11:04:46.039 INFO:tasks.workunit.client.0.smithi144.stdout:1/576: chown d3/d6/da/d13/d19/d3a/d3e/f50 30 1 2024-07-26T11:04:46.041 INFO:tasks.workunit.client.0.smithi144.stdout:1/577: getdents d3/d6/d75 0 2024-07-26T11:04:46.043 INFO:tasks.workunit.client.0.smithi144.stdout:1/578: rename d3/d3d/d58/f7d to d3/fb2 0 2024-07-26T11:04:46.044 INFO:tasks.workunit.client.0.smithi144.stdout:1/579: symlink d3/d6/da/d70/lb3 0 2024-07-26T11:04:46.046 INFO:tasks.workunit.client.0.smithi144.stdout:6/567: dwrite d8/d50/d98/fb4 [8388608,4194304] 0 2024-07-26T11:04:46.047 INFO:tasks.workunit.client.0.smithi144.stdout:6/568: truncate d8/f86 458053 0 2024-07-26T11:04:46.053 INFO:tasks.workunit.client.0.smithi144.stdout:3/335: dread d1/d5/f15 [0,4194304] 0 2024-07-26T11:04:46.055 INFO:tasks.workunit.client.0.smithi144.stdout:3/336: creat d1/d28/d54/d3d/d42/d44/f5f x:0 0 0 2024-07-26T11:04:46.055 INFO:tasks.workunit.client.0.smithi144.stdout:3/337: chown d1/d5/f24 37670 1 2024-07-26T11:04:46.056 INFO:tasks.workunit.client.0.smithi144.stdout:3/338: readlink d1/d5/l21 0 2024-07-26T11:04:46.058 INFO:tasks.workunit.client.0.smithi144.stdout:3/339: rename d1/d28/d54/d26/d37/d46 to d1/d28/d54/d3d/d42/d44/d60 0 2024-07-26T11:04:46.059 INFO:tasks.workunit.client.0.smithi144.stdout:3/340: mkdir d1/d28/d54/d61 0 2024-07-26T11:04:46.069 INFO:tasks.workunit.client.0.smithi144.stdout:9/452: dwrite d9/de/f6e [0,4194304] 0 2024-07-26T11:04:46.071 INFO:tasks.workunit.client.0.smithi144.stdout:9/453: getdents d9/de/d14/d23/d39 0 2024-07-26T11:04:46.079 INFO:tasks.workunit.client.0.smithi144.stdout:9/454: read d9/de/d14/d23/d39/f66 [814827,14811] 0 2024-07-26T11:04:46.079 INFO:tasks.workunit.client.0.smithi144.stdout:9/455: chown d9/de/d14/d23/c55 29010548 1 2024-07-26T11:04:46.090 INFO:tasks.workunit.client.0.smithi144.stdout:6/569: dwrite d8/d42/fa9 [0,4194304] 0 2024-07-26T11:04:46.093 INFO:tasks.workunit.client.0.smithi144.stdout:6/570: creat d8/d50/d98/fc4 x:0 0 0 2024-07-26T11:04:46.093 INFO:tasks.workunit.client.0.smithi144.stdout:6/571: stat d8/d15/d57/d5f/f84 0 2024-07-26T11:04:46.103 INFO:tasks.workunit.client.0.smithi144.stdout:1/580: dwrite d3/d6/da/d13/d22/f7f [0,4194304] 0 2024-07-26T11:04:46.103 INFO:tasks.workunit.client.0.smithi144.stdout:1/581: chown d3/d6/da/d5b 0 1 2024-07-26T11:04:46.103 INFO:tasks.workunit.client.0.smithi144.stdout:6/572: dread d8/db/da6/fa3 [0,4194304] 0 2024-07-26T11:04:46.105 INFO:tasks.workunit.client.0.smithi144.stdout:1/582: creat d3/d6/da/d21/d5c/fb4 x:0 0 0 2024-07-26T11:04:46.106 INFO:tasks.workunit.client.0.smithi144.stdout:6/573: creat d8/d50/d98/fc5 x:0 0 0 2024-07-26T11:04:46.107 INFO:tasks.workunit.client.0.smithi144.stdout:1/583: creat d3/d6/da/d21/d5c/d62/d68/fb5 x:0 0 0 2024-07-26T11:04:46.109 INFO:tasks.workunit.client.0.smithi144.stdout:1/584: mkdir d3/d6/da/d13/d19/d3a/d69/d36/db6 0 2024-07-26T11:04:46.110 INFO:tasks.workunit.client.0.smithi144.stdout:1/585: write d3/d6/da/d21/d5c/d62/d6e/f8f [2335974,3401] 0 2024-07-26T11:04:46.111 INFO:tasks.workunit.client.0.smithi144.stdout:1/586: rename d3/d6/da/d21/d5c/d62/d6e to d3/d6/da/d13/d19/d3a/d69/db7 0 2024-07-26T11:04:46.134 INFO:tasks.workunit.client.0.smithi144.stdout:9/456: dwrite d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:46.138 INFO:tasks.workunit.client.0.smithi144.stdout:9/457: creat d9/de/d4e/f93 x:0 0 0 2024-07-26T11:04:46.138 INFO:tasks.workunit.client.0.smithi144.stdout:9/458: mknod d9/de/d14/d79/c94 0 2024-07-26T11:04:46.138 INFO:tasks.workunit.client.0.smithi144.stdout:9/459: symlink d9/de/d14/l95 0 2024-07-26T11:04:46.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/574: dwrite d8/d50/f85 [0,4194304] 0 2024-07-26T11:04:46.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/575: getdents d8/db/dbe/d5c/da7 0 2024-07-26T11:04:46.149 INFO:tasks.workunit.client.0.smithi144.stdout:9/460: dread d9/de/f46 [0,4194304] 0 2024-07-26T11:04:46.150 INFO:tasks.workunit.client.0.smithi144.stdout:9/461: truncate d9/d3b/d5d/f8b 522190 0 2024-07-26T11:04:46.175 INFO:tasks.workunit.client.0.smithi144.stdout:9/462: read d9/de/d14/d23/f47 [1899985,39676] 0 2024-07-26T11:04:46.176 INFO:tasks.workunit.client.0.smithi144.stdout:9/463: rename d9/de/l12 to d9/de/d4e/l96 0 2024-07-26T11:04:46.177 INFO:tasks.workunit.client.0.smithi144.stdout:6/576: dread d8/d50/f85 [0,4194304] 0 2024-07-26T11:04:46.177 INFO:tasks.workunit.client.0.smithi144.stdout:9/464: mknod d9/de/d14/d23/d39/d7f/c97 0 2024-07-26T11:04:46.178 INFO:tasks.workunit.client.0.smithi144.stdout:6/577: symlink d8/d15/lc6 0 2024-07-26T11:04:46.179 INFO:tasks.workunit.client.0.smithi144.stdout:6/578: symlink d8/db/dbe/d52/d58/dbf/lc7 0 2024-07-26T11:04:46.180 INFO:tasks.workunit.client.0.smithi144.stdout:6/579: getdents d8/db/dbe/d74 0 2024-07-26T11:04:46.181 INFO:tasks.workunit.client.0.smithi144.stdout:6/580: symlink d8/d42/lc8 0 2024-07-26T11:04:46.198 INFO:tasks.workunit.client.0.smithi144.stdout:8/452: sync 2024-07-26T11:04:46.199 INFO:tasks.workunit.client.0.smithi144.stdout:8/453: mkdir d1a/d32/d8c/d9b 0 2024-07-26T11:04:46.201 INFO:tasks.workunit.client.0.smithi144.stdout:8/454: creat d1a/d32/d6c/d82/f9c x:0 0 0 2024-07-26T11:04:46.201 INFO:tasks.workunit.client.0.smithi144.stdout:8/455: getdents d1a/d32/d49/d73/d7b/d43/d4f 0 2024-07-26T11:04:46.201 INFO:tasks.workunit.client.0.smithi144.stdout:8/456: rmdir d1a/d32/d49/d73/d7b/d4e 39 2024-07-26T11:04:46.203 INFO:tasks.workunit.client.0.smithi144.stdout:8/457: rename d1a/d32/d49/d73/d7b/d43/d4c/d55 to d1a/d32/d49/d73/d2c/d9d 0 2024-07-26T11:04:46.204 INFO:tasks.workunit.client.0.smithi144.stdout:8/458: stat d1a/d32/d49/d73/d7b/d43/d4c/f77 0 2024-07-26T11:04:46.216 INFO:tasks.workunit.client.0.smithi144.stdout:9/465: read d9/de/ff [1082881,82119] 0 2024-07-26T11:04:46.216 INFO:tasks.workunit.client.0.smithi144.stdout:9/466: stat d9/de/d14/d79/c86 0 2024-07-26T11:04:46.216 INFO:tasks.workunit.client.0.smithi144.stdout:9/467: truncate d9/d3b/f78 1240233 0 2024-07-26T11:04:46.216 INFO:tasks.workunit.client.0.smithi144.stdout:9/468: write d9/de/d14/d3f/d5e/f62 [1170367,48465] 0 2024-07-26T11:04:46.216 INFO:tasks.workunit.client.0.smithi144.stdout:9/469: write f4 [2799137,78187] 0 2024-07-26T11:04:46.218 INFO:tasks.workunit.client.0.smithi144.stdout:9/470: mkdir d9/de/d14/d23/d98 0 2024-07-26T11:04:46.219 INFO:tasks.workunit.client.0.smithi144.stdout:9/471: creat d9/de/d14/d3f/f99 x:0 0 0 2024-07-26T11:04:46.219 INFO:tasks.workunit.client.0.smithi144.stdout:9/472: chown d9/d3b/d5d/c68 781107 1 2024-07-26T11:04:46.221 INFO:tasks.workunit.client.0.smithi144.stdout:9/473: symlink d9/de/d4e/d6a/l9a 0 2024-07-26T11:04:46.221 INFO:tasks.workunit.client.0.smithi144.stdout:9/474: chown d9/d3b/f56 1442488 1 2024-07-26T11:04:46.221 INFO:tasks.workunit.client.0.smithi144.stdout:9/475: chown d9/de/d4e/l96 2 1 2024-07-26T11:04:46.222 INFO:tasks.workunit.client.0.smithi144.stdout:9/476: mkdir d9/d3b/d76/d9b 0 2024-07-26T11:04:46.222 INFO:tasks.workunit.client.0.smithi144.stdout:9/477: chown d9/de/d14/d23/f50 8661920 1 2024-07-26T11:04:46.223 INFO:tasks.workunit.client.0.smithi144.stdout:9/478: read d9/d3b/f64 [3406486,53922] 0 2024-07-26T11:04:46.225 INFO:tasks.workunit.client.0.smithi144.stdout:9/479: write d9/fc [2002168,112339] 0 2024-07-26T11:04:46.226 INFO:tasks.workunit.client.0.smithi144.stdout:9/480: stat d9/de/d4e/d6a/l85 0 2024-07-26T11:04:46.226 INFO:tasks.workunit.client.0.smithi144.stdout:9/481: truncate d9/de/d14/d3f/d5e/f62 1624576 0 2024-07-26T11:04:46.236 INFO:tasks.workunit.client.0.smithi144.stdout:6/581: dwrite d8/db/fa1 [0,4194304] 0 2024-07-26T11:04:46.237 INFO:tasks.workunit.client.0.smithi144.stdout:9/482: dread d9/de/d14/d3f/f53 [0,4194304] 0 2024-07-26T11:04:46.237 INFO:tasks.workunit.client.0.smithi144.stdout:9/483: write d9/d3b/f71 [1359911,100827] 0 2024-07-26T11:04:46.238 INFO:tasks.workunit.client.0.smithi144.stdout:6/582: creat d8/db/dbe/fc9 x:0 0 0 2024-07-26T11:04:46.240 INFO:tasks.workunit.client.0.smithi144.stdout:9/484: link d9/de/d14/d23/d39/d7f/c97 d9/de/d4e/c9c 0 2024-07-26T11:04:46.240 INFO:tasks.workunit.client.0.smithi144.stdout:9/485: fsync d9/de/d14/d79/f82 0 2024-07-26T11:04:46.240 INFO:tasks.workunit.client.0.smithi144.stdout:9/486: dread - d9/de/d14/d79/f82 zero size 2024-07-26T11:04:46.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/583: mkdir d8/dca 0 2024-07-26T11:04:46.258 INFO:tasks.workunit.client.0.smithi144.stdout:9/487: dread d9/d3b/d5d/f8b [0,4194304] 0 2024-07-26T11:04:46.259 INFO:tasks.workunit.client.0.smithi144.stdout:9/488: write d9/de/f15 [686560,41516] 0 2024-07-26T11:04:46.259 INFO:tasks.workunit.client.0.smithi144.stdout:9/489: truncate d9/de/d14/d3f/f61 274530 0 2024-07-26T11:04:46.261 INFO:tasks.workunit.client.0.smithi144.stdout:8/459: dwrite d1a/d32/d49/d73/d2c/f88 [0,4194304] 0 2024-07-26T11:04:46.261 INFO:tasks.workunit.client.0.smithi144.stdout:8/460: truncate d1a/d1c/f22 5321223 0 2024-07-26T11:04:46.261 INFO:tasks.workunit.client.0.smithi144.stdout:9/490: rmdir d9/de/d14/d79 39 2024-07-26T11:04:46.261 INFO:tasks.workunit.client.0.smithi144.stdout:9/491: write d9/fc [267565,25174] 0 2024-07-26T11:04:46.262 INFO:tasks.workunit.client.0.smithi144.stdout:9/492: stat d9/de/d14/d79/c94 0 2024-07-26T11:04:46.263 INFO:tasks.workunit.client.0.smithi144.stdout:8/461: creat d1a/d32/d6c/d82/f9e x:0 0 0 2024-07-26T11:04:46.265 INFO:tasks.workunit.client.0.smithi144.stdout:8/462: creat d1a/d32/d49/d73/d7b/d43/f9f x:0 0 0 2024-07-26T11:04:46.271 INFO:tasks.workunit.client.0.smithi144.stdout:6/584: dread d8/db/f2c [0,4194304] 0 2024-07-26T11:04:46.315 INFO:tasks.workunit.client.0.smithi144.stdout:6/585: dwrite d8/db/dbe/fc9 [0,4194304] 0 2024-07-26T11:04:46.318 INFO:tasks.workunit.client.0.smithi144.stdout:6/586: creat d8/db/dbe/d52/fcb x:0 0 0 2024-07-26T11:04:46.323 INFO:tasks.workunit.client.0.smithi144.stdout:6/587: mknod d8/d15/ccc 0 2024-07-26T11:04:46.323 INFO:tasks.workunit.client.0.smithi144.stdout:6/588: write d8/db/dbe/d52/f5a [4378234,100162] 0 2024-07-26T11:04:46.324 INFO:tasks.workunit.client.0.smithi144.stdout:6/589: write d8/db/fa1 [4254004,104076] 0 2024-07-26T11:04:46.332 INFO:tasks.workunit.client.0.smithi144.stdout:9/493: dwrite d9/de/d14/d3f/f99 [0,4194304] 0 2024-07-26T11:04:46.334 INFO:tasks.workunit.client.0.smithi144.stdout:9/494: creat d9/de/d4e/f9d x:0 0 0 2024-07-26T11:04:46.336 INFO:tasks.workunit.client.0.smithi144.stdout:9/495: rename d9/de/d14/d23/l8f to d9/de/d4e/d6a/l9e 0 2024-07-26T11:04:46.345 INFO:tasks.workunit.client.0.smithi144.stdout:9/496: write d9/de/f15 [242295,104487] 0 2024-07-26T11:04:46.346 INFO:tasks.workunit.client.0.smithi144.stdout:9/497: stat d9/l77 0 2024-07-26T11:04:46.349 INFO:tasks.workunit.client.0.smithi144.stdout:4/510: sync 2024-07-26T11:04:46.350 INFO:tasks.workunit.client.0.smithi144.stdout:4/511: rename d9/d26/d48/d98/c9b to d9/da/cac 0 2024-07-26T11:04:46.351 INFO:tasks.workunit.client.0.smithi144.stdout:4/512: fsync d9/d26/f62 0 2024-07-26T11:04:46.351 INFO:tasks.workunit.client.0.smithi144.stdout:4/513: write d9/f51 [1504984,64859] 0 2024-07-26T11:04:46.351 INFO:tasks.workunit.client.0.smithi144.stdout:4/514: write d9/da/f52 [581886,110222] 0 2024-07-26T11:04:46.353 INFO:tasks.workunit.client.0.smithi144.stdout:4/515: creat d9/da/fad x:0 0 0 2024-07-26T11:04:46.355 INFO:tasks.workunit.client.0.smithi144.stdout:9/498: dread d9/d3b/d5d/f8b [0,4194304] 0 2024-07-26T11:04:46.356 INFO:tasks.workunit.client.0.smithi144.stdout:9/499: creat d9/de/d14/d23/d39/f9f x:0 0 0 2024-07-26T11:04:46.360 INFO:tasks.workunit.client.0.smithi144.stdout:9/500: mknod d9/de/d4e/d6a/ca0 0 2024-07-26T11:04:46.371 INFO:tasks.workunit.client.0.smithi144.stdout:6/590: dwrite d8/db/dbe/daa/fac [0,4194304] 0 2024-07-26T11:04:46.412 INFO:tasks.workunit.client.0.smithi144.stdout:4/516: dread d9/f22 [0,4194304] 0 2024-07-26T11:04:46.443 INFO:tasks.workunit.client.0.smithi144.stdout:9/501: dwrite d9/d3b/d5d/f6b [0,4194304] 0 2024-07-26T11:04:46.443 INFO:tasks.workunit.client.0.smithi144.stdout:9/502: fdatasync d9/d3b/f78 0 2024-07-26T11:04:46.443 INFO:tasks.workunit.client.0.smithi144.stdout:9/503: write d9/d3b/f7c [221100,16649] 0 2024-07-26T11:04:46.444 INFO:tasks.workunit.client.0.smithi144.stdout:9/504: rmdir d9/de/d14 39 2024-07-26T11:04:46.444 INFO:tasks.workunit.client.0.smithi144.stdout:9/505: truncate d9/d3b/f64 4686769 0 2024-07-26T11:04:46.478 INFO:tasks.workunit.client.0.smithi144.stdout:6/591: dwrite d8/d15/f21 [0,4194304] 0 2024-07-26T11:04:46.478 INFO:tasks.workunit.client.0.smithi144.stdout:6/592: chown d8/d15/d57 513514 1 2024-07-26T11:04:46.478 INFO:tasks.workunit.client.0.smithi144.stdout:6/593: truncate d8/db/dbe/f72 10678 0 2024-07-26T11:04:46.478 INFO:tasks.workunit.client.0.smithi144.stdout:4/517: dread d9/d26/d48/d50/f59 [0,4194304] 0 2024-07-26T11:04:46.482 INFO:tasks.workunit.client.0.smithi144.stdout:6/594: creat d8/d42/fcd x:0 0 0 2024-07-26T11:04:46.483 INFO:tasks.workunit.client.0.smithi144.stdout:4/518: symlink d9/lae 0 2024-07-26T11:04:46.485 INFO:tasks.workunit.client.0.smithi144.stdout:6/595: rmdir d8/d50/d92 39 2024-07-26T11:04:46.488 INFO:tasks.workunit.client.0.smithi144.stdout:6/596: getdents d8/d15/d57/d5f 0 2024-07-26T11:04:46.493 INFO:tasks.workunit.client.0.smithi144.stdout:6/597: creat d8/d15/d57/d5f/fce x:0 0 0 2024-07-26T11:04:46.497 INFO:tasks.workunit.client.0.smithi144.stdout:4/519: getdents d9 0 2024-07-26T11:04:46.497 INFO:tasks.workunit.client.0.smithi144.stdout:5/494: sync 2024-07-26T11:04:46.497 INFO:tasks.workunit.client.0.smithi144.stdout:2/527: sync 2024-07-26T11:04:46.499 INFO:tasks.workunit.client.0.smithi144.stdout:8/463: sync 2024-07-26T11:04:46.499 INFO:tasks.workunit.client.0.smithi144.stdout:8/464: fdatasync d1a/d1c/f70 0 2024-07-26T11:04:46.499 INFO:tasks.workunit.client.0.smithi144.stdout:8/465: read d1a/f21 [673695,68928] 0 2024-07-26T11:04:46.500 INFO:tasks.workunit.client.0.smithi144.stdout:7/497: sync 2024-07-26T11:04:46.501 INFO:tasks.workunit.client.0.smithi144.stdout:1/587: sync 2024-07-26T11:04:46.501 INFO:tasks.workunit.client.0.smithi144.stdout:0/437: sync 2024-07-26T11:04:46.501 INFO:tasks.workunit.client.0.smithi144.stdout:3/341: sync 2024-07-26T11:04:46.502 INFO:tasks.workunit.client.0.smithi144.stdout:5/495: mkdir d1/da8 0 2024-07-26T11:04:46.502 INFO:tasks.workunit.client.0.smithi144.stdout:0/438: fdatasync d0/f7 0 2024-07-26T11:04:46.502 INFO:tasks.workunit.client.0.smithi144.stdout:0/439: fdatasync d0/d3e/d5f/f8d 0 2024-07-26T11:04:46.502 INFO:tasks.workunit.client.0.smithi144.stdout:5/496: chown d1/f96 323030010 1 2024-07-26T11:04:46.502 INFO:tasks.workunit.client.0.smithi144.stdout:5/497: dread - d1/f96 zero size 2024-07-26T11:04:46.508 INFO:tasks.workunit.client.0.smithi144.stdout:8/466: mkdir d1a/d32/d66/da0 0 2024-07-26T11:04:46.508 INFO:tasks.workunit.client.0.smithi144.stdout:8/467: truncate f11 1204590 0 2024-07-26T11:04:46.508 INFO:tasks.workunit.client.0.smithi144.stdout:8/468: chown d1a/d32/d49/d73/d7b/f37 15423 1 2024-07-26T11:04:46.510 INFO:tasks.workunit.client.0.smithi144.stdout:1/588: mkdir d3/d6/db8 0 2024-07-26T11:04:46.512 INFO:tasks.workunit.client.0.smithi144.stdout:3/342: mknod d1/d28/d54/d26/d37/c62 0 2024-07-26T11:04:46.513 INFO:tasks.workunit.client.0.smithi144.stdout:5/498: truncate d1/d9/d44/f62 4182994 0 2024-07-26T11:04:46.517 INFO:tasks.workunit.client.0.smithi144.stdout:9/506: dwrite d9/de/d14/d23/d39/f9f [0,4194304] 0 2024-07-26T11:04:46.517 INFO:tasks.workunit.client.0.smithi144.stdout:8/469: symlink d1a/d32/d49/d73/d7b/d43/d4f/la1 0 2024-07-26T11:04:46.517 INFO:tasks.workunit.client.0.smithi144.stdout:8/470: truncate d1a/d32/d49/d73/d2c/f8a 856108 0 2024-07-26T11:04:46.517 INFO:tasks.workunit.client.0.smithi144.stdout:8/471: chown d1a/d32/d49/d73/d7b/d43/d4f 62113569 1 2024-07-26T11:04:46.517 INFO:tasks.workunit.client.0.smithi144.stdout:1/589: rename d3/d6/da/d13/d19/d3a/la5 to d3/d6/da/d13/d22/lb9 0 2024-07-26T11:04:46.518 INFO:tasks.workunit.client.0.smithi144.stdout:7/498: link d0/d6/d9/d45/d57/f7e d0/d6/d85/d2f/d68/fb8 0 2024-07-26T11:04:46.518 INFO:tasks.workunit.client.0.smithi144.stdout:7/499: write d0/f2 [2770668,39614] 0 2024-07-26T11:04:46.518 INFO:tasks.workunit.client.0.smithi144.stdout:3/343: creat d1/d28/d54/d61/f63 x:0 0 0 2024-07-26T11:04:46.519 INFO:tasks.workunit.client.0.smithi144.stdout:3/344: write d1/d28/d54/d26/d37/f41 [646889,76278] 0 2024-07-26T11:04:46.523 INFO:tasks.workunit.client.0.smithi144.stdout:5/499: symlink d1/d18/d24/d35/la9 0 2024-07-26T11:04:46.523 INFO:tasks.workunit.client.0.smithi144.stdout:9/507: mknod d9/d3b/ca1 0 2024-07-26T11:04:46.523 INFO:tasks.workunit.client.0.smithi144.stdout:8/472: mkdir d1a/d32/d49/d73/d7b/d43/d4f/da2 0 2024-07-26T11:04:46.523 INFO:tasks.workunit.client.0.smithi144.stdout:5/500: read d1/d18/d24/d27/d71/f80 [2898242,98120] 0 2024-07-26T11:04:46.523 INFO:tasks.workunit.client.0.smithi144.stdout:7/500: stat d0/c29 0 2024-07-26T11:04:46.523 INFO:tasks.workunit.client.0.smithi144.stdout:1/590: creat d3/d6/da/d13/d19/d3a/d69/fba x:0 0 0 2024-07-26T11:04:46.524 INFO:tasks.workunit.client.0.smithi144.stdout:9/508: mknod d9/d3b/d81/ca2 0 2024-07-26T11:04:46.525 INFO:tasks.workunit.client.0.smithi144.stdout:9/509: write d9/d3b/f78 [1490486,121714] 0 2024-07-26T11:04:46.525 INFO:tasks.workunit.client.0.smithi144.stdout:9/510: readlink d9/de/d14/l72 0 2024-07-26T11:04:46.525 INFO:tasks.workunit.client.0.smithi144.stdout:8/473: mkdir d1a/d32/d49/d73/d2c/d38/d58/da3 0 2024-07-26T11:04:46.527 INFO:tasks.workunit.client.0.smithi144.stdout:5/501: mkdir d1/d9/d44/d45/daa 0 2024-07-26T11:04:46.527 INFO:tasks.workunit.client.0.smithi144.stdout:4/520: dread d9/d26/d48/d50/f59 [0,4194304] 0 2024-07-26T11:04:46.534 INFO:tasks.workunit.client.0.smithi144.stdout:0/440: write d0/d25/d5e/d1e/f36 [6053754,14351] 0 2024-07-26T11:04:46.535 INFO:tasks.workunit.client.0.smithi144.stdout:1/591: creat d3/d6/da/d13/d19/d3a/d69/d36/db6/fbb x:0 0 0 2024-07-26T11:04:46.535 INFO:tasks.workunit.client.0.smithi144.stdout:9/511: creat d9/de/d14/d23/fa3 x:0 0 0 2024-07-26T11:04:46.535 INFO:tasks.workunit.client.0.smithi144.stdout:2/528: dread d0/d24/d2b/f59 [4194304,4194304] 0 2024-07-26T11:04:46.536 INFO:tasks.workunit.client.0.smithi144.stdout:5/502: symlink d1/d9/d44/d45/daa/lab 0 2024-07-26T11:04:46.536 INFO:tasks.workunit.client.0.smithi144.stdout:4/521: creat d9/d26/d3c/faf x:0 0 0 2024-07-26T11:04:46.538 INFO:tasks.workunit.client.0.smithi144.stdout:7/501: rename d0/d6/d85/d2f/f54 to d0/d6/d85/d2f/d68/fb9 0 2024-07-26T11:04:46.538 INFO:tasks.workunit.client.0.smithi144.stdout:7/502: write d0/f11 [145219,48505] 0 2024-07-26T11:04:46.539 INFO:tasks.workunit.client.0.smithi144.stdout:0/441: mknod d0/d25/d5e/d1e/d30/d67/c93 0 2024-07-26T11:04:46.540 INFO:tasks.workunit.client.0.smithi144.stdout:1/592: symlink d3/d6/da/d13/d19/d3a/d3e/lbc 0 2024-07-26T11:04:46.543 INFO:tasks.workunit.client.0.smithi144.stdout:9/512: creat d9/de/d14/d23/d39/d7f/fa4 x:0 0 0 2024-07-26T11:04:46.544 INFO:tasks.workunit.client.0.smithi144.stdout:9/513: readlink d9/de/d14/l1b 0 2024-07-26T11:04:46.544 INFO:tasks.workunit.client.0.smithi144.stdout:2/529: getdents d0/d24/d48/d6d/d28/d2e/d77 0 2024-07-26T11:04:46.550 INFO:tasks.workunit.client.0.smithi144.stdout:8/474: rename d1a/d32/d6c/f7f to d1a/d32/d6c/d82/fa4 0 2024-07-26T11:04:46.550 INFO:tasks.workunit.client.0.smithi144.stdout:7/503: creat d0/d6/d85/d2f/d47/d5e/fba x:0 0 0 2024-07-26T11:04:46.554 INFO:tasks.workunit.client.0.smithi144.stdout:9/514: mknod d9/d3b/d5d/ca5 0 2024-07-26T11:04:46.554 INFO:tasks.workunit.client.0.smithi144.stdout:9/515: fsync d9/d3b/d5d/f8b 0 2024-07-26T11:04:46.554 INFO:tasks.workunit.client.0.smithi144.stdout:2/530: link d0/d24/d2b/d68/d82/f5a d0/d24/d48/d63/fa7 0 2024-07-26T11:04:46.554 INFO:tasks.workunit.client.0.smithi144.stdout:8/475: symlink d1a/d32/d49/d73/d7b/d43/d4f/la5 0 2024-07-26T11:04:46.554 INFO:tasks.workunit.client.0.smithi144.stdout:4/522: rename d9/d26/d3c/d6f/d8c/f95 to d9/da/fb0 0 2024-07-26T11:04:46.554 INFO:tasks.workunit.client.0.smithi144.stdout:8/476: write d1a/f20 [31270,91327] 0 2024-07-26T11:04:46.556 INFO:tasks.workunit.client.0.smithi144.stdout:2/531: truncate d0/d24/d2b/f9f 2794121 0 2024-07-26T11:04:46.558 INFO:tasks.workunit.client.0.smithi144.stdout:4/523: creat d9/d26/d48/d98/fb1 x:0 0 0 2024-07-26T11:04:46.559 INFO:tasks.workunit.client.0.smithi144.stdout:8/477: rename d1a/d32/d49/d73/d7b/d43/c4b to d1a/d32/d49/d73/d2c/d9d/ca6 0 2024-07-26T11:04:46.561 INFO:tasks.workunit.client.0.smithi144.stdout:2/532: unlink d0/d24/d48/d6d/d28/d2e/d77/l42 0 2024-07-26T11:04:46.562 INFO:tasks.workunit.client.0.smithi144.stdout:4/524: mknod d9/cb2 0 2024-07-26T11:04:46.564 INFO:tasks.workunit.client.0.smithi144.stdout:2/533: mkdir d0/d24/d48/d6d/d28/da8 0 2024-07-26T11:04:46.564 INFO:tasks.workunit.client.0.smithi144.stdout:0/442: dread d0/d3e/d45/f4a [0,4194304] 0 2024-07-26T11:04:46.564 INFO:tasks.workunit.client.0.smithi144.stdout:4/525: symlink d9/d26/d3c/d6f/d8c/lb3 0 2024-07-26T11:04:46.565 INFO:tasks.workunit.client.0.smithi144.stdout:4/526: chown d9/da/f52 528105074 1 2024-07-26T11:04:46.565 INFO:tasks.workunit.client.0.smithi144.stdout:4/527: write d9/d57/fa4 [2742237,124653] 0 2024-07-26T11:04:46.569 INFO:tasks.workunit.client.0.smithi144.stdout:2/534: mknod d0/d24/d48/d6d/d28/d2e/ca9 0 2024-07-26T11:04:46.570 INFO:tasks.workunit.client.0.smithi144.stdout:2/535: truncate d0/d6/f1d 744559 0 2024-07-26T11:04:46.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/593: dread d3/d6/da/d13/d19/d3a/d69/f28 [0,4194304] 0 2024-07-26T11:04:46.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/594: readlink d3/d6/l61 0 2024-07-26T11:04:46.576 INFO:tasks.workunit.client.0.smithi144.stdout:3/345: dwrite d1/d28/d54/d3d/f53 [0,4194304] 0 2024-07-26T11:04:46.576 INFO:tasks.workunit.client.0.smithi144.stdout:3/346: fsync d1/d28/d54/d3d/f53 0 2024-07-26T11:04:46.579 INFO:tasks.workunit.client.0.smithi144.stdout:9/516: dread d9/de/d14/d3f/f99 [0,4194304] 0 2024-07-26T11:04:46.579 INFO:tasks.workunit.client.0.smithi144.stdout:9/517: write d9/de/d14/d23/d39/d7f/fa4 [545538,123507] 0 2024-07-26T11:04:46.588 INFO:tasks.workunit.client.0.smithi144.stdout:4/528: dread d9/f51 [0,4194304] 0 2024-07-26T11:04:46.589 INFO:tasks.workunit.client.0.smithi144.stdout:1/595: link d3/d6/da/d21/d5c/d62/f77 d3/d6/da/d70/fbd 0 2024-07-26T11:04:46.589 INFO:tasks.workunit.client.0.smithi144.stdout:9/518: unlink d9/de/l32 0 2024-07-26T11:04:46.590 INFO:tasks.workunit.client.0.smithi144.stdout:7/504: write d0/f49 [915189,81598] 0 2024-07-26T11:04:46.591 INFO:tasks.workunit.client.0.smithi144.stdout:4/529: mknod d9/d57/cb4 0 2024-07-26T11:04:46.591 INFO:tasks.workunit.client.0.smithi144.stdout:0/443: dread d0/d25/d5e/d1e/f36 [4194304,4194304] 0 2024-07-26T11:04:46.592 INFO:tasks.workunit.client.0.smithi144.stdout:0/444: dread - d0/d25/d5e/f68 zero size 2024-07-26T11:04:46.592 INFO:tasks.workunit.client.0.smithi144.stdout:0/445: getdents d0/d25/d5e/d1e/d91 0 2024-07-26T11:04:46.592 INFO:tasks.workunit.client.0.smithi144.stdout:9/519: link d9/de/d4e/d6a/l9a d9/la6 0 2024-07-26T11:04:46.593 INFO:tasks.workunit.client.0.smithi144.stdout:7/505: mknod d0/d71/da1/db5/cbb 0 2024-07-26T11:04:46.594 INFO:tasks.workunit.client.0.smithi144.stdout:0/446: truncate d0/d25/d5e/d1e/d30/f87 2064795 0 2024-07-26T11:04:46.594 INFO:tasks.workunit.client.0.smithi144.stdout:0/447: write d0/d25/d5e/d29/f86 [851111,37152] 0 2024-07-26T11:04:46.596 INFO:tasks.workunit.client.0.smithi144.stdout:9/520: rename d9/d3b/d5d/f6b to d9/de/d14/d79/fa7 0 2024-07-26T11:04:46.596 INFO:tasks.workunit.client.0.smithi144.stdout:9/521: write d9/d3b/d5d/f70 [379020,50655] 0 2024-07-26T11:04:46.596 INFO:tasks.workunit.client.0.smithi144.stdout:7/506: creat d0/d6/d9/d64/fbc x:0 0 0 2024-07-26T11:04:46.598 INFO:tasks.workunit.client.0.smithi144.stdout:9/522: symlink d9/de/d14/la8 0 2024-07-26T11:04:46.602 INFO:tasks.workunit.client.0.smithi144.stdout:3/347: dread d1/f32 [0,4194304] 0 2024-07-26T11:04:46.606 INFO:tasks.workunit.client.0.smithi144.stdout:3/348: mkdir d1/d28/d54/d26/d64 0 2024-07-26T11:04:46.608 INFO:tasks.workunit.client.0.smithi144.stdout:3/349: mkdir d1/d28/d54/d26/d30/d4d/d65 0 2024-07-26T11:04:46.609 INFO:tasks.workunit.client.0.smithi144.stdout:3/350: getdents d1/d28/d54/d61 0 2024-07-26T11:04:46.609 INFO:tasks.workunit.client.0.smithi144.stdout:3/351: fsync d1/f34 0 2024-07-26T11:04:46.610 INFO:tasks.workunit.client.0.smithi144.stdout:3/352: mkdir d1/d28/d54/d26/d37/d66 0 2024-07-26T11:04:46.615 INFO:tasks.workunit.client.0.smithi144.stdout:8/478: dwrite d1a/d32/d49/d73/d2c/f8a [0,4194304] 0 2024-07-26T11:04:46.618 INFO:tasks.workunit.client.0.smithi144.stdout:8/479: symlink d1a/d32/d49/d73/d7b/d43/la7 0 2024-07-26T11:04:46.618 INFO:tasks.workunit.client.0.smithi144.stdout:2/536: dread d0/d24/d48/d6d/d5f/f85 [0,4194304] 0 2024-07-26T11:04:46.619 INFO:tasks.workunit.client.0.smithi144.stdout:8/480: creat d1a/d32/d49/d73/d2c/d38/d58/fa8 x:0 0 0 2024-07-26T11:04:46.619 INFO:tasks.workunit.client.0.smithi144.stdout:8/481: read d1a/f21 [3545417,36615] 0 2024-07-26T11:04:46.619 INFO:tasks.workunit.client.0.smithi144.stdout:8/482: chown d1a/d32/d49/d73/d7b/f5e 53212968 1 2024-07-26T11:04:46.621 INFO:tasks.workunit.client.0.smithi144.stdout:8/483: unlink d1a/d32/c85 0 2024-07-26T11:04:46.623 INFO:tasks.workunit.client.0.smithi144.stdout:8/484: link d1a/d32/d49/d73/f80 d1a/d32/fa9 0 2024-07-26T11:04:46.623 INFO:tasks.workunit.client.0.smithi144.stdout:9/523: write d9/de/f4b [52618,59948] 0 2024-07-26T11:04:46.624 INFO:tasks.workunit.client.0.smithi144.stdout:8/485: mkdir d1a/d32/d8c/daa 0 2024-07-26T11:04:46.625 INFO:tasks.workunit.client.0.smithi144.stdout:9/524: mknod d9/de/ca9 0 2024-07-26T11:04:46.627 INFO:tasks.workunit.client.0.smithi144.stdout:8/486: creat d1a/d1c/fab x:0 0 0 2024-07-26T11:04:46.635 INFO:tasks.workunit.client.0.smithi144.stdout:9/525: creat d9/de/faa x:0 0 0 2024-07-26T11:04:46.635 INFO:tasks.workunit.client.0.smithi144.stdout:9/526: readlink d9/de/d14/l95 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:0/448: write d0/d3e/d45/d4c/f79 [3061146,98542] 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:0/449: fsync d0/d25/d5e/d1e/d30/d4b/f58 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:0/450: fdatasync d0/f8 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:1/596: read d3/d6/f18 [251797,48757] 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:3/353: dread d1/d28/d54/d26/f2b [0,4194304] 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:1/597: mknod d3/cbe 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:3/354: write d1/d28/d54/d61/f63 [224030,39387] 0 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:3/355: chown d1/c18 5514218 1 2024-07-26T11:04:46.636 INFO:tasks.workunit.client.0.smithi144.stdout:3/356: dread - d1/d28/d54/f4a zero size 2024-07-26T11:04:46.640 INFO:tasks.workunit.client.0.smithi144.stdout:3/357: rmdir d1/d28/d54/d26/d30/d4d/d65 0 2024-07-26T11:04:46.642 INFO:tasks.workunit.client.0.smithi144.stdout:3/358: rmdir d1/d28/d54/d3d/d42 39 2024-07-26T11:04:46.644 INFO:tasks.workunit.client.0.smithi144.stdout:3/359: truncate d1/d28/d54/d26/f2e 1758254 0 2024-07-26T11:04:46.653 INFO:tasks.workunit.client.0.smithi144.stdout:1/598: dread d3/d6/da/d13/d22/f81 [0,4194304] 0 2024-07-26T11:04:46.654 INFO:tasks.workunit.client.0.smithi144.stdout:1/599: truncate d3/d53/f87 155311 0 2024-07-26T11:04:46.657 INFO:tasks.workunit.client.0.smithi144.stdout:7/507: dwrite d0/d6/d9/d45/d57/f77 [4194304,4194304] 0 2024-07-26T11:04:46.657 INFO:tasks.workunit.client.0.smithi144.stdout:7/508: dread - d0/d6/d9/fb3 zero size 2024-07-26T11:04:46.660 INFO:tasks.workunit.client.0.smithi144.stdout:7/509: readlink d0/d6/d85/d2f/l32 0 2024-07-26T11:04:46.667 INFO:tasks.workunit.client.0.smithi144.stdout:7/510: rename d0/d6/d85/d2f/f33 to d0/d71/fbd 0 2024-07-26T11:04:46.686 INFO:tasks.workunit.client.0.smithi144.stdout:2/537: dwrite d0/d24/d48/d6d/d5f/f85 [0,4194304] 0 2024-07-26T11:04:46.688 INFO:tasks.workunit.client.0.smithi144.stdout:1/600: dread d3/d6/da/d13/d22/f7f [0,4194304] 0 2024-07-26T11:04:46.688 INFO:tasks.workunit.client.0.smithi144.stdout:1/601: fsync d3/f3b 0 2024-07-26T11:04:46.688 INFO:tasks.workunit.client.0.smithi144.stdout:2/538: rename d0/d6/d34/f43 to d0/d6/d9/faa 0 2024-07-26T11:04:46.689 INFO:tasks.workunit.client.0.smithi144.stdout:2/539: chown d0/d24/d48/d6d/fa0 0 1 2024-07-26T11:04:46.692 INFO:tasks.workunit.client.0.smithi144.stdout:8/487: dwrite d1a/d32/d49/d73/f2e [0,4194304] 0 2024-07-26T11:04:46.692 INFO:tasks.workunit.client.0.smithi144.stdout:8/488: getdents d1a/d32/d66/da0 0 2024-07-26T11:04:46.694 INFO:tasks.workunit.client.0.smithi144.stdout:4/530: dwrite d9/d26/d48/d50/d60/f88 [0,4194304] 0 2024-07-26T11:04:46.694 INFO:tasks.workunit.client.0.smithi144.stdout:4/531: write d9/d26/f30 [2494111,37924] 0 2024-07-26T11:04:46.694 INFO:tasks.workunit.client.0.smithi144.stdout:4/532: chown d9/f76 180604 1 2024-07-26T11:04:46.696 INFO:tasks.workunit.client.0.smithi144.stdout:4/533: creat d9/d26/d3c/d6f/fb5 x:0 0 0 2024-07-26T11:04:46.696 INFO:tasks.workunit.client.0.smithi144.stdout:4/534: chown d9/d57/f77 5671 1 2024-07-26T11:04:46.697 INFO:tasks.workunit.client.0.smithi144.stdout:4/535: creat d9/d26/fb6 x:0 0 0 2024-07-26T11:04:46.697 INFO:tasks.workunit.client.0.smithi144.stdout:4/536: chown d9/d26/d48/d50/c67 2 1 2024-07-26T11:04:46.697 INFO:tasks.workunit.client.0.smithi144.stdout:9/527: dwrite d9/de/f46 [0,4194304] 0 2024-07-26T11:04:46.698 INFO:tasks.workunit.client.0.smithi144.stdout:9/528: chown d9/de/d14/d3f/l5c 5963 1 2024-07-26T11:04:46.701 INFO:tasks.workunit.client.0.smithi144.stdout:4/537: creat d9/da/fb7 x:0 0 0 2024-07-26T11:04:46.702 INFO:tasks.workunit.client.0.smithi144.stdout:9/529: creat d9/de/d14/d23/d39/fab x:0 0 0 2024-07-26T11:04:46.702 INFO:tasks.workunit.client.0.smithi144.stdout:9/530: fsync d9/de/d14/f2c 0 2024-07-26T11:04:46.702 INFO:tasks.workunit.client.0.smithi144.stdout:8/489: dread d1a/d32/d49/d73/d7b/d4e/d78/f90 [0,4194304] 0 2024-07-26T11:04:46.704 INFO:tasks.workunit.client.0.smithi144.stdout:9/531: link d9/de/d14/d23/d39/f9f d9/d3b/d5d/fac 0 2024-07-26T11:04:46.706 INFO:tasks.workunit.client.0.smithi144.stdout:9/532: symlink d9/de/d14/d23/d39/lad 0 2024-07-26T11:04:46.706 INFO:tasks.workunit.client.0.smithi144.stdout:9/533: rename d9/de to d9/de/d14/d3f/dae 22 2024-07-26T11:04:46.706 INFO:tasks.workunit.client.0.smithi144.stdout:9/534: chown d9/de/d14/d23/d39/f9f 1 1 2024-07-26T11:04:46.712 INFO:tasks.workunit.client.0.smithi144.stdout:7/511: dwrite d0/d6/d9/d3f/f83 [0,4194304] 0 2024-07-26T11:04:46.712 INFO:tasks.workunit.client.0.smithi144.stdout:7/512: write d0/d6/d85/d2f/d47/d5e/fba [609631,14665] 0 2024-07-26T11:04:46.712 INFO:tasks.workunit.client.0.smithi144.stdout:7/513: dread - d0/d6/d85/d2f/d47/fae zero size 2024-07-26T11:04:46.715 INFO:tasks.workunit.client.0.smithi144.stdout:7/514: creat d0/d6/d85/d2f/da2/fbe x:0 0 0 2024-07-26T11:04:46.724 INFO:tasks.workunit.client.0.smithi144.stdout:7/515: rename d0/d6/d85 to d0/d6/d9/d98/dbf 0 2024-07-26T11:04:46.724 INFO:tasks.workunit.client.0.smithi144.stdout:7/516: truncate d0/d6/d9/d12/fb1 300984 0 2024-07-26T11:04:46.726 INFO:tasks.workunit.client.0.smithi144.stdout:7/517: mkdir d0/d71/dc0 0 2024-07-26T11:04:46.726 INFO:tasks.workunit.client.0.smithi144.stdout:7/518: read d0/f11 [837880,29982] 0 2024-07-26T11:04:46.727 INFO:tasks.workunit.client.0.smithi144.stdout:7/519: chown d0/d6/d9/d45/f2a 1223 1 2024-07-26T11:04:46.728 INFO:tasks.workunit.client.0.smithi144.stdout:7/520: creat d0/d6/d9/d12/d6c/fc1 x:0 0 0 2024-07-26T11:04:46.730 INFO:tasks.workunit.client.0.smithi144.stdout:1/602: dwrite d3/d6/da/d13/d19/d3a/d3e/f6f [0,4194304] 0 2024-07-26T11:04:46.731 INFO:tasks.workunit.client.0.smithi144.stdout:8/490: write d1a/d32/d49/d73/d2c/f8a [1786295,35088] 0 2024-07-26T11:04:46.731 INFO:tasks.workunit.client.0.smithi144.stdout:8/491: truncate d1a/d32/d49/d73/d7b/d43/d4c/f54 8542463 0 2024-07-26T11:04:46.731 INFO:tasks.workunit.client.0.smithi144.stdout:8/492: truncate d1a/d32/d49/d73/f79 736164 0 2024-07-26T11:04:46.733 INFO:tasks.workunit.client.0.smithi144.stdout:1/603: unlink d3/d6/da/f4a 0 2024-07-26T11:04:46.734 INFO:tasks.workunit.client.0.smithi144.stdout:1/604: chown d3/fb2 209 1 2024-07-26T11:04:46.734 INFO:tasks.workunit.client.0.smithi144.stdout:8/493: mkdir d1a/d32/d49/d73/d2c/d9d/d74/dac 0 2024-07-26T11:04:46.741 INFO:tasks.workunit.client.0.smithi144.stdout:8/494: mknod d1a/d32/d49/d73/d2c/d38/d58/da3/cad 0 2024-07-26T11:04:46.741 INFO:tasks.workunit.client.0.smithi144.stdout:8/495: truncate d1a/d1c/f70 76492 0 2024-07-26T11:04:46.741 INFO:tasks.workunit.client.0.smithi144.stdout:8/496: write d1a/f21 [2691682,85192] 0 2024-07-26T11:04:46.754 INFO:tasks.workunit.client.0.smithi144.stdout:4/538: dwrite d9/d57/f77 [0,4194304] 0 2024-07-26T11:04:46.756 INFO:tasks.workunit.client.0.smithi144.stdout:4/539: truncate d9/d26/d3c/f40 384821 0 2024-07-26T11:04:46.758 INFO:tasks.workunit.client.0.smithi144.stdout:4/540: mkdir d9/d26/d48/d98/db8 0 2024-07-26T11:04:46.759 INFO:tasks.workunit.client.0.smithi144.stdout:9/535: dwrite d9/de/d14/d23/d39/fab [0,4194304] 0 2024-07-26T11:04:46.759 INFO:tasks.workunit.client.0.smithi144.stdout:9/536: write d9/de/d4e/f9d [846480,64928] 0 2024-07-26T11:04:46.762 INFO:tasks.workunit.client.0.smithi144.stdout:9/537: mknod d9/de/d14/d3f/d5e/caf 0 2024-07-26T11:04:46.765 INFO:tasks.workunit.client.0.smithi144.stdout:9/538: rename d9/de/d14/d23/d39/c88 to d9/de/d4e/d6a/cb0 0 2024-07-26T11:04:46.766 INFO:tasks.workunit.client.0.smithi144.stdout:8/497: write d1a/f1b [2110737,16654] 0 2024-07-26T11:04:46.766 INFO:tasks.workunit.client.0.smithi144.stdout:4/541: read d9/f51 [122886,1818] 0 2024-07-26T11:04:46.771 INFO:tasks.workunit.client.0.smithi144.stdout:8/498: mkdir d1a/d32/d49/d73/d7b/d43/dae 0 2024-07-26T11:04:46.771 INFO:tasks.workunit.client.0.smithi144.stdout:4/542: rmdir d9/d26/d48 39 2024-07-26T11:04:46.771 INFO:tasks.workunit.client.0.smithi144.stdout:4/543: chown d9/d26/d48/f66 271 1 2024-07-26T11:04:46.771 INFO:tasks.workunit.client.0.smithi144.stdout:8/499: read - d1a/d32/d49/d73/d7b/f8b zero size 2024-07-26T11:04:46.772 INFO:tasks.workunit.client.0.smithi144.stdout:4/544: creat d9/d26/d48/d50/d60/fb9 x:0 0 0 2024-07-26T11:04:46.773 INFO:tasks.workunit.client.0.smithi144.stdout:8/500: mknod d1a/d32/d49/d73/d2c/d38/d84/caf 0 2024-07-26T11:04:46.776 INFO:tasks.workunit.client.0.smithi144.stdout:8/501: rename d1a/d32/d66/d96 to d1a/d32/d49/d73/d2c/d38/d84/db0 0 2024-07-26T11:04:46.776 INFO:tasks.workunit.client.0.smithi144.stdout:8/502: dread - d1a/d1c/f97 zero size 2024-07-26T11:04:46.779 INFO:tasks.workunit.client.0.smithi144.stdout:1/605: dwrite d3/d6/da/d13/d19/d3a/d69/db7/f8f [0,4194304] 0 2024-07-26T11:04:46.781 INFO:tasks.workunit.client.0.smithi144.stdout:1/606: truncate d3/d6/da/d13/d19/d3a/d3e/f50 2607911 0 2024-07-26T11:04:46.787 INFO:tasks.workunit.client.0.smithi144.stdout:1/607: link d3/d6/da/d21/la8 d3/d6/da/d13/lbf 0 2024-07-26T11:04:46.787 INFO:tasks.workunit.client.0.smithi144.stdout:1/608: fdatasync d3/d6/da/d78/f7e 0 2024-07-26T11:04:46.787 INFO:tasks.workunit.client.0.smithi144.stdout:9/539: dread d9/de/d14/d23/f3a [0,4194304] 0 2024-07-26T11:04:46.788 INFO:tasks.workunit.client.0.smithi144.stdout:9/540: chown d9/de/d14/d23 1639 1 2024-07-26T11:04:46.788 INFO:tasks.workunit.client.0.smithi144.stdout:1/609: mkdir d3/d6/dc0 0 2024-07-26T11:04:46.790 INFO:tasks.workunit.client.0.smithi144.stdout:1/610: rename d3/d53/fa6 to d3/d6/da/d21/fc1 0 2024-07-26T11:04:46.791 INFO:tasks.workunit.client.0.smithi144.stdout:1/611: truncate d3/f2d 1442325 0 2024-07-26T11:04:46.791 INFO:tasks.workunit.client.0.smithi144.stdout:1/612: getdents d3/d6/db8 0 2024-07-26T11:04:46.802 INFO:tasks.workunit.client.0.smithi144.stdout:8/503: dread f6 [0,4194304] 0 2024-07-26T11:04:46.806 INFO:tasks.workunit.client.0.smithi144.stdout:1/613: truncate d3/f2d 1027417 0 2024-07-26T11:04:46.808 INFO:tasks.workunit.client.0.smithi144.stdout:1/614: mknod d3/d6/d9c/d9f/cc2 0 2024-07-26T11:04:46.809 INFO:tasks.workunit.client.0.smithi144.stdout:1/615: creat d3/d6/da/d21/d67/fc3 x:0 0 0 2024-07-26T11:04:46.809 INFO:tasks.workunit.client.0.smithi144.stdout:1/616: chown d3/d6/da/d21/f24 125116 1 2024-07-26T11:04:46.811 INFO:tasks.workunit.client.0.smithi144.stdout:1/617: creat d3/d6/d75/d88/da9/dac/fc4 x:0 0 0 2024-07-26T11:04:46.812 INFO:tasks.workunit.client.0.smithi144.stdout:1/618: chown d3/d6/da/d21/d5c/f5e 0 1 2024-07-26T11:04:46.812 INFO:tasks.workunit.client.0.smithi144.stdout:4/545: dwrite d9/d26/d48/d50/d60/f6e [0,4194304] 0 2024-07-26T11:04:46.813 INFO:tasks.workunit.client.0.smithi144.stdout:4/546: fsync d9/d26/d3c/d6f/f7c 0 2024-07-26T11:04:46.815 INFO:tasks.workunit.client.0.smithi144.stdout:4/547: rename d9/d26/l68 to d9/d26/d48/d50/d60/lba 0 2024-07-26T11:04:46.819 INFO:tasks.workunit.client.0.smithi144.stdout:9/541: read d9/de/f51 [1240600,72457] 0 2024-07-26T11:04:46.830 INFO:tasks.workunit.client.0.smithi144.stdout:9/542: dread d9/de/d4e/f9d [0,4194304] 0 2024-07-26T11:04:46.831 INFO:tasks.workunit.client.0.smithi144.stdout:9/543: link d9/de/d14/d3f/d5e/caf d9/de/d14/d23/cb1 0 2024-07-26T11:04:46.831 INFO:tasks.workunit.client.0.smithi144.stdout:9/544: chown d9/de/d14/f22 1974284 1 2024-07-26T11:04:46.832 INFO:tasks.workunit.client.0.smithi144.stdout:9/545: truncate d9/de/f30 1182860 0 2024-07-26T11:04:46.835 INFO:tasks.workunit.client.0.smithi144.stdout:5/503: sync 2024-07-26T11:04:46.835 INFO:tasks.workunit.client.0.smithi144.stdout:6/598: sync 2024-07-26T11:04:46.835 INFO:tasks.workunit.client.0.smithi144.stdout:6/599: truncate d8/f86 4443587 0 2024-07-26T11:04:46.835 INFO:tasks.workunit.client.0.smithi144.stdout:6/600: write d8/d15/f94 [5177895,7082] 0 2024-07-26T11:04:46.836 INFO:tasks.workunit.client.0.smithi144.stdout:5/504: creat d1/d18/d24/d27/d71/fac x:0 0 0 2024-07-26T11:04:46.837 INFO:tasks.workunit.client.0.smithi144.stdout:5/505: chown d1/d18/f90 5169 1 2024-07-26T11:04:46.838 INFO:tasks.workunit.client.0.smithi144.stdout:1/619: write d3/d6/da/d21/f51 [2408087,48586] 0 2024-07-26T11:04:46.840 INFO:tasks.workunit.client.0.smithi144.stdout:5/506: creat d1/d9/d44/da1/fad x:0 0 0 2024-07-26T11:04:46.841 INFO:tasks.workunit.client.0.smithi144.stdout:1/620: mkdir d3/d6/da/d13/dc5 0 2024-07-26T11:04:46.841 INFO:tasks.workunit.client.0.smithi144.stdout:1/621: readlink d3/d6/da/d21/la8 0 2024-07-26T11:04:46.842 INFO:tasks.workunit.client.0.smithi144.stdout:5/507: getdents d1/d9/d44 0 2024-07-26T11:04:46.843 INFO:tasks.workunit.client.0.smithi144.stdout:5/508: readlink d1/l55 0 2024-07-26T11:04:46.845 INFO:tasks.workunit.client.0.smithi144.stdout:1/622: creat d3/d3d/fc6 x:0 0 0 2024-07-26T11:04:46.845 INFO:tasks.workunit.client.0.smithi144.stdout:1/623: fsync d3/d6/da/d13/d19/d3a/d69/f28 0 2024-07-26T11:04:46.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/509: truncate d1/d18/d24/d35/f77 134503 0 2024-07-26T11:04:46.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/510: chown d1/d9/d5b/d4d/c4b 3667770 1 2024-07-26T11:04:46.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/511: readlink d1/d9/d44/l95 0 2024-07-26T11:04:46.848 INFO:tasks.workunit.client.0.smithi144.stdout:1/624: getdents d3/d6/d9c 0 2024-07-26T11:04:46.850 INFO:tasks.workunit.client.0.smithi144.stdout:1/625: rmdir d3/d6/da/d13/d19/d3a/d69/d36 39 2024-07-26T11:04:46.850 INFO:tasks.workunit.client.0.smithi144.stdout:1/626: stat d3/d3d/la0 0 2024-07-26T11:04:46.850 INFO:tasks.workunit.client.0.smithi144.stdout:1/627: chown d3/d6/d75 26 1 2024-07-26T11:04:46.850 INFO:tasks.workunit.client.0.smithi144.stdout:1/628: truncate d3/d6/da/d13/d22/f81 1489610 0 2024-07-26T11:04:46.852 INFO:tasks.workunit.client.0.smithi144.stdout:1/629: creat d3/d6/d75/fc7 x:0 0 0 2024-07-26T11:04:46.854 INFO:tasks.workunit.client.0.smithi144.stdout:5/512: dread d1/d9/d5b/d4d/fa5 [0,4194304] 0 2024-07-26T11:04:46.854 INFO:tasks.workunit.client.0.smithi144.stdout:5/513: dread - d1/f96 zero size 2024-07-26T11:04:46.854 INFO:tasks.workunit.client.0.smithi144.stdout:1/630: symlink d3/d6/dc0/lc8 0 2024-07-26T11:04:46.856 INFO:tasks.workunit.client.0.smithi144.stdout:1/631: unlink d3/d3d/l8b 0 2024-07-26T11:04:46.856 INFO:tasks.workunit.client.0.smithi144.stdout:1/632: chown d3/d6/da/l1a 26 1 2024-07-26T11:04:46.856 INFO:tasks.workunit.client.0.smithi144.stdout:1/633: chown d3/d6/da/d21/f4d 3826722 1 2024-07-26T11:04:46.857 INFO:tasks.workunit.client.0.smithi144.stdout:5/514: mkdir d1/d5/dae 0 2024-07-26T11:04:46.858 INFO:tasks.workunit.client.0.smithi144.stdout:5/515: write d1/d18/d24/fa0 [765366,41208] 0 2024-07-26T11:04:46.858 INFO:tasks.workunit.client.0.smithi144.stdout:5/516: truncate d1/d9/d44/da1/fad 510928 0 2024-07-26T11:04:46.858 INFO:tasks.workunit.client.0.smithi144.stdout:5/517: chown d1/d18 3779455 1 2024-07-26T11:04:46.862 INFO:tasks.workunit.client.0.smithi144.stdout:6/601: dread d8/d15/d57/d5f/d88/f8a [0,4194304] 0 2024-07-26T11:04:46.862 INFO:tasks.workunit.client.0.smithi144.stdout:4/548: dwrite d9/da/f3b [0,4194304] 0 2024-07-26T11:04:46.862 INFO:tasks.workunit.client.0.smithi144.stdout:5/518: creat d1/d9/d44/d45/d98/faf x:0 0 0 2024-07-26T11:04:46.862 INFO:tasks.workunit.client.0.smithi144.stdout:5/519: chown d1/d9/d44/da1/la6 897 1 2024-07-26T11:04:46.862 INFO:tasks.workunit.client.0.smithi144.stdout:5/520: dread - d1/d18/d24/d35/f9c zero size 2024-07-26T11:04:46.863 INFO:tasks.workunit.client.0.smithi144.stdout:5/521: creat d1/d9/d44/da1/fb0 x:0 0 0 2024-07-26T11:04:46.863 INFO:tasks.workunit.client.0.smithi144.stdout:5/522: truncate d1/d9/d44/f8a 1036393 0 2024-07-26T11:04:46.864 INFO:tasks.workunit.client.0.smithi144.stdout:5/523: fsync d1/d9/d44/f4e 0 2024-07-26T11:04:46.865 INFO:tasks.workunit.client.0.smithi144.stdout:5/524: creat d1/d18/d24/fb1 x:0 0 0 2024-07-26T11:04:46.877 INFO:tasks.workunit.client.0.smithi144.stdout:5/525: unlink d1/d18/l64 0 2024-07-26T11:04:46.877 INFO:tasks.workunit.client.0.smithi144.stdout:5/526: getdents d1/d9/d44/d87 0 2024-07-26T11:04:46.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/527: mkdir d1/d9/d44/d45/d98/db2 0 2024-07-26T11:04:46.880 INFO:tasks.workunit.client.0.smithi144.stdout:5/528: mkdir d1/d18/da4/db3 0 2024-07-26T11:04:46.880 INFO:tasks.workunit.client.0.smithi144.stdout:5/529: dread - d1/d9/f8b zero size 2024-07-26T11:04:46.887 INFO:tasks.workunit.client.0.smithi144.stdout:9/546: dwrite d9/de/d4e/f93 [0,4194304] 0 2024-07-26T11:04:46.887 INFO:tasks.workunit.client.0.smithi144.stdout:9/547: chown d9/de/d14/d3f/d5e/f62 833410 1 2024-07-26T11:04:46.887 INFO:tasks.workunit.client.0.smithi144.stdout:9/548: stat d9/de/d14/d23/d39/c48 0 2024-07-26T11:04:46.887 INFO:tasks.workunit.client.0.smithi144.stdout:9/549: dread - d9/de/d4e/d6a/f8a zero size 2024-07-26T11:04:46.888 INFO:tasks.workunit.client.0.smithi144.stdout:9/550: mkdir d9/de/d14/d79/db2 0 2024-07-26T11:04:46.889 INFO:tasks.workunit.client.0.smithi144.stdout:9/551: fsync d9/de/d14/d23/d39/fab 0 2024-07-26T11:04:46.890 INFO:tasks.workunit.client.0.smithi144.stdout:9/552: getdents d9/de/d4e/d6a 0 2024-07-26T11:04:46.912 INFO:tasks.workunit.client.0.smithi144.stdout:4/549: dwrite d9/d26/d48/d50/d60/f6e [0,4194304] 0 2024-07-26T11:04:46.913 INFO:tasks.workunit.client.0.smithi144.stdout:6/602: dwrite d8/f16 [4194304,4194304] 0 2024-07-26T11:04:46.913 INFO:tasks.workunit.client.0.smithi144.stdout:6/603: readlink d8/d15/d57/l65 0 2024-07-26T11:04:46.914 INFO:tasks.workunit.client.0.smithi144.stdout:6/604: mknod d8/db/ccf 0 2024-07-26T11:04:46.915 INFO:tasks.workunit.client.0.smithi144.stdout:4/550: symlink d9/d26/d3c/lbb 0 2024-07-26T11:04:46.919 INFO:tasks.workunit.client.0.smithi144.stdout:6/605: mknod d8/db/dbe/daa/cd0 0 2024-07-26T11:04:46.924 INFO:tasks.workunit.client.0.smithi144.stdout:1/634: dwrite d3/d6/f23 [0,4194304] 0 2024-07-26T11:04:46.924 INFO:tasks.workunit.client.0.smithi144.stdout:1/635: dread - d3/d6/da/d70/fbd zero size 2024-07-26T11:04:46.927 INFO:tasks.workunit.client.0.smithi144.stdout:4/551: dread d9/f15 [0,4194304] 0 2024-07-26T11:04:46.930 INFO:tasks.workunit.client.0.smithi144.stdout:4/552: rename d9/ca5 to d9/da/da1/cbc 0 2024-07-26T11:04:46.932 INFO:tasks.workunit.client.0.smithi144.stdout:4/553: symlink d9/d26/d48/d98/lbd 0 2024-07-26T11:04:46.932 INFO:tasks.workunit.client.0.smithi144.stdout:4/554: truncate d9/d26/f83 2761763 0 2024-07-26T11:04:46.932 INFO:tasks.workunit.client.0.smithi144.stdout:4/555: read d9/d26/d48/d50/f59 [1133599,93371] 0 2024-07-26T11:04:46.933 INFO:tasks.workunit.client.0.smithi144.stdout:4/556: link d9/c78 d9/da/da1/cbe 0 2024-07-26T11:04:46.940 INFO:tasks.workunit.client.0.smithi144.stdout:5/530: dwrite d1/d9/d44/f9d [0,4194304] 0 2024-07-26T11:04:46.943 INFO:tasks.workunit.client.0.smithi144.stdout:5/531: mknod d1/d9/d44/d45/d98/db2/cb4 0 2024-07-26T11:04:46.943 INFO:tasks.workunit.client.0.smithi144.stdout:5/532: write d1/d9/d44/f4e [4087037,77510] 0 2024-07-26T11:04:46.944 INFO:tasks.workunit.client.0.smithi144.stdout:5/533: creat d1/d5/dae/fb5 x:0 0 0 2024-07-26T11:04:46.946 INFO:tasks.workunit.client.0.smithi144.stdout:5/534: unlink d1/d9/d5b/d4d/f79 0 2024-07-26T11:04:46.946 INFO:tasks.workunit.client.0.smithi144.stdout:5/535: write d1/d9/d5b/d4d/f3e [3074936,29496] 0 2024-07-26T11:04:46.946 INFO:tasks.workunit.client.0.smithi144.stdout:5/536: read d1/d9/d5b/d4d/f3e [1176103,66035] 0 2024-07-26T11:04:46.950 INFO:tasks.workunit.client.0.smithi144.stdout:9/553: dwrite d9/de/f30 [0,4194304] 0 2024-07-26T11:04:46.950 INFO:tasks.workunit.client.0.smithi144.stdout:9/554: fdatasync d9/de/d14/f2c 0 2024-07-26T11:04:46.951 INFO:tasks.workunit.client.0.smithi144.stdout:9/555: mknod d9/d3b/d76/d9b/cb3 0 2024-07-26T11:04:46.951 INFO:tasks.workunit.client.0.smithi144.stdout:9/556: truncate d9/de/d14/d3f/d5e/f6c 584273 0 2024-07-26T11:04:46.951 INFO:tasks.workunit.client.0.smithi144.stdout:9/557: write d9/de/f3d [1877482,883] 0 2024-07-26T11:04:46.951 INFO:tasks.workunit.client.0.smithi144.stdout:9/558: creat d9/d3b/d81/fb4 x:0 0 0 2024-07-26T11:04:46.969 INFO:tasks.workunit.client.0.smithi144.stdout:5/537: write d1/f96 [2563646,11842] 0 2024-07-26T11:04:46.969 INFO:tasks.workunit.client.0.smithi144.stdout:6/606: dwrite d8/f7d [0,4194304] 0 2024-07-26T11:04:46.971 INFO:tasks.workunit.client.0.smithi144.stdout:1/636: dwrite d3/d6/da/d13/d19/d3a/d69/f28 [0,4194304] 0 2024-07-26T11:04:46.975 INFO:tasks.workunit.client.0.smithi144.stdout:5/538: mknod d1/d9/d44/d45/d98/cb6 0 2024-07-26T11:04:46.976 INFO:tasks.workunit.client.0.smithi144.stdout:9/559: dread d9/de/f4d [0,4194304] 0 2024-07-26T11:04:46.984 INFO:tasks.workunit.client.0.smithi144.stdout:1/637: rename d3/d6/da/d13/l1c to d3/d6/da/d13/d19/d3a/d69/d36/lc9 0 2024-07-26T11:04:46.985 INFO:tasks.workunit.client.0.smithi144.stdout:5/539: mknod d1/d9/d44/d45/cb7 0 2024-07-26T11:04:46.986 INFO:tasks.workunit.client.0.smithi144.stdout:1/638: creat d3/d6/da/fca x:0 0 0 2024-07-26T11:04:46.996 INFO:tasks.workunit.client.0.smithi144.stdout:6/607: dread d8/f82 [0,4194304] 0 2024-07-26T11:04:46.999 INFO:tasks.workunit.client.0.smithi144.stdout:6/608: rename d8/db/dbe/d74/f9c to d8/d42/dad/fd1 0 2024-07-26T11:04:47.005 INFO:tasks.workunit.client.0.smithi144.stdout:1/639: dread d3/d6/da/d13/d22/f30 [0,4194304] 0 2024-07-26T11:04:47.005 INFO:tasks.workunit.client.0.smithi144.stdout:1/640: fdatasync d3/d6/da/d70/f90 0 2024-07-26T11:04:47.005 INFO:tasks.workunit.client.0.smithi144.stdout:1/641: truncate d3/d6/da/d5b/f8d 560673 0 2024-07-26T11:04:47.005 INFO:tasks.workunit.client.0.smithi144.stdout:1/642: getdents d3/d6/da/d13/dc5 0 2024-07-26T11:04:47.009 INFO:tasks.workunit.client.0.smithi144.stdout:1/643: getdents d3/d6/da/d13/d19/d3a/d69/d36 0 2024-07-26T11:04:47.009 INFO:tasks.workunit.client.0.smithi144.stdout:1/644: readlink d3/d6/da/d13/d19/d3a/l3f 0 2024-07-26T11:04:47.012 INFO:tasks.workunit.client.0.smithi144.stdout:1/645: read d3/d6/da/d13/d22/f3c [179483,62516] 0 2024-07-26T11:04:47.013 INFO:tasks.workunit.client.0.smithi144.stdout:1/646: dread - d3/d6/d75/d88/da9/dac/fc4 zero size 2024-07-26T11:04:47.014 INFO:tasks.workunit.client.0.smithi144.stdout:1/647: creat d3/fcb x:0 0 0 2024-07-26T11:04:47.014 INFO:tasks.workunit.client.0.smithi144.stdout:1/648: fsync d3/d6/f18 0 2024-07-26T11:04:47.026 INFO:tasks.workunit.client.0.smithi144.stdout:0/451: sync 2024-07-26T11:04:47.028 INFO:tasks.workunit.client.0.smithi144.stdout:0/452: rename d0/d25/d5e/l35 to d0/d3e/d5f/l94 0 2024-07-26T11:04:47.029 INFO:tasks.workunit.client.0.smithi144.stdout:0/453: mknod d0/d25/d5e/d1e/d91/c95 0 2024-07-26T11:04:47.029 INFO:tasks.workunit.client.0.smithi144.stdout:0/454: stat d0/d25/d5e/d1e/d30/d4b 0 2024-07-26T11:04:47.058 INFO:tasks.workunit.client.0.smithi144.stdout:6/609: dwrite d8/d15/d26/f28 [0,4194304] 0 2024-07-26T11:04:47.060 INFO:tasks.workunit.client.0.smithi144.stdout:6/610: rename d8/db/dbe/d52/fbd to d8/d42/dad/fd2 0 2024-07-26T11:04:47.061 INFO:tasks.workunit.client.0.smithi144.stdout:6/611: fdatasync d8/d50/d98/f9b 0 2024-07-26T11:04:47.062 INFO:tasks.workunit.client.0.smithi144.stdout:6/612: mkdir d8/d15/d57/d5f/dd3 0 2024-07-26T11:04:47.062 INFO:tasks.workunit.client.0.smithi144.stdout:6/613: fsync d8/d15/f61 0 2024-07-26T11:04:47.063 INFO:tasks.workunit.client.0.smithi144.stdout:0/455: dread d0/d25/d5e/d29/f2d [0,4194304] 0 2024-07-26T11:04:47.063 INFO:tasks.workunit.client.0.smithi144.stdout:6/614: creat d8/d15/fd4 x:0 0 0 2024-07-26T11:04:47.065 INFO:tasks.workunit.client.0.smithi144.stdout:6/615: rename d8/db/f13 to d8/d42/dad/fd5 0 2024-07-26T11:04:47.066 INFO:tasks.workunit.client.0.smithi144.stdout:0/456: link d0/d25/d5e/d29/f6d d0/d25/d5e/d1e/d91/f96 0 2024-07-26T11:04:47.066 INFO:tasks.workunit.client.0.smithi144.stdout:0/457: fdatasync d0/d25/f7b 0 2024-07-26T11:04:47.066 INFO:tasks.workunit.client.0.smithi144.stdout:0/458: stat d0/f27 0 2024-07-26T11:04:47.067 INFO:tasks.workunit.client.0.smithi144.stdout:0/459: write d0/f27 [1303891,84729] 0 2024-07-26T11:04:47.067 INFO:tasks.workunit.client.0.smithi144.stdout:6/616: creat d8/d15/d57/d5f/dd3/fd6 x:0 0 0 2024-07-26T11:04:47.067 INFO:tasks.workunit.client.0.smithi144.stdout:6/617: chown d8/l22 4 1 2024-07-26T11:04:47.069 INFO:tasks.workunit.client.0.smithi144.stdout:6/618: symlink d8/d15/d57/d5f/d6e/ld7 0 2024-07-26T11:04:47.071 INFO:tasks.workunit.client.0.smithi144.stdout:6/619: creat d8/d42/dad/fd8 x:0 0 0 2024-07-26T11:04:47.072 INFO:tasks.workunit.client.0.smithi144.stdout:6/620: rename d8/db/dbe/d5c/cab to d8/d15/d57/d5f/d6e/cd9 0 2024-07-26T11:04:47.117 INFO:tasks.workunit.client.0.smithi144.stdout:3/360: sync 2024-07-26T11:04:47.118 INFO:tasks.workunit.client.0.smithi144.stdout:7/521: sync 2024-07-26T11:04:47.118 INFO:tasks.workunit.client.0.smithi144.stdout:3/361: chown d1/f32 228056 1 2024-07-26T11:04:47.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/504: sync 2024-07-26T11:04:47.121 INFO:tasks.workunit.client.0.smithi144.stdout:2/540: sync 2024-07-26T11:04:47.121 INFO:tasks.workunit.client.0.smithi144.stdout:3/362: rmdir d1/d28/d54/d3d/d42/d44/d48 39 2024-07-26T11:04:47.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/505: mkdir d1a/d32/d49/d73/d2c/d38/d58/da3/db1 0 2024-07-26T11:04:47.122 INFO:tasks.workunit.client.0.smithi144.stdout:8/506: stat d1a/d32/d6c/l59 0 2024-07-26T11:04:47.124 INFO:tasks.workunit.client.0.smithi144.stdout:8/507: mknod d1a/d32/d49/d73/d2c/d38/d58/da3/cb2 0 2024-07-26T11:04:47.127 INFO:tasks.workunit.client.0.smithi144.stdout:8/508: write d1a/d32/f71 [6669,3023] 0 2024-07-26T11:04:47.134 INFO:tasks.workunit.client.0.smithi144.stdout:7/522: dread d0/d6/d9/d64/f65 [0,4194304] 0 2024-07-26T11:04:47.135 INFO:tasks.workunit.client.0.smithi144.stdout:7/523: dread - d0/d6/d9/d98/dbf/d2f/da2/fa6 zero size 2024-07-26T11:04:47.135 INFO:tasks.workunit.client.0.smithi144.stdout:7/524: dread - d0/d6/d9/d45/d57/f7e zero size 2024-07-26T11:04:47.137 INFO:tasks.workunit.client.0.smithi144.stdout:7/525: creat d0/d6/d9/d64/fc2 x:0 0 0 2024-07-26T11:04:47.138 INFO:tasks.workunit.client.0.smithi144.stdout:7/526: mknod d0/d6/d9/d12/d6c/cc3 0 2024-07-26T11:04:47.138 INFO:tasks.workunit.client.0.smithi144.stdout:7/527: truncate d0/d6/d9/d98/dbf/d67/f9b 470578 0 2024-07-26T11:04:47.144 INFO:tasks.workunit.client.0.smithi144.stdout:7/528: rename d0/d6/d9/d64 to d0/d71/d95/dc4 0 2024-07-26T11:04:47.146 INFO:tasks.workunit.client.0.smithi144.stdout:7/529: creat d0/d6/d9/d98/dbf/d2f/d53/fc5 x:0 0 0 2024-07-26T11:04:47.146 INFO:tasks.workunit.client.0.smithi144.stdout:7/530: dread - d0/d6/d9/d45/fa3 zero size 2024-07-26T11:04:47.155 INFO:tasks.workunit.client.0.smithi144.stdout:6/621: dwrite d8/d50/d98/fc5 [0,4194304] 0 2024-07-26T11:04:47.161 INFO:tasks.workunit.client.0.smithi144.stdout:6/622: creat d8/d42/dad/fda x:0 0 0 2024-07-26T11:04:47.162 INFO:tasks.workunit.client.0.smithi144.stdout:6/623: mkdir d8/db/dbe/d52/ddb 0 2024-07-26T11:04:47.163 INFO:tasks.workunit.client.0.smithi144.stdout:6/624: rename d8/d50/d98/fc4 to d8/d50/d98/fdc 0 2024-07-26T11:04:47.165 INFO:tasks.workunit.client.0.smithi144.stdout:6/625: symlink d8/db/dbe/ldd 0 2024-07-26T11:04:47.165 INFO:tasks.workunit.client.0.smithi144.stdout:6/626: write d8/d15/d57/d5f/d88/f95 [619892,128550] 0 2024-07-26T11:04:47.166 INFO:tasks.workunit.client.0.smithi144.stdout:6/627: read - d8/d15/d57/d5f/fc3 zero size 2024-07-26T11:04:47.166 INFO:tasks.workunit.client.0.smithi144.stdout:6/628: chown d8/d50/d98/fb4 2 1 2024-07-26T11:04:47.166 INFO:tasks.workunit.client.0.smithi144.stdout:6/629: dread - d8/d50/d98/fdc zero size 2024-07-26T11:04:47.166 INFO:tasks.workunit.client.0.smithi144.stdout:6/630: write d8/f96 [4597774,5803] 0 2024-07-26T11:04:47.190 INFO:tasks.workunit.client.0.smithi144.stdout:7/531: read d0/d6/d9/d12/f6d [4442573,41902] 0 2024-07-26T11:04:47.193 INFO:tasks.workunit.client.0.smithi144.stdout:7/532: link d0/d6/d9/d98/dbf/fb2 d0/d6/d9/d98/dbf/d2f/d68/fc6 0 2024-07-26T11:04:47.193 INFO:tasks.workunit.client.0.smithi144.stdout:2/541: dwrite d0/d6/d34/f8f [0,4194304] 0 2024-07-26T11:04:47.193 INFO:tasks.workunit.client.0.smithi144.stdout:7/533: write d0/d6/d9/d98/dbf/d2f/d53/fa8 [677149,114941] 0 2024-07-26T11:04:47.193 INFO:tasks.workunit.client.0.smithi144.stdout:2/542: truncate d0/f1e 5230566 0 2024-07-26T11:04:47.194 INFO:tasks.workunit.client.0.smithi144.stdout:7/534: unlink d0/d6/d9/c1f 0 2024-07-26T11:04:47.199 INFO:tasks.workunit.client.0.smithi144.stdout:6/631: read d8/f16 [4849533,85510] 0 2024-07-26T11:04:47.200 INFO:tasks.workunit.client.0.smithi144.stdout:8/509: dwrite d1a/f20 [0,4194304] 0 2024-07-26T11:04:47.203 INFO:tasks.workunit.client.0.smithi144.stdout:8/510: mknod d1a/d32/d6c/d82/cb3 0 2024-07-26T11:04:47.203 INFO:tasks.workunit.client.0.smithi144.stdout:8/511: dread - d1a/d32/d6c/d82/fa4 zero size 2024-07-26T11:04:47.207 INFO:tasks.workunit.client.0.smithi144.stdout:3/363: sync 2024-07-26T11:04:47.208 INFO:tasks.workunit.client.0.smithi144.stdout:3/364: chown d1/d28/l2c 1255551971 1 2024-07-26T11:04:47.208 INFO:tasks.workunit.client.0.smithi144.stdout:5/540: sync 2024-07-26T11:04:47.208 INFO:tasks.workunit.client.0.smithi144.stdout:9/560: sync 2024-07-26T11:04:47.208 INFO:tasks.workunit.client.0.smithi144.stdout:9/561: truncate d9/de/d14/f22 4782273 0 2024-07-26T11:04:47.208 INFO:tasks.workunit.client.0.smithi144.stdout:0/460: sync 2024-07-26T11:04:47.208 INFO:tasks.workunit.client.0.smithi144.stdout:0/461: read - d0/d25/d5e/f2c zero size 2024-07-26T11:04:47.209 INFO:tasks.workunit.client.0.smithi144.stdout:5/541: unlink d1/d5/l20 0 2024-07-26T11:04:47.212 INFO:tasks.workunit.client.0.smithi144.stdout:4/557: sync 2024-07-26T11:04:47.212 INFO:tasks.workunit.client.0.smithi144.stdout:1/649: sync 2024-07-26T11:04:47.214 INFO:tasks.workunit.client.0.smithi144.stdout:1/650: mkdir d3/d6/dcc 0 2024-07-26T11:04:47.222 INFO:tasks.workunit.client.0.smithi144.stdout:4/558: creat d9/da/fbf x:0 0 0 2024-07-26T11:04:47.222 INFO:tasks.workunit.client.0.smithi144.stdout:1/651: creat d3/d6/da/d97/fcd x:0 0 0 2024-07-26T11:04:47.222 INFO:tasks.workunit.client.0.smithi144.stdout:2/543: dread d0/d24/d48/d6d/d28/d2e/d5d/d6b/f73 [0,4194304] 0 2024-07-26T11:04:47.222 INFO:tasks.workunit.client.0.smithi144.stdout:2/544: truncate d0/d24/d2b/d68/d82/f98 308348 0 2024-07-26T11:04:47.227 INFO:tasks.workunit.client.0.smithi144.stdout:8/512: dread d1a/d32/d49/d73/d7b/f37 [0,4194304] 0 2024-07-26T11:04:47.228 INFO:tasks.workunit.client.0.smithi144.stdout:8/513: rmdir d1a/d32/d8c/daa 0 2024-07-26T11:04:47.229 INFO:tasks.workunit.client.0.smithi144.stdout:8/514: mknod d1a/d32/d66/da0/cb4 0 2024-07-26T11:04:47.230 INFO:tasks.workunit.client.0.smithi144.stdout:8/515: mknod d1a/d32/d49/d73/d7b/d43/d4f/cb5 0 2024-07-26T11:04:47.232 INFO:tasks.workunit.client.0.smithi144.stdout:8/516: symlink d1a/d32/d49/d73/d7b/d43/dae/lb6 0 2024-07-26T11:04:47.232 INFO:tasks.workunit.client.0.smithi144.stdout:8/517: read d1a/d32/f71 [6824,19598] 0 2024-07-26T11:04:47.232 INFO:tasks.workunit.client.0.smithi144.stdout:8/518: fdatasync d1a/d32/d49/d73/f80 0 2024-07-26T11:04:47.232 INFO:tasks.workunit.client.0.smithi144.stdout:8/519: write d1a/d32/d49/d73/d7b/f8b [210037,41381] 0 2024-07-26T11:04:47.232 INFO:tasks.workunit.client.0.smithi144.stdout:8/520: stat d1a/d32/d49/d73/d2c/d38/d84 0 2024-07-26T11:04:47.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/632: write d8/db/dbe/fc9 [174891,1351] 0 2024-07-26T11:04:47.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/633: fdatasync f6 0 2024-07-26T11:04:47.244 INFO:tasks.workunit.client.0.smithi144.stdout:6/634: fdatasync d8/d42/f47 0 2024-07-26T11:04:47.245 INFO:tasks.workunit.client.0.smithi144.stdout:3/365: dread d1/d5/fa [0,4194304] 0 2024-07-26T11:04:47.251 INFO:tasks.workunit.client.0.smithi144.stdout:3/366: truncate d1/d28/d54/d3d/f40 2840136 0 2024-07-26T11:04:47.254 INFO:tasks.workunit.client.0.smithi144.stdout:3/367: creat d1/d28/d54/d3d/d42/d44/f67 x:0 0 0 2024-07-26T11:04:47.254 INFO:tasks.workunit.client.0.smithi144.stdout:3/368: chown d1/l10 14 1 2024-07-26T11:04:47.259 INFO:tasks.workunit.client.0.smithi144.stdout:1/652: dread d3/d6/da/d21/d5c/d62/f7c [4194304,4194304] 0 2024-07-26T11:04:47.264 INFO:tasks.workunit.client.0.smithi144.stdout:9/562: dread d9/de/d14/f22 [4194304,4194304] 0 2024-07-26T11:04:47.266 INFO:tasks.workunit.client.0.smithi144.stdout:7/535: dwrite d0/d6/d9/d98/dbf/d2f/f89 [0,4194304] 0 2024-07-26T11:04:47.269 INFO:tasks.workunit.client.0.smithi144.stdout:8/521: dread d1a/d32/d49/d73/f80 [0,4194304] 0 2024-07-26T11:04:47.269 INFO:tasks.workunit.client.0.smithi144.stdout:8/522: fsync d1a/d32/d49/d73/d7b/d43/d4c/f54 0 2024-07-26T11:04:47.276 INFO:tasks.workunit.client.0.smithi144.stdout:0/462: dwrite d0/d25/d5e/d29/f6e [0,4194304] 0 2024-07-26T11:04:47.276 INFO:tasks.workunit.client.0.smithi144.stdout:9/563: truncate d9/de/f3d 1742875 0 2024-07-26T11:04:47.278 INFO:tasks.workunit.client.0.smithi144.stdout:7/536: symlink d0/lc7 0 2024-07-26T11:04:47.281 INFO:tasks.workunit.client.0.smithi144.stdout:7/537: truncate d0/d6/d9/d98/dbf/d67/f9b 857901 0 2024-07-26T11:04:47.281 INFO:tasks.workunit.client.0.smithi144.stdout:7/538: write d0/d6/d9/d98/dbf/f41 [1684031,11768] 0 2024-07-26T11:04:47.286 INFO:tasks.workunit.client.0.smithi144.stdout:3/369: dread d1/d28/d54/d26/f2e [0,4194304] 0 2024-07-26T11:04:47.290 INFO:tasks.workunit.client.0.smithi144.stdout:8/523: rename d1a/d32/d49/d73/d2c/f94 to d1a/d32/d49/fb7 0 2024-07-26T11:04:47.290 INFO:tasks.workunit.client.0.smithi144.stdout:9/564: rmdir d9/de/d14/d23/d98 0 2024-07-26T11:04:47.290 INFO:tasks.workunit.client.0.smithi144.stdout:9/565: truncate d9/de/f52 796756 0 2024-07-26T11:04:47.290 INFO:tasks.workunit.client.0.smithi144.stdout:9/566: write d9/de/f4b [1489484,88106] 0 2024-07-26T11:04:47.290 INFO:tasks.workunit.client.0.smithi144.stdout:4/559: dread d9/f12 [0,4194304] 0 2024-07-26T11:04:47.291 INFO:tasks.workunit.client.0.smithi144.stdout:9/567: write d9/de/d14/d3f/f53 [811366,130192] 0 2024-07-26T11:04:47.291 INFO:tasks.workunit.client.0.smithi144.stdout:0/463: rmdir d0/d25/d5e/d29 39 2024-07-26T11:04:47.295 INFO:tasks.workunit.client.0.smithi144.stdout:0/464: creat d0/d25/d5e/d1e/d6c/f97 x:0 0 0 2024-07-26T11:04:47.295 INFO:tasks.workunit.client.0.smithi144.stdout:9/568: creat d9/d3b/d76/fb5 x:0 0 0 2024-07-26T11:04:47.295 INFO:tasks.workunit.client.0.smithi144.stdout:0/465: write d0/d3e/d45/f55 [1448590,111540] 0 2024-07-26T11:04:47.295 INFO:tasks.workunit.client.0.smithi144.stdout:3/370: rename d1/d28/d54/d26/d37/d66 to d1/d28/d54/d26/d64/d68 0 2024-07-26T11:04:47.299 INFO:tasks.workunit.client.0.smithi144.stdout:2/545: write d0/d24/d2b/d68/d82/f7c [4147659,31783] 0 2024-07-26T11:04:47.299 INFO:tasks.workunit.client.0.smithi144.stdout:9/569: mknod d9/de/d14/cb6 0 2024-07-26T11:04:47.303 INFO:tasks.workunit.client.0.smithi144.stdout:5/542: dwrite d1/d9/f10 [4194304,4194304] 0 2024-07-26T11:04:47.303 INFO:tasks.workunit.client.0.smithi144.stdout:5/543: fdatasync d1/f97 0 2024-07-26T11:04:47.304 INFO:tasks.workunit.client.0.smithi144.stdout:2/546: getdents d0/d24/d48/d6d 0 2024-07-26T11:04:47.304 INFO:tasks.workunit.client.0.smithi144.stdout:2/547: dread - d0/d24/d48/d6d/f96 zero size 2024-07-26T11:04:47.306 INFO:tasks.workunit.client.0.smithi144.stdout:9/570: creat d9/d3b/d76/d9b/fb7 x:0 0 0 2024-07-26T11:04:47.307 INFO:tasks.workunit.client.0.smithi144.stdout:9/571: chown d9/d3b/d76/fb5 25 1 2024-07-26T11:04:47.308 INFO:tasks.workunit.client.0.smithi144.stdout:2/548: creat d0/d24/d2b/d68/d82/fab x:0 0 0 2024-07-26T11:04:47.312 INFO:tasks.workunit.client.0.smithi144.stdout:9/572: rename d9/de/l33 to d9/de/lb8 0 2024-07-26T11:04:47.312 INFO:tasks.workunit.client.0.smithi144.stdout:9/573: write d9/de/d14/d79/f91 [717052,69631] 0 2024-07-26T11:04:47.312 INFO:tasks.workunit.client.0.smithi144.stdout:9/574: truncate d9/de/d14/d23/d39/f5b 1003637 0 2024-07-26T11:04:47.335 INFO:tasks.workunit.client.0.smithi144.stdout:8/524: dread d1a/d32/d49/d73/d2c/d38/d58/f83 [0,4194304] 0 2024-07-26T11:04:47.339 INFO:tasks.workunit.client.0.smithi144.stdout:8/525: symlink d1a/d32/d49/d73/d2c/d38/d58/da3/db1/lb8 0 2024-07-26T11:04:47.339 INFO:tasks.workunit.client.0.smithi144.stdout:8/526: stat d1a/d32/d6c/c6f 0 2024-07-26T11:04:47.340 INFO:tasks.workunit.client.0.smithi144.stdout:5/544: dread d1/d5/f7f [0,4194304] 0 2024-07-26T11:04:47.340 INFO:tasks.workunit.client.0.smithi144.stdout:6/635: dwrite d8/f96 [0,4194304] 0 2024-07-26T11:04:47.340 INFO:tasks.workunit.client.0.smithi144.stdout:6/636: chown d8/db/fa8 7589 1 2024-07-26T11:04:47.340 INFO:tasks.workunit.client.0.smithi144.stdout:5/545: unlink d1/d9/l1e 0 2024-07-26T11:04:47.347 INFO:tasks.workunit.client.0.smithi144.stdout:2/549: write d0/d6/d9/faa [2680670,23813] 0 2024-07-26T11:04:47.347 INFO:tasks.workunit.client.0.smithi144.stdout:7/539: dwrite d0/d6/d9/d98/dbf/d2f/d47/d5e/fba [0,4194304] 0 2024-07-26T11:04:47.347 INFO:tasks.workunit.client.0.smithi144.stdout:6/637: rename d8/f7d to d8/d15/d57/d5f/d6e/fde 0 2024-07-26T11:04:47.351 INFO:tasks.workunit.client.0.smithi144.stdout:2/550: symlink d0/d24/d48/d6d/d28/da8/lac 0 2024-07-26T11:04:47.351 INFO:tasks.workunit.client.0.smithi144.stdout:6/638: mknod d8/dca/cdf 0 2024-07-26T11:04:47.354 INFO:tasks.workunit.client.0.smithi144.stdout:6/639: symlink d8/db/dbe/d5c/le0 0 2024-07-26T11:04:47.354 INFO:tasks.workunit.client.0.smithi144.stdout:6/640: chown d8/db/dbe/ldd 314369914 1 2024-07-26T11:04:47.354 INFO:tasks.workunit.client.0.smithi144.stdout:6/641: write d8/d15/f94 [9000760,23295] 0 2024-07-26T11:04:47.366 INFO:tasks.workunit.client.0.smithi144.stdout:6/642: mknod d8/d15/d57/d5f/ce1 0 2024-07-26T11:04:47.366 INFO:tasks.workunit.client.0.smithi144.stdout:2/551: dread d0/d24/d2b/f8d [0,4194304] 0 2024-07-26T11:04:47.367 INFO:tasks.workunit.client.0.smithi144.stdout:2/552: readlink d0/d24/d48/d6d/l5b 0 2024-07-26T11:04:47.367 INFO:tasks.workunit.client.0.smithi144.stdout:2/553: write d0/d24/d48/d6d/d28/f72 [886435,103282] 0 2024-07-26T11:04:47.367 INFO:tasks.workunit.client.0.smithi144.stdout:4/560: dwrite d9/d26/d48/d50/d60/fb9 [0,4194304] 0 2024-07-26T11:04:47.367 INFO:tasks.workunit.client.0.smithi144.stdout:4/561: dread - d9/da/f5b zero size 2024-07-26T11:04:47.367 INFO:tasks.workunit.client.0.smithi144.stdout:4/562: fdatasync d9/d26/d48/d50/d60/fb9 0 2024-07-26T11:04:47.367 INFO:tasks.workunit.client.0.smithi144.stdout:4/563: dread - d9/d26/fab zero size 2024-07-26T11:04:47.369 INFO:tasks.workunit.client.0.smithi144.stdout:2/554: link d0/d24/d48/c54 d0/d24/d48/d6d/d28/d2e/d5d/cad 0 2024-07-26T11:04:47.372 INFO:tasks.workunit.client.0.smithi144.stdout:5/546: dread d1/d18/d24/d35/f3d [0,4194304] 0 2024-07-26T11:04:47.372 INFO:tasks.workunit.client.0.smithi144.stdout:2/555: mkdir d0/d24/d48/d6d/d28/d2e/d5d/dae 0 2024-07-26T11:04:47.378 INFO:tasks.workunit.client.0.smithi144.stdout:5/547: dread d1/d9/d44/da1/fad [0,4194304] 0 2024-07-26T11:04:47.380 INFO:tasks.workunit.client.0.smithi144.stdout:5/548: creat d1/d18/da4/db3/fb8 x:0 0 0 2024-07-26T11:04:47.381 INFO:tasks.workunit.client.0.smithi144.stdout:5/549: creat d1/d18/d24/d35/fb9 x:0 0 0 2024-07-26T11:04:47.384 INFO:tasks.workunit.client.0.smithi144.stdout:9/575: dread d9/de/d14/d3f/f53 [0,4194304] 0 2024-07-26T11:04:47.384 INFO:tasks.workunit.client.0.smithi144.stdout:0/466: read d0/f32 [1850304,72024] 0 2024-07-26T11:04:47.385 INFO:tasks.workunit.client.0.smithi144.stdout:0/467: dread - d0/d25/d5e/d29/f82 zero size 2024-07-26T11:04:47.386 INFO:tasks.workunit.client.0.smithi144.stdout:1/653: dwrite d3/d6/da/d5b/f8d [0,4194304] 0 2024-07-26T11:04:47.386 INFO:tasks.workunit.client.0.smithi144.stdout:5/550: unlink d1/l55 0 2024-07-26T11:04:47.389 INFO:tasks.workunit.client.0.smithi144.stdout:3/371: dwrite d1/d28/d54/d26/f43 [0,4194304] 0 2024-07-26T11:04:47.390 INFO:tasks.workunit.client.0.smithi144.stdout:0/468: link d0/d25/l63 d0/d3e/d45/d4c/l98 0 2024-07-26T11:04:47.390 INFO:tasks.workunit.client.0.smithi144.stdout:0/469: write d0/d3e/d45/f50 [1164556,77481] 0 2024-07-26T11:04:47.391 INFO:tasks.workunit.client.0.smithi144.stdout:1/654: link d3/d6/da/laa d3/d3d/lce 0 2024-07-26T11:04:47.392 INFO:tasks.workunit.client.0.smithi144.stdout:8/527: dwrite d1a/d32/d6c/d3c/f53 [0,4194304] 0 2024-07-26T11:04:47.392 INFO:tasks.workunit.client.0.smithi144.stdout:8/528: chown d1a/d32/d49/d73/d2c/d38/d58/l7e 1402009 1 2024-07-26T11:04:47.401 INFO:tasks.workunit.client.0.smithi144.stdout:5/551: rename d1/d5/f7f to d1/d9/d5b/d86/fba 0 2024-07-26T11:04:47.401 INFO:tasks.workunit.client.0.smithi144.stdout:0/470: creat d0/d25/d5e/d1e/d5c/f99 x:0 0 0 2024-07-26T11:04:47.401 INFO:tasks.workunit.client.0.smithi144.stdout:0/471: readlink d0/l44 0 2024-07-26T11:04:47.401 INFO:tasks.workunit.client.0.smithi144.stdout:0/472: fsync d0/d25/d5e/d29/f2d 0 2024-07-26T11:04:47.401 INFO:tasks.workunit.client.0.smithi144.stdout:0/473: chown d0/d25/c5b 2274 1 2024-07-26T11:04:47.401 INFO:tasks.workunit.client.0.smithi144.stdout:0/474: write d0/f27 [1860808,64747] 0 2024-07-26T11:04:47.403 INFO:tasks.workunit.client.0.smithi144.stdout:4/564: dread d9/d26/d48/d50/f64 [0,4194304] 0 2024-07-26T11:04:47.403 INFO:tasks.workunit.client.0.smithi144.stdout:4/565: dread - d9/d26/d48/f4d zero size 2024-07-26T11:04:47.403 INFO:tasks.workunit.client.0.smithi144.stdout:1/655: rename d3/d6/da/d13/d22/lb9 to d3/d6/d75/d88/da9/dac/lcf 0 2024-07-26T11:04:47.404 INFO:tasks.workunit.client.0.smithi144.stdout:1/656: truncate d3/d53/f87 739004 0 2024-07-26T11:04:47.404 INFO:tasks.workunit.client.0.smithi144.stdout:8/529: mknod d1a/d32/d49/cb9 0 2024-07-26T11:04:47.405 INFO:tasks.workunit.client.0.smithi144.stdout:1/657: link d3/d6/da/d13/d22/f7f d3/d6/d9c/d9f/fd0 0 2024-07-26T11:04:47.407 INFO:tasks.workunit.client.0.smithi144.stdout:1/658: unlink d3/d53/l6c 0 2024-07-26T11:04:47.407 INFO:tasks.workunit.client.0.smithi144.stdout:1/659: chown d3/f9b 1195 1 2024-07-26T11:04:47.408 INFO:tasks.workunit.client.0.smithi144.stdout:6/643: dread d8/d42/f7c [0,4194304] 0 2024-07-26T11:04:47.408 INFO:tasks.workunit.client.0.smithi144.stdout:6/644: dread - d8/d50/d98/f9f zero size 2024-07-26T11:04:47.410 INFO:tasks.workunit.client.0.smithi144.stdout:9/576: dread d9/de/f4d [0,4194304] 0 2024-07-26T11:04:47.417 INFO:tasks.workunit.client.0.smithi144.stdout:9/577: symlink d9/de/d14/lb9 0 2024-07-26T11:04:47.418 INFO:tasks.workunit.client.0.smithi144.stdout:4/566: dread d9/d26/d3c/d6f/f7c [4194304,4194304] 0 2024-07-26T11:04:47.419 INFO:tasks.workunit.client.0.smithi144.stdout:9/578: mknod d9/de/cba 0 2024-07-26T11:04:47.420 INFO:tasks.workunit.client.0.smithi144.stdout:4/567: creat d9/fc0 x:0 0 0 2024-07-26T11:04:47.420 INFO:tasks.workunit.client.0.smithi144.stdout:4/568: write d9/d57/fa4 [4231424,20742] 0 2024-07-26T11:04:47.420 INFO:tasks.workunit.client.0.smithi144.stdout:4/569: chown d9/d26/d48 1 1 2024-07-26T11:04:47.421 INFO:tasks.workunit.client.0.smithi144.stdout:9/579: creat d9/de/d14/fbb x:0 0 0 2024-07-26T11:04:47.427 INFO:tasks.workunit.client.0.smithi144.stdout:4/570: creat d9/da/da1/fc1 x:0 0 0 2024-07-26T11:04:47.427 INFO:tasks.workunit.client.0.smithi144.stdout:4/571: dread - d9/d26/d3c/d6f/f86 zero size 2024-07-26T11:04:47.429 INFO:tasks.workunit.client.0.smithi144.stdout:4/572: rename d9/d26/d3c/f40 to d9/d26/d48/d98/fc2 0 2024-07-26T11:04:47.430 INFO:tasks.workunit.client.0.smithi144.stdout:3/372: dwrite d1/d5/f24 [0,4194304] 0 2024-07-26T11:04:47.431 INFO:tasks.workunit.client.0.smithi144.stdout:4/573: read d9/d26/f28 [643387,37405] 0 2024-07-26T11:04:47.434 INFO:tasks.workunit.client.0.smithi144.stdout:6/645: write d8/db/dbe/f72 [603806,87295] 0 2024-07-26T11:04:47.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/580: dread d9/de/d14/f22 [0,4194304] 0 2024-07-26T11:04:47.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/581: dread - d9/de/faa zero size 2024-07-26T11:04:47.461 INFO:tasks.workunit.client.0.smithi144.stdout:1/660: dread d3/d6/da/d13/d19/d3a/d69/db7/f8f [0,4194304] 0 2024-07-26T11:04:47.462 INFO:tasks.workunit.client.0.smithi144.stdout:4/574: write d9/d26/d48/d50/d60/f88 [2747202,11170] 0 2024-07-26T11:04:47.463 INFO:tasks.workunit.client.0.smithi144.stdout:5/552: dwrite d1/d9/d44/f41 [0,4194304] 0 2024-07-26T11:04:47.463 INFO:tasks.workunit.client.0.smithi144.stdout:5/553: readlink d1/d9/d44/la7 0 2024-07-26T11:04:47.463 INFO:tasks.workunit.client.0.smithi144.stdout:8/530: dwrite d1a/d32/d49/d73/d2c/f4d [0,4194304] 0 2024-07-26T11:04:47.463 INFO:tasks.workunit.client.0.smithi144.stdout:8/531: getdents d1a/d32/d49/d73/d7b/d43/d4f/da2 0 2024-07-26T11:04:47.468 INFO:tasks.workunit.client.0.smithi144.stdout:6/646: dread d8/f16 [0,4194304] 0 2024-07-26T11:04:47.469 INFO:tasks.workunit.client.0.smithi144.stdout:1/661: symlink d3/d6/da/d21/d5c/ld1 0 2024-07-26T11:04:47.470 INFO:tasks.workunit.client.0.smithi144.stdout:4/575: creat d9/d26/d48/d50/d60/fc3 x:0 0 0 2024-07-26T11:04:47.471 INFO:tasks.workunit.client.0.smithi144.stdout:5/554: creat d1/d9/d44/d45/daa/fbb x:0 0 0 2024-07-26T11:04:47.472 INFO:tasks.workunit.client.0.smithi144.stdout:6/647: mkdir d8/d50/de2 0 2024-07-26T11:04:47.474 INFO:tasks.workunit.client.0.smithi144.stdout:1/662: creat d3/d6/da/d21/d5c/d62/fd2 x:0 0 0 2024-07-26T11:04:47.480 INFO:tasks.workunit.client.0.smithi144.stdout:1/663: rename d3/d6/da/d21/c76 to d3/d6/d75/d88/cd3 0 2024-07-26T11:04:47.481 INFO:tasks.workunit.client.0.smithi144.stdout:3/373: dwrite d1/d5/f3a [0,4194304] 0 2024-07-26T11:04:47.493 INFO:tasks.workunit.client.0.smithi144.stdout:8/532: read d1a/d32/d49/d73/d2c/d38/d58/f6e [3333866,58033] 0 2024-07-26T11:04:47.517 INFO:tasks.workunit.client.0.smithi144.stdout:8/533: dread d1a/d1c/f22 [4194304,4194304] 0 2024-07-26T11:04:47.517 INFO:tasks.workunit.client.0.smithi144.stdout:8/534: stat d1a/d32/d49/d73/d7b/d43/d4f/l56 0 2024-07-26T11:04:47.529 INFO:tasks.workunit.client.0.smithi144.stdout:4/576: dwrite d9/d26/d48/d50/fa7 [0,4194304] 0 2024-07-26T11:04:47.529 INFO:tasks.workunit.client.0.smithi144.stdout:4/577: dread - d9/da/f5b zero size 2024-07-26T11:04:47.535 INFO:tasks.workunit.client.0.smithi144.stdout:4/578: rmdir d9 39 2024-07-26T11:04:47.539 INFO:tasks.workunit.client.0.smithi144.stdout:4/579: mkdir d9/d7b/dc4 0 2024-07-26T11:04:47.548 INFO:tasks.workunit.client.0.smithi144.stdout:5/555: dwrite d1/d9/d44/da1/fb0 [0,4194304] 0 2024-07-26T11:04:47.558 INFO:tasks.workunit.client.0.smithi144.stdout:6/648: dwrite d8/db/dbe/f62 [0,4194304] 0 2024-07-26T11:04:47.558 INFO:tasks.workunit.client.0.smithi144.stdout:6/649: truncate d8/d50/f85 4481766 0 2024-07-26T11:04:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:47 smithi160 ceph-mon[36715]: pgmap v530: 129 pgs: 129 active+clean; 992 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 154 MiB/s wr, 711 op/s 2024-07-26T11:04:47.561 INFO:tasks.workunit.client.0.smithi144.stdout:6/650: mkdir d8/db/dbe/d52/ddb/de3 0 2024-07-26T11:04:47.561 INFO:tasks.workunit.client.0.smithi144.stdout:6/651: chown d8/d15/d57/d5f/fc3 180333 1 2024-07-26T11:04:47.561 INFO:tasks.workunit.client.0.smithi144.stdout:6/652: readlink d8/l22 0 2024-07-26T11:04:47.572 INFO:tasks.workunit.client.0.smithi144.stdout:1/664: dwrite d3/d6/da/d21/f4d [0,4194304] 0 2024-07-26T11:04:47.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/580: dread d9/da/f3b [0,4194304] 0 2024-07-26T11:04:47.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/581: write d9/da/fb0 [1138469,10819] 0 2024-07-26T11:04:47.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/665: link d3/d6/da/d78/f9d d3/d6/da/d13/d19/d3a/d69/fd4 0 2024-07-26T11:04:47.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/666: dread - d3/d6/d75/d88/da9/dac/fc4 zero size 2024-07-26T11:04:47.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/667: dread - d3/d6/da/d13/d19/d3a/d69/d36/f96 zero size 2024-07-26T11:04:47.580 INFO:tasks.workunit.client.0.smithi144.stdout:8/535: dwrite d1a/d32/d6c/d82/f9c [0,4194304] 0 2024-07-26T11:04:47.584 INFO:tasks.workunit.client.0.smithi144.stdout:4/582: read d9/d26/f30 [1531973,126059] 0 2024-07-26T11:04:47.589 INFO:tasks.workunit.client.0.smithi144.stdout:6/653: dread d8/f1d [0,4194304] 0 2024-07-26T11:04:47.589 INFO:tasks.workunit.client.0.smithi144.stdout:6/654: fdatasync d8/d15/d57/d5f/d6e/fb7 0 2024-07-26T11:04:47.591 INFO:tasks.workunit.client.0.smithi144.stdout:8/536: write d1a/d32/d49/d73/f79 [22008,97610] 0 2024-07-26T11:04:47.603 INFO:tasks.workunit.client.0.smithi144.stdout:6/655: symlink d8/le4 0 2024-07-26T11:04:47.604 INFO:tasks.workunit.client.0.smithi144.stdout:6/656: write d8/db/dbe/d52/f8d [765017,125375] 0 2024-07-26T11:04:47.607 INFO:tasks.workunit.client.0.smithi144.stdout:6/657: getdents d8/db/dbe/d5c 0 2024-07-26T11:04:47.607 INFO:tasks.workunit.client.0.smithi144.stdout:6/658: getdents d8/db/dbe/d52/d58 0 2024-07-26T11:04:47.616 INFO:tasks.workunit.client.0.smithi144.stdout:5/556: dwrite d1/d5/f2e [0,4194304] 0 2024-07-26T11:04:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:47 smithi183 ceph-mon[37273]: pgmap v530: 129 pgs: 129 active+clean; 992 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 154 MiB/s wr, 711 op/s 2024-07-26T11:04:47.636 INFO:tasks.workunit.client.0.smithi144.stdout:1/668: dwrite d3/d6/da/d21/f24 [0,4194304] 0 2024-07-26T11:04:47.636 INFO:tasks.workunit.client.0.smithi144.stdout:1/669: chown d3/d6/d75/d88/da9/dac/fc4 2243666 1 2024-07-26T11:04:47.636 INFO:tasks.workunit.client.0.smithi144.stdout:1/670: dread - d3/d3d/fc6 zero size 2024-07-26T11:04:47.636 INFO:tasks.workunit.client.0.smithi144.stdout:4/583: dwrite d9/d57/f77 [0,4194304] 0 2024-07-26T11:04:47.637 INFO:tasks.workunit.client.0.smithi144.stdout:6/659: dread d8/f53 [0,4194304] 0 2024-07-26T11:04:47.637 INFO:tasks.workunit.client.0.smithi144.stdout:6/660: fdatasync d8/d42/dad/fd2 0 2024-07-26T11:04:47.637 INFO:tasks.workunit.client.0.smithi144.stdout:1/671: link d3/d6/d9c/cb0 d3/d6/da/d21/d5c/d62/d68/cd5 0 2024-07-26T11:04:47.638 INFO:tasks.workunit.client.0.smithi144.stdout:6/661: creat d8/d15/d57/d5f/fe5 x:0 0 0 2024-07-26T11:04:47.639 INFO:tasks.workunit.client.0.smithi144.stdout:6/662: creat d8/db/dbe/d52/ddb/fe6 x:0 0 0 2024-07-26T11:04:47.644 INFO:tasks.workunit.client.0.smithi144.stdout:5/557: write d1/d5/f2d [5669092,30007] 0 2024-07-26T11:04:47.644 INFO:tasks.workunit.client.0.smithi144.stdout:8/537: dwrite d1a/d1c/fab [0,4194304] 0 2024-07-26T11:04:47.644 INFO:tasks.workunit.client.0.smithi144.stdout:8/538: truncate d1a/d1c/f70 617633 0 2024-07-26T11:04:47.646 INFO:tasks.workunit.client.0.smithi144.stdout:5/558: link d1/d5/f6d d1/d9/d44/d85/fbc 0 2024-07-26T11:04:47.646 INFO:tasks.workunit.client.0.smithi144.stdout:4/584: dread d9/f15 [0,4194304] 0 2024-07-26T11:04:47.656 INFO:tasks.workunit.client.0.smithi144.stdout:5/559: creat d1/d5/dae/fbd x:0 0 0 2024-07-26T11:04:47.658 INFO:tasks.workunit.client.0.smithi144.stdout:4/585: mkdir d9/d26/d48/d50/d60/dc5 0 2024-07-26T11:04:47.658 INFO:tasks.workunit.client.0.smithi144.stdout:4/586: readlink d9/d26/d48/l6d 0 2024-07-26T11:04:47.658 INFO:tasks.workunit.client.0.smithi144.stdout:4/587: write d9/d26/f97 [766989,52745] 0 2024-07-26T11:04:47.658 INFO:tasks.workunit.client.0.smithi144.stdout:4/588: write d9/d26/d3c/d6f/f86 [878349,18563] 0 2024-07-26T11:04:47.659 INFO:tasks.workunit.client.0.smithi144.stdout:5/560: creat d1/d9/d44/d45/d98/fbe x:0 0 0 2024-07-26T11:04:47.661 INFO:tasks.workunit.client.0.smithi144.stdout:4/589: creat d9/fc6 x:0 0 0 2024-07-26T11:04:47.663 INFO:tasks.workunit.client.0.smithi144.stdout:5/561: rename d1/d18/d24/f46 to d1/d9/d44/d45/d81/fbf 0 2024-07-26T11:04:47.665 INFO:tasks.workunit.client.0.smithi144.stdout:5/562: creat d1/d9/d5b/d4d/fc0 x:0 0 0 2024-07-26T11:04:47.665 INFO:tasks.workunit.client.0.smithi144.stdout:5/563: dread - d1/d9/d44/d85/fbc zero size 2024-07-26T11:04:47.665 INFO:tasks.workunit.client.0.smithi144.stdout:5/564: dread - d1/d9/d44/d45/daa/fbb zero size 2024-07-26T11:04:47.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/565: creat d1/d9/d44/d45/fc1 x:0 0 0 2024-07-26T11:04:47.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/566: dread - d1/d9/d44/d85/fbc zero size 2024-07-26T11:04:47.676 INFO:tasks.workunit.client.0.smithi144.stdout:4/590: dread d9/d26/f92 [0,4194304] 0 2024-07-26T11:04:47.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/591: rmdir d9/d26/d48/d50/d60 39 2024-07-26T11:04:47.681 INFO:tasks.workunit.client.0.smithi144.stdout:4/592: truncate d9/d57/f77 2731140 0 2024-07-26T11:04:47.687 INFO:tasks.workunit.client.0.smithi144.stdout:1/672: dwrite d3/d6/da/d13/d22/f7f [0,4194304] 0 2024-07-26T11:04:47.689 INFO:tasks.workunit.client.0.smithi144.stdout:1/673: mkdir d3/d6/da/d70/dd6 0 2024-07-26T11:04:47.689 INFO:tasks.workunit.client.0.smithi144.stdout:1/674: dread - d3/d6/d75/f93 zero size 2024-07-26T11:04:47.691 INFO:tasks.workunit.client.0.smithi144.stdout:1/675: getdents d3/d6/da/d13/d22/d5d 0 2024-07-26T11:04:47.693 INFO:tasks.workunit.client.0.smithi144.stdout:1/676: rename d3/cbe to d3/d6/da/d13/d19/cd7 0 2024-07-26T11:04:47.693 INFO:tasks.workunit.client.0.smithi144.stdout:1/677: chown d3/d6/da/d21/d5c/d62/d68 102486 1 2024-07-26T11:04:47.693 INFO:tasks.workunit.client.0.smithi144.stdout:6/663: dwrite d8/d15/d57/d5f/d88/f95 [0,4194304] 0 2024-07-26T11:04:47.694 INFO:tasks.workunit.client.0.smithi144.stdout:1/678: dread - d3/d3d/fc6 zero size 2024-07-26T11:04:47.696 INFO:tasks.workunit.client.0.smithi144.stdout:6/664: rename d8/d42/dad/fd1 to d8/db/fe7 0 2024-07-26T11:04:47.698 INFO:tasks.workunit.client.0.smithi144.stdout:6/665: creat d8/db/dbe/d52/d58/dbf/fe8 x:0 0 0 2024-07-26T11:04:47.707 INFO:tasks.workunit.client.0.smithi144.stdout:6/666: dread d8/db/dbe/d5c/fa0 [0,4194304] 0 2024-07-26T11:04:47.708 INFO:tasks.workunit.client.0.smithi144.stdout:6/667: symlink d8/db/dbe/d52/le9 0 2024-07-26T11:04:47.708 INFO:tasks.workunit.client.0.smithi144.stdout:6/668: write d8/d50/f79 [1390310,76416] 0 2024-07-26T11:04:47.709 INFO:tasks.workunit.client.0.smithi144.stdout:6/669: write d8/f1d [4796985,113410] 0 2024-07-26T11:04:47.710 INFO:tasks.workunit.client.0.smithi144.stdout:6/670: mkdir d8/d50/d92/dea 0 2024-07-26T11:04:47.711 INFO:tasks.workunit.client.0.smithi144.stdout:6/671: mknod d8/d15/d57/d5f/dd3/ceb 0 2024-07-26T11:04:47.718 INFO:tasks.workunit.client.0.smithi144.stdout:1/679: dread d3/f39 [0,4194304] 0 2024-07-26T11:04:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:47 smithi144 ceph-mon[32129]: pgmap v530: 129 pgs: 129 active+clean; 992 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 154 MiB/s wr, 711 op/s 2024-07-26T11:04:47.720 INFO:tasks.workunit.client.0.smithi144.stdout:1/680: rename d3/d6/da/d13/d19/d3a/d69/l86 to d3/d6/d75/d88/ld8 0 2024-07-26T11:04:47.720 INFO:tasks.workunit.client.0.smithi144.stdout:1/681: truncate d3/d6/da/d13/d22/f8a 861362 0 2024-07-26T11:04:47.723 INFO:tasks.workunit.client.0.smithi144.stdout:1/682: creat d3/d6/dcc/fd9 x:0 0 0 2024-07-26T11:04:47.724 INFO:tasks.workunit.client.0.smithi144.stdout:1/683: creat d3/d6/da/d21/fda x:0 0 0 2024-07-26T11:04:47.724 INFO:tasks.workunit.client.0.smithi144.stdout:1/684: write d3/d6/da/d13/d22/f30 [1117334,95682] 0 2024-07-26T11:04:47.792 INFO:tasks.workunit.client.0.smithi144.stdout:7/540: sync 2024-07-26T11:04:47.793 INFO:tasks.workunit.client.0.smithi144.stdout:3/374: sync 2024-07-26T11:04:47.793 INFO:tasks.workunit.client.0.smithi144.stdout:9/582: sync 2024-07-26T11:04:47.793 INFO:tasks.workunit.client.0.smithi144.stdout:0/475: sync 2024-07-26T11:04:47.793 INFO:tasks.workunit.client.0.smithi144.stdout:2/556: sync 2024-07-26T11:04:47.793 INFO:tasks.workunit.client.0.smithi144.stdout:2/557: chown d0/d24/d48/fa1 0 1 2024-07-26T11:04:47.797 INFO:tasks.workunit.client.0.smithi144.stdout:0/476: truncate d0/d25/d5e/d1e/d30/d4b/f58 3598227 0 2024-07-26T11:04:47.828 INFO:tasks.workunit.client.0.smithi144.stdout:0/477: write d0/d25/d5e/f41 [1127199,98435] 0 2024-07-26T11:04:47.829 INFO:tasks.workunit.client.0.smithi144.stdout:0/478: mknod d0/c9a 0 2024-07-26T11:04:47.830 INFO:tasks.workunit.client.0.smithi144.stdout:0/479: creat d0/d3e/d45/f9b x:0 0 0 2024-07-26T11:04:47.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/480: mknod d0/d3e/c9c 0 2024-07-26T11:04:47.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/481: chown d0/d25/d5e/c2b 1227857 1 2024-07-26T11:04:47.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/482: mknod d0/d25/d5e/d1e/d30/d67/c9d 0 2024-07-26T11:04:47.850 INFO:tasks.workunit.client.0.smithi144.stdout:5/567: dwrite d1/d5/f6d [0,4194304] 0 2024-07-26T11:04:47.850 INFO:tasks.workunit.client.0.smithi144.stdout:9/583: dwrite d9/de/f46 [4194304,4194304] 0 2024-07-26T11:04:47.850 INFO:tasks.workunit.client.0.smithi144.stdout:9/584: write d9/de/d14/d3f/d5e/f62 [2404225,60194] 0 2024-07-26T11:04:47.858 INFO:tasks.workunit.client.0.smithi144.stdout:3/375: dwrite d1/f4f [0,4194304] 0 2024-07-26T11:04:47.859 INFO:tasks.workunit.client.0.smithi144.stdout:3/376: write d1/d28/d54/d26/d30/d4d/f55 [640066,17840] 0 2024-07-26T11:04:47.872 INFO:tasks.workunit.client.0.smithi144.stdout:3/377: mknod d1/d28/d54/c69 0 2024-07-26T11:04:47.873 INFO:tasks.workunit.client.0.smithi144.stdout:7/541: read d0/d6/d55/f6a [1174115,103231] 0 2024-07-26T11:04:47.873 INFO:tasks.workunit.client.0.smithi144.stdout:7/542: chown d0/d6/d9/d98/dbf/d2f/d47/f4f 31926 1 2024-07-26T11:04:47.875 INFO:tasks.workunit.client.0.smithi144.stdout:3/378: mkdir d1/d28/d6a 0 2024-07-26T11:04:47.875 INFO:tasks.workunit.client.0.smithi144.stdout:3/379: fsync d1/f34 0 2024-07-26T11:04:47.875 INFO:tasks.workunit.client.0.smithi144.stdout:5/568: write d1/d18/d24/d35/f3d [2342125,82474] 0 2024-07-26T11:04:47.875 INFO:tasks.workunit.client.0.smithi144.stdout:7/543: truncate d0/d6/d9/d98/dbf/d2f/f89 3090296 0 2024-07-26T11:04:47.875 INFO:tasks.workunit.client.0.smithi144.stdout:9/585: write d9/de/d14/d23/f3a [1517334,46510] 0 2024-07-26T11:04:47.875 INFO:tasks.workunit.client.0.smithi144.stdout:9/586: fsync d9/de/d4e/f93 0 2024-07-26T11:04:47.876 INFO:tasks.workunit.client.0.smithi144.stdout:3/380: mknod d1/d5/d5e/c6b 0 2024-07-26T11:04:47.877 INFO:tasks.workunit.client.0.smithi144.stdout:5/569: rename d1/d9/d44/c3a to d1/d5/dae/cc2 0 2024-07-26T11:04:47.878 INFO:tasks.workunit.client.0.smithi144.stdout:3/381: symlink d1/d28/d54/d26/d64/l6c 0 2024-07-26T11:04:47.879 INFO:tasks.workunit.client.0.smithi144.stdout:3/382: dread - d1/d28/d54/d3d/d42/d44/d48/f56 zero size 2024-07-26T11:04:47.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/570: mknod d1/d18/d24/d35/cc3 0 2024-07-26T11:04:47.879 INFO:tasks.workunit.client.0.smithi144.stdout:5/571: chown d1/d18/c89 3695592 1 2024-07-26T11:04:47.881 INFO:tasks.workunit.client.0.smithi144.stdout:5/572: mkdir d1/d18/dc4 0 2024-07-26T11:04:47.882 INFO:tasks.workunit.client.0.smithi144.stdout:5/573: mknod d1/d9/d44/d45/daa/cc5 0 2024-07-26T11:04:47.889 INFO:tasks.workunit.client.0.smithi144.stdout:5/574: write d1/f5e [617738,101077] 0 2024-07-26T11:04:47.892 INFO:tasks.workunit.client.0.smithi144.stdout:5/575: truncate d1/d18/d24/d35/f77 768843 0 2024-07-26T11:04:47.901 INFO:tasks.workunit.client.0.smithi144.stdout:0/483: dwrite d0/d25/d5e/f41 [0,4194304] 0 2024-07-26T11:04:47.908 INFO:tasks.workunit.client.0.smithi144.stdout:3/383: dread d1/d28/d54/d26/f2d [0,4194304] 0 2024-07-26T11:04:47.912 INFO:tasks.workunit.client.0.smithi144.stdout:0/484: unlink d0/d25/c7e 0 2024-07-26T11:04:47.913 INFO:tasks.workunit.client.0.smithi144.stdout:3/384: creat d1/d28/d54/d26/d64/d68/f6d x:0 0 0 2024-07-26T11:04:47.913 INFO:tasks.workunit.client.0.smithi144.stdout:3/385: chown d1/d28/d54/d26/d30/d4d/f55 7200 1 2024-07-26T11:04:47.915 INFO:tasks.workunit.client.0.smithi144.stdout:3/386: mkdir d1/d28/d54/d26/d30/d4d/d6e 0 2024-07-26T11:04:47.915 INFO:tasks.workunit.client.0.smithi144.stdout:7/544: dread d0/d6/d9/d45/f30 [0,4194304] 0 2024-07-26T11:04:47.915 INFO:tasks.workunit.client.0.smithi144.stdout:3/387: write d1/f32 [1642915,70471] 0 2024-07-26T11:04:47.916 INFO:tasks.workunit.client.0.smithi144.stdout:3/388: dread - d1/f39 zero size 2024-07-26T11:04:47.916 INFO:tasks.workunit.client.0.smithi144.stdout:3/389: rmdir d1/d28/d54 39 2024-07-26T11:04:47.917 INFO:tasks.workunit.client.0.smithi144.stdout:3/390: symlink d1/d28/d54/d3d/d42/d44/d60/l6f 0 2024-07-26T11:04:47.923 INFO:tasks.workunit.client.0.smithi144.stdout:7/545: dread d0/d6/d9/d45/f90 [0,4194304] 0 2024-07-26T11:04:47.924 INFO:tasks.workunit.client.0.smithi144.stdout:7/546: read - d0/d71/d95/dc4/fbc zero size 2024-07-26T11:04:47.924 INFO:tasks.workunit.client.0.smithi144.stdout:0/485: write d0/d25/d5e/d1e/d30/f87 [2613780,88258] 0 2024-07-26T11:04:47.925 INFO:tasks.workunit.client.0.smithi144.stdout:7/547: creat d0/d6/d9/d98/dbf/d2f/d53/fc8 x:0 0 0 2024-07-26T11:04:47.927 INFO:tasks.workunit.client.0.smithi144.stdout:9/587: dwrite d9/d3b/f64 [0,4194304] 0 2024-07-26T11:04:47.929 INFO:tasks.workunit.client.0.smithi144.stdout:9/588: mkdir d9/de/d14/d23/d39/d7f/dbc 0 2024-07-26T11:04:47.929 INFO:tasks.workunit.client.0.smithi144.stdout:9/589: write d9/de/d14/d23/f3a [3905661,43910] 0 2024-07-26T11:04:47.932 INFO:tasks.workunit.client.0.smithi144.stdout:6/672: dwrite d8/db/dbe/daa/fac [0,4194304] 0 2024-07-26T11:04:47.934 INFO:tasks.workunit.client.0.smithi144.stdout:6/673: creat d8/d50/fec x:0 0 0 2024-07-26T11:04:47.941 INFO:tasks.workunit.client.0.smithi144.stdout:3/391: write d1/f17 [1671767,100678] 0 2024-07-26T11:04:47.941 INFO:tasks.workunit.client.0.smithi144.stdout:3/392: dread - d1/f39 zero size 2024-07-26T11:04:47.945 INFO:tasks.workunit.client.0.smithi144.stdout:1/685: dwrite d3/d6/da/d13/d19/d3a/d69/fba [0,4194304] 0 2024-07-26T11:04:47.945 INFO:tasks.workunit.client.0.smithi144.stdout:1/686: chown d3/d6/da/d21/f4d 0 1 2024-07-26T11:04:47.949 INFO:tasks.workunit.client.0.smithi144.stdout:1/687: mkdir d3/d6/da/d13/d19/d3a/d3e/ddb 0 2024-07-26T11:04:47.949 INFO:tasks.workunit.client.0.smithi144.stdout:1/688: chown d3/d6/l61 0 1 2024-07-26T11:04:47.951 INFO:tasks.workunit.client.0.smithi144.stdout:5/576: dwrite d1/d18/d24/f2a [0,4194304] 0 2024-07-26T11:04:47.953 INFO:tasks.workunit.client.0.smithi144.stdout:1/689: mknod d3/d6/d9c/cdc 0 2024-07-26T11:04:47.953 INFO:tasks.workunit.client.0.smithi144.stdout:1/690: read d3/d6/da/d13/d22/f30 [129150,117963] 0 2024-07-26T11:04:47.958 INFO:tasks.workunit.client.0.smithi144.stdout:1/691: rename d3/d6/da/d97/fcd to d3/d6/da/d13/d19/fdd 0 2024-07-26T11:04:47.958 INFO:tasks.workunit.client.0.smithi144.stdout:1/692: chown d3/d6/fd 1637 1 2024-07-26T11:04:47.958 INFO:tasks.workunit.client.0.smithi144.stdout:7/548: write d0/d6/d9/d45/d57/f77 [6923031,64371] 0 2024-07-26T11:04:47.958 INFO:tasks.workunit.client.0.smithi144.stdout:7/549: link d0/d6/d9/d98/dbf/d2f/d68/c9a d0/d6/d9/d45/d57/cc9 0 2024-07-26T11:04:47.958 INFO:tasks.workunit.client.0.smithi144.stdout:7/550: chown d0/d6/d9/d45/d57 31700313 1 2024-07-26T11:04:47.959 INFO:tasks.workunit.client.0.smithi144.stdout:0/486: write d0/d25/d5e/d1e/f24 [1592793,66479] 0 2024-07-26T11:04:47.961 INFO:tasks.workunit.client.0.smithi144.stdout:0/487: creat d0/d25/d5e/d29/d92/f9e x:0 0 0 2024-07-26T11:04:47.962 INFO:tasks.workunit.client.0.smithi144.stdout:0/488: chown d0/d25/d5e/d29/d92 8451 1 2024-07-26T11:04:47.969 INFO:tasks.workunit.client.0.smithi144.stdout:9/590: dread d9/d3b/f78 [0,4194304] 0 2024-07-26T11:04:47.969 INFO:tasks.workunit.client.0.smithi144.stdout:9/591: chown d9/de/d14/la8 12121 1 2024-07-26T11:04:47.969 INFO:tasks.workunit.client.0.smithi144.stdout:0/489: rename d0/d25/d5e/d29 to d0/d3e/d45/d9f 0 2024-07-26T11:04:47.969 INFO:tasks.workunit.client.0.smithi144.stdout:0/490: truncate d0/d3e/d45/d9f/f6d 618209 0 2024-07-26T11:04:47.970 INFO:tasks.workunit.client.0.smithi144.stdout:3/393: dread d1/f34 [0,4194304] 0 2024-07-26T11:04:47.981 INFO:tasks.workunit.client.0.smithi144.stdout:5/577: dread d1/d9/d44/f41 [0,4194304] 0 2024-07-26T11:04:47.982 INFO:tasks.workunit.client.0.smithi144.stdout:6/674: write d8/d50/d98/fc5 [1384833,16896] 0 2024-07-26T11:04:47.984 INFO:tasks.workunit.client.0.smithi144.stdout:6/675: symlink d8/db/dbe/d5c/da7/led 0 2024-07-26T11:04:47.984 INFO:tasks.workunit.client.0.smithi144.stdout:6/676: chown d8/db/ccf 1310 1 2024-07-26T11:04:47.984 INFO:tasks.workunit.client.0.smithi144.stdout:5/578: mkdir d1/d9/d5b/d86/dc6 0 2024-07-26T11:04:47.985 INFO:tasks.workunit.client.0.smithi144.stdout:5/579: write d1/d18/f51 [389618,102140] 0 2024-07-26T11:04:47.985 INFO:tasks.workunit.client.0.smithi144.stdout:6/677: mknod d8/db/da6/cee 0 2024-07-26T11:04:47.985 INFO:tasks.workunit.client.0.smithi144.stdout:1/693: dread d3/d6/da/d13/d19/d3a/f4b [0,4194304] 0 2024-07-26T11:04:47.985 INFO:tasks.workunit.client.0.smithi144.stdout:1/694: fdatasync d3/d6/da/d13/d22/f3c 0 2024-07-26T11:04:47.986 INFO:tasks.workunit.client.0.smithi144.stdout:5/580: symlink d1/d9/d5b/lc7 0 2024-07-26T11:04:47.988 INFO:tasks.workunit.client.0.smithi144.stdout:6/678: rename d8/db/dbe/d52/c93 to d8/db/dbe/d52/cef 0 2024-07-26T11:04:47.990 INFO:tasks.workunit.client.0.smithi144.stdout:6/679: symlink d8/d42/lf0 0 2024-07-26T11:04:48.022 INFO:tasks.workunit.client.0.smithi144.stdout:6/680: dread d8/db/dbe/f63 [0,4194304] 0 2024-07-26T11:04:48.028 INFO:tasks.workunit.client.0.smithi144.stdout:6/681: creat d8/d42/dad/ff1 x:0 0 0 2024-07-26T11:04:48.028 INFO:tasks.workunit.client.0.smithi144.stdout:6/682: chown d8/db/dbe/f63 1 1 2024-07-26T11:04:48.029 INFO:tasks.workunit.client.0.smithi144.stdout:7/551: dwrite d0/d6/d9/d98/dbf/f79 [0,4194304] 0 2024-07-26T11:04:48.034 INFO:tasks.workunit.client.0.smithi144.stdout:6/683: rmdir d8/d15/d57/d5f 39 2024-07-26T11:04:48.034 INFO:tasks.workunit.client.0.smithi144.stdout:7/552: rename d0/f2d to d0/d6/d9/d98/dbf/d2f/d53/db6/fca 0 2024-07-26T11:04:48.035 INFO:tasks.workunit.client.0.smithi144.stdout:7/553: symlink d0/d6/d9/d98/dbf/d2f/d68/lcb 0 2024-07-26T11:04:48.038 INFO:tasks.workunit.client.0.smithi144.stdout:1/695: dread d3/d6/da/d13/d19/d3a/d3e/f6f [0,4194304] 0 2024-07-26T11:04:48.038 INFO:tasks.workunit.client.0.smithi144.stdout:1/696: readlink d3/d6/da/l2e 0 2024-07-26T11:04:48.038 INFO:tasks.workunit.client.0.smithi144.stdout:1/697: chown d3/d6/d9c/cdc 671 1 2024-07-26T11:04:48.038 INFO:tasks.workunit.client.0.smithi144.stdout:1/698: chown d3/d6/da/d70/fa2 1947793730 1 2024-07-26T11:04:48.039 INFO:tasks.workunit.client.0.smithi144.stdout:1/699: symlink d3/d6/dcc/lde 0 2024-07-26T11:04:48.040 INFO:tasks.workunit.client.0.smithi144.stdout:1/700: creat d3/d6/da/fdf x:0 0 0 2024-07-26T11:04:48.044 INFO:tasks.workunit.client.0.smithi144.stdout:9/592: dwrite d9/d3b/f7c [4194304,4194304] 0 2024-07-26T11:04:48.044 INFO:tasks.workunit.client.0.smithi144.stdout:9/593: rename d9/de/d14 to d9/de/d14/d79/db2/dbd 22 2024-07-26T11:04:48.045 INFO:tasks.workunit.client.0.smithi144.stdout:9/594: creat d9/de/d14/d23/d39/fbe x:0 0 0 2024-07-26T11:04:48.048 INFO:tasks.workunit.client.0.smithi144.stdout:6/684: dread d8/db/da6/fa3 [0,4194304] 0 2024-07-26T11:04:48.048 INFO:tasks.workunit.client.0.smithi144.stdout:9/595: creat d9/de/fbf x:0 0 0 2024-07-26T11:04:48.056 INFO:tasks.workunit.client.0.smithi144.stdout:9/596: creat d9/de/d14/d3f/d5e/fc0 x:0 0 0 2024-07-26T11:04:48.056 INFO:tasks.workunit.client.0.smithi144.stdout:9/597: fsync d9/de/d14/d3f/d5e/f62 0 2024-07-26T11:04:48.056 INFO:tasks.workunit.client.0.smithi144.stdout:5/581: dwrite d1/d18/d24/d27/d71/f99 [0,4194304] 0 2024-07-26T11:04:48.071 INFO:tasks.workunit.client.0.smithi144.stdout:0/491: dwrite d0/d25/d5e/d1e/d91/f96 [0,4194304] 0 2024-07-26T11:04:48.071 INFO:tasks.workunit.client.0.smithi144.stdout:0/492: fsync d0/d3e/d45/d4c/f8f 0 2024-07-26T11:04:48.071 INFO:tasks.workunit.client.0.smithi144.stdout:0/493: readlink d0/d25/d5e/d1e/d30/l4e 0 2024-07-26T11:04:48.073 INFO:tasks.workunit.client.0.smithi144.stdout:0/494: creat d0/d25/fa0 x:0 0 0 2024-07-26T11:04:48.073 INFO:tasks.workunit.client.0.smithi144.stdout:0/495: readlink d0/d3e/d45/d4c/l98 0 2024-07-26T11:04:48.080 INFO:tasks.workunit.client.0.smithi144.stdout:0/496: getdents d0/d3e/d45/d4c 0 2024-07-26T11:04:48.080 INFO:tasks.workunit.client.0.smithi144.stdout:5/582: dread d1/d9/f19 [0,4194304] 0 2024-07-26T11:04:48.080 INFO:tasks.workunit.client.0.smithi144.stdout:5/583: truncate d1/d9/d44/d45/d81/fbf 666744 0 2024-07-26T11:04:48.081 INFO:tasks.workunit.client.0.smithi144.stdout:0/497: link d0/d3e/d45/d9f/l31 d0/d3e/d45/d9f/d65/la1 0 2024-07-26T11:04:48.081 INFO:tasks.workunit.client.0.smithi144.stdout:0/498: readlink d0/d3e/d5f/l94 0 2024-07-26T11:04:48.081 INFO:tasks.workunit.client.0.smithi144.stdout:0/499: chown d0/c9a 143 1 2024-07-26T11:04:48.083 INFO:tasks.workunit.client.0.smithi144.stdout:0/500: creat d0/d25/d5e/d1e/d30/d6a/d6f/fa2 x:0 0 0 2024-07-26T11:04:48.084 INFO:tasks.workunit.client.0.smithi144.stdout:0/501: mknod d0/d3e/d5f/ca3 0 2024-07-26T11:04:48.085 INFO:tasks.workunit.client.0.smithi144.stdout:0/502: rename d0/d25/d5e/d1e/d30/d67/f83 to d0/d25/d5e/d1e/d30/d4b/fa4 0 2024-07-26T11:04:48.086 INFO:tasks.workunit.client.0.smithi144.stdout:0/503: stat d0/d25/d5e/f41 0 2024-07-26T11:04:48.090 INFO:tasks.workunit.client.0.smithi144.stdout:1/701: dwrite d3/fb2 [4194304,4194304] 0 2024-07-26T11:04:48.090 INFO:tasks.workunit.client.0.smithi144.stdout:1/702: fdatasync d3/d6/da/d13/d19/d3a/d69/db7/f8f 0 2024-07-26T11:04:48.090 INFO:tasks.workunit.client.0.smithi144.stdout:1/703: dread - d3/d6/da/d21/d5c/d62/d68/fb5 zero size 2024-07-26T11:04:48.091 INFO:tasks.workunit.client.0.smithi144.stdout:1/704: chown d3/d6/da/d13/d22/c59 501 1 2024-07-26T11:04:48.091 INFO:tasks.workunit.client.0.smithi144.stdout:0/504: rename d0/d25/d5e/d1e/d30/d4b/fa4 to d0/d25/fa5 0 2024-07-26T11:04:48.093 INFO:tasks.workunit.client.0.smithi144.stdout:1/705: unlink d3/l44 0 2024-07-26T11:04:48.093 INFO:tasks.workunit.client.0.smithi144.stdout:0/505: symlink d0/la6 0 2024-07-26T11:04:48.101 INFO:tasks.workunit.client.0.smithi144.stdout:6/685: dwrite d8/d42/f7c [0,4194304] 0 2024-07-26T11:04:48.101 INFO:tasks.workunit.client.0.smithi144.stdout:1/706: rename d3/d6/da/d13/d19/d3a/d69/d36/d37/fa1 to d3/d6/da/d13/d22/d5d/d9a/fe0 0 2024-07-26T11:04:48.104 INFO:tasks.workunit.client.0.smithi144.stdout:1/707: rmdir d3/d6/da/d13/d19/d3a/d69/d36/db6 39 2024-07-26T11:04:48.105 INFO:tasks.workunit.client.0.smithi144.stdout:6/686: mknod d8/dca/cf2 0 2024-07-26T11:04:48.105 INFO:tasks.workunit.client.0.smithi144.stdout:1/708: chown d3/d6/d9c/d9f/fd0 93404 1 2024-07-26T11:04:48.105 INFO:tasks.workunit.client.0.smithi144.stdout:1/709: write d3/d6/da/d21/d5c/d62/f7c [2366514,115327] 0 2024-07-26T11:04:48.107 INFO:tasks.workunit.client.0.smithi144.stdout:1/710: mkdir d3/d6/da/d13/d22/d5d/d9a/de1 0 2024-07-26T11:04:48.107 INFO:tasks.workunit.client.0.smithi144.stdout:1/711: dread - d3/d3d/fae zero size 2024-07-26T11:04:48.108 INFO:tasks.workunit.client.0.smithi144.stdout:6/687: creat d8/d15/d57/d5f/d88/ff3 x:0 0 0 2024-07-26T11:04:48.108 INFO:tasks.workunit.client.0.smithi144.stdout:1/712: fsync d3/d6/da/d13/d19/d3a/f4b 0 2024-07-26T11:04:48.110 INFO:tasks.workunit.client.0.smithi144.stdout:1/713: creat d3/d6/d9c/d9f/fe2 x:0 0 0 2024-07-26T11:04:48.115 INFO:tasks.workunit.client.0.smithi144.stdout:6/688: dread d8/db/dbe/f7e [0,4194304] 0 2024-07-26T11:04:48.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/689: unlink d8/d42/c8b 0 2024-07-26T11:04:48.119 INFO:tasks.workunit.client.0.smithi144.stdout:6/690: rmdir d8/db/da6 39 2024-07-26T11:04:48.122 INFO:tasks.workunit.client.0.smithi144.stdout:6/691: symlink d8/d42/lf4 0 2024-07-26T11:04:48.122 INFO:tasks.workunit.client.0.smithi144.stdout:9/598: dwrite d9/de/d14/d3f/d5e/f62 [0,4194304] 0 2024-07-26T11:04:48.123 INFO:tasks.workunit.client.0.smithi144.stdout:6/692: mkdir d8/dca/df5 0 2024-07-26T11:04:48.123 INFO:tasks.workunit.client.0.smithi144.stdout:6/693: fdatasync d8/d42/fcd 0 2024-07-26T11:04:48.123 INFO:tasks.workunit.client.0.smithi144.stdout:6/694: dread - d8/d50/d98/f9f zero size 2024-07-26T11:04:48.123 INFO:tasks.workunit.client.0.smithi144.stdout:9/599: getdents d9/de/d14/d23/d39 0 2024-07-26T11:04:48.124 INFO:tasks.workunit.client.0.smithi144.stdout:6/695: rmdir d8/db/dbe/d52/d58 39 2024-07-26T11:04:48.125 INFO:tasks.workunit.client.0.smithi144.stdout:9/600: creat d9/de/d14/d23/d39/fc1 x:0 0 0 2024-07-26T11:04:48.126 INFO:tasks.workunit.client.0.smithi144.stdout:0/506: write d0/d25/d5e/d1e/f3c [3307390,67771] 0 2024-07-26T11:04:48.126 INFO:tasks.workunit.client.0.smithi144.stdout:0/507: truncate d0/d25/d5e/f2c 32348 0 2024-07-26T11:04:48.126 INFO:tasks.workunit.client.0.smithi144.stdout:0/508: dread - d0/d25/d5e/d1e/d30/d6a/d6f/fa2 zero size 2024-07-26T11:04:48.126 INFO:tasks.workunit.client.0.smithi144.stdout:6/696: creat d8/db/ff6 x:0 0 0 2024-07-26T11:04:48.126 INFO:tasks.workunit.client.0.smithi144.stdout:0/509: chown d0/d3e/d45/d9f/d65/c8b 6989409 1 2024-07-26T11:04:48.127 INFO:tasks.workunit.client.0.smithi144.stdout:0/510: creat d0/d25/fa7 x:0 0 0 2024-07-26T11:04:48.127 INFO:tasks.workunit.client.0.smithi144.stdout:0/511: dread - d0/d25/d5e/d1e/d30/d6a/d6f/fa2 zero size 2024-07-26T11:04:48.129 INFO:tasks.workunit.client.0.smithi144.stdout:0/512: link d0/d25/d5e/d1e/d5c/f99 d0/d3e/d45/d9f/fa8 0 2024-07-26T11:04:48.129 INFO:tasks.workunit.client.0.smithi144.stdout:0/513: chown d0/d3e/f43 23857269 1 2024-07-26T11:04:48.129 INFO:tasks.workunit.client.0.smithi144.stdout:0/514: mknod d0/d25/d5e/d1e/d30/d4b/ca9 0 2024-07-26T11:04:48.131 INFO:tasks.workunit.client.0.smithi144.stdout:0/515: creat d0/d25/d5e/d1e/d30/d6a/faa x:0 0 0 2024-07-26T11:04:48.132 INFO:tasks.workunit.client.0.smithi144.stdout:0/516: mkdir d0/d25/d5e/d1e/d30/dab 0 2024-07-26T11:04:48.134 INFO:tasks.workunit.client.0.smithi144.stdout:0/517: stat d0/d25/d5e/d1e/d30/d4b/c8e 0 2024-07-26T11:04:48.134 INFO:tasks.workunit.client.0.smithi144.stdout:0/518: chown d0/d25/d5e/d1e/d30/d4b/l4f 7662500 1 2024-07-26T11:04:48.137 INFO:tasks.workunit.client.0.smithi144.stdout:0/519: mkdir d0/d25/d5e/d1e/d30/d6a/dac 0 2024-07-26T11:04:48.138 INFO:tasks.workunit.client.0.smithi144.stdout:5/584: dwrite d1/d18/f3b [0,4194304] 0 2024-07-26T11:04:48.138 INFO:tasks.workunit.client.0.smithi144.stdout:5/585: write d1/d9/d44/d45/fc1 [689950,60347] 0 2024-07-26T11:04:48.138 INFO:tasks.workunit.client.0.smithi144.stdout:5/586: stat d1/d5 0 2024-07-26T11:04:48.138 INFO:tasks.workunit.client.0.smithi144.stdout:0/520: creat d0/d3e/d45/fad x:0 0 0 2024-07-26T11:04:48.140 INFO:tasks.workunit.client.0.smithi144.stdout:5/587: mkdir d1/d9/d44/d87/dc8 0 2024-07-26T11:04:48.140 INFO:tasks.workunit.client.0.smithi144.stdout:5/588: fdatasync d1/f39 0 2024-07-26T11:04:48.140 INFO:tasks.workunit.client.0.smithi144.stdout:5/589: write d1/d18/d24/d35/f9c [634454,23940] 0 2024-07-26T11:04:48.141 INFO:tasks.workunit.client.0.smithi144.stdout:5/590: truncate d1/d9/d5b/f21 2998159 0 2024-07-26T11:04:48.143 INFO:tasks.workunit.client.0.smithi144.stdout:5/591: rmdir d1/d9/d44/d45/d98/db2 39 2024-07-26T11:04:48.147 INFO:tasks.workunit.client.0.smithi144.stdout:6/697: write d8/f86 [4191008,106110] 0 2024-07-26T11:04:48.149 INFO:tasks.workunit.client.0.smithi144.stdout:9/601: write d9/de/f4d [1295708,128017] 0 2024-07-26T11:04:48.149 INFO:tasks.workunit.client.0.smithi144.stdout:9/602: write d9/de/d14/fbb [922286,124590] 0 2024-07-26T11:04:48.158 INFO:tasks.workunit.client.0.smithi144.stdout:6/698: write d8/db/dbe/f72 [394107,120429] 0 2024-07-26T11:04:48.159 INFO:tasks.workunit.client.0.smithi144.stdout:9/603: dread d9/de/d14/d3f/f53 [0,4194304] 0 2024-07-26T11:04:48.164 INFO:tasks.workunit.client.0.smithi144.stdout:5/592: write d1/d9/f10 [4665930,93439] 0 2024-07-26T11:04:48.164 INFO:tasks.workunit.client.0.smithi144.stdout:6/699: creat d8/d50/de2/ff7 x:0 0 0 2024-07-26T11:04:48.166 INFO:tasks.workunit.client.0.smithi144.stdout:9/604: getdents d9/de/d14/d23 0 2024-07-26T11:04:48.166 INFO:tasks.workunit.client.0.smithi144.stdout:5/593: creat d1/d18/da4/db3/fc9 x:0 0 0 2024-07-26T11:04:48.170 INFO:tasks.workunit.client.0.smithi144.stdout:5/594: rmdir d1/d9/d5b/d86/dc6 0 2024-07-26T11:04:48.175 INFO:tasks.workunit.client.0.smithi144.stdout:1/714: dwrite d3/d6/d75/d88/da9/dac/fc4 [0,4194304] 0 2024-07-26T11:04:48.186 INFO:tasks.workunit.client.0.smithi144.stdout:1/715: write d3/d6/da/d78/f99 [113579,48241] 0 2024-07-26T11:04:48.187 INFO:tasks.workunit.client.0.smithi144.stdout:1/716: truncate d3/d6/f47 9242790 0 2024-07-26T11:04:48.188 INFO:tasks.workunit.client.0.smithi144.stdout:1/717: creat d3/d6/da/d21/fe3 x:0 0 0 2024-07-26T11:04:48.190 INFO:tasks.workunit.client.0.smithi144.stdout:1/718: rename d3/d6/da/d78/f99 to d3/d3d/d58/fe4 0 2024-07-26T11:04:48.191 INFO:tasks.workunit.client.0.smithi144.stdout:0/521: dwrite d0/d25/d5e/d1e/d5c/f99 [0,4194304] 0 2024-07-26T11:04:48.193 INFO:tasks.workunit.client.0.smithi144.stdout:0/522: rmdir d0/d25/d5e/d1e/d6c 39 2024-07-26T11:04:48.223 INFO:tasks.workunit.client.0.smithi144.stdout:1/719: read d3/d6/da/d21/f24 [1330748,20096] 0 2024-07-26T11:04:48.223 INFO:tasks.workunit.client.0.smithi144.stdout:1/720: write d3/d6/da/d78/f7e [910060,66958] 0 2024-07-26T11:04:48.226 INFO:tasks.workunit.client.0.smithi144.stdout:6/700: dwrite d8/d15/d26/f2e [0,4194304] 0 2024-07-26T11:04:48.226 INFO:tasks.workunit.client.0.smithi144.stdout:6/701: readlink d8/d15/d26/l7f 0 2024-07-26T11:04:48.231 INFO:tasks.workunit.client.0.smithi144.stdout:6/702: mkdir d8/d50/d92/dea/df8 0 2024-07-26T11:04:48.231 INFO:tasks.workunit.client.0.smithi144.stdout:5/595: dwrite d1/d9/f5a [0,4194304] 0 2024-07-26T11:04:48.231 INFO:tasks.workunit.client.0.smithi144.stdout:5/596: truncate d1/d9/d5b/f7d 1109167 0 2024-07-26T11:04:48.232 INFO:tasks.workunit.client.0.smithi144.stdout:1/721: dread d3/d6/da/d13/d22/f8a [0,4194304] 0 2024-07-26T11:04:48.234 INFO:tasks.workunit.client.0.smithi144.stdout:5/597: rename d1/d9/d5b/l5d to d1/d9/d44/d45/d98/db2/lca 0 2024-07-26T11:04:48.240 INFO:tasks.workunit.client.0.smithi144.stdout:1/722: symlink d3/d6/da/d13/d19/d3a/d69/d36/db6/le5 0 2024-07-26T11:04:48.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/703: write d8/d15/d57/d5f/d88/f8a [924385,105704] 0 2024-07-26T11:04:48.240 INFO:tasks.workunit.client.0.smithi144.stdout:5/598: creat d1/d18/d24/d27/fcb x:0 0 0 2024-07-26T11:04:48.242 INFO:tasks.workunit.client.0.smithi144.stdout:6/704: truncate d8/db/dbe/f51 446709 0 2024-07-26T11:04:48.246 INFO:tasks.workunit.client.0.smithi144.stdout:9/605: dwrite d9/d3b/f56 [0,4194304] 0 2024-07-26T11:04:48.247 INFO:tasks.workunit.client.0.smithi144.stdout:5/599: symlink d1/d9/d44/d45/d81/lcc 0 2024-07-26T11:04:48.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.135+0000 7f6ad7e00640 1 -- 172.21.15.144:0/951401073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad01057e0 msgr2=0x7f6ad0105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.135+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/951401073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad01057e0 0x7f6ad0105bf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4009860 tx=0x7f6ac402a340 comp rx=0 tx=0).stop 2024-07-26T11:04:48.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 -- 172.21.15.144:0/951401073 shutdown_connections 2024-07-26T11:04:48.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/951401073 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad010ba70 0x7f6ad01122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/951401073 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 0x7f6ad010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/951401073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad01057e0 0x7f6ad0105bf0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 -- 172.21.15.144:0/951401073 >> 172.21.15.144:0/951401073 conn(0x7f6ad01011f0 msgr2=0x7f6ad0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 -- 172.21.15.144:0/951401073 shutdown_connections 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.136+0000 7f6ad7e00640 1 -- 172.21.15.144:0/951401073 wait complete. 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.139+0000 7f6ad7e00640 1 Processor -- start 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.140+0000 7f6ad7e00640 1 -- start start 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 0x7f6ad019c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 0x7f6ad019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad010ba70 0x7f6ad01a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.254 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad7e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad0115720 con 0x7f6ad01057e0 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01155a0 con 0x7f6ad010b120 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01158a0 con 0x7f6ad010ba70 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad5400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 0x7f6ad019c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.141+0000 7f6ad5400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 0x7f6ad019c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55480/0 (socket says 172.21.15.144:55480) 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5400640 1 -- 172.21.15.144:0/1551322469 learned_addr learned my addr 172.21.15.144:0/1551322469 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad4a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 0x7f6ad019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad010ba70 0x7f6ad01a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5400640 1 -- 172.21.15.144:0/1551322469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad010ba70 msgr2=0x7f6ad01a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5400640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad010ba70 0x7f6ad01a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5400640 1 -- 172.21.15.144:0/1551322469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 msgr2=0x7f6ad019c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5400640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 0x7f6ad019c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad5400640 1 -- 172.21.15.144:0/1551322469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad01a20a0 con 0x7f6ad01057e0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.142+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 0x7f6ad019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.143+0000 7f6ad5400640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 0x7f6ad019c130 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4035040 tx=0x7f6ac4032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.143+0000 7f6ace000640 1 -- 172.21.15.144:0/1551322469 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4037040 con 0x7f6ad01057e0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.143+0000 7f6ace000640 1 -- 172.21.15.144:0/1551322469 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac4041070 con 0x7f6ad01057e0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.143+0000 7f6ace000640 1 -- 172.21.15.144:0/1551322469 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac40035f0 con 0x7f6ad01057e0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.143+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad01a2330 con 0x7f6ad01057e0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.143+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad01a27b0 con 0x7f6ad01057e0 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.152+0000 7f6adebfd740 1 Processor -- start 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.152+0000 7f6adebfd740 1 -- start start 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.153+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed2bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.153+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed2c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.153+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed2c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.153+0000 7f6adebfd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55dc1eca2b90 con 0x55dc1e966390 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.153+0000 7f6adebfd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55dc1ec92090 con 0x55dc1e95e1b0 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.153+0000 7f6adebfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dc1ec960d0 con 0x55dc1e987070 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.154+0000 7f6ad5e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed2bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.154+0000 7f6ad5e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed2bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:41378/0 (socket says 172.21.15.144:41378) 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.154+0000 7f6ad5e00640 1 -- 172.21.15.144:0/4097274093 learned_addr learned my addr 172.21.15.144:0/4097274093 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.154+0000 7f6ad5400640 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed2c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.155+0000 7f6ad5e00640 1 -- 172.21.15.144:0/4097274093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 msgr2=0x55dc1ed2c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.155+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed2c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.155+0000 7f6ad5e00640 1 -- 172.21.15.144:0/4097274093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 msgr2=0x55dc1ed2c870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.155+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed2c870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.155+0000 7f6ad5e00640 1 -- 172.21.15.144:0/4097274093 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1e9f2650 con 0x55dc1e987070 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.157+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed2bdf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009fd0 tx=0x7f6ac80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=153b41a675b9850f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.157+0000 7f6ace000640 1 -- 172.21.15.144:0/1551322469 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f6ac4004050 con 0x7f6ad01057e0 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.160+0000 7f6ace000640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6ab80b3dc0 0x7f6ab80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.160+0000 7f6ace000640 1 -- 172.21.15.144:0/1551322469 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6ac4103fc0 con 0x7f6ad01057e0 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.160+0000 7f6ac2600640 1 -- 172.21.15.144:0/4097274093 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8004d20 con 0x55dc1e987070 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.161+0000 7f6ac2600640 1 -- 172.21.15.144:0/4097274093 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac8004050 con 0x55dc1e987070 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.161+0000 7f6ac2600640 1 -- 172.21.15.144:0/4097274093 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8005240 con 0x55dc1e987070 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.162+0000 7f6adebfd740 1 -- 172.21.15.144:0/4097274093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 msgr2=0x55dc1ed2bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.162+0000 7f6adebfd740 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed2bdf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009fd0 tx=0x7f6ac80118d0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6adebfd740 1 -- 172.21.15.144:0/4097274093 shutdown_connections 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6adebfd740 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed2c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6adebfd740 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed2c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6adebfd740 1 --2- 172.21.15.144:0/4097274093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed2bdf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6adebfd740 1 -- 172.21.15.144:0/4097274093 >> 172.21.15.144:0/4097274093 conn(0x55dc1e97b460 msgr2=0x55dc1ed29ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6ad5400640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6ab80b3dc0 0x7f6ab80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.163+0000 7f6adebfd740 1 -- 172.21.15.144:0/4097274093 shutdown_connections 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.164+0000 7f6ad5400640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6ab80b3dc0 0x7f6ab80b62d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55dc1ed2aa70 tx=0x7f6ac402a340 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.164+0000 7f6adebfd740 1 -- 172.21.15.144:0/4097274093 wait complete. 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.165+0000 7f6adebfd740 1 -- start start 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.165+0000 7f6adebfd740 1 Processor -- start 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.166+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed40500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.166+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed40ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.166+0000 7f6ad5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed40500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6ad5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed40500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51920/0 (socket says 172.21.15.144:51920) 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6ad5e00640 1 -- 172.21.15.144:0/2400424923 learned_addr learned my addr 172.21.15.144:0/2400424923 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed45b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55dc1eaf5700 con 0x55dc1e966390 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55dc1ec2e660 con 0x55dc1e95e1b0 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dc1ed3a5d0 con 0x55dc1e987070 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6adebfd740 10 client.0 _ll_register_callbacks cb 0x55dc1ec3e110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.167+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed40ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed45b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 -- 172.21.15.144:0/2400424923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 msgr2=0x55dc1ed45b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed45b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 -- 172.21.15.144:0/2400424923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 msgr2=0x55dc1ed40ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed40ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1ec45800 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed40500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8005c40 tx=0x7f6ac8009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3ada6e08c9a8803 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.168+0000 7f6ab6a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.169+0000 7f6ab6a00640 1 -- 172.21.15.144:0/2400424923 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8004c10 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.169+0000 7f6ab6a00640 1 -- 172.21.15.144:0/2400424923 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6ac801f040 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.169+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ac8004050 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.169+0000 7f6ab6a00640 1 -- 172.21.15.144:0/2400424923 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6ac8004c10 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.169+0000 7f6adebfd740 10 client.25847 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.169+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc1ed4ea50 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6ab6a00640 1 -- 172.21.15.144:0/2400424923 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac8004050 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6ab6a00640 1 -- 172.21.15.144:0/2400424923 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6ac8005040 con 0x55dc1e95e1b0 2024-07-26T11:04:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6adebfd740 10 client.25847 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6adebfd740 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6adebfd740 20 client.25847 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e9459f0 0x55dc1ed51db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6adebfd740 4 client.25847 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.170+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc1eca4ae0 con 0x55dc1e9459f0 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.171+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e9459f0 0x55dc1ed51db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.172+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e9459f0 0x55dc1ed51db0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1bc61a9c670fca5 server_cookie=1195cb5cbc5efbe in_seq=0 out_seq=0 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.172+0000 7f6ab6a00640 10 client.25847 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.172+0000 7f6ab6a00640 1 -- 172.21.15.144:0/2400424923 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc1eca4ae0 con 0x55dc1e9459f0 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.173+0000 7f6ab6a00640 10 client.25847 handle_command_reply: tid=0 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.173+0000 7f6adebfd740 1 client.25847 shutdown 2024-07-26T11:04:48.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 msgr2=0x55dc1ed40500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed40500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8005c40 tx=0x7f6ac8009c80 comp rx=0 tx=0).stop 2024-07-26T11:04:48.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 shutdown_connections 2024-07-26T11:04:48.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e9459f0 0x55dc1ed51db0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed45b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e966390 0x55dc1ed40ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 --2- 172.21.15.144:0/2400424923 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed40500 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 >> 172.21.15.144:0/2400424923 conn(0x55dc1e97b460 msgr2=0x55dc1ed2ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6ab6a00640 0 client.25847 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.176+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 shutdown_connections 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.177+0000 7f6adebfd740 1 -- 172.21.15.144:0/2400424923 wait complete. 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.178+0000 7f6adebfd740 20 client.25847 trim_cache size 0 max 16384 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.218+0000 7f6adebfd740 1 Processor -- start 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 -- start start 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e966390 0x55dc1ed398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 0x55dc1ed2d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01b1050 con 0x55dc1e95e1b0 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01ae920 con 0x55dc1e987070 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.219+0000 7f6adebfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01ac730 con 0x55dc1e966390 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 0x55dc1ed2d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 0x55dc1ed2d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51936/0 (socket says 172.21.15.144:51936) 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 -- 172.21.15.144:0/1386040914 learned_addr learned my addr 172.21.15.144:0/1386040914 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e966390 0x55dc1ed398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 -- 172.21.15.144:0/1386040914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e966390 msgr2=0x55dc1ed398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e966390 0x55dc1ed398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 -- 172.21.15.144:0/1386040914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 msgr2=0x55dc1ed422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.220+0000 7f6ad5e00640 1 -- 172.21.15.144:0/1386040914 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1ed3e800 con 0x55dc1e987070 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.221+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 0x55dc1ed2d7e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009fd0 tx=0x7f6ac80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95289a8664d1e2ec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.221+0000 7f6ac1c00640 1 -- 172.21.15.144:0/1386040914 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8004570 con 0x55dc1e987070 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.221+0000 7f6ac1c00640 1 -- 172.21.15.144:0/1386040914 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac802c070 con 0x55dc1e987070 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.221+0000 7f6ac1c00640 1 -- 172.21.15.144:0/1386040914 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac800f860 con 0x55dc1e987070 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.222+0000 7f6adebfd740 1 -- 172.21.15.144:0/1386040914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 msgr2=0x55dc1ed2d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.222+0000 7f6adebfd740 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 0x55dc1ed2d7e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009fd0 tx=0x7f6ac80118d0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 -- 172.21.15.144:0/1386040914 shutdown_connections 2024-07-26T11:04:48.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e987070 0x55dc1ed2d7e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e966390 0x55dc1ed398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 --2- 172.21.15.144:0/1386040914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 -- 172.21.15.144:0/1386040914 >> 172.21.15.144:0/1386040914 conn(0x55dc1e97b460 msgr2=0x55dc1ed2ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 -- 172.21.15.144:0/1386040914 shutdown_connections 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.223+0000 7f6adebfd740 1 -- 172.21.15.144:0/1386040914 wait complete. 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.224+0000 7f6adebfd740 1 -- start start 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.224+0000 7f6adebfd740 1 Processor -- start 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.225+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed56a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.225+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 0x55dc1ed56f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.225+0000 7f6adebfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6adebfd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01acde0 con 0x55dc1e95e1b0 2024-07-26T11:04:48.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6adebfd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01ae250 con 0x55dc1e966390 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6adebfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01ad4b0 con 0x55dc1e987070 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6adebfd740 10 client.0 _ll_register_callbacks cb 0x7f6ac8004220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6ad4a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 0x55dc1ed56f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6ad4a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 0x55dc1ed56f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51952/0 (socket says 172.21.15.144:51952) 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6ad4a00640 1 -- 172.21.15.144:0/3528021473 learned_addr learned my addr 172.21.15.144:0/3528021473 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "no active scrubs running", 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": {} 2024-07-26T11:04:48.268 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.226+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed56a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.227+0000 7f6ad4a00640 1 -- 172.21.15.144:0/3528021473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 msgr2=0x55dc1ed574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.227+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.227+0000 7f6ad4a00640 1 -- 172.21.15.144:0/3528021473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 msgr2=0x55dc1ed56a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.227+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed56a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.227+0000 7f6ad4a00640 1 -- 172.21.15.144:0/3528021473 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1edc94c0 con 0x55dc1e966390 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.228+0000 7f6ad4a00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 0x55dc1ed56f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6ad006b560 tx=0x7f6abc005410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=730adb65cf6a5bbc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.228+0000 7f6ab6a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:48.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.228+0000 7f6ab6a00640 1 -- 172.21.15.144:0/3528021473 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc004e60 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.228+0000 7f6ab6a00640 1 -- 172.21.15.144:0/3528021473 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6abc0035f0 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.228+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ac802c070 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.229+0000 7f6ab6a00640 1 -- 172.21.15.144:0/3528021473 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6abc02e050 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.229+0000 7f6adebfd740 10 client.25853 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.229+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc1edca7b0 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.229+0000 7f6ab6a00640 1 -- 172.21.15.144:0/3528021473 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac802c070 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.231+0000 7f6ab6a00640 1 -- 172.21.15.144:0/3528021473 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6abc025070 con 0x55dc1e966390 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.231+0000 7f6adebfd740 10 client.25853 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.231+0000 7f6adebfd740 10 client.25853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.231+0000 7f6adebfd740 20 client.25853 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:48.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.231+0000 7f6adebfd740 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e91ada0 0x55dc1edcd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.232+0000 7f6adebfd740 4 client.25853 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.232+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc1eca0580 con 0x55dc1e91ada0 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.232+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e91ada0 0x55dc1edcd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.233+0000 7f6ab6a00640 10 client.25853 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.233+0000 7f6ad5e00640 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e91ada0 0x55dc1edcd000 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0507214a1a54010 server_cookie=ffb87785507fb588 in_seq=0 out_seq=0 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6ab6a00640 1 -- 172.21.15.144:0/3528021473 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dc1eca0580 con 0x55dc1e91ada0 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6ab6a00640 10 client.25853 handle_command_reply: tid=0 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 client.25853 shutdown 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 msgr2=0x55dc1ed56f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 0x55dc1ed56f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6ad006b560 tx=0x7f6abc005410 comp rx=0 tx=0).stop 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 shutdown_connections 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55dc1e91ada0 0x55dc1edcd000 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc1e987070 0x55dc1ed574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc1e966390 0x55dc1ed56f60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 --2- 172.21.15.144:0/3528021473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc1e95e1b0 0x55dc1ed56a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.236+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 >> 172.21.15.144:0/3528021473 conn(0x55dc1e97b460 msgr2=0x55dc1ed2d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.237+0000 7f6ab6a00640 0 client.25853 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.237+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 shutdown_connections 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.237+0000 7f6adebfd740 1 -- 172.21.15.144:0/3528021473 wait complete. 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.237+0000 7f6adebfd740 20 client.25853 trim_cache size 0 max 16384 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.238+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6ab80b3dc0 msgr2=0x7f6ab80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.238+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6ab80b3dc0 0x7f6ab80b62d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55dc1ed2aa70 tx=0x7f6ac402a340 comp rx=0 tx=0).stop 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.238+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 msgr2=0x7f6ad019c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.238+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 0x7f6ad019c130 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4035040 tx=0x7f6ac4032d50 comp rx=0 tx=0).stop 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 shutdown_connections 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6ab80b3dc0 0x7f6ab80b62d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6ad010ba70 0x7f6ad01a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6ad010b120 0x7f6ad019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 --2- 172.21.15.144:0/1551322469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad01057e0 0x7f6ad019c130 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:48.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 >> 172.21.15.144:0/1551322469 conn(0x7f6ad01011f0 msgr2=0x7f6ad0103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:48.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.239+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 shutdown_connections 2024-07-26T11:04:48.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:48.240+0000 7f6ad7e00640 1 -- 172.21.15.144:0/1551322469 wait complete. 2024-07-26T11:04:48.273 INFO:tasks.workunit.client.0.smithi144.stdout:0/523: dwrite d0/d25/fa5 [0,4194304] 0 2024-07-26T11:04:48.275 INFO:tasks.workunit.client.0.smithi144.stdout:0/524: getdents d0/d25 0 2024-07-26T11:04:48.292 INFO:tasks.workunit.client.0.smithi144.stdout:1/723: dwrite d3/d6/fa4 [0,4194304] 0 2024-07-26T11:04:48.296 INFO:tasks.workunit.client.0.smithi144.stdout:6/705: dwrite d8/db/dbe/d5c/f7b [0,4194304] 0 2024-07-26T11:04:48.300 INFO:tasks.workunit.client.0.smithi144.stdout:5/600: dread d1/d18/d24/f2a [0,4194304] 0 2024-07-26T11:04:48.301 INFO:tasks.workunit.client.0.smithi144.stdout:1/724: creat d3/d6/da/d13/d22/d5d/d9a/fe6 x:0 0 0 2024-07-26T11:04:48.301 INFO:tasks.workunit.client.0.smithi144.stdout:1/725: truncate d3/d6/da/d21/f24 4626897 0 2024-07-26T11:04:48.301 INFO:tasks.workunit.client.0.smithi144.stdout:0/525: dread d0/d25/d5e/d1e/d91/f96 [0,4194304] 0 2024-07-26T11:04:48.301 INFO:tasks.workunit.client.0.smithi144.stdout:5/601: truncate d1/d9/d44/d85/fbc 127972 0 2024-07-26T11:04:48.301 INFO:tasks.workunit.client.0.smithi144.stdout:0/526: write d0/d25/fa0 [394478,63202] 0 2024-07-26T11:04:48.302 INFO:tasks.workunit.client.0.smithi144.stdout:1/726: creat d3/d6/da/d13/d22/fe7 x:0 0 0 2024-07-26T11:04:48.302 INFO:tasks.workunit.client.0.smithi144.stdout:0/527: getdents d0/d25/d5e/d1e/d30/d6a/d6f 0 2024-07-26T11:04:48.303 INFO:tasks.workunit.client.0.smithi144.stdout:5/602: mknod d1/d9/d44/d45/d98/ccd 0 2024-07-26T11:04:48.303 INFO:tasks.workunit.client.0.smithi144.stdout:5/603: chown d1/c47 44 1 2024-07-26T11:04:48.304 INFO:tasks.workunit.client.0.smithi144.stdout:1/727: creat d3/d6/da/d70/dd6/fe8 x:0 0 0 2024-07-26T11:04:48.305 INFO:tasks.workunit.client.0.smithi144.stdout:9/606: dwrite d9/fc [0,4194304] 0 2024-07-26T11:04:48.305 INFO:tasks.workunit.client.0.smithi144.stdout:9/607: fdatasync d9/de/d14/f2c 0 2024-07-26T11:04:48.306 INFO:tasks.workunit.client.0.smithi144.stdout:0/528: truncate d0/f32 1774088 0 2024-07-26T11:04:48.309 INFO:tasks.workunit.client.0.smithi144.stdout:5/604: creat d1/d9/d44/fce x:0 0 0 2024-07-26T11:04:48.309 INFO:tasks.workunit.client.0.smithi144.stdout:1/728: creat d3/d3d/fe9 x:0 0 0 2024-07-26T11:04:48.312 INFO:tasks.workunit.client.0.smithi144.stdout:9/608: symlink d9/de/lc2 0 2024-07-26T11:04:48.313 INFO:tasks.workunit.client.0.smithi144.stdout:5/605: link d1/d9/l8e d1/d18/da4/lcf 0 2024-07-26T11:04:48.314 INFO:tasks.workunit.client.0.smithi144.stdout:1/729: mkdir d3/d6/dc0/dea 0 2024-07-26T11:04:48.314 INFO:tasks.workunit.client.0.smithi144.stdout:1/730: chown d3/d6/da/d21/d5c/d62/f7c 1277343 1 2024-07-26T11:04:48.315 INFO:tasks.workunit.client.0.smithi144.stdout:9/609: symlink d9/de/d14/d3f/d5e/lc3 0 2024-07-26T11:04:48.315 INFO:tasks.workunit.client.0.smithi144.stdout:9/610: read - d9/de/d14/d79/f82 zero size 2024-07-26T11:04:48.315 INFO:tasks.workunit.client.0.smithi144.stdout:9/611: write d9/de/f15 [304928,99276] 0 2024-07-26T11:04:48.315 INFO:tasks.workunit.client.0.smithi144.stdout:5/606: link d1/d5/dae/cc2 d1/d9/d44/d45/d98/cd0 0 2024-07-26T11:04:48.316 INFO:tasks.workunit.client.0.smithi144.stdout:1/731: link d3/d6/da/d21/d67/f95 d3/d6/da/d13/feb 0 2024-07-26T11:04:48.317 INFO:tasks.workunit.client.0.smithi144.stdout:9/612: mknod d9/de/d14/d23/cc4 0 2024-07-26T11:04:48.317 INFO:tasks.workunit.client.0.smithi144.stdout:9/613: fdatasync d9/d3b/d81/fb4 0 2024-07-26T11:04:48.321 INFO:tasks.workunit.client.0.smithi144.stdout:5/607: truncate d1/d9/f19 205343 0 2024-07-26T11:04:48.321 INFO:tasks.workunit.client.0.smithi144.stdout:9/614: mknod d9/cc5 0 2024-07-26T11:04:48.321 INFO:tasks.workunit.client.0.smithi144.stdout:1/732: symlink d3/d6/da/d13/d19/d3a/d69/d36/d37/lec 0 2024-07-26T11:04:48.323 INFO:tasks.workunit.client.0.smithi144.stdout:1/733: symlink d3/d6/dc0/led 0 2024-07-26T11:04:48.328 INFO:tasks.workunit.client.0.smithi144.stdout:1/734: rename d3/d6/da/d13/d22/c59 to d3/d6/dc0/cee 0 2024-07-26T11:04:48.330 INFO:tasks.workunit.client.0.smithi144.stdout:1/735: creat d3/d6/da/d13/d19/d3a/fef x:0 0 0 2024-07-26T11:04:48.331 INFO:tasks.workunit.client.0.smithi144.stdout:6/706: write d8/d15/d26/f28 [710739,50689] 0 2024-07-26T11:04:48.331 INFO:tasks.workunit.client.0.smithi144.stdout:6/707: fdatasync f2 0 2024-07-26T11:04:48.331 INFO:tasks.workunit.client.0.smithi144.stdout:6/708: fdatasync d8/db/dbe/d52/d58/f97 0 2024-07-26T11:04:48.332 INFO:tasks.workunit.client.0.smithi144.stdout:1/736: creat d3/d6/da/d13/d19/d3a/d3e/ff0 x:0 0 0 2024-07-26T11:04:48.342 INFO:tasks.workunit.client.0.smithi144.stdout:6/709: mknod d8/d15/cf9 0 2024-07-26T11:04:48.343 INFO:tasks.workunit.client.0.smithi144.stdout:1/737: symlink d3/d6/da/d21/lf1 0 2024-07-26T11:04:48.348 INFO:tasks.workunit.client.0.smithi144.stdout:4/593: sync 2024-07-26T11:04:48.349 INFO:tasks.workunit.client.0.smithi144.stdout:8/539: sync 2024-07-26T11:04:48.350 INFO:tasks.workunit.client.0.smithi144.stdout:1/738: unlink d3/d6/da/d13/d19/d3a/d69/f28 0 2024-07-26T11:04:48.350 INFO:tasks.workunit.client.0.smithi144.stdout:1/739: fdatasync d3/d6/da/d21/d5c/d62/d68/f7b 0 2024-07-26T11:04:48.350 INFO:tasks.workunit.client.0.smithi144.stdout:4/594: creat d9/d7b/dc4/fc7 x:0 0 0 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:3/394: sync 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:2/558: sync 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:3/395: chown d1/d28/d6a 3977 1 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:3/396: stat d1/c51 0 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:2/559: truncate d0/d24/d48/d63/fa7 2037606 0 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:2/560: chown d0/d24/d48/d6d/d28/l9b 1 1 2024-07-26T11:04:48.351 INFO:tasks.workunit.client.0.smithi144.stdout:3/397: write d1/f39 [836938,94386] 0 2024-07-26T11:04:48.352 INFO:tasks.workunit.client.0.smithi144.stdout:7/554: sync 2024-07-26T11:04:48.352 INFO:tasks.workunit.client.0.smithi144.stdout:7/555: read d0/d6/d9/d45/f90 [521142,118848] 0 2024-07-26T11:04:48.352 INFO:tasks.workunit.client.0.smithi144.stdout:1/740: rmdir d3/d6/da/d13/d19/d3a 39 2024-07-26T11:04:48.352 INFO:tasks.workunit.client.0.smithi144.stdout:4/595: getdents d9/d26/d3c/d6f/d8c 0 2024-07-26T11:04:48.353 INFO:tasks.workunit.client.0.smithi144.stdout:8/540: mkdir d1a/d32/d49/d73/d2c/d38/d84/db0/dba 0 2024-07-26T11:04:48.353 INFO:tasks.workunit.client.0.smithi144.stdout:4/596: chown d9/d26/d48/d50/l61 462 1 2024-07-26T11:04:48.355 INFO:tasks.workunit.client.0.smithi144.stdout:1/741: creat d3/d6/d9c/d9f/ff2 x:0 0 0 2024-07-26T11:04:48.357 INFO:tasks.workunit.client.0.smithi144.stdout:4/597: creat d9/d26/d48/d98/fc8 x:0 0 0 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/742: symlink d3/d6/dc0/lf3 0 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/743: truncate d3/d6/da/d13/d22/d5d/d9a/fe0 78570 0 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/744: read - d3/d6/d9c/d9f/fe2 zero size 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/745: read - d3/d6/da/d70/fbd zero size 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/746: dread - d3/d6/da/d13/d19/d3a/d69/fd4 zero size 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/747: chown d3/d6/da/d13/d19/d3a/d3e/f50 1 1 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:1/748: chown d3/d6/da/cf 5176 1 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:2/561: rename d0/d24/d2b/f9f to d0/d6/d34/faf 0 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:2/562: chown d0/d24/d48/d6d/d28/d2e/d5d/d61 2 1 2024-07-26T11:04:48.360 INFO:tasks.workunit.client.0.smithi144.stdout:3/398: symlink d1/d28/d54/d26/d37/l70 0 2024-07-26T11:04:48.361 INFO:tasks.workunit.client.0.smithi144.stdout:3/399: mknod d1/d28/d54/d26/d64/d68/c71 0 2024-07-26T11:04:48.371 INFO:tasks.workunit.client.0.smithi144.stdout:0/529: dwrite d0/d3e/f43 [0,4194304] 0 2024-07-26T11:04:48.372 INFO:tasks.workunit.client.0.smithi144.stdout:0/530: stat d0/d25/d5e/d1e/l40 0 2024-07-26T11:04:48.372 INFO:tasks.workunit.client.0.smithi144.stdout:4/598: rename d9/c2c to d9/d26/d48/d98/cc9 0 2024-07-26T11:04:48.373 INFO:tasks.workunit.client.0.smithi144.stdout:3/400: link d1/d28/d54/d26/d37/l38 d1/d28/d54/d3d/d42/l72 0 2024-07-26T11:04:48.373 INFO:tasks.workunit.client.0.smithi144.stdout:3/401: chown d1/d28/d54/d26/d37/c62 1844 1 2024-07-26T11:04:48.374 INFO:tasks.workunit.client.0.smithi144.stdout:3/402: write d1/d28/d54/f4a [872088,5495] 0 2024-07-26T11:04:48.379 INFO:tasks.workunit.client.0.smithi144.stdout:9/615: dwrite d9/de/d14/d23/f50 [0,4194304] 0 2024-07-26T11:04:48.379 INFO:tasks.workunit.client.0.smithi144.stdout:3/403: symlink d1/d28/d54/l73 0 2024-07-26T11:04:48.381 INFO:tasks.workunit.client.0.smithi144.stdout:9/616: symlink d9/d3b/d76/lc6 0 2024-07-26T11:04:48.381 INFO:tasks.workunit.client.0.smithi144.stdout:4/599: mknod d9/da/cca 0 2024-07-26T11:04:48.382 INFO:tasks.workunit.client.0.smithi144.stdout:3/404: mkdir d1/d28/d54/d26/d64/d68/d74 0 2024-07-26T11:04:48.385 INFO:tasks.workunit.client.0.smithi144.stdout:9/617: mknod d9/de/d14/d79/db2/cc7 0 2024-07-26T11:04:48.385 INFO:tasks.workunit.client.0.smithi144.stdout:4/600: mknod d9/d26/d3c/d6f/d8c/ccb 0 2024-07-26T11:04:48.386 INFO:tasks.workunit.client.0.smithi144.stdout:3/405: mkdir d1/d5/d5e/d75 0 2024-07-26T11:04:48.386 INFO:tasks.workunit.client.0.smithi144.stdout:3/406: chown d1/d28/d54/d3d/d42/d44/f67 410967 1 2024-07-26T11:04:48.388 INFO:tasks.workunit.client.0.smithi144.stdout:3/407: rename d1/d28/c59 to d1/d28/d6a/c76 0 2024-07-26T11:04:48.388 INFO:tasks.workunit.client.0.smithi144.stdout:3/408: fsync d1/d28/d54/d3d/d42/d44/f4e 0 2024-07-26T11:04:48.388 INFO:tasks.workunit.client.0.smithi144.stdout:3/409: fsync d1/f17 0 2024-07-26T11:04:48.391 INFO:tasks.workunit.client.0.smithi144.stdout:2/563: dread d0/f3d [0,4194304] 0 2024-07-26T11:04:48.391 INFO:tasks.workunit.client.0.smithi144.stdout:2/564: chown d0/d6/d9 6988 1 2024-07-26T11:04:48.410 INFO:tasks.workunit.client.0.smithi144.stdout:5/608: dwrite d1/d18/d24/d27/d71/f80 [0,4194304] 0 2024-07-26T11:04:48.410 INFO:tasks.workunit.client.0.smithi144.stdout:1/749: dread d3/d6/da/d13/d19/d3a/d69/d36/f55 [0,4194304] 0 2024-07-26T11:04:48.410 INFO:tasks.workunit.client.0.smithi144.stdout:1/750: chown d3/d6/da/d78/ca3 28761 1 2024-07-26T11:04:48.413 INFO:tasks.workunit.client.0.smithi144.stdout:9/618: dread d9/d3b/d5d/fac [0,4194304] 0 2024-07-26T11:04:48.413 INFO:tasks.workunit.client.0.smithi144.stdout:1/751: creat d3/d6/dc0/dea/ff4 x:0 0 0 2024-07-26T11:04:48.414 INFO:tasks.workunit.client.0.smithi144.stdout:8/541: dwrite d1a/d32/f48 [0,4194304] 0 2024-07-26T11:04:48.418 INFO:tasks.workunit.client.0.smithi144.stdout:6/710: dwrite d8/db/da6/fc2 [0,4194304] 0 2024-07-26T11:04:48.419 INFO:tasks.workunit.client.0.smithi144.stdout:9/619: creat d9/de/d14/d79/db2/fc8 x:0 0 0 2024-07-26T11:04:48.419 INFO:tasks.workunit.client.0.smithi144.stdout:1/752: mkdir d3/d6/da/d13/d19/d3a/d69/df5 0 2024-07-26T11:04:48.419 INFO:tasks.workunit.client.0.smithi144.stdout:1/753: chown d3/d6/dcc/lde 7531 1 2024-07-26T11:04:48.419 INFO:tasks.workunit.client.0.smithi144.stdout:1/754: write d3/d6/da/d13/d22/f3c [1141914,43769] 0 2024-07-26T11:04:48.420 INFO:tasks.workunit.client.0.smithi144.stdout:8/542: rename d1a/d32/d6c/c63 to d1a/d32/cbb 0 2024-07-26T11:04:48.422 INFO:tasks.workunit.client.0.smithi144.stdout:6/711: symlink d8/d15/d57/d5f/dd3/lfa 0 2024-07-26T11:04:48.423 INFO:tasks.workunit.client.0.smithi144.stdout:6/712: dread - d8/d42/dad/ff1 zero size 2024-07-26T11:04:48.423 INFO:tasks.workunit.client.0.smithi144.stdout:1/755: unlink d3/d6/da/d21/d5c/d62/d68/f7b 0 2024-07-26T11:04:48.425 INFO:tasks.workunit.client.0.smithi144.stdout:8/543: rename d1a/d32/d6c/d3c/c8f to d1a/d32/d49/d73/d2c/d38/d58/da3/db1/cbc 0 2024-07-26T11:04:48.428 INFO:tasks.workunit.client.0.smithi144.stdout:1/756: rmdir d3/d6/da/d78 39 2024-07-26T11:04:48.428 INFO:tasks.workunit.client.0.smithi144.stdout:8/544: mkdir d1a/d32/d49/d73/d2c/d38/dbd 0 2024-07-26T11:04:48.429 INFO:tasks.workunit.client.0.smithi144.stdout:1/757: write d3/d6/da/d70/fbd [90693,122459] 0 2024-07-26T11:04:48.429 INFO:tasks.workunit.client.0.smithi144.stdout:6/713: symlink d8/d15/d26/lfb 0 2024-07-26T11:04:48.431 INFO:tasks.workunit.client.0.smithi144.stdout:8/545: mkdir d1a/d32/d49/d73/d2c/d9d/d74/dac/dbe 0 2024-07-26T11:04:48.432 INFO:tasks.workunit.client.0.smithi144.stdout:6/714: mknod d8/db/dbe/d52/ddb/de3/cfc 0 2024-07-26T11:04:48.432 INFO:tasks.workunit.client.0.smithi144.stdout:6/715: dread - d8/d50/de2/ff7 zero size 2024-07-26T11:04:48.434 INFO:tasks.workunit.client.0.smithi144.stdout:6/716: link d8/db/dbe/d52/l90 d8/d50/d92/dea/df8/lfd 0 2024-07-26T11:04:48.438 INFO:tasks.workunit.client.0.smithi144.stdout:2/565: write d0/d6/d34/f8f [2529371,85360] 0 2024-07-26T11:04:48.442 INFO:tasks.workunit.client.0.smithi144.stdout:3/410: dread d1/f39 [0,4194304] 0 2024-07-26T11:04:48.444 INFO:tasks.workunit.client.0.smithi144.stdout:3/411: creat d1/f77 x:0 0 0 2024-07-26T11:04:48.446 INFO:tasks.workunit.client.0.smithi144.stdout:3/412: rename d1/d5/d5e/d75 to d1/d28/d54/d26/d30/d78 0 2024-07-26T11:04:48.449 INFO:tasks.workunit.client.0.smithi144.stdout:0/531: dwrite d0/d25/d5e/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:04:48.449 INFO:tasks.workunit.client.0.smithi144.stdout:9/620: dread d9/d3b/d5d/f70 [0,4194304] 0 2024-07-26T11:04:48.451 INFO:tasks.workunit.client.0.smithi144.stdout:0/532: getdents d0/d3e 0 2024-07-26T11:04:48.452 INFO:tasks.workunit.client.0.smithi144.stdout:0/533: creat d0/d3e/d45/fae x:0 0 0 2024-07-26T11:04:48.462 INFO:tasks.workunit.client.0.smithi144.stdout:8/546: dread d1a/d32/d49/d73/f2e [0,4194304] 0 2024-07-26T11:04:48.463 INFO:tasks.workunit.client.0.smithi144.stdout:5/609: dwrite d1/d18/f90 [0,4194304] 0 2024-07-26T11:04:48.463 INFO:tasks.workunit.client.0.smithi144.stdout:5/610: truncate d1/d5/dae/fb5 452503 0 2024-07-26T11:04:48.465 INFO:tasks.workunit.client.0.smithi144.stdout:5/611: unlink d1/d18/d24/d27/l4c 0 2024-07-26T11:04:48.465 INFO:tasks.workunit.client.0.smithi144.stdout:5/612: fdatasync d1/d9/d44/d45/d81/f91 0 2024-07-26T11:04:48.477 INFO:tasks.workunit.client.0.smithi144.stdout:2/566: dread d0/d24/d48/d6d/d28/d2e/d5d/d6b/f73 [0,4194304] 0 2024-07-26T11:04:48.482 INFO:tasks.workunit.client.0.smithi144.stdout:0/534: read d0/d3e/d45/d9f/f6e [3863028,96355] 0 2024-07-26T11:04:48.483 INFO:tasks.workunit.client.0.smithi144.stdout:0/535: chown d0/f8 2063697 1 2024-07-26T11:04:48.483 INFO:tasks.workunit.client.0.smithi144.stdout:0/536: truncate d0/d3e/d45/d9f/f82 443916 0 2024-07-26T11:04:48.484 INFO:tasks.workunit.client.0.smithi144.stdout:0/537: mknod d0/d3e/d5f/caf 0 2024-07-26T11:04:48.486 INFO:tasks.workunit.client.0.smithi144.stdout:0/538: truncate d0/d25/d5e/d1e/d30/f39 1961303 0 2024-07-26T11:04:48.491 INFO:tasks.workunit.client.0.smithi144.stdout:1/758: write d3/d6/da/d13/d22/d5d/faf [3071062,109949] 0 2024-07-26T11:04:48.491 INFO:tasks.workunit.client.0.smithi144.stdout:9/621: dread d9/de/d14/d23/d39/f9f [0,4194304] 0 2024-07-26T11:04:48.491 INFO:tasks.workunit.client.0.smithi144.stdout:1/759: chown d3/d6/dc0 357309909 1 2024-07-26T11:04:48.492 INFO:tasks.workunit.client.0.smithi144.stdout:1/760: chown d3/d6/da/d13/d19/d3a/d69/db7/f8f 5887354 1 2024-07-26T11:04:48.494 INFO:tasks.workunit.client.0.smithi144.stdout:9/622: truncate d9/de/f4d 1340433 0 2024-07-26T11:04:48.496 INFO:tasks.workunit.client.0.smithi144.stdout:5/613: dread d1/d9/d44/d45/d81/f91 [4194304,4194304] 0 2024-07-26T11:04:48.496 INFO:tasks.workunit.client.0.smithi144.stdout:3/413: dwrite d1/d28/d54/d26/f2e [4194304,4194304] 0 2024-07-26T11:04:48.496 INFO:tasks.workunit.client.0.smithi144.stdout:5/614: chown d1/d9/d44/d45/d98/ccd 79 1 2024-07-26T11:04:48.504 INFO:tasks.workunit.client.0.smithi144.stdout:2/567: read d0/d24/d2b/f59 [1511125,96986] 0 2024-07-26T11:04:48.504 INFO:tasks.workunit.client.0.smithi144.stdout:5/615: rmdir d1/d9 39 2024-07-26T11:04:48.506 INFO:tasks.workunit.client.0.smithi144.stdout:5/616: truncate d1/d9/d5b/d4d/f3e 2857493 0 2024-07-26T11:04:48.507 INFO:tasks.workunit.client.0.smithi144.stdout:5/617: fdatasync d1/d18/d24/d35/fb9 0 2024-07-26T11:04:48.508 INFO:tasks.workunit.client.0.smithi144.stdout:2/568: creat d0/d24/d7b/fb0 x:0 0 0 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:5/618: rmdir d1/d18/da4 39 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:1/761: fdatasync d3/d6/da/d21/d5c/d62/f77 0 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:0/539: read d0/d3e/d45/d9f/f86 [42224,53521] 0 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:0/540: chown d0/d25/d5e/d1e/f24 836850 1 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:1/762: dread - d3/d3d/fae zero size 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:0/541: write d0/d25/f7b [540601,110325] 0 2024-07-26T11:04:48.510 INFO:tasks.workunit.client.0.smithi144.stdout:0/542: stat d0/d25/d5e/d1e/d30/f3b 0 2024-07-26T11:04:48.511 INFO:tasks.workunit.client.0.smithi144.stdout:0/543: write d0/f32 [2061767,14035] 0 2024-07-26T11:04:48.511 INFO:tasks.workunit.client.0.smithi144.stdout:6/717: dwrite d8/d50/d98/f9f [0,4194304] 0 2024-07-26T11:04:48.511 INFO:tasks.workunit.client.0.smithi144.stdout:8/547: read d1a/d32/fa9 [2442810,88319] 0 2024-07-26T11:04:48.515 INFO:tasks.workunit.client.0.smithi144.stdout:9/623: dread d9/de/f20 [0,4194304] 0 2024-07-26T11:04:48.516 INFO:tasks.workunit.client.0.smithi144.stdout:9/624: write d9/f29 [223566,81174] 0 2024-07-26T11:04:48.516 INFO:tasks.workunit.client.0.smithi144.stdout:1/763: rename d3/d6/db8 to d3/d6/da/d13/d19/d3a/d3e/ddb/df6 0 2024-07-26T11:04:48.516 INFO:tasks.workunit.client.0.smithi144.stdout:2/569: creat d0/d24/d7b/fb1 x:0 0 0 2024-07-26T11:04:48.516 INFO:tasks.workunit.client.0.smithi144.stdout:1/764: write d3/d6/dcc/fd9 [242071,2605] 0 2024-07-26T11:04:48.526 INFO:tasks.workunit.client.0.smithi144.stdout:6/718: mknod d8/db/dbe/d5c/cfe 0 2024-07-26T11:04:48.526 INFO:tasks.workunit.client.0.smithi144.stdout:8/548: symlink d1a/d32/d49/d73/d7b/d4e/d78/lbf 0 2024-07-26T11:04:48.526 INFO:tasks.workunit.client.0.smithi144.stdout:6/719: chown d8/d50/l59 59422884 1 2024-07-26T11:04:48.526 INFO:tasks.workunit.client.0.smithi144.stdout:8/549: stat d1a/d32/f48 0 2024-07-26T11:04:48.526 INFO:tasks.workunit.client.0.smithi144.stdout:1/765: rmdir d3/d6/da/d13/d22 39 2024-07-26T11:04:48.527 INFO:tasks.workunit.client.0.smithi144.stdout:3/414: dread d1/d28/d54/d26/f27 [0,4194304] 0 2024-07-26T11:04:48.527 INFO:tasks.workunit.client.0.smithi144.stdout:3/415: fsync d1/d28/d54/d26/d30/d4d/f55 0 2024-07-26T11:04:48.528 INFO:tasks.workunit.client.0.smithi144.stdout:2/570: creat d0/d24/d48/d6d/d28/d2e/d77/fb2 x:0 0 0 2024-07-26T11:04:48.528 INFO:tasks.workunit.client.0.smithi144.stdout:2/571: fsync d0/d6/f89 0 2024-07-26T11:04:48.528 INFO:tasks.workunit.client.0.smithi144.stdout:2/572: dread - d0/d6/d34/d6c/f86 zero size 2024-07-26T11:04:48.533 INFO:tasks.workunit.client.0.smithi144.stdout:8/550: mknod d1a/d32/d49/d73/d7b/d4e/d78/cc0 0 2024-07-26T11:04:48.533 INFO:tasks.workunit.client.0.smithi144.stdout:8/551: truncate d1a/d32/f62 1653548 0 2024-07-26T11:04:48.533 INFO:tasks.workunit.client.0.smithi144.stdout:0/544: dread d0/d25/fa5 [0,4194304] 0 2024-07-26T11:04:48.534 INFO:tasks.workunit.client.0.smithi144.stdout:1/766: truncate d3/d6/da/d21/fc1 261204 0 2024-07-26T11:04:48.537 INFO:tasks.workunit.client.0.smithi144.stdout:3/416: mkdir d1/d28/d54/d26/d30/d79 0 2024-07-26T11:04:48.538 INFO:tasks.workunit.client.0.smithi144.stdout:3/417: readlink d1/d5/l21 0 2024-07-26T11:04:48.538 INFO:tasks.workunit.client.0.smithi144.stdout:2/573: link d0/d6/c1a d0/d24/d48/d6d/d28/da8/cb3 0 2024-07-26T11:04:48.538 INFO:tasks.workunit.client.0.smithi144.stdout:2/574: fdatasync d0/d24/d48/d6d/d28/d2e/d5d/f7a 0 2024-07-26T11:04:48.538 INFO:tasks.workunit.client.0.smithi144.stdout:2/575: chown d0/d24/d48/d6d/d28/d2e/d5d/d6b 31821509 1 2024-07-26T11:04:48.538 INFO:tasks.workunit.client.0.smithi144.stdout:0/545: getdents d0/d3e/d45/d9f/d92 0 2024-07-26T11:04:48.540 INFO:tasks.workunit.client.0.smithi144.stdout:2/576: symlink d0/d24/d48/d6d/d28/d2e/d5d/dae/lb4 0 2024-07-26T11:04:48.540 INFO:tasks.workunit.client.0.smithi144.stdout:2/577: chown d0/d6/d34/d6c/f75 991169010 1 2024-07-26T11:04:48.542 INFO:tasks.workunit.client.0.smithi144.stdout:0/546: mkdir d0/d3e/d45/d9f/d65/db0 0 2024-07-26T11:04:48.542 INFO:tasks.workunit.client.0.smithi144.stdout:0/547: truncate d0/d25/d5e/d1e/d30/d4b/f75 918147 0 2024-07-26T11:04:48.542 INFO:tasks.workunit.client.0.smithi144.stdout:0/548: stat d0/d3e/d45/d9f/f82 0 2024-07-26T11:04:48.542 INFO:tasks.workunit.client.0.smithi144.stdout:0/549: fdatasync d0/d3e/d45/d9f/f6e 0 2024-07-26T11:04:48.544 INFO:tasks.workunit.client.0.smithi144.stdout:0/550: creat d0/d3e/d5f/fb1 x:0 0 0 2024-07-26T11:04:48.544 INFO:tasks.workunit.client.0.smithi144.stdout:0/551: fdatasync d0/d25/d5e/d1e/f3c 0 2024-07-26T11:04:48.547 INFO:tasks.workunit.client.0.smithi144.stdout:0/552: mkdir d0/d25/d5e/d1e/d30/d67/d84/db2 0 2024-07-26T11:04:48.555 INFO:tasks.workunit.client.0.smithi144.stdout:0/553: mknod d0/d25/d5e/d1e/d30/d67/d84/cb3 0 2024-07-26T11:04:48.557 INFO:tasks.workunit.client.0.smithi144.stdout:0/554: mkdir d0/d3e/db4 0 2024-07-26T11:04:48.559 INFO:tasks.workunit.client.0.smithi144.stdout:0/555: creat d0/d3e/d45/d9f/d65/fb5 x:0 0 0 2024-07-26T11:04:48.566 INFO:tasks.workunit.client.0.smithi144.stdout:8/552: dread d1a/d32/fa9 [0,4194304] 0 2024-07-26T11:04:48.566 INFO:tasks.workunit.client.0.smithi144.stdout:8/553: chown cc 25550876 1 2024-07-26T11:04:48.567 INFO:tasks.workunit.client.0.smithi144.stdout:8/554: write d1a/f1b [2687277,58676] 0 2024-07-26T11:04:48.567 INFO:tasks.workunit.client.0.smithi144.stdout:8/555: truncate d1a/d32/f62 2700473 0 2024-07-26T11:04:48.570 INFO:tasks.workunit.client.0.smithi144.stdout:1/767: write d3/d6/da/d13/d19/f45 [1633315,105035] 0 2024-07-26T11:04:48.571 INFO:tasks.workunit.client.0.smithi144.stdout:3/418: dread d1/d5/f24 [0,4194304] 0 2024-07-26T11:04:48.571 INFO:tasks.workunit.client.0.smithi144.stdout:1/768: mkdir d3/d6/da/d78/df7 0 2024-07-26T11:04:48.572 INFO:tasks.workunit.client.0.smithi144.stdout:1/769: stat d3/d6/da/d13/d19/d3a/d69/d36/d37/l74 0 2024-07-26T11:04:48.573 INFO:tasks.workunit.client.0.smithi144.stdout:1/770: getdents d3/d6/da/d13 0 2024-07-26T11:04:48.574 INFO:tasks.workunit.client.0.smithi144.stdout:1/771: chown d3/d6 532 1 2024-07-26T11:04:48.574 INFO:tasks.workunit.client.0.smithi144.stdout:1/772: chown d3/d6/da/d21/c4c 1841420546 1 2024-07-26T11:04:48.575 INFO:tasks.workunit.client.0.smithi144.stdout:1/773: mknod d3/d53/cf8 0 2024-07-26T11:04:48.575 INFO:tasks.workunit.client.0.smithi144.stdout:1/774: chown d3/d6/dc0/cee 4 1 2024-07-26T11:04:48.575 INFO:tasks.workunit.client.0.smithi144.stdout:1/775: write d3/d6/da/d13/d22/fe7 [885045,125266] 0 2024-07-26T11:04:48.576 INFO:tasks.workunit.client.0.smithi144.stdout:1/776: mknod d3/d6/da/d13/d19/d3a/d69/cf9 0 2024-07-26T11:04:48.578 INFO:tasks.workunit.client.0.smithi144.stdout:1/777: symlink d3/d6/da/d78/lfa 0 2024-07-26T11:04:48.578 INFO:tasks.workunit.client.0.smithi144.stdout:1/778: write d3/d6/da/d21/d67/fc3 [600200,123679] 0 2024-07-26T11:04:48.578 INFO:tasks.workunit.client.0.smithi144.stdout:1/779: write d3/d6/da/d13/d19/d3a/d69/d36/f55 [1544153,30005] 0 2024-07-26T11:04:48.578 INFO:tasks.workunit.client.0.smithi144.stdout:9/625: dwrite d9/d3b/d81/fb4 [0,4194304] 0 2024-07-26T11:04:48.580 INFO:tasks.workunit.client.0.smithi144.stdout:1/780: mkdir d3/d6/da/d97/dfb 0 2024-07-26T11:04:48.580 INFO:tasks.workunit.client.0.smithi144.stdout:9/626: mknod d9/de/d4e/cc9 0 2024-07-26T11:04:48.581 INFO:tasks.workunit.client.0.smithi144.stdout:1/781: creat d3/d6/da/d21/d5c/d62/ffc x:0 0 0 2024-07-26T11:04:48.582 INFO:tasks.workunit.client.0.smithi144.stdout:0/556: dread d0/d25/fa5 [0,4194304] 0 2024-07-26T11:04:48.582 INFO:tasks.workunit.client.0.smithi144.stdout:0/557: getdents d0/d25 0 2024-07-26T11:04:48.583 INFO:tasks.workunit.client.0.smithi144.stdout:9/627: creat d9/fca x:0 0 0 2024-07-26T11:04:48.586 INFO:tasks.workunit.client.0.smithi144.stdout:1/782: symlink d3/d6/da/d13/d19/d3a/d69/db7/lfd 0 2024-07-26T11:04:48.588 INFO:tasks.workunit.client.0.smithi144.stdout:0/558: creat d0/d25/d5e/d1e/d30/d4b/fb6 x:0 0 0 2024-07-26T11:04:48.589 INFO:tasks.workunit.client.0.smithi144.stdout:9/628: symlink d9/d3b/d81/lcb 0 2024-07-26T11:04:48.590 INFO:tasks.workunit.client.0.smithi144.stdout:9/629: stat d9/d3b/d81/fb4 0 2024-07-26T11:04:48.592 INFO:tasks.workunit.client.0.smithi144.stdout:0/559: rename d0/d3e/d45/d4c/l98 to d0/d25/d5e/d1e/d6c/lb7 0 2024-07-26T11:04:48.594 INFO:tasks.workunit.client.0.smithi144.stdout:0/560: mkdir d0/d3e/db8 0 2024-07-26T11:04:48.606 INFO:tasks.workunit.client.0.smithi144.stdout:2/578: dwrite d0/d24/d48/d6d/f47 [0,4194304] 0 2024-07-26T11:04:48.606 INFO:tasks.workunit.client.0.smithi144.stdout:2/579: chown d0/d6/f89 33 1 2024-07-26T11:04:48.608 INFO:tasks.workunit.client.0.smithi144.stdout:3/419: write d1/d28/d54/d26/f2e [438218,7027] 0 2024-07-26T11:04:48.608 INFO:tasks.workunit.client.0.smithi144.stdout:6/720: dwrite d8/db/dbe/f51 [0,4194304] 0 2024-07-26T11:04:48.619 INFO:tasks.workunit.client.0.smithi144.stdout:8/556: dwrite f6 [0,4194304] 0 2024-07-26T11:04:48.620 INFO:tasks.workunit.client.0.smithi144.stdout:8/557: read d1a/d32/d49/d73/d2c/d38/d58/f6e [2292013,67584] 0 2024-07-26T11:04:48.620 INFO:tasks.workunit.client.0.smithi144.stdout:8/558: write d1a/d1c/f2d [5142709,87662] 0 2024-07-26T11:04:48.636 INFO:tasks.workunit.client.0.smithi144.stdout:0/561: read d0/f10 [787833,54985] 0 2024-07-26T11:04:48.636 INFO:tasks.workunit.client.0.smithi144.stdout:0/562: read - d0/d25/d5e/d1e/d30/d4b/fb6 zero size 2024-07-26T11:04:48.638 INFO:tasks.workunit.client.0.smithi144.stdout:0/563: creat d0/d25/d5e/d1e/d30/fb9 x:0 0 0 2024-07-26T11:04:48.638 INFO:tasks.workunit.client.0.smithi144.stdout:6/721: dread d8/db/dbe/daa/fac [0,4194304] 0 2024-07-26T11:04:48.639 INFO:tasks.workunit.client.0.smithi144.stdout:0/564: mknod d0/d25/cba 0 2024-07-26T11:04:48.639 INFO:tasks.workunit.client.0.smithi144.stdout:8/559: dread d1a/d32/d49/d73/d7b/d43/d4c/f54 [4194304,4194304] 0 2024-07-26T11:04:48.640 INFO:tasks.workunit.client.0.smithi144.stdout:6/722: symlink d8/db/dbe/d5c/lff 0 2024-07-26T11:04:48.640 INFO:tasks.workunit.client.0.smithi144.stdout:6/723: chown d8/d42/lf4 1265668 1 2024-07-26T11:04:48.640 INFO:tasks.workunit.client.0.smithi144.stdout:6/724: dread - d8/f67 zero size 2024-07-26T11:04:48.641 INFO:tasks.workunit.client.0.smithi144.stdout:0/565: rename d0/d3e/f7d to d0/d3e/d45/d9f/d65/fbb 0 2024-07-26T11:04:48.641 INFO:tasks.workunit.client.0.smithi144.stdout:6/725: creat d8/d50/d92/dea/df8/f100 x:0 0 0 2024-07-26T11:04:48.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/580: write d0/d24/d7b/f87 [3737227,118662] 0 2024-07-26T11:04:48.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/581: truncate d0/d6/d34/f8f 4322252 0 2024-07-26T11:04:48.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/582: read d0/d24/d2b/f8d [82127,90219] 0 2024-07-26T11:04:48.643 INFO:tasks.workunit.client.0.smithi144.stdout:0/566: rename d0/d3e/d45/f50 to d0/d25/d5e/d1e/d30/d6a/dac/fbc 0 2024-07-26T11:04:48.643 INFO:tasks.workunit.client.0.smithi144.stdout:0/567: write d0/f27 [940193,58727] 0 2024-07-26T11:04:48.643 INFO:tasks.workunit.client.0.smithi144.stdout:6/726: creat d8/d50/de2/f101 x:0 0 0 2024-07-26T11:04:48.645 INFO:tasks.workunit.client.0.smithi144.stdout:2/583: getdents d0/d24/d48/d6d/d28/d2e 0 2024-07-26T11:04:48.646 INFO:tasks.workunit.client.0.smithi144.stdout:8/560: dread d1a/d32/f62 [0,4194304] 0 2024-07-26T11:04:48.646 INFO:tasks.workunit.client.0.smithi144.stdout:0/568: creat d0/d25/d5e/d1e/d30/d67/d84/fbd x:0 0 0 2024-07-26T11:04:48.648 INFO:tasks.workunit.client.0.smithi144.stdout:8/561: mkdir d1a/d32/d49/d73/d7b/d4e/d9a/dc1 0 2024-07-26T11:04:48.649 INFO:tasks.workunit.client.0.smithi144.stdout:0/569: symlink d0/d25/d5e/d1e/d30/d4b/lbe 0 2024-07-26T11:04:48.649 INFO:tasks.workunit.client.0.smithi144.stdout:0/570: chown d0/d25/d5e/d1e/d30/d6a/d6f 15031168 1 2024-07-26T11:04:48.649 INFO:tasks.workunit.client.0.smithi144.stdout:2/584: truncate d0/d24/d48/d6d/d28/f5c 983930 0 2024-07-26T11:04:48.651 INFO:tasks.workunit.client.0.smithi144.stdout:8/562: rename d1a/d32/d49/d73/d7b/d43/d4f/da2 to d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2 0 2024-07-26T11:04:48.653 INFO:tasks.workunit.client.0.smithi144.stdout:0/571: truncate d0/d3e/d45/d4c/f79 1411709 0 2024-07-26T11:04:48.653 INFO:tasks.workunit.client.0.smithi144.stdout:0/572: chown d0/d25/d5e/d1e/d30/d4b/c8e 78385356 1 2024-07-26T11:04:48.653 INFO:tasks.workunit.client.0.smithi144.stdout:0/573: chown d0/d25/d5e/c1f 183277628 1 2024-07-26T11:04:48.654 INFO:tasks.workunit.client.0.smithi144.stdout:2/585: creat d0/d24/d48/d6d/d28/d2e/d5d/d6b/fb5 x:0 0 0 2024-07-26T11:04:48.655 INFO:tasks.workunit.client.0.smithi144.stdout:8/563: symlink d1a/d32/d49/d73/d2c/d9d/d74/dac/dbe/lc3 0 2024-07-26T11:04:48.655 INFO:tasks.workunit.client.0.smithi144.stdout:8/564: readlink d1a/d32/d6c/l59 0 2024-07-26T11:04:48.656 INFO:tasks.workunit.client.0.smithi144.stdout:0/574: creat d0/d25/d5e/fbf x:0 0 0 2024-07-26T11:04:48.668 INFO:tasks.workunit.client.0.smithi144.stdout:8/565: mkdir d1a/d32/d50/dc4 0 2024-07-26T11:04:48.670 INFO:tasks.workunit.client.0.smithi144.stdout:0/575: creat d0/d25/d5e/d1e/d30/dab/fc0 x:0 0 0 2024-07-26T11:04:48.671 INFO:tasks.workunit.client.0.smithi144.stdout:0/576: mkdir d0/d25/d5e/d1e/dc1 0 2024-07-26T11:04:48.697 INFO:tasks.workunit.client.0.smithi144.stdout:2/586: dwrite d0/d24/d48/d6d/d28/d2e/d5d/d6b/f73 [0,4194304] 0 2024-07-26T11:04:48.697 INFO:tasks.workunit.client.0.smithi144.stdout:2/587: fdatasync d0/d24/d48/d6d/d28/d2e/d5d/d6b/f99 0 2024-07-26T11:04:48.699 INFO:tasks.workunit.client.0.smithi144.stdout:2/588: mknod d0/d24/d48/d63/da3/cb6 0 2024-07-26T11:04:48.699 INFO:tasks.workunit.client.0.smithi144.stdout:2/589: fsync d0/d24/d48/d6d/f96 0 2024-07-26T11:04:48.705 INFO:tasks.workunit.client.0.smithi144.stdout:8/566: dwrite d1a/d1c/f1f [0,4194304] 0 2024-07-26T11:04:48.712 INFO:tasks.workunit.client.0.smithi144.stdout:0/577: dwrite d0/f18 [0,4194304] 0 2024-07-26T11:04:48.733 INFO:tasks.workunit.client.0.smithi144.stdout:0/578: dread d0/d25/d5e/d1e/f24 [0,4194304] 0 2024-07-26T11:04:48.749 INFO:tasks.workunit.client.0.smithi144.stdout:2/590: dwrite d0/d6/f95 [0,4194304] 0 2024-07-26T11:04:48.753 INFO:tasks.workunit.client.0.smithi144.stdout:2/591: creat d0/d24/d48/d6d/d28/d2e/fb7 x:0 0 0 2024-07-26T11:04:48.754 INFO:tasks.workunit.client.0.smithi144.stdout:2/592: write d0/d24/d48/fa1 [818637,14981] 0 2024-07-26T11:04:48.754 INFO:tasks.workunit.client.0.smithi144.stdout:2/593: fdatasync d0/d24/d2b/f8e 0 2024-07-26T11:04:48.759 INFO:tasks.workunit.client.0.smithi144.stdout:8/567: dwrite d1a/d1c/f24 [0,4194304] 0 2024-07-26T11:04:48.759 INFO:tasks.workunit.client.0.smithi144.stdout:8/568: chown d1a/d32/d6c/c6d 701 1 2024-07-26T11:04:48.761 INFO:tasks.workunit.client.0.smithi144.stdout:8/569: rename d1a/d32/d49/d73/d7b/d43/d4c/l87 to d1a/d32/d49/d73/d2c/d38/lc5 0 2024-07-26T11:04:48.761 INFO:tasks.workunit.client.0.smithi144.stdout:8/570: rmdir d1a 39 2024-07-26T11:04:48.782 INFO:tasks.workunit.client.0.smithi144.stdout:0/579: dwrite d0/d25/d5e/d1e/d30/d4b/f75 [0,4194304] 0 2024-07-26T11:04:48.782 INFO:tasks.workunit.client.0.smithi144.stdout:0/580: dread - d0/d25/d5e/d1e/d6c/f97 zero size 2024-07-26T11:04:48.784 INFO:tasks.workunit.client.0.smithi144.stdout:0/581: rename d0/d3e/d45/d9f/d92 to d0/d25/d5e/d1e/dc2 0 2024-07-26T11:04:48.785 INFO:tasks.workunit.client.0.smithi144.stdout:0/582: link d0/d3e/d45/d9f/d65/c8b d0/d25/d5e/d1e/d6c/cc3 0 2024-07-26T11:04:48.786 INFO:tasks.workunit.client.0.smithi144.stdout:0/583: fsync d0/d25/d5e/f41 0 2024-07-26T11:04:48.786 INFO:tasks.workunit.client.0.smithi144.stdout:0/584: stat d0/d25/d5e/d1e/d30/d4b/f75 0 2024-07-26T11:04:48.786 INFO:tasks.workunit.client.0.smithi144.stdout:0/585: write d0/d3e/d45/fae [327945,122963] 0 2024-07-26T11:04:48.788 INFO:tasks.workunit.client.0.smithi144.stdout:0/586: rmdir d0/d3e/d45/d9f/d65/db0 0 2024-07-26T11:04:48.788 INFO:tasks.workunit.client.0.smithi144.stdout:0/587: write d0/d25/d5e/f68 [973120,111262] 0 2024-07-26T11:04:48.788 INFO:tasks.workunit.client.0.smithi144.stdout:0/588: readlink d0/l16 0 2024-07-26T11:04:48.788 INFO:tasks.workunit.client.0.smithi144.stdout:0/589: dread - d0/d3e/d5f/f8d zero size 2024-07-26T11:04:48.789 INFO:tasks.workunit.client.0.smithi144.stdout:0/590: mknod d0/d25/d5e/d1e/d91/cc4 0 2024-07-26T11:04:48.795 INFO:tasks.workunit.client.0.smithi144.stdout:8/571: dwrite d1a/d32/d49/d73/d2c/d38/d58/fa8 [0,4194304] 0 2024-07-26T11:04:48.797 INFO:tasks.workunit.client.0.smithi144.stdout:8/572: link d1a/d32/d49/d73/d2c/d9d/d74/l7d d1a/d32/d49/d73/d7b/d4e/d78/lc6 0 2024-07-26T11:04:48.838 INFO:tasks.workunit.client.0.smithi144.stdout:0/591: dwrite d0/d25/d5e/d1e/f3c [0,4194304] 0 2024-07-26T11:04:48.838 INFO:tasks.workunit.client.0.smithi144.stdout:0/592: chown d0/d25/d5e/d1e/d30/d4b/f58 73130999 1 2024-07-26T11:04:48.888 INFO:tasks.workunit.client.0.smithi144.stdout:0/593: dwrite d0/d25/d5e/f2c [0,4194304] 0 2024-07-26T11:04:48.891 INFO:tasks.workunit.client.0.smithi144.stdout:0/594: creat d0/d25/d5e/d1e/d30/d67/fc5 x:0 0 0 2024-07-26T11:04:48.893 INFO:tasks.workunit.client.0.smithi144.stdout:0/595: mkdir d0/d25/d5e/d1e/d6c/dc6 0 2024-07-26T11:04:48.964 INFO:tasks.workunit.client.0.smithi144.stdout:0/596: dwrite d0/f27 [0,4194304] 0 2024-07-26T11:04:48.964 INFO:tasks.workunit.client.0.smithi144.stdout:0/597: fdatasync d0/f27 0 2024-07-26T11:04:48.967 INFO:tasks.workunit.client.0.smithi144.stdout:0/598: unlink d0/d25/d5e/c1d 0 2024-07-26T11:04:48.967 INFO:tasks.workunit.client.0.smithi144.stdout:0/599: truncate d0/d25/d5e/d1e/d30/f87 3041646 0 2024-07-26T11:04:48.969 INFO:tasks.workunit.client.0.smithi144.stdout:0/600: rename d0/d3e/c77 to d0/d3e/db4/cc7 0 2024-07-26T11:04:48.973 INFO:tasks.workunit.client.0.smithi144.stdout:4/601: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:3/420: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:5/619: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:1/783: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:4/602: chown d9/lae 23834 1 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:9/630: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:7/556: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:2/594: sync 2024-07-26T11:04:48.974 INFO:tasks.workunit.client.0.smithi144.stdout:6/727: sync 2024-07-26T11:04:48.976 INFO:tasks.workunit.client.0.smithi144.stdout:4/603: creat d9/d26/fcc x:0 0 0 2024-07-26T11:04:48.976 INFO:tasks.workunit.client.0.smithi144.stdout:4/604: dread - d9/d26/fb6 zero size 2024-07-26T11:04:48.977 INFO:tasks.workunit.client.0.smithi144.stdout:9/631: mkdir d9/d3b/d76/d9b/dcc 0 2024-07-26T11:04:48.977 INFO:tasks.workunit.client.0.smithi144.stdout:9/632: truncate d9/d3b/d76/d9b/fb7 177304 0 2024-07-26T11:04:48.979 INFO:tasks.workunit.client.0.smithi144.stdout:1/784: rename d3/d6/da/d13/d22 to d3/d6/da/d13/d19/d3a/d3e/ddb/df6/dfe 0 2024-07-26T11:04:48.980 INFO:tasks.workunit.client.0.smithi144.stdout:2/595: mkdir d0/db8 0 2024-07-26T11:04:48.980 INFO:tasks.workunit.client.0.smithi144.stdout:2/596: chown d0/d6/d34/c3b 11665027 1 2024-07-26T11:04:48.981 INFO:tasks.workunit.client.0.smithi144.stdout:2/597: chown d0/d24/d48/d6d/fa0 351137365 1 2024-07-26T11:04:48.982 INFO:tasks.workunit.client.0.smithi144.stdout:9/633: mkdir d9/de/d14/d3f/d5e/dcd 0 2024-07-26T11:04:48.992 INFO:tasks.workunit.client.0.smithi144.stdout:1/785: truncate d3/d6/da/d13/f26 3981542 0 2024-07-26T11:04:48.992 INFO:tasks.workunit.client.0.smithi144.stdout:9/634: getdents d9/de/d14/d79 0 2024-07-26T11:04:48.992 INFO:tasks.workunit.client.0.smithi144.stdout:1/786: read - d3/d6/da/fca zero size 2024-07-26T11:04:49.004 INFO:tasks.workunit.client.0.smithi144.stdout:9/635: mkdir d9/d3b/dce 0 2024-07-26T11:04:49.005 INFO:tasks.workunit.client.0.smithi144.stdout:6/728: dread d8/f2d [0,4194304] 0 2024-07-26T11:04:49.021 INFO:tasks.workunit.client.0.smithi144.stdout:9/636: mknod d9/de/d14/d3f/d5e/ccf 0 2024-07-26T11:04:49.021 INFO:tasks.workunit.client.0.smithi144.stdout:9/637: chown d9/de/lb8 47974818 1 2024-07-26T11:04:49.022 INFO:tasks.workunit.client.0.smithi144.stdout:9/638: chown d9/d3b/d5d/f70 690689 1 2024-07-26T11:04:49.022 INFO:tasks.workunit.client.0.smithi144.stdout:9/639: fdatasync d9/de/f30 0 2024-07-26T11:04:49.025 INFO:tasks.workunit.client.0.smithi144.stdout:7/557: dread d0/d6/d9/d12/f7f [0,4194304] 0 2024-07-26T11:04:49.025 INFO:tasks.workunit.client.0.smithi144.stdout:5/620: dread d1/d9/d5b/f21 [0,4194304] 0 2024-07-26T11:04:49.025 INFO:tasks.workunit.client.0.smithi144.stdout:5/621: write d1/d18/d24/d27/f70 [354594,131032] 0 2024-07-26T11:04:49.025 INFO:tasks.workunit.client.0.smithi144.stdout:9/640: creat d9/d3b/d76/d9b/dcc/fd0 x:0 0 0 2024-07-26T11:04:49.026 INFO:tasks.workunit.client.0.smithi144.stdout:6/729: symlink d8/d15/d57/d5f/l102 0 2024-07-26T11:04:49.027 INFO:tasks.workunit.client.0.smithi144.stdout:5/622: rename d1/d9/d44/d45/daa/fbb to d1/d9/fd1 0 2024-07-26T11:04:49.028 INFO:tasks.workunit.client.0.smithi144.stdout:9/641: creat d9/de/d4e/fd1 x:0 0 0 2024-07-26T11:04:49.028 INFO:tasks.workunit.client.0.smithi144.stdout:9/642: readlink d9/de/d14/d23/d39/lad 0 2024-07-26T11:04:49.029 INFO:tasks.workunit.client.0.smithi144.stdout:9/643: truncate f4 3248968 0 2024-07-26T11:04:49.032 INFO:tasks.workunit.client.0.smithi144.stdout:5/623: symlink d1/d18/d24/ld2 0 2024-07-26T11:04:49.032 INFO:tasks.workunit.client.0.smithi144.stdout:5/624: readlink d1/d9/d5b/lc7 0 2024-07-26T11:04:49.032 INFO:tasks.workunit.client.0.smithi144.stdout:5/625: chown d1/d9/d5b/d4d/f63 1 1 2024-07-26T11:04:49.032 INFO:tasks.workunit.client.0.smithi144.stdout:3/421: dread d1/d5/f3a [0,4194304] 0 2024-07-26T11:04:49.032 INFO:tasks.workunit.client.0.smithi144.stdout:3/422: mknod d1/d28/d54/d3d/d42/d44/d60/c7a 0 2024-07-26T11:04:49.032 INFO:tasks.workunit.client.0.smithi144.stdout:5/626: creat d1/d5/dae/fd3 x:0 0 0 2024-07-26T11:04:49.057 INFO:tasks.workunit.client.0.smithi144.stdout:7/558: dread d0/d6/f17 [0,4194304] 0 2024-07-26T11:04:49.057 INFO:tasks.workunit.client.0.smithi144.stdout:7/559: chown d0/d71/cad 15545 1 2024-07-26T11:04:49.062 INFO:tasks.workunit.client.0.smithi144.stdout:4/605: dwrite d9/f63 [0,4194304] 0 2024-07-26T11:04:49.062 INFO:tasks.workunit.client.0.smithi144.stdout:4/606: write d9/f12 [4923625,64938] 0 2024-07-26T11:04:49.062 INFO:tasks.workunit.client.0.smithi144.stdout:7/560: getdents d0/d6/d9/d98/dbf/d2f/d68 0 2024-07-26T11:04:49.064 INFO:tasks.workunit.client.0.smithi144.stdout:3/423: dread d1/d5/f3a [0,4194304] 0 2024-07-26T11:04:49.066 INFO:tasks.workunit.client.0.smithi144.stdout:3/424: link d1/l10 d1/d28/d54/d3d/d42/l7b 0 2024-07-26T11:04:49.066 INFO:tasks.workunit.client.0.smithi144.stdout:0/601: dwrite d0/d3e/d45/d9f/f6e [0,4194304] 0 2024-07-26T11:04:49.068 INFO:tasks.workunit.client.0.smithi144.stdout:5/627: dread d1/d18/d24/d35/f9b [0,4194304] 0 2024-07-26T11:04:49.068 INFO:tasks.workunit.client.0.smithi144.stdout:3/425: rename d1/d5/f15 to d1/d28/d54/d26/f7c 0 2024-07-26T11:04:49.068 INFO:tasks.workunit.client.0.smithi144.stdout:3/426: fdatasync d1/d28/d54/d26/f2e 0 2024-07-26T11:04:49.069 INFO:tasks.workunit.client.0.smithi144.stdout:3/427: write d1/f17 [2357762,130209] 0 2024-07-26T11:04:49.070 INFO:tasks.workunit.client.0.smithi144.stdout:0/602: creat d0/d25/d5e/d1e/d5c/fc8 x:0 0 0 2024-07-26T11:04:49.072 INFO:tasks.workunit.client.0.smithi144.stdout:5/628: unlink d1/d18/f3b 0 2024-07-26T11:04:49.072 INFO:tasks.workunit.client.0.smithi144.stdout:5/629: stat d1/d9/d5b/d4d/f3e 0 2024-07-26T11:04:49.080 INFO:tasks.workunit.client.0.smithi144.stdout:0/603: link d0/d25/d5e/d1e/d6c/cc3 d0/d25/d5e/d1e/d6c/cc9 0 2024-07-26T11:04:49.097 INFO:tasks.workunit.client.0.smithi144.stdout:2/598: dwrite d0/d24/d2b/d68/d82/f7c [0,4194304] 0 2024-07-26T11:04:49.104 INFO:tasks.workunit.client.0.smithi144.stdout:1/787: dwrite d3/d6/d9c/d9f/fa7 [0,4194304] 0 2024-07-26T11:04:49.106 INFO:tasks.workunit.client.0.smithi144.stdout:6/730: write d8/db/dbe/d5c/f7b [2740349,103706] 0 2024-07-26T11:04:49.108 INFO:tasks.workunit.client.0.smithi144.stdout:6/731: mkdir d8/d50/d92/db3/d103 0 2024-07-26T11:04:49.109 INFO:tasks.workunit.client.0.smithi144.stdout:6/732: rmdir d8/d50/d92/dea 39 2024-07-26T11:04:49.109 INFO:tasks.workunit.client.0.smithi144.stdout:2/599: dread d0/d24/d2b/d68/f6a [0,4194304] 0 2024-07-26T11:04:49.111 INFO:tasks.workunit.client.0.smithi144.stdout:2/600: symlink d0/d24/d48/d63/da3/lb9 0 2024-07-26T11:04:49.112 INFO:tasks.workunit.client.0.smithi144.stdout:9/644: read d9/de/d14/d3f/d5e/f62 [4168955,116812] 0 2024-07-26T11:04:49.113 INFO:tasks.workunit.client.0.smithi144.stdout:9/645: mknod d9/de/d14/d23/d39/cd2 0 2024-07-26T11:04:49.114 INFO:tasks.workunit.client.0.smithi144.stdout:9/646: creat d9/de/d14/d3f/d5e/dcd/fd3 x:0 0 0 2024-07-26T11:04:49.124 INFO:tasks.workunit.client.0.smithi144.stdout:9/647: mknod d9/d3b/d76/cd4 0 2024-07-26T11:04:49.124 INFO:tasks.workunit.client.0.smithi144.stdout:4/607: dwrite d9/f41 [0,4194304] 0 2024-07-26T11:04:49.125 INFO:tasks.workunit.client.0.smithi144.stdout:8/573: sync 2024-07-26T11:04:49.126 INFO:tasks.workunit.client.0.smithi144.stdout:8/574: symlink d1a/d32/d49/d73/d7b/d43/lc7 0 2024-07-26T11:04:49.127 INFO:tasks.workunit.client.0.smithi144.stdout:8/575: creat d1a/fc8 x:0 0 0 2024-07-26T11:04:49.130 INFO:tasks.workunit.client.0.smithi144.stdout:7/561: dwrite d0/d71/d95/dc4/fc2 [0,4194304] 0 2024-07-26T11:04:49.130 INFO:tasks.workunit.client.0.smithi144.stdout:9/648: dread d9/de/d14/d23/d39/d7f/fa4 [0,4194304] 0 2024-07-26T11:04:49.131 INFO:tasks.workunit.client.0.smithi144.stdout:9/649: stat d9/de/d14/d79/db2/fc8 0 2024-07-26T11:04:49.131 INFO:tasks.workunit.client.0.smithi144.stdout:9/650: readlink d9/de/d14/d3f/l60 0 2024-07-26T11:04:49.131 INFO:tasks.workunit.client.0.smithi144.stdout:9/651: truncate d9/de/d4e/d6a/f8a 259267 0 2024-07-26T11:04:49.133 INFO:tasks.workunit.client.0.smithi144.stdout:7/562: creat d0/d6/d9/d12/fcc x:0 0 0 2024-07-26T11:04:49.133 INFO:tasks.workunit.client.0.smithi144.stdout:7/563: write d0/d6/d9/fc [1762268,1185] 0 2024-07-26T11:04:49.137 INFO:tasks.workunit.client.0.smithi144.stdout:4/608: write d9/f41 [1409960,121978] 0 2024-07-26T11:04:49.140 INFO:tasks.workunit.client.0.smithi144.stdout:7/564: dread d0/d6/d9/d98/dbf/d2f/d47/d5e/f88 [0,4194304] 0 2024-07-26T11:04:49.141 INFO:tasks.workunit.client.0.smithi144.stdout:4/609: creat d9/d7b/fcd x:0 0 0 2024-07-26T11:04:49.151 INFO:tasks.workunit.client.0.smithi144.stdout:1/788: dwrite d3/d6/da/d70/fa2 [0,4194304] 0 2024-07-26T11:04:49.171 INFO:tasks.workunit.client.0.smithi144.stdout:9/652: dwrite f4 [0,4194304] 0 2024-07-26T11:04:49.173 INFO:tasks.workunit.client.0.smithi144.stdout:2/601: dread d0/d24/d48/d6d/d28/d2e/d5d/d6b/f99 [0,4194304] 0 2024-07-26T11:04:49.177 INFO:tasks.workunit.client.0.smithi144.stdout:1/789: write d3/d6/da/d70/fa2 [62657,117003] 0 2024-07-26T11:04:49.177 INFO:tasks.workunit.client.0.smithi144.stdout:1/790: truncate d3/d6/da/fca 86694 0 2024-07-26T11:04:49.180 INFO:tasks.workunit.client.0.smithi144.stdout:9/653: write d9/de/d14/d3f/f53 [485562,79722] 0 2024-07-26T11:04:49.181 INFO:tasks.workunit.client.0.smithi144.stdout:9/654: mknod d9/d3b/d5d/cd5 0 2024-07-26T11:04:49.183 INFO:tasks.workunit.client.0.smithi144.stdout:8/576: dread d1a/d32/d49/d73/d2c/d38/f5f [0,4194304] 0 2024-07-26T11:04:49.183 INFO:tasks.workunit.client.0.smithi144.stdout:0/604: dwrite d0/f8 [0,4194304] 0 2024-07-26T11:04:49.185 INFO:tasks.workunit.client.0.smithi144.stdout:0/605: link d0/d25/d5e/d1e/d30/d67/fc5 d0/d3e/d45/d4c/fca 0 2024-07-26T11:04:49.186 INFO:tasks.workunit.client.0.smithi144.stdout:8/577: creat d1a/d32/d49/d73/d7b/d43/d4f/fc9 x:0 0 0 2024-07-26T11:04:49.186 INFO:tasks.workunit.client.0.smithi144.stdout:0/606: mknod d0/ccb 0 2024-07-26T11:04:49.186 INFO:tasks.workunit.client.0.smithi144.stdout:0/607: write d0/d3e/d45/d4c/f79 [2262596,2612] 0 2024-07-26T11:04:49.189 INFO:tasks.workunit.client.0.smithi144.stdout:0/608: rename d0/l17 to d0/d25/d5e/d1e/d30/d67/d84/db2/lcc 0 2024-07-26T11:04:49.191 INFO:tasks.workunit.client.0.smithi144.stdout:4/610: read d9/d26/f62 [175555,94849] 0 2024-07-26T11:04:49.192 INFO:tasks.workunit.client.0.smithi144.stdout:4/611: mknod d9/d26/d48/d50/cce 0 2024-07-26T11:04:49.193 INFO:tasks.workunit.client.0.smithi144.stdout:4/612: symlink d9/d26/d48/d50/lcf 0 2024-07-26T11:04:49.195 INFO:tasks.workunit.client.0.smithi144.stdout:4/613: mkdir d9/d26/d3c/d6f/dd0 0 2024-07-26T11:04:49.207 INFO:tasks.workunit.client.0.smithi144.stdout:8/578: dread d1a/d1c/f22 [4194304,4194304] 0 2024-07-26T11:04:49.207 INFO:tasks.workunit.client.0.smithi144.stdout:7/565: dwrite d0/d6/d9/d98/dbf/d2f/d47/d5e/f69 [4194304,4194304] 0 2024-07-26T11:04:49.210 INFO:tasks.workunit.client.0.smithi144.stdout:8/579: mknod d1a/d32/d49/d73/d2c/d38/dbd/cca 0 2024-07-26T11:04:49.210 INFO:tasks.workunit.client.0.smithi144.stdout:7/566: truncate d0/f4c 4155899 0 2024-07-26T11:04:49.225 INFO:tasks.workunit.client.0.smithi144.stdout:2/602: dwrite d0/d6/d34/faf [0,4194304] 0 2024-07-26T11:04:49.227 INFO:tasks.workunit.client.0.smithi144.stdout:4/614: dread d9/f11 [0,4194304] 0 2024-07-26T11:04:49.230 INFO:tasks.workunit.client.0.smithi144.stdout:2/603: symlink d0/d24/d7b/lba 0 2024-07-26T11:04:49.231 INFO:tasks.workunit.client.0.smithi144.stdout:2/604: chown d0/c1b 93986 1 2024-07-26T11:04:49.233 INFO:tasks.workunit.client.0.smithi144.stdout:2/605: creat d0/d24/d2b/fbb x:0 0 0 2024-07-26T11:04:49.235 INFO:tasks.workunit.client.0.smithi144.stdout:2/606: rmdir d0/d24/d48 39 2024-07-26T11:04:49.237 INFO:tasks.workunit.client.0.smithi144.stdout:2/607: creat d0/d24/d48/d63/da3/fbc x:0 0 0 2024-07-26T11:04:49.237 INFO:tasks.workunit.client.0.smithi144.stdout:2/608: chown d0/d24/d48/d6d/d28 713776 1 2024-07-26T11:04:49.240 INFO:tasks.workunit.client.0.smithi144.stdout:2/609: rename d0/d24/d48/d6d/d28/d2e/d5d/f7a to d0/d24/d48/d6d/d28/d2e/d77/fbd 0 2024-07-26T11:04:49.243 INFO:tasks.workunit.client.0.smithi144.stdout:1/791: dwrite d3/d6/da/d13/d19/d3a/f4b [0,4194304] 0 2024-07-26T11:04:49.243 INFO:tasks.workunit.client.0.smithi144.stdout:1/792: chown d3/d6/da/d78/ca3 17009 1 2024-07-26T11:04:49.243 INFO:tasks.workunit.client.0.smithi144.stdout:1/793: write d3/d6/da/d21/fda [118542,128135] 0 2024-07-26T11:04:49.245 INFO:tasks.workunit.client.0.smithi144.stdout:1/794: symlink d3/d6/lff 0 2024-07-26T11:04:49.247 INFO:tasks.workunit.client.0.smithi144.stdout:9/655: read d9/fc [2786928,24255] 0 2024-07-26T11:04:49.248 INFO:tasks.workunit.client.0.smithi144.stdout:9/656: chown d9/la6 12188 1 2024-07-26T11:04:49.248 INFO:tasks.workunit.client.0.smithi144.stdout:9/657: chown d9/de/d14/d3f/d5e/lc3 26311576 1 2024-07-26T11:04:49.248 INFO:tasks.workunit.client.0.smithi144.stdout:1/795: truncate d3/d6/da/d70/f90 1363970 0 2024-07-26T11:04:49.252 INFO:tasks.workunit.client.0.smithi144.stdout:0/609: dwrite d0/f32 [0,4194304] 0 2024-07-26T11:04:49.252 INFO:tasks.workunit.client.0.smithi144.stdout:0/610: read d0/d3e/d45/d9f/f86 [47765,40972] 0 2024-07-26T11:04:49.254 INFO:tasks.workunit.client.0.smithi144.stdout:0/611: read d0/d3e/d45/d9f/f82 [348835,60882] 0 2024-07-26T11:04:49.254 INFO:tasks.workunit.client.0.smithi144.stdout:0/612: fsync d0/d3e/d5f/f8d 0 2024-07-26T11:04:49.255 INFO:tasks.workunit.client.0.smithi144.stdout:7/567: dread d0/d6/d9/d12/f5d [0,4194304] 0 2024-07-26T11:04:49.255 INFO:tasks.workunit.client.0.smithi144.stdout:7/568: readlink d0/d6/d9/l50 0 2024-07-26T11:04:49.255 INFO:tasks.workunit.client.0.smithi144.stdout:7/569: stat d0/d71/d95/dc4/fbc 0 2024-07-26T11:04:49.255 INFO:tasks.workunit.client.0.smithi144.stdout:7/570: read - d0/d6/d9/d98/dbf/d2f/da2/fa6 zero size 2024-07-26T11:04:49.257 INFO:tasks.workunit.client.0.smithi144.stdout:0/613: link d0/d25/d5e/d1e/d30/d67/d84/cb3 d0/d25/d5e/d1e/d30/d67/d84/ccd 0 2024-07-26T11:04:49.257 INFO:tasks.workunit.client.0.smithi144.stdout:0/614: chown d0/d3e/db8 14094 1 2024-07-26T11:04:49.257 INFO:tasks.workunit.client.0.smithi144.stdout:7/571: creat d0/d6/fcd x:0 0 0 2024-07-26T11:04:49.260 INFO:tasks.workunit.client.0.smithi144.stdout:7/572: chown d0/d6/d9/d98/dbf/d2f/lac 3 1 2024-07-26T11:04:49.262 INFO:tasks.workunit.client.0.smithi144.stdout:7/573: creat d0/d6/d9/d98/dbf/d2f/d47/d5e/fce x:0 0 0 2024-07-26T11:04:49.265 INFO:tasks.workunit.client.0.smithi144.stdout:8/580: dwrite d1a/d32/d49/d73/d2c/f36 [4194304,4194304] 0 2024-07-26T11:04:49.265 INFO:tasks.workunit.client.0.smithi144.stdout:8/581: fdatasync d1a/d32/d49/d73/d7b/f40 0 2024-07-26T11:04:49.265 INFO:tasks.workunit.client.0.smithi144.stdout:7/574: creat d0/d6/d9/d98/dbf/d2f/d53/fcf x:0 0 0 2024-07-26T11:04:49.265 INFO:tasks.workunit.client.0.smithi144.stdout:7/575: truncate d0/d6/d9/d45/d57/f7e 654572 0 2024-07-26T11:04:49.270 INFO:tasks.workunit.client.0.smithi144.stdout:8/582: creat d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2/fcb x:0 0 0 2024-07-26T11:04:49.284 INFO:tasks.workunit.client.0.smithi144.stdout:4/615: dwrite d9/d26/d48/d50/f6c [0,4194304] 0 2024-07-26T11:04:49.287 INFO:tasks.workunit.client.0.smithi144.stdout:4/616: creat d9/d26/d48/d98/fd1 x:0 0 0 2024-07-26T11:04:49.289 INFO:tasks.workunit.client.0.smithi144.stdout:7/576: rename d0/d6/d9/d98 to d0/dd0 0 2024-07-26T11:04:49.289 INFO:tasks.workunit.client.0.smithi144.stdout:7/577: stat d0/d6/d9/d12/c72 0 2024-07-26T11:04:49.289 INFO:tasks.workunit.client.0.smithi144.stdout:7/578: write d0/f4b [4717947,97522] 0 2024-07-26T11:04:49.289 INFO:tasks.workunit.client.0.smithi144.stdout:7/579: fdatasync d0/dd0/dbf/d67/f9d 0 2024-07-26T11:04:49.290 INFO:tasks.workunit.client.0.smithi144.stdout:4/617: creat d9/d26/fd2 x:0 0 0 2024-07-26T11:04:49.290 INFO:tasks.workunit.client.0.smithi144.stdout:7/580: mknod d0/d6/d9/d12/d6c/cd1 0 2024-07-26T11:04:49.292 INFO:tasks.workunit.client.0.smithi144.stdout:4/618: creat d9/fd3 x:0 0 0 2024-07-26T11:04:49.305 INFO:tasks.workunit.client.0.smithi144.stdout:2/610: dwrite d0/d24/d48/d6d/d28/d2e/d77/fb2 [0,4194304] 0 2024-07-26T11:04:49.306 INFO:tasks.workunit.client.0.smithi144.stdout:2/611: truncate d0/d24/d48/d6d/d28/d2e/d5d/d6b/f99 4352500 0 2024-07-26T11:04:49.310 INFO:tasks.workunit.client.0.smithi144.stdout:2/612: unlink d0/d24/d48/d6d/d28/da8/lac 0 2024-07-26T11:04:49.313 INFO:tasks.workunit.client.0.smithi144.stdout:5/630: sync 2024-07-26T11:04:49.313 INFO:tasks.workunit.client.0.smithi144.stdout:6/733: sync 2024-07-26T11:04:49.314 INFO:tasks.workunit.client.0.smithi144.stdout:6/734: chown d8/db/f2c 21 1 2024-07-26T11:04:49.315 INFO:tasks.workunit.client.0.smithi144.stdout:3/428: sync 2024-07-26T11:04:49.315 INFO:tasks.workunit.client.0.smithi144.stdout:3/429: dread - d1/d28/d54/d26/d64/d68/f6d zero size 2024-07-26T11:04:49.315 INFO:tasks.workunit.client.0.smithi144.stdout:6/735: mkdir d8/d50/d98/d104 0 2024-07-26T11:04:49.315 INFO:tasks.workunit.client.0.smithi144.stdout:6/736: truncate d8/db/dbe/fc9 4894019 0 2024-07-26T11:04:49.317 INFO:tasks.workunit.client.0.smithi144.stdout:8/583: dwrite d1a/d32/fa9 [0,4194304] 0 2024-07-26T11:04:49.318 INFO:tasks.workunit.client.0.smithi144.stdout:4/619: dread d9/da/f2d [0,4194304] 0 2024-07-26T11:04:49.321 INFO:tasks.workunit.client.0.smithi144.stdout:7/581: dread d0/dd0/dbf/d2f/f70 [0,4194304] 0 2024-07-26T11:04:49.321 INFO:tasks.workunit.client.0.smithi144.stdout:7/582: dread - d0/dd0/dbf/d2f/d53/fc5 zero size 2024-07-26T11:04:49.321 INFO:tasks.workunit.client.0.smithi144.stdout:4/620: link d9/da/c16 d9/d26/d48/d98/db8/cd4 0 2024-07-26T11:04:49.322 INFO:tasks.workunit.client.0.smithi144.stdout:1/796: dwrite d3/d3d/d58/fe4 [0,4194304] 0 2024-07-26T11:04:49.323 INFO:tasks.workunit.client.0.smithi144.stdout:4/621: symlink d9/d26/d48/d98/db8/ld5 0 2024-07-26T11:04:49.328 INFO:tasks.workunit.client.0.smithi144.stdout:0/615: dwrite d0/d25/d5e/d1e/d30/fb9 [0,4194304] 0 2024-07-26T11:04:49.328 INFO:tasks.workunit.client.0.smithi144.stdout:1/797: mkdir d3/d6/da/d13/d19/d3a/d100 0 2024-07-26T11:04:49.328 INFO:tasks.workunit.client.0.smithi144.stdout:4/622: creat d9/d26/d48/d98/fd6 x:0 0 0 2024-07-26T11:04:49.329 INFO:tasks.workunit.client.0.smithi144.stdout:1/798: write d3/f3b [941806,69582] 0 2024-07-26T11:04:49.332 INFO:tasks.workunit.client.0.smithi144.stdout:1/799: rename d3/d6/da/d13/d19/d3a/d69 to d3/d3d/d58/d101 0 2024-07-26T11:04:49.332 INFO:tasks.workunit.client.0.smithi144.stdout:1/800: fsync d3/d6/da/d13/d19/d3a/d3e/ff0 0 2024-07-26T11:04:49.348 INFO:tasks.workunit.client.0.smithi144.stdout:0/616: read d0/d25/d5e/f41 [6545676,86290] 0 2024-07-26T11:04:49.356 INFO:tasks.workunit.client.0.smithi144.stdout:2/613: dwrite d0/d6/f1d [0,4194304] 0 2024-07-26T11:04:49.360 INFO:tasks.workunit.client.0.smithi144.stdout:6/737: dwrite d8/d50/d98/f9b [0,4194304] 0 2024-07-26T11:04:49.360 INFO:tasks.workunit.client.0.smithi144.stdout:6/738: chown d8/f82 0 1 2024-07-26T11:04:49.360 INFO:tasks.workunit.client.0.smithi144.stdout:6/739: chown d8/d50/d98/f9f 58352285 1 2024-07-26T11:04:49.362 INFO:tasks.workunit.client.0.smithi144.stdout:6/740: creat d8/d15/d57/f105 x:0 0 0 2024-07-26T11:04:49.363 INFO:tasks.workunit.client.0.smithi144.stdout:6/741: symlink d8/d50/l106 0 2024-07-26T11:04:49.365 INFO:tasks.workunit.client.0.smithi144.stdout:6/742: symlink d8/db/dbe/d52/ddb/l107 0 2024-07-26T11:04:49.377 INFO:tasks.workunit.client.0.smithi144.stdout:7/583: dwrite d0/dd0/dbf/d2f/d47/d5e/f69 [8388608,4194304] 0 2024-07-26T11:04:49.377 INFO:tasks.workunit.client.0.smithi144.stdout:7/584: write d0/dd0/dbf/d67/f9f [816284,41508] 0 2024-07-26T11:04:49.383 INFO:tasks.workunit.client.0.smithi144.stdout:7/585: symlink d0/d6/d55/ld2 0 2024-07-26T11:04:49.385 INFO:tasks.workunit.client.0.smithi144.stdout:7/586: mkdir d0/d6/d9/d3f/dd3 0 2024-07-26T11:04:49.385 INFO:tasks.workunit.client.0.smithi144.stdout:7/587: chown d0/dd0/dbf/f92 12 1 2024-07-26T11:04:49.385 INFO:tasks.workunit.client.0.smithi144.stdout:7/588: getdents d0/d6/d9/d3f/dd3 0 2024-07-26T11:04:49.385 INFO:tasks.workunit.client.0.smithi144.stdout:7/589: chown d0/dd0/dbf/d67/f9d 35020 1 2024-07-26T11:04:49.393 INFO:tasks.workunit.client.0.smithi144.stdout:1/801: dwrite d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:49.399 INFO:tasks.workunit.client.0.smithi144.stdout:2/614: dwrite d0/d24/d2b/d68/d82/fab [0,4194304] 0 2024-07-26T11:04:49.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/623: dwrite d9/da/f39 [0,4194304] 0 2024-07-26T11:04:49.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/624: chown d9/c78 1104805670 1 2024-07-26T11:04:49.425 INFO:tasks.workunit.client.0.smithi144.stdout:7/590: dwrite d0/dd0/dbf/d2f/d47/d5e/f69 [8388608,4194304] 0 2024-07-26T11:04:49.425 INFO:tasks.workunit.client.0.smithi144.stdout:7/591: read d0/d71/fbd [4186541,43729] 0 2024-07-26T11:04:49.428 INFO:tasks.workunit.client.0.smithi144.stdout:7/592: rename d0/dd0/dbf/d2f/d47/f91 to d0/d71/fd4 0 2024-07-26T11:04:49.428 INFO:tasks.workunit.client.0.smithi144.stdout:7/593: chown d0/dd0/dbf/d2f/d47/f4a 29226 1 2024-07-26T11:04:49.428 INFO:tasks.workunit.client.0.smithi144.stdout:7/594: fsync d0/dd0/dbf/d2f/d47/d5e/f63 0 2024-07-26T11:04:49.429 INFO:tasks.workunit.client.0.smithi144.stdout:7/595: write d0/f49 [2248208,56935] 0 2024-07-26T11:04:49.429 INFO:tasks.workunit.client.0.smithi144.stdout:7/596: write d0/d6/d9/d12/fb1 [1186105,39219] 0 2024-07-26T11:04:49.432 INFO:tasks.workunit.client.0.smithi144.stdout:7/597: truncate d0/dd0/dbf/d67/f9b 426813 0 2024-07-26T11:04:49.432 INFO:tasks.workunit.client.0.smithi144.stdout:7/598: write d0/d6/d9/d12/fb1 [1677338,129373] 0 2024-07-26T11:04:49.432 INFO:tasks.workunit.client.0.smithi144.stdout:1/802: dwrite d3/d6/da/d13/d19/d3a/d3e/f50 [0,4194304] 0 2024-07-26T11:04:49.433 INFO:tasks.workunit.client.0.smithi144.stdout:4/625: write d9/f4e [1652075,15005] 0 2024-07-26T11:04:49.434 INFO:tasks.workunit.client.0.smithi144.stdout:4/626: symlink d9/d57/ld7 0 2024-07-26T11:04:49.434 INFO:tasks.workunit.client.0.smithi144.stdout:4/627: fsync d9/d26/d48/d50/f6c 0 2024-07-26T11:04:49.436 INFO:tasks.workunit.client.0.smithi144.stdout:1/803: write d3/f2d [44106,13011] 0 2024-07-26T11:04:49.443 INFO:tasks.workunit.client.0.smithi144.stdout:1/804: write d3/d6/da/d13/d19/d3a/d3e/ddb/df6/dfe/f3c [561174,66884] 0 2024-07-26T11:04:49.444 INFO:tasks.workunit.client.0.smithi144.stdout:4/628: dread d9/d26/f99 [0,4194304] 0 2024-07-26T11:04:49.444 INFO:tasks.workunit.client.0.smithi144.stdout:4/629: fdatasync d9/d26/d48/d50/d60/f6e 0 2024-07-26T11:04:49.445 INFO:tasks.workunit.client.0.smithi144.stdout:4/630: chown d9/d26/d3c/d6f/dd0 18 1 2024-07-26T11:04:49.449 INFO:tasks.workunit.client.0.smithi144.stdout:4/631: mkdir d9/d26/d48/dd8 0 2024-07-26T11:04:49.450 INFO:tasks.workunit.client.0.smithi144.stdout:4/632: mkdir d9/d26/d48/d50/d60/dc5/dd9 0 2024-07-26T11:04:49.450 INFO:tasks.workunit.client.0.smithi144.stdout:4/633: truncate d9/d26/d3c/d6f/fb5 418598 0 2024-07-26T11:04:49.460 INFO:tasks.workunit.client.0.smithi144.stdout:4/634: dread d9/d26/d48/d98/fc2 [0,4194304] 0 2024-07-26T11:04:49.461 INFO:tasks.workunit.client.0.smithi144.stdout:4/635: mknod d9/d26/cda 0 2024-07-26T11:04:49.461 INFO:tasks.workunit.client.0.smithi144.stdout:4/636: chown d9/da/f69 199302384 1 2024-07-26T11:04:49.462 INFO:tasks.workunit.client.0.smithi144.stdout:4/637: creat d9/d26/d48/d50/d60/fdb x:0 0 0 2024-07-26T11:04:49.462 INFO:tasks.workunit.client.0.smithi144.stdout:4/638: chown d9/d26/f99 14313121 1 2024-07-26T11:04:49.464 INFO:tasks.workunit.client.0.smithi144.stdout:1/805: write d3/d6/d75/d88/da9/dac/fc4 [3025149,20649] 0 2024-07-26T11:04:49.464 INFO:tasks.workunit.client.0.smithi144.stdout:4/639: rename d9/da/c9e to d9/d26/d48/dd8/cdc 0 2024-07-26T11:04:49.464 INFO:tasks.workunit.client.0.smithi144.stdout:4/640: write d9/d26/d3c/d6f/fb5 [1151072,53720] 0 2024-07-26T11:04:49.464 INFO:tasks.workunit.client.0.smithi144.stdout:4/641: write d9/d26/d3c/d6f/f81 [996163,82643] 0 2024-07-26T11:04:49.465 INFO:tasks.workunit.client.0.smithi144.stdout:1/806: rename d3/d6/da/d21/d5c/d62/d68/f92 to d3/d6/f102 0 2024-07-26T11:04:49.466 INFO:tasks.workunit.client.0.smithi144.stdout:4/642: mkdir d9/da/ddd 0 2024-07-26T11:04:49.467 INFO:tasks.workunit.client.0.smithi144.stdout:1/807: mknod d3/d6/da/d21/d5c/d62/c103 0 2024-07-26T11:04:49.473 INFO:tasks.workunit.client.0.smithi144.stdout:1/808: getdents d3/d6 0 2024-07-26T11:04:49.474 INFO:tasks.workunit.client.0.smithi144.stdout:7/599: dwrite d0/dd0/dbf/f41 [0,4194304] 0 2024-07-26T11:04:49.476 INFO:tasks.workunit.client.0.smithi144.stdout:1/809: rename d3/d6/da/d13/d19/d3a/d3e to d3/d6/d75/d88/da9/dac/d104 0 2024-07-26T11:04:49.477 INFO:tasks.workunit.client.0.smithi144.stdout:7/600: truncate d0/d6/d55/f6a 1566301 0 2024-07-26T11:04:49.479 INFO:tasks.workunit.client.0.smithi144.stdout:7/601: creat d0/dd0/dbf/d2f/d47/fd5 x:0 0 0 2024-07-26T11:04:49.481 INFO:tasks.workunit.client.0.smithi144.stdout:7/602: symlink d0/dd0/dbf/d2f/ld6 0 2024-07-26T11:04:49.492 INFO:tasks.workunit.client.0.smithi144.stdout:9/658: sync 2024-07-26T11:04:49.492 INFO:tasks.workunit.client.0.smithi144.stdout:9/659: chown d9/de/d14/d23/l28 6 1 2024-07-26T11:04:49.530 INFO:tasks.workunit.client.0.smithi144.stdout:4/643: dwrite d9/d26/d3c/d6f/f86 [0,4194304] 0 2024-07-26T11:04:49.530 INFO:tasks.workunit.client.0.smithi144.stdout:7/603: dread d0/d6/d9/d12/f60 [0,4194304] 0 2024-07-26T11:04:49.531 INFO:tasks.workunit.client.0.smithi144.stdout:4/644: rename d9/fc0 to d9/d26/d48/d50/d60/dc5/dd9/fde 0 2024-07-26T11:04:49.532 INFO:tasks.workunit.client.0.smithi144.stdout:4/645: chown d9/d26/d48/l6d 1 1 2024-07-26T11:04:49.532 INFO:tasks.workunit.client.0.smithi144.stdout:7/604: symlink d0/dd0/dbf/d2f/d53/ld7 0 2024-07-26T11:04:49.539 INFO:tasks.workunit.client.0.smithi144.stdout:7/605: read d0/d6/d9/d12/fb1 [3029,52585] 0 2024-07-26T11:04:49.540 INFO:tasks.workunit.client.0.smithi144.stdout:7/606: link d0/dd0/dbf/d2f/d47/fb7 d0/d6/d9/d12/d6c/fd8 0 2024-07-26T11:04:49.541 INFO:tasks.workunit.client.0.smithi144.stdout:7/607: mknod d0/d6/d9/d12/d6c/cd9 0 2024-07-26T11:04:49.542 INFO:tasks.workunit.client.0.smithi144.stdout:7/608: rename d0 to d0/dd0/dbf/d2f/d68/dda 22 2024-07-26T11:04:49.543 INFO:tasks.workunit.client.0.smithi144.stdout:7/609: mknod d0/d6/d9/d3f/cdb 0 2024-07-26T11:04:49.543 INFO:tasks.workunit.client.0.smithi144.stdout:7/610: fsync d0/d6/d9/d45/d57/f7b 0 2024-07-26T11:04:49.543 INFO:tasks.workunit.client.0.smithi144.stdout:7/611: creat d0/dd0/dbf/d2f/d53/fdc x:0 0 0 2024-07-26T11:04:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:49 smithi160 ceph-mon[36715]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s rd, 158 MiB/s wr, 625 op/s 2024-07-26T11:04:49.562 INFO:tasks.workunit.client.0.smithi144.stdout:7/612: dread d0/dd0/dbf/f42 [0,4194304] 0 2024-07-26T11:04:49.564 INFO:tasks.workunit.client.0.smithi144.stdout:7/613: symlink d0/dd0/dbf/d67/ldd 0 2024-07-26T11:04:49.564 INFO:tasks.workunit.client.0.smithi144.stdout:7/614: stat d0/d6/f52 0 2024-07-26T11:04:49.564 INFO:tasks.workunit.client.0.smithi144.stdout:7/615: write d0/d6/d9/d45/d57/f7b [474535,126322] 0 2024-07-26T11:04:49.564 INFO:tasks.workunit.client.0.smithi144.stdout:7/616: read - d0/dd0/dbf/d2f/d68/f8d zero size 2024-07-26T11:04:49.564 INFO:tasks.workunit.client.0.smithi144.stdout:1/810: dwrite d3/d6/da/fca [0,4194304] 0 2024-07-26T11:04:49.566 INFO:tasks.workunit.client.0.smithi144.stdout:7/617: rmdir d0 39 2024-07-26T11:04:49.567 INFO:tasks.workunit.client.0.smithi144.stdout:1/811: rename d3/d3d/c72 to d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/de1/c105 0 2024-07-26T11:04:49.568 INFO:tasks.workunit.client.0.smithi144.stdout:7/618: symlink d0/d6/d9/d45/d57/lde 0 2024-07-26T11:04:49.568 INFO:tasks.workunit.client.0.smithi144.stdout:7/619: fsync d0/dd0/dbf/d2f/d47/d5e/f88 0 2024-07-26T11:04:49.568 INFO:tasks.workunit.client.0.smithi144.stdout:7/620: dread - d0/dd0/dbf/d2f/da2/fa6 zero size 2024-07-26T11:04:49.571 INFO:tasks.workunit.client.0.smithi144.stdout:1/812: rmdir d3/d6 39 2024-07-26T11:04:49.572 INFO:tasks.workunit.client.0.smithi144.stdout:9/660: dwrite d9/d3b/d5d/f8b [0,4194304] 0 2024-07-26T11:04:49.572 INFO:tasks.workunit.client.0.smithi144.stdout:9/661: readlink d9/l6f 0 2024-07-26T11:04:49.574 INFO:tasks.workunit.client.0.smithi144.stdout:9/662: unlink d9/de/d4e/d6a/f8a 0 2024-07-26T11:04:49.584 INFO:tasks.workunit.client.0.smithi144.stdout:9/663: dread d9/de/d14/d23/d39/f5b [0,4194304] 0 2024-07-26T11:04:49.586 INFO:tasks.workunit.client.0.smithi144.stdout:9/664: symlink d9/d3b/d81/ld6 0 2024-07-26T11:04:49.587 INFO:tasks.workunit.client.0.smithi144.stdout:9/665: mkdir d9/d3b/d76/d9b/dd7 0 2024-07-26T11:04:49.588 INFO:tasks.workunit.client.0.smithi144.stdout:7/621: write d0/dd0/dbf/d2f/d47/d5e/fba [349056,46677] 0 2024-07-26T11:04:49.588 INFO:tasks.workunit.client.0.smithi144.stdout:9/666: creat d9/de/fd8 x:0 0 0 2024-07-26T11:04:49.588 INFO:tasks.workunit.client.0.smithi144.stdout:9/667: chown d9/d3b/l3c 22692 1 2024-07-26T11:04:49.590 INFO:tasks.workunit.client.0.smithi144.stdout:7/622: rename d0/d71/fbd to d0/d71/dc0/fdf 0 2024-07-26T11:04:49.590 INFO:tasks.workunit.client.0.smithi144.stdout:7/623: readlink d0/d6/d9/d45/l7d 0 2024-07-26T11:04:49.591 INFO:tasks.workunit.client.0.smithi144.stdout:9/668: unlink d9/de/d14/d23/f50 0 2024-07-26T11:04:49.595 INFO:tasks.workunit.client.0.smithi144.stdout:7/624: mkdir d0/dd0/dbf/d67/de0 0 2024-07-26T11:04:49.596 INFO:tasks.workunit.client.0.smithi144.stdout:7/625: creat d0/dd0/dbf/d2f/d47/d5e/fe1 x:0 0 0 2024-07-26T11:04:49.598 INFO:tasks.workunit.client.0.smithi144.stdout:7/626: mkdir d0/dd0/dbf/d2f/d68/de2 0 2024-07-26T11:04:49.598 INFO:tasks.workunit.client.0.smithi144.stdout:7/627: chown d0/d6/f52 64700377 1 2024-07-26T11:04:49.598 INFO:tasks.workunit.client.0.smithi144.stdout:7/628: readlink d0/d6/d9/d12/l8c 0 2024-07-26T11:04:49.598 INFO:tasks.workunit.client.0.smithi144.stdout:7/629: write d0/d71/da1/faf [762883,5729] 0 2024-07-26T11:04:49.599 INFO:tasks.workunit.client.0.smithi144.stdout:7/630: symlink d0/dd0/dbf/d2f/d47/le3 0 2024-07-26T11:04:49.602 INFO:tasks.workunit.client.0.smithi144.stdout:7/631: getdents d0/dd0/dbf/d2f/d53 0 2024-07-26T11:04:49.602 INFO:tasks.workunit.client.0.smithi144.stdout:7/632: chown d0/d6/d9/f13 1 1 2024-07-26T11:04:49.603 INFO:tasks.workunit.client.0.smithi144.stdout:7/633: mkdir d0/d71/dc0/de4 0 2024-07-26T11:04:49.611 INFO:tasks.workunit.client.0.smithi144.stdout:9/669: write d9/de/d14/d23/d39/fab [1649331,76090] 0 2024-07-26T11:04:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:49 smithi183 ceph-mon[37273]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s rd, 158 MiB/s wr, 625 op/s 2024-07-26T11:04:49.642 INFO:tasks.workunit.client.0.smithi144.stdout:9/670: write d9/d3b/d81/fb4 [18733,7165] 0 2024-07-26T11:04:49.642 INFO:tasks.workunit.client.0.smithi144.stdout:9/671: chown d9/de/d14/d3f/d5e/ccf 103414000 1 2024-07-26T11:04:49.644 INFO:tasks.workunit.client.0.smithi144.stdout:9/672: creat d9/de/d14/fd9 x:0 0 0 2024-07-26T11:04:49.644 INFO:tasks.workunit.client.0.smithi144.stdout:9/673: chown d9/de/d14/d3f/d5e/caf 28502383 1 2024-07-26T11:04:49.645 INFO:tasks.workunit.client.0.smithi144.stdout:9/674: rmdir d9/de/d14/d23/d39/d7f 39 2024-07-26T11:04:49.665 INFO:tasks.workunit.client.0.smithi144.stdout:7/634: dwrite d0/dd0/dbf/d2f/f8a [0,4194304] 0 2024-07-26T11:04:49.668 INFO:tasks.workunit.client.0.smithi144.stdout:7/635: unlink d0/d6/d9/d12/fb1 0 2024-07-26T11:04:49.669 INFO:tasks.workunit.client.0.smithi144.stdout:7/636: creat d0/d6/d9/d45/fe5 x:0 0 0 2024-07-26T11:04:49.671 INFO:tasks.workunit.client.0.smithi144.stdout:7/637: unlink d0/dd0/dbf/d67/ldd 0 2024-07-26T11:04:49.690 INFO:tasks.workunit.client.0.smithi144.stdout:3/430: sync 2024-07-26T11:04:49.692 INFO:tasks.workunit.client.0.smithi144.stdout:5/631: sync 2024-07-26T11:04:49.692 INFO:tasks.workunit.client.0.smithi144.stdout:8/584: sync 2024-07-26T11:04:49.692 INFO:tasks.workunit.client.0.smithi144.stdout:8/585: stat d1a/d32/d49/d73/d2c/d38/dbd/cca 0 2024-07-26T11:04:49.692 INFO:tasks.workunit.client.0.smithi144.stdout:5/632: read d1/d18/d24/d35/f9b [254666,1038] 0 2024-07-26T11:04:49.692 INFO:tasks.workunit.client.0.smithi144.stdout:8/586: chown d1a/d1c 777613123 1 2024-07-26T11:04:49.692 INFO:tasks.workunit.client.0.smithi144.stdout:5/633: dread - d1/d9/fd1 zero size 2024-07-26T11:04:49.693 INFO:tasks.workunit.client.0.smithi144.stdout:3/431: rename d1/d28/f3b to d1/d28/d54/d26/d64/d68/f7d 0 2024-07-26T11:04:49.693 INFO:tasks.workunit.client.0.smithi144.stdout:3/432: write d1/d28/d54/d3d/d42/d44/f5f [341317,35818] 0 2024-07-26T11:04:49.693 INFO:tasks.workunit.client.0.smithi144.stdout:5/634: rmdir d1 39 2024-07-26T11:04:49.694 INFO:tasks.workunit.client.0.smithi144.stdout:3/433: fdatasync d1/f39 0 2024-07-26T11:04:49.694 INFO:tasks.workunit.client.0.smithi144.stdout:8/587: truncate d1a/f20 3700259 0 2024-07-26T11:04:49.696 INFO:tasks.workunit.client.0.smithi144.stdout:3/434: mkdir d1/d28/d7e 0 2024-07-26T11:04:49.696 INFO:tasks.workunit.client.0.smithi144.stdout:8/588: symlink d1a/d1c/lcc 0 2024-07-26T11:04:49.696 INFO:tasks.workunit.client.0.smithi144.stdout:8/589: chown d1a/d32/d49/d73/d7b/d43/d4f/la5 2 1 2024-07-26T11:04:49.697 INFO:tasks.workunit.client.0.smithi144.stdout:3/435: creat d1/d28/d54/d26/d30/f7f x:0 0 0 2024-07-26T11:04:49.697 INFO:tasks.workunit.client.0.smithi144.stdout:3/436: fsync d1/f1a 0 2024-07-26T11:04:49.698 INFO:tasks.workunit.client.0.smithi144.stdout:9/675: dread d9/de/ff [0,4194304] 0 2024-07-26T11:04:49.698 INFO:tasks.workunit.client.0.smithi144.stdout:3/437: read d1/d28/d54/d26/f2d [2333431,58295] 0 2024-07-26T11:04:49.698 INFO:tasks.workunit.client.0.smithi144.stdout:3/438: chown d1/d28/d54/d26/d64 545 1 2024-07-26T11:04:49.698 INFO:tasks.workunit.client.0.smithi144.stdout:9/676: mknod d9/de/d14/d3f/cda 0 2024-07-26T11:04:49.700 INFO:tasks.workunit.client.0.smithi144.stdout:3/439: mkdir d1/d28/d54/d26/d30/d78/d80 0 2024-07-26T11:04:49.701 INFO:tasks.workunit.client.0.smithi144.stdout:9/677: symlink d9/d3b/ldb 0 2024-07-26T11:04:49.705 INFO:tasks.workunit.client.0.smithi144.stdout:3/440: stat d1/d28/d54/d3d/d42/l72 0 2024-07-26T11:04:49.706 INFO:tasks.workunit.client.0.smithi144.stdout:3/441: creat d1/d28/d54/d26/d64/f81 x:0 0 0 2024-07-26T11:04:49.710 INFO:tasks.workunit.client.0.smithi144.stdout:7/638: write d0/d6/f17 [1665428,127440] 0 2024-07-26T11:04:49.710 INFO:tasks.workunit.client.0.smithi144.stdout:7/639: truncate d0/dd0/dbf/d2f/d47/fae 956783 0 2024-07-26T11:04:49.711 INFO:tasks.workunit.client.0.smithi144.stdout:7/640: chown d0/d6/f56 2681 1 2024-07-26T11:04:49.713 INFO:tasks.workunit.client.0.smithi144.stdout:7/641: rename d0/dd0/dbf/d67/fa7 to d0/dd0/dbf/d2f/d53/db6/fe6 0 2024-07-26T11:04:49.723 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:49 smithi144 ceph-mon[32129]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s rd, 158 MiB/s wr, 625 op/s 2024-07-26T11:04:49.759 INFO:tasks.workunit.client.0.smithi144.stdout:8/590: dwrite d1a/f95 [0,4194304] 0 2024-07-26T11:04:49.763 INFO:tasks.workunit.client.0.smithi144.stdout:8/591: truncate d1a/f21 3767599 0 2024-07-26T11:04:49.763 INFO:tasks.workunit.client.0.smithi144.stdout:5/635: dwrite d1/d18/d24/d27/fcb [0,4194304] 0 2024-07-26T11:04:49.766 INFO:tasks.workunit.client.0.smithi144.stdout:5/636: mkdir d1/d9/d44/d45/d81/dd4 0 2024-07-26T11:04:49.776 INFO:tasks.workunit.client.0.smithi144.stdout:9/678: dwrite d9/de/d14/d3f/d5e/dcd/fd3 [0,4194304] 0 2024-07-26T11:04:49.776 INFO:tasks.workunit.client.0.smithi144.stdout:9/679: truncate d9/d3b/f56 4194544 0 2024-07-26T11:04:49.776 INFO:tasks.workunit.client.0.smithi144.stdout:9/680: fsync d9/de/d4e/f93 0 2024-07-26T11:04:49.776 INFO:tasks.workunit.client.0.smithi144.stdout:3/442: dwrite d1/d28/d54/d26/f47 [0,4194304] 0 2024-07-26T11:04:49.785 INFO:tasks.workunit.client.0.smithi144.stdout:3/443: mknod d1/d28/d54/d3d/d42/d44/d60/c82 0 2024-07-26T11:04:49.786 INFO:tasks.workunit.client.0.smithi144.stdout:5/637: dread d1/d5/f78 [4194304,4194304] 0 2024-07-26T11:04:49.786 INFO:tasks.workunit.client.0.smithi144.stdout:9/681: truncate d9/de/f4d 478466 0 2024-07-26T11:04:49.788 INFO:tasks.workunit.client.0.smithi144.stdout:3/444: unlink d1/d28/d54/l73 0 2024-07-26T11:04:49.788 INFO:tasks.workunit.client.0.smithi144.stdout:3/445: write d1/d28/d54/d3d/d42/d44/f4e [804563,54530] 0 2024-07-26T11:04:49.789 INFO:tasks.workunit.client.0.smithi144.stdout:5/638: rename d1/d18/l40 to d1/d5/dae/ld5 0 2024-07-26T11:04:49.789 INFO:tasks.workunit.client.0.smithi144.stdout:5/639: write d1/d5/dae/fb5 [1006267,75797] 0 2024-07-26T11:04:49.792 INFO:tasks.workunit.client.0.smithi144.stdout:3/446: rename d1/d28/d54/d3d/d42/d44/d60/f4b to d1/d5/d5e/f83 0 2024-07-26T11:04:49.792 INFO:tasks.workunit.client.0.smithi144.stdout:5/640: chown d1/d9/d44/l3c 495 1 2024-07-26T11:04:49.792 INFO:tasks.workunit.client.0.smithi144.stdout:5/641: fsync d1/d9/d44/d45/d98/faf 0 2024-07-26T11:04:49.804 INFO:tasks.workunit.client.0.smithi144.stdout:1/813: sync 2024-07-26T11:04:49.805 INFO:tasks.workunit.client.0.smithi144.stdout:1/814: write d3/d6/da/d21/f24 [2943880,7847] 0 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:6/743: sync 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:0/617: sync 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:4/646: sync 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:4/647: chown d9/d26/d48/d98/db8/cd4 32 1 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:4/648: fdatasync d9/d26/f83 0 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:1/815: mknod d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/c106 0 2024-07-26T11:04:49.808 INFO:tasks.workunit.client.0.smithi144.stdout:2/615: sync 2024-07-26T11:04:49.809 INFO:tasks.workunit.client.0.smithi144.stdout:2/616: getdents d0/d24/d48/d6d/d28/d27/d8b 0 2024-07-26T11:04:49.815 INFO:tasks.workunit.client.0.smithi144.stdout:6/744: rmdir d8/d50 39 2024-07-26T11:04:49.815 INFO:tasks.workunit.client.0.smithi144.stdout:4/649: symlink d9/d26/d48/d98/ldf 0 2024-07-26T11:04:49.815 INFO:tasks.workunit.client.0.smithi144.stdout:4/650: read - d9/da/fad zero size 2024-07-26T11:04:49.815 INFO:tasks.workunit.client.0.smithi144.stdout:4/651: fdatasync d9/d26/f92 0 2024-07-26T11:04:49.816 INFO:tasks.workunit.client.0.smithi144.stdout:4/652: creat d9/da/ddd/fe0 x:0 0 0 2024-07-26T11:04:49.821 INFO:tasks.workunit.client.0.smithi144.stdout:6/745: rename d8/db/fa8 to d8/db/da6/f108 0 2024-07-26T11:04:49.821 INFO:tasks.workunit.client.0.smithi144.stdout:2/617: write d0/d24/d48/d6d/d28/d2e/d77/f40 [1057275,74481] 0 2024-07-26T11:04:49.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/653: mkdir d9/d26/d48/d50/d60/dc5/dd9/de1 0 2024-07-26T11:04:49.826 INFO:tasks.workunit.client.0.smithi144.stdout:2/618: link d0/d6/c1a d0/d24/d48/d6d/d28/cbe 0 2024-07-26T11:04:49.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/654: fsync d9/d26/fcc 0 2024-07-26T11:04:49.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/655: write d9/f51 [1540998,57139] 0 2024-07-26T11:04:49.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/656: rename d9/d26/d48/d50/d60/fdb to d9/d26/d48/dd8/fe2 0 2024-07-26T11:04:49.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/657: write d9/d7b/dc4/fc7 [418407,46838] 0 2024-07-26T11:04:49.836 INFO:tasks.workunit.client.0.smithi144.stdout:3/447: dwrite d1/d28/d54/d3d/f40 [0,4194304] 0 2024-07-26T11:04:49.837 INFO:tasks.workunit.client.0.smithi144.stdout:5/642: dread d1/d9/d44/f4e [0,4194304] 0 2024-07-26T11:04:49.838 INFO:tasks.workunit.client.0.smithi144.stdout:5/643: truncate d1/d18/d24/f76 209859 0 2024-07-26T11:04:49.838 INFO:tasks.workunit.client.0.smithi144.stdout:5/644: fsync d1/d9/d5b/d4d/f63 0 2024-07-26T11:04:49.842 INFO:tasks.workunit.client.0.smithi144.stdout:3/448: link d1/d28/d54/d3d/d42/d44/f4e d1/d28/d54/d26/d64/d68/f84 0 2024-07-26T11:04:49.842 INFO:tasks.workunit.client.0.smithi144.stdout:8/592: dwrite d1a/f1b [0,4194304] 0 2024-07-26T11:04:49.843 INFO:tasks.workunit.client.0.smithi144.stdout:8/593: readlink d1a/d32/d49/d73/d7b/d43/d4f/la1 0 2024-07-26T11:04:49.843 INFO:tasks.workunit.client.0.smithi144.stdout:8/594: rename d1a/d32/d6c/d3c to d1a/d32/d66/dcd 0 2024-07-26T11:04:49.845 INFO:tasks.workunit.client.0.smithi144.stdout:8/595: unlink d1a/l92 0 2024-07-26T11:04:49.845 INFO:tasks.workunit.client.0.smithi144.stdout:8/596: dread - d1a/d32/d49/fb7 zero size 2024-07-26T11:04:49.845 INFO:tasks.workunit.client.0.smithi144.stdout:8/597: chown d1a/d32/f71 14394875 1 2024-07-26T11:04:49.846 INFO:tasks.workunit.client.0.smithi144.stdout:9/682: dwrite d9/de/ff [4194304,4194304] 0 2024-07-26T11:04:49.846 INFO:tasks.workunit.client.0.smithi144.stdout:8/598: getdents d1a/d32/d6c/d82 0 2024-07-26T11:04:49.847 INFO:tasks.workunit.client.0.smithi144.stdout:8/599: fsync d1a/d32/d49/d73/d7b/f40 0 2024-07-26T11:04:49.848 INFO:tasks.workunit.client.0.smithi144.stdout:9/683: symlink d9/d3b/d81/ldc 0 2024-07-26T11:04:49.854 INFO:tasks.workunit.client.0.smithi144.stdout:8/600: symlink d1a/d32/d6c/lce 0 2024-07-26T11:04:49.854 INFO:tasks.workunit.client.0.smithi144.stdout:9/684: mkdir d9/d3b/d81/ddd 0 2024-07-26T11:04:49.854 INFO:tasks.workunit.client.0.smithi144.stdout:9/685: creat d9/d3b/d76/d9b/dcc/fde x:0 0 0 2024-07-26T11:04:49.855 INFO:tasks.workunit.client.0.smithi144.stdout:2/619: dread d0/d6/d9/faa [0,4194304] 0 2024-07-26T11:04:49.855 INFO:tasks.workunit.client.0.smithi144.stdout:9/686: creat d9/de/d14/d3f/d5e/fdf x:0 0 0 2024-07-26T11:04:49.855 INFO:tasks.workunit.client.0.smithi144.stdout:2/620: chown d0/d24/d2b/d68/d82/f7c 1448149113 1 2024-07-26T11:04:49.855 INFO:tasks.workunit.client.0.smithi144.stdout:9/687: link d9/de/d14/d79/c94 d9/de/d14/d3f/ce0 0 2024-07-26T11:04:49.857 INFO:tasks.workunit.client.0.smithi144.stdout:2/621: creat d0/d24/d48/d6d/d28/d27/fbf x:0 0 0 2024-07-26T11:04:49.857 INFO:tasks.workunit.client.0.smithi144.stdout:2/622: write d0/f3d [4748439,83941] 0 2024-07-26T11:04:49.858 INFO:tasks.workunit.client.0.smithi144.stdout:9/688: creat d9/de/d14/d3f/d5e/dcd/fe1 x:0 0 0 2024-07-26T11:04:49.862 INFO:tasks.workunit.client.0.smithi144.stdout:2/623: rmdir d0/d24/d48 39 2024-07-26T11:04:49.862 INFO:tasks.workunit.client.0.smithi144.stdout:6/746: dread d8/db/dbe/fc9 [0,4194304] 0 2024-07-26T11:04:49.862 INFO:tasks.workunit.client.0.smithi144.stdout:6/747: dread - d8/f1a zero size 2024-07-26T11:04:49.863 INFO:tasks.workunit.client.0.smithi144.stdout:6/748: dread - d8/d15/d57/d5f/d88/ff3 zero size 2024-07-26T11:04:49.863 INFO:tasks.workunit.client.0.smithi144.stdout:6/749: write d8/d15/d26/f3f [665966,81557] 0 2024-07-26T11:04:49.864 INFO:tasks.workunit.client.0.smithi144.stdout:3/449: write d1/d28/d54/d26/f2e [6446372,116747] 0 2024-07-26T11:04:49.865 INFO:tasks.workunit.client.0.smithi144.stdout:6/750: symlink d8/d15/d57/d5f/dd3/l109 0 2024-07-26T11:04:49.866 INFO:tasks.workunit.client.0.smithi144.stdout:3/450: creat d1/d28/d54/f85 x:0 0 0 2024-07-26T11:04:49.868 INFO:tasks.workunit.client.0.smithi144.stdout:6/751: creat d8/d15/d57/d5f/f10a x:0 0 0 2024-07-26T11:04:49.869 INFO:tasks.workunit.client.0.smithi144.stdout:6/752: chown d8/d15/d57/d5f/d88/f95 205511 1 2024-07-26T11:04:49.869 INFO:tasks.workunit.client.0.smithi144.stdout:6/753: write d8/db/dbe/d52/fcb [209544,3248] 0 2024-07-26T11:04:49.878 INFO:tasks.workunit.client.0.smithi144.stdout:5/645: write d1/d9/d44/f4e [987853,54364] 0 2024-07-26T11:04:49.885 INFO:tasks.workunit.client.0.smithi144.stdout:1/816: dwrite d3/d6/d75/d88/da9/dac/d104/f50 [0,4194304] 0 2024-07-26T11:04:49.885 INFO:tasks.workunit.client.0.smithi144.stdout:0/618: dwrite d0/d25/d5e/d1e/dc2/f9e [0,4194304] 0 2024-07-26T11:04:49.885 INFO:tasks.workunit.client.0.smithi144.stdout:6/754: creat d8/d50/d98/f10b x:0 0 0 2024-07-26T11:04:49.886 INFO:tasks.workunit.client.0.smithi144.stdout:5/646: creat d1/d18/d24/fd6 x:0 0 0 2024-07-26T11:04:49.886 INFO:tasks.workunit.client.0.smithi144.stdout:8/601: dread d1a/f21 [0,4194304] 0 2024-07-26T11:04:49.891 INFO:tasks.workunit.client.0.smithi144.stdout:6/755: mknod d8/d50/d92/db3/c10c 0 2024-07-26T11:04:49.891 INFO:tasks.workunit.client.0.smithi144.stdout:5/647: mkdir d1/d18/d24/d35/dd7 0 2024-07-26T11:04:49.892 INFO:tasks.workunit.client.0.smithi144.stdout:0/619: read d0/d25/d5e/d1e/d30/f39 [440591,113308] 0 2024-07-26T11:04:49.892 INFO:tasks.workunit.client.0.smithi144.stdout:8/602: getdents d1a/d32/d49/d73/d2c/d38/d58/da3/db1 0 2024-07-26T11:04:49.895 INFO:tasks.workunit.client.0.smithi144.stdout:6/756: rmdir d8/d50/d92/db3 39 2024-07-26T11:04:49.895 INFO:tasks.workunit.client.0.smithi144.stdout:8/603: unlink d1a/d32/cbb 0 2024-07-26T11:04:49.910 INFO:tasks.workunit.client.0.smithi144.stdout:1/817: dread d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/faf [0,4194304] 0 2024-07-26T11:04:49.910 INFO:tasks.workunit.client.0.smithi144.stdout:1/818: truncate d3/d3d/d58/d101/d36/f96 609370 0 2024-07-26T11:04:49.912 INFO:tasks.workunit.client.0.smithi144.stdout:1/819: link d3/d3d/d58/d101/db7/c85 d3/d6/d75/d88/da9/dac/d104/ddb/c107 0 2024-07-26T11:04:49.914 INFO:tasks.workunit.client.0.smithi144.stdout:1/820: creat d3/d3d/d58/d101/d36/db6/f108 x:0 0 0 2024-07-26T11:04:49.921 INFO:tasks.workunit.client.0.smithi144.stdout:9/689: dwrite d9/de/d14/d3f/d5e/dcd/fd3 [0,4194304] 0 2024-07-26T11:04:49.921 INFO:tasks.workunit.client.0.smithi144.stdout:9/690: chown d9/de/d14/d79/f82 16288 1 2024-07-26T11:04:49.925 INFO:tasks.workunit.client.0.smithi144.stdout:9/691: mkdir d9/de/d14/d79/db2/de2 0 2024-07-26T11:04:49.926 INFO:tasks.workunit.client.0.smithi144.stdout:9/692: creat d9/d3b/d5d/fe3 x:0 0 0 2024-07-26T11:04:49.931 INFO:tasks.workunit.client.0.smithi144.stdout:0/620: write d0/d25/d5e/d1e/d30/fb9 [1489907,72167] 0 2024-07-26T11:04:49.932 INFO:tasks.workunit.client.0.smithi144.stdout:0/621: dread - d0/d3e/d45/d9f/d65/fb5 zero size 2024-07-26T11:04:49.961 INFO:tasks.workunit.client.0.smithi144.stdout:7/642: sync 2024-07-26T11:04:49.963 INFO:tasks.workunit.client.0.smithi144.stdout:7/643: mkdir d0/d71/d95/de7 0 2024-07-26T11:04:49.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/451: write d1/d28/d54/d26/f2b [984611,3974] 0 2024-07-26T11:04:49.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/452: truncate d1/d28/d54/d26/d64/d68/f7d 858345 0 2024-07-26T11:04:49.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/453: write d1/d28/d54/f85 [314037,94310] 0 2024-07-26T11:04:49.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/454: dread - d1/d28/d54/d26/f33 zero size 2024-07-26T11:04:49.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/455: chown d1/d28/d54/d3d/d42/d44/f5f 225052 1 2024-07-26T11:04:49.969 INFO:tasks.workunit.client.0.smithi144.stdout:5/648: dwrite d1/d18/d24/d27/f4f [4194304,4194304] 0 2024-07-26T11:04:49.969 INFO:tasks.workunit.client.0.smithi144.stdout:2/624: dwrite d0/d24/d2b/f59 [0,4194304] 0 2024-07-26T11:04:49.976 INFO:tasks.workunit.client.0.smithi144.stdout:3/456: link d1/d28/d54/d3d/d42/d44/d60/c82 d1/d28/d54/d26/d30/d78/c86 0 2024-07-26T11:04:49.976 INFO:tasks.workunit.client.0.smithi144.stdout:3/457: write d1/d28/d54/d26/d30/f5d [997410,47749] 0 2024-07-26T11:04:49.976 INFO:tasks.workunit.client.0.smithi144.stdout:5/649: link d1/d18/d24/fa3 d1/d9/d44/d45/d98/db2/fd8 0 2024-07-26T11:04:49.978 INFO:tasks.workunit.client.0.smithi144.stdout:2/625: symlink d0/d24/lc0 0 2024-07-26T11:04:49.978 INFO:tasks.workunit.client.0.smithi144.stdout:5/650: link d1/f8f d1/d9/d44/d45/d98/fd9 0 2024-07-26T11:04:49.978 INFO:tasks.workunit.client.0.smithi144.stdout:7/644: dread d0/d6/d55/f6a [0,4194304] 0 2024-07-26T11:04:49.978 INFO:tasks.workunit.client.0.smithi144.stdout:2/626: chown d0/d24 0 1 2024-07-26T11:04:49.979 INFO:tasks.workunit.client.0.smithi144.stdout:2/627: write d0/d6/d9/f26 [2843774,52067] 0 2024-07-26T11:04:49.979 INFO:tasks.workunit.client.0.smithi144.stdout:9/693: dwrite d9/de/d4e/f93 [0,4194304] 0 2024-07-26T11:04:49.981 INFO:tasks.workunit.client.0.smithi144.stdout:6/757: dwrite d8/d15/d57/d5f/f84 [0,4194304] 0 2024-07-26T11:04:49.981 INFO:tasks.workunit.client.0.smithi144.stdout:6/758: dread - d8/d42/dad/fd8 zero size 2024-07-26T11:04:49.984 INFO:tasks.workunit.client.0.smithi144.stdout:7/645: mkdir d0/d71/d95/dc4/de8 0 2024-07-26T11:04:49.984 INFO:tasks.workunit.client.0.smithi144.stdout:5/651: rename d1/d18/d24/d27/c52 to d1/d9/d44/d85/cda 0 2024-07-26T11:04:49.984 INFO:tasks.workunit.client.0.smithi144.stdout:5/652: chown d1/d9 60082024 1 2024-07-26T11:04:49.984 INFO:tasks.workunit.client.0.smithi144.stdout:5/653: stat d1/d9/l30 0 2024-07-26T11:04:49.985 INFO:tasks.workunit.client.0.smithi144.stdout:0/622: dwrite d0/d3e/d45/d9f/d65/fbb [0,4194304] 0 2024-07-26T11:04:49.986 INFO:tasks.workunit.client.0.smithi144.stdout:9/694: mkdir d9/d3b/dce/de4 0 2024-07-26T11:04:49.986 INFO:tasks.workunit.client.0.smithi144.stdout:6/759: creat d8/d15/d57/d5f/f10d x:0 0 0 2024-07-26T11:04:49.990 INFO:tasks.workunit.client.0.smithi144.stdout:9/695: rename d9/de/d14/d3f/f53 to d9/de/d14/d79/db2/de2/fe5 0 2024-07-26T11:04:49.992 INFO:tasks.workunit.client.0.smithi144.stdout:0/623: unlink d0/l44 0 2024-07-26T11:04:49.992 INFO:tasks.workunit.client.0.smithi144.stdout:0/624: chown d0/d25/l47 884286 1 2024-07-26T11:04:49.993 INFO:tasks.workunit.client.0.smithi144.stdout:5/654: mkdir d1/d9/d44/da1/ddb 0 2024-07-26T11:04:49.997 INFO:tasks.workunit.client.0.smithi144.stdout:9/696: link d9/fc d9/d3b/dce/fe6 0 2024-07-26T11:04:49.997 INFO:tasks.workunit.client.0.smithi144.stdout:0/625: mknod d0/d25/d5e/d1e/d30/dab/cce 0 2024-07-26T11:04:50.001 INFO:tasks.workunit.client.0.smithi144.stdout:5/655: creat d1/d18/dc4/fdc x:0 0 0 2024-07-26T11:04:50.002 INFO:tasks.workunit.client.0.smithi144.stdout:9/697: rename d9/de/d14/cb6 to d9/de/d14/ce7 0 2024-07-26T11:04:50.002 INFO:tasks.workunit.client.0.smithi144.stdout:0/626: truncate d0/f7 5681258 0 2024-07-26T11:04:50.004 INFO:tasks.workunit.client.0.smithi144.stdout:1/821: dwrite d3/d6/da/d78/f9d [0,4194304] 0 2024-07-26T11:04:50.004 INFO:tasks.workunit.client.0.smithi144.stdout:5/656: unlink d1/d18/d24/d27/f4f 0 2024-07-26T11:04:50.005 INFO:tasks.workunit.client.0.smithi144.stdout:9/698: link d9/de/d14/d3f/d5e/fdf d9/d3b/dce/fe8 0 2024-07-26T11:04:50.006 INFO:tasks.workunit.client.0.smithi144.stdout:9/699: truncate d9/de/faa 274134 0 2024-07-26T11:04:50.009 INFO:tasks.workunit.client.0.smithi144.stdout:0/627: link d0/d25/d5e/d1e/d30/d6a/d6f/fa2 d0/d3e/db8/fcf 0 2024-07-26T11:04:50.016 INFO:tasks.workunit.client.0.smithi144.stdout:1/822: creat d3/d6/da/d97/dfb/f109 x:0 0 0 2024-07-26T11:04:50.016 INFO:tasks.workunit.client.0.smithi144.stdout:1/823: truncate d3/d6/da/fca 4318114 0 2024-07-26T11:04:50.016 INFO:tasks.workunit.client.0.smithi144.stdout:1/824: chown d3/d6/da/d78/lfa 535957100 1 2024-07-26T11:04:50.016 INFO:tasks.workunit.client.0.smithi144.stdout:5/657: creat d1/d9/d5b/fdd x:0 0 0 2024-07-26T11:04:50.016 INFO:tasks.workunit.client.0.smithi144.stdout:9/700: truncate d9/de/f52 989931 0 2024-07-26T11:04:50.017 INFO:tasks.workunit.client.0.smithi144.stdout:5/658: truncate d1/d18/d24/d35/f9c 1012267 0 2024-07-26T11:04:50.017 INFO:tasks.workunit.client.0.smithi144.stdout:9/701: write d9/de/d14/fbb [922313,59843] 0 2024-07-26T11:04:50.017 INFO:tasks.workunit.client.0.smithi144.stdout:9/702: write d9/de/d14/d79/db2/fc8 [518637,103542] 0 2024-07-26T11:04:50.017 INFO:tasks.workunit.client.0.smithi144.stdout:0/628: mknod d0/d25/d5e/cd0 0 2024-07-26T11:04:50.019 INFO:tasks.workunit.client.0.smithi144.stdout:9/703: rmdir d9/d3b/d5d 39 2024-07-26T11:04:50.020 INFO:tasks.workunit.client.0.smithi144.stdout:3/458: dwrite d1/d28/d54/d26/f47 [0,4194304] 0 2024-07-26T11:04:50.020 INFO:tasks.workunit.client.0.smithi144.stdout:0/629: rmdir d0/d25/d5e/d1e/d30/d4b 39 2024-07-26T11:04:50.020 INFO:tasks.workunit.client.0.smithi144.stdout:0/630: read d0/d3e/d45/d9f/f86 [621091,3063] 0 2024-07-26T11:04:50.023 INFO:tasks.workunit.client.0.smithi144.stdout:6/760: read d8/f96 [3296777,73909] 0 2024-07-26T11:04:50.025 INFO:tasks.workunit.client.0.smithi144.stdout:1/825: rename d3/d3d/d58/d101/d36/d37/c91 to d3/d6/c10a 0 2024-07-26T11:04:50.025 INFO:tasks.workunit.client.0.smithi144.stdout:6/761: unlink d8/d15/ccc 0 2024-07-26T11:04:50.030 INFO:tasks.workunit.client.0.smithi144.stdout:3/459: rename d1/d28/c5c to d1/d5/d5e/c87 0 2024-07-26T11:04:50.030 INFO:tasks.workunit.client.0.smithi144.stdout:1/826: creat d3/d3d/d58/d101/f10b x:0 0 0 2024-07-26T11:04:50.032 INFO:tasks.workunit.client.0.smithi144.stdout:1/827: rmdir d3/d6/dc0 39 2024-07-26T11:04:50.038 INFO:tasks.workunit.client.0.smithi144.stdout:3/460: dread d1/d28/d54/d61/f63 [0,4194304] 0 2024-07-26T11:04:50.039 INFO:tasks.workunit.client.0.smithi144.stdout:3/461: chown d1/d28/d54/d26/f2d 13619636 1 2024-07-26T11:04:50.039 INFO:tasks.workunit.client.0.smithi144.stdout:3/462: write d1/f77 [697646,85176] 0 2024-07-26T11:04:50.042 INFO:tasks.workunit.client.0.smithi144.stdout:3/463: symlink d1/d28/d54/d3d/d42/d44/d48/l88 0 2024-07-26T11:04:50.045 INFO:tasks.workunit.client.0.smithi144.stdout:2/628: dwrite d0/d24/d2b/d68/d82/f7c [0,4194304] 0 2024-07-26T11:04:50.048 INFO:tasks.workunit.client.0.smithi144.stdout:5/659: dread d1/d9/d44/da1/fb0 [0,4194304] 0 2024-07-26T11:04:50.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/631: dread d0/d3e/d45/d9f/f6d [0,4194304] 0 2024-07-26T11:04:50.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/632: stat d0/d25/d5e/d1e/d5c/f99 0 2024-07-26T11:04:50.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/633: truncate d0/d25/fa7 956044 0 2024-07-26T11:04:50.049 INFO:tasks.workunit.client.0.smithi144.stdout:7/646: dwrite d0/f4b [0,4194304] 0 2024-07-26T11:04:50.049 INFO:tasks.workunit.client.0.smithi144.stdout:7/647: fdatasync d0/d6/d9/d45/f26 0 2024-07-26T11:04:50.049 INFO:tasks.workunit.client.0.smithi144.stdout:9/704: dread d9/de/d14/fbb [0,4194304] 0 2024-07-26T11:04:50.049 INFO:tasks.workunit.client.0.smithi144.stdout:9/705: fdatasync d9/de/d14/fd9 0 2024-07-26T11:04:50.049 INFO:tasks.workunit.client.0.smithi144.stdout:5/660: symlink d1/d18/dc4/lde 0 2024-07-26T11:04:50.050 INFO:tasks.workunit.client.0.smithi144.stdout:0/634: truncate d0/d25/d5e/d1e/d30/f3b 489733 0 2024-07-26T11:04:50.050 INFO:tasks.workunit.client.0.smithi144.stdout:0/635: truncate d0/d3e/d5f/f8d 656822 0 2024-07-26T11:04:50.050 INFO:tasks.workunit.client.0.smithi144.stdout:0/636: chown d0/d25/d5e/d1e/d30/d4b/c8e 1664334 1 2024-07-26T11:04:50.050 INFO:tasks.workunit.client.0.smithi144.stdout:7/648: rename d0/dd0/dbf/d2f/d53/fdc to d0/d6/d9/d45/fe9 0 2024-07-26T11:04:50.051 INFO:tasks.workunit.client.0.smithi144.stdout:7/649: dread - d0/dd0/dbf/d2f/da2/fa6 zero size 2024-07-26T11:04:50.062 INFO:tasks.workunit.client.0.smithi144.stdout:0/637: rename d0/d3e/d45/d9f/d65 to d0/d25/d5e/d1e/d91/dd1 0 2024-07-26T11:04:50.063 INFO:tasks.workunit.client.0.smithi144.stdout:6/762: write d8/d42/f7c [2687616,41147] 0 2024-07-26T11:04:50.063 INFO:tasks.workunit.client.0.smithi144.stdout:7/650: getdents d0 0 2024-07-26T11:04:50.066 INFO:tasks.workunit.client.0.smithi144.stdout:6/763: rename d8/db/dbe/daa/fac to d8/d50/d92/dea/f10e 0 2024-07-26T11:04:50.067 INFO:tasks.workunit.client.0.smithi144.stdout:7/651: unlink d0/dd0/dbf/f42 0 2024-07-26T11:04:50.069 INFO:tasks.workunit.client.0.smithi144.stdout:6/764: creat d8/db/dbe/d52/ddb/de3/f10f x:0 0 0 2024-07-26T11:04:50.069 INFO:tasks.workunit.client.0.smithi144.stdout:7/652: link d0/dd0/dbf/l43 d0/d71/d95/lea 0 2024-07-26T11:04:50.069 INFO:tasks.workunit.client.0.smithi144.stdout:6/765: chown d8/d50/d92 12182 1 2024-07-26T11:04:50.069 INFO:tasks.workunit.client.0.smithi144.stdout:6/766: fsync d8/db/dbe/f7e 0 2024-07-26T11:04:50.070 INFO:tasks.workunit.client.0.smithi144.stdout:6/767: chown d8/d15/d57/l6c 10 1 2024-07-26T11:04:50.070 INFO:tasks.workunit.client.0.smithi144.stdout:6/768: write d8/d15/f5e [264292,25532] 0 2024-07-26T11:04:50.070 INFO:tasks.workunit.client.0.smithi144.stdout:7/653: mkdir d0/d6/d9/d3f/dd3/deb 0 2024-07-26T11:04:50.070 INFO:tasks.workunit.client.0.smithi144.stdout:7/654: dread - d0/dd0/dbf/d2f/d68/f8d zero size 2024-07-26T11:04:50.070 INFO:tasks.workunit.client.0.smithi144.stdout:7/655: write d0/d71/d95/dc4/f65 [596661,64764] 0 2024-07-26T11:04:50.071 INFO:tasks.workunit.client.0.smithi144.stdout:6/769: symlink d8/d15/l110 0 2024-07-26T11:04:50.071 INFO:tasks.workunit.client.0.smithi144.stdout:6/770: stat d8/db/f2c 0 2024-07-26T11:04:50.077 INFO:tasks.workunit.client.0.smithi144.stdout:1/828: write d3/d6/d75/d88/da9/dac/d104/f50 [553116,102794] 0 2024-07-26T11:04:50.077 INFO:tasks.workunit.client.0.smithi144.stdout:1/829: stat d3/d6/da/d21/d67/c8c 0 2024-07-26T11:04:50.080 INFO:tasks.workunit.client.0.smithi144.stdout:8/604: sync 2024-07-26T11:04:50.080 INFO:tasks.workunit.client.0.smithi144.stdout:4/658: sync 2024-07-26T11:04:50.081 INFO:tasks.workunit.client.0.smithi144.stdout:4/659: dread - d9/d26/d48/d98/fd6 zero size 2024-07-26T11:04:50.081 INFO:tasks.workunit.client.0.smithi144.stdout:4/660: stat d9/c10 0 2024-07-26T11:04:50.081 INFO:tasks.workunit.client.0.smithi144.stdout:4/661: stat d9/da/f3b 0 2024-07-26T11:04:50.081 INFO:tasks.workunit.client.0.smithi144.stdout:8/605: write d1a/d32/d49/d73/f79 [952434,60544] 0 2024-07-26T11:04:50.081 INFO:tasks.workunit.client.0.smithi144.stdout:8/606: chown d1a/d32/d66/dcd/l76 91284 1 2024-07-26T11:04:50.082 INFO:tasks.workunit.client.0.smithi144.stdout:4/662: mknod d9/d26/d3c/d6f/dd0/ce3 0 2024-07-26T11:04:50.086 INFO:tasks.workunit.client.0.smithi144.stdout:4/663: rename d9/fc6 to d9/d26/d48/d98/fe4 0 2024-07-26T11:04:50.086 INFO:tasks.workunit.client.0.smithi144.stdout:4/664: chown d9/c10 279264 1 2024-07-26T11:04:50.086 INFO:tasks.workunit.client.0.smithi144.stdout:5/661: dread d1/d18/d24/d27/fcb [0,4194304] 0 2024-07-26T11:04:50.086 INFO:tasks.workunit.client.0.smithi144.stdout:3/464: dread d1/d28/d54/d26/f2b [0,4194304] 0 2024-07-26T11:04:50.086 INFO:tasks.workunit.client.0.smithi144.stdout:3/465: chown d1/d28/d54/d26/f2d 165706 1 2024-07-26T11:04:50.089 INFO:tasks.workunit.client.0.smithi144.stdout:6/771: dread d8/db/da6/fa3 [0,4194304] 0 2024-07-26T11:04:50.092 INFO:tasks.workunit.client.0.smithi144.stdout:8/607: dread d1a/d32/f62 [0,4194304] 0 2024-07-26T11:04:50.097 INFO:tasks.workunit.client.0.smithi144.stdout:7/656: dread d0/d6/f52 [0,4194304] 0 2024-07-26T11:04:50.098 INFO:tasks.workunit.client.0.smithi144.stdout:5/662: creat d1/d18/fdf x:0 0 0 2024-07-26T11:04:50.098 INFO:tasks.workunit.client.0.smithi144.stdout:5/663: fdatasync d1/d9/d44/d45/d81/f9e 0 2024-07-26T11:04:50.098 INFO:tasks.workunit.client.0.smithi144.stdout:5/664: truncate d1/d18/d24/f82 544852 0 2024-07-26T11:04:50.098 INFO:tasks.workunit.client.0.smithi144.stdout:6/772: creat d8/db/dbe/d52/ddb/f111 x:0 0 0 2024-07-26T11:04:50.099 INFO:tasks.workunit.client.0.smithi144.stdout:8/608: symlink d1a/d32/d6c/d82/lcf 0 2024-07-26T11:04:50.100 INFO:tasks.workunit.client.0.smithi144.stdout:5/665: mkdir d1/d18/dc4/de0 0 2024-07-26T11:04:50.101 INFO:tasks.workunit.client.0.smithi144.stdout:3/466: dread d1/d28/d54/d61/f63 [0,4194304] 0 2024-07-26T11:04:50.102 INFO:tasks.workunit.client.0.smithi144.stdout:8/609: mkdir d1a/d32/d8c/dd0 0 2024-07-26T11:04:50.102 INFO:tasks.workunit.client.0.smithi144.stdout:6/773: link d8/d50/d92/db3/c10c d8/db/dbe/d52/ddb/de3/c112 0 2024-07-26T11:04:50.103 INFO:tasks.workunit.client.0.smithi144.stdout:3/467: mkdir d1/d28/d54/d26/d64/d89 0 2024-07-26T11:04:50.103 INFO:tasks.workunit.client.0.smithi144.stdout:3/468: fdatasync d1/d28/d54/d26/d30/f7f 0 2024-07-26T11:04:50.103 INFO:tasks.workunit.client.0.smithi144.stdout:8/610: mknod d1a/d32/d49/d73/d2c/d38/d84/db0/cd1 0 2024-07-26T11:04:50.104 INFO:tasks.workunit.client.0.smithi144.stdout:8/611: chown d1a/d32/d50/c61 32301737 1 2024-07-26T11:04:50.104 INFO:tasks.workunit.client.0.smithi144.stdout:8/612: fdatasync d1a/d32/d49/d73/f3f 0 2024-07-26T11:04:50.104 INFO:tasks.workunit.client.0.smithi144.stdout:6/774: creat d8/d50/d98/d104/f113 x:0 0 0 2024-07-26T11:04:50.105 INFO:tasks.workunit.client.0.smithi144.stdout:8/613: symlink d1a/d32/d49/d73/d7b/d4e/d9a/ld2 0 2024-07-26T11:04:50.107 INFO:tasks.workunit.client.0.smithi144.stdout:0/638: write d0/d25/d5e/d1e/f3c [2920516,41414] 0 2024-07-26T11:04:50.107 INFO:tasks.workunit.client.0.smithi144.stdout:6/775: rename d8/db/dbe/fc9 to d8/d15/d26/f114 0 2024-07-26T11:04:50.107 INFO:tasks.workunit.client.0.smithi144.stdout:0/639: mkdir d0/d25/d5e/d1e/dd2 0 2024-07-26T11:04:50.108 INFO:tasks.workunit.client.0.smithi144.stdout:6/776: symlink d8/db/da6/l115 0 2024-07-26T11:04:50.108 INFO:tasks.workunit.client.0.smithi144.stdout:6/777: fsync d8/d50/f85 0 2024-07-26T11:04:50.109 INFO:tasks.workunit.client.0.smithi144.stdout:0/640: rmdir d0/d25/d5e/d1e/d6c 39 2024-07-26T11:04:50.109 INFO:tasks.workunit.client.0.smithi144.stdout:0/641: fsync d0/d3e/f71 0 2024-07-26T11:04:50.109 INFO:tasks.workunit.client.0.smithi144.stdout:1/830: write d3/d6/da/d78/f9d [3731495,77010] 0 2024-07-26T11:04:50.109 INFO:tasks.workunit.client.0.smithi144.stdout:1/831: readlink d3/d6/da/d13/lbf 0 2024-07-26T11:04:50.110 INFO:tasks.workunit.client.0.smithi144.stdout:6/778: link d8/dca/cdf d8/d15/c116 0 2024-07-26T11:04:50.110 INFO:tasks.workunit.client.0.smithi144.stdout:6/779: write d8/db/dbe/f72 [1095049,101020] 0 2024-07-26T11:04:50.110 INFO:tasks.workunit.client.0.smithi144.stdout:6/780: fsync d8/d15/f39 0 2024-07-26T11:04:50.110 INFO:tasks.workunit.client.0.smithi144.stdout:6/781: read d8/db/dbe/d52/f5a [1136425,119515] 0 2024-07-26T11:04:50.111 INFO:tasks.workunit.client.0.smithi144.stdout:6/782: chown d8/d15/f38 32627904 1 2024-07-26T11:04:50.111 INFO:tasks.workunit.client.0.smithi144.stdout:0/642: getdents d0/d25/d5e 0 2024-07-26T11:04:50.111 INFO:tasks.workunit.client.0.smithi144.stdout:0/643: chown d0/d25/d5e/d1e/d30/d6a/faa 274468082 1 2024-07-26T11:04:50.111 INFO:tasks.workunit.client.0.smithi144.stdout:1/832: rename d3/d6/da/d13/c60 to d3/d6/da/d21/d67/c10c 0 2024-07-26T11:04:50.112 INFO:tasks.workunit.client.0.smithi144.stdout:1/833: mkdir d3/d3d/d58/d101/d10d 0 2024-07-26T11:04:50.113 INFO:tasks.workunit.client.0.smithi144.stdout:1/834: stat d3/d53/f87 0 2024-07-26T11:04:50.124 INFO:tasks.workunit.client.0.smithi144.stdout:1/835: truncate d3/d6/da/d21/d5c/d62/f7c 5949971 0 2024-07-26T11:04:50.128 INFO:tasks.workunit.client.0.smithi144.stdout:1/836: unlink d3/d3d/l46 0 2024-07-26T11:04:50.128 INFO:tasks.workunit.client.0.smithi144.stdout:1/837: chown d3/d6/d75/d88/da9/dac/fc4 0 1 2024-07-26T11:04:50.130 INFO:tasks.workunit.client.0.smithi144.stdout:1/838: unlink d3/d6/d75/d88/cd3 0 2024-07-26T11:04:50.133 INFO:tasks.workunit.client.0.smithi144.stdout:1/839: creat d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/f10e x:0 0 0 2024-07-26T11:04:50.133 INFO:tasks.workunit.client.0.smithi144.stdout:1/840: chown d3/d6/d75/d88/da9/dac/d104/ddb 697568 1 2024-07-26T11:04:50.138 INFO:tasks.workunit.client.0.smithi144.stdout:4/665: dwrite d9/d26/d48/d50/f6c [0,4194304] 0 2024-07-26T11:04:50.138 INFO:tasks.workunit.client.0.smithi144.stdout:1/841: mknod d3/d6/dcc/c10f 0 2024-07-26T11:04:50.140 INFO:tasks.workunit.client.0.smithi144.stdout:4/666: creat d9/d26/d48/d98/db8/fe5 x:0 0 0 2024-07-26T11:04:50.142 INFO:tasks.workunit.client.0.smithi144.stdout:6/783: dread d8/d50/d98/fc5 [0,4194304] 0 2024-07-26T11:04:50.143 INFO:tasks.workunit.client.0.smithi144.stdout:1/842: creat d3/d6/da/d21/d5c/f110 x:0 0 0 2024-07-26T11:04:50.145 INFO:tasks.workunit.client.0.smithi144.stdout:4/667: creat d9/d7b/fe6 x:0 0 0 2024-07-26T11:04:50.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/784: link d8/d15/d57/d5f/d6e/cd9 d8/db/dbe/d52/d58/dbf/c117 0 2024-07-26T11:04:50.147 INFO:tasks.workunit.client.0.smithi144.stdout:6/785: fdatasync d8/d42/dad/fd8 0 2024-07-26T11:04:50.153 INFO:tasks.workunit.client.0.smithi144.stdout:2/629: dwrite d0/d6/d9/f26 [0,4194304] 0 2024-07-26T11:04:50.154 INFO:tasks.workunit.client.0.smithi144.stdout:2/630: fsync d0/d6/d9/d44/fa2 0 2024-07-26T11:04:50.160 INFO:tasks.workunit.client.0.smithi144.stdout:4/668: creat d9/d26/d48/dd8/fe7 x:0 0 0 2024-07-26T11:04:50.161 INFO:tasks.workunit.client.0.smithi144.stdout:6/786: creat d8/d50/d92/db3/f118 x:0 0 0 2024-07-26T11:04:50.161 INFO:tasks.workunit.client.0.smithi144.stdout:6/787: readlink d8/d50/d92/dea/df8/lfd 0 2024-07-26T11:04:50.161 INFO:tasks.workunit.client.0.smithi144.stdout:0/644: dread d0/d3e/d45/d9f/fa8 [0,4194304] 0 2024-07-26T11:04:50.161 INFO:tasks.workunit.client.0.smithi144.stdout:0/645: chown d0/d25/d5e/d1e/d30/d6a 3 1 2024-07-26T11:04:50.162 INFO:tasks.workunit.client.0.smithi144.stdout:0/646: write d0/d3e/d45/d9f/f82 [567810,54271] 0 2024-07-26T11:04:50.162 INFO:tasks.workunit.client.0.smithi144.stdout:2/631: rename d0/d24/d48/d63/da3/cb6 to d0/db8/cc1 0 2024-07-26T11:04:50.163 INFO:tasks.workunit.client.0.smithi144.stdout:2/632: write d0/d24/d2b/fbb [966665,3856] 0 2024-07-26T11:04:50.163 INFO:tasks.workunit.client.0.smithi144.stdout:4/669: symlink d9/d26/d48/d98/db8/le8 0 2024-07-26T11:04:50.166 INFO:tasks.workunit.client.0.smithi144.stdout:0/647: symlink d0/d25/d5e/d1e/d30/d67/ld3 0 2024-07-26T11:04:50.166 INFO:tasks.workunit.client.0.smithi144.stdout:0/648: readlink d0/d3e/d45/d4c/l74 0 2024-07-26T11:04:50.168 INFO:tasks.workunit.client.0.smithi144.stdout:6/788: rename d8/db/dbe/d52/d58/f97 to d8/db/dbe/d52/f119 0 2024-07-26T11:04:50.168 INFO:tasks.workunit.client.0.smithi144.stdout:6/789: chown d8/d42/fa9 1190667 1 2024-07-26T11:04:50.174 INFO:tasks.workunit.client.0.smithi144.stdout:8/614: dwrite d1a/d32/d49/d73/d7b/d43/f9f [0,4194304] 0 2024-07-26T11:04:50.175 INFO:tasks.workunit.client.0.smithi144.stdout:8/615: truncate d1a/d32/d49/d73/f79 1574847 0 2024-07-26T11:04:50.175 INFO:tasks.workunit.client.0.smithi144.stdout:8/616: write d1a/d32/d49/d73/d7b/d43/d4c/f77 [858825,88669] 0 2024-07-26T11:04:50.190 INFO:tasks.workunit.client.0.smithi144.stdout:1/843: dread d3/d6/da/f12 [0,4194304] 0 2024-07-26T11:04:50.190 INFO:tasks.workunit.client.0.smithi144.stdout:5/666: dwrite d1/d9/d44/da1/fad [0,4194304] 0 2024-07-26T11:04:50.192 INFO:tasks.workunit.client.0.smithi144.stdout:1/844: creat d3/d6/da/f111 x:0 0 0 2024-07-26T11:04:50.194 INFO:tasks.workunit.client.0.smithi144.stdout:4/670: read d9/d26/d3c/d6f/f7c [615201,13453] 0 2024-07-26T11:04:50.196 INFO:tasks.workunit.client.0.smithi144.stdout:4/671: rename d9/d26/d48/d50/f59 to d9/d7b/fe9 0 2024-07-26T11:04:50.196 INFO:tasks.workunit.client.0.smithi144.stdout:5/667: link d1/d9/d5b/d4d/fa5 d1/d9/d44/d45/d81/fe1 0 2024-07-26T11:04:50.196 INFO:tasks.workunit.client.0.smithi144.stdout:4/672: write d9/d26/d48/f71 [5003928,36244] 0 2024-07-26T11:04:50.197 INFO:tasks.workunit.client.0.smithi144.stdout:5/668: rmdir d1/d9/d44/d45/d81 39 2024-07-26T11:04:50.198 INFO:tasks.workunit.client.0.smithi144.stdout:6/790: dread d8/d50/d98/fc5 [0,4194304] 0 2024-07-26T11:04:50.200 INFO:tasks.workunit.client.0.smithi144.stdout:4/673: read d9/d7b/dc4/fc7 [162660,97067] 0 2024-07-26T11:04:50.200 INFO:tasks.workunit.client.0.smithi144.stdout:4/674: truncate d9/da/fad 933109 0 2024-07-26T11:04:50.200 INFO:tasks.workunit.client.0.smithi144.stdout:5/669: link d1/d9/d5b/d4d/f3e d1/d18/fe2 0 2024-07-26T11:04:50.201 INFO:tasks.workunit.client.0.smithi144.stdout:5/670: dread - d1/d18/fdf zero size 2024-07-26T11:04:50.201 INFO:tasks.workunit.client.0.smithi144.stdout:5/671: chown d1 594407896 1 2024-07-26T11:04:50.201 INFO:tasks.workunit.client.0.smithi144.stdout:6/791: mkdir d8/db/dbe/daa/d11a 0 2024-07-26T11:04:50.202 INFO:tasks.workunit.client.0.smithi144.stdout:6/792: dread - d8/d15/d57/d5f/fc3 zero size 2024-07-26T11:04:50.202 INFO:tasks.workunit.client.0.smithi144.stdout:0/649: dread d0/f8 [0,4194304] 0 2024-07-26T11:04:50.203 INFO:tasks.workunit.client.0.smithi144.stdout:5/672: unlink d1/d9/d44/d45/d98/db2/fd8 0 2024-07-26T11:04:50.203 INFO:tasks.workunit.client.0.smithi144.stdout:5/673: dread - d1/d5/dae/fd3 zero size 2024-07-26T11:04:50.203 INFO:tasks.workunit.client.0.smithi144.stdout:5/674: fdatasync d1/d5/f2d 0 2024-07-26T11:04:50.203 INFO:tasks.workunit.client.0.smithi144.stdout:5/675: truncate d1/d9/d44/d45/d81/fe1 979450 0 2024-07-26T11:04:50.204 INFO:tasks.workunit.client.0.smithi144.stdout:1/845: dread d3/d53/f87 [0,4194304] 0 2024-07-26T11:04:50.205 INFO:tasks.workunit.client.0.smithi144.stdout:6/793: symlink d8/d15/l11b 0 2024-07-26T11:04:50.207 INFO:tasks.workunit.client.0.smithi144.stdout:0/650: mkdir d0/d3e/d45/dd4 0 2024-07-26T11:04:50.208 INFO:tasks.workunit.client.0.smithi144.stdout:4/675: dread d9/f76 [0,4194304] 0 2024-07-26T11:04:50.209 INFO:tasks.workunit.client.0.smithi144.stdout:5/676: link d1/d9/le d1/d9/d44/d45/le3 0 2024-07-26T11:04:50.212 INFO:tasks.workunit.client.0.smithi144.stdout:1/846: creat d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f112 x:0 0 0 2024-07-26T11:04:50.214 INFO:tasks.workunit.client.0.smithi144.stdout:0/651: link d0/d25/d5e/d1e/d30/fb9 d0/d25/d5e/d1e/d30/dab/fd5 0 2024-07-26T11:04:50.216 INFO:tasks.workunit.client.0.smithi144.stdout:4/676: rmdir d9/d26 39 2024-07-26T11:04:50.216 INFO:tasks.workunit.client.0.smithi144.stdout:4/677: truncate d9/d26/d48/d98/fd6 6061 0 2024-07-26T11:04:50.216 INFO:tasks.workunit.client.0.smithi144.stdout:4/678: readlink d9/l2e 0 2024-07-26T11:04:50.217 INFO:tasks.workunit.client.0.smithi144.stdout:4/679: readlink d9/d57/ld7 0 2024-07-26T11:04:50.217 INFO:tasks.workunit.client.0.smithi144.stdout:2/633: dwrite d0/d24/d48/d6d/f35 [0,4194304] 0 2024-07-26T11:04:50.219 INFO:tasks.workunit.client.0.smithi144.stdout:5/677: dread d1/d9/d44/d85/fbc [0,4194304] 0 2024-07-26T11:04:50.220 INFO:tasks.workunit.client.0.smithi144.stdout:4/680: creat d9/da/fea x:0 0 0 2024-07-26T11:04:50.222 INFO:tasks.workunit.client.0.smithi144.stdout:5/678: mkdir d1/d18/d24/d35/de4 0 2024-07-26T11:04:50.223 INFO:tasks.workunit.client.0.smithi144.stdout:5/679: mknod d1/d9/d5b/d4d/ce5 0 2024-07-26T11:04:50.225 INFO:tasks.workunit.client.0.smithi144.stdout:5/680: symlink d1/da8/le6 0 2024-07-26T11:04:50.225 INFO:tasks.workunit.client.0.smithi144.stdout:5/681: stat d1/d9/f19 0 2024-07-26T11:04:50.225 INFO:tasks.workunit.client.0.smithi144.stdout:5/682: stat d1/c2 0 2024-07-26T11:04:50.227 INFO:tasks.workunit.client.0.smithi144.stdout:2/634: dread d0/d24/d48/d6d/d28/f5c [0,4194304] 0 2024-07-26T11:04:50.241 INFO:tasks.workunit.client.0.smithi144.stdout:1/847: dread d3/d6/d9c/d9f/fd0 [0,4194304] 0 2024-07-26T11:04:50.242 INFO:tasks.workunit.client.0.smithi144.stdout:1/848: mkdir d3/d6/d75/d88/d113 0 2024-07-26T11:04:50.245 INFO:tasks.workunit.client.0.smithi144.stdout:4/681: write d9/d26/d3c/f7a [3832928,110727] 0 2024-07-26T11:04:50.247 INFO:tasks.workunit.client.0.smithi144.stdout:6/794: dwrite d8/d50/d98/f10b [0,4194304] 0 2024-07-26T11:04:50.248 INFO:tasks.workunit.client.0.smithi144.stdout:6/795: chown d8/d15/d26/lc0 0 1 2024-07-26T11:04:50.254 INFO:tasks.workunit.client.0.smithi144.stdout:2/635: dread d0/d24/d2b/d68/d82/fab [0,4194304] 0 2024-07-26T11:04:50.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/636: mkdir d0/d24/d48/d6d/d28/d2e/d5d/d6b/dc2 0 2024-07-26T11:04:50.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/637: fdatasync d0/d6/d9/f33 0 2024-07-26T11:04:50.267 INFO:tasks.workunit.client.0.smithi144.stdout:1/849: write d3/fb2 [7700087,9413] 0 2024-07-26T11:04:50.267 INFO:tasks.workunit.client.0.smithi144.stdout:2/638: mkdir d0/d6/d9/d44/dc3 0 2024-07-26T11:04:50.269 INFO:tasks.workunit.client.0.smithi144.stdout:2/639: creat d0/d24/d2b/d68/d82/fc4 x:0 0 0 2024-07-26T11:04:50.271 INFO:tasks.workunit.client.0.smithi144.stdout:2/640: symlink d0/d24/d48/d6d/d5f/lc5 0 2024-07-26T11:04:50.271 INFO:tasks.workunit.client.0.smithi144.stdout:2/641: fdatasync d0/d6/f89 0 2024-07-26T11:04:50.271 INFO:tasks.workunit.client.0.smithi144.stdout:4/682: dread d9/f41 [0,4194304] 0 2024-07-26T11:04:50.272 INFO:tasks.workunit.client.0.smithi144.stdout:2/642: symlink d0/d24/d48/d6d/d28/d2e/d5d/d6b/dc2/lc6 0 2024-07-26T11:04:50.273 INFO:tasks.workunit.client.0.smithi144.stdout:2/643: truncate d0/d6/f1d 1447417 0 2024-07-26T11:04:50.274 INFO:tasks.workunit.client.0.smithi144.stdout:1/850: dread d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f40 [0,4194304] 0 2024-07-26T11:04:50.274 INFO:tasks.workunit.client.0.smithi144.stdout:1/851: readlink d3/d53/l6d 0 2024-07-26T11:04:50.278 INFO:tasks.workunit.client.0.smithi144.stdout:1/852: truncate d3/d6/da/d21/d67/fc3 236733 0 2024-07-26T11:04:50.278 INFO:tasks.workunit.client.0.smithi144.stdout:1/853: read d3/f2d [421285,38083] 0 2024-07-26T11:04:50.280 INFO:tasks.workunit.client.0.smithi144.stdout:1/854: rmdir d3/d6/da/d78 39 2024-07-26T11:04:50.291 INFO:tasks.workunit.client.0.smithi144.stdout:5/683: dwrite d1/d9/d44/fce [0,4194304] 0 2024-07-26T11:04:50.296 INFO:tasks.workunit.client.0.smithi144.stdout:5/684: link d1/d9/l30 d1/d18/dc4/le7 0 2024-07-26T11:04:50.311 INFO:tasks.workunit.client.0.smithi144.stdout:4/683: read d9/d26/f2b [2473953,28186] 0 2024-07-26T11:04:50.311 INFO:tasks.workunit.client.0.smithi144.stdout:4/684: truncate d9/d57/f77 3367326 0 2024-07-26T11:04:50.311 INFO:tasks.workunit.client.0.smithi144.stdout:6/796: dwrite d8/d15/fd4 [0,4194304] 0 2024-07-26T11:04:50.313 INFO:tasks.workunit.client.0.smithi144.stdout:2/644: dread d0/f1 [0,4194304] 0 2024-07-26T11:04:50.314 INFO:tasks.workunit.client.0.smithi144.stdout:2/645: mknod d0/d24/d48/d6d/d28/d27/d8b/cc7 0 2024-07-26T11:04:50.315 INFO:tasks.workunit.client.0.smithi144.stdout:2/646: symlink d0/d24/d48/d6d/d28/d2e/d5d/d6b/lc8 0 2024-07-26T11:04:50.315 INFO:tasks.workunit.client.0.smithi144.stdout:2/647: getdents d0/d24/d48/d6d 0 2024-07-26T11:04:50.323 INFO:tasks.workunit.client.0.smithi144.stdout:1/855: dwrite d3/d6/da/d70/f90 [0,4194304] 0 2024-07-26T11:04:50.323 INFO:tasks.workunit.client.0.smithi144.stdout:1/856: fdatasync d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f30 0 2024-07-26T11:04:50.323 INFO:tasks.workunit.client.0.smithi144.stdout:1/857: chown d3/d3d/d58/d101 329 1 2024-07-26T11:04:50.323 INFO:tasks.workunit.client.0.smithi144.stdout:1/858: read - d3/d6/d9c/d9f/ff2 zero size 2024-07-26T11:04:50.323 INFO:tasks.workunit.client.0.smithi144.stdout:1/859: truncate d3/d6/da/d21/f4d 4530135 0 2024-07-26T11:04:50.325 INFO:tasks.workunit.client.0.smithi144.stdout:1/860: rmdir d3/d6 39 2024-07-26T11:04:50.329 INFO:tasks.workunit.client.0.smithi144.stdout:2/648: dread d0/d24/d48/d6d/d28/d2e/d77/fbd [0,4194304] 0 2024-07-26T11:04:50.331 INFO:tasks.workunit.client.0.smithi144.stdout:2/649: creat d0/d24/d48/d6d/d28/d2e/d5d/d6b/fc9 x:0 0 0 2024-07-26T11:04:50.331 INFO:tasks.workunit.client.0.smithi144.stdout:1/861: write d3/d6/da/d78/f7e [749902,31088] 0 2024-07-26T11:04:50.331 INFO:tasks.workunit.client.0.smithi144.stdout:2/650: rmdir d0/d24/d7b 39 2024-07-26T11:04:50.335 INFO:tasks.workunit.client.0.smithi144.stdout:1/862: symlink d3/l114 0 2024-07-26T11:04:50.336 INFO:tasks.workunit.client.0.smithi144.stdout:1/863: dread - d3/d6/d75/fc7 zero size 2024-07-26T11:04:50.336 INFO:tasks.workunit.client.0.smithi144.stdout:1/864: write d3/d3d/d58/d101/fd4 [1333540,47820] 0 2024-07-26T11:04:50.342 INFO:tasks.workunit.client.0.smithi144.stdout:2/651: dread d0/d24/d2b/d68/f6a [0,4194304] 0 2024-07-26T11:04:50.344 INFO:tasks.workunit.client.0.smithi144.stdout:5/685: dwrite d1/d18/d24/d27/d71/fac [0,4194304] 0 2024-07-26T11:04:50.361 INFO:tasks.workunit.client.0.smithi144.stdout:4/685: dread d9/d26/d3c/d6f/f86 [0,4194304] 0 2024-07-26T11:04:50.377 INFO:tasks.workunit.client.0.smithi144.stdout:1/865: dwrite d3/d6/d75/fc7 [0,4194304] 0 2024-07-26T11:04:50.380 INFO:tasks.workunit.client.0.smithi144.stdout:4/686: rename d9/d26/d3c/d6f/d8c/f96 to d9/d57/feb 0 2024-07-26T11:04:50.380 INFO:tasks.workunit.client.0.smithi144.stdout:2/652: creat d0/d6/d34/fca x:0 0 0 2024-07-26T11:04:50.390 INFO:tasks.workunit.client.0.smithi144.stdout:2/653: getdents d0/d24/d48/d6d/d5f 0 2024-07-26T11:04:50.403 INFO:tasks.workunit.client.0.smithi144.stdout:2/654: creat d0/d24/d48/d6d/d28/da8/fcb x:0 0 0 2024-07-26T11:04:50.427 INFO:tasks.workunit.client.0.smithi144.stdout:5/686: unlink d1/d9/cb 0 2024-07-26T11:04:50.429 INFO:tasks.workunit.client.0.smithi144.stdout:5/687: truncate d1/d9/d44/d45/f56 362028 0 2024-07-26T11:04:50.435 INFO:tasks.workunit.client.0.smithi144.stdout:1/866: dwrite d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f8a [0,4194304] 0 2024-07-26T11:04:50.443 INFO:tasks.workunit.client.0.smithi144.stdout:2/655: creat d0/d24/d48/d6d/fcc x:0 0 0 2024-07-26T11:04:50.471 INFO:tasks.workunit.client.0.smithi144.stdout:1/867: write d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f30 [3640178,41392] 0 2024-07-26T11:04:50.472 INFO:tasks.workunit.client.0.smithi144.stdout:4/687: dwrite f6 [0,4194304] 0 2024-07-26T11:04:50.472 INFO:tasks.workunit.client.0.smithi144.stdout:4/688: fdatasync d9/d26/f2f 0 2024-07-26T11:04:50.474 INFO:tasks.workunit.client.0.smithi144.stdout:1/868: creat d3/d6/d75/f115 x:0 0 0 2024-07-26T11:04:50.475 INFO:tasks.workunit.client.0.smithi144.stdout:1/869: dread - d3/d3d/fc6 zero size 2024-07-26T11:04:50.477 INFO:tasks.workunit.client.0.smithi144.stdout:1/870: creat d3/d6/da/d21/d5c/d62/d68/f116 x:0 0 0 2024-07-26T11:04:50.477 INFO:tasks.workunit.client.0.smithi144.stdout:1/871: chown d3/d6/d9c 176 1 2024-07-26T11:04:50.477 INFO:tasks.workunit.client.0.smithi144.stdout:1/872: fdatasync d3/d6/da/d21/fc1 0 2024-07-26T11:04:50.477 INFO:tasks.workunit.client.0.smithi144.stdout:1/873: readlink d3/d6/da/d21/lf1 0 2024-07-26T11:04:50.477 INFO:tasks.workunit.client.0.smithi144.stdout:1/874: fdatasync d3/d6/dcc/fd9 0 2024-07-26T11:04:50.478 INFO:tasks.workunit.client.0.smithi144.stdout:1/875: symlink d3/d6/da/d21/l117 0 2024-07-26T11:04:50.478 INFO:tasks.workunit.client.0.smithi144.stdout:1/876: fdatasync d3/f2d 0 2024-07-26T11:04:50.483 INFO:tasks.workunit.client.0.smithi144.stdout:5/688: dwrite d1/d9/d5b/d4d/f3e [0,4194304] 0 2024-07-26T11:04:50.492 INFO:tasks.workunit.client.0.smithi144.stdout:2/656: dwrite d0/d24/d2b/d68/d82/f98 [0,4194304] 0 2024-07-26T11:04:50.511 INFO:tasks.workunit.client.0.smithi144.stdout:4/689: dwrite d9/d26/d48/d50/f6c [0,4194304] 0 2024-07-26T11:04:50.553 INFO:tasks.workunit.client.0.smithi144.stdout:1/877: rename d3/d6/da/d21/fe3 to d3/d6/d9c/f118 0 2024-07-26T11:04:50.555 INFO:tasks.workunit.client.0.smithi144.stdout:5/689: symlink d1/d9/d44/d45/daa/le8 0 2024-07-26T11:04:50.555 INFO:tasks.workunit.client.0.smithi144.stdout:5/690: dread - d1/d18/d24/fd6 zero size 2024-07-26T11:04:50.556 INFO:tasks.workunit.client.0.smithi144.stdout:2/657: unlink d0/d24/d48/d6d/d28/d2e/d5d/d6b/dc2/lc6 0 2024-07-26T11:04:50.558 INFO:tasks.workunit.client.0.smithi144.stdout:4/690: mkdir d9/d7b/dc4/dec 0 2024-07-26T11:04:50.559 INFO:tasks.workunit.client.0.smithi144.stdout:4/691: write d9/f8d [841667,114927] 0 2024-07-26T11:04:50.559 INFO:tasks.workunit.client.0.smithi144.stdout:4/692: fdatasync d9/f15 0 2024-07-26T11:04:50.559 INFO:tasks.workunit.client.0.smithi144.stdout:5/691: symlink d1/d18/d24/d27/d71/le9 0 2024-07-26T11:04:50.563 INFO:tasks.workunit.client.0.smithi144.stdout:2/658: symlink d0/lcd 0 2024-07-26T11:04:50.573 INFO:tasks.workunit.client.0.smithi144.stdout:0/652: sync 2024-07-26T11:04:50.573 INFO:tasks.workunit.client.0.smithi144.stdout:6/797: sync 2024-07-26T11:04:50.574 INFO:tasks.workunit.client.0.smithi144.stdout:8/617: sync 2024-07-26T11:04:50.574 INFO:tasks.workunit.client.0.smithi144.stdout:7/657: sync 2024-07-26T11:04:50.574 INFO:tasks.workunit.client.0.smithi144.stdout:3/469: sync 2024-07-26T11:04:50.574 INFO:tasks.workunit.client.0.smithi144.stdout:9/706: sync 2024-07-26T11:04:50.574 INFO:tasks.workunit.client.0.smithi144.stdout:0/653: mknod d0/d3e/db4/cd6 0 2024-07-26T11:04:50.574 INFO:tasks.workunit.client.0.smithi144.stdout:0/654: readlink d0/d25/d5e/d1e/d30/d6a/d6f/l7f 0 2024-07-26T11:04:50.575 INFO:tasks.workunit.client.0.smithi144.stdout:6/798: getdents d8/d50/d98 0 2024-07-26T11:04:50.577 INFO:tasks.workunit.client.0.smithi144.stdout:7/658: creat d0/dd0/dbf/d67/de0/fec x:0 0 0 2024-07-26T11:04:50.577 INFO:tasks.workunit.client.0.smithi144.stdout:3/470: truncate d1/f34 2093377 0 2024-07-26T11:04:50.577 INFO:tasks.workunit.client.0.smithi144.stdout:7/659: write d0/d6/f8b [1219852,124399] 0 2024-07-26T11:04:50.577 INFO:tasks.workunit.client.0.smithi144.stdout:7/660: readlink d0/d71/d95/dc4/l73 0 2024-07-26T11:04:50.578 INFO:tasks.workunit.client.0.smithi144.stdout:9/707: getdents d9/de/d14/d79/db2/de2 0 2024-07-26T11:04:50.578 INFO:tasks.workunit.client.0.smithi144.stdout:8/618: symlink d1a/d32/d50/dc4/ld3 0 2024-07-26T11:04:50.579 INFO:tasks.workunit.client.0.smithi144.stdout:0/655: symlink d0/d25/d5e/d1e/dd2/ld7 0 2024-07-26T11:04:50.589 INFO:tasks.workunit.client.0.smithi144.stdout:6/799: creat d8/d50/d92/dea/df8/f11c x:0 0 0 2024-07-26T11:04:50.589 INFO:tasks.workunit.client.0.smithi144.stdout:9/708: rename d9/de/d14/d23/d39/d7f/fa4 to d9/d3b/d76/d9b/dd7/fe9 0 2024-07-26T11:04:50.590 INFO:tasks.workunit.client.0.smithi144.stdout:7/661: rmdir d0/d6 39 2024-07-26T11:04:50.592 INFO:tasks.workunit.client.0.smithi144.stdout:8/619: mkdir d1a/d32/dd4 0 2024-07-26T11:04:50.593 INFO:tasks.workunit.client.0.smithi144.stdout:0/656: mkdir d0/d25/d5e/d1e/dc1/dd8 0 2024-07-26T11:04:50.593 INFO:tasks.workunit.client.0.smithi144.stdout:0/657: getdents d0/d25/d5e/d1e/dc2 0 2024-07-26T11:04:50.593 INFO:tasks.workunit.client.0.smithi144.stdout:4/693: write d9/f76 [325167,58066] 0 2024-07-26T11:04:50.593 INFO:tasks.workunit.client.0.smithi144.stdout:4/694: dread - d9/d26/d48/d50/d60/dc5/dd9/fde zero size 2024-07-26T11:04:50.593 INFO:tasks.workunit.client.0.smithi144.stdout:6/800: creat d8/d50/f11d x:0 0 0 2024-07-26T11:04:50.593 INFO:tasks.workunit.client.0.smithi144.stdout:6/801: chown d8/d50/d92/dea 91020 1 2024-07-26T11:04:50.596 INFO:tasks.workunit.client.0.smithi144.stdout:7/662: rename d0/d6/f17 to d0/dd0/dbf/d2f/d53/db6/fed 0 2024-07-26T11:04:50.596 INFO:tasks.workunit.client.0.smithi144.stdout:7/663: stat d0/d6/d9/d12/f7f 0 2024-07-26T11:04:50.597 INFO:tasks.workunit.client.0.smithi144.stdout:0/658: mknod d0/d3e/db4/cd9 0 2024-07-26T11:04:50.597 INFO:tasks.workunit.client.0.smithi144.stdout:8/620: link d1a/d32/d50/c5d d1a/d32/d6c/d82/cd5 0 2024-07-26T11:04:50.599 INFO:tasks.workunit.client.0.smithi144.stdout:7/664: truncate d0/dd0/dbf/d2f/d47/f4a 1612873 0 2024-07-26T11:04:50.604 INFO:tasks.workunit.client.0.smithi144.stdout:0/659: rename d0/l16 to d0/d25/d5e/d1e/d30/lda 0 2024-07-26T11:04:50.605 INFO:tasks.workunit.client.0.smithi144.stdout:0/660: fdatasync d0/f18 0 2024-07-26T11:04:50.605 INFO:tasks.workunit.client.0.smithi144.stdout:7/665: symlink d0/dd0/lee 0 2024-07-26T11:04:50.611 INFO:tasks.workunit.client.0.smithi144.stdout:0/661: link d0/d3e/d45/d4c/c70 d0/d25/d5e/d1e/d30/d4b/cdb 0 2024-07-26T11:04:50.618 INFO:tasks.workunit.client.0.smithi144.stdout:3/471: dread d1/f14 [0,4194304] 0 2024-07-26T11:04:50.618 INFO:tasks.workunit.client.0.smithi144.stdout:3/472: readlink d1/d28/d54/d3d/d42/d44/d48/l88 0 2024-07-26T11:04:50.618 INFO:tasks.workunit.client.0.smithi144.stdout:3/473: write d1/d5/f1c [1333272,73537] 0 2024-07-26T11:04:50.628 INFO:tasks.workunit.client.0.smithi144.stdout:9/709: dread d9/de/f46 [4194304,4194304] 0 2024-07-26T11:04:50.635 INFO:tasks.workunit.client.0.smithi144.stdout:5/692: dwrite d1/d18/d24/d27/d71/f80 [0,4194304] 0 2024-07-26T11:04:50.635 INFO:tasks.workunit.client.0.smithi144.stdout:1/878: dwrite d3/d6/d9c/d9f/fd0 [0,4194304] 0 2024-07-26T11:04:50.636 INFO:tasks.workunit.client.0.smithi144.stdout:5/693: stat d1/f97 0 2024-07-26T11:04:50.641 INFO:tasks.workunit.client.0.smithi144.stdout:5/694: mkdir d1/d9/d44/d85/dea 0 2024-07-26T11:04:50.643 INFO:tasks.workunit.client.0.smithi144.stdout:2/659: dwrite d0/d24/d48/fa1 [0,4194304] 0 2024-07-26T11:04:50.647 INFO:tasks.workunit.client.0.smithi144.stdout:2/660: creat d0/d24/d48/fce x:0 0 0 2024-07-26T11:04:50.647 INFO:tasks.workunit.client.0.smithi144.stdout:2/661: write d0/d24/d48/fce [164058,75847] 0 2024-07-26T11:04:50.647 INFO:tasks.workunit.client.0.smithi144.stdout:2/662: getdents d0/d6/d9/d44/dc3 0 2024-07-26T11:04:50.647 INFO:tasks.workunit.client.0.smithi144.stdout:2/663: dread - d0/d24/d7b/fb0 zero size 2024-07-26T11:04:50.648 INFO:tasks.workunit.client.0.smithi144.stdout:2/664: fdatasync d0/d6/d9/d44/fa2 0 2024-07-26T11:04:50.648 INFO:tasks.workunit.client.0.smithi144.stdout:2/665: write d0/d24/d48/d6d/d28/d2e/d77/fbd [1961427,31858] 0 2024-07-26T11:04:50.648 INFO:tasks.workunit.client.0.smithi144.stdout:2/666: unlink d0/d24/d48/d6d/d28/da8/fcb 0 2024-07-26T11:04:50.650 INFO:tasks.workunit.client.0.smithi144.stdout:2/667: rename d0/d24/d48/d6d/d5f/lc5 to d0/d6/d9/d44/dc3/lcf 0 2024-07-26T11:04:50.650 INFO:tasks.workunit.client.0.smithi144.stdout:2/668: fsync d0/d6/d9/faa 0 2024-07-26T11:04:50.652 INFO:tasks.workunit.client.0.smithi144.stdout:2/669: read d0/d24/d2b/d68/f6a [251573,26134] 0 2024-07-26T11:04:50.659 INFO:tasks.workunit.client.0.smithi144.stdout:2/670: write d0/d24/d48/d6d/d28/d2e/d77/f40 [1221348,89464] 0 2024-07-26T11:04:50.659 INFO:tasks.workunit.client.0.smithi144.stdout:2/671: chown d0/d6/f1d 1469 1 2024-07-26T11:04:50.660 INFO:tasks.workunit.client.0.smithi144.stdout:2/672: fsync d0/d24/d48/d6d/d28/d2e/fb7 0 2024-07-26T11:04:50.661 INFO:tasks.workunit.client.0.smithi144.stdout:2/673: symlink d0/d24/d48/d6d/d28/d2e/d5d/d6b/dc2/ld0 0 2024-07-26T11:04:50.662 INFO:tasks.workunit.client.0.smithi144.stdout:2/674: symlink d0/db8/ld1 0 2024-07-26T11:04:50.663 INFO:tasks.workunit.client.0.smithi144.stdout:2/675: creat d0/d6/d34/fd2 x:0 0 0 2024-07-26T11:04:50.664 INFO:tasks.workunit.client.0.smithi144.stdout:7/666: write d0/dd0/dbf/d2f/d47/f4f [419299,5341] 0 2024-07-26T11:04:50.664 INFO:tasks.workunit.client.0.smithi144.stdout:2/676: symlink d0/d24/d48/d6d/d28/d27/d8b/ld3 0 2024-07-26T11:04:50.665 INFO:tasks.workunit.client.0.smithi144.stdout:7/667: symlink d0/d71/d95/lef 0 2024-07-26T11:04:50.667 INFO:tasks.workunit.client.0.smithi144.stdout:4/695: dwrite d9/da/f69 [0,4194304] 0 2024-07-26T11:04:50.667 INFO:tasks.workunit.client.0.smithi144.stdout:7/668: unlink d0/d6/d9/d3f/f58 0 2024-07-26T11:04:50.668 INFO:tasks.workunit.client.0.smithi144.stdout:4/696: mknod d9/d57/ced 0 2024-07-26T11:04:50.671 INFO:tasks.workunit.client.0.smithi144.stdout:7/669: symlink d0/dd0/dbf/d2f/da2/lf0 0 2024-07-26T11:04:50.672 INFO:tasks.workunit.client.0.smithi144.stdout:4/697: getdents d9/d26/d3c 0 2024-07-26T11:04:50.672 INFO:tasks.workunit.client.0.smithi144.stdout:4/698: write d9/d26/f97 [1163717,23562] 0 2024-07-26T11:04:50.672 INFO:tasks.workunit.client.0.smithi144.stdout:4/699: readlink d9/d26/l8b 0 2024-07-26T11:04:50.672 INFO:tasks.workunit.client.0.smithi144.stdout:4/700: read d9/d26/d3c/d6f/f7c [411394,48854] 0 2024-07-26T11:04:50.677 INFO:tasks.workunit.client.0.smithi144.stdout:7/670: truncate d0/f4b 281985 0 2024-07-26T11:04:50.677 INFO:tasks.workunit.client.0.smithi144.stdout:7/671: link d0/dd0/dbf/d2f/d47/f4f d0/d71/ff1 0 2024-07-26T11:04:50.677 INFO:tasks.workunit.client.0.smithi144.stdout:7/672: chown d0/d6/d9/d12/d6c 189194 1 2024-07-26T11:04:50.677 INFO:tasks.workunit.client.0.smithi144.stdout:8/621: dwrite d1a/d1c/f22 [0,4194304] 0 2024-07-26T11:04:50.682 INFO:tasks.workunit.client.0.smithi144.stdout:7/673: mkdir d0/d6/d9/d3f/dd3/df2 0 2024-07-26T11:04:50.698 INFO:tasks.workunit.client.0.smithi144.stdout:2/677: dread d0/f3d [0,4194304] 0 2024-07-26T11:04:50.698 INFO:tasks.workunit.client.0.smithi144.stdout:9/710: dwrite d9/de/f51 [0,4194304] 0 2024-07-26T11:04:50.698 INFO:tasks.workunit.client.0.smithi144.stdout:2/678: mknod d0/d24/cd4 0 2024-07-26T11:04:50.702 INFO:tasks.workunit.client.0.smithi144.stdout:9/711: mkdir d9/d3b/d76/d9b/dcc/dea 0 2024-07-26T11:04:50.702 INFO:tasks.workunit.client.0.smithi144.stdout:9/712: dread - d9/fca zero size 2024-07-26T11:04:50.703 INFO:tasks.workunit.client.0.smithi144.stdout:2/679: truncate d0/d24/d2b/f59 4852564 0 2024-07-26T11:04:50.710 INFO:tasks.workunit.client.0.smithi144.stdout:9/713: mkdir d9/de/d4e/d6a/deb 0 2024-07-26T11:04:50.710 INFO:tasks.workunit.client.0.smithi144.stdout:2/680: truncate d0/d24/d48/d6d/d5f/f85 1170083 0 2024-07-26T11:04:50.713 INFO:tasks.workunit.client.0.smithi144.stdout:2/681: link d0/d6/d34/l3a d0/d24/d48/ld5 0 2024-07-26T11:04:50.737 INFO:tasks.workunit.client.0.smithi144.stdout:8/622: dwrite d1a/d32/d49/d73/d2c/f88 [4194304,4194304] 0 2024-07-26T11:04:50.739 INFO:tasks.workunit.client.0.smithi144.stdout:8/623: creat d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2/fd6 x:0 0 0 2024-07-26T11:04:50.740 INFO:tasks.workunit.client.0.smithi144.stdout:8/624: chown d1a/d32/f71 64430870 1 2024-07-26T11:04:50.742 INFO:tasks.workunit.client.0.smithi144.stdout:8/625: rmdir d1a/d32/d8c 39 2024-07-26T11:04:50.744 INFO:tasks.workunit.client.0.smithi144.stdout:8/626: mkdir d1a/d32/d49/d73/d2c/d38/d58/da3/dd7 0 2024-07-26T11:04:50.746 INFO:tasks.workunit.client.0.smithi144.stdout:8/627: mkdir d1a/d32/d49/d73/d2c/d38/d84/db0/dd8 0 2024-07-26T11:04:50.749 INFO:tasks.workunit.client.0.smithi144.stdout:9/714: dread d9/de/d14/f22 [4194304,4194304] 0 2024-07-26T11:04:50.751 INFO:tasks.workunit.client.0.smithi144.stdout:9/715: symlink d9/de/lec 0 2024-07-26T11:04:50.752 INFO:tasks.workunit.client.0.smithi144.stdout:9/716: mknod d9/de/d4e/d6a/deb/ced 0 2024-07-26T11:04:50.753 INFO:tasks.workunit.client.0.smithi144.stdout:4/701: dwrite d9/d26/d3c/d6f/f86 [0,4194304] 0 2024-07-26T11:04:50.753 INFO:tasks.workunit.client.0.smithi144.stdout:4/702: write d9/da/da1/fc1 [398065,120039] 0 2024-07-26T11:04:50.754 INFO:tasks.workunit.client.0.smithi144.stdout:9/717: rename d9/de/d14/d3f/d5e/f62 to d9/de/d14/d79/db2/fee 0 2024-07-26T11:04:50.754 INFO:tasks.workunit.client.0.smithi144.stdout:2/682: read d0/d24/d2b/f59 [454466,126816] 0 2024-07-26T11:04:50.754 INFO:tasks.workunit.client.0.smithi144.stdout:9/718: write d9/de/f4d [1164825,113847] 0 2024-07-26T11:04:50.754 INFO:tasks.workunit.client.0.smithi144.stdout:9/719: write d9/f29 [903617,79946] 0 2024-07-26T11:04:50.755 INFO:tasks.workunit.client.0.smithi144.stdout:2/683: write d0/d24/d48/d6d/fa0 [462515,118406] 0 2024-07-26T11:04:50.755 INFO:tasks.workunit.client.0.smithi144.stdout:7/674: dwrite d0/dd0/dbf/d2f/f70 [0,4194304] 0 2024-07-26T11:04:50.756 INFO:tasks.workunit.client.0.smithi144.stdout:4/703: creat d9/d26/d48/d50/d60/dc5/fee x:0 0 0 2024-07-26T11:04:50.756 INFO:tasks.workunit.client.0.smithi144.stdout:9/720: symlink d9/d3b/d81/ddd/lef 0 2024-07-26T11:04:50.756 INFO:tasks.workunit.client.0.smithi144.stdout:7/675: dread - d0/dd0/dbf/fb2 zero size 2024-07-26T11:04:50.757 INFO:tasks.workunit.client.0.smithi144.stdout:4/704: creat d9/da/fef x:0 0 0 2024-07-26T11:04:50.758 INFO:tasks.workunit.client.0.smithi144.stdout:9/721: mknod d9/de/d14/d23/d39/d7f/cf0 0 2024-07-26T11:04:50.758 INFO:tasks.workunit.client.0.smithi144.stdout:9/722: write d9/d3b/d76/d9b/dcc/fd0 [120415,106698] 0 2024-07-26T11:04:50.758 INFO:tasks.workunit.client.0.smithi144.stdout:9/723: readlink d9/d3b/d76/l7b 0 2024-07-26T11:04:50.759 INFO:tasks.workunit.client.0.smithi144.stdout:9/724: stat d9/fc 0 2024-07-26T11:04:50.759 INFO:tasks.workunit.client.0.smithi144.stdout:4/705: rmdir d9/d26/d48/d50/d60/dc5/dd9 39 2024-07-26T11:04:50.760 INFO:tasks.workunit.client.0.smithi144.stdout:9/725: truncate d9/de/d14/d23/d39/fab 1689559 0 2024-07-26T11:04:50.763 INFO:tasks.workunit.client.0.smithi144.stdout:7/676: symlink d0/dd0/dbf/d2f/d53/lf3 0 2024-07-26T11:04:50.763 INFO:tasks.workunit.client.0.smithi144.stdout:4/706: rmdir d9/d7b/dc4 39 2024-07-26T11:04:50.764 INFO:tasks.workunit.client.0.smithi144.stdout:7/677: mknod d0/d6/cf4 0 2024-07-26T11:04:50.766 INFO:tasks.workunit.client.0.smithi144.stdout:4/707: creat d9/d26/d48/d50/ff0 x:0 0 0 2024-07-26T11:04:50.768 INFO:tasks.workunit.client.0.smithi144.stdout:8/628: dread d1a/f20 [0,4194304] 0 2024-07-26T11:04:50.769 INFO:tasks.workunit.client.0.smithi144.stdout:8/629: mknod d1a/d32/dd4/cd9 0 2024-07-26T11:04:50.782 INFO:tasks.workunit.client.0.smithi144.stdout:8/630: symlink d1a/d32/d8c/dd0/lda 0 2024-07-26T11:04:50.782 INFO:tasks.workunit.client.0.smithi144.stdout:8/631: readlink d1a/d32/d49/d73/d2c/d9d/d74/l7d 0 2024-07-26T11:04:50.782 INFO:tasks.workunit.client.0.smithi144.stdout:8/632: getdents d1a/d32/d49/d73/d7b 0 2024-07-26T11:04:50.784 INFO:tasks.workunit.client.0.smithi144.stdout:8/633: symlink d1a/d32/d50/ldb 0 2024-07-26T11:04:50.786 INFO:tasks.workunit.client.0.smithi144.stdout:8/634: write d1a/d32/f62 [1551507,39570] 0 2024-07-26T11:04:50.787 INFO:tasks.workunit.client.0.smithi144.stdout:8/635: mkdir d1a/d32/d66/da0/ddc 0 2024-07-26T11:04:50.792 INFO:tasks.workunit.client.0.smithi144.stdout:8/636: link d1a/l5b d1a/d32/d49/d73/ldd 0 2024-07-26T11:04:50.792 INFO:tasks.workunit.client.0.smithi144.stdout:8/637: write d1a/d1c/f97 [427956,84938] 0 2024-07-26T11:04:50.809 INFO:tasks.workunit.client.0.smithi144.stdout:2/684: dwrite d0/d24/d48/d6d/f35 [0,4194304] 0 2024-07-26T11:04:50.809 INFO:tasks.workunit.client.0.smithi144.stdout:2/685: readlink d0/d24/d48/d6d/d28/l5 0 2024-07-26T11:04:50.818 INFO:tasks.workunit.client.0.smithi144.stdout:8/638: dread d1a/d32/d66/dcd/f53 [0,4194304] 0 2024-07-26T11:04:50.818 INFO:tasks.workunit.client.0.smithi144.stdout:8/639: rename d1a to d1a/d32/d49/d73/d7b/d4e/d78/dde 22 2024-07-26T11:04:50.824 INFO:tasks.workunit.client.0.smithi144.stdout:4/708: dwrite d9/d26/d48/d98/fb1 [0,4194304] 0 2024-07-26T11:04:50.824 INFO:tasks.workunit.client.0.smithi144.stdout:9/726: dwrite d9/de/f20 [0,4194304] 0 2024-07-26T11:04:50.824 INFO:tasks.workunit.client.0.smithi144.stdout:9/727: creat d9/d3b/ff1 x:0 0 0 2024-07-26T11:04:50.824 INFO:tasks.workunit.client.0.smithi144.stdout:9/728: stat d9/de/d14/d23/l28 0 2024-07-26T11:04:50.825 INFO:tasks.workunit.client.0.smithi144.stdout:8/640: creat d1a/d32/d8c/d9b/fdf x:0 0 0 2024-07-26T11:04:50.825 INFO:tasks.workunit.client.0.smithi144.stdout:7/678: dwrite d0/dd0/dbf/d2f/d53/db6/fca [0,4194304] 0 2024-07-26T11:04:50.834 INFO:tasks.workunit.client.0.smithi144.stdout:8/641: truncate d1a/d32/d49/d73/f3f 6823814 0 2024-07-26T11:04:50.835 INFO:tasks.workunit.client.0.smithi144.stdout:7/679: mkdir d0/d71/df5 0 2024-07-26T11:04:50.845 INFO:tasks.workunit.client.0.smithi144.stdout:2/686: dread d0/d24/d48/d6d/d28/d2e/d77/fbd [0,4194304] 0 2024-07-26T11:04:50.850 INFO:tasks.workunit.client.0.smithi144.stdout:2/687: symlink d0/d24/d48/d6d/d28/d2e/d5d/dae/ld6 0 2024-07-26T11:04:50.850 INFO:tasks.workunit.client.0.smithi144.stdout:2/688: unlink d0/d24/d2b/d68/d82/l97 0 2024-07-26T11:04:50.859 INFO:tasks.workunit.client.0.smithi144.stdout:8/642: dread d1a/d32/d49/d73/d7b/f5e [0,4194304] 0 2024-07-26T11:04:50.862 INFO:tasks.workunit.client.0.smithi144.stdout:7/680: dread d0/dd0/dbf/f41 [0,4194304] 0 2024-07-26T11:04:50.862 INFO:tasks.workunit.client.0.smithi144.stdout:7/681: fsync d0/d6/d9/d3f/f6e 0 2024-07-26T11:04:50.862 INFO:tasks.workunit.client.0.smithi144.stdout:7/682: truncate d0/dd0/dbf/d2f/d68/fc6 847653 0 2024-07-26T11:04:50.862 INFO:tasks.workunit.client.0.smithi144.stdout:7/683: chown d0/dd0/dbf/d2f/d68 0 1 2024-07-26T11:04:50.883 INFO:tasks.workunit.client.0.smithi144.stdout:8/643: dread d1a/d32/d49/d73/d7b/f37 [0,4194304] 0 2024-07-26T11:04:50.901 INFO:tasks.workunit.client.0.smithi144.stdout:2/689: dwrite d0/d6/f89 [0,4194304] 0 2024-07-26T11:04:50.901 INFO:tasks.workunit.client.0.smithi144.stdout:2/690: write d0/d24/d48/d6d/d28/d2e/d5d/d6b/fc9 [808530,76596] 0 2024-07-26T11:04:50.901 INFO:tasks.workunit.client.0.smithi144.stdout:2/691: write d0/d24/d2b/f59 [3083570,23712] 0 2024-07-26T11:04:50.901 INFO:tasks.workunit.client.0.smithi144.stdout:7/684: dwrite d0/dd0/dbf/d2f/d47/fa0 [0,4194304] 0 2024-07-26T11:04:50.905 INFO:tasks.workunit.client.0.smithi144.stdout:7/685: rmdir d0/dd0/dbf/d2f/d68 39 2024-07-26T11:04:50.906 INFO:tasks.workunit.client.0.smithi144.stdout:2/692: rmdir d0/d24/d48/d6d/d28/d2e/d5d/dae 39 2024-07-26T11:04:50.908 INFO:tasks.workunit.client.0.smithi144.stdout:2/693: rename d0/d24/d48/d6d/d5f/l7d to d0/d24/d2b/d68/d82/ld7 0 2024-07-26T11:04:50.908 INFO:tasks.workunit.client.0.smithi144.stdout:2/694: write d0/d24/d48/d6d/d28/d2e/d77/fbd [473121,82371] 0 2024-07-26T11:04:50.909 INFO:tasks.workunit.client.0.smithi144.stdout:2/695: creat d0/d24/d48/d6d/d28/da8/fd8 x:0 0 0 2024-07-26T11:04:50.909 INFO:tasks.workunit.client.0.smithi144.stdout:2/696: write d0/d24/d48/d6d/f96 [515437,122398] 0 2024-07-26T11:04:50.911 INFO:tasks.workunit.client.0.smithi144.stdout:8/644: read d1a/d1c/f41 [3823076,18338] 0 2024-07-26T11:04:50.913 INFO:tasks.workunit.client.0.smithi144.stdout:8/645: rename d1a/d32/d6c/d82/f9e to d1a/d32/d49/d73/d2c/d38/fe0 0 2024-07-26T11:04:50.914 INFO:tasks.workunit.client.0.smithi144.stdout:8/646: rmdir d1a/d32 39 2024-07-26T11:04:50.914 INFO:tasks.workunit.client.0.smithi144.stdout:8/647: fsync d1a/f2b 0 2024-07-26T11:04:50.947 INFO:tasks.workunit.client.0.smithi144.stdout:2/697: write d0/d24/d2b/d68/d82/fab [1949755,13603] 0 2024-07-26T11:04:50.948 INFO:tasks.workunit.client.0.smithi144.stdout:2/698: dread - d0/d24/d48/d63/da3/fbc zero size 2024-07-26T11:04:50.987 INFO:tasks.workunit.client.0.smithi144.stdout:7/686: dwrite d0/dd0/dbf/d2f/da2/fbe [0,4194304] 0 2024-07-26T11:04:50.987 INFO:tasks.workunit.client.0.smithi144.stdout:8/648: dwrite d1a/d32/d49/d73/d7b/f39 [0,4194304] 0 2024-07-26T11:04:50.991 INFO:tasks.workunit.client.0.smithi144.stdout:2/699: dwrite d0/d24/d48/fa1 [0,4194304] 0 2024-07-26T11:04:51.006 INFO:tasks.workunit.client.0.smithi144.stdout:8/649: dread d1a/d32/d49/d73/d2c/d38/d58/f6e [0,4194304] 0 2024-07-26T11:04:51.051 INFO:tasks.workunit.client.0.smithi144.stdout:8/650: dwrite d1a/f2b [0,4194304] 0 2024-07-26T11:04:51.078 INFO:tasks.workunit.client.0.smithi144.stdout:8/651: write d1a/d32/d49/d73/f80 [437500,76957] 0 2024-07-26T11:04:51.244 INFO:tasks.workunit.client.0.smithi144.stdout:7/687: link d0/dd0/dbf/d2f/d68/fb8 d0/dd0/dbf/d2f/d68/de2/ff6 0 2024-07-26T11:04:51.245 INFO:tasks.workunit.client.0.smithi144.stdout:2/700: mkdir d0/d24/d48/d6d/dd9 0 2024-07-26T11:04:51.245 INFO:tasks.workunit.client.0.smithi144.stdout:2/701: chown d0/d24/d48/d6d/d28/d2e/d5d 182 1 2024-07-26T11:04:51.249 INFO:tasks.workunit.client.0.smithi144.stdout:8/652: mknod d1a/d32/d49/d73/d7b/d4e/d9a/dc1/ce1 0 2024-07-26T11:04:51.250 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:04:51.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:04:51.252 INFO:tasks.workunit.client.0.smithi144.stdout:7/688: truncate d0/f11 4182 0 2024-07-26T11:04:51.252 INFO:tasks.workunit.client.0.smithi144.stdout:2/702: symlink d0/d24/d48/d6d/d28/d2e/d5d/d6b/dc2/lda 0 2024-07-26T11:04:51.262 INFO:tasks.workunit.client.0.smithi144.stdout:8/653: mknod d1a/d32/d8c/ce2 0 2024-07-26T11:04:51.262 INFO:tasks.workunit.client.0.smithi144.stdout:8/654: mknod d1a/d32/d49/d73/d2c/d9d/d74/dac/dbe/ce3 0 2024-07-26T11:04:51.262 INFO:tasks.workunit.client.0.smithi144.stdout:7/689: mknod d0/dd0/dbf/d67/de0/cf7 0 2024-07-26T11:04:51.263 INFO:tasks.workunit.client.0.smithi144.stdout:7/690: write d0/d71/d95/dc4/f65 [770706,36157] 0 2024-07-26T11:04:51.271 INFO:tasks.workunit.client.0.smithi144.stdout:7/691: creat d0/d71/d95/de7/ff8 x:0 0 0 2024-07-26T11:04:51.282 INFO:tasks.workunit.client.0.smithi144.stdout:8/655: dread d1a/d32/d66/dcd/f53 [0,4194304] 0 2024-07-26T11:04:51.283 INFO:tasks.workunit.client.0.smithi144.stdout:8/656: mkdir d1a/d32/d49/d73/d2c/d38/d58/de4 0 2024-07-26T11:04:51.285 INFO:tasks.workunit.client.0.smithi144.stdout:2/703: write d0/d6/f95 [1554458,41613] 0 2024-07-26T11:04:51.285 INFO:tasks.workunit.client.0.smithi144.stdout:8/657: rename d1a/d32/d6c/l89 to d1a/d32/d49/d73/d7b/d4e/d9a/dc1/le5 0 2024-07-26T11:04:51.287 INFO:tasks.workunit.client.0.smithi144.stdout:2/704: mkdir d0/d24/d2b/d68/d82/ddb 0 2024-07-26T11:04:51.287 INFO:tasks.workunit.client.0.smithi144.stdout:2/705: fdatasync d0/d24/d48/d6d/d28/d2e/d77/f40 0 2024-07-26T11:04:51.289 INFO:tasks.workunit.client.0.smithi144.stdout:2/706: getdents d0/d6/d9/d44 0 2024-07-26T11:04:51.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/707: stat d0/d6/d9/d44/dc3/lcf 0 2024-07-26T11:04:51.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/708: readlink d0/d24/d7b/la5 0 2024-07-26T11:04:51.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/709: truncate d0/f3d 5559314 0 2024-07-26T11:04:51.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/710: truncate d0/d24/d48/d6d/fa0 1315230 0 2024-07-26T11:04:51.294 INFO:tasks.workunit.client.0.smithi144.stdout:2/711: rename d0/d24/d48/d6d to d0/d24/d7b/ddc 0 2024-07-26T11:04:51.295 INFO:tasks.workunit.client.0.smithi144.stdout:2/712: mknod d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2/cdd 0 2024-07-26T11:04:51.298 INFO:tasks.workunit.client.0.smithi144.stdout:7/692: dread d0/d6/d9/d12/f59 [0,4194304] 0 2024-07-26T11:04:51.298 INFO:tasks.workunit.client.0.smithi144.stdout:2/713: read - d0/d6/d9/d44/fa2 zero size 2024-07-26T11:04:51.299 INFO:tasks.workunit.client.0.smithi144.stdout:7/693: mknod d0/d71/dc0/de4/cf9 0 2024-07-26T11:04:51.301 INFO:tasks.workunit.client.0.smithi144.stdout:2/714: unlink d0/d24/d7b/ddc/d28/f5c 0 2024-07-26T11:04:51.303 INFO:tasks.workunit.client.0.smithi144.stdout:2/715: creat d0/db8/fde x:0 0 0 2024-07-26T11:04:51.304 INFO:tasks.workunit.client.0.smithi144.stdout:2/716: chown d0/d24/d7b/ddc/f35 4542 1 2024-07-26T11:04:51.305 INFO:tasks.workunit.client.0.smithi144.stdout:2/717: link d0/d24/d2b/d68/l9c d0/d6/d34/ldf 0 2024-07-26T11:04:51.322 INFO:tasks.workunit.client.0.smithi144.stdout:8/658: read d1a/f20 [2521741,45753] 0 2024-07-26T11:04:51.354 INFO:tasks.workunit.client.0.smithi144.stdout:8/659: dread d1a/d32/d49/d73/d2c/d38/f5f [0,4194304] 0 2024-07-26T11:04:51.361 INFO:tasks.workunit.client.0.smithi144.stdout:8/660: rmdir d1a/d32/d49 39 2024-07-26T11:04:51.396 INFO:tasks.workunit.client.0.smithi144.stdout:8/661: dwrite d1a/d32/d49/d73/f80 [0,4194304] 0 2024-07-26T11:04:51.397 INFO:tasks.workunit.client.0.smithi144.stdout:8/662: symlink d1a/d32/d49/d73/d2c/d38/le6 0 2024-07-26T11:04:51.397 INFO:tasks.workunit.client.0.smithi144.stdout:8/663: dread - d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2/fd6 zero size 2024-07-26T11:04:51.398 INFO:tasks.workunit.client.0.smithi144.stdout:8/664: unlink d1a/d32/c42 0 2024-07-26T11:04:51.399 INFO:tasks.workunit.client.0.smithi144.stdout:8/665: link d1a/d32/d66/da0/cb4 d1a/d32/d49/d73/d2c/d9d/ce7 0 2024-07-26T11:04:51.400 INFO:tasks.workunit.client.0.smithi144.stdout:8/666: write d1a/d32/d49/d73/d7b/f5e [2414505,14686] 0 2024-07-26T11:04:51.421 INFO:tasks.workunit.client.0.smithi144.stdout:8/667: dread d1a/d32/d49/d73/f80 [0,4194304] 0 2024-07-26T11:04:51.421 INFO:tasks.workunit.client.0.smithi144.stdout:8/668: fsync d1a/d32/d49/d73/d7b/f39 0 2024-07-26T11:04:51.422 INFO:tasks.workunit.client.0.smithi144.stdout:8/669: creat d1a/d32/d8c/d9b/fe8 x:0 0 0 2024-07-26T11:04:51.423 INFO:tasks.workunit.client.0.smithi144.stdout:8/670: mknod d1a/d32/d49/d73/d7b/d4e/d9a/dc1/ce9 0 2024-07-26T11:04:51.439 INFO:tasks.workunit.client.0.smithi144.stdout:8/671: write d1a/d32/d49/d73/d2c/d38/d58/f83 [3684307,86644] 0 2024-07-26T11:04:51.439 INFO:tasks.workunit.client.0.smithi144.stdout:8/672: write d1a/d32/f71 [503632,42247] 0 2024-07-26T11:04:51.439 INFO:tasks.workunit.client.0.smithi144.stdout:8/673: truncate d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2/fcb 394222 0 2024-07-26T11:04:51.441 INFO:tasks.workunit.client.0.smithi144.stdout:8/674: creat d1a/d32/d66/da0/fea x:0 0 0 2024-07-26T11:04:51.451 INFO:tasks.workunit.client.0.smithi144.stdout:8/675: dread d1a/d32/d49/d73/f79 [0,4194304] 0 2024-07-26T11:04:51.452 INFO:tasks.workunit.client.0.smithi144.stdout:8/676: mknod d1a/d32/d49/d73/d2c/d9d/d74/ceb 0 2024-07-26T11:04:51.452 INFO:tasks.workunit.client.0.smithi144.stdout:8/677: stat d1a/d32/d6c/d82/cb3 0 2024-07-26T11:04:51.452 INFO:tasks.workunit.client.0.smithi144.stdout:8/678: fsync d1a/d32/d49/d73/d7b/f37 0 2024-07-26T11:04:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:51 smithi144 ceph-mon[32129]: pgmap v532: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s rd, 168 MiB/s wr, 639 op/s 2024-07-26T11:04:51.492 INFO:tasks.workunit.client.0.smithi144.stdout:8/679: dwrite d1a/d32/d49/d73/d2c/d38/d58/fa8 [0,4194304] 0 2024-07-26T11:04:51.493 INFO:tasks.workunit.client.0.smithi144.stdout:8/680: unlink d1a/d32/d49/d73/d7b/d4e/l8e 0 2024-07-26T11:04:51.493 INFO:tasks.workunit.client.0.smithi144.stdout:8/681: fdatasync d1a/d32/d49/d73/d2c/f4d 0 2024-07-26T11:04:51.494 INFO:tasks.workunit.client.0.smithi144.stdout:8/682: creat d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/fec x:0 0 0 2024-07-26T11:04:51.495 INFO:tasks.workunit.client.0.smithi144.stdout:8/683: stat d1a/d32/d49/d73/d2c/d9d/ce7 0 2024-07-26T11:04:51.499 INFO:tasks.workunit.client.0.smithi144.stdout:8/684: getdents d1a/d32/d49/d73/d2c/d9d 0 2024-07-26T11:04:51.518 INFO:tasks.workunit.client.0.smithi144.stdout:5/695: sync 2024-07-26T11:04:51.518 INFO:tasks.workunit.client.0.smithi144.stdout:9/729: sync 2024-07-26T11:04:51.518 INFO:tasks.workunit.client.0.smithi144.stdout:3/474: sync 2024-07-26T11:04:51.519 INFO:tasks.workunit.client.0.smithi144.stdout:4/709: sync 2024-07-26T11:04:51.519 INFO:tasks.workunit.client.0.smithi144.stdout:6/802: sync 2024-07-26T11:04:51.519 INFO:tasks.workunit.client.0.smithi144.stdout:0/662: sync 2024-07-26T11:04:51.520 INFO:tasks.workunit.client.0.smithi144.stdout:1/879: creat d3/d6/d75/d88/da9/dac/d104/ddb/f119 x:0 0 0 2024-07-26T11:04:51.520 INFO:tasks.workunit.client.0.smithi144.stdout:6/803: chown d8/d15/d57/l65 2172961 1 2024-07-26T11:04:51.522 INFO:tasks.workunit.client.0.smithi144.stdout:9/730: unlink d9/d3b/d76/fb5 0 2024-07-26T11:04:51.522 INFO:tasks.workunit.client.0.smithi144.stdout:5/696: mkdir d1/d18/dc4/de0/deb 0 2024-07-26T11:04:51.522 INFO:tasks.workunit.client.0.smithi144.stdout:3/475: creat d1/d28/d54/d26/d30/d78/f8a x:0 0 0 2024-07-26T11:04:51.522 INFO:tasks.workunit.client.0.smithi144.stdout:3/476: stat d1/d28 0 2024-07-26T11:04:51.523 INFO:tasks.workunit.client.0.smithi144.stdout:1/880: symlink d3/d6/da/d70/dd6/l11a 0 2024-07-26T11:04:51.523 INFO:tasks.workunit.client.0.smithi144.stdout:1/881: chown d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/faf 31486 1 2024-07-26T11:04:51.523 INFO:tasks.workunit.client.0.smithi144.stdout:0/663: rename d0/d25/fa7 to d0/d25/d5e/d1e/d6c/dc6/fdc 0 2024-07-26T11:04:51.525 INFO:tasks.workunit.client.0.smithi144.stdout:8/685: write f6 [3083397,55653] 0 2024-07-26T11:04:51.525 INFO:tasks.workunit.client.0.smithi144.stdout:6/804: mknod d8/db/dbe/d52/d58/c11e 0 2024-07-26T11:04:51.525 INFO:tasks.workunit.client.0.smithi144.stdout:5/697: unlink d1/d9/l33 0 2024-07-26T11:04:51.525 INFO:tasks.workunit.client.0.smithi144.stdout:1/882: creat d3/d6/da/d97/f11b x:0 0 0 2024-07-26T11:04:51.525 INFO:tasks.workunit.client.0.smithi144.stdout:5/698: write d1/d5/dae/fbd [673500,113265] 0 2024-07-26T11:04:51.525 INFO:tasks.workunit.client.0.smithi144.stdout:9/731: creat d9/d3b/dce/de4/ff2 x:0 0 0 2024-07-26T11:04:51.526 INFO:tasks.workunit.client.0.smithi144.stdout:0/664: truncate d0/d3e/d45/f55 389071 0 2024-07-26T11:04:51.526 INFO:tasks.workunit.client.0.smithi144.stdout:6/805: rmdir d8/db/dbe/d52/d58/dbf 39 2024-07-26T11:04:51.527 INFO:tasks.workunit.client.0.smithi144.stdout:1/883: creat d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/f11c x:0 0 0 2024-07-26T11:04:51.527 INFO:tasks.workunit.client.0.smithi144.stdout:1/884: readlink d3/d3d/lce 0 2024-07-26T11:04:51.527 INFO:tasks.workunit.client.0.smithi144.stdout:1/885: write d3/f2d [989305,48280] 0 2024-07-26T11:04:51.527 INFO:tasks.workunit.client.0.smithi144.stdout:5/699: rmdir d1/d18/d24/d35/de4 0 2024-07-26T11:04:51.528 INFO:tasks.workunit.client.0.smithi144.stdout:9/732: creat d9/d3b/d76/d9b/dcc/ff3 x:0 0 0 2024-07-26T11:04:51.528 INFO:tasks.workunit.client.0.smithi144.stdout:0/665: symlink d0/d25/d5e/d1e/d5c/ldd 0 2024-07-26T11:04:51.528 INFO:tasks.workunit.client.0.smithi144.stdout:0/666: fdatasync d0/d25/d5e/d1e/d30/d67/fc5 0 2024-07-26T11:04:51.528 INFO:tasks.workunit.client.0.smithi144.stdout:0/667: fdatasync d0/d25/d5e/d1e/d91/f96 0 2024-07-26T11:04:51.528 INFO:tasks.workunit.client.0.smithi144.stdout:0/668: fsync d0/d3e/d45/d4c/f64 0 2024-07-26T11:04:51.529 INFO:tasks.workunit.client.0.smithi144.stdout:1/886: mknod d3/d6/da/d13/d19/d3a/d100/c11d 0 2024-07-26T11:04:51.529 INFO:tasks.workunit.client.0.smithi144.stdout:1/887: chown d3/d6/da/d21/lf1 8331336 1 2024-07-26T11:04:51.529 INFO:tasks.workunit.client.0.smithi144.stdout:5/700: rename d1/d5/c7 to d1/cec 0 2024-07-26T11:04:51.529 INFO:tasks.workunit.client.0.smithi144.stdout:0/669: mknod d0/d3e/cde 0 2024-07-26T11:04:51.530 INFO:tasks.workunit.client.0.smithi144.stdout:9/733: creat d9/d3b/d76/d9b/dd7/ff4 x:0 0 0 2024-07-26T11:04:51.530 INFO:tasks.workunit.client.0.smithi144.stdout:0/670: creat d0/d25/d5e/d1e/d6c/fdf x:0 0 0 2024-07-26T11:04:51.532 INFO:tasks.workunit.client.0.smithi144.stdout:5/701: rename d1/d9/d44/d87 to d1/d18/dc4/ded 0 2024-07-26T11:04:51.532 INFO:tasks.workunit.client.0.smithi144.stdout:6/806: unlink d8/d15/d57/c5d 0 2024-07-26T11:04:51.532 INFO:tasks.workunit.client.0.smithi144.stdout:0/671: mkdir d0/d3e/d45/dd4/de0 0 2024-07-26T11:04:51.532 INFO:tasks.workunit.client.0.smithi144.stdout:0/672: truncate d0/f78 1044628 0 2024-07-26T11:04:51.533 INFO:tasks.workunit.client.0.smithi144.stdout:5/702: creat d1/d9/d44/fee x:0 0 0 2024-07-26T11:04:51.534 INFO:tasks.workunit.client.0.smithi144.stdout:0/673: symlink d0/d3e/d5f/le1 0 2024-07-26T11:04:51.534 INFO:tasks.workunit.client.0.smithi144.stdout:9/734: creat d9/ff5 x:0 0 0 2024-07-26T11:04:51.535 INFO:tasks.workunit.client.0.smithi144.stdout:5/703: stat d1/d9/d44/d85/l94 0 2024-07-26T11:04:51.535 INFO:tasks.workunit.client.0.smithi144.stdout:6/807: write d8/d15/d26/f28 [2021076,99124] 0 2024-07-26T11:04:51.536 INFO:tasks.workunit.client.0.smithi144.stdout:9/735: mknod d9/cf6 0 2024-07-26T11:04:51.537 INFO:tasks.workunit.client.0.smithi144.stdout:3/477: write d1/d28/d54/d26/d30/d4d/f55 [430461,53243] 0 2024-07-26T11:04:51.539 INFO:tasks.workunit.client.0.smithi144.stdout:6/808: link d8/db/dbe/d5c/f73 d8/d15/d57/d5f/d6e/f11f 0 2024-07-26T11:04:51.539 INFO:tasks.workunit.client.0.smithi144.stdout:5/704: rename d1/d5/ca to d1/d9/d5b/cef 0 2024-07-26T11:04:51.542 INFO:tasks.workunit.client.0.smithi144.stdout:3/478: mknod d1/d28/d54/d3d/d42/d44/d48/c8b 0 2024-07-26T11:04:51.543 INFO:tasks.workunit.client.0.smithi144.stdout:9/736: mknod d9/de/d14/d23/d39/cf7 0 2024-07-26T11:04:51.544 INFO:tasks.workunit.client.0.smithi144.stdout:8/686: dread d1a/d32/f62 [0,4194304] 0 2024-07-26T11:04:51.548 INFO:tasks.workunit.client.0.smithi144.stdout:5/705: unlink d1/d5/dae/fb5 0 2024-07-26T11:04:51.548 INFO:tasks.workunit.client.0.smithi144.stdout:6/809: mkdir d8/db/dbe/d52/d58/d120 0 2024-07-26T11:04:51.548 INFO:tasks.workunit.client.0.smithi144.stdout:3/479: creat d1/d28/d54/d3d/d42/d44/d60/f8c x:0 0 0 2024-07-26T11:04:51.549 INFO:tasks.workunit.client.0.smithi144.stdout:9/737: truncate d9/de/f15 158710 0 2024-07-26T11:04:51.549 INFO:tasks.workunit.client.0.smithi144.stdout:9/738: chown d9/de/d14/d3f/d5e/l6d 110918 1 2024-07-26T11:04:51.549 INFO:tasks.workunit.client.0.smithi144.stdout:8/687: mknod d1a/d32/d49/d73/d7b/d43/ced 0 2024-07-26T11:04:51.549 INFO:tasks.workunit.client.0.smithi144.stdout:8/688: stat d1a/d32/d49/d73/d2c/d38/dbd 0 2024-07-26T11:04:51.549 INFO:tasks.workunit.client.0.smithi144.stdout:6/810: unlink d8/d15/c116 0 2024-07-26T11:04:51.552 INFO:tasks.workunit.client.0.smithi144.stdout:5/706: fdatasync d1/d9/d5b/f21 0 2024-07-26T11:04:51.555 INFO:tasks.workunit.client.0.smithi144.stdout:9/739: dread d9/d3b/d76/d9b/fb7 [0,4194304] 0 2024-07-26T11:04:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:51 smithi160 ceph-mon[36715]: pgmap v532: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s rd, 168 MiB/s wr, 639 op/s 2024-07-26T11:04:51.560 INFO:tasks.workunit.client.0.smithi144.stdout:9/740: symlink d9/d3b/d5d/lf8 0 2024-07-26T11:04:51.564 INFO:tasks.workunit.client.0.smithi144.stdout:9/741: creat d9/de/d4e/d6a/ff9 x:0 0 0 2024-07-26T11:04:51.564 INFO:tasks.workunit.client.0.smithi144.stdout:1/888: dread d3/d6/da/d78/f9d [0,4194304] 0 2024-07-26T11:04:51.564 INFO:tasks.workunit.client.0.smithi144.stdout:3/480: write d1/f39 [258189,23543] 0 2024-07-26T11:04:51.565 INFO:tasks.workunit.client.0.smithi144.stdout:4/710: dwrite d9/d26/fb6 [0,4194304] 0 2024-07-26T11:04:51.565 INFO:tasks.workunit.client.0.smithi144.stdout:4/711: read - d9/d7b/fe6 zero size 2024-07-26T11:04:51.566 INFO:tasks.workunit.client.0.smithi144.stdout:3/481: mknod d1/d28/d54/d3d/d42/d44/d48/c8d 0 2024-07-26T11:04:51.566 INFO:tasks.workunit.client.0.smithi144.stdout:1/889: symlink d3/d6/da/d21/l11e 0 2024-07-26T11:04:51.568 INFO:tasks.workunit.client.0.smithi144.stdout:9/742: symlink d9/de/d14/d3f/lfa 0 2024-07-26T11:04:51.571 INFO:tasks.workunit.client.0.smithi144.stdout:9/743: mknod d9/d3b/dce/de4/cfb 0 2024-07-26T11:04:51.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/712: truncate d9/d26/f28 62131 0 2024-07-26T11:04:51.575 INFO:tasks.workunit.client.0.smithi144.stdout:4/713: readlink d9/d57/l74 0 2024-07-26T11:04:51.575 INFO:tasks.workunit.client.0.smithi144.stdout:9/744: truncate d9/d3b/f56 2506898 0 2024-07-26T11:04:51.583 INFO:tasks.workunit.client.0.smithi144.stdout:9/745: truncate d9/de/d14/f2c 3034642 0 2024-07-26T11:04:51.583 INFO:tasks.workunit.client.0.smithi144.stdout:9/746: stat d9/de/d14/d3f/l60 0 2024-07-26T11:04:51.587 INFO:tasks.workunit.client.0.smithi144.stdout:9/747: mkdir d9/dfc 0 2024-07-26T11:04:51.587 INFO:tasks.workunit.client.0.smithi144.stdout:9/748: chown d9/de/f52 671191148 1 2024-07-26T11:04:51.627 INFO:tasks.workunit.client.0.smithi144.stdout:6/811: dread d8/db/dbe/f51 [0,4194304] 0 2024-07-26T11:04:51.632 INFO:tasks.workunit.client.0.smithi144.stdout:0/674: dwrite d0/d3e/d45/d9f/fa8 [0,4194304] 0 2024-07-26T11:04:51.632 INFO:tasks.workunit.client.0.smithi144.stdout:0/675: dread - d0/d25/d5e/d1e/d30/d67/d84/fbd zero size 2024-07-26T11:04:51.634 INFO:tasks.workunit.client.0.smithi144.stdout:4/714: dread d9/d26/d3c/d6f/f86 [0,4194304] 0 2024-07-26T11:04:51.634 INFO:tasks.workunit.client.0.smithi144.stdout:4/715: chown d9/d26/d48/d50/d60/la8 2 1 2024-07-26T11:04:51.634 INFO:tasks.workunit.client.0.smithi144.stdout:4/716: read d9/f12 [1008492,129479] 0 2024-07-26T11:04:51.635 INFO:tasks.workunit.client.0.smithi144.stdout:6/812: creat d8/d15/d26/f121 x:0 0 0 2024-07-26T11:04:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:51 smithi183 ceph-mon[37273]: pgmap v532: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s rd, 168 MiB/s wr, 639 op/s 2024-07-26T11:04:51.635 INFO:tasks.workunit.client.0.smithi144.stdout:0/676: creat d0/d3e/db4/fe2 x:0 0 0 2024-07-26T11:04:51.635 INFO:tasks.workunit.client.0.smithi144.stdout:0/677: chown d0/d25/d5e/d1e/d30/d67/c93 0 1 2024-07-26T11:04:51.637 INFO:tasks.workunit.client.0.smithi144.stdout:0/678: link d0/d25/d5e/c2b d0/d3e/db4/ce3 0 2024-07-26T11:04:51.640 INFO:tasks.workunit.client.0.smithi144.stdout:0/679: unlink d0/d3e/d45/f46 0 2024-07-26T11:04:51.642 INFO:tasks.workunit.client.0.smithi144.stdout:0/680: mkdir d0/d25/d5e/d1e/d30/d6a/de4 0 2024-07-26T11:04:51.642 INFO:tasks.workunit.client.0.smithi144.stdout:0/681: chown d0/d25/d5e/d1e/d30/d67/d84/cb3 3 1 2024-07-26T11:04:51.643 INFO:tasks.workunit.client.0.smithi144.stdout:6/813: symlink d8/d15/l122 0 2024-07-26T11:04:51.643 INFO:tasks.workunit.client.0.smithi144.stdout:0/682: creat d0/d25/d5e/d1e/d30/d6a/dac/fe5 x:0 0 0 2024-07-26T11:04:51.646 INFO:tasks.workunit.client.0.smithi144.stdout:0/683: link d0/d3e/d5f/l80 d0/d25/d5e/d1e/dd2/le6 0 2024-07-26T11:04:51.647 INFO:tasks.workunit.client.0.smithi144.stdout:5/707: dwrite d1/d9/d44/d45/d81/f91 [4194304,4194304] 0 2024-07-26T11:04:51.652 INFO:tasks.workunit.client.0.smithi144.stdout:0/684: rename d0/d25/d5e/fbf to d0/d25/d5e/d1e/dc1/fe7 0 2024-07-26T11:04:51.653 INFO:tasks.workunit.client.0.smithi144.stdout:6/814: creat d8/db/dbe/d52/d58/f123 x:0 0 0 2024-07-26T11:04:51.653 INFO:tasks.workunit.client.0.smithi144.stdout:5/708: mknod d1/d18/d24/d35/cf0 0 2024-07-26T11:04:51.654 INFO:tasks.workunit.client.0.smithi144.stdout:8/689: dwrite d1a/d32/d49/fb7 [0,4194304] 0 2024-07-26T11:04:51.654 INFO:tasks.workunit.client.0.smithi144.stdout:8/690: write d1a/d32/d6c/d82/fa4 [674494,91957] 0 2024-07-26T11:04:51.654 INFO:tasks.workunit.client.0.smithi144.stdout:8/691: stat d1a/d1c/f97 0 2024-07-26T11:04:51.657 INFO:tasks.workunit.client.0.smithi144.stdout:5/709: link d1/d5/l22 d1/d9/d5b/d4d/lf1 0 2024-07-26T11:04:51.657 INFO:tasks.workunit.client.0.smithi144.stdout:8/692: truncate d1a/d32/d49/d73/d7b/d4e/d78/f90 747063 0 2024-07-26T11:04:51.659 INFO:tasks.workunit.client.0.smithi144.stdout:5/710: symlink d1/d9/d5b/lf2 0 2024-07-26T11:04:51.659 INFO:tasks.workunit.client.0.smithi144.stdout:5/711: fdatasync d1/d9/d5b/d86/fba 0 2024-07-26T11:04:51.659 INFO:tasks.workunit.client.0.smithi144.stdout:5/712: write d1/f5e [1413929,62221] 0 2024-07-26T11:04:51.660 INFO:tasks.workunit.client.0.smithi144.stdout:8/693: mkdir d1a/d32/d66/dcd/dee 0 2024-07-26T11:04:51.660 INFO:tasks.workunit.client.0.smithi144.stdout:8/694: chown d1a/d32/d49/d73/d7b/d4e/d9a/dc1/le5 206602094 1 2024-07-26T11:04:51.660 INFO:tasks.workunit.client.0.smithi144.stdout:8/695: stat d1a/d32/d49/d73/d2c/d38/d84/db0 0 2024-07-26T11:04:51.660 INFO:tasks.workunit.client.0.smithi144.stdout:8/696: write fe [1369943,4561] 0 2024-07-26T11:04:51.660 INFO:tasks.workunit.client.0.smithi144.stdout:6/815: creat d8/dca/df5/f124 x:0 0 0 2024-07-26T11:04:51.661 INFO:tasks.workunit.client.0.smithi144.stdout:5/713: creat d1/d18/d24/d35/ff3 x:0 0 0 2024-07-26T11:04:51.662 INFO:tasks.workunit.client.0.smithi144.stdout:8/697: getdents d1a/d32/d49/d73/d2c/d38/d84/db0/dd8 0 2024-07-26T11:04:51.662 INFO:tasks.workunit.client.0.smithi144.stdout:1/890: dwrite d3/d6/da/d21/d5c/f110 [0,4194304] 0 2024-07-26T11:04:51.665 INFO:tasks.workunit.client.0.smithi144.stdout:8/698: chown d1a/d32/d49/d73/d2c/d38/d58 5677786 1 2024-07-26T11:04:51.666 INFO:tasks.workunit.client.0.smithi144.stdout:8/699: creat d1a/d32/d66/da0/ddc/fef x:0 0 0 2024-07-26T11:04:51.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/714: creat d1/d9/d5b/d86/ff4 x:0 0 0 2024-07-26T11:04:51.668 INFO:tasks.workunit.client.0.smithi144.stdout:1/891: creat d3/d6/da/d21/d5c/f11f x:0 0 0 2024-07-26T11:04:51.668 INFO:tasks.workunit.client.0.smithi144.stdout:1/892: fdatasync d3/d6/dcc/fd9 0 2024-07-26T11:04:51.668 INFO:tasks.workunit.client.0.smithi144.stdout:1/893: truncate d3/d6/da/d21/d5c/d62/fd2 476462 0 2024-07-26T11:04:51.677 INFO:tasks.workunit.client.0.smithi144.stdout:3/482: dwrite d1/d28/d54/d3d/d42/d44/d60/f8c [0,4194304] 0 2024-07-26T11:04:51.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/717: write d9/da/f69 [190801,78286] 0 2024-07-26T11:04:51.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/718: stat d9/d26/d48/d98/db8/le8 0 2024-07-26T11:04:51.677 INFO:tasks.workunit.client.0.smithi144.stdout:1/894: symlink d3/d6/d75/l120 0 2024-07-26T11:04:51.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/719: truncate f1 204960 0 2024-07-26T11:04:51.678 INFO:tasks.workunit.client.0.smithi144.stdout:4/720: truncate d9/d26/d48/d50/d60/f88 3145910 0 2024-07-26T11:04:51.679 INFO:tasks.workunit.client.0.smithi144.stdout:4/721: creat d9/da/da1/ff1 x:0 0 0 2024-07-26T11:04:51.680 INFO:tasks.workunit.client.0.smithi144.stdout:1/895: creat d3/d6/da/f121 x:0 0 0 2024-07-26T11:04:51.680 INFO:tasks.workunit.client.0.smithi144.stdout:1/896: chown d3/d3d/d58/d101/d36/db6/le5 83624157 1 2024-07-26T11:04:51.680 INFO:tasks.workunit.client.0.smithi144.stdout:1/897: chown d3/d6/da/d13/d19/d3a/fef 11199 1 2024-07-26T11:04:51.680 INFO:tasks.workunit.client.0.smithi144.stdout:1/898: chown d3/d6/d75/d88/da9/dac/d104/ddb/c107 3725054 1 2024-07-26T11:04:51.681 INFO:tasks.workunit.client.0.smithi144.stdout:4/722: creat d9/d57/ff2 x:0 0 0 2024-07-26T11:04:51.685 INFO:tasks.workunit.client.0.smithi144.stdout:1/899: truncate d3/d6/f102 3688101 0 2024-07-26T11:04:51.685 INFO:tasks.workunit.client.0.smithi144.stdout:1/900: stat d3/d6/da/d13/c14 0 2024-07-26T11:04:51.688 INFO:tasks.workunit.client.0.smithi144.stdout:1/901: rmdir d3/d3d/d58/d101/db7 39 2024-07-26T11:04:51.692 INFO:tasks.workunit.client.0.smithi144.stdout:0/685: write d0/d25/d5e/d1e/d91/dd1/fbb [1189504,8316] 0 2024-07-26T11:04:51.695 INFO:tasks.workunit.client.0.smithi144.stdout:1/902: read d3/d6/dcc/fd9 [120150,86155] 0 2024-07-26T11:04:51.695 INFO:tasks.workunit.client.0.smithi144.stdout:0/686: unlink d0/d3e/d45/c53 0 2024-07-26T11:04:51.698 INFO:tasks.workunit.client.0.smithi144.stdout:3/483: dread d1/d5/f1c [0,4194304] 0 2024-07-26T11:04:51.698 INFO:tasks.workunit.client.0.smithi144.stdout:3/484: chown d1/d28/d54/d3d/d42/d44 680332 1 2024-07-26T11:04:51.700 INFO:tasks.workunit.client.0.smithi144.stdout:3/485: rename d1/l16 to d1/l8e 0 2024-07-26T11:04:51.701 INFO:tasks.workunit.client.0.smithi144.stdout:6/816: dread d8/db/dbe/d52/f8d [0,4194304] 0 2024-07-26T11:04:51.701 INFO:tasks.workunit.client.0.smithi144.stdout:6/817: dread - d8/d42/dad/ff1 zero size 2024-07-26T11:04:51.702 INFO:tasks.workunit.client.0.smithi144.stdout:3/486: truncate d1/f1b 219367 0 2024-07-26T11:04:51.710 INFO:tasks.workunit.client.0.smithi144.stdout:3/487: mknod d1/d28/d54/d3d/c8f 0 2024-07-26T11:04:51.724 INFO:tasks.workunit.client.0.smithi144.stdout:3/488: mknod d1/d28/d54/d61/c90 0 2024-07-26T11:04:51.724 INFO:tasks.workunit.client.0.smithi144.stdout:3/489: chown d1/d28/d54/d3d/d42/d44/d60 0 1 2024-07-26T11:04:51.734 INFO:tasks.workunit.client.0.smithi144.stdout:5/715: dwrite d1/d9/d44/d45/fc1 [0,4194304] 0 2024-07-26T11:04:51.734 INFO:tasks.workunit.client.0.smithi144.stdout:0/687: dread d0/d3e/d45/d9f/fa8 [0,4194304] 0 2024-07-26T11:04:51.737 INFO:tasks.workunit.client.0.smithi144.stdout:5/716: rename d1/d18/d24/d35/f9c to d1/d18/dc4/de0/ff5 0 2024-07-26T11:04:51.740 INFO:tasks.workunit.client.0.smithi144.stdout:0/688: rename d0/d3e/d45/d4c/l88 to d0/d25/d5e/d1e/dd2/le8 0 2024-07-26T11:04:51.740 INFO:tasks.workunit.client.0.smithi144.stdout:0/689: dread - d0/d25/d5e/d1e/d30/d67/d84/fbd zero size 2024-07-26T11:04:51.741 INFO:tasks.workunit.client.0.smithi144.stdout:5/717: mknod d1/d9/d44/da1/ddb/cf6 0 2024-07-26T11:04:51.744 INFO:tasks.workunit.client.0.smithi144.stdout:5/718: mkdir d1/d9/d44/da1/ddb/df7 0 2024-07-26T11:04:51.746 INFO:tasks.workunit.client.0.smithi144.stdout:5/719: mknod d1/d5/dae/cf8 0 2024-07-26T11:04:51.753 INFO:tasks.workunit.client.0.smithi144.stdout:1/903: dwrite d3/d6/d75/d88/da9/dac/d104/ff0 [0,4194304] 0 2024-07-26T11:04:51.754 INFO:tasks.workunit.client.0.smithi144.stdout:4/723: dread d9/d7b/dc4/fc7 [0,4194304] 0 2024-07-26T11:04:51.756 INFO:tasks.workunit.client.0.smithi144.stdout:4/724: mkdir d9/d26/d3c/df3 0 2024-07-26T11:04:51.756 INFO:tasks.workunit.client.0.smithi144.stdout:4/725: chown d9/f51 98202 1 2024-07-26T11:04:51.761 INFO:tasks.workunit.client.0.smithi144.stdout:6/818: dwrite d8/d50/de2/ff7 [0,4194304] 0 2024-07-26T11:04:51.765 INFO:tasks.workunit.client.0.smithi144.stdout:6/819: mkdir d8/d50/d92/db3/d103/d125 0 2024-07-26T11:04:51.769 INFO:tasks.workunit.client.0.smithi144.stdout:3/490: dwrite d1/d28/d54/f85 [0,4194304] 0 2024-07-26T11:04:51.772 INFO:tasks.workunit.client.0.smithi144.stdout:4/726: rename d9/d26/fb6 to d9/d26/d3c/d6f/d8c/ff4 0 2024-07-26T11:04:51.774 INFO:tasks.workunit.client.0.smithi144.stdout:3/491: mkdir d1/d28/d54/d26/d37/d91 0 2024-07-26T11:04:51.775 INFO:tasks.workunit.client.0.smithi144.stdout:4/727: creat d9/d26/d3c/ff5 x:0 0 0 2024-07-26T11:04:51.778 INFO:tasks.workunit.client.0.smithi144.stdout:3/492: rename d1/d28/d54/c2a to d1/d28/d54/d26/d64/d89/c92 0 2024-07-26T11:04:51.779 INFO:tasks.workunit.client.0.smithi144.stdout:4/728: read d9/d26/f97 [1037625,114163] 0 2024-07-26T11:04:51.783 INFO:tasks.workunit.client.0.smithi144.stdout:3/493: symlink d1/d28/d54/d3d/d42/d44/d60/l93 0 2024-07-26T11:04:51.783 INFO:tasks.workunit.client.0.smithi144.stdout:3/494: chown d1/d28/d54/f85 472240 1 2024-07-26T11:04:51.783 INFO:tasks.workunit.client.0.smithi144.stdout:3/495: chown d1/d28/d54/d3d/d42/d44/d60/c82 6625 1 2024-07-26T11:04:51.783 INFO:tasks.workunit.client.0.smithi144.stdout:3/496: chown d1/d28/d54/l45 557 1 2024-07-26T11:04:51.786 INFO:tasks.workunit.client.0.smithi144.stdout:0/690: dwrite d0/d25/d5e/d1e/f21 [4194304,4194304] 0 2024-07-26T11:04:51.789 INFO:tasks.workunit.client.0.smithi144.stdout:4/729: unlink d9/d26/d48/d50/d60/dc5/fee 0 2024-07-26T11:04:51.789 INFO:tasks.workunit.client.0.smithi144.stdout:5/720: dwrite d1/d18/d24/d27/fcb [0,4194304] 0 2024-07-26T11:04:51.794 INFO:tasks.workunit.client.0.smithi144.stdout:1/904: dwrite d3/d6/da/d21/fc1 [0,4194304] 0 2024-07-26T11:04:51.795 INFO:tasks.workunit.client.0.smithi144.stdout:6/820: dread d8/d50/d98/f9f [0,4194304] 0 2024-07-26T11:04:51.796 INFO:tasks.workunit.client.0.smithi144.stdout:1/905: mkdir d3/d6/da/d13/d122 0 2024-07-26T11:04:51.799 INFO:tasks.workunit.client.0.smithi144.stdout:4/730: rmdir d9/d26/d3c/df3 0 2024-07-26T11:04:51.799 INFO:tasks.workunit.client.0.smithi144.stdout:4/731: write d9/da/fea [962646,103676] 0 2024-07-26T11:04:51.800 INFO:tasks.workunit.client.0.smithi144.stdout:3/497: rename d1/d28/d54/d3d/d42/d44 to d1/d28/d54/d26/d37/d91/d94 0 2024-07-26T11:04:51.800 INFO:tasks.workunit.client.0.smithi144.stdout:5/721: truncate d1/d9/d5b/d4d/f63 3825148 0 2024-07-26T11:04:51.805 INFO:tasks.workunit.client.0.smithi144.stdout:3/498: link d1/d5/fa d1/f95 0 2024-07-26T11:04:51.818 INFO:tasks.workunit.client.0.smithi144.stdout:3/499: truncate d1/d28/d54/d3d/f40 538809 0 2024-07-26T11:04:51.820 INFO:tasks.workunit.client.0.smithi144.stdout:1/906: dread d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f81 [0,4194304] 0 2024-07-26T11:04:51.825 INFO:tasks.workunit.client.0.smithi144.stdout:1/907: rename d3/d6/da/d78/f7e to d3/d6/da/d21/d5c/d62/d68/f123 0 2024-07-26T11:04:51.828 INFO:tasks.workunit.client.0.smithi144.stdout:1/908: mknod d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/c124 0 2024-07-26T11:04:51.835 INFO:tasks.workunit.client.0.smithi144.stdout:1/909: mknod d3/d3d/d58/d101/d36/d37/c125 0 2024-07-26T11:04:51.835 INFO:tasks.workunit.client.0.smithi144.stdout:1/910: truncate d3/d3d/fe9 180306 0 2024-07-26T11:04:51.837 INFO:tasks.workunit.client.0.smithi144.stdout:1/911: link d3/d6/da/d21/d5c/d62/d68/f116 d3/d6/d75/d88/da9/dac/d104/f126 0 2024-07-26T11:04:51.840 INFO:tasks.workunit.client.0.smithi144.stdout:0/691: dwrite d0/f27 [0,4194304] 0 2024-07-26T11:04:51.845 INFO:tasks.workunit.client.0.smithi144.stdout:3/500: dread d1/d28/d54/d26/f2e [4194304,4194304] 0 2024-07-26T11:04:51.845 INFO:tasks.workunit.client.0.smithi144.stdout:4/732: dwrite d9/d26/d3c/d6f/f85 [0,4194304] 0 2024-07-26T11:04:51.845 INFO:tasks.workunit.client.0.smithi144.stdout:4/733: read - d9/d57/feb zero size 2024-07-26T11:04:51.846 INFO:tasks.workunit.client.0.smithi144.stdout:3/501: rmdir d1/d28/d54/d26/d37/d91/d94/d48 39 2024-07-26T11:04:51.847 INFO:tasks.workunit.client.0.smithi144.stdout:3/502: chown d1/d28/d54/d26/f47 7946 1 2024-07-26T11:04:51.848 INFO:tasks.workunit.client.0.smithi144.stdout:4/734: rename d9/d26/d3c/c54 to d9/d26/d48/d98/db8/cf6 0 2024-07-26T11:04:51.851 INFO:tasks.workunit.client.0.smithi144.stdout:3/503: mknod d1/d28/d54/d26/d64/d68/c96 0 2024-07-26T11:04:51.851 INFO:tasks.workunit.client.0.smithi144.stdout:4/735: symlink d9/d26/d3c/d6f/dd0/lf7 0 2024-07-26T11:04:51.852 INFO:tasks.workunit.client.0.smithi144.stdout:3/504: rename d1/d28/d54/d26/d37/d91/d94 to d1/d5/d97 0 2024-07-26T11:04:51.854 INFO:tasks.workunit.client.0.smithi144.stdout:4/736: creat d9/da/ff8 x:0 0 0 2024-07-26T11:04:51.856 INFO:tasks.workunit.client.0.smithi144.stdout:4/737: getdents d9/d26/d48/d50 0 2024-07-26T11:04:51.864 INFO:tasks.workunit.client.0.smithi144.stdout:0/692: dread d0/d3e/f43 [0,4194304] 0 2024-07-26T11:04:51.864 INFO:tasks.workunit.client.0.smithi144.stdout:0/693: chown d0/d3e/d45/f9b 5 1 2024-07-26T11:04:51.866 INFO:tasks.workunit.client.0.smithi144.stdout:6/821: dwrite d8/d15/d57/d5f/fc3 [0,4194304] 0 2024-07-26T11:04:51.889 INFO:tasks.workunit.client.0.smithi144.stdout:0/694: dread d0/d25/d5e/d1e/d30/dab/fd5 [0,4194304] 0 2024-07-26T11:04:51.892 INFO:tasks.workunit.client.0.smithi144.stdout:3/505: write d1/f1a [1616899,51230] 0 2024-07-26T11:04:51.894 INFO:tasks.workunit.client.0.smithi144.stdout:0/695: link d0/d25/d5e/d1e/d30/d67/c7a d0/d25/d5e/d1e/dd2/ce9 0 2024-07-26T11:04:51.895 INFO:tasks.workunit.client.0.smithi144.stdout:4/738: dread d9/d26/d3c/d6f/f86 [0,4194304] 0 2024-07-26T11:04:51.901 INFO:tasks.workunit.client.0.smithi144.stdout:0/696: write d0/d25/f7b [429606,23679] 0 2024-07-26T11:04:51.902 INFO:tasks.workunit.client.0.smithi144.stdout:0/697: symlink d0/d3e/d45/dd4/de0/lea 0 2024-07-26T11:04:51.903 INFO:tasks.workunit.client.0.smithi144.stdout:0/698: creat d0/d25/d5e/d1e/dc1/feb x:0 0 0 2024-07-26T11:04:51.906 INFO:tasks.workunit.client.0.smithi144.stdout:6/822: dwrite d8/db/da6/fa3 [0,4194304] 0 2024-07-26T11:04:51.906 INFO:tasks.workunit.client.0.smithi144.stdout:6/823: fsync d8/d15/d57/d5f/f10d 0 2024-07-26T11:04:51.912 INFO:tasks.workunit.client.0.smithi144.stdout:6/824: link d8/db/c4d d8/d50/c126 0 2024-07-26T11:04:51.922 INFO:tasks.workunit.client.0.smithi144.stdout:3/506: dread d1/d5/f6 [0,4194304] 0 2024-07-26T11:04:51.929 INFO:tasks.workunit.client.0.smithi144.stdout:4/739: dread d9/f63 [0,4194304] 0 2024-07-26T11:04:51.931 INFO:tasks.workunit.client.0.smithi144.stdout:4/740: getdents d9/d26/d48/d50/d60/dc5/dd9 0 2024-07-26T11:04:51.932 INFO:tasks.workunit.client.0.smithi144.stdout:4/741: dread - d9/d26/fab zero size 2024-07-26T11:04:51.937 INFO:tasks.workunit.client.0.smithi144.stdout:4/742: symlink d9/d26/d3c/lf9 0 2024-07-26T11:04:51.942 INFO:tasks.workunit.client.0.smithi144.stdout:0/699: write d0/f27 [1757122,122704] 0 2024-07-26T11:04:51.942 INFO:tasks.workunit.client.0.smithi144.stdout:0/700: write d0/f18 [5156875,43678] 0 2024-07-26T11:04:51.943 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:04:51.944 INFO:tasks.workunit.client.0.smithi144.stdout:0/701: mkdir d0/d25/d5e/d1e/d30/d6a/d6f/dec 0 2024-07-26T11:04:51.947 INFO:tasks.workunit.client.0.smithi144.stdout:0/702: rename d0/d25/d5e/d1e/f36 to d0/d25/d5e/d1e/d91/fed 0 2024-07-26T11:04:51.947 INFO:tasks.workunit.client.0.smithi144.stdout:0/703: stat d0/l11 0 2024-07-26T11:04:51.947 INFO:tasks.workunit.client.0.smithi144.stdout:0/704: truncate d0/d3e/d45/f55 1016668 0 2024-07-26T11:04:51.954 INFO:tasks.workunit.client.0.smithi144.stdout:0/705: mknod d0/d3e/d5f/cee 0 2024-07-26T11:04:51.965 INFO:tasks.workunit.client.0.smithi144.stdout:3/507: dwrite d1/d28/d54/d26/f2e [0,4194304] 0 2024-07-26T11:04:51.971 INFO:tasks.workunit.client.0.smithi144.stdout:6/825: dwrite d8/d42/dad/fd5 [0,4194304] 0 2024-07-26T11:04:51.973 INFO:tasks.workunit.client.0.smithi144.stdout:6/826: getdents d8/d50/d92/db3/d103 0 2024-07-26T11:04:51.975 INFO:tasks.workunit.client.0.smithi144.stdout:6/827: truncate d8/db/dbe/d5c/f7b 300798 0 2024-07-26T11:04:51.980 INFO:tasks.workunit.client.0.smithi144.stdout:0/706: dread d0/d3e/d45/f4a [0,4194304] 0 2024-07-26T11:04:51.981 INFO:tasks.workunit.client.0.smithi144.stdout:0/707: mkdir d0/d25/d5e/def 0 2024-07-26T11:04:52.003 INFO:tasks.workunit.client.0.smithi144.stdout:6/828: write d8/d50/d98/f9f [3147515,81590] 0 2024-07-26T11:04:52.006 INFO:tasks.workunit.client.0.smithi144.stdout:6/829: creat d8/d50/f127 x:0 0 0 2024-07-26T11:04:52.008 INFO:tasks.workunit.client.0.smithi144.stdout:6/830: mkdir d8/d50/d92/db3/d103/d128 0 2024-07-26T11:04:52.009 INFO:tasks.workunit.client.0.smithi144.stdout:6/831: getdents d8/d50/d92/db3 0 2024-07-26T11:04:52.009 INFO:tasks.workunit.client.0.smithi144.stdout:6/832: getdents d8/d50/d92 0 2024-07-26T11:04:52.010 INFO:tasks.workunit.client.0.smithi144.stdout:6/833: write d8/db/dbe/d74/fb5 [114853,118309] 0 2024-07-26T11:04:52.011 INFO:tasks.workunit.client.0.smithi144.stdout:6/834: creat d8/d50/d92/db3/d103/d125/f129 x:0 0 0 2024-07-26T11:04:52.011 INFO:tasks.workunit.client.0.smithi144.stdout:6/835: dread - d8/d15/d57/d5f/fe5 zero size 2024-07-26T11:04:52.012 INFO:tasks.workunit.client.0.smithi144.stdout:3/508: dwrite d1/f50 [0,4194304] 0 2024-07-26T11:04:52.012 INFO:tasks.workunit.client.0.smithi144.stdout:0/708: dread d0/d25/d5e/f2c [0,4194304] 0 2024-07-26T11:04:52.013 INFO:tasks.workunit.client.0.smithi144.stdout:3/509: mknod d1/d28/d54/d26/d30/d78/d80/c98 0 2024-07-26T11:04:52.014 INFO:tasks.workunit.client.0.smithi144.stdout:0/709: unlink d0/d3e/d45/d9f/l31 0 2024-07-26T11:04:52.015 INFO:tasks.workunit.client.0.smithi144.stdout:0/710: rename d0/d3e/f71 to d0/d25/d5e/d1e/d30/dab/ff0 0 2024-07-26T11:04:52.021 INFO:tasks.workunit.client.0.smithi144.stdout:6/836: mkdir d8/d50/d92/db3/d103/d12a 0 2024-07-26T11:04:52.021 INFO:tasks.workunit.client.0.smithi144.stdout:6/837: stat d8/d50/d92/db3 0 2024-07-26T11:04:52.021 INFO:tasks.workunit.client.0.smithi144.stdout:3/510: write d1/d28/d54/d26/d64/d68/f7d [1200944,87344] 0 2024-07-26T11:04:52.021 INFO:tasks.workunit.client.0.smithi144.stdout:3/511: chown d1/d5/l21 22484 1 2024-07-26T11:04:52.022 INFO:tasks.workunit.client.0.smithi144.stdout:6/838: creat d8/db/dbe/d52/f12b x:0 0 0 2024-07-26T11:04:52.038 INFO:tasks.workunit.client.0.smithi144.stdout:0/711: dread d0/d3e/f43 [0,4194304] 0 2024-07-26T11:04:52.041 INFO:tasks.workunit.client.0.smithi144.stdout:0/712: creat d0/d25/d5e/d1e/d30/d6a/de4/ff1 x:0 0 0 2024-07-26T11:04:52.042 INFO:tasks.workunit.client.0.smithi144.stdout:0/713: write d0/d25/d5e/d1e/d30/d67/d84/fbd [80151,67075] 0 2024-07-26T11:04:52.044 INFO:tasks.workunit.client.0.smithi144.stdout:3/512: dread d1/f1a [0,4194304] 0 2024-07-26T11:04:52.044 INFO:tasks.workunit.client.0.smithi144.stdout:0/714: getdents d0/d25/d5e/d1e/dd2 0 2024-07-26T11:04:52.046 INFO:tasks.workunit.client.0.smithi144.stdout:3/513: link d1/d28/d54/d26/d30/d78/d80/c98 d1/d28/d54/d26/d37/c99 0 2024-07-26T11:04:52.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/715: symlink d0/d25/d5e/d1e/d30/d6a/d6f/lf2 0 2024-07-26T11:04:52.048 INFO:tasks.workunit.client.0.smithi144.stdout:0/716: fsync d0/d25/d5e/d1e/d30/dab/ff0 0 2024-07-26T11:04:52.048 INFO:tasks.workunit.client.0.smithi144.stdout:3/514: creat d1/d28/d54/d3d/f9a x:0 0 0 2024-07-26T11:04:52.050 INFO:tasks.workunit.client.0.smithi144.stdout:3/515: creat d1/d5/f9b x:0 0 0 2024-07-26T11:04:52.052 INFO:tasks.workunit.client.0.smithi144.stdout:3/516: rmdir d1/d28/d54/d26/d30/d4d/d6e 0 2024-07-26T11:04:52.066 INFO:tasks.workunit.client.0.smithi144.stdout:6/839: dwrite d8/f33 [4194304,4194304] 0 2024-07-26T11:04:52.067 INFO:tasks.workunit.client.0.smithi144.stdout:6/840: truncate d8/d15/d26/f3f 2090782 0 2024-07-26T11:04:52.068 INFO:tasks.workunit.client.0.smithi144.stdout:0/717: write d0/f18 [1781251,12541] 0 2024-07-26T11:04:52.068 INFO:tasks.workunit.client.0.smithi144.stdout:0/718: chown d0/d3e/d5f/caf 1708700 1 2024-07-26T11:04:52.069 INFO:tasks.workunit.client.0.smithi144.stdout:6/841: creat d8/db/dbe/d52/ddb/de3/f12c x:0 0 0 2024-07-26T11:04:52.069 INFO:tasks.workunit.client.0.smithi144.stdout:6/842: write d8/d42/dad/fd2 [882107,43428] 0 2024-07-26T11:04:52.079 INFO:tasks.workunit.client.0.smithi144.stdout:3/517: dread d1/f4f [0,4194304] 0 2024-07-26T11:04:52.080 INFO:tasks.workunit.client.0.smithi144.stdout:3/518: link d1/d5/f1c d1/d28/d54/d26/d37/d91/f9c 0 2024-07-26T11:04:52.089 INFO:tasks.workunit.client.0.smithi144.stdout:6/843: dread d8/f2d [0,4194304] 0 2024-07-26T11:04:52.099 INFO:tasks.workunit.client.0.smithi144.stdout:7/694: sync 2024-07-26T11:04:52.099 INFO:tasks.workunit.client.0.smithi144.stdout:2/718: sync 2024-07-26T11:04:52.106 INFO:tasks.workunit.client.0.smithi144.stdout:2/719: mknod d0/d24/d7b/ddc/d28/d27/d8b/ce0 0 2024-07-26T11:04:52.106 INFO:tasks.workunit.client.0.smithi144.stdout:7/695: mknod d0/d6/d9/d3f/dd3/cfa 0 2024-07-26T11:04:52.113 INFO:tasks.workunit.client.0.smithi144.stdout:7/696: dread d0/dd0/dbf/d2f/d53/fa8 [0,4194304] 0 2024-07-26T11:04:52.116 INFO:tasks.workunit.client.0.smithi144.stdout:7/697: mknod d0/dd0/dbf/d2f/cfb 0 2024-07-26T11:04:52.116 INFO:tasks.workunit.client.0.smithi144.stdout:7/698: write d0/dd0/dbf/d2f/d53/db6/fe6 [356620,81259] 0 2024-07-26T11:04:52.116 INFO:tasks.workunit.client.0.smithi144.stdout:7/699: chown d0/d6/d9/d45/d57/lab 0 1 2024-07-26T11:04:52.117 INFO:tasks.workunit.client.0.smithi144.stdout:6/844: dread d8/db/dbe/f7e [0,4194304] 0 2024-07-26T11:04:52.117 INFO:tasks.workunit.client.0.smithi144.stdout:7/700: getdents d0/d6 0 2024-07-26T11:04:52.117 INFO:tasks.workunit.client.0.smithi144.stdout:7/701: readlink d0/dd0/dbf/d2f/d47/l5b 0 2024-07-26T11:04:52.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/845: creat d8/d50/d98/d104/f12d x:0 0 0 2024-07-26T11:04:52.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/846: chown d8/d42/dad/fda 70501 1 2024-07-26T11:04:52.120 INFO:tasks.workunit.client.0.smithi144.stdout:6/847: creat d8/d15/d57/d5f/d6e/f12e x:0 0 0 2024-07-26T11:04:52.131 INFO:tasks.workunit.client.0.smithi144.stdout:0/719: dwrite d0/d25/d5e/d1e/d30/dab/fc0 [0,4194304] 0 2024-07-26T11:04:52.131 INFO:tasks.workunit.client.0.smithi144.stdout:0/720: chown d0/d3e/d45/f9b 14554782 1 2024-07-26T11:04:52.133 INFO:tasks.workunit.client.0.smithi144.stdout:0/721: mkdir d0/d25/d5e/d1e/d30/dab/df3 0 2024-07-26T11:04:52.133 INFO:tasks.workunit.client.0.smithi144.stdout:0/722: write d0/d25/d5e/d1e/dc1/feb [856713,68625] 0 2024-07-26T11:04:52.133 INFO:tasks.workunit.client.0.smithi144.stdout:0/723: stat d0/d25/d5e/d1e/d30/d67 0 2024-07-26T11:04:52.161 INFO:tasks.workunit.client.0.smithi144.stdout:0/724: read d0/f7 [2857855,107373] 0 2024-07-26T11:04:52.161 INFO:tasks.workunit.client.0.smithi144.stdout:0/725: fdatasync d0/d3e/d45/d4c/f64 0 2024-07-26T11:04:52.161 INFO:tasks.workunit.client.0.smithi144.stdout:2/720: dwrite d0/d24/d7b/ddc/f35 [0,4194304] 0 2024-07-26T11:04:52.162 INFO:tasks.workunit.client.0.smithi144.stdout:0/726: mkdir d0/d3e/d45/d9f/df4 0 2024-07-26T11:04:52.163 INFO:tasks.workunit.client.0.smithi144.stdout:2/721: creat d0/d24/d7b/ddc/d28/d27/d8b/fe1 x:0 0 0 2024-07-26T11:04:52.164 INFO:tasks.workunit.client.0.smithi144.stdout:0/727: rename d0/d25/d5e/d1e/d6c/l8c to d0/d25/d5e/d1e/d30/d4b/lf5 0 2024-07-26T11:04:52.164 INFO:tasks.workunit.client.0.smithi144.stdout:0/728: chown d0/d25/d5e/d1e/d91/f96 383995 1 2024-07-26T11:04:52.164 INFO:tasks.workunit.client.0.smithi144.stdout:2/722: mknod d0/d6/d9/d44/ce2 0 2024-07-26T11:04:52.165 INFO:tasks.workunit.client.0.smithi144.stdout:0/729: symlink d0/d3e/db8/lf6 0 2024-07-26T11:04:52.169 INFO:tasks.workunit.client.0.smithi144.stdout:6/848: dwrite f2 [0,4194304] 0 2024-07-26T11:04:52.169 INFO:tasks.workunit.client.0.smithi144.stdout:3/519: dwrite d1/d28/d54/d26/f2b [0,4194304] 0 2024-07-26T11:04:52.171 INFO:tasks.workunit.client.0.smithi144.stdout:6/849: getdents d8/db/da6 0 2024-07-26T11:04:52.178 INFO:tasks.workunit.client.0.smithi144.stdout:2/723: dread d0/d24/d7b/ddc/d5f/f85 [0,4194304] 0 2024-07-26T11:04:52.178 INFO:tasks.workunit.client.0.smithi144.stdout:2/724: chown d0/d24/d7b/ddc 104891 1 2024-07-26T11:04:52.178 INFO:tasks.workunit.client.0.smithi144.stdout:9/749: sync 2024-07-26T11:04:52.179 INFO:tasks.workunit.client.0.smithi144.stdout:2/725: mkdir d0/d24/d7b/ddc/d28/d2e/d77/d84/de3 0 2024-07-26T11:04:52.179 INFO:tasks.workunit.client.0.smithi144.stdout:2/726: write d0/d24/d48/fce [1246879,47548] 0 2024-07-26T11:04:52.179 INFO:tasks.workunit.client.0.smithi144.stdout:9/750: mkdir d9/de/d4e/d6a/deb/dfd 0 2024-07-26T11:04:52.179 INFO:tasks.workunit.client.0.smithi144.stdout:2/727: read d0/d24/d7b/f87 [654575,101674] 0 2024-07-26T11:04:52.179 INFO:tasks.workunit.client.0.smithi144.stdout:2/728: write d0/d6/d34/d6c/f75 [779126,56420] 0 2024-07-26T11:04:52.222 INFO:tasks.workunit.client.0.smithi144.stdout:2/729: dwrite d0/f3d [4194304,4194304] 0 2024-07-26T11:04:52.224 INFO:tasks.workunit.client.0.smithi144.stdout:2/730: rmdir d0/d24/d7b/ddc/d28/d2e 39 2024-07-26T11:04:52.225 INFO:tasks.workunit.client.0.smithi144.stdout:2/731: mknod d0/d24/d7b/ddc/d28/d2e/d77/d84/de3/ce4 0 2024-07-26T11:04:52.227 INFO:tasks.workunit.client.0.smithi144.stdout:0/730: dwrite d0/d25/d5e/f2c [0,4194304] 0 2024-07-26T11:04:52.227 INFO:tasks.workunit.client.0.smithi144.stdout:0/731: stat d0/d25/d5e/d1e/d30/d6a/d6f/l7f 0 2024-07-26T11:04:52.233 INFO:tasks.workunit.client.0.smithi144.stdout:9/751: dwrite d9/d3b/d76/d9b/dd7/fe9 [0,4194304] 0 2024-07-26T11:04:52.234 INFO:tasks.workunit.client.0.smithi144.stdout:6/850: dwrite d8/d15/d57/d5f/f10a [0,4194304] 0 2024-07-26T11:04:52.235 INFO:tasks.workunit.client.0.smithi144.stdout:2/732: getdents d0/d24/d7b/ddc/d28/d2e/d77/d84/de3 0 2024-07-26T11:04:52.237 INFO:tasks.workunit.client.0.smithi144.stdout:2/733: mknod d0/d24/d48/d63/ce5 0 2024-07-26T11:04:52.237 INFO:tasks.workunit.client.0.smithi144.stdout:9/752: rename d9/de/d14/d23/l28 to d9/de/d4e/d6a/lfe 0 2024-07-26T11:04:52.239 INFO:tasks.workunit.client.0.smithi144.stdout:3/520: dwrite d1/f39 [0,4194304] 0 2024-07-26T11:04:52.239 INFO:tasks.workunit.client.0.smithi144.stdout:3/521: stat d1/d28/d54/d3d/d42/l72 0 2024-07-26T11:04:52.240 INFO:tasks.workunit.client.0.smithi144.stdout:6/851: mknod d8/d50/d92/db3/c12f 0 2024-07-26T11:04:52.241 INFO:tasks.workunit.client.0.smithi144.stdout:6/852: fsync f2 0 2024-07-26T11:04:52.241 INFO:tasks.workunit.client.0.smithi144.stdout:9/753: rename d9/de/d14/d79/db2/de2 to d9/de/d4e/dff 0 2024-07-26T11:04:52.248 INFO:tasks.workunit.client.0.smithi144.stdout:9/754: dread d9/d3b/d76/d9b/fb7 [0,4194304] 0 2024-07-26T11:04:52.249 INFO:tasks.workunit.client.0.smithi144.stdout:0/732: write d0/d25/d5e/d1e/d30/dab/ff0 [2474306,26112] 0 2024-07-26T11:04:52.249 INFO:tasks.workunit.client.0.smithi144.stdout:3/522: getdents d1/d5 0 2024-07-26T11:04:52.249 INFO:tasks.workunit.client.0.smithi144.stdout:3/523: readlink d1/d5/d97/d48/l88 0 2024-07-26T11:04:52.249 INFO:tasks.workunit.client.0.smithi144.stdout:9/755: rmdir d9/de/d14/d23 39 2024-07-26T11:04:52.250 INFO:tasks.workunit.client.0.smithi144.stdout:9/756: rmdir d9/d3b/d76 39 2024-07-26T11:04:52.250 INFO:tasks.workunit.client.0.smithi144.stdout:0/733: symlink d0/d25/d5e/d1e/d30/d6a/dac/lf7 0 2024-07-26T11:04:52.250 INFO:tasks.workunit.client.0.smithi144.stdout:3/524: creat d1/d28/f9d x:0 0 0 2024-07-26T11:04:52.251 INFO:tasks.workunit.client.0.smithi144.stdout:0/734: mknod d0/d25/d5e/cf8 0 2024-07-26T11:04:52.251 INFO:tasks.workunit.client.0.smithi144.stdout:0/735: truncate d0/d3e/d45/d9f/f82 1536867 0 2024-07-26T11:04:52.252 INFO:tasks.workunit.client.0.smithi144.stdout:3/525: creat d1/d5/f9e x:0 0 0 2024-07-26T11:04:52.253 INFO:tasks.workunit.client.0.smithi144.stdout:0/736: truncate d0/f10 2920377 0 2024-07-26T11:04:52.278 INFO:tasks.workunit.client.0.smithi144.stdout:0/737: write d0/d25/d5e/f41 [1087842,42580] 0 2024-07-26T11:04:52.286 INFO:tasks.workunit.client.0.smithi144.stdout:2/734: dread d0/d24/d2b/d68/d82/f7c [0,4194304] 0 2024-07-26T11:04:52.293 INFO:tasks.workunit.client.0.smithi144.stdout:6/853: dwrite d8/db/fa1 [4194304,4194304] 0 2024-07-26T11:04:52.294 INFO:tasks.workunit.client.0.smithi144.stdout:6/854: read d8/d50/d98/fc5 [379616,11004] 0 2024-07-26T11:04:52.297 INFO:tasks.workunit.client.0.smithi144.stdout:6/855: mkdir d8/db/dbe/d52/d58/d130 0 2024-07-26T11:04:52.305 INFO:tasks.workunit.client.0.smithi144.stdout:0/738: dread d0/f10 [0,4194304] 0 2024-07-26T11:04:52.308 INFO:tasks.workunit.client.0.smithi144.stdout:9/757: dwrite d9/d3b/d76/d9b/dcc/ff3 [0,4194304] 0 2024-07-26T11:04:52.308 INFO:tasks.workunit.client.0.smithi144.stdout:9/758: write d9/de/d14/d79/db2/fee [1147076,25533] 0 2024-07-26T11:04:52.308 INFO:tasks.workunit.client.0.smithi144.stdout:9/759: fsync d9/de/d14/d23/d39/fab 0 2024-07-26T11:04:52.308 INFO:tasks.workunit.client.0.smithi144.stdout:9/760: chown d9/d3b/d76/d9b/dd7/ff4 61542 1 2024-07-26T11:04:52.311 INFO:tasks.workunit.client.0.smithi144.stdout:9/761: unlink d9/de/d14/d3f/d5e/lc3 0 2024-07-26T11:04:52.311 INFO:tasks.workunit.client.0.smithi144.stdout:3/526: dread d1/f32 [0,4194304] 0 2024-07-26T11:04:52.312 INFO:tasks.workunit.client.0.smithi144.stdout:9/762: getdents d9/d3b/d5d 0 2024-07-26T11:04:52.312 INFO:tasks.workunit.client.0.smithi144.stdout:3/527: rmdir d1/d5 39 2024-07-26T11:04:52.315 INFO:tasks.workunit.client.0.smithi144.stdout:3/528: mkdir d1/d28/d54/d3d/d9f 0 2024-07-26T11:04:52.315 INFO:tasks.workunit.client.0.smithi144.stdout:9/763: creat d9/de/d14/d79/db2/f100 x:0 0 0 2024-07-26T11:04:52.315 INFO:tasks.workunit.client.0.smithi144.stdout:9/764: fsync d9/de/d14/d23/d39/f9f 0 2024-07-26T11:04:52.315 INFO:tasks.workunit.client.0.smithi144.stdout:9/765: dread - d9/fca zero size 2024-07-26T11:04:52.316 INFO:tasks.workunit.client.0.smithi144.stdout:3/529: link d1/d28/d54/f85 d1/d28/d54/d26/d30/d78/d80/fa0 0 2024-07-26T11:04:52.317 INFO:tasks.workunit.client.0.smithi144.stdout:3/530: mkdir d1/d28/d54/d26/d37/da1 0 2024-07-26T11:04:52.326 INFO:tasks.workunit.client.0.smithi144.stdout:9/766: dread d9/d3b/f78 [0,4194304] 0 2024-07-26T11:04:52.326 INFO:tasks.workunit.client.0.smithi144.stdout:9/767: chown d9/de/d14/d23/d39/fc1 354 1 2024-07-26T11:04:52.336 INFO:tasks.workunit.client.0.smithi144.stdout:3/531: dread d1/d5/f6 [0,4194304] 0 2024-07-26T11:04:52.337 INFO:tasks.workunit.client.0.smithi144.stdout:3/532: creat d1/d5/d97/fa2 x:0 0 0 2024-07-26T11:04:52.338 INFO:tasks.workunit.client.0.smithi144.stdout:3/533: creat d1/d28/d54/d26/d64/d68/fa3 x:0 0 0 2024-07-26T11:04:52.338 INFO:tasks.workunit.client.0.smithi144.stdout:3/534: fdatasync d1/f1a 0 2024-07-26T11:04:52.343 INFO:tasks.workunit.client.0.smithi144.stdout:3/535: read d1/d28/d54/d26/d37/d91/f9c [954789,17101] 0 2024-07-26T11:04:52.349 INFO:tasks.workunit.client.0.smithi144.stdout:2/735: dwrite d0/d24/d7b/ddc/d28/d2e/d5d/d6b/f73 [0,4194304] 0 2024-07-26T11:04:52.349 INFO:tasks.workunit.client.0.smithi144.stdout:2/736: dread - d0/d6/d9/d44/fa2 zero size 2024-07-26T11:04:52.349 INFO:tasks.workunit.client.0.smithi144.stdout:2/737: read - d0/d6/d34/fd2 zero size 2024-07-26T11:04:52.350 INFO:tasks.workunit.client.0.smithi144.stdout:2/738: write d0/d24/d7b/ddc/fa0 [1257196,29076] 0 2024-07-26T11:04:52.351 INFO:tasks.workunit.client.0.smithi144.stdout:2/739: creat d0/d24/d7b/ddc/d5f/fe6 x:0 0 0 2024-07-26T11:04:52.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/740: unlink d0/d24/d7b/ddc/d28/d2e/l78 0 2024-07-26T11:04:52.353 INFO:tasks.workunit.client.0.smithi144.stdout:2/741: truncate d0/d6/d9/f33 1589625 0 2024-07-26T11:04:52.370 INFO:tasks.workunit.client.0.smithi144.stdout:0/739: dwrite d0/d3e/d45/fae [0,4194304] 0 2024-07-26T11:04:52.374 INFO:tasks.workunit.client.0.smithi144.stdout:0/740: mknod d0/d25/d5e/d1e/cf9 0 2024-07-26T11:04:52.387 INFO:tasks.workunit.client.0.smithi144.stdout:2/742: read d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fc9 [448138,86943] 0 2024-07-26T11:04:52.387 INFO:tasks.workunit.client.0.smithi144.stdout:2/743: fdatasync d0/d24/d7b/ddc/d28/d2e/fb7 0 2024-07-26T11:04:52.387 INFO:tasks.workunit.client.0.smithi144.stdout:9/768: dwrite d9/de/d14/d23/d39/f66 [0,4194304] 0 2024-07-26T11:04:52.387 INFO:tasks.workunit.client.0.smithi144.stdout:9/769: stat d9/d3b/d5d/c68 0 2024-07-26T11:04:52.388 INFO:tasks.workunit.client.0.smithi144.stdout:9/770: fsync d9/de/d14/d79/f91 0 2024-07-26T11:04:52.388 INFO:tasks.workunit.client.0.smithi144.stdout:2/744: symlink d0/d6/d34/le7 0 2024-07-26T11:04:52.388 INFO:tasks.workunit.client.0.smithi144.stdout:2/745: truncate d0/d24/d7b/ddc/d28/d2e/d77/fb2 5129688 0 2024-07-26T11:04:52.390 INFO:tasks.workunit.client.0.smithi144.stdout:9/771: rename d9/de/d14/d23/c55 to d9/d3b/d76/d9b/dcc/c101 0 2024-07-26T11:04:52.439 INFO:tasks.workunit.client.0.smithi144.stdout:2/746: dwrite d0/d6/d34/d6c/f75 [0,4194304] 0 2024-07-26T11:04:52.439 INFO:tasks.workunit.client.0.smithi144.stdout:2/747: dread - d0/d24/d7b/ddc/fcc zero size 2024-07-26T11:04:52.441 INFO:tasks.workunit.client.0.smithi144.stdout:2/748: symlink d0/d24/d7b/ddc/d28/d2e/d5d/le8 0 2024-07-26T11:04:52.447 INFO:tasks.workunit.client.0.smithi144.stdout:9/772: dread d9/d3b/f64 [0,4194304] 0 2024-07-26T11:04:52.447 INFO:tasks.workunit.client.0.smithi144.stdout:0/741: dwrite d0/d25/d5e/d1e/d30/d4b/f58 [4194304,4194304] 0 2024-07-26T11:04:52.448 INFO:tasks.workunit.client.0.smithi144.stdout:0/742: rmdir d0/d25/d5e/d1e 39 2024-07-26T11:04:52.448 INFO:tasks.workunit.client.0.smithi144.stdout:9/773: creat d9/d3b/d76/d9b/dd7/f102 x:0 0 0 2024-07-26T11:04:52.449 INFO:tasks.workunit.client.0.smithi144.stdout:9/774: chown d9/de/f46 878556964 1 2024-07-26T11:04:52.449 INFO:tasks.workunit.client.0.smithi144.stdout:9/775: fsync d9/de/d14/d23/d39/fab 0 2024-07-26T11:04:52.449 INFO:tasks.workunit.client.0.smithi144.stdout:9/776: dread - d9/de/fbf zero size 2024-07-26T11:04:52.449 INFO:tasks.workunit.client.0.smithi144.stdout:0/743: symlink d0/d3e/d45/d4c/lfa 0 2024-07-26T11:04:52.450 INFO:tasks.workunit.client.0.smithi144.stdout:9/777: symlink d9/de/d14/d23/d39/d7f/l103 0 2024-07-26T11:04:52.451 INFO:tasks.workunit.client.0.smithi144.stdout:9/778: mknod d9/d3b/d5d/c104 0 2024-07-26T11:04:52.454 INFO:tasks.workunit.client.0.smithi144.stdout:3/536: dwrite d1/d5/d97/d60/f8c [0,4194304] 0 2024-07-26T11:04:52.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/779: symlink d9/de/d14/d3f/d5e/l105 0 2024-07-26T11:04:52.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/780: chown d9/de/d14/d23 61 1 2024-07-26T11:04:52.454 INFO:tasks.workunit.client.0.smithi144.stdout:9/781: truncate d9/de/d14/d23/fa3 290721 0 2024-07-26T11:04:52.455 INFO:tasks.workunit.client.0.smithi144.stdout:3/537: unlink d1/d5/d97/d48/l88 0 2024-07-26T11:04:52.456 INFO:tasks.workunit.client.0.smithi144.stdout:3/538: mkdir d1/d28/d54/d26/d64/da4 0 2024-07-26T11:04:52.457 INFO:tasks.workunit.client.0.smithi144.stdout:3/539: creat d1/d28/d54/d26/d30/d78/fa5 x:0 0 0 2024-07-26T11:04:52.458 INFO:tasks.workunit.client.0.smithi144.stdout:3/540: mknod d1/d28/d54/d26/d30/d79/ca6 0 2024-07-26T11:04:52.459 INFO:tasks.workunit.client.0.smithi144.stdout:2/749: write d0/d6/f1d [566563,25144] 0 2024-07-26T11:04:52.482 INFO:tasks.workunit.client.0.smithi144.stdout:2/750: write d0/d24/d7b/ddc/f35 [3226528,22173] 0 2024-07-26T11:04:52.482 INFO:tasks.workunit.client.0.smithi144.stdout:2/751: chown d0/d6 20833 1 2024-07-26T11:04:52.489 INFO:tasks.workunit.client.0.smithi144.stdout:4/743: sync 2024-07-26T11:04:52.489 INFO:tasks.workunit.client.0.smithi144.stdout:8/700: sync 2024-07-26T11:04:52.489 INFO:tasks.workunit.client.0.smithi144.stdout:1/912: sync 2024-07-26T11:04:52.490 INFO:tasks.workunit.client.0.smithi144.stdout:6/856: sync 2024-07-26T11:04:52.490 INFO:tasks.workunit.client.0.smithi144.stdout:5/722: sync 2024-07-26T11:04:52.490 INFO:tasks.workunit.client.0.smithi144.stdout:7/702: sync 2024-07-26T11:04:52.491 INFO:tasks.workunit.client.0.smithi144.stdout:4/744: getdents d9/da/da1 0 2024-07-26T11:04:52.493 INFO:tasks.workunit.client.0.smithi144.stdout:1/913: link d3/d6/dc0/lc8 d3/d6/d75/d88/l127 0 2024-07-26T11:04:52.498 INFO:tasks.workunit.client.0.smithi144.stdout:1/914: mkdir d3/d3d/d128 0 2024-07-26T11:04:52.498 INFO:tasks.workunit.client.0.smithi144.stdout:7/703: mkdir d0/dd0/dbf/d2f/d53/db6/dfc 0 2024-07-26T11:04:52.498 INFO:tasks.workunit.client.0.smithi144.stdout:5/723: mkdir d1/d9/d44/d85/dea/df9 0 2024-07-26T11:04:52.498 INFO:tasks.workunit.client.0.smithi144.stdout:4/745: mknod d9/d7b/dc4/dec/cfa 0 2024-07-26T11:04:52.498 INFO:tasks.workunit.client.0.smithi144.stdout:5/724: truncate d1/d9/d5b/f21 3639248 0 2024-07-26T11:04:52.499 INFO:tasks.workunit.client.0.smithi144.stdout:3/541: dread d1/f34 [0,4194304] 0 2024-07-26T11:04:52.499 INFO:tasks.workunit.client.0.smithi144.stdout:3/542: fdatasync d1/d28/d54/d26/d30/f5d 0 2024-07-26T11:04:52.499 INFO:tasks.workunit.client.0.smithi144.stdout:7/704: symlink d0/dd0/dbf/d67/lfd 0 2024-07-26T11:04:52.506 INFO:tasks.workunit.client.0.smithi144.stdout:8/701: dread d1a/d32/d49/d73/d7b/f40 [0,4194304] 0 2024-07-26T11:04:52.512 INFO:tasks.workunit.client.0.smithi144.stdout:9/782: dwrite d9/de/fd8 [0,4194304] 0 2024-07-26T11:04:52.512 INFO:tasks.workunit.client.0.smithi144.stdout:9/783: chown d9/de/d14/d23/d39/d7f/cf0 512 1 2024-07-26T11:04:52.512 INFO:tasks.workunit.client.0.smithi144.stdout:8/702: rmdir d1a/d32/d49/d73/d2c/d38/d58/da3/dd7 0 2024-07-26T11:04:52.512 INFO:tasks.workunit.client.0.smithi144.stdout:8/703: fsync d1a/d32/d49/d73/f3f 0 2024-07-26T11:04:52.516 INFO:tasks.workunit.client.0.smithi144.stdout:3/543: dread d1/d5/d97/f4e [0,4194304] 0 2024-07-26T11:04:52.517 INFO:tasks.workunit.client.0.smithi144.stdout:3/544: write d1/d28/f9d [811534,82353] 0 2024-07-26T11:04:52.517 INFO:tasks.workunit.client.0.smithi144.stdout:9/784: mkdir d9/de/d4e/d106 0 2024-07-26T11:04:52.519 INFO:tasks.workunit.client.0.smithi144.stdout:9/785: mknod d9/de/d14/c107 0 2024-07-26T11:04:52.528 INFO:tasks.workunit.client.0.smithi144.stdout:8/704: getdents d1a/d32/d66 0 2024-07-26T11:04:52.528 INFO:tasks.workunit.client.0.smithi144.stdout:8/705: fsync d1a/d32/d49/d73/d2c/f8a 0 2024-07-26T11:04:52.528 INFO:tasks.workunit.client.0.smithi144.stdout:3/545: rmdir d1/d28/d54/d3d/d9f 0 2024-07-26T11:04:52.529 INFO:tasks.workunit.client.0.smithi144.stdout:8/706: creat d1a/d32/d66/ff0 x:0 0 0 2024-07-26T11:04:52.530 INFO:tasks.workunit.client.0.smithi144.stdout:8/707: fsync d1a/d32/d49/fb7 0 2024-07-26T11:04:52.531 INFO:tasks.workunit.client.0.smithi144.stdout:3/546: mkdir d1/d28/d54/d26/d30/da7 0 2024-07-26T11:04:52.533 INFO:tasks.workunit.client.0.smithi144.stdout:8/708: mkdir d1a/d32/d49/d73/d7b/d4e/df1 0 2024-07-26T11:04:52.533 INFO:tasks.workunit.client.0.smithi144.stdout:8/709: write d1a/d32/d49/d73/d7b/f39 [2784320,116625] 0 2024-07-26T11:04:52.533 INFO:tasks.workunit.client.0.smithi144.stdout:8/710: chown d1a/d32/d49/d73/d7b/c60 2433512 1 2024-07-26T11:04:52.533 INFO:tasks.workunit.client.0.smithi144.stdout:3/547: rmdir d1/d5/d5e 39 2024-07-26T11:04:52.534 INFO:tasks.workunit.client.0.smithi144.stdout:4/746: write d9/d26/d48/f66 [1231429,104168] 0 2024-07-26T11:04:52.534 INFO:tasks.workunit.client.0.smithi144.stdout:4/747: truncate d9/da/f69 4693819 0 2024-07-26T11:04:52.534 INFO:tasks.workunit.client.0.smithi144.stdout:0/744: dwrite d0/d25/d5e/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:04:52.535 INFO:tasks.workunit.client.0.smithi144.stdout:8/711: creat d1a/d32/d49/d73/d2c/ff2 x:0 0 0 2024-07-26T11:04:52.537 INFO:tasks.workunit.client.0.smithi144.stdout:3/548: creat d1/d28/d6a/fa8 x:0 0 0 2024-07-26T11:04:52.539 INFO:tasks.workunit.client.0.smithi144.stdout:4/748: link d9/f76 d9/d26/d3c/ffb 0 2024-07-26T11:04:52.540 INFO:tasks.workunit.client.0.smithi144.stdout:4/749: creat d9/d26/d48/d50/ffc x:0 0 0 2024-07-26T11:04:52.540 INFO:tasks.workunit.client.0.smithi144.stdout:4/750: fdatasync d9/f51 0 2024-07-26T11:04:52.540 INFO:tasks.workunit.client.0.smithi144.stdout:4/751: fdatasync d9/d57/feb 0 2024-07-26T11:04:52.541 INFO:tasks.workunit.client.0.smithi144.stdout:4/752: write d9/d26/d48/d50/ffc [1039786,73002] 0 2024-07-26T11:04:52.541 INFO:tasks.workunit.client.0.smithi144.stdout:4/753: fdatasync d9/d26/d3c/d6f/f7c 0 2024-07-26T11:04:52.541 INFO:tasks.workunit.client.0.smithi144.stdout:4/754: creat d9/d26/d48/d50/ffd x:0 0 0 2024-07-26T11:04:52.542 INFO:tasks.workunit.client.0.smithi144.stdout:8/712: write d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2/fcb [386980,54951] 0 2024-07-26T11:04:52.543 INFO:tasks.workunit.client.0.smithi144.stdout:2/752: dwrite d0/d24/d2b/d68/d82/fc4 [0,4194304] 0 2024-07-26T11:04:52.543 INFO:tasks.workunit.client.0.smithi144.stdout:2/753: fsync d0/d24/d7b/ddc/fcc 0 2024-07-26T11:04:52.544 INFO:tasks.workunit.client.0.smithi144.stdout:5/725: dread d1/d9/d44/f41 [0,4194304] 0 2024-07-26T11:04:52.545 INFO:tasks.workunit.client.0.smithi144.stdout:8/713: mknod d1a/d32/d66/cf3 0 2024-07-26T11:04:52.546 INFO:tasks.workunit.client.0.smithi144.stdout:2/754: symlink d0/d24/d48/le9 0 2024-07-26T11:04:52.546 INFO:tasks.workunit.client.0.smithi144.stdout:2/755: chown d0/d6/d9/d44 1132088 1 2024-07-26T11:04:52.546 INFO:tasks.workunit.client.0.smithi144.stdout:2/756: chown d0/f3d 2006191 1 2024-07-26T11:04:52.548 INFO:tasks.workunit.client.0.smithi144.stdout:5/726: truncate d1/d9/f10 1537167 0 2024-07-26T11:04:52.548 INFO:tasks.workunit.client.0.smithi144.stdout:8/714: rmdir d1a/d32/d49/d73/d7b/d43 39 2024-07-26T11:04:52.548 INFO:tasks.workunit.client.0.smithi144.stdout:8/715: chown f11 0 1 2024-07-26T11:04:52.548 INFO:tasks.workunit.client.0.smithi144.stdout:2/757: truncate d0/f3d 2401839 0 2024-07-26T11:04:52.550 INFO:tasks.workunit.client.0.smithi144.stdout:5/727: creat d1/d18/d24/d27/ffa x:0 0 0 2024-07-26T11:04:52.553 INFO:tasks.workunit.client.0.smithi144.stdout:5/728: write d1/d9/d5b/d4d/fc0 [475954,84672] 0 2024-07-26T11:04:52.557 INFO:tasks.workunit.client.0.smithi144.stdout:7/705: dwrite d0/dd0/dbf/f41 [0,4194304] 0 2024-07-26T11:04:52.559 INFO:tasks.workunit.client.0.smithi144.stdout:8/716: truncate d1a/d1c/f2d 352821 0 2024-07-26T11:04:52.560 INFO:tasks.workunit.client.0.smithi144.stdout:2/758: unlink d0/d24/d7b/ddc/fcc 0 2024-07-26T11:04:52.560 INFO:tasks.workunit.client.0.smithi144.stdout:2/759: chown d0/d24/d7b/ddc/d5f/l8a 1148552 1 2024-07-26T11:04:52.560 INFO:tasks.workunit.client.0.smithi144.stdout:2/760: fdatasync d0/d24/d2b/d68/d82/fab 0 2024-07-26T11:04:52.560 INFO:tasks.workunit.client.0.smithi144.stdout:5/729: mkdir d1/d18/d24/dfb 0 2024-07-26T11:04:52.562 INFO:tasks.workunit.client.0.smithi144.stdout:7/706: creat d0/d6/d9/d3f/dd3/deb/ffe x:0 0 0 2024-07-26T11:04:52.565 INFO:tasks.workunit.client.0.smithi144.stdout:8/717: symlink d1a/d32/d49/d73/d2c/d9d/d74/lf4 0 2024-07-26T11:04:52.567 INFO:tasks.workunit.client.0.smithi144.stdout:6/857: dwrite d8/f82 [4194304,4194304] 0 2024-07-26T11:04:52.568 INFO:tasks.workunit.client.0.smithi144.stdout:7/707: creat d0/d6/d55/fff x:0 0 0 2024-07-26T11:04:52.569 INFO:tasks.workunit.client.0.smithi144.stdout:6/858: getdents d8/db/dbe/d52/d58 0 2024-07-26T11:04:52.576 INFO:tasks.workunit.client.0.smithi144.stdout:2/761: symlink d0/d24/d7b/lea 0 2024-07-26T11:04:52.576 INFO:tasks.workunit.client.0.smithi144.stdout:8/718: symlink d1a/d32/d8c/lf5 0 2024-07-26T11:04:52.577 INFO:tasks.workunit.client.0.smithi144.stdout:8/719: truncate d1a/d1c/f97 1056318 0 2024-07-26T11:04:52.577 INFO:tasks.workunit.client.0.smithi144.stdout:9/786: dread d9/d3b/f71 [0,4194304] 0 2024-07-26T11:04:52.577 INFO:tasks.workunit.client.0.smithi144.stdout:9/787: write d9/de/f4d [2046371,83714] 0 2024-07-26T11:04:52.579 INFO:tasks.workunit.client.0.smithi144.stdout:2/762: mkdir d0/d6/deb 0 2024-07-26T11:04:52.579 INFO:tasks.workunit.client.0.smithi144.stdout:2/763: write d0/d6/d9/f33 [2032780,22377] 0 2024-07-26T11:04:52.579 INFO:tasks.workunit.client.0.smithi144.stdout:2/764: read d0/d6/d9/faa [1039912,34920] 0 2024-07-26T11:04:52.586 INFO:tasks.workunit.client.0.smithi144.stdout:8/720: mkdir d1a/d32/d50/dc4/df6 0 2024-07-26T11:04:52.586 INFO:tasks.workunit.client.0.smithi144.stdout:8/721: fsync d1a/d32/d49/d73/d2c/d38/f5f 0 2024-07-26T11:04:52.586 INFO:tasks.workunit.client.0.smithi144.stdout:8/722: readlink d1a/d32/d49/d73/d7b/d43/d4f/l56 0 2024-07-26T11:04:52.601 INFO:tasks.workunit.client.0.smithi144.stdout:0/745: dwrite d0/d25/d5e/d1e/dc1/feb [0,4194304] 0 2024-07-26T11:04:52.604 INFO:tasks.workunit.client.0.smithi144.stdout:0/746: symlink d0/d25/d5e/d1e/d91/lfb 0 2024-07-26T11:04:52.609 INFO:tasks.workunit.client.0.smithi144.stdout:1/915: dwrite d3/fcb [0,4194304] 0 2024-07-26T11:04:52.615 INFO:tasks.workunit.client.0.smithi144.stdout:4/755: dwrite d9/d7b/fe6 [0,4194304] 0 2024-07-26T11:04:52.616 INFO:tasks.workunit.client.0.smithi144.stdout:4/756: creat d9/d26/d3c/d6f/dd0/ffe x:0 0 0 2024-07-26T11:04:52.618 INFO:tasks.workunit.client.0.smithi144.stdout:4/757: creat d9/d26/d3c/d6f/d8c/fff x:0 0 0 2024-07-26T11:04:52.622 INFO:tasks.workunit.client.0.smithi144.stdout:1/916: symlink d3/d6/d75/d88/da9/l129 0 2024-07-26T11:04:52.625 INFO:tasks.workunit.client.0.smithi144.stdout:2/765: dread d0/f91 [0,4194304] 0 2024-07-26T11:04:52.625 INFO:tasks.workunit.client.0.smithi144.stdout:2/766: chown d0/d24/d7b/ddc/d28/d27/d8b 2 1 2024-07-26T11:04:52.630 INFO:tasks.workunit.client.0.smithi144.stdout:1/917: mkdir d3/d6/d9c/d12a 0 2024-07-26T11:04:52.649 INFO:tasks.workunit.client.0.smithi144.stdout:5/730: dwrite d1/d9/d44/d45/d81/f9e [0,4194304] 0 2024-07-26T11:04:52.651 INFO:tasks.workunit.client.0.smithi144.stdout:5/731: getdents d1 0 2024-07-26T11:04:52.656 INFO:tasks.workunit.client.0.smithi144.stdout:0/747: dread d0/d3e/d45/f4a [0,4194304] 0 2024-07-26T11:04:52.656 INFO:tasks.workunit.client.0.smithi144.stdout:0/748: write d0/f18 [537313,94564] 0 2024-07-26T11:04:52.669 INFO:tasks.workunit.client.0.smithi144.stdout:4/758: read d9/da/f39 [3771439,92997] 0 2024-07-26T11:04:52.676 INFO:tasks.workunit.client.0.smithi144.stdout:8/723: read d1a/d32/d49/d73/d2c/d38/d58/fa8 [541185,102194] 0 2024-07-26T11:04:52.676 INFO:tasks.workunit.client.0.smithi144.stdout:8/724: truncate d1a/d32/d49/d73/d2c/ff2 21212 0 2024-07-26T11:04:52.676 INFO:tasks.workunit.client.0.smithi144.stdout:6/859: dwrite d8/f1a [0,4194304] 0 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/759: unlink d9/d26/d3c/d6f/f86 0 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:9/788: dread d9/de/faa [0,4194304] 0 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:9/789: readlink d9/de/d14/l72 0 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:9/790: chown d9/de/d14/d3f/d5e/dcd/fd3 0 1 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/760: creat d9/d7b/f100 x:0 0 0 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/761: fdatasync d9/d57/ff2 0 2024-07-26T11:04:52.677 INFO:tasks.workunit.client.0.smithi144.stdout:4/762: write d9/d26/f97 [1716229,125879] 0 2024-07-26T11:04:52.698 INFO:tasks.workunit.client.0.smithi144.stdout:2/767: dwrite d0/d24/d7b/ddc/d28/d2e/d5d/d6b/f99 [0,4194304] 0 2024-07-26T11:04:52.698 INFO:tasks.workunit.client.0.smithi144.stdout:7/708: dwrite d0/d6/d9/d12/f6d [0,4194304] 0 2024-07-26T11:04:52.706 INFO:tasks.workunit.client.0.smithi144.stdout:4/763: dread d9/d26/d3c/d6f/d8c/ff4 [0,4194304] 0 2024-07-26T11:04:52.707 INFO:tasks.workunit.client.0.smithi144.stdout:6/860: rename d8/d42/lf4 to d8/d15/d57/l131 0 2024-07-26T11:04:52.712 INFO:tasks.workunit.client.0.smithi144.stdout:4/764: link d9/da/da1/cbe d9/d26/d48/dd8/c101 0 2024-07-26T11:04:52.712 INFO:tasks.workunit.client.0.smithi144.stdout:4/765: dread - d9/da/ddd/fe0 zero size 2024-07-26T11:04:52.712 INFO:tasks.workunit.client.0.smithi144.stdout:4/766: write d9/da/f5b [1753,103824] 0 2024-07-26T11:04:52.714 INFO:tasks.workunit.client.0.smithi144.stdout:5/732: dwrite d1/d18/d24/d27/d71/fac [0,4194304] 0 2024-07-26T11:04:52.715 INFO:tasks.workunit.client.0.smithi144.stdout:4/767: rename d9/da/da1/fc1 to d9/f102 0 2024-07-26T11:04:52.716 INFO:tasks.workunit.client.0.smithi144.stdout:4/768: symlink d9/d26/d48/d50/d60/dc5/dd9/de1/l103 0 2024-07-26T11:04:52.731 INFO:tasks.workunit.client.0.smithi144.stdout:4/769: symlink d9/d7b/l104 0 2024-07-26T11:04:52.731 INFO:tasks.workunit.client.0.smithi144.stdout:0/749: dwrite d0/d25/d5e/d1e/d30/d67/fc5 [0,4194304] 0 2024-07-26T11:04:52.732 INFO:tasks.workunit.client.0.smithi144.stdout:1/918: dwrite d3/d6/d75/d88/da9/dac/d104/f126 [0,4194304] 0 2024-07-26T11:04:52.733 INFO:tasks.workunit.client.0.smithi144.stdout:2/768: write d0/d24/d7b/ddc/d28/d2e/d77/fb2 [88244,86131] 0 2024-07-26T11:04:52.735 INFO:tasks.workunit.client.0.smithi144.stdout:0/750: creat d0/d25/d5e/d1e/dc2/ffc x:0 0 0 2024-07-26T11:04:52.735 INFO:tasks.workunit.client.0.smithi144.stdout:0/751: chown d0/d3e/d45/dd4/de0/lea 100486272 1 2024-07-26T11:04:52.735 INFO:tasks.workunit.client.0.smithi144.stdout:0/752: chown d0/d25/d5e 2638755 1 2024-07-26T11:04:52.735 INFO:tasks.workunit.client.0.smithi144.stdout:0/753: fsync d0/d25/d5e/d1e/d30/d6a/de4/ff1 0 2024-07-26T11:04:52.737 INFO:tasks.workunit.client.0.smithi144.stdout:0/754: creat d0/d25/d5e/def/ffd x:0 0 0 2024-07-26T11:04:52.738 INFO:tasks.workunit.client.0.smithi144.stdout:0/755: truncate d0/d3e/d45/f55 1793968 0 2024-07-26T11:04:52.738 INFO:tasks.workunit.client.0.smithi144.stdout:1/919: truncate d3/f39 5280823 0 2024-07-26T11:04:52.739 INFO:tasks.workunit.client.0.smithi144.stdout:2/769: unlink d0/d24/d7b/l94 0 2024-07-26T11:04:52.742 INFO:tasks.workunit.client.0.smithi144.stdout:9/791: dwrite d9/de/d14/d23/d39/fbe [0,4194304] 0 2024-07-26T11:04:52.743 INFO:tasks.workunit.client.0.smithi144.stdout:9/792: write d9/de/d14/d79/db2/f100 [904731,67843] 0 2024-07-26T11:04:52.746 INFO:tasks.workunit.client.0.smithi144.stdout:9/793: mkdir d9/dfc/d108 0 2024-07-26T11:04:52.746 INFO:tasks.workunit.client.0.smithi144.stdout:9/794: readlink d9/de/d14/d3f/l43 0 2024-07-26T11:04:52.746 INFO:tasks.workunit.client.0.smithi144.stdout:9/795: truncate d9/de/d14/d23/d39/fc1 43370 0 2024-07-26T11:04:52.749 INFO:tasks.workunit.client.0.smithi144.stdout:4/770: write d9/d26/d3c/d6f/f85 [143517,90039] 0 2024-07-26T11:04:52.773 INFO:tasks.workunit.client.0.smithi144.stdout:1/920: dread d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f8a [0,4194304] 0 2024-07-26T11:04:52.774 INFO:tasks.workunit.client.0.smithi144.stdout:1/921: getdents d3/d6/da/d97/dfb 0 2024-07-26T11:04:52.775 INFO:tasks.workunit.client.0.smithi144.stdout:1/922: getdents d3/d6/d75/d88/da9/dac/d104/ddb/df6 0 2024-07-26T11:04:52.775 INFO:tasks.workunit.client.0.smithi144.stdout:1/923: write d3/d53/f54 [907189,34668] 0 2024-07-26T11:04:52.775 INFO:tasks.workunit.client.0.smithi144.stdout:1/924: truncate d3/d6/da/f121 358817 0 2024-07-26T11:04:52.776 INFO:tasks.workunit.client.0.smithi144.stdout:9/796: dread d9/fa [0,4194304] 0 2024-07-26T11:04:52.779 INFO:tasks.workunit.client.0.smithi144.stdout:5/733: dwrite d1/d18/d24/d27/f2f [0,4194304] 0 2024-07-26T11:04:52.779 INFO:tasks.workunit.client.0.smithi144.stdout:7/709: dwrite d0/dd0/dbf/f79 [0,4194304] 0 2024-07-26T11:04:52.779 INFO:tasks.workunit.client.0.smithi144.stdout:9/797: creat d9/d3b/dce/f109 x:0 0 0 2024-07-26T11:04:52.780 INFO:tasks.workunit.client.0.smithi144.stdout:5/734: symlink d1/d5/dae/lfc 0 2024-07-26T11:04:52.781 INFO:tasks.workunit.client.0.smithi144.stdout:7/710: mkdir d0/dd0/dbf/d67/de0/d100 0 2024-07-26T11:04:52.781 INFO:tasks.workunit.client.0.smithi144.stdout:7/711: chown d0/d6/d9/d12/f59 1135912 1 2024-07-26T11:04:52.781 INFO:tasks.workunit.client.0.smithi144.stdout:9/798: creat d9/de/d4e/d6a/f10a x:0 0 0 2024-07-26T11:04:52.781 INFO:tasks.workunit.client.0.smithi144.stdout:9/799: write d9/de/d14/d79/db2/f100 [1428939,63026] 0 2024-07-26T11:04:52.781 INFO:tasks.workunit.client.0.smithi144.stdout:5/735: truncate d1/d9/d5b/d4d/fa5 334612 0 2024-07-26T11:04:52.783 INFO:tasks.workunit.client.0.smithi144.stdout:7/712: mknod d0/d71/c101 0 2024-07-26T11:04:52.783 INFO:tasks.workunit.client.0.smithi144.stdout:9/800: mknod d9/de/d14/d23/c10b 0 2024-07-26T11:04:52.799 INFO:tasks.workunit.client.0.smithi144.stdout:0/756: dwrite d0/d3e/d45/d4c/f8f [0,4194304] 0 2024-07-26T11:04:52.808 INFO:tasks.workunit.client.0.smithi144.stdout:4/771: dread d9/f1f [0,4194304] 0 2024-07-26T11:04:52.808 INFO:tasks.workunit.client.0.smithi144.stdout:2/770: dwrite d0/d24/d48/d63/da3/fbc [0,4194304] 0 2024-07-26T11:04:52.808 INFO:tasks.workunit.client.0.smithi144.stdout:2/771: read - d0/d6/d34/fca zero size 2024-07-26T11:04:52.809 INFO:tasks.workunit.client.0.smithi144.stdout:2/772: stat d0/d6/d34/ldf 0 2024-07-26T11:04:52.811 INFO:tasks.workunit.client.0.smithi144.stdout:2/773: creat d0/fec x:0 0 0 2024-07-26T11:04:52.812 INFO:tasks.workunit.client.0.smithi144.stdout:2/774: getdents d0/d6/d34 0 2024-07-26T11:04:52.829 INFO:tasks.workunit.client.0.smithi144.stdout:4/772: dread d9/f1f [0,4194304] 0 2024-07-26T11:04:52.829 INFO:tasks.workunit.client.0.smithi144.stdout:4/773: chown d9/d26/d48/d50/cce 99640829 1 2024-07-26T11:04:52.829 INFO:tasks.workunit.client.0.smithi144.stdout:4/774: chown d9/d7b/l104 14 1 2024-07-26T11:04:52.831 INFO:tasks.workunit.client.0.smithi144.stdout:7/713: read d0/dd0/dbf/d2f/d47/fa0 [3543978,77274] 0 2024-07-26T11:04:52.831 INFO:tasks.workunit.client.0.smithi144.stdout:9/801: dwrite d9/d3b/d5d/fe3 [0,4194304] 0 2024-07-26T11:04:52.832 INFO:tasks.workunit.client.0.smithi144.stdout:7/714: mknod d0/d6/d9/d12/c102 0 2024-07-26T11:04:52.833 INFO:tasks.workunit.client.0.smithi144.stdout:9/802: mkdir d9/de/d14/d23/d39/d7f/dbc/d10c 0 2024-07-26T11:04:52.835 INFO:tasks.workunit.client.0.smithi144.stdout:7/715: creat d0/dd0/dbf/d2f/d53/db6/dfc/f103 x:0 0 0 2024-07-26T11:04:52.835 INFO:tasks.workunit.client.0.smithi144.stdout:9/803: link d9/de/d14/d79/db2/f100 d9/de/d14/d23/d39/d7f/f10d 0 2024-07-26T11:04:52.835 INFO:tasks.workunit.client.0.smithi144.stdout:9/804: stat d9/de/d14/f2c 0 2024-07-26T11:04:52.835 INFO:tasks.workunit.client.0.smithi144.stdout:9/805: write d9/d3b/dce/f109 [30919,84906] 0 2024-07-26T11:04:52.835 INFO:tasks.workunit.client.0.smithi144.stdout:7/716: rmdir d0/d6/d9/d45 39 2024-07-26T11:04:52.836 INFO:tasks.workunit.client.0.smithi144.stdout:7/717: chown d0/d6/d9/d12/f1d 2 1 2024-07-26T11:04:52.836 INFO:tasks.workunit.client.0.smithi144.stdout:7/718: fdatasync d0/f11 0 2024-07-26T11:04:52.839 INFO:tasks.workunit.client.0.smithi144.stdout:9/806: getdents d9/de/d14/d23/d39/d7f 0 2024-07-26T11:04:52.862 INFO:tasks.workunit.client.0.smithi144.stdout:0/757: dwrite d0/f32 [4194304,4194304] 0 2024-07-26T11:04:52.864 INFO:tasks.workunit.client.0.smithi144.stdout:0/758: symlink d0/d3e/d45/dd4/de0/lfe 0 2024-07-26T11:04:52.865 INFO:tasks.workunit.client.0.smithi144.stdout:0/759: mknod d0/d25/d5e/d1e/d30/d6a/d6f/cff 0 2024-07-26T11:04:52.870 INFO:tasks.workunit.client.0.smithi144.stdout:7/719: read d0/d71/f99 [531350,28134] 0 2024-07-26T11:04:52.870 INFO:tasks.workunit.client.0.smithi144.stdout:7/720: dread - d0/dd0/dbf/d2f/d53/db6/dfc/f103 zero size 2024-07-26T11:04:52.871 INFO:tasks.workunit.client.0.smithi144.stdout:7/721: mknod d0/dd0/dbf/d67/c104 0 2024-07-26T11:04:52.871 INFO:tasks.workunit.client.0.smithi144.stdout:7/722: fsync d0/dd0/dbf/d67/f9f 0 2024-07-26T11:04:52.871 INFO:tasks.workunit.client.0.smithi144.stdout:7/723: readlink d0/d71/d95/lea 0 2024-07-26T11:04:52.872 INFO:tasks.workunit.client.0.smithi144.stdout:7/724: symlink d0/dd0/dbf/d2f/d47/l105 0 2024-07-26T11:04:52.873 INFO:tasks.workunit.client.0.smithi144.stdout:9/807: dread d9/de/d14/f2c [0,4194304] 0 2024-07-26T11:04:52.877 INFO:tasks.workunit.client.0.smithi144.stdout:4/775: dwrite d9/da/ff8 [0,4194304] 0 2024-07-26T11:04:52.877 INFO:tasks.workunit.client.0.smithi144.stdout:4/776: write d9/d26/f30 [2833548,66636] 0 2024-07-26T11:04:52.880 INFO:tasks.workunit.client.0.smithi144.stdout:4/777: symlink d9/l105 0 2024-07-26T11:04:52.880 INFO:tasks.workunit.client.0.smithi144.stdout:4/778: chown d9/d26/d48/d50/d60/l94 493 1 2024-07-26T11:04:52.881 INFO:tasks.workunit.client.0.smithi144.stdout:4/779: symlink d9/d26/d48/d98/db8/l106 0 2024-07-26T11:04:52.882 INFO:tasks.workunit.client.0.smithi144.stdout:7/725: dread d0/dd0/dbf/d67/f9f [0,4194304] 0 2024-07-26T11:04:52.882 INFO:tasks.workunit.client.0.smithi144.stdout:7/726: dread - d0/d6/d55/fff zero size 2024-07-26T11:04:52.883 INFO:tasks.workunit.client.0.smithi144.stdout:4/780: mknod d9/d26/d48/dd8/c107 0 2024-07-26T11:04:52.890 INFO:tasks.workunit.client.0.smithi144.stdout:7/727: mkdir d0/dd0/dbf/d2f/d53/db6/d106 0 2024-07-26T11:04:52.890 INFO:tasks.workunit.client.0.smithi144.stdout:0/760: write d0/f32 [5406216,109074] 0 2024-07-26T11:04:52.890 INFO:tasks.workunit.client.0.smithi144.stdout:4/781: creat d9/d26/d3c/d6f/dd0/f108 x:0 0 0 2024-07-26T11:04:52.890 INFO:tasks.workunit.client.0.smithi144.stdout:4/782: chown d9/d26/d48/dd8 272 1 2024-07-26T11:04:52.890 INFO:tasks.workunit.client.0.smithi144.stdout:4/783: write d9/f102 [719468,72742] 0 2024-07-26T11:04:52.893 INFO:tasks.workunit.client.0.smithi144.stdout:7/728: link d0/d6/d9/d12/c72 d0/d71/d95/dc4/c107 0 2024-07-26T11:04:52.893 INFO:tasks.workunit.client.0.smithi144.stdout:4/784: creat d9/d26/d3c/d6f/dd0/f109 x:0 0 0 2024-07-26T11:04:52.895 INFO:tasks.workunit.client.0.smithi144.stdout:7/729: creat d0/dd0/dbf/d2f/d47/f108 x:0 0 0 2024-07-26T11:04:52.895 INFO:tasks.workunit.client.0.smithi144.stdout:7/730: read d0/d6/d55/f6a [610686,46685] 0 2024-07-26T11:04:52.916 INFO:tasks.workunit.client.0.smithi144.stdout:4/785: dread d9/d26/f62 [0,4194304] 0 2024-07-26T11:04:52.916 INFO:tasks.workunit.client.0.smithi144.stdout:4/786: truncate d9/d26/d3c/d6f/f7c 4743693 0 2024-07-26T11:04:52.916 INFO:tasks.workunit.client.0.smithi144.stdout:4/787: fsync d9/f11 0 2024-07-26T11:04:52.916 INFO:tasks.workunit.client.0.smithi144.stdout:4/788: chown d9/d26/d48/dd8/fe2 3 1 2024-07-26T11:04:52.919 INFO:tasks.workunit.client.0.smithi144.stdout:4/789: unlink d9/d26/l4f 0 2024-07-26T11:04:52.919 INFO:tasks.workunit.client.0.smithi144.stdout:4/790: truncate d9/d26/d48/d50/d60/f88 3574410 0 2024-07-26T11:04:52.921 INFO:tasks.workunit.client.0.smithi144.stdout:7/731: dread d0/dd0/dbf/d2f/d47/d5e/fba [0,4194304] 0 2024-07-26T11:04:52.921 INFO:tasks.workunit.client.0.smithi144.stdout:4/791: read d9/f11 [17597,67508] 0 2024-07-26T11:04:52.922 INFO:tasks.workunit.client.0.smithi144.stdout:0/761: dwrite d0/d25/d5e/d1e/d5c/f99 [0,4194304] 0 2024-07-26T11:04:52.923 INFO:tasks.workunit.client.0.smithi144.stdout:7/732: creat d0/dd0/dbf/d2f/d53/db6/dfc/f109 x:0 0 0 2024-07-26T11:04:52.923 INFO:tasks.workunit.client.0.smithi144.stdout:4/792: rmdir d9/d7b/dc4 39 2024-07-26T11:04:52.924 INFO:tasks.workunit.client.0.smithi144.stdout:0/762: mkdir d0/d25/d5e/d1e/d30/d67/d100 0 2024-07-26T11:04:52.924 INFO:tasks.workunit.client.0.smithi144.stdout:7/733: creat d0/dd0/dbf/d67/de0/f10a x:0 0 0 2024-07-26T11:04:52.925 INFO:tasks.workunit.client.0.smithi144.stdout:0/763: mkdir d0/d25/d5e/d1e/d5c/d101 0 2024-07-26T11:04:52.927 INFO:tasks.workunit.client.0.smithi144.stdout:0/764: symlink d0/d25/d5e/d1e/d30/d67/d84/l102 0 2024-07-26T11:04:52.927 INFO:tasks.workunit.client.0.smithi144.stdout:0/765: write d0/d25/d5e/d1e/d30/d67/fc5 [5236976,75908] 0 2024-07-26T11:04:52.927 INFO:tasks.workunit.client.0.smithi144.stdout:0/766: readlink d0/d25/d5e/d1e/d91/lfb 0 2024-07-26T11:04:52.928 INFO:tasks.workunit.client.0.smithi144.stdout:0/767: rename d0/d25/d5e/d1e/c85 to d0/d25/d5e/d1e/dd2/c103 0 2024-07-26T11:04:52.940 INFO:tasks.workunit.client.0.smithi144.stdout:0/768: truncate d0/d3e/d45/d4c/fca 3095338 0 2024-07-26T11:04:52.940 INFO:tasks.workunit.client.0.smithi144.stdout:0/769: stat d0/f10 0 2024-07-26T11:04:52.940 INFO:tasks.workunit.client.0.smithi144.stdout:0/770: stat d0/d25/d5e/d1e/dd2/ld7 0 2024-07-26T11:04:52.952 INFO:tasks.workunit.client.0.smithi144.stdout:7/734: write d0/d6/d9/d12/f6d [3939682,17652] 0 2024-07-26T11:04:52.976 INFO:tasks.workunit.client.0.smithi144.stdout:7/735: write d0/dd0/dbf/d2f/f35 [64148,113454] 0 2024-07-26T11:04:52.976 INFO:tasks.workunit.client.0.smithi144.stdout:4/793: dwrite d9/d26/d3c/d6f/f81 [0,4194304] 0 2024-07-26T11:04:52.979 INFO:tasks.workunit.client.0.smithi144.stdout:7/736: mknod d0/dd0/dbf/d67/c10b 0 2024-07-26T11:04:52.993 INFO:tasks.workunit.client.0.smithi144.stdout:0/771: dwrite d0/d25/d5e/d1e/d30/f87 [0,4194304] 0 2024-07-26T11:04:52.993 INFO:tasks.workunit.client.0.smithi144.stdout:0/772: chown d0/d25/d5e/def 32148917 1 2024-07-26T11:04:52.996 INFO:tasks.workunit.client.0.smithi144.stdout:4/794: dread d9/f102 [0,4194304] 0 2024-07-26T11:04:52.997 INFO:tasks.workunit.client.0.smithi144.stdout:0/773: truncate d0/d25/d5e/d1e/f24 1764465 0 2024-07-26T11:04:52.998 INFO:tasks.workunit.client.0.smithi144.stdout:4/795: creat d9/da/ddd/f10a x:0 0 0 2024-07-26T11:04:52.999 INFO:tasks.workunit.client.0.smithi144.stdout:0/774: rename d0/d25/d5e/d1e/d5c to d0/d25/d5e/d1e/d30/d6a/de4/d104 0 2024-07-26T11:04:53.000 INFO:tasks.workunit.client.0.smithi144.stdout:0/775: getdents d0/d25/d5e 0 2024-07-26T11:04:53.000 INFO:tasks.workunit.client.0.smithi144.stdout:0/776: stat d0/d25/fa0 0 2024-07-26T11:04:53.021 INFO:tasks.workunit.client.0.smithi144.stdout:0/777: dread d0/d3e/d45/d4c/f8f [0,4194304] 0 2024-07-26T11:04:53.021 INFO:tasks.workunit.client.0.smithi144.stdout:0/778: fsync d0/d25/d5e/d1e/d30/d67/f90 0 2024-07-26T11:04:53.022 INFO:tasks.workunit.client.0.smithi144.stdout:0/779: mkdir d0/d3e/d45/d9f/df4/d105 0 2024-07-26T11:04:53.022 INFO:tasks.workunit.client.0.smithi144.stdout:0/780: read - d0/d25/d5e/def/ffd zero size 2024-07-26T11:04:53.024 INFO:tasks.workunit.client.0.smithi144.stdout:0/781: creat d0/d25/d5e/d1e/dd2/f106 x:0 0 0 2024-07-26T11:04:53.024 INFO:tasks.workunit.client.0.smithi144.stdout:0/782: fdatasync d0/d3e/f43 0 2024-07-26T11:04:53.025 INFO:tasks.workunit.client.0.smithi144.stdout:3/549: sync 2024-07-26T11:04:53.025 INFO:tasks.workunit.client.0.smithi144.stdout:8/725: sync 2024-07-26T11:04:53.025 INFO:tasks.workunit.client.0.smithi144.stdout:3/550: fsync d1/d5/f24 0 2024-07-26T11:04:53.039 INFO:tasks.workunit.client.0.smithi144.stdout:4/796: dwrite d9/d26/d48/dd8/fe2 [0,4194304] 0 2024-07-26T11:04:53.041 INFO:tasks.workunit.client.0.smithi144.stdout:4/797: truncate d9/f8d 527369 0 2024-07-26T11:04:53.042 INFO:tasks.workunit.client.0.smithi144.stdout:7/737: dwrite d0/d6/fcd [0,4194304] 0 2024-07-26T11:04:53.042 INFO:tasks.workunit.client.0.smithi144.stdout:4/798: mkdir d9/d26/d48/d98/d10b 0 2024-07-26T11:04:53.043 INFO:tasks.workunit.client.0.smithi144.stdout:0/783: dread d0/d3e/d45/d9f/fa8 [0,4194304] 0 2024-07-26T11:04:53.044 INFO:tasks.workunit.client.0.smithi144.stdout:7/738: rename d0/d6/d9/d45 to d0/d6/d10c 0 2024-07-26T11:04:53.046 INFO:tasks.workunit.client.0.smithi144.stdout:7/739: creat d0/d6/d9/f10d x:0 0 0 2024-07-26T11:04:53.071 INFO:tasks.workunit.client.0.smithi144.stdout:7/740: read d0/dd0/dbf/f41 [4073626,119099] 0 2024-07-26T11:04:53.075 INFO:tasks.workunit.client.0.smithi144.stdout:0/784: read d0/d25/d5e/d1e/dc1/feb [718401,93229] 0 2024-07-26T11:04:53.075 INFO:tasks.workunit.client.0.smithi144.stdout:8/726: dwrite d1a/d32/d66/da0/ddc/fef [0,4194304] 0 2024-07-26T11:04:53.077 INFO:tasks.workunit.client.0.smithi144.stdout:0/785: mkdir d0/d25/d5e/d1e/d30/dab/d107 0 2024-07-26T11:04:53.081 INFO:tasks.workunit.client.0.smithi144.stdout:8/727: getdents d1a/d32/d66/da0 0 2024-07-26T11:04:53.081 INFO:tasks.workunit.client.0.smithi144.stdout:8/728: chown d1a/d32/d6c 84375 1 2024-07-26T11:04:53.081 INFO:tasks.workunit.client.0.smithi144.stdout:7/741: dread d0/dd0/dbf/d67/f9f [0,4194304] 0 2024-07-26T11:04:53.086 INFO:tasks.workunit.client.0.smithi144.stdout:4/799: dwrite d9/f41 [0,4194304] 0 2024-07-26T11:04:53.086 INFO:tasks.workunit.client.0.smithi144.stdout:4/800: truncate d9/d26/d48/f8f 47558 0 2024-07-26T11:04:53.086 INFO:tasks.workunit.client.0.smithi144.stdout:8/729: creat d1a/d32/d66/da0/ddc/ff7 x:0 0 0 2024-07-26T11:04:53.086 INFO:tasks.workunit.client.0.smithi144.stdout:4/801: truncate d9/d26/d48/f8f 329036 0 2024-07-26T11:04:53.086 INFO:tasks.workunit.client.0.smithi144.stdout:7/742: symlink d0/dd0/l10e 0 2024-07-26T11:04:53.086 INFO:tasks.workunit.client.0.smithi144.stdout:7/743: chown d0/d6/c31 1865 1 2024-07-26T11:04:53.087 INFO:tasks.workunit.client.0.smithi144.stdout:4/802: rmdir d9/d26/d48/d98 39 2024-07-26T11:04:53.088 INFO:tasks.workunit.client.0.smithi144.stdout:4/803: mkdir d9/d26/d3c/d10c 0 2024-07-26T11:04:53.088 INFO:tasks.workunit.client.0.smithi144.stdout:4/804: write d9/da/ddd/fe0 [487487,36649] 0 2024-07-26T11:04:53.089 INFO:tasks.workunit.client.0.smithi144.stdout:4/805: truncate d9/d26/d48/d50/ffc 665233 0 2024-07-26T11:04:53.089 INFO:tasks.workunit.client.0.smithi144.stdout:4/806: write d9/d26/f62 [4621856,55515] 0 2024-07-26T11:04:53.107 INFO:tasks.workunit.client.0.smithi144.stdout:0/786: dread d0/d25/d5e/d1e/d91/dd1/fbb [0,4194304] 0 2024-07-26T11:04:53.107 INFO:tasks.workunit.client.0.smithi144.stdout:0/787: stat d0/d25/d5e/def/ffd 0 2024-07-26T11:04:53.108 INFO:tasks.workunit.client.0.smithi144.stdout:0/788: symlink d0/d25/d5e/d1e/d6c/dc6/l108 0 2024-07-26T11:04:53.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/730: dread d1a/d1c/f24 [0,4194304] 0 2024-07-26T11:04:53.122 INFO:tasks.workunit.client.0.smithi144.stdout:8/731: creat d1a/d32/d66/da0/ddc/ff8 x:0 0 0 2024-07-26T11:04:53.125 INFO:tasks.workunit.client.0.smithi144.stdout:7/744: read d0/dd0/dbf/d67/f9d [2404255,76904] 0 2024-07-26T11:04:53.128 INFO:tasks.workunit.client.0.smithi144.stdout:4/807: write d9/d26/d3c/d6f/f81 [980811,129629] 0 2024-07-26T11:04:53.128 INFO:tasks.workunit.client.0.smithi144.stdout:4/808: dread - d9/da/fb7 zero size 2024-07-26T11:04:53.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/809: mknod d9/da/c10d 0 2024-07-26T11:04:53.132 INFO:tasks.workunit.client.0.smithi144.stdout:4/810: mknod d9/da/c10e 0 2024-07-26T11:04:53.132 INFO:tasks.workunit.client.0.smithi144.stdout:4/811: write d9/f76 [259837,10026] 0 2024-07-26T11:04:53.134 INFO:tasks.workunit.client.0.smithi144.stdout:4/812: unlink d9/l105 0 2024-07-26T11:04:53.135 INFO:tasks.workunit.client.0.smithi144.stdout:6/861: sync 2024-07-26T11:04:53.136 INFO:tasks.workunit.client.0.smithi144.stdout:4/813: symlink d9/d26/d3c/d10c/l10f 0 2024-07-26T11:04:53.137 INFO:tasks.workunit.client.0.smithi144.stdout:6/862: mkdir d8/d50/d92/db3/d132 0 2024-07-26T11:04:53.138 INFO:tasks.workunit.client.0.smithi144.stdout:4/814: creat d9/d7b/dc4/dec/f110 x:0 0 0 2024-07-26T11:04:53.138 INFO:tasks.workunit.client.0.smithi144.stdout:4/815: chown d9/d26/d3c/ff5 18 1 2024-07-26T11:04:53.138 INFO:tasks.workunit.client.0.smithi144.stdout:6/863: mkdir d8/d50/d92/dea/df8/d133 0 2024-07-26T11:04:53.140 INFO:tasks.workunit.client.0.smithi144.stdout:6/864: mknod d8/d50/d98/d104/c134 0 2024-07-26T11:04:53.142 INFO:tasks.workunit.client.0.smithi144.stdout:6/865: creat d8/d50/d92/db3/d132/f135 x:0 0 0 2024-07-26T11:04:53.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/866: mkdir d8/db/da6/d136 0 2024-07-26T11:04:53.146 INFO:tasks.workunit.client.0.smithi144.stdout:6/867: chown d8/d15/fd4 392 1 2024-07-26T11:04:53.147 INFO:tasks.workunit.client.0.smithi144.stdout:6/868: mknod d8/db/dbe/d52/d58/d120/c137 0 2024-07-26T11:04:53.147 INFO:tasks.workunit.client.0.smithi144.stdout:6/869: chown d8/db/da6/fc2 20469964 1 2024-07-26T11:04:53.153 INFO:tasks.workunit.client.0.smithi144.stdout:0/789: dwrite d0/d25/d5e/d1e/d30/d6a/dac/fbc [0,4194304] 0 2024-07-26T11:04:53.185 INFO:tasks.workunit.client.0.smithi144.stdout:8/732: dwrite d1a/d1c/f1f [0,4194304] 0 2024-07-26T11:04:53.185 INFO:tasks.workunit.client.0.smithi144.stdout:8/733: chown d1a/d1c/lcc 387 1 2024-07-26T11:04:53.189 INFO:tasks.workunit.client.0.smithi144.stdout:6/870: dwrite d8/d15/d57/d5f/d88/f8a [0,4194304] 0 2024-07-26T11:04:53.189 INFO:tasks.workunit.client.0.smithi144.stdout:6/871: chown d8/d50/d92/db3/d103/d125/f129 274764694 1 2024-07-26T11:04:53.192 INFO:tasks.workunit.client.0.smithi144.stdout:6/872: truncate d8/db/f2c 5543797 0 2024-07-26T11:04:53.203 INFO:tasks.workunit.client.0.smithi144.stdout:4/816: dwrite d9/d26/f99 [0,4194304] 0 2024-07-26T11:04:53.203 INFO:tasks.workunit.client.0.smithi144.stdout:4/817: fdatasync d9/d26/d3c/d6f/dd0/ffe 0 2024-07-26T11:04:53.208 INFO:tasks.workunit.client.0.smithi144.stdout:0/790: dwrite d0/d25/d5e/d1e/d30/d4b/fb6 [0,4194304] 0 2024-07-26T11:04:53.212 INFO:tasks.workunit.client.0.smithi144.stdout:0/791: creat d0/d25/d5e/d1e/d30/d67/d84/f109 x:0 0 0 2024-07-26T11:04:53.212 INFO:tasks.workunit.client.0.smithi144.stdout:0/792: fdatasync d0/d25/d5e/d1e/dc2/f9e 0 2024-07-26T11:04:53.212 INFO:tasks.workunit.client.0.smithi144.stdout:0/793: write d0/d25/d5e/d1e/f3c [3774361,99168] 0 2024-07-26T11:04:53.216 INFO:tasks.workunit.client.0.smithi144.stdout:6/873: write d8/d50/de2/ff7 [2067744,56626] 0 2024-07-26T11:04:53.216 INFO:tasks.workunit.client.0.smithi144.stdout:6/874: chown d8/db/ccf 907563594 1 2024-07-26T11:04:53.216 INFO:tasks.workunit.client.0.smithi144.stdout:6/875: write d8/db/f27 [11601,71170] 0 2024-07-26T11:04:53.216 INFO:tasks.workunit.client.0.smithi144.stdout:6/876: truncate d8/db/dbe/d52/d58/faf 118098 0 2024-07-26T11:04:53.229 INFO:tasks.workunit.client.0.smithi144.stdout:9/808: sync 2024-07-26T11:04:53.231 INFO:tasks.workunit.client.0.smithi144.stdout:6/877: rename d8/d15/d26/f3f to d8/d50/d92/db3/d103/d12a/f138 0 2024-07-26T11:04:53.233 INFO:tasks.workunit.client.0.smithi144.stdout:9/809: symlink d9/de/d4e/d6a/l10e 0 2024-07-26T11:04:53.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/794: dread d0/d25/d5e/d1e/d30/fb9 [0,4194304] 0 2024-07-26T11:04:53.241 INFO:tasks.workunit.client.0.smithi144.stdout:0/795: rmdir d0/d25/d5e/d1e/d30/d4b 39 2024-07-26T11:04:53.242 INFO:tasks.workunit.client.0.smithi144.stdout:0/796: mknod d0/d25/d5e/d1e/d30/d6a/d6f/dec/c10a 0 2024-07-26T11:04:53.251 INFO:tasks.workunit.client.0.smithi144.stdout:0/797: creat d0/d3e/d45/d9f/df4/f10b x:0 0 0 2024-07-26T11:04:53.252 INFO:tasks.workunit.client.0.smithi144.stdout:8/734: dwrite d1a/d32/d8c/d9b/fe8 [0,4194304] 0 2024-07-26T11:04:53.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/775: sync 2024-07-26T11:04:53.256 INFO:tasks.workunit.client.0.smithi144.stdout:5/736: sync 2024-07-26T11:04:53.256 INFO:tasks.workunit.client.0.smithi144.stdout:1/925: sync 2024-07-26T11:04:53.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/776: chown d0/d24/d2b/d68/d82/ld7 703851 1 2024-07-26T11:04:53.256 INFO:tasks.workunit.client.0.smithi144.stdout:2/777: read - d0/fec zero size 2024-07-26T11:04:53.257 INFO:tasks.workunit.client.0.smithi144.stdout:2/778: write d0/d24/d48/d63/da3/fbc [4985667,17889] 0 2024-07-26T11:04:53.262 INFO:tasks.workunit.client.0.smithi144.stdout:1/926: truncate d3/d6/d75/d88/da9/dac/d104/ff0 2477872 0 2024-07-26T11:04:53.262 INFO:tasks.workunit.client.0.smithi144.stdout:4/818: dwrite d9/da/fad [0,4194304] 0 2024-07-26T11:04:53.262 INFO:tasks.workunit.client.0.smithi144.stdout:4/819: fsync d9/d26/f62 0 2024-07-26T11:04:53.267 INFO:tasks.workunit.client.0.smithi144.stdout:0/798: rmdir d0/d25/d5e/d1e/d30/d6a/de4/d104 39 2024-07-26T11:04:53.267 INFO:tasks.workunit.client.0.smithi144.stdout:8/735: symlink d1a/d32/d49/d73/d7b/lf9 0 2024-07-26T11:04:53.267 INFO:tasks.workunit.client.0.smithi144.stdout:5/737: mknod d1/d9/d5b/d4d/cfd 0 2024-07-26T11:04:53.268 INFO:tasks.workunit.client.0.smithi144.stdout:4/820: mkdir d9/d7b/dc4/d111 0 2024-07-26T11:04:53.268 INFO:tasks.workunit.client.0.smithi144.stdout:5/738: symlink d1/d9/d44/d45/d98/lfe 0 2024-07-26T11:04:53.268 INFO:tasks.workunit.client.0.smithi144.stdout:2/779: symlink d0/d24/d7b/ddc/d28/led 0 2024-07-26T11:04:53.269 INFO:tasks.workunit.client.0.smithi144.stdout:4/821: creat d9/d7b/dc4/d111/f112 x:0 0 0 2024-07-26T11:04:53.269 INFO:tasks.workunit.client.0.smithi144.stdout:4/822: chown d9/d26/d48/d50/ffd 10578 1 2024-07-26T11:04:53.269 INFO:tasks.workunit.client.0.smithi144.stdout:4/823: chown d9/cb2 941911 1 2024-07-26T11:04:53.270 INFO:tasks.workunit.client.0.smithi144.stdout:1/927: dread d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f81 [0,4194304] 0 2024-07-26T11:04:53.271 INFO:tasks.workunit.client.0.smithi144.stdout:2/780: rename d0/d24/d7b/ddc/d28/d27/l5e to d0/d6/d34/lee 0 2024-07-26T11:04:53.271 INFO:tasks.workunit.client.0.smithi144.stdout:3/551: sync 2024-07-26T11:04:53.271 INFO:tasks.workunit.client.0.smithi144.stdout:7/745: sync 2024-07-26T11:04:53.271 INFO:tasks.workunit.client.0.smithi144.stdout:3/552: fsync d1/f50 0 2024-07-26T11:04:53.271 INFO:tasks.workunit.client.0.smithi144.stdout:3/553: stat d1/d28/d54/l3f 0 2024-07-26T11:04:53.271 INFO:tasks.workunit.client.0.smithi144.stdout:7/746: write d0/f49 [2856925,5574] 0 2024-07-26T11:04:53.272 INFO:tasks.workunit.client.0.smithi144.stdout:8/736: dread d1a/d1c/f70 [0,4194304] 0 2024-07-26T11:04:53.273 INFO:tasks.workunit.client.0.smithi144.stdout:9/810: dwrite d9/de/d14/d23/d39/fc1 [0,4194304] 0 2024-07-26T11:04:53.274 INFO:tasks.workunit.client.0.smithi144.stdout:5/739: creat d1/d18/d24/d35/dd7/fff x:0 0 0 2024-07-26T11:04:53.277 INFO:tasks.workunit.client.0.smithi144.stdout:4/824: mknod d9/d57/c113 0 2024-07-26T11:04:53.277 INFO:tasks.workunit.client.0.smithi144.stdout:2/781: unlink d0/d24/d7b/ddc/d28/d2e/d77/fbd 0 2024-07-26T11:04:53.278 INFO:tasks.workunit.client.0.smithi144.stdout:7/747: unlink d0/d6/d9/d12/d6c/f9c 0 2024-07-26T11:04:53.279 INFO:tasks.workunit.client.0.smithi144.stdout:8/737: creat d1a/d1c/ffa x:0 0 0 2024-07-26T11:04:53.279 INFO:tasks.workunit.client.0.smithi144.stdout:5/740: stat d1/d9/d44/la7 0 2024-07-26T11:04:53.282 INFO:tasks.workunit.client.0.smithi144.stdout:4/825: link d9/d26/f2b d9/d7b/dc4/f114 0 2024-07-26T11:04:53.283 INFO:tasks.workunit.client.0.smithi144.stdout:2/782: creat d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fef x:0 0 0 2024-07-26T11:04:53.284 INFO:tasks.workunit.client.0.smithi144.stdout:8/738: creat d1a/d32/d66/da0/ddc/ffb x:0 0 0 2024-07-26T11:04:53.287 INFO:tasks.workunit.client.0.smithi144.stdout:5/741: write d1/d5/f6d [38748,28409] 0 2024-07-26T11:04:53.287 INFO:tasks.workunit.client.0.smithi144.stdout:1/928: dread d3/d53/f54 [0,4194304] 0 2024-07-26T11:04:53.287 INFO:tasks.workunit.client.0.smithi144.stdout:4/826: mkdir d9/da/da1/d115 0 2024-07-26T11:04:53.287 INFO:tasks.workunit.client.0.smithi144.stdout:4/827: chown d9/d7b/fe9 22795416 1 2024-07-26T11:04:53.287 INFO:tasks.workunit.client.0.smithi144.stdout:4/828: readlink d9/d57/l74 0 2024-07-26T11:04:53.288 INFO:tasks.workunit.client.0.smithi144.stdout:7/748: symlink d0/d6/d10c/d57/l10f 0 2024-07-26T11:04:53.289 INFO:tasks.workunit.client.0.smithi144.stdout:1/929: truncate d3/d6/f102 1025946 0 2024-07-26T11:04:53.289 INFO:tasks.workunit.client.0.smithi144.stdout:6/878: dwrite d8/f82 [0,4194304] 0 2024-07-26T11:04:53.290 INFO:tasks.workunit.client.0.smithi144.stdout:8/739: getdents d1a/d32/d8c/d9b 0 2024-07-26T11:04:53.290 INFO:tasks.workunit.client.0.smithi144.stdout:8/740: read - d1a/fc8 zero size 2024-07-26T11:04:53.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/783: rename d0/d24/d2b/d68/f6a to d0/d24/d7b/ddc/d28/ff0 0 2024-07-26T11:04:53.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/784: write d0/d6/d9/faa [1798601,8776] 0 2024-07-26T11:04:53.290 INFO:tasks.workunit.client.0.smithi144.stdout:2/785: chown d0/d24/d7b/ddc/d28/d2e/c32 650613527 1 2024-07-26T11:04:53.292 INFO:tasks.workunit.client.0.smithi144.stdout:6/879: creat d8/db/dbe/daa/f139 x:0 0 0 2024-07-26T11:04:53.292 INFO:tasks.workunit.client.0.smithi144.stdout:7/749: unlink d0/dd0/dbf/d2f/d68/fb8 0 2024-07-26T11:04:53.296 INFO:tasks.workunit.client.0.smithi144.stdout:7/750: truncate d0/dd0/dbf/f41 3159950 0 2024-07-26T11:04:53.296 INFO:tasks.workunit.client.0.smithi144.stdout:5/742: rename d1/d18/d24/d27/f70 to d1/d9/d44/da1/ddb/f100 0 2024-07-26T11:04:53.297 INFO:tasks.workunit.client.0.smithi144.stdout:5/743: rmdir d1/d18/d24/d35 39 2024-07-26T11:04:53.297 INFO:tasks.workunit.client.0.smithi144.stdout:5/744: dread - d1/d9/d5b/fdd zero size 2024-07-26T11:04:53.297 INFO:tasks.workunit.client.0.smithi144.stdout:6/880: mknod d8/db/dbe/d52/ddb/c13a 0 2024-07-26T11:04:53.300 INFO:tasks.workunit.client.0.smithi144.stdout:5/745: symlink d1/d9/d44/da1/ddb/l101 0 2024-07-26T11:04:53.300 INFO:tasks.workunit.client.0.smithi144.stdout:1/930: creat d3/d6/da/d21/d5c/d62/d68/f12b x:0 0 0 2024-07-26T11:04:53.300 INFO:tasks.workunit.client.0.smithi144.stdout:6/881: mknod d8/db/dbe/d52/c13b 0 2024-07-26T11:04:53.302 INFO:tasks.workunit.client.0.smithi144.stdout:8/741: dread d1a/d32/f71 [0,4194304] 0 2024-07-26T11:04:53.316 INFO:tasks.workunit.client.0.smithi144.stdout:1/931: creat d3/d3d/d128/f12c x:0 0 0 2024-07-26T11:04:53.316 INFO:tasks.workunit.client.0.smithi144.stdout:1/932: dread - d3/d6/d75/f93 zero size 2024-07-26T11:04:53.350 INFO:tasks.workunit.client.0.smithi144.stdout:3/554: dwrite d1/f4f [4194304,4194304] 0 2024-07-26T11:04:53.360 INFO:tasks.workunit.client.0.smithi144.stdout:8/742: dread d1a/d32/d66/da0/ddc/fef [0,4194304] 0 2024-07-26T11:04:53.368 INFO:tasks.workunit.client.0.smithi144.stdout:6/882: dread d8/d15/f21 [0,4194304] 0 2024-07-26T11:04:53.385 INFO:tasks.workunit.client.0.smithi144.stdout:0/799: dwrite d0/d25/d5e/d1e/d30/f39 [0,4194304] 0 2024-07-26T11:04:53.388 INFO:tasks.workunit.client.0.smithi144.stdout:5/746: dwrite d1/d9/f5a [0,4194304] 0 2024-07-26T11:04:53.388 INFO:tasks.workunit.client.0.smithi144.stdout:0/800: symlink d0/d25/d5e/l10c 0 2024-07-26T11:04:53.388 INFO:tasks.workunit.client.0.smithi144.stdout:0/801: fdatasync d0/d3e/d45/fae 0 2024-07-26T11:04:53.388 INFO:tasks.workunit.client.0.smithi144.stdout:0/802: chown d0/d25/d5e/d1e/d30/d4b/c59 940050 1 2024-07-26T11:04:53.390 INFO:tasks.workunit.client.0.smithi144.stdout:5/747: creat d1/d18/dc4/ded/dc8/f102 x:0 0 0 2024-07-26T11:04:53.394 INFO:tasks.workunit.client.0.smithi144.stdout:6/883: getdents d8/db/dbe/d52/ddb 0 2024-07-26T11:04:53.394 INFO:tasks.workunit.client.0.smithi144.stdout:6/884: dread - d8/d15/d57/d5f/f10d zero size 2024-07-26T11:04:53.394 INFO:tasks.workunit.client.0.smithi144.stdout:6/885: write d8/d15/d57/d5f/dd3/fd6 [621884,108394] 0 2024-07-26T11:04:53.394 INFO:tasks.workunit.client.0.smithi144.stdout:6/886: write d8/d15/d26/f4c [1195595,43112] 0 2024-07-26T11:04:53.399 INFO:tasks.workunit.client.0.smithi144.stdout:6/887: creat d8/d50/d92/f13c x:0 0 0 2024-07-26T11:04:53.403 INFO:tasks.workunit.client.0.smithi144.stdout:9/811: dwrite d9/fc [0,4194304] 0 2024-07-26T11:04:53.408 INFO:tasks.workunit.client.0.smithi144.stdout:2/786: dwrite d0/d24/d7b/ddc/d28/d2e/d77/f40 [0,4194304] 0 2024-07-26T11:04:53.408 INFO:tasks.workunit.client.0.smithi144.stdout:6/888: symlink d8/db/dbe/d52/d58/l13d 0 2024-07-26T11:04:53.411 INFO:tasks.workunit.client.0.smithi144.stdout:6/889: mknod d8/d50/d92/dea/c13e 0 2024-07-26T11:04:53.415 INFO:tasks.workunit.client.0.smithi144.stdout:6/890: mknod d8/db/dbe/d74/c13f 0 2024-07-26T11:04:53.415 INFO:tasks.workunit.client.0.smithi144.stdout:6/891: readlink d8/d15/d57/l65 0 2024-07-26T11:04:53.416 INFO:tasks.workunit.client.0.smithi144.stdout:7/751: dwrite d0/dd0/dbf/d2f/f89 [0,4194304] 0 2024-07-26T11:04:53.418 INFO:tasks.workunit.client.0.smithi144.stdout:7/752: creat d0/f110 x:0 0 0 2024-07-26T11:04:53.423 INFO:tasks.workunit.client.0.smithi144.stdout:7/753: symlink d0/d71/l111 0 2024-07-26T11:04:53.423 INFO:tasks.workunit.client.0.smithi144.stdout:4/829: dwrite d9/d26/f2f [0,4194304] 0 2024-07-26T11:04:53.423 INFO:tasks.workunit.client.0.smithi144.stdout:8/743: dread d1a/d1c/fab [0,4194304] 0 2024-07-26T11:04:53.427 INFO:tasks.workunit.client.0.smithi144.stdout:4/830: mkdir d9/d7b/dc4/d111/d116 0 2024-07-26T11:04:53.427 INFO:tasks.workunit.client.0.smithi144.stdout:7/754: dread d0/dd0/dbf/d2f/d68/fc6 [0,4194304] 0 2024-07-26T11:04:53.427 INFO:tasks.workunit.client.0.smithi144.stdout:4/831: getdents d9/d26/d3c/d6f/dd0 0 2024-07-26T11:04:53.432 INFO:tasks.workunit.client.0.smithi144.stdout:7/755: mkdir d0/d112 0 2024-07-26T11:04:53.432 INFO:tasks.workunit.client.0.smithi144.stdout:7/756: chown d0/d6/d9/l36 21122784 1 2024-07-26T11:04:53.438 INFO:tasks.workunit.client.0.smithi144.stdout:7/757: write d0/dd0/dbf/fb2 [550652,16267] 0 2024-07-26T11:04:53.441 INFO:tasks.workunit.client.0.smithi144.stdout:7/758: symlink d0/d71/d95/de7/l113 0 2024-07-26T11:04:53.441 INFO:tasks.workunit.client.0.smithi144.stdout:3/555: dwrite d1/d5/d97/d48/f56 [0,4194304] 0 2024-07-26T11:04:53.442 INFO:tasks.workunit.client.0.smithi144.stdout:3/556: getdents d1/d28 0 2024-07-26T11:04:53.444 INFO:tasks.workunit.client.0.smithi144.stdout:3/557: mkdir d1/d5/d97/da9 0 2024-07-26T11:04:53.446 INFO:tasks.workunit.client.0.smithi144.stdout:3/558: creat d1/d28/d54/d26/d37/faa x:0 0 0 2024-07-26T11:04:53.456 INFO:tasks.workunit.client.0.smithi144.stdout:5/748: dwrite d1/d5/dae/fd3 [0,4194304] 0 2024-07-26T11:04:53.459 INFO:tasks.workunit.client.0.smithi144.stdout:9/812: dwrite d9/de/d14/f35 [0,4194304] 0 2024-07-26T11:04:53.463 INFO:tasks.workunit.client.0.smithi144.stdout:9/813: creat d9/d3b/dce/f10f x:0 0 0 2024-07-26T11:04:53.468 INFO:tasks.workunit.client.0.smithi144.stdout:4/832: dread d9/d26/d48/d50/f6c [0,4194304] 0 2024-07-26T11:04:53.468 INFO:tasks.workunit.client.0.smithi144.stdout:9/814: creat d9/d3b/d5d/f110 x:0 0 0 2024-07-26T11:04:53.468 INFO:tasks.workunit.client.0.smithi144.stdout:9/815: creat d9/de/d14/d23/d39/d7f/f111 x:0 0 0 2024-07-26T11:04:53.486 INFO:tasks.workunit.client.0.smithi144.stdout:7/759: dwrite d0/dd0/dbf/d2f/d53/fc5 [0,4194304] 0 2024-07-26T11:04:53.499 INFO:tasks.workunit.client.0.smithi144.stdout:6/892: dwrite d8/d15/f6b [0,4194304] 0 2024-07-26T11:04:53.507 INFO:tasks.workunit.client.0.smithi144.stdout:3/559: dwrite d1/d5/f35 [0,4194304] 0 2024-07-26T11:04:53.510 INFO:tasks.workunit.client.0.smithi144.stdout:3/560: mknod d1/d28/d6a/cab 0 2024-07-26T11:04:53.511 INFO:tasks.workunit.client.0.smithi144.stdout:5/749: dwrite d1/d18/da4/db3/fb8 [0,4194304] 0 2024-07-26T11:04:53.512 INFO:tasks.workunit.client.0.smithi144.stdout:3/561: mknod d1/d28/d54/d3d/d42/cac 0 2024-07-26T11:04:53.513 INFO:tasks.workunit.client.0.smithi144.stdout:5/750: mknod d1/d9/d5b/c103 0 2024-07-26T11:04:53.517 INFO:tasks.workunit.client.0.smithi144.stdout:4/833: dwrite d9/d26/f58 [0,4194304] 0 2024-07-26T11:04:53.517 INFO:tasks.workunit.client.0.smithi144.stdout:4/834: write d9/d26/d3c/d6f/d8c/ff4 [4953551,79456] 0 2024-07-26T11:04:53.519 INFO:tasks.workunit.client.0.smithi144.stdout:4/835: mkdir d9/d26/d48/d50/d60/d117 0 2024-07-26T11:04:53.521 INFO:tasks.workunit.client.0.smithi144.stdout:3/562: dread d1/d5/f1c [0,4194304] 0 2024-07-26T11:04:53.523 INFO:tasks.workunit.client.0.smithi144.stdout:3/563: creat d1/d5/d97/d48/fad x:0 0 0 2024-07-26T11:04:53.525 INFO:tasks.workunit.client.0.smithi144.stdout:3/564: mknod d1/d28/d54/d26/d64/da4/cae 0 2024-07-26T11:04:53.527 INFO:tasks.workunit.client.0.smithi144.stdout:3/565: truncate d1/f34 320642 0 2024-07-26T11:04:53.527 INFO:tasks.workunit.client.0.smithi144.stdout:3/566: chown d1/lc 954 1 2024-07-26T11:04:53.527 INFO:tasks.workunit.client.0.smithi144.stdout:3/567: readlink d1/d28/d54/d26/d37/l70 0 2024-07-26T11:04:53.539 INFO:tasks.workunit.client.0.smithi144.stdout:5/751: dread d1/d9/d44/f41 [0,4194304] 0 2024-07-26T11:04:53.558 INFO:tasks.workunit.client.0.smithi144.stdout:4/836: dwrite d9/d57/f77 [0,4194304] 0 2024-07-26T11:04:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:53 smithi160 ceph-mon[36715]: pgmap v533: 129 pgs: 129 active+clean; 1.3 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s rd, 156 MiB/s wr, 616 op/s 2024-07-26T11:04:53.567 INFO:tasks.workunit.client.0.smithi144.stdout:3/568: dwrite d1/f34 [0,4194304] 0 2024-07-26T11:04:53.570 INFO:tasks.workunit.client.0.smithi144.stdout:3/569: rename d1/d28/d54/d26/d37/faa to d1/d5/d97/d60/faf 0 2024-07-26T11:04:53.571 INFO:tasks.workunit.client.0.smithi144.stdout:3/570: write d1/d28/d54/d26/d64/d68/fa3 [878530,95899] 0 2024-07-26T11:04:53.573 INFO:tasks.workunit.client.0.smithi144.stdout:3/571: getdents d1/d28/d54/d3d 0 2024-07-26T11:04:53.573 INFO:tasks.workunit.client.0.smithi144.stdout:3/572: chown d1/d28/d54/d26/d30/d78 16553855 1 2024-07-26T11:04:53.576 INFO:tasks.workunit.client.0.smithi144.stdout:3/573: link d1/d5/d97/d60/f8c d1/d28/d54/d26/d64/da4/fb0 0 2024-07-26T11:04:53.576 INFO:tasks.workunit.client.0.smithi144.stdout:3/574: write d1/d28/d54/d26/f33 [578493,26163] 0 2024-07-26T11:04:53.582 INFO:tasks.workunit.client.0.smithi144.stdout:5/752: dwrite d1/d9/d44/da1/ddb/f100 [0,4194304] 0 2024-07-26T11:04:53.584 INFO:tasks.workunit.client.0.smithi144.stdout:5/753: symlink d1/d18/dc4/de0/l104 0 2024-07-26T11:04:53.584 INFO:tasks.workunit.client.0.smithi144.stdout:5/754: chown d1/d9 1 1 2024-07-26T11:04:53.584 INFO:tasks.workunit.client.0.smithi144.stdout:5/755: chown d1/d18/d24/d27/d71/f80 4609597 1 2024-07-26T11:04:53.584 INFO:tasks.workunit.client.0.smithi144.stdout:5/756: symlink d1/d9/d44/d45/d98/l105 0 2024-07-26T11:04:53.586 INFO:tasks.workunit.client.0.smithi144.stdout:5/757: creat d1/d9/d5b/f106 x:0 0 0 2024-07-26T11:04:53.586 INFO:tasks.workunit.client.0.smithi144.stdout:5/758: write d1/f97 [104690,37117] 0 2024-07-26T11:04:53.586 INFO:tasks.workunit.client.0.smithi144.stdout:5/759: write d1/d9/d44/d45/d81/f91 [635372,126128] 0 2024-07-26T11:04:53.587 INFO:tasks.workunit.client.0.smithi144.stdout:3/575: write d1/f95 [504549,99938] 0 2024-07-26T11:04:53.588 INFO:tasks.workunit.client.0.smithi144.stdout:5/760: rename d1/d18/d24/d35/f9f to d1/d9/d44/f107 0 2024-07-26T11:04:53.590 INFO:tasks.workunit.client.0.smithi144.stdout:3/576: unlink d1/d28/d6a/cab 0 2024-07-26T11:04:53.591 INFO:tasks.workunit.client.0.smithi144.stdout:5/761: rename d1/d9/d44/da1/ddb/f100 to d1/d18/f108 0 2024-07-26T11:04:53.593 INFO:tasks.workunit.client.0.smithi144.stdout:3/577: unlink d1/d5/f6 0 2024-07-26T11:04:53.594 INFO:tasks.workunit.client.0.smithi144.stdout:5/762: creat d1/d18/dc4/f109 x:0 0 0 2024-07-26T11:04:53.594 INFO:tasks.workunit.client.0.smithi144.stdout:5/763: write d1/d18/d24/d35/ff3 [375316,80629] 0 2024-07-26T11:04:53.594 INFO:tasks.workunit.client.0.smithi144.stdout:3/578: creat d1/d5/d97/da9/fb1 x:0 0 0 2024-07-26T11:04:53.595 INFO:tasks.workunit.client.0.smithi144.stdout:3/579: chown d1/d28/d54/d26/f2b 494 1 2024-07-26T11:04:53.597 INFO:tasks.workunit.client.0.smithi144.stdout:3/580: link d1/d28/d6a/c76 d1/d28/d54/d26/d30/cb2 0 2024-07-26T11:04:53.598 INFO:tasks.workunit.client.0.smithi144.stdout:3/581: symlink d1/d28/d54/d26/d64/da4/lb3 0 2024-07-26T11:04:53.600 INFO:tasks.workunit.client.0.smithi144.stdout:3/582: truncate d1/d28/d54/d26/f2b 2406408 0 2024-07-26T11:04:53.601 INFO:tasks.workunit.client.0.smithi144.stdout:3/583: mkdir d1/d28/d54/d26/d64/d68/db4 0 2024-07-26T11:04:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:53 smithi183 ceph-mon[37273]: pgmap v533: 129 pgs: 129 active+clean; 1.3 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s rd, 156 MiB/s wr, 616 op/s 2024-07-26T11:04:53.651 INFO:tasks.workunit.client.0.smithi144.stdout:5/764: dwrite d1/d9/d44/d45/fa2 [0,4194304] 0 2024-07-26T11:04:53.651 INFO:tasks.workunit.client.0.smithi144.stdout:3/584: dwrite d1/d5/d97/d60/f8c [0,4194304] 0 2024-07-26T11:04:53.653 INFO:tasks.workunit.client.0.smithi144.stdout:5/765: symlink d1/d9/d44/d85/dea/df9/l10a 0 2024-07-26T11:04:53.654 INFO:tasks.workunit.client.0.smithi144.stdout:5/766: getdents d1/d9/d44/d45/d98/db2 0 2024-07-26T11:04:53.654 INFO:tasks.workunit.client.0.smithi144.stdout:5/767: truncate d1/d18/da4/db3/fb8 4563432 0 2024-07-26T11:04:53.655 INFO:tasks.workunit.client.0.smithi144.stdout:5/768: fdatasync d1/d5/f78 0 2024-07-26T11:04:53.671 INFO:tasks.workunit.client.0.smithi144.stdout:3/585: write d1/d5/d97/d60/f8c [3659470,83598] 0 2024-07-26T11:04:53.671 INFO:tasks.workunit.client.0.smithi144.stdout:3/586: chown d1/d28/d7e 0 1 2024-07-26T11:04:53.673 INFO:tasks.workunit.client.0.smithi144.stdout:3/587: mkdir d1/d28/d54/d26/d64/d68/db5 0 2024-07-26T11:04:53.676 INFO:tasks.workunit.client.0.smithi144.stdout:5/769: dread d1/d18/fe2 [0,4194304] 0 2024-07-26T11:04:53.676 INFO:tasks.workunit.client.0.smithi144.stdout:5/770: write d1/d9/d44/d45/f56 [1231863,130440] 0 2024-07-26T11:04:53.678 INFO:tasks.workunit.client.0.smithi144.stdout:5/771: getdents d1/d18/da4/db3 0 2024-07-26T11:04:53.678 INFO:tasks.workunit.client.0.smithi144.stdout:5/772: truncate d1/d18/d24/d27/d71/f80 4281004 0 2024-07-26T11:04:53.678 INFO:tasks.workunit.client.0.smithi144.stdout:5/773: readlink d1/d9/l30 0 2024-07-26T11:04:53.679 INFO:tasks.workunit.client.0.smithi144.stdout:5/774: symlink d1/d9/l10b 0 2024-07-26T11:04:53.680 INFO:tasks.workunit.client.0.smithi144.stdout:5/775: fdatasync d1/d18/f108 0 2024-07-26T11:04:53.682 INFO:tasks.workunit.client.0.smithi144.stdout:5/776: unlink d1/d9/d5b/d4d/fa5 0 2024-07-26T11:04:53.682 INFO:tasks.workunit.client.0.smithi144.stdout:5/777: fdatasync d1/d9/d5b/d4d/f63 0 2024-07-26T11:04:53.682 INFO:tasks.workunit.client.0.smithi144.stdout:5/778: chown d1/d9/d44/la7 6 1 2024-07-26T11:04:53.682 INFO:tasks.workunit.client.0.smithi144.stdout:5/779: chown d1/d9/d5b/d4d/f63 118958922 1 2024-07-26T11:04:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:53 smithi144 ceph-mon[32129]: pgmap v533: 129 pgs: 129 active+clean; 1.3 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s rd, 156 MiB/s wr, 616 op/s 2024-07-26T11:04:53.726 INFO:tasks.workunit.client.0.smithi144.stdout:3/588: dwrite d1/d28/d54/d26/d64/da4/fb0 [0,4194304] 0 2024-07-26T11:04:53.727 INFO:tasks.workunit.client.0.smithi144.stdout:3/589: symlink d1/d28/d54/d26/d64/d89/lb6 0 2024-07-26T11:04:53.729 INFO:tasks.workunit.client.0.smithi144.stdout:3/590: mknod d1/d28/d54/d3d/d42/cb7 0 2024-07-26T11:04:53.730 INFO:tasks.workunit.client.0.smithi144.stdout:5/780: dwrite d1/d18/fdf [0,4194304] 0 2024-07-26T11:04:53.733 INFO:tasks.workunit.client.0.smithi144.stdout:5/781: symlink d1/d9/d44/d45/d81/dd4/l10c 0 2024-07-26T11:04:53.738 INFO:tasks.workunit.client.0.smithi144.stdout:5/782: read d1/d18/d24/fa0 [40062,11522] 0 2024-07-26T11:04:53.740 INFO:tasks.workunit.client.0.smithi144.stdout:5/783: rename d1/d9/d44/da1/fb0 to d1/d9/d44/da1/ddb/df7/f10d 0 2024-07-26T11:04:53.741 INFO:tasks.workunit.client.0.smithi144.stdout:5/784: creat d1/d18/d24/d27/f10e x:0 0 0 2024-07-26T11:04:53.741 INFO:tasks.workunit.client.0.smithi144.stdout:5/785: write d1/d5/f5c [2948463,18416] 0 2024-07-26T11:04:53.743 INFO:tasks.workunit.client.0.smithi144.stdout:5/786: rename d1/d9/d44/d85 to d1/d18/d24/d35/d10f 0 2024-07-26T11:04:53.744 INFO:tasks.workunit.client.0.smithi144.stdout:5/787: mkdir d1/d9/d110 0 2024-07-26T11:04:53.745 INFO:tasks.workunit.client.0.smithi144.stdout:5/788: getdents d1/da8 0 2024-07-26T11:04:53.746 INFO:tasks.workunit.client.0.smithi144.stdout:5/789: creat d1/d9/d44/d45/d81/dd4/f111 x:0 0 0 2024-07-26T11:04:53.746 INFO:tasks.workunit.client.0.smithi144.stdout:5/790: rmdir d1/d18/dc4 39 2024-07-26T11:04:53.794 INFO:tasks.workunit.client.0.smithi144.stdout:5/791: dwrite d1/d18/d24/d35/d10f/fbc [0,4194304] 0 2024-07-26T11:04:53.796 INFO:tasks.workunit.client.0.smithi144.stdout:5/792: rename d1/d18/d24/d27/d71/fac to d1/d9/d44/d45/f112 0 2024-07-26T11:04:53.797 INFO:tasks.workunit.client.0.smithi144.stdout:5/793: truncate d1/d5/f2d 8023133 0 2024-07-26T11:04:53.798 INFO:tasks.workunit.client.0.smithi144.stdout:5/794: symlink d1/d18/dc4/de0/deb/l113 0 2024-07-26T11:04:53.799 INFO:tasks.workunit.client.0.smithi144.stdout:5/795: getdents d1/d18/d24/d35/dd7 0 2024-07-26T11:04:53.799 INFO:tasks.workunit.client.0.smithi144.stdout:5/796: dread - d1/d9/d5b/f106 zero size 2024-07-26T11:04:53.799 INFO:tasks.workunit.client.0.smithi144.stdout:5/797: readlink d1/d9/d5b/l7e 0 2024-07-26T11:04:55.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:55 smithi144 ceph-mon[32129]: pgmap v534: 129 pgs: 129 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s rd, 155 MiB/s wr, 577 op/s 2024-07-26T11:04:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:55 smithi160 ceph-mon[36715]: pgmap v534: 129 pgs: 129 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s rd, 155 MiB/s wr, 577 op/s 2024-07-26T11:04:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:55 smithi183 ceph-mon[37273]: pgmap v534: 129 pgs: 129 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s rd, 155 MiB/s wr, 577 op/s 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 -- 172.21.15.144:0/2979229415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 msgr2=0x7fcafc105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 --2- 172.21.15.144:0/2979229415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc105bc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009900 tx=0x7fcaec02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 -- 172.21.15.144:0/2979229415 shutdown_connections 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 --2- 172.21.15.144:0/2979229415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 0x7fcafc112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 --2- 172.21.15.144:0/2979229415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 0x7fcafc10b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 --2- 172.21.15.144:0/2979229415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc105bc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.732+0000 7fcb01a00640 1 -- 172.21.15.144:0/2979229415 >> 172.21.15.144:0/2979229415 conn(0x7fcafc101240 msgr2=0x7fcafc103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.733+0000 7fcb01a00640 1 -- 172.21.15.144:0/2979229415 shutdown_connections 2024-07-26T11:04:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.733+0000 7fcb01a00640 1 -- 172.21.15.144:0/2979229415 wait complete. 2024-07-26T11:04:55.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.734+0000 7fcb01a00640 1 Processor -- start 2024-07-26T11:04:55.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.734+0000 7fcb01a00640 1 -- start start 2024-07-26T11:04:55.736 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcb01a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc19c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:55.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcb01a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 0x7fcafc19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:55.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcb01a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 0x7fcafc1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:55.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcb01a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc1156f0 con 0x7fcafc1057b0 2024-07-26T11:04:55.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcb01a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc115570 con 0x7fcafc10ba40 2024-07-26T11:04:55.737 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcb01a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc115870 con 0x7fcafc10b0f0 2024-07-26T11:04:55.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcafaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc19c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:55.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcaf3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 0x7fcafc19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:55.738 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcafaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc19c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:56148/0 (socket says 172.21.15.144:56148) 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcafaa00640 1 -- 172.21.15.144:0/1460107483 learned_addr learned my addr 172.21.15.144:0/1460107483 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.735+0000 7fcafb400640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 0x7fcafc1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3e00640 1 -- 172.21.15.144:0/1460107483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 msgr2=0x7fcafc1a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3e00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 0x7fcafc1a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3e00640 1 -- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 msgr2=0x7fcafc19c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3e00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc19c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3e00640 1 -- 172.21.15.144:0/1460107483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcafc1a2130 con 0x7fcafc10b0f0 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcafb400640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 0x7fcafc1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcafaa00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc19c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:55.739 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3e00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 0x7fcafc19c760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcae4005fd0 tx=0x7fcae400dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:55.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.736+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4009c00 con 0x7fcafc10b0f0 2024-07-26T11:04:55.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.737+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcafc1a2420 con 0x7fcafc10b0f0 2024-07-26T11:04:55.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.737+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcae400be30 con 0x7fcafc10b0f0 2024-07-26T11:04:55.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.737+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4003d00 con 0x7fcafc10b0f0 2024-07-26T11:04:55.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.737+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcafc1a8b80 con 0x7fcafc10b0f0 2024-07-26T11:04:55.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.738+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcafc105bc0 con 0x7fcafc10b0f0 2024-07-26T11:04:55.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.740+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fcae4003ea0 con 0x7fcafc10b0f0 2024-07-26T11:04:55.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.742+0000 7fcaf3400640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcae00b3d40 0x7fcae00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:55.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.742+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fcae40e3ea0 con 0x7fcafc10b0f0 2024-07-26T11:04:55.744 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.742+0000 7fcafaa00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcae00b3d40 0x7fcae00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:55.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.743+0000 7fcafaa00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcae00b3d40 0x7fcae00b6250 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcaec031440 tx=0x7fcaec0058d0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:55.745 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.744+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcae4017020 con 0x7fcafc10b0f0 2024-07-26T11:04:55.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.985+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcafc19d2f0 con 0x7fcafc10b0f0 2024-07-26T11:04:55.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.986+0000 7fcaf3400640 1 -- 172.21.15.144:0/1460107483 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11934 (secure 0 0 0) 0x7fcae40aade0 con 0x7fcafc10b0f0 2024-07-26T11:04:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:04:55.989 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":27,"btime":"2024-07-26T11:04:00:389009+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T11:04:00.338790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24728,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24728":{"gid":24728,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.160:6839/1074049609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1074049609},{"type":"v1","addr":"172.21.15.160:6839","nonce":1074049609}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24728,"qdb_cluster":[24728,24715,24703]},"id":1}]} 2024-07-26T11:04:55.989 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 27 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.989+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcae00b3d40 msgr2=0x7fcae00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.989+0000 7fcb01a00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcae00b3d40 0x7fcae00b6250 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcaec031440 tx=0x7fcaec0058d0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 msgr2=0x7fcafc19c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 0x7fcafc19c760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcae4005fd0 tx=0x7fcae400dd20 comp rx=0 tx=0).stop 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 shutdown_connections 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcae00b3d40 0x7fcae00b6250 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcafc10ba40 0x7fcafc1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcafc10b0f0 0x7fcafc19c760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 --2- 172.21.15.144:0/1460107483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc1057b0 0x7fcafc19c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 >> 172.21.15.144:0/1460107483 conn(0x7fcafc101240 msgr2=0x7fcafc103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 shutdown_connections 2024-07-26T11:04:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:04:55.990+0000 7fcb01a00640 1 -- 172.21.15.144:0/1460107483 wait complete. 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.285+0000 7fd743e00640 1 -- 172.21.15.144:0/309014301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c108e60 msgr2=0x7fd73c109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.285+0000 7fd743e00640 1 --2- 172.21.15.144:0/309014301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c108e60 0x7fd73c109270 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd728009860 tx=0x7fd72802a340 comp rx=0 tx=0).stop 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 -- 172.21.15.144:0/309014301 shutdown_connections 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 --2- 172.21.15.144:0/309014301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c1097b0 0x7fd73c10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 --2- 172.21.15.144:0/309014301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c108e60 0x7fd73c109270 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 --2- 172.21.15.144:0/309014301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 -- 172.21.15.144:0/309014301 >> 172.21.15.144:0/309014301 conn(0x7fd73c0fef30 msgr2=0x7fd73c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 -- 172.21.15.144:0/309014301 shutdown_connections 2024-07-26T11:04:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.286+0000 7fd743e00640 1 -- 172.21.15.144:0/309014301 wait complete. 2024-07-26T11:04:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 Processor -- start 2024-07-26T11:04:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 -- start start 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c108e60 0x7fd73c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1097b0 0x7fd73c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c113420 con 0x7fd73c103530 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1132a0 con 0x7fd73c108e60 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.287+0000 7fd743e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1135a0 con 0x7fd73c1097b0 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd741400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd741e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1097b0 0x7fd73c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd740a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c108e60 0x7fd73c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd741e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1097b0 0x7fd73c19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54888/0 (socket says 172.21.15.144:54888) 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd741400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:56176/0 (socket says 172.21.15.144:56176) 2024-07-26T11:04:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd741e00640 1 -- 172.21.15.144:0/1534103607 learned_addr learned my addr 172.21.15.144:0/1534103607 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd740a00640 1 -- 172.21.15.144:0/1534103607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1097b0 msgr2=0x7fd73c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd740a00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1097b0 0x7fd73c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd740a00640 1 -- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 msgr2=0x7fd73c199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd740a00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd740a00640 1 -- 172.21.15.144:0/1534103607 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c19fdf0 con 0x7fd73c108e60 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.288+0000 7fd741400640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.289+0000 7fd740a00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c108e60 0x7fd73c19a460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd72802a860 tx=0x7fd728032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.289+0000 7fd73a000640 1 -- 172.21.15.144:0/1534103607 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728040020 con 0x7fd73c108e60 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.289+0000 7fd73a000640 1 -- 172.21.15.144:0/1534103607 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd728030e10 con 0x7fd73c108e60 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.289+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c1a0080 con 0x7fd73c108e60 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.289+0000 7fd73a000640 1 -- 172.21.15.144:0/1534103607 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728037040 con 0x7fd73c108e60 2024-07-26T11:04:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.289+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c1a6870 con 0x7fd73c108e60 2024-07-26T11:04:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.292+0000 7fd74adc8740 1 Processor -- start 2024-07-26T11:04:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 -- start start 2024-07-26T11:04:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd73a000640 1 -- 172.21.15.144:0/1534103607 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd728043020 con 0x7fd73c108e60 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31b96a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31b96f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31b974c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563d31b0b270 con 0x563d317f31b0 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563d31afa770 con 0x563d317ca2f0 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.293+0000 7fd74adc8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563d31afe7b0 con 0x563d317d24d0 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31b96a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31b96a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:56178/0 (socket says 172.21.15.144:56178) 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31b96f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 -- 172.21.15.144:0/1190772340 learned_addr learned my addr 172.21.15.144:0/1190772340 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd740a00640 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31b974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 -- 172.21.15.144:0/1190772340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 msgr2=0x563d31b974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31b974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 -- 172.21.15.144:0/1190772340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 msgr2=0x563d31b96f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31b96f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 -- 172.21.15.144:0/1190772340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3185e790 con 0x563d317f31b0 2024-07-26T11:04:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741400640 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31b96f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.294+0000 7fd741e00640 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31b96a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd734009d60 tx=0x7fd734011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cd760a88ce83d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.295+0000 7fd72e600640 1 -- 172.21.15.144:0/1190772340 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73401e070 con 0x563d317f31b0 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.295+0000 7fd72e600640 1 -- 172.21.15.144:0/1190772340 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd734019ae0 con 0x563d317f31b0 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.295+0000 7fd72e600640 1 -- 172.21.15.144:0/1190772340 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73400fa90 con 0x563d317f31b0 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.295+0000 7fd73a000640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd7240b3d40 0x7fd7240b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.295+0000 7fd73a000640 1 -- 172.21.15.144:0/1534103607 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd728103e90 con 0x7fd73c108e60 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.295+0000 7fd741400640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd7240b3d40 0x7fd7240b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 -- 172.21.15.144:0/1190772340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 msgr2=0x563d31b96a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31b96a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd734009d60 tx=0x7fd734011810 comp rx=0 tx=0).stop 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd741400640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd7240b3d40 0x7fd7240b6250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x563d31b956c0 tx=0x7fd7300095c0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 -- 172.21.15.144:0/1190772340 shutdown_connections 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31b974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31b96f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 --2- 172.21.15.144:0/1190772340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31b96a40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 -- 172.21.15.144:0/1190772340 >> 172.21.15.144:0/1190772340 conn(0x563d317e7300 msgr2=0x563d31b94930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 -- 172.21.15.144:0/1190772340 shutdown_connections 2024-07-26T11:04:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.296+0000 7fd74adc8740 1 -- 172.21.15.144:0/1190772340 wait complete. 2024-07-26T11:04:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.297+0000 7fd74adc8740 1 -- start start 2024-07-26T11:04:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.297+0000 7fd74adc8740 1 Processor -- start 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.297+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 0x563d31ba6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31ba70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317f31b0 0x563d31bb28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd74adc8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563d31a34ab0 con 0x563d317ca2f0 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd74adc8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563d31a97d50 con 0x563d317f31b0 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd74adc8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563d31ba95a0 con 0x563d317d24d0 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd74adc8740 10 client.0 _ll_register_callbacks cb 0x563d3180aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 0x563d31ba6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 0x563d31ba6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:56194/0 (socket says 172.21.15.144:56194) 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 -- 172.21.15.144:0/795261688 learned_addr learned my addr 172.21.15.144:0/795261688 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd740a00640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31ba70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741e00640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317f31b0 0x563d31bb28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 -- 172.21.15.144:0/795261688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 msgr2=0x563d31ba70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31ba70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 -- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317f31b0 msgr2=0x563d31bb28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317f31b0 0x563d31bb28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.298+0000 7fd741400640 1 -- 172.21.15.144:0/795261688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d318d7620 con 0x563d317ca2f0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd741400640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 0x563d31ba6ad0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd73000bf50 tx=0x7fd730004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3e03b27bd2bce0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd722a00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd722a00640 1 -- 172.21.15.144:0/795261688 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730029020 con 0x563d317ca2f0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd722a00640 1 -- 172.21.15.144:0/795261688 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd7300255f0 con 0x563d317ca2f0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd734019ae0 con 0x563d317ca2f0 2024-07-26T11:04:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd722a00640 1 -- 172.21.15.144:0/795261688 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd730034050 con 0x563d317ca2f0 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd74adc8740 10 client.15714 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.299+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563d31bb5710 con 0x563d317ca2f0 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd722a00640 1 -- 172.21.15.144:0/795261688 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd734019ae0 con 0x563d317ca2f0 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd722a00640 1 -- 172.21.15.144:0/795261688 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd730024030 con 0x563d317ca2f0 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd74adc8740 10 client.15714 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd74adc8740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd74adc8740 20 client.15714 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd74adc8740 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317b1d80 0x563d31bb8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd74adc8740 4 client.15714 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:04:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d31b0d1c0 con 0x563d317b1d80 2024-07-26T11:04:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.300+0000 7fd740a00640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317b1d80 0x563d31bb8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.301+0000 7fd740a00640 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317b1d80 0x563d31bb8d90 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75e7bff0ac378fda server_cookie=94505887fc8be1e in_seq=0 out_seq=0 2024-07-26T11:04:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.301+0000 7fd722a00640 10 client.15714 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd722a00640 1 -- 172.21.15.144:0/795261688 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563d31b0d1c0 con 0x563d317b1d80 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd722a00640 10 client.15714 handle_command_reply: tid=0 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 client.15714 shutdown 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 msgr2=0x563d31ba6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 0x563d31ba6ad0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd73000bf50 tx=0x7fd730004ab0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 shutdown_connections 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317b1d80 0x563d31bb8d90 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317f31b0 0x563d31bb28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317d24d0 0x563d31ba70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 --2- 172.21.15.144:0/795261688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317ca2f0 0x563d31ba6ad0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd722a00640 0 client.15714 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 >> 172.21.15.144:0/795261688 conn(0x563d317e7300 msgr2=0x563d31b930b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 shutdown_connections 2024-07-26T11:04:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 1 -- 172.21.15.144:0/795261688 wait complete. 2024-07-26T11:04:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.302+0000 7fd74adc8740 20 client.15714 trim_cache size 0 max 16384 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 Processor -- start 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 -- start start 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317ca2f0 0x563d31b93720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 0x563d31b94960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31bb26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1b0f00 con 0x563d317f31b0 2024-07-26T11:04:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1ae7d0 con 0x563d317d24d0 2024-07-26T11:04:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.319+0000 7fd74adc8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1ac620 con 0x563d317ca2f0 2024-07-26T11:04:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 0x563d31b94960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd741e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31bb26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 0x563d31b94960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51100/0 (socket says 172.21.15.144:51100) 2024-07-26T11:04:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 -- 172.21.15.144:0/885045624 learned_addr learned my addr 172.21.15.144:0/885045624 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 -- 172.21.15.144:0/885045624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317ca2f0 msgr2=0x563d31b93720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317ca2f0 0x563d31b93720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 -- 172.21.15.144:0/885045624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 msgr2=0x563d31bb26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31bb26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 -- 172.21.15.144:0/885045624 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d31b9c4a0 con 0x563d317d24d0 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd740a00640 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 0x563d31b94960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7281076e0 tx=0x7fd7281062e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338c31e1a40d87db server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.320+0000 7fd72dc00640 1 -- 172.21.15.144:0/885045624 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280ca5b0 con 0x563d317d24d0 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd72dc00640 1 -- 172.21.15.144:0/885045624 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7280d31d0 con 0x563d317d24d0 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd72dc00640 1 -- 172.21.15.144:0/885045624 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280c9f50 con 0x563d317d24d0 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 -- 172.21.15.144:0/885045624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 msgr2=0x563d31b94960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 0x563d31b94960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7281076e0 tx=0x7fd7281062e0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 -- 172.21.15.144:0/885045624 shutdown_connections 2024-07-26T11:04:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317f31b0 0x563d31bb26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317d24d0 0x563d31b94960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 --2- 172.21.15.144:0/885045624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317ca2f0 0x563d31b93720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.321+0000 7fd74adc8740 1 -- 172.21.15.144:0/885045624 >> 172.21.15.144:0/885045624 conn(0x563d317e7300 msgr2=0x563d31b94250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.322+0000 7fd74adc8740 1 -- 172.21.15.144:0/885045624 shutdown_connections 2024-07-26T11:04:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.322+0000 7fd74adc8740 1 -- 172.21.15.144:0/885045624 wait complete. 2024-07-26T11:04:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.322+0000 7fd74adc8740 1 -- start start 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.322+0000 7fd74adc8740 1 Processor -- start 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31baf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 0x563d31baa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317f31b0 0x563d31baac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1acc90 con 0x563d317d24d0 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1ae100 con 0x563d317ca2f0 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1ad360 con 0x563d317f31b0 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd74adc8740 10 client.0 _ll_register_callbacks cb 0x7fd730009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd740a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 0x563d31baa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317f31b0 0x563d31baac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317f31b0 0x563d31baac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54910/0 (socket says 172.21.15.144:54910) 2024-07-26T11:04:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31baf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd740a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 0x563d31baa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:56210/0 (socket says 172.21.15.144:56210) 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741e00640 1 -- 172.21.15.144:0/335512904 learned_addr learned my addr 172.21.15.144:0/335512904 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741400640 1 -- 172.21.15.144:0/335512904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317f31b0 msgr2=0x563d31baac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741400640 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317f31b0 0x563d31baac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741400640 1 -- 172.21.15.144:0/335512904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 msgr2=0x563d31baa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741400640 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 0x563d31baa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd741400640 1 -- 172.21.15.144:0/335512904 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d31c20520 con 0x563d317ca2f0 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.323+0000 7fd740a00640 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 0x563d31baa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd741400640 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31baf0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd730019a50 tx=0x7fd730019d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc4f325175a70e68 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd722a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd722a00640 1 -- 172.21.15.144:0/335512904 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73003c030 con 0x563d317ca2f0 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd722a00640 1 -- 172.21.15.144:0/335512904 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fd7300195e0 con 0x563d317ca2f0 2024-07-26T11:04:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563d31b08c60 con 0x563d317ca2f0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd722a00640 1 -- 172.21.15.144:0/335512904 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd730046050 con 0x563d317ca2f0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd74adc8740 10 client.25877 fetch_fsmap learned FSMap version 27 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.324+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563d31c21650 con 0x563d317ca2f0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd722a00640 1 -- 172.21.15.144:0/335512904 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563d31b08c60 con 0x563d317ca2f0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd722a00640 1 -- 172.21.15.144:0/335512904 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd730024040 con 0x563d317ca2f0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd74adc8740 10 client.25877 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd74adc8740 10 client.25877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd74adc8740 20 client.25877 populate_metadata read hostname 'smithi144' 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd74adc8740 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317871c0 0x563d31c23ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd74adc8740 4 client.25877 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.325+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d31b063a0 con 0x563d317871c0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.326+0000 7fd740a00640 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317871c0 0x563d31c23ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.326+0000 7fd740a00640 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317871c0 0x563d31c23ec0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c7563936be4faa2 server_cookie=5729ce7796be350c in_seq=0 out_seq=0 2024-07-26T11:04:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.326+0000 7fd722a00640 10 client.25877 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:56 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/1460107483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:04:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:56 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/1460107483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:04:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd722a00640 1 -- 172.21.15.144:0/335512904 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d31b063a0 con 0x563d317871c0 2024-07-26T11:04:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd722a00640 10 client.25877 handle_command_reply: tid=0 2024-07-26T11:04:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 client.25877 shutdown 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 msgr2=0x563d31baf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31baf0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd730019a50 tx=0x7fd730019d80 comp rx=0 tx=0).stop 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 shutdown_connections 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563d317871c0 0x563d31c23ec0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563d317f31b0 0x563d31baac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563d317d24d0 0x563d31baa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd722a00640 0 client.25877 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 --2- 172.21.15.144:0/335512904 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563d317ca2f0 0x563d31baf0d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 >> 172.21.15.144:0/335512904 conn(0x563d317e7300 msgr2=0x563d31b97a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 shutdown_connections 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.672+0000 7fd74adc8740 1 -- 172.21.15.144:0/335512904 wait complete. 2024-07-26T11:04:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.673+0000 7fd74adc8740 20 client.25877 trim_cache size 0 max 16384 2024-07-26T11:04:56.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:04:56.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:04:56.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:04:56.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:04:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd7240b3d40 msgr2=0x7fd7240b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd7240b3d40 0x7fd7240b6250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x563d31b956c0 tx=0x7fd7300095c0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c108e60 msgr2=0x7fd73c19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c108e60 0x7fd73c19a460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd72802a860 tx=0x7fd728032d50 comp rx=0 tx=0).stop 2024-07-26T11:04:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 shutdown_connections 2024-07-26T11:04:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd7240b3d40 0x7fd7240b6250 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1097b0 0x7fd73c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd73c108e60 0x7fd73c19a460 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 --2- 172.21.15.144:0/1534103607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c103530 0x7fd73c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.674+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 >> 172.21.15.144:0/1534103607 conn(0x7fd73c0fef30 msgr2=0x7fd73c108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.675+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 shutdown_connections 2024-07-26T11:04:56.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:04:56.675+0000 7fd743e00640 1 -- 172.21.15.144:0/1534103607 wait complete. 2024-07-26T11:04:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:56 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/1460107483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:04:56.895 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:04:57.402 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:04:57.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:57 smithi144 ceph-mon[32129]: pgmap v535: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s rd, 148 MiB/s wr, 426 op/s 2024-07-26T11:04:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:57 smithi160 ceph-mon[36715]: pgmap v535: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s rd, 148 MiB/s wr, 426 op/s 2024-07-26T11:04:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:57 smithi183 ceph-mon[37273]: pgmap v535: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s rd, 148 MiB/s wr, 426 op/s 2024-07-26T11:04:57.894 INFO:tasks.workunit.client.0.smithi144.stdout:3/591: sync 2024-07-26T11:04:57.894 INFO:tasks.workunit.client.0.smithi144.stdout:5/798: sync 2024-07-26T11:04:57.894 INFO:tasks.workunit.client.0.smithi144.stdout:8/744: sync 2024-07-26T11:04:57.894 INFO:tasks.workunit.client.0.smithi144.stdout:6/893: sync 2024-07-26T11:04:57.894 INFO:tasks.workunit.client.0.smithi144.stdout:6/894: truncate d8/d50/d92/db3/d103/d125/f129 845931 0 2024-07-26T11:04:57.894 INFO:tasks.workunit.client.0.smithi144.stdout:1/933: sync 2024-07-26T11:04:57.895 INFO:tasks.workunit.client.0.smithi144.stdout:4/837: sync 2024-07-26T11:04:57.895 INFO:tasks.workunit.client.0.smithi144.stdout:0/803: sync 2024-07-26T11:04:57.895 INFO:tasks.workunit.client.0.smithi144.stdout:2/787: sync 2024-07-26T11:04:57.895 INFO:tasks.workunit.client.0.smithi144.stdout:1/934: stat d3/d3d/d58/d101/d36/db6/le5 0 2024-07-26T11:04:57.895 INFO:tasks.workunit.client.0.smithi144.stdout:7/760: sync 2024-07-26T11:04:57.895 INFO:tasks.workunit.client.0.smithi144.stdout:9/816: sync 2024-07-26T11:04:57.900 INFO:tasks.workunit.client.0.smithi144.stdout:1/935: creat d3/d6/d75/d88/da9/f12d x:0 0 0 2024-07-26T11:04:57.900 INFO:tasks.workunit.client.0.smithi144.stdout:6/895: link d8/db/f2c d8/d50/d98/d104/f140 0 2024-07-26T11:04:57.900 INFO:tasks.workunit.client.0.smithi144.stdout:1/936: write d3/d3d/d58/d101/d36/f55 [2103917,72041] 0 2024-07-26T11:04:57.900 INFO:tasks.workunit.client.0.smithi144.stdout:6/896: chown d8/d50/d92/db3 572608946 1 2024-07-26T11:04:57.901 INFO:tasks.workunit.client.0.smithi144.stdout:6/897: readlink d8/d15/d57/d5f/d6e/ld7 0 2024-07-26T11:04:57.901 INFO:tasks.workunit.client.0.smithi144.stdout:1/937: chown d3/d6/da/d13/d19/d3a/l89 204964702 1 2024-07-26T11:04:57.901 INFO:tasks.workunit.client.0.smithi144.stdout:3/592: creat d1/d5/d97/d60/fb8 x:0 0 0 2024-07-26T11:04:57.901 INFO:tasks.workunit.client.0.smithi144.stdout:3/593: fsync d1/d28/d54/d26/f2e 0 2024-07-26T11:04:57.901 INFO:tasks.workunit.client.0.smithi144.stdout:5/799: creat d1/d18/d24/d27/d71/f114 x:0 0 0 2024-07-26T11:04:57.901 INFO:tasks.workunit.client.0.smithi144.stdout:5/800: readlink d1/d9/d44/d45/le3 0 2024-07-26T11:04:57.904 INFO:tasks.workunit.client.0.smithi144.stdout:8/745: rename d1a/d32/d49/d73/d7b/lf9 to d1a/d32/d49/d73/d7b/d43/d4f/lfc 0 2024-07-26T11:04:57.904 INFO:tasks.workunit.client.0.smithi144.stdout:8/746: fsync d1a/d32/d49/d73/d2c/f8a 0 2024-07-26T11:04:57.905 INFO:tasks.workunit.client.0.smithi144.stdout:2/788: getdents d0/d24/d2b 0 2024-07-26T11:04:57.905 INFO:tasks.workunit.client.0.smithi144.stdout:2/789: truncate d0/d24/d48/d63/fa7 4894915 0 2024-07-26T11:04:57.905 INFO:tasks.workunit.client.0.smithi144.stdout:2/790: fdatasync d0/d24/d48/d63/fa7 0 2024-07-26T11:04:57.907 INFO:tasks.workunit.client.0.smithi144.stdout:4/838: mknod d9/d26/d3c/d6f/dd0/c118 0 2024-07-26T11:04:57.919 INFO:tasks.workunit.client.0.smithi144.stdout:6/898: creat d8/d50/d92/db3/f141 x:0 0 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:6/899: chown d8/d15/d26/f121 194 1 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:6/900: fsync d8/f33 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:7/761: mkdir d0/d6/d9/d3f/dd3/d114 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:3/594: mkdir d1/d28/d54/d61/db9 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:3/595: fdatasync d1/d5/d97/f67 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:5/801: rename d1/d9/f19 to d1/d9/d44/d45/d98/db2/f115 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:3/596: write d1/ff [2538343,63697] 0 2024-07-26T11:04:57.920 INFO:tasks.workunit.client.0.smithi144.stdout:7/762: write d0/dd0/dbf/d2f/d47/d5e/f63 [797306,54385] 0 2024-07-26T11:04:57.921 INFO:tasks.workunit.client.0.smithi144.stdout:7/763: write d0/d71/d95/dc4/f97 [924277,79402] 0 2024-07-26T11:04:57.921 INFO:tasks.workunit.client.0.smithi144.stdout:7/764: chown d0/dd0/dbf/d2f/da2/lb0 1279241 1 2024-07-26T11:04:57.921 INFO:tasks.workunit.client.0.smithi144.stdout:3/597: write d1/d28/d54/d26/d30/d78/fa5 [408302,121388] 0 2024-07-26T11:04:57.921 INFO:tasks.workunit.client.0.smithi144.stdout:3/598: read d1/d28/d54/d26/d64/d68/f84 [850809,12364] 0 2024-07-26T11:04:57.921 INFO:tasks.workunit.client.0.smithi144.stdout:8/747: mkdir d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/dfd 0 2024-07-26T11:04:57.937 INFO:tasks.workunit.client.0.smithi144.stdout:2/791: creat d0/d6/d34/ff1 x:0 0 0 2024-07-26T11:04:57.937 INFO:tasks.workunit.client.0.smithi144.stdout:2/792: dread - d0/d24/d7b/ddc/d28/d27/d8b/fe1 zero size 2024-07-26T11:04:57.937 INFO:tasks.workunit.client.0.smithi144.stdout:5/802: mkdir d1/d18/dc4/de0/deb/d116 0 2024-07-26T11:04:57.937 INFO:tasks.workunit.client.0.smithi144.stdout:9/817: unlink d9/de/d14/d23/d39/lad 0 2024-07-26T11:04:57.937 INFO:tasks.workunit.client.0.smithi144.stdout:9/818: rename d9/de/d14/d3f/d5e/dcd to d9/de/d14/d23/d112 0 2024-07-26T11:04:57.937 INFO:tasks.workunit.client.0.smithi144.stdout:9/819: getdents d9/de/d4e/d106 0 2024-07-26T11:04:57.938 INFO:tasks.workunit.client.0.smithi144.stdout:4/839: write f1 [803910,74628] 0 2024-07-26T11:04:57.938 INFO:tasks.workunit.client.0.smithi144.stdout:4/840: dread - d9/d57/feb zero size 2024-07-26T11:04:57.958 INFO:tasks.workunit.client.0.smithi144.stdout:4/841: truncate d9/d26/d3c/d6f/f7c 2811790 0 2024-07-26T11:04:57.958 INFO:tasks.workunit.client.0.smithi144.stdout:4/842: write d9/f11 [453776,72998] 0 2024-07-26T11:04:57.958 INFO:tasks.workunit.client.0.smithi144.stdout:4/843: chown d9/d26/cda 1749748 1 2024-07-26T11:04:57.958 INFO:tasks.workunit.client.0.smithi144.stdout:5/803: write d1/d9/f5a [3760360,102303] 0 2024-07-26T11:04:57.959 INFO:tasks.workunit.client.0.smithi144.stdout:4/844: truncate d9/d26/d3c/d6f/fb5 1047346 0 2024-07-26T11:04:57.961 INFO:tasks.workunit.client.0.smithi144.stdout:5/804: creat d1/d18/dc4/de0/f117 x:0 0 0 2024-07-26T11:04:57.962 INFO:tasks.workunit.client.0.smithi144.stdout:5/805: symlink d1/d18/d24/d35/l118 0 2024-07-26T11:04:57.972 INFO:tasks.workunit.client.0.smithi144.stdout:8/748: dread d1a/d32/d49/d73/d7b/f40 [0,4194304] 0 2024-07-26T11:04:57.972 INFO:tasks.workunit.client.0.smithi144.stdout:8/749: chown d1a/d32/d49/d73/d7b/d43/dae/lb6 54 1 2024-07-26T11:04:57.972 INFO:tasks.workunit.client.0.smithi144.stdout:8/750: chown d1a/d32/d49/d73/d2c/d38/d58/da3/cb2 54 1 2024-07-26T11:04:57.997 INFO:tasks.workunit.client.0.smithi144.stdout:7/765: dread d0/dd0/dbf/d2f/d53/fc5 [0,4194304] 0 2024-07-26T11:04:57.997 INFO:tasks.workunit.client.0.smithi144.stdout:7/766: truncate d0/dd0/dbf/d67/de0/fec 324335 0 2024-07-26T11:04:57.999 INFO:tasks.workunit.client.0.smithi144.stdout:0/804: dwrite d0/d25/d5e/d1e/dd2/f106 [0,4194304] 0 2024-07-26T11:04:58.006 INFO:tasks.workunit.client.0.smithi144.stdout:0/805: truncate d0/d25/d5e/d1e/f24 1821534 0 2024-07-26T11:04:58.006 INFO:tasks.workunit.client.0.smithi144.stdout:0/806: chown d0/d25/d5e/d1e/d30/d4b/cdb 46905 1 2024-07-26T11:04:58.008 INFO:tasks.workunit.client.0.smithi144.stdout:1/938: dwrite d3/d6/da/d21/f51 [0,4194304] 0 2024-07-26T11:04:58.011 INFO:tasks.workunit.client.0.smithi144.stdout:6/901: dwrite d8/d15/d57/d5f/d88/f8a [0,4194304] 0 2024-07-26T11:04:58.011 INFO:tasks.workunit.client.0.smithi144.stdout:6/902: fsync d8/d50/d92/db3/d132/f135 0 2024-07-26T11:04:58.012 INFO:tasks.workunit.client.0.smithi144.stdout:6/903: write d8/d15/d26/f114 [5887212,96925] 0 2024-07-26T11:04:58.013 INFO:tasks.workunit.client.0.smithi144.stdout:4/845: dread d9/d26/d3c/d6f/d8c/ff4 [4194304,4194304] 0 2024-07-26T11:04:58.017 INFO:tasks.workunit.client.0.smithi144.stdout:4/846: creat d9/d26/d48/d50/d60/d117/f119 x:0 0 0 2024-07-26T11:04:58.019 INFO:tasks.workunit.client.0.smithi144.stdout:6/904: creat d8/d50/d92/dea/f142 x:0 0 0 2024-07-26T11:04:58.023 INFO:tasks.workunit.client.0.smithi144.stdout:6/905: creat d8/d42/dad/f143 x:0 0 0 2024-07-26T11:04:58.023 INFO:tasks.workunit.client.0.smithi144.stdout:6/906: readlink d8/db/le 0 2024-07-26T11:04:58.024 INFO:tasks.workunit.client.0.smithi144.stdout:7/767: dread d0/d71/f84 [0,4194304] 0 2024-07-26T11:04:58.031 INFO:tasks.workunit.client.0.smithi144.stdout:8/751: dwrite d1a/d32/d49/d73/d2c/f36 [4194304,4194304] 0 2024-07-26T11:04:58.036 INFO:tasks.workunit.client.0.smithi144.stdout:8/752: creat d1a/d32/d49/ffe x:0 0 0 2024-07-26T11:04:58.038 INFO:tasks.workunit.client.0.smithi144.stdout:1/939: mknod d3/d3d/d58/d101/c12e 0 2024-07-26T11:04:58.044 INFO:tasks.workunit.client.0.smithi144.stdout:3/599: dwrite d1/d5/d97/d60/f8c [0,4194304] 0 2024-07-26T11:04:58.044 INFO:tasks.workunit.client.0.smithi144.stdout:1/940: mkdir d3/d6/da/d13/d12f 0 2024-07-26T11:04:58.050 INFO:tasks.workunit.client.0.smithi144.stdout:6/907: read d8/d15/f21 [1817555,111594] 0 2024-07-26T11:04:58.052 INFO:tasks.workunit.client.0.smithi144.stdout:3/600: mknod d1/d5/cba 0 2024-07-26T11:04:58.057 INFO:tasks.workunit.client.0.smithi144.stdout:3/601: creat d1/d28/d54/d26/d37/fbb x:0 0 0 2024-07-26T11:04:58.062 INFO:tasks.workunit.client.0.smithi144.stdout:5/806: dwrite d1/d9/f10 [0,4194304] 0 2024-07-26T11:04:58.063 INFO:tasks.workunit.client.0.smithi144.stdout:3/602: rename d1/d28/d54/d26/d37/f41 to d1/d28/d54/d26/d30/d79/fbc 0 2024-07-26T11:04:58.063 INFO:tasks.workunit.client.0.smithi144.stdout:3/603: write d1/f1a [2034439,117995] 0 2024-07-26T11:04:58.063 INFO:tasks.workunit.client.0.smithi144.stdout:6/908: mkdir d8/d50/d98/d144 0 2024-07-26T11:04:58.064 INFO:tasks.workunit.client.0.smithi144.stdout:3/604: rmdir d1/d28/d7e 0 2024-07-26T11:04:58.065 INFO:tasks.workunit.client.0.smithi144.stdout:3/605: creat d1/d28/d54/d26/d64/fbd x:0 0 0 2024-07-26T11:04:58.067 INFO:tasks.workunit.client.0.smithi144.stdout:3/606: creat d1/d28/d54/d26/d64/fbe x:0 0 0 2024-07-26T11:04:58.070 INFO:tasks.workunit.client.0.smithi144.stdout:3/607: creat d1/d28/d54/d26/d64/d68/db5/fbf x:0 0 0 2024-07-26T11:04:58.082 INFO:tasks.workunit.client.0.smithi144.stdout:5/807: dread d1/d18/d24/d35/ff3 [0,4194304] 0 2024-07-26T11:04:58.082 INFO:tasks.workunit.client.0.smithi144.stdout:5/808: write d1/d18/d24/fa0 [1113554,89164] 0 2024-07-26T11:04:58.082 INFO:tasks.workunit.client.0.smithi144.stdout:5/809: chown d1/d9/d44/d45/daa/lab 25548 1 2024-07-26T11:04:58.083 INFO:tasks.workunit.client.0.smithi144.stdout:5/810: getdents d1/d9/d44/d45/d81 0 2024-07-26T11:04:58.085 INFO:tasks.workunit.client.0.smithi144.stdout:4/847: dwrite d9/da/f69 [0,4194304] 0 2024-07-26T11:04:58.087 INFO:tasks.workunit.client.0.smithi144.stdout:4/848: rename d9/d26/d3c/f7a to d9/d26/f11a 0 2024-07-26T11:04:58.089 INFO:tasks.workunit.client.0.smithi144.stdout:4/849: creat d9/d26/d48/d50/d60/d117/f11b x:0 0 0 2024-07-26T11:04:58.090 INFO:tasks.workunit.client.0.smithi144.stdout:4/850: creat d9/d26/d48/d98/f11c x:0 0 0 2024-07-26T11:04:58.091 INFO:tasks.workunit.client.0.smithi144.stdout:4/851: link d9/d57/cb4 d9/d26/d48/d50/d60/c11d 0 2024-07-26T11:04:58.092 INFO:tasks.workunit.client.0.smithi144.stdout:8/753: dwrite d1a/d32/d66/ff0 [0,4194304] 0 2024-07-26T11:04:58.093 INFO:tasks.workunit.client.0.smithi144.stdout:4/852: creat d9/d26/d48/dd8/f11e x:0 0 0 2024-07-26T11:04:58.093 INFO:tasks.workunit.client.0.smithi144.stdout:4/853: fdatasync d9/f4e 0 2024-07-26T11:04:58.093 INFO:tasks.workunit.client.0.smithi144.stdout:3/608: dread d1/f39 [0,4194304] 0 2024-07-26T11:04:58.094 INFO:tasks.workunit.client.0.smithi144.stdout:8/754: unlink d1a/d32/d49/d73/ldd 0 2024-07-26T11:04:58.094 INFO:tasks.workunit.client.0.smithi144.stdout:8/755: readlink d1a/d32/d49/d73/d7b/d43/d4f/l56 0 2024-07-26T11:04:58.095 INFO:tasks.workunit.client.0.smithi144.stdout:4/854: truncate d9/d26/d48/f66 4097370 0 2024-07-26T11:04:58.096 INFO:tasks.workunit.client.0.smithi144.stdout:4/855: fdatasync d9/d57/fa4 0 2024-07-26T11:04:58.099 INFO:tasks.workunit.client.0.smithi144.stdout:3/609: mknod d1/d28/d54/d26/d30/cc0 0 2024-07-26T11:04:58.099 INFO:tasks.workunit.client.0.smithi144.stdout:8/756: creat d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/dfd/fff x:0 0 0 2024-07-26T11:04:58.100 INFO:tasks.workunit.client.0.smithi144.stdout:8/757: write d1a/d32/d49/d73/d7b/f40 [1357709,52611] 0 2024-07-26T11:04:58.102 INFO:tasks.workunit.client.0.smithi144.stdout:8/758: symlink d1a/d32/d8c/d9b/l100 0 2024-07-26T11:04:58.102 INFO:tasks.workunit.client.0.smithi144.stdout:8/759: fsync d1a/d32/d49/d73/d2c/d38/f5a 0 2024-07-26T11:04:58.102 INFO:tasks.workunit.client.0.smithi144.stdout:8/760: fdatasync d1a/d1c/f2d 0 2024-07-26T11:04:58.102 INFO:tasks.workunit.client.0.smithi144.stdout:8/761: chown d1a/f95 894699 1 2024-07-26T11:04:58.110 INFO:tasks.workunit.client.0.smithi144.stdout:0/807: dwrite d0/d25/d5e/d1e/d30/d67/f90 [0,4194304] 0 2024-07-26T11:04:58.110 INFO:tasks.workunit.client.0.smithi144.stdout:0/808: fsync d0/d25/d5e/d1e/dc2/f9e 0 2024-07-26T11:04:58.110 INFO:tasks.workunit.client.0.smithi144.stdout:0/809: write d0/d25/d5e/d1e/d30/d6a/faa [816105,68503] 0 2024-07-26T11:04:58.110 INFO:tasks.workunit.client.0.smithi144.stdout:7/768: dread d0/d6/f2e [0,4194304] 0 2024-07-26T11:04:58.110 INFO:tasks.workunit.client.0.smithi144.stdout:0/810: read d0/d3e/f69 [300214,115210] 0 2024-07-26T11:04:58.111 INFO:tasks.workunit.client.0.smithi144.stdout:5/811: dread d1/d18/d24/d27/d71/f99 [0,4194304] 0 2024-07-26T11:04:58.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/909: dread d8/f1a [0,4194304] 0 2024-07-26T11:04:58.118 INFO:tasks.workunit.client.0.smithi144.stdout:6/910: write d8/d15/d57/d5f/d6e/fb7 [201474,125003] 0 2024-07-26T11:04:58.118 INFO:tasks.workunit.client.0.smithi144.stdout:7/769: rename d0/d6/d9/d3f/dd3 to d0/d6/d9/d12/d115 0 2024-07-26T11:04:58.118 INFO:tasks.workunit.client.0.smithi144.stdout:7/770: readlink d0/dd0/dbf/d2f/d68/lcb 0 2024-07-26T11:04:58.118 INFO:tasks.workunit.client.0.smithi144.stdout:7/771: chown d0/d6/d9/d3f/c44 559330 1 2024-07-26T11:04:58.119 INFO:tasks.workunit.client.0.smithi144.stdout:7/772: fdatasync d0/dd0/dbf/d2f/d53/db6/fca 0 2024-07-26T11:04:58.119 INFO:tasks.workunit.client.0.smithi144.stdout:7/773: fsync d0/d6/d10c/fe5 0 2024-07-26T11:04:58.119 INFO:tasks.workunit.client.0.smithi144.stdout:6/911: creat d8/db/dbe/daa/d11a/f145 x:0 0 0 2024-07-26T11:04:58.119 INFO:tasks.workunit.client.0.smithi144.stdout:6/912: mknod d8/d50/d92/db3/d103/d128/c146 0 2024-07-26T11:04:58.119 INFO:tasks.workunit.client.0.smithi144.stdout:6/913: chown d8/d50/d92/db3/d132 123 1 2024-07-26T11:04:58.119 INFO:tasks.workunit.client.0.smithi144.stdout:6/914: write d8/d50/f85 [4615898,7057] 0 2024-07-26T11:04:58.121 INFO:tasks.workunit.client.0.smithi144.stdout:6/915: link d8/c75 d8/d50/d92/db3/c147 0 2024-07-26T11:04:58.125 INFO:tasks.workunit.client.0.smithi144.stdout:7/774: unlink d0/d6/d9/d12/c3c 0 2024-07-26T11:04:58.125 INFO:tasks.workunit.client.0.smithi144.stdout:3/610: write d1/d5/f3a [1379301,124674] 0 2024-07-26T11:04:58.125 INFO:tasks.workunit.client.0.smithi144.stdout:3/611: truncate d1/d28/d54/d26/d37/d91/f9c 2313247 0 2024-07-26T11:04:58.125 INFO:tasks.workunit.client.0.smithi144.stdout:5/812: truncate d1/d9/d44/f8a 401395 0 2024-07-26T11:04:58.125 INFO:tasks.workunit.client.0.smithi144.stdout:5/813: fdatasync d1/d9/d5b/f69 0 2024-07-26T11:04:58.126 INFO:tasks.workunit.client.0.smithi144.stdout:6/916: creat d8/d50/d92/db3/d103/d125/f148 x:0 0 0 2024-07-26T11:04:58.128 INFO:tasks.workunit.client.0.smithi144.stdout:6/917: rename d8/db/dbe/d52/ddb to d8/d15/d26/d149 0 2024-07-26T11:04:58.128 INFO:tasks.workunit.client.0.smithi144.stdout:3/612: mknod d1/d28/d54/d26/d64/d68/db4/cc1 0 2024-07-26T11:04:58.129 INFO:tasks.workunit.client.0.smithi144.stdout:5/814: stat d1/d9/d5b/d4d/l25 0 2024-07-26T11:04:58.129 INFO:tasks.workunit.client.0.smithi144.stdout:5/815: chown d1/d18/d24/d27/ffa 1715824 1 2024-07-26T11:04:58.130 INFO:tasks.workunit.client.0.smithi144.stdout:3/613: creat d1/d28/fc2 x:0 0 0 2024-07-26T11:04:58.130 INFO:tasks.workunit.client.0.smithi144.stdout:6/918: rename d8/db/dbe/d74/c13f to d8/d15/d26/d149/c14a 0 2024-07-26T11:04:58.130 INFO:tasks.workunit.client.0.smithi144.stdout:5/816: mkdir d1/d18/dc4/ded/d119 0 2024-07-26T11:04:58.131 INFO:tasks.workunit.client.0.smithi144.stdout:5/817: chown d1/d9/d44/d45/d98/cb6 4721429 1 2024-07-26T11:04:58.132 INFO:tasks.workunit.client.0.smithi144.stdout:5/818: rmdir d1/d18/da4 39 2024-07-26T11:04:58.134 INFO:tasks.workunit.client.0.smithi144.stdout:6/919: unlink d8/db/dbe/f51 0 2024-07-26T11:04:58.145 INFO:tasks.workunit.client.0.smithi144.stdout:0/811: dread d0/d25/d5e/d1e/d30/d6a/dac/fbc [0,4194304] 0 2024-07-26T11:04:58.159 INFO:tasks.workunit.client.0.smithi144.stdout:3/614: write d1/f39 [1425879,69862] 0 2024-07-26T11:04:58.164 INFO:tasks.workunit.client.0.smithi144.stdout:3/615: symlink d1/d28/d54/d26/d64/d68/db5/lc3 0 2024-07-26T11:04:58.165 INFO:tasks.workunit.client.0.smithi144.stdout:3/616: rmdir d1/d28/d54/d26/d30/d4d 39 2024-07-26T11:04:58.165 INFO:tasks.workunit.client.0.smithi144.stdout:3/617: truncate d1/d28/d54/d3d/f52 6874 0 2024-07-26T11:04:58.171 INFO:tasks.workunit.client.0.smithi144.stdout:4/856: dwrite d9/d26/d48/d50/ffc [0,4194304] 0 2024-07-26T11:04:58.172 INFO:tasks.workunit.client.0.smithi144.stdout:4/857: getdents d9/d26/d48/d98/d10b 0 2024-07-26T11:04:58.172 INFO:tasks.workunit.client.0.smithi144.stdout:4/858: read - d9/d26/d3c/d6f/dd0/ffe zero size 2024-07-26T11:04:58.172 INFO:tasks.workunit.client.0.smithi144.stdout:4/859: fdatasync d9/d26/f58 0 2024-07-26T11:04:58.198 INFO:tasks.workunit.client.0.smithi144.stdout:3/618: write d1/d28/d54/d3d/f53 [2057837,89065] 0 2024-07-26T11:04:58.199 INFO:tasks.workunit.client.0.smithi144.stdout:3/619: mkdir d1/d28/d54/d3d/d42/dc4 0 2024-07-26T11:04:58.200 INFO:tasks.workunit.client.0.smithi144.stdout:3/620: mkdir d1/d5/d5e/dc5 0 2024-07-26T11:04:58.200 INFO:tasks.workunit.client.0.smithi144.stdout:3/621: chown d1/d5/d97/d48/fad 50 1 2024-07-26T11:04:58.201 INFO:tasks.workunit.client.0.smithi144.stdout:4/860: dread d9/d26/d48/d50/d60/fb9 [0,4194304] 0 2024-07-26T11:04:58.201 INFO:tasks.workunit.client.0.smithi144.stdout:4/861: chown d9/d26/d3c/ffb 43494736 1 2024-07-26T11:04:58.201 INFO:tasks.workunit.client.0.smithi144.stdout:3/622: creat d1/d28/d54/d26/d64/da4/fc6 x:0 0 0 2024-07-26T11:04:58.202 INFO:tasks.workunit.client.0.smithi144.stdout:3/623: getdents d1/d28/d54/d61/db9 0 2024-07-26T11:04:58.203 INFO:tasks.workunit.client.0.smithi144.stdout:3/624: rename d1/d28/d54/d26/d30/d78/d80 to d1/d28/d54/d26/dc7 0 2024-07-26T11:04:58.203 INFO:tasks.workunit.client.0.smithi144.stdout:3/625: chown d1/d28/d54/f85 1 1 2024-07-26T11:04:58.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/626: creat d1/d28/d54/d26/d64/d68/fc8 x:0 0 0 2024-07-26T11:04:58.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/627: chown d1/d28/d54/d26/f27 0 1 2024-07-26T11:04:58.205 INFO:tasks.workunit.client.0.smithi144.stdout:6/920: dwrite d8/d50/d92/dea/df8/f11c [0,4194304] 0 2024-07-26T11:04:58.205 INFO:tasks.workunit.client.0.smithi144.stdout:6/921: dread - d8/d42/fcd zero size 2024-07-26T11:04:58.210 INFO:tasks.workunit.client.0.smithi144.stdout:5/819: dwrite d1/d18/d24/fb1 [0,4194304] 0 2024-07-26T11:04:58.211 INFO:tasks.workunit.client.0.smithi144.stdout:6/922: creat d8/d42/f14b x:0 0 0 2024-07-26T11:04:58.211 INFO:tasks.workunit.client.0.smithi144.stdout:6/923: write d8/db/dbe/f72 [1414319,55483] 0 2024-07-26T11:04:58.211 INFO:tasks.workunit.client.0.smithi144.stdout:0/812: dwrite d0/d25/d5e/d1e/d6c/f97 [0,4194304] 0 2024-07-26T11:04:58.211 INFO:tasks.workunit.client.0.smithi144.stdout:6/924: getdents d8/d15/d57/d5f/d6e 0 2024-07-26T11:04:58.211 INFO:tasks.workunit.client.0.smithi144.stdout:7/775: dwrite d0/d71/d95/dc4/fc2 [0,4194304] 0 2024-07-26T11:04:58.211 INFO:tasks.workunit.client.0.smithi144.stdout:5/820: creat d1/d9/d44/d45/d98/db2/f11a x:0 0 0 2024-07-26T11:04:58.212 INFO:tasks.workunit.client.0.smithi144.stdout:7/776: mkdir d0/dd0/dbf/d2f/d47/d5e/d116 0 2024-07-26T11:04:58.213 INFO:tasks.workunit.client.0.smithi144.stdout:6/925: symlink d8/d42/dad/l14c 0 2024-07-26T11:04:58.213 INFO:tasks.workunit.client.0.smithi144.stdout:7/777: write d0/d6/d10c/d57/fa4 [987377,110675] 0 2024-07-26T11:04:58.213 INFO:tasks.workunit.client.0.smithi144.stdout:6/926: write d8/d42/dad/f143 [723004,64269] 0 2024-07-26T11:04:58.213 INFO:tasks.workunit.client.0.smithi144.stdout:5/821: mkdir d1/d18/dc4/de0/deb/d116/d11b 0 2024-07-26T11:04:58.215 INFO:tasks.workunit.client.0.smithi144.stdout:7/778: creat d0/d6/d9/d12/d115/deb/f117 x:0 0 0 2024-07-26T11:04:58.218 INFO:tasks.workunit.client.0.smithi144.stdout:6/927: truncate d8/d42/f47 2754632 0 2024-07-26T11:04:58.218 INFO:tasks.workunit.client.0.smithi144.stdout:6/928: readlink d8/d15/d26/d149/l107 0 2024-07-26T11:04:58.219 INFO:tasks.workunit.client.0.smithi144.stdout:4/862: dread d9/da/f5b [0,4194304] 0 2024-07-26T11:04:58.219 INFO:tasks.workunit.client.0.smithi144.stdout:5/822: creat d1/d18/d24/f11c x:0 0 0 2024-07-26T11:04:58.221 INFO:tasks.workunit.client.0.smithi144.stdout:4/863: mknod d9/d26/d48/d50/d60/dc5/dd9/c11f 0 2024-07-26T11:04:58.268 INFO:tasks.workunit.client.0.smithi144.stdout:7/779: dwrite d0/d6/d9/d12/d115/deb/f117 [0,4194304] 0 2024-07-26T11:04:58.268 INFO:tasks.workunit.client.0.smithi144.stdout:7/780: readlink d0/l1c 0 2024-07-26T11:04:58.271 INFO:tasks.workunit.client.0.smithi144.stdout:7/781: creat d0/d6/d9/d12/f118 x:0 0 0 2024-07-26T11:04:58.273 INFO:tasks.workunit.client.0.smithi144.stdout:7/782: link d0/dd0/dbf/d2f/d47/c5c d0/dd0/dbf/c119 0 2024-07-26T11:04:58.275 INFO:tasks.workunit.client.0.smithi144.stdout:7/783: mknod d0/d6/d9/d12/d115/df2/c11a 0 2024-07-26T11:04:58.282 INFO:tasks.workunit.client.0.smithi144.stdout:4/864: dread d9/d26/f2f [0,4194304] 0 2024-07-26T11:04:58.282 INFO:tasks.workunit.client.0.smithi144.stdout:4/865: chown d9/d26/l8b 0 1 2024-07-26T11:04:58.282 INFO:tasks.workunit.client.0.smithi144.stdout:4/866: fdatasync d9/da/f52 0 2024-07-26T11:04:58.285 INFO:tasks.workunit.client.0.smithi144.stdout:6/929: dwrite d8/db/dbe/d52/fcb [0,4194304] 0 2024-07-26T11:04:58.287 INFO:tasks.workunit.client.0.smithi144.stdout:5/823: read d1/d5/f6d [1360985,84455] 0 2024-07-26T11:04:58.288 INFO:tasks.workunit.client.0.smithi144.stdout:6/930: mknod d8/d50/d92/db3/d103/d125/c14d 0 2024-07-26T11:04:58.288 INFO:tasks.workunit.client.0.smithi144.stdout:6/931: readlink d8/d15/l110 0 2024-07-26T11:04:58.288 INFO:tasks.workunit.client.0.smithi144.stdout:6/932: chown d8/d50/d92/lc1 251356 1 2024-07-26T11:04:58.288 INFO:tasks.workunit.client.0.smithi144.stdout:3/628: dwrite d1/d28/d54/d3d/f9a [0,4194304] 0 2024-07-26T11:04:58.288 INFO:tasks.workunit.client.0.smithi144.stdout:6/933: write d8/db/dbe/f7e [684136,79386] 0 2024-07-26T11:04:58.290 INFO:tasks.workunit.client.0.smithi144.stdout:3/629: mkdir d1/d28/d54/d61/db9/dc9 0 2024-07-26T11:04:58.292 INFO:tasks.workunit.client.0.smithi144.stdout:6/934: rmdir d8/d50/d92/dea/df8/d133 0 2024-07-26T11:04:58.297 INFO:tasks.workunit.client.0.smithi144.stdout:6/935: mknod d8/d15/d57/d5f/d6e/c14e 0 2024-07-26T11:04:58.306 INFO:tasks.workunit.client.0.smithi144.stdout:7/784: dread d0/dd0/dbf/f79 [0,4194304] 0 2024-07-26T11:04:58.307 INFO:tasks.workunit.client.0.smithi144.stdout:7/785: creat d0/d6/d10c/f11b x:0 0 0 2024-07-26T11:04:58.308 INFO:tasks.workunit.client.0.smithi144.stdout:7/786: write d0/d6/d55/f6a [433946,21445] 0 2024-07-26T11:04:58.308 INFO:tasks.workunit.client.0.smithi144.stdout:7/787: read d0/d6/d9/fc [1946279,47332] 0 2024-07-26T11:04:58.308 INFO:tasks.workunit.client.0.smithi144.stdout:7/788: fdatasync d0/dd0/dbf/d2f/d68/fb9 0 2024-07-26T11:04:58.308 INFO:tasks.workunit.client.0.smithi144.stdout:7/789: dread - d0/f110 zero size 2024-07-26T11:04:58.328 INFO:tasks.workunit.client.0.smithi144.stdout:4/867: dwrite d9/da/da1/ff1 [0,4194304] 0 2024-07-26T11:04:58.331 INFO:tasks.workunit.client.0.smithi144.stdout:4/868: mknod d9/d26/d48/d98/d10b/c120 0 2024-07-26T11:04:58.332 INFO:tasks.workunit.client.0.smithi144.stdout:4/869: mknod d9/d26/d48/d98/d10b/c121 0 2024-07-26T11:04:58.343 INFO:tasks.workunit.client.0.smithi144.stdout:6/936: dwrite d8/d15/d26/f121 [0,4194304] 0 2024-07-26T11:04:58.347 INFO:tasks.workunit.client.0.smithi144.stdout:6/937: mkdir d8/d42/d14f 0 2024-07-26T11:04:58.347 INFO:tasks.workunit.client.0.smithi144.stdout:6/938: readlink d8/d50/l106 0 2024-07-26T11:04:58.347 INFO:tasks.workunit.client.0.smithi144.stdout:6/939: fsync d8/db/dbe/daa/f139 0 2024-07-26T11:04:58.347 INFO:tasks.workunit.client.0.smithi144.stdout:6/940: chown d8/d50/d98/d104/f113 13 1 2024-07-26T11:04:58.354 INFO:tasks.workunit.client.0.smithi144.stdout:4/870: dread d9/d26/f99 [0,4194304] 0 2024-07-26T11:04:58.354 INFO:tasks.workunit.client.0.smithi144.stdout:7/790: dwrite d0/dd0/dbf/d2f/d53/fa8 [0,4194304] 0 2024-07-26T11:04:58.355 INFO:tasks.workunit.client.0.smithi144.stdout:7/791: fsync d0/dd0/dbf/f92 0 2024-07-26T11:04:58.357 INFO:tasks.workunit.client.0.smithi144.stdout:7/792: truncate d0/dd0/dbf/d67/f9b 486087 0 2024-07-26T11:04:58.357 INFO:tasks.workunit.client.0.smithi144.stdout:7/793: truncate d0/dd0/dbf/d67/f9f 1348132 0 2024-07-26T11:04:58.358 INFO:tasks.workunit.client.0.smithi144.stdout:7/794: creat d0/dd0/dbf/d67/f11c x:0 0 0 2024-07-26T11:04:58.359 INFO:tasks.workunit.client.0.smithi144.stdout:7/795: getdents d0 0 2024-07-26T11:04:58.368 INFO:tasks.workunit.client.0.smithi144.stdout:6/941: dread d8/d42/dad/fd5 [0,4194304] 0 2024-07-26T11:04:58.375 INFO:tasks.workunit.client.0.smithi144.stdout:6/942: dread d8/d50/d92/db3/d103/d125/f129 [0,4194304] 0 2024-07-26T11:04:58.377 INFO:tasks.workunit.client.0.smithi144.stdout:6/943: mkdir d8/d150 0 2024-07-26T11:04:58.378 INFO:tasks.workunit.client.0.smithi144.stdout:6/944: chown d8/d15/d26/d149/de3/c112 21031624 1 2024-07-26T11:04:58.383 INFO:tasks.workunit.client.0.smithi144.stdout:7/796: dread d0/d6/d10c/d57/f77 [4194304,4194304] 0 2024-07-26T11:04:58.387 INFO:tasks.workunit.client.0.smithi144.stdout:7/797: dread d0/dd0/dbf/d67/de0/fec [0,4194304] 0 2024-07-26T11:04:58.389 INFO:tasks.workunit.client.0.smithi144.stdout:7/798: link d0/d71/da1/faf d0/dd0/dbf/d2f/d47/d5e/f11d 0 2024-07-26T11:04:58.391 INFO:tasks.workunit.client.0.smithi144.stdout:4/871: dwrite d9/d26/d48/d50/ffd [0,4194304] 0 2024-07-26T11:04:58.393 INFO:tasks.workunit.client.0.smithi144.stdout:4/872: mkdir d9/da/da1/d115/d122 0 2024-07-26T11:04:58.405 INFO:tasks.workunit.client.0.smithi144.stdout:6/945: dread d8/d15/d57/d5f/f10a [0,4194304] 0 2024-07-26T11:04:58.405 INFO:tasks.workunit.client.0.smithi144.stdout:6/946: dread - d8/d42/dad/ff1 zero size 2024-07-26T11:04:58.405 INFO:tasks.workunit.client.0.smithi144.stdout:6/947: chown d8/db/dbe/d52/d58/d120 13749849 1 2024-07-26T11:04:58.405 INFO:tasks.workunit.client.0.smithi144.stdout:6/948: chown d8/d15/d57/f105 997 1 2024-07-26T11:04:58.406 INFO:tasks.workunit.client.0.smithi144.stdout:4/873: link d9/d26/f2f d9/d57/f123 0 2024-07-26T11:04:58.406 INFO:tasks.workunit.client.0.smithi144.stdout:6/949: unlink d8/d15/d26/d149/de3/c112 0 2024-07-26T11:04:58.410 INFO:tasks.workunit.client.0.smithi144.stdout:4/874: rename d9/c10 to d9/d26/d3c/d6f/dd0/c124 0 2024-07-26T11:04:58.411 INFO:tasks.workunit.client.0.smithi144.stdout:4/875: getdents d9/d26/d3c/d10c 0 2024-07-26T11:04:58.422 INFO:tasks.workunit.client.0.smithi144.stdout:4/876: link d9/d26/d48/d50/d60/d117/f119 d9/d7b/dc4/f125 0 2024-07-26T11:04:58.422 INFO:tasks.workunit.client.0.smithi144.stdout:4/877: stat d9/d26/d48/d50/lcf 0 2024-07-26T11:04:58.424 INFO:tasks.workunit.client.0.smithi144.stdout:4/878: rename d9/d26/f2b to d9/d26/d48/d98/f126 0 2024-07-26T11:04:58.432 INFO:tasks.workunit.client.0.smithi144.stdout:7/799: dwrite d0/d6/d10c/f90 [0,4194304] 0 2024-07-26T11:04:58.435 INFO:tasks.workunit.client.0.smithi144.stdout:7/800: mknod d0/d6/d9/d12/d6c/c11e 0 2024-07-26T11:04:58.437 INFO:tasks.workunit.client.0.smithi144.stdout:7/801: mknod d0/dd0/dbf/d2f/d47/d5e/d116/c11f 0 2024-07-26T11:04:58.437 INFO:tasks.workunit.client.0.smithi144.stdout:7/802: chown d0/d6/d9/d12/c102 17141 1 2024-07-26T11:04:58.438 INFO:tasks.workunit.client.0.smithi144.stdout:7/803: read d0/d6/d55/f6a [1030298,63027] 0 2024-07-26T11:04:58.438 INFO:tasks.workunit.client.0.smithi144.stdout:7/804: dread - d0/d6/d10c/f11b zero size 2024-07-26T11:04:58.440 INFO:tasks.workunit.client.0.smithi144.stdout:7/805: rename d0/d71/f84 to d0/d6/d9/d12/d115/f120 0 2024-07-26T11:04:58.442 INFO:tasks.workunit.client.0.smithi144.stdout:7/806: mknod d0/dd0/dbf/d2f/d47/c121 0 2024-07-26T11:04:58.444 INFO:tasks.workunit.client.0.smithi144.stdout:7/807: rename d0/dd0/dbf/l9e to d0/dd0/dbf/d67/l122 0 2024-07-26T11:04:58.444 INFO:tasks.workunit.client.0.smithi144.stdout:7/808: truncate d0/dd0/dbf/d2f/d47/d5e/fe1 714439 0 2024-07-26T11:04:58.456 INFO:tasks.workunit.client.0.smithi144.stdout:6/950: dwrite d8/d50/de2/f101 [0,4194304] 0 2024-07-26T11:04:58.456 INFO:tasks.workunit.client.0.smithi144.stdout:6/951: stat d8/d15/f21 0 2024-07-26T11:04:58.479 INFO:tasks.workunit.client.0.smithi144.stdout:6/952: write d8/f33 [7536113,121233] 0 2024-07-26T11:04:58.479 INFO:tasks.workunit.client.0.smithi144.stdout:6/953: write d8/db/fa1 [2364497,62680] 0 2024-07-26T11:04:58.480 INFO:tasks.workunit.client.0.smithi144.stdout:6/954: chown d8/db/dbe/d52/d58/dbf/c117 0 1 2024-07-26T11:04:58.480 INFO:tasks.workunit.client.0.smithi144.stdout:6/955: truncate d8/db/dbe/d52/d58/f123 1035142 0 2024-07-26T11:04:58.502 INFO:tasks.workunit.client.0.smithi144.stdout:7/809: dwrite d0/dd0/dbf/d2f/d47/fae [0,4194304] 0 2024-07-26T11:04:58.518 INFO:tasks.workunit.client.0.smithi144.stdout:6/956: dwrite d8/d15/d57/f105 [0,4194304] 0 2024-07-26T11:04:58.522 INFO:tasks.workunit.client.0.smithi144.stdout:6/957: unlink d8/d15/d57/f105 0 2024-07-26T11:04:58.523 INFO:tasks.workunit.client.0.smithi144.stdout:6/958: mknod d8/d15/c151 0 2024-07-26T11:04:58.525 INFO:tasks.workunit.client.0.smithi144.stdout:7/810: dread d0/d6/f8 [0,4194304] 0 2024-07-26T11:04:58.525 INFO:tasks.workunit.client.0.smithi144.stdout:7/811: write d0/d71/d95/dc4/f65 [597924,43292] 0 2024-07-26T11:04:58.525 INFO:tasks.workunit.client.0.smithi144.stdout:7/812: truncate d0/d6/d10c/fe9 729531 0 2024-07-26T11:04:58.526 INFO:tasks.workunit.client.0.smithi144.stdout:7/813: truncate d0/d6/d10c/f30 2407798 0 2024-07-26T11:04:58.527 INFO:tasks.workunit.client.0.smithi144.stdout:6/959: rename d8/d50/d98/fb4 to d8/db/dbe/d52/d58/d130/f152 0 2024-07-26T11:04:58.581 INFO:tasks.workunit.client.0.smithi144.stdout:6/960: dwrite d8/d15/d57/d5f/d6e/f11f [0,4194304] 0 2024-07-26T11:04:58.593 INFO:tasks.workunit.client.0.smithi144.stdout:6/961: dread d8/d42/dad/fd2 [0,4194304] 0 2024-07-26T11:04:58.595 INFO:tasks.workunit.client.0.smithi144.stdout:6/962: rename d8/d50/d92/db3/d103/d125/c14d to d8/d50/d92/db3/d103/d128/c153 0 2024-07-26T11:04:58.605 INFO:tasks.workunit.client.0.smithi144.stdout:7/814: dwrite d0/f4c [0,4194304] 0 2024-07-26T11:04:58.605 INFO:tasks.workunit.client.0.smithi144.stdout:7/815: read - d0/dd0/dbf/d2f/d53/db6/dfc/f109 zero size 2024-07-26T11:04:58.617 INFO:tasks.workunit.client.0.smithi144.stdout:6/963: dread d8/d50/d92/dea/f10e [0,4194304] 0 2024-07-26T11:04:58.619 INFO:tasks.workunit.client.0.smithi144.stdout:6/964: mkdir d8/d15/d57/d5f/d88/d154 0 2024-07-26T11:04:58.622 INFO:tasks.workunit.client.0.smithi144.stdout:6/965: creat d8/d15/d26/d149/de3/f155 x:0 0 0 2024-07-26T11:04:58.624 INFO:tasks.workunit.client.0.smithi144.stdout:6/966: rename d8/l14 to d8/d50/d92/dea/l156 0 2024-07-26T11:04:58.624 INFO:tasks.workunit.client.0.smithi144.stdout:6/967: chown d8/db/dbe/d5c/da7/led 41922696 1 2024-07-26T11:04:58.625 INFO:tasks.workunit.client.0.smithi144.stdout:7/816: dread d0/d6/d9/d12/f6d [0,4194304] 0 2024-07-26T11:04:58.627 INFO:tasks.workunit.client.0.smithi144.stdout:7/817: write d0/dd0/dbf/d2f/d68/de2/ff6 [227015,30165] 0 2024-07-26T11:04:58.627 INFO:tasks.workunit.client.0.smithi144.stdout:7/818: readlink d0/d6/d55/ld2 0 2024-07-26T11:04:58.629 INFO:tasks.workunit.client.0.smithi144.stdout:7/819: creat d0/dd0/dbf/d2f/d47/f123 x:0 0 0 2024-07-26T11:04:58.629 INFO:tasks.workunit.client.0.smithi144.stdout:7/820: write d0/dd0/dbf/d2f/d47/fa0 [2202155,55276] 0 2024-07-26T11:04:58.629 INFO:tasks.workunit.client.0.smithi144.stdout:7/821: write d0/d71/d95/dc4/f65 [1517347,71973] 0 2024-07-26T11:04:58.631 INFO:tasks.workunit.client.0.smithi144.stdout:7/822: mkdir d0/d71/dc0/de4/d124 0 2024-07-26T11:04:58.633 INFO:tasks.workunit.client.0.smithi144.stdout:7/823: rename d0/d71/da1 to d0/d71/dc0/de4/d125 0 2024-07-26T11:04:58.642 INFO:tasks.workunit.client.0.smithi144.stdout:7/824: write d0/d6/f8b [460426,69343] 0 2024-07-26T11:04:58.645 INFO:tasks.workunit.client.0.smithi144.stdout:7/825: unlink d0/dd0/dbf/d2f/d53/fc8 0 2024-07-26T11:04:58.670 INFO:tasks.workunit.client.0.smithi144.stdout:6/968: dwrite d8/d15/d57/d5f/d88/f8a [4194304,4194304] 0 2024-07-26T11:04:58.670 INFO:tasks.workunit.client.0.smithi144.stdout:6/969: write d8/d15/f21 [3469627,118787] 0 2024-07-26T11:04:58.670 INFO:tasks.workunit.client.0.smithi144.stdout:6/970: write d8/d15/d26/d149/de3/f155 [839772,25982] 0 2024-07-26T11:04:58.672 INFO:tasks.workunit.client.0.smithi144.stdout:6/971: rmdir d8/dca 39 2024-07-26T11:04:58.672 INFO:tasks.workunit.client.0.smithi144.stdout:6/972: truncate d8/d50/d92/db3/f118 561696 0 2024-07-26T11:04:58.672 INFO:tasks.workunit.client.0.smithi144.stdout:6/973: dread - d8/db/dbe/daa/f139 zero size 2024-07-26T11:04:58.674 INFO:tasks.workunit.client.0.smithi144.stdout:6/974: truncate d8/d15/d57/d5f/fc3 2316954 0 2024-07-26T11:04:58.674 INFO:tasks.workunit.client.0.smithi144.stdout:6/975: chown d8/d50/d98/d104/f140 0 1 2024-07-26T11:04:58.674 INFO:tasks.workunit.client.0.smithi144.stdout:7/826: dread d0/d6/d10c/d57/f7b [0,4194304] 0 2024-07-26T11:04:58.684 INFO:tasks.workunit.client.0.smithi144.stdout:6/976: unlink d8/d50/c126 0 2024-07-26T11:04:58.690 INFO:tasks.workunit.client.0.smithi144.stdout:6/977: getdents d8/db/dbe/d52/d58/dbf 0 2024-07-26T11:04:58.703 INFO:tasks.workunit.client.0.smithi144.stdout:6/978: dread d8/db/fe7 [0,4194304] 0 2024-07-26T11:04:58.704 INFO:tasks.workunit.client.0.smithi144.stdout:6/979: mknod d8/d42/c157 0 2024-07-26T11:04:58.706 INFO:tasks.workunit.client.0.smithi144.stdout:6/980: unlink d8/d50/d92/dea/df8/f100 0 2024-07-26T11:04:58.738 INFO:tasks.workunit.client.0.smithi144.stdout:6/981: write d8/d50/d92/db3/d103/d12a/f138 [2953596,125060] 0 2024-07-26T11:04:58.738 INFO:tasks.workunit.client.0.smithi144.stdout:6/982: fdatasync d8/d15/d26/d149/de3/f12c 0 2024-07-26T11:04:58.738 INFO:tasks.workunit.client.0.smithi144.stdout:7/827: dwrite d0/f49 [0,4194304] 0 2024-07-26T11:04:58.740 INFO:tasks.workunit.client.0.smithi144.stdout:6/983: write d8/d50/d92/db3/d103/d125/f129 [496072,76957] 0 2024-07-26T11:04:58.741 INFO:tasks.workunit.client.0.smithi144.stdout:7/828: rename d0/d71/l111 to d0/dd0/dbf/d67/l126 0 2024-07-26T11:04:58.741 INFO:tasks.workunit.client.0.smithi144.stdout:7/829: write d0/d6/d9/d12/f118 [215095,57716] 0 2024-07-26T11:04:58.742 INFO:tasks.workunit.client.0.smithi144.stdout:6/984: mknod d8/db/dbe/c158 0 2024-07-26T11:04:58.742 INFO:tasks.workunit.client.0.smithi144.stdout:7/830: getdents d0/dd0/dbf 0 2024-07-26T11:04:58.743 INFO:tasks.workunit.client.0.smithi144.stdout:7/831: rename d0/dd0/dbf/d67/c104 to d0/d71/dc0/c127 0 2024-07-26T11:04:58.745 INFO:tasks.workunit.client.0.smithi144.stdout:7/832: creat d0/dd0/dbf/d67/de0/d100/f128 x:0 0 0 2024-07-26T11:04:58.789 INFO:tasks.workunit.client.0.smithi144.stdout:7/833: dread d0/d6/f56 [0,4194304] 0 2024-07-26T11:04:58.790 INFO:tasks.workunit.client.0.smithi144.stdout:7/834: chown d0/dd0/dbf/d2f/d47/d5e/f63 231064 1 2024-07-26T11:04:58.791 INFO:tasks.workunit.client.0.smithi144.stdout:7/835: creat d0/d112/f129 x:0 0 0 2024-07-26T11:04:58.792 INFO:tasks.workunit.client.0.smithi144.stdout:7/836: symlink d0/l12a 0 2024-07-26T11:04:58.799 INFO:tasks.workunit.client.0.smithi144.stdout:6/985: dwrite d8/d42/dad/ff1 [0,4194304] 0 2024-07-26T11:04:58.799 INFO:tasks.workunit.client.0.smithi144.stdout:6/986: chown d8/db/dbe/daa/cd0 116491 1 2024-07-26T11:04:58.808 INFO:tasks.workunit.client.0.smithi144.stdout:6/987: symlink d8/db/dbe/d52/l159 0 2024-07-26T11:04:58.818 INFO:tasks.workunit.client.0.smithi144.stdout:6/988: mkdir d8/d42/dad/d15a 0 2024-07-26T11:04:58.818 INFO:tasks.workunit.client.0.smithi144.stdout:6/989: readlink d8/d15/l122 0 2024-07-26T11:04:58.826 INFO:tasks.workunit.client.0.smithi144.stdout:6/990: mkdir d8/d50/d92/db3/d103/d15b 0 2024-07-26T11:04:58.826 INFO:tasks.workunit.client.0.smithi144.stdout:8/762: sync 2024-07-26T11:04:58.826 INFO:tasks.workunit.client.0.smithi144.stdout:2/793: sync 2024-07-26T11:04:58.826 INFO:tasks.workunit.client.0.smithi144.stdout:4/879: sync 2024-07-26T11:04:58.826 INFO:tasks.workunit.client.0.smithi144.stdout:3/630: sync 2024-07-26T11:04:58.827 INFO:tasks.workunit.client.0.smithi144.stdout:5/824: sync 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:0/813: sync 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:9/820: sync 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:1/941: sync 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:9/821: getdents d9/de/d14/d23/d39/d7f/dbc/d10c 0 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:2/794: rmdir d0/d24/d7b/ddc/d28/d27 39 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:2/795: stat d0/d24/d7b/ddc/d28/d27/d8b/fe1 0 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:6/991: creat d8/d150/f15c x:0 0 0 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:2/796: write d0/d24/d7b/ddc/d28/d2e/d5d/d6b/f99 [4712171,33129] 0 2024-07-26T11:04:58.828 INFO:tasks.workunit.client.0.smithi144.stdout:3/631: write d1/f17 [1939337,81166] 0 2024-07-26T11:04:58.830 INFO:tasks.workunit.client.0.smithi144.stdout:8/763: rename d1a/d32/d49/d73/d2c/d38/d58/da3/db1/lb8 to d1a/d32/d49/d73/d7b/d43/d4c/l101 0 2024-07-26T11:04:58.830 INFO:tasks.workunit.client.0.smithi144.stdout:8/764: dread - d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/fec zero size 2024-07-26T11:04:58.832 INFO:tasks.workunit.client.0.smithi144.stdout:9/822: creat d9/de/d4e/d6a/deb/dfd/f113 x:0 0 0 2024-07-26T11:04:58.832 INFO:tasks.workunit.client.0.smithi144.stdout:9/823: chown d9/d3b/d76/d9b/cb3 97840472 1 2024-07-26T11:04:58.835 INFO:tasks.workunit.client.0.smithi144.stdout:0/814: mkdir d0/d25/d5e/d1e/d30/d6a/d6f/dec/d10d 0 2024-07-26T11:04:58.835 INFO:tasks.workunit.client.0.smithi144.stdout:3/632: mkdir d1/d28/d54/d26/d30/d79/dca 0 2024-07-26T11:04:58.835 INFO:tasks.workunit.client.0.smithi144.stdout:3/633: chown d1/d5/d5e/dc5 239581730 1 2024-07-26T11:04:58.835 INFO:tasks.workunit.client.0.smithi144.stdout:3/634: fsync d1/d28/fc2 0 2024-07-26T11:04:58.835 INFO:tasks.workunit.client.0.smithi144.stdout:5/825: creat d1/d9/d44/d45/d98/db2/f11d x:0 0 0 2024-07-26T11:04:58.836 INFO:tasks.workunit.client.0.smithi144.stdout:5/826: write d1/d18/dc4/de0/ff5 [1392691,35048] 0 2024-07-26T11:04:58.836 INFO:tasks.workunit.client.0.smithi144.stdout:9/824: rename d9/d3b/d76/d9b to d9/dfc/d114 0 2024-07-26T11:04:58.837 INFO:tasks.workunit.client.0.smithi144.stdout:0/815: mkdir d0/d25/d5e/d1e/dc2/d10e 0 2024-07-26T11:04:58.837 INFO:tasks.workunit.client.0.smithi144.stdout:8/765: mkdir d1a/d32/d49/d102 0 2024-07-26T11:04:58.839 INFO:tasks.workunit.client.0.smithi144.stdout:7/837: dread d0/d71/d95/dc4/fc2 [0,4194304] 0 2024-07-26T11:04:58.840 INFO:tasks.workunit.client.0.smithi144.stdout:5/827: unlink d1/l7a 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:9/825: link d9/la6 d9/de/d14/d23/d39/d7f/dbc/d10c/l115 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:0/816: write d0/d25/d5e/d1e/d91/dd1/fb5 [805497,78718] 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:8/766: rename d1a/d1c/f1f to d1a/d32/d6c/f103 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:7/838: creat d0/d6/d55/f12b x:0 0 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/828: symlink d1/d18/d24/l11e 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/829: fdatasync d1/d18/d24/f82 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/830: fdatasync d1/d18/d24/f82 0 2024-07-26T11:04:58.847 INFO:tasks.workunit.client.0.smithi144.stdout:4/880: dread d9/d7b/dc4/fc7 [0,4194304] 0 2024-07-26T11:04:58.850 INFO:tasks.workunit.client.0.smithi144.stdout:3/635: write d1/d28/d54/d26/d64/d68/f84 [455595,37745] 0 2024-07-26T11:04:58.850 INFO:tasks.workunit.client.0.smithi144.stdout:8/767: mkdir d1a/d32/d49/d73/d2c/d9d/d74/d104 0 2024-07-26T11:04:58.850 INFO:tasks.workunit.client.0.smithi144.stdout:0/817: mkdir d0/d25/d5e/d1e/d30/d6a/de4/d104/d101/d10f 0 2024-07-26T11:04:58.851 INFO:tasks.workunit.client.0.smithi144.stdout:3/636: chown d1/d28/d54/d26/d64/da4/lb3 93 1 2024-07-26T11:04:58.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/839: creat d0/dd0/dbf/d2f/d53/db6/d106/f12c x:0 0 0 2024-07-26T11:04:58.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/840: fdatasync d0/dd0/dbf/d2f/d53/db6/dfc/f109 0 2024-07-26T11:04:58.851 INFO:tasks.workunit.client.0.smithi144.stdout:3/637: dread - d1/d5/f9e zero size 2024-07-26T11:04:58.854 INFO:tasks.workunit.client.0.smithi144.stdout:4/881: mknod d9/d26/d48/d98/c127 0 2024-07-26T11:04:58.855 INFO:tasks.workunit.client.0.smithi144.stdout:0/818: creat d0/d25/d5e/def/f110 x:0 0 0 2024-07-26T11:04:58.855 INFO:tasks.workunit.client.0.smithi144.stdout:0/819: read d0/d3e/f69 [1503254,125405] 0 2024-07-26T11:04:58.857 INFO:tasks.workunit.client.0.smithi144.stdout:3/638: rename d1/d28/d54/d26/d64/d68/c96 to d1/d5/d97/d60/ccb 0 2024-07-26T11:04:58.857 INFO:tasks.workunit.client.0.smithi144.stdout:3/639: write d1/d28/d54/d26/d30/f7f [761461,44878] 0 2024-07-26T11:04:58.859 INFO:tasks.workunit.client.0.smithi144.stdout:0/820: unlink d0/d3e/d45/d4c/c70 0 2024-07-26T11:04:58.861 INFO:tasks.workunit.client.0.smithi144.stdout:3/640: creat d1/d28/d54/d26/d37/da1/fcc x:0 0 0 2024-07-26T11:04:58.861 INFO:tasks.workunit.client.0.smithi144.stdout:3/641: write d1/d28/d54/d26/f7c [3627764,34668] 0 2024-07-26T11:04:58.862 INFO:tasks.workunit.client.0.smithi144.stdout:0/821: rmdir d0/d3e/db8 39 2024-07-26T11:04:58.863 INFO:tasks.workunit.client.0.smithi144.stdout:2/797: write d0/d24/d7b/ddc/f47 [45367,65917] 0 2024-07-26T11:04:58.863 INFO:tasks.workunit.client.0.smithi144.stdout:2/798: fdatasync d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fef 0 2024-07-26T11:04:58.864 INFO:tasks.workunit.client.0.smithi144.stdout:1/942: dread d3/d6/da/f121 [0,4194304] 0 2024-07-26T11:04:58.874 INFO:tasks.workunit.client.0.smithi144.stdout:2/799: symlink d0/d24/d2b/d68/d82/lf2 0 2024-07-26T11:04:58.875 INFO:tasks.workunit.client.0.smithi144.stdout:2/800: readlink d0/d24/lc0 0 2024-07-26T11:04:58.877 INFO:tasks.workunit.client.0.smithi144.stdout:9/826: read d9/de/d14/d79/fa7 [629700,19929] 0 2024-07-26T11:04:58.877 INFO:tasks.workunit.client.0.smithi144.stdout:2/801: unlink d0/d6/d34/f8f 0 2024-07-26T11:04:58.878 INFO:tasks.workunit.client.0.smithi144.stdout:6/992: dread d8/d42/fa9 [0,4194304] 0 2024-07-26T11:04:58.878 INFO:tasks.workunit.client.0.smithi144.stdout:6/993: chown d8/d15/d26/f7a 9461942 1 2024-07-26T11:04:58.878 INFO:tasks.workunit.client.0.smithi144.stdout:6/994: chown d8/l1c 59 1 2024-07-26T11:04:58.879 INFO:tasks.workunit.client.0.smithi144.stdout:9/827: mkdir d9/dfc/d114/d116 0 2024-07-26T11:04:58.887 INFO:tasks.workunit.client.0.smithi144.stdout:2/802: symlink d0/d24/d2b/d68/d82/lf3 0 2024-07-26T11:04:58.887 INFO:tasks.workunit.client.0.smithi144.stdout:9/828: rmdir d9/de 39 2024-07-26T11:04:58.888 INFO:tasks.workunit.client.0.smithi144.stdout:4/882: dread d9/da/f2d [0,4194304] 0 2024-07-26T11:04:58.889 INFO:tasks.workunit.client.0.smithi144.stdout:0/822: dread d0/d25/f7b [0,4194304] 0 2024-07-26T11:04:58.890 INFO:tasks.workunit.client.0.smithi144.stdout:0/823: chown d0/d25/d5e/d1e/l60 199084802 1 2024-07-26T11:04:58.892 INFO:tasks.workunit.client.0.smithi144.stdout:0/824: creat d0/d25/d5e/d1e/d30/d67/f111 x:0 0 0 2024-07-26T11:04:58.894 INFO:tasks.workunit.client.0.smithi144.stdout:0/825: getdents d0/d3e/d5f 0 2024-07-26T11:04:58.895 INFO:tasks.workunit.client.0.smithi144.stdout:0/826: write d0/d25/d5e/d1e/dc2/ffc [868354,6327] 0 2024-07-26T11:04:58.895 INFO:tasks.workunit.client.0.smithi144.stdout:0/827: getdents d0/d25/d5e/d1e/d30/d6a/d6f/dec/d10d 0 2024-07-26T11:04:58.900 INFO:tasks.workunit.client.0.smithi144.stdout:0/828: symlink d0/d25/d5e/d1e/d30/dab/d107/l112 0 2024-07-26T11:04:58.900 INFO:tasks.workunit.client.0.smithi144.stdout:0/829: truncate d0/d3e/d45/f9b 42459 0 2024-07-26T11:04:58.900 INFO:tasks.workunit.client.0.smithi144.stdout:0/830: truncate d0/d3e/d45/d4c/f64 966739 0 2024-07-26T11:04:58.901 INFO:tasks.workunit.client.0.smithi144.stdout:0/831: rmdir d0/d3e/db8 39 2024-07-26T11:04:58.901 INFO:tasks.workunit.client.0.smithi144.stdout:0/832: chown d0/d25/d5e/d1e/d30/d6a/faa 64521218 1 2024-07-26T11:04:58.903 INFO:tasks.workunit.client.0.smithi144.stdout:0/833: rename d0/d25/d5e/d1e/dd2/c103 to d0/d3e/d45/c113 0 2024-07-26T11:04:58.905 INFO:tasks.workunit.client.0.smithi144.stdout:0/834: link d0/d3e/db4/ce3 d0/d25/d5e/d1e/d30/d6a/de4/d104/d101/d10f/c114 0 2024-07-26T11:04:58.916 INFO:tasks.workunit.client.0.smithi144.stdout:1/943: dread d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f7f [0,4194304] 0 2024-07-26T11:04:58.917 INFO:tasks.workunit.client.0.smithi144.stdout:9/829: read d9/de/d14/d23/d112/fd3 [578950,27544] 0 2024-07-26T11:04:58.918 INFO:tasks.workunit.client.0.smithi144.stdout:1/944: mkdir d3/d3d/d58/d101/d36/d130 0 2024-07-26T11:04:58.920 INFO:tasks.workunit.client.0.smithi144.stdout:9/830: symlink d9/dfc/d114/dd7/l117 0 2024-07-26T11:04:58.925 INFO:tasks.workunit.client.0.smithi144.stdout:1/945: rename d3/d6/f23 to d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/de1/f131 0 2024-07-26T11:04:58.925 INFO:tasks.workunit.client.0.smithi144.stdout:1/946: rename d3/d6/da to d3/d6/da/d13/d19/d3a/d100/d132 22 2024-07-26T11:04:58.925 INFO:tasks.workunit.client.0.smithi144.stdout:1/947: chown d3/d6/d75 25760 1 2024-07-26T11:04:58.926 INFO:tasks.workunit.client.0.smithi144.stdout:7/841: dwrite d0/dd0/dbf/d2f/d47/fae [0,4194304] 0 2024-07-26T11:04:58.928 INFO:tasks.workunit.client.0.smithi144.stdout:1/948: mknod d3/d6/da/d78/c133 0 2024-07-26T11:04:58.928 INFO:tasks.workunit.client.0.smithi144.stdout:6/995: write d8/db/dbe/d52/d58/f123 [123217,19576] 0 2024-07-26T11:04:58.929 INFO:tasks.workunit.client.0.smithi144.stdout:1/949: creat d3/d3d/d58/d101/d10d/f134 x:0 0 0 2024-07-26T11:04:58.930 INFO:tasks.workunit.client.0.smithi144.stdout:3/642: dwrite d1/d28/d54/d26/f2d [0,4194304] 0 2024-07-26T11:04:58.932 INFO:tasks.workunit.client.0.smithi144.stdout:3/643: symlink d1/d5/d97/d60/lcd 0 2024-07-26T11:04:58.933 INFO:tasks.workunit.client.0.smithi144.stdout:1/950: unlink d3/fcb 0 2024-07-26T11:04:58.938 INFO:tasks.workunit.client.0.smithi144.stdout:1/951: creat d3/d6/d9c/f135 x:0 0 0 2024-07-26T11:04:58.943 INFO:tasks.workunit.client.0.smithi144.stdout:0/835: write d0/d25/d5e/d1e/d30/d6a/dac/fbc [3923702,117220] 0 2024-07-26T11:04:58.943 INFO:tasks.workunit.client.0.smithi144.stdout:8/768: dwrite d1a/d32/d66/da0/ddc/ff7 [0,4194304] 0 2024-07-26T11:04:58.946 INFO:tasks.workunit.client.0.smithi144.stdout:2/803: dread d0/f1 [0,4194304] 0 2024-07-26T11:04:58.950 INFO:tasks.workunit.client.0.smithi144.stdout:7/842: write d0/dd0/dbf/d2f/d47/d5e/fe1 [1652373,76804] 0 2024-07-26T11:04:58.954 INFO:tasks.workunit.client.0.smithi144.stdout:9/831: dread d9/de/f4b [0,4194304] 0 2024-07-26T11:04:58.956 INFO:tasks.workunit.client.0.smithi144.stdout:9/832: creat d9/de/d4e/d6a/deb/f118 x:0 0 0 2024-07-26T11:04:58.957 INFO:tasks.workunit.client.0.smithi144.stdout:9/833: creat d9/de/d4e/d6a/deb/f119 x:0 0 0 2024-07-26T11:04:58.958 INFO:tasks.workunit.client.0.smithi144.stdout:6/996: read d8/d15/d26/d149/de3/f155 [752077,77885] 0 2024-07-26T11:04:58.961 INFO:tasks.workunit.client.0.smithi144.stdout:2/804: dread d0/d24/d7b/ddc/d5f/f85 [0,4194304] 0 2024-07-26T11:04:58.965 INFO:tasks.workunit.client.0.smithi144.stdout:2/805: mkdir d0/db8/df4 0 2024-07-26T11:04:58.965 INFO:tasks.workunit.client.0.smithi144.stdout:2/806: chown d0/d24/d7b/lba 1605 1 2024-07-26T11:04:58.967 INFO:tasks.workunit.client.0.smithi144.stdout:2/807: getdents d0/d24/d7b/ddc/d28/d27/d8b 0 2024-07-26T11:04:58.968 INFO:tasks.workunit.client.0.smithi144.stdout:2/808: creat d0/d24/d7b/ddc/d28/d2e/d5d/dae/ff5 x:0 0 0 2024-07-26T11:04:58.968 INFO:tasks.workunit.client.0.smithi144.stdout:2/809: fdatasync d0/d24/d7b/ddc/d28/f72 0 2024-07-26T11:04:58.974 INFO:tasks.workunit.client.0.smithi144.stdout:7/843: dread d0/dd0/dbf/d2f/d47/d5e/fba [0,4194304] 0 2024-07-26T11:04:58.979 INFO:tasks.workunit.client.0.smithi144.stdout:4/883: dwrite d9/d26/fab [0,4194304] 0 2024-07-26T11:04:58.979 INFO:tasks.workunit.client.0.smithi144.stdout:7/844: write d0/dd0/dbf/d2f/d47/d5e/fa5 [397218,105007] 0 2024-07-26T11:04:58.982 INFO:tasks.workunit.client.0.smithi144.stdout:9/834: symlink d9/de/d4e/l11a 0 2024-07-26T11:04:58.982 INFO:tasks.workunit.client.0.smithi144.stdout:6/997: getdents d8/d42 0 2024-07-26T11:04:58.983 INFO:tasks.workunit.client.0.smithi144.stdout:1/952: dread d3/d6/d9c/d9f/fd0 [0,4194304] 0 2024-07-26T11:04:58.984 INFO:tasks.workunit.client.0.smithi144.stdout:4/884: link d9/d26/l36 d9/da/l128 0 2024-07-26T11:04:58.985 INFO:tasks.workunit.client.0.smithi144.stdout:7/845: truncate d0/d6/f8 2952901 0 2024-07-26T11:04:58.987 INFO:tasks.workunit.client.0.smithi144.stdout:6/998: creat d8/db/dbe/f15d x:0 0 0 2024-07-26T11:04:58.988 INFO:tasks.workunit.client.0.smithi144.stdout:7/846: creat d0/dd0/dbf/f12d x:0 0 0 2024-07-26T11:04:58.999 INFO:tasks.workunit.client.0.smithi144.stdout:3/644: dwrite d1/f34 [0,4194304] 0 2024-07-26T11:04:59.017 INFO:tasks.workunit.client.0.smithi144.stdout:8/769: dwrite d1a/d32/d49/d73/d7b/f37 [0,4194304] 0 2024-07-26T11:04:59.020 INFO:tasks.workunit.client.0.smithi144.stdout:0/836: dwrite d0/f10 [0,4194304] 0 2024-07-26T11:04:59.020 INFO:tasks.workunit.client.0.smithi144.stdout:8/770: link f6 d1a/d32/d49/d73/d2c/d38/d84/f105 0 2024-07-26T11:04:59.030 INFO:tasks.workunit.client.0.smithi144.stdout:8/771: dread d1a/d32/d49/d73/d2c/d38/d84/db0/dba/dc2/fcb [0,4194304] 0 2024-07-26T11:04:59.030 INFO:tasks.workunit.client.0.smithi144.stdout:8/772: fdatasync d1a/d32/d49/d73/d7b/d43/d4c/f77 0 2024-07-26T11:04:59.030 INFO:tasks.workunit.client.0.smithi144.stdout:2/810: dwrite d0/d24/d7b/f87 [4194304,4194304] 0 2024-07-26T11:04:59.031 INFO:tasks.workunit.client.0.smithi144.stdout:8/773: mkdir d1a/d32/d49/d73/d7b/d43/d4c/d106 0 2024-07-26T11:04:59.032 INFO:tasks.workunit.client.0.smithi144.stdout:2/811: truncate d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fc9 721977 0 2024-07-26T11:04:59.033 INFO:tasks.workunit.client.0.smithi144.stdout:8/774: mkdir d1a/d32/d49/d73/d7b/d43/d4c/d107 0 2024-07-26T11:04:59.036 INFO:tasks.workunit.client.0.smithi144.stdout:2/812: rename d0/d6/d34/fd2 to d0/ff6 0 2024-07-26T11:04:59.037 INFO:tasks.workunit.client.0.smithi144.stdout:8/775: rmdir d1a/d32/d49/d73/d7b/d4e/d9a 39 2024-07-26T11:04:59.038 INFO:tasks.workunit.client.0.smithi144.stdout:2/813: creat d0/d24/d48/d63/da3/ff7 x:0 0 0 2024-07-26T11:04:59.049 INFO:tasks.workunit.client.0.smithi144.stdout:1/953: dwrite d3/d6/da/d70/fbd [0,4194304] 0 2024-07-26T11:04:59.055 INFO:tasks.workunit.client.0.smithi144.stdout:2/814: dread d0/f3d [0,4194304] 0 2024-07-26T11:04:59.055 INFO:tasks.workunit.client.0.smithi144.stdout:2/815: chown d0/d24/d7b/ddc/d28/d2e/d77/fb2 258 1 2024-07-26T11:04:59.068 INFO:tasks.workunit.client.0.smithi144.stdout:3/645: dwrite d1/d28/d54/d26/d64/f81 [0,4194304] 0 2024-07-26T11:04:59.068 INFO:tasks.workunit.client.0.smithi144.stdout:3/646: write d1/d5/f9e [877425,126756] 0 2024-07-26T11:04:59.068 INFO:tasks.workunit.client.0.smithi144.stdout:3/647: chown d1/d28/d54/d3d/d42 246 1 2024-07-26T11:04:59.068 INFO:tasks.workunit.client.0.smithi144.stdout:3/648: truncate d1/d28/d54/d26/d64/d68/fa3 2006748 0 2024-07-26T11:04:59.070 INFO:tasks.workunit.client.0.smithi144.stdout:3/649: truncate d1/d28/d54/d26/d30/f7f 106682 0 2024-07-26T11:04:59.070 INFO:tasks.workunit.client.0.smithi144.stdout:3/650: chown d1/d28/d54/d26/d64/d68/fa3 15 1 2024-07-26T11:04:59.070 INFO:tasks.workunit.client.0.smithi144.stdout:6/999: dwrite d8/d15/d57/d5f/d88/f8a [0,4194304] 0 2024-07-26T11:04:59.078 INFO:tasks.workunit.client.0.smithi144.stdout:3/651: write d1/f20 [600096,25473] 0 2024-07-26T11:04:59.081 INFO:tasks.workunit.client.0.smithi144.stdout:3/652: unlink d1/d28/d54/d26/d30/l3c 0 2024-07-26T11:04:59.081 INFO:tasks.workunit.client.0.smithi144.stdout:3/653: truncate d1/d28/d54/d26/d64/d68/fc8 230486 0 2024-07-26T11:04:59.081 INFO:tasks.workunit.client.0.smithi144.stdout:3/654: dread - d1/d5/d97/d60/fb8 zero size 2024-07-26T11:04:59.084 INFO:tasks.workunit.client.0.smithi144.stdout:0/837: read d0/d25/d5e/d1e/d30/d6a/f8a [523287,57324] 0 2024-07-26T11:04:59.092 INFO:tasks.workunit.client.0.smithi144.stdout:8/776: dwrite d1a/d1c/f22 [4194304,4194304] 0 2024-07-26T11:04:59.092 INFO:tasks.workunit.client.0.smithi144.stdout:0/838: dread d0/f78 [0,4194304] 0 2024-07-26T11:04:59.092 INFO:tasks.workunit.client.0.smithi144.stdout:9/835: dwrite d9/de/d14/d23/d39/d7f/f111 [0,4194304] 0 2024-07-26T11:04:59.092 INFO:tasks.workunit.client.0.smithi144.stdout:9/836: dread - d9/fca zero size 2024-07-26T11:04:59.093 INFO:tasks.workunit.client.0.smithi144.stdout:8/777: rmdir d1a 39 2024-07-26T11:04:59.093 INFO:tasks.workunit.client.0.smithi144.stdout:0/839: chown d0/d25/d5e/d1e/d30/d6a/de4/d104/d101/d10f 11 1 2024-07-26T11:04:59.094 INFO:tasks.workunit.client.0.smithi144.stdout:0/840: write d0/d25/d5e/d1e/d6c/fdf [614088,130249] 0 2024-07-26T11:04:59.095 INFO:tasks.workunit.client.0.smithi144.stdout:8/778: truncate d1a/d32/d49/d73/d7b/d4e/d78/f90 220550 0 2024-07-26T11:04:59.095 INFO:tasks.workunit.client.0.smithi144.stdout:8/779: readlink d1a/d32/d49/d73/d7b/d43/dae/lb6 0 2024-07-26T11:04:59.096 INFO:tasks.workunit.client.0.smithi144.stdout:9/837: symlink d9/de/d14/d23/d39/d7f/dbc/d10c/l11b 0 2024-07-26T11:04:59.096 INFO:tasks.workunit.client.0.smithi144.stdout:9/838: read - d9/ff5 zero size 2024-07-26T11:04:59.096 INFO:tasks.workunit.client.0.smithi144.stdout:9/839: readlink d9/l69 0 2024-07-26T11:04:59.096 INFO:tasks.workunit.client.0.smithi144.stdout:9/840: fdatasync d9/de/d14/d79/fa7 0 2024-07-26T11:04:59.098 INFO:tasks.workunit.client.0.smithi144.stdout:8/780: getdents d1a/d32/d49/d73/d2c/d38 0 2024-07-26T11:04:59.098 INFO:tasks.workunit.client.0.smithi144.stdout:8/781: read d1a/d32/d49/d73/d2c/d38/d58/f6e [3247072,30555] 0 2024-07-26T11:04:59.099 INFO:tasks.workunit.client.0.smithi144.stdout:9/841: mknod d9/dfc/d108/c11c 0 2024-07-26T11:04:59.099 INFO:tasks.workunit.client.0.smithi144.stdout:0/841: mknod d0/d25/d5e/d1e/d30/d4b/c115 0 2024-07-26T11:04:59.102 INFO:tasks.workunit.client.0.smithi144.stdout:9/842: mknod d9/d3b/dce/c11d 0 2024-07-26T11:04:59.102 INFO:tasks.workunit.client.0.smithi144.stdout:0/842: truncate d0/d25/d5e/d1e/d30/dab/fd5 2609172 0 2024-07-26T11:04:59.104 INFO:tasks.workunit.client.0.smithi144.stdout:0/843: chown d0/d25/d5e/d1e/d91/dd1/la1 3 1 2024-07-26T11:04:59.104 INFO:tasks.workunit.client.0.smithi144.stdout:9/843: link d9/d3b/d81/lcb d9/de/d4e/l11e 0 2024-07-26T11:04:59.105 INFO:tasks.workunit.client.0.smithi144.stdout:9/844: mknod d9/dfc/d114/c11f 0 2024-07-26T11:04:59.105 INFO:tasks.workunit.client.0.smithi144.stdout:0/844: rename d0/d3e/d45/d4c/f8f to d0/d25/d5e/d1e/d30/dab/f116 0 2024-07-26T11:04:59.106 INFO:tasks.workunit.client.0.smithi144.stdout:9/845: symlink d9/dfc/l120 0 2024-07-26T11:04:59.108 INFO:tasks.workunit.client.0.smithi144.stdout:2/816: dwrite d0/db8/fde [0,4194304] 0 2024-07-26T11:04:59.117 INFO:tasks.workunit.client.0.smithi144.stdout:9/846: dread d9/de/d14/d3f/d5e/f6c [0,4194304] 0 2024-07-26T11:04:59.121 INFO:tasks.workunit.client.0.smithi144.stdout:2/817: dread - d0/d24/d2b/d68/f9d zero size 2024-07-26T11:04:59.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/782: read d1a/d32/d49/d73/d2c/f88 [6551182,33192] 0 2024-07-26T11:04:59.121 INFO:tasks.workunit.client.0.smithi144.stdout:8/783: dread - d1a/d32/d8c/d9b/fdf zero size 2024-07-26T11:04:59.124 INFO:tasks.workunit.client.0.smithi144.stdout:2/818: creat d0/d24/d7b/ddc/d5f/ff8 x:0 0 0 2024-07-26T11:04:59.125 INFO:tasks.workunit.client.0.smithi144.stdout:8/784: symlink d1a/d32/d49/d73/d7b/d43/d4c/d107/l108 0 2024-07-26T11:04:59.125 INFO:tasks.workunit.client.0.smithi144.stdout:8/785: fdatasync d1a/d1c/f41 0 2024-07-26T11:04:59.127 INFO:tasks.workunit.client.0.smithi144.stdout:8/786: creat d1a/d32/d49/d73/d7b/d4e/d78/f109 x:0 0 0 2024-07-26T11:04:59.128 INFO:tasks.workunit.client.0.smithi144.stdout:0/845: dread d0/d25/d5e/d1e/d30/f39 [0,4194304] 0 2024-07-26T11:04:59.128 INFO:tasks.workunit.client.0.smithi144.stdout:0/846: chown d0/d3e/d45/f55 70907 1 2024-07-26T11:04:59.177 INFO:tasks.workunit.client.0.smithi144.stdout:8/787: dwrite d1a/d32/d49/d73/d2c/d38/d58/f83 [0,4194304] 0 2024-07-26T11:04:59.179 INFO:tasks.workunit.client.0.smithi144.stdout:0/847: read d0/d25/fa5 [1422786,34496] 0 2024-07-26T11:04:59.179 INFO:tasks.workunit.client.0.smithi144.stdout:2/819: dwrite d0/d24/d7b/ddc/d5f/fe6 [0,4194304] 0 2024-07-26T11:04:59.180 INFO:tasks.workunit.client.0.smithi144.stdout:8/788: rename d1a/d32/d50/l6a to d1a/d32/d49/d73/d2c/d9d/d74/d104/l10a 0 2024-07-26T11:04:59.182 INFO:tasks.workunit.client.0.smithi144.stdout:2/820: truncate d0/f3d 2129848 0 2024-07-26T11:04:59.182 INFO:tasks.workunit.client.0.smithi144.stdout:2/821: truncate d0/d24/d2b/f8e 764268 0 2024-07-26T11:04:59.183 INFO:tasks.workunit.client.0.smithi144.stdout:2/822: truncate d0/d6/d9/f33 1105553 0 2024-07-26T11:04:59.185 INFO:tasks.workunit.client.0.smithi144.stdout:2/823: mkdir d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2/df9 0 2024-07-26T11:04:59.205 INFO:tasks.workunit.client.0.smithi144.stdout:2/824: dread d0/d24/d7b/ddc/d28/f72 [0,4194304] 0 2024-07-26T11:04:59.206 INFO:tasks.workunit.client.0.smithi144.stdout:2/825: dread - d0/d24/d48/d63/da3/ff7 zero size 2024-07-26T11:04:59.206 INFO:tasks.workunit.client.0.smithi144.stdout:2/826: write d0/db8/fde [4962957,5698] 0 2024-07-26T11:04:59.206 INFO:tasks.workunit.client.0.smithi144.stdout:2/827: readlink d0/d24/d7b/lba 0 2024-07-26T11:04:59.207 INFO:tasks.workunit.client.0.smithi144.stdout:2/828: creat d0/d24/d7b/ddc/ffa x:0 0 0 2024-07-26T11:04:59.208 INFO:tasks.workunit.client.0.smithi144.stdout:2/829: link d0/d24/d7b/ddc/ffa d0/d24/d7b/ddc/d28/d27/ffb 0 2024-07-26T11:04:59.208 INFO:tasks.workunit.client.0.smithi144.stdout:2/830: fdatasync d0/d24/d7b/ddc/f35 0 2024-07-26T11:04:59.209 INFO:tasks.workunit.client.0.smithi144.stdout:2/831: getdents d0/d24/d7b/ddc/d28/d2e/d77/d84 0 2024-07-26T11:04:59.210 INFO:tasks.workunit.client.0.smithi144.stdout:2/832: mknod d0/d24/d2b/cfc 0 2024-07-26T11:04:59.215 INFO:tasks.workunit.client.0.smithi144.stdout:8/789: dread d1a/f2b [0,4194304] 0 2024-07-26T11:04:59.219 INFO:tasks.workunit.client.0.smithi144.stdout:8/790: getdents d1a/d32/d66/da0/ddc 0 2024-07-26T11:04:59.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/791: creat d1a/d32/d49/d73/d2c/d38/d58/f10b x:0 0 0 2024-07-26T11:04:59.221 INFO:tasks.workunit.client.0.smithi144.stdout:8/792: mkdir d1a/d32/d49/d73/d7b/d4e/d10c 0 2024-07-26T11:04:59.223 INFO:tasks.workunit.client.0.smithi144.stdout:8/793: truncate d1a/d32/d6c/d82/fa4 765567 0 2024-07-26T11:04:59.223 INFO:tasks.workunit.client.0.smithi144.stdout:8/794: fdatasync d1a/d32/f48 0 2024-07-26T11:04:59.233 INFO:tasks.workunit.client.0.smithi144.stdout:0/848: dwrite d0/d25/d5e/d1e/d30/d6a/dac/fe5 [0,4194304] 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/849: creat d0/d3e/d45/dd4/f117 x:0 0 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/850: write d0/d3e/d5f/fb1 [680221,90601] 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/851: chown d0/d3e/db8 317812156 1 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/852: stat d0/d3e/d5f/l94 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/853: truncate d0/f18 6077738 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:2/833: read d0/d24/d7b/ddc/d28/d2e/d77/f40 [1070426,16240] 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/854: creat d0/d25/d5e/d1e/d30/f118 x:0 0 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/855: fsync d0/d25/d5e/d1e/d30/fb9 0 2024-07-26T11:04:59.239 INFO:tasks.workunit.client.0.smithi144.stdout:0/856: readlink d0/d25/d5e/d1e/d30/d67/d84/l102 0 2024-07-26T11:04:59.240 INFO:tasks.workunit.client.0.smithi144.stdout:0/857: symlink d0/d25/d5e/d1e/d30/dab/df3/l119 0 2024-07-26T11:04:59.251 INFO:tasks.workunit.client.0.smithi144.stdout:8/795: write d1a/d32/d66/ff0 [2376460,120444] 0 2024-07-26T11:04:59.258 INFO:tasks.workunit.client.0.smithi144.stdout:8/796: write d1a/d32/d49/d73/d7b/d43/d4c/f77 [206480,77097] 0 2024-07-26T11:04:59.274 INFO:tasks.workunit.client.0.smithi144.stdout:2/834: dread d0/db8/fde [0,4194304] 0 2024-07-26T11:04:59.274 INFO:tasks.workunit.client.0.smithi144.stdout:2/835: write d0/d24/d2b/f59 [1090668,82350] 0 2024-07-26T11:04:59.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/836: rename d0/d6/d9/d44/ce2 to d0/d24/d7b/ddc/d28/d2e/d5d/dae/cfd 0 2024-07-26T11:04:59.276 INFO:tasks.workunit.client.0.smithi144.stdout:2/837: write d0/d6/d9/d44/fa2 [758733,9269] 0 2024-07-26T11:04:59.286 INFO:tasks.workunit.client.0.smithi144.stdout:8/797: dread d1a/d32/d66/dcd/f53 [0,4194304] 0 2024-07-26T11:04:59.287 INFO:tasks.workunit.client.0.smithi144.stdout:8/798: chown d1a/d1c/ffa 15 1 2024-07-26T11:04:59.291 INFO:tasks.workunit.client.0.smithi144.stdout:8/799: link d1a/d32/d49/d73/d7b/d43/d4f/c6b d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/dfd/c10d 0 2024-07-26T11:04:59.298 INFO:tasks.workunit.client.0.smithi144.stdout:8/800: mknod d1a/d32/d49/d73/d7b/d43/c10e 0 2024-07-26T11:04:59.298 INFO:tasks.workunit.client.0.smithi144.stdout:8/801: fsync d1a/d1c/f97 0 2024-07-26T11:04:59.298 INFO:tasks.workunit.client.0.smithi144.stdout:8/802: chown d1a/d32/d49/d73/f3f 61 1 2024-07-26T11:04:59.299 INFO:tasks.workunit.client.0.smithi144.stdout:2/838: read d0/d24/d2b/d68/d82/f7c [2095825,82275] 0 2024-07-26T11:04:59.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/803: creat d1a/d32/d49/d73/d7b/d43/f10f x:0 0 0 2024-07-26T11:04:59.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/804: truncate d1a/d1c/ffa 672912 0 2024-07-26T11:04:59.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/805: read d1a/d32/d49/d73/d7b/d43/d4c/f54 [1681449,21747] 0 2024-07-26T11:04:59.303 INFO:tasks.workunit.client.0.smithi144.stdout:8/806: rmdir d1a/d32/d50 39 2024-07-26T11:04:59.307 INFO:tasks.workunit.client.0.smithi144.stdout:2/839: symlink d0/d24/d7b/lfe 0 2024-07-26T11:04:59.348 INFO:tasks.workunit.client.0.smithi144.stdout:2/840: dwrite d0/d24/d7b/ddc/d28/d2e/d77/fb2 [0,4194304] 0 2024-07-26T11:04:59.348 INFO:tasks.workunit.client.0.smithi144.stdout:2/841: truncate d0/ff6 543187 0 2024-07-26T11:04:59.350 INFO:tasks.workunit.client.0.smithi144.stdout:2/842: rename d0/d6/d34/c3b to d0/d6/d34/d6c/cff 0 2024-07-26T11:04:59.354 INFO:tasks.workunit.client.0.smithi144.stdout:8/807: dwrite d1a/d32/d49/d73/d2c/d38/d84/f105 [0,4194304] 0 2024-07-26T11:04:59.354 INFO:tasks.workunit.client.0.smithi144.stdout:8/808: write d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/fec [415275,48457] 0 2024-07-26T11:04:59.355 INFO:tasks.workunit.client.0.smithi144.stdout:8/809: write d1a/d32/d66/ff0 [4999639,77182] 0 2024-07-26T11:04:59.357 INFO:tasks.workunit.client.0.smithi144.stdout:8/810: link d1a/d32/d49/d73/d7b/d4e/d78/cc0 d1a/d32/d49/d73/d2c/d38/d84/db0/dba/c110 0 2024-07-26T11:04:59.357 INFO:tasks.workunit.client.0.smithi144.stdout:8/811: readlink d1a/d32/d49/d73/d2c/d38/lc5 0 2024-07-26T11:04:59.432 INFO:tasks.workunit.client.0.smithi144.stdout:8/812: dwrite d1a/d1c/ffa [0,4194304] 0 2024-07-26T11:04:59.434 INFO:tasks.workunit.client.0.smithi144.stdout:8/813: symlink d1a/d32/d49/d73/d2c/d9d/d74/dac/l111 0 2024-07-26T11:04:59.436 INFO:tasks.workunit.client.0.smithi144.stdout:8/814: creat d1a/d32/d49/d73/d2c/d38/f112 x:0 0 0 2024-07-26T11:04:59.437 INFO:tasks.workunit.client.0.smithi144.stdout:8/815: stat d1a/d32/f48 0 2024-07-26T11:04:59.437 INFO:tasks.workunit.client.0.smithi144.stdout:8/816: stat d1a/d32/d49/d73/d2c/d9d/d74/d104 0 2024-07-26T11:04:59.448 INFO:tasks.workunit.client.0.smithi144.stdout:8/817: dread f11 [0,4194304] 0 2024-07-26T11:04:59.448 INFO:tasks.workunit.client.0.smithi144.stdout:8/818: truncate d1a/f1b 5039883 0 2024-07-26T11:04:59.448 INFO:tasks.workunit.client.0.smithi144.stdout:8/819: write d1a/d32/d66/da0/ddc/ff8 [619177,102022] 0 2024-07-26T11:04:59.448 INFO:tasks.workunit.client.0.smithi144.stdout:8/820: stat d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/fec 0 2024-07-26T11:04:59.448 INFO:tasks.workunit.client.0.smithi144.stdout:8/821: fsync d1a/d32/d49/d73/f79 0 2024-07-26T11:04:59.449 INFO:tasks.workunit.client.0.smithi144.stdout:8/822: rmdir d1a/d32/d50 39 2024-07-26T11:04:59.451 INFO:tasks.workunit.client.0.smithi144.stdout:8/823: mkdir d1a/d32/d49/d73/d7b/d43/d113 0 2024-07-26T11:04:59.454 INFO:tasks.workunit.client.0.smithi144.stdout:8/824: rename d1a/d32/f71 to d1a/d32/d49/d73/d2c/f114 0 2024-07-26T11:04:59.457 INFO:tasks.workunit.client.0.smithi144.stdout:8/825: link d1a/d32/d49/d73/d2c/d38/d58/fa8 d1a/d32/d49/d73/d7b/f115 0 2024-07-26T11:04:59.458 INFO:tasks.workunit.client.0.smithi144.stdout:8/826: mknod d1a/d32/d49/d73/d7b/d4e/d10c/c116 0 2024-07-26T11:04:59.460 INFO:tasks.workunit.client.0.smithi144.stdout:8/827: symlink d1a/d32/d49/d102/l117 0 2024-07-26T11:04:59.529 INFO:tasks.workunit.client.0.smithi144.stdout:8/828: dwrite d1a/d32/d66/da0/ddc/ffb [0,4194304] 0 2024-07-26T11:04:59.532 INFO:tasks.workunit.client.0.smithi144.stdout:8/829: mknod d1a/d32/d66/c118 0 2024-07-26T11:04:59.534 INFO:tasks.workunit.client.0.smithi144.stdout:8/830: creat d1a/d32/d49/d73/d2c/d38/d58/de4/f119 x:0 0 0 2024-07-26T11:04:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:04:59 smithi160 ceph-mon[36715]: pgmap v536: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 116 MiB/s wr, 336 op/s 2024-07-26T11:04:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:04:59 smithi183 ceph-mon[37273]: pgmap v536: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 116 MiB/s wr, 336 op/s 2024-07-26T11:04:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:04:59 smithi144 ceph-mon[32129]: pgmap v536: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 116 MiB/s wr, 336 op/s 2024-07-26T11:05:01.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 -- 172.21.15.144:0/2975800778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df01055d0 msgr2=0x7f2df01059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 --2- 172.21.15.144:0/2975800778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df01055d0 0x7f2df01059e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009900 tx=0x7f2de402a410 comp rx=0 tx=0).stop 2024-07-26T11:05:01.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 -- 172.21.15.144:0/2975800778 shutdown_connections 2024-07-26T11:05:01.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 --2- 172.21.15.144:0/2975800778 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010b860 0x7f2df0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 --2- 172.21.15.144:0/2975800778 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2df010af10 0x7f2df010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 --2- 172.21.15.144:0/2975800778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df01055d0 0x7f2df01059e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 -- 172.21.15.144:0/2975800778 >> 172.21.15.144:0/2975800778 conn(0x7f2df0101040 msgr2=0x7f2df0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:01.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 -- 172.21.15.144:0/2975800778 shutdown_connections 2024-07-26T11:05:01.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.092+0000 7f2df7200640 1 -- 172.21.15.144:0/2975800778 wait complete. 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.093+0000 7f2df7200640 1 Processor -- start 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.093+0000 7f2df7200640 1 -- start start 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2df01055d0 0x7f2df019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df7200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 0x7f2df019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df7200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 0x7f2df01a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df7200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2df0115530 con 0x7f2df010b860 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df7200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01153b0 con 0x7f2df01055d0 2024-07-26T11:05:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df7200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01156b0 con 0x7f2df010af10 2024-07-26T11:05:01.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 0x7f2df01a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 0x7f2df019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 0x7f2df01a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37048/0 (socket says 172.21.15.144:37048) 2024-07-26T11:05:01.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2df01055d0 0x7f2df019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 0x7f2df019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34864/0 (socket says 172.21.15.144:34864) 2024-07-26T11:05:01.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.094+0000 7f2df6800640 1 -- 172.21.15.144:0/151281716 learned_addr learned my addr 172.21.15.144:0/151281716 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df5400640 1 -- 172.21.15.144:0/151281716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2df01055d0 msgr2=0x7f2df019bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df5400640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2df01055d0 0x7f2df019bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df5400640 1 -- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 msgr2=0x7f2df01a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df5400640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 0x7f2df01a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df5400640 1 -- 172.21.15.144:0/151281716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df01a1e90 con 0x7f2df010af10 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df6800640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 0x7f2df01a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2df5400640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 0x7f2df019c4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2df006c270 tx=0x7f2de000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.095+0000 7f2deaa00640 1 -- 172.21.15.144:0/151281716 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de000cc40 con 0x7f2df010af10 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.096+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df01a2180 con 0x7f2df010af10 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.096+0000 7f2deaa00640 1 -- 172.21.15.144:0/151281716 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2de000cde0 con 0x7f2df010af10 2024-07-26T11:05:01.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.096+0000 7f2deaa00640 1 -- 172.21.15.144:0/151281716 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de0018760 con 0x7f2df010af10 2024-07-26T11:05:01.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.096+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df01a88e0 con 0x7f2df010af10 2024-07-26T11:05:01.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.099+0000 7f2dfeffd740 1 Processor -- start 2024-07-26T11:05:01.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.099+0000 7f2dfeffd740 1 -- start start 2024-07-26T11:05:01.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c65b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c65b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c65ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55ec0c5d1bd0 con 0x55ec0c2b6070 2024-07-26T11:05:01.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ec0c5c10d0 con 0x55ec0c28d1b0 2024-07-26T11:05:01.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ec0c5c5110 con 0x55ec0c295390 2024-07-26T11:05:01.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2deaa00640 1 -- 172.21.15.144:0/151281716 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f2de0010a90 con 0x7f2df010af10 2024-07-26T11:05:01.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2df6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c65b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c65b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2df6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c65b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37050/0 (socket says 172.21.15.144:37050) 2024-07-26T11:05:01.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2df6800640 1 -- 172.21.15.144:0/3287567069 learned_addr learned my addr 172.21.15.144:0/3287567069 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:01.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.100+0000 7f2df5400640 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c65ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df6800640 1 -- 172.21.15.144:0/3287567069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 msgr2=0x55ec0c65ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df6800640 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c65ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df6800640 1 -- 172.21.15.144:0/3287567069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 msgr2=0x55ec0c65b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df6800640 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c65b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df6800640 1 -- 172.21.15.144:0/3287567069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec0c321650 con 0x55ec0c2b6070 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df5e00640 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c65b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:01.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df5400640 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c65ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2df6800640 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c65b010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0118b0 tx=0x7f2dec011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe20547f80c2cb9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.101+0000 7f2ddf000640 1 -- 172.21.15.144:0/3287567069 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec01e070 con 0x55ec0c2b6070 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.102+0000 7f2ddf000640 1 -- 172.21.15.144:0/3287567069 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dec00fe70 con 0x55ec0c2b6070 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.102+0000 7f2ddf000640 1 -- 172.21.15.144:0/3287567069 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec019d00 con 0x55ec0c2b6070 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.102+0000 7f2deaa00640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2dd80b3d40 0x7f2dd80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.102+0000 7f2deaa00640 1 -- 172.21.15.144:0/151281716 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f2de00e4620 con 0x7f2df010af10 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.102+0000 7f2df5e00640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2dd80b3d40 0x7f2dd80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3287567069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 msgr2=0x55ec0c65b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c65b010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0118b0 tx=0x7f2dec011d70 comp rx=0 tx=0).stop 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3287567069 shutdown_connections 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c65ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c65b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3287567069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c65b010 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3287567069 >> 172.21.15.144:0/3287567069 conn(0x55ec0c2aa460 msgr2=0x55ec0c658f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3287567069 shutdown_connections 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3287567069 wait complete. 2024-07-26T11:05:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.103+0000 7f2df5e00640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2dd80b3d40 0x7f2dd80b6250 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2de4031440 tx=0x7f2de4009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:01.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.104+0000 7f2dfeffd740 1 -- start start 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.104+0000 7f2dfeffd740 1 Processor -- start 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c66f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c66fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c2b6070 0x55ec0c674b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55ec0c424700 con 0x55ec0c28d1b0 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ec0c55d6a0 con 0x55ec0c2b6070 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ec0c6697f0 con 0x55ec0c295390 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2dfeffd740 10 client.0 _ll_register_callbacks cb 0x55ec0c56d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c66f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c66f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37066/0 (socket says 172.21.15.144:37066) 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5e00640 1 -- 172.21.15.144:0/3957226429 learned_addr learned my addr 172.21.15.144:0/3957226429 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:01.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5400640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c66fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df6800640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c2b6070 0x55ec0c674b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5400640 1 -- 172.21.15.144:0/3957226429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c2b6070 msgr2=0x55ec0c674b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5400640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c2b6070 0x55ec0c674b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5400640 1 -- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 msgr2=0x55ec0c66f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5400640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c66f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.105+0000 7f2df5400640 1 -- 172.21.15.144:0/3957226429 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0c574840 con 0x55ec0c295390 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2df5400640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c66fb70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2de00e1a00 tx=0x7f2de00e1f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe6f2a39eec1b9e3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dd3400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dd3400640 1 -- 172.21.15.144:0/3957226429 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de00e4460 con 0x55ec0c295390 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dd3400640 1 -- 172.21.15.144:0/3957226429 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f2de00aa1d0 con 0x55ec0c295390 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dec00fe70 con 0x55ec0c295390 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dd3400640 1 -- 172.21.15.144:0/3957226429 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2de00aaf70 con 0x55ec0c295390 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dfeffd740 10 client.25864 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:01.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.106+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ec0c67dae0 con 0x55ec0c295390 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dd3400640 1 -- 172.21.15.144:0/3957226429 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dec00fe70 con 0x55ec0c295390 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dd3400640 1 -- 172.21.15.144:0/3957226429 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2de00a9cd0 con 0x55ec0c295390 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dfeffd740 10 client.25864 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dfeffd740 10 client.25864 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dfeffd740 20 client.25864 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c2749f0 0x55ec0c680f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dfeffd740 4 client.25864 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.107+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec0c5d3b20 con 0x55ec0c2749f0 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.108+0000 7f2df5e00640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c2749f0 0x55ec0c680f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.108+0000 7f2df5e00640 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c2749f0 0x55ec0c680f90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1548b5f45c58f51 server_cookie=eb85deb515acc30b in_seq=0 out_seq=0 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.108+0000 7f2dd3400640 10 client.25864 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dd3400640 1 -- 172.21.15.144:0/3957226429 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ec0c5d3b20 con 0x55ec0c2749f0 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dd3400640 10 client.25864 handle_command_reply: tid=0 2024-07-26T11:05:01.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 client.25864 shutdown 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 msgr2=0x55ec0c66fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c66fb70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2de00e1a00 tx=0x7f2de00e1f80 comp rx=0 tx=0).stop 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 shutdown_connections 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c2749f0 0x55ec0c680f90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c2b6070 0x55ec0c674b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c66fb70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3957226429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c66f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dd3400640 0 client.25864 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.109+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 >> 172.21.15.144:0/3957226429 conn(0x55ec0c2aa460 msgr2=0x55ec0c65e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.110+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 shutdown_connections 2024-07-26T11:05:01.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.110+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3957226429 wait complete. 2024-07-26T11:05:01.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.110+0000 7f2dfeffd740 20 client.25864 trim_cache size 0 max 16384 2024-07-26T11:05:01.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.124+0000 7f2dfeffd740 1 Processor -- start 2024-07-26T11:05:01.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.124+0000 7f2dfeffd740 1 -- start start 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c658010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c673360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c66aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01b0d60 con 0x55ec0c2b6070 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01ae630 con 0x55ec0c28d1b0 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01ac480 con 0x55ec0c295390 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2df6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c66aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c658010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2df6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c66aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37080/0 (socket says 172.21.15.144:37080) 2024-07-26T11:05:01.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2df6800640 1 -- 172.21.15.144:0/3851642124 learned_addr learned my addr 172.21.15.144:0/3851642124 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:01.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2df5400640 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c673360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.125+0000 7f2df5e00640 1 -- 172.21.15.144:0/3851642124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 msgr2=0x55ec0c673360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df5e00640 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c673360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df5e00640 1 -- 172.21.15.144:0/3851642124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 msgr2=0x55ec0c66aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df5e00640 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c66aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df5e00640 1 -- 172.21.15.144:0/3851642124 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec0c66d890 con 0x55ec0c28d1b0 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df6800640 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c66aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df5400640 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c673360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2df5e00640 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c658010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009690 tx=0x7f2de403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f12fc26129136e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2dde600640 1 -- 172.21.15.144:0/3851642124 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4035040 con 0x55ec0c28d1b0 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2dde600640 1 -- 172.21.15.144:0/3851642124 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2de40035f0 con 0x55ec0c28d1b0 2024-07-26T11:05:01.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.126+0000 7f2dde600640 1 -- 172.21.15.144:0/3851642124 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4053610 con 0x55ec0c28d1b0 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3851642124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 msgr2=0x55ec0c658010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c658010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009690 tx=0x7f2de403f000 comp rx=0 tx=0).stop 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3851642124 shutdown_connections 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c2b6070 0x55ec0c66aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c295390 0x55ec0c673360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/3851642124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c28d1b0 0x55ec0c658010 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3851642124 >> 172.21.15.144:0/3851642124 conn(0x55ec0c2aa460 msgr2=0x55ec0c6708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3851642124 shutdown_connections 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.127+0000 7f2dfeffd740 1 -- 172.21.15.144:0/3851642124 wait complete. 2024-07-26T11:05:01.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.128+0000 7f2dfeffd740 1 -- start start 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.128+0000 7f2dfeffd740 1 Processor -- start 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c686950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c295390 0x55ec0c686f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c2b6070 0x55ec0c67ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01acaf0 con 0x55ec0c28d1b0 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01adf60 con 0x55ec0c295390 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2df01ad1c0 con 0x55ec0c2b6070 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2dfeffd740 10 client.0 _ll_register_callbacks cb 0x7f2de00aad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df6800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c2b6070 0x55ec0c67ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c295390 0x55ec0c686f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df6800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c2b6070 0x55ec0c67ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34910/0 (socket says 172.21.15.144:34910) 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c686950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df6800640 1 -- 172.21.15.144:0/1087716814 learned_addr learned my addr 172.21.15.144:0/1087716814 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:01.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5400640 1 -- 172.21.15.144:0/1087716814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c2b6070 msgr2=0x55ec0c67ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5400640 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c2b6070 0x55ec0c67ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5400640 1 -- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 msgr2=0x55ec0c686950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:01.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5400640 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c686950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.129+0000 7f2df5400640 1 -- 172.21.15.144:0/1087716814 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0c67b410 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2df5400640 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c295390 0x55ec0c686f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2de00aac70 tx=0x7f2de00af3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbf64c779e56f4ce server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dd3400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dd3400640 1 -- 172.21.15.144:0/1087716814 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de00aa6e0 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2de40035f0 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dd3400640 1 -- 172.21.15.144:0/1087716814 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f2de00aa1d0 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dd3400640 1 -- 172.21.15.144:0/1087716814 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2de00a9e50 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dfeffd740 10 client.25895 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.130+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ec0c668770 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dd3400640 1 -- 172.21.15.144:0/1087716814 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2de40035f0 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dd3400640 1 -- 172.21.15.144:0/1087716814 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2de00b06e0 con 0x55ec0c295390 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dfeffd740 10 client.25895 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:01.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dfeffd740 10 client.25895 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dfeffd740 20 client.25895 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c249da0 0x55ec0c6ebb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dfeffd740 4 client.25895 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec0c5cf5c0 con 0x55ec0c249da0 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.131+0000 7f2df5e00640 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c249da0 0x55ec0c6ebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.132+0000 7f2dd3400640 10 client.25895 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.132+0000 7f2df5e00640 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c249da0 0x55ec0c6ebb70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c269acf84619f11 server_cookie=2f66f891e4f74420 in_seq=0 out_seq=0 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.132+0000 7f2dd3400640 1 -- 172.21.15.144:0/1087716814 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec0c5cf5c0 con 0x55ec0c249da0 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.132+0000 7f2dd3400640 10 client.25895 handle_command_reply: tid=0 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.132+0000 7f2dfeffd740 1 client.25895 shutdown 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c295390 msgr2=0x55ec0c686f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c295390 0x55ec0c686f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2de00aac70 tx=0x7f2de00af3e0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 shutdown_connections 2024-07-26T11:05:01.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55ec0c249da0 0x55ec0c6ebb70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec0c2b6070 0x55ec0c67ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ec0c295390 0x55ec0c686f10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 --2- 172.21.15.144:0/1087716814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec0c28d1b0 0x55ec0c686950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 >> 172.21.15.144:0/1087716814 conn(0x55ec0c2aa460 msgr2=0x55ec0c673130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:01.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dd3400640 0 client.25895 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:05:01.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 shutdown_connections 2024-07-26T11:05:01.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 1 -- 172.21.15.144:0/1087716814 wait complete. 2024-07-26T11:05:01.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.133+0000 7f2dfeffd740 20 client.25895 trim_cache size 0 max 16384 2024-07-26T11:05:01.138 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2dd80b3d40 msgr2=0x7f2dd80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2dd80b3d40 0x7f2dd80b6250 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2de4031440 tx=0x7f2de4009560 comp rx=0 tx=0).stop 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 msgr2=0x7f2df019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 0x7f2df019c4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2df006c270 tx=0x7f2de000ee50 comp rx=0 tx=0).stop 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 shutdown_connections 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2dd80b3d40 0x7f2dd80b6250 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2df010b860 0x7f2df01a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2df010af10 0x7f2df019c4c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 --2- 172.21.15.144:0/151281716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2df01055d0 0x7f2df019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 >> 172.21.15.144:0/151281716 conn(0x7f2df0101040 msgr2=0x7f2df0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 shutdown_connections 2024-07-26T11:05:01.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:01.134+0000 7f2df7200640 1 -- 172.21.15.144:0/151281716 wait complete. 2024-07-26T11:05:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:01 smithi144 ceph-mon[32129]: pgmap v537: 129 pgs: 129 active+clean; 1.7 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 101 MiB/s wr, 371 op/s 2024-07-26T11:05:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:01 smithi160 ceph-mon[36715]: pgmap v537: 129 pgs: 129 active+clean; 1.7 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 101 MiB/s wr, 371 op/s 2024-07-26T11:05:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:01 smithi183 ceph-mon[37273]: pgmap v537: 129 pgs: 129 active+clean; 1.7 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 101 MiB/s wr, 371 op/s 2024-07-26T11:05:02.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:05:02.492 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:8/831: sync 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:1/954: sync 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:5/831: sync 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:1/955: chown d3/d6/da/cf 29120346 1 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:2/843: sync 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:9/847: sync 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:5/832: chown d1/d9/d44/d45/d98/db2/lca 0 1 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:0/858: sync 2024-07-26T11:05:02.683 INFO:tasks.workunit.client.0.smithi144.stdout:0/859: chown d0/d25/d5e/d1e/f3c 289697 1 2024-07-26T11:05:02.684 INFO:tasks.workunit.client.0.smithi144.stdout:7/847: sync 2024-07-26T11:05:02.684 INFO:tasks.workunit.client.0.smithi144.stdout:3/655: sync 2024-07-26T11:05:02.684 INFO:tasks.workunit.client.0.smithi144.stdout:4/885: sync 2024-07-26T11:05:02.684 INFO:tasks.workunit.client.0.smithi144.stdout:7/848: write d0/dd0/dbf/d2f/d47/d5e/fce [769485,15607] 0 2024-07-26T11:05:02.684 INFO:tasks.workunit.client.0.smithi144.stdout:1/956: creat d3/d3d/d58/d101/d10d/f136 x:0 0 0 2024-07-26T11:05:02.684 INFO:tasks.workunit.client.0.smithi144.stdout:1/957: dread - d3/d6/d75/f115 zero size 2024-07-26T11:05:02.687 INFO:tasks.workunit.client.0.smithi144.stdout:8/832: link d1a/d32/d49/d73/d7b/d4e/d10c/c116 d1a/d32/d49/d73/d7b/d4e/d10c/c11a 0 2024-07-26T11:05:02.687 INFO:tasks.workunit.client.0.smithi144.stdout:8/833: read - d1a/d32/d49/d73/d2c/d38/d58/f10b zero size 2024-07-26T11:05:02.687 INFO:tasks.workunit.client.0.smithi144.stdout:8/834: chown d1a/d32/d49/d73/f79 0 1 2024-07-26T11:05:02.690 INFO:tasks.workunit.client.0.smithi144.stdout:8/835: write d1a/d32/d8c/d9b/fdf [326537,74334] 0 2024-07-26T11:05:02.690 INFO:tasks.workunit.client.0.smithi144.stdout:5/833: symlink d1/d9/d5b/l11f 0 2024-07-26T11:05:02.690 INFO:tasks.workunit.client.0.smithi144.stdout:2/844: creat d0/d24/f100 x:0 0 0 2024-07-26T11:05:02.691 INFO:tasks.workunit.client.0.smithi144.stdout:0/860: creat d0/d25/d5e/d1e/d30/d67/d84/f11a x:0 0 0 2024-07-26T11:05:02.694 INFO:tasks.workunit.client.0.smithi144.stdout:4/886: unlink d9/fd3 0 2024-07-26T11:05:02.694 INFO:tasks.workunit.client.0.smithi144.stdout:3/656: creat d1/d5/fce x:0 0 0 2024-07-26T11:05:02.698 INFO:tasks.workunit.client.0.smithi144.stdout:5/834: mknod d1/d18/da4/db3/c120 0 2024-07-26T11:05:02.703 INFO:tasks.workunit.client.0.smithi144.stdout:1/958: rename d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/de1/f131 to d3/d6/d9c/f137 0 2024-07-26T11:05:02.703 INFO:tasks.workunit.client.0.smithi144.stdout:3/657: symlink d1/d28/d54/d26/d30/d79/lcf 0 2024-07-26T11:05:02.703 INFO:tasks.workunit.client.0.smithi144.stdout:7/849: mknod d0/dd0/dbf/d67/de0/c12e 0 2024-07-26T11:05:02.703 INFO:tasks.workunit.client.0.smithi144.stdout:2/845: mkdir d0/d6/deb/d101 0 2024-07-26T11:05:02.704 INFO:tasks.workunit.client.0.smithi144.stdout:7/850: readlink d0/l1c 0 2024-07-26T11:05:02.704 INFO:tasks.workunit.client.0.smithi144.stdout:4/887: mkdir d9/d7b/dc4/d111/d116/d129 0 2024-07-26T11:05:02.704 INFO:tasks.workunit.client.0.smithi144.stdout:4/888: truncate d9/d26/d48/d50/ffd 4238723 0 2024-07-26T11:05:02.704 INFO:tasks.workunit.client.0.smithi144.stdout:4/889: chown d9/d26/d48/d98/db8/l106 1341086 1 2024-07-26T11:05:02.704 INFO:tasks.workunit.client.0.smithi144.stdout:2/846: creat d0/d24/d7b/ddc/d28/d2e/d77/d84/de3/f102 x:0 0 0 2024-07-26T11:05:02.704 INFO:tasks.workunit.client.0.smithi144.stdout:2/847: stat d0/d24/f100 0 2024-07-26T11:05:02.705 INFO:tasks.workunit.client.0.smithi144.stdout:4/890: rename d9/d26/d3c/d6f/f7c to d9/d26/d3c/d6f/dd0/f12a 0 2024-07-26T11:05:02.705 INFO:tasks.workunit.client.0.smithi144.stdout:4/891: fdatasync d9/f51 0 2024-07-26T11:05:02.707 INFO:tasks.workunit.client.0.smithi144.stdout:4/892: rmdir d9/d7b/dc4/d111/d116 39 2024-07-26T11:05:02.708 INFO:tasks.workunit.client.0.smithi144.stdout:4/893: symlink d9/d7b/dc4/d111/l12b 0 2024-07-26T11:05:02.708 INFO:tasks.workunit.client.0.smithi144.stdout:4/894: dread - d9/d26/d3c/ff5 zero size 2024-07-26T11:05:02.714 INFO:tasks.workunit.client.0.smithi144.stdout:4/895: creat d9/d7b/f12c x:0 0 0 2024-07-26T11:05:02.716 INFO:tasks.workunit.client.0.smithi144.stdout:4/896: symlink d9/d26/d48/d50/d60/dc5/dd9/de1/l12d 0 2024-07-26T11:05:02.736 INFO:tasks.workunit.client.0.smithi144.stdout:3/658: dread d1/d28/d54/d26/f2b [0,4194304] 0 2024-07-26T11:05:02.737 INFO:tasks.workunit.client.0.smithi144.stdout:3/659: write d1/f39 [3096573,86967] 0 2024-07-26T11:05:02.737 INFO:tasks.workunit.client.0.smithi144.stdout:1/959: dread d3/d6/d9c/f137 [0,4194304] 0 2024-07-26T11:05:02.738 INFO:tasks.workunit.client.0.smithi144.stdout:1/960: dread - d3/d6/d9c/d9f/ff2 zero size 2024-07-26T11:05:02.738 INFO:tasks.workunit.client.0.smithi144.stdout:1/961: chown d3/d6/da/d21/l11e 209627941 1 2024-07-26T11:05:02.740 INFO:tasks.workunit.client.0.smithi144.stdout:3/660: mkdir d1/d28/d54/d26/d64/d68/d74/dd0 0 2024-07-26T11:05:02.740 INFO:tasks.workunit.client.0.smithi144.stdout:3/661: fsync d1/d28/d54/d26/f2b 0 2024-07-26T11:05:02.740 INFO:tasks.workunit.client.0.smithi144.stdout:9/848: dread d9/de/d14/d23/d39/fbe [0,4194304] 0 2024-07-26T11:05:02.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/662: rename d1/d28/d54/d26/d64/fbe to d1/d28/d54/d26/d64/d68/d74/fd1 0 2024-07-26T11:05:02.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/663: chown d1/f39 31113638 1 2024-07-26T11:05:02.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/664: readlink d1/d5/d97/d60/lcd 0 2024-07-26T11:05:02.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/665: read - d1/d28/d54/d26/d37/f58 zero size 2024-07-26T11:05:02.742 INFO:tasks.workunit.client.0.smithi144.stdout:3/666: readlink d1/d28/d54/d26/d64/d68/db5/lc3 0 2024-07-26T11:05:02.747 INFO:tasks.workunit.client.0.smithi144.stdout:9/849: creat d9/de/d14/d23/d112/f121 x:0 0 0 2024-07-26T11:05:02.747 INFO:tasks.workunit.client.0.smithi144.stdout:9/850: truncate d9/de/d14/d79/f91 1794241 0 2024-07-26T11:05:02.761 INFO:tasks.workunit.client.0.smithi144.stdout:1/962: read d3/d53/f87 [653580,65306] 0 2024-07-26T11:05:02.761 INFO:tasks.workunit.client.0.smithi144.stdout:1/963: fdatasync d3/d6/d9c/f135 0 2024-07-26T11:05:02.761 INFO:tasks.workunit.client.0.smithi144.stdout:1/964: fdatasync d3/d6/da/d21/d67/fc3 0 2024-07-26T11:05:02.761 INFO:tasks.workunit.client.0.smithi144.stdout:1/965: chown d3/d6/da/d97/f11b 1852211883 1 2024-07-26T11:05:02.768 INFO:tasks.workunit.client.0.smithi144.stdout:1/966: rename d3/d6/da/d70/fa2 to d3/d6/d75/d88/da9/dac/d104/ddb/df6/f138 0 2024-07-26T11:05:02.770 INFO:tasks.workunit.client.0.smithi144.stdout:5/835: dread d1/d9/d44/f4e [0,4194304] 0 2024-07-26T11:05:02.770 INFO:tasks.workunit.client.0.smithi144.stdout:1/967: mkdir d3/d6/d75/d88/da9/dac/d104/ddb/d139 0 2024-07-26T11:05:02.776 INFO:tasks.workunit.client.0.smithi144.stdout:5/836: rename d1/d9/d44/f9d to d1/f121 0 2024-07-26T11:05:02.783 INFO:tasks.workunit.client.0.smithi144.stdout:5/837: mknod d1/d18/dc4/de0/deb/d116/c122 0 2024-07-26T11:05:02.789 INFO:tasks.workunit.client.0.smithi144.stdout:7/851: dread d0/f49 [0,4194304] 0 2024-07-26T11:05:02.792 INFO:tasks.workunit.client.0.smithi144.stdout:7/852: creat d0/d6/d9/f12f x:0 0 0 2024-07-26T11:05:02.792 INFO:tasks.workunit.client.0.smithi144.stdout:7/853: fsync d0/dd0/dbf/d2f/d47/d5e/f69 0 2024-07-26T11:05:02.792 INFO:tasks.workunit.client.0.smithi144.stdout:7/854: chown d0/dd0/dbf/d2f/d53/db6/fed 1007 1 2024-07-26T11:05:02.792 INFO:tasks.workunit.client.0.smithi144.stdout:7/855: stat d0/d6/d10c/d57/cc9 0 2024-07-26T11:05:02.796 INFO:tasks.workunit.client.0.smithi144.stdout:0/861: dwrite d0/d3e/d45/d9f/f82 [0,4194304] 0 2024-07-26T11:05:02.796 INFO:tasks.workunit.client.0.smithi144.stdout:0/862: chown d0/d25/d5e/d1e/d6c/dc6/l108 1656798151 1 2024-07-26T11:05:02.807 INFO:tasks.workunit.client.0.smithi144.stdout:3/667: dwrite d1/d28/d54/d3d/f40 [0,4194304] 0 2024-07-26T11:05:02.807 INFO:tasks.workunit.client.0.smithi144.stdout:3/668: truncate d1/d28/d54/d26/d64/da4/fc6 418209 0 2024-07-26T11:05:02.807 INFO:tasks.workunit.client.0.smithi144.stdout:3/669: write d1/d28/d54/d26/d30/d4d/f55 [532519,6846] 0 2024-07-26T11:05:02.807 INFO:tasks.workunit.client.0.smithi144.stdout:0/863: truncate d0/d25/d5e/d1e/d6c/f97 784713 0 2024-07-26T11:05:02.808 INFO:tasks.workunit.client.0.smithi144.stdout:9/851: dread d9/de/d14/fbb [0,4194304] 0 2024-07-26T11:05:02.809 INFO:tasks.workunit.client.0.smithi144.stdout:1/968: dread d3/d6/da/d13/d19/d3a/f79 [0,4194304] 0 2024-07-26T11:05:02.809 INFO:tasks.workunit.client.0.smithi144.stdout:1/969: dread - d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/f10e zero size 2024-07-26T11:05:02.811 INFO:tasks.workunit.client.0.smithi144.stdout:9/852: creat d9/de/d14/d3f/d5e/f122 x:0 0 0 2024-07-26T11:05:02.812 INFO:tasks.workunit.client.0.smithi144.stdout:3/670: rename d1/d28/d54/d26/d64/d68/f6d to d1/d28/d54/d26/d30/d78/fd2 0 2024-07-26T11:05:02.813 INFO:tasks.workunit.client.0.smithi144.stdout:0/864: symlink d0/d25/d5e/d1e/dc2/l11b 0 2024-07-26T11:05:02.815 INFO:tasks.workunit.client.0.smithi144.stdout:9/853: mkdir d9/de/d14/d23/d39/d7f/d123 0 2024-07-26T11:05:02.824 INFO:tasks.workunit.client.0.smithi144.stdout:3/671: symlink d1/d28/d54/d26/d37/d91/ld3 0 2024-07-26T11:05:02.824 INFO:tasks.workunit.client.0.smithi144.stdout:3/672: fdatasync d1/d5/d97/da9/fb1 0 2024-07-26T11:05:02.824 INFO:tasks.workunit.client.0.smithi144.stdout:9/854: mkdir d9/de/d14/d79/d124 0 2024-07-26T11:05:02.824 INFO:tasks.workunit.client.0.smithi144.stdout:7/856: dread d0/d6/f52 [0,4194304] 0 2024-07-26T11:05:02.825 INFO:tasks.workunit.client.0.smithi144.stdout:9/855: symlink d9/dfc/d114/dcc/l125 0 2024-07-26T11:05:02.825 INFO:tasks.workunit.client.0.smithi144.stdout:2/848: dwrite d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fc9 [0,4194304] 0 2024-07-26T11:05:02.826 INFO:tasks.workunit.client.0.smithi144.stdout:5/838: dread d1/d9/d5b/f8c [0,4194304] 0 2024-07-26T11:05:02.827 INFO:tasks.workunit.client.0.smithi144.stdout:8/836: dwrite d1a/d32/d49/d73/d2c/d38/d84/db0/dd8/dfd/fff [0,4194304] 0 2024-07-26T11:05:02.827 INFO:tasks.workunit.client.0.smithi144.stdout:8/837: readlink d1a/d32/d49/d73/d7b/d43/d4c/d107/l108 0 2024-07-26T11:05:02.827 INFO:tasks.workunit.client.0.smithi144.stdout:7/857: getdents d0 0 2024-07-26T11:05:02.829 INFO:tasks.workunit.client.0.smithi144.stdout:9/856: rename d9/de/d14/f22 to d9/de/d14/d23/d39/d7f/dbc/f126 0 2024-07-26T11:05:02.829 INFO:tasks.workunit.client.0.smithi144.stdout:2/849: mknod d0/d24/d7b/ddc/d28/d2e/c103 0 2024-07-26T11:05:02.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/865: dread d0/d25/d5e/d1e/d30/d6a/f8a [0,4194304] 0 2024-07-26T11:05:02.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/866: chown d0/d25/cba 105609748 1 2024-07-26T11:05:02.831 INFO:tasks.workunit.client.0.smithi144.stdout:0/867: chown d0/l54 61 1 2024-07-26T11:05:02.833 INFO:tasks.workunit.client.0.smithi144.stdout:5/839: symlink d1/d9/l123 0 2024-07-26T11:05:02.835 INFO:tasks.workunit.client.0.smithi144.stdout:4/897: dwrite d9/d26/f83 [0,4194304] 0 2024-07-26T11:05:02.835 INFO:tasks.workunit.client.0.smithi144.stdout:8/838: unlink d1a/d32/d49/d73/d2c/f36 0 2024-07-26T11:05:02.839 INFO:tasks.workunit.client.0.smithi144.stdout:2/850: rename d0/d24/d7b/ddc/l5b to d0/d24/d7b/ddc/d28/l104 0 2024-07-26T11:05:02.839 INFO:tasks.workunit.client.0.smithi144.stdout:2/851: fsync d0/d24/d2b/f8d 0 2024-07-26T11:05:02.839 INFO:tasks.workunit.client.0.smithi144.stdout:2/852: fsync d0/d24/d2b/f8d 0 2024-07-26T11:05:02.839 INFO:tasks.workunit.client.0.smithi144.stdout:2/853: stat d0/d6/f95 0 2024-07-26T11:05:02.840 INFO:tasks.workunit.client.0.smithi144.stdout:0/868: creat d0/d25/d5e/d1e/d6c/dc6/f11c x:0 0 0 2024-07-26T11:05:02.843 INFO:tasks.workunit.client.0.smithi144.stdout:5/840: creat d1/d18/dc4/de0/deb/d116/d11b/f124 x:0 0 0 2024-07-26T11:05:02.843 INFO:tasks.workunit.client.0.smithi144.stdout:5/841: chown d1/d18/d24/d27/f10e 0 1 2024-07-26T11:05:02.844 INFO:tasks.workunit.client.0.smithi144.stdout:4/898: mkdir d9/d7b/dc4/d111/d116/d12e 0 2024-07-26T11:05:02.846 INFO:tasks.workunit.client.0.smithi144.stdout:2/854: link d0/d24/d2b/cfc d0/d24/d48/c105 0 2024-07-26T11:05:02.847 INFO:tasks.workunit.client.0.smithi144.stdout:5/842: creat d1/d5/dae/f125 x:0 0 0 2024-07-26T11:05:02.852 INFO:tasks.workunit.client.0.smithi144.stdout:2/855: unlink d0/d6/d9/d44/fa2 0 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/843: creat d1/d9/d5b/d86/f126 x:0 0 0 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:3/673: dread d1/f17 [0,4194304] 0 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/844: write d1/d9/d5b/d4d/f63 [2503848,118228] 0 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/845: chown d1/d9/d44/d45/d98/db2/f11a 832892 1 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/846: dread - d1/d18/dc4/ded/dc8/f102 zero size 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/847: write d1/d9/d5b/f69 [4510425,8196] 0 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/848: getdents d1/d18/d24/d35/d10f/dea/df9 0 2024-07-26T11:05:02.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/849: rename d1/d9/d44/d45/daa to d1/d18/d127 0 2024-07-26T11:05:02.854 INFO:tasks.workunit.client.0.smithi144.stdout:5/850: mkdir d1/d18/da4/d128 0 2024-07-26T11:05:02.856 INFO:tasks.workunit.client.0.smithi144.stdout:5/851: creat d1/d18/d24/d27/d71/f129 x:0 0 0 2024-07-26T11:05:02.865 INFO:tasks.workunit.client.0.smithi144.stdout:7/858: dread d0/dd0/dbf/d2f/d53/db6/fca [0,4194304] 0 2024-07-26T11:05:02.869 INFO:tasks.workunit.client.0.smithi144.stdout:0/869: dread d0/d25/d5e/d1e/dc1/feb [0,4194304] 0 2024-07-26T11:05:02.884 INFO:tasks.workunit.client.0.smithi144.stdout:2/856: write d0/d24/d48/d63/da3/fbc [2331438,119747] 0 2024-07-26T11:05:02.885 INFO:tasks.workunit.client.0.smithi144.stdout:2/857: rmdir d0/d6/d34 39 2024-07-26T11:05:02.885 INFO:tasks.workunit.client.0.smithi144.stdout:2/858: chown d0/d24/d2b/f7e 144682911 1 2024-07-26T11:05:02.886 INFO:tasks.workunit.client.0.smithi144.stdout:2/859: getdents d0/d6/d34/d6c 0 2024-07-26T11:05:02.886 INFO:tasks.workunit.client.0.smithi144.stdout:9/857: dwrite d9/de/d14/d23/d39/fc1 [0,4194304] 0 2024-07-26T11:05:02.891 INFO:tasks.workunit.client.0.smithi144.stdout:8/839: dwrite d1a/d32/d49/d73/f79 [0,4194304] 0 2024-07-26T11:05:02.891 INFO:tasks.workunit.client.0.smithi144.stdout:9/858: rename d9/de/d14/d23/d112/fd3 to d9/d3b/d81/ddd/f127 0 2024-07-26T11:05:02.892 INFO:tasks.workunit.client.0.smithi144.stdout:5/852: dread d1/d9/d44/f62 [0,4194304] 0 2024-07-26T11:05:02.893 INFO:tasks.workunit.client.0.smithi144.stdout:9/859: symlink d9/de/d14/d23/d112/l128 0 2024-07-26T11:05:02.893 INFO:tasks.workunit.client.0.smithi144.stdout:5/853: rmdir d1/d18/dc4/de0/deb 39 2024-07-26T11:05:02.894 INFO:tasks.workunit.client.0.smithi144.stdout:9/860: creat d9/d3b/dce/de4/f129 x:0 0 0 2024-07-26T11:05:02.896 INFO:tasks.workunit.client.0.smithi144.stdout:9/861: creat d9/de/d14/d23/d39/f12a x:0 0 0 2024-07-26T11:05:02.898 INFO:tasks.workunit.client.0.smithi144.stdout:9/862: creat d9/de/d14/f12b x:0 0 0 2024-07-26T11:05:02.902 INFO:tasks.workunit.client.0.smithi144.stdout:5/854: truncate d1/d5/f2d 1445816 0 2024-07-26T11:05:02.911 INFO:tasks.workunit.client.0.smithi144.stdout:5/855: creat d1/d5/dae/f12a x:0 0 0 2024-07-26T11:05:02.913 INFO:tasks.workunit.client.0.smithi144.stdout:3/674: dwrite d1/ff [4194304,4194304] 0 2024-07-26T11:05:02.913 INFO:tasks.workunit.client.0.smithi144.stdout:3/675: dread - d1/d28/d54/d26/d37/da1/fcc zero size 2024-07-26T11:05:02.913 INFO:tasks.workunit.client.0.smithi144.stdout:3/676: write d1/d5/d97/d48/fad [531295,79635] 0 2024-07-26T11:05:02.926 INFO:tasks.workunit.client.0.smithi144.stdout:9/863: dread d9/de/d14/d79/fa7 [0,4194304] 0 2024-07-26T11:05:02.926 INFO:tasks.workunit.client.0.smithi144.stdout:9/864: stat d9/dfc/d114/dcc/l125 0 2024-07-26T11:05:02.926 INFO:tasks.workunit.client.0.smithi144.stdout:9/865: symlink d9/de/d14/l12c 0 2024-07-26T11:05:02.926 INFO:tasks.workunit.client.0.smithi144.stdout:9/866: stat d9/c17 0 2024-07-26T11:05:02.931 INFO:tasks.workunit.client.0.smithi144.stdout:0/870: write d0/d25/d5e/d1e/d91/f96 [207966,41717] 0 2024-07-26T11:05:02.931 INFO:tasks.workunit.client.0.smithi144.stdout:0/871: stat d0/d3e/d45/d9f/df4/d105 0 2024-07-26T11:05:02.932 INFO:tasks.workunit.client.0.smithi144.stdout:7/859: dwrite d0/dd0/dbf/d2f/d68/fc6 [0,4194304] 0 2024-07-26T11:05:02.933 INFO:tasks.workunit.client.0.smithi144.stdout:0/872: creat d0/d25/d5e/d1e/d30/d6a/dac/f11d x:0 0 0 2024-07-26T11:05:02.937 INFO:tasks.workunit.client.0.smithi144.stdout:0/873: getdents d0/d3e 0 2024-07-26T11:05:02.938 INFO:tasks.workunit.client.0.smithi144.stdout:0/874: fsync d0/d3e/d45/d9f/fa8 0 2024-07-26T11:05:02.941 INFO:tasks.workunit.client.0.smithi144.stdout:0/875: mkdir d0/d3e/db4/d11e 0 2024-07-26T11:05:02.945 INFO:tasks.workunit.client.0.smithi144.stdout:0/876: rename d0/d25 to d0/d3e/d45/dd4/d11f 0 2024-07-26T11:05:02.946 INFO:tasks.workunit.client.0.smithi144.stdout:3/677: dread d1/d28/d54/d26/d64/da4/fb0 [0,4194304] 0 2024-07-26T11:05:02.946 INFO:tasks.workunit.client.0.smithi144.stdout:3/678: read d1/d5/fa [427278,22210] 0 2024-07-26T11:05:02.946 INFO:tasks.workunit.client.0.smithi144.stdout:0/877: rmdir d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/d6f/dec 39 2024-07-26T11:05:02.946 INFO:tasks.workunit.client.0.smithi144.stdout:0/878: stat d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/d84/l102 0 2024-07-26T11:05:02.948 INFO:tasks.workunit.client.0.smithi144.stdout:0/879: link d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d4b/c59 d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/d84/c120 0 2024-07-26T11:05:02.949 INFO:tasks.workunit.client.0.smithi144.stdout:7/860: dread d0/dd0/dbf/d2f/d53/db6/fed [0,4194304] 0 2024-07-26T11:05:02.949 INFO:tasks.workunit.client.0.smithi144.stdout:0/880: symlink d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/d84/l121 0 2024-07-26T11:05:02.950 INFO:tasks.workunit.client.0.smithi144.stdout:0/881: readlink d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d4b/lf5 0 2024-07-26T11:05:02.952 INFO:tasks.workunit.client.0.smithi144.stdout:5/856: dwrite d1/d9/d5b/d86/ff4 [0,4194304] 0 2024-07-26T11:05:02.952 INFO:tasks.workunit.client.0.smithi144.stdout:5/857: chown d1/d5/dae/f12a 15 1 2024-07-26T11:05:02.952 INFO:tasks.workunit.client.0.smithi144.stdout:5/858: write d1/d9/d44/d45/d98/faf [219904,120005] 0 2024-07-26T11:05:02.952 INFO:tasks.workunit.client.0.smithi144.stdout:0/882: mkdir d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/d122 0 2024-07-26T11:05:02.952 INFO:tasks.workunit.client.0.smithi144.stdout:0/883: write d0/d3e/d45/d9f/df4/f10b [738233,34590] 0 2024-07-26T11:05:02.954 INFO:tasks.workunit.client.0.smithi144.stdout:5/859: mkdir d1/d18/d12b 0 2024-07-26T11:05:02.962 INFO:tasks.workunit.client.0.smithi144.stdout:9/867: dwrite d9/dfc/d114/dcc/fde [0,4194304] 0 2024-07-26T11:05:02.965 INFO:tasks.workunit.client.0.smithi144.stdout:9/868: mkdir d9/dfc/d114/d116/d12d 0 2024-07-26T11:05:02.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/679: write d1/f50 [3120493,39481] 0 2024-07-26T11:05:02.968 INFO:tasks.workunit.client.0.smithi144.stdout:3/680: chown d1/d28/d54/d26/d37/da1 1449066 1 2024-07-26T11:05:02.970 INFO:tasks.workunit.client.0.smithi144.stdout:3/681: truncate d1/d28/d54/d26/d30/d78/fa5 350293 0 2024-07-26T11:05:02.976 INFO:tasks.workunit.client.0.smithi144.stdout:5/860: write d1/f39 [4122028,97972] 0 2024-07-26T11:05:02.985 INFO:tasks.workunit.client.0.smithi144.stdout:7/861: write d0/d6/d10c/f90 [3081467,93383] 0 2024-07-26T11:05:02.990 INFO:tasks.workunit.client.0.smithi144.stdout:5/861: dread d1/d9/d44/d45/f56 [0,4194304] 0 2024-07-26T11:05:02.995 INFO:tasks.workunit.client.0.smithi144.stdout:0/884: dwrite d0/d3e/d5f/f8d [0,4194304] 0 2024-07-26T11:05:02.995 INFO:tasks.workunit.client.0.smithi144.stdout:9/869: dread d9/d3b/d81/ddd/f127 [0,4194304] 0 2024-07-26T11:05:02.995 INFO:tasks.workunit.client.0.smithi144.stdout:0/885: write d0/d3e/db8/fcf [368879,52010] 0 2024-07-26T11:05:02.995 INFO:tasks.workunit.client.0.smithi144.stdout:0/886: truncate d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/d84/fbd 1150325 0 2024-07-26T11:05:02.996 INFO:tasks.workunit.client.0.smithi144.stdout:9/870: getdents d9/d3b/dce 0 2024-07-26T11:05:02.999 INFO:tasks.workunit.client.0.smithi144.stdout:9/871: rename d9/d3b/d5d/f110 to d9/dfc/d114/dcc/dea/f12e 0 2024-07-26T11:05:02.999 INFO:tasks.workunit.client.0.smithi144.stdout:7/862: mknod d0/d6/d9/d3f/c130 0 2024-07-26T11:05:03.001 INFO:tasks.workunit.client.0.smithi144.stdout:9/872: mknod d9/de/d14/c12f 0 2024-07-26T11:05:03.001 INFO:tasks.workunit.client.0.smithi144.stdout:9/873: write d9/de/d14/d3f/d5e/fc0 [725501,114694] 0 2024-07-26T11:05:03.001 INFO:tasks.workunit.client.0.smithi144.stdout:9/874: write d9/f29 [1031580,18228] 0 2024-07-26T11:05:03.002 INFO:tasks.workunit.client.0.smithi144.stdout:9/875: chown d9/de/fd8 14 1 2024-07-26T11:05:03.002 INFO:tasks.workunit.client.0.smithi144.stdout:7/863: creat d0/d71/dc0/de4/f131 x:0 0 0 2024-07-26T11:05:03.003 INFO:tasks.workunit.client.0.smithi144.stdout:7/864: creat d0/d6/d9/f132 x:0 0 0 2024-07-26T11:05:03.008 INFO:tasks.workunit.client.0.smithi144.stdout:9/876: dread d9/de/d14/d3f/f61 [0,4194304] 0 2024-07-26T11:05:03.018 INFO:tasks.workunit.client.0.smithi144.stdout:9/877: symlink d9/d3b/l130 0 2024-07-26T11:05:03.019 INFO:tasks.workunit.client.0.smithi144.stdout:9/878: getdents d9/dfc/d114/d116/d12d 0 2024-07-26T11:05:03.020 INFO:tasks.workunit.client.0.smithi144.stdout:5/862: write d1/d9/f10 [2117956,115998] 0 2024-07-26T11:05:03.023 INFO:tasks.workunit.client.0.smithi144.stdout:0/887: dread d0/d3e/d45/dd4/d11f/d5e/f41 [4194304,4194304] 0 2024-07-26T11:05:03.025 INFO:tasks.workunit.client.0.smithi144.stdout:9/879: symlink d9/de/d4e/d6a/deb/l131 0 2024-07-26T11:05:03.026 INFO:tasks.workunit.client.0.smithi144.stdout:0/888: mkdir d0/d3e/d45/dd4/d11f/d5e/d1e/d6c/dc6/d123 0 2024-07-26T11:05:03.027 INFO:tasks.workunit.client.0.smithi144.stdout:9/880: creat d9/de/d14/d23/d112/f132 x:0 0 0 2024-07-26T11:05:03.028 INFO:tasks.workunit.client.0.smithi144.stdout:3/682: dwrite d1/d5/d97/f4e [0,4194304] 0 2024-07-26T11:05:03.028 INFO:tasks.workunit.client.0.smithi144.stdout:3/683: fdatasync d1/d5/d97/f4e 0 2024-07-26T11:05:03.029 INFO:tasks.workunit.client.0.smithi144.stdout:7/865: dread d0/f49 [0,4194304] 0 2024-07-26T11:05:03.029 INFO:tasks.workunit.client.0.smithi144.stdout:7/866: chown d0/d71/d95/lea 81765339 1 2024-07-26T11:05:03.031 INFO:tasks.workunit.client.0.smithi144.stdout:3/684: creat d1/d28/d54/d3d/d42/fd4 x:0 0 0 2024-07-26T11:05:03.031 INFO:tasks.workunit.client.0.smithi144.stdout:7/867: creat d0/d6/d9/d12/d115/f133 x:0 0 0 2024-07-26T11:05:03.031 INFO:tasks.workunit.client.0.smithi144.stdout:7/868: fsync d0/dd0/dbf/d2f/da2/fbe 0 2024-07-26T11:05:03.032 INFO:tasks.workunit.client.0.smithi144.stdout:3/685: unlink d1/d28/d54/d61/c90 0 2024-07-26T11:05:03.034 INFO:tasks.workunit.client.0.smithi144.stdout:7/869: mkdir d0/d71/d95/d134 0 2024-07-26T11:05:03.034 INFO:tasks.workunit.client.0.smithi144.stdout:7/870: getdents d0/d71/df5 0 2024-07-26T11:05:03.036 INFO:tasks.workunit.client.0.smithi144.stdout:3/686: mkdir d1/d28/d54/d26/d64/dd5 0 2024-07-26T11:05:03.036 INFO:tasks.workunit.client.0.smithi144.stdout:3/687: chown d1/d28/d54/d26/d64/fbd 57246 1 2024-07-26T11:05:03.046 INFO:tasks.workunit.client.0.smithi144.stdout:7/871: rename d0/d6/d9/d12/c102 to d0/dd0/dbf/d2f/d47/d5e/d116/c135 0 2024-07-26T11:05:03.047 INFO:tasks.workunit.client.0.smithi144.stdout:7/872: read - d0/dd0/dbf/d2f/d47/fb7 zero size 2024-07-26T11:05:03.048 INFO:tasks.workunit.client.0.smithi144.stdout:3/688: unlink d1/d5/d97/d60/c57 0 2024-07-26T11:05:03.048 INFO:tasks.workunit.client.0.smithi144.stdout:7/873: mkdir d0/d71/d95/de7/d136 0 2024-07-26T11:05:03.049 INFO:tasks.workunit.client.0.smithi144.stdout:0/889: dread d0/d3e/d45/dd4/d11f/d5e/d1e/d30/f39 [0,4194304] 0 2024-07-26T11:05:03.050 INFO:tasks.workunit.client.0.smithi144.stdout:0/890: mknod d0/d3e/d45/dd4/d11f/d5e/d1e/dd2/c124 0 2024-07-26T11:05:03.051 INFO:tasks.workunit.client.0.smithi144.stdout:0/891: rename d0/d3e/d45/dd4/d11f/d5e/c2b to d0/d3e/d45/dd4/d11f/d5e/c125 0 2024-07-26T11:05:03.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/892: rmdir d0/d3e/d45/dd4/d11f/d5e/d1e/d6c/dc6/d123 0 2024-07-26T11:05:03.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/893: chown d0/f10 46024874 1 2024-07-26T11:05:03.053 INFO:tasks.workunit.client.0.smithi144.stdout:0/894: write d0/d3e/d45/dd4/d11f/d5e/d1e/dc2/ffc [948299,3736] 0 2024-07-26T11:05:03.078 INFO:tasks.workunit.client.0.smithi144.stdout:7/874: dread d0/dd0/dbf/d2f/d68/fc6 [0,4194304] 0 2024-07-26T11:05:03.078 INFO:tasks.workunit.client.0.smithi144.stdout:7/875: fsync d0/dd0/dbf/d2f/d68/f8d 0 2024-07-26T11:05:03.081 INFO:tasks.workunit.client.0.smithi144.stdout:5/863: dwrite d1/d9/d44/d45/f112 [0,4194304] 0 2024-07-26T11:05:03.081 INFO:tasks.workunit.client.0.smithi144.stdout:5/864: fsync d1/d9/d5b/d86/f126 0 2024-07-26T11:05:03.082 INFO:tasks.workunit.client.0.smithi144.stdout:5/865: getdents d1/d18/d24/d35/dd7 0 2024-07-26T11:05:03.082 INFO:tasks.workunit.client.0.smithi144.stdout:7/876: mkdir d0/d6/d10c/d137 0 2024-07-26T11:05:03.083 INFO:tasks.workunit.client.0.smithi144.stdout:5/866: rename d1/d9/d44/d45/d98 to d1/d18/d24/d35/dd7/d12c 0 2024-07-26T11:05:03.085 INFO:tasks.workunit.client.0.smithi144.stdout:7/877: unlink d0/dd0/dbf/d2f/d53/l75 0 2024-07-26T11:05:03.085 INFO:tasks.workunit.client.0.smithi144.stdout:7/878: truncate d0/dd0/dbf/d2f/d47/d5e/f63 1308735 0 2024-07-26T11:05:03.085 INFO:tasks.workunit.client.0.smithi144.stdout:5/867: symlink d1/d18/dc4/l12d 0 2024-07-26T11:05:03.085 INFO:tasks.workunit.client.0.smithi144.stdout:5/868: chown d1/d9/d5b/d4d/f16 11 1 2024-07-26T11:05:03.104 INFO:tasks.workunit.client.0.smithi144.stdout:0/895: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/fc8 [0,4194304] 0 2024-07-26T11:05:03.110 INFO:tasks.workunit.client.0.smithi144.stdout:0/896: mknod d0/d3e/d45/dd4/d11f/d5e/d1e/dc1/dd8/c126 0 2024-07-26T11:05:03.110 INFO:tasks.workunit.client.0.smithi144.stdout:9/881: dwrite d9/de/d14/d23/d39/f12a [0,4194304] 0 2024-07-26T11:05:03.111 INFO:tasks.workunit.client.0.smithi144.stdout:9/882: symlink d9/de/d4e/d6a/deb/l133 0 2024-07-26T11:05:03.111 INFO:tasks.workunit.client.0.smithi144.stdout:9/883: chown d9/de/d4e/d6a/deb/dfd/f113 1 1 2024-07-26T11:05:03.113 INFO:tasks.workunit.client.0.smithi144.stdout:9/884: symlink d9/dfc/d114/dcc/dea/l134 0 2024-07-26T11:05:03.114 INFO:tasks.workunit.client.0.smithi144.stdout:9/885: mknod d9/de/d14/d23/c135 0 2024-07-26T11:05:03.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/886: mkdir d9/de/d14/d79/d124/d136 0 2024-07-26T11:05:03.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/887: stat d9/de/d14/d23/d112/l128 0 2024-07-26T11:05:03.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/888: readlink d9/de/d14/d3f/d5e/l87 0 2024-07-26T11:05:03.116 INFO:tasks.workunit.client.0.smithi144.stdout:9/889: fdatasync d9/de/d14/f35 0 2024-07-26T11:05:03.121 INFO:tasks.workunit.client.0.smithi144.stdout:9/890: symlink d9/de/d14/l137 0 2024-07-26T11:05:03.122 INFO:tasks.workunit.client.0.smithi144.stdout:9/891: mknod d9/de/d4e/d6a/deb/c138 0 2024-07-26T11:05:03.123 INFO:tasks.workunit.client.0.smithi144.stdout:9/892: stat d9/c11 0 2024-07-26T11:05:03.124 INFO:tasks.workunit.client.0.smithi144.stdout:9/893: readlink d9/d3b/l3c 0 2024-07-26T11:05:03.130 INFO:tasks.workunit.client.0.smithi144.stdout:9/894: rename d9/de/lec to d9/de/d4e/l139 0 2024-07-26T11:05:03.130 INFO:tasks.workunit.client.0.smithi144.stdout:9/895: chown d9/de/f52 278141 1 2024-07-26T11:05:03.141 INFO:tasks.workunit.client.0.smithi144.stdout:5/869: dwrite d1/f97 [0,4194304] 0 2024-07-26T11:05:03.144 INFO:tasks.workunit.client.0.smithi144.stdout:5/870: link d1/c83 d1/d5/dae/c12e 0 2024-07-26T11:05:03.144 INFO:tasks.workunit.client.0.smithi144.stdout:5/871: dread - d1/d18/d24/fd6 zero size 2024-07-26T11:05:03.145 INFO:tasks.workunit.client.0.smithi144.stdout:5/872: creat d1/d18/f12f x:0 0 0 2024-07-26T11:05:03.146 INFO:tasks.workunit.client.0.smithi144.stdout:5/873: creat d1/d18/da4/f130 x:0 0 0 2024-07-26T11:05:03.147 INFO:tasks.workunit.client.0.smithi144.stdout:5/874: fsync d1/d5/dae/f125 0 2024-07-26T11:05:03.147 INFO:tasks.workunit.client.0.smithi144.stdout:5/875: truncate d1/d18/d24/fd6 341982 0 2024-07-26T11:05:03.147 INFO:tasks.workunit.client.0.smithi144.stdout:5/876: write d1/d18/d24/f82 [669967,84288] 0 2024-07-26T11:05:03.148 INFO:tasks.workunit.client.0.smithi144.stdout:5/877: mknod d1/d18/d24/d35/dd7/d12c/db2/c131 0 2024-07-26T11:05:03.148 INFO:tasks.workunit.client.0.smithi144.stdout:5/878: truncate d1/d9/d44/da1/ddb/df7/f10d 5103960 0 2024-07-26T11:05:03.148 INFO:tasks.workunit.client.0.smithi144.stdout:5/879: fsync d1/d18/d24/fb1 0 2024-07-26T11:05:03.151 INFO:tasks.workunit.client.0.smithi144.stdout:9/896: write d9/d3b/dce/fe6 [1384827,98179] 0 2024-07-26T11:05:03.151 INFO:tasks.workunit.client.0.smithi144.stdout:0/897: dread d0/d3e/f69 [0,4194304] 0 2024-07-26T11:05:03.153 INFO:tasks.workunit.client.0.smithi144.stdout:0/898: write d0/d3e/d45/d4c/f79 [289146,6033] 0 2024-07-26T11:05:03.153 INFO:tasks.workunit.client.0.smithi144.stdout:9/897: truncate d9/de/d14/d23/fa3 1093699 0 2024-07-26T11:05:03.153 INFO:tasks.workunit.client.0.smithi144.stdout:9/898: truncate d9/d3b/f78 1823711 0 2024-07-26T11:05:03.153 INFO:tasks.workunit.client.0.smithi144.stdout:7/879: dwrite d0/dd0/dbf/d67/f9b [0,4194304] 0 2024-07-26T11:05:03.155 INFO:tasks.workunit.client.0.smithi144.stdout:7/880: creat d0/dd0/dbf/d2f/f138 x:0 0 0 2024-07-26T11:05:03.155 INFO:tasks.workunit.client.0.smithi144.stdout:7/881: write d0/dd0/dbf/d67/de0/f10a [478995,932] 0 2024-07-26T11:05:03.157 INFO:tasks.workunit.client.0.smithi144.stdout:7/882: mkdir d0/d71/df5/d139 0 2024-07-26T11:05:03.159 INFO:tasks.workunit.client.0.smithi144.stdout:7/883: mknod d0/c13a 0 2024-07-26T11:05:03.160 INFO:tasks.workunit.client.0.smithi144.stdout:7/884: chown d0/dd0/dbf/d2f/d53/db6/dfc/f103 8 1 2024-07-26T11:05:03.162 INFO:tasks.workunit.client.0.smithi144.stdout:7/885: unlink d0/dd0/dbf/d2f/d53/db6/dfc/f109 0 2024-07-26T11:05:03.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/880: dread d1/d9/d44/f62 [0,4194304] 0 2024-07-26T11:05:03.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/881: chown d1/d9/d44/d45/d81/fe1 26085 1 2024-07-26T11:05:03.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/882: chown d1/d18/da4/d128 1197639625 1 2024-07-26T11:05:03.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/883: readlink d1/d18/d24/l5f 0 2024-07-26T11:05:03.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/884: chown d1/d9/d44/d45/d81/f9e 461792779 1 2024-07-26T11:05:03.168 INFO:tasks.workunit.client.0.smithi144.stdout:5/885: truncate d1/d18/fdf 2856353 0 2024-07-26T11:05:03.169 INFO:tasks.workunit.client.0.smithi144.stdout:5/886: mkdir d1/d18/da4/d132 0 2024-07-26T11:05:03.171 INFO:tasks.workunit.client.0.smithi144.stdout:5/887: mkdir d1/d18/d24/d35/d10f/d133 0 2024-07-26T11:05:03.174 INFO:tasks.workunit.client.0.smithi144.stdout:9/899: dread d9/de/d14/d23/d39/fc1 [0,4194304] 0 2024-07-26T11:05:03.182 INFO:tasks.workunit.client.0.smithi144.stdout:9/900: dread d9/de/f15 [0,4194304] 0 2024-07-26T11:05:03.187 INFO:tasks.workunit.client.0.smithi144.stdout:7/886: dread d0/dd0/dbf/d67/f9b [0,4194304] 0 2024-07-26T11:05:03.187 INFO:tasks.workunit.client.0.smithi144.stdout:7/887: truncate d0/dd0/dbf/f12d 560889 0 2024-07-26T11:05:03.188 INFO:tasks.workunit.client.0.smithi144.stdout:7/888: link d0/d6/d55/f6a d0/dd0/dbf/d2f/da2/f13b 0 2024-07-26T11:05:03.188 INFO:tasks.workunit.client.0.smithi144.stdout:7/889: write d0/f2 [3437398,44910] 0 2024-07-26T11:05:03.202 INFO:tasks.workunit.client.0.smithi144.stdout:0/899: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/f90 [0,4194304] 0 2024-07-26T11:05:03.204 INFO:tasks.workunit.client.0.smithi144.stdout:0/900: unlink d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d4b/c59 0 2024-07-26T11:05:03.209 INFO:tasks.workunit.client.0.smithi144.stdout:5/888: dwrite d1/d18/d24/d35/dd7/d12c/fbe [0,4194304] 0 2024-07-26T11:05:03.212 INFO:tasks.workunit.client.0.smithi144.stdout:5/889: symlink d1/d9/d44/d45/d81/l134 0 2024-07-26T11:05:03.247 INFO:tasks.workunit.client.0.smithi144.stdout:9/901: dwrite d9/de/d14/d23/d39/d7f/f84 [0,4194304] 0 2024-07-26T11:05:03.247 INFO:tasks.workunit.client.0.smithi144.stdout:9/902: stat d9/de/d14/d3f/f61 0 2024-07-26T11:05:03.247 INFO:tasks.workunit.client.0.smithi144.stdout:0/901: dread d0/d3e/d45/d9f/f6d [0,4194304] 0 2024-07-26T11:05:03.251 INFO:tasks.workunit.client.0.smithi144.stdout:9/903: link d9/d3b/d81/fb4 d9/de/d14/d23/d39/d7f/dbc/f13a 0 2024-07-26T11:05:03.253 INFO:tasks.workunit.client.0.smithi144.stdout:9/904: symlink d9/de/l13b 0 2024-07-26T11:05:03.255 INFO:tasks.workunit.client.0.smithi144.stdout:9/905: mknod d9/d3b/d76/c13c 0 2024-07-26T11:05:03.256 INFO:tasks.workunit.client.0.smithi144.stdout:9/906: mknod d9/de/d14/d3f/d5e/c13d 0 2024-07-26T11:05:03.258 INFO:tasks.workunit.client.0.smithi144.stdout:9/907: creat d9/de/d14/d3f/d5e/f13e x:0 0 0 2024-07-26T11:05:03.265 INFO:tasks.workunit.client.0.smithi144.stdout:7/890: dwrite d0/dd0/dbf/d67/f11c [0,4194304] 0 2024-07-26T11:05:03.265 INFO:tasks.workunit.client.0.smithi144.stdout:7/891: write d0/d6/d10c/d57/f77 [8033216,4764] 0 2024-07-26T11:05:03.265 INFO:tasks.workunit.client.0.smithi144.stdout:7/892: chown d0/d6/d9/f10d 207805 1 2024-07-26T11:05:03.265 INFO:tasks.workunit.client.0.smithi144.stdout:7/893: chown d0/d6/d10c/f8e 32 1 2024-07-26T11:05:03.266 INFO:tasks.workunit.client.0.smithi144.stdout:7/894: dread - d0/d6/d9/d12/d6c/fc1 zero size 2024-07-26T11:05:03.266 INFO:tasks.workunit.client.0.smithi144.stdout:7/895: write d0/d6/d9/d12/f5d [5125914,91395] 0 2024-07-26T11:05:03.266 INFO:tasks.workunit.client.0.smithi144.stdout:7/896: write d0/d6/d10c/f11b [175335,55993] 0 2024-07-26T11:05:03.266 INFO:tasks.workunit.client.0.smithi144.stdout:9/908: dread d9/de/d14/d3f/d5e/f6c [0,4194304] 0 2024-07-26T11:05:03.271 INFO:tasks.workunit.client.0.smithi144.stdout:9/909: mknod d9/de/d14/d23/d39/d7f/dbc/d10c/c13f 0 2024-07-26T11:05:03.279 INFO:tasks.workunit.client.0.smithi144.stdout:0/902: read d0/d3e/d45/dd4/d11f/d5e/f2c [2450628,82292] 0 2024-07-26T11:05:03.284 INFO:tasks.workunit.client.0.smithi144.stdout:5/890: dwrite d1/d9/d44/d45/fc1 [0,4194304] 0 2024-07-26T11:05:03.302 INFO:tasks.workunit.client.0.smithi144.stdout:0/903: dread d0/d3e/d45/d9f/f6d [0,4194304] 0 2024-07-26T11:05:03.324 INFO:tasks.workunit.client.0.smithi144.stdout:7/897: dwrite d0/d6/d10c/f8e [0,4194304] 0 2024-07-26T11:05:03.324 INFO:tasks.workunit.client.0.smithi144.stdout:1/970: sync 2024-07-26T11:05:03.326 INFO:tasks.workunit.client.0.smithi144.stdout:7/898: rmdir d0/d6/d10c 39 2024-07-26T11:05:03.329 INFO:tasks.workunit.client.0.smithi144.stdout:1/971: rmdir d3/d3d/d128 39 2024-07-26T11:05:03.333 INFO:tasks.workunit.client.0.smithi144.stdout:9/910: dwrite d9/de/f4b [0,4194304] 0 2024-07-26T11:05:03.335 INFO:tasks.workunit.client.0.smithi144.stdout:9/911: mkdir d9/de/d140 0 2024-07-26T11:05:03.335 INFO:tasks.workunit.client.0.smithi144.stdout:1/972: rename d3/d3d/d58/d101/f5a to d3/d6/da/d13/d19/d3a/f13a 0 2024-07-26T11:05:03.336 INFO:tasks.workunit.client.0.smithi144.stdout:0/904: dread d0/d3e/d5f/f8d [0,4194304] 0 2024-07-26T11:05:03.338 INFO:tasks.workunit.client.0.smithi144.stdout:9/912: symlink d9/d3b/d81/ddd/l141 0 2024-07-26T11:05:03.344 INFO:tasks.workunit.client.0.smithi144.stdout:9/913: mkdir d9/dfc/d114/dcc/d142 0 2024-07-26T11:05:03.344 INFO:tasks.workunit.client.0.smithi144.stdout:9/914: dread - d9/de/d14/d23/d112/fe1 zero size 2024-07-26T11:05:03.344 INFO:tasks.workunit.client.0.smithi144.stdout:9/915: write d9/d3b/dce/fe8 [432773,67034] 0 2024-07-26T11:05:03.344 INFO:tasks.workunit.client.0.smithi144.stdout:9/916: fdatasync d9/d3b/d5d/fe3 0 2024-07-26T11:05:03.344 INFO:tasks.workunit.client.0.smithi144.stdout:1/973: creat d3/d6/d75/f13b x:0 0 0 2024-07-26T11:05:03.345 INFO:tasks.workunit.client.0.smithi144.stdout:5/891: dwrite d1/d18/d24/d35/dd7/fff [0,4194304] 0 2024-07-26T11:05:03.346 INFO:tasks.workunit.client.0.smithi144.stdout:1/974: mkdir d3/d3d/d58/d101/d36/db6/d13c 0 2024-07-26T11:05:03.346 INFO:tasks.workunit.client.0.smithi144.stdout:1/975: getdents d3/d6/da/d13/d12f 0 2024-07-26T11:05:03.349 INFO:tasks.workunit.client.0.smithi144.stdout:5/892: creat d1/d9/d44/da1/ddb/df7/f135 x:0 0 0 2024-07-26T11:05:03.349 INFO:tasks.workunit.client.0.smithi144.stdout:5/893: chown d1/d9/d5b/lf2 342045794 1 2024-07-26T11:05:03.349 INFO:tasks.workunit.client.0.smithi144.stdout:1/976: truncate d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/f30 1466251 0 2024-07-26T11:05:03.351 INFO:tasks.workunit.client.0.smithi144.stdout:0/905: rename d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d4b/fb6 to d0/d3e/d45/dd4/d11f/f127 0 2024-07-26T11:05:03.352 INFO:tasks.workunit.client.0.smithi144.stdout:1/977: creat d3/d6/d75/d88/da9/dac/d104/ddb/d139/f13d x:0 0 0 2024-07-26T11:05:03.352 INFO:tasks.workunit.client.0.smithi144.stdout:1/978: chown d3/d6/da/d21/d5c/d62/fd2 10714407 1 2024-07-26T11:05:03.364 INFO:tasks.workunit.client.0.smithi144.stdout:1/979: getdents d3/d3d/d58/d101 0 2024-07-26T11:05:03.377 INFO:tasks.workunit.client.0.smithi144.stdout:1/980: mkdir d3/d6/da/d13/d12f/d13e 0 2024-07-26T11:05:03.382 INFO:tasks.workunit.client.0.smithi144.stdout:3/689: sync 2024-07-26T11:05:03.382 INFO:tasks.workunit.client.0.smithi144.stdout:8/840: sync 2024-07-26T11:05:03.382 INFO:tasks.workunit.client.0.smithi144.stdout:2/860: sync 2024-07-26T11:05:03.382 INFO:tasks.workunit.client.0.smithi144.stdout:4/899: sync 2024-07-26T11:05:03.382 INFO:tasks.workunit.client.0.smithi144.stdout:3/690: dread - d1/d5/d97/fa2 zero size 2024-07-26T11:05:03.383 INFO:tasks.workunit.client.0.smithi144.stdout:4/900: write d9/da/fb7 [102963,117997] 0 2024-07-26T11:05:03.383 INFO:tasks.workunit.client.0.smithi144.stdout:1/981: rmdir d3/d6/d75/d88/da9/dac/d104/ddb/df6/dfe/d5d/d9a/de1 39 2024-07-26T11:05:03.383 INFO:tasks.workunit.client.0.smithi144.stdout:1/982: write d3/d6/da/d21/fda [950970,49317] 0 2024-07-26T11:05:03.383 INFO:tasks.workunit.client.0.smithi144.stdout:5/894: dread d1/d9/d5b/f8c [0,4194304] 0 2024-07-26T11:05:03.386 INFO:tasks.workunit.client.0.smithi144.stdout:2/861: creat d0/d6/d9/d44/f106 x:0 0 0 2024-07-26T11:05:03.386 INFO:tasks.workunit.client.0.smithi144.stdout:2/862: fdatasync d0/d24/d2b/d68/d82/fc4 0 2024-07-26T11:05:03.387 INFO:tasks.workunit.client.0.smithi144.stdout:9/917: dread d9/d3b/dce/fe8 [0,4194304] 0 2024-07-26T11:05:03.387 INFO:tasks.workunit.client.0.smithi144.stdout:8/841: creat d1a/d32/d49/f11b x:0 0 0 2024-07-26T11:05:03.387 INFO:tasks.workunit.client.0.smithi144.stdout:2/863: mkdir d0/d6/d34/d6c/d107 0 2024-07-26T11:05:03.387 INFO:tasks.workunit.client.0.smithi144.stdout:2/864: write d0/d24/d7b/ddc/d28/d2e/d77/f40 [1020240,32362] 0 2024-07-26T11:05:03.388 INFO:tasks.workunit.client.0.smithi144.stdout:8/842: creat d1a/d32/d49/d73/d2c/d38/d58/da3/db1/f11c x:0 0 0 2024-07-26T11:05:03.389 INFO:tasks.workunit.client.0.smithi144.stdout:9/918: getdents d9/de/d14/d23/d39/d7f 0 2024-07-26T11:05:03.390 INFO:tasks.workunit.client.0.smithi144.stdout:8/843: truncate d1a/f21 1211743 0 2024-07-26T11:05:03.395 INFO:tasks.workunit.client.0.smithi144.stdout:1/983: unlink d3/d3d/d58/d101/fba 0 2024-07-26T11:05:03.395 INFO:tasks.workunit.client.0.smithi144.stdout:1/984: dread - d3/d6/d9c/d9f/ff2 zero size 2024-07-26T11:05:03.395 INFO:tasks.workunit.client.0.smithi144.stdout:7/899: dwrite d0/dd0/dbf/fb2 [4194304,4194304] 0 2024-07-26T11:05:03.399 INFO:tasks.workunit.client.0.smithi144.stdout:9/919: rename d9/de/d4e/d6a/ff9 to d9/de/d14/d79/d124/f143 0 2024-07-26T11:05:03.399 INFO:tasks.workunit.client.0.smithi144.stdout:9/920: stat d9/dfc/d114/dd7 0 2024-07-26T11:05:03.409 INFO:tasks.workunit.client.0.smithi144.stdout:9/921: creat d9/de/d4e/d6a/deb/dfd/f144 x:0 0 0 2024-07-26T11:05:03.410 INFO:tasks.workunit.client.0.smithi144.stdout:9/922: write d9/dfc/d114/dcc/dea/f12e [291427,121407] 0 2024-07-26T11:05:03.413 INFO:tasks.workunit.client.0.smithi144.stdout:8/844: dread d1a/d32/d49/d73/d2c/f114 [0,4194304] 0 2024-07-26T11:05:03.416 INFO:tasks.workunit.client.0.smithi144.stdout:8/845: symlink d1a/l11d 0 2024-07-26T11:05:03.416 INFO:tasks.workunit.client.0.smithi144.stdout:9/923: unlink d9/de/d14/f35 0 2024-07-26T11:05:03.420 INFO:tasks.workunit.client.0.smithi144.stdout:8/846: getdents d1a/d32/d49/d73/d2c/d38/d58 0 2024-07-26T11:05:03.422 INFO:tasks.workunit.client.0.smithi144.stdout:9/924: link d9/cf6 d9/de/d14/d23/d39/d7f/d123/c145 0 2024-07-26T11:05:03.422 INFO:tasks.workunit.client.0.smithi144.stdout:9/925: fsync d9/d3b/f64 0 2024-07-26T11:05:03.425 INFO:tasks.workunit.client.0.smithi144.stdout:9/926: rename d9/de/d14/d23/fa3 to d9/de/d4e/dff/f146 0 2024-07-26T11:05:03.429 INFO:tasks.workunit.client.0.smithi144.stdout:9/927: creat d9/de/d4e/d106/f147 x:0 0 0 2024-07-26T11:05:03.432 INFO:tasks.workunit.client.0.smithi144.stdout:1/985: dread d3/f9b [0,4194304] 0 2024-07-26T11:05:03.432 INFO:tasks.workunit.client.0.smithi144.stdout:0/906: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/d84/f11a [0,4194304] 0 2024-07-26T11:05:03.434 INFO:tasks.workunit.client.0.smithi144.stdout:0/907: symlink d0/l128 0 2024-07-26T11:05:03.434 INFO:tasks.workunit.client.0.smithi144.stdout:0/908: chown d0/d3e/d45/d4c/f64 306 1 2024-07-26T11:05:03.434 INFO:tasks.workunit.client.0.smithi144.stdout:1/986: mkdir d3/d6/dc0/d13f 0 2024-07-26T11:05:03.437 INFO:tasks.workunit.client.0.smithi144.stdout:0/909: rename d0/d3e/d45/dd4/d11f/l47 to d0/d3e/d45/dd4/d11f/d5e/l129 0 2024-07-26T11:05:03.451 INFO:tasks.workunit.client.0.smithi144.stdout:8/847: dread d1a/d32/d49/d73/d7b/d43/d4c/f77 [0,4194304] 0 2024-07-26T11:05:03.451 INFO:tasks.workunit.client.0.smithi144.stdout:9/928: write d9/dfc/d114/dcc/ff3 [2465338,24533] 0 2024-07-26T11:05:03.453 INFO:tasks.workunit.client.0.smithi144.stdout:3/691: dwrite d1/d5/d97/d60/f8c [4194304,4194304] 0 2024-07-26T11:05:03.455 INFO:tasks.workunit.client.0.smithi144.stdout:9/929: rename d9/de/d14/lb9 to d9/dfc/d114/d116/l148 0 2024-07-26T11:05:03.457 INFO:tasks.workunit.client.0.smithi144.stdout:7/900: dwrite d0/d6/d9/d12/d115/deb/f117 [0,4194304] 0 2024-07-26T11:05:03.462 INFO:tasks.workunit.client.0.smithi144.stdout:2/865: dwrite d0/d24/d2b/fbb [0,4194304] 0 2024-07-26T11:05:03.462 INFO:tasks.workunit.client.0.smithi144.stdout:8/848: symlink d1a/d32/d49/d73/d2c/d38/d58/l11e 0 2024-07-26T11:05:03.464 INFO:tasks.workunit.client.0.smithi144.stdout:3/692: rmdir d1/d28/d54/d26/d37/da1 39 2024-07-26T11:05:03.467 INFO:tasks.workunit.client.0.smithi144.stdout:2/866: creat d0/d6/d9/f108 x:0 0 0 2024-07-26T11:05:03.474 INFO:tasks.workunit.client.0.smithi144.stdout:3/693: creat d1/d28/d54/d26/d64/fd6 x:0 0 0 2024-07-26T11:05:03.474 INFO:tasks.workunit.client.0.smithi144.stdout:8/849: creat d1a/d32/d8c/f11f x:0 0 0 2024-07-26T11:05:03.476 INFO:tasks.workunit.client.0.smithi144.stdout:8/850: truncate d1a/d1c/f22 167001 0 2024-07-26T11:05:03.482 INFO:tasks.workunit.client.0.smithi144.stdout:3/694: mknod d1/d28/d54/d3d/d42/dc4/cd7 0 2024-07-26T11:05:03.482 INFO:tasks.workunit.client.0.smithi144.stdout:2/867: dread d0/d24/d7b/ddc/fa0 [0,4194304] 0 2024-07-26T11:05:03.483 INFO:tasks.workunit.client.0.smithi144.stdout:7/901: dread d0/d6/d9/d12/f60 [0,4194304] 0 2024-07-26T11:05:03.485 INFO:tasks.workunit.client.0.smithi144.stdout:2/868: rename d0/d6/d34/d6c/f86 to d0/d24/d7b/ddc/d28/d27/d8b/f109 0 2024-07-26T11:05:03.485 INFO:tasks.workunit.client.0.smithi144.stdout:2/869: stat d0/d24/cd4 0 2024-07-26T11:05:03.486 INFO:tasks.workunit.client.0.smithi144.stdout:2/870: symlink d0/d6/l10a 0 2024-07-26T11:05:03.489 INFO:tasks.workunit.client.0.smithi144.stdout:2/871: rename d0/c21 to d0/d24/d7b/ddc/d28/c10b 0 2024-07-26T11:05:03.489 INFO:tasks.workunit.client.0.smithi144.stdout:0/910: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/fc8 [0,4194304] 0 2024-07-26T11:05:03.489 INFO:tasks.workunit.client.0.smithi144.stdout:2/872: rmdir d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2 39 2024-07-26T11:05:03.491 INFO:tasks.workunit.client.0.smithi144.stdout:0/911: creat d0/d3e/d45/dd4/d11f/d5e/d1e/dd2/f12a x:0 0 0 2024-07-26T11:05:03.500 INFO:tasks.workunit.client.0.smithi144.stdout:0/912: rmdir d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/d6f/dec 39 2024-07-26T11:05:03.510 INFO:tasks.workunit.client.0.smithi144.stdout:8/851: write d1a/f1b [3562025,127523] 0 2024-07-26T11:05:03.510 INFO:tasks.workunit.client.0.smithi144.stdout:8/852: dread - d1a/d32/d49/d73/d2c/d38/fe0 zero size 2024-07-26T11:05:03.511 INFO:tasks.workunit.client.0.smithi144.stdout:9/930: dwrite d9/de/d14/d23/d39/f12a [4194304,4194304] 0 2024-07-26T11:05:03.512 INFO:tasks.workunit.client.0.smithi144.stdout:3/695: write d1/d5/f24 [2869322,96877] 0 2024-07-26T11:05:03.513 INFO:tasks.workunit.client.0.smithi144.stdout:9/931: mknod d9/de/d14/d23/c149 0 2024-07-26T11:05:03.513 INFO:tasks.workunit.client.0.smithi144.stdout:0/913: mknod d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/c12b 0 2024-07-26T11:05:03.514 INFO:tasks.workunit.client.0.smithi144.stdout:8/853: getdents d1a/d32/d49/d73/d2c/d9d/d74/dac 0 2024-07-26T11:05:03.517 INFO:tasks.workunit.client.0.smithi144.stdout:3/696: read d1/d5/d97/d48/fad [386704,16225] 0 2024-07-26T11:05:03.517 INFO:tasks.workunit.client.0.smithi144.stdout:3/697: fsync d1/f4f 0 2024-07-26T11:05:03.519 INFO:tasks.workunit.client.0.smithi144.stdout:3/698: creat d1/d28/d54/d26/d64/dd5/fd8 x:0 0 0 2024-07-26T11:05:03.520 INFO:tasks.workunit.client.0.smithi144.stdout:3/699: chown d1/d5/d97/d48/f56 3606 1 2024-07-26T11:05:03.520 INFO:tasks.workunit.client.0.smithi144.stdout:3/700: fdatasync d1/f50 0 2024-07-26T11:05:03.541 INFO:tasks.workunit.client.0.smithi144.stdout:7/902: dwrite d0/dd0/dbf/d2f/d68/de2/ff6 [0,4194304] 0 2024-07-26T11:05:03.544 INFO:tasks.workunit.client.0.smithi144.stdout:7/903: unlink d0/dd0/dbf/d2f/d53/db6/d106/f12c 0 2024-07-26T11:05:03.546 INFO:tasks.workunit.client.0.smithi144.stdout:7/904: mkdir d0/dd0/dbf/d13c 0 2024-07-26T11:05:03.547 INFO:tasks.workunit.client.0.smithi144.stdout:7/905: mknod d0/dd0/dbf/d2f/d47/c13d 0 2024-07-26T11:05:03.549 INFO:tasks.workunit.client.0.smithi144.stdout:8/854: dwrite d1a/d32/d49/f11b [0,4194304] 0 2024-07-26T11:05:03.551 INFO:tasks.workunit.client.0.smithi144.stdout:9/932: dwrite d9/f29 [0,4194304] 0 2024-07-26T11:05:03.554 INFO:tasks.workunit.client.0.smithi144.stdout:9/933: link d9/de/d14/d3f/cda d9/de/d14/d23/d39/d7f/c14a 0 2024-07-26T11:05:03.554 INFO:tasks.workunit.client.0.smithi144.stdout:9/934: chown d9/fc 104685797 1 2024-07-26T11:05:03.554 INFO:tasks.workunit.client.0.smithi144.stdout:9/935: write d9/de/d14/d3f/d5e/fc0 [1668906,43717] 0 2024-07-26T11:05:03.555 INFO:tasks.workunit.client.0.smithi144.stdout:7/906: write d0/d6/d9/d3f/f6e [41600,45545] 0 2024-07-26T11:05:03.555 INFO:tasks.workunit.client.0.smithi144.stdout:0/914: write d0/d3e/d45/d9f/f6e [3352580,96270] 0 2024-07-26T11:05:03.556 INFO:tasks.workunit.client.0.smithi144.stdout:9/936: creat d9/dfc/d114/dcc/dea/f14b x:0 0 0 2024-07-26T11:05:03.558 INFO:tasks.workunit.client.0.smithi144.stdout:0/915: link d0/d3e/d5f/fb1 d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/dac/f12c 0 2024-07-26T11:05:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:03 smithi160 ceph-mon[36715]: pgmap v538: 129 pgs: 129 active+clean; 1.7 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 80 MiB/s wr, 378 op/s 2024-07-26T11:05:03.560 INFO:tasks.workunit.client.0.smithi144.stdout:0/916: rename d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/d101 to d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/d12d 0 2024-07-26T11:05:03.561 INFO:tasks.workunit.client.0.smithi144.stdout:0/917: link d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/fc8 d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/df3/f12e 0 2024-07-26T11:05:03.562 INFO:tasks.workunit.client.0.smithi144.stdout:0/918: mkdir d0/d3e/db4/d11e/d12f 0 2024-07-26T11:05:03.562 INFO:tasks.workunit.client.0.smithi144.stdout:0/919: chown d0/f32 8800 1 2024-07-26T11:05:03.576 INFO:tasks.workunit.client.0.smithi144.stdout:3/701: dwrite d1/d28/d54/d26/d37/f58 [0,4194304] 0 2024-07-26T11:05:03.601 INFO:tasks.workunit.client.0.smithi144.stdout:9/937: dread d9/de/d14/d23/d39/f5b [0,4194304] 0 2024-07-26T11:05:03.607 INFO:tasks.workunit.client.0.smithi144.stdout:7/907: write d0/d6/d9/d12/f59 [1164438,92306] 0 2024-07-26T11:05:03.607 INFO:tasks.workunit.client.0.smithi144.stdout:7/908: fdatasync d0/d6/d10c/f26 0 2024-07-26T11:05:03.613 INFO:tasks.workunit.client.0.smithi144.stdout:0/920: dread d0/d3e/d45/dd4/d11f/d5e/d1e/d30/f3b [0,4194304] 0 2024-07-26T11:05:03.613 INFO:tasks.workunit.client.0.smithi144.stdout:8/855: dwrite d1a/d32/d49/d73/d7b/d43/d4c/f54 [4194304,4194304] 0 2024-07-26T11:05:03.613 INFO:tasks.workunit.client.0.smithi144.stdout:0/921: read d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/dac/fbc [2377465,90444] 0 2024-07-26T11:05:03.613 INFO:tasks.workunit.client.0.smithi144.stdout:8/856: chown d1a/d32/d49/d73/d2c/d38/d58/da3/db1 45579258 1 2024-07-26T11:05:03.613 INFO:tasks.workunit.client.0.smithi144.stdout:8/857: write d1a/fc8 [294996,71242] 0 2024-07-26T11:05:03.614 INFO:tasks.workunit.client.0.smithi144.stdout:0/922: mknod d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/d107/c130 0 2024-07-26T11:05:03.616 INFO:tasks.workunit.client.0.smithi144.stdout:0/923: mknod d0/d3e/d45/dd4/d11f/d5e/d1e/d91/dd1/c131 0 2024-07-26T11:05:03.618 INFO:tasks.workunit.client.0.smithi144.stdout:0/924: mknod d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/c132 0 2024-07-26T11:05:03.626 INFO:tasks.workunit.client.0.smithi144.stdout:0/925: creat d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/f133 x:0 0 0 2024-07-26T11:05:03.627 INFO:tasks.workunit.client.0.smithi144.stdout:0/926: chown d0/d3e/d45/dd4/d11f/fa5 426 1 2024-07-26T11:05:03.627 INFO:tasks.workunit.client.0.smithi144.stdout:0/927: chown d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/cce 93603036 1 2024-07-26T11:05:03.627 INFO:tasks.workunit.client.0.smithi144.stdout:0/928: chown d0/d3e/d45/dd4/d11f/d5e/d1e/d6c/cc9 55 1 2024-07-26T11:05:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:03 smithi183 ceph-mon[37273]: pgmap v538: 129 pgs: 129 active+clean; 1.7 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 80 MiB/s wr, 378 op/s 2024-07-26T11:05:03.650 INFO:tasks.workunit.client.0.smithi144.stdout:9/938: dwrite d9/de/d14/d3f/d5e/f6c [0,4194304] 0 2024-07-26T11:05:03.660 INFO:tasks.workunit.client.0.smithi144.stdout:0/929: dread d0/f10 [0,4194304] 0 2024-07-26T11:05:03.660 INFO:tasks.workunit.client.0.smithi144.stdout:8/858: dwrite d1a/d32/d8c/f11f [0,4194304] 0 2024-07-26T11:05:03.660 INFO:tasks.workunit.client.0.smithi144.stdout:7/909: dwrite d0/d6/d55/fff [0,4194304] 0 2024-07-26T11:05:03.660 INFO:tasks.workunit.client.0.smithi144.stdout:7/910: chown d0/d6/d9/d12/d115/cfa 55 1 2024-07-26T11:05:03.667 INFO:tasks.workunit.client.0.smithi144.stdout:0/930: creat d0/d3e/d45/dd4/d11f/d5e/d1e/d91/dd1/f134 x:0 0 0 2024-07-26T11:05:03.669 INFO:tasks.workunit.client.0.smithi144.stdout:0/931: creat d0/d3e/d45/dd4/d11f/d5e/d1e/dc2/d10e/f135 x:0 0 0 2024-07-26T11:05:03.671 INFO:tasks.workunit.client.0.smithi144.stdout:0/932: creat d0/d3e/d5f/f136 x:0 0 0 2024-07-26T11:05:03.672 INFO:tasks.workunit.client.0.smithi144.stdout:0/933: creat d0/d3e/d45/d4c/f137 x:0 0 0 2024-07-26T11:05:03.673 INFO:tasks.workunit.client.0.smithi144.stdout:0/934: symlink d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/dac/l138 0 2024-07-26T11:05:03.681 INFO:tasks.workunit.client.0.smithi144.stdout:8/859: write d1a/d32/d49/fb7 [4055093,75078] 0 2024-07-26T11:05:03.688 INFO:tasks.workunit.client.0.smithi144.stdout:7/911: dread d0/d6/d10c/f90 [0,4194304] 0 2024-07-26T11:05:03.690 INFO:tasks.workunit.client.0.smithi144.stdout:7/912: creat d0/d6/d9/f13e x:0 0 0 2024-07-26T11:05:03.701 INFO:tasks.workunit.client.0.smithi144.stdout:9/939: dwrite d9/d3b/f7c [4194304,4194304] 0 2024-07-26T11:05:03.703 INFO:tasks.workunit.client.0.smithi144.stdout:9/940: mknod d9/d3b/d81/c14c 0 2024-07-26T11:05:03.709 INFO:tasks.workunit.client.0.smithi144.stdout:8/860: write d1a/d1c/fab [2667883,65657] 0 2024-07-26T11:05:03.709 INFO:tasks.workunit.client.0.smithi144.stdout:8/861: write d1a/d32/d49/d73/d2c/d38/d58/f10b [619436,105484] 0 2024-07-26T11:05:03.711 INFO:tasks.workunit.client.0.smithi144.stdout:8/862: write d1a/d32/d66/da0/fea [244014,118110] 0 2024-07-26T11:05:03.711 INFO:tasks.workunit.client.0.smithi144.stdout:8/863: chown d1a/d32/d49/d73/d7b/d43/d4f/la1 11415 1 2024-07-26T11:05:03.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:03 smithi144 ceph-mon[32129]: pgmap v538: 129 pgs: 129 active+clean; 1.7 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 80 MiB/s wr, 378 op/s 2024-07-26T11:05:03.731 INFO:tasks.workunit.client.0.smithi144.stdout:0/935: dwrite d0/d3e/d45/d4c/fca [0,4194304] 0 2024-07-26T11:05:03.732 INFO:tasks.workunit.client.0.smithi144.stdout:0/936: creat d0/f139 x:0 0 0 2024-07-26T11:05:03.732 INFO:tasks.workunit.client.0.smithi144.stdout:0/937: write d0/d3e/d45/dd4/d11f/d5e/d1e/d91/dd1/fbb [4517767,119024] 0 2024-07-26T11:05:03.737 INFO:tasks.workunit.client.0.smithi144.stdout:7/913: dwrite d0/d6/d9/f12f [0,4194304] 0 2024-07-26T11:05:03.741 INFO:tasks.workunit.client.0.smithi144.stdout:7/914: creat d0/d6/d9/d12/d115/df2/f13f x:0 0 0 2024-07-26T11:05:03.742 INFO:tasks.workunit.client.0.smithi144.stdout:7/915: mknod d0/d6/d9/d12/c140 0 2024-07-26T11:05:03.743 INFO:tasks.workunit.client.0.smithi144.stdout:7/916: creat d0/d71/dc0/f141 x:0 0 0 2024-07-26T11:05:03.744 INFO:tasks.workunit.client.0.smithi144.stdout:7/917: creat d0/f142 x:0 0 0 2024-07-26T11:05:03.744 INFO:tasks.workunit.client.0.smithi144.stdout:7/918: chown d0/dd0/dbf/d67/de0/d100/f128 179807451 1 2024-07-26T11:05:03.759 INFO:tasks.workunit.client.0.smithi144.stdout:8/864: dwrite d1a/d32/d49/d73/d2c/d38/f112 [0,4194304] 0 2024-07-26T11:05:03.759 INFO:tasks.workunit.client.0.smithi144.stdout:0/938: dread d0/d3e/d45/dd4/d11f/d5e/d1e/d30/fb9 [0,4194304] 0 2024-07-26T11:05:03.761 INFO:tasks.workunit.client.0.smithi144.stdout:9/941: dwrite d9/de/faa [0,4194304] 0 2024-07-26T11:05:03.765 INFO:tasks.workunit.client.0.smithi144.stdout:8/865: chown d1a/d32/d6c/d82/cd5 23876572 1 2024-07-26T11:05:03.766 INFO:tasks.workunit.client.0.smithi144.stdout:0/939: truncate d0/d3e/d45/dd4/d11f/f127 3226400 0 2024-07-26T11:05:03.766 INFO:tasks.workunit.client.0.smithi144.stdout:0/940: chown d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/dac/fe5 17767 1 2024-07-26T11:05:03.767 INFO:tasks.workunit.client.0.smithi144.stdout:8/866: creat d1a/d32/d49/d73/f120 x:0 0 0 2024-07-26T11:05:03.768 INFO:tasks.workunit.client.0.smithi144.stdout:7/919: dread d0/d6/d9/d12/f60 [0,4194304] 0 2024-07-26T11:05:03.768 INFO:tasks.workunit.client.0.smithi144.stdout:7/920: write d0/d6/d9/d12/fcc [437182,80758] 0 2024-07-26T11:05:03.778 INFO:tasks.workunit.client.0.smithi144.stdout:0/941: truncate d0/d3e/d45/dd4/d11f/d5e/d1e/d6c/dc6/fdc 117137 0 2024-07-26T11:05:03.783 INFO:tasks.workunit.client.0.smithi144.stdout:7/921: creat d0/dd0/dbf/d2f/d53/db6/f143 x:0 0 0 2024-07-26T11:05:03.783 INFO:tasks.workunit.client.0.smithi144.stdout:7/922: write d0/dd0/dbf/d2f/d47/fb7 [60581,98869] 0 2024-07-26T11:05:03.784 INFO:tasks.workunit.client.0.smithi144.stdout:7/923: mkdir d0/dd0/dbf/d2f/d144 0 2024-07-26T11:05:03.784 INFO:tasks.workunit.client.0.smithi144.stdout:7/924: chown d0/d71/dc0/de4/d125/faf 1616071 1 2024-07-26T11:05:03.794 INFO:tasks.workunit.client.0.smithi144.stdout:7/925: dread d0/dd0/dbf/d2f/d53/db6/fe6 [0,4194304] 0 2024-07-26T11:05:03.798 INFO:tasks.workunit.client.0.smithi144.stdout:7/926: creat d0/dd0/dbf/d2f/d68/de2/f145 x:0 0 0 2024-07-26T11:05:03.800 INFO:tasks.workunit.client.0.smithi144.stdout:7/927: link d0/d6/d10c/f2a d0/dd0/dbf/d2f/f146 0 2024-07-26T11:05:03.802 INFO:tasks.workunit.client.0.smithi144.stdout:7/928: mknod d0/dd0/c147 0 2024-07-26T11:05:03.802 INFO:tasks.workunit.client.0.smithi144.stdout:7/929: read d0/d6/d9/d3f/f6e [317381,9052] 0 2024-07-26T11:05:03.808 INFO:tasks.workunit.client.0.smithi144.stdout:8/867: dwrite d1a/d32/d66/ff0 [0,4194304] 0 2024-07-26T11:05:03.808 INFO:tasks.workunit.client.0.smithi144.stdout:8/868: fsync d1a/d32/d49/d73/d7b/f40 0 2024-07-26T11:05:03.808 INFO:tasks.workunit.client.0.smithi144.stdout:8/869: rename d1a/d32/d49/d73/d2c/d38/d58/da3 to d1a/d32/d49/d73/d2c/d38/d58/da3/db1/d121 22 2024-07-26T11:05:03.811 INFO:tasks.workunit.client.0.smithi144.stdout:8/870: creat d1a/d1c/f122 x:0 0 0 2024-07-26T11:05:03.811 INFO:tasks.workunit.client.0.smithi144.stdout:9/942: dwrite d9/d3b/dce/f109 [0,4194304] 0 2024-07-26T11:05:03.811 INFO:tasks.workunit.client.0.smithi144.stdout:8/871: write d1a/d32/d49/d73/d2c/d38/fe0 [543400,51729] 0 2024-07-26T11:05:03.851 INFO:tasks.workunit.client.0.smithi144.stdout:8/872: write d1a/f95 [405796,47649] 0 2024-07-26T11:05:03.851 INFO:tasks.workunit.client.0.smithi144.stdout:0/942: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/dc2/f9e [0,4194304] 0 2024-07-26T11:05:03.855 INFO:tasks.workunit.client.0.smithi144.stdout:8/873: symlink d1a/d32/d49/d73/d7b/d4e/d9a/dc1/l123 0 2024-07-26T11:05:03.858 INFO:tasks.workunit.client.0.smithi144.stdout:8/874: mkdir d1a/d32/d49/d73/d2c/d38/d58/da3/d124 0 2024-07-26T11:05:03.858 INFO:tasks.workunit.client.0.smithi144.stdout:8/875: chown d1a/d32/d49/d73/d2c/d38/f5f 110105 1 2024-07-26T11:05:03.860 INFO:tasks.workunit.client.0.smithi144.stdout:0/943: read d0/d3e/d45/dd4/d11f/d5e/f68 [467832,125900] 0 2024-07-26T11:05:03.860 INFO:tasks.workunit.client.0.smithi144.stdout:0/944: chown d0/d3e/d5f/f136 82 1 2024-07-26T11:05:03.866 INFO:tasks.workunit.client.0.smithi144.stdout:0/945: rename d0/d3e/d45/dd4/d11f/d5e/c1f to d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d4b/c13a 0 2024-07-26T11:05:03.867 INFO:tasks.workunit.client.0.smithi144.stdout:0/946: truncate d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d67/fc5 2610601 0 2024-07-26T11:05:03.867 INFO:tasks.workunit.client.0.smithi144.stdout:0/947: fdatasync d0/f139 0 2024-07-26T11:05:03.869 INFO:tasks.workunit.client.0.smithi144.stdout:9/943: dwrite d9/d3b/f64 [0,4194304] 0 2024-07-26T11:05:03.869 INFO:tasks.workunit.client.0.smithi144.stdout:7/930: dwrite d0/dd0/dbf/d67/f9f [0,4194304] 0 2024-07-26T11:05:03.870 INFO:tasks.workunit.client.0.smithi144.stdout:9/944: creat d9/f14d x:0 0 0 2024-07-26T11:05:03.875 INFO:tasks.workunit.client.0.smithi144.stdout:7/931: rename d0/d71/dc0/de4/d125/la9 to d0/d71/d95/de7/d136/l148 0 2024-07-26T11:05:03.875 INFO:tasks.workunit.client.0.smithi144.stdout:7/932: truncate d0/d6/d10c/d57/f77 8856963 0 2024-07-26T11:05:03.875 INFO:tasks.workunit.client.0.smithi144.stdout:7/933: dread - d0/d112/f129 zero size 2024-07-26T11:05:03.875 INFO:tasks.workunit.client.0.smithi144.stdout:7/934: truncate d0/dd0/dbf/d2f/d47/d5e/fe1 2436525 0 2024-07-26T11:05:03.875 INFO:tasks.workunit.client.0.smithi144.stdout:7/935: read - d0/d6/d9/f10d zero size 2024-07-26T11:05:03.881 INFO:tasks.workunit.client.0.smithi144.stdout:7/936: rename d0/d71/d95/de7/d136/l148 to d0/d6/d9/d12/d115/deb/l149 0 2024-07-26T11:05:03.882 INFO:tasks.workunit.client.0.smithi144.stdout:7/937: unlink d0/d6/d9/d12/f59 0 2024-07-26T11:05:03.893 INFO:tasks.workunit.client.0.smithi144.stdout:0/948: dread d0/d3e/d45/dd4/d11f/fa5 [0,4194304] 0 2024-07-26T11:05:03.893 INFO:tasks.workunit.client.0.smithi144.stdout:0/949: write d0/d3e/d45/dd4/d11f/d5e/d1e/d91/dd1/fbb [2678393,120944] 0 2024-07-26T11:05:03.901 INFO:tasks.workunit.client.0.smithi144.stdout:8/876: dwrite f6 [0,4194304] 0 2024-07-26T11:05:03.901 INFO:tasks.workunit.client.0.smithi144.stdout:8/877: fsync d1a/d32/d49/d73/d7b/d43/f9f 0 2024-07-26T11:05:03.902 INFO:tasks.workunit.client.0.smithi144.stdout:8/878: mknod d1a/d32/d49/d73/d7b/d43/d4f/c125 0 2024-07-26T11:05:03.904 INFO:tasks.workunit.client.0.smithi144.stdout:8/879: truncate d1a/d1c/fab 1085037 0 2024-07-26T11:05:03.932 INFO:tasks.workunit.client.0.smithi144.stdout:7/938: dwrite d0/d112/f129 [0,4194304] 0 2024-07-26T11:05:03.936 INFO:tasks.workunit.client.0.smithi144.stdout:7/939: symlink d0/d71/d95/l14a 0 2024-07-26T11:05:03.938 INFO:tasks.workunit.client.0.smithi144.stdout:7/940: unlink d0/lc7 0 2024-07-26T11:05:03.946 INFO:tasks.workunit.client.0.smithi144.stdout:0/950: write d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/df3/f12e [1650705,95507] 0 2024-07-26T11:05:03.946 INFO:tasks.workunit.client.0.smithi144.stdout:0/951: dread - d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/ff1 zero size 2024-07-26T11:05:03.947 INFO:tasks.workunit.client.0.smithi144.stdout:0/952: rename d0/d3e/d45/dd4/d11f/d5e/d1e/d6c/dc6/fdc to d0/d3e/db8/f13b 0 2024-07-26T11:05:03.947 INFO:tasks.workunit.client.0.smithi144.stdout:0/953: fdatasync d0/d3e/d45/dd4/d11f/fa0 0 2024-07-26T11:05:03.948 INFO:tasks.workunit.client.0.smithi144.stdout:0/954: truncate d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/f99 1312095 0 2024-07-26T11:05:03.960 INFO:tasks.workunit.client.0.smithi144.stdout:7/941: write d0/dd0/dbf/d67/f9f [1490450,80567] 0 2024-07-26T11:05:03.960 INFO:tasks.workunit.client.0.smithi144.stdout:7/942: write d0/d6/d10c/f1e [801974,7235] 0 2024-07-26T11:05:03.965 INFO:tasks.workunit.client.0.smithi144.stdout:8/880: dwrite d1a/f20 [0,4194304] 0 2024-07-26T11:05:03.967 INFO:tasks.workunit.client.0.smithi144.stdout:8/881: creat d1a/d32/d49/d73/d7b/d4e/d10c/f126 x:0 0 0 2024-07-26T11:05:03.967 INFO:tasks.workunit.client.0.smithi144.stdout:8/882: read d1a/d32/d8c/d9b/fdf [349399,43546] 0 2024-07-26T11:05:03.967 INFO:tasks.workunit.client.0.smithi144.stdout:8/883: dread - d1a/d32/d49/ffe zero size 2024-07-26T11:05:03.972 INFO:tasks.workunit.client.0.smithi144.stdout:0/955: dread d0/d3e/d45/f4a [0,4194304] 0 2024-07-26T11:05:03.973 INFO:tasks.workunit.client.0.smithi144.stdout:0/956: mknod d0/d3e/d45/dd4/d11f/c13c 0 2024-07-26T11:05:03.974 INFO:tasks.workunit.client.0.smithi144.stdout:0/957: symlink d0/d3e/d45/d4c/l13d 0 2024-07-26T11:05:03.975 INFO:tasks.workunit.client.0.smithi144.stdout:0/958: rmdir d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/d12d 39 2024-07-26T11:05:03.976 INFO:tasks.workunit.client.0.smithi144.stdout:0/959: fdatasync d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/d104/f99 0 2024-07-26T11:05:03.976 INFO:tasks.workunit.client.0.smithi144.stdout:0/960: write d0/d3e/d45/dd4/d11f/d5e/d1e/d91/f96 [2562594,109723] 0 2024-07-26T11:05:03.988 INFO:tasks.workunit.client.0.smithi144.stdout:8/884: write d1a/d32/d49/d73/f79 [1819946,17399] 0 2024-07-26T11:05:03.991 INFO:tasks.workunit.client.0.smithi144.stdout:8/885: creat d1a/d32/d50/dc4/df6/f127 x:0 0 0 2024-07-26T11:05:03.993 INFO:tasks.workunit.client.0.smithi144.stdout:7/943: dwrite d0/dd0/dbf/d2f/d47/fb7 [0,4194304] 0 2024-07-26T11:05:03.993 INFO:tasks.workunit.client.0.smithi144.stdout:8/886: rename d1a/d32/d49/d73/d2c to d1a/d32/d49/d73/d7b/d43/d4c/d128 0 2024-07-26T11:05:03.993 INFO:tasks.workunit.client.0.smithi144.stdout:8/887: fsync d1a/f21 0 2024-07-26T11:05:03.993 INFO:tasks.workunit.client.0.smithi144.stdout:8/888: dread - d1a/d32/d49/d73/d7b/d4e/d10c/f126 zero size 2024-07-26T11:05:03.995 INFO:tasks.workunit.client.0.smithi144.stdout:7/944: getdents d0/dd0/dbf/d67 0 2024-07-26T11:05:03.995 INFO:tasks.workunit.client.0.smithi144.stdout:7/945: write d0/d6/d9/f13 [4550915,31275] 0 2024-07-26T11:05:03.997 INFO:tasks.workunit.client.0.smithi144.stdout:7/946: rename d0/d6/d10c/f90 to d0/d6/d9/d3f/f14b 0 2024-07-26T11:05:03.998 INFO:tasks.workunit.client.0.smithi144.stdout:7/947: truncate d0/dd0/dbf/d2f/d53/fc5 2888526 0 2024-07-26T11:05:04.000 INFO:tasks.workunit.client.0.smithi144.stdout:7/948: rename d0/dd0/dbf/d67/de0/c12e to d0/d71/dc0/c14c 0 2024-07-26T11:05:04.012 INFO:tasks.workunit.client.0.smithi144.stdout:8/889: write d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/f112 [262059,7132] 0 2024-07-26T11:05:04.016 INFO:tasks.workunit.client.0.smithi144.stdout:8/890: truncate d1a/d32/d49/d73/d7b/d4e/d78/f90 591060 0 2024-07-26T11:05:04.017 INFO:tasks.workunit.client.0.smithi144.stdout:8/891: chown d1a/d32/d49/d73/d7b/c60 10650 1 2024-07-26T11:05:04.019 INFO:tasks.workunit.client.0.smithi144.stdout:8/892: rename d1a/d32/d8c/ce2 to d1a/d32/d49/d73/d7b/d43/d4c/d128/d9d/c129 0 2024-07-26T11:05:04.021 INFO:tasks.workunit.client.0.smithi144.stdout:7/949: write d0/dd0/dbf/d2f/f8a [2477551,6817] 0 2024-07-26T11:05:04.021 INFO:tasks.workunit.client.0.smithi144.stdout:7/950: dread - d0/d6/d9/f132 zero size 2024-07-26T11:05:04.021 INFO:tasks.workunit.client.0.smithi144.stdout:8/893: symlink d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/l12a 0 2024-07-26T11:05:04.021 INFO:tasks.workunit.client.0.smithi144.stdout:0/961: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/dc2/d10e/f135 [0,4194304] 0 2024-07-26T11:05:04.026 INFO:tasks.workunit.client.0.smithi144.stdout:8/894: creat d1a/d32/d49/d73/d7b/d43/d4c/d128/d9d/f12b x:0 0 0 2024-07-26T11:05:04.026 INFO:tasks.workunit.client.0.smithi144.stdout:8/895: chown d1a/d32/d49/d73/d7b/f40 227547 1 2024-07-26T11:05:04.026 INFO:tasks.workunit.client.0.smithi144.stdout:8/896: stat d1a/d32/d49/d73/f80 0 2024-07-26T11:05:04.028 INFO:tasks.workunit.client.0.smithi144.stdout:0/962: creat d0/d3e/f13e x:0 0 0 2024-07-26T11:05:04.032 INFO:tasks.workunit.client.0.smithi144.stdout:8/897: truncate d1a/d32/d49/d73/f79 4122826 0 2024-07-26T11:05:04.032 INFO:tasks.workunit.client.0.smithi144.stdout:8/898: stat d1a/d1c/fab 0 2024-07-26T11:05:04.033 INFO:tasks.workunit.client.0.smithi144.stdout:8/899: chown d1a/d32/d49/d73/f2e 4194 1 2024-07-26T11:05:04.043 INFO:tasks.workunit.client.0.smithi144.stdout:8/900: mkdir d1a/d32/d50/dc4/d12c 0 2024-07-26T11:05:04.046 INFO:tasks.workunit.client.0.smithi144.stdout:7/951: dread d0/d6/d9/d12/d115/deb/f117 [0,4194304] 0 2024-07-26T11:05:04.051 INFO:tasks.workunit.client.0.smithi144.stdout:7/952: fsync d0/d6/d10c/f1e 0 2024-07-26T11:05:04.059 INFO:tasks.workunit.client.0.smithi144.stdout:4/901: sync 2024-07-26T11:05:04.059 INFO:tasks.workunit.client.0.smithi144.stdout:4/902: fdatasync d9/d26/d48/d50/f64 0 2024-07-26T11:05:04.059 INFO:tasks.workunit.client.0.smithi144.stdout:4/903: fsync d9/d7b/fcd 0 2024-07-26T11:05:04.059 INFO:tasks.workunit.client.0.smithi144.stdout:1/987: sync 2024-07-26T11:05:04.059 INFO:tasks.workunit.client.0.smithi144.stdout:1/988: dread - d3/d6/dc0/dea/ff4 zero size 2024-07-26T11:05:04.059 INFO:tasks.workunit.client.0.smithi144.stdout:5/895: sync 2024-07-26T11:05:04.062 INFO:tasks.workunit.client.0.smithi144.stdout:3/702: sync 2024-07-26T11:05:04.062 INFO:tasks.workunit.client.0.smithi144.stdout:2/873: sync 2024-07-26T11:05:04.062 INFO:tasks.workunit.client.0.smithi144.stdout:4/904: getdents d9/d7b 0 2024-07-26T11:05:04.062 INFO:tasks.workunit.client.0.smithi144.stdout:4/905: dread - d9/d26/d48/d98/fe4 zero size 2024-07-26T11:05:04.062 INFO:tasks.workunit.client.0.smithi144.stdout:4/906: fsync d9/d26/d3c/d6f/fb5 0 2024-07-26T11:05:04.063 INFO:tasks.workunit.client.0.smithi144.stdout:1/989: mknod d3/d6/da/d78/c140 0 2024-07-26T11:05:04.063 INFO:tasks.workunit.client.0.smithi144.stdout:5/896: creat d1/f136 x:0 0 0 2024-07-26T11:05:04.063 INFO:tasks.workunit.client.0.smithi144.stdout:5/897: chown d1/d9/d5b/d4d/f63 8 1 2024-07-26T11:05:04.063 INFO:tasks.workunit.client.0.smithi144.stdout:1/990: creat d3/d6/da/d13/d122/f141 x:0 0 0 2024-07-26T11:05:04.063 INFO:tasks.workunit.client.0.smithi144.stdout:2/874: rename d0/d24/d7b/ddc/d28/d2e/d77/ca4 to d0/d24/d2b/d68/d82/c10c 0 2024-07-26T11:05:04.065 INFO:tasks.workunit.client.0.smithi144.stdout:1/991: mkdir d3/d53/d142 0 2024-07-26T11:05:04.065 INFO:tasks.workunit.client.0.smithi144.stdout:1/992: getdents d3/d3d/d58/d101/d36/d130 0 2024-07-26T11:05:04.065 INFO:tasks.workunit.client.0.smithi144.stdout:9/945: sync 2024-07-26T11:05:04.065 INFO:tasks.workunit.client.0.smithi144.stdout:1/993: write d3/d6/f18 [973748,128770] 0 2024-07-26T11:05:04.065 INFO:tasks.workunit.client.0.smithi144.stdout:4/907: creat d9/da/da1/d115/f12f x:0 0 0 2024-07-26T11:05:04.066 INFO:tasks.workunit.client.0.smithi144.stdout:4/908: write d9/d26/fcc [475717,130907] 0 2024-07-26T11:05:04.066 INFO:tasks.workunit.client.0.smithi144.stdout:4/909: read - d9/d26/d3c/faf zero size 2024-07-26T11:05:04.067 INFO:tasks.workunit.client.0.smithi144.stdout:5/898: creat d1/d18/dc4/de0/deb/d116/d11b/f137 x:0 0 0 2024-07-26T11:05:04.068 INFO:tasks.workunit.client.0.smithi144.stdout:2/875: symlink d0/l10d 0 2024-07-26T11:05:04.068 INFO:tasks.workunit.client.0.smithi144.stdout:2/876: chown d0/d24/d7b/ddc/d28/d27/d8b/f109 1 1 2024-07-26T11:05:04.071 INFO:tasks.workunit.client.0.smithi144.stdout:5/899: truncate d1/d9/d5b/d4d/f6c 3174019 0 2024-07-26T11:05:04.071 INFO:tasks.workunit.client.0.smithi144.stdout:5/900: chown d1/d18/d24/d35/dd7/d12c/db2 4 1 2024-07-26T11:05:04.071 INFO:tasks.workunit.client.0.smithi144.stdout:2/877: rmdir d0/d24/d7b/ddc/d28/d2e/d5d/dae 39 2024-07-26T11:05:04.072 INFO:tasks.workunit.client.0.smithi144.stdout:5/901: symlink d1/d18/dc4/ded/dc8/l138 0 2024-07-26T11:05:04.074 INFO:tasks.workunit.client.0.smithi144.stdout:2/878: creat d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2/f10e x:0 0 0 2024-07-26T11:05:04.074 INFO:tasks.workunit.client.0.smithi144.stdout:5/902: creat d1/d9/d44/d45/d81/f139 x:0 0 0 2024-07-26T11:05:04.075 INFO:tasks.workunit.client.0.smithi144.stdout:2/879: creat d0/d24/d2b/d68/f10f x:0 0 0 2024-07-26T11:05:04.076 INFO:tasks.workunit.client.0.smithi144.stdout:5/903: mkdir d1/d18/dc4/ded/dc8/d13a 0 2024-07-26T11:05:04.089 INFO:tasks.workunit.client.0.smithi144.stdout:5/904: unlink d1/d9/d5b/c50 0 2024-07-26T11:05:04.089 INFO:tasks.workunit.client.0.smithi144.stdout:5/905: fsync d1/d9/d44/d45/d81/fbf 0 2024-07-26T11:05:04.092 INFO:tasks.workunit.client.0.smithi144.stdout:2/880: dread d0/d24/d7b/ddc/d28/f72 [0,4194304] 0 2024-07-26T11:05:04.092 INFO:tasks.workunit.client.0.smithi144.stdout:5/906: creat d1/d18/d24/d35/dd7/d12c/db2/f13b x:0 0 0 2024-07-26T11:05:04.093 INFO:tasks.workunit.client.0.smithi144.stdout:2/881: write d0/d24/d7b/ddc/d28/d27/d8b/fe1 [694055,114100] 0 2024-07-26T11:05:04.093 INFO:tasks.workunit.client.0.smithi144.stdout:2/882: write d0/d24/f100 [1025892,16253] 0 2024-07-26T11:05:04.093 INFO:tasks.workunit.client.0.smithi144.stdout:2/883: write d0/d6/f95 [3507747,112308] 0 2024-07-26T11:05:04.093 INFO:tasks.workunit.client.0.smithi144.stdout:2/884: chown d0/d24/d2b/d68 2316134 1 2024-07-26T11:05:04.096 INFO:tasks.workunit.client.0.smithi144.stdout:2/885: unlink d0/d24/d2b/d68/d82/fab 0 2024-07-26T11:05:04.097 INFO:tasks.workunit.client.0.smithi144.stdout:2/886: creat d0/d24/d7b/ddc/d28/d2e/d5d/dae/f110 x:0 0 0 2024-07-26T11:05:04.101 INFO:tasks.workunit.client.0.smithi144.stdout:2/887: rename d0/db8/fde to d0/d24/f111 0 2024-07-26T11:05:04.102 INFO:tasks.workunit.client.0.smithi144.stdout:2/888: readlink d0/d24/d2b/d68/l9c 0 2024-07-26T11:05:04.109 INFO:tasks.workunit.client.0.smithi144.stdout:8/901: dwrite d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/fd6 [0,4194304] 0 2024-07-26T11:05:04.109 INFO:tasks.workunit.client.0.smithi144.stdout:8/902: dread - d1a/d32/d49/d73/d7b/d43/d4f/fc9 zero size 2024-07-26T11:05:04.109 INFO:tasks.workunit.client.0.smithi144.stdout:8/903: stat d1a/d32/d49/d73/d7b/d4e/d78/lbf 0 2024-07-26T11:05:04.115 INFO:tasks.workunit.client.0.smithi144.stdout:8/904: rmdir d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0 39 2024-07-26T11:05:04.115 INFO:tasks.workunit.client.0.smithi144.stdout:8/905: truncate d1a/fc8 1059487 0 2024-07-26T11:05:04.116 INFO:tasks.workunit.client.0.smithi144.stdout:8/906: fdatasync d1a/d32/d66/da0/ddc/ff7 0 2024-07-26T11:05:04.116 INFO:tasks.workunit.client.0.smithi144.stdout:3/703: write d1/d28/d54/d3d/f40 [1204105,104501] 0 2024-07-26T11:05:04.116 INFO:tasks.workunit.client.0.smithi144.stdout:3/704: dread - d1/d28/d54/d26/d64/dd5/fd8 zero size 2024-07-26T11:05:04.116 INFO:tasks.workunit.client.0.smithi144.stdout:0/963: dwrite d0/f18 [4194304,4194304] 0 2024-07-26T11:05:04.116 INFO:tasks.workunit.client.0.smithi144.stdout:3/705: mknod d1/d5/d97/cd9 0 2024-07-26T11:05:04.116 INFO:tasks.workunit.client.0.smithi144.stdout:0/964: rmdir d0/d3e/d45/dd4 39 2024-07-26T11:05:04.119 INFO:tasks.workunit.client.0.smithi144.stdout:0/965: symlink d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/d107/l13f 0 2024-07-26T11:05:04.120 INFO:tasks.workunit.client.0.smithi144.stdout:9/946: dwrite d9/d3b/dce/f109 [0,4194304] 0 2024-07-26T11:05:04.120 INFO:tasks.workunit.client.0.smithi144.stdout:9/947: read d9/d3b/d81/ddd/f127 [1000993,130455] 0 2024-07-26T11:05:04.125 INFO:tasks.workunit.client.0.smithi144.stdout:4/910: dwrite d9/d26/d48/d50/d60/f6e [0,4194304] 0 2024-07-26T11:05:04.125 INFO:tasks.workunit.client.0.smithi144.stdout:4/911: readlink d9/d26/d48/d50/d60/dc5/dd9/de1/l103 0 2024-07-26T11:05:04.125 INFO:tasks.workunit.client.0.smithi144.stdout:4/912: readlink d9/d26/d48/d50/lcf 0 2024-07-26T11:05:04.127 INFO:tasks.workunit.client.0.smithi144.stdout:4/913: mknod d9/d26/d48/d50/d60/c130 0 2024-07-26T11:05:04.128 INFO:tasks.workunit.client.0.smithi144.stdout:4/914: write d9/da/ddd/fe0 [1236838,112313] 0 2024-07-26T11:05:04.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/915: mkdir d9/d26/d48/d50/d60/d131 0 2024-07-26T11:05:04.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/916: stat d9/d57/l91 0 2024-07-26T11:05:04.130 INFO:tasks.workunit.client.0.smithi144.stdout:4/917: fdatasync d9/d26/d3c/d6f/dd0/f12a 0 2024-07-26T11:05:04.131 INFO:tasks.workunit.client.0.smithi144.stdout:4/918: readlink d9/lae 0 2024-07-26T11:05:04.131 INFO:tasks.workunit.client.0.smithi144.stdout:4/919: readlink d9/d26/l27 0 2024-07-26T11:05:04.131 INFO:tasks.workunit.client.0.smithi144.stdout:4/920: write d9/d26/d48/fa0 [1304140,76286] 0 2024-07-26T11:05:04.131 INFO:tasks.workunit.client.0.smithi144.stdout:4/921: chown d9/d26/d48/f66 73048 1 2024-07-26T11:05:04.131 INFO:tasks.workunit.client.0.smithi144.stdout:9/948: rename d9/la6 to d9/d3b/d76/l14e 0 2024-07-26T11:05:04.135 INFO:tasks.workunit.client.0.smithi144.stdout:5/907: write d1/d9/d5b/d4d/f3e [2309358,28902] 0 2024-07-26T11:05:04.136 INFO:tasks.workunit.client.0.smithi144.stdout:1/994: dwrite d3/d6/da/d21/d5c/d62/f7c [4194304,4194304] 0 2024-07-26T11:05:04.137 INFO:tasks.workunit.client.0.smithi144.stdout:5/908: creat d1/d18/d127/f13c x:0 0 0 2024-07-26T11:05:04.144 INFO:tasks.workunit.client.0.smithi144.stdout:2/889: write d0/d24/d2b/fbb [3539634,49815] 0 2024-07-26T11:05:04.150 INFO:tasks.workunit.client.0.smithi144.stdout:3/706: dread d1/f50 [0,4194304] 0 2024-07-26T11:05:04.150 INFO:tasks.workunit.client.0.smithi144.stdout:3/707: read d1/d5/f3a [830664,60730] 0 2024-07-26T11:05:04.151 INFO:tasks.workunit.client.0.smithi144.stdout:5/909: rmdir d1/d18/da4/d132 0 2024-07-26T11:05:04.155 INFO:tasks.workunit.client.0.smithi144.stdout:2/890: mknod d0/d24/d7b/ddc/c112 0 2024-07-26T11:05:04.155 INFO:tasks.workunit.client.0.smithi144.stdout:3/708: truncate d1/d28/d54/d26/d64/d68/f7d 238163 0 2024-07-26T11:05:04.155 INFO:tasks.workunit.client.0.smithi144.stdout:2/891: dread - d0/d24/d7b/ddc/d28/d2e/d77/d84/de3/f102 zero size 2024-07-26T11:05:04.155 INFO:tasks.workunit.client.0.smithi144.stdout:5/910: mkdir d1/d18/d24/d35/dd7/d12c/d13d 0 2024-07-26T11:05:04.155 INFO:tasks.workunit.client.0.smithi144.stdout:5/911: chown d1/d18/d24/d35/d10f/dea/df9/l10a 2943 1 2024-07-26T11:05:04.156 INFO:tasks.workunit.client.0.smithi144.stdout:8/907: dread d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/f6e [0,4194304] 0 2024-07-26T11:05:04.156 INFO:tasks.workunit.client.0.smithi144.stdout:5/912: creat d1/d18/d24/d35/d10f/dea/f13e x:0 0 0 2024-07-26T11:05:04.157 INFO:tasks.workunit.client.0.smithi144.stdout:5/913: getdents d1/d18/dc4/ded/dc8 0 2024-07-26T11:05:04.158 INFO:tasks.workunit.client.0.smithi144.stdout:5/914: fsync d1/f5e 0 2024-07-26T11:05:04.159 INFO:tasks.workunit.client.0.smithi144.stdout:5/915: rename d1/d18/d24/d35/fb9 to d1/d18/d24/d35/dd7/d12c/f13f 0 2024-07-26T11:05:04.160 INFO:tasks.workunit.client.0.smithi144.stdout:5/916: rmdir d1/d18/dc4/de0/deb 39 2024-07-26T11:05:04.161 INFO:tasks.workunit.client.0.smithi144.stdout:5/917: creat d1/d9/f140 x:0 0 0 2024-07-26T11:05:04.164 INFO:tasks.workunit.client.0.smithi144.stdout:5/918: rmdir d1/d18/d24/dfb 0 2024-07-26T11:05:04.164 INFO:tasks.workunit.client.0.smithi144.stdout:5/919: chown d1/d9/d5b/d4d/ce5 50257319 1 2024-07-26T11:05:04.166 INFO:tasks.workunit.client.0.smithi144.stdout:5/920: link d1/d18/d24/d27/l68 d1/d18/dc4/de0/deb/l141 0 2024-07-26T11:05:04.166 INFO:tasks.workunit.client.0.smithi144.stdout:5/921: rename d1/da8 to d1/da8/d142 22 2024-07-26T11:05:04.167 INFO:tasks.workunit.client.0.smithi144.stdout:5/922: creat d1/da8/f143 x:0 0 0 2024-07-26T11:05:04.171 INFO:tasks.workunit.client.0.smithi144.stdout:5/923: creat d1/d5/dae/f144 x:0 0 0 2024-07-26T11:05:04.171 INFO:tasks.workunit.client.0.smithi144.stdout:5/924: dread - d1/d18/dc4/de0/f117 zero size 2024-07-26T11:05:04.172 INFO:tasks.workunit.client.0.smithi144.stdout:1/995: write d3/d6/fd [1062626,1630] 0 2024-07-26T11:05:04.172 INFO:tasks.workunit.client.0.smithi144.stdout:1/996: dread - d3/d6/d75/f115 zero size 2024-07-26T11:05:04.172 INFO:tasks.workunit.client.0.smithi144.stdout:5/925: mknod d1/d18/d24/d35/dd7/c145 0 2024-07-26T11:05:04.172 INFO:tasks.workunit.client.0.smithi144.stdout:5/926: chown d1/d18/dc4/de0/deb 108 1 2024-07-26T11:05:04.174 INFO:tasks.workunit.client.0.smithi144.stdout:5/927: symlink d1/d9/d44/da1/ddb/df7/l146 0 2024-07-26T11:05:04.174 INFO:tasks.workunit.client.0.smithi144.stdout:5/928: dread - d1/d9/f8b zero size 2024-07-26T11:05:04.176 INFO:tasks.workunit.client.0.smithi144.stdout:5/929: mknod d1/d9/d44/c147 0 2024-07-26T11:05:04.180 INFO:tasks.workunit.client.0.smithi144.stdout:1/997: rename d3/d6/da/d13/d19/d3a/c64 to d3/d3d/d128/c143 0 2024-07-26T11:05:04.180 INFO:tasks.workunit.client.0.smithi144.stdout:1/998: dread - d3/d6/dc0/dea/ff4 zero size 2024-07-26T11:05:04.181 INFO:tasks.workunit.client.0.smithi144.stdout:1/999: chown d3/d6/da/d21/d5c/d62/d68 137 1 2024-07-26T11:05:04.184 INFO:tasks.workunit.client.0.smithi144.stdout:5/930: rename d1/d9/d44/da1/fad to d1/d18/dc4/ded/d119/f148 0 2024-07-26T11:05:04.184 INFO:tasks.workunit.client.0.smithi144.stdout:5/931: chown d1/d9/d44/d45 10 1 2024-07-26T11:05:04.184 INFO:tasks.workunit.client.0.smithi144.stdout:5/932: truncate d1/d9/d44/f107 844001 0 2024-07-26T11:05:04.191 INFO:tasks.workunit.client.0.smithi144.stdout:0/966: dwrite d0/d3e/d45/dd4/d11f/d5e/d1e/d30/dab/fc0 [0,4194304] 0 2024-07-26T11:05:04.191 INFO:tasks.workunit.client.0.smithi144.stdout:0/967: chown d0/d3e/d45/dd4/d11f/d5e/d1e/l26 2424997 1 2024-07-26T11:05:04.193 INFO:tasks.workunit.client.0.smithi144.stdout:0/968: getdents d0/d3e/d45/dd4/d11f/d5e/d1e/dc1 0 2024-07-26T11:05:04.193 INFO:tasks.workunit.client.0.smithi144.stdout:0/969: write d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/de4/ff1 [1012741,50873] 0 2024-07-26T11:05:04.195 INFO:tasks.workunit.client.0.smithi144.stdout:4/922: dwrite d9/d26/f2f [0,4194304] 0 2024-07-26T11:05:04.197 INFO:tasks.workunit.client.0.smithi144.stdout:0/970: creat d0/d3e/f140 x:0 0 0 2024-07-26T11:05:04.200 INFO:tasks.workunit.client.0.smithi144.stdout:4/923: mkdir d9/d26/d3c/d10c/d132 0 2024-07-26T11:05:04.200 INFO:tasks.workunit.client.0.smithi144.stdout:4/924: stat d9/d57/l90 0 2024-07-26T11:05:04.202 INFO:tasks.workunit.client.0.smithi144.stdout:5/933: getdents d1/d18/d24/d35/dd7/d12c 0 2024-07-26T11:05:04.202 INFO:tasks.workunit.client.0.smithi144.stdout:5/934: fsync d1/d18/d24/d35/dd7/fff 0 2024-07-26T11:05:04.203 INFO:tasks.workunit.client.0.smithi144.stdout:0/971: rmdir d0/d3e/d45/dd4/d11f/d5e/d1e/d30/d6a/dac 39 2024-07-26T11:05:04.203 INFO:tasks.workunit.client.0.smithi144.stdout:0/972: write d0/d3e/d45/dd4/d11f/d5e/d1e/d91/f96 [649850,95001] 0 2024-07-26T11:05:04.207 INFO:tasks.workunit.client.0.smithi144.stdout:2/892: dwrite d0/d24/d7b/ddc/f96 [0,4194304] 0 2024-07-26T11:05:04.207 INFO:tasks.workunit.client.0.smithi144.stdout:8/908: dread d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/fcb [0,4194304] 0 2024-07-26T11:05:04.207 INFO:tasks.workunit.client.0.smithi144.stdout:5/935: mkdir d1/d18/dc4/ded/d149 0 2024-07-26T11:05:04.207 INFO:tasks.workunit.client.0.smithi144.stdout:9/949: dwrite d9/de/faa [0,4194304] 0 2024-07-26T11:05:04.208 INFO:tasks.workunit.client.0.smithi144.stdout:9/950: readlink d9/dfc/l120 0 2024-07-26T11:05:04.214 INFO:tasks.workunit.client.0.smithi144.stdout:3/709: write d1/d28/d54/d26/d64/d68/f84 [3736654,130429] 0 2024-07-26T11:05:04.215 INFO:tasks.workunit.client.0.smithi144.stdout:3/710: dread - d1/d5/d5e/f83 zero size 2024-07-26T11:05:04.215 INFO:tasks.workunit.client.0.smithi144.stdout:2/893: mkdir d0/d24/d7b/ddc/d28/da8/d113 0 2024-07-26T11:05:04.215 INFO:tasks.workunit.client.0.smithi144.stdout:8/909: creat d1a/d32/d49/d73/f12d x:0 0 0 2024-07-26T11:05:04.215 INFO:tasks.workunit.client.0.smithi144.stdout:9/951: symlink d9/de/d14/d79/db2/l14f 0 2024-07-26T11:05:04.218 INFO:tasks.workunit.client.0.smithi144.stdout:8/910: creat d1a/d32/d66/da0/f12e x:0 0 0 2024-07-26T11:05:04.219 INFO:tasks.workunit.client.0.smithi144.stdout:2/894: creat d0/d24/d48/d63/da3/f114 x:0 0 0 2024-07-26T11:05:04.219 INFO:tasks.workunit.client.0.smithi144.stdout:0/973: dread d0/d3e/d45/f55 [0,4194304] 0 2024-07-26T11:05:04.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/911: truncate d1a/d32/d49/d73/f2e 957932 0 2024-07-26T11:05:04.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/912: chown d1a/d32/d49/d73 1 1 2024-07-26T11:05:04.220 INFO:tasks.workunit.client.0.smithi144.stdout:8/913: chown d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/caf 21 1 2024-07-26T11:05:04.222 INFO:tasks.workunit.client.0.smithi144.stdout:0/974: rename d0/d3e/d45/dd4 to d0/d141 0 2024-07-26T11:05:04.225 INFO:tasks.workunit.client.0.smithi144.stdout:8/914: link d1a/f2b d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/f12f 0 2024-07-26T11:05:04.225 INFO:tasks.workunit.client.0.smithi144.stdout:8/915: mknod d1a/d32/d49/d102/c130 0 2024-07-26T11:05:04.227 INFO:tasks.workunit.client.0.smithi144.stdout:8/916: creat d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/f131 x:0 0 0 2024-07-26T11:05:04.230 INFO:tasks.workunit.client.0.smithi144.stdout:8/917: chown d1a/d32/d49/d73/d7b/d43/dae 1997066272 1 2024-07-26T11:05:04.230 INFO:tasks.workunit.client.0.smithi144.stdout:4/925: dread d9/d26/d48/d98/f126 [0,4194304] 0 2024-07-26T11:05:04.233 INFO:tasks.workunit.client.0.smithi144.stdout:4/926: unlink d9/d26/d3c/d6f/d8c/lb3 0 2024-07-26T11:05:04.235 INFO:tasks.workunit.client.0.smithi144.stdout:4/927: creat d9/d26/d3c/d10c/f133 x:0 0 0 2024-07-26T11:05:04.239 INFO:tasks.workunit.client.0.smithi144.stdout:8/918: dread d1a/d32/d49/d73/d7b/f8b [0,4194304] 0 2024-07-26T11:05:04.243 INFO:tasks.workunit.client.0.smithi144.stdout:8/919: symlink d1a/d32/d49/d73/d7b/d4e/l132 0 2024-07-26T11:05:04.244 INFO:tasks.workunit.client.0.smithi144.stdout:8/920: read d1a/d32/f62 [269956,107654] 0 2024-07-26T11:05:04.249 INFO:tasks.workunit.client.0.smithi144.stdout:0/975: write d0/d3e/d45/fae [3128004,15779] 0 2024-07-26T11:05:04.260 INFO:tasks.workunit.client.0.smithi144.stdout:0/976: creat d0/d141/d11f/d5e/d1e/d30/d6a/d6f/dec/f142 x:0 0 0 2024-07-26T11:05:04.263 INFO:tasks.workunit.client.0.smithi144.stdout:0/977: rename d0/d141/d11f/d5e/d1e/d30/d6a/dac/lf7 to d0/d141/d11f/d5e/d1e/dc1/dd8/l143 0 2024-07-26T11:05:04.263 INFO:tasks.workunit.client.0.smithi144.stdout:0/978: fsync d0/d141/d11f/d5e/d1e/d91/f96 0 2024-07-26T11:05:04.264 INFO:tasks.workunit.client.0.smithi144.stdout:0/979: symlink d0/d3e/l144 0 2024-07-26T11:05:04.265 INFO:tasks.workunit.client.0.smithi144.stdout:0/980: mknod d0/d3e/d45/d4c/c145 0 2024-07-26T11:05:04.268 INFO:tasks.workunit.client.0.smithi144.stdout:0/981: mkdir d0/d3e/d45/d4c/d146 0 2024-07-26T11:05:04.273 INFO:tasks.workunit.client.0.smithi144.stdout:4/928: read d9/d26/d48/d98/fb1 [1256248,16681] 0 2024-07-26T11:05:04.275 INFO:tasks.workunit.client.0.smithi144.stdout:4/929: link d9/f1f d9/d26/d3c/d6f/dd0/f134 0 2024-07-26T11:05:04.279 INFO:tasks.workunit.client.0.smithi144.stdout:4/930: mkdir d9/d26/d48/d135 0 2024-07-26T11:05:04.282 INFO:tasks.workunit.client.0.smithi144.stdout:4/931: creat d9/d7b/dc4/d111/d116/f136 x:0 0 0 2024-07-26T11:05:04.283 INFO:tasks.workunit.client.0.smithi144.stdout:4/932: link d9/da/f39 d9/d7b/f137 0 2024-07-26T11:05:04.284 INFO:tasks.workunit.client.0.smithi144.stdout:4/933: fsync d9/d26/d48/d50/d60/f88 0 2024-07-26T11:05:04.287 INFO:tasks.workunit.client.0.smithi144.stdout:3/711: dwrite d1/d28/d54/d26/d64/dd5/fd8 [0,4194304] 0 2024-07-26T11:05:04.293 INFO:tasks.workunit.client.0.smithi144.stdout:4/934: creat d9/d26/d3c/d6f/dd0/f138 x:0 0 0 2024-07-26T11:05:04.294 INFO:tasks.workunit.client.0.smithi144.stdout:3/712: creat d1/d28/d54/d26/d64/d68/db5/fda x:0 0 0 2024-07-26T11:05:04.297 INFO:tasks.workunit.client.0.smithi144.stdout:9/952: dwrite d9/de/d4e/f9d [0,4194304] 0 2024-07-26T11:05:04.298 INFO:tasks.workunit.client.0.smithi144.stdout:0/982: read d0/d141/d11f/d5e/d1e/d30/d67/f90 [1670912,16211] 0 2024-07-26T11:05:04.298 INFO:tasks.workunit.client.0.smithi144.stdout:0/983: write d0/d3e/d45/d9f/f6d [3377697,60545] 0 2024-07-26T11:05:04.300 INFO:tasks.workunit.client.0.smithi144.stdout:3/713: link d1/d28/d54/d26/d37/d91/f9c d1/d28/d54/d26/d37/da1/fdb 0 2024-07-26T11:05:04.300 INFO:tasks.workunit.client.0.smithi144.stdout:8/921: dwrite d1a/d32/d49/d73/d7b/d43/d4c/d128/f8a [4194304,4194304] 0 2024-07-26T11:05:04.301 INFO:tasks.workunit.client.0.smithi144.stdout:0/984: creat d0/d3e/d5f/f147 x:0 0 0 2024-07-26T11:05:04.305 INFO:tasks.workunit.client.0.smithi144.stdout:8/922: symlink d1a/d32/d49/d73/d7b/d4e/d78/l133 0 2024-07-26T11:05:04.307 INFO:tasks.workunit.client.0.smithi144.stdout:0/985: truncate d0/d141/d11f/d5e/d1e/d30/dab/f116 3103472 0 2024-07-26T11:05:04.309 INFO:tasks.workunit.client.0.smithi144.stdout:2/895: dwrite d0/d24/d48/d63/da3/ff7 [0,4194304] 0 2024-07-26T11:05:04.309 INFO:tasks.workunit.client.0.smithi144.stdout:8/923: creat d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/f134 x:0 0 0 2024-07-26T11:05:04.310 INFO:tasks.workunit.client.0.smithi144.stdout:0/986: creat d0/d3e/db4/f148 x:0 0 0 2024-07-26T11:05:04.317 INFO:tasks.workunit.client.0.smithi144.stdout:2/896: rmdir d0/d24/d7b/ddc/d28/d2e/d77/d84/de3 39 2024-07-26T11:05:04.319 INFO:tasks.workunit.client.0.smithi144.stdout:8/924: link d1a/d32/d49/d73/c5c d1a/d32/d49/d73/d7b/d43/d113/c135 0 2024-07-26T11:05:04.322 INFO:tasks.workunit.client.0.smithi144.stdout:2/897: creat d0/d6/f115 x:0 0 0 2024-07-26T11:05:04.322 INFO:tasks.workunit.client.0.smithi144.stdout:8/925: creat d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/da3/f136 x:0 0 0 2024-07-26T11:05:04.323 INFO:tasks.workunit.client.0.smithi144.stdout:8/926: rmdir d1a/d32/d49 39 2024-07-26T11:05:04.339 INFO:tasks.workunit.client.0.smithi144.stdout:4/935: dwrite d9/d26/d3c/d6f/d8c/ff4 [0,4194304] 0 2024-07-26T11:05:04.339 INFO:tasks.workunit.client.0.smithi144.stdout:4/936: chown d9/d26/l8b 136520684 1 2024-07-26T11:05:04.339 INFO:tasks.workunit.client.0.smithi144.stdout:4/937: fdatasync d9/d26/d48/d50/d60/fc3 0 2024-07-26T11:05:04.339 INFO:tasks.workunit.client.0.smithi144.stdout:4/938: fsync d9/f22 0 2024-07-26T11:05:04.342 INFO:tasks.workunit.client.0.smithi144.stdout:4/939: link d9/d26/d48/d50/d60/la8 d9/l139 0 2024-07-26T11:05:04.343 INFO:tasks.workunit.client.0.smithi144.stdout:4/940: mknod d9/d26/d48/c13a 0 2024-07-26T11:05:04.343 INFO:tasks.workunit.client.0.smithi144.stdout:4/941: write d9/f12 [2739412,49069] 0 2024-07-26T11:05:04.343 INFO:tasks.workunit.client.0.smithi144.stdout:4/942: dread - d9/d26/d3c/d6f/dd0/f138 zero size 2024-07-26T11:05:04.346 INFO:tasks.workunit.client.0.smithi144.stdout:9/953: dwrite d9/fca [0,4194304] 0 2024-07-26T11:05:04.349 INFO:tasks.workunit.client.0.smithi144.stdout:9/954: creat d9/dfc/d114/d116/d12d/f150 x:0 0 0 2024-07-26T11:05:04.352 INFO:tasks.workunit.client.0.smithi144.stdout:9/955: mknod d9/de/d4e/d6a/deb/c151 0 2024-07-26T11:05:04.352 INFO:tasks.workunit.client.0.smithi144.stdout:9/956: chown d9/de/d14/d23/d39/c48 58 1 2024-07-26T11:05:04.353 INFO:tasks.workunit.client.0.smithi144.stdout:3/714: dwrite d1/d28/d54/d26/d64/fd6 [0,4194304] 0 2024-07-26T11:05:04.354 INFO:tasks.workunit.client.0.smithi144.stdout:3/715: mknod d1/d28/d54/d61/cdc 0 2024-07-26T11:05:04.358 INFO:tasks.workunit.client.0.smithi144.stdout:3/716: mkdir d1/d28/d54/d26/d64/da4/ddd 0 2024-07-26T11:05:04.358 INFO:tasks.workunit.client.0.smithi144.stdout:3/717: fdatasync d1/d28/f9d 0 2024-07-26T11:05:04.358 INFO:tasks.workunit.client.0.smithi144.stdout:3/718: write d1/d28/d54/d26/f7c [3341358,125503] 0 2024-07-26T11:05:04.358 INFO:tasks.workunit.client.0.smithi144.stdout:3/719: link d1/d28/d54/d26/d37/d91/f9c d1/d28/d54/d26/d37/da1/fde 0 2024-07-26T11:05:04.361 INFO:tasks.workunit.client.0.smithi144.stdout:0/987: dwrite d0/f27 [0,4194304] 0 2024-07-26T11:05:04.368 INFO:tasks.workunit.client.0.smithi144.stdout:9/957: dread d9/de/f3d [0,4194304] 0 2024-07-26T11:05:04.369 INFO:tasks.workunit.client.0.smithi144.stdout:9/958: creat d9/de/d14/d23/d39/d7f/d123/f152 x:0 0 0 2024-07-26T11:05:04.370 INFO:tasks.workunit.client.0.smithi144.stdout:8/927: dwrite d1a/d32/d8c/f11f [0,4194304] 0 2024-07-26T11:05:04.371 INFO:tasks.workunit.client.0.smithi144.stdout:9/959: creat d9/de/d14/d23/d39/d7f/dbc/d10c/f153 x:0 0 0 2024-07-26T11:05:04.371 INFO:tasks.workunit.client.0.smithi144.stdout:9/960: chown d9/de/d14/d3f/l43 1553401 1 2024-07-26T11:05:04.373 INFO:tasks.workunit.client.0.smithi144.stdout:9/961: unlink d9/de/d4e/f9d 0 2024-07-26T11:05:04.374 INFO:tasks.workunit.client.0.smithi144.stdout:0/988: rename d0/d141/d11f/d5e/d1e/d30/d6a/d6f/dec/d10d to d0/d141/d11f/d5e/d1e/d91/dd1/d149 0 2024-07-26T11:05:04.378 INFO:tasks.workunit.client.0.smithi144.stdout:0/989: unlink d0/d3e/d45/f55 0 2024-07-26T11:05:04.379 INFO:tasks.workunit.client.0.smithi144.stdout:9/962: write d9/de/d14/d3f/d5e/fc0 [266531,81101] 0 2024-07-26T11:05:04.380 INFO:tasks.workunit.client.0.smithi144.stdout:0/990: creat d0/d141/d11f/d5e/d1e/d30/dab/d12d/d10f/f14a x:0 0 0 2024-07-26T11:05:04.385 INFO:tasks.workunit.client.0.smithi144.stdout:9/963: unlink d9/de/d14/d23/cb1 0 2024-07-26T11:05:04.387 INFO:tasks.workunit.client.0.smithi144.stdout:9/964: mknod d9/de/d4e/d6a/deb/c154 0 2024-07-26T11:05:04.395 INFO:tasks.workunit.client.0.smithi144.stdout:3/720: dwrite d1/d5/d5e/f83 [0,4194304] 0 2024-07-26T11:05:04.395 INFO:tasks.workunit.client.0.smithi144.stdout:8/928: write d1a/d32/d49/d73/d7b/d43/d4c/f54 [7358519,47128] 0 2024-07-26T11:05:04.396 INFO:tasks.workunit.client.0.smithi144.stdout:8/929: write d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dd8/fec [1041009,127042] 0 2024-07-26T11:05:04.396 INFO:tasks.workunit.client.0.smithi144.stdout:8/930: fdatasync d1a/d32/d49/d73/f2e 0 2024-07-26T11:05:04.396 INFO:tasks.workunit.client.0.smithi144.stdout:9/965: getdents d9/de/d14/d23 0 2024-07-26T11:05:04.396 INFO:tasks.workunit.client.0.smithi144.stdout:9/966: chown d9/d3b/d5d/fac 2034345 1 2024-07-26T11:05:04.400 INFO:tasks.workunit.client.0.smithi144.stdout:3/721: dread d1/d28/d54/d26/d30/f7f [0,4194304] 0 2024-07-26T11:05:04.400 INFO:tasks.workunit.client.0.smithi144.stdout:3/722: stat d1/d28/d54/d26/f2d 0 2024-07-26T11:05:04.400 INFO:tasks.workunit.client.0.smithi144.stdout:3/723: write d1/d28/d54/d26/f27 [3094324,104187] 0 2024-07-26T11:05:04.400 INFO:tasks.workunit.client.0.smithi144.stdout:8/931: rename d1a/d32/d8c/d9b/l100 to d1a/d1c/l137 0 2024-07-26T11:05:04.406 INFO:tasks.workunit.client.0.smithi144.stdout:8/932: dread d1a/d1c/f99 [0,4194304] 0 2024-07-26T11:05:04.407 INFO:tasks.workunit.client.0.smithi144.stdout:8/933: write d1a/d32/d49/d73/d7b/d43/d4c/d128/f8a [1352190,91055] 0 2024-07-26T11:05:04.407 INFO:tasks.workunit.client.0.smithi144.stdout:8/934: chown d1a/d32/d49/d73/d7b/d43/d4c/d128/f4d 19761942 1 2024-07-26T11:05:04.407 INFO:tasks.workunit.client.0.smithi144.stdout:8/935: fdatasync d1a/d32/d49/f11b 0 2024-07-26T11:05:04.408 INFO:tasks.workunit.client.0.smithi144.stdout:8/936: symlink d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/l138 0 2024-07-26T11:05:04.408 INFO:tasks.workunit.client.0.smithi144.stdout:8/937: stat d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/f131 0 2024-07-26T11:05:04.408 INFO:tasks.workunit.client.0.smithi144.stdout:8/938: dread - d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/f131 zero size 2024-07-26T11:05:04.410 INFO:tasks.workunit.client.0.smithi144.stdout:3/724: dread d1/d28/d54/d26/d37/da1/fdb [0,4194304] 0 2024-07-26T11:05:04.411 INFO:tasks.workunit.client.0.smithi144.stdout:3/725: symlink d1/d28/d54/d26/d64/ldf 0 2024-07-26T11:05:04.435 INFO:tasks.workunit.client.0.smithi144.stdout:8/939: write d1a/d32/d49/d73/d7b/d43/d4c/d128/f4d [1132919,93524] 0 2024-07-26T11:05:04.437 INFO:tasks.workunit.client.0.smithi144.stdout:0/991: dwrite d0/d3e/f140 [0,4194304] 0 2024-07-26T11:05:04.437 INFO:tasks.workunit.client.0.smithi144.stdout:8/940: rmdir d1a/d1c 39 2024-07-26T11:05:04.439 INFO:tasks.workunit.client.0.smithi144.stdout:9/967: dwrite d9/de/fd8 [0,4194304] 0 2024-07-26T11:05:04.439 INFO:tasks.workunit.client.0.smithi144.stdout:0/992: mknod d0/d141/d11f/d5e/d1e/d30/dab/d107/c14b 0 2024-07-26T11:05:04.441 INFO:tasks.workunit.client.0.smithi144.stdout:0/993: rename d0/d3e/d45/d4c/f64 to d0/d3e/db4/d11e/d12f/f14c 0 2024-07-26T11:05:04.443 INFO:tasks.workunit.client.0.smithi144.stdout:0/994: symlink d0/d141/d11f/d5e/def/l14d 0 2024-07-26T11:05:04.458 INFO:tasks.workunit.client.0.smithi144.stdout:9/968: dread d9/de/f30 [0,4194304] 0 2024-07-26T11:05:04.458 INFO:tasks.workunit.client.0.smithi144.stdout:9/969: read - d9/d3b/dce/de4/f129 zero size 2024-07-26T11:05:04.459 INFO:tasks.workunit.client.0.smithi144.stdout:9/970: mknod d9/d3b/d5d/c155 0 2024-07-26T11:05:04.460 INFO:tasks.workunit.client.0.smithi144.stdout:9/971: rmdir d9/de/d4e/dff 39 2024-07-26T11:05:04.461 INFO:tasks.workunit.client.0.smithi144.stdout:9/972: creat d9/de/d14/d23/f156 x:0 0 0 2024-07-26T11:05:04.461 INFO:tasks.workunit.client.0.smithi144.stdout:9/973: stat d9/de/d14/d79/db2/fc8 0 2024-07-26T11:05:04.462 INFO:tasks.workunit.client.0.smithi144.stdout:9/974: mkdir d9/de/d14/d79/d157 0 2024-07-26T11:05:04.463 INFO:tasks.workunit.client.0.smithi144.stdout:9/975: mknod d9/de/d14/d23/c158 0 2024-07-26T11:05:04.464 INFO:tasks.workunit.client.0.smithi144.stdout:9/976: rmdir d9/de/d14/d79 39 2024-07-26T11:05:04.465 INFO:tasks.workunit.client.0.smithi144.stdout:9/977: creat d9/de/d14/d79/d124/d136/f159 x:0 0 0 2024-07-26T11:05:04.465 INFO:tasks.workunit.client.0.smithi144.stdout:9/978: write d9/de/fbf [28852,79545] 0 2024-07-26T11:05:04.466 INFO:tasks.workunit.client.0.smithi144.stdout:9/979: creat d9/dfc/d114/f15a x:0 0 0 2024-07-26T11:05:04.468 INFO:tasks.workunit.client.0.smithi144.stdout:9/980: unlink d9/de/d14/d23/d39/d7f/l103 0 2024-07-26T11:05:04.469 INFO:tasks.workunit.client.0.smithi144.stdout:9/981: truncate d9/d3b/f78 1638077 0 2024-07-26T11:05:04.469 INFO:tasks.workunit.client.0.smithi144.stdout:9/982: stat d9/l69 0 2024-07-26T11:05:04.471 INFO:tasks.workunit.client.0.smithi144.stdout:9/983: creat d9/de/d4e/dff/f15b x:0 0 0 2024-07-26T11:05:04.474 INFO:tasks.workunit.client.0.smithi144.stdout:9/984: truncate d9/dfc/d114/dcc/ff3 3164836 0 2024-07-26T11:05:04.474 INFO:tasks.workunit.client.0.smithi144.stdout:9/985: dread - d9/de/d4e/dff/f15b zero size 2024-07-26T11:05:04.475 INFO:tasks.workunit.client.0.smithi144.stdout:9/986: creat d9/dfc/d114/dcc/dea/f15c x:0 0 0 2024-07-26T11:05:04.477 INFO:tasks.workunit.client.0.smithi144.stdout:9/987: creat d9/dfc/d108/f15d x:0 0 0 2024-07-26T11:05:04.478 INFO:tasks.workunit.client.0.smithi144.stdout:9/988: chown f4 1313605336 1 2024-07-26T11:05:04.480 INFO:tasks.workunit.client.0.smithi144.stdout:9/989: mknod d9/d3b/d76/c15e 0 2024-07-26T11:05:04.484 INFO:tasks.workunit.client.0.smithi144.stdout:0/995: dwrite d0/d141/d11f/d5e/d1e/d91/dd1/f134 [0,4194304] 0 2024-07-26T11:05:04.485 INFO:tasks.workunit.client.0.smithi144.stdout:0/996: readlink d0/d141/d11f/d5e/d1e/d30/d4b/lbe 0 2024-07-26T11:05:04.486 INFO:tasks.workunit.client.0.smithi144.stdout:0/997: rmdir d0/d3e/d45 39 2024-07-26T11:05:04.487 INFO:tasks.workunit.client.0.smithi144.stdout:0/998: rmdir d0/d141/d11f/d5e/d1e/d6c/dc6 39 2024-07-26T11:05:04.534 INFO:tasks.workunit.client.0.smithi144.stdout:9/990: dwrite d9/dfc/d114/f15a [0,4194304] 0 2024-07-26T11:05:04.535 INFO:tasks.workunit.client.0.smithi144.stdout:0/999: dread d0/d3e/d45/fae [0,4194304] 0 2024-07-26T11:05:04.536 INFO:tasks.workunit.client.0.smithi144.stdout:9/991: mkdir d9/de/d14/d3f/d5e/d15f 0 2024-07-26T11:05:04.538 INFO:tasks.workunit.client.0.smithi144.stdout:9/992: rename d9/de/d4e/c9c to d9/de/d4e/dff/c160 0 2024-07-26T11:05:04.558 INFO:tasks.workunit.client.0.smithi144.stdout:9/993: dread d9/d3b/f64 [0,4194304] 0 2024-07-26T11:05:04.558 INFO:tasks.workunit.client.0.smithi144.stdout:9/994: write d9/d3b/f71 [1632805,7928] 0 2024-07-26T11:05:04.609 INFO:tasks.workunit.client.0.smithi144.stdout:9/995: dwrite d9/fc [0,4194304] 0 2024-07-26T11:05:04.614 INFO:tasks.workunit.client.0.smithi144.stdout:9/996: truncate d9/de/d14/d23/d39/fbe 3028385 0 2024-07-26T11:05:04.629 INFO:tasks.workunit.client.0.smithi144.stdout:9/997: dread d9/de/d14/d79/db2/fc8 [0,4194304] 0 2024-07-26T11:05:04.631 INFO:tasks.workunit.client.0.smithi144.stdout:9/998: mknod d9/d3b/d76/c161 0 2024-07-26T11:05:04.632 INFO:tasks.workunit.client.0.smithi144.stdout:9/999: chown d9/de/d14/d23/d39/d7f/f111 301041 1 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:8/941: sync 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:3/726: sync 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:2/898: sync 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:7/953: sync 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:4/943: sync 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:5/936: sync 2024-07-26T11:05:05.387 INFO:tasks.workunit.client.0.smithi144.stdout:3/727: dread - d1/d28/d54/d26/d37/fbb zero size 2024-07-26T11:05:05.388 INFO:tasks.workunit.client.0.smithi144.stdout:2/899: fsync d0/d24/d7b/fb0 0 2024-07-26T11:05:05.388 INFO:tasks.workunit.client.0.smithi144.stdout:5/937: read d1/d18/d24/d35/dd7/d12c/faf [260190,12052] 0 2024-07-26T11:05:05.388 INFO:tasks.workunit.client.0.smithi144.stdout:5/938: getdents d1/d18/d24/d35/d10f/dea/df9 0 2024-07-26T11:05:05.391 INFO:tasks.workunit.client.0.smithi144.stdout:8/942: mkdir d1a/d32/d50/d139 0 2024-07-26T11:05:05.392 INFO:tasks.workunit.client.0.smithi144.stdout:8/943: fdatasync d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/de4/f119 0 2024-07-26T11:05:05.397 INFO:tasks.workunit.client.0.smithi144.stdout:3/728: dread d1/d28/d54/d61/f63 [0,4194304] 0 2024-07-26T11:05:05.398 INFO:tasks.workunit.client.0.smithi144.stdout:4/944: creat d9/da/da1/d115/f13b x:0 0 0 2024-07-26T11:05:05.400 INFO:tasks.workunit.client.0.smithi144.stdout:5/939: mkdir d1/d9/d110/d14a 0 2024-07-26T11:05:05.404 INFO:tasks.workunit.client.0.smithi144.stdout:3/729: mknod d1/d28/d54/d26/ce0 0 2024-07-26T11:05:05.415 INFO:tasks.workunit.client.0.smithi144.stdout:7/954: creat d0/d71/d95/f14d x:0 0 0 2024-07-26T11:05:05.423 INFO:tasks.workunit.client.0.smithi144.stdout:7/955: creat d0/d71/dc0/f14e x:0 0 0 2024-07-26T11:05:05.433 INFO:tasks.workunit.client.0.smithi144.stdout:5/940: dread d1/d9/d44/f8a [0,4194304] 0 2024-07-26T11:05:05.435 INFO:tasks.workunit.client.0.smithi144.stdout:4/945: dread d9/d57/f72 [0,4194304] 0 2024-07-26T11:05:05.435 INFO:tasks.workunit.client.0.smithi144.stdout:4/946: write d9/da/fbf [442634,96690] 0 2024-07-26T11:05:05.436 INFO:tasks.workunit.client.0.smithi144.stdout:2/900: read d0/d24/d2b/d68/d82/fc4 [1576905,66110] 0 2024-07-26T11:05:05.436 INFO:tasks.workunit.client.0.smithi144.stdout:2/901: truncate d0/d24/d7b/ddc/ffa 868492 0 2024-07-26T11:05:05.437 INFO:tasks.workunit.client.0.smithi144.stdout:4/947: mknod d9/d26/d3c/d6f/dd0/c13c 0 2024-07-26T11:05:05.441 INFO:tasks.workunit.client.0.smithi144.stdout:2/902: truncate d0/d24/d2b/d68/d82/fc4 2584308 0 2024-07-26T11:05:05.442 INFO:tasks.workunit.client.0.smithi144.stdout:2/903: write d0/d24/d7b/ddc/d28/d2e/d5d/d6b/f99 [5366557,39592] 0 2024-07-26T11:05:05.443 INFO:tasks.workunit.client.0.smithi144.stdout:2/904: readlink d0/d6/d34/l55 0 2024-07-26T11:05:05.443 INFO:tasks.workunit.client.0.smithi144.stdout:2/905: fdatasync d0/d24/d7b/f87 0 2024-07-26T11:05:05.443 INFO:tasks.workunit.client.0.smithi144.stdout:2/906: readlink d0/d24/d7b/ddc/d28/d2e/d5d/dae/ld6 0 2024-07-26T11:05:05.448 INFO:tasks.workunit.client.0.smithi144.stdout:2/907: truncate d0/d24/d48/d63/fa7 1466569 0 2024-07-26T11:05:05.470 INFO:tasks.workunit.client.0.smithi144.stdout:8/944: dread d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/f83 [0,4194304] 0 2024-07-26T11:05:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:05 smithi144 ceph-mon[32129]: pgmap v539: 129 pgs: 129 active+clean; 1.8 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s rd, 70 MiB/s wr, 376 op/s 2024-07-26T11:05:05.475 INFO:tasks.workunit.client.0.smithi144.stdout:7/956: write d0/dd0/dbf/d2f/d68/fc6 [4348830,22833] 0 2024-07-26T11:05:05.477 INFO:tasks.workunit.client.0.smithi144.stdout:8/945: getdents d1a/d32/d8c/dd0 0 2024-07-26T11:05:05.478 INFO:tasks.workunit.client.0.smithi144.stdout:3/730: dread d1/d5/d97/f4e [0,4194304] 0 2024-07-26T11:05:05.478 INFO:tasks.workunit.client.0.smithi144.stdout:3/731: chown d1/d5/f24 295 1 2024-07-26T11:05:05.482 INFO:tasks.workunit.client.0.smithi144.stdout:8/946: mkdir d1a/d32/d8c/d9b/d13a 0 2024-07-26T11:05:05.482 INFO:tasks.workunit.client.0.smithi144.stdout:8/947: fdatasync d1a/d32/d49/d73/d7b/d43/d4f/fc9 0 2024-07-26T11:05:05.484 INFO:tasks.workunit.client.0.smithi144.stdout:8/948: getdents d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58 0 2024-07-26T11:05:05.484 INFO:tasks.workunit.client.0.smithi144.stdout:8/949: chown d1a/d32/d66/c118 800358 1 2024-07-26T11:05:05.484 INFO:tasks.workunit.client.0.smithi144.stdout:8/950: readlink l13 0 2024-07-26T11:05:05.487 INFO:tasks.workunit.client.0.smithi144.stdout:4/948: dread d9/d26/d48/d50/ffc [0,4194304] 0 2024-07-26T11:05:05.487 INFO:tasks.workunit.client.0.smithi144.stdout:8/951: creat d1a/d1c/f13b x:0 0 0 2024-07-26T11:05:05.512 INFO:tasks.workunit.client.0.smithi144.stdout:2/908: dwrite d0/f1 [4194304,4194304] 0 2024-07-26T11:05:05.528 INFO:tasks.workunit.client.0.smithi144.stdout:2/909: dread d0/d24/f100 [0,4194304] 0 2024-07-26T11:05:05.528 INFO:tasks.workunit.client.0.smithi144.stdout:2/910: fsync d0/d6/d34/fca 0 2024-07-26T11:05:05.528 INFO:tasks.workunit.client.0.smithi144.stdout:2/911: read - d0/d6/f115 zero size 2024-07-26T11:05:05.529 INFO:tasks.workunit.client.0.smithi144.stdout:4/949: write d9/f8d [929978,26994] 0 2024-07-26T11:05:05.530 INFO:tasks.workunit.client.0.smithi144.stdout:4/950: chown d9/da/c10d 22489374 1 2024-07-26T11:05:05.530 INFO:tasks.workunit.client.0.smithi144.stdout:2/912: symlink d0/d24/d7b/ddc/d28/d2e/d77/l116 0 2024-07-26T11:05:05.530 INFO:tasks.workunit.client.0.smithi144.stdout:2/913: fsync d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fb5 0 2024-07-26T11:05:05.531 INFO:tasks.workunit.client.0.smithi144.stdout:4/951: creat d9/d26/d3c/d6f/dd0/f13d x:0 0 0 2024-07-26T11:05:05.532 INFO:tasks.workunit.client.0.smithi144.stdout:8/952: dwrite d1a/d32/d49/d73/d7b/f8b [0,4194304] 0 2024-07-26T11:05:05.534 INFO:tasks.workunit.client.0.smithi144.stdout:5/941: dwrite d1/d18/fdf [0,4194304] 0 2024-07-26T11:05:05.534 INFO:tasks.workunit.client.0.smithi144.stdout:4/952: creat d9/d26/d48/d50/d60/d131/f13e x:0 0 0 2024-07-26T11:05:05.535 INFO:tasks.workunit.client.0.smithi144.stdout:8/953: symlink d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/da3/db1/l13c 0 2024-07-26T11:05:05.537 INFO:tasks.workunit.client.0.smithi144.stdout:5/942: rename d1/d9/d5b/d86/ff4 to d1/d18/dc4/de0/f14b 0 2024-07-26T11:05:05.539 INFO:tasks.workunit.client.0.smithi144.stdout:4/953: link d9/d26/d48/d50/d60/d117/f119 d9/d26/d48/d98/d10b/f13f 0 2024-07-26T11:05:05.539 INFO:tasks.workunit.client.0.smithi144.stdout:4/954: dread - d9/d26/d48/d50/d60/dc5/dd9/fde zero size 2024-07-26T11:05:05.539 INFO:tasks.workunit.client.0.smithi144.stdout:4/955: getdents d9/d7b/dc4/d111/d116/d129 0 2024-07-26T11:05:05.540 INFO:tasks.workunit.client.0.smithi144.stdout:8/954: creat d1a/d32/d50/d139/f13d x:0 0 0 2024-07-26T11:05:05.540 INFO:tasks.workunit.client.0.smithi144.stdout:8/955: chown d1a/d32/d8c/d9b/d13a 6337910 1 2024-07-26T11:05:05.541 INFO:tasks.workunit.client.0.smithi144.stdout:8/956: readlink d1a/l5b 0 2024-07-26T11:05:05.543 INFO:tasks.workunit.client.0.smithi144.stdout:8/957: rename d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dd8/dfd/fff to d1a/d32/d50/d139/f13e 0 2024-07-26T11:05:05.545 INFO:tasks.workunit.client.0.smithi144.stdout:4/956: read d9/da/ddd/fe0 [1144085,128397] 0 2024-07-26T11:05:05.545 INFO:tasks.workunit.client.0.smithi144.stdout:4/957: stat d9/d26/d3c/l73 0 2024-07-26T11:05:05.545 INFO:tasks.workunit.client.0.smithi144.stdout:4/958: truncate d9/da/da1/d115/f12f 677732 0 2024-07-26T11:05:05.546 INFO:tasks.workunit.client.0.smithi144.stdout:4/959: mknod d9/d7b/c140 0 2024-07-26T11:05:05.549 INFO:tasks.workunit.client.0.smithi144.stdout:4/960: rename d9/d26/f97 to d9/da/f141 0 2024-07-26T11:05:05.559 INFO:tasks.workunit.client.0.smithi144.stdout:4/961: getdents d9/d26 0 2024-07-26T11:05:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:05 smithi160 ceph-mon[36715]: pgmap v539: 129 pgs: 129 active+clean; 1.8 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s rd, 70 MiB/s wr, 376 op/s 2024-07-26T11:05:05.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/962: dread d9/da/fb0 [0,4194304] 0 2024-07-26T11:05:05.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/963: dread - d9/d26/d48/d50/d60/fc3 zero size 2024-07-26T11:05:05.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/964: fdatasync d9/d26/d48/d50/d60/fc3 0 2024-07-26T11:05:05.574 INFO:tasks.workunit.client.0.smithi144.stdout:4/965: readlink d9/d26/d48/d50/l53 0 2024-07-26T11:05:05.586 INFO:tasks.workunit.client.0.smithi144.stdout:2/914: dwrite d0/d24/d2b/f8d [0,4194304] 0 2024-07-26T11:05:05.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/943: dwrite d1/d18/d24/d35/dd7/fff [0,4194304] 0 2024-07-26T11:05:05.590 INFO:tasks.workunit.client.0.smithi144.stdout:4/966: dread d9/d26/f58 [0,4194304] 0 2024-07-26T11:05:05.590 INFO:tasks.workunit.client.0.smithi144.stdout:5/944: chown d1/d9/d5b/d86/f126 720726014 1 2024-07-26T11:05:05.591 INFO:tasks.workunit.client.0.smithi144.stdout:4/967: creat d9/d26/d48/d50/d60/d131/f142 x:0 0 0 2024-07-26T11:05:05.592 INFO:tasks.workunit.client.0.smithi144.stdout:5/945: creat d1/d18/d12b/f14c x:0 0 0 2024-07-26T11:05:05.594 INFO:tasks.workunit.client.0.smithi144.stdout:5/946: mkdir d1/d18/d14d 0 2024-07-26T11:05:05.614 INFO:tasks.workunit.client.0.smithi144.stdout:2/915: dread d0/f1e [0,4194304] 0 2024-07-26T11:05:05.614 INFO:tasks.workunit.client.0.smithi144.stdout:2/916: dread - d0/d24/d2b/d68/f9d zero size 2024-07-26T11:05:05.616 INFO:tasks.workunit.client.0.smithi144.stdout:2/917: rename d0/d6/d34/d6c to d0/d24/d7b/ddc/d28/d2e/d5d/d61/d117 0 2024-07-26T11:05:05.617 INFO:tasks.workunit.client.0.smithi144.stdout:4/968: write d9/d26/d48/d50/f64 [1190230,118638] 0 2024-07-26T11:05:05.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/969: rename d9/d26/f92 to d9/d26/d48/d98/f143 0 2024-07-26T11:05:05.619 INFO:tasks.workunit.client.0.smithi144.stdout:4/970: chown d9/d26/d48/d50/c9c 0 1 2024-07-26T11:05:05.620 INFO:tasks.workunit.client.0.smithi144.stdout:4/971: creat d9/d26/d48/d98/db8/f144 x:0 0 0 2024-07-26T11:05:05.622 INFO:tasks.workunit.client.0.smithi144.stdout:4/972: rename d9/f22 to d9/d7b/dc4/d111/d116/d12e/f145 0 2024-07-26T11:05:05.622 INFO:tasks.workunit.client.0.smithi144.stdout:4/973: stat d9/d7b/fe6 0 2024-07-26T11:05:05.622 INFO:tasks.workunit.client.0.smithi144.stdout:4/974: write d9/d7b/dc4/f125 [891799,31771] 0 2024-07-26T11:05:05.623 INFO:tasks.workunit.client.0.smithi144.stdout:4/975: readlink d9/d57/l74 0 2024-07-26T11:05:05.624 INFO:tasks.workunit.client.0.smithi144.stdout:4/976: creat d9/d7b/dc4/d111/f146 x:0 0 0 2024-07-26T11:05:05.624 INFO:tasks.workunit.client.0.smithi144.stdout:4/977: write d9/d26/d3c/faf [63855,74633] 0 2024-07-26T11:05:05.624 INFO:tasks.workunit.client.0.smithi144.stdout:4/978: getdents d9/da/da1/d115/d122 0 2024-07-26T11:05:05.625 INFO:tasks.workunit.client.0.smithi144.stdout:4/979: stat d9/d7b/la2 0 2024-07-26T11:05:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:05 smithi183 ceph-mon[37273]: pgmap v539: 129 pgs: 129 active+clean; 1.8 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s rd, 70 MiB/s wr, 376 op/s 2024-07-26T11:05:05.671 INFO:tasks.workunit.client.0.smithi144.stdout:4/980: read d9/d26/d3c/d6f/dd0/f134 [2039744,95371] 0 2024-07-26T11:05:05.685 INFO:tasks.workunit.client.0.smithi144.stdout:2/918: dwrite d0/d24/d48/d63/da3/f114 [0,4194304] 0 2024-07-26T11:05:05.688 INFO:tasks.workunit.client.0.smithi144.stdout:2/919: mkdir d0/d118 0 2024-07-26T11:05:05.690 INFO:tasks.workunit.client.0.smithi144.stdout:2/920: unlink d0/d24/d7b/lea 0 2024-07-26T11:05:05.729 INFO:tasks.workunit.client.0.smithi144.stdout:2/921: read d0/d24/d7b/ddc/d28/d2e/d5d/d61/d117/f75 [1482083,16793] 0 2024-07-26T11:05:05.743 INFO:tasks.workunit.client.0.smithi144.stdout:4/981: dwrite d9/d26/d3c/d6f/dd0/f109 [0,4194304] 0 2024-07-26T11:05:05.743 INFO:tasks.workunit.client.0.smithi144.stdout:4/982: fsync d9/da/fb0 0 2024-07-26T11:05:05.758 INFO:tasks.workunit.client.0.smithi144.stdout:2/922: write d0/d24/d7b/ddc/d28/d2e/d77/fb2 [1724588,54063] 0 2024-07-26T11:05:05.759 INFO:tasks.workunit.client.0.smithi144.stdout:2/923: write d0/d24/d7b/ddc/d28/ff0 [502363,119242] 0 2024-07-26T11:05:05.759 INFO:tasks.workunit.client.0.smithi144.stdout:2/924: getdents d0/d118 0 2024-07-26T11:05:05.759 INFO:tasks.workunit.client.0.smithi144.stdout:2/925: write d0/d24/d2b/d68/f9d [695691,122504] 0 2024-07-26T11:05:05.807 INFO:tasks.workunit.client.0.smithi144.stdout:2/926: dwrite d0/d24/d7b/ddc/d28/d2e/d5d/d61/d117/f75 [0,4194304] 0 2024-07-26T11:05:05.807 INFO:tasks.workunit.client.0.smithi144.stdout:2/927: dread - d0/d6/d9/f108 zero size 2024-07-26T11:05:05.807 INFO:tasks.workunit.client.0.smithi144.stdout:2/928: dread - d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fb5 zero size 2024-07-26T11:05:05.858 INFO:tasks.workunit.client.0.smithi144.stdout:2/929: write d0/d24/d48/d63/da3/f114 [1488088,58524] 0 2024-07-26T11:05:05.858 INFO:tasks.workunit.client.0.smithi144.stdout:2/930: chown d0/d24/d7b/ddc/d28/d2e/d5d/cad 928072 1 2024-07-26T11:05:06.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 -- 172.21.15.144:0/295955142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88113fa0 msgr2=0x7f6d881163c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 --2- 172.21.15.144:0/295955142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88113fa0 0x7f6d881163c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009ca0 tx=0x7f6d8002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 -- 172.21.15.144:0/295955142 shutdown_connections 2024-07-26T11:05:06.913 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 --2- 172.21.15.144:0/295955142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88113fa0 0x7f6d881163c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 --2- 172.21.15.144:0/295955142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d88113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 --2- 172.21.15.144:0/295955142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d88077370 0x7f6d88077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.910+0000 7f6d8d800640 1 -- 172.21.15.144:0/295955142 >> 172.21.15.144:0/295955142 conn(0x7f6d881010b0 msgr2=0x7f6d88103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:06.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.911+0000 7f6d8d800640 1 -- 172.21.15.144:0/295955142 shutdown_connections 2024-07-26T11:05:06.914 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.911+0000 7f6d8d800640 1 -- 172.21.15.144:0/295955142 wait complete. 2024-07-26T11:05:06.915 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.912+0000 7f6d8d800640 1 Processor -- start 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.912+0000 7f6d8d800640 1 -- start start 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.912+0000 7f6d8d800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 0x7f6d881a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.912+0000 7f6d8d800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d881a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.912+0000 7f6d8d800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d88113fa0 0x7f6d881a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d8d800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6d88119920 con 0x7f6d88077370 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d8d800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881197a0 con 0x7f6d88113fa0 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d8d800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d88119aa0 con 0x7f6d88077cc0 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d881a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 0x7f6d881a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d881a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34918/0 (socket says 172.21.15.144:34918) 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 -- 172.21.15.144:0/4064230665 learned_addr learned my addr 172.21.15.144:0/4064230665 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:06.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 0x7f6d881a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37118/0 (socket says 172.21.15.144:37118) 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d88113fa0 0x7f6d881a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 -- 172.21.15.144:0/4064230665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d88113fa0 msgr2=0x7f6d881a5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d88113fa0 0x7f6d881a5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 -- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 msgr2=0x7f6d881a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 0x7f6d881a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d7fe00640 1 -- 172.21.15.144:0/4064230665 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d881a6260 con 0x7f6d88077cc0 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.913+0000 7f6d87e00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 0x7f6d881a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.914+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d881a0890 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e9c0 tx=0x7f6d7000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:06.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.914+0000 7f6d85600640 1 -- 172.21.15.144:0/4064230665 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7000ccc0 con 0x7f6d88077cc0 2024-07-26T11:05:06.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.914+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d881a6550 con 0x7f6d88077cc0 2024-07-26T11:05:06.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.914+0000 7f6d85600640 1 -- 172.21.15.144:0/4064230665 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d700054d0 con 0x7f6d88077cc0 2024-07-26T11:05:06.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.914+0000 7f6d85600640 1 -- 172.21.15.144:0/4064230665 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70010430 con 0x7f6d88077cc0 2024-07-26T11:05:06.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.914+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d881accb0 con 0x7f6d88077cc0 2024-07-26T11:05:06.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.918+0000 7f6d959bd740 1 Processor -- start 2024-07-26T11:05:06.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 -- start start 2024-07-26T11:05:06.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e23b070 0x55b72e5e0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 0x55b72e5e0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5e0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b72e556bf0 con 0x55b72e23b070 2024-07-26T11:05:06.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b72e5460f0 con 0x55b72e21a390 2024-07-26T11:05:06.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.919+0000 7f6d959bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b72e54a130 con 0x55b72e2121b0 2024-07-26T11:05:06.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d85600640 1 -- 172.21.15.144:0/4064230665 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f6d70005020 con 0x7f6d88077cc0 2024-07-26T11:05:06.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 0x55b72e5e0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d8ce00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e23b070 0x55b72e5e0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 0x55b72e5e0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:34934/0 (socket says 172.21.15.144:34934) 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 -- 172.21.15.144:0/2516807533 learned_addr learned my addr 172.21.15.144:0/2516807533 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5e0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 -- 172.21.15.144:0/2516807533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 msgr2=0x55b72e5e0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5e0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 -- 172.21.15.144:0/2516807533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e23b070 msgr2=0x55b72e5e0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e23b070 0x55b72e5e0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d87e00640 1 -- 172.21.15.144:0/2516807533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b72e2a6650 con 0x55b72e2121b0 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e23b070 0x55b72e5e0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:06.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.920+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5e0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:05:06.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.921+0000 7f6d87e00640 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 0x55b72e5e0570 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6d780115f0 tx=0x7f6d78011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2abd82c356987a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:06.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.921+0000 7f6d7d600640 1 -- 172.21.15.144:0/2516807533 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7801e070 con 0x55b72e2121b0 2024-07-26T11:05:06.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.921+0000 7f6d7d600640 1 -- 172.21.15.144:0/2516807533 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d7800faf0 con 0x55b72e2121b0 2024-07-26T11:05:06.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.921+0000 7f6d7d600640 1 -- 172.21.15.144:0/2516807533 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78019b50 con 0x55b72e2121b0 2024-07-26T11:05:06.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.921+0000 7f6d85600640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6d6c0b3d40 0x7f6d6c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d85600640 1 -- 172.21.15.144:0/4064230665 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6d700e46f0 con 0x7f6d88077cc0 2024-07-26T11:05:06.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6d6c0b3d40 0x7f6d6c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 -- 172.21.15.144:0/2516807533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 msgr2=0x55b72e5e0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 0x55b72e5e0570 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6d780115f0 tx=0x7f6d78011ab0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 -- 172.21.15.144:0/2516807533 shutdown_connections 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5e0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e2121b0 0x55b72e5e0570 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 --2- 172.21.15.144:0/2516807533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e23b070 0x55b72e5e0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 -- 172.21.15.144:0/2516807533 >> 172.21.15.144:0/2516807533 conn(0x55b72e22f460 msgr2=0x55b72e5ddf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.922+0000 7f6d959bd740 1 -- 172.21.15.144:0/2516807533 shutdown_connections 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.923+0000 7f6d959bd740 1 -- 172.21.15.144:0/2516807533 wait complete. 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.923+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6d6c0b3d40 0x7f6d6c0b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d8002f500 tx=0x7f6d8002f490 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.923+0000 7f6d959bd740 1 -- start start 2024-07-26T11:05:06.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.923+0000 7f6d959bd740 1 Processor -- start 2024-07-26T11:05:06.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5eb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5ebb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5ec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b72e3a9700 con 0x55b72e2121b0 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b72e4e26c0 con 0x55b72e21a390 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b72e5ee810 con 0x55b72e23b070 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d959bd740 10 client.0 _ll_register_callbacks cb 0x55b72e4f2170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5eb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37136/0 (socket says 172.21.15.144:37136) 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d87e00640 1 -- 172.21.15.144:0/994367313 learned_addr learned my addr 172.21.15.144:0/994367313 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:06.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5ebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.924+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5ec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d87e00640 1 -- 172.21.15.144:0/994367313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 msgr2=0x55b72e5ec0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d87e00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5ec0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d87e00640 1 -- 172.21.15.144:0/994367313 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 msgr2=0x55b72e5ebb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d87e00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5ebb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d87e00640 1 -- 172.21.15.144:0/994367313 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72e4f9860 con 0x55b72e2121b0 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5ec0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5ebb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d87e00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5eb630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800fff0 tx=0x7f6d78005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d18bef872e2b99b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.925+0000 7f6d76000640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:05:06.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d76000640 1 -- 172.21.15.144:0/994367313 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78027060 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d76000640 1 -- 172.21.15.144:0/994367313 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6d7800f780 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d7800faf0 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d76000640 1 -- 172.21.15.144:0/994367313 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d7802e050 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d959bd740 10 client.15750 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b72e5fa3d0 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d76000640 1 -- 172.21.15.144:0/994367313 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d7800faf0 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.926+0000 7f6d76000640 1 -- 172.21.15.144:0/994367313 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6d78021030 con 0x55b72e2121b0 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d959bd740 10 client.15750 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d959bd740 10 client.15750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d959bd740 20 client.15750 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d959bd740 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1f99f0 0x55b72e5fd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d959bd740 4 client.15750 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:06.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b72e558b40 con 0x55b72e1f99f0 2024-07-26T11:05:06.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.927+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1f99f0 0x55b72e5fd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.928+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1f99f0 0x55b72e5fd7e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7870bc96814f250 server_cookie=e15603c1a6ccd3b5 in_seq=0 out_seq=0 2024-07-26T11:05:06.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.928+0000 7f6d76000640 10 client.15750 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.928+0000 7f6d76000640 1 -- 172.21.15.144:0/994367313 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b72e558b40 con 0x55b72e1f99f0 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.928+0000 7f6d76000640 10 client.15750 handle_command_reply: tid=0 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 client.15750 shutdown 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 msgr2=0x55b72e5eb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5eb630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800fff0 tx=0x7f6d78005d90 comp rx=0 tx=0).stop 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 shutdown_connections 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5ec0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1f99f0 0x55b72e5fd7e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5ebb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 --2- 172.21.15.144:0/994367313 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5eb630 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d76000640 0 client.15750 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:05:06.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 >> 172.21.15.144:0/994367313 conn(0x55b72e22f460 msgr2=0x55b72e5e3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:06.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 shutdown_connections 2024-07-26T11:05:06.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 1 -- 172.21.15.144:0/994367313 wait complete. 2024-07-26T11:05:06.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.929+0000 7f6d959bd740 20 client.15750 trim_cache size 0 max 16384 2024-07-26T11:05:06.945 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 Processor -- start 2024-07-26T11:05:06.946 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 -- start start 2024-07-26T11:05:06.947 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5dcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.947 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e21a390 0x55b72e5e1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.947 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e23b070 0x55b72e602e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881b9540 con 0x55b72e2121b0 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881b6e10 con 0x55b72e23b070 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d959bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881b4c60 con 0x55b72e21a390 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5dcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.944+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5dcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37150/0 (socket says 172.21.15.144:37150) 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 -- 172.21.15.144:0/255822336 learned_addr learned my addr 172.21.15.144:0/255822336 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e23b070 0x55b72e602e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e21a390 0x55b72e5e1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 -- 172.21.15.144:0/255822336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e21a390 msgr2=0x55b72e5e1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e21a390 0x55b72e5e1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 -- 172.21.15.144:0/255822336 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e23b070 msgr2=0x55b72e602e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e23b070 0x55b72e602e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.948 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 -- 172.21.15.144:0/255822336 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b72e5e2800 con 0x55b72e2121b0 2024-07-26T11:05:06.949 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.945+0000 7f6d87e00640 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5dcf20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6d78000c00 tx=0x7f6d78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c4c4f585ad5821 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:06.949 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.946+0000 7f6d7cc00640 1 -- 172.21.15.144:0/255822336 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78027070 con 0x55b72e2121b0 2024-07-26T11:05:06.949 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.946+0000 7f6d7cc00640 1 -- 172.21.15.144:0/255822336 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d780055b0 con 0x55b72e2121b0 2024-07-26T11:05:06.949 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.946+0000 7f6d7cc00640 1 -- 172.21.15.144:0/255822336 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7801e040 con 0x55b72e2121b0 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 -- 172.21.15.144:0/255822336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 msgr2=0x55b72e5dcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5dcf20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6d78000c00 tx=0x7f6d78009b50 comp rx=0 tx=0).stop 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 -- 172.21.15.144:0/255822336 shutdown_connections 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e23b070 0x55b72e602e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e21a390 0x55b72e5e1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 --2- 172.21.15.144:0/255822336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5dcf20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 -- 172.21.15.144:0/255822336 >> 172.21.15.144:0/255822336 conn(0x55b72e22f460 msgr2=0x55b72e5e1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 -- 172.21.15.144:0/255822336 shutdown_connections 2024-07-26T11:05:06.950 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 -- 172.21.15.144:0/255822336 wait complete. 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.947+0000 7f6d959bd740 1 -- start start 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 Processor -- start 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5f1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5f1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881b52d0 con 0x55b72e2121b0 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881b6740 con 0x55b72e21a390 2024-07-26T11:05:06.951 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d881b59a0 con 0x55b72e23b070 2024-07-26T11:05:06.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d959bd740 10 client.0 _ll_register_callbacks cb 0x7f6d78011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:06.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.948+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.954 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d87e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5f0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37154/0 (socket says 172.21.15.144:37154) 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d87e00640 1 -- 172.21.15.144:0/3854941162 learned_addr learned my addr 172.21.15.144:0/3854941162 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d8ce00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 -- 172.21.15.144:0/3854941162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 msgr2=0x55b72e5f1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5f1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 -- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 msgr2=0x55b72e5f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 -- 172.21.15.144:0/3854941162 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72e663f60 con 0x55b72e21a390 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d87e00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d7fe00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5f1000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d700e1a70 tx=0x7f6d700053d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59eeb66d265c2700 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d76000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:05:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.949+0000 7f6d76000640 1 -- 172.21.15.144:0/3854941162 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7008be80 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d76000640 1 -- 172.21.15.144:0/3854941162 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f6d700ab190 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d780055b0 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d76000640 1 -- 172.21.15.144:0/3854941162 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d700e1f00 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d959bd740 10 client.25910 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b72e665240 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d76000640 1 -- 172.21.15.144:0/3854941162 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d780055b0 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.950+0000 7f6d76000640 1 -- 172.21.15.144:0/3854941162 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6d700b3b50 con 0x55b72e21a390 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d959bd740 10 client.25910 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d959bd740 10 client.25910 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d959bd740 20 client.25910 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d959bd740 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1ceda0 0x55b72e6679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d959bd740 4 client.25910 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:05:06.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 --> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b72e5545e0 con 0x55b72e1ceda0 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.951+0000 7f6d87e00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1ceda0 0x55b72e6679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d87e00640 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1ceda0 0x55b72e6679b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=426af3b06d9ecb99 server_cookie=49f7c3c62ace3bb4 in_seq=0 out_seq=0 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d76000640 10 client.25910 ms_handle_connect on v2:172.21.15.144:6840/1389845259 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d76000640 1 -- 172.21.15.144:0/3854941162 <== mds.1 v2:172.21.15.144:6840/1389845259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b72e5545e0 con 0x55b72e1ceda0 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d76000640 10 client.25910 handle_command_reply: tid=0 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 client.25910 shutdown 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 msgr2=0x55b72e5f1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5f1000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d700e1a70 tx=0x7f6d700053d0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 shutdown_connections 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b72e23b070 0x55b72e5f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:6840/1389845259,v1:172.21.15.144:6841/1389845259] conn(0x55b72e1ceda0 0x55b72e6679b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b72e21a390 0x55b72e5f1000 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 --2- 172.21.15.144:0/3854941162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b72e2121b0 0x55b72e5f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d76000640 0 client.25910 ms_handle_reset on v2:172.21.15.144:6840/1389845259 2024-07-26T11:05:06.958 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 >> 172.21.15.144:0/3854941162 conn(0x55b72e22f460 msgr2=0x55b72e5dccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:06.958 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 shutdown_connections 2024-07-26T11:05:06.958 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.952+0000 7f6d959bd740 1 -- 172.21.15.144:0/3854941162 wait complete. 2024-07-26T11:05:06.958 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d959bd740 20 client.25910 trim_cache size 0 max 16384 2024-07-26T11:05:06.958 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6d6c0b3d40 msgr2=0x7f6d6c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6d6c0b3d40 0x7f6d6c0b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d8002f500 tx=0x7f6d8002f490 comp rx=0 tx=0).stop 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 msgr2=0x7f6d881a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d881a0890 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e9c0 tx=0x7f6d7000ee80 comp rx=0 tx=0).stop 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 shutdown_connections 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f6d6c0b3d40 0x7f6d6c0b6250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d88113fa0 0x7f6d881a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d88077cc0 0x7f6d881a0890 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 --2- 172.21.15.144:0/4064230665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d88077370 0x7f6d881a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.953+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 >> 172.21.15.144:0/4064230665 conn(0x7f6d881010b0 msgr2=0x7f6d88115fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.954+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 shutdown_connections 2024-07-26T11:05:06.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:06.954+0000 7f6d8d800640 1 -- 172.21.15.144:0/4064230665 wait complete. 2024-07-26T11:05:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:07 smithi160 ceph-mon[36715]: pgmap v540: 129 pgs: 129 active+clean; 2.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s rd, 60 MiB/s wr, 355 op/s 2024-07-26T11:05:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:07 smithi183 ceph-mon[37273]: pgmap v540: 129 pgs: 129 active+clean; 2.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s rd, 60 MiB/s wr, 355 op/s 2024-07-26T11:05:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:07 smithi144 ceph-mon[32129]: pgmap v540: 129 pgs: 129 active+clean; 2.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s rd, 60 MiB/s wr, 355 op/s 2024-07-26T11:05:07.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:05:08.287 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:09 smithi144 ceph-mon[32129]: pgmap v541: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 49 MiB/s wr, 336 op/s 2024-07-26T11:05:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:09 smithi160 ceph-mon[36715]: pgmap v541: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 49 MiB/s wr, 336 op/s 2024-07-26T11:05:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:09 smithi183 ceph-mon[37273]: pgmap v541: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 49 MiB/s wr, 336 op/s 2024-07-26T11:05:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 -- 172.21.15.144:0/3987406925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef810b980 msgr2=0x7f7ef81121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 --2- 172.21.15.144:0/3987406925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef810b980 0x7f7ef81121b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009ca0 tx=0x7f7ef002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 -- 172.21.15.144:0/3987406925 shutdown_connections 2024-07-26T11:05:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 --2- 172.21.15.144:0/3987406925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef810b980 0x7f7ef81121b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 --2- 172.21.15.144:0/3987406925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b030 0x7f7ef810b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 --2- 172.21.15.144:0/3987406925 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ef8105700 0x7f7ef8105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 -- 172.21.15.144:0/3987406925 >> 172.21.15.144:0/3987406925 conn(0x7f7ef8101120 msgr2=0x7f7ef8103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 -- 172.21.15.144:0/3987406925 shutdown_connections 2024-07-26T11:05:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.881+0000 7f7eff000640 1 -- 172.21.15.144:0/3987406925 wait complete. 2024-07-26T11:05:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.882+0000 7f7eff000640 1 Processor -- start 2024-07-26T11:05:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 -- start start 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 0x7f7ef819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ef810b030 0x7f7ef819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b980 0x7f7ef81a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8115630 con 0x7f7ef810b980 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81154b0 con 0x7f7ef810b030 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.883+0000 7f7eff000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81157b0 con 0x7f7ef8105700 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 0x7f7ef819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 0x7f7ef819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54180/0 (socket says 172.21.15.144:54180) 2024-07-26T11:05:09.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 -- 172.21.15.144:0/2095242129 learned_addr learned my addr 172.21.15.144:0/2095242129 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7400640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ef810b030 0x7f7ef819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7efd000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b980 0x7f7ef81a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 -- 172.21.15.144:0/2095242129 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ef810b030 msgr2=0x7f7ef819c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ef810b030 0x7f7ef819c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 -- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b980 msgr2=0x7f7ef81a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b980 0x7f7ef81a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 -- 172.21.15.144:0/2095242129 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef81a1f40 con 0x7f7ef8105700 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7efd000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b980 0x7f7ef81a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:05:09.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.884+0000 7f7ef7e00640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 0x7f7ef819c010 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9f0 tx=0x7f7ee800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.885+0000 7f7ef4c00640 1 -- 172.21.15.144:0/2095242129 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800cd80 con 0x7f7ef8105700 2024-07-26T11:05:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.885+0000 7f7ef4c00640 1 -- 172.21.15.144:0/2095242129 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee80108e0 con 0x7f7ef8105700 2024-07-26T11:05:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.885+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef81a2230 con 0x7f7ef8105700 2024-07-26T11:05:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.885+0000 7f7ef4c00640 1 -- 172.21.15.144:0/2095242129 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8018980 con 0x7f7ef8105700 2024-07-26T11:05:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.885+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef81a89c0 con 0x7f7ef8105700 2024-07-26T11:05:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.888+0000 7f7f06017740 1 Processor -- start 2024-07-26T11:05:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.888+0000 7f7f06017740 1 -- start start 2024-07-26T11:05:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7f06017740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 0x55b1eb92fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7f06017740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb5632f0 0x55b1eb92ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7f06017740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb56b4d0 0x55b1eb9304c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7f06017740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b1eb8a4270 con 0x55b1eb5632f0 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7f06017740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b1eb893770 con 0x55b1eb56b4d0 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7f06017740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b1eb8977b0 con 0x55b1eb58c1b0 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7ef4c00640 1 -- 172.21.15.144:0/2095242129 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7ee8018b20 con 0x7f7ef8105700 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7efd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 0x55b1eb92fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7efd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 0x55b1eb92fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54194/0 (socket says 172.21.15.144:54194) 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7efd000640 1 -- 172.21.15.144:0/3561048691 learned_addr learned my addr 172.21.15.144:0/3561048691 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7ef7400640 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb56b4d0 0x55b1eb9304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.889+0000 7f7ef7e00640 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb5632f0 0x55b1eb92ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7efd000640 1 -- 172.21.15.144:0/3561048691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb56b4d0 msgr2=0x55b1eb9304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7efd000640 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb56b4d0 0x55b1eb9304c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7efd000640 1 -- 172.21.15.144:0/3561048691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb5632f0 msgr2=0x55b1eb92ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7efd000640 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb5632f0 0x55b1eb92ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7efd000640 1 -- 172.21.15.144:0/3561048691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1eb5f7790 con 0x55b1eb58c1b0 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7efd000640 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 0x55b1eb92fa40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00094a0 tx=0x7f7ef0009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d690bdcc170eb4c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7eed600640 1 -- 172.21.15.144:0/3561048691 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0031db0 con 0x55b1eb58c1b0 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7eed600640 1 -- 172.21.15.144:0/3561048691 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef0040070 con 0x55b1eb58c1b0 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.890+0000 7f7eed600640 1 -- 172.21.15.144:0/3561048691 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00042a0 con 0x55b1eb58c1b0 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.891+0000 7f7ef4c00640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7edc0b3d40 0x7f7edc0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.891+0000 7f7ef4c00640 1 -- 172.21.15.144:0/2095242129 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f7ee80e4820 con 0x7f7ef8105700 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.891+0000 7f7ef7e00640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7edc0b3d40 0x7f7edc0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 -- 172.21.15.144:0/3561048691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 msgr2=0x55b1eb92fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 0x55b1eb92fa40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00094a0 tx=0x7f7ef0009800 comp rx=0 tx=0).stop 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7ef7e00640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7edc0b3d40 0x7f7edc0b6250 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55b1eb92e6c0 tx=0x7f7ee8000bc0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 -- 172.21.15.144:0/3561048691 shutdown_connections 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb56b4d0 0x55b1eb9304c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb5632f0 0x55b1eb92ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 --2- 172.21.15.144:0/3561048691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb58c1b0 0x55b1eb92fa40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 -- 172.21.15.144:0/3561048691 >> 172.21.15.144:0/3561048691 conn(0x55b1eb580300 msgr2=0x55b1eb92d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 -- 172.21.15.144:0/3561048691 shutdown_connections 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 -- 172.21.15.144:0/3561048691 wait complete. 2024-07-26T11:05:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.892+0000 7f7f06017740 1 -- start start 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 Processor -- start 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb93b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb93bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb947480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55b1eb7cdab0 con 0x55b1eb56b4d0 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b1eb830d50 con 0x55b1eb58c1b0 2024-07-26T11:05:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b1eb93e160 con 0x55b1eb5632f0 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7f06017740 10 client.0 _ll_register_callbacks cb 0x55b1eb5a3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7efd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb93b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7ef7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb93bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7efd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb93b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54202/0 (socket says 172.21.15.144:54202) 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.893+0000 7f7efd000640 1 -- 172.21.15.144:0/2584706549 learned_addr learned my addr 172.21.15.144:0/2584706549 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb947480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 -- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 msgr2=0x55b1eb947480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb947480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 -- 172.21.15.144:0/2584706549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 msgr2=0x55b1eb93bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb93bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 -- 172.21.15.144:0/2584706549 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1eb670620 con 0x55b1eb5632f0 2024-07-26T11:05:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7ef7400640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb93bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7efd000640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb93b6b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009620 tx=0x7f7ef003e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d682859ecc3707c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.894+0000 7f7ee2000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7ee2000640 1 -- 172.21.15.144:0/2584706549 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0004440 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef0040070 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7ee2000640 1 -- 172.21.15.144:0/2584706549 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f7ef00045e0 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7ee2000640 1 -- 172.21.15.144:0/2584706549 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7ef00047e0 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 10 client.25900 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1eb94a250 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7ee2000640 1 -- 172.21.15.144:0/2584706549 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ef0040070 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7ee2000640 1 -- 172.21.15.144:0/2584706549 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7ef0053040 con 0x55b1eb5632f0 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 10 client.25900 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 10 client.25900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 20 client.25900 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb54ad80 0x55b1eb94d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 4 client.25900 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.895+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1eb8a61c0 con 0x55b1eb54ad80 2024-07-26T11:05:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.896+0000 7f7ef7400640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb54ad80 0x55b1eb94d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.896+0000 7f7ef7400640 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb54ad80 0x55b1eb94d5b0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc40c83711321b6e server_cookie=23e095514b544483 in_seq=0 out_seq=0 2024-07-26T11:05:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.896+0000 7f7ee2000640 10 client.25900 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.897+0000 7f7ee2000640 1 -- 172.21.15.144:0/2584706549 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b1eb8a61c0 con 0x55b1eb54ad80 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.897+0000 7f7ee2000640 10 client.25900 handle_command_reply: tid=0 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.897+0000 7f7f06017740 1 client.25900 shutdown 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.897+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 msgr2=0x55b1eb93b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.897+0000 7f7f06017740 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb93b6b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009620 tx=0x7f7ef003e040 comp rx=0 tx=0).stop 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 shutdown_connections 2024-07-26T11:05:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb54ad80 0x55b1eb94d5b0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb947480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7ee2000640 0 client.25900 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb93bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 --2- 172.21.15.144:0/2584706549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb93b6b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 >> 172.21.15.144:0/2584706549 conn(0x55b1eb580300 msgr2=0x55b1eb930a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 shutdown_connections 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 1 -- 172.21.15.144:0/2584706549 wait complete. 2024-07-26T11:05:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.898+0000 7f7f06017740 20 client.25900 trim_cache size 0 max 16384 2024-07-26T11:05:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.928+0000 7f7f06017740 1 Processor -- start 2024-07-26T11:05:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 -- start start 2024-07-26T11:05:09.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb935280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb938670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb938bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b5390 con 0x55b1eb56b4d0 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b2c60 con 0x55b1eb58c1b0 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7f06017740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b0ab0 con 0x55b1eb5632f0 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.929+0000 7f7ef7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb938670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb938670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51166/0 (socket says 172.21.15.144:51166) 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 -- 172.21.15.144:0/961042283 learned_addr learned my addr 172.21.15.144:0/961042283 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7efd000640 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb935280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7efd000640 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb938bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 -- 172.21.15.144:0/961042283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 msgr2=0x55b1eb935280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb935280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 -- 172.21.15.144:0/961042283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 msgr2=0x55b1eb938bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb938bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 -- 172.21.15.144:0/961042283 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1eb932850 con 0x55b1eb56b4d0 2024-07-26T11:05:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.930+0000 7f7ef7400640 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb938670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8109a60 tx=0x7f7ee4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d720b7a0ed6fe4b2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.931+0000 7f7eecc00640 1 -- 172.21.15.144:0/961042283 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4010630 con 0x55b1eb56b4d0 2024-07-26T11:05:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.931+0000 7f7eecc00640 1 -- 172.21.15.144:0/961042283 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee4010c80 con 0x55b1eb56b4d0 2024-07-26T11:05:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.931+0000 7f7eecc00640 1 -- 172.21.15.144:0/961042283 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee401ecd0 con 0x55b1eb56b4d0 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 -- 172.21.15.144:0/961042283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 msgr2=0x55b1eb938670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb938670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8109a60 tx=0x7f7ee4005f90 comp rx=0 tx=0).stop 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 -- 172.21.15.144:0/961042283 shutdown_connections 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb938bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb938670 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 --2- 172.21.15.144:0/961042283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb935280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 -- 172.21.15.144:0/961042283 >> 172.21.15.144:0/961042283 conn(0x55b1eb580300 msgr2=0x55b1eb934b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 -- 172.21.15.144:0/961042283 shutdown_connections 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 -- 172.21.15.144:0/961042283 wait complete. 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 -- start start 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.932+0000 7f7f06017740 1 Processor -- start 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb94d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb94d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb94de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b1120 con 0x55b1eb56b4d0 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b2590 con 0x55b1eb58c1b0 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b17f0 con 0x55b1eb5632f0 2024-07-26T11:05:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7f06017740 10 client.0 _ll_register_callbacks cb 0x7f7ee4009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7ef7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb94d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7ef7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb94d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51182/0 (socket says 172.21.15.144:51182) 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7ef7400640 1 -- 172.21.15.144:0/715567503 learned_addr learned my addr 172.21.15.144:0/715567503 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7efd000640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb94d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.933+0000 7f7efd000640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb94de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ef7400640 1 -- 172.21.15.144:0/715567503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 msgr2=0x55b1eb94d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ef7400640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb94d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ef7400640 1 -- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 msgr2=0x55b1eb94de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ef7400640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb94de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ef7400640 1 -- 172.21.15.144:0/715567503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1eb9b3af0 con 0x55b1eb56b4d0 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ef7400640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb94d920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4005eb0 tx=0x7f7ee4010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f183168fe5c3ae server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ee2000640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ee2000640 1 -- 172.21.15.144:0/715567503 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4004230 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7ee2000640 1 -- 172.21.15.144:0/715567503 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f7ee40043d0 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.934+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee4010c80 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7ee2000640 1 -- 172.21.15.144:0/715567503 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7ee4039050 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 10 client.15774 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1eb9b4cb0 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7ee2000640 1 -- 172.21.15.144:0/715567503 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee4010c80 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7ee2000640 1 -- 172.21.15.144:0/715567503 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7ee40053f0 con 0x55b1eb56b4d0 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 10 client.15774 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 20 client.15774 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb5201c0 0x55b1eb9b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 4 client.15774 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:05:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b1eb8a1c60 con 0x55b1eb5201c0 2024-07-26T11:05:09.942 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.935+0000 7f7efd000640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb5201c0 0x55b1eb9b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:09.942 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.936+0000 7f7efd000640 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb5201c0 0x55b1eb9b74a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c9af4c95043f72 server_cookie=a66b9602a8df5c1e in_seq=0 out_seq=0 2024-07-26T11:05:09.942 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:09.936+0000 7f7ee2000640 10 client.15774 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:10.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 -- 172.21.15.144:0/2615745633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c81022a0 msgr2=0x7f02c81026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2615745633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c81022a0 0x7f02c81026b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009900 tx=0x7f02b802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 -- 172.21.15.144:0/2615745633 shutdown_connections 2024-07-26T11:05:10.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2615745633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c8108530 0x7f02c810ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2615745633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02c8107be0 0x7f02c8107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2615745633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c81022a0 0x7f02c81026b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 -- 172.21.15.144:0/2615745633 >> 172.21.15.144:0/2615745633 conn(0x7f02c80fdcf0 msgr2=0x7f02c8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:10.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 -- 172.21.15.144:0/2615745633 shutdown_connections 2024-07-26T11:05:10.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.487+0000 7f02c7e00640 1 -- 172.21.15.144:0/2615745633 wait complete. 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.488+0000 7f02c7e00640 1 Processor -- start 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.488+0000 7f02c7e00640 1 -- start start 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 0x7f02c819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02c8107be0 0x7f02c819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c8108530 0x7f02c81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8113350 con 0x7f02c8108530 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81131d0 con 0x7f02c8107be0 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81134d0 con 0x7f02c81022a0 2024-07-26T11:05:10.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 0x7f02c819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c8108530 0x7f02c81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 0x7f02c819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54234/0 (socket says 172.21.15.144:54234) 2024-07-26T11:05:10.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c6a00640 1 -- 172.21.15.144:0/2630112629 learned_addr learned my addr 172.21.15.144:0/2630112629 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:10.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.489+0000 7f02c6000640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02c8107be0 0x7f02c819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c6a00640 1 -- 172.21.15.144:0/2630112629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02c8107be0 msgr2=0x7f02c819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02c8107be0 0x7f02c819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c6a00640 1 -- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c8108530 msgr2=0x7f02c81a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c8108530 0x7f02c81a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c6a00640 1 -- 172.21.15.144:0/2630112629 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c81a1eb0 con 0x7f02c81022a0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 0x7f02c819bf40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005ab0 tx=0x7f02b8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02bf400640 1 -- 172.21.15.144:0/2630112629 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b803f070 con 0x7f02c81022a0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02bf400640 1 -- 172.21.15.144:0/2630112629 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b8031b60 con 0x7f02c81022a0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02bf400640 1 -- 172.21.15.144:0/2630112629 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8035030 con 0x7f02c81022a0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.490+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c81a2140 con 0x7f02c81022a0 2024-07-26T11:05:10.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.491+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c81a8900 con 0x7f02c81022a0 2024-07-26T11:05:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 Processor -- start 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 -- start start 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02bf400640 1 -- 172.21.15.144:0/2630112629 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f02b8042070 con 0x7f02c81022a0 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b3d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f04b347bd0 con 0x55f04b00b390 2024-07-26T11:05:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f04b3370d0 con 0x55f04b02c070 2024-07-26T11:05:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.494+0000 7f02d444b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f04b33b110 con 0x55f04b0031b0 2024-07-26T11:05:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b3d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b3d1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55534/0 (socket says 172.21.15.144:55534) 2024-07-26T11:05:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 -- 172.21.15.144:0/2882439264 learned_addr learned my addr 172.21.15.144:0/2882439264 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c6000640 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 -- 172.21.15.144:0/2882439264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 msgr2=0x55f04b3d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 -- 172.21.15.144:0/2882439264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 msgr2=0x55f04b3d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.498 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c7400640 1 -- 172.21.15.144:0/2882439264 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f04b097650 con 0x55f04b02c070 2024-07-26T11:05:10.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.495+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:10.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.496+0000 7f02c7400640 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b3d1010 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009fd0 tx=0x7f02c00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9808b49f66f43c19 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:10.499 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.496+0000 7f02b7e00640 1 -- 172.21.15.144:0/2882439264 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0004d20 con 0x55f04b02c070 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.496+0000 7f02b7e00640 1 -- 172.21.15.144:0/2882439264 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c0004050 con 0x55f04b02c070 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.496+0000 7f02b7e00640 1 -- 172.21.15.144:0/2882439264 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0005280 con 0x55f04b02c070 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.496+0000 7f02bf400640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02a80b3dc0 0x7f02a80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02bf400640 1 -- 172.21.15.144:0/2630112629 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f02b8104970 con 0x7f02c81022a0 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02a80b3dc0 0x7f02a80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 -- 172.21.15.144:0/2882439264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 msgr2=0x55f04b3d1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b3d1010 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009fd0 tx=0x7f02c00118d0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 -- 172.21.15.144:0/2882439264 shutdown_connections 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 --2- 172.21.15.144:0/2882439264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b3d1010 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 -- 172.21.15.144:0/2882439264 >> 172.21.15.144:0/2882439264 conn(0x55f04b020460 msgr2=0x55f04b3cef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 -- 172.21.15.144:0/2882439264 shutdown_connections 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02d444b740 1 -- 172.21.15.144:0/2882439264 wait complete. 2024-07-26T11:05:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.497+0000 7f02c6a00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02a80b3dc0 0x7f02a80b62d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55f04b3cfc90 tx=0x7f02b810a480 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.498+0000 7f02d444b740 1 -- start start 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.498+0000 7f02d444b740 1 Processor -- start 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b0031b0 0x55f04b3dc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3dd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f04b19a700 con 0x55f04b00b390 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f04b2d36a0 con 0x55f04b0031b0 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f04b3df7f0 con 0x55f04b02c070 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02d444b740 10 client.0 _ll_register_callbacks cb 0x55f04b2e3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02c6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02c6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3dcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51222/0 (socket says 172.21.15.144:51222) 2024-07-26T11:05:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02c6000640 1 -- 172.21.15.144:0/3634936666 learned_addr learned my addr 172.21.15.144:0/3634936666 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02c7400640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b0031b0 0x55f04b3dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.499+0000 7f02c7400640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02c6000640 1 -- 172.21.15.144:0/3634936666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 msgr2=0x55f04b3dd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02c6000640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3dd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02c6000640 1 -- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b0031b0 msgr2=0x55f04b3dc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02c6000640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b0031b0 0x55f04b3dc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.503 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02c6000640 1 -- 172.21.15.144:0/3634936666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f04b2ea840 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02c6000640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3dcad0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f02b0012700 tx=0x7f02b0012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4c919f86a80c7e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.500+0000 7f02b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02b4c00640 1 -- 172.21.15.144:0/3634936666 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b000f920 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c0004050 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02b4c00640 1 -- 172.21.15.144:0/3634936666 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f02b000fac0 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02b4c00640 1 -- 172.21.15.144:0/3634936666 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f02b000fcc0 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02d444b740 10 client.15786 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f04b3eb4a0 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02b4c00640 1 -- 172.21.15.144:0/3634936666 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02c0004050 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.501+0000 7f02b4c00640 1 -- 172.21.15.144:0/3634936666 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f02b001c420 con 0x55f04b00b390 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02d444b740 10 client.15786 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02d444b740 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:10.505 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02d444b740 20 client.15786 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02d444b740 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afea9f0 0x55f04b3ee970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02d444b740 4 client.15786 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f04b349b20 con 0x55f04afea9f0 2024-07-26T11:05:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.502+0000 7f02c7400640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afea9f0 0x55f04b3ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.503+0000 7f02c7400640 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afea9f0 0x55f04b3ee970 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8370e5acc42bfbbb server_cookie=d634242419944a9d in_seq=0 out_seq=0 2024-07-26T11:05:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.503+0000 7f02b4c00640 10 client.15786 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.503+0000 7f02b4c00640 1 -- 172.21.15.144:0/3634936666 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f04b349b20 con 0x55f04afea9f0 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.503+0000 7f02b4c00640 10 client.15786 handle_command_reply: tid=0 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.503+0000 7f02d444b740 1 client.15786 shutdown 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 msgr2=0x55f04b3dcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3dcad0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f02b0012700 tx=0x7f02b0012bc0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 shutdown_connections 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afea9f0 0x55f04b3ee970 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3dd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3dcad0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 --2- 172.21.15.144:0/3634936666 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b0031b0 0x55f04b3dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.507 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02b4c00640 0 client.15786 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:10.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 >> 172.21.15.144:0/3634936666 conn(0x55f04b020460 msgr2=0x55f04b3d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:10.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 shutdown_connections 2024-07-26T11:05:10.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 1 -- 172.21.15.144:0/3634936666 wait complete. 2024-07-26T11:05:10.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.504+0000 7f02d444b740 20 client.15786 trim_cache size 0 max 16384 2024-07-26T11:05:10.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.518+0000 7f02d444b740 1 Processor -- start 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.518+0000 7f02d444b740 1 -- start start 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02d444b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b0031b0 0x55f04b3d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02d444b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b00b390 0x55f04b3dcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02d444b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3f4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02d444b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b5160 con 0x55f04b0031b0 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02d444b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b2a30 con 0x55f04b00b390 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02d444b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b0880 con 0x55f04b02c070 2024-07-26T11:05:10.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02c7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b0031b0 0x55f04b3d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02c7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b0031b0 0x55f04b3d2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51224/0 (socket says 172.21.15.144:51224) 2024-07-26T11:05:10.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02c7400640 1 -- 172.21.15.144:0/820838319 learned_addr learned my addr 172.21.15.144:0/820838319 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:10.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02c6000640 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b00b390 0x55f04b3dcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.519+0000 7f02c7400640 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02c6000640 1 -- 172.21.15.144:0/820838319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 msgr2=0x55f04b3f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02c6000640 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02c6000640 1 -- 172.21.15.144:0/820838319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b0031b0 msgr2=0x55f04b3d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02c6000640 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b0031b0 0x55f04b3d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02c6000640 1 -- 172.21.15.144:0/820838319 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f04b3cfcc0 con 0x55f04b00b390 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02c6000640 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b00b390 0x55f04b3dcc10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009e00 tx=0x7f02b000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1eeb64c164d6044 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02b7400640 1 -- 172.21.15.144:0/820838319 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b000f920 con 0x55f04b00b390 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02b7400640 1 -- 172.21.15.144:0/820838319 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b000fac0 con 0x55f04b00b390 2024-07-26T11:05:10.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.520+0000 7f02b7400640 1 -- 172.21.15.144:0/820838319 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b00035f0 con 0x55f04b00b390 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 -- 172.21.15.144:0/820838319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b00b390 msgr2=0x55f04b3dcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b00b390 0x55f04b3dcc10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009e00 tx=0x7f02b000f790 comp rx=0 tx=0).stop 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 -- 172.21.15.144:0/820838319 shutdown_connections 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b02c070 0x55f04b3f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b00b390 0x55f04b3dcc10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 --2- 172.21.15.144:0/820838319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b0031b0 0x55f04b3d2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 -- 172.21.15.144:0/820838319 >> 172.21.15.144:0/820838319 conn(0x55f04b020460 msgr2=0x55f04b3dc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:10.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 -- 172.21.15.144:0/820838319 shutdown_connections 2024-07-26T11:05:10.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.521+0000 7f02d444b740 1 -- 172.21.15.144:0/820838319 wait complete. 2024-07-26T11:05:10.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.522+0000 7f02d444b740 1 -- start start 2024-07-26T11:05:10.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.522+0000 7f02d444b740 1 Processor -- start 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3e2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3e27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b455140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b0ef0 con 0x55f04b00b390 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b2360 con 0x55f04b02c070 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b15c0 con 0x55f04b0031b0 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02d444b740 10 client.0 _ll_register_callbacks cb 0x7f02b0009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3e2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3e27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c6000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3e27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51230/0 (socket says 172.21.15.144:51230) 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3e2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54278/0 (socket says 172.21.15.144:54278) 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c6000640 1 -- 172.21.15.144:0/448367127 learned_addr learned my addr 172.21.15.144:0/448367127 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c7400640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b455140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.523+0000 7f02c7400640 1 -- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 msgr2=0x55f04b455140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02c7400640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b455140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02c7400640 1 -- 172.21.15.144:0/448367127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 msgr2=0x55f04b3e27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02c7400640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3e27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02c7400640 1 -- 172.21.15.144:0/448367127 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f04b4557b0 con 0x55f04b0031b0 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02c6000640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3e27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02c7400640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3e2210 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02c001b040 tx=0x7f02c00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33fd46dfa27b93bb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:10.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02b4c00640 1 -- 172.21.15.144:0/448367127 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c003c060 con 0x55f04b0031b0 2024-07-26T11:05:10.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02b4c00640 1 -- 172.21.15.144:0/448367127 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f02c001f040 con 0x55f04b0031b0 2024-07-26T11:05:10.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.524+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02b000fac0 con 0x55f04b0031b0 2024-07-26T11:05:10.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02b4c00640 1 -- 172.21.15.144:0/448367127 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f02c0043050 con 0x55f04b0031b0 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 10 client.25924 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f04b456a30 con 0x55f04b0031b0 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02b4c00640 1 -- 172.21.15.144:0/448367127 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b000fac0 con 0x55f04b0031b0 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02b4c00640 1 -- 172.21.15.144:0/448367127 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f02c000f7b0 con 0x55f04b0031b0 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 10 client.25924 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 10 client.25924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 20 client.25924 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afbfda0 0x55f04b4590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 4 client.25924 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.525+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f04b3455c0 con 0x55f04afbfda0 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.526+0000 7f02c6000640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afbfda0 0x55f04b4590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:10.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.526+0000 7f02c6000640 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afbfda0 0x55f04b4590e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b668ccfc5b4366b server_cookie=b5a7ada8505833bb in_seq=0 out_seq=0 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.526+0000 7f02b4c00640 10 client.25924 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.526+0000 7f02b4c00640 1 -- 172.21.15.144:0/448367127 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f04b3455c0 con 0x55f04afbfda0 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.526+0000 7f02b4c00640 10 client.25924 handle_command_reply: tid=0 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 client.25924 shutdown 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 msgr2=0x55f04b3e2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3e2210 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02c001b040 tx=0x7f02c00118d0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 shutdown_connections 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55f04afbfda0 0x55f04b4590e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f04b02c070 0x55f04b455140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f04b00b390 0x55f04b3e27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 --2- 172.21.15.144:0/448367127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f04b0031b0 0x55f04b3e2210 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 >> 172.21.15.144:0/448367127 conn(0x55f04b020460 msgr2=0x55f04b3d27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:10.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02b4c00640 0 client.25924 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:10.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 shutdown_connections 2024-07-26T11:05:10.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 1 -- 172.21.15.144:0/448367127 wait complete. 2024-07-26T11:05:10.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.527+0000 7f02d444b740 20 client.25924 trim_cache size 0 max 16384 2024-07-26T11:05:10.532 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02a80b3dc0 msgr2=0x7f02a80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02a80b3dc0 0x7f02a80b62d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55f04b3cfc90 tx=0x7f02b810a480 comp rx=0 tx=0).stop 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 msgr2=0x7f02c819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 0x7f02c819bf40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005ab0 tx=0x7f02b8003c10 comp rx=0 tx=0).stop 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 shutdown_connections 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f02a80b3dc0 0x7f02a80b62d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02c8108530 0x7f02c81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f02c8107be0 0x7f02c819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 --2- 172.21.15.144:0/2630112629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02c81022a0 0x7f02c819bf40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 >> 172.21.15.144:0/2630112629 conn(0x7f02c80fdcf0 msgr2=0x7f02c8107280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 shutdown_connections 2024-07-26T11:05:10.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:10.528+0000 7f02c7e00640 1 -- 172.21.15.144:0/2630112629 wait complete. 2024-07-26T11:05:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:11 smithi160 ceph-mon[36715]: pgmap v542: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s rd, 45 MiB/s wr, 260 op/s 2024-07-26T11:05:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:11 smithi183 ceph-mon[37273]: pgmap v542: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s rd, 45 MiB/s wr, 260 op/s 2024-07-26T11:05:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:11 smithi144 ceph-mon[32129]: pgmap v542: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s rd, 45 MiB/s wr, 260 op/s 2024-07-26T11:05:12.445 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:05:12.445 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:05:12.923 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:13.432 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:13 smithi144 ceph-mon[32129]: pgmap v543: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s rd, 37 MiB/s wr, 194 op/s 2024-07-26T11:05:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:13 smithi160 ceph-mon[36715]: pgmap v543: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s rd, 37 MiB/s wr, 194 op/s 2024-07-26T11:05:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:13 smithi183 ceph-mon[37273]: pgmap v543: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s rd, 37 MiB/s wr, 194 op/s 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 -- 172.21.15.144:0/3926051397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c10f330 msgr2=0x7f660c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 --2- 172.21.15.144:0/3926051397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c10f330 0x7f660c10f740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009900 tx=0x7f65f402a410 comp rx=0 tx=0).stop 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 -- 172.21.15.144:0/3926051397 shutdown_connections 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 --2- 172.21.15.144:0/3926051397 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f660c10fc80 0x7f660c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 --2- 172.21.15.144:0/3926051397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c10f330 0x7f660c10f740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 --2- 172.21.15.144:0/3926051397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f660c1099f0 0x7f660c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 -- 172.21.15.144:0/3926051397 >> 172.21.15.144:0/3926051397 conn(0x7f660c078ff0 msgr2=0x7f660c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 -- 172.21.15.144:0/3926051397 shutdown_connections 2024-07-26T11:05:15.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.308+0000 7f6610a00640 1 -- 172.21.15.144:0/3926051397 wait complete. 2024-07-26T11:05:15.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.309+0000 7f6610a00640 1 Processor -- start 2024-07-26T11:05:15.312 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 -- start start 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 0x7f660c1a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f660c10f330 0x7f660c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f660c10fc80 0x7f660c1a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f660c119940 con 0x7f660c1099f0 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1197c0 con 0x7f660c10f330 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f6610a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f660c119ac0 con 0x7f660c10fc80 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f660b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 0x7f660c1a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f660b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 0x7f660c1a0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51252/0 (socket says 172.21.15.144:51252) 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f660b400640 1 -- 172.21.15.144:0/2734250635 learned_addr learned my addr 172.21.15.144:0/2734250635 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f660aa00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f660c10f330 0x7f660c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.310+0000 7f660be00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f660c10fc80 0x7f660c1a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.311+0000 7f660b400640 1 -- 172.21.15.144:0/2734250635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f660c10fc80 msgr2=0x7f660c1a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.311+0000 7f660b400640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f660c10fc80 0x7f660c1a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.311+0000 7f660b400640 1 -- 172.21.15.144:0/2734250635 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f660c10f330 msgr2=0x7f660c1a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.311+0000 7f660b400640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f660c10f330 0x7f660c1a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.311+0000 7f660b400640 1 -- 172.21.15.144:0/2734250635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c1a62c0 con 0x7f660c1099f0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.311+0000 7f660b400640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 0x7f660c1a0350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00eb90 tx=0x7f65fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.312+0000 7f6603e00640 1 -- 172.21.15.144:0/2734250635 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc004d20 con 0x7f660c1099f0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.312+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c1a65b0 con 0x7f660c1099f0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.312+0000 7f6603e00640 1 -- 172.21.15.144:0/2734250635 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65fc00ee90 con 0x7f660c1099f0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.312+0000 7f6603e00640 1 -- 172.21.15.144:0/2734250635 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc010670 con 0x7f660c1099f0 2024-07-26T11:05:15.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.312+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c1acd10 con 0x7f660c1099f0 2024-07-26T11:05:15.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.315+0000 7f6618cc6740 1 Processor -- start 2024-07-26T11:05:15.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.315+0000 7f6618cc6740 1 -- start start 2024-07-26T11:05:15.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6603e00640 1 -- 172.21.15.144:0/2734250635 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f65fc005020 con 0x7f660c1099f0 2024-07-26T11:05:15.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda824ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda824f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda824f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6618cc6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55cda81c5be0 con 0x55cda7e81490 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6618cc6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55cda81b50e0 con 0x55cda7e89670 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f6618cc6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cda81b9120 con 0x55cda7eaa350 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f660b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda824f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f660be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda824ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f660b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda824f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51266/0 (socket says 172.21.15.144:51266) 2024-07-26T11:05:15.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f660b400640 1 -- 172.21.15.144:0/3654972740 learned_addr learned my addr 172.21.15.144:0/3654972740 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.316+0000 7f660be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda824ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:54312/0 (socket says 172.21.15.144:54312) 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660aa00640 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda824f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660be00640 1 -- 172.21.15.144:0/3654972740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 msgr2=0x55cda824f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660be00640 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda824f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660be00640 1 -- 172.21.15.144:0/3654972740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 msgr2=0x55cda824f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660be00640 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda824f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660be00640 1 -- 172.21.15.144:0/3654972740 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda7f15930 con 0x55cda7eaa350 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660aa00640 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda824f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f660be00640 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda824ef50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6604009b80 tx=0x7f6604011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33b707a01bcfcc98 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f65fbe00640 1 -- 172.21.15.144:0/3654972740 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604004030 con 0x55cda7eaa350 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f65fbe00640 1 -- 172.21.15.144:0/3654972740 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6604005020 con 0x55cda7eaa350 2024-07-26T11:05:15.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.317+0000 7f65fbe00640 1 -- 172.21.15.144:0/3654972740 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604004ca0 con 0x55cda7eaa350 2024-07-26T11:05:15.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.318+0000 7f6603e00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f65f00b3d40 0x7f65f00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.318+0000 7f6603e00640 1 -- 172.21.15.144:0/2734250635 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f65fc0e4a00 con 0x7f660c1099f0 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.318+0000 7f660b400640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f65f00b3d40 0x7f65f00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 -- 172.21.15.144:0/3654972740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 msgr2=0x55cda824ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda824ef50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6604009b80 tx=0x7f6604011810 comp rx=0 tx=0).stop 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 -- 172.21.15.144:0/3654972740 shutdown_connections 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda824f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda824f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 --2- 172.21.15.144:0/3654972740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda824ef50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 -- 172.21.15.144:0/3654972740 >> 172.21.15.144:0/3654972740 conn(0x55cda7e9e740 msgr2=0x55cda824ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f660b400640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f65f00b3d40 0x7f65f00b6250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00cba0 tx=0x7f65fc00cf90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 -- 172.21.15.144:0/3654972740 shutdown_connections 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 -- 172.21.15.144:0/3654972740 wait complete. 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.319+0000 7f6618cc6740 1 -- start start 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 Processor -- start 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e81490 0x55cda825a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e89670 0x55cda825abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda8266a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55cda8018890 con 0x55cda7eaa350 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55cda8151320 con 0x55cda7e81490 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cda825d730 con 0x55cda7e89670 2024-07-26T11:05:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.320+0000 7f6618cc6740 10 client.0 _ll_register_callbacks cb 0x55cda817d000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e81490 0x55cda825a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e81490 0x55cda825a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55594/0 (socket says 172.21.15.144:55594) 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660be00640 1 -- 172.21.15.144:0/1387590194 learned_addr learned my addr 172.21.15.144:0/1387590194 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660aa00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e89670 0x55cda825abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660be00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda8266a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660aa00640 1 -- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e81490 msgr2=0x55cda825a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660aa00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e81490 0x55cda825a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660aa00640 1 -- 172.21.15.144:0/1387590194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 msgr2=0x55cda8266a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660aa00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda8266a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.321+0000 7f660aa00640 1 -- 172.21.15.144:0/1387590194 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda825b440 con 0x55cda7e89670 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f660aa00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e89670 0x55cda825abc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f65f402a930 tx=0x7f65f4032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=768a1f8a7212bf5f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f65f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f65f8c00640 1 -- 172.21.15.144:0/1387590194 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4040070 con 0x55cda7e89670 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6604005020 con 0x55cda7e89670 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f65f8c00640 1 -- 172.21.15.144:0/1387590194 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f65f40318b0 con 0x55cda7e89670 2024-07-26T11:05:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f65f8c00640 1 -- 172.21.15.144:0/1387590194 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f65f404d050 con 0x55cda7e89670 2024-07-26T11:05:15.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f6618cc6740 10 client.25936 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:15.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.322+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cda82695c0 con 0x55cda7e89670 2024-07-26T11:05:15.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f65f8c00640 1 -- 172.21.15.144:0/1387590194 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6604005020 con 0x55cda7e89670 2024-07-26T11:05:15.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f65f8c00640 1 -- 172.21.15.144:0/1387590194 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f65f4004750 con 0x55cda7e89670 2024-07-26T11:05:15.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f6618cc6740 10 client.25936 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:15.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f6618cc6740 10 client.25936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f6618cc6740 20 client.25936 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e68cd0 0x55cda826cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f6618cc6740 4 client.25936 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.323+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cda81c7b30 con 0x55cda7e68cd0 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.324+0000 7f660be00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e68cd0 0x55cda826cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.324+0000 7f660be00640 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e68cd0 0x55cda826cb10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a41c7d340455a39 server_cookie=43640a802c4bd4b9 in_seq=0 out_seq=0 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.324+0000 7f65f8c00640 10 client.25936 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f65f8c00640 1 -- 172.21.15.144:0/1387590194 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cda81c7b30 con 0x55cda7e68cd0 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f65f8c00640 10 client.25936 handle_command_reply: tid=0 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 client.25936 shutdown 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e89670 msgr2=0x55cda825abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e89670 0x55cda825abc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f65f402a930 tx=0x7f65f4032d50 comp rx=0 tx=0).stop 2024-07-26T11:05:15.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 shutdown_connections 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e68cd0 0x55cda826cb10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda8266a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e89670 0x55cda825abc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1387590194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e81490 0x55cda825a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f65f8c00640 0 client.25936 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.325+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 >> 172.21.15.144:0/1387590194 conn(0x55cda7e9e740 msgr2=0x55cda82520e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.326+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 shutdown_connections 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.326+0000 7f6618cc6740 1 -- 172.21.15.144:0/1387590194 wait complete. 2024-07-26T11:05:15.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.326+0000 7f6618cc6740 20 client.25936 trim_cache size 0 max 16384 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.346+0000 7f6618cc6740 1 Processor -- start 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.346+0000 7f6618cc6740 1 -- start start 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e81490 0x55cda8255e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda8250940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda825c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f6618cc6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b9600 con 0x55cda7eaa350 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f6618cc6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b6ed0 con 0x55cda7e89670 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f6618cc6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b4d20 con 0x55cda7e81490 2024-07-26T11:05:15.349 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f660be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e81490 0x55cda8255e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.350 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:15 smithi144 ceph-mon[32129]: pgmap v544: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s rd, 32 MiB/s wr, 201 op/s 2024-07-26T11:05:15.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f660aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda8250940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f660aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda8250940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55596/0 (socket says 172.21.15.144:55596) 2024-07-26T11:05:15.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.347+0000 7f660aa00640 1 -- 172.21.15.144:0/1791027254 learned_addr learned my addr 172.21.15.144:0/1791027254 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:15.351 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660be00640 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda825c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660aa00640 1 -- 172.21.15.144:0/1791027254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e81490 msgr2=0x55cda8255e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660aa00640 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e81490 0x55cda8255e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660aa00640 1 -- 172.21.15.144:0/1791027254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 msgr2=0x55cda825c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660aa00640 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda825c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660aa00640 1 -- 172.21.15.144:0/1791027254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda824d380 con 0x55cda7e89670 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.348+0000 7f660aa00640 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda8250940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f65f40057a0 tx=0x7f65f4031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe09481d2e99c7be server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.349+0000 7f65fb400640 1 -- 172.21.15.144:0/1791027254 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4040040 con 0x55cda7e89670 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.349+0000 7f65fb400640 1 -- 172.21.15.144:0/1791027254 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f4056070 con 0x55cda7e89670 2024-07-26T11:05:15.352 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.349+0000 7f65fb400640 1 -- 172.21.15.144:0/1791027254 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4035040 con 0x55cda7e89670 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 -- 172.21.15.144:0/1791027254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 msgr2=0x55cda8250940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda8250940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f65f40057a0 tx=0x7f65f4031400 comp rx=0 tx=0).stop 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 -- 172.21.15.144:0/1791027254 shutdown_connections 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7eaa350 0x55cda825c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda8250940 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1791027254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7e81490 0x55cda8255e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.353 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.350+0000 7f6618cc6740 1 -- 172.21.15.144:0/1791027254 >> 172.21.15.144:0/1791027254 conn(0x55cda7e9e740 msgr2=0x55cda825bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.351+0000 7f6618cc6740 1 -- 172.21.15.144:0/1791027254 shutdown_connections 2024-07-26T11:05:15.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.351+0000 7f6618cc6740 1 -- 172.21.15.144:0/1791027254 wait complete. 2024-07-26T11:05:15.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.351+0000 7f6618cc6740 1 -- start start 2024-07-26T11:05:15.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 Processor -- start 2024-07-26T11:05:15.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda825e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda825ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda825f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b5390 con 0x55cda7e81490 2024-07-26T11:05:15.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b6800 con 0x55cda7e89670 2024-07-26T11:05:15.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.352+0000 7f6618cc6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b5a60 con 0x55cda7eaa350 2024-07-26T11:05:15.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.353+0000 7f6618cc6740 10 client.0 _ll_register_callbacks cb 0x7f65fc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.353+0000 7f660be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda825e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.353+0000 7f660be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda825e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51308/0 (socket says 172.21.15.144:51308) 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.353+0000 7f660be00640 1 -- 172.21.15.144:0/1589741565 learned_addr learned my addr 172.21.15.144:0/1589741565 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.353+0000 7f660be00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda825f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660aa00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda825ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660be00640 1 -- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 msgr2=0x55cda825ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660be00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda825ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660be00640 1 -- 172.21.15.144:0/1589741565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 msgr2=0x55cda825e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660be00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda825e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660be00640 1 -- 172.21.15.144:0/1589741565 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda82d4370 con 0x55cda7eaa350 2024-07-26T11:05:15.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f660be00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda825f3c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6604009b80 tx=0x7f6604000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ffe56f78607852a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:15.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.354+0000 7f65f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f65f8c00640 1 -- 172.21.15.144:0/1589741565 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604004030 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f65f8c00640 1 -- 172.21.15.144:0/1589741565 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f66040041d0 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f4056070 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f65f8c00640 1 -- 172.21.15.144:0/1589741565 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f66040043d0 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f6618cc6740 10 client.25942 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cda82d5690 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.355+0000 7f65f8c00640 1 -- 172.21.15.144:0/1589741565 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f4056070 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f65f8c00640 1 -- 172.21.15.144:0/1589741565 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6604024040 con 0x55cda7eaa350 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f6618cc6740 10 client.25942 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f6618cc6740 10 client.25942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f6618cc6740 20 client.25942 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:15.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e3e080 0x55cda82d7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:15.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f6618cc6740 4 client.25942 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:05:15.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cda81c35d0 con 0x55cda7e3e080 2024-07-26T11:05:15.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.356+0000 7f660aa00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e3e080 0x55cda82d7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:15.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.357+0000 7f65f8c00640 10 client.25942 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:15.359 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.357+0000 7f660aa00640 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e3e080 0x55cda82d7d40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=834e2812731b5f14 server_cookie=4cb5fc194c9cff82 in_seq=0 out_seq=0 2024-07-26T11:05:15.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f65f8c00640 1 -- 172.21.15.144:0/1589741565 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cda81c35d0 con 0x55cda7e3e080 2024-07-26T11:05:15.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f65f8c00640 10 client.25942 handle_command_reply: tid=0 2024-07-26T11:05:15.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 client.25942 shutdown 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 msgr2=0x55cda825f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda825f3c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6604009b80 tx=0x7f6604000bc0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 shutdown_connections 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55cda7e3e080 0x55cda82d7d40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cda7eaa350 0x55cda825f3c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cda7e89670 0x55cda825ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 --2- 172.21.15.144:0/1589741565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55cda7e81490 0x55cda825e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 >> 172.21.15.144:0/1589741565 conn(0x55cda7e9e740 msgr2=0x55cda8250710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f65f8c00640 0 client.25942 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.368+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 shutdown_connections 2024-07-26T11:05:15.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.369+0000 7f6618cc6740 1 -- 172.21.15.144:0/1589741565 wait complete. 2024-07-26T11:05:15.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.369+0000 7f6618cc6740 20 client.25942 trim_cache size 0 max 16384 2024-07-26T11:05:15.372 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:05:15.372 INFO:teuthology.orchestra.run.smithi144.stdout: "return_code": 0, 2024-07-26T11:05:15.372 INFO:teuthology.orchestra.run.smithi144.stdout: "scrub_tag": "5707e68b-2051-495c-91d1-212436abd999", 2024-07-26T11:05:15.372 INFO:teuthology.orchestra.run.smithi144.stdout: "mode": "asynchronous" 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f65f00b3d40 msgr2=0x7f65f00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f65f00b3d40 0x7f65f00b6250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00cba0 tx=0x7f65fc00cf90 comp rx=0 tx=0).stop 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 msgr2=0x7f660c1a0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 0x7f660c1a0350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00eb90 tx=0x7f65fc00c6a0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 shutdown_connections 2024-07-26T11:05:15.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f65f00b3d40 0x7f65f00b6250 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f660c10fc80 0x7f660c1a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f660c10f330 0x7f660c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 --2- 172.21.15.144:0/2734250635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f660c1099f0 0x7f660c1a0350 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 >> 172.21.15.144:0/2734250635 conn(0x7f660c078ff0 msgr2=0x7f660c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 shutdown_connections 2024-07-26T11:05:15.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:15.370+0000 7f6610a00640 1 -- 172.21.15.144:0/2734250635 wait complete. 2024-07-26T11:05:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:15 smithi160 ceph-mon[36715]: pgmap v544: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s rd, 32 MiB/s wr, 201 op/s 2024-07-26T11:05:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.611+0000 7f7ee2000640 1 -- 172.21.15.144:0/715567503 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b1eb8a1c60 con 0x55b1eb5201c0 2024-07-26T11:05:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.611+0000 7f7ee2000640 10 client.15774 handle_command_reply: tid=0 2024-07-26T11:05:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.611+0000 7f7f06017740 1 client.15774 shutdown 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 msgr2=0x55b1eb94d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb94d920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4005eb0 tx=0x7f7ee4010f00 comp rx=0 tx=0).stop 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 shutdown_connections 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x55b1eb5201c0 0x55b1eb9b74a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b1eb58c1b0 0x55b1eb94de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b1eb56b4d0 0x55b1eb94d920 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 --2- 172.21.15.144:0/715567503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b1eb5632f0 0x55b1eb94d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 >> 172.21.15.144:0/715567503 conn(0x55b1eb580300 msgr2=0x55b1eb93ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.612+0000 7f7ee2000640 0 client.15774 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.613+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 shutdown_connections 2024-07-26T11:05:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.613+0000 7f7f06017740 1 -- 172.21.15.144:0/715567503 wait complete. 2024-07-26T11:05:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.613+0000 7f7f06017740 20 client.15774 trim_cache size 0 max 16384 2024-07-26T11:05:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:05:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:05:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:05:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:05:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7edc0b3d40 msgr2=0x7f7edc0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7edc0b3d40 0x7f7edc0b6250 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55b1eb92e6c0 tx=0x7f7ee8000bc0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 msgr2=0x7f7ef819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 0x7f7ef819c010 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9f0 tx=0x7f7ee800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 shutdown_connections 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7edc0b3d40 0x7f7edc0b6250 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef810b980 0x7f7ef81a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ef810b030 0x7f7ef819c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 --2- 172.21.15.144:0/2095242129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ef8105700 0x7f7ef819c010 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9f0 tx=0x7f7ee800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 >> 172.21.15.144:0/2095242129 conn(0x7f7ef8101120 msgr2=0x7f7ef8103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.615+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 shutdown_connections 2024-07-26T11:05:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:15.616+0000 7f7eff000640 1 -- 172.21.15.144:0/2095242129 wait complete. 2024-07-26T11:05:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:15 smithi183 ceph-mon[37273]: pgmap v544: 129 pgs: 129 active+clean; 2.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s rd, 32 MiB/s wr, 201 op/s 2024-07-26T11:05:15.636 INFO:tasks.workunit.client.0.smithi144.stdout:7/957: mkdir d0/d14f 0 2024-07-26T11:05:15.655 INFO:tasks.workunit.client.0.smithi144.stdout:7/958: write d0/d6/d10c/f11b [601183,40954] 0 2024-07-26T11:05:15.656 INFO:tasks.workunit.client.0.smithi144.stdout:7/959: dread - d0/dd0/dbf/d2f/d53/fcf zero size 2024-07-26T11:05:15.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/947: sync 2024-07-26T11:05:15.666 INFO:tasks.workunit.client.0.smithi144.stdout:8/958: sync 2024-07-26T11:05:15.666 INFO:tasks.workunit.client.0.smithi144.stdout:3/732: sync 2024-07-26T11:05:15.666 INFO:tasks.workunit.client.0.smithi144.stdout:4/983: sync 2024-07-26T11:05:15.666 INFO:tasks.workunit.client.0.smithi144.stdout:5/948: chown d1/d18/da4/db3/fb8 23739 1 2024-07-26T11:05:15.669 INFO:tasks.workunit.client.0.smithi144.stdout:2/931: sync 2024-07-26T11:05:15.669 INFO:tasks.workunit.client.0.smithi144.stdout:2/932: dread - d0/d6/d34/fca zero size 2024-07-26T11:05:15.670 INFO:tasks.workunit.client.0.smithi144.stdout:3/733: mkdir d1/d28/d54/d26/d37/da1/de1 0 2024-07-26T11:05:15.670 INFO:tasks.workunit.client.0.smithi144.stdout:3/734: chown d1/d28/d54/d26/f7c 1991642589 1 2024-07-26T11:05:15.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/949: symlink d1/d18/d24/d35/dd7/d12c/l14e 0 2024-07-26T11:05:15.670 INFO:tasks.workunit.client.0.smithi144.stdout:5/950: mkdir d1/d18/d24/d35/d14f 0 2024-07-26T11:05:15.671 INFO:tasks.workunit.client.0.smithi144.stdout:3/735: rename d1/l9 to d1/d28/d54/d26/d64/d68/db5/le2 0 2024-07-26T11:05:15.672 INFO:tasks.workunit.client.0.smithi144.stdout:5/951: truncate d1/d18/d24/d27/d71/f80 2444451 0 2024-07-26T11:05:15.672 INFO:tasks.workunit.client.0.smithi144.stdout:5/952: stat d1/d18/d24/d35/dd7/d12c/l105 0 2024-07-26T11:05:15.673 INFO:tasks.workunit.client.0.smithi144.stdout:3/736: link d1/f34 d1/d28/d54/d26/d30/d4d/fe3 0 2024-07-26T11:05:15.674 INFO:tasks.workunit.client.0.smithi144.stdout:5/953: link d1/d18/dc4/ded/dc8/l138 d1/d18/dc4/ded/l150 0 2024-07-26T11:05:15.674 INFO:tasks.workunit.client.0.smithi144.stdout:5/954: write d1/d9/d44/f4e [2335892,52321] 0 2024-07-26T11:05:15.675 INFO:tasks.workunit.client.0.smithi144.stdout:3/737: link d1/d5/d97/d60/fb8 d1/d28/d54/d26/d30/d79/fe4 0 2024-07-26T11:05:15.677 INFO:tasks.workunit.client.0.smithi144.stdout:5/955: mkdir d1/d18/dc4/ded/d149/d151 0 2024-07-26T11:05:15.677 INFO:tasks.workunit.client.0.smithi144.stdout:5/956: chown d1/d18/dc4/ded 37135757 1 2024-07-26T11:05:15.677 INFO:tasks.workunit.client.0.smithi144.stdout:3/738: mknod d1/d28/d54/d3d/ce5 0 2024-07-26T11:05:15.678 INFO:tasks.workunit.client.0.smithi144.stdout:5/957: symlink d1/d18/dc4/de0/deb/d116/d11b/l152 0 2024-07-26T11:05:15.680 INFO:tasks.workunit.client.0.smithi144.stdout:3/739: mknod d1/d28/ce6 0 2024-07-26T11:05:15.680 INFO:tasks.workunit.client.0.smithi144.stdout:5/958: symlink d1/d18/dc4/ded/l153 0 2024-07-26T11:05:15.680 INFO:tasks.workunit.client.0.smithi144.stdout:5/959: dread - d1/d18/da4/db3/fc9 zero size 2024-07-26T11:05:15.680 INFO:tasks.workunit.client.0.smithi144.stdout:3/740: getdents d1/d28/d54/d3d/d42/dc4 0 2024-07-26T11:05:15.688 INFO:tasks.workunit.client.0.smithi144.stdout:3/741: link d1/d5/d97/d48/f56 d1/d28/d54/d26/d37/da1/de1/fe7 0 2024-07-26T11:05:15.689 INFO:tasks.workunit.client.0.smithi144.stdout:3/742: fdatasync d1/d28/d54/d26/d64/fd6 0 2024-07-26T11:05:15.689 INFO:tasks.workunit.client.0.smithi144.stdout:3/743: truncate d1/d28/d54/d26/d30/d79/fe4 897670 0 2024-07-26T11:05:15.690 INFO:tasks.workunit.client.0.smithi144.stdout:4/984: dread d9/da/f141 [0,4194304] 0 2024-07-26T11:05:15.691 INFO:tasks.workunit.client.0.smithi144.stdout:3/744: symlink d1/d28/d54/d26/d37/da1/le8 0 2024-07-26T11:05:15.692 INFO:tasks.workunit.client.0.smithi144.stdout:4/985: creat d9/d26/d3c/d10c/f147 x:0 0 0 2024-07-26T11:05:15.694 INFO:tasks.workunit.client.0.smithi144.stdout:4/986: rename d9/d7b/f12c to d9/d26/d48/d50/d60/f148 0 2024-07-26T11:05:15.694 INFO:tasks.workunit.client.0.smithi144.stdout:4/987: chown d9/d26/d48/d50/l53 29 1 2024-07-26T11:05:15.703 INFO:tasks.workunit.client.0.smithi144.stdout:7/960: dread d0/dd0/dbf/d2f/f39 [0,4194304] 0 2024-07-26T11:05:15.726 INFO:tasks.workunit.client.0.smithi144.stdout:2/933: write d0/f1 [2390411,33169] 0 2024-07-26T11:05:15.726 INFO:tasks.workunit.client.0.smithi144.stdout:5/960: dwrite d1/d9/d44/d45/d81/dd4/f111 [0,4194304] 0 2024-07-26T11:05:15.726 INFO:tasks.workunit.client.0.smithi144.stdout:5/961: chown d1/d18/d24/f76 7191565 1 2024-07-26T11:05:15.727 INFO:tasks.workunit.client.0.smithi144.stdout:4/988: dread d9/d57/fa4 [0,4194304] 0 2024-07-26T11:05:15.730 INFO:tasks.workunit.client.0.smithi144.stdout:5/962: symlink d1/d18/d24/l154 0 2024-07-26T11:05:15.730 INFO:tasks.workunit.client.0.smithi144.stdout:5/963: fsync d1/d9/d5b/d4d/f6c 0 2024-07-26T11:05:15.730 INFO:tasks.workunit.client.0.smithi144.stdout:4/989: creat d9/da/da1/d115/f149 x:0 0 0 2024-07-26T11:05:15.732 INFO:tasks.workunit.client.0.smithi144.stdout:5/964: mkdir d1/d18/d24/d35/dd7/d12c/d155 0 2024-07-26T11:05:15.733 INFO:tasks.workunit.client.0.smithi144.stdout:5/965: mkdir d1/d18/d24/d35/dd7/d12c/db2/d156 0 2024-07-26T11:05:15.734 INFO:tasks.workunit.client.0.smithi144.stdout:5/966: readlink d1/d9/d44/d45/d81/dd4/l10c 0 2024-07-26T11:05:15.734 INFO:tasks.workunit.client.0.smithi144.stdout:5/967: write d1/d18/d12b/f14c [300440,83374] 0 2024-07-26T11:05:15.758 INFO:tasks.workunit.client.0.smithi144.stdout:2/934: write d0/d24/d2b/d68/d82/f5a [1022564,95416] 0 2024-07-26T11:05:15.765 INFO:tasks.workunit.client.0.smithi144.stdout:5/968: dread d1/d18/d24/d35/dd7/fff [0,4194304] 0 2024-07-26T11:05:15.765 INFO:tasks.workunit.client.0.smithi144.stdout:5/969: write d1/d18/d24/d35/d10f/dea/f13e [892922,12910] 0 2024-07-26T11:05:15.767 INFO:tasks.workunit.client.0.smithi144.stdout:5/970: mknod d1/d9/d5b/c157 0 2024-07-26T11:05:15.769 INFO:tasks.workunit.client.0.smithi144.stdout:5/971: truncate d1/d9/d44/f41 2955207 0 2024-07-26T11:05:15.770 INFO:tasks.workunit.client.0.smithi144.stdout:5/972: chown d1/d18/dc4/de0/deb/d116/d11b/f124 468684 1 2024-07-26T11:05:15.770 INFO:tasks.workunit.client.0.smithi144.stdout:4/990: write d9/da/f69 [3083986,52904] 0 2024-07-26T11:05:15.777 INFO:tasks.workunit.client.0.smithi144.stdout:7/961: dwrite d0/d71/d95/dc4/f6b [0,4194304] 0 2024-07-26T11:05:15.777 INFO:tasks.workunit.client.0.smithi144.stdout:7/962: write d0/d71/d95/de7/ff8 [998771,112586] 0 2024-07-26T11:05:15.777 INFO:tasks.workunit.client.0.smithi144.stdout:7/963: fdatasync d0/d6/f8 0 2024-07-26T11:05:15.777 INFO:tasks.workunit.client.0.smithi144.stdout:7/964: stat d0/dd0/dbf/d2f/da2/lb0 0 2024-07-26T11:05:15.780 INFO:tasks.workunit.client.0.smithi144.stdout:7/965: read d0/dd0/dbf/d2f/d53/db6/fe6 [220519,22598] 0 2024-07-26T11:05:15.782 INFO:tasks.workunit.client.0.smithi144.stdout:5/973: mknod d1/d18/d24/d35/c158 0 2024-07-26T11:05:15.786 INFO:tasks.workunit.client.0.smithi144.stdout:4/991: rename d9/da/f2d to d9/d26/d48/d50/f14a 0 2024-07-26T11:05:15.790 INFO:tasks.workunit.client.0.smithi144.stdout:7/966: rmdir d0/d71/d95/de7/d136 0 2024-07-26T11:05:15.790 INFO:tasks.workunit.client.0.smithi144.stdout:7/967: write d0/dd0/dbf/d2f/d53/fcf [786733,110285] 0 2024-07-26T11:05:15.793 INFO:tasks.workunit.client.0.smithi144.stdout:4/992: getdents d9/d26/d48/d50 0 2024-07-26T11:05:15.793 INFO:tasks.workunit.client.0.smithi144.stdout:5/974: write d1/d18/d24/d35/dd7/d12c/f13f [935876,27698] 0 2024-07-26T11:05:15.793 INFO:tasks.workunit.client.0.smithi144.stdout:5/975: dread - d1/d5/dae/f12a zero size 2024-07-26T11:05:15.796 INFO:tasks.workunit.client.0.smithi144.stdout:7/968: creat d0/d6/d9/d12/d115/d114/f150 x:0 0 0 2024-07-26T11:05:15.796 INFO:tasks.workunit.client.0.smithi144.stdout:5/976: rename d1/d9/d44/c147 to d1/d18/dc4/ded/d149/c159 0 2024-07-26T11:05:15.796 INFO:tasks.workunit.client.0.smithi144.stdout:5/977: truncate d1/d9/d44/d45/d81/f139 43589 0 2024-07-26T11:05:15.798 INFO:tasks.workunit.client.0.smithi144.stdout:5/978: rename d1/d18/dc4/de0/f117 to d1/d18/dc4/ded/f15a 0 2024-07-26T11:05:15.798 INFO:tasks.workunit.client.0.smithi144.stdout:7/969: rmdir d0/d14f 0 2024-07-26T11:05:15.799 INFO:tasks.workunit.client.0.smithi144.stdout:7/970: fdatasync d0/d6/d10c/f1e 0 2024-07-26T11:05:15.799 INFO:tasks.workunit.client.0.smithi144.stdout:7/971: readlink d0/dd0/dbf/d2f/l32 0 2024-07-26T11:05:15.806 INFO:tasks.workunit.client.0.smithi144.stdout:7/972: rename d0/d6/d10c/c34 to d0/dd0/dbf/d2f/d144/c151 0 2024-07-26T11:05:15.806 INFO:tasks.workunit.client.0.smithi144.stdout:7/973: fsync d0/d6/d10c/f3e 0 2024-07-26T11:05:15.808 INFO:tasks.workunit.client.0.smithi144.stdout:2/935: dwrite d0/d24/d7b/ddc/d28/ff0 [0,4194304] 0 2024-07-26T11:05:15.811 INFO:tasks.workunit.client.0.smithi144.stdout:7/974: mknod d0/d71/d95/dc4/de8/c152 0 2024-07-26T11:05:15.813 INFO:tasks.workunit.client.0.smithi144.stdout:7/975: symlink d0/d71/df5/d139/l153 0 2024-07-26T11:05:15.813 INFO:tasks.workunit.client.0.smithi144.stdout:7/976: fdatasync d0/dd0/dbf/d2f/d68/fb9 0 2024-07-26T11:05:15.814 INFO:tasks.workunit.client.0.smithi144.stdout:7/977: truncate d0/f49 5138625 0 2024-07-26T11:05:15.815 INFO:tasks.workunit.client.0.smithi144.stdout:7/978: truncate d0/d6/d9/f13 86276 0 2024-07-26T11:05:15.817 INFO:tasks.workunit.client.0.smithi144.stdout:7/979: getdents d0/d6/d9/d12 0 2024-07-26T11:05:15.817 INFO:tasks.workunit.client.0.smithi144.stdout:7/980: truncate d0/d71/dc0/de4/f131 164993 0 2024-07-26T11:05:15.847 INFO:tasks.workunit.client.0.smithi144.stdout:7/981: dread d0/d6/d10c/d57/f77 [0,4194304] 0 2024-07-26T11:05:15.851 INFO:tasks.workunit.client.0.smithi144.stdout:7/982: rename d0/dd0/dbf/d2f/d144/c151 to d0/dd0/dbf/d2f/d53/db6/c154 0 2024-07-26T11:05:15.853 INFO:tasks.workunit.client.0.smithi144.stdout:7/983: symlink d0/d6/d9/d12/d115/deb/l155 0 2024-07-26T11:05:15.856 INFO:tasks.workunit.client.0.smithi144.stdout:4/993: dwrite d9/f11 [0,4194304] 0 2024-07-26T11:05:15.857 INFO:tasks.workunit.client.0.smithi144.stdout:7/984: rmdir d0/dd0/dbf/d67/de0/d100 39 2024-07-26T11:05:15.857 INFO:tasks.workunit.client.0.smithi144.stdout:4/994: mknod d9/d26/c14b 0 2024-07-26T11:05:15.857 INFO:tasks.workunit.client.0.smithi144.stdout:4/995: chown d9/d26/d48/d50/c9c 4838815 1 2024-07-26T11:05:15.859 INFO:tasks.workunit.client.0.smithi144.stdout:4/996: link d9/d26/d48/c87 d9/d26/c14c 0 2024-07-26T11:05:15.861 INFO:tasks.workunit.client.0.smithi144.stdout:4/997: rename d9/d7b/dc4/d111/f146 to d9/d26/d3c/d6f/f14d 0 2024-07-26T11:05:15.861 INFO:tasks.workunit.client.0.smithi144.stdout:4/998: fsync d9/d26/d48/dd8/fe2 0 2024-07-26T11:05:15.862 INFO:tasks.workunit.client.0.smithi144.stdout:4/999: rmdir d9/d26/d48/d50/d60 39 2024-07-26T11:05:15.879 INFO:tasks.workunit.client.0.smithi144.stdout:7/985: write d0/d6/d9/d12/d6c/fd8 [3330834,69446] 0 2024-07-26T11:05:15.915 INFO:tasks.workunit.client.0.smithi144.stdout:7/986: dwrite d0/dd0/dbf/d2f/d53/db6/fe6 [0,4194304] 0 2024-07-26T11:05:15.915 INFO:tasks.workunit.client.0.smithi144.stdout:7/987: stat d0/dd0/dbf/f41 0 2024-07-26T11:05:15.916 INFO:tasks.workunit.client.0.smithi144.stdout:7/988: chown d0/dd0/dbf/d67 585790 1 2024-07-26T11:05:15.924 INFO:tasks.workunit.client.0.smithi144.stdout:7/989: mkdir d0/d71/d156 0 2024-07-26T11:05:15.945 INFO:tasks.workunit.client.0.smithi144.stdout:7/990: dread d0/d71/fd4 [0,4194304] 0 2024-07-26T11:05:15.947 INFO:tasks.workunit.client.0.smithi144.stdout:7/991: symlink d0/dd0/dbf/d2f/da2/l157 0 2024-07-26T11:05:15.953 INFO:tasks.workunit.client.0.smithi144.stdout:7/992: dread d0/dd0/dbf/f12d [0,4194304] 0 2024-07-26T11:05:16.007 INFO:tasks.workunit.client.0.smithi144.stdout:7/993: dwrite d0/d6/d9/f10d [0,4194304] 0 2024-07-26T11:05:16.007 INFO:tasks.workunit.client.0.smithi144.stdout:7/994: readlink d0/dd0/dbf/d67/l126 0 2024-07-26T11:05:16.021 INFO:tasks.workunit.client.0.smithi144.stdout:7/995: dread d0/d6/d9/d12/f5d [4194304,4194304] 0 2024-07-26T11:05:16.026 INFO:tasks.workunit.client.0.smithi144.stdout:7/996: rmdir d0/d6/d9 39 2024-07-26T11:05:16.034 INFO:tasks.workunit.client.0.smithi144.stdout:7/997: dread d0/d6/d10c/f11b [0,4194304] 0 2024-07-26T11:05:16.035 INFO:tasks.workunit.client.0.smithi144.stdout:7/998: write d0/dd0/dbf/d2f/d47/fa0 [4812836,109482] 0 2024-07-26T11:05:16.037 INFO:tasks.workunit.client.0.smithi144.stdout:7/999: mkdir d0/dd0/dbf/d2f/da2/d158 0 2024-07-26T11:05:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:16 smithi160 ceph-mon[36715]: scrub queued for path: / 2024-07-26T11:05:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:16 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:16 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:16 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:16 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:16 smithi183 ceph-mon[37273]: scrub queued for path: / 2024-07-26T11:05:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:16 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:16 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:16 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:16 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:16 smithi144 ceph-mon[32129]: scrub queued for path: / 2024-07-26T11:05:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:16 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:16 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:16 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:16 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:17 smithi144 ceph-mon[32129]: pgmap v545: 129 pgs: 129 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 22 MiB/s wr, 325 op/s 2024-07-26T11:05:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:17 smithi160 ceph-mon[36715]: pgmap v545: 129 pgs: 129 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 22 MiB/s wr, 325 op/s 2024-07-26T11:05:17.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:05:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:17 smithi183 ceph-mon[37273]: pgmap v545: 129 pgs: 129 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 22 MiB/s wr, 325 op/s 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:3/745: sync 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:5/979: sync 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:2/936: sync 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:3/746: chown d1/d28/d54/d26/f2d 126977835 1 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:2/937: chown d0/d6/d34 0 1 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:3/747: dread - d1/d28/d54/d26/d64/d68/d74/fd1 zero size 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:3/748: chown d1/d28/d54/d3d/d42/l72 125544 1 2024-07-26T11:05:17.680 INFO:tasks.workunit.client.0.smithi144.stdout:2/938: write d0/d24/d2b/f7e [5132415,116519] 0 2024-07-26T11:05:17.683 INFO:tasks.workunit.client.0.smithi144.stdout:5/980: mknod d1/d18/da4/db3/c15b 0 2024-07-26T11:05:17.683 INFO:tasks.workunit.client.0.smithi144.stdout:3/749: mkdir d1/d28/d54/d26/d37/de9 0 2024-07-26T11:05:17.684 INFO:tasks.workunit.client.0.smithi144.stdout:5/981: getdents d1/d18/d24/d27 0 2024-07-26T11:05:17.685 INFO:tasks.workunit.client.0.smithi144.stdout:5/982: symlink d1/d18/dc4/de0/deb/l15c 0 2024-07-26T11:05:17.686 INFO:tasks.workunit.client.0.smithi144.stdout:5/983: mknod d1/d18/d24/d35/d10f/c15d 0 2024-07-26T11:05:17.705 INFO:tasks.workunit.client.0.smithi144.stdout:2/939: dread d0/d24/d7b/ddc/fa0 [0,4194304] 0 2024-07-26T11:05:17.705 INFO:tasks.workunit.client.0.smithi144.stdout:2/940: truncate d0/d24/d7b/ddc/d28/d2e/fb7 445520 0 2024-07-26T11:05:17.705 INFO:tasks.workunit.client.0.smithi144.stdout:2/941: chown d0/d24/d2b/d68/d82 445503615 1 2024-07-26T11:05:17.709 INFO:tasks.workunit.client.0.smithi144.stdout:2/942: mkdir d0/d24/d7b/ddc/dd9/d119 0 2024-07-26T11:05:17.709 INFO:tasks.workunit.client.0.smithi144.stdout:2/943: creat d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2/f11a x:0 0 0 2024-07-26T11:05:17.710 INFO:tasks.workunit.client.0.smithi144.stdout:2/944: write d0/d24/d2b/f59 [397356,81007] 0 2024-07-26T11:05:17.730 INFO:tasks.workunit.client.0.smithi144.stdout:2/945: read d0/d6/d34/faf [1831786,40854] 0 2024-07-26T11:05:17.747 INFO:tasks.workunit.client.0.smithi144.stdout:2/946: write d0/d24/d48/d63/da3/fbc [4769027,74527] 0 2024-07-26T11:05:17.747 INFO:tasks.workunit.client.0.smithi144.stdout:5/984: dwrite d1/d18/d127/f13c [0,4194304] 0 2024-07-26T11:05:17.754 INFO:tasks.workunit.client.0.smithi144.stdout:3/750: dwrite d1/f1a [0,4194304] 0 2024-07-26T11:05:17.755 INFO:tasks.workunit.client.0.smithi144.stdout:3/751: getdents d1/d28/d54/d26/d37/d91 0 2024-07-26T11:05:17.755 INFO:tasks.workunit.client.0.smithi144.stdout:2/947: read d0/d24/d7b/ddc/d28/d27/ffb [854704,70732] 0 2024-07-26T11:05:17.756 INFO:tasks.workunit.client.0.smithi144.stdout:3/752: mkdir d1/d28/d54/d26/d37/da1/dea 0 2024-07-26T11:05:17.757 INFO:tasks.workunit.client.0.smithi144.stdout:3/753: rmdir d1/d28/d54/d26/d37/da1 39 2024-07-26T11:05:17.760 INFO:tasks.workunit.client.0.smithi144.stdout:3/754: mkdir d1/d5/d5e/dc5/deb 0 2024-07-26T11:05:17.760 INFO:tasks.workunit.client.0.smithi144.stdout:3/755: chown d1/d28/d54/d26/d37/da1/fdb 0 1 2024-07-26T11:05:17.802 INFO:tasks.workunit.client.0.smithi144.stdout:5/985: dwrite d1/f136 [0,4194304] 0 2024-07-26T11:05:17.804 INFO:tasks.workunit.client.0.smithi144.stdout:5/986: link d1/d18/dc4/lde d1/d9/l15e 0 2024-07-26T11:05:17.812 INFO:tasks.workunit.client.0.smithi144.stdout:2/948: dwrite d0/d24/d2b/fbb [0,4194304] 0 2024-07-26T11:05:17.813 INFO:tasks.workunit.client.0.smithi144.stdout:2/949: dread - d0/d24/d7b/fb1 zero size 2024-07-26T11:05:17.820 INFO:tasks.workunit.client.0.smithi144.stdout:2/950: creat d0/d24/d7b/ddc/d28/d2e/d5d/f11b x:0 0 0 2024-07-26T11:05:17.850 INFO:tasks.workunit.client.0.smithi144.stdout:5/987: dwrite d1/d9/d44/d45/d81/dd4/f111 [0,4194304] 0 2024-07-26T11:05:17.850 INFO:tasks.workunit.client.0.smithi144.stdout:5/988: chown d1/d18/dc4/ded/dc8 2904672 1 2024-07-26T11:05:17.852 INFO:tasks.workunit.client.0.smithi144.stdout:5/989: rename d1/d18/d24/d27 to d1/d5/d15f 0 2024-07-26T11:05:17.852 INFO:tasks.workunit.client.0.smithi144.stdout:2/951: dwrite d0/d24/d2b/d68/f10f [0,4194304] 0 2024-07-26T11:05:17.853 INFO:tasks.workunit.client.0.smithi144.stdout:5/990: mknod d1/d5/dae/c160 0 2024-07-26T11:05:17.858 INFO:tasks.workunit.client.0.smithi144.stdout:2/952: creat d0/d24/d7b/ddc/d28/d2e/d5d/d61/f11c x:0 0 0 2024-07-26T11:05:17.859 INFO:tasks.workunit.client.0.smithi144.stdout:5/991: dread d1/d18/d24/f76 [0,4194304] 0 2024-07-26T11:05:17.899 INFO:tasks.workunit.client.0.smithi144.stdout:5/992: dwrite d1/d5/f6d [4194304,4194304] 0 2024-07-26T11:05:17.899 INFO:tasks.workunit.client.0.smithi144.stdout:5/993: fsync d1/d5/dae/f12a 0 2024-07-26T11:05:17.906 INFO:tasks.workunit.client.0.smithi144.stdout:5/994: read d1/d9/d5b/d4d/fc0 [91498,94978] 0 2024-07-26T11:05:17.906 INFO:tasks.workunit.client.0.smithi144.stdout:5/995: dread - d1/d18/d24/f11c zero size 2024-07-26T11:05:17.927 INFO:tasks.workunit.client.0.smithi144.stdout:5/996: dread d1/d18/d24/d35/dd7/fff [0,4194304] 0 2024-07-26T11:05:17.929 INFO:tasks.workunit.client.0.smithi144.stdout:5/997: rename d1/d5/d15f/d71 to d1/d18/dc4/ded/d119/d161 0 2024-07-26T11:05:17.933 INFO:tasks.workunit.client.0.smithi144.stdout:5/998: link d1/c47 d1/d18/c162 0 2024-07-26T11:05:17.935 INFO:tasks.workunit.client.0.smithi144.stdout:5/999: link d1/d18/dc4/de0/f14b d1/d18/dc4/de0/deb/d116/d11b/f163 0 2024-07-26T11:05:18.130 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:18.436 INFO:tasks.workunit.client.0.smithi144.stdout:2/953: sync 2024-07-26T11:05:18.436 INFO:tasks.workunit.client.0.smithi144.stdout:3/756: sync 2024-07-26T11:05:18.438 INFO:tasks.workunit.client.0.smithi144.stdout:2/954: symlink d0/d24/d7b/ddc/d28/da8/d113/l11d 0 2024-07-26T11:05:18.440 INFO:tasks.workunit.client.0.smithi144.stdout:3/757: creat d1/d5/d97/d48/fec x:0 0 0 2024-07-26T11:05:18.442 INFO:tasks.workunit.client.0.smithi144.stdout:2/955: rename d0/d24/f100 to d0/f11e 0 2024-07-26T11:05:18.443 INFO:tasks.workunit.client.0.smithi144.stdout:3/758: mkdir d1/d28/d54/d26/d64/ded 0 2024-07-26T11:05:18.444 INFO:tasks.workunit.client.0.smithi144.stdout:2/956: rmdir d0/d6/deb 39 2024-07-26T11:05:18.446 INFO:tasks.workunit.client.0.smithi144.stdout:3/759: creat d1/d28/d54/d3d/fee x:0 0 0 2024-07-26T11:05:18.446 INFO:tasks.workunit.client.0.smithi144.stdout:3/760: chown d1/d28/d54/d26/d37/d91/ld3 145921 1 2024-07-26T11:05:18.446 INFO:tasks.workunit.client.0.smithi144.stdout:3/761: truncate d1/d28/fc2 673820 0 2024-07-26T11:05:18.446 INFO:tasks.workunit.client.0.smithi144.stdout:2/957: rmdir d0/d6/d9/d44 39 2024-07-26T11:05:18.448 INFO:tasks.workunit.client.0.smithi144.stdout:3/762: creat d1/d5/d5e/fef x:0 0 0 2024-07-26T11:05:18.449 INFO:tasks.workunit.client.0.smithi144.stdout:2/958: creat d0/d24/d7b/ddc/d28/d2e/f11f x:0 0 0 2024-07-26T11:05:18.450 INFO:tasks.workunit.client.0.smithi144.stdout:3/763: creat d1/d28/d54/d26/d64/d89/ff0 x:0 0 0 2024-07-26T11:05:18.451 INFO:tasks.workunit.client.0.smithi144.stdout:2/959: stat d0/d24/d48/d63/l65 0 2024-07-26T11:05:18.453 INFO:tasks.workunit.client.0.smithi144.stdout:2/960: unlink d0/f3d 0 2024-07-26T11:05:18.482 INFO:tasks.workunit.client.0.smithi144.stdout:2/961: dread d0/f11e [0,4194304] 0 2024-07-26T11:05:18.496 INFO:tasks.workunit.client.0.smithi144.stdout:3/764: dread d1/d28/d54/d26/d64/dd5/fd8 [0,4194304] 0 2024-07-26T11:05:18.496 INFO:tasks.workunit.client.0.smithi144.stdout:3/765: chown d1/d28/d54/d26/d64/d68/db5 121989146 1 2024-07-26T11:05:18.507 INFO:tasks.workunit.client.0.smithi144.stdout:3/766: symlink d1/d5/d5e/lf1 0 2024-07-26T11:05:18.507 INFO:tasks.workunit.client.0.smithi144.stdout:3/767: chown d1/f77 251020 1 2024-07-26T11:05:18.510 INFO:tasks.workunit.client.0.smithi144.stdout:3/768: getdents d1/d28/d54/d26/d64/d68 0 2024-07-26T11:05:18.514 INFO:tasks.workunit.client.0.smithi144.stdout:3/769: mknod d1/d28/d54/d26/d37/da1/de1/cf2 0 2024-07-26T11:05:18.514 INFO:tasks.workunit.client.0.smithi144.stdout:3/770: write d1/d5/d97/da9/fb1 [647974,130861] 0 2024-07-26T11:05:18.519 INFO:tasks.workunit.client.0.smithi144.stdout:3/771: truncate d1/d28/d54/d26/f7c 3527001 0 2024-07-26T11:05:18.519 INFO:tasks.workunit.client.0.smithi144.stdout:3/772: chown d1/d5/d97/d48/c8b 7786 1 2024-07-26T11:05:18.519 INFO:tasks.workunit.client.0.smithi144.stdout:3/773: chown d1/d5/d97/d60/l93 24837 1 2024-07-26T11:05:18.521 INFO:tasks.workunit.client.0.smithi144.stdout:3/774: fdatasync d1/d28/d54/d26/d30/d4d/f55 0 2024-07-26T11:05:18.534 INFO:tasks.workunit.client.0.smithi144.stdout:3/775: creat d1/d28/d54/d26/d64/da4/ff3 x:0 0 0 2024-07-26T11:05:18.536 INFO:tasks.workunit.client.0.smithi144.stdout:3/776: mkdir d1/d28/d54/d61/df4 0 2024-07-26T11:05:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:18 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:18 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:18.564 INFO:tasks.workunit.client.0.smithi144.stdout:2/962: dwrite d0/d24/d2b/fbb [4194304,4194304] 0 2024-07-26T11:05:18.614 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:18 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:18.614 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:18 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:18.626 INFO:tasks.workunit.client.0.smithi144.stdout:3/777: write d1/d28/d54/d26/d37/f58 [3442062,104513] 0 2024-07-26T11:05:18.626 INFO:tasks.workunit.client.0.smithi144.stdout:3/778: fsync d1/d28/d54/d26/d37/da1/de1/fe7 0 2024-07-26T11:05:18.632 INFO:tasks.workunit.client.0.smithi144.stdout:3/779: write d1/d28/d54/d26/d30/d78/fd2 [837111,51348] 0 2024-07-26T11:05:18.634 INFO:tasks.workunit.client.0.smithi144.stdout:3/780: rename d1/d28/fc2 to d1/d28/d54/d61/db9/dc9/ff5 0 2024-07-26T11:05:18.634 INFO:tasks.workunit.client.0.smithi144.stdout:3/781: truncate d1/d28/d54/d3d/d42/fd4 178421 0 2024-07-26T11:05:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:18 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:18 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:18.640 INFO:tasks.workunit.client.0.smithi144.stdout:2/963: dwrite d0/d6/d9/d44/f106 [0,4194304] 0 2024-07-26T11:05:18.640 INFO:tasks.workunit.client.0.smithi144.stdout:2/964: getdents d0/db8/df4 0 2024-07-26T11:05:18.697 INFO:tasks.workunit.client.0.smithi144.stdout:2/965: dwrite d0/d24/d7b/ddc/d28/d2e/d5d/d6b/fb5 [0,4194304] 0 2024-07-26T11:05:18.698 INFO:tasks.workunit.client.0.smithi144.stdout:2/966: mknod d0/d24/d7b/ddc/dd9/d119/c120 0 2024-07-26T11:05:18.700 INFO:tasks.workunit.client.0.smithi144.stdout:2/967: mkdir d0/d24/d2b/d68/d82/d121 0 2024-07-26T11:05:18.703 INFO:tasks.workunit.client.0.smithi144.stdout:2/968: mkdir d0/d24/d7b/ddc/d28/d2e/d5d/d61/d117/d122 0 2024-07-26T11:05:18.704 INFO:tasks.workunit.client.0.smithi144.stdout:2/969: mkdir d0/d24/d7b/ddc/d5f/d123 0 2024-07-26T11:05:18.708 INFO:tasks.workunit.client.0.smithi144.stdout:2/970: mknod d0/d6/d9/d44/dc3/c124 0 2024-07-26T11:05:18.713 INFO:tasks.workunit.client.0.smithi144.stdout:3/782: dwrite d1/d28/d54/d26/f7c [0,4194304] 0 2024-07-26T11:05:18.714 INFO:tasks.workunit.client.0.smithi144.stdout:3/783: mkdir d1/d28/d54/df6 0 2024-07-26T11:05:18.723 INFO:tasks.workunit.client.0.smithi144.stdout:3/784: write d1/d28/d54/d26/d30/d79/fe4 [1451825,60279] 0 2024-07-26T11:05:18.771 INFO:tasks.workunit.client.0.smithi144.stdout:2/971: dwrite d0/f91 [0,4194304] 0 2024-07-26T11:05:18.773 INFO:tasks.workunit.client.0.smithi144.stdout:2/972: symlink d0/d24/d7b/ddc/dd9/d119/l125 0 2024-07-26T11:05:18.774 INFO:tasks.workunit.client.0.smithi144.stdout:2/973: link d0/d24/f111 d0/d6/d9/f126 0 2024-07-26T11:05:18.775 INFO:tasks.workunit.client.0.smithi144.stdout:2/974: creat d0/d24/f127 x:0 0 0 2024-07-26T11:05:18.776 INFO:tasks.workunit.client.0.smithi144.stdout:2/975: write d0/d24/d7b/ddc/d28/d2e/d5d/dae/f110 [645339,108256] 0 2024-07-26T11:05:18.776 INFO:tasks.workunit.client.0.smithi144.stdout:2/976: fsync d0/d6/f1d 0 2024-07-26T11:05:18.776 INFO:tasks.workunit.client.0.smithi144.stdout:2/977: stat d0/d24/d7b/ddc/f47 0 2024-07-26T11:05:18.801 INFO:tasks.workunit.client.0.smithi144.stdout:3/785: dwrite d1/d28/d54/d26/f7c [0,4194304] 0 2024-07-26T11:05:18.802 INFO:tasks.workunit.client.0.smithi144.stdout:3/786: symlink d1/d28/d54/d26/d30/d79/lf7 0 2024-07-26T11:05:18.811 INFO:tasks.workunit.client.0.smithi144.stdout:3/787: write d1/d28/d54/d26/d64/d68/d74/fd1 [1040870,120037] 0 2024-07-26T11:05:18.812 INFO:tasks.workunit.client.0.smithi144.stdout:3/788: symlink d1/d28/d6a/lf8 0 2024-07-26T11:05:18.813 INFO:tasks.workunit.client.0.smithi144.stdout:3/789: unlink d1/d5/d5e/c6b 0 2024-07-26T11:05:18.815 INFO:tasks.workunit.client.0.smithi144.stdout:3/790: creat d1/d28/d54/d26/d64/da4/ddd/ff9 x:0 0 0 2024-07-26T11:05:18.815 INFO:tasks.workunit.client.0.smithi144.stdout:3/791: readlink d1/lc 0 2024-07-26T11:05:18.827 INFO:tasks.workunit.client.0.smithi144.stdout:2/978: dwrite d0/d6/d9/d44/f106 [4194304,4194304] 0 2024-07-26T11:05:18.827 INFO:tasks.workunit.client.0.smithi144.stdout:2/979: read d0/d24/d7b/ddc/f47 [468397,85420] 0 2024-07-26T11:05:18.871 INFO:tasks.workunit.client.0.smithi144.stdout:2/980: dread d0/d24/d7b/ddc/d28/d2e/d77/f40 [0,4194304] 0 2024-07-26T11:05:18.871 INFO:tasks.workunit.client.0.smithi144.stdout:2/981: fdatasync d0/fec 0 2024-07-26T11:05:18.872 INFO:tasks.workunit.client.0.smithi144.stdout:2/982: write d0/d24/d7b/ddc/d28/da8/fd8 [10005,68138] 0 2024-07-26T11:05:18.873 INFO:tasks.workunit.client.0.smithi144.stdout:2/983: creat d0/d24/d7b/ddc/d5f/f128 x:0 0 0 2024-07-26T11:05:18.890 INFO:tasks.workunit.client.0.smithi144.stdout:3/792: dwrite d1/d28/d54/d3d/fee [0,4194304] 0 2024-07-26T11:05:18.892 INFO:tasks.workunit.client.0.smithi144.stdout:3/793: symlink d1/d28/d54/d61/db9/lfa 0 2024-07-26T11:05:18.892 INFO:tasks.workunit.client.0.smithi144.stdout:3/794: readlink d1/d28/d54/d26/d30/d79/lcf 0 2024-07-26T11:05:18.909 INFO:tasks.workunit.client.0.smithi144.stdout:2/984: dread d0/d24/d7b/ddc/f96 [0,4194304] 0 2024-07-26T11:05:18.909 INFO:tasks.workunit.client.0.smithi144.stdout:2/985: chown d0/d24/cd4 793802 1 2024-07-26T11:05:18.912 INFO:tasks.workunit.client.0.smithi144.stdout:2/986: getdents d0/d24/d7b/ddc/d28/d2e/d77 0 2024-07-26T11:05:18.913 INFO:tasks.workunit.client.0.smithi144.stdout:2/987: truncate d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2/f10e 212278 0 2024-07-26T11:05:18.913 INFO:tasks.workunit.client.0.smithi144.stdout:2/988: creat d0/d24/d2b/d68/d82/d121/f129 x:0 0 0 2024-07-26T11:05:18.914 INFO:tasks.workunit.client.0.smithi144.stdout:2/989: mkdir d0/d24/d7b/ddc/d28/d2e/d5d/d61/d12a 0 2024-07-26T11:05:18.916 INFO:tasks.workunit.client.0.smithi144.stdout:2/990: symlink d0/d24/d7b/ddc/d28/d2e/d5d/d61/l12b 0 2024-07-26T11:05:18.917 INFO:tasks.workunit.client.0.smithi144.stdout:2/991: unlink d0/d6/d9/f108 0 2024-07-26T11:05:18.919 INFO:tasks.workunit.client.0.smithi144.stdout:2/992: mknod d0/d24/d2b/d68/c12c 0 2024-07-26T11:05:18.922 INFO:tasks.workunit.client.0.smithi144.stdout:2/993: link d0/d24/d7b/ddc/d28/c10b d0/d24/d7b/c12d 0 2024-07-26T11:05:18.923 INFO:tasks.workunit.client.0.smithi144.stdout:2/994: mknod d0/c12e 0 2024-07-26T11:05:18.923 INFO:tasks.workunit.client.0.smithi144.stdout:2/995: chown d0/d24/d7b/ddc/d28/d2e/d5d/d6b/dc2/cdd 1 1 2024-07-26T11:05:18.925 INFO:tasks.workunit.client.0.smithi144.stdout:2/996: symlink d0/d24/d7b/ddc/d28/d2e/d77/d84/l12f 0 2024-07-26T11:05:18.925 INFO:tasks.workunit.client.0.smithi144.stdout:2/997: fsync d0/d24/d7b/ddc/d28/d2e/d77/f40 0 2024-07-26T11:05:18.926 INFO:tasks.workunit.client.0.smithi144.stdout:2/998: write d0/d24/d7b/ddc/d5f/f128 [824203,122106] 0 2024-07-26T11:05:18.926 INFO:tasks.workunit.client.0.smithi144.stdout:3/795: write d1/d28/d54/f85 [2440833,48061] 0 2024-07-26T11:05:18.930 INFO:tasks.workunit.client.0.smithi144.stdout:3/796: creat d1/d28/d54/d26/d64/d89/ffb x:0 0 0 2024-07-26T11:05:18.939 INFO:tasks.workunit.client.0.smithi144.stdout:2/999: dread d0/d24/d7b/ddc/d28/d27/ffb [0,4194304] 0 2024-07-26T11:05:18.995 INFO:tasks.workunit.client.0.smithi144.stdout:3/797: dwrite d1/d28/d54/d26/d64/d89/ff0 [0,4194304] 0 2024-07-26T11:05:19.004 INFO:tasks.workunit.client.0.smithi144.stdout:3/798: dread d1/d28/d54/f4a [0,4194304] 0 2024-07-26T11:05:19.004 INFO:tasks.workunit.client.0.smithi144.stdout:3/799: fsync d1/d28/d54/d26/d64/d68/db5/fbf 0 2024-07-26T11:05:19.005 INFO:tasks.workunit.client.0.smithi144.stdout:3/800: mkdir d1/d28/d54/d3d/dfc 0 2024-07-26T11:05:19.005 INFO:tasks.workunit.client.0.smithi144.stdout:3/801: chown d1/d28/d54/d26/d37/da1/de1/fe7 3 1 2024-07-26T11:05:19.006 INFO:tasks.workunit.client.0.smithi144.stdout:3/802: write d1/d5/d97/d48/fec [57836,43719] 0 2024-07-26T11:05:19.007 INFO:tasks.workunit.client.0.smithi144.stdout:3/803: link d1/d28/d54/d26/d30/d79/lf7 d1/d28/d54/d26/d64/d89/lfd 0 2024-07-26T11:05:19.007 INFO:tasks.workunit.client.0.smithi144.stdout:3/804: stat d1/ff 0 2024-07-26T11:05:19.008 INFO:tasks.workunit.client.0.smithi144.stdout:3/805: symlink d1/d28/d54/d26/d37/lfe 0 2024-07-26T11:05:19.008 INFO:tasks.workunit.client.0.smithi144.stdout:3/806: creat d1/d5/fff x:0 0 0 2024-07-26T11:05:19.009 INFO:tasks.workunit.client.0.smithi144.stdout:3/807: mknod d1/c100 0 2024-07-26T11:05:19.010 INFO:tasks.workunit.client.0.smithi144.stdout:3/808: symlink d1/d28/d54/d26/d64/ded/l101 0 2024-07-26T11:05:19.011 INFO:tasks.workunit.client.0.smithi144.stdout:3/809: chown d1/d28/d54/d26/d30/d79/fbc 161830305 1 2024-07-26T11:05:19.011 INFO:tasks.workunit.client.0.smithi144.stdout:3/810: mkdir d1/d28/d54/d102 0 2024-07-26T11:05:19.012 INFO:tasks.workunit.client.0.smithi144.stdout:3/811: mkdir d1/d28/d54/d26/d37/d103 0 2024-07-26T11:05:19.014 INFO:tasks.workunit.client.0.smithi144.stdout:3/812: creat d1/d28/d54/d102/f104 x:0 0 0 2024-07-26T11:05:19.014 INFO:tasks.workunit.client.0.smithi144.stdout:3/813: truncate d1/d5/d97/d48/fec 670053 0 2024-07-26T11:05:19.015 INFO:tasks.workunit.client.0.smithi144.stdout:3/814: unlink d1/d28/d54/d26/d37/l38 0 2024-07-26T11:05:19.015 INFO:tasks.workunit.client.0.smithi144.stdout:3/815: fdatasync d1/d28/d54/d102/f104 0 2024-07-26T11:05:19.016 INFO:tasks.workunit.client.0.smithi144.stdout:3/816: creat d1/d28/f105 x:0 0 0 2024-07-26T11:05:19.018 INFO:tasks.workunit.client.0.smithi144.stdout:3/817: rename d1/d28/d54/d26/d64/d68/fc8 to d1/d28/d54/d26/d30/f106 0 2024-07-26T11:05:19.019 INFO:tasks.workunit.client.0.smithi144.stdout:3/818: mknod d1/d5/d97/da9/c107 0 2024-07-26T11:05:19.021 INFO:tasks.workunit.client.0.smithi144.stdout:3/819: unlink d1/d5/d5e/fef 0 2024-07-26T11:05:19.021 INFO:tasks.workunit.client.0.smithi144.stdout:3/820: fsync d1/d28/d54/d3d/f9a 0 2024-07-26T11:05:19.021 INFO:tasks.workunit.client.0.smithi144.stdout:3/821: read - d1/d28/d6a/fa8 zero size 2024-07-26T11:05:19.022 INFO:tasks.workunit.client.0.smithi144.stdout:3/822: rmdir d1/d28/d54/d26/d64/d68/db5 39 2024-07-26T11:05:19.024 INFO:tasks.workunit.client.0.smithi144.stdout:3/823: mkdir d1/d28/d54/d3d/d42/dc4/d108 0 2024-07-26T11:05:19.024 INFO:tasks.workunit.client.0.smithi144.stdout:3/824: chown d1/d5/d97/f5f 1524763 1 2024-07-26T11:05:19.024 INFO:tasks.workunit.client.0.smithi144.stdout:3/825: readlink d1/d28/d54/d3d/d42/l7b 0 2024-07-26T11:05:19.024 INFO:tasks.workunit.client.0.smithi144.stdout:3/826: truncate d1/d28/d54/d26/dc7/fa0 4259819 0 2024-07-26T11:05:19.024 INFO:tasks.workunit.client.0.smithi144.stdout:3/827: chown d1/d28/d54/d26/d64/da4 1098 1 2024-07-26T11:05:19.048 INFO:tasks.workunit.client.0.smithi144.stdout:3/828: write d1/d28/d54/d26/f2d [2261124,108869] 0 2024-07-26T11:05:19.048 INFO:tasks.workunit.client.0.smithi144.stdout:3/829: readlink d1/d28/d54/d26/d37/d91/ld3 0 2024-07-26T11:05:19.106 INFO:tasks.workunit.client.0.smithi144.stdout:3/830: dwrite d1/d28/d54/d26/d64/d68/fa3 [0,4194304] 0 2024-07-26T11:05:19.109 INFO:tasks.workunit.client.0.smithi144.stdout:3/831: mkdir d1/d28/d54/d26/d30/d109 0 2024-07-26T11:05:19.109 INFO:tasks.workunit.client.0.smithi144.stdout:3/832: readlink d1/lc 0 2024-07-26T11:05:19.109 INFO:tasks.workunit.client.0.smithi144.stdout:3/833: write d1/d28/d54/d26/d37/da1/fde [1405779,11701] 0 2024-07-26T11:05:19.112 INFO:tasks.workunit.client.0.smithi144.stdout:3/834: creat d1/d28/d54/d3d/d42/dc4/d108/f10a x:0 0 0 2024-07-26T11:05:19.112 INFO:tasks.workunit.client.0.smithi144.stdout:3/835: mknod d1/d28/d54/d61/df4/c10b 0 2024-07-26T11:05:19.114 INFO:tasks.workunit.client.0.smithi144.stdout:3/836: mkdir d1/d28/d54/d26/d37/da1/d10c 0 2024-07-26T11:05:19.114 INFO:tasks.workunit.client.0.smithi144.stdout:3/837: chown d1/d28/d54/d26/d64/fbd 14329 1 2024-07-26T11:05:19.114 INFO:tasks.workunit.client.0.smithi144.stdout:3/838: readlink d1/d5/l21 0 2024-07-26T11:05:19.115 INFO:tasks.workunit.client.0.smithi144.stdout:3/839: truncate d1/f1a 4158934 0 2024-07-26T11:05:19.115 INFO:tasks.workunit.client.0.smithi144.stdout:3/840: write d1/d28/d54/d26/d37/da1/fde [2355236,74040] 0 2024-07-26T11:05:19.115 INFO:tasks.workunit.client.0.smithi144.stdout:3/841: fdatasync d1/d28/d54/d26/f2e 0 2024-07-26T11:05:19.116 INFO:tasks.workunit.client.0.smithi144.stdout:3/842: mkdir d1/d28/d54/d26/d64/ded/d10d 0 2024-07-26T11:05:19.161 INFO:tasks.workunit.client.0.smithi144.stdout:3/843: write d1/d5/d97/d60/f8c [4123164,49430] 0 2024-07-26T11:05:19.162 INFO:tasks.workunit.client.0.smithi144.stdout:3/844: truncate d1/d28/d54/d26/d64/d68/db5/fbf 604390 0 2024-07-26T11:05:19.162 INFO:tasks.workunit.client.0.smithi144.stdout:3/845: fsync d1/d28/d54/d26/d30/f7f 0 2024-07-26T11:05:19.163 INFO:tasks.workunit.client.0.smithi144.stdout:3/846: creat d1/d28/d54/d26/d64/d68/db5/f10e x:0 0 0 2024-07-26T11:05:19.176 INFO:tasks.workunit.client.0.smithi144.stdout:3/847: dread d1/d28/d54/d26/d64/da4/fc6 [0,4194304] 0 2024-07-26T11:05:19.226 INFO:tasks.workunit.client.0.smithi144.stdout:3/848: write d1/d28/d54/d26/d64/d89/ff0 [1722419,76791] 0 2024-07-26T11:05:19.269 INFO:tasks.workunit.client.0.smithi144.stdout:3/849: dread d1/d28/d54/d26/f2d [0,4194304] 0 2024-07-26T11:05:19.269 INFO:tasks.workunit.client.0.smithi144.stdout:3/850: write d1/d28/d54/d102/f104 [838949,85038] 0 2024-07-26T11:05:19.343 INFO:tasks.workunit.client.0.smithi144.stdout:3/851: dwrite d1/d28/d54/d26/d37/da1/de1/fe7 [4194304,4194304] 0 2024-07-26T11:05:19.350 INFO:tasks.workunit.client.0.smithi144.stdout:3/852: fsync d1/d28/d54/d26/d37/f58 0 2024-07-26T11:05:19.350 INFO:tasks.workunit.client.0.smithi144.stdout:3/853: chown d1/d28/d54/d26/f2b 215182 1 2024-07-26T11:05:19.350 INFO:tasks.workunit.client.0.smithi144.stdout:3/854: chown d1/d28/d54/d3d/f9a 59178 1 2024-07-26T11:05:19.388 INFO:tasks.workunit.client.0.smithi144.stdout:3/855: dread d1/f39 [0,4194304] 0 2024-07-26T11:05:19.393 INFO:tasks.workunit.client.0.smithi144.stdout:3/856: truncate d1/d28/d54/d26/d37/da1/de1/fe7 3217163 0 2024-07-26T11:05:19.466 INFO:tasks.workunit.client.0.smithi144.stdout:3/857: dwrite d1/d28/d54/d26/d37/da1/de1/fe7 [0,4194304] 0 2024-07-26T11:05:19.467 INFO:tasks.workunit.client.0.smithi144.stdout:3/858: mknod d1/d28/d54/d61/c10f 0 2024-07-26T11:05:19.469 INFO:tasks.workunit.client.0.smithi144.stdout:3/859: creat d1/d28/d54/d61/df4/f110 x:0 0 0 2024-07-26T11:05:19.469 INFO:tasks.workunit.client.0.smithi144.stdout:3/860: chown d1/d5/f1c 687708776 1 2024-07-26T11:05:19.477 INFO:tasks.workunit.client.0.smithi144.stdout:3/861: write d1/f39 [2365701,44227] 0 2024-07-26T11:05:19.478 INFO:tasks.workunit.client.0.smithi144.stdout:3/862: creat d1/d5/d97/d60/f111 x:0 0 0 2024-07-26T11:05:19.540 INFO:tasks.workunit.client.0.smithi144.stdout:3/863: dread d1/d5/d97/d60/fb8 [0,4194304] 0 2024-07-26T11:05:19.540 INFO:tasks.workunit.client.0.smithi144.stdout:3/864: dread - d1/d28/f105 zero size 2024-07-26T11:05:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:19 smithi160 ceph-mon[36715]: pgmap v546: 129 pgs: 129 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 10 MiB/s wr, 511 op/s 2024-07-26T11:05:19.622 INFO:tasks.workunit.client.0.smithi144.stdout:3/865: dwrite d1/d28/d54/d26/d64/d68/f84 [0,4194304] 0 2024-07-26T11:05:19.624 INFO:tasks.workunit.client.0.smithi144.stdout:3/866: creat d1/d28/d54/d26/d37/da1/de1/f112 x:0 0 0 2024-07-26T11:05:19.627 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:19 smithi144 ceph-mon[32129]: pgmap v546: 129 pgs: 129 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 10 MiB/s wr, 511 op/s 2024-07-26T11:05:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:19 smithi183 ceph-mon[37273]: pgmap v546: 129 pgs: 129 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 10 MiB/s wr, 511 op/s 2024-07-26T11:05:19.663 INFO:tasks.workunit.client.0.smithi144.stdout:3/867: write d1/f34 [2370970,25791] 0 2024-07-26T11:05:19.665 INFO:tasks.workunit.client.0.smithi144.stdout:3/868: rename d1/c51 to d1/d28/d54/d26/d37/da1/dea/c113 0 2024-07-26T11:05:19.666 INFO:tasks.workunit.client.0.smithi144.stdout:3/869: chown d1/d28/d54/l3f 435 1 2024-07-26T11:05:19.666 INFO:tasks.workunit.client.0.smithi144.stdout:3/870: stat d1/d5/d97 0 2024-07-26T11:05:19.737 INFO:tasks.workunit.client.0.smithi144.stdout:3/871: dwrite d1/d28/d54/d26/d64/da4/ff3 [0,4194304] 0 2024-07-26T11:05:19.756 INFO:tasks.workunit.client.0.smithi144.stdout:3/872: read d1/d28/d54/d26/f2b [1146235,44251] 0 2024-07-26T11:05:19.757 INFO:tasks.workunit.client.0.smithi144.stdout:3/873: rmdir d1/d28/d54/d26/d37 39 2024-07-26T11:05:19.759 INFO:tasks.workunit.client.0.smithi144.stdout:3/874: mknod d1/d28/d54/d26/d64/c114 0 2024-07-26T11:05:20.032 INFO:tasks.workunit.client.0.smithi144.stdout:3/875: sync 2024-07-26T11:05:20.032 INFO:tasks.workunit.client.0.smithi144.stdout:3/876: stat d1/d28/d54/d26/d30/d4d 0 2024-07-26T11:05:20.089 INFO:tasks.workunit.client.0.smithi144.stdout:3/877: dwrite d1/d28/d54/d26/d64/d68/f7d [0,4194304] 0 2024-07-26T11:05:20.173 INFO:tasks.workunit.client.0.smithi144.stdout:3/878: dwrite d1/d28/d54/d26/d64/d89/ffb [0,4194304] 0 2024-07-26T11:05:20.176 INFO:tasks.workunit.client.0.smithi144.stdout:3/879: truncate d1/d28/d54/d26/d37/da1/fdb 358028 0 2024-07-26T11:05:20.182 INFO:tasks.workunit.client.0.smithi144.stdout:3/880: write d1/d5/d97/d48/fad [1323928,129593] 0 2024-07-26T11:05:20.184 INFO:tasks.workunit.client.0.smithi144.stdout:3/881: mknod d1/d28/d54/d26/d64/d89/c115 0 2024-07-26T11:05:20.202 INFO:tasks.workunit.client.0.smithi144.stdout:3/882: dread d1/d28/d54/d26/f33 [0,4194304] 0 2024-07-26T11:05:20.202 INFO:tasks.workunit.client.0.smithi144.stdout:3/883: stat d1/d28/d54/d3d/d42/cac 0 2024-07-26T11:05:20.203 INFO:tasks.workunit.client.0.smithi144.stdout:3/884: write d1/d28/d54/d26/d37/da1/fcc [188209,86258] 0 2024-07-26T11:05:20.203 INFO:tasks.workunit.client.0.smithi144.stdout:3/885: read d1/d28/d54/d26/f2b [833105,114599] 0 2024-07-26T11:05:20.440 INFO:tasks.workunit.client.0.smithi144.stdout:3/886: sync 2024-07-26T11:05:20.528 INFO:tasks.workunit.client.0.smithi144.stdout:3/887: dwrite d1/d28/d54/d3d/d42/fd4 [0,4194304] 0 2024-07-26T11:05:20.541 INFO:tasks.workunit.client.0.smithi144.stdout:3/888: creat d1/d28/d54/d3d/dfc/f116 x:0 0 0 2024-07-26T11:05:20.549 INFO:tasks.workunit.client.0.smithi144.stdout:8/959: rename d1a/f21 to d1a/d32/d6c/f13f 0 2024-07-26T11:05:20.551 INFO:tasks.workunit.client.0.smithi144.stdout:8/960: mknod d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/dbd/c140 0 2024-07-26T11:05:20.577 INFO:tasks.workunit.client.0.smithi144.stdout:8/961: dread d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/f6e [0,4194304] 0 2024-07-26T11:05:20.742 INFO:tasks.workunit.client.0.smithi144.stdout:8/962: sync 2024-07-26T11:05:20.742 INFO:tasks.workunit.client.0.smithi144.stdout:8/963: readlink d1a/d32/d49/d73/d7b/d43/d4c/d128/l33 0 2024-07-26T11:05:20.778 INFO:tasks.workunit.client.0.smithi144.stdout:8/964: dread f6 [0,4194304] 0 2024-07-26T11:05:20.778 INFO:tasks.workunit.client.0.smithi144.stdout:8/965: chown d1a/d32/d50/dc4/ld3 125527600 1 2024-07-26T11:05:20.782 INFO:tasks.workunit.client.0.smithi144.stdout:8/966: link d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/c110 d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/da3/db1/c141 0 2024-07-26T11:05:20.785 INFO:tasks.workunit.client.0.smithi144.stdout:8/967: creat d1a/d32/d6c/d82/f142 x:0 0 0 2024-07-26T11:05:20.790 INFO:tasks.workunit.client.0.smithi144.stdout:8/968: rename d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/f12f to d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/f143 0 2024-07-26T11:05:20.790 INFO:tasks.workunit.client.0.smithi144.stdout:8/969: chown d1a/d32/d49/d73/d7b/d43/la7 7446461 1 2024-07-26T11:05:20.793 INFO:tasks.workunit.client.0.smithi144.stdout:8/970: mkdir d1a/d32/d50/dc4/df6/d144 0 2024-07-26T11:05:20.839 INFO:tasks.workunit.client.0.smithi144.stdout:8/971: dread d1a/d1c/f41 [0,4194304] 0 2024-07-26T11:05:20.909 INFO:tasks.workunit.client.0.smithi144.stdout:8/972: dwrite d1a/d1c/f122 [0,4194304] 0 2024-07-26T11:05:20.912 INFO:tasks.workunit.client.0.smithi144.stdout:8/973: write d1a/d32/f62 [1969010,100704] 0 2024-07-26T11:05:20.984 INFO:tasks.workunit.client.0.smithi144.stdout:8/974: dwrite d1a/d32/d49/d73/d7b/d43/d4c/d128/d9d/f12b [0,4194304] 0 2024-07-26T11:05:20.992 INFO:tasks.workunit.client.0.smithi144.stdout:8/975: getdents d1a/d32/dd4 0 2024-07-26T11:05:20.992 INFO:tasks.workunit.client.0.smithi144.stdout:8/976: write d1a/d1c/f70 [1086136,100989] 0 2024-07-26T11:05:21.000 INFO:tasks.workunit.client.0.smithi144.stdout:8/977: unlink d1a/d32/d49/d73/d7b/d43/d4c/d128/d9d/ce7 0 2024-07-26T11:05:21.042 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:21 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:21.042 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:21 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:21.042 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:21 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:21.042 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:21 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:21.064 INFO:tasks.workunit.client.0.smithi144.stdout:8/978: dwrite d1a/d32/d49/d73/d7b/f5e [0,4194304] 0 2024-07-26T11:05:21.066 INFO:tasks.workunit.client.0.smithi144.stdout:8/979: link d1a/d32/d49/d73/f12d d1a/d32/d66/dcd/f145 0 2024-07-26T11:05:21.116 INFO:tasks.workunit.client.0.smithi144.stdout:8/980: dwrite d1a/d32/d49/d73/d7b/f37 [4194304,4194304] 0 2024-07-26T11:05:21.139 INFO:tasks.workunit.client.0.smithi144.stdout:8/981: read d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/f143 [2801480,52967] 0 2024-07-26T11:05:21.140 INFO:tasks.workunit.client.0.smithi144.stdout:8/982: read - d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/da3/db1/f11c zero size 2024-07-26T11:05:21.148 INFO:tasks.workunit.client.0.smithi144.stdout:8/983: dread d1a/d32/d49/d73/d7b/d43/d4c/d128/ff2 [0,4194304] 0 2024-07-26T11:05:21.162 INFO:tasks.workunit.client.0.smithi144.stdout:8/984: write d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/fe0 [460313,114612] 0 2024-07-26T11:05:21.162 INFO:tasks.workunit.client.0.smithi144.stdout:8/985: fdatasync d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/dc2/fd6 0 2024-07-26T11:05:21.162 INFO:tasks.workunit.client.0.smithi144.stdout:8/986: fdatasync d1a/d32/d66/da0/ddc/ffb 0 2024-07-26T11:05:21.167 INFO:tasks.workunit.client.0.smithi144.stdout:8/987: creat d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d84/db0/dba/f146 x:0 0 0 2024-07-26T11:05:21.172 INFO:tasks.workunit.client.0.smithi144.stdout:8/988: mknod d1a/d32/d49/d73/d7b/d43/d4c/d128/d9d/d74/d104/c147 0 2024-07-26T11:05:21.172 INFO:tasks.workunit.client.0.smithi144.stdout:8/989: chown d1a/d32/f62 1300346040 1 2024-07-26T11:05:21.176 INFO:tasks.workunit.client.0.smithi144.stdout:8/990: symlink d1a/d32/d8c/dd0/l148 0 2024-07-26T11:05:21.179 INFO:tasks.workunit.client.0.smithi144.stdout:8/991: mkdir d1a/d32/d49/d73/d7b/d43/d4c/d149 0 2024-07-26T11:05:21.179 INFO:tasks.workunit.client.0.smithi144.stdout:8/992: chown d1a/d32/d49/d73/d7b/d43/d4c/d128/d9d/d74/dac/dbe 1319 1 2024-07-26T11:05:21.183 INFO:tasks.workunit.client.0.smithi144.stdout:8/993: link d1a/d32/d49/d73/d7b/d4e/d9a/dc1/ce9 d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/c14a 0 2024-07-26T11:05:21.189 INFO:tasks.workunit.client.0.smithi144.stdout:8/994: truncate d1a/d1c/f41 428518 0 2024-07-26T11:05:21.189 INFO:tasks.workunit.client.0.smithi144.stdout:8/995: stat d1a/d32/d49/d73/d7b/d43/d4c/d128/d38/d58/da3/d124 0 2024-07-26T11:05:21.189 INFO:tasks.workunit.client.0.smithi144.stdout:8/996: truncate d1a/d1c/f22 673886 0 2024-07-26T11:05:21.189 INFO:tasks.workunit.client.0.smithi144.stdout:8/997: write d1a/d1c/f122 [4821107,22311] 0 2024-07-26T11:05:21.192 INFO:tasks.workunit.client.0.smithi144.stdout:8/998: symlink d1a/d32/d8c/dd0/l14b 0 2024-07-26T11:05:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:21 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:21 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:21 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:21 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:21 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:21 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:21 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:21 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:21.456 INFO:tasks.workunit.client.0.smithi144.stdout:8/999: sync 2024-07-26T11:05:21.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 -- 172.21.15.144:0/2510872588 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd3410af10 msgr2=0x7fbd3410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 --2- 172.21.15.144:0/2510872588 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd3410af10 0x7fbd3410b320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009900 tx=0x7fbd2002a410 comp rx=0 tx=0).stop 2024-07-26T11:05:21.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 -- 172.21.15.144:0/2510872588 shutdown_connections 2024-07-26T11:05:21.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 --2- 172.21.15.144:0/2510872588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410b860 0x7fbd34112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 --2- 172.21.15.144:0/2510872588 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd3410af10 0x7fbd3410b320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 --2- 172.21.15.144:0/2510872588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd341055d0 0x7fbd341059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 -- 172.21.15.144:0/2510872588 >> 172.21.15.144:0/2510872588 conn(0x7fbd34101040 msgr2=0x7fbd34103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.409+0000 7fbd39e00640 1 -- 172.21.15.144:0/2510872588 shutdown_connections 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.410+0000 7fbd39e00640 1 -- 172.21.15.144:0/2510872588 wait complete. 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 Processor -- start 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 -- start start 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 0x7fbd3419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410af10 0x7fbd3419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd3410b860 0x7fbd341a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbd34115540 con 0x7fbd3410b860 2024-07-26T11:05:21.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341153c0 con 0x7fbd341055d0 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.411+0000 7fbd39e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341156c0 con 0x7fbd3410af10 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 0x7fbd3419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd33e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410af10 0x7fbd3419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 0x7fbd3419bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47296/0 (socket says 172.21.15.144:47296) 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 -- 172.21.15.144:0/671700657 learned_addr learned my addr 172.21.15.144:0/671700657 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd39400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd3410b860 0x7fbd341a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd33e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410af10 0x7fbd3419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:40600/0 (socket says 172.21.15.144:40600) 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 -- 172.21.15.144:0/671700657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410af10 msgr2=0x7fbd3419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410af10 0x7fbd3419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 -- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd3410b860 msgr2=0x7fbd341a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd3410b860 0x7fbd341a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 -- 172.21.15.144:0/671700657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd341a1ec0 con 0x7fbd341055d0 2024-07-26T11:05:21.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.412+0000 7fbd38a00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 0x7fbd3419bf50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800e9f0 tx=0x7fbd2800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.413+0000 7fbd31600640 1 -- 172.21.15.144:0/671700657 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2800cd80 con 0x7fbd341055d0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.413+0000 7fbd31600640 1 -- 172.21.15.144:0/671700657 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd280108e0 con 0x7fbd341055d0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.413+0000 7fbd31600640 1 -- 172.21.15.144:0/671700657 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28018980 con 0x7fbd341055d0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.413+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd341a21b0 con 0x7fbd341055d0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.413+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd341a8910 con 0x7fbd341055d0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 Processor -- start 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 -- start start 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd31600640 1 -- 172.21.15.144:0/671700657 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fbd28010430 con 0x7fbd341055d0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1f1390 0x564f0f5b7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564f0f52dbc0 con 0x564f0f1e91b0 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564f0f51d0c0 con 0x564f0f1f1390 2024-07-26T11:05:21.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.417+0000 7fbd420c2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564f0f521100 con 0x564f0f212070 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd38a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:40606/0 (socket says 172.21.15.144:40606) 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 -- 172.21.15.144:0/449168785 learned_addr learned my addr 172.21.15.144:0/449168785 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd33e00640 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1f1390 0x564f0f5b7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 -- 172.21.15.144:0/449168785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1f1390 msgr2=0x564f0f5b7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1f1390 0x564f0f5b7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 -- 172.21.15.144:0/449168785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 msgr2=0x564f0f5b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.418+0000 7fbd39400640 1 -- 172.21.15.144:0/449168785 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f0f27d650 con 0x564f0f212070 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.419+0000 7fbd39400640 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5b71f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009fd0 tx=0x7fbd2c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf8bfdf6a1796f2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.419+0000 7fbd25c00640 1 -- 172.21.15.144:0/449168785 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c004d20 con 0x564f0f212070 2024-07-26T11:05:21.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.419+0000 7fbd25c00640 1 -- 172.21.15.144:0/449168785 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd2c004050 con 0x564f0f212070 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.419+0000 7fbd25c00640 1 -- 172.21.15.144:0/449168785 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c005280 con 0x564f0f212070 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.419+0000 7fbd31600640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbd1c0b3dc0 0x7fbd1c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd31600640 1 -- 172.21.15.144:0/671700657 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fbd280e4610 con 0x7fbd341055d0 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd38a00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbd1c0b3dc0 0x7fbd1c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 -- 172.21.15.144:0/449168785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 msgr2=0x564f0f5b71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5b71f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009fd0 tx=0x7fbd2c0118d0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 -- 172.21.15.144:0/449168785 shutdown_connections 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1f1390 0x564f0f5b7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 --2- 172.21.15.144:0/449168785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5b71f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 -- 172.21.15.144:0/449168785 >> 172.21.15.144:0/449168785 conn(0x564f0f206460 msgr2=0x564f0f5b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 -- 172.21.15.144:0/449168785 shutdown_connections 2024-07-26T11:05:21.610 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.420+0000 7fbd420c2740 1 -- 172.21.15.144:0/449168785 wait complete. 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.421+0000 7fbd38a00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbd1c0b3dc0 0x7fbd1c0b62d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd280e1aa0 tx=0x7fbd280ea550 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.421+0000 7fbd420c2740 1 -- start start 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.421+0000 7fbd420c2740 1 Processor -- start 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 0x564f0f5c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564f0f380700 con 0x564f0f1f1390 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564f0f4b9690 con 0x564f0f1e91b0 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564f0f5c59d0 con 0x564f0f212070 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd420c2740 10 client.0 _ll_register_callbacks cb 0x564f0f4c9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd39400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 0x564f0f5c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd39400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 0x564f0f5c27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47320/0 (socket says 172.21.15.144:47320) 2024-07-26T11:05:21.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd39400640 1 -- 172.21.15.144:0/2096469485 learned_addr learned my addr 172.21.15.144:0/2096469485 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd33e00640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.422+0000 7fbd39400640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd39400640 1 -- 172.21.15.144:0/2096469485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 msgr2=0x564f0f5c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd39400640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5c3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd39400640 1 -- 172.21.15.144:0/2096469485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 msgr2=0x564f0f5c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd39400640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5c2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd39400640 1 -- 172.21.15.144:0/2096469485 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f0f4d0830 con 0x564f0f1e91b0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd39400640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 0x564f0f5c27c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c003fd0 tx=0x7fbd2c005a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ca20c1913099f6e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd1a000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd1a000640 1 -- 172.21.15.144:0/2096469485 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c02c020 con 0x564f0f1e91b0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.423+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd2c004050 con 0x564f0f1e91b0 2024-07-26T11:05:21.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.424+0000 7fbd1a000640 1 -- 172.21.15.144:0/2096469485 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fbd2c00f4e0 con 0x564f0f1e91b0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.424+0000 7fbd1a000640 1 -- 172.21.15.144:0/2096469485 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd2c037050 con 0x564f0f1e91b0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.424+0000 7fbd420c2740 10 client.25973 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.424+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f0f5d1600 con 0x564f0f1e91b0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.424+0000 7fbd1a000640 1 -- 172.21.15.144:0/2096469485 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd2c004050 con 0x564f0f1e91b0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.424+0000 7fbd1a000640 1 -- 172.21.15.144:0/2096469485 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbd2c028450 con 0x564f0f1e91b0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd420c2740 10 client.25973 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd420c2740 10 client.25973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd420c2740 20 client.25973 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f1d09f0 0x564f0f5d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd420c2740 4 client.25973 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f0f52fb10 con 0x564f0f1d09f0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd33e00640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f1d09f0 0x564f0f5d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd33e00640 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f1d09f0 0x564f0f5d4a20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c3967e588c3de09 server_cookie=c663888d47a8fbbe in_seq=0 out_seq=0 2024-07-26T11:05:21.613 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.425+0000 7fbd1a000640 10 client.25973 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.426+0000 7fbd1a000640 1 -- 172.21.15.144:0/2096469485 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f0f52fb10 con 0x564f0f1d09f0 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.426+0000 7fbd1a000640 10 client.25973 handle_command_reply: tid=0 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.426+0000 7fbd420c2740 1 client.25973 shutdown 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.426+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 msgr2=0x564f0f5c27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.426+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 0x564f0f5c27c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c003fd0 tx=0x7fbd2c005a50 comp rx=0 tx=0).stop 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 shutdown_connections 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f1d09f0 0x564f0f5d4a20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f212070 0x564f0f5c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2096469485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f1e91b0 0x564f0f5c27c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd1a000640 0 client.25973 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 >> 172.21.15.144:0/2096469485 conn(0x564f0f206460 msgr2=0x564f0f5ba380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 shutdown_connections 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 1 -- 172.21.15.144:0/2096469485 wait complete. 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.427+0000 7fbd420c2740 20 client.25973 trim_cache size 0 max 16384 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 Processor -- start 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 -- start start 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1e91b0 0x564f0f5b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5b8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5da120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341b51f0 con 0x564f0f1f1390 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341b2ac0 con 0x564f0f212070 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.446+0000 7fbd420c2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341b0910 con 0x564f0f1e91b0 2024-07-26T11:05:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5b8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33664/0 (socket says 172.21.15.144:33664) 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 -- 172.21.15.144:0/802776291 learned_addr learned my addr 172.21.15.144:0/802776291 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd39400640 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1e91b0 0x564f0f5b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 -- 172.21.15.144:0/802776291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1e91b0 msgr2=0x564f0f5b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1e91b0 0x564f0f5b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 -- 172.21.15.144:0/802776291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 msgr2=0x564f0f5da120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5da120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 -- 172.21.15.144:0/802776291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f0f5c3a80 con 0x564f0f1f1390 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.447+0000 7fbd33e00640 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5b8930 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbd20031440 tx=0x7fbd20005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4d52e235c6aa4a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.448+0000 7fbd25200640 1 -- 172.21.15.144:0/802776291 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20035040 con 0x564f0f1f1390 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.448+0000 7fbd25200640 1 -- 172.21.15.144:0/802776291 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd20004750 con 0x564f0f1f1390 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.448+0000 7fbd25200640 1 -- 172.21.15.144:0/802776291 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20003850 con 0x564f0f1f1390 2024-07-26T11:05:21.616 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 -- 172.21.15.144:0/802776291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 msgr2=0x564f0f5b8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5b8930 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbd20031440 tx=0x7fbd20005a20 comp rx=0 tx=0).stop 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 -- 172.21.15.144:0/802776291 shutdown_connections 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5da120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1f1390 0x564f0f5b8930 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 --2- 172.21.15.144:0/802776291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1e91b0 0x564f0f5b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 -- 172.21.15.144:0/802776291 >> 172.21.15.144:0/802776291 conn(0x564f0f206460 msgr2=0x564f0f5b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 -- 172.21.15.144:0/802776291 shutdown_connections 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.449+0000 7fbd420c2740 1 -- 172.21.15.144:0/802776291 wait complete. 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 -- start start 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 Processor -- start 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.617 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1a5da0 0x564f0f6350c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341b0f80 con 0x564f0f1e91b0 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341b23f0 con 0x564f0f212070 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341b1650 con 0x564f0f1a5da0 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.450+0000 7fbd420c2740 10 client.0 _ll_register_callbacks cb 0x7fbd2c003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33670/0 (socket says 172.21.15.144:33670) 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 -- 172.21.15.144:0/2552755840 learned_addr learned my addr 172.21.15.144:0/2552755840 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1a5da0 0x564f0f6350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd33e00640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 -- 172.21.15.144:0/2552755840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1a5da0 msgr2=0x564f0f6350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1a5da0 0x564f0f6350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.618 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 -- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 msgr2=0x564f0f5c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.451+0000 7fbd39400640 1 -- 172.21.15.144:0/2552755840 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f0f635730 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd39400640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5c8450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c006010 tx=0x7fbd2c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f56f9b1f124cb1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd1a000640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd1a000640 1 -- 172.21.15.144:0/2552755840 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c02c050 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd20004750 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd1a000640 1 -- 172.21.15.144:0/2552755840 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7fbd2c004000 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd1a000640 1 -- 172.21.15.144:0/2552755840 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd2c005b40 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd420c2740 10 client.15828 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.452+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f0f63ca80 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd1a000640 1 -- 172.21.15.144:0/2552755840 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd20004750 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd1a000640 1 -- 172.21.15.144:0/2552755840 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbd2c0284e0 con 0x564f0f1e91b0 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd420c2740 10 client.15828 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:21.619 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd420c2740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd420c2740 20 client.15828 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f63d230 0x564f0f63f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd420c2740 4 client.15828 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f0f52b5b0 con 0x564f0f63d230 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.453+0000 7fbd33e00640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f63d230 0x564f0f63f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd33e00640 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f63d230 0x564f0f63f650 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd6afef5816822f3 server_cookie=79eb7edbbfff04e6 in_seq=0 out_seq=0 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd1a000640 10 client.15828 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd1a000640 1 -- 172.21.15.144:0/2552755840 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x564f0f52b5b0 con 0x564f0f63d230 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd1a000640 10 client.15828 handle_command_reply: tid=0 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 client.15828 shutdown 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 msgr2=0x564f0f5c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5c8450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c006010 tx=0x7fbd2c000f40 comp rx=0 tx=0).stop 2024-07-26T11:05:21.620 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 shutdown_connections 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x564f0f63d230 0x564f0f63f650 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f0f1a5da0 0x564f0f6350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f0f212070 0x564f0f5c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 --2- 172.21.15.144:0/2552755840 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f0f1e91b0 0x564f0f5c8450 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd1a000640 0 client.15828 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.454+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 >> 172.21.15.144:0/2552755840 conn(0x564f0f206460 msgr2=0x564f0f5b3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.455+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 shutdown_connections 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.455+0000 7fbd420c2740 1 -- 172.21.15.144:0/2552755840 wait complete. 2024-07-26T11:05:21.621 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "scrub active (154 inodes in the stack)", 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: "5707e68b-2051-495c-91d1-212436abd999": { 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "5707e68b-2051-495c-91d1-212436abd999", 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:05:21.622 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.455+0000 7fbd420c2740 20 client.15828 trim_cache size 0 max 16384 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbd1c0b3dc0 msgr2=0x7fbd1c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbd1c0b3dc0 0x7fbd1c0b62d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd280e1aa0 tx=0x7fbd280ea550 comp rx=0 tx=0).stop 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 msgr2=0x7fbd3419bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 0x7fbd3419bf50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800e9f0 tx=0x7fbd2800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 shutdown_connections 2024-07-26T11:05:21.623 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbd1c0b3dc0 0x7fbd1c0b62d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd3410b860 0x7fbd341a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd3410af10 0x7fbd3419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 --2- 172.21.15.144:0/671700657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbd341055d0 0x7fbd3419bf50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:21.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 >> 172.21.15.144:0/671700657 conn(0x7fbd34101040 msgr2=0x7fbd34103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:21.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 shutdown_connections 2024-07-26T11:05:21.624 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:21.456+0000 7fbd39e00640 1 -- 172.21.15.144:0/671700657 wait complete. 2024-07-26T11:05:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:22 smithi160 ceph-mon[36715]: pgmap v547: 129 pgs: 129 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.5 MiB/s wr, 713 op/s 2024-07-26T11:05:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:22 smithi183 ceph-mon[37273]: pgmap v547: 129 pgs: 129 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.5 MiB/s wr, 713 op/s 2024-07-26T11:05:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:22 smithi144 ceph-mon[32129]: pgmap v547: 129 pgs: 129 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.5 MiB/s wr, 713 op/s 2024-07-26T11:05:22.567 INFO:tasks.cephfs.filesystem:scrub status for tag:5707e68b-2051-495c-91d1-212436abd999 - {'path': '/', 'tag': '5707e68b-2051-495c-91d1-212436abd999', 'options': 'recursive,force'} 2024-07-26T11:05:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:24 smithi160 ceph-mon[36715]: pgmap v548: 129 pgs: 129 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.8 MiB/s wr, 743 op/s 2024-07-26T11:05:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:24 smithi183 ceph-mon[37273]: pgmap v548: 129 pgs: 129 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.8 MiB/s wr, 743 op/s 2024-07-26T11:05:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:24 smithi144 ceph-mon[32129]: pgmap v548: 129 pgs: 129 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.8 MiB/s wr, 743 op/s 2024-07-26T11:05:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:26 smithi160 ceph-mon[36715]: pgmap v549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 733 op/s 2024-07-26T11:05:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:26 smithi183 ceph-mon[37273]: pgmap v549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 733 op/s 2024-07-26T11:05:26.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:26 smithi144 ceph-mon[32129]: pgmap v549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 733 op/s 2024-07-26T11:05:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:28 smithi160 ceph-mon[36715]: pgmap v550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 687 op/s 2024-07-26T11:05:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:28 smithi183 ceph-mon[37273]: pgmap v550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 687 op/s 2024-07-26T11:05:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:28 smithi144 ceph-mon[32129]: pgmap v550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 687 op/s 2024-07-26T11:05:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 -- 172.21.15.144:0/3726526297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac10f9b0 msgr2=0x7f70ac111dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 --2- 172.21.15.144:0/3726526297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac10f9b0 0x7f70ac111dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009900 tx=0x7f70a402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 -- 172.21.15.144:0/3726526297 shutdown_connections 2024-07-26T11:05:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 --2- 172.21.15.144:0/3726526297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac10f9b0 0x7f70ac111dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 --2- 172.21.15.144:0/3726526297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70ac102750 0x7f70ac10f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 --2- 172.21.15.144:0/3726526297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac101e00 0x7f70ac102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 -- 172.21.15.144:0/3726526297 >> 172.21.15.144:0/3726526297 conn(0x7f70ac0fdc60 msgr2=0x7f70ac1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 -- 172.21.15.144:0/3726526297 shutdown_connections 2024-07-26T11:05:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.845+0000 7f70abe00640 1 -- 172.21.15.144:0/3726526297 wait complete. 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.846+0000 7f70abe00640 1 Processor -- start 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 -- start start 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70ac101e00 0x7f70ac19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac102750 0x7f70ac19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac10f9b0 0x7f70ac1a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac115440 con 0x7f70ac101e00 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1152c0 con 0x7f70ac102750 2024-07-26T11:05:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70abe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1155c0 con 0x7f70ac10f9b0 2024-07-26T11:05:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70a2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac102750 0x7f70ac19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70ab400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac10f9b0 0x7f70ac1a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70ac101e00 0x7f70ac19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70a2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac102750 0x7f70ac19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47336/0 (socket says 172.21.15.144:47336) 2024-07-26T11:05:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.847+0000 7f70a2000640 1 -- 172.21.15.144:0/2771387350 learned_addr learned my addr 172.21.15.144:0/2771387350 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70ab400640 1 -- 172.21.15.144:0/2771387350 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac102750 msgr2=0x7f70ac19c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70ab400640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac102750 0x7f70ac19c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70ab400640 1 -- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70ac101e00 msgr2=0x7f70ac19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70ab400640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70ac101e00 0x7f70ac19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70ab400640 1 -- 172.21.15.144:0/2771387350 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac1a1d70 con 0x7f70ac10f9b0 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70ab400640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac10f9b0 0x7f70ac1a1670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f70a4005940 tx=0x7f70a4031a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70b0a00640 1 -- 172.21.15.144:0/2771387350 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a403f030 con 0x7f70ac10f9b0 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ac1a2000 con 0x7f70ac10f9b0 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70b0a00640 1 -- 172.21.15.144:0/2771387350 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a40043d0 con 0x7f70ac10f9b0 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.848+0000 7f70b0a00640 1 -- 172.21.15.144:0/2771387350 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4035070 con 0x7f70ac10f9b0 2024-07-26T11:05:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.849+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ac1a87f0 con 0x7f70ac10f9b0 2024-07-26T11:05:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.852+0000 7f70b8f79740 1 Processor -- start 2024-07-26T11:05:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.852+0000 7f70b0a00640 1 -- 172.21.15.144:0/2771387350 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f70a4042030 con 0x7f70ac10f9b0 2024-07-26T11:05:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.852+0000 7f70b8f79740 1 -- start start 2024-07-26T11:05:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692cb41b0 0x556693057c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693058160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930586a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70b8f79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x556692fcc270 con 0x556692c8b2f0 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70b8f79740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x556692fbb770 con 0x556692c934d0 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70b8f79740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556692fbf7b0 con 0x556692cb41b0 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70a2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70a2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47340/0 (socket says 172.21.15.144:47340) 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693058160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.853+0000 7f70a2000640 1 -- 172.21.15.144:0/3149581508 learned_addr learned my addr 172.21.15.144:0/3149581508 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692cb41b0 0x556693057c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 -- 172.21.15.144:0/3149581508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692cb41b0 msgr2=0x556693057c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692cb41b0 0x556693057c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 -- 172.21.15.144:0/3149581508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 msgr2=0x556693058160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693058160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 -- 172.21.15.144:0/3149581508 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556692d1f790 con 0x556692c934d0 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70aaa00640 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693058160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70b0a00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f70900b3d40 0x7f70900b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70a2000640 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930586a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7094011860 tx=0x7f7094009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bed35af050a42bf3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.854+0000 7f70b0a00640 1 -- 172.21.15.144:0/2771387350 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f70a4104630 con 0x7f70ac10f9b0 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.855+0000 7f709be00640 1 -- 172.21.15.144:0/3149581508 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094005020 con 0x556692c934d0 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.855+0000 7f70aaa00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f70900b3d40 0x7f70900b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.855+0000 7f709be00640 1 -- 172.21.15.144:0/3149581508 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70940047a0 con 0x556692c934d0 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.855+0000 7f709be00640 1 -- 172.21.15.144:0/3149581508 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709400fa00 con 0x556692c934d0 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 -- 172.21.15.144:0/3149581508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 msgr2=0x5566930586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70aaa00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f70900b3d40 0x7f70900b6250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f709c009fd0 tx=0x7f709c009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930586a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7094011860 tx=0x7f7094009550 comp rx=0 tx=0).stop 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 -- 172.21.15.144:0/3149581508 shutdown_connections 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930586a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693058160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 --2- 172.21.15.144:0/3149581508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692cb41b0 0x556693057c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 -- 172.21.15.144:0/3149581508 >> 172.21.15.144:0/3149581508 conn(0x556692ca8300 msgr2=0x556693055b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 -- 172.21.15.144:0/3149581508 shutdown_connections 2024-07-26T11:05:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 -- 172.21.15.144:0/3149581508 wait complete. 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.856+0000 7f70b8f79740 1 -- start start 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 Processor -- start 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x556693063860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c934d0 0x556693063e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669306f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x556692ef5ab0 con 0x556692c934d0 2024-07-26T11:05:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x556692f58d50 con 0x556692cb41b0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5566930663a0 con 0x556692c8b2f0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70b8f79740 10 client.0 _ll_register_callbacks cb 0x556692ccba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.857+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x556693063860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70ab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669306f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x556693063860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:40674/0 (socket says 172.21.15.144:40674) 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 -- 172.21.15.144:0/1857664962 learned_addr learned my addr 172.21.15.144:0/1857664962 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70a2000640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c934d0 0x556693063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 -- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 msgr2=0x55669306f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669306f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 -- 172.21.15.144:0/1857664962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c934d0 msgr2=0x556693063e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c934d0 0x556693063e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 -- 172.21.15.144:0/1857664962 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556692d98620 con 0x556692c8b2f0 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70ab400640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669306f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f70aaa00640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x556693063860 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f709c0115b0 tx=0x7f709c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6adfe172f516ba1d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.858+0000 7f7098c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f7098c00640 1 -- 172.21.15.144:0/1857664962 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c00fe20 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70940047a0 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f7098c00640 1 -- 172.21.15.144:0/1857664962 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f709c017030 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f7098c00640 1 -- 172.21.15.144:0/1857664962 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f709c003b80 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f70b8f79740 10 client.25969 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5566930724e0 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f7098c00640 1 -- 172.21.15.144:0/1857664962 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70940047a0 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.859+0000 7f7098c00640 1 -- 172.21.15.144:0/1857664962 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f709c0043e0 con 0x556692c8b2f0 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70b8f79740 10 client.25969 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70b8f79740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70b8f79740 20 client.25969 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c72d80 0x556693075a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70b8f79740 4 client.25969 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556692fce1c0 con 0x556692c72d80 2024-07-26T11:05:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.860+0000 7f70a2000640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c72d80 0x556693075a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.861+0000 7f70a2000640 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c72d80 0x556693075a80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=528cd3a5ed047b5 server_cookie=cf091b0e017d60e7 in_seq=0 out_seq=0 2024-07-26T11:05:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.861+0000 7f7098c00640 10 client.25969 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.861+0000 7f7098c00640 1 -- 172.21.15.144:0/1857664962 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556692fce1c0 con 0x556692c72d80 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.861+0000 7f7098c00640 10 client.25969 handle_command_reply: tid=0 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.861+0000 7f70b8f79740 1 client.25969 shutdown 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 msgr2=0x556693063860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x556693063860 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f709c0115b0 tx=0x7f709c004ab0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 shutdown_connections 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c72d80 0x556693075a80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669306f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c934d0 0x556693063e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f7098c00640 0 client.25969 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1857664962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x556693063860 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 >> 172.21.15.144:0/1857664962 conn(0x556692ca8300 msgr2=0x556693058c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 shutdown_connections 2024-07-26T11:05:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 1 -- 172.21.15.144:0/1857664962 wait complete. 2024-07-26T11:05:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.862+0000 7f70b8f79740 20 client.25969 trim_cache size 0 max 16384 2024-07-26T11:05:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.878+0000 7f70b8f79740 1 Processor -- start 2024-07-26T11:05:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.878+0000 7f70b8f79740 1 -- start start 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693065280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c934d0 0x55669307b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669307b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70b8f79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1b50c0 con 0x556692c8b2f0 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70b8f79740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1b2990 con 0x556692cb41b0 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70b8f79740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1b07e0 con 0x556692c934d0 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693065280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693065280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33726/0 (socket says 172.21.15.144:33726) 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70aaa00640 1 -- 172.21.15.144:0/1307392822 learned_addr learned my addr 172.21.15.144:0/1307392822 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70a2000640 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c934d0 0x55669307b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70ab400640 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669307b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70a2000640 1 -- 172.21.15.144:0/1307392822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 msgr2=0x55669307b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70a2000640 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669307b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.879+0000 7f70a2000640 1 -- 172.21.15.144:0/1307392822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 msgr2=0x556693065280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f70a2000640 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693065280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f70a2000640 1 -- 172.21.15.144:0/1307392822 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556693064260 con 0x556692c934d0 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f70aaa00640 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693065280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f70a2000640 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c934d0 0x55669307b240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7094004fd0 tx=0x7f7094009cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd245a4e46bb1565 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f709b400640 1 -- 172.21.15.144:0/1307392822 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094004720 con 0x556692c934d0 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f709b400640 1 -- 172.21.15.144:0/1307392822 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094004d90 con 0x556692c934d0 2024-07-26T11:05:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.880+0000 7f709b400640 1 -- 172.21.15.144:0/1307392822 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709400fac0 con 0x556692c934d0 2024-07-26T11:05:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 -- 172.21.15.144:0/1307392822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c934d0 msgr2=0x55669307b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c934d0 0x55669307b240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7094004fd0 tx=0x7f7094009cd0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 -- 172.21.15.144:0/1307392822 shutdown_connections 2024-07-26T11:05:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692cb41b0 0x55669307b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c934d0 0x55669307b240 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 --2- 172.21.15.144:0/1307392822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692c8b2f0 0x556693065280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 -- 172.21.15.144:0/1307392822 >> 172.21.15.144:0/1307392822 conn(0x556692ca8300 msgr2=0x556693064b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 -- 172.21.15.144:0/1307392822 shutdown_connections 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 -- 172.21.15.144:0/1307392822 wait complete. 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.881+0000 7f70b8f79740 1 -- start start 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 Processor -- start 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x55669306bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930665b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692cb41b0 0x556693066af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1b0e50 con 0x556692cb41b0 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1b22c0 con 0x556692c934d0 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1b1520 con 0x556692c8b2f0 2024-07-26T11:05:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.882+0000 7f70b8f79740 10 client.0 _ll_register_callbacks cb 0x7f709c009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x55669306bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70a2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70ab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692cb41b0 0x556693066af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70a2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930665b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47366/0 (socket says 172.21.15.144:47366) 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70a2000640 1 -- 172.21.15.144:0/2654220561 learned_addr learned my addr 172.21.15.144:0/2654220561 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x55669306bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:40692/0 (socket says 172.21.15.144:40692) 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 -- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 msgr2=0x5566930665b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930665b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 -- 172.21.15.144:0/2654220561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692cb41b0 msgr2=0x556693066af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692cb41b0 0x556693066af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 -- 172.21.15.144:0/2654220561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556693067230 con 0x556692c8b2f0 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70aaa00640 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x55669306bc80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f709c004a80 tx=0x7f709c01cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7c82c174e70331f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f7098c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f7098c00640 1 -- 172.21.15.144:0/2654220561 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c01fa00 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.883+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7094004d90 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f7098c00640 1 -- 172.21.15.144:0/2654220561 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f709c01fba0 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f7098c00640 1 -- 172.21.15.144:0/2654220561 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f709c0113f0 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 10 client.25975 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5566930dd0e0 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f7098c00640 1 -- 172.21.15.144:0/2654220561 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094004d90 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f7098c00640 1 -- 172.21.15.144:0/2654220561 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f709c03e030 con 0x556692c8b2f0 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 10 client.25975 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 10 client.25975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 20 client.25975 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c481c0 0x5566930df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 4 client.25975 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:05:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.884+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556692fc9c60 con 0x556692c481c0 2024-07-26T11:05:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.885+0000 7f70a2000640 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c481c0 0x5566930df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.885+0000 7f7098c00640 10 client.25975 ms_handle_connect on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:28.885+0000 7f70a2000640 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c481c0 0x5566930df980 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86a9e62fdde92de7 server_cookie=b3eda1fd1c980fe2 in_seq=0 out_seq=0 2024-07-26T11:05:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:30 smithi160 ceph-mon[36715]: pgmap v551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 11 MiB/s wr, 551 op/s 2024-07-26T11:05:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:30 smithi183 ceph-mon[37273]: pgmap v551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 11 MiB/s wr, 551 op/s 2024-07-26T11:05:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:30 smithi144 ceph-mon[32129]: pgmap v551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 11 MiB/s wr, 551 op/s 2024-07-26T11:05:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:32 smithi160 ceph-mon[36715]: pgmap v552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.3 MiB/s wr, 344 op/s 2024-07-26T11:05:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:32 smithi183 ceph-mon[37273]: pgmap v552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.3 MiB/s wr, 344 op/s 2024-07-26T11:05:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:32 smithi144 ceph-mon[32129]: pgmap v552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.3 MiB/s wr, 344 op/s 2024-07-26T11:05:32.809 INFO:tasks.workunit.client.0.smithi144.stdout:3/889: link d1/d28/d54/d26/f2b d1/d28/d54/d26/d30/d79/dca/f117 0 2024-07-26T11:05:32.811 INFO:tasks.workunit.client.0.smithi144.stdout:3/890: symlink d1/d28/d54/d26/d64/d68/db5/l118 0 2024-07-26T11:05:32.824 INFO:tasks.workunit.client.0.smithi144.stdout:3/891: write d1/d28/d54/d3d/f52 [882356,32249] 0 2024-07-26T11:05:32.824 INFO:tasks.workunit.client.0.smithi144.stdout:3/892: write d1/f20 [1250839,42773] 0 2024-07-26T11:05:32.826 INFO:tasks.workunit.client.0.smithi144.stdout:3/893: truncate d1/d28/d54/d26/d37/da1/de1/fe7 3861918 0 2024-07-26T11:05:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f7098c00640 1 -- 172.21.15.144:0/2654220561 <== mds.0 v2:172.21.15.160:6838/1074049609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556692fc9c60 con 0x556692c481c0 2024-07-26T11:05:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f7098c00640 10 client.25975 handle_command_reply: tid=0 2024-07-26T11:05:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 client.25975 shutdown 2024-07-26T11:05:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 msgr2=0x55669306bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x55669306bc80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f709c004a80 tx=0x7f709c01cb30 comp rx=0 tx=0).stop 2024-07-26T11:05:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 shutdown_connections 2024-07-26T11:05:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x556692c481c0 0x5566930df980 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556692cb41b0 0x556693066af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556692c934d0 0x5566930665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 --2- 172.21.15.144:0/2654220561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556692c8b2f0 0x55669306bc80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.833+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 >> 172.21.15.144:0/2654220561 conn(0x556692ca8300 msgr2=0x556693062b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.834+0000 7f7098c00640 0 client.25975 ms_handle_reset on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.834+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 shutdown_connections 2024-07-26T11:05:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.834+0000 7f70b8f79740 1 -- 172.21.15.144:0/2654220561 wait complete. 2024-07-26T11:05:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.834+0000 7f70b8f79740 20 client.25975 trim_cache size 0 max 16384 2024-07-26T11:05:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:{ 2024-07-26T11:05:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "message": "", 2024-07-26T11:05:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout: "return_code": 0 2024-07-26T11:05:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stdout:} 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f70900b3d40 msgr2=0x7f70900b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f70900b3d40 0x7f70900b6250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f709c009fd0 tx=0x7f709c009500 comp rx=0 tx=0).stop 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac10f9b0 msgr2=0x7f70ac1a1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac10f9b0 0x7f70ac1a1670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f70a4005940 tx=0x7f70a4031a10 comp rx=0 tx=0).stop 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 shutdown_connections 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f70900b3d40 0x7f70900b6250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70ac10f9b0 0x7f70ac1a1670 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f70ac102750 0x7f70ac19c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.835+0000 7f70abe00640 1 --2- 172.21.15.144:0/2771387350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70ac101e00 0x7f70ac19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.836+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 >> 172.21.15.144:0/2771387350 conn(0x7f70ac0fdc60 msgr2=0x7f70ac1094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.836+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 shutdown_connections 2024-07-26T11:05:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:32.836+0000 7f70abe00640 1 -- 172.21.15.144:0/2771387350 wait complete. 2024-07-26T11:05:32.902 INFO:tasks.workunit.client.0.smithi144.stdout:3/894: dwrite d1/d5/f35 [0,4194304] 0 2024-07-26T11:05:32.905 INFO:tasks.workunit.client.0.smithi144.stdout:3/895: symlink d1/d28/d54/d26/d30/d79/l119 0 2024-07-26T11:05:32.905 INFO:tasks.workunit.client.0.smithi144.stdout:3/896: chown d1/d28/d54/d26/d30/d78/fa5 191 1 2024-07-26T11:05:32.907 INFO:tasks.workunit.client.0.smithi144.stdout:3/897: symlink d1/d5/d97/d60/l11a 0 2024-07-26T11:05:32.921 INFO:tasks.workunit.client.0.smithi144.stdout:3/898: dread d1/d28/f9d [0,4194304] 0 2024-07-26T11:05:32.924 INFO:tasks.workunit.client.0.smithi144.stdout:3/899: rename d1/f50 to d1/d28/d54/d26/d64/d68/d74/f11b 0 2024-07-26T11:05:32.927 INFO:tasks.workunit.client.0.smithi144.stdout:3/900: creat d1/d28/d54/d26/d64/ded/d10d/f11c x:0 0 0 2024-07-26T11:05:32.933 INFO:tasks.workunit.client.0.smithi144.stdout:3/901: unlink d1/d5/d5e/c87 0 2024-07-26T11:05:32.936 INFO:tasks.workunit.client.0.smithi144.stdout:3/902: mkdir d1/d28/d54/d26/d37/d91/d11d 0 2024-07-26T11:05:32.937 INFO:tasks.workunit.client.0.smithi144.stdout:3/903: fdatasync d1/d28/d54/d26/d64/da4/fc6 0 2024-07-26T11:05:32.937 INFO:tasks.workunit.client.0.smithi144.stdout:3/904: truncate d1/d28/d54/d3d/d42/dc4/d108/f10a 429493 0 2024-07-26T11:05:32.993 INFO:tasks.workunit.client.0.smithi144.stdout:3/905: dwrite d1/f32 [0,4194304] 0 2024-07-26T11:05:33.039 INFO:tasks.workunit.client.0.smithi144.stdout:3/906: dwrite d1/d28/d54/d26/f27 [0,4194304] 0 2024-07-26T11:05:33.041 INFO:tasks.workunit.client.0.smithi144.stdout:3/907: creat d1/d28/d54/d61/f11e x:0 0 0 2024-07-26T11:05:33.041 INFO:tasks.workunit.client.0.smithi144.stdout:3/908: chown d1/d28/d54/d26/d30/d4d 99598760 1 2024-07-26T11:05:33.042 INFO:tasks.workunit.client.0.smithi144.stdout:3/909: getdents d1/d28/d54/d26/d37/da1/dea 0 2024-07-26T11:05:33.044 INFO:tasks.workunit.client.0.smithi144.stdout:3/910: symlink d1/d28/d54/d3d/dfc/l11f 0 2024-07-26T11:05:33.044 INFO:tasks.workunit.client.0.smithi144.stdout:3/911: readlink d1/l4c 0 2024-07-26T11:05:33.044 INFO:tasks.workunit.client.0.smithi144.stdout:3/912: truncate d1/d28/d54/d26/dc7/fa0 5170058 0 2024-07-26T11:05:33.046 INFO:tasks.workunit.client.0.smithi144.stdout:3/913: symlink d1/d28/d54/d26/d64/d68/l120 0 2024-07-26T11:05:33.049 INFO:tasks.workunit.client.0.smithi144.stdout:3/914: link d1/d5/d97/d60/l6f d1/d28/d54/d26/d30/d4d/l121 0 2024-07-26T11:05:33.051 INFO:tasks.workunit.client.0.smithi144.stdout:3/915: creat d1/d28/d54/d26/d37/da1/de1/f122 x:0 0 0 2024-07-26T11:05:33.051 INFO:tasks.workunit.client.0.smithi144.stdout:3/916: dread - d1/d28/d54/d26/d37/da1/de1/f112 zero size 2024-07-26T11:05:33.053 INFO:tasks.workunit.client.0.smithi144.stdout:3/917: mkdir d1/d5/d5e/d123 0 2024-07-26T11:05:33.055 INFO:tasks.workunit.client.0.smithi144.stdout:3/918: creat d1/d28/d54/d61/df4/f124 x:0 0 0 2024-07-26T11:05:33.067 INFO:tasks.workunit.client.0.smithi144.stdout:3/919: write d1/d28/d54/d26/d30/d4d/fe3 [275369,45442] 0 2024-07-26T11:05:33.073 INFO:tasks.workunit.client.0.smithi144.stdout:3/920: write d1/d28/d54/d26/f33 [31849,49542] 0 2024-07-26T11:05:33.076 INFO:tasks.workunit.client.0.smithi144.stdout:3/921: symlink d1/d5/d5e/dc5/l125 0 2024-07-26T11:05:33.132 INFO:tasks.workunit.client.0.smithi144.stdout:3/922: dwrite d1/d28/f9d [0,4194304] 0 2024-07-26T11:05:33.139 INFO:tasks.workunit.client.0.smithi144.stdout:3/923: creat d1/d28/d54/d61/f126 x:0 0 0 2024-07-26T11:05:33.139 INFO:tasks.workunit.client.0.smithi144.stdout:3/924: write d1/d5/f9b [672021,67254] 0 2024-07-26T11:05:33.140 INFO:tasks.workunit.client.0.smithi144.stdout:3/925: mkdir d1/d28/d54/d26/d30/d4d/d127 0 2024-07-26T11:05:33.142 INFO:tasks.workunit.client.0.smithi144.stdout:3/926: mkdir d1/d28/d54/d26/d37/d103/d128 0 2024-07-26T11:05:33.144 INFO:tasks.workunit.client.0.smithi144.stdout:3/927: mkdir d1/d28/d54/d26/d37/d103/d128/d129 0 2024-07-26T11:05:33.149 INFO:tasks.workunit.client.0.smithi144.stdout:3/928: mknod d1/c12a 0 2024-07-26T11:05:33.151 INFO:tasks.workunit.client.0.smithi144.stdout:3/929: truncate d1/f22 701739 0 2024-07-26T11:05:33.154 INFO:tasks.workunit.client.0.smithi144.stdout:3/930: rmdir d1/d28/d54 39 2024-07-26T11:05:33.180 INFO:tasks.workunit.client.0.smithi144.stdout:3/931: dread d1/d28/d54/d26/dc7/fa0 [0,4194304] 0 2024-07-26T11:05:33.180 INFO:tasks.workunit.client.0.smithi144.stdout:3/932: stat d1/d28/d54/d26/d37/da1/de1/cf2 0 2024-07-26T11:05:33.183 INFO:tasks.workunit.client.0.smithi144.stdout:3/933: creat d1/d5/d5e/dc5/f12b x:0 0 0 2024-07-26T11:05:33.187 INFO:tasks.workunit.client.0.smithi144.stdout:3/934: link d1/d28/d54/d26/f2d d1/d28/d54/d26/d30/d78/f12c 0 2024-07-26T11:05:33.187 INFO:tasks.workunit.client.0.smithi144.stdout:3/935: fdatasync d1/d28/d54/d26/d30/f106 0 2024-07-26T11:05:33.187 INFO:tasks.workunit.client.0.smithi144.stdout:3/936: chown d1/d28/d54/d26/d30/d78 0 1 2024-07-26T11:05:33.195 INFO:tasks.workunit.client.0.smithi144.stdout:3/937: link d1/d28/d54/d26/d64/d68/l120 d1/d28/d54/d26/d30/d79/dca/l12d 0 2024-07-26T11:05:33.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/938: rename d1/d28/d54/d26/d64/ded/l101 to d1/d28/d54/d102/l12e 0 2024-07-26T11:05:33.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/939: write d1/d5/d97/fa2 [379435,1612] 0 2024-07-26T11:05:33.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/940: chown d1/d5/d97/d60/lcd 415 1 2024-07-26T11:05:33.209 INFO:tasks.workunit.client.0.smithi144.stdout:3/941: mknod d1/d5/d5e/dc5/deb/c12f 0 2024-07-26T11:05:33.209 INFO:tasks.workunit.client.0.smithi144.stdout:3/942: write d1/d28/d54/d61/df4/f110 [191227,43189] 0 2024-07-26T11:05:33.216 INFO:tasks.workunit.client.0.smithi144.stdout:3/943: unlink d1/d28/d54/d26/d64/d89/c92 0 2024-07-26T11:05:33.220 INFO:tasks.workunit.client.0.smithi144.stdout:3/944: creat d1/d28/d54/d26/d64/da4/ddd/f130 x:0 0 0 2024-07-26T11:05:33.223 INFO:tasks.workunit.client.0.smithi144.stdout:3/945: rmdir d1/d28/d54/d26/d30/d78 39 2024-07-26T11:05:33.230 INFO:tasks.workunit.client.0.smithi144.stdout:3/946: creat d1/d28/d54/d26/d64/f131 x:0 0 0 2024-07-26T11:05:33.230 INFO:tasks.workunit.client.0.smithi144.stdout:3/947: chown d1/d28/d54/d26/f47 4394852 1 2024-07-26T11:05:33.247 INFO:tasks.workunit.client.0.smithi144.stdout:3/948: write d1/d28/d54/d26/d64/d68/d74/fd1 [494916,68766] 0 2024-07-26T11:05:33.271 INFO:tasks.workunit.client.0.smithi144.stdout:3/949: dread d1/d5/d97/fa2 [0,4194304] 0 2024-07-26T11:05:33.274 INFO:tasks.workunit.client.0.smithi144.stdout:3/950: mknod d1/d28/d54/d26/d30/d4d/c132 0 2024-07-26T11:05:33.276 INFO:tasks.workunit.client.0.smithi144.stdout:3/951: mkdir d1/d28/d54/d61/db9/dc9/d133 0 2024-07-26T11:05:33.277 INFO:tasks.workunit.client.0.smithi144.stdout:3/952: write d1/d28/d54/d26/d37/da1/fde [645382,80645] 0 2024-07-26T11:05:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:33 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:33 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:33.717 INFO:tasks.workunit.client.0.smithi144.stdout:3/953: sync 2024-07-26T11:05:33.717 INFO:tasks.workunit.client.0.smithi144.stdout:3/954: write d1/d28/d54/d61/df4/f110 [768333,663] 0 2024-07-26T11:05:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:33 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:33.755 INFO:tasks.workunit.client.0.smithi144.stdout:3/955: dread d1/d28/d54/d26/d64/fd6 [0,4194304] 0 2024-07-26T11:05:33.758 INFO:tasks.workunit.client.0.smithi144.stdout:3/956: symlink d1/d28/d54/d3d/dfc/l134 0 2024-07-26T11:05:33.830 INFO:tasks.workunit.client.0.smithi144.stdout:3/957: dwrite d1/d28/d54/d26/d64/da4/ff3 [4194304,4194304] 0 2024-07-26T11:05:33.830 INFO:tasks.workunit.client.0.smithi144.stdout:3/958: chown d1/d28/d54/d26/d37/lfe 2 1 2024-07-26T11:05:33.830 INFO:tasks.workunit.client.0.smithi144.stdout:3/959: read - d1/d28/d54/d26/d64/f131 zero size 2024-07-26T11:05:33.833 INFO:tasks.workunit.client.0.smithi144.stdout:3/960: symlink d1/d28/d6a/l135 0 2024-07-26T11:05:33.874 INFO:tasks.workunit.client.0.smithi144.stdout:3/961: dread d1/d28/f9d [0,4194304] 0 2024-07-26T11:05:33.978 INFO:tasks.workunit.client.0.smithi144.stdout:3/962: dwrite d1/d28/d54/d26/d64/d68/db5/fda [0,4194304] 0 2024-07-26T11:05:33.982 INFO:tasks.workunit.client.0.smithi144.stdout:3/963: rmdir d1/d28/d54/d26/d30 39 2024-07-26T11:05:33.982 INFO:tasks.workunit.client.0.smithi144.stdout:3/964: chown d1/d28/d54/d26/d37/d91 7873 1 2024-07-26T11:05:33.985 INFO:tasks.workunit.client.0.smithi144.stdout:3/965: getdents d1/d28/d54/d26/d30/d4d 0 2024-07-26T11:05:34.116 INFO:tasks.workunit.client.0.smithi144.stdout:3/966: sync 2024-07-26T11:05:34.117 INFO:tasks.workunit.client.0.smithi144.stdout:3/967: stat d1/d28/d54/d26/d64/d68/c71 0 2024-07-26T11:05:34.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/968: dwrite d1/d5/d97/d60/fb8 [0,4194304] 0 2024-07-26T11:05:34.204 INFO:tasks.workunit.client.0.smithi144.stdout:3/969: write d1/d28/d54/d26/d30/f106 [627005,38573] 0 2024-07-26T11:05:34.214 INFO:tasks.workunit.client.0.smithi144.stdout:3/970: unlink d1/d28/d54/d3d/c8f 0 2024-07-26T11:05:34.284 INFO:tasks.workunit.client.0.smithi144.stdout:3/971: dwrite d1/d28/d54/d26/d30/f5d [0,4194304] 0 2024-07-26T11:05:34.291 INFO:tasks.workunit.client.0.smithi144.stdout:3/972: creat d1/d28/d54/d61/db9/dc9/f136 x:0 0 0 2024-07-26T11:05:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:05:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-mon[36715]: pgmap v553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.6 MiB/s wr, 126 op/s 2024-07-26T11:05:34.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:05:34.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: *** Caught signal (Aborted) ** 2024-07-26T11:05:34.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: in thread 7f775ae62640 thread_name:ms_dispatch 2024-07-26T11:05:34.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:34.560 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f77602396f0] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f776028694c] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 3: raise() 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 4: abort() 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 5: /lib64/libc.so.6(+0x2871b) [0x7f776022371b] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 6: /lib64/libc.so.6(+0x37386) [0x7f7760232386] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5600564a26b9] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56005666080c] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5600565aecde] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560056570763] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560056571a1b] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560056555752] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5600564a3a56] 2024-07-26T11:05:34.561 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5600564a7637] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5600564a7f5e] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56005648b03a] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7760b06407] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 18: (DispatchQueue::entry()+0x60f) [0x7f7760b06cf9] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7760be35a1] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 20: (Thread::entry_wrapper()+0x43) [0x7f7760990cdd] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7760990cf9] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7760284c02] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7760309c40] 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.299+0000 7f775ae62640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: in thread 7f775ae62640 thread_name:ms_dispatch 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.562 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f77602396f0] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f776028694c] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 3: raise() 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 4: abort() 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 5: /lib64/libc.so.6(+0x2871b) [0x7f776022371b] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 6: /lib64/libc.so.6(+0x37386) [0x7f7760232386] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5600564a26b9] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56005666080c] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5600565aecde] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560056570763] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560056571a1b] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560056555752] 2024-07-26T11:05:34.563 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5600564a3a56] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5600564a7637] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5600564a7f5e] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56005648b03a] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7760b06407] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 18: (DispatchQueue::entry()+0x60f) [0x7f7760b06cf9] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7760be35a1] 2024-07-26T11:05:34.564 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 20: (Thread::entry_wrapper()+0x43) [0x7f7760990cdd] 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7760990cf9] 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7760284c02] 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7760309c40] 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 0> 2024-07-26T11:05:34.299+0000 7f775ae62640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: in thread 7f775ae62640 thread_name:ms_dispatch 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f77602396f0] 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f776028694c] 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 3: raise() 2024-07-26T11:05:34.565 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 4: abort() 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 5: /lib64/libc.so.6(+0x2871b) [0x7f776022371b] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 6: /lib64/libc.so.6(+0x37386) [0x7f7760232386] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5600564a26b9] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56005666080c] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5600565aecde] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560056570763] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560056571a1b] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560056555752] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5600564a3a56] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5600564a7637] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5600564a7f5e] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56005648b03a] 2024-07-26T11:05:34.566 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7760b06407] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 18: (DispatchQueue::entry()+0x60f) [0x7f7760b06cf9] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7760be35a1] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 20: (Thread::entry_wrapper()+0x43) [0x7f7760990cdd] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7760990cf9] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7760284c02] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7760309c40] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 0> 2024-07-26T11:05:34.299+0000 7f775ae62640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: in thread 7f775ae62640 thread_name:ms_dispatch 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f77602396f0] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f776028694c] 2024-07-26T11:05:34.567 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 3: raise() 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 4: abort() 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 5: /lib64/libc.so.6(+0x2871b) [0x7f776022371b] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 6: /lib64/libc.so.6(+0x37386) [0x7f7760232386] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5600564a26b9] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56005666080c] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5600565aecde] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560056570763] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560056571a1b] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560056555752] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5600564a3a56] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5600564a7637] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5600564a7f5e] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56005648b03a] 2024-07-26T11:05:34.568 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7760b06407] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 18: (DispatchQueue::entry()+0x60f) [0x7f7760b06cf9] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7760be35a1] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 20: (Thread::entry_wrapper()+0x43) [0x7f7760990cdd] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7760990cf9] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7760284c02] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7760309c40] 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[67875]: 2024-07-26T11:05:34.569 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:34 smithi160 podman[70578]: 2024-07-26 11:05:34.363185207 +0000 UTC m=+0.019448209 container died decd0104d935384312348803fcc9a89f27815def9e83d5379e3334612dece527 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:05:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:34 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:05:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:34 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:34 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:34 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:34 smithi183 ceph-mon[37273]: pgmap v553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.6 MiB/s wr, 126 op/s 2024-07-26T11:05:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:34 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:05:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:34 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:34 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:34 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:05:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:34 smithi144 ceph-mon[32129]: pgmap v553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.6 MiB/s wr, 126 op/s 2024-07-26T11:05:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:35.592 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:35 smithi160 podman[70578]: 2024-07-26 11:05:35.338492549 +0000 UTC m=+0.994755584 container remove decd0104d935384312348803fcc9a89f27815def9e83d5379e3334612dece527 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:05:35.592 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:35 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:05:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:36.059 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:35 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T11:05:36.059 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:35 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.b.service: Consumed 59.504s CPU time. 2024-07-26T11:05:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: pgmap v554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 87 op/s 2024-07-26T11:05:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: pgmap v554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 87 op/s 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:05:36.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:05:36.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:36.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:36.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: pgmap v554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 87 op/s 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:05:36.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:05:36.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:36.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:36.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:05:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:38 smithi183 ceph-mon[37273]: pgmap v555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 94 op/s 2024-07-26T11:05:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:38 smithi144 ceph-mon[32129]: pgmap v555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 94 op/s 2024-07-26T11:05:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:38 smithi160 ceph-mon[36715]: pgmap v555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 94 op/s 2024-07-26T11:05:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:40 smithi183 ceph-mon[37273]: pgmap v556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 99 op/s 2024-07-26T11:05:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:40 smithi144 ceph-mon[32129]: pgmap v556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 99 op/s 2024-07-26T11:05:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:40 smithi160 ceph-mon[36715]: pgmap v556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 99 op/s 2024-07-26T11:05:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:42 smithi183 ceph-mon[37273]: pgmap v557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 91 op/s 2024-07-26T11:05:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:42 smithi144 ceph-mon[32129]: pgmap v557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 91 op/s 2024-07-26T11:05:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:42 smithi160 ceph-mon[36715]: pgmap v557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 91 op/s 2024-07-26T11:05:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:44 smithi183 ceph-mon[37273]: pgmap v558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 79 op/s 2024-07-26T11:05:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:44 smithi144 ceph-mon[32129]: pgmap v558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 79 op/s 2024-07-26T11:05:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:44 smithi160 ceph-mon[36715]: pgmap v558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 79 op/s 2024-07-26T11:05:46.076 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:45 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:05:46.076 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:45 smithi160 systemd[1]: Stopped Ceph mds.b for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:05:46.076 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:45 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.b.service: Consumed 59.504s CPU time. 2024-07-26T11:05:46.076 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:45 smithi160 systemd[1]: Starting Ceph mds.b for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:05:46.076 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 podman[70805]: 2024-07-26 11:05:45.951966803 +0000 UTC m=+0.017407686 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:05:46.076 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 podman[70805]: 2024-07-26 11:05:46.074890906 +0000 UTC m=+0.140331786 container create c445898fed95ab8d1db89c47573a9d537b21be0d4236b4d8e9ffc4976fc36df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 -- 172.21.15.144:0/9830891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74b010fbe0 msgr2=0x7f74b0116410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 --2- 172.21.15.144:0/9830891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74b010fbe0 0x7f74b0116410 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009ca0 tx=0x7f74a802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 -- 172.21.15.144:0/9830891 shutdown_connections 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 --2- 172.21.15.144:0/9830891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74b010fbe0 0x7f74b0116410 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 --2- 172.21.15.144:0/9830891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010f290 0x7f74b010f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 --2- 172.21.15.144:0/9830891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b0109990 0x7f74b0109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 -- 172.21.15.144:0/9830891 >> 172.21.15.144:0/9830891 conn(0x7f74b00790b0 msgr2=0x7f74b0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 -- 172.21.15.144:0/9830891 shutdown_connections 2024-07-26T11:05:46.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.081+0000 7f74afe00640 1 -- 172.21.15.144:0/9830891 wait complete. 2024-07-26T11:05:46.085 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.082+0000 7f74afe00640 1 Processor -- start 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.082+0000 7f74afe00640 1 -- start start 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74afe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74b0109990 0x7f74b01a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b010f290 0x7f74b01a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74afe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010fbe0 0x7f74b01a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74afe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01198a0 con 0x7f74b010fbe0 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74afe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0119720 con 0x7f74b0109990 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74afe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0119a20 con 0x7f74b010f290 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010fbe0 0x7f74b01a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010fbe0 0x7f74b01a5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35216/0 (socket says 172.21.15.144:35216) 2024-07-26T11:05:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 learned_addr learned my addr 172.21.15.144:0/108036915 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74aea00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74b0109990 0x7f74b01a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.083+0000 7f74a7e00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b010f290 0x7f74b01a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74aea00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b010f290 msgr2=0x7f74b01a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74aea00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b010f290 0x7f74b01a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74aea00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010fbe0 msgr2=0x7f74b01a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74aea00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010fbe0 0x7f74b01a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74aea00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b01a6130 con 0x7f74b0109990 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b010fbe0 0x7f74b01a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74aea00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74b0109990 0x7f74b01a0200 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e9f0 tx=0x7f74a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a000cd80 con 0x7f74b0109990 2024-07-26T11:05:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74afe00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b01a6420 con 0x7f74b0109990 2024-07-26T11:05:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74a00035f0 con 0x7f74b0109990 2024-07-26T11:05:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0010690 con 0x7f74b0109990 2024-07-26T11:05:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.084+0000 7f74afe00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b01acbb0 con 0x7f74b0109990 2024-07-26T11:05:46.090 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 Processor -- start 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 -- start start 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f74a00037a0 con 0x7f74b0109990 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4411b0 0x563c6c7e4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4182f0 0x563c6c7e5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4204d0 0x563c6c7e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563c6c759270 con 0x563c6c4411b0 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563c6c748770 con 0x563c6c4204d0 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.088+0000 7f74bcdc6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c6c74c7b0 con 0x563c6c4182f0 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74af400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4411b0 0x563c6c7e4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74aea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4182f0 0x563c6c7e5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4204d0 0x563c6c7e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4204d0 0x563c6c7e56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:35090/0 (socket says 172.21.15.144:35090) 2024-07-26T11:05:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74a7e00640 1 -- 172.21.15.144:0/2863628809 learned_addr learned my addr 172.21.15.144:0/2863628809 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74aea00640 1 -- 172.21.15.144:0/2863628809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4204d0 msgr2=0x563c6c7e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74aea00640 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4204d0 0x563c6c7e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74aea00640 1 -- 172.21.15.144:0/2863628809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4411b0 msgr2=0x563c6c7e4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74aea00640 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4411b0 0x563c6c7e4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.089+0000 7f74aea00640 1 -- 172.21.15.144:0/2863628809 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6c4ac790 con 0x563c6c4182f0 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f74aea00640 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4182f0 0x563c6c7e5180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f74a00e1a40 tx=0x7f74a00eaef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf967ab202778c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f749fe00640 1 -- 172.21.15.144:0/2863628809 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00f3070 con 0x563c6c4182f0 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f74a00e46b0 con 0x7f74b0109990 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f749fe00640 1 -- 172.21.15.144:0/2863628809 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74a00aaa40 con 0x563c6c4182f0 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.090+0000 7f749fe00640 1 -- 172.21.15.144:0/2863628809 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00aa6a0 con 0x563c6c4182f0 2024-07-26T11:05:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 -- 172.21.15.144:0/2863628809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4182f0 msgr2=0x563c6c7e5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4182f0 0x563c6c7e5180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f74a00e1a40 tx=0x7f74a00eaef0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900b6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005e30 tx=0x7f74a8005c50 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 -- 172.21.15.144:0/2863628809 shutdown_connections 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4204d0 0x563c6c7e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4182f0 0x563c6c7e5180 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 --2- 172.21.15.144:0/2863628809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4411b0 0x563c6c7e4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 -- 172.21.15.144:0/2863628809 >> 172.21.15.144:0/2863628809 conn(0x563c6c435300 msgr2=0x563c6c7e2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 -- 172.21.15.144:0/2863628809 shutdown_connections 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.091+0000 7f74bcdc6740 1 -- 172.21.15.144:0/2863628809 wait complete. 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.092+0000 7f74bcdc6740 1 -- start start 2024-07-26T11:05:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.092+0000 7f74bcdc6740 1 Processor -- start 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4182f0 0x563c6c7f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4204d0 0x563c6c7f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4411b0 0x563c6c7fc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563c6c682ab0 con 0x563c6c4182f0 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563c6c6e5d50 con 0x563c6c4411b0 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c6c7f33c0 con 0x563c6c4204d0 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74bcdc6740 10 client.0 _ll_register_callbacks cb 0x563c6c458a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4182f0 0x563c6c7f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4182f0 0x563c6c7f07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35234/0 (socket says 172.21.15.144:35234) 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 -- 172.21.15.144:0/3349571383 learned_addr learned my addr 172.21.15.144:0/3349571383 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74af400640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4411b0 0x563c6c7fc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4204d0 0x563c6c7f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4204d0 msgr2=0x563c6c7f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c6c4204d0 0x563c6c7f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4411b0 msgr2=0x563c6c7fc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c6c4411b0 0x563c6c7fc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.093+0000 7f74aea00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6c525620 con 0x563c6c4182f0 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f74aea00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c6c4182f0 0x563c6c7f07f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f74a000cfd0 tx=0x7f74a00ea300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b2145f9be81f94 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f749cc00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:05:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00f3070 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9394+0+0 (secure 0 0 0) 0x7f74a0009310 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f74bcdc6740 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a00aaa40 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74a00aa6a0 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f74bcdc6740 10 client.15858 fetch_fsmap learned FSMap version 27 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f74bcdc6740 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c6c7ff400 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.094+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74a00aaa40 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f74a00a71c0 con 0x563c6c4182f0 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74bcdc6740 10 client.15858 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74bcdc6740 10 client.15858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74bcdc6740 20 client.15858 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74bcdc6740 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74bcdc6740 4 client.15858 mds_command: new command op to 24728 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74bcdc6740 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c6c75b1c0 con 0x563c6c3ffd80 2024-07-26T11:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f749cc00640 1 client.15858 ms_handle_refused on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T11:05:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.297+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.297+0000 7f749cc00640 1 client.15858 ms_handle_refused on v2:172.21.15.160:6838/1074049609 2024-07-26T11:05:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.297+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T11:05:46.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:46 smithi160 ceph-mon[36715]: pgmap v559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 74 op/s 2024-07-26T11:05:46.607 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 podman[70805]: 2024-07-26 11:05:46.327999678 +0000 UTC m=+0.393440558 container init c445898fed95ab8d1db89c47573a9d537b21be0d4236b4d8e9ffc4976fc36df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T11:05:46.607 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 podman[70805]: 2024-07-26 11:05:46.331084936 +0000 UTC m=+0.396525824 container start c445898fed95ab8d1db89c47573a9d537b21be0d4236b4d8e9ffc4976fc36df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T11:05:46.607 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-b[70820]: starting mds.b at 2024-07-26T11:05:46.607 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 bash[70805]: c445898fed95ab8d1db89c47573a9d537b21be0d4236b4d8e9ffc4976fc36df9 2024-07-26T11:05:46.608 INFO:journalctl@ceph.mds.b.smithi160.stdout:Jul 26 11:05:46 smithi160 systemd[1]: Started Ceph mds.b for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:05:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:46 smithi183 ceph-mon[37273]: pgmap v559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 74 op/s 2024-07-26T11:05:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.699+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:46.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.700+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.700+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.700+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:46.700+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T11:05:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:46 smithi144 ceph-mon[32129]: pgmap v559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 74 op/s 2024-07-26T11:05:47.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:47.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:47.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:47.504+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:47.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:47.504+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:47.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:47.504+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:47.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:47.504+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: daemon mds.b restarted 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6838/1392250925,v1:172.21.15.160:6839/1392250925] up:boot 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {1=a=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:05:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:47 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:replay,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: daemon mds.b restarted 2024-07-26T11:05:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:05:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:05:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T11:05:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6838/1392250925,v1:172.21.15.160:6839/1392250925] up:boot 2024-07-26T11:05:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T11:05:47.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {1=a=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:05:47.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:47 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:replay,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: daemon mds.b restarted 2024-07-26T11:05:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:05:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:05:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T11:05:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6838/1392250925,v1:172.21.15.160:6839/1392250925] up:boot 2024-07-26T11:05:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T11:05:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {1=a=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:05:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:47 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:replay,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:48 smithi144 ceph-mon[32129]: pgmap v561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 18 op/s 2024-07-26T11:05:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:resolve 2024-07-26T11:05:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:48 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:resolve,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:48 smithi160 ceph-mon[36715]: pgmap v561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 18 op/s 2024-07-26T11:05:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:resolve 2024-07-26T11:05:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:48 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:resolve,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:48 smithi183 ceph-mon[37273]: pgmap v561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 18 op/s 2024-07-26T11:05:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:resolve 2024-07-26T11:05:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:48 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:resolve,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:49.106+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:49.107+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:49.107+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:49.107+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:49.107+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T11:05:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:reconnect 2024-07-26T11:05:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:49 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:reconnect,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:49 smithi144 ceph-mon[32129]: reconnect by client.25502 172.21.15.144:0/3053566356 after 0.00999985 2024-07-26T11:05:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:reconnect 2024-07-26T11:05:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:49 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:reconnect,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:49 smithi160 ceph-mon[36715]: reconnect by client.25502 172.21.15.144:0/3053566356 after 0.00999985 2024-07-26T11:05:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:reconnect 2024-07-26T11:05:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:49 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:reconnect,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:49 smithi183 ceph-mon[37273]: reconnect by client.25502 172.21.15.144:0/3053566356 after 0.00999985 2024-07-26T11:05:50.427 INFO:tasks.workunit.client.0.smithi144.stdout:3/973: rename d1/d5/d97 to d1/d5/d5e/dc5/d137 0 2024-07-26T11:05:50.441 INFO:tasks.workunit.client.0.smithi144.stdout:3/974: fsync d1/d28/d54/d26/f33 0 2024-07-26T11:05:50.444 INFO:tasks.workunit.client.0.smithi144.stdout:3/975: fsync d1/d28/d54/d26/f27 0 2024-07-26T11:05:50.452 INFO:tasks.workunit.client.0.smithi144.stdout:3/976: truncate d1/d28/d54/d26/d64/d68/d74/fd1 1223049 0 2024-07-26T11:05:50.458 INFO:tasks.workunit.client.0.smithi144.stdout:3/977: write d1/d5/d5e/dc5/d137/d60/f111 [422877,56529] 0 2024-07-26T11:05:50.490 INFO:tasks.workunit.client.0.smithi144.stdout:3/978: dread d1/f14 [0,4194304] 0 2024-07-26T11:05:50.492 INFO:tasks.workunit.client.0.smithi144.stdout:3/979: readlink d1/l23 0 2024-07-26T11:05:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.498992 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: pgmap v562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:rejoin 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:rejoin,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:05:50.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:50 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:05:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.498992 2024-07-26T11:05:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: pgmap v562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-07-26T11:05:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:rejoin 2024-07-26T11:05:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:rejoin,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T11:05:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:05:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:05:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:50 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.498992 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: pgmap v562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:rejoin 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:rejoin,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:05:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:50 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:05:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:51 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:51.767 INFO:tasks.workunit.client.0.smithi144.stdout:3/980: sync 2024-07-26T11:05:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:51 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:51 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:51.900 INFO:tasks.workunit.client.0.smithi144.stdout:3/981: sync 2024-07-26T11:05:52.003 INFO:tasks.workunit.client.0.smithi144.stdout:3/982: dwrite d1/d28/d54/d26/d37/fbb [0,4194304] 0 2024-07-26T11:05:52.014 INFO:tasks.workunit.client.0.smithi144.stdout:3/983: truncate d1/d28/d54/d26/d30/d79/fe4 1744201 0 2024-07-26T11:05:52.022 INFO:tasks.workunit.client.0.smithi144.stdout:3/984: fsync d1/d28/d54/d3d/d42/fd4 0 2024-07-26T11:05:52.074 INFO:tasks.workunit.client.0.smithi144.stdout:3/985: write d1/d28/d54/d26/f7c [3392330,78955] 0 2024-07-26T11:05:52.081 INFO:tasks.workunit.client.0.smithi144.stdout:3/986: mknod d1/d28/d54/d26/d37/d103/c138 0 2024-07-26T11:05:52.090 INFO:tasks.workunit.client.0.smithi144.stdout:3/987: creat d1/d28/d54/d61/db9/dc9/f139 x:0 0 0 2024-07-26T11:05:52.098 INFO:tasks.workunit.client.0.smithi144.stdout:3/988: mknod d1/d28/d54/d26/d64/ded/d10d/c13a 0 2024-07-26T11:05:52.104 INFO:tasks.workunit.client.0.smithi144.stdout:3/989: symlink d1/d28/d54/d26/d64/dd5/l13b 0 2024-07-26T11:05:52.107 INFO:tasks.workunit.client.0.smithi144.stdout:3/990: fdatasync d1/d28/d54/d26/d64/ded/d10d/f11c 0 2024-07-26T11:05:52.110 INFO:tasks.workunit.client.0.smithi144.stdout:3/991: read - d1/d28/d54/d26/d64/f131 zero size 2024-07-26T11:05:52.116 INFO:tasks.workunit.client.0.smithi144.stdout:3/992: mkdir d1/d28/d54/d26/d64/dd5/d13c 0 2024-07-26T11:05:52.124 INFO:tasks.workunit.client.0.smithi144.stdout:3/993: symlink d1/d5/d5e/d123/l13d 0 2024-07-26T11:05:52.128 INFO:tasks.workunit.client.0.smithi144.stdout:3/994: fsync d1/d28/d54/d61/db9/dc9/f136 0 2024-07-26T11:05:52.139 INFO:tasks.workunit.client.0.smithi144.stdout:3/995: getdents d1/d28/d54/d26/d37/da1/dea 0 2024-07-26T11:05:52.141 INFO:tasks.workunit.client.0.smithi144.stdout:3/996: chown d1/d28/d54/d26/d64/d68/f7d 242980585 1 2024-07-26T11:05:52.166 INFO:tasks.workunit.client.0.smithi144.stdout:3/997: dread d1/d28/d54/d26/d30/d79/fbc [0,4194304] 0 2024-07-26T11:05:52.174 INFO:tasks.workunit.client.0.smithi144.stdout:3/998: mkdir d1/d5/d5e/dc5/d137/d48/d13e 0 2024-07-26T11:05:52.181 INFO:tasks.workunit.client.0.smithi144.stdout:3/999: write d1/d5/fff [935396,57826] 0 2024-07-26T11:05:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:+ rm -rf -- ./tmp.Q0elH8bjWH 2024-07-26T11:05:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:52.310+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:52.311+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:52.312+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:52.312+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:52.312+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T11:05:52.448 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:05:52 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:05:52.149+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T11:05:52.448 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:05:52 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:05:52.149+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:05:52.448 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:05:52 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:05:52.149+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T11:05:52.569 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:05:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:52 smithi144 ceph-mon[32129]: pgmap v563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 225 KiB/s wr, 13 op/s 2024-07-26T11:05:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:52 smithi160 ceph-mon[36715]: pgmap v563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 225 KiB/s wr, 13 op/s 2024-07-26T11:05:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:52 smithi183 ceph-mon[37273]: pgmap v563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 225 KiB/s wr, 13 op/s 2024-07-26T11:05:53.048 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:54 smithi144 ceph-mon[32129]: pgmap v564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 920 KiB/s wr, 21 op/s 2024-07-26T11:05:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:54 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:54 smithi160 ceph-mon[36715]: pgmap v564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 920 KiB/s wr, 21 op/s 2024-07-26T11:05:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:54 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:54 smithi183 ceph-mon[37273]: pgmap v564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 920 KiB/s wr, 21 op/s 2024-07-26T11:05:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:54 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.314+0000 7f2071400640 1 -- 172.21.15.144:0/918729799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10af30 msgr2=0x7f206c10b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.314+0000 7f2071400640 1 --2- 172.21.15.144:0/918729799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10af30 0x7f206c10b340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2058009900 tx=0x7f205802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 -- 172.21.15.144:0/918729799 shutdown_connections 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 --2- 172.21.15.144:0/918729799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c10b880 0x7f206c1120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 --2- 172.21.15.144:0/918729799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10af30 0x7f206c10b340 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 --2- 172.21.15.144:0/918729799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c1055f0 0x7f206c105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 -- 172.21.15.144:0/918729799 >> 172.21.15.144:0/918729799 conn(0x7f206c101060 msgr2=0x7f206c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 -- 172.21.15.144:0/918729799 shutdown_connections 2024-07-26T11:05:55.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.315+0000 7f2071400640 1 -- 172.21.15.144:0/918729799 wait complete. 2024-07-26T11:05:55.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.316+0000 7f2071400640 1 Processor -- start 2024-07-26T11:05:55.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.316+0000 7f2071400640 1 -- start start 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2071400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c1055f0 0x7f206c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2071400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c10af30 0x7f206c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2071400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10b880 0x7f206c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2071400640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f206c115560 con 0x7f206c10b880 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2071400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1153e0 con 0x7f206c10af30 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2071400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1156e0 con 0x7f206c1055f0 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f206be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c1055f0 0x7f206c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f2070a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10b880 0x7f206c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f206be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c1055f0 0x7f206c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56648/0 (socket says 172.21.15.144:56648) 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f206be00640 1 -- 172.21.15.144:0/584893955 learned_addr learned my addr 172.21.15.144:0/584893955 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.317+0000 7f206b400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c10af30 0x7f206c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f206b400640 1 -- 172.21.15.144:0/584893955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c1055f0 msgr2=0x7f206c19bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f206b400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c1055f0 0x7f206c19bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f206b400640 1 -- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10b880 msgr2=0x7f206c1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f206b400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10b880 0x7f206c1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f206b400640 1 -- 172.21.15.144:0/584893955 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c1a1e80 con 0x7f206c10af30 2024-07-26T11:05:55.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f2070a00640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10b880 0x7f206c1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:05:55.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f206b400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c10af30 0x7f206c19c4b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2058009230 tx=0x7f2058003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:55.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.318+0000 7f2068c00640 1 -- 172.21.15.144:0/584893955 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205803f070 con 0x7f206c10af30 2024-07-26T11:05:55.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.319+0000 7f2068c00640 1 -- 172.21.15.144:0/584893955 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2058031b50 con 0x7f206c10af30 2024-07-26T11:05:55.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.319+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c1a2110 con 0x7f206c10af30 2024-07-26T11:05:55.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.319+0000 7f2068c00640 1 -- 172.21.15.144:0/584893955 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058035030 con 0x7f206c10af30 2024-07-26T11:05:55.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.319+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c1a88d0 con 0x7f206c10af30 2024-07-26T11:05:55.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.322+0000 7f20796ca740 1 Processor -- start 2024-07-26T11:05:55.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 -- start start 2024-07-26T11:05:55.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 0x56339c8d3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c50d390 0x56339c8d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56339c849b90 con 0x56339c52e070 2024-07-26T11:05:55.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56339c839090 con 0x56339c50d390 2024-07-26T11:05:55.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.323+0000 7f20796ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56339c83d0d0 con 0x56339c5051b0 2024-07-26T11:05:55.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 0x56339c8d3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f2070a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 0x56339c8d3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56662/0 (socket says 172.21.15.144:56662) 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 -- 172.21.15.144:0/112494192 learned_addr learned my addr 172.21.15.144:0/112494192 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 -- 172.21.15.144:0/112494192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c50d390 msgr2=0x56339c8d3a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c50d390 0x56339c8d3a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 -- 172.21.15.144:0/112494192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 msgr2=0x56339c8d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f206be00640 1 -- 172.21.15.144:0/112494192 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56339c599650 con 0x56339c5051b0 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.324+0000 7f2068c00640 1 -- 172.21.15.144:0/584893955 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f2058042070 con 0x7f206c10af30 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.325+0000 7f206be00640 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 0x56339c8d3510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f205c009e10 tx=0x7f205c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98bf6aca1b3550f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:55.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.325+0000 7f2061200640 1 -- 172.21.15.144:0/112494192 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c01e070 con 0x56339c5051b0 2024-07-26T11:05:55.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.325+0000 7f2061200640 1 -- 172.21.15.144:0/112494192 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c019ae0 con 0x56339c5051b0 2024-07-26T11:05:55.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.325+0000 7f2061200640 1 -- 172.21.15.144:0/112494192 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c00fa90 con 0x56339c5051b0 2024-07-26T11:05:55.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 -- 172.21.15.144:0/112494192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 msgr2=0x56339c8d3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 0x56339c8d3510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f205c009e10 tx=0x7f205c011810 comp rx=0 tx=0).stop 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 -- 172.21.15.144:0/112494192 shutdown_connections 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c50d390 0x56339c8d3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c5051b0 0x56339c8d3510 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 --2- 172.21.15.144:0/112494192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 -- 172.21.15.144:0/112494192 >> 172.21.15.144:0/112494192 conn(0x56339c522460 msgr2=0x56339c8d0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.326+0000 7f20796ca740 1 -- 172.21.15.144:0/112494192 shutdown_connections 2024-07-26T11:05:55.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.327+0000 7f20796ca740 1 -- 172.21.15.144:0/112494192 wait complete. 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.327+0000 7f2068c00640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f20500b3cd0 0x7f20500b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.327+0000 7f20796ca740 1 -- start start 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.327+0000 7f2068c00640 1 -- 172.21.15.144:0/584893955 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7f2058104610 con 0x7f206c10af30 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.327+0000 7f20796ca740 1 Processor -- start 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f2070a00640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f20500b3cd0 0x7f20500b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f20796ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f20796ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8deae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f20796ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8df020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f20796ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56339c69c700 con 0x56339c52e070 2024-07-26T11:05:55.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f20796ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56339c7d5660 con 0x56339c5051b0 2024-07-26T11:05:55.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.328+0000 7f20796ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56339c8e17b0 con 0x56339c50d390 2024-07-26T11:05:55.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f20796ca740 10 client.0 _ll_register_callbacks cb 0x56339c7e5110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f2070a00640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f20500b3cd0 0x7f20500b61e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2064005fd0 tx=0x7f2064009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8deae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8de5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:48294/0 (socket says 172.21.15.144:48294) 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 -- 172.21.15.144:0/3082980379 learned_addr learned my addr 172.21.15.144:0/3082980379 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8deae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56672/0 (socket says 172.21.15.144:56672) 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f2070a00640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8df020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 -- 172.21.15.144:0/3082980379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 msgr2=0x56339c8deae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8deae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 -- 172.21.15.144:0/3082980379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 msgr2=0x56339c8df020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8df020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.329+0000 7f206be00640 1 -- 172.21.15.144:0/3082980379 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56339c7ec800 con 0x56339c5051b0 2024-07-26T11:05:55.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.330+0000 7f206b400640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8deae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:55.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.330+0000 7f2070a00640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8df020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:05:55.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.330+0000 7f2055600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:05:55.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.330+0000 7f206be00640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8de5a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f205c01afd0 tx=0x7f205c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bb6ec18fdd70a4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.330+0000 7f2055600640 1 -- 172.21.15.144:0/3082980379 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c026020 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f205c019ae0 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f2055600640 1 -- 172.21.15.144:0/3082980379 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7f205c004440 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f2055600640 1 -- 172.21.15.144:0/3082980379 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f205c01e040 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f20796ca740 10 client.26021 fetch_fsmap learned FSMap version 34 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56339c8ed3e0 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f2055600640 1 -- 172.21.15.144:0/3082980379 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c019ae0 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f2055600640 1 -- 172.21.15.144:0/3082980379 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f205c00f810 con 0x56339c5051b0 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.331+0000 7f20796ca740 10 client.26021 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.332+0000 7f20796ca740 10 client.26021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.332+0000 7f20796ca740 20 client.26021 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:55.335 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.332+0000 7f20796ca740 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4ec9f0 0x56339c8f0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.332+0000 7f20796ca740 4 client.26021 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:05:55.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.332+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56339c84bae0 con 0x56339c4ec9f0 2024-07-26T11:05:55.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.332+0000 7f206b400640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4ec9f0 0x56339c8f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.333+0000 7f206b400640 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4ec9f0 0x56339c8f0740 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a37d940685293c1 server_cookie=c5db27cad6b97b79 in_seq=0 out_seq=0 2024-07-26T11:05:55.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.333+0000 7f2055600640 10 client.26021 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f2055600640 1 -- 172.21.15.144:0/3082980379 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56339c84bae0 con 0x56339c4ec9f0 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f2055600640 10 client.26021 handle_command_reply: tid=0 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 client.26021 shutdown 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 msgr2=0x56339c8de5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8de5a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f205c01afd0 tx=0x7f205c005dc0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 shutdown_connections 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8df020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8deae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4ec9f0 0x56339c8f0740 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 --2- 172.21.15.144:0/3082980379 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8de5a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f2055600640 0 client.26021 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:05:55.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.334+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 >> 172.21.15.144:0/3082980379 conn(0x56339c522460 msgr2=0x56339c8d61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.335+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 shutdown_connections 2024-07-26T11:05:55.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.335+0000 7f20796ca740 1 -- 172.21.15.144:0/3082980379 wait complete. 2024-07-26T11:05:55.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.335+0000 7f20796ca740 20 client.26021 trim_cache size 0 max 16384 2024-07-26T11:05:55.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.369+0000 7f20796ca740 1 Processor -- start 2024-07-26T11:05:55.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.369+0000 7f20796ca740 1 -- start start 2024-07-26T11:05:55.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f20796ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f20796ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8dea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f20796ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8defc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f20796ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1b5140 con 0x56339c52e070 2024-07-26T11:05:55.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f20796ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1b29e0 con 0x56339c5051b0 2024-07-26T11:05:55.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f20796ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1b0830 con 0x56339c50d390 2024-07-26T11:05:55.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f206b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f206be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8d49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f206b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8dea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56684/0 (socket says 172.21.15.144:56684) 2024-07-26T11:05:55.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.370+0000 7f206b400640 1 -- 172.21.15.144:0/3281348110 learned_addr learned my addr 172.21.15.144:0/3281348110 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f206be00640 1 -- 172.21.15.144:0/3281348110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 msgr2=0x56339c8dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f206be00640 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8dea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f206be00640 1 -- 172.21.15.144:0/3281348110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 msgr2=0x56339c8defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f2070a00640 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f206be00640 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8defc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f206be00640 1 -- 172.21.15.144:0/3281348110 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56339c8dcac0 con 0x56339c5051b0 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f206b400640 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.371+0000 7f2070a00640 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.372+0000 7f206be00640 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8d49c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f205c009d60 tx=0x7f205c009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc50e1f07324de6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:55.374 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.372+0000 7f2057e00640 1 -- 172.21.15.144:0/3281348110 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c026070 con 0x56339c5051b0 2024-07-26T11:05:55.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.372+0000 7f2057e00640 1 -- 172.21.15.144:0/3281348110 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c004240 con 0x56339c5051b0 2024-07-26T11:05:55.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.372+0000 7f2057e00640 1 -- 172.21.15.144:0/3281348110 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c003ac0 con 0x56339c5051b0 2024-07-26T11:05:55.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.373+0000 7f20796ca740 1 -- 172.21.15.144:0/3281348110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 msgr2=0x56339c8d49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.373+0000 7f20796ca740 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8d49c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f205c009d60 tx=0x7f205c009ad0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 -- 172.21.15.144:0/3281348110 shutdown_connections 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c52e070 0x56339c8defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c50d390 0x56339c8dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 --2- 172.21.15.144:0/3281348110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8d49c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 -- 172.21.15.144:0/3281348110 >> 172.21.15.144:0/3281348110 conn(0x56339c522460 msgr2=0x56339c8de370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 -- 172.21.15.144:0/3281348110 shutdown_connections 2024-07-26T11:05:55.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 -- 172.21.15.144:0/3281348110 wait complete. 2024-07-26T11:05:55.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.374+0000 7f20796ca740 1 -- start start 2024-07-26T11:05:55.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.375+0000 7f20796ca740 1 Processor -- start 2024-07-26T11:05:55.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.375+0000 7f20796ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.375+0000 7f20796ca740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c50d390 0x56339c8e4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.375+0000 7f20796ca740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c52e070 0x56339c956d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f20796ca740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1b0ea0 con 0x56339c50d390 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f20796ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1b2310 con 0x56339c5051b0 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f20796ca740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1b1570 con 0x56339c52e070 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f20796ca740 10 client.0 _ll_register_callbacks cb 0x7f205c009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f2070a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c52e070 0x56339c956d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f2070a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c52e070 0x56339c956d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56690/0 (socket says 172.21.15.144:56690) 2024-07-26T11:05:55.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f2070a00640 1 -- 172.21.15.144:0/3487078297 learned_addr learned my addr 172.21.15.144:0/3487078297 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:55.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206b400640 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c50d390 0x56339c8e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206be00640 1 -- 172.21.15.144:0/3487078297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c52e070 msgr2=0x56339c956d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206be00640 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c52e070 0x56339c956d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206be00640 1 -- 172.21.15.144:0/3487078297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c50d390 msgr2=0x56339c8e4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206be00640 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c50d390 0x56339c8e4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.376+0000 7f206be00640 1 -- 172.21.15.144:0/3487078297 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56339c9572b0 con 0x56339c5051b0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f206be00640 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8e3ff0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f205c009e10 tx=0x7f205c00f620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f47005bbfb54a936 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f2055600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f2055600640 1 -- 172.21.15.144:0/3487078297 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c042070 con 0x56339c5051b0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f2055600640 1 -- 172.21.15.144:0/3487078297 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7f205c0037f0 con 0x56339c5051b0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f205c004240 con 0x56339c5051b0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f2055600640 1 -- 172.21.15.144:0/3487078297 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f205c026040 con 0x56339c5051b0 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f20796ca740 10 client.26027 fetch_fsmap learned FSMap version 34 2024-07-26T11:05:55.380 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.377+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56339c958480 con 0x56339c5051b0 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f2055600640 1 -- 172.21.15.144:0/3487078297 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c004240 con 0x56339c5051b0 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f2055600640 1 -- 172.21.15.144:0/3487078297 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f205c0477b0 con 0x56339c5051b0 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f20796ca740 10 client.26027 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f20796ca740 10 client.26027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f20796ca740 20 client.26027 populate_metadata read hostname 'smithi144' 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f20796ca740 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4c1da0 0x56339c95aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f20796ca740 4 client.26027 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:05:55.381 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56339c847580 con 0x56339c4c1da0 2024-07-26T11:05:55.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.378+0000 7f206b400640 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4c1da0 0x56339c95aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:55.382 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.380+0000 7f2055600640 10 client.26027 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.380+0000 7f206b400640 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4c1da0 0x56339c95aa00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd1fe2a31c344f03 server_cookie=1058f140bd2a1090 in_seq=0 out_seq=0 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.380+0000 7f2055600640 1 -- 172.21.15.144:0/3487078297 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56339c847580 con 0x56339c4c1da0 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.380+0000 7f2055600640 10 client.26027 handle_command_reply: tid=0 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.380+0000 7f20796ca740 1 client.26027 shutdown 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 msgr2=0x56339c8e3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8e3ff0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f205c009e10 tx=0x7f205c00f620 comp rx=0 tx=0).stop 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 shutdown_connections 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56339c52e070 0x56339c956d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56339c50d390 0x56339c8e4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56339c4c1da0 0x56339c95aa00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 --2- 172.21.15.144:0/3487078297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56339c5051b0 0x56339c8e3ff0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.383 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 >> 172.21.15.144:0/3487078297 conn(0x56339c522460 msgr2=0x56339c8d4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f2055600640 0 client.26027 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:05:55.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.381+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 shutdown_connections 2024-07-26T11:05:55.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.382+0000 7f20796ca740 1 -- 172.21.15.144:0/3487078297 wait complete. 2024-07-26T11:05:55.384 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.382+0000 7f20796ca740 20 client.26027 trim_cache size 0 max 16384 2024-07-26T11:05:55.385 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:05:55.385 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "no active scrubs running", 2024-07-26T11:05:55.385 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": {} 2024-07-26T11:05:55.385 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:05:55.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.382+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f20500b3cd0 msgr2=0x7f20500b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.382+0000 7f2071400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f20500b3cd0 0x7f20500b61e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2064005fd0 tx=0x7f2064009500 comp rx=0 tx=0).stop 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.382+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c10af30 msgr2=0x7f206c19c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.382+0000 7f2071400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c10af30 0x7f206c19c4b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2058009230 tx=0x7f2058003b90 comp rx=0 tx=0).stop 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 shutdown_connections 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f20500b3cd0 0x7f20500b61e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f206c10b880 0x7f206c1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f206c10af30 0x7f206c19c4b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 --2- 172.21.15.144:0/584893955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c1055f0 0x7f206c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 >> 172.21.15.144:0/584893955 conn(0x7f206c101060 msgr2=0x7f206c103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 shutdown_connections 2024-07-26T11:05:55.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:55.383+0000 7f2071400640 1 -- 172.21.15.144:0/584893955 wait complete. 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: *** Caught signal (Aborted) ** 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: in thread 7f782ea00640 thread_name:ms_dispatch 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7834c3e6f0] 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7834c8b94c] 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 3: raise() 2024-07-26T11:05:56.097 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 4: abort() 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7834c2871b] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 6: /lib64/libc.so.6(+0x37386) [0x7f7834c37386] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a428ffe6b9] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a4291bc80c] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a4290fc2ca] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a4290cc753] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a4290cda1b] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a4290b1752] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a428fffa56] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a429003637] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a429003f5e] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a428fe703a] 2024-07-26T11:05:56.098 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f783579b407] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 18: (DispatchQueue::entry()+0x60f) [0x7f783579bcf9] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78358785a1] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 20: (Thread::entry_wrapper()+0x43) [0x7f7835625cdd] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7835625cf9] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7834c89c02] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7834d0ec40] 2024-07-26T11:05:56.099 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.035+0000 7f782ea00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: in thread 7f782ea00640 thread_name:ms_dispatch 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7834c3e6f0] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7834c8b94c] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 3: raise() 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 4: abort() 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7834c2871b] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 6: /lib64/libc.so.6(+0x37386) [0x7f7834c37386] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a428ffe6b9] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a4291bc80c] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a4290fc2ca] 2024-07-26T11:05:56.100 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a4290cc753] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a4290cda1b] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a4290b1752] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a428fffa56] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a429003637] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a429003f5e] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a428fe703a] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f783579b407] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 18: (DispatchQueue::entry()+0x60f) [0x7f783579bcf9] 2024-07-26T11:05:56.101 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78358785a1] 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 20: (Thread::entry_wrapper()+0x43) [0x7f7835625cdd] 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7835625cf9] 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7834c89c02] 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7834d0ec40] 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 0> 2024-07-26T11:05:56.035+0000 7f782ea00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:05:56.102 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: in thread 7f782ea00640 thread_name:ms_dispatch 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7834c3e6f0] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7834c8b94c] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 3: raise() 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 4: abort() 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7834c2871b] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 6: /lib64/libc.so.6(+0x37386) [0x7f7834c37386] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a428ffe6b9] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a4291bc80c] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a4290fc2ca] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a4290cc753] 2024-07-26T11:05:56.103 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a4290cda1b] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a4290b1752] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a428fffa56] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a429003637] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a429003f5e] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a428fe703a] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f783579b407] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 18: (DispatchQueue::entry()+0x60f) [0x7f783579bcf9] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78358785a1] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 20: (Thread::entry_wrapper()+0x43) [0x7f7835625cdd] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7835625cf9] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7834c89c02] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7834d0ec40] 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:05:56.104 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 0> 2024-07-26T11:05:56.035+0000 7f782ea00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:05:56.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: in thread 7f782ea00640 thread_name:ms_dispatch 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7834c3e6f0] 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7834c8b94c] 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 3: raise() 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 4: abort() 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7834c2871b] 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 6: /lib64/libc.so.6(+0x37386) [0x7f7834c37386] 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a428ffe6b9] 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a4291bc80c] 2024-07-26T11:05:56.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a4290fc2ca] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a4290cc753] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a4290cda1b] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a4290b1752] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a428fffa56] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a429003637] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a429003f5e] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a428fe703a] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f783579b407] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 18: (DispatchQueue::entry()+0x60f) [0x7f783579bcf9] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78358785a1] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 20: (Thread::entry_wrapper()+0x43) [0x7f7835625cdd] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7835625cf9] 2024-07-26T11:05:56.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7834c89c02] 2024-07-26T11:05:56.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7834d0ec40] 2024-07-26T11:05:56.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:05:56.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[78529]: 2024-07-26T11:05:56.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:56 smithi144 podman[101780]: 2024-07-26 11:05:56.21285389 +0000 UTC m=+0.036917684 container died c881e9e79670e090912a5655ddb577ad4a9f0c1a1be0089e89c58e8a52896662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T11:05:56.241 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:05:56.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:05:56.464 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:56 smithi144 ceph-mon[32129]: pgmap v565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 40 op/s 2024-07-26T11:05:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:56 smithi160 ceph-mon[36715]: pgmap v565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 40 op/s 2024-07-26T11:05:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:56 smithi183 ceph-mon[37273]: pgmap v565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 40 op/s 2024-07-26T11:05:57.094 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:57 smithi144 podman[101780]: 2024-07-26 11:05:57.011611859 +0000 UTC m=+0.835675679 container remove c881e9e79670e090912a5655ddb577ad4a9f0c1a1be0089e89c58e8a52896662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T11:05:57.094 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:57 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:05:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:57.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:57 smithi144 podman[101834]: 2024-07-26 11:05:57.707914192 +0000 UTC m=+0.634903483 container died 6db35845bf8f802e37af95ab0cbf62998ee10f57a06e906a4ec6091f0fe34ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=quizzical_dhawan, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T11:05:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:58.718+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:58.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:58.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:58.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:05:58.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T11:05:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:58 smithi160 ceph-mon[36715]: pgmap v566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 54 op/s 2024-07-26T11:05:58.822 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:58 smithi144 ceph-mon[32129]: pgmap v566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 54 op/s 2024-07-26T11:05:58.823 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:58 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:05:58.823 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:05:58 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 22.182s CPU time. 2024-07-26T11:05:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:58 smithi183 ceph-mon[37273]: pgmap v566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 54 op/s 2024-07-26T11:05:59.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 -- 172.21.15.144:0/4134638700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 msgr2=0x7fc630109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:59.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 --2- 172.21.15.144:0/4134638700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630109130 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc618009900 tx=0x7fc61802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 -- 172.21.15.144:0/4134638700 shutdown_connections 2024-07-26T11:05:59.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 --2- 172.21.15.144:0/4134638700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc630109670 0x7fc63010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 --2- 172.21.15.144:0/4134638700 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630109130 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 --2- 172.21.15.144:0/4134638700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 0x7fc6301037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 -- 172.21.15.144:0/4134638700 >> 172.21.15.144:0/4134638700 conn(0x7fc6300fee70 msgr2=0x7fc6301012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:59.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 -- 172.21.15.144:0/4134638700 shutdown_connections 2024-07-26T11:05:59.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.395+0000 7fc635c00640 1 -- 172.21.15.144:0/4134638700 wait complete. 2024-07-26T11:05:59.399 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.396+0000 7fc635c00640 1 Processor -- start 2024-07-26T11:05:59.399 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.396+0000 7fc635c00640 1 -- start start 2024-07-26T11:05:59.399 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 0x7fc630075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:59.399 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:59.399 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc630109670 0x7fc630075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:59.399 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc630113300 con 0x7fc630109670 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc630113180 con 0x7fc630108d20 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc630113480 con 0x7fc6301033e0 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 0x7fc630075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc635200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc630109670 0x7fc630075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 0x7fc630075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56706/0 (socket says 172.21.15.144:56706) 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc62fe00640 1 -- 172.21.15.144:0/3879938952 learned_addr learned my addr 172.21.15.144:0/3879938952 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:05:59.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.397+0000 7fc627400640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc627400640 1 -- 172.21.15.144:0/3879938952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 msgr2=0x7fc630075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc627400640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 0x7fc630075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc627400640 1 -- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc630109670 msgr2=0x7fc630075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc627400640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc630109670 0x7fc630075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc627400640 1 -- 172.21.15.144:0/3879938952 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6301a8030 con 0x7fc630108d20 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc627400640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630075680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc61802a910 tx=0x7fc618031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61803f070 con 0x7fc630108d20 2024-07-26T11:05:59.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc618003aa0 con 0x7fc630108d20 2024-07-26T11:05:59.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc618035070 con 0x7fc630108d20 2024-07-26T11:05:59.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.398+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6301a82c0 con 0x7fc630108d20 2024-07-26T11:05:59.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.399+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6301a87f0 con 0x7fc630108d20 2024-07-26T11:05:59.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.400+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6301037f0 con 0x7fc630108d20 2024-07-26T11:05:59.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.402+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fc618003c50 con 0x7fc630108d20 2024-07-26T11:05:59.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.404+0000 7fc62d600640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6140b3dc0 0x7fc6140b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:59.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.404+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7fc6181046e0 con 0x7fc630108d20 2024-07-26T11:05:59.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.404+0000 7fc62fe00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6140b3dc0 0x7fc6140b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:59.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.405+0000 7fc62fe00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6140b3dc0 0x7fc6140b62d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc630076b90 tx=0x7fc620005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:59.407 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.406+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc6180cb5d0 con 0x7fc630108d20 2024-07-26T11:05:59.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.646+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc630076440 con 0x7fc630108d20 2024-07-26T11:05:59.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.647+0000 7fc62d600640 1 -- 172.21.15.144:0/3879938952 <== mon.2 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11937 (secure 0 0 0) 0x7fc630076440 con 0x7fc630108d20 2024-07-26T11:05:59.649 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-07-26T11:05:59.649 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":35,"btime":"2024-07-26T11:05:58:477747+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3205660369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3205660369},{"type":"v1","addr":"172.21.15.144:6837","nonce":3205660369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/4222472223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4222472223},{"type":"v1","addr":"172.21.15.183:6833","nonce":4222472223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24647,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/1864776442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":1864776442},{"type":"v1","addr":"172.21.15.160:6833","nonce":1864776442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24653,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2179784040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2179784040},{"type":"v1","addr":"172.21.15.144:6835","nonce":2179784040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24667,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/15323047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":15323047},{"type":"v1","addr":"172.21.15.183:6835","nonce":15323047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2490842163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2490842163},{"type":"v1","addr":"172.21.15.183:6837","nonce":2490842163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T10:57:25.002914+0000","modified":"2024-07-26T11:05:57.892593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24701,"mds_1":24715,"mds_2":24703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24677":{"gid":24677,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/405635522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":405635522},{"type":"v1","addr":"172.21.15.160:6835","nonce":405635522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24691":{"gid":24691,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6839/3458220569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":3458220569},{"type":"v1","addr":"172.21.15.183:6839","nonce":3458220569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"h","rank":0,"incarnation":29,"state":"up:active","state_seq":139,"addr":"172.21.15.160:6837/4027087859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":4027087859},{"type":"v1","addr":"172.21.15.160:6837","nonce":4027087859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.144:6839/2313694680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":2313694680},{"type":"v1","addr":"172.21.15.144:6839","nonce":2313694680}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.144:6841/1389845259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6840","nonce":1389845259},{"type":"v1","addr":"172.21.15.144:6841","nonce":1389845259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26012":{"gid":26012,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6839/1392250925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":1392250925},{"type":"v1","addr":"172.21.15.160:6839","nonce":1392250925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24701,24703,24715]},"id":1}]} 2024-07-26T11:05:59.650 INFO:teuthology.orchestra.run.smithi144.stderr:dumped fsmap epoch 35 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6140b3dc0 msgr2=0x7fc6140b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6140b3dc0 0x7fc6140b62d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc630076b90 tx=0x7fc620005f90 comp rx=0 tx=0).stop 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 msgr2=0x7fc630075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630075680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc61802a910 tx=0x7fc618031a20 comp rx=0 tx=0).stop 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 shutdown_connections 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc6140b3dc0 0x7fc6140b62d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc630109670 0x7fc630075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc630108d20 0x7fc630075680 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 --2- 172.21.15.144:0/3879938952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6301033e0 0x7fc630075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:59.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.650+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 >> 172.21.15.144:0/3879938952 conn(0x7fc6300fee70 msgr2=0x7fc630101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:59.653 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.651+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 shutdown_connections 2024-07-26T11:05:59.654 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:05:59.651+0000 7fc635c00640 1 -- 172.21.15.144:0/3879938952 wait complete. 2024-07-26T11:05:59.775 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:05:59 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:05:59 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:05:59.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:05:59 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:00.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:06:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:00 smithi160 ceph-mon[36715]: pgmap v567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 71 op/s 2024-07-26T11:06:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:00 smithi160 ceph-mon[36715]: from='client.? 172.21.15.144:0/3879938952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:06:00.862 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:00 smithi144 ceph-mon[32129]: pgmap v567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 71 op/s 2024-07-26T11:06:00.862 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:00 smithi144 ceph-mon[32129]: from='client.? 172.21.15.144:0/3879938952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:06:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:00 smithi183 ceph-mon[37273]: pgmap v567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 71 op/s 2024-07-26T11:06:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:00 smithi183 ceph-mon[37273]: from='client.? 172.21.15.144:0/3879938952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:06:01.005 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:02 smithi160 ceph-mon[36715]: pgmap v568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 72 op/s 2024-07-26T11:06:02.814 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:02 smithi144 ceph-mon[32129]: pgmap v568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 72 op/s 2024-07-26T11:06:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:02 smithi183 ceph-mon[37273]: pgmap v568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 72 op/s 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 -- 172.21.15.144:0/3913145595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc104220 msgr2=0x7ffbbc104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/3913145595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc104220 0x7ffbbc104630 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009900 tx=0x7ffbac02a410 comp rx=0 tx=0).stop 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 -- 172.21.15.144:0/3913145595 shutdown_connections 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/3913145595 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc10d680 0x7ffbbc10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/3913145595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104b70 0x7ffbbc10d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/3913145595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc104220 0x7ffbbc104630 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 -- 172.21.15.144:0/3913145595 >> 172.21.15.144:0/3913145595 conn(0x7ffbbc0fdd40 msgr2=0x7ffbbc100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 -- 172.21.15.144:0/3913145595 shutdown_connections 2024-07-26T11:06:03.385 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.382+0000 7ffbc0e00640 1 -- 172.21.15.144:0/3913145595 wait complete. 2024-07-26T11:06:03.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 Processor -- start 2024-07-26T11:06:03.386 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 -- start start 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104220 0x7ffbbc101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 0x7ffbbc1024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc10d680 0x7ffbbc1029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc113360 con 0x7ffbbc10d680 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1131e0 con 0x7ffbbc104b70 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.384+0000 7ffbc0e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1134e0 con 0x7ffbbc104220 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 0x7ffbbc1024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 0x7ffbbc1024a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:50662/0 (socket says 172.21.15.144:50662) 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2380872685 learned_addr learned my addr 172.21.15.144:0/2380872685 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104220 0x7ffbbc101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.387 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc10d680 0x7ffbbc1029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbb400640 1 -- 172.21.15.144:0/2380872685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 msgr2=0x7ffbbc1024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 0x7ffbbc1024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbb400640 1 -- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc10d680 msgr2=0x7ffbbc1029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc10d680 0x7ffbbc1029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbb400640 1 -- 172.21.15.144:0/2380872685 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbbc1a80f0 con 0x7ffbbc104220 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.385+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 0x7ffbbc1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.386+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104220 0x7ffbbc101f60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffbac02aa00 tx=0x7ffbac003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.386+0000 7ffbb3e00640 1 -- 172.21.15.144:0/2380872685 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac03f070 con 0x7ffbbc104220 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.386+0000 7ffbb3e00640 1 -- 172.21.15.144:0/2380872685 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbac031b50 con 0x7ffbbc104220 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.386+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbbc1a8380 con 0x7ffbbc104220 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.386+0000 7ffbb3e00640 1 -- 172.21.15.144:0/2380872685 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac035030 con 0x7ffbbc104220 2024-07-26T11:06:03.389 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.386+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbbc1a8830 con 0x7ffbbc104220 2024-07-26T11:06:03.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.389+0000 7ffbc907a740 1 Processor -- start 2024-07-26T11:06:03.391 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 -- start start 2024-07-26T11:06:03.392 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbb3e00640 1 -- 172.21.15.144:0/2380872685 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ffbac042070 con 0x7ffbbc104220 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7398030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fca1b0 0x5636a7398570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 0x5636a7398ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5636a730ebf0 con 0x5636a6fca1b0 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5636a72fe0f0 con 0x5636a6fd2390 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbc907a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5636a7302130 con 0x5636a6ff3070 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fca1b0 0x5636a7398570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 0x5636a7398ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 0x5636a7398ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:50670/0 (socket says 172.21.15.144:50670) 2024-07-26T11:06:03.393 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2508419910 learned_addr learned my addr 172.21.15.144:0/2508419910 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:03.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.390+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fca1b0 0x5636a7398570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39766/0 (socket says 172.21.15.144:39766) 2024-07-26T11:06:03.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7398030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2508419910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 msgr2=0x5636a7398030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7398030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2508419910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fca1b0 msgr2=0x5636a7398570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fca1b0 0x5636a7398570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2508419910 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a705e650 con 0x5636a6fd2390 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7398030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.391+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 0x5636a7398ab0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffba40118b0 tx=0x7ffba4011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ec3ddf79f7d48 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.392+0000 7ffbb0c00640 1 -- 172.21.15.144:0/2508419910 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4011310 con 0x5636a6fd2390 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.392+0000 7ffbb0c00640 1 -- 172.21.15.144:0/2508419910 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba4003970 con 0x5636a6fd2390 2024-07-26T11:06:03.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.392+0000 7ffbb0c00640 1 -- 172.21.15.144:0/2508419910 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4004020 con 0x5636a6fd2390 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.392+0000 7ffbb3e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ffb9c0b3d40 0x7ffb9c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.392+0000 7ffbb3e00640 1 -- 172.21.15.144:0/2380872685 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7ffbac104970 con 0x7ffbbc104220 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.392+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ffb9c0b3d40 0x7ffb9c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 -- 172.21.15.144:0/2508419910 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 msgr2=0x5636a7398ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 0x5636a7398ab0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffba40118b0 tx=0x7ffba4011d70 comp rx=0 tx=0).stop 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 -- 172.21.15.144:0/2508419910 shutdown_connections 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fd2390 0x5636a7398ab0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fca1b0 0x5636a7398570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2508419910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7398030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 -- 172.21.15.144:0/2508419910 >> 172.21.15.144:0/2508419910 conn(0x5636a6fe7460 msgr2=0x5636a7395f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:03.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 -- 172.21.15.144:0/2508419910 shutdown_connections 2024-07-26T11:06:03.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ffb9c0b3d40 0x7ffb9c0b6250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009cd0 tx=0x7ffbb4005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:03.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.393+0000 7ffbc907a740 1 -- 172.21.15.144:0/2508419910 wait complete. 2024-07-26T11:06:03.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.394+0000 7ffbc907a740 1 -- start start 2024-07-26T11:06:03.398 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.394+0000 7ffbc907a740 1 Processor -- start 2024-07-26T11:06:03.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.400 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a73a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5636a7161700 con 0x5636a6fd2390 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5636a729a6c0 con 0x5636a6fca1b0 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5636a73a6810 con 0x5636a6ff3070 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbc907a740 10 client.0 _ll_register_callbacks cb 0x5636a72aa170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39776/0 (socket says 172.21.15.144:39776) 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbbaa00640 1 -- 172.21.15.144:0/1685292785 learned_addr learned my addr 172.21.15.144:0/1685292785 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbbb400640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.395+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a73a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.401 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbb400640 1 -- 172.21.15.144:0/1685292785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 msgr2=0x5636a73a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbb400640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a73a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbb400640 1 -- 172.21.15.144:0/1685292785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 msgr2=0x5636a73a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbb400640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbb400640 1 -- 172.21.15.144:0/1685292785 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a72b1860 con 0x5636a6fca1b0 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbbb400640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffbac101ca0 tx=0x7ffbac02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c381dd6df8010ef server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffba9600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffba9600640 1 -- 172.21.15.144:0/1685292785 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac049070 con 0x5636a6fca1b0 2024-07-26T11:06:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba4003970 con 0x5636a6fca1b0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffba9600640 1 -- 172.21.15.144:0/1685292785 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7ffbac0cb5d0 con 0x5636a6fca1b0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.396+0000 7ffba9600640 1 -- 172.21.15.144:0/1685292785 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ffbac0c7020 con 0x5636a6fca1b0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 10 client.26045 fetch_fsmap learned FSMap version 35 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5636a73b2350 con 0x5636a6fca1b0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffba9600640 1 -- 172.21.15.144:0/1685292785 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba4003970 con 0x5636a6fca1b0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffba9600640 1 -- 172.21.15.144:0/1685292785 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ffbac0d3d80 con 0x5636a6fca1b0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 10 client.26045 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 10 client.26045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 20 client.26045 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6fb19f0 0x5636a73b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 4 client.26045 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.397+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636a7310b40 con 0x5636a6fb19f0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.398+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6fb19f0 0x5636a73b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.403 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.398+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6fb19f0 0x5636a73b5930 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42c867de5b15eb45 server_cookie=3f4d56445f80c188 in_seq=0 out_seq=0 2024-07-26T11:06:03.404 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.398+0000 7ffba9600640 10 client.26045 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffba9600640 1 -- 172.21.15.144:0/1685292785 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5636a7310b40 con 0x5636a6fb19f0 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffba9600640 10 client.26045 handle_command_reply: tid=0 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 client.26045 shutdown 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 msgr2=0x5636a73a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffbac101ca0 tx=0x7ffbac02a410 comp rx=0 tx=0).stop 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 shutdown_connections 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a73a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6fb19f0 0x5636a73b5930 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 --2- 172.21.15.144:0/1685292785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3570 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 >> 172.21.15.144:0/1685292785 conn(0x5636a6fe7460 msgr2=0x5636a739b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:03.405 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.399+0000 7ffba9600640 0 client.26045 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:03.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.400+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 shutdown_connections 2024-07-26T11:06:03.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.400+0000 7ffbc907a740 1 -- 172.21.15.144:0/1685292785 wait complete. 2024-07-26T11:06:03.406 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.400+0000 7ffbc907a740 20 client.26045 trim_cache size 0 max 16384 2024-07-26T11:06:03.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 Processor -- start 2024-07-26T11:06:03.416 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 -- start start 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a7394560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7399770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1b51e0 con 0x5636a6fd2390 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1b2a50 con 0x5636a6fca1b0 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.414+0000 7ffbc907a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1b08a0 con 0x5636a6ff3070 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a7394560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.417 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a7394560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39792/0 (socket says 172.21.15.144:39792) 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2200877080 learned_addr learned my addr 172.21.15.144:0/2200877080 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7399770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2200877080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 msgr2=0x5636a7399770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7399770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2200877080 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 msgr2=0x5636a73a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.418 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 -- 172.21.15.144:0/2200877080 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a739a320 con 0x5636a6fd2390 2024-07-26T11:06:03.419 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.415+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a7394560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffba40118b0 tx=0x7ffba400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e5513ccedf1b73 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:03.419 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.416+0000 7ffbabe00640 1 -- 172.21.15.144:0/2200877080 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4031070 con 0x5636a6fd2390 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.416+0000 7ffbabe00640 1 -- 172.21.15.144:0/2200877080 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba4029070 con 0x5636a6fd2390 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.416+0000 7ffbabe00640 1 -- 172.21.15.144:0/2200877080 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba401d040 con 0x5636a6fd2390 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 -- 172.21.15.144:0/2200877080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 msgr2=0x5636a7394560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a7394560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffba40118b0 tx=0x7ffba400f840 comp rx=0 tx=0).stop 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 -- 172.21.15.144:0/2200877080 shutdown_connections 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6ff3070 0x5636a7399770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a7394560 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2200877080 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6fca1b0 0x5636a73a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 -- 172.21.15.144:0/2200877080 >> 172.21.15.144:0/2200877080 conn(0x5636a6fe7460 msgr2=0x5636a7399060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 -- 172.21.15.144:0/2200877080 shutdown_connections 2024-07-26T11:06:03.420 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 -- 172.21.15.144:0/2200877080 wait complete. 2024-07-26T11:06:03.421 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.417+0000 7ffbc907a740 1 -- start start 2024-07-26T11:06:03.421 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 Processor -- start 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 0x5636a73ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6ff3070 0x5636a73a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1b0f10 con 0x5636a6fd2390 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1b2380 con 0x5636a6ff3070 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1b15e0 con 0x5636a6fca1b0 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbc907a740 10 client.0 _ll_register_callbacks cb 0x7ffbac10a220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 0x5636a73ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.423 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.418+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6ff3070 0x5636a73a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 0x5636a73ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35058/0 (socket says 172.21.15.144:35058) 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 -- 172.21.15.144:0/2007730007 learned_addr learned my addr 172.21.15.144:0/2007730007 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 -- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6ff3070 msgr2=0x5636a73a9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6ff3070 0x5636a73a9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 -- 172.21.15.144:0/2007730007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 msgr2=0x5636a73a9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 -- 172.21.15.144:0/2007730007 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a73aa0f0 con 0x5636a6fca1b0 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbbe00640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6ff3070 0x5636a73a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffbbb400640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 0x5636a73ac050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0c71e0 tx=0x7ffbac02aa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23b8505fa4949bf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffba9600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:06:03.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.419+0000 7ffba9600640 1 -- 172.21.15.144:0/2007730007 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac049070 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffba9600640 1 -- 172.21.15.144:0/2007730007 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7ffbac0ca620 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba4029070 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffba9600640 1 -- 172.21.15.144:0/2007730007 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ffbac0ca8d0 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffbc907a740 10 client.26035 fetch_fsmap learned FSMap version 35 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5636a741dec0 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffba9600640 1 -- 172.21.15.144:0/2007730007 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba4029070 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffba9600640 1 -- 172.21.15.144:0/2007730007 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ffbac0d2ae0 con 0x5636a6fca1b0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffbc907a740 10 client.26035 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.420+0000 7ffbc907a740 10 client.26035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffbc907a740 20 client.26035 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6f86da0 0x5636a7420410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffbc907a740 4 client.26035 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636a730c5e0 con 0x5636a6f86da0 2024-07-26T11:06:03.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6f86da0 0x5636a7420410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffba9600640 10 client.26035 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.421+0000 7ffbbaa00640 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6f86da0 0x5636a7420410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1813248dc13d9565 server_cookie=8a0829a613ae551d in_seq=0 out_seq=0 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffba9600640 1 -- 172.21.15.144:0/2007730007 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636a730c5e0 con 0x5636a6f86da0 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffba9600640 10 client.26035 handle_command_reply: tid=0 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 client.26035 shutdown 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 msgr2=0x5636a73ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 0x5636a73ac050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0c71e0 tx=0x7ffbac02aa90 comp rx=0 tx=0).stop 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 shutdown_connections 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5636a6f86da0 0x5636a7420410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5636a6ff3070 0x5636a73a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5636a6fd2390 0x5636a73a9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffba9600640 0 client.26035 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:03.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 --2- 172.21.15.144:0/2007730007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5636a6fca1b0 0x5636a73ac050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 >> 172.21.15.144:0/2007730007 conn(0x5636a6fe7460 msgr2=0x5636a73a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:03.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 shutdown_connections 2024-07-26T11:06:03.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 1 -- 172.21.15.144:0/2007730007 wait complete. 2024-07-26T11:06:03.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.422+0000 7ffbc907a740 20 client.26035 trim_cache size 0 max 16384 2024-07-26T11:06:03.428 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.423+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ffb9c0b3d40 msgr2=0x7ffb9c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.423+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ffb9c0b3d40 0x7ffb9c0b6250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009cd0 tx=0x7ffbb4005f90 comp rx=0 tx=0).stop 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.423+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104220 msgr2=0x7ffbbc101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.423+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104220 0x7ffbbc101f60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffbac02aa00 tx=0x7ffbac003b90 comp rx=0 tx=0).stop 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 shutdown_connections 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7ffb9c0b3d40 0x7ffb9c0b6250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbbc10d680 0x7ffbbc1029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffbbc104b70 0x7ffbbc1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 --2- 172.21.15.144:0/2380872685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbbc104220 0x7ffbbc101f60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:03.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 >> 172.21.15.144:0/2380872685 conn(0x7ffbbc0fdd40 msgr2=0x7ffbbc1071a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:03.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 shutdown_connections 2024-07-26T11:06:03.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:03.424+0000 7ffbc0e00640 1 -- 172.21.15.144:0/2380872685 wait complete. 2024-07-26T11:06:04.350 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:06:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:04 smithi144 ceph-mon[32129]: pgmap v569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 69 op/s 2024-07-26T11:06:04.804 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:04 smithi160 ceph-mon[36715]: pgmap v569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 69 op/s 2024-07-26T11:06:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:04 smithi183 ceph-mon[37273]: pgmap v569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 69 op/s 2024-07-26T11:06:06.781 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:06 smithi144 ceph-mon[32129]: pgmap v570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 66 op/s 2024-07-26T11:06:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:06 smithi160 ceph-mon[36715]: pgmap v570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 66 op/s 2024-07-26T11:06:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:06 smithi183 ceph-mon[37273]: pgmap v570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 66 op/s 2024-07-26T11:06:07.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 -- 172.21.15.144:0/4071994000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 msgr2=0x7f212c075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 --2- 172.21.15.144:0/4071994000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c075590 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f211c009900 tx=0x7f211c02a410 comp rx=0 tx=0).stop 2024-07-26T11:06:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 -- 172.21.15.144:0/4071994000 shutdown_connections 2024-07-26T11:06:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 --2- 172.21.15.144:0/4071994000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c10f9a0 0x7f212c111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 --2- 172.21.15.144:0/4071994000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f212c075ad0 0x7f212c10f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 --2- 172.21.15.144:0/4071994000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c075590 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 -- 172.21.15.144:0/4071994000 >> 172.21.15.144:0/4071994000 conn(0x7f212c0fee30 msgr2=0x7f212c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 -- 172.21.15.144:0/4071994000 shutdown_connections 2024-07-26T11:06:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.094+0000 7f2131200640 1 -- 172.21.15.144:0/4071994000 wait complete. 2024-07-26T11:06:07.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.095+0000 7f2131200640 1 Processor -- start 2024-07-26T11:06:07.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.095+0000 7f2131200640 1 -- start start 2024-07-26T11:06:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f2131200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c19bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f2131200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f212c075ad0 0x7f212c19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f2131200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c10f9a0 0x7f212c1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f2131200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f212c115550 con 0x7f212c075ad0 2024-07-26T11:06:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f2131200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1153d0 con 0x7f212c10f9a0 2024-07-26T11:06:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f2131200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1156d0 con 0x7f212c075180 2024-07-26T11:06:07.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f212aa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f212c075ad0 0x7f212c19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f212b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c19bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f212b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c19bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35082/0 (socket says 172.21.15.144:35082) 2024-07-26T11:06:07.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f212b400640 1 -- 172.21.15.144:0/3892550774 learned_addr learned my addr 172.21.15.144:0/3892550774 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:07.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.096+0000 7f212be00640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c10f9a0 0x7f212c1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f212b400640 1 -- 172.21.15.144:0/3892550774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c10f9a0 msgr2=0x7f212c1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f212b400640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c10f9a0 0x7f212c1a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f212b400640 1 -- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f212c075ad0 msgr2=0x7f212c19c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f212b400640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f212c075ad0 0x7f212c19c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f212b400640 1 -- 172.21.15.144:0/3892550774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c1a1ef0 con 0x7f212c075180 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f212b400640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c19bf80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f211c005ab0 tx=0x7f211c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f2123e00640 1 -- 172.21.15.144:0/3892550774 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c03f070 con 0x7f212c075180 2024-07-26T11:06:07.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f2123e00640 1 -- 172.21.15.144:0/3892550774 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f211c031b60 con 0x7f212c075180 2024-07-26T11:06:07.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.097+0000 7f2123e00640 1 -- 172.21.15.144:0/3892550774 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c035030 con 0x7f212c075180 2024-07-26T11:06:07.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.098+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212c1a2180 con 0x7f212c075180 2024-07-26T11:06:07.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.098+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f212c1a8940 con 0x7f212c075180 2024-07-26T11:06:07.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.101+0000 7f2139483740 1 Processor -- start 2024-07-26T11:06:07.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 -- start start 2024-07-26T11:06:07.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdc3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9f51b0 0x55e90bdc3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9fd390 0x55e90bdc3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e90bd39bd0 con 0x55e90ba1e070 2024-07-26T11:06:07.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e90bd290d0 con 0x55e90b9fd390 2024-07-26T11:06:07.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2139483740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e90bd2d110 con 0x55e90b9f51b0 2024-07-26T11:06:07.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f2123e00640 1 -- 172.21.15.144:0/3892550774 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f211c042070 con 0x7f212c075180 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f212be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdc3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f212be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdc3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39818/0 (socket says 172.21.15.144:39818) 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f212be00640 1 -- 172.21.15.144:0/414849828 learned_addr learned my addr 172.21.15.144:0/414849828 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.102+0000 7f212aa00640 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9fd390 0x55e90bdc3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212b400640 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9f51b0 0x55e90bdc3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212aa00640 1 -- 172.21.15.144:0/414849828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9f51b0 msgr2=0x55e90bdc3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212aa00640 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9f51b0 0x55e90bdc3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212aa00640 1 -- 172.21.15.144:0/414849828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 msgr2=0x55e90bdc3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212aa00640 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdc3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212aa00640 1 -- 172.21.15.144:0/414849828 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e90ba89650 con 0x55e90b9fd390 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212be00640 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdc3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.103+0000 7f212aa00640 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9fd390 0x55e90bdc3a90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f212c06b360 tx=0x7f2118011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e7fddb7df6cf8c7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.104+0000 7f2120c00640 1 -- 172.21.15.144:0/414849828 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118004050 con 0x55e90b9fd390 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.104+0000 7f2120c00640 1 -- 172.21.15.144:0/414849828 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21180054d0 con 0x55e90b9fd390 2024-07-26T11:06:07.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.104+0000 7f2120c00640 1 -- 172.21.15.144:0/414849828 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118004c00 con 0x55e90b9fd390 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.104+0000 7f2123e00640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21100b3dc0 0x7f21100b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.104+0000 7f2123e00640 1 -- 172.21.15.144:0/3892550774 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7f211c104a00 con 0x7f212c075180 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.105+0000 7f212be00640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21100b3dc0 0x7f21100b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.105+0000 7f2139483740 1 -- 172.21.15.144:0/414849828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9fd390 msgr2=0x55e90bdc3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.105+0000 7f2139483740 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9fd390 0x55e90bdc3a90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f212c06b360 tx=0x7f2118011810 comp rx=0 tx=0).stop 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 -- 172.21.15.144:0/414849828 shutdown_connections 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9fd390 0x55e90bdc3a90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9f51b0 0x55e90bdc3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 --2- 172.21.15.144:0/414849828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdc3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 -- 172.21.15.144:0/414849828 >> 172.21.15.144:0/414849828 conn(0x55e90ba12460 msgr2=0x55e90bdc0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f212be00640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21100b3dc0 0x7f21100b62d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2124009fd0 tx=0x7f2124009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 -- 172.21.15.144:0/414849828 shutdown_connections 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.106+0000 7f2139483740 1 -- 172.21.15.144:0/414849828 wait complete. 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.107+0000 7f2139483740 1 -- start start 2024-07-26T11:06:07.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.107+0000 7f2139483740 1 Processor -- start 2024-07-26T11:06:07.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 0x55e90bdce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90ba1e070 0x55e90bdcf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e90bb8c700 con 0x55e90b9f51b0 2024-07-26T11:06:07.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e90bcc56a0 con 0x55e90ba1e070 2024-07-26T11:06:07.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e90bdd17f0 con 0x55e90b9fd390 2024-07-26T11:06:07.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f2139483740 10 client.0 _ll_register_callbacks cb 0x55e90bcd5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:07.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f212aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f212b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 0x55e90bdce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f212be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90ba1e070 0x55e90bdcf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f212b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 0x55e90bdce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39828/0 (socket says 172.21.15.144:39828) 2024-07-26T11:06:07.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.108+0000 7f212b400640 1 -- 172.21.15.144:0/269777492 learned_addr learned my addr 172.21.15.144:0/269777492 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:07.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f212b400640 1 -- 172.21.15.144:0/269777492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 msgr2=0x55e90bdceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f212b400640 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f212b400640 1 -- 172.21.15.144:0/269777492 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90ba1e070 msgr2=0x55e90bdcf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f212b400640 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90ba1e070 0x55e90bdcf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f212b400640 1 -- 172.21.15.144:0/269777492 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90bcdc840 con 0x55e90b9f51b0 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f212b400640 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 0x55e90bdce630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f211c031440 tx=0x7f211c0c7280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c265fd666288ac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f2115600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f2115600640 1 -- 172.21.15.144:0/269777492 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c049070 con 0x55e90b9f51b0 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f2115600640 1 -- 172.21.15.144:0/269777492 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7f211c0c7590 con 0x55e90b9f51b0 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.109+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21180054d0 con 0x55e90b9f51b0 2024-07-26T11:06:07.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.110+0000 7f2115600640 1 -- 172.21.15.144:0/269777492 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f211c0caa70 con 0x55e90b9f51b0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.110+0000 7f2139483740 10 client.15909 fetch_fsmap learned FSMap version 35 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.110+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e90bddd430 con 0x55e90b9f51b0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.110+0000 7f2115600640 1 -- 172.21.15.144:0/269777492 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21180054d0 con 0x55e90b9f51b0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.110+0000 7f2115600640 1 -- 172.21.15.144:0/269777492 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f211c03b460 con 0x55e90b9f51b0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.110+0000 7f2139483740 10 client.15909 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.111+0000 7f2139483740 10 client.15909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.111+0000 7f2139483740 20 client.15909 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.111+0000 7f2139483740 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9dc9f0 0x55e90bde0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.111+0000 7f2139483740 4 client.15909 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.111+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e90bd3bb20 con 0x55e90b9dc9f0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.111+0000 7f212aa00640 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9dc9f0 0x55e90bde0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.112+0000 7f212aa00640 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9dc9f0 0x55e90bde0a40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=572c37665322d725 server_cookie=ebdd289359b4f936 in_seq=0 out_seq=0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.112+0000 7f2115600640 10 client.15909 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2115600640 1 -- 172.21.15.144:0/269777492 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e90bd3bb20 con 0x55e90b9dc9f0 2024-07-26T11:06:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2115600640 10 client.15909 handle_command_reply: tid=0 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 client.15909 shutdown 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 msgr2=0x55e90bdce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 0x55e90bdce630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f211c031440 tx=0x7f211c0c7280 comp rx=0 tx=0).stop 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 shutdown_connections 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9dc9f0 0x55e90bde0a40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2115600640 0 client.15909 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90ba1e070 0x55e90bdcf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 --2- 172.21.15.144:0/269777492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9f51b0 0x55e90bdce630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.113+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 >> 172.21.15.144:0/269777492 conn(0x55e90ba12460 msgr2=0x55e90bdc61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.114+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 shutdown_connections 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.114+0000 7f2139483740 1 -- 172.21.15.144:0/269777492 wait complete. 2024-07-26T11:06:07.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.114+0000 7f2139483740 20 client.15909 trim_cache size 0 max 16384 2024-07-26T11:06:07.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.129+0000 7f2139483740 1 Processor -- start 2024-07-26T11:06:07.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.129+0000 7f2139483740 1 -- start start 2024-07-26T11:06:07.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.129+0000 7f2139483740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f2139483740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 0x55e90bde6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f2139483740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90ba1e070 0x55e90bde6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f2139483740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1b0e40 con 0x55e90b9fd390 2024-07-26T11:06:07.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f2139483740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1ae6b0 con 0x55e90b9f51b0 2024-07-26T11:06:07.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f2139483740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1ac540 con 0x55e90ba1e070 2024-07-26T11:06:07.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212aa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 0x55e90bde6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90ba1e070 0x55e90bde6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212aa00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 0x55e90bde6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39834/0 (socket says 172.21.15.144:39834) 2024-07-26T11:06:07.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212aa00640 1 -- 172.21.15.144:0/2020264167 learned_addr learned my addr 172.21.15.144:0/2020264167 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:07.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212aa00640 1 -- 172.21.15.144:0/2020264167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90ba1e070 msgr2=0x55e90bde6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.130+0000 7f212aa00640 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90ba1e070 0x55e90bde6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f212aa00640 1 -- 172.21.15.144:0/2020264167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 msgr2=0x55e90bdd0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f212aa00640 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f212aa00640 1 -- 172.21.15.144:0/2020264167 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e90bdccb30 con 0x55e90b9fd390 2024-07-26T11:06:07.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f212be00640 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90ba1e070 0x55e90bde6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:07.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f212aa00640 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 0x55e90bde6240 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f212c06b360 tx=0x7f2118011fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b279d74d7067a92a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:07.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f2117e00640 1 -- 172.21.15.144:0/2020264167 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118004640 con 0x55e90b9fd390 2024-07-26T11:06:07.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f2117e00640 1 -- 172.21.15.144:0/2020264167 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2118009d70 con 0x55e90b9fd390 2024-07-26T11:06:07.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.131+0000 7f2117e00640 1 -- 172.21.15.144:0/2020264167 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118003d00 con 0x55e90b9fd390 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 -- 172.21.15.144:0/2020264167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 msgr2=0x55e90bde6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 0x55e90bde6240 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f212c06b360 tx=0x7f2118011fb0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 -- 172.21.15.144:0/2020264167 shutdown_connections 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90ba1e070 0x55e90bde6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90b9fd390 0x55e90bde6240 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 --2- 172.21.15.144:0/2020264167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 -- 172.21.15.144:0/2020264167 >> 172.21.15.144:0/2020264167 conn(0x55e90ba12460 msgr2=0x55e90bdc4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:07.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.132+0000 7f2139483740 1 -- 172.21.15.144:0/2020264167 shutdown_connections 2024-07-26T11:06:07.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.133+0000 7f2139483740 1 -- 172.21.15.144:0/2020264167 wait complete. 2024-07-26T11:06:07.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.133+0000 7f2139483740 1 -- start start 2024-07-26T11:06:07.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.133+0000 7f2139483740 1 Processor -- start 2024-07-26T11:06:07.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdd45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdd4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1acb90 con 0x55e90ba1e070 2024-07-26T11:06:07.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1adfe0 con 0x55e90b9f51b0 2024-07-26T11:06:07.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1ad240 con 0x55e90b9fd390 2024-07-26T11:06:07.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f2139483740 10 client.0 _ll_register_callbacks cb 0x7f211c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:07.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f212be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdd4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f212be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdd4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39842/0 (socket says 172.21.15.144:39842) 2024-07-26T11:06:07.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f212be00640 1 -- 172.21.15.144:0/1930123647 learned_addr learned my addr 172.21.15.144:0/1930123647 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:07.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f212b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.134+0000 7f212aa00640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdd45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212b400640 1 -- 172.21.15.144:0/1930123647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 msgr2=0x55e90bdd45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212b400640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdd45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212b400640 1 -- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 msgr2=0x55e90bdd4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:07.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212b400640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdd4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212b400640 1 -- 172.21.15.144:0/1930123647 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90bdd51f0 con 0x55e90b9f51b0 2024-07-26T11:06:07.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212aa00640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdd45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:07.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212be00640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdd4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:07.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f212b400640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd70d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f211c0c8570 tx=0x7f211c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7293f926d883f0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:07.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f2115600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:06:07.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.135+0000 7f2115600640 1 -- 172.21.15.144:0/1930123647 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c049070 con 0x55e90b9f51b0 2024-07-26T11:06:07.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2115600640 1 -- 172.21.15.144:0/1930123647 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9480+0+0 (secure 0 0 0) 0x7f211c0d2de0 con 0x55e90b9f51b0 2024-07-26T11:06:07.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2118009d70 con 0x55e90b9f51b0 2024-07-26T11:06:07.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2115600640 1 -- 172.21.15.144:0/1930123647 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f211c0cae10 con 0x55e90b9f51b0 2024-07-26T11:06:07.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 10 client.26069 fetch_fsmap learned FSMap version 35 2024-07-26T11:06:07.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e90be490e0 con 0x55e90b9f51b0 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2115600640 1 -- 172.21.15.144:0/1930123647 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2118009d70 con 0x55e90b9f51b0 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2115600640 1 -- 172.21.15.144:0/1930123647 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f211c0ca560 con 0x55e90b9f51b0 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 10 client.26069 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 10 client.26069 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 20 client.26069 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9b1da0 0x55e90be4b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 4 client.26069 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:06:07.149 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.136+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 --> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e90bd375c0 con 0x55e90b9b1da0 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.137+0000 7f212aa00640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9b1da0 0x55e90be4b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.137+0000 7f212aa00640 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9b1da0 0x55e90be4b7e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=538638efd3f0a1a6 server_cookie=1d5588d53fbf2df5 in_seq=0 out_seq=0 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.137+0000 7f2115600640 10 client.26069 ms_handle_connect on v2:172.21.15.144:6838/2313694680 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2115600640 1 -- 172.21.15.144:0/1930123647 <== mds.2 v2:172.21.15.144:6838/2313694680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e90bd375c0 con 0x55e90b9b1da0 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2115600640 10 client.26069 handle_command_reply: tid=0 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 client.26069 shutdown 2024-07-26T11:06:07.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 msgr2=0x55e90bdd70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd70d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f211c0c8570 tx=0x7f211c000bc0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 shutdown_connections 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:6838/2313694680,v1:172.21.15.144:6839/2313694680] conn(0x55e90b9b1da0 0x55e90be4b7e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e90ba1e070 0x55e90bdd4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e90b9fd390 0x55e90bdd45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 --2- 172.21.15.144:0/1930123647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e90b9f51b0 0x55e90bdd70d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2115600640 0 client.26069 ms_handle_reset on v2:172.21.15.144:6838/2313694680 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 >> 172.21.15.144:0/1930123647 conn(0x55e90ba12460 msgr2=0x55e90bdc4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:07.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 shutdown_connections 2024-07-26T11:06:07.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 1 -- 172.21.15.144:0/1930123647 wait complete. 2024-07-26T11:06:07.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.138+0000 7f2139483740 20 client.26069 trim_cache size 0 max 16384 2024-07-26T11:06:07.152 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:06:07.153 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21100b3dc0 msgr2=0x7f21100b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21100b3dc0 0x7f21100b62d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2124009fd0 tx=0x7f2124009500 comp rx=0 tx=0).stop 2024-07-26T11:06:07.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 msgr2=0x7f212c19bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:07.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c19bf80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f211c005ab0 tx=0x7f211c003c10 comp rx=0 tx=0).stop 2024-07-26T11:06:07.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 shutdown_connections 2024-07-26T11:06:07.154 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21100b3dc0 0x7f21100b62d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c10f9a0 0x7f212c1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f212c075ad0 0x7f212c19c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 --2- 172.21.15.144:0/3892550774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f212c075180 0x7f212c19bf80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:07.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 >> 172.21.15.144:0/3892550774 conn(0x7f212c0fee30 msgr2=0x7f212c1094d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:07.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.139+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 shutdown_connections 2024-07-26T11:06:07.155 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:07.140+0000 7f2131200640 1 -- 172.21.15.144:0/3892550774 wait complete. 2024-07-26T11:06:08.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:06:08.566 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:08.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:08 smithi144 ceph-mon[32129]: pgmap v571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 748 KiB/s wr, 49 op/s 2024-07-26T11:06:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:08 smithi160 ceph-mon[36715]: pgmap v571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 748 KiB/s wr, 49 op/s 2024-07-26T11:06:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:08 smithi183 ceph-mon[37273]: pgmap v571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 748 KiB/s wr, 49 op/s 2024-07-26T11:06:09.027 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:08 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:06:09.027 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:08 smithi144 systemd[1]: Stopped Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:09.028 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:08 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 22.182s CPU time. 2024-07-26T11:06:09.028 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:08 smithi144 systemd[1]: Starting Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:06:09.716 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:09 smithi144 podman[102473]: 2024-07-26 11:06:09.577789286 +0000 UTC m=+0.130341912 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:06:10.311 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 podman[102473]: 2024-07-26 11:06:10.128254889 +0000 UTC m=+0.680807484 container create e5194ea418b74534bd198533b29b9faa03f5ed768b9236fec60f2627de5670d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:06:10.620 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mon[32129]: pgmap v572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 292 KiB/s wr, 35 op/s 2024-07-26T11:06:10.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mon[32129]: Replacing daemon mds.a as rank 1 with standby daemon mds.i 2024-07-26T11:06:10.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mon[32129]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:06:10.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:10.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mon[32129]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T11:06:10.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:06:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:10 smithi160 ceph-mon[36715]: pgmap v572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 292 KiB/s wr, 35 op/s 2024-07-26T11:06:10.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:10 smithi160 ceph-mon[36715]: Replacing daemon mds.a as rank 1 with standby daemon mds.i 2024-07-26T11:06:10.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:10 smithi160 ceph-mon[36715]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:06:10.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:10 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:10.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:10 smithi160 ceph-mon[36715]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T11:06:10.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:10 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:06:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:10 smithi183 ceph-mon[37273]: pgmap v572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 292 KiB/s wr, 35 op/s 2024-07-26T11:06:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:10 smithi183 ceph-mon[37273]: Replacing daemon mds.a as rank 1 with standby daemon mds.i 2024-07-26T11:06:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:10 smithi183 ceph-mon[37273]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:06:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:10 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:10 smithi183 ceph-mon[37273]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T11:06:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:10 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:06:10.963 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 podman[102473]: 2024-07-26 11:06:10.642318998 +0000 UTC m=+1.194871615 container init e5194ea418b74534bd198533b29b9faa03f5ed768b9236fec60f2627de5670d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T11:06:10.963 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 podman[102473]: 2024-07-26 11:06:10.649903452 +0000 UTC m=+1.202456075 container start e5194ea418b74534bd198533b29b9faa03f5ed768b9236fec60f2627de5670d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:06:10.963 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 bash[102473]: e5194ea418b74534bd198533b29b9faa03f5ed768b9236fec60f2627de5670d8 2024-07-26T11:06:10.963 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 systemd[1]: Started Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:10.963 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-mds[102530]: -- 172.21.15.144:0/421343118 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56414bcfd860 con 0x56414fe9f180 2024-07-26T11:06:10.963 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:10 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: starting mds.a at 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 -- 172.21.15.144:0/3274889982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 msgr2=0x7efc7c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 --2- 172.21.15.144:0/3274889982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c10fb40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc74009ca0 tx=0x7efc7402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 -- 172.21.15.144:0/3274889982 shutdown_connections 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 --2- 172.21.15.144:0/3274889982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c10fb40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 --2- 172.21.15.144:0/3274889982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c104b40 0x7efc7c10d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 --2- 172.21.15.144:0/3274889982 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c1041f0 0x7efc7c104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 -- 172.21.15.144:0/3274889982 >> 172.21.15.144:0/3274889982 conn(0x7efc7c0fdcf0 msgr2=0x7efc7c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 -- 172.21.15.144:0/3274889982 shutdown_connections 2024-07-26T11:06:11.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.059+0000 7efc80a00640 1 -- 172.21.15.144:0/3274889982 wait complete. 2024-07-26T11:06:11.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.061+0000 7efc80a00640 1 Processor -- start 2024-07-26T11:06:11.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.061+0000 7efc80a00640 1 -- start start 2024-07-26T11:06:11.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc80a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1041f0 0x7efc7c199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc80a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c104b40 0x7efc7c19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc80a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc80a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c113310 con 0x7efc7c10d650 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc80a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c113190 con 0x7efc7c104b40 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc80a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c113490 con 0x7efc7c1041f0 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc7b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1041f0 0x7efc7c199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc72a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c104b40 0x7efc7c19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37376/0 (socket says 172.21.15.144:37376) 2024-07-26T11:06:11.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc7b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1041f0 0x7efc7c199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39098/0 (socket says 172.21.15.144:39098) 2024-07-26T11:06:11.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.062+0000 7efc7be00640 1 -- 172.21.15.144:0/2319684483 learned_addr learned my addr 172.21.15.144:0/2319684483 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc72a00640 1 -- 172.21.15.144:0/2319684483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1041f0 msgr2=0x7efc7c199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc72a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1041f0 0x7efc7c199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc72a00640 1 -- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 msgr2=0x7efc7c19f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc72a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c19f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc72a00640 1 -- 172.21.15.144:0/2319684483 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7c19fc90 con 0x7efc7c104b40 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc72a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c104b40 0x7efc7c19a2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06a4c0 tx=0x7efc6400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc78c00640 1 -- 172.21.15.144:0/2319684483 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6400be60 con 0x7efc7c104b40 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7c19ff80 con 0x7efc7c104b40 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc78c00640 1 -- 172.21.15.144:0/2319684483 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc640035f0 con 0x7efc7c104b40 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc78c00640 1 -- 172.21.15.144:0/2319684483 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64019670 con 0x7efc7c104b40 2024-07-26T11:06:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.063+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc7c101e10 con 0x7efc7c104b40 2024-07-26T11:06:11.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.066+0000 7efc88c93740 1 Processor -- start 2024-07-26T11:06:11.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 -- start start 2024-07-26T11:06:11.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add75030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 0x5590add75570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add75ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5590adcebbf0 con 0x5590ad9d0070 2024-07-26T11:06:11.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5590adcdb0f0 con 0x5590ad9a71b0 2024-07-26T11:06:11.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc88c93740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5590adcdf130 con 0x5590ad9af390 2024-07-26T11:06:11.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add75030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc7b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 0x5590add75570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.067+0000 7efc7b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 0x5590add75570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36610/0 (socket says 172.21.15.144:36610) 2024-07-26T11:06:11.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add75030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37390/0 (socket says 172.21.15.144:37390) 2024-07-26T11:06:11.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 -- 172.21.15.144:0/1935454888 learned_addr learned my addr 172.21.15.144:0/1935454888 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:11.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc78c00640 1 -- 172.21.15.144:0/2319684483 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7efc64009310 con 0x7efc7c104b40 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 -- 172.21.15.144:0/1935454888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 msgr2=0x5590add75ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add75ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 -- 172.21.15.144:0/1935454888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 msgr2=0x5590add75030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add75030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 -- 172.21.15.144:0/1935454888 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ada3b650 con 0x5590ad9a71b0 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7be00640 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add75030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.068+0000 7efc7b400640 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 0x5590add75570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009cf0 tx=0x7efc6c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480f1e0c10c023fb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.069+0000 7efc6be00640 1 -- 172.21.15.144:0/1935454888 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c004230 con 0x5590ad9a71b0 2024-07-26T11:06:11.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.069+0000 7efc6be00640 1 -- 172.21.15.144:0/1935454888 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c0043d0 con 0x5590ad9a71b0 2024-07-26T11:06:11.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.069+0000 7efc6be00640 1 -- 172.21.15.144:0/1935454888 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c019820 con 0x5590ad9a71b0 2024-07-26T11:06:11.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc78c00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efc5c0b3d40 0x7efc5c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 -- 172.21.15.144:0/1935454888 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 msgr2=0x5590add75570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 0x5590add75570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009cf0 tx=0x7efc6c011810 comp rx=0 tx=0).stop 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 -- 172.21.15.144:0/1935454888 shutdown_connections 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc78c00640 1 -- 172.21.15.144:0/2319684483 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7efc640e7bf0 con 0x7efc7c104b40 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add75ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9a71b0 0x5590add75570 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 --2- 172.21.15.144:0/1935454888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add75030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 -- 172.21.15.144:0/1935454888 >> 172.21.15.144:0/1935454888 conn(0x5590ad9c4460 msgr2=0x5590add72f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc7be00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efc5c0b3d40 0x7efc5c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 -- 172.21.15.144:0/1935454888 shutdown_connections 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.070+0000 7efc88c93740 1 -- 172.21.15.144:0/1935454888 wait complete. 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc7be00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efc5c0b3d40 0x7efc5c0b6250 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc74009a60 tx=0x7efc74005d80 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 -- start start 2024-07-26T11:06:11.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 Processor -- start 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9a71b0 0x5590add80720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add80c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9d0070 0x5590add811c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5590adb3e700 con 0x5590ad9a71b0 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5590adc776c0 con 0x5590ad9d0070 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.071+0000 7efc88c93740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5590add83810 con 0x5590ad9af390 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc88c93740 10 client.0 _ll_register_callbacks cb 0x5590adc87170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc7b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9a71b0 0x5590add80720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc72a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add80c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc7b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9a71b0 0x5590add80720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37400/0 (socket says 172.21.15.144:37400) 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc7b400640 1 -- 172.21.15.144:0/1329369624 learned_addr learned my addr 172.21.15.144:0/1329369624 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc7be00640 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9d0070 0x5590add811c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc72a00640 1 -- 172.21.15.144:0/1329369624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9d0070 msgr2=0x5590add811c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc72a00640 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9d0070 0x5590add811c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc72a00640 1 -- 172.21.15.144:0/1329369624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9a71b0 msgr2=0x5590add80720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc72a00640 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9a71b0 0x5590add80720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.072+0000 7efc72a00640 1 -- 172.21.15.144:0/1329369624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590adc8e860 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc72a00640 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add80c60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efc640e4f10 tx=0x7efc64003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e0a1a05ba3dee7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc68c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc68c00640 1 -- 172.21.15.144:0/1329369624 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc640e53a0 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc6c0043d0 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc68c00640 1 -- 172.21.15.144:0/1329369624 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7efc640a8ba0 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc68c00640 1 -- 172.21.15.144:0/1329369624 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7efc640b7480 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc88c93740 10 client.26068 fetch_fsmap learned FSMap version 36 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5590add8f4e0 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.073+0000 7efc68c00640 1 -- 172.21.15.144:0/1329369624 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc6c0043d0 con 0x5590ad9af390 2024-07-26T11:06:11.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc68c00640 1 -- 172.21.15.144:0/1329369624 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 36) ==== 7761+0+0 (secure 0 0 0) 0x7efc640ad150 con 0x5590ad9af390 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc88c93740 10 client.26068 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc88c93740 10 client.26068 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc88c93740 20 client.26068 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc88c93740 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad98e9f0 0x5590add92950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc88c93740 4 client.26068 mds_command: new command op to 24691 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 --> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590adcedb40 con 0x5590ad98e9f0 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.074+0000 7efc7b400640 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad98e9f0 0x5590add92950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.075+0000 7efc7b400640 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad98e9f0 0x5590add92950 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=378291615c23a6ec server_cookie=e4ecd5e5ddee01cb in_seq=0 out_seq=0 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.075+0000 7efc68c00640 10 client.26068 ms_handle_connect on v2:172.21.15.183:6838/3458220569 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc68c00640 1 -- 172.21.15.144:0/1329369624 <== mds.1 v2:172.21.15.183:6838/3458220569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5590adcedb40 con 0x5590ad98e9f0 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc68c00640 10 client.26068 handle_command_reply: tid=0 2024-07-26T11:06:11.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 client.26068 shutdown 2024-07-26T11:06:11.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 msgr2=0x5590add80c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add80c60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efc640e4f10 tx=0x7efc64003c60 comp rx=0 tx=0).stop 2024-07-26T11:06:11.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 shutdown_connections 2024-07-26T11:06:11.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad98e9f0 0x5590add92950 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9d0070 0x5590add811c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9af390 0x5590add80c60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 --2- 172.21.15.144:0/1329369624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9a71b0 0x5590add80720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc68c00640 0 client.26068 ms_handle_reset on v2:172.21.15.183:6838/3458220569 2024-07-26T11:06:11.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 >> 172.21.15.144:0/1329369624 conn(0x5590ad9c4460 msgr2=0x5590add78210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:11.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 shutdown_connections 2024-07-26T11:06:11.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 1 -- 172.21.15.144:0/1329369624 wait complete. 2024-07-26T11:06:11.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.076+0000 7efc88c93740 20 client.26068 trim_cache size 0 max 16384 2024-07-26T11:06:11.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 Processor -- start 2024-07-26T11:06:11.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 -- start start 2024-07-26T11:06:11.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add7be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add76a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add80da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c1b0860 con 0x5590ad9d0070 2024-07-26T11:06:11.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c1ae7c0 con 0x5590ad9af390 2024-07-26T11:06:11.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.091+0000 7efc88c93740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c1ac610 con 0x5590ad9a71b0 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add80da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add80da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37410/0 (socket says 172.21.15.144:37410) 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 -- 172.21.15.144:0/3178996648 learned_addr learned my addr 172.21.15.144:0/3178996648 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7b400640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add7be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc72a00640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add76a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 -- 172.21.15.144:0/3178996648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 msgr2=0x5590add7be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add7be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 -- 172.21.15.144:0/3178996648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 msgr2=0x5590add76a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add76a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7be00640 1 -- 172.21.15.144:0/3178996648 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590add77320 con 0x5590ad9d0070 2024-07-26T11:06:11.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc7b400640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add7be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:11.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.092+0000 7efc72a00640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add76a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:11.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.093+0000 7efc7be00640 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add80da0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efc74006010 tx=0x7efc74041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc2d41098d91fc4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:11.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.093+0000 7efc6b400640 1 -- 172.21.15.144:0/3178996648 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc74031650 con 0x5590ad9d0070 2024-07-26T11:06:11.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.093+0000 7efc6b400640 1 -- 172.21.15.144:0/3178996648 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc74036050 con 0x5590ad9d0070 2024-07-26T11:06:11.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.093+0000 7efc6b400640 1 -- 172.21.15.144:0/3178996648 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc74004540 con 0x5590ad9d0070 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 -- 172.21.15.144:0/3178996648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 msgr2=0x5590add80da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add80da0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efc74006010 tx=0x7efc74041000 comp rx=0 tx=0).stop 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 -- 172.21.15.144:0/3178996648 shutdown_connections 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add80da0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add76a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 --2- 172.21.15.144:0/3178996648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add7be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 -- 172.21.15.144:0/3178996648 >> 172.21.15.144:0/3178996648 conn(0x5590ad9c4460 msgr2=0x5590add80690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 -- 172.21.15.144:0/3178996648 shutdown_connections 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.094+0000 7efc88c93740 1 -- 172.21.15.144:0/3178996648 wait complete. 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 -- start start 2024-07-26T11:06:11.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 Processor -- start 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add88e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add842f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add84830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c1acc80 con 0x5590ad9d0070 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c1ae0f0 con 0x5590ad9af390 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c1ad350 con 0x5590ad9a71b0 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.095+0000 7efc88c93740 10 client.0 _ll_register_callbacks cb 0x7efc640adbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc7be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add84830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc7b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add88e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc7b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add88e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39114/0 (socket says 172.21.15.144:39114) 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc7b400640 1 -- 172.21.15.144:0/2437159445 learned_addr learned my addr 172.21.15.144:0/2437159445 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add842f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 -- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 msgr2=0x5590add88e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add88e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 -- 172.21.15.144:0/2437159445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 msgr2=0x5590add84830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add84830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 -- 172.21.15.144:0/2437159445 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590add84d70 con 0x5590ad9af390 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc72a00640 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add842f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efc640ae410 tx=0x7efc640e7ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3811d92d57738c0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.096+0000 7efc68c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc68c00640 1 -- 172.21.15.144:0/2437159445 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc640a8db0 con 0x5590ad9af390 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc74036050 con 0x5590ad9af390 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc68c00640 1 -- 172.21.15.144:0/2437159445 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7efc640ad6e0 con 0x5590ad9af390 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc68c00640 1 -- 172.21.15.144:0/2437159445 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7efc6408f050 con 0x5590ad9af390 2024-07-26T11:06:11.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc88c93740 10 client.26093 fetch_fsmap learned FSMap version 36 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5590addfad00 con 0x5590ad9af390 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.097+0000 7efc68c00640 1 -- 172.21.15.144:0/2437159445 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc74036050 con 0x5590ad9af390 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc68c00640 1 -- 172.21.15.144:0/2437159445 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 36) ==== 7761+0+0 (secure 0 0 0) 0x7efc640cf320 con 0x5590ad9af390 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc88c93740 10 client.26093 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc88c93740 10 client.26093 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc88c93740 20 client.26093 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc88c93740 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad963da0 0x5590addfd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc88c93740 4 client.26093 mds_command: new command op to 24691 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 --> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5590adce95e0 con 0x5590ad963da0 2024-07-26T11:06:11.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc7b400640 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad963da0 0x5590addfd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc7b400640 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad963da0 0x5590addfd2d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa1996bcc38ac3a8 server_cookie=2472585f8c4712ca in_seq=0 out_seq=0 2024-07-26T11:06:11.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.098+0000 7efc68c00640 10 client.26093 ms_handle_connect on v2:172.21.15.183:6838/3458220569 2024-07-26T11:06:11.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc68c00640 1 -- 172.21.15.144:0/2437159445 <== mds.1 v2:172.21.15.183:6838/3458220569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5590adce95e0 con 0x5590ad963da0 2024-07-26T11:06:11.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc68c00640 10 client.26093 handle_command_reply: tid=0 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 client.26093 shutdown 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 msgr2=0x5590add842f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add842f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efc640ae410 tx=0x7efc640e7ee0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 shutdown_connections 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] conn(0x5590ad963da0 0x5590addfd2d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5590ad9d0070 0x5590add84830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5590ad9af390 0x5590add842f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 --2- 172.21.15.144:0/2437159445 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5590ad9a71b0 0x5590add88e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 >> 172.21.15.144:0/2437159445 conn(0x5590ad9c4460 msgr2=0x5590add82590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc68c00640 0 client.26093 ms_handle_reset on v2:172.21.15.183:6838/3458220569 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 shutdown_connections 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 1 -- 172.21.15.144:0/2437159445 wait complete. 2024-07-26T11:06:11.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.099+0000 7efc88c93740 20 client.26093 trim_cache size 0 max 16384 2024-07-26T11:06:11.105 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2024-07-26T11:06:11.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efc5c0b3d40 0x7efc5c0b6250 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc74009a60 tx=0x7efc74005d80 comp rx=0 tx=0).stop 2024-07-26T11:06:11.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c104b40 msgr2=0x7efc7c19a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c104b40 0x7efc7c19a2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06a4c0 tx=0x7efc6400b690 comp rx=0 tx=0).stop 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 shutdown_connections 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efc5c0b3d40 0x7efc5c0b6250 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc7c10d650 0x7efc7c19f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efc7c104b40 0x7efc7c19a2c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 --2- 172.21.15.144:0/2319684483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1041f0 0x7efc7c199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 >> 172.21.15.144:0/2319684483 conn(0x7efc7c0fdcf0 msgr2=0x7efc7c107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 shutdown_connections 2024-07-26T11:06:11.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:11.100+0000 7efc80a00640 1 -- 172.21.15.144:0/2319684483 wait complete. 2024-07-26T11:06:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:11.533+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:11.535+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:11.535+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:11.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:11.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:12.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:12 smithi144 ceph-mon[32129]: pgmap v574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-07-26T11:06:12.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:06:12.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:12 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:boot 2024-07-26T11:06:12.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:12 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:resolve 2024-07-26T11:06:12.789 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:12 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:12 smithi160 ceph-mon[36715]: pgmap v574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-07-26T11:06:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:06:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:12 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:boot 2024-07-26T11:06:12.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:12 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:resolve 2024-07-26T11:06:12.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:12 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:12 smithi183 ceph-mon[37273]: pgmap v574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-07-26T11:06:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:06:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:12 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:boot 2024-07-26T11:06:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:12 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:resolve 2024-07-26T11:06:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:12 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:13.090 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:06:13.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:06:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:13 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:reconnect 2024-07-26T11:06:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:13 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:13 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0119997 2024-07-26T11:06:13.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:13 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:reconnect 2024-07-26T11:06:13.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:13 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:13.907 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:13 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0119997 2024-07-26T11:06:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:13 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:reconnect 2024-07-26T11:06:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:13 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:13 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0119997 2024-07-26T11:06:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-mon[37273]: pgmap v575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-07-26T11:06:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:rejoin 2024-07-26T11:06:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-mon[37273]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: *** Caught signal (Aborted) ** 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: in thread 7fb15476c640 thread_name:ms_dispatch 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb159b436f0] 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb159b9094c] 2024-07-26T11:06:14.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 3: raise() 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 4: abort() 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb159b2d71b] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 6: /lib64/libc.so.6(+0x37386) [0x7fb159b3c386] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ac2d286b9] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ac2ee680c] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559ac2e262ca] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559ac2df6753] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ac2df7a1b] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ac2ddb752] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ac2d29a56] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ac2d2d637] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ac2d2df5e] 2024-07-26T11:06:14.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ac2d1103a] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb15a410407] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 18: (DispatchQueue::entry()+0x60f) [0x7fb15a410cf9] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb15a4ed5a1] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 20: (Thread::entry_wrapper()+0x43) [0x7fb15a29acdd] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb15a29acf9] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb159b8ec02] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb159c13c40] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.591+0000 7fb15476c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: in thread 7fb15476c640 thread_name:ms_dispatch 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb159b436f0] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb159b9094c] 2024-07-26T11:06:14.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 3: raise() 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 4: abort() 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb159b2d71b] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 6: /lib64/libc.so.6(+0x37386) [0x7fb159b3c386] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ac2d286b9] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ac2ee680c] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559ac2e262ca] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559ac2df6753] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ac2df7a1b] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ac2ddb752] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ac2d29a56] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ac2d2d637] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ac2d2df5e] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ac2d1103a] 2024-07-26T11:06:14.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb15a410407] 2024-07-26T11:06:14.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 18: (DispatchQueue::entry()+0x60f) [0x7fb15a410cf9] 2024-07-26T11:06:14.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb15a4ed5a1] 2024-07-26T11:06:14.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 20: (Thread::entry_wrapper()+0x43) [0x7fb15a29acdd] 2024-07-26T11:06:14.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb15a29acf9] 2024-07-26T11:06:14.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb159b8ec02] 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb159c13c40] 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 0> 2024-07-26T11:06:14.591+0000 7fb15476c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: in thread 7fb15476c640 thread_name:ms_dispatch 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb159b436f0] 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb159b9094c] 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 3: raise() 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 4: abort() 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb159b2d71b] 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 6: /lib64/libc.so.6(+0x37386) [0x7fb159b3c386] 2024-07-26T11:06:14.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ac2d286b9] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ac2ee680c] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559ac2e262ca] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559ac2df6753] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ac2df7a1b] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ac2ddb752] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ac2d29a56] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ac2d2d637] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ac2d2df5e] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ac2d1103a] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb15a410407] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 18: (DispatchQueue::entry()+0x60f) [0x7fb15a410cf9] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb15a4ed5a1] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 20: (Thread::entry_wrapper()+0x43) [0x7fb15a29acdd] 2024-07-26T11:06:14.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb15a29acf9] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb159b8ec02] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb159c13c40] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 0> 2024-07-26T11:06:14.591+0000 7fb15476c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: in thread 7fb15476c640 thread_name:ms_dispatch 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb159b436f0] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb159b9094c] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 3: raise() 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 4: abort() 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb159b2d71b] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 6: /lib64/libc.so.6(+0x37386) [0x7fb159b3c386] 2024-07-26T11:06:14.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ac2d286b9] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ac2ee680c] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559ac2e262ca] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559ac2df6753] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ac2df7a1b] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ac2ddb752] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ac2d29a56] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ac2d2d637] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ac2d2df5e] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ac2d1103a] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb15a410407] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 18: (DispatchQueue::entry()+0x60f) [0x7fb15a410cf9] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb15a4ed5a1] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 20: (Thread::entry_wrapper()+0x43) [0x7fb15a29acdd] 2024-07-26T11:06:14.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb15a29acf9] 2024-07-26T11:06:14.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb159b8ec02] 2024-07-26T11:06:14.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb159c13c40] 2024-07-26T11:06:14.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:14.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[68934]: 2024-07-26T11:06:14.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:14 smithi183 podman[71614]: 2024-07-26 11:06:14.655438354 +0000 UTC m=+0.020294358 container died 73dd7ee98bd64c9841f78a45039e53411b63ccfca0d38a3194c136c8c6c41815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:06:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:14 smithi144 ceph-mon[32129]: pgmap v575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-07-26T11:06:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:14 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:rejoin 2024-07-26T11:06:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:14 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:14 smithi144 ceph-mon[32129]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:06:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:14 smithi160 ceph-mon[36715]: pgmap v575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-07-26T11:06:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:14 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:rejoin 2024-07-26T11:06:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:14 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:14 smithi160 ceph-mon[36715]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:06:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:15 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:15 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:06:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:active 2024-07-26T11:06:15.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:15 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:15.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:15 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:15.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:15 smithi183 podman[71614]: 2024-07-26 11:06:15.372503796 +0000 UTC m=+0.737359787 container remove 73dd7ee98bd64c9841f78a45039e53411b63ccfca0d38a3194c136c8c6c41815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T11:06:15.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:06:15.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:06:15.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 2.800s CPU time. 2024-07-26T11:06:15.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 -- 172.21.15.144:0/3701750305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346410af10 msgr2=0x7f346410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 --2- 172.21.15.144:0/3701750305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346410af10 0x7f346410b320 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3450009900 tx=0x7f345002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:15 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:15.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:15 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:06:15.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:active 2024-07-26T11:06:15.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:15 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:15.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:15 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:15.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 -- 172.21.15.144:0/3701750305 shutdown_connections 2024-07-26T11:06:15.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 --2- 172.21.15.144:0/3701750305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f346410b860 0x7f3464112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 --2- 172.21.15.144:0/3701750305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346410af10 0x7f346410b320 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 --2- 172.21.15.144:0/3701750305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34641055d0 0x7f34641059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 -- 172.21.15.144:0/3701750305 >> 172.21.15.144:0/3701750305 conn(0x7f3464101040 msgr2=0x7f3464103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 -- 172.21.15.144:0/3701750305 shutdown_connections 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.886+0000 7f346ac00640 1 -- 172.21.15.144:0/3701750305 wait complete. 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.887+0000 7f346ac00640 1 Processor -- start 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 -- start start 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34641055d0 0x7f346419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f346410af10 0x7f346419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f346410b860 0x7f34641a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3464115520 con 0x7f34641055d0 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34641153a0 con 0x7f346410af10 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f346ac00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34641156a0 con 0x7f346410b860 2024-07-26T11:06:15.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.888+0000 7f3469800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34641055d0 0x7f346419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f3469800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34641055d0 0x7f346419bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37432/0 (socket says 172.21.15.144:37432) 2024-07-26T11:06:15.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f3469800640 1 -- 172.21.15.144:0/3877072062 learned_addr learned my addr 172.21.15.144:0/3877072062 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:15.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f3468e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f346410af10 0x7f346419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f346410b860 0x7f34641a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 -- 172.21.15.144:0/3877072062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f346410af10 msgr2=0x7f346419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f346410af10 0x7f346419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 -- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34641055d0 msgr2=0x7f346419bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34641055d0 0x7f346419bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 -- 172.21.15.144:0/3877072062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34641a1ec0 con 0x7f346410b860 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.889+0000 7f346a200640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f346410b860 0x7f34641a1790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f345c009fd0 tx=0x7f345c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.890+0000 7f3462000640 1 -- 172.21.15.144:0/3877072062 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c00ce20 con 0x7f346410b860 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.890+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34641a21b0 con 0x7f346410b860 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.890+0000 7f3462000640 1 -- 172.21.15.144:0/3877072062 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345c005020 con 0x7f346410b860 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.890+0000 7f3462000640 1 -- 172.21.15.144:0/3877072062 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c010690 con 0x7f346410b860 2024-07-26T11:06:15.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.890+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34641a8910 con 0x7f346410b860 2024-07-26T11:06:15.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.893+0000 7f3472e16740 1 Processor -- start 2024-07-26T11:06:15.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.893+0000 7f3462000640 1 -- 172.21.15.144:0/3877072062 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f345c0054d0 con 0x7f346410b860 2024-07-26T11:06:15.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 -- start start 2024-07-26T11:06:15.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 0x55f0e16c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1301670 0x55f0e16c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f0e163dbe0 con 0x55f0e1322350 2024-07-26T11:06:15.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f0e162d0e0 con 0x55f0e12f9490 2024-07-26T11:06:15.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3472e16740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f0e1631120 con 0x55f0e1301670 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3469800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 0x55f0e16c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3469800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 0x55f0e16c7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36664/0 (socket says 172.21.15.144:36664) 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.894+0000 7f3469800640 1 -- 172.21.15.144:0/3213717203 learned_addr learned my addr 172.21.15.144:0/3213717203 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3468e00640 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3468e00640 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1301670 0x55f0e16c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3469800640 1 -- 172.21.15.144:0/3213717203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1301670 msgr2=0x55f0e16c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3469800640 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1301670 0x55f0e16c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3469800640 1 -- 172.21.15.144:0/3213717203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 msgr2=0x55f0e16c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3469800640 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3469800640 1 -- 172.21.15.144:0/3213717203 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0e138d930 con 0x55f0e12f9490 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3468e00640 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1301670 0x55f0e16c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3469800640 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 0x55f0e16c7490 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3458009dc0 tx=0x7f3458011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94054d8d8071179 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:15.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.895+0000 7f3462000640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f344c0b3d40 0x7f344c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.896+0000 7f3456a00640 1 -- 172.21.15.144:0/3213717203 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458005220 con 0x55f0e12f9490 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.896+0000 7f3462000640 1 -- 172.21.15.144:0/3877072062 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7f345c0e4b30 con 0x7f346410b860 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.896+0000 7f3456a00640 1 -- 172.21.15.144:0/3213717203 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3458004c50 con 0x55f0e12f9490 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.896+0000 7f3456a00640 1 -- 172.21.15.144:0/3213717203 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458019810 con 0x55f0e12f9490 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.896+0000 7f3468e00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f344c0b3d40 0x7f344c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 -- 172.21.15.144:0/3213717203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 msgr2=0x55f0e16c7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3468e00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f344c0b3d40 0x7f344c0b6250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f345002a9e0 tx=0x7f3450005780 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 0x55f0e16c7490 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3458009dc0 tx=0x7f3458011810 comp rx=0 tx=0).stop 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 -- 172.21.15.144:0/3213717203 shutdown_connections 2024-07-26T11:06:15.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1301670 0x55f0e16c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e12f9490 0x55f0e16c7490 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 --2- 172.21.15.144:0/3213717203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 -- 172.21.15.144:0/3213717203 >> 172.21.15.144:0/3213717203 conn(0x55f0e1316740 msgr2=0x55f0e16c4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 -- 172.21.15.144:0/3213717203 shutdown_connections 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.897+0000 7f3472e16740 1 -- 172.21.15.144:0/3213717203 wait complete. 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 -- start start 2024-07-26T11:06:15.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 Processor -- start 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 0x55f0e16d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16dea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55f0e1490890 con 0x55f0e1322350 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f0e15c9320 con 0x55f0e1301670 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.898+0000 7f3472e16740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f0e16d5730 con 0x55f0e12f9490 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3472e16740 10 client.0 _ll_register_callbacks cb 0x55f0e15f5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:15.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 0x55f0e16d2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 0x55f0e16d2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39162/0 (socket says 172.21.15.144:39162) 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f346a200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16dea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 -- 172.21.15.144:0/3152445261 learned_addr learned my addr 172.21.15.144:0/3152445261 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3468e00640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 -- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 msgr2=0x55f0e16d2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 -- 172.21.15.144:0/3152445261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 msgr2=0x55f0e16dea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16dea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3469800640 1 -- 172.21.15.144:0/3152445261 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0e16d3400 con 0x55f0e12f9490 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.899+0000 7f3468e00640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f3469800640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 0x55f0e16d2670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3458005700 tx=0x7f3458004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bacde9ad5e1c2175 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f344b400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:06:15.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f344b400640 1 -- 172.21.15.144:0/3152445261 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458027070 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f344b400640 1 -- 172.21.15.144:0/3152445261 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7f3458005210 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3458004c50 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f344b400640 1 -- 172.21.15.144:0/3152445261 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3458033050 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f3472e16740 10 client.26083 fetch_fsmap learned FSMap version 41 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f0e16e15c0 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.900+0000 7f344b400640 1 -- 172.21.15.144:0/3152445261 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3458004c50 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f344b400640 1 -- 172.21.15.144:0/3152445261 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 41) ==== 8362+0+0 (secure 0 0 0) 0x7f3458003cc0 con 0x55f0e12f9490 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3472e16740 10 client.26083 fetch_fsmap finished waiting for FSMap version 41 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3472e16740 10 client.26083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3472e16740 20 client.26083 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3472e16740 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e12e0cd0 0x55f0e16e4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3472e16740 4 client.26083 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0e163fb30 con 0x55f0e12e0cd0 2024-07-26T11:06:15.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.901+0000 7f3468e00640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e12e0cd0 0x55f0e16e4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.902+0000 7f3468e00640 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e12e0cd0 0x55f0e16e4980 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64be2c19ef8bbd19 server_cookie=8895de87ac340150 in_seq=0 out_seq=0 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.902+0000 7f344b400640 10 client.26083 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f344b400640 1 -- 172.21.15.144:0/3152445261 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f0e163fb30 con 0x55f0e12e0cd0 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f344b400640 10 client.26083 handle_command_reply: tid=0 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 client.26083 shutdown 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 msgr2=0x55f0e16d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 0x55f0e16d2670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3458005700 tx=0x7f3458004270 comp rx=0 tx=0).stop 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 shutdown_connections 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e1322350 0x55f0e16dea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e12e0cd0 0x55f0e16e4980 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 --2- 172.21.15.144:0/3152445261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e12f9490 0x55f0e16d2670 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 >> 172.21.15.144:0/3152445261 conn(0x55f0e1316740 msgr2=0x55f0e16ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f344b400640 0 client.26083 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:15.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 shutdown_connections 2024-07-26T11:06:15.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.903+0000 7f3472e16740 1 -- 172.21.15.144:0/3152445261 wait complete. 2024-07-26T11:06:15.908 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.904+0000 7f3472e16740 20 client.26083 trim_cache size 0 max 16384 2024-07-26T11:06:15.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 Processor -- start 2024-07-26T11:06:15.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 -- start start 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16de840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16ded80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16ea050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34641b5340 con 0x55f0e12b6080 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34641b2bb0 con 0x55f0e1301670 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.923+0000 7f3472e16740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34641b0a00 con 0x55f0e1322350 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3469800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16de840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16ded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f346a200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16ea050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16ded80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39178/0 (socket says 172.21.15.144:39178) 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3469800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16de840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36680/0 (socket says 172.21.15.144:36680) 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 -- 172.21.15.144:0/2303920863 learned_addr learned my addr 172.21.15.144:0/2303920863 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 -- 172.21.15.144:0/2303920863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 msgr2=0x55f0e16de840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16de840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 -- 172.21.15.144:0/2303920863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 msgr2=0x55f0e16ea050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16ea050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 -- 172.21.15.144:0/2303920863 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0e16c9240 con 0x55f0e1322350 2024-07-26T11:06:15.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.924+0000 7f3468e00640 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16ded80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f346406c270 tx=0x7f345003f070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62dcba0286feb9e0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:15.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.925+0000 7f3456000640 1 -- 172.21.15.144:0/2303920863 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450035040 con 0x55f0e1322350 2024-07-26T11:06:15.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.925+0000 7f3456000640 1 -- 172.21.15.144:0/2303920863 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345005a070 con 0x55f0e1322350 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.925+0000 7f3456000640 1 -- 172.21.15.144:0/2303920863 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34500318c0 con 0x55f0e1322350 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 -- 172.21.15.144:0/2303920863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 msgr2=0x55f0e16ded80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16ded80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f346406c270 tx=0x7f345003f070 comp rx=0 tx=0).stop 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 -- 172.21.15.144:0/2303920863 shutdown_connections 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16ea050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16ded80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 --2- 172.21.15.144:0/2303920863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16de840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 -- 172.21.15.144:0/2303920863 >> 172.21.15.144:0/2303920863 conn(0x55f0e1316740 msgr2=0x55f0e16d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 -- 172.21.15.144:0/2303920863 shutdown_connections 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 -- 172.21.15.144:0/2303920863 wait complete. 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 -- start start 2024-07-26T11:06:15.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.926+0000 7f3472e16740 1 Processor -- start 2024-07-26T11:06:15.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16db130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16d6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34641b1070 con 0x55f0e12b6080 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34641b24e0 con 0x55f0e1301670 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34641b1740 con 0x55f0e1322350 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3472e16740 10 client.0 _ll_register_callbacks cb 0x7f3458009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3469800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16db130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f346a200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16d6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f3468e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f346a200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16d6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39194/0 (socket says 172.21.15.144:39194) 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.927+0000 7f346a200640 1 -- 172.21.15.144:0/870294039 learned_addr learned my addr 172.21.15.144:0/870294039 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f346a200640 1 -- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 msgr2=0x55f0e16d65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f346a200640 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f346a200640 1 -- 172.21.15.144:0/870294039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 msgr2=0x55f0e16db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f346a200640 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16db130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f346a200640 1 -- 172.21.15.144:0/870294039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0e16d7350 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f346a200640 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16d6bc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f345c0a7350 tx=0x7f345c0abb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8285dd2e3b0e215 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f344b400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f344b400640 1 -- 172.21.15.144:0/870294039 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c0b4380 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f344b400640 1 -- 172.21.15.144:0/870294039 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7f345c0b4100 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.928+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f345005a070 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f344b400640 1 -- 172.21.15.144:0/870294039 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f345c0aa530 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 10 client.26089 fetch_fsmap learned FSMap version 41 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f0e174d040 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f344b400640 1 -- 172.21.15.144:0/870294039 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345005a070 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f344b400640 1 -- 172.21.15.144:0/870294039 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 41) ==== 8362+0+0 (secure 0 0 0) 0x7f345c0aa030 con 0x55f0e1322350 2024-07-26T11:06:15.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 10 client.26089 fetch_fsmap finished waiting for FSMap version 41 2024-07-26T11:06:15.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:15.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 20 client.26089 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:15.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e174d790 0x55f0e174fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:15.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 4 client.26089 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:06:15.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f0e163b5d0 con 0x55f0e174d790 2024-07-26T11:06:15.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.929+0000 7f3469800640 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e174d790 0x55f0e174fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:15.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.930+0000 7f3469800640 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e174d790 0x55f0e174fbb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252d80da8817b818 server_cookie=55206d6e33608b5c in_seq=0 out_seq=0 2024-07-26T11:06:15.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.930+0000 7f344b400640 10 client.26089 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:15.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.931+0000 7f344b400640 1 -- 172.21.15.144:0/870294039 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f0e163b5d0 con 0x55f0e174d790 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.931+0000 7f344b400640 10 client.26089 handle_command_reply: tid=0 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.931+0000 7f3472e16740 1 client.26089 shutdown 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.931+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 msgr2=0x55f0e16d6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.931+0000 7f3472e16740 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16d6bc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f345c0a7350 tx=0x7f345c0abb40 comp rx=0 tx=0).stop 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 shutdown_connections 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f0e1322350 0x55f0e16d6bc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55f0e174d790 0x55f0e174fbb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f0e1301670 0x55f0e16d65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 --2- 172.21.15.144:0/870294039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55f0e12b6080 0x55f0e16db130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f344b400640 0 client.26089 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 >> 172.21.15.144:0/870294039 conn(0x55f0e1316740 msgr2=0x55f0e16d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 shutdown_connections 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 1 -- 172.21.15.144:0/870294039 wait complete. 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.932+0000 7f3472e16740 20 client.26089 trim_cache size 0 max 16384 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stdout: "return_code": 0, 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stdout: "scrub_tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stdout: "mode": "asynchronous" 2024-07-26T11:06:15.937 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:06:15.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f344c0b3d40 msgr2=0x7f344c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f344c0b3d40 0x7f344c0b6250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f345002a9e0 tx=0x7f3450005780 comp rx=0 tx=0).stop 2024-07-26T11:06:15.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f346410b860 msgr2=0x7f34641a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:15.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f346410b860 0x7f34641a1790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f345c009fd0 tx=0x7f345c00ef10 comp rx=0 tx=0).stop 2024-07-26T11:06:15.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 shutdown_connections 2024-07-26T11:06:15.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f344c0b3d40 0x7f344c0b6250 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f346410b860 0x7f34641a1790 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f346410af10 0x7f346419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 --2- 172.21.15.144:0/3877072062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34641055d0 0x7f346419bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 >> 172.21.15.144:0/3877072062 conn(0x7f3464101040 msgr2=0x7f3464103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 shutdown_connections 2024-07-26T11:06:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:15.933+0000 7f346ac00640 1 -- 172.21.15.144:0/3877072062 wait complete. 2024-07-26T11:06:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:15 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:15 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:06:16.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/3458220569,v1:172.21.15.183:6839/3458220569] up:active 2024-07-26T11:06:16.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:15 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:16.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:15 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:16 smithi144 ceph-mon[32129]: pgmap v576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 10 op/s 2024-07-26T11:06:16.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:06:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:16 smithi183 ceph-mon[37273]: pgmap v576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 10 op/s 2024-07-26T11:06:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:16 smithi160 ceph-mon[36715]: pgmap v576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 10 op/s 2024-07-26T11:06:17.349 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:17.825 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:17 smithi144 ceph-mon[32129]: scrub queued for path: / 2024-07-26T11:06:17.825 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:17 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:06:17.825 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:17 smithi144 ceph-mon[32129]: scrub summary: active paths [/] 2024-07-26T11:06:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:17 smithi183 ceph-mon[37273]: scrub queued for path: / 2024-07-26T11:06:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:17 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:06:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:17 smithi183 ceph-mon[37273]: scrub summary: active paths [/] 2024-07-26T11:06:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:17 smithi160 ceph-mon[36715]: scrub queued for path: / 2024-07-26T11:06:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:17 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:06:18.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:17 smithi160 ceph-mon[36715]: scrub summary: active paths [/] 2024-07-26T11:06:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:18 smithi183 ceph-mon[37273]: pgmap v577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.8 KiB/s wr, 10 op/s 2024-07-26T11:06:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:18 smithi144 ceph-mon[32129]: pgmap v577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.8 KiB/s wr, 10 op/s 2024-07-26T11:06:19.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:18 smithi160 ceph-mon[36715]: pgmap v577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.8 KiB/s wr, 10 op/s 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 -- 172.21.15.144:0/3123827357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 msgr2=0x7fc8f0077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 --2- 172.21.15.144:0/3123827357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 0x7fc8f0077780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0009900 tx=0x7fc8e002a410 comp rx=0 tx=0).stop 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 -- 172.21.15.144:0/3123827357 shutdown_connections 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 --2- 172.21.15.144:0/3123827357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8f01141a0 0x7fc8f01165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 --2- 172.21.15.144:0/3123827357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f0113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 --2- 172.21.15.144:0/3123827357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 0x7fc8f0077780 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 -- 172.21.15.144:0/3123827357 >> 172.21.15.144:0/3123827357 conn(0x7fc8f01011f0 msgr2=0x7fc8f0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:20.042 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 -- 172.21.15.144:0/3123827357 shutdown_connections 2024-07-26T11:06:20.042 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.038+0000 7fc8f5600640 1 -- 172.21.15.144:0/3123827357 wait complete. 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 Processor -- start 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 -- start start 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 0x7fc8f01a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f01a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8f01141a0 0x7fc8f01a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f0119b20 con 0x7fc8f01141a0 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01199a0 con 0x7fc8f0077370 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.040+0000 7fc8f5600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f0119ca0 con 0x7fc8f0077cc0 2024-07-26T11:06:20.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f01a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f01a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57114/0 (socket says 172.21.15.144:57114) 2024-07-26T11:06:20.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 -- 172.21.15.144:0/1026408991 learned_addr learned my addr 172.21.15.144:0/1026408991 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:20.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8eea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 0x7fc8f01a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ef400640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8f01141a0 0x7fc8f01a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 -- 172.21.15.144:0/1026408991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 msgr2=0x7fc8f01a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 0x7fc8f01a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 -- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8f01141a0 msgr2=0x7fc8f01a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8f01141a0 0x7fc8f01a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.041+0000 7fc8ee000640 1 -- 172.21.15.144:0/1026408991 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f01a6480 con 0x7fc8f0077cc0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.042+0000 7fc8ee000640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f01a0ab0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00e9c0 tx=0x7fc8dc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.042+0000 7fc8e7400640 1 -- 172.21.15.144:0/1026408991 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00ccc0 con 0x7fc8f0077cc0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.042+0000 7fc8e7400640 1 -- 172.21.15.144:0/1026408991 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8dc0054d0 con 0x7fc8f0077cc0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.042+0000 7fc8e7400640 1 -- 172.21.15.144:0/1026408991 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc010430 con 0x7fc8f0077cc0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.042+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f01a6770 con 0x7fc8f0077cc0 2024-07-26T11:06:20.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.042+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8f01aced0 con 0x7fc8f0077cc0 2024-07-26T11:06:20.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.045+0000 7fc8fc2a8740 1 Processor -- start 2024-07-26T11:06:20.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 -- start start 2024-07-26T11:06:20.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5410df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d5411330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d504b390 0x55c9d5411870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55c9d5387b90 con 0x55c9d504b390 2024-07-26T11:06:20.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c9d5377090 con 0x55c9d50431b0 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c9d537b0d0 con 0x55c9d506c070 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8e7400640 1 -- 172.21.15.144:0/1026408991 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fc8dc005020 con 0x7fc8f0077cc0 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8eea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d5411330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8eea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d5411330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:44682/0 (socket says 172.21.15.144:44682) 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8eea00640 1 -- 172.21.15.144:0/2758023415 learned_addr learned my addr 172.21.15.144:0/2758023415 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.046+0000 7fc8ef400640 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5410df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ee000640 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d504b390 0x55c9d5411870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ef400640 1 -- 172.21.15.144:0/2758023415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 msgr2=0x55c9d5411330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ef400640 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d5411330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ef400640 1 -- 172.21.15.144:0/2758023415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d504b390 msgr2=0x55c9d5411870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ef400640 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d504b390 0x55c9d5411870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ef400640 1 -- 172.21.15.144:0/2758023415 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9d50d7650 con 0x55c9d506c070 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.047+0000 7fc8ef400640 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5410df0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009f00 tx=0x7fc8e8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e84265be1d45634c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.048+0000 7fc8db400640 1 -- 172.21.15.144:0/2758023415 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8004d20 con 0x55c9d506c070 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.048+0000 7fc8db400640 1 -- 172.21.15.144:0/2758023415 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8e8004030 con 0x55c9d506c070 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.048+0000 7fc8db400640 1 -- 172.21.15.144:0/2758023415 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8005280 con 0x55c9d506c070 2024-07-26T11:06:20.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.048+0000 7fc8e7400640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc8d40b3d40 0x7fc8d40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2758023415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 msgr2=0x55c9d5410df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5410df0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009f00 tx=0x7fc8e8011810 comp rx=0 tx=0).stop 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8e7400640 1 -- 172.21.15.144:0/1026408991 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7fc8dc0e4790 con 0x7fc8f0077cc0 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8eea00640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc8d40b3d40 0x7fc8d40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2758023415 shutdown_connections 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d504b390 0x55c9d5411870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d5411330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2758023415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5410df0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2758023415 >> 172.21.15.144:0/2758023415 conn(0x55c9d5060460 msgr2=0x55c9d540ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2758023415 shutdown_connections 2024-07-26T11:06:20.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.049+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2758023415 wait complete. 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.050+0000 7fc8fc2a8740 1 -- start start 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.050+0000 7fc8eea00640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc8d40b3d40 0x7fc8d40b6250 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55c9d540fa70 tx=0x7fc8e0009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.050+0000 7fc8fc2a8740 1 Processor -- start 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.050+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d541c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d541ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55c9d51da700 con 0x55c9d506c070 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c9d5313660 con 0x55c9d50431b0 2024-07-26T11:06:20.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c9d541f5d0 con 0x55c9d504b390 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8fc2a8740 10 client.0 _ll_register_callbacks cb 0x55c9d5323110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d541c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d541c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57128/0 (socket says 172.21.15.144:57128) 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 -- 172.21.15.144:0/1432773399 learned_addr learned my addr 172.21.15.144:0/1432773399 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ef400640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d541ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8eea00640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 -- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 msgr2=0x55c9d541c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 -- 172.21.15.144:0/1432773399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 msgr2=0x55c9d541ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d541ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.051+0000 7fc8ee000640 1 -- 172.21.15.144:0/1432773399 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9d532a800 con 0x55c9d504b390 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8ee000640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d541c930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc004650 tx=0x7fc8dc0aad00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be80e68c073f5ab7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8cfe00640 1 -- 172.21.15.144:0/1432773399 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00ce60 con 0x55c9d504b390 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8e8004030 con 0x55c9d504b390 2024-07-26T11:06:20.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8cfe00640 1 -- 172.21.15.144:0/1432773399 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7fc8dc0b3b50 con 0x55c9d504b390 2024-07-26T11:06:20.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8cfe00640 1 -- 172.21.15.144:0/1432773399 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc8dc0e1f00 con 0x55c9d504b390 2024-07-26T11:06:20.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8fc2a8740 10 client.26101 fetch_fsmap learned FSMap version 41 2024-07-26T11:06:20.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c9d542b300 con 0x55c9d504b390 2024-07-26T11:06:20.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.052+0000 7fc8cfe00640 1 -- 172.21.15.144:0/1432773399 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8e8004030 con 0x55c9d504b390 2024-07-26T11:06:20.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8cfe00640 1 -- 172.21.15.144:0/1432773399 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 41) ==== 8362+0+0 (secure 0 0 0) 0x7fc8dc0ae030 con 0x55c9d504b390 2024-07-26T11:06:20.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8fc2a8740 10 client.26101 fetch_fsmap finished waiting for FSMap version 41 2024-07-26T11:06:20.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8fc2a8740 10 client.26101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8fc2a8740 20 client.26101 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d502a9f0 0x55c9d542e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8fc2a8740 4 client.26101 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9d5389ae0 con 0x55c9d502a9f0 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.053+0000 7fc8eea00640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d502a9f0 0x55c9d542e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.054+0000 7fc8eea00640 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d502a9f0 0x55c9d542e850 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8118d48585f00f0e server_cookie=1dc253c08b0858f8 in_seq=0 out_seq=0 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.054+0000 7fc8cfe00640 10 client.26101 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8cfe00640 1 -- 172.21.15.144:0/1432773399 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c9d5389ae0 con 0x55c9d502a9f0 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8cfe00640 10 client.26101 handle_command_reply: tid=0 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 client.26101 shutdown 2024-07-26T11:06:20.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 msgr2=0x55c9d541c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d541c930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc004650 tx=0x7fc8dc0aad00 comp rx=0 tx=0).stop 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 shutdown_connections 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d541ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d541c930 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d502a9f0 0x55c9d542e850 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/1432773399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 >> 172.21.15.144:0/1432773399 conn(0x55c9d5060460 msgr2=0x55c9d5413f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8cfe00640 0 client.26101 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 shutdown_connections 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/1432773399 wait complete. 2024-07-26T11:06:20.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.055+0000 7fc8fc2a8740 20 client.26101 trim_cache size 0 max 16384 2024-07-26T11:06:20.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.074+0000 7fc8fc2a8740 1 Processor -- start 2024-07-26T11:06:20.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.074+0000 7fc8fc2a8740 1 -- start start 2024-07-26T11:06:20.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.074+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.074+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d540d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d5412530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01b97a0 con 0x55c9d506c070 2024-07-26T11:06:20.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01b7010 con 0x55c9d50431b0 2024-07-26T11:06:20.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01b4e60 con 0x55c9d504b390 2024-07-26T11:06:20.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d540d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d540d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57138/0 (socket says 172.21.15.144:57138) 2024-07-26T11:06:20.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 -- 172.21.15.144:0/2733010651 learned_addr learned my addr 172.21.15.144:0/2733010651 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:20.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8eea00640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ef400640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d5412530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 -- 172.21.15.144:0/2733010651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 msgr2=0x55c9d541e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 -- 172.21.15.144:0/2733010651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 msgr2=0x55c9d5412530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d5412530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ee000640 1 -- 172.21.15.144:0/2733010651 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9d5413c50 con 0x55c9d504b390 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8eea00640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.075+0000 7fc8ef400640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d5412530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.076+0000 7fc8ee000640 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d540d3a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00c6a0 tx=0x7fc8dc0e4a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=382493fa724cb2cc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.076+0000 7fc8daa00640 1 -- 172.21.15.144:0/2733010651 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00ce60 con 0x55c9d504b390 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.076+0000 7fc8daa00640 1 -- 172.21.15.144:0/2733010651 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8dc005310 con 0x55c9d504b390 2024-07-26T11:06:20.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.076+0000 7fc8daa00640 1 -- 172.21.15.144:0/2733010651 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0aa880 con 0x55c9d504b390 2024-07-26T11:06:20.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.077+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2733010651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 msgr2=0x55c9d540d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.077+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d540d3a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00c6a0 tx=0x7fc8dc0e4a10 comp rx=0 tx=0).stop 2024-07-26T11:06:20.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2733010651 shutdown_connections 2024-07-26T11:06:20.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d506c070 0x55c9d5412530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d504b390 0x55c9d540d3a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/2733010651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d50431b0 0x55c9d541e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2733010651 >> 172.21.15.144:0/2733010651 conn(0x55c9d5060460 msgr2=0x55c9d5411e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2733010651 shutdown_connections 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/2733010651 wait complete. 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.078+0000 7fc8fc2a8740 1 -- start start 2024-07-26T11:06:20.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 Processor -- start 2024-07-26T11:06:20.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 0x55c9d5425170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d504b390 0x55c9d54226c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5422dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01b54d0 con 0x55c9d50431b0 2024-07-26T11:06:20.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01b6940 con 0x55c9d504b390 2024-07-26T11:06:20.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f01b5ba0 con 0x55c9d506c070 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8fc2a8740 10 client.0 _ll_register_callbacks cb 0x7fc8dc004920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.079+0000 7fc8eea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 0x55c9d5425170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8ee000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d504b390 0x55c9d54226c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 0x55c9d5425170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:36286/0 (socket says 172.21.15.144:36286) 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 -- 172.21.15.144:0/428753640 learned_addr learned my addr 172.21.15.144:0/428753640 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8ef400640 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5422dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 -- 172.21.15.144:0/428753640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 msgr2=0x55c9d5422dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5422dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 -- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d504b390 msgr2=0x55c9d54226c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d504b390 0x55c9d54226c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.083 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 -- 172.21.15.144:0/428753640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9d5495fd0 con 0x55c9d50431b0 2024-07-26T11:06:20.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8eea00640 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 0x55c9d5425170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0005940 tx=0x7fc8e002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1337d4dfa0123c78 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:20.084 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.080+0000 7fc8cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8cfe00640 1 -- 172.21.15.144:0/428753640 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0035040 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8cfe00640 1 -- 172.21.15.144:0/428753640 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9566+0+0 (secure 0 0 0) 0x7fc8e00035f0 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c9d5385580 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8cfe00640 1 -- 172.21.15.144:0/428753640 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc8e00037f0 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8fc2a8740 10 client.15975 fetch_fsmap learned FSMap version 41 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c9d5497200 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.081+0000 7fc8cfe00640 1 -- 172.21.15.144:0/428753640 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c9d5385580 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8cfe00640 1 -- 172.21.15.144:0/428753640 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 41) ==== 8362+0+0 (secure 0 0 0) 0x7fc8e00538a0 con 0x55c9d50431b0 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8fc2a8740 10 client.15975 fetch_fsmap finished waiting for FSMap version 41 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8fc2a8740 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:20.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8fc2a8740 20 client.15975 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d4fffda0 0x55c9d54999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8fc2a8740 4 client.15975 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c9d5382cc0 con 0x55c9d4fffda0 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.082+0000 7fc8ee000640 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d4fffda0 0x55c9d54999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8ee000640 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d4fffda0 0x55c9d54999c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8c7c981c5e0e25f server_cookie=cfcdf3d375511b48 in_seq=0 out_seq=0 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8cfe00640 10 client.15975 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8cfe00640 1 -- 172.21.15.144:0/428753640 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55c9d5382cc0 con 0x55c9d4fffda0 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8cfe00640 10 client.15975 handle_command_reply: tid=0 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 client.15975 shutdown 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 msgr2=0x55c9d5425170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 0x55c9d5425170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0005940 tx=0x7fc8e002a410 comp rx=0 tx=0).stop 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 shutdown_connections 2024-07-26T11:06:20.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c9d506c070 0x55c9d5422dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c9d4fffda0 0x55c9d54999c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c9d504b390 0x55c9d54226c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 --2- 172.21.15.144:0/428753640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c9d50431b0 0x55c9d5425170 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8cfe00640 0 client.15975 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.083+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 >> 172.21.15.144:0/428753640 conn(0x55c9d5060460 msgr2=0x55c9d54125b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 shutdown_connections 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8fc2a8740 1 -- 172.21.15.144:0/428753640 wait complete. 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8fc2a8740 20 client.15975 trim_cache size 0 max 16384 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:06:20.088 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "scrub active (12 inodes in the stack)", 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:06:20.089 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc8d40b3d40 msgr2=0x7fc8d40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8f5600640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc8d40b3d40 0x7fc8d40b6250 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55c9d540fa70 tx=0x7fc8e0009560 comp rx=0 tx=0).stop 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 msgr2=0x7fc8f01a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.084+0000 7fc8f5600640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f01a0ab0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00e9c0 tx=0x7fc8dc00ee80 comp rx=0 tx=0).stop 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 shutdown_connections 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc8d40b3d40 0x7fc8d40b6250 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8f01141a0 0x7fc8f01a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8f0077cc0 0x7fc8f01a0ab0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 --2- 172.21.15.144:0/1026408991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc8f0077370 0x7fc8f01a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 >> 172.21.15.144:0/1026408991 conn(0x7fc8f01011f0 msgr2=0x7fc8f0103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 shutdown_connections 2024-07-26T11:06:20.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:20.085+0000 7fc8f5600640 1 -- 172.21.15.144:0/1026408991 wait complete. 2024-07-26T11:06:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:20 smithi183 ceph-mon[37273]: pgmap v578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 13 KiB/s wr, 11 op/s 2024-07-26T11:06:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:20 smithi144 ceph-mon[32129]: pgmap v578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 13 KiB/s wr, 11 op/s 2024-07-26T11:06:21.041 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:06:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:20 smithi160 ceph-mon[36715]: pgmap v578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 13 KiB/s wr, 11 op/s 2024-07-26T11:06:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:21 smithi183 ceph-mon[37273]: scrub summary: idle 2024-07-26T11:06:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:21 smithi144 ceph-mon[32129]: scrub summary: idle 2024-07-26T11:06:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:21 smithi160 ceph-mon[36715]: scrub summary: idle 2024-07-26T11:06:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:22 smithi183 ceph-mon[37273]: pgmap v579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 12 KiB/s wr, 10 op/s 2024-07-26T11:06:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:22 smithi144 ceph-mon[32129]: pgmap v579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 12 KiB/s wr, 10 op/s 2024-07-26T11:06:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:22 smithi160 ceph-mon[36715]: pgmap v579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 12 KiB/s wr, 10 op/s 2024-07-26T11:06:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:24 smithi183 ceph-mon[37273]: pgmap v580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 KiB/s wr, 9 op/s 2024-07-26T11:06:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:24 smithi144 ceph-mon[32129]: pgmap v580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 KiB/s wr, 9 op/s 2024-07-26T11:06:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:24 smithi160 ceph-mon[36715]: pgmap v580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 KiB/s wr, 9 op/s 2024-07-26T11:06:25.899 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:25 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:06:25.899 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:25 smithi183 systemd[1]: Stopped Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:25.900 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:25 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 2.800s CPU time. 2024-07-26T11:06:25.900 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:25 smithi183 systemd[1]: Starting Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:06:25.900 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:25 smithi183 podman[71721]: 2024-07-26 11:06:25.770715119 +0000 UTC m=+0.018182879 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:06:25.900 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:25 smithi183 podman[71721]: 2024-07-26 11:06:25.898390733 +0000 UTC m=+0.145858486 container create a8a067ab399467b5b14345b46625358159fcef7364b5ab18b732ebbd084f2226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:06:26.172 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:26 smithi183 podman[71721]: 2024-07-26 11:06:26.135108195 +0000 UTC m=+0.382575959 container init a8a067ab399467b5b14345b46625358159fcef7364b5ab18b732ebbd084f2226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T11:06:26.172 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:26 smithi183 podman[71721]: 2024-07-26 11:06:26.138040638 +0000 UTC m=+0.385508395 container start a8a067ab399467b5b14345b46625358159fcef7364b5ab18b732ebbd084f2226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:06:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:26.551+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:26.552+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:26.552+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:26.552+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:26.552+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:26.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mds[71740]: -- 172.21.15.183:0/1576603147 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1fcad9860 con 0x55c200c7ba80 2024-07-26T11:06:26.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: starting mds.i at 2024-07-26T11:06:26.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:26 smithi183 bash[71721]: a8a067ab399467b5b14345b46625358159fcef7364b5ab18b732ebbd084f2226 2024-07-26T11:06:26.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:26 smithi183 systemd[1]: Started Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: pgmap v581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 KiB/s wr, 12 op/s 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: daemon mds.i restarted 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:boot 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:06:26.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:26 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: pgmap v581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 KiB/s wr, 12 op/s 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: daemon mds.i restarted 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:boot 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:06:27.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:26 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: pgmap v581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 KiB/s wr, 12 op/s 2024-07-26T11:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: daemon mds.i restarted 2024-07-26T11:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T11:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:boot 2024-07-26T11:06:27.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:06:27.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:06:27.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:26 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:28.466 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:28 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:resolve 2024-07-26T11:06:28.467 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:28 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:28 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:resolve 2024-07-26T11:06:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:28 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:28 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:resolve 2024-07-26T11:06:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:28 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:29 smithi144 ceph-mon[32129]: pgmap v583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 11 KiB/s wr, 9 op/s 2024-07-26T11:06:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:29 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:reconnect 2024-07-26T11:06:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:29 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:29.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:29 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:06:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:29 smithi160 ceph-mon[36715]: pgmap v583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 11 KiB/s wr, 9 op/s 2024-07-26T11:06:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:29 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:reconnect 2024-07-26T11:06:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:29 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:29 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:06:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:29 smithi183 ceph-mon[37273]: pgmap v583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 11 KiB/s wr, 9 op/s 2024-07-26T11:06:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:29 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:reconnect 2024-07-26T11:06:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:29 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:29 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:06:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:30 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:rejoin 2024-07-26T11:06:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:30 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:30 smithi160 ceph-mon[36715]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:06:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:30 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:rejoin 2024-07-26T11:06:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:30 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:30 smithi183 ceph-mon[37273]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:06:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:rejoin 2024-07-26T11:06:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-mon[32129]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: *** Caught signal (Aborted) ** 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: in thread 7f660e000640 thread_name:ms_dispatch 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f661423e6f0] 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f661428b94c] 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 3: raise() 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 4: abort() 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 5: /lib64/libc.so.6(+0x2871b) [0x7f661422871b] 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 6: /lib64/libc.so.6(+0x37386) [0x7f6614237386] 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5641263756b9] 2024-07-26T11:06:30.722 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56412653380c] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5641264732ca] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564126443753] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564126444a1b] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564126428752] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564126376a56] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56412637a637] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56412637af5e] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56412635e03a] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6614d9b407] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 18: (DispatchQueue::entry()+0x60f) [0x7f6614d9bcf9] 2024-07-26T11:06:30.723 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6614e785a1] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 20: (Thread::entry_wrapper()+0x43) [0x7f6614c25cdd] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6614c25cf9] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6614289c02] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f661430ec40] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.289+0000 7f660e000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: in thread 7f660e000640 thread_name:ms_dispatch 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f661423e6f0] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f661428b94c] 2024-07-26T11:06:30.724 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 3: raise() 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 4: abort() 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 5: /lib64/libc.so.6(+0x2871b) [0x7f661422871b] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 6: /lib64/libc.so.6(+0x37386) [0x7f6614237386] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5641263756b9] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56412653380c] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5641264732ca] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564126443753] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564126444a1b] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564126428752] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564126376a56] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56412637a637] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56412637af5e] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56412635e03a] 2024-07-26T11:06:30.725 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6614d9b407] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 18: (DispatchQueue::entry()+0x60f) [0x7f6614d9bcf9] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6614e785a1] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 20: (Thread::entry_wrapper()+0x43) [0x7f6614c25cdd] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6614c25cf9] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6614289c02] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f661430ec40] 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.726 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 0> 2024-07-26T11:06:30.289+0000 7f660e000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: in thread 7f660e000640 thread_name:ms_dispatch 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f661423e6f0] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f661428b94c] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 3: raise() 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 4: abort() 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 5: /lib64/libc.so.6(+0x2871b) [0x7f661422871b] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 6: /lib64/libc.so.6(+0x37386) [0x7f6614237386] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5641263756b9] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56412653380c] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5641264732ca] 2024-07-26T11:06:30.727 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564126443753] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564126444a1b] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564126428752] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564126376a56] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56412637a637] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56412637af5e] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56412635e03a] 2024-07-26T11:06:30.728 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6614d9b407] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 18: (DispatchQueue::entry()+0x60f) [0x7f6614d9bcf9] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6614e785a1] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 20: (Thread::entry_wrapper()+0x43) [0x7f6614c25cdd] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6614c25cf9] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6614289c02] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f661430ec40] 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 0> 2024-07-26T11:06:30.289+0000 7f660e000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: in thread 7f660e000640 thread_name:ms_dispatch 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:30.729 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f661423e6f0] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f661428b94c] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 3: raise() 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 4: abort() 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 5: /lib64/libc.so.6(+0x2871b) [0x7f661422871b] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 6: /lib64/libc.so.6(+0x37386) [0x7f6614237386] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5641263756b9] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56412653380c] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5641264732ca] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564126443753] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564126444a1b] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564126428752] 2024-07-26T11:06:30.730 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564126376a56] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56412637a637] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56412637af5e] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56412635e03a] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6614d9b407] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 18: (DispatchQueue::entry()+0x60f) [0x7f6614d9bcf9] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6614e785a1] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 20: (Thread::entry_wrapper()+0x43) [0x7f6614c25cdd] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6614c25cf9] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6614289c02] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f661430ec40] 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[102518]: 2024-07-26T11:06:30.731 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:30 smithi144 podman[102981]: 2024-07-26 11:06:30.426506399 +0000 UTC m=+0.033002500 container died e5194ea418b74534bd198533b29b9faa03f5ed768b9236fec60f2627de5670d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-26T11:06:31.470 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:31 smithi144 podman[102981]: 2024-07-26 11:06:31.180279512 +0000 UTC m=+0.786775634 container remove e5194ea418b74534bd198533b29b9faa03f5ed768b9236fec60f2627de5670d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T11:06:31.470 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:31 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:06:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:31 smithi144 ceph-mon[32129]: pgmap v584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-07-26T11:06:31.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:31 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:31.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:31 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:06:31.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:active 2024-07-26T11:06:31.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:31 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:31.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:31 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:31 smithi160 ceph-mon[36715]: pgmap v584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-07-26T11:06:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:31 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:31 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:06:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:active 2024-07-26T11:06:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:31 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:31 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:31 smithi183 ceph-mon[37273]: pgmap v584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-07-26T11:06:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:31 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:31 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:06:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1399766908,v1:172.21.15.144:6841/1399766908] up:active 2024-07-26T11:06:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:31 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:31 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:32.220 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:31 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:06:32.220 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:31 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.387s CPU time. 2024-07-26T11:06:32.486 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:32 smithi144 ceph-mon[32129]: pgmap v585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T11:06:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:32 smithi160 ceph-mon[36715]: pgmap v585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T11:06:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:32 smithi183 ceph-mon[37273]: pgmap v585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-07-26T11:06:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:34 smithi160 ceph-mon[36715]: pgmap v586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-07-26T11:06:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:34 smithi183 ceph-mon[37273]: pgmap v586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-07-26T11:06:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:34 smithi144 ceph-mon[32129]: pgmap v586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-07-26T11:06:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: pgmap v587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: pgmap v587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: pgmap v587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:06:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:37.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:37.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:06:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:06:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:06:37.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:06:37.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:37.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:37.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:06:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:38 smithi160 ceph-mon[36715]: pgmap v588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-07-26T11:06:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:38 smithi183 ceph-mon[37273]: pgmap v588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-07-26T11:06:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:38 smithi144 ceph-mon[32129]: pgmap v588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-07-26T11:06:40.905 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:40 smithi144 ceph-mon[32129]: pgmap v589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-07-26T11:06:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:40 smithi160 ceph-mon[36715]: pgmap v589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-07-26T11:06:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:40 smithi183 ceph-mon[37273]: pgmap v589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-07-26T11:06:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:41.567+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:41.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:41.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:41.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:41.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:42.219 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:41 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:06:42.220 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:41 smithi144 systemd[1]: Stopped Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:42.220 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:41 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.387s CPU time. 2024-07-26T11:06:42.220 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:41 smithi144 systemd[1]: Starting Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:06:42.678 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 podman[103212]: 2024-07-26 11:06:42.282320926 +0000 UTC m=+0.033536327 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:06:42.678 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 podman[103212]: 2024-07-26 11:06:42.405575767 +0000 UTC m=+0.156791212 container create ddd473e7ebdb817c101faa7ab65f3e8331518e3751f92f7a71c3a9b17672108f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T11:06:42.679 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 podman[103212]: 2024-07-26 11:06:42.676353579 +0000 UTC m=+0.427569014 container init ddd473e7ebdb817c101faa7ab65f3e8331518e3751f92f7a71c3a9b17672108f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-07-26T11:06:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:42 smithi144 ceph-mon[32129]: pgmap v590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-07-26T11:06:42.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 podman[103212]: 2024-07-26 11:06:42.685891512 +0000 UTC m=+0.437106902 container start ddd473e7ebdb817c101faa7ab65f3e8331518e3751f92f7a71c3a9b17672108f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T11:06:42.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 bash[103212]: ddd473e7ebdb817c101faa7ab65f3e8331518e3751f92f7a71c3a9b17672108f 2024-07-26T11:06:42.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 systemd[1]: Started Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:42.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 ceph-mds[103231]: -- 172.21.15.144:0/615540977 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573a0e57860 con 0x5573a4ff9a80 2024-07-26T11:06:42.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:06:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: starting mds.a at 2024-07-26T11:06:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:42 smithi160 ceph-mon[36715]: pgmap v590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-07-26T11:06:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:42 smithi183 ceph-mon[37273]: pgmap v590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-07-26T11:06:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: daemon mds.a restarted 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:boot 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:06:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:43 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: daemon mds.a restarted 2024-07-26T11:06:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T11:06:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:boot 2024-07-26T11:06:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:06:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:06:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:06:44.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:43 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: daemon mds.a restarted 2024-07-26T11:06:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:06:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T11:06:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:boot 2024-07-26T11:06:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:06:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:06:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:06:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:43 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:44 smithi183 ceph-mon[37273]: pgmap v592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 KiB/s wr, 7 op/s 2024-07-26T11:06:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:44 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:resolve 2024-07-26T11:06:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:44 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:44 smithi144 ceph-mon[32129]: pgmap v592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 KiB/s wr, 7 op/s 2024-07-26T11:06:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:44 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:resolve 2024-07-26T11:06:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:44 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:44 smithi160 ceph-mon[36715]: pgmap v592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 KiB/s wr, 7 op/s 2024-07-26T11:06:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:44 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:resolve 2024-07-26T11:06:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:44 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:46 smithi160 ceph-mon[36715]: pgmap v593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:06:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:46 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:reconnect 2024-07-26T11:06:47.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:46 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:47.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:46 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:06:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:46 smithi183 ceph-mon[37273]: pgmap v593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:06:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:46 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:reconnect 2024-07-26T11:06:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:46 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:46 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:06:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:46 smithi144 ceph-mon[32129]: pgmap v593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:06:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:46 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:reconnect 2024-07-26T11:06:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:46 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:46 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:06:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:47 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:rejoin 2024-07-26T11:06:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:47 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:48.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:47 smithi160 ceph-mon[36715]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:06:48.099 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:47 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:rejoin 2024-07-26T11:06:48.100 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:47 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:48.100 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:47 smithi144 ceph-mon[32129]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:06:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:rejoin 2024-07-26T11:06:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-mon[37273]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:06:48.135 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: *** Caught signal (Aborted) ** 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: in thread 7f1d2310e640 thread_name:ms_dispatch 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d284e56f0] 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d2853294c] 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 3: raise() 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 4: abort() 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d284cf71b] 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d284de386] 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c1fa6066b9] 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c1fa7c480c] 2024-07-26T11:06:48.136 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c1fa7042ca] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c1fa6d4753] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c1fa6d5a1b] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c1fa6b9752] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c1fa607a56] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c1fa60b637] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c1fa60bf5e] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c1fa5ef03a] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d28db2407] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d28db2cf9] 2024-07-26T11:06:48.137 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d28e8f5a1] 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d28c3ccdd] 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d28c3ccf9] 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d28530c02] 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d285b5c40] 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:47.809+0000 7f1d2310e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: in thread 7f1d2310e640 thread_name:ms_dispatch 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:48.138 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d284e56f0] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d2853294c] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 3: raise() 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 4: abort() 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d284cf71b] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d284de386] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c1fa6066b9] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c1fa7c480c] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c1fa7042ca] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c1fa6d4753] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c1fa6d5a1b] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c1fa6b9752] 2024-07-26T11:06:48.139 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c1fa607a56] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c1fa60b637] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c1fa60bf5e] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c1fa5ef03a] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d28db2407] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d28db2cf9] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d28e8f5a1] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d28c3ccdd] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d28c3ccf9] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d28530c02] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d285b5c40] 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 0> 2024-07-26T11:06:47.809+0000 7f1d2310e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:48.140 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: in thread 7f1d2310e640 thread_name:ms_dispatch 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d284e56f0] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d2853294c] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 3: raise() 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 4: abort() 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d284cf71b] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d284de386] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c1fa6066b9] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c1fa7c480c] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c1fa7042ca] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c1fa6d4753] 2024-07-26T11:06:48.141 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c1fa6d5a1b] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c1fa6b9752] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c1fa607a56] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c1fa60b637] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c1fa60bf5e] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c1fa5ef03a] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d28db2407] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d28db2cf9] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d28e8f5a1] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d28c3ccdd] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d28c3ccf9] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d28530c02] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d285b5c40] 2024-07-26T11:06:48.142 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 0> 2024-07-26T11:06:47.809+0000 7f1d2310e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: in thread 7f1d2310e640 thread_name:ms_dispatch 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d284e56f0] 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d2853294c] 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 3: raise() 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 4: abort() 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d284cf71b] 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d284de386] 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c1fa6066b9] 2024-07-26T11:06:48.143 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c1fa7c480c] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c1fa7042ca] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c1fa6d4753] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c1fa6d5a1b] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c1fa6b9752] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c1fa607a56] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c1fa60b637] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c1fa60bf5e] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c1fa5ef03a] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d28db2407] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d28db2cf9] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d28e8f5a1] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d28c3ccdd] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d28c3ccf9] 2024-07-26T11:06:48.144 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d28530c02] 2024-07-26T11:06:48.145 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d285b5c40] 2024-07-26T11:06:48.145 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:06:48.145 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[71735]: 2024-07-26T11:06:48.145 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:47 smithi183 podman[71917]: 2024-07-26 11:06:47.879065196 +0000 UTC m=+0.023636340 container died a8a067ab399467b5b14345b46625358159fcef7364b5ab18b732ebbd084f2226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True) 2024-07-26T11:06:48.766 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:48 smithi183 podman[71917]: 2024-07-26 11:06:48.430864423 +0000 UTC m=+0.575435553 container remove a8a067ab399467b5b14345b46625358159fcef7364b5ab18b732ebbd084f2226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724) 2024-07-26T11:06:48.766 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:48 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:06:48.766 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:48 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:06:48.766 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:48 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 1.935s CPU time. 2024-07-26T11:06:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:48 smithi160 ceph-mon[36715]: pgmap v594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:06:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:48 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:49.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:48 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:06:49.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:active 2024-07-26T11:06:49.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:48 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:49.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:48 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:48 smithi183 ceph-mon[37273]: pgmap v594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:48 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:48 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:active 2024-07-26T11:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:48 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:48 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:48 smithi144 ceph-mon[32129]: pgmap v594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:06:49.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:48 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:06:49.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:48 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:06:49.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2805870176,v1:172.21.15.183:6839/2805870176] up:active 2024-07-26T11:06:49.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:48 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:06:49.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:48 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:06:51.042 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:06:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:50 smithi160 ceph-mon[36715]: pgmap v595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 KiB/s wr, 28 op/s 2024-07-26T11:06:51.125 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:50 smithi144 ceph-mon[32129]: pgmap v595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 KiB/s wr, 28 op/s 2024-07-26T11:06:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:50 smithi183 ceph-mon[37273]: pgmap v595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 KiB/s wr, 28 op/s 2024-07-26T11:06:51.517 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:52.935 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:52 smithi144 ceph-mon[32129]: pgmap v596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 KiB/s wr, 51 op/s 2024-07-26T11:06:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:52 smithi160 ceph-mon[36715]: pgmap v596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 KiB/s wr, 51 op/s 2024-07-26T11:06:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:52 smithi183 ceph-mon[37273]: pgmap v596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 KiB/s wr, 51 op/s 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 -- 172.21.15.144:0/1413820058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c109650 msgr2=0x7f197c10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 --2- 172.21.15.144:0/1413820058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c109650 0x7f197c10fe80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1974009ca0 tx=0x7f197402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 -- 172.21.15.144:0/1413820058 shutdown_connections 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 --2- 172.21.15.144:0/1413820058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c109650 0x7f197c10fe80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 --2- 172.21.15.144:0/1413820058 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f197c108d00 0x7f197c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 --2- 172.21.15.144:0/1413820058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197c1033c0 0x7f197c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 -- 172.21.15.144:0/1413820058 >> 172.21.15.144:0/1413820058 conn(0x7f197c0fee10 msgr2=0x7f197c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 -- 172.21.15.144:0/1413820058 shutdown_connections 2024-07-26T11:06:54.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.007+0000 7f197b200640 1 -- 172.21.15.144:0/1413820058 wait complete. 2024-07-26T11:06:54.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.008+0000 7f197b200640 1 Processor -- start 2024-07-26T11:06:54.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 -- start start 2024-07-26T11:06:54.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 0x7f197c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197c108d00 0x7f197c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f197c109650 0x7f197c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f197c113320 con 0x7f197c1033c0 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1131a0 con 0x7f197c109650 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f197b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1134a0 con 0x7f197c108d00 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f1979e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 0x7f197c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f1979e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 0x7f197c19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57636/0 (socket says 172.21.15.144:57636) 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.009+0000 7f1979e00640 1 -- 172.21.15.144:0/4165430916 learned_addr learned my addr 172.21.15.144:0/4165430916 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f197a800640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f197c109650 0x7f197c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979400640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197c108d00 0x7f197c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979e00640 1 -- 172.21.15.144:0/4165430916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197c108d00 msgr2=0x7f197c19c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979e00640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197c108d00 0x7f197c19c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979e00640 1 -- 172.21.15.144:0/4165430916 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f197c109650 msgr2=0x7f197c1a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979e00640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f197c109650 0x7f197c1a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979e00640 1 -- 172.21.15.144:0/4165430916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c1a1e60 con 0x7f197c1033c0 2024-07-26T11:06:54.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.010+0000 7f1979e00640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 0x7f197c19bef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f196c00e9f0 tx=0x7f196c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:54.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.011+0000 7f1972a00640 1 -- 172.21.15.144:0/4165430916 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c00cd80 con 0x7f197c1033c0 2024-07-26T11:06:54.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.011+0000 7f1972a00640 1 -- 172.21.15.144:0/4165430916 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c0054d0 con 0x7f197c1033c0 2024-07-26T11:06:54.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.011+0000 7f1972a00640 1 -- 172.21.15.144:0/4165430916 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c010690 con 0x7f197c1033c0 2024-07-26T11:06:54.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.011+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197c1a2150 con 0x7f197c1033c0 2024-07-26T11:06:54.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.011+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197c1a88b0 con 0x7f197c1033c0 2024-07-26T11:06:54.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.014+0000 7f1987e26740 1 Processor -- start 2024-07-26T11:06:54.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1972a00640 1 -- 172.21.15.144:0/4165430916 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f196c004e90 con 0x7f197c1033c0 2024-07-26T11:06:54.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 -- start start 2024-07-26T11:06:54.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c31041b1b0 0x55c3107e9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107e9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55c31075fb90 con 0x55c310444070 2024-07-26T11:06:54.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c31074f090 con 0x55c31041b1b0 2024-07-26T11:06:54.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.015+0000 7f1987e26740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c3107530d0 con 0x55c310423390 2024-07-26T11:06:54.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f197a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f197a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107e8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57640/0 (socket says 172.21.15.144:57640) 2024-07-26T11:06:54.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f197a800640 1 -- 172.21.15.144:0/931353999 learned_addr learned my addr 172.21.15.144:0/931353999 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:54.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979e00640 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c31041b1b0 0x55c3107e9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979400640 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107e9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979e00640 1 -- 172.21.15.144:0/931353999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 msgr2=0x55c3107e9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979e00640 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107e9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979e00640 1 -- 172.21.15.144:0/931353999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 msgr2=0x55c3107e8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979e00640 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107e8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.016+0000 7f1979e00640 1 -- 172.21.15.144:0/931353999 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3104af650 con 0x55c31041b1b0 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f1979e00640 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c31041b1b0 0x55c3107e9510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f196c0e1ae0 tx=0x7f196c0eab20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5718c8c4dc1f18b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f1972a00640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19600b3dc0 0x7f19600b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f1967400640 1 -- 172.21.15.144:0/931353999 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0f3070 con 0x55c31041b1b0 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f1967400640 1 -- 172.21.15.144:0/931353999 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c0aaab0 con 0x55c31041b1b0 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f1967400640 1 -- 172.21.15.144:0/931353999 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0c09f0 con 0x55c31041b1b0 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f1972a00640 1 -- 172.21.15.144:0/4165430916 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9738+0+0 (secure 0 0 0) 0x7f196c0e48b0 con 0x7f197c1033c0 2024-07-26T11:06:54.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.017+0000 7f197a800640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19600b3dc0 0x7f19600b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 -- 172.21.15.144:0/931353999 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c31041b1b0 msgr2=0x55c3107e9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c31041b1b0 0x55c3107e9510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f196c0e1ae0 tx=0x7f196c0eab20 comp rx=0 tx=0).stop 2024-07-26T11:06:54.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f197a800640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19600b3dc0 0x7f19600b62d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f197402f500 tx=0x7f197402f310 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:54.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 -- 172.21.15.144:0/931353999 shutdown_connections 2024-07-26T11:06:54.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c31041b1b0 0x55c3107e9510 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 --2- 172.21.15.144:0/931353999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107e8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 -- 172.21.15.144:0/931353999 >> 172.21.15.144:0/931353999 conn(0x55c310438460 msgr2=0x55c3107e6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.018+0000 7f1987e26740 1 -- 172.21.15.144:0/931353999 shutdown_connections 2024-07-26T11:06:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.019+0000 7f1987e26740 1 -- 172.21.15.144:0/931353999 wait complete. 2024-07-26T11:06:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.019+0000 7f1987e26740 1 -- start start 2024-07-26T11:06:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.019+0000 7f1987e26740 1 Processor -- start 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 0x55c3107f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310423390 0x55c3107f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107f5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55c3105b2700 con 0x55c310444070 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c3106eb660 con 0x55c310423390 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c3107f77b0 con 0x55c31041b1b0 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1987e26740 10 client.0 _ll_register_callbacks cb 0x55c3106fb110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1979e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 0x55c3107f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1979e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 0x55c3107f45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:50268/0 (socket says 172.21.15.144:50268) 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1979e00640 1 -- 172.21.15.144:0/748811805 learned_addr learned my addr 172.21.15.144:0/748811805 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:54.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f197a800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.020+0000 7f1979400640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310423390 0x55c3107f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979e00640 1 -- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310423390 msgr2=0x55c3107f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979e00640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310423390 0x55c3107f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979e00640 1 -- 172.21.15.144:0/748811805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 msgr2=0x55c3107f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979e00640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979e00640 1 -- 172.21.15.144:0/748811805 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c310702800 con 0x55c31041b1b0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979400640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310423390 0x55c3107f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1979e00640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 0x55c3107f45b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f196c0aaca0 tx=0x7f196c0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=568efd8eb116d7b9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f195be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f195be00640 1 -- 172.21.15.144:0/748811805 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0c3f60 con 0x55c31041b1b0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f196c0aaab0 con 0x55c31041b1b0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.021+0000 7f195be00640 1 -- 172.21.15.144:0/748811805 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9738+0+0 (secure 0 0 0) 0x7f196c0ab200 con 0x55c31041b1b0 2024-07-26T11:06:54.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f195be00640 1 -- 172.21.15.144:0/748811805 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f196c0c2e60 con 0x55c31041b1b0 2024-07-26T11:06:54.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 10 client.26131 fetch_fsmap learned FSMap version 55 2024-07-26T11:06:54.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55c310803330 con 0x55c31041b1b0 2024-07-26T11:06:54.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f195be00640 1 -- 172.21.15.144:0/748811805 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c0aaab0 con 0x55c31041b1b0 2024-07-26T11:06:54.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f195be00640 1 -- 172.21.15.144:0/748811805 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f196c0b3bc0 con 0x55c31041b1b0 2024-07-26T11:06:54.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 10 client.26131 fetch_fsmap finished waiting for FSMap version 55 2024-07-26T11:06:54.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 10 client.26131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:54.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 20 client.26131 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c3104029f0 0x55c310806690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 4 client.26131 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.022+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c310761ae0 con 0x55c3104029f0 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.023+0000 7f1979400640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c3104029f0 0x55c310806690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.023+0000 7f1979400640 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c3104029f0 0x55c310806690 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c001af16dd187571 server_cookie=c3e05708eb65db5d in_seq=0 out_seq=0 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.023+0000 7f195be00640 10 client.26131 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.024+0000 7f195be00640 1 -- 172.21.15.144:0/748811805 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c310761ae0 con 0x55c3104029f0 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.024+0000 7f195be00640 10 client.26131 handle_command_reply: tid=0 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.024+0000 7f1987e26740 1 client.26131 shutdown 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.024+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 msgr2=0x55c3107f45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.024+0000 7f1987e26740 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 0x55c3107f45b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f196c0aaca0 tx=0x7f196c0048f0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 shutdown_connections 2024-07-26T11:06:54.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310444070 0x55c3107f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c3104029f0 0x55c310806690 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310423390 0x55c3107f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 --2- 172.21.15.144:0/748811805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c31041b1b0 0x55c3107f45b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 >> 172.21.15.144:0/748811805 conn(0x55c310438460 msgr2=0x55c3107ec1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f195be00640 0 client.26131 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 shutdown_connections 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 1 -- 172.21.15.144:0/748811805 wait complete. 2024-07-26T11:06:54.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.025+0000 7f1987e26740 20 client.26131 trim_cache size 0 max 16384 2024-07-26T11:06:54.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 Processor -- start 2024-07-26T11:06:54.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 -- start start 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 0x55c3107f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c310800140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b5410 con 0x55c31041b1b0 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b2c80 con 0x55c310444070 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1987e26740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b0ad0 con 0x55c310423390 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.044+0000 7f1979e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 0x55c3107f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 0x55c3107f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57664/0 (socket says 172.21.15.144:57664) 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 -- 172.21.15.144:0/1292774178 learned_addr learned my addr 172.21.15.144:0/1292774178 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f197a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c310800140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979400640 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 -- 172.21.15.144:0/1292774178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 msgr2=0x55c3107ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 -- 172.21.15.144:0/1292774178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 msgr2=0x55c310800140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.048 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c310800140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979e00640 1 -- 172.21.15.144:0/1292774178 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3107e7ac0 con 0x55c31041b1b0 2024-07-26T11:06:54.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.045+0000 7f1979400640 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:54.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.046+0000 7f1979e00640 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 0x55c3107f4ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f196c0a8210 tx=0x7f196c0eab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc46276b927cbff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:54.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.046+0000 7f1966a00640 1 -- 172.21.15.144:0/1292774178 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0f3020 con 0x55c31041b1b0 2024-07-26T11:06:54.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.046+0000 7f1966a00640 1 -- 172.21.15.144:0/1292774178 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c0aaab0 con 0x55c31041b1b0 2024-07-26T11:06:54.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.046+0000 7f1966a00640 1 -- 172.21.15.144:0/1292774178 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0c2530 con 0x55c31041b1b0 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 -- 172.21.15.144:0/1292774178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 msgr2=0x55c3107f4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 0x55c3107f4ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f196c0a8210 tx=0x7f196c0eab20 comp rx=0 tx=0).stop 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 -- 172.21.15.144:0/1292774178 shutdown_connections 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c310800140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c310423390 0x55c3107ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 --2- 172.21.15.144:0/1292774178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c31041b1b0 0x55c3107f4ff0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 -- 172.21.15.144:0/1292774178 >> 172.21.15.144:0/1292774178 conn(0x55c310438460 msgr2=0x55c3107ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 -- 172.21.15.144:0/1292774178 shutdown_connections 2024-07-26T11:06:54.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.047+0000 7f1987e26740 1 -- 172.21.15.144:0/1292774178 wait complete. 2024-07-26T11:06:54.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.048+0000 7f1987e26740 1 -- start start 2024-07-26T11:06:54.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.048+0000 7f1987e26740 1 Processor -- start 2024-07-26T11:06:54.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.048+0000 7f1987e26740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 0x55c310809f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1987e26740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c3108083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1987e26740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3103d7da0 0x55c310808a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1987e26740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b1140 con 0x55c310423390 2024-07-26T11:06:54.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1987e26740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b25b0 con 0x55c310444070 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1987e26740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b1810 con 0x55c3103d7da0 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1987e26740 10 client.0 _ll_register_callbacks cb 0x7f196c0ea8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 0x55c310809f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 0x55c310809f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57672/0 (socket says 172.21.15.144:57672) 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 -- 172.21.15.144:0/1009832822 learned_addr learned my addr 172.21.15.144:0/1009832822 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f197a800640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3103d7da0 0x55c310808a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979400640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c3108083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 -- 172.21.15.144:0/1009832822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3103d7da0 msgr2=0x55c310808a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3103d7da0 0x55c310808a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 -- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 msgr2=0x55c3108083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c3108083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f1979e00640 1 -- 172.21.15.144:0/1009832822 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c310809210 con 0x55c310423390 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.049+0000 7f197a800640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3103d7da0 0x55c310808a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f1979e00640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 0x55c310809f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f196c0c07e0 tx=0x7f196c0e1f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681919516f37b915 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f195be00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f195be00640 1 -- 172.21.15.144:0/1009832822 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0f3070 con 0x55c310423390 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f196c0aaab0 con 0x55c310423390 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f195be00640 1 -- 172.21.15.144:0/1009832822 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9738+0+0 (secure 0 0 0) 0x7f196c0ab200 con 0x55c310423390 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f195be00640 1 -- 172.21.15.144:0/1009832822 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f196c0c2700 con 0x55c310423390 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f1987e26740 10 client.15999 fetch_fsmap learned FSMap version 55 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55c31086dca0 con 0x55c310423390 2024-07-26T11:06:54.054 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.050+0000 7f195be00640 1 -- 172.21.15.144:0/1009832822 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c0aaab0 con 0x55c310423390 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f195be00640 1 -- 172.21.15.144:0/1009832822 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 55) ==== 8362+0+0 (secure 0 0 0) 0x7f196c0c5ae0 con 0x55c310423390 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1987e26740 10 client.15999 fetch_fsmap finished waiting for FSMap version 55 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1987e26740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1987e26740 20 client.15999 populate_metadata read hostname 'smithi144' 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1987e26740 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c31086e170 0x55c310870590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1987e26740 4 client.15999 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c3107e6360 con 0x55c31086e170 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.051+0000 7f1979400640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c31086e170 0x55c310870590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.052+0000 7f195be00640 10 client.15999 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.052+0000 7f1979400640 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c31086e170 0x55c310870590 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da77d686796e46f6 server_cookie=e24eb0f3af4f7b28 in_seq=0 out_seq=0 2024-07-26T11:06:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.052+0000 7f195be00640 1 -- 172.21.15.144:0/1009832822 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55c3107e6360 con 0x55c31086e170 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.052+0000 7f195be00640 10 client.15999 handle_command_reply: tid=0 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.052+0000 7f1987e26740 1 client.15999 shutdown 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 msgr2=0x55c310809f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 0x55c310809f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f196c0c07e0 tx=0x7f196c0e1f70 comp rx=0 tx=0).stop 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 shutdown_connections 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3103d7da0 0x55c310808a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c31086e170 0x55c310870590 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c310444070 0x55c3108083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 --2- 172.21.15.144:0/1009832822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c310423390 0x55c310809f00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f195be00640 0 client.15999 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 >> 172.21.15.144:0/1009832822 conn(0x55c310438460 msgr2=0x55c3107f4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 shutdown_connections 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 1 -- 172.21.15.144:0/1009832822 wait complete. 2024-07-26T11:06:54.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.053+0000 7f1987e26740 20 client.15999 trim_cache size 0 max 16384 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "scrub active (1 inodes in the stack)", 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:06:54.058 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:06:54.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19600b3dc0 msgr2=0x7f19600b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19600b3dc0 0x7f19600b62d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f197402f500 tx=0x7f197402f310 comp rx=0 tx=0).stop 2024-07-26T11:06:54.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 msgr2=0x7f197c19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 0x7f197c19bef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f196c00e9f0 tx=0x7f196c00edf0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 shutdown_connections 2024-07-26T11:06:54.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f19600b3dc0 0x7f19600b62d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f197c109650 0x7f197c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197c108d00 0x7f197c19c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 --2- 172.21.15.144:0/4165430916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f197c1033c0 0x7f197c19bef0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 >> 172.21.15.144:0/4165430916 conn(0x7f197c0fee10 msgr2=0x7f197c1062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 shutdown_connections 2024-07-26T11:06:54.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:06:54.054+0000 7f197b200640 1 -- 172.21.15.144:0/4165430916 wait complete. 2024-07-26T11:06:54.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:54 smithi144 ceph-mon[32129]: pgmap v597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 KiB/s wr, 69 op/s 2024-07-26T11:06:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:54 smithi160 ceph-mon[36715]: pgmap v597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 KiB/s wr, 69 op/s 2024-07-26T11:06:55.085 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:06:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:54 smithi183 ceph-mon[37273]: pgmap v597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 KiB/s wr, 69 op/s 2024-07-26T11:06:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:56.585+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:56.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:56.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:56.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:06:56.587+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:56 smithi160 ceph-mon[36715]: pgmap v598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 62 op/s 2024-07-26T11:06:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:56 smithi183 ceph-mon[37273]: pgmap v598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 62 op/s 2024-07-26T11:06:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:56 smithi144 ceph-mon[32129]: pgmap v598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 62 op/s 2024-07-26T11:06:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:58 smithi160 ceph-mon[36715]: pgmap v599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 60 op/s 2024-07-26T11:06:59.118 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:58 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:06:59.118 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:58 smithi183 systemd[1]: Stopped Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:06:59.119 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:58 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 1.935s CPU time. 2024-07-26T11:06:59.119 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:58 smithi183 systemd[1]: Starting Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:06:59.119 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:58 smithi183 ceph-mon[37273]: pgmap v599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 60 op/s 2024-07-26T11:06:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:58 smithi144 ceph-mon[32129]: pgmap v599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 60 op/s 2024-07-26T11:06:59.369 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 podman[72005]: 2024-07-26 11:06:59.020159275 +0000 UTC m=+0.017637297 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:06:59.369 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 podman[72005]: 2024-07-26 11:06:59.131318697 +0000 UTC m=+0.128796709 container create 2b882ce2ab3dfb032029fe63422e44f54fc994a45262d9e1ff5b71315e46ad3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:06:59.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 podman[72005]: 2024-07-26 11:06:59.367696167 +0000 UTC m=+0.365174180 container init 2b882ce2ab3dfb032029fe63422e44f54fc994a45262d9e1ff5b71315e46ad3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:06:59.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 podman[72005]: 2024-07-26 11:06:59.370878235 +0000 UTC m=+0.368356249 container start 2b882ce2ab3dfb032029fe63422e44f54fc994a45262d9e1ff5b71315e46ad3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-26T11:06:59.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mds[72024]: -- 172.21.15.183:0/3175980739 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638124bf860 con 0x563816661600 2024-07-26T11:06:59.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: starting mds.i at 2024-07-26T11:06:59.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 bash[72005]: 2b882ce2ab3dfb032029fe63422e44f54fc994a45262d9e1ff5b71315e46ad3a 2024-07-26T11:06:59.636 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:06:59 smithi183 systemd[1]: Started Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: daemon mds.i restarted 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:boot 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:06:59 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: daemon mds.i restarted 2024-07-26T11:07:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:07:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T11:07:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:boot 2024-07-26T11:07:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:07:00.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:00.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:06:59 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: daemon mds.i restarted 2024-07-26T11:07:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:07:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T11:07:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:boot 2024-07-26T11:07:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:07:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:06:59 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:00 smithi183 ceph-mon[37273]: pgmap v600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 61 op/s 2024-07-26T11:07:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:00 smithi144 ceph-mon[32129]: pgmap v600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 61 op/s 2024-07-26T11:07:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:00 smithi160 ceph-mon[36715]: pgmap v600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 KiB/s wr, 61 op/s 2024-07-26T11:07:02.094 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:01 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:resolve 2024-07-26T11:07:02.094 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:01 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:01 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:resolve 2024-07-26T11:07:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:01 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:01 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:resolve 2024-07-26T11:07:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:01 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:02 smithi183 ceph-mon[37273]: pgmap v602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 op/s 2024-07-26T11:07:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:02 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:reconnect 2024-07-26T11:07:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:02 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:02 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:02 smithi144 ceph-mon[32129]: pgmap v602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 op/s 2024-07-26T11:07:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:02 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:reconnect 2024-07-26T11:07:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:02 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:02 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:02 smithi160 ceph-mon[36715]: pgmap v602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 op/s 2024-07-26T11:07:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:02 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:reconnect 2024-07-26T11:07:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:02 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:03.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:02 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:rejoin 2024-07-26T11:07:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-mon[37273]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: *** Caught signal (Aborted) ** 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: in thread 7f58404f0640 thread_name:ms_dispatch 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f58458c76f0] 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f584591494c] 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 3: raise() 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 4: abort() 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58458b171b] 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 6: /lib64/libc.so.6(+0x37386) [0x7f58458c0386] 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562978e316b9] 2024-07-26T11:07:04.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562978fef80c] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562978f2f2ca] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562978eff753] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562978f00a1b] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562978ee4752] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562978e32a56] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562978e36637] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562978e36f5e] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562978e1a03a] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5846194407] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 18: (DispatchQueue::entry()+0x60f) [0x7f5846194cf9] 2024-07-26T11:07:04.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f58462715a1] 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 20: (Thread::entry_wrapper()+0x43) [0x7f584601ecdd] 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 21: (Thread::_entry_func(void*)+0xd) [0x7f584601ecf9] 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5845912c02] 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5845997c40] 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:03.901+0000 7f58404f0640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: in thread 7f58404f0640 thread_name:ms_dispatch 2024-07-26T11:07:04.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f58458c76f0] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f584591494c] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 3: raise() 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 4: abort() 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58458b171b] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 6: /lib64/libc.so.6(+0x37386) [0x7f58458c0386] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562978e316b9] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562978fef80c] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562978f2f2ca] 2024-07-26T11:07:04.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562978eff753] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562978f00a1b] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562978ee4752] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562978e32a56] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562978e36637] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562978e36f5e] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562978e1a03a] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5846194407] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 18: (DispatchQueue::entry()+0x60f) [0x7f5846194cf9] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f58462715a1] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 20: (Thread::entry_wrapper()+0x43) [0x7f584601ecdd] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 21: (Thread::_entry_func(void*)+0xd) [0x7f584601ecf9] 2024-07-26T11:07:04.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5845912c02] 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5845997c40] 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 0> 2024-07-26T11:07:03.901+0000 7f58404f0640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: in thread 7f58404f0640 thread_name:ms_dispatch 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f58458c76f0] 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f584591494c] 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 3: raise() 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 4: abort() 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58458b171b] 2024-07-26T11:07:04.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 6: /lib64/libc.so.6(+0x37386) [0x7f58458c0386] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562978e316b9] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562978fef80c] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562978f2f2ca] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562978eff753] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562978f00a1b] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562978ee4752] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562978e32a56] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562978e36637] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562978e36f5e] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562978e1a03a] 2024-07-26T11:07:04.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5846194407] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 18: (DispatchQueue::entry()+0x60f) [0x7f5846194cf9] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f58462715a1] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 20: (Thread::entry_wrapper()+0x43) [0x7f584601ecdd] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 21: (Thread::_entry_func(void*)+0xd) [0x7f584601ecf9] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5845912c02] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5845997c40] 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:04.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: -1> 2024-07-26T11:07:03.901+0000 7f58404f0640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: in thread 7f58404f0640 thread_name:ms_dispatch 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f58458c76f0] 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f584591494c] 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 3: raise() 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 4: abort() 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58458b171b] 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 6: /lib64/libc.so.6(+0x37386) [0x7f58458c0386] 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562978e316b9] 2024-07-26T11:07:04.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562978fef80c] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562978f2f2ca] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562978eff753] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562978f00a1b] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562978ee4752] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562978e32a56] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562978e36637] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562978e36f5e] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562978e1a03a] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5846194407] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 18: (DispatchQueue::entry()+0x60f) [0x7f5846194cf9] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f58462715a1] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 20: (Thread::entry_wrapper()+0x43) [0x7f584601ecdd] 2024-07-26T11:07:04.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 21: (Thread::_entry_func(void*)+0xd) [0x7f584601ecf9] 2024-07-26T11:07:04.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5845912c02] 2024-07-26T11:07:04.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5845997c40] 2024-07-26T11:07:04.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:04.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[68633]: 2024-07-26T11:07:04.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:03 smithi183 podman[72070]: 2024-07-26 11:07:03.975944974 +0000 UTC m=+0.018855360 container died 34383d6ad4a10d76bae1a62e2e7d74b3cae237812d13822ddf43c6846341c8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:07:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:03 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:rejoin 2024-07-26T11:07:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:03 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:03 smithi144 ceph-mon[32129]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:07:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:03 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:rejoin 2024-07-26T11:07:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:03 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:03 smithi160 ceph-mon[36715]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:07:04.997 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:04 smithi183 podman[72070]: 2024-07-26 11:07:04.739924148 +0000 UTC m=+0.782834532 container remove 34383d6ad4a10d76bae1a62e2e7d74b3cae237812d13822ddf43c6846341c8fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-07-26T11:07:04.997 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:04 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:07:04.997 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:04 smithi183 ceph-mon[37273]: pgmap v603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:07:04.997 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:04 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:04.998 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:04 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:07:04.998 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:04 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:active 2024-07-26T11:07:04.998 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:04 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:04.998 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:04 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:04 smithi144 ceph-mon[32129]: pgmap v603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:04 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:04 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:04 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:active 2024-07-26T11:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:04 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:04 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:04 smithi160 ceph-mon[36715]: pgmap v603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:07:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:04 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:04 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:07:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:04 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2490842163,v1:172.21.15.183:6837/2490842163] up:active 2024-07-26T11:07:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:04 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:05.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:04 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:05.385 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:04 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:07:05.385 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:04 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 2.160s CPU time. 2024-07-26T11:07:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:06 smithi183 ceph-mon[37273]: pgmap v604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 KiB/s wr, 7 op/s 2024-07-26T11:07:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:06 smithi144 ceph-mon[32129]: pgmap v604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 KiB/s wr, 7 op/s 2024-07-26T11:07:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:06 smithi160 ceph-mon[36715]: pgmap v604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 KiB/s wr, 7 op/s 2024-07-26T11:07:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:08 smithi183 ceph-mon[37273]: pgmap v605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-07-26T11:07:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:08 smithi144 ceph-mon[32129]: pgmap v605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-07-26T11:07:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:08 smithi160 ceph-mon[36715]: pgmap v605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-07-26T11:07:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:10 smithi183 ceph-mon[37273]: pgmap v606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-07-26T11:07:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:10 smithi144 ceph-mon[32129]: pgmap v606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-07-26T11:07:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:10 smithi160 ceph-mon[36715]: pgmap v606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-07-26T11:07:11.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:11.602+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:11.604+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:11.604+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:11.604+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:11.604+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:12 smithi183 ceph-mon[37273]: pgmap v607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-07-26T11:07:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:12 smithi144 ceph-mon[32129]: pgmap v607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-07-26T11:07:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:12 smithi160 ceph-mon[36715]: pgmap v607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-07-26T11:07:15.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:14 smithi183 ceph-mon[37273]: pgmap v608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-07-26T11:07:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:14 smithi144 ceph-mon[32129]: pgmap v608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-07-26T11:07:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:14 smithi160 ceph-mon[36715]: pgmap v608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-07-26T11:07:15.504 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:07:15.504 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 systemd[1]: Stopped Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:15.504 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 2.160s CPU time. 2024-07-26T11:07:15.504 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 systemd[1]: Starting Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:07:15.504 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 podman[72161]: 2024-07-26 11:07:15.277709096 +0000 UTC m=+0.018185685 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:07:15.504 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 podman[72161]: 2024-07-26 11:07:15.382012524 +0000 UTC m=+0.122489097 container create d1e65fcf791f5feea5ff65843f728be9cb4fb42db2423dde586023cb5189e4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-26T11:07:15.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 podman[72161]: 2024-07-26 11:07:15.618152978 +0000 UTC m=+0.358629567 container init d1e65fcf791f5feea5ff65843f728be9cb4fb42db2423dde586023cb5189e4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-07-26T11:07:15.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 podman[72161]: 2024-07-26 11:07:15.620755435 +0000 UTC m=+0.361232017 container start d1e65fcf791f5feea5ff65843f728be9cb4fb42db2423dde586023cb5189e4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-26T11:07:15.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mds[72180]: -- 172.21.15.183:0/1785168174 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56499d063860 con 0x5649a1205600 2024-07-26T11:07:15.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: starting mds.c at 2024-07-26T11:07:15.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 bash[72161]: d1e65fcf791f5feea5ff65843f728be9cb4fb42db2423dde586023cb5189e4de 2024-07-26T11:07:15.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:15 smithi183 systemd[1]: Started Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: daemon mds.c restarted 2024-07-26T11:07:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:16.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:07:16.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T11:07:16.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:boot 2024-07-26T11:07:16.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:07:16.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:16.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:15 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: daemon mds.c restarted 2024-07-26T11:07:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:07:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T11:07:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:boot 2024-07-26T11:07:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:07:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:15 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: daemon mds.c restarted 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:boot 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:15 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:16 smithi144 ceph-mon[32129]: pgmap v609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 KiB/s wr, 10 op/s 2024-07-26T11:07:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:16 smithi160 ceph-mon[36715]: pgmap v609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 KiB/s wr, 10 op/s 2024-07-26T11:07:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:16 smithi183 ceph-mon[37273]: pgmap v609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 KiB/s wr, 10 op/s 2024-07-26T11:07:18.179 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:17 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:resolve 2024-07-26T11:07:18.179 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:17 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:17 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:resolve 2024-07-26T11:07:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:17 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:17 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:resolve 2024-07-26T11:07:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:17 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:18 smithi144 ceph-mon[32129]: pgmap v611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 6 op/s 2024-07-26T11:07:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:18 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:reconnect 2024-07-26T11:07:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:18 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:18 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:18 smithi160 ceph-mon[36715]: pgmap v611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 6 op/s 2024-07-26T11:07:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:18 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:reconnect 2024-07-26T11:07:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:18 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:18 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:18 smithi183 ceph-mon[37273]: pgmap v611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 6 op/s 2024-07-26T11:07:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:18 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:reconnect 2024-07-26T11:07:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:18 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:18 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:19 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:rejoin 2024-07-26T11:07:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:19 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:19 smithi144 ceph-mon[32129]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:07:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:19 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:rejoin 2024-07-26T11:07:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:19 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:19 smithi160 ceph-mon[36715]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:rejoin 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-mon[37273]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: *** Caught signal (Aborted) ** 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: in thread 7f69db78c640 thread_name:ms_dispatch 2024-07-26T11:07:20.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f69e0b636f0] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f69e0bb094c] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 3: raise() 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 4: abort() 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 5: /lib64/libc.so.6(+0x2871b) [0x7f69e0b4d71b] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 6: /lib64/libc.so.6(+0x37386) [0x7f69e0b5c386] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56380f6e96b9] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56380f8a780c] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56380f7e72ca] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56380f7b7753] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56380f7b8a1b] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56380f79c752] 2024-07-26T11:07:20.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56380f6eaa56] 2024-07-26T11:07:20.387 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56380f6ee637] 2024-07-26T11:07:20.387 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56380f6eef5e] 2024-07-26T11:07:20.387 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56380f6d203a] 2024-07-26T11:07:20.387 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f69e1430407] 2024-07-26T11:07:20.387 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 18: (DispatchQueue::entry()+0x60f) [0x7f69e1430cf9] 2024-07-26T11:07:20.387 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69e150d5a1] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 20: (Thread::entry_wrapper()+0x43) [0x7f69e12bacdd] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 21: (Thread::_entry_func(void*)+0xd) [0x7f69e12bacf9] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 22: /lib64/libc.so.6(+0x89c02) [0x7f69e0baec02] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f69e0c33c40] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:19.995+0000 7f69db78c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: in thread 7f69db78c640 thread_name:ms_dispatch 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f69e0b636f0] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f69e0bb094c] 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 3: raise() 2024-07-26T11:07:20.388 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 4: abort() 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 5: /lib64/libc.so.6(+0x2871b) [0x7f69e0b4d71b] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 6: /lib64/libc.so.6(+0x37386) [0x7f69e0b5c386] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56380f6e96b9] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56380f8a780c] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56380f7e72ca] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56380f7b7753] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56380f7b8a1b] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56380f79c752] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56380f6eaa56] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56380f6ee637] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56380f6eef5e] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56380f6d203a] 2024-07-26T11:07:20.389 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f69e1430407] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 18: (DispatchQueue::entry()+0x60f) [0x7f69e1430cf9] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69e150d5a1] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 20: (Thread::entry_wrapper()+0x43) [0x7f69e12bacdd] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 21: (Thread::_entry_func(void*)+0xd) [0x7f69e12bacf9] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 22: /lib64/libc.so.6(+0x89c02) [0x7f69e0baec02] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f69e0c33c40] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:19 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 0> 2024-07-26T11:07:19.995+0000 7f69db78c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: in thread 7f69db78c640 thread_name:ms_dispatch 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f69e0b636f0] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f69e0bb094c] 2024-07-26T11:07:20.390 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 3: raise() 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 4: abort() 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 5: /lib64/libc.so.6(+0x2871b) [0x7f69e0b4d71b] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 6: /lib64/libc.so.6(+0x37386) [0x7f69e0b5c386] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56380f6e96b9] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56380f8a780c] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56380f7e72ca] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56380f7b7753] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56380f7b8a1b] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56380f79c752] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56380f6eaa56] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56380f6ee637] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56380f6eef5e] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56380f6d203a] 2024-07-26T11:07:20.391 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f69e1430407] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 18: (DispatchQueue::entry()+0x60f) [0x7f69e1430cf9] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69e150d5a1] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 20: (Thread::entry_wrapper()+0x43) [0x7f69e12bacdd] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 21: (Thread::_entry_func(void*)+0xd) [0x7f69e12bacf9] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 22: /lib64/libc.so.6(+0x89c02) [0x7f69e0baec02] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f69e0c33c40] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 0> 2024-07-26T11:07:19.995+0000 7f69db78c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: in thread 7f69db78c640 thread_name:ms_dispatch 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f69e0b636f0] 2024-07-26T11:07:20.392 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f69e0bb094c] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 3: raise() 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 4: abort() 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 5: /lib64/libc.so.6(+0x2871b) [0x7f69e0b4d71b] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 6: /lib64/libc.so.6(+0x37386) [0x7f69e0b5c386] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56380f6e96b9] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56380f8a780c] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56380f7e72ca] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56380f7b7753] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56380f7b8a1b] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56380f79c752] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56380f6eaa56] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56380f6ee637] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56380f6eef5e] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56380f6d203a] 2024-07-26T11:07:20.393 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f69e1430407] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 18: (DispatchQueue::entry()+0x60f) [0x7f69e1430cf9] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69e150d5a1] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 20: (Thread::entry_wrapper()+0x43) [0x7f69e12bacdd] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 21: (Thread::_entry_func(void*)+0xd) [0x7f69e12bacf9] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 22: /lib64/libc.so.6(+0x89c02) [0x7f69e0baec02] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f69e0c33c40] 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72019]: 2024-07-26T11:07:20.394 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 podman[72214]: 2024-07-26 11:07:20.071276477 +0000 UTC m=+0.021191162 container died 2b882ce2ab3dfb032029fe63422e44f54fc994a45262d9e1ff5b71315e46ad3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:07:20.954 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 podman[72214]: 2024-07-26 11:07:20.665018982 +0000 UTC m=+0.614933687 container remove 2b882ce2ab3dfb032029fe63422e44f54fc994a45262d9e1ff5b71315e46ad3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-26T11:07:20.954 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:07:20.954 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:07:20.954 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:20 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 2.025s CPU time. 2024-07-26T11:07:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: pgmap v612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-07-26T11:07:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:07:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:active 2024-07-26T11:07:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:21.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:21.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:20 smithi144 ceph-mon[32129]: scrub summary: idle+waiting paths [/] 2024-07-26T11:07:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: pgmap v612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-07-26T11:07:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:07:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:active 2024-07-26T11:07:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:20 smithi160 ceph-mon[36715]: scrub summary: idle+waiting paths [/] 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: pgmap v612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2565569132,v1:172.21.15.183:6839/2565569132] up:active 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:20 smithi183 ceph-mon[37273]: scrub summary: idle+waiting paths [/] 2024-07-26T11:07:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:22 smithi144 ceph-mon[32129]: pgmap v613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-07-26T11:07:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:22 smithi160 ceph-mon[36715]: pgmap v613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-07-26T11:07:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:22 smithi183 ceph-mon[37273]: pgmap v613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-07-26T11:07:25.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:07:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:24 smithi144 ceph-mon[32129]: pgmap v614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:07:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:24 smithi160 ceph-mon[36715]: pgmap v614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:07:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:24 smithi183 ceph-mon[37273]: pgmap v614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:07:25.566 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:07:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:26.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:26.622+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:26.622+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:26.622+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:26.622+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:26 smithi160 ceph-mon[36715]: pgmap v615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-07-26T11:07:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:26 smithi183 ceph-mon[37273]: pgmap v615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-07-26T11:07:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:26 smithi144 ceph-mon[32129]: pgmap v615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-07-26T11:07:28.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 -- 172.21.15.144:0/3594405755 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f942410b880 msgr2=0x7f94241120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 --2- 172.21.15.144:0/3594405755 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f942410b880 0x7f94241120b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f941c009be0 tx=0x7f941c02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 -- 172.21.15.144:0/3594405755 shutdown_connections 2024-07-26T11:07:28.040 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 --2- 172.21.15.144:0/3594405755 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f942410b880 0x7f94241120b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 --2- 172.21.15.144:0/3594405755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f942410af30 0x7f942410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 --2- 172.21.15.144:0/3594405755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94241055f0 0x7f9424105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 -- 172.21.15.144:0/3594405755 >> 172.21.15.144:0/3594405755 conn(0x7f9424101040 msgr2=0x7f9424103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:28.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 -- 172.21.15.144:0/3594405755 shutdown_connections 2024-07-26T11:07:28.041 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.037+0000 7f9428a00640 1 -- 172.21.15.144:0/3594405755 wait complete. 2024-07-26T11:07:28.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 Processor -- start 2024-07-26T11:07:28.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 -- start start 2024-07-26T11:07:28.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 0x7f942419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.043 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f942410af30 0x7f942419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f942410b880 0x7f94241a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9424115550 con 0x7f942410b880 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94241153d0 con 0x7f94241055f0 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9428a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f94241156d0 con 0x7f942410af30 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9422800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 0x7f942419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.039+0000 7f9423200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f942410b880 0x7f94241a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 0x7f942419bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49814/0 (socket says 172.21.15.144:49814) 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9423200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f942410b880 0x7f94241a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50868/0 (socket says 172.21.15.144:50868) 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 -- 172.21.15.144:0/2732408449 learned_addr learned my addr 172.21.15.144:0/2732408449 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f941be00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f942410af30 0x7f942419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.044 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 -- 172.21.15.144:0/2732408449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f942410af30 msgr2=0x7f942419c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f942410af30 0x7f942419c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 -- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f942410b880 msgr2=0x7f94241a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f942410b880 0x7f94241a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 -- 172.21.15.144:0/2732408449 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94241a1e30 con 0x7f94241055f0 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.040+0000 7f9422800640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 0x7f942419bec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f941400e9c0 tx=0x7f941400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.041+0000 7f941b400640 1 -- 172.21.15.144:0/2732408449 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941400cde0 con 0x7f94241055f0 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.041+0000 7f941b400640 1 -- 172.21.15.144:0/2732408449 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94140035f0 con 0x7f94241055f0 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.041+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94241a2120 con 0x7f94241055f0 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.041+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94241a8880 con 0x7f94241055f0 2024-07-26T11:07:28.045 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.041+0000 7f941b400640 1 -- 172.21.15.144:0/2732408449 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414010690 con 0x7f94241055f0 2024-07-26T11:07:28.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.044+0000 7f9430095740 1 Processor -- start 2024-07-26T11:07:28.046 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.044+0000 7f9430095740 1 -- start start 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.044+0000 7f941b400640 1 -- 172.21.15.144:0/2732408449 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f9414004000 con 0x7f94241055f0 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9430095740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b23000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9430095740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b23540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9430095740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b23a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9430095740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563de3a99bc0 con 0x563de375d390 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9430095740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563de3a890c0 con 0x563de377e070 2024-07-26T11:07:28.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9430095740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563de3a8d100 con 0x563de37551b0 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f941be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b23a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f941be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b23a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50874/0 (socket says 172.21.15.144:50874) 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f941be00640 1 -- 172.21.15.144:0/2404011364 learned_addr learned my addr 172.21.15.144:0/2404011364 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9423200640 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b23000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.045+0000 7f9422800640 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b23540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f9423200640 1 -- 172.21.15.144:0/2404011364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 msgr2=0x563de3b23540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f9423200640 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b23540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f9423200640 1 -- 172.21.15.144:0/2404011364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 msgr2=0x563de3b23a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f9423200640 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b23a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.049 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f9423200640 1 -- 172.21.15.144:0/2404011364 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563de37e9650 con 0x563de377e070 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f9423200640 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b23000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f941c009b30 tx=0x7f941c030430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55006f9fcb40820a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.046+0000 7f941b400640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f94080b3dc0 0x7f94080b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.047+0000 7f9413400640 1 -- 172.21.15.144:0/2404011364 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c041030 con 0x563de377e070 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.047+0000 7f9413400640 1 -- 172.21.15.144:0/2404011364 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f941c003e20 con 0x563de377e070 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.047+0000 7f9413400640 1 -- 172.21.15.144:0/2404011364 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c036040 con 0x563de377e070 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.047+0000 7f941b400640 1 -- 172.21.15.144:0/2732408449 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9910+0+0 (secure 0 0 0) 0x7f94140e4850 con 0x7f94241055f0 2024-07-26T11:07:28.050 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.047+0000 7f9422800640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f94080b3dc0 0x7f94080b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.051 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 -- 172.21.15.144:0/2404011364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 msgr2=0x563de3b23000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9422800640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f94080b3dc0 0x7f94080b62d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x563de3b21c80 tx=0x7f94140ea530 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b23000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f941c009b30 tx=0x7f941c030430 comp rx=0 tx=0).stop 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 -- 172.21.15.144:0/2404011364 shutdown_connections 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b23a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b23540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 --2- 172.21.15.144:0/2404011364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b23000 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 -- 172.21.15.144:0/2404011364 >> 172.21.15.144:0/2404011364 conn(0x563de3772460 msgr2=0x563de3b20ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:28.052 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 -- 172.21.15.144:0/2404011364 shutdown_connections 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.048+0000 7f9430095740 1 -- 172.21.15.144:0/2404011364 wait complete. 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.049+0000 7f9430095740 1 -- start start 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.049+0000 7f9430095740 1 Processor -- start 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b2e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 0x563de3b2eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3b2f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563de38ec700 con 0x563de377e070 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563de3a25690 con 0x563de375d390 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563de3b31780 con 0x563de37551b0 2024-07-26T11:07:28.053 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9430095740 10 client.0 _ll_register_callbacks cb 0x563de3a35140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:07:28.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 0x563de3b2eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 0x563de3b2eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:49840/0 (socket says 172.21.15.144:49840) 2024-07-26T11:07:28.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 -- 172.21.15.144:0/2768050740 learned_addr learned my addr 172.21.15.144:0/2768050740 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9423200640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b2e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f9423200640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3b2f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 -- 172.21.15.144:0/2768050740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 msgr2=0x563de3b2e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b2e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 -- 172.21.15.144:0/2768050740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 msgr2=0x563de3b2f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.050+0000 7f941be00640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3b2f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f941be00640 1 -- 172.21.15.144:0/2768050740 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563de3a3c830 con 0x563de375d390 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9423200640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3b2f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f941be00640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 0x563de3b2eae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f940c012810 tx=0x7f940c012cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b072d267c2e1e01 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9407e00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9407e00640 1 -- 172.21.15.144:0/2768050740 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c004050 con 0x563de375d390 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f941c003e20 con 0x563de375d390 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9407e00640 1 -- 172.21.15.144:0/2768050740 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9910+0+0 (secure 0 0 0) 0x7f940c0041f0 con 0x563de375d390 2024-07-26T11:07:28.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9407e00640 1 -- 172.21.15.144:0/2768050740 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f940c0043f0 con 0x563de375d390 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9430095740 10 client.26174 fetch_fsmap learned FSMap version 69 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.051+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x563de3b3d430 con 0x563de375d390 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9407e00640 1 -- 172.21.15.144:0/2768050740 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f941c003e20 con 0x563de375d390 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9407e00640 1 -- 172.21.15.144:0/2768050740 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 69) ==== 8362+0+0 (secure 0 0 0) 0x7f940c003b50 con 0x563de375d390 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9430095740 10 client.26174 fetch_fsmap finished waiting for FSMap version 69 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9430095740 10 client.26174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9430095740 20 client.26174 populate_metadata read hostname 'smithi144' 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9430095740 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de373c9f0 0x563de3b408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9430095740 4 client.26174 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.052+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563de3a9bb10 con 0x563de373c9f0 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.053+0000 7f9423200640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de373c9f0 0x563de3b408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.053+0000 7f9423200640 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de373c9f0 0x563de3b408d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fcfd04729a84475 server_cookie=8c1c2fd3eca1aa8b in_seq=0 out_seq=0 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.053+0000 7f9407e00640 10 client.26174 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:07:28.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9407e00640 1 -- 172.21.15.144:0/2768050740 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563de3a9bb10 con 0x563de373c9f0 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9407e00640 10 client.26174 handle_command_reply: tid=0 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 client.26174 shutdown 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 msgr2=0x563de3b2eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 0x563de3b2eae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f940c012810 tx=0x7f940c012cd0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 shutdown_connections 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3b2f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de373c9f0 0x563de3b408d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de375d390 0x563de3b2eae0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 --2- 172.21.15.144:0/2768050740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b2e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 >> 172.21.15.144:0/2768050740 conn(0x563de3772460 msgr2=0x563de3b261e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9407e00640 0 client.26174 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 shutdown_connections 2024-07-26T11:07:28.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.054+0000 7f9430095740 1 -- 172.21.15.144:0/2768050740 wait complete. 2024-07-26T11:07:28.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.055+0000 7f9430095740 20 client.26174 trim_cache size 0 max 16384 2024-07-26T11:07:28.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.073+0000 7f9430095740 1 Processor -- start 2024-07-26T11:07:28.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.073+0000 7f9430095740 1 -- start start 2024-07-26T11:07:28.075 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9430095740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b30480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9430095740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b1f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9430095740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b24740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9430095740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b5400 con 0x563de375d390 2024-07-26T11:07:28.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9430095740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b2cd0 con 0x563de377e070 2024-07-26T11:07:28.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9430095740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b0b20 con 0x563de37551b0 2024-07-26T11:07:28.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f941be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b1f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9423200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b30480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f941be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b1f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50892/0 (socket says 172.21.15.144:50892) 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f941be00640 1 -- 172.21.15.144:0/3589359227 learned_addr learned my addr 172.21.15.144:0/3589359227 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.074+0000 7f9423200640 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b24740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f941be00640 1 -- 172.21.15.144:0/3589359227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 msgr2=0x563de3b30480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f941be00640 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b30480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f941be00640 1 -- 172.21.15.144:0/3589359227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 msgr2=0x563de3b24740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f941be00640 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b24740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f941be00640 1 -- 172.21.15.144:0/3589359227 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563de3b21cb0 con 0x563de375d390 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f9423200640 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b30480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:07:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f941be00640 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b1f530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f940c012560 tx=0x7f940c010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e380da60b67778 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:28.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.075+0000 7f9412a00640 1 -- 172.21.15.144:0/3589359227 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c02d020 con 0x563de375d390 2024-07-26T11:07:28.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.076+0000 7f9412a00640 1 -- 172.21.15.144:0/3589359227 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c004080 con 0x563de375d390 2024-07-26T11:07:28.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.076+0000 7f9412a00640 1 -- 172.21.15.144:0/3589359227 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c005040 con 0x563de375d390 2024-07-26T11:07:28.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 -- 172.21.15.144:0/3589359227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 msgr2=0x563de3b1f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b1f530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f940c012560 tx=0x7f940c010930 comp rx=0 tx=0).stop 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 -- 172.21.15.144:0/3589359227 shutdown_connections 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de377e070 0x563de3b24740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de375d390 0x563de3b1f530 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 --2- 172.21.15.144:0/3589359227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de37551b0 0x563de3b30480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 -- 172.21.15.144:0/3589359227 >> 172.21.15.144:0/3589359227 conn(0x563de3772460 msgr2=0x563de3b24030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 -- 172.21.15.144:0/3589359227 shutdown_connections 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 -- 172.21.15.144:0/3589359227 wait complete. 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.077+0000 7f9430095740 1 -- start start 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 Processor -- start 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de37551b0 0x563de3b341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.081 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de375d390 0x563de3b347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3ba0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b1190 con 0x563de377e070 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b2600 con 0x563de37551b0 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b1860 con 0x563de375d390 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9430095740 10 client.0 _ll_register_callbacks cb 0x7f940c012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.078+0000 7f9423200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3ba0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de375d390 0x563de3b347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f9423200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3ba0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50896/0 (socket says 172.21.15.144:50896) 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f9423200640 1 -- 172.21.15.144:0/1459944776 learned_addr learned my addr 172.21.15.144:0/1459944776 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:07:28.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f9423200640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de37551b0 0x563de3b341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 -- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de37551b0 msgr2=0x563de3b341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de37551b0 0x563de3b341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 -- 172.21.15.144:0/1459944776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 msgr2=0x563de3ba0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3ba0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 -- 172.21.15.144:0/1459944776 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563de3ba1660 con 0x563de375d390 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f9423200640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3ba0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:07:28.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f941be00640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de375d390 0x563de3b347a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x563de36f9b60 tx=0x7f940c00ffd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2e510e54ddfcc11 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.079+0000 7f9407e00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9407e00640 1 -- 172.21.15.144:0/1459944776 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c02d030 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f940c004080 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9407e00640 1 -- 172.21.15.144:0/1459944776 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9910+0+0 (secure 0 0 0) 0x7f940c0109a0 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9407e00640 1 -- 172.21.15.144:0/1459944776 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f940c044050 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9430095740 10 client.26167 fetch_fsmap learned FSMap version 69 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=4+}) -- 0x563de3ba89b0 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.080+0000 7f9407e00640 1 -- 172.21.15.144:0/1459944776 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c004080 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9407e00640 1 -- 172.21.15.144:0/1459944776 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 69) ==== 8362+0+0 (secure 0 0 0) 0x7f940c0035f0 con 0x563de375d390 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9430095740 10 client.26167 fetch_fsmap finished waiting for FSMap version 69 2024-07-26T11:07:28.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9430095740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9430095740 20 client.26167 populate_metadata read hostname 'smithi144' 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9430095740 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de3711da0 0x563de3bab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9430095740 4 client.26167 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563de3a975b0 con 0x563de3711da0 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.081+0000 7f9423200640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de3711da0 0x563de3bab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9423200640 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de3711da0 0x563de3bab0f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64ea7ce829b482d2 server_cookie=60f4090ae72ca83a in_seq=0 out_seq=0 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9407e00640 10 client.26167 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9407e00640 1 -- 172.21.15.144:0/1459944776 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563de3a975b0 con 0x563de3711da0 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9407e00640 10 client.26167 handle_command_reply: tid=0 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9430095740 1 client.26167 shutdown 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de375d390 msgr2=0x563de3b347a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.082+0000 7f9430095740 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de375d390 0x563de3b347a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x563de36f9b60 tx=0x7f940c00ffd0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 shutdown_connections 2024-07-26T11:07:28.087 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563de377e070 0x563de3ba0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563de375d390 0x563de3b347a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563de3711da0 0x563de3bab0f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 --2- 172.21.15.144:0/1459944776 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563de37551b0 0x563de3b341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 >> 172.21.15.144:0/1459944776 conn(0x563de3772460 msgr2=0x563de3b30220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9407e00640 0 client.26167 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 shutdown_connections 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 1 -- 172.21.15.144:0/1459944776 wait complete. 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.083+0000 7f9430095740 20 client.26167 trim_cache size 0 max 16384 2024-07-26T11:07:28.088 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:07:28.089 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f94080b3dc0 msgr2=0x7f94080b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f94080b3dc0 0x7f94080b62d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x563de3b21c80 tx=0x7f94140ea530 comp rx=0 tx=0).stop 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 msgr2=0x7f942419bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 0x7f942419bec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f941400e9c0 tx=0x7f941400ee80 comp rx=0 tx=0).stop 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 shutdown_connections 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f94080b3dc0 0x7f94080b62d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f942410b880 0x7f94241a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f942410af30 0x7f942419c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 --2- 172.21.15.144:0/2732408449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94241055f0 0x7f942419bec0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:28.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 >> 172.21.15.144:0/2732408449 conn(0x7f9424101040 msgr2=0x7f9424103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:28.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 shutdown_connections 2024-07-26T11:07:28.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:07:28.084+0000 7f9428a00640 1 -- 172.21.15.144:0/2732408449 wait complete. 2024-07-26T11:07:28.888 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:07:29.208 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:28 smithi144 ceph-mon[32129]: pgmap v616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-07-26T11:07:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:28 smithi160 ceph-mon[36715]: pgmap v616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-07-26T11:07:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:28 smithi183 ceph-mon[37273]: pgmap v616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-07-26T11:07:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:30 smithi160 ceph-mon[36715]: pgmap v617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-07-26T11:07:31.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:30 smithi183 ceph-mon[37273]: pgmap v617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-07-26T11:07:31.368 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:07:31.368 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 systemd[1]: Stopped Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:31.368 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 2.025s CPU time. 2024-07-26T11:07:31.368 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 systemd[1]: Starting Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:07:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:30 smithi144 ceph-mon[32129]: pgmap v617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-07-26T11:07:31.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 podman[72304]: 2024-07-26 11:07:31.269518382 +0000 UTC m=+0.017481979 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:07:31.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 podman[72304]: 2024-07-26 11:07:31.373804729 +0000 UTC m=+0.121768313 container create 4b1d9971842a18a572798870f2e11029fd84d81669eeff29867192f1b300e131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-26T11:07:31.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 podman[72304]: 2024-07-26 11:07:31.601448389 +0000 UTC m=+0.349411989 container init 4b1d9971842a18a572798870f2e11029fd84d81669eeff29867192f1b300e131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:07:31.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 podman[72304]: 2024-07-26 11:07:31.604023578 +0000 UTC m=+0.351987163 container start 4b1d9971842a18a572798870f2e11029fd84d81669eeff29867192f1b300e131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T11:07:31.994 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mds[72322]: -- 172.21.15.183:0/1502708430 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e06e9d860 con 0x557e0b03f600 2024-07-26T11:07:31.994 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: starting mds.i at 2024-07-26T11:07:31.994 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 bash[72304]: 4b1d9971842a18a572798870f2e11029fd84d81669eeff29867192f1b300e131 2024-07-26T11:07:31.994 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:31 smithi183 systemd[1]: Started Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: daemon mds.i restarted 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:boot 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:31 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: daemon mds.i restarted 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:boot 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:31 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: daemon mds.i restarted 2024-07-26T11:07:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:07:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T11:07:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:boot 2024-07-26T11:07:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:07:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:32.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:31 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:32 smithi160 ceph-mon[36715]: pgmap v618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:07:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:32 smithi183 ceph-mon[37273]: pgmap v618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:07:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:32 smithi144 ceph-mon[32129]: pgmap v618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:07:34.271 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:34 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:resolve 2024-07-26T11:07:34.271 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:34 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:34 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:resolve 2024-07-26T11:07:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:34 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:34 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:resolve 2024-07-26T11:07:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:34 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:35 smithi160 ceph-mon[36715]: pgmap v620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:35 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:reconnect 2024-07-26T11:07:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:35 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:35.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:35 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-mon[37273]: pgmap v620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:reconnect 2024-07-26T11:07:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:35 smithi144 ceph-mon[32129]: pgmap v620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:35 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:reconnect 2024-07-26T11:07:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:35 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:35 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:35.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:07:35.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: *** Caught signal (Aborted) ** 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: in thread 7fa2a5dd7640 thread_name:ms_dispatch 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa2ab1ae6f0] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa2ab1fb94c] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 3: raise() 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 4: abort() 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa2ab19871b] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 6: /lib64/libc.so.6(+0x37386) [0x7fa2ab1a7386] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56499c0936b9] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56499c25180c] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56499c1912ca] 2024-07-26T11:07:35.886 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56499c161753] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56499c162a1b] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56499c146752] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56499c094a56] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56499c098637] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56499c098f5e] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56499c07c03a] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa2aba7b407] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 18: (DispatchQueue::entry()+0x60f) [0x7fa2aba7bcf9] 2024-07-26T11:07:35.887 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa2abb585a1] 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 20: (Thread::entry_wrapper()+0x43) [0x7fa2ab905cdd] 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa2ab905cf9] 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa2ab1f9c02] 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa2ab27ec40] 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.513+0000 7fa2a5dd7640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: in thread 7fa2a5dd7640 thread_name:ms_dispatch 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.888 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa2ab1ae6f0] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa2ab1fb94c] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 3: raise() 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 4: abort() 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa2ab19871b] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 6: /lib64/libc.so.6(+0x37386) [0x7fa2ab1a7386] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56499c0936b9] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56499c25180c] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56499c1912ca] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56499c161753] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56499c162a1b] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56499c146752] 2024-07-26T11:07:35.889 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56499c094a56] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56499c098637] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56499c098f5e] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56499c07c03a] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa2aba7b407] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 18: (DispatchQueue::entry()+0x60f) [0x7fa2aba7bcf9] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa2abb585a1] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 20: (Thread::entry_wrapper()+0x43) [0x7fa2ab905cdd] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa2ab905cf9] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa2ab1f9c02] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa2ab27ec40] 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.890 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 0> 2024-07-26T11:07:35.513+0000 7fa2a5dd7640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: in thread 7fa2a5dd7640 thread_name:ms_dispatch 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa2ab1ae6f0] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa2ab1fb94c] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 3: raise() 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 4: abort() 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa2ab19871b] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 6: /lib64/libc.so.6(+0x37386) [0x7fa2ab1a7386] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56499c0936b9] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56499c25180c] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56499c1912ca] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56499c161753] 2024-07-26T11:07:35.891 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56499c162a1b] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56499c146752] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56499c094a56] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56499c098637] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56499c098f5e] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56499c07c03a] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa2aba7b407] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 18: (DispatchQueue::entry()+0x60f) [0x7fa2aba7bcf9] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa2abb585a1] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 20: (Thread::entry_wrapper()+0x43) [0x7fa2ab905cdd] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa2ab905cf9] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa2ab1f9c02] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa2ab27ec40] 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:35.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 0> 2024-07-26T11:07:35.513+0000 7fa2a5dd7640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: in thread 7fa2a5dd7640 thread_name:ms_dispatch 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa2ab1ae6f0] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa2ab1fb94c] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 3: raise() 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 4: abort() 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa2ab19871b] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 6: /lib64/libc.so.6(+0x37386) [0x7fa2ab1a7386] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56499c0936b9] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56499c25180c] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56499c1912ca] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56499c161753] 2024-07-26T11:07:35.893 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56499c162a1b] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56499c146752] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56499c094a56] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56499c098637] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56499c098f5e] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56499c07c03a] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa2aba7b407] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 18: (DispatchQueue::entry()+0x60f) [0x7fa2aba7bcf9] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa2abb585a1] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 20: (Thread::entry_wrapper()+0x43) [0x7fa2ab905cdd] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa2ab905cf9] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa2ab1f9c02] 2024-07-26T11:07:35.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa2ab27ec40] 2024-07-26T11:07:35.895 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:35.895 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72176]: 2024-07-26T11:07:35.895 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:35 smithi183 podman[72356]: 2024-07-26 11:07:35.590980558 +0000 UTC m=+0.019392017 container died d1e65fcf791f5feea5ff65843f728be9cb4fb42db2423dde586023cb5189e4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-07-26T11:07:36.266 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:rejoin 2024-07-26T11:07:36.266 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:07:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:07:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:active 2024-07-26T11:07:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:36 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:rejoin 2024-07-26T11:07:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:07:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:07:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:active 2024-07-26T11:07:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:36 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:rejoin 2024-07-26T11:07:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:07:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:07:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3814743773,v1:172.21.15.183:6837/3814743773] up:active 2024-07-26T11:07:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:36 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:36.635 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:36 smithi183 podman[72356]: 2024-07-26 11:07:36.265379108 +0000 UTC m=+0.693790576 container remove d1e65fcf791f5feea5ff65843f728be9cb4fb42db2423dde586023cb5189e4de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-07-26T11:07:36.635 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:36 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:07:36.635 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:36 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:07:36.635 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:36 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.978s CPU time. 2024-07-26T11:07:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:37 smithi160 ceph-mon[36715]: pgmap v621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:37.309 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:07:37 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:07:37.156+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:07:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:37 smithi183 ceph-mon[37273]: pgmap v621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:37 smithi144 ceph-mon[32129]: pgmap v621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:07:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:07:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:07:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:39 smithi183 ceph-mon[37273]: pgmap v622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:07:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:39 smithi144 ceph-mon[32129]: pgmap v622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:07:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:39 smithi160 ceph-mon[36715]: pgmap v622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:07:41.179 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:41 smithi144 ceph-mon[32129]: pgmap v623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:41 smithi183 ceph-mon[37273]: pgmap v623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:41 smithi160 ceph-mon[36715]: pgmap v623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:41.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:41.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:41.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:41.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:41.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:43 smithi183 ceph-mon[37273]: pgmap v624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:43 smithi144 ceph-mon[32129]: pgmap v624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:43 smithi160 ceph-mon[36715]: pgmap v624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:45 smithi183 ceph-mon[37273]: pgmap v625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 175 B/s wr, 6 op/s 2024-07-26T11:07:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:45 smithi144 ceph-mon[32129]: pgmap v625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 175 B/s wr, 6 op/s 2024-07-26T11:07:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:45 smithi160 ceph-mon[36715]: pgmap v625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 175 B/s wr, 6 op/s 2024-07-26T11:07:46.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:46 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:07:46.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:46 smithi183 systemd[1]: Stopped Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:46.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:46 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.978s CPU time. 2024-07-26T11:07:46.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:46 smithi183 systemd[1]: Starting Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:07:46.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:46 smithi183 podman[72578]: 2024-07-26 11:07:46.771536119 +0000 UTC m=+0.017617462 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:07:46.892 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:46 smithi183 podman[72578]: 2024-07-26 11:07:46.890864042 +0000 UTC m=+0.136945384 container create a1c91b4c8c14c309f8b09b400175d3205271eaf49e5f10b41b771c2d47ae9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:07:47.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:47 smithi183 ceph-mon[37273]: pgmap v626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:07:47.163 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:47 smithi183 podman[72578]: 2024-07-26 11:07:47.127404936 +0000 UTC m=+0.373486272 container init a1c91b4c8c14c309f8b09b400175d3205271eaf49e5f10b41b771c2d47ae9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T11:07:47.164 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:47 smithi183 podman[72578]: 2024-07-26 11:07:47.130976476 +0000 UTC m=+0.377057813 container start a1c91b4c8c14c309f8b09b400175d3205271eaf49e5f10b41b771c2d47ae9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:07:47.446 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:47 smithi183 ceph-mds[72598]: -- 172.21.15.183:0/1653001279 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4afd09860 con 0x55c4b3eab180 2024-07-26T11:07:47.446 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:47 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: starting mds.c at 2024-07-26T11:07:47.446 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:47 smithi183 bash[72578]: a1c91b4c8c14c309f8b09b400175d3205271eaf49e5f10b41b771c2d47ae9108 2024-07-26T11:07:47.446 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:07:47 smithi183 systemd[1]: Started Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:07:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:47 smithi144 ceph-mon[32129]: pgmap v626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:07:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:47 smithi160 ceph-mon[36715]: pgmap v626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: daemon mds.c restarted 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:boot 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:48.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:48 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: daemon mds.c restarted 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:boot 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:48 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: daemon mds.c restarted 2024-07-26T11:07:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:07:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:07:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T11:07:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:boot 2024-07-26T11:07:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:07:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:07:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:48 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:49.459 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:49 smithi144 ceph-mon[32129]: pgmap v628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:49.459 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:resolve 2024-07-26T11:07:49.459 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:49 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:49 smithi160 ceph-mon[36715]: pgmap v628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:resolve 2024-07-26T11:07:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:49 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:49 smithi183 ceph-mon[37273]: pgmap v628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:07:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:resolve 2024-07-26T11:07:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:49 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:50.498 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:50 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:reconnect 2024-07-26T11:07:50.499 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:50 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:50.499 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:50 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:50.523 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:reconnect 2024-07-26T11:07:50.523 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:50.523 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:50 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:reconnect 2024-07-26T11:07:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:50 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:50 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:07:50.885 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: *** Caught signal (Aborted) ** 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: in thread 7f247070f640 thread_name:ms_dispatch 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2475ae66f0] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2475b3394c] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 3: raise() 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 4: abort() 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2475ad071b] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 6: /lib64/libc.so.6(+0x37386) [0x7f2475adf386] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e051ea6b9] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e053a880c] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e052e82ca] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e052b8753] 2024-07-26T11:07:50.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e052b9a1b] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e0529d752] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e051eba56] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e051ef637] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e051eff5e] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e051d303a] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f24763b3407] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 18: (DispatchQueue::entry()+0x60f) [0x7f24763b3cf9] 2024-07-26T11:07:50.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f24764905a1] 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 20: (Thread::entry_wrapper()+0x43) [0x7f247623dcdd] 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 21: (Thread::_entry_func(void*)+0xd) [0x7f247623dcf9] 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2475b31c02] 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2475bb6c40] 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.522+0000 7f247070f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: in thread 7f247070f640 thread_name:ms_dispatch 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2475ae66f0] 2024-07-26T11:07:50.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2475b3394c] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 3: raise() 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 4: abort() 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2475ad071b] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 6: /lib64/libc.so.6(+0x37386) [0x7f2475adf386] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e051ea6b9] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e053a880c] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e052e82ca] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e052b8753] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e052b9a1b] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e0529d752] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e051eba56] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e051ef637] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e051eff5e] 2024-07-26T11:07:50.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e051d303a] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f24763b3407] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 18: (DispatchQueue::entry()+0x60f) [0x7f24763b3cf9] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f24764905a1] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 20: (Thread::entry_wrapper()+0x43) [0x7f247623dcdd] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 21: (Thread::_entry_func(void*)+0xd) [0x7f247623dcf9] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2475b31c02] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2475bb6c40] 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 0> 2024-07-26T11:07:50.522+0000 7f247070f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: in thread 7f247070f640 thread_name:ms_dispatch 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:50.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2475ae66f0] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2475b3394c] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 3: raise() 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 4: abort() 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2475ad071b] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 6: /lib64/libc.so.6(+0x37386) [0x7f2475adf386] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e051ea6b9] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e053a880c] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e052e82ca] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e052b8753] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e052b9a1b] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e0529d752] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e051eba56] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e051ef637] 2024-07-26T11:07:50.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e051eff5e] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e051d303a] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f24763b3407] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 18: (DispatchQueue::entry()+0x60f) [0x7f24763b3cf9] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f24764905a1] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 20: (Thread::entry_wrapper()+0x43) [0x7f247623dcdd] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 21: (Thread::_entry_func(void*)+0xd) [0x7f247623dcf9] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2475b31c02] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2475bb6c40] 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 0> 2024-07-26T11:07:50.522+0000 7f247070f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: in thread 7f247070f640 thread_name:ms_dispatch 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:07:50.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2475ae66f0] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2475b3394c] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 3: raise() 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 4: abort() 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2475ad071b] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 6: /lib64/libc.so.6(+0x37386) [0x7f2475adf386] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e051ea6b9] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e053a880c] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e052e82ca] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e052b8753] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e052b9a1b] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e0529d752] 2024-07-26T11:07:50.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e051eba56] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e051ef637] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e051eff5e] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e051d303a] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f24763b3407] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 18: (DispatchQueue::entry()+0x60f) [0x7f24763b3cf9] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f24764905a1] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 20: (Thread::entry_wrapper()+0x43) [0x7f247623dcdd] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 21: (Thread::_entry_func(void*)+0xd) [0x7f247623dcf9] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2475b31c02] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2475bb6c40] 2024-07-26T11:07:50.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:07:50.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72318]: 2024-07-26T11:07:50.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:50 smithi183 podman[72643]: 2024-07-26 11:07:50.583247639 +0000 UTC m=+0.018689516 container died 4b1d9971842a18a572798870f2e11029fd84d81669eeff29867192f1b300e131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: pgmap v629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:rejoin 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:active 2024-07-26T11:07:51.552 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:51 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: pgmap v629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:07:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:rejoin 2024-07-26T11:07:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:07:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:07:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:active 2024-07-26T11:07:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:51 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:51.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:51 smithi183 podman[72643]: 2024-07-26 11:07:51.191779636 +0000 UTC m=+0.627221516 container remove 4b1d9971842a18a572798870f2e11029fd84d81669eeff29867192f1b300e131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:07:51.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:51 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:07:51.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:51 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:07:51.635 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:07:51 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 1.953s CPU time. 2024-07-26T11:07:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: pgmap v629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:07:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:rejoin 2024-07-26T11:07:51.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:07:51.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:07:51.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:07:51.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:07:51.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/222659679,v1:172.21.15.183:6839/222659679] up:active 2024-07-26T11:07:51.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:51 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:07:52.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:07:52 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:07:52.158+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:07:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:53 smithi160 ceph-mon[36715]: pgmap v630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:07:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:53 smithi183 ceph-mon[37273]: pgmap v630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:07:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:53 smithi144 ceph-mon[32129]: pgmap v630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:07:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:55 smithi160 ceph-mon[36715]: pgmap v631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:07:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:55 smithi183 ceph-mon[37273]: pgmap v631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:07:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:55 smithi144 ceph-mon[32129]: pgmap v631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:07:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:56.654+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:56.655+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:56.656+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:56.656+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:07:56.656+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:57 smithi160 ceph-mon[36715]: pgmap v632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:57 smithi183 ceph-mon[37273]: pgmap v632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:57 smithi144 ceph-mon[32129]: pgmap v632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:07:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:07:58 smithi160 ceph-mon[36715]: pgmap v633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 202 B/s wr, 7 op/s 2024-07-26T11:07:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:07:58 smithi183 ceph-mon[37273]: pgmap v633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 202 B/s wr, 7 op/s 2024-07-26T11:07:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:07:58 smithi144 ceph-mon[32129]: pgmap v633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 202 B/s wr, 7 op/s 2024-07-26T11:07:58.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:07:59.380 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:08:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:00 smithi183 ceph-mon[37273]: pgmap v634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:00 smithi144 ceph-mon[32129]: pgmap v634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:00 smithi160 ceph-mon[36715]: pgmap v634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:01.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 -- 172.21.15.144:0/428485301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed41055d0 msgr2=0x7f7ed41059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 --2- 172.21.15.144:0/428485301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed41055d0 0x7f7ed41059e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009900 tx=0x7f7ec802a410 comp rx=0 tx=0).stop 2024-07-26T11:08:01.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 -- 172.21.15.144:0/428485301 shutdown_connections 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 --2- 172.21.15.144:0/428485301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 0x7f7ed4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 --2- 172.21.15.144:0/428485301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ed410af10 0x7f7ed410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 --2- 172.21.15.144:0/428485301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed41055d0 0x7f7ed41059e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 -- 172.21.15.144:0/428485301 >> 172.21.15.144:0/428485301 conn(0x7f7ed4101040 msgr2=0x7f7ed4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 -- 172.21.15.144:0/428485301 shutdown_connections 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.838+0000 7f7edac00640 1 -- 172.21.15.144:0/428485301 wait complete. 2024-07-26T11:08:01.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.839+0000 7f7edac00640 1 Processor -- start 2024-07-26T11:08:01.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 -- start start 2024-07-26T11:08:01.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ed41055d0 0x7f7ed419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 0x7f7ed419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 0x7f7ed41a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed4115580 con 0x7f7ed410af10 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed4115400 con 0x7f7ed41055d0 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7edac00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed4115700 con 0x7f7ed410b860 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.840+0000 7f7ed8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 0x7f7ed419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7eda200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 0x7f7ed41a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 0x7f7ed419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38330/0 (socket says 172.21.15.144:38330) 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 -- 172.21.15.144:0/4056356878 learned_addr learned my addr 172.21.15.144:0/4056356878 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed9800640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ed41055d0 0x7f7ed419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 -- 172.21.15.144:0/4056356878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 msgr2=0x7f7ed41a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 0x7f7ed41a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 -- 172.21.15.144:0/4056356878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ed41055d0 msgr2=0x7f7ed419bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ed41055d0 0x7f7ed419bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 -- 172.21.15.144:0/4056356878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed41a1ee0 con 0x7f7ed410af10 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7eda200640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 0x7f7ed41a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.841+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 0x7f7ed419c510 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000e9f0 tx=0x7f7ec000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.842+0000 7f7ed2000640 1 -- 172.21.15.144:0/4056356878 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec000ccc0 con 0x7f7ed410af10 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.842+0000 7f7ed2000640 1 -- 172.21.15.144:0/4056356878 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec00108e0 con 0x7f7ed410af10 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.842+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed41a21d0 con 0x7f7ed410af10 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.842+0000 7f7ed2000640 1 -- 172.21.15.144:0/4056356878 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0018860 con 0x7f7ed410af10 2024-07-26T11:08:01.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.842+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed41a8930 con 0x7f7ed410af10 2024-07-26T11:08:01.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.845+0000 7f7ee2fd0740 1 Processor -- start 2024-07-26T11:08:01.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 -- start start 2024-07-26T11:08:01.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ed2000640 1 -- 172.21.15.144:0/4056356878 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7ec0010430 con 0x7f7ed410af10 2024-07-26T11:08:01.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d5144730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5144c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5642d50babc0 con 0x5642d4d761b0 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642d50aa0c0 con 0x5642d4d7e390 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642d50ae100 con 0x5642d4d9f070 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7eda200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7eda200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56216/0 (socket says 172.21.15.144:56216) 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7eda200640 1 -- 172.21.15.144:0/2814800973 learned_addr learned my addr 172.21.15.144:0/2814800973 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ed9800640 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d5144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.846+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7eda200640 1 -- 172.21.15.144:0/2814800973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 msgr2=0x5642d5144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7eda200640 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7eda200640 1 -- 172.21.15.144:0/2814800973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 msgr2=0x5642d5144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7eda200640 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d5144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7eda200640 1 -- 172.21.15.144:0/2814800973 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642d4e0a650 con 0x5642d4d9f070 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:08:01.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7eda200640 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51441f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0117e0 tx=0x7f7ecc011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ccc92a9623076df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:01.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.847+0000 7f7ec6a00640 1 -- 172.21.15.144:0/2814800973 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011310 con 0x5642d4d9f070 2024-07-26T11:08:01.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.848+0000 7f7ec6a00640 1 -- 172.21.15.144:0/2814800973 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ecc003e20 con 0x5642d4d9f070 2024-07-26T11:08:01.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.848+0000 7f7ec6a00640 1 -- 172.21.15.144:0/2814800973 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc004490 con 0x5642d4d9f070 2024-07-26T11:08:01.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.848+0000 7f7ed2000640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7ebc0b3dc0 0x7f7ebc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.848+0000 7f7ed2000640 1 -- 172.21.15.144:0/4056356878 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 10082+0+0 (secure 0 0 0) 0x7f7ec00e4890 con 0x7f7ed410af10 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.848+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2814800973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 msgr2=0x5642d51441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.848+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51441f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0117e0 tx=0x7f7ecc011ca0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ed9800640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7ebc0b3dc0 0x7f7ebc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2814800973 shutdown_connections 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d5144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2814800973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51441f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2814800973 >> 172.21.15.144:0/2814800973 conn(0x5642d4d93460 msgr2=0x5642d51420e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2814800973 shutdown_connections 2024-07-26T11:08:01.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2814800973 wait complete. 2024-07-26T11:08:01.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.849+0000 7f7ed9800640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7ebc0b3dc0 0x7f7ebc0b62d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005ab0 tx=0x7f7ec8005a20 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:01.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.850+0000 7f7ee2fd0740 1 -- start start 2024-07-26T11:08:01.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.850+0000 7f7ee2fd0740 1 Processor -- start 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.850+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d514f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.850+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d514fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.850+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51501a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5642d4f0d700 con 0x5642d4d761b0 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5642d5046690 con 0x5642d4d7e390 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5642d51529d0 con 0x5642d4d9f070 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ee2fd0740 10 client.0 _ll_register_callbacks cb 0x5642d5056140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d514f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d514f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38340/0 (socket says 172.21.15.144:38340) 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 -- 172.21.15.144:0/3880287091 learned_addr learned my addr 172.21.15.144:0/3880287091 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d514fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7eda200640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 -- 172.21.15.144:0/3880287091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 msgr2=0x5642d51501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 -- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 msgr2=0x5642d514fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d514fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.851+0000 7f7ed9800640 1 -- 172.21.15.144:0/3880287091 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642d505d830 con 0x5642d4d761b0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.852+0000 7f7ed9800640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d514f720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009900 tx=0x7f7ec8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e898467da81a597f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.852+0000 7f7ebb400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.852+0000 7f7ebb400640 1 -- 172.21.15.144:0/3880287091 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec804e070 con 0x5642d4d761b0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.852+0000 7f7ebb400640 1 -- 172.21.15.144:0/3880287091 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 10082+0+0 (secure 0 0 0) 0x7f7ec8004970 con 0x5642d4d761b0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.852+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ecc003e20 con 0x5642d4d761b0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.852+0000 7f7ebb400640 1 -- 172.21.15.144:0/3880287091 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f7ec805a050 con 0x5642d4d761b0 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 10 client.16029 fetch_fsmap learned FSMap version 81 2024-07-26T11:08:01.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x5642d515e520 con 0x5642d4d761b0 2024-07-26T11:08:01.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ebb400640 1 -- 172.21.15.144:0/3880287091 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ecc003e20 con 0x5642d4d761b0 2024-07-26T11:08:01.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ebb400640 1 -- 172.21.15.144:0/3880287091 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 81) ==== 8362+0+0 (secure 0 0 0) 0x7f7ec80324b0 con 0x5642d4d761b0 2024-07-26T11:08:01.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 10 client.16029 fetch_fsmap finished waiting for FSMap version 81 2024-07-26T11:08:01.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:08:01.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 20 client.16029 populate_metadata read hostname 'smithi144' 2024-07-26T11:08:01.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d5d9f0 0x5642d5161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 4 client.16029 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.853+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642d50bcb10 con 0x5642d4d5d9f0 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.854+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d5d9f0 0x5642d5161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.854+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d5d9f0 0x5642d5161a60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22888f715ce0f062 server_cookie=df91a0aea5557567 in_seq=0 out_seq=0 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.854+0000 7f7ebb400640 10 client.16029 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.855+0000 7f7ebb400640 1 -- 172.21.15.144:0/3880287091 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5642d50bcb10 con 0x5642d4d5d9f0 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.855+0000 7f7ebb400640 10 client.16029 handle_command_reply: tid=0 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.855+0000 7f7ee2fd0740 1 client.16029 shutdown 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 msgr2=0x5642d514f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d514f720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009900 tx=0x7f7ec8009260 comp rx=0 tx=0).stop 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 shutdown_connections 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d9f070 0x5642d51501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d5d9f0 0x5642d5161a60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d514fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ebb400640 0 client.16029 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3880287091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d761b0 0x5642d514f720 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 >> 172.21.15.144:0/3880287091 conn(0x5642d4d93460 msgr2=0x5642d5147380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 shutdown_connections 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3880287091 wait complete. 2024-07-26T11:08:01.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.856+0000 7f7ee2fd0740 20 client.16029 trim_cache size 0 max 16384 2024-07-26T11:08:01.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 Processor -- start 2024-07-26T11:08:01.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 -- start start 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5151710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5167220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d5167760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41b5830 con 0x5642d4d9f070 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41b3100 con 0x5642d4d7e390 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.875+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41b0f50 con 0x5642d4d761b0 2024-07-26T11:08:01.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:56234/0 (socket says 172.21.15.144:56234) 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 -- 172.21.15.144:0/2381133767 learned_addr learned my addr 172.21.15.144:0/2381133767 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5167220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7eda200640 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d5167760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 -- 172.21.15.144:0/2381133767 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 msgr2=0x5642d5167220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5167220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 -- 172.21.15.144:0/2381133767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 msgr2=0x5642d5167760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d5167760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 -- 172.21.15.144:0/2381133767 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642d5161df0 con 0x5642d4d761b0 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7eda200640 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d5167760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:08:01.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.876+0000 7f7ed9800640 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5151710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8032900 tx=0x7f7ec8005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=813adddd7a8e88fb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:01.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.877+0000 7f7ec6000640 1 -- 172.21.15.144:0/2381133767 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec804e070 con 0x5642d4d761b0 2024-07-26T11:08:01.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.877+0000 7f7ec6000640 1 -- 172.21.15.144:0/2381133767 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec8004770 con 0x5642d4d761b0 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.877+0000 7f7ec6000640 1 -- 172.21.15.144:0/2381133767 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8035040 con 0x5642d4d761b0 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.877+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2381133767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 msgr2=0x5642d5151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.877+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5151710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8032900 tx=0x7f7ec8005f00 comp rx=0 tx=0).stop 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2381133767 shutdown_connections 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d5167760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5167220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/2381133767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5151710 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2381133767 >> 172.21.15.144:0/2381133767 conn(0x5642d4d93460 msgr2=0x5642d5150050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2381133767 shutdown_connections 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/2381133767 wait complete. 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 -- start start 2024-07-26T11:08:01.881 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.878+0000 7f7ee2fd0740 1 Processor -- start 2024-07-26T11:08:01.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5155f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5156450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d51c9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41b15c0 con 0x5642d4d9f070 2024-07-26T11:08:01.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41b2a30 con 0x5642d4d7e390 2024-07-26T11:08:01.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41b1c90 con 0x5642d4d761b0 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ee2fd0740 10 client.0 _ll_register_callbacks cb 0x7f7ec80044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ed8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7eda200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d51c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ed8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5156450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56078/0 (socket says 172.21.15.144:56078) 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ed8e00640 1 -- 172.21.15.144:0/3765446086 learned_addr learned my addr 172.21.15.144:0/3765446086 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.879+0000 7f7ed9800640 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ed8e00640 1 -- 172.21.15.144:0/3765446086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 msgr2=0x5642d5155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ed8e00640 1 -- 172.21.15.144:0/3765446086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 msgr2=0x5642d51c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d51c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ed8e00640 1 -- 172.21.15.144:0/3765446086 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642d51c9740 con 0x5642d4d7e390 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ed8e00640 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5156450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00a8140 tx=0x7f7ec000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94b3b3d2f0fa1d1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:01.884 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ebb400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ebb400640 1 -- 172.21.15.144:0/3765446086 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec00af420 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ec8004770 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.880+0000 7f7ebb400640 1 -- 172.21.15.144:0/3765446086 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 10082+0+0 (secure 0 0 0) 0x7f7ec00af160 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ebb400640 1 -- 172.21.15.144:0/3765446086 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f7ec00b3ed0 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 10 client.26204 fetch_fsmap learned FSMap version 81 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x5642d51ca7e0 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ebb400640 1 -- 172.21.15.144:0/3765446086 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec8004770 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ebb400640 1 -- 172.21.15.144:0/3765446086 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 81) ==== 8362+0+0 (secure 0 0 0) 0x7f7ec00e6ce0 con 0x5642d4d7e390 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 10 client.26204 fetch_fsmap finished waiting for FSMap version 81 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 10 client.26204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 20 client.26204 populate_metadata read hostname 'smithi144' 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d32da0 0x5642d51ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 4 client.26204 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:08:01.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.881+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642d50b85b0 con 0x5642d4d32da0 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.882+0000 7f7ed9800640 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d32da0 0x5642d51ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.882+0000 7f7ed9800640 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d32da0 0x5642d51ccf00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=747d508e4c73f46d server_cookie=e404312f8e654241 in_seq=0 out_seq=0 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.882+0000 7f7ebb400640 10 client.26204 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.882+0000 7f7ebb400640 1 -- 172.21.15.144:0/3765446086 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5642d50b85b0 con 0x5642d4d32da0 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.882+0000 7f7ebb400640 10 client.26204 handle_command_reply: tid=0 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.882+0000 7f7ee2fd0740 1 client.26204 shutdown 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 msgr2=0x5642d5156450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5156450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00a8140 tx=0x7f7ec000cef0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 shutdown_connections 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5642d4d9f070 0x5642d51c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5642d4d32da0 0x5642d51ccf00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5642d4d7e390 0x5642d5156450 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 --2- 172.21.15.144:0/3765446086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5642d4d761b0 0x5642d5155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ebb400640 0 client.26204 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 >> 172.21.15.144:0/3765446086 conn(0x5642d4d93460 msgr2=0x5642d5166210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 shutdown_connections 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 1 -- 172.21.15.144:0/3765446086 wait complete. 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.883+0000 7f7ee2fd0740 20 client.26204 trim_cache size 0 max 16384 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:08:01.888 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:08:01.889 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:08:01.889 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:08:01.889 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:08:01.889 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:08:01.889 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:08:01.889 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7ebc0b3dc0 msgr2=0x7f7ebc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7ebc0b3dc0 0x7f7ebc0b62d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005ab0 tx=0x7f7ec8005a20 comp rx=0 tx=0).stop 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 msgr2=0x7f7ed419c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 0x7f7ed419c510 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000e9f0 tx=0x7f7ec000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 shutdown_connections 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f7ebc0b3dc0 0x7f7ebc0b62d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ed410b860 0x7f7ed41a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed410af10 0x7f7ed419c510 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 --2- 172.21.15.144:0/4056356878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ed41055d0 0x7f7ed419bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:01.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 >> 172.21.15.144:0/4056356878 conn(0x7f7ed4101040 msgr2=0x7f7ed4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:01.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 shutdown_connections 2024-07-26T11:08:01.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:01.884+0000 7f7edac00640 1 -- 172.21.15.144:0/4056356878 wait complete. 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:01 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:01 smithi183 systemd[1]: Stopped Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:01 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 1.953s CPU time. 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:01 smithi183 systemd[1]: Starting Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:01 smithi183 podman[72736]: 2024-07-26 11:08:01.770280727 +0000 UTC m=+0.017182566 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:01 smithi183 podman[72736]: 2024-07-26 11:08:01.875520714 +0000 UTC m=+0.122422584 container create 7d8116aecdf260e979942825fa54ce848a479cd2ec8b17256bdb45dafe756089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:08:02.362 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:02 smithi183 podman[72736]: 2024-07-26 11:08:02.119827657 +0000 UTC m=+0.366729485 container init 7d8116aecdf260e979942825fa54ce848a479cd2ec8b17256bdb45dafe756089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T11:08:02.362 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:02 smithi183 podman[72736]: 2024-07-26 11:08:02.122433204 +0000 UTC m=+0.369335033 container start 7d8116aecdf260e979942825fa54ce848a479cd2ec8b17256bdb45dafe756089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-07-26T11:08:02.362 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mds[72755]: -- 172.21.15.183:0/2908501125 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce963dd860 con 0x55ce9a57f180 2024-07-26T11:08:02.362 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: starting mds.i at 2024-07-26T11:08:02.362 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:02 smithi183 bash[72736]: 7d8116aecdf260e979942825fa54ce848a479cd2ec8b17256bdb45dafe756089 2024-07-26T11:08:02.362 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:02 smithi183 systemd[1]: Started Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:02.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: pgmap v635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:08:02.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: daemon mds.i restarted 2024-07-26T11:08:02.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:02.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:08:02.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T11:08:02.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:boot 2024-07-26T11:08:02.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:08:02.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:08:02.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:02 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: pgmap v635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: daemon mds.i restarted 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:boot 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:02 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: pgmap v635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: daemon mds.i restarted 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:boot 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:08:02.653 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:02 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:02.712 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:08:04.633 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:04 smithi144 ceph-mon[32129]: pgmap v637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:04.633 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:04 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:resolve 2024-07-26T11:08:04.633 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:04 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:04 smithi183 ceph-mon[37273]: pgmap v637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:04 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:resolve 2024-07-26T11:08:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:04 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:04 smithi160 ceph-mon[36715]: pgmap v637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:04 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:resolve 2024-07-26T11:08:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:04 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:05 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:reconnect 2024-07-26T11:08:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:05 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:05 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:08:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:05 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:reconnect 2024-07-26T11:08:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:05 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:05 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:08:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:05 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:reconnect 2024-07-26T11:08:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:05 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:05 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:08:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:06 smithi144 ceph-mon[32129]: pgmap v638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:06 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:rejoin 2024-07-26T11:08:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:06 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:06 smithi144 ceph-mon[32129]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:08:06.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-mon[37273]: pgmap v638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:06.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:rejoin 2024-07-26T11:08:06.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:06.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-mon[37273]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:08:06.784 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: *** Caught signal (Aborted) ** 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: in thread 7f0b94a58640 thread_name:ms_dispatch 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0b99e2f6f0] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0b99e7c94c] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 3: raise() 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 4: abort() 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0b99e1971b] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 6: /lib64/libc.so.6(+0x37386) [0x7f0b99e28386] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4ad34d6b9] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4ad50b80c] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c4ad44b2ca] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4ad41b753] 2024-07-26T11:08:06.785 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4ad41ca1b] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4ad400752] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c4ad34ea56] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c4ad352637] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c4ad352f5e] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c4ad33603a] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0b9a6fc407] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 18: (DispatchQueue::entry()+0x60f) [0x7f0b9a6fccf9] 2024-07-26T11:08:06.786 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0b9a7d95a1] 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 20: (Thread::entry_wrapper()+0x43) [0x7f0b9a586cdd] 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0b9a586cf9] 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0b99e7ac02] 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0b99effc40] 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.441+0000 7f0b94a58640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: in thread 7f0b94a58640 thread_name:ms_dispatch 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:06.787 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0b99e2f6f0] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0b99e7c94c] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 3: raise() 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 4: abort() 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0b99e1971b] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 6: /lib64/libc.so.6(+0x37386) [0x7f0b99e28386] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4ad34d6b9] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4ad50b80c] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c4ad44b2ca] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4ad41b753] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4ad41ca1b] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4ad400752] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c4ad34ea56] 2024-07-26T11:08:06.788 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c4ad352637] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c4ad352f5e] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c4ad33603a] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0b9a6fc407] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 18: (DispatchQueue::entry()+0x60f) [0x7f0b9a6fccf9] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0b9a7d95a1] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 20: (Thread::entry_wrapper()+0x43) [0x7f0b9a586cdd] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0b9a586cf9] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0b99e7ac02] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0b99effc40] 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 0> 2024-07-26T11:08:06.441+0000 7f0b94a58640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: in thread 7f0b94a58640 thread_name:ms_dispatch 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.789 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0b99e2f6f0] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0b99e7c94c] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 3: raise() 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 4: abort() 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0b99e1971b] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 6: /lib64/libc.so.6(+0x37386) [0x7f0b99e28386] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4ad34d6b9] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4ad50b80c] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c4ad44b2ca] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4ad41b753] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4ad41ca1b] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4ad400752] 2024-07-26T11:08:06.790 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c4ad34ea56] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c4ad352637] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c4ad352f5e] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c4ad33603a] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0b9a6fc407] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 18: (DispatchQueue::entry()+0x60f) [0x7f0b9a6fccf9] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0b9a7d95a1] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 20: (Thread::entry_wrapper()+0x43) [0x7f0b9a586cdd] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0b9a586cf9] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0b99e7ac02] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0b99effc40] 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.791 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 0> 2024-07-26T11:08:06.441+0000 7f0b94a58640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: in thread 7f0b94a58640 thread_name:ms_dispatch 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0b99e2f6f0] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0b99e7c94c] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 3: raise() 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 4: abort() 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0b99e1971b] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 6: /lib64/libc.so.6(+0x37386) [0x7f0b99e28386] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4ad34d6b9] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4ad50b80c] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c4ad44b2ca] 2024-07-26T11:08:06.792 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4ad41b753] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4ad41ca1b] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4ad400752] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c4ad34ea56] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c4ad352637] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c4ad352f5e] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c4ad33603a] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0b9a6fc407] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 18: (DispatchQueue::entry()+0x60f) [0x7f0b9a6fccf9] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0b9a7d95a1] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 20: (Thread::entry_wrapper()+0x43) [0x7f0b9a586cdd] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0b9a586cf9] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0b99e7ac02] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0b99effc40] 2024-07-26T11:08:06.793 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:06.794 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72593]: 2024-07-26T11:08:06.794 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:06 smithi183 podman[72789]: 2024-07-26 11:08:06.501463678 +0000 UTC m=+0.018445045 container died a1c91b4c8c14c309f8b09b400175d3205271eaf49e5f10b41b771c2d47ae9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T11:08:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:06 smithi160 ceph-mon[36715]: pgmap v638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:06 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:rejoin 2024-07-26T11:08:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:06 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:06.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:06 smithi160 ceph-mon[36715]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:08:07.423 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:08:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:08:07.159+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:08:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:07 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:07 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:08:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:07 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:active 2024-07-26T11:08:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:07 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:07 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:08:07.635 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:07 smithi183 podman[72789]: 2024-07-26 11:08:07.209334708 +0000 UTC m=+0.726316063 container remove a1c91b4c8c14c309f8b09b400175d3205271eaf49e5f10b41b771c2d47ae9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:08:07.636 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:07 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:08:07.636 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:07 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:08:07.636 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:07 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 2.072s CPU time. 2024-07-26T11:08:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:07 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:07 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:08:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:07 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:active 2024-07-26T11:08:07.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:07 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:07.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:07 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:08:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:07 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:07 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:08:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:07 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3213465258,v1:172.21.15.183:6837/3213465258] up:active 2024-07-26T11:08:07.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:07 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:07.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:07 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:08:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:08 smithi144 ceph-mon[32129]: pgmap v639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 6 op/s 2024-07-26T11:08:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:08 smithi160 ceph-mon[36715]: pgmap v639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 6 op/s 2024-07-26T11:08:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:08 smithi183 ceph-mon[37273]: pgmap v639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 6 op/s 2024-07-26T11:08:10.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:10 smithi144 ceph-mon[32129]: pgmap v640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:08:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:10 smithi160 ceph-mon[36715]: pgmap v640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:08:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:10 smithi183 ceph-mon[37273]: pgmap v640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:08:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:11.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:11.672+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:11.673+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:11.673+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:11.673+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:12 smithi144 ceph-mon[32129]: pgmap v641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:12 smithi160 ceph-mon[36715]: pgmap v641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:12 smithi183 ceph-mon[37273]: pgmap v641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:14 smithi144 ceph-mon[32129]: pgmap v642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 183 B/s wr, 7 op/s 2024-07-26T11:08:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:14 smithi160 ceph-mon[36715]: pgmap v642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 183 B/s wr, 7 op/s 2024-07-26T11:08:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:14 smithi183 ceph-mon[37273]: pgmap v642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 183 B/s wr, 7 op/s 2024-07-26T11:08:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:16 smithi144 ceph-mon[32129]: pgmap v643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:08:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:16 smithi160 ceph-mon[36715]: pgmap v643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:08:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:16 smithi183 ceph-mon[37273]: pgmap v643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:08:17.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:17 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:08:17.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:17 smithi183 systemd[1]: Stopped Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:17.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:17 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 2.072s CPU time. 2024-07-26T11:08:17.894 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:17 smithi183 systemd[1]: Starting Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:08:17.895 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:17 smithi183 podman[72879]: 2024-07-26 11:08:17.771256196 +0000 UTC m=+0.017839798 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:08:18.156 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:17 smithi183 podman[72879]: 2024-07-26 11:08:17.893341771 +0000 UTC m=+0.139925419 container create ac43f6fd2cc2445a3a420f3e3e6f5d5fe653e605394bf652a5070f0f4dbd25f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724) 2024-07-26T11:08:18.156 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:18 smithi183 podman[72879]: 2024-07-26 11:08:18.121141866 +0000 UTC m=+0.367725469 container init ac43f6fd2cc2445a3a420f3e3e6f5d5fe653e605394bf652a5070f0f4dbd25f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:08:18.156 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:18 smithi183 podman[72879]: 2024-07-26 11:08:18.123787889 +0000 UTC m=+0.370371502 container start ac43f6fd2cc2445a3a420f3e3e6f5d5fe653e605394bf652a5070f0f4dbd25f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:08:18.464 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mds[72899]: -- 172.21.15.183:0/1394810357 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562995355860 con 0x5629994f7180 2024-07-26T11:08:18.464 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: starting mds.c at 2024-07-26T11:08:18.464 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:18 smithi183 bash[72879]: ac43f6fd2cc2445a3a420f3e3e6f5d5fe653e605394bf652a5070f0f4dbd25f6 2024-07-26T11:08:18.464 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:18 smithi183 systemd[1]: Started Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: pgmap v644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: daemon mds.c restarted 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:boot 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:08:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:18 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: pgmap v644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: daemon mds.c restarted 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:boot 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:08:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:18 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: pgmap v644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: daemon mds.c restarted 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:boot 2024-07-26T11:08:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-07-26T11:08:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-07-26T11:08:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:18 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:replay,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:20.733 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:20 smithi144 ceph-mon[32129]: pgmap v646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:08:20.734 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:20 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:resolve 2024-07-26T11:08:20.734 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:20 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:20 smithi160 ceph-mon[36715]: pgmap v646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:08:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:20 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:resolve 2024-07-26T11:08:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:20 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:20 smithi183 ceph-mon[37273]: pgmap v646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:08:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:20 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:resolve 2024-07-26T11:08:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:20 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:resolve,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:21 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:reconnect 2024-07-26T11:08:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:21 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:21 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:08:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:21 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:reconnect 2024-07-26T11:08:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:21 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:21 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:08:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:21 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:reconnect 2024-07-26T11:08:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:21 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:reconnect,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:21 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:08:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:22 smithi160 ceph-mon[36715]: pgmap v647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:08:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:22 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:rejoin 2024-07-26T11:08:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:22 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:22 smithi160 ceph-mon[36715]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:08:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-mon[37273]: pgmap v647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:08:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:rejoin 2024-07-26T11:08:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-mon[37273]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: *** Caught signal (Aborted) ** 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: in thread 7f974bf81640 thread_name:ms_dispatch 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f97513586f0] 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f97513a594c] 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 3: raise() 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 4: abort() 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 5: /lib64/libc.so.6(+0x2871b) [0x7f975134271b] 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 6: /lib64/libc.so.6(+0x37386) [0x7f9751351386] 2024-07-26T11:08:22.886 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ce93fcd6b9] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ce9418b80c] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ce940cb2ca] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ce9409b753] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ce9409ca1b] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ce94080752] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ce93fcea56] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ce93fd2637] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ce93fd2f5e] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ce93fb603a] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9751c25407] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 18: (DispatchQueue::entry()+0x60f) [0x7f9751c25cf9] 2024-07-26T11:08:22.887 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9751d025a1] 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 20: (Thread::entry_wrapper()+0x43) [0x7f9751aafcdd] 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9751aafcf9] 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 22: /lib64/libc.so.6(+0x89c02) [0x7f97513a3c02] 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9751428c40] 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.546+0000 7f974bf81640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: in thread 7f974bf81640 thread_name:ms_dispatch 2024-07-26T11:08:22.888 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f97513586f0] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f97513a594c] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 3: raise() 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 4: abort() 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 5: /lib64/libc.so.6(+0x2871b) [0x7f975134271b] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 6: /lib64/libc.so.6(+0x37386) [0x7f9751351386] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ce93fcd6b9] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ce9418b80c] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ce940cb2ca] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ce9409b753] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ce9409ca1b] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ce94080752] 2024-07-26T11:08:22.889 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ce93fcea56] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ce93fd2637] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ce93fd2f5e] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ce93fb603a] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9751c25407] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 18: (DispatchQueue::entry()+0x60f) [0x7f9751c25cf9] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9751d025a1] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 20: (Thread::entry_wrapper()+0x43) [0x7f9751aafcdd] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9751aafcf9] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 22: /lib64/libc.so.6(+0x89c02) [0x7f97513a3c02] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9751428c40] 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.890 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 0> 2024-07-26T11:08:22.546+0000 7f974bf81640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: in thread 7f974bf81640 thread_name:ms_dispatch 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f97513586f0] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f97513a594c] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 3: raise() 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 4: abort() 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 5: /lib64/libc.so.6(+0x2871b) [0x7f975134271b] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 6: /lib64/libc.so.6(+0x37386) [0x7f9751351386] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ce93fcd6b9] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ce9418b80c] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ce940cb2ca] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ce9409b753] 2024-07-26T11:08:22.891 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ce9409ca1b] 2024-07-26T11:08:22.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ce94080752] 2024-07-26T11:08:22.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ce93fcea56] 2024-07-26T11:08:22.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ce93fd2637] 2024-07-26T11:08:22.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ce93fd2f5e] 2024-07-26T11:08:22.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ce93fb603a] 2024-07-26T11:08:22.892 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9751c25407] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 18: (DispatchQueue::entry()+0x60f) [0x7f9751c25cf9] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9751d025a1] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 20: (Thread::entry_wrapper()+0x43) [0x7f9751aafcdd] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9751aafcf9] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 22: /lib64/libc.so.6(+0x89c02) [0x7f97513a3c02] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9751428c40] 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 0> 2024-07-26T11:08:22.546+0000 7f974bf81640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: in thread 7f974bf81640 thread_name:ms_dispatch 2024-07-26T11:08:22.893 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f97513586f0] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f97513a594c] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 3: raise() 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 4: abort() 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 5: /lib64/libc.so.6(+0x2871b) [0x7f975134271b] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 6: /lib64/libc.so.6(+0x37386) [0x7f9751351386] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ce93fcd6b9] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ce9418b80c] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ce940cb2ca] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ce9409b753] 2024-07-26T11:08:22.894 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ce9409ca1b] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ce94080752] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ce93fcea56] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ce93fd2637] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ce93fd2f5e] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ce93fb603a] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9751c25407] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 18: (DispatchQueue::entry()+0x60f) [0x7f9751c25cf9] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9751d025a1] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 20: (Thread::entry_wrapper()+0x43) [0x7f9751aafcdd] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9751aafcf9] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 22: /lib64/libc.so.6(+0x89c02) [0x7f97513a3c02] 2024-07-26T11:08:22.895 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9751428c40] 2024-07-26T11:08:22.896 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:22.896 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i[72751]: 2024-07-26T11:08:22.896 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:22 smithi183 podman[72933]: 2024-07-26 11:08:22.619152309 +0000 UTC m=+0.028610144 container died 7d8116aecdf260e979942825fa54ce848a479cd2ec8b17256bdb45dafe756089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux ) 2024-07-26T11:08:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:22 smithi144 ceph-mon[32129]: pgmap v647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:08:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:22 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:rejoin 2024-07-26T11:08:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:22 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=i=up:rejoin,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:22 smithi144 ceph-mon[32129]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:23 smithi183 podman[72933]: 2024-07-26 11:08:23.451578169 +0000 UTC m=+0.861036031 container remove 7d8116aecdf260e979942825fa54ce848a479cd2ec8b17256bdb45dafe756089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-i, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:23 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:23 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:23 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:23 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:active 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:23 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:23.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:23 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:08:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:23 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:23 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:08:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:23 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:active 2024-07-26T11:08:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:23 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:23 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:08:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:23 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:23 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:08:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:23 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6838/2371226522,v1:172.21.15.183:6839/2371226522] up:active 2024-07-26T11:08:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:23 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 7 up:standby 2024-07-26T11:08:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:23 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-07-26T11:08:24.135 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:23 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:08:24.135 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:23 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 1.922s CPU time. 2024-07-26T11:08:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:24 smithi160 ceph-mon[36715]: pgmap v648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:08:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:24 smithi183 ceph-mon[37273]: pgmap v648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:08:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:24 smithi144 ceph-mon[32129]: pgmap v648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:08:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:26.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:26.689+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:26.689+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:26.689+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:26.689+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:26 smithi160 ceph-mon[36715]: pgmap v649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:26 smithi183 ceph-mon[37273]: pgmap v649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:26 smithi144 ceph-mon[32129]: pgmap v649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:28 smithi160 ceph-mon[36715]: pgmap v650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:28 smithi183 ceph-mon[37273]: pgmap v650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:28 smithi144 ceph-mon[32129]: pgmap v650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:30 smithi160 ceph-mon[36715]: pgmap v651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 183 B/s wr, 8 op/s 2024-07-26T11:08:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:30 smithi183 ceph-mon[37273]: pgmap v651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 183 B/s wr, 8 op/s 2024-07-26T11:08:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:30 smithi144 ceph-mon[32129]: pgmap v651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 183 B/s wr, 8 op/s 2024-07-26T11:08:32.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:08:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:32 smithi160 ceph-mon[36715]: pgmap v652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:08:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:32 smithi183 ceph-mon[37273]: pgmap v652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:08:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:32 smithi144 ceph-mon[32129]: pgmap v652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:08:33.179 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:08:34.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:33 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:08:34.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:33 smithi183 systemd[1]: Stopped Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:34.385 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:33 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Consumed 1.922s CPU time. 2024-07-26T11:08:34.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:33 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Start request repeated too quickly. 2024-07-26T11:08:34.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:33 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:08:34.386 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 26 11:08:33 smithi183 systemd[1]: Failed to start Ceph mds.i for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:34 smithi160 ceph-mon[36715]: pgmap v653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:34 smithi183 ceph-mon[37273]: pgmap v653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:34.908 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:34 smithi144 ceph-mon[32129]: pgmap v653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 -- 172.21.15.144:0/443279790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c1099f0 msgr2=0x7f783c109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 --2- 172.21.15.144:0/443279790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c1099f0 0x7f783c109e00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7830009ca0 tx=0x7f783002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 -- 172.21.15.144:0/443279790 shutdown_connections 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 --2- 172.21.15.144:0/443279790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783c10fc80 0x7f783c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 --2- 172.21.15.144:0/443279790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783c10f330 0x7f783c10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 --2- 172.21.15.144:0/443279790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c1099f0 0x7f783c109e00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.473 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.470+0000 7f7842a00640 1 -- 172.21.15.144:0/443279790 >> 172.21.15.144:0/443279790 conn(0x7f783c078ff0 msgr2=0x7f783c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:35.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.471+0000 7f7842a00640 1 -- 172.21.15.144:0/443279790 shutdown_connections 2024-07-26T11:08:35.474 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.471+0000 7f7842a00640 1 -- 172.21.15.144:0/443279790 wait complete. 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 Processor -- start 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 -- start start 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783c1099f0 0x7f783c1a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 0x7f783c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783c10fc80 0x7f783c1a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f783c119940 con 0x7f783c10f330 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1197c0 con 0x7f783c1099f0 2024-07-26T11:08:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.472+0000 7f7842a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f783c119ac0 con 0x7f783c10fc80 2024-07-26T11:08:35.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 0x7f783c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 0x7f783c1a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42258/0 (socket says 172.21.15.144:42258) 2024-07-26T11:08:35.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 -- 172.21.15.144:0/2745945902 learned_addr learned my addr 172.21.15.144:0/2745945902 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:35.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7841600640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783c1099f0 0x7f783c1a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7842000640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783c10fc80 0x7f783c1a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 -- 172.21.15.144:0/2745945902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783c10fc80 msgr2=0x7f783c1a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783c10fc80 0x7f783c1a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 -- 172.21.15.144:0/2745945902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783c1099f0 msgr2=0x7f783c1a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783c1099f0 0x7f783c1a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.473+0000 7f7840c00640 1 -- 172.21.15.144:0/2745945902 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c1a62c0 con 0x7f783c10f330 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.474+0000 7f7840c00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 0x7f783c1a08f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f782800e9c0 tx=0x7f782800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.474+0000 7f783a000640 1 -- 172.21.15.144:0/2745945902 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782800ccc0 con 0x7f783c10f330 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.474+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783c1a65b0 con 0x7f783c10f330 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.474+0000 7f783a000640 1 -- 172.21.15.144:0/2745945902 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78280054d0 con 0x7f783c10f330 2024-07-26T11:08:35.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.474+0000 7f783a000640 1 -- 172.21.15.144:0/2745945902 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828010430 con 0x7f783c10f330 2024-07-26T11:08:35.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.474+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783c1acd10 con 0x7f783c10f330 2024-07-26T11:08:35.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.477+0000 7f784ae31740 1 Processor -- start 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 -- start start 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f783a000640 1 -- 172.21.15.144:0/2745945902 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7828004f40 con 0x7f783c10f330 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86affd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86b0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564de8626b90 con 0x564de82e21b0 2024-07-26T11:08:35.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564de8616090 con 0x564de82ea390 2024-07-26T11:08:35.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.478+0000 7f784ae31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564de861a0d0 con 0x564de830b070 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7841600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86affd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:46370/0 (socket says 172.21.15.144:46370) 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 -- 172.21.15.144:0/4020892720 learned_addr learned my addr 172.21.15.144:0/4020892720 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7840c00640 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 -- 172.21.15.144:0/4020892720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 msgr2=0x564de86b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 -- 172.21.15.144:0/4020892720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 msgr2=0x564de86b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 -- 172.21.15.144:0/4020892720 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564de8376650 con 0x564de830b070 2024-07-26T11:08:35.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7840c00640 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:08:35.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.479+0000 7f7842000640 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86affd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7834009c90 tx=0x7f7834011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33c20b7441f4a591 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:35.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.480+0000 7f782e600640 1 -- 172.21.15.144:0/4020892720 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834004230 con 0x564de830b070 2024-07-26T11:08:35.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.480+0000 7f783a000640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f78240b3dc0 0x7f78240b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.480+0000 7f782e600640 1 -- 172.21.15.144:0/4020892720 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7834009e40 con 0x564de830b070 2024-07-26T11:08:35.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.480+0000 7f782e600640 1 -- 172.21.15.144:0/4020892720 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834019710 con 0x564de830b070 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.480+0000 7f783a000640 1 -- 172.21.15.144:0/2745945902 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 10254+0+0 (secure 0 0 0) 0x7f78280e4af0 con 0x7f783c10f330 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.480+0000 7f7841600640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f78240b3dc0 0x7f78240b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 -- 172.21.15.144:0/4020892720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 msgr2=0x564de86affd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86affd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7834009c90 tx=0x7f7834011810 comp rx=0 tx=0).stop 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 -- 172.21.15.144:0/4020892720 shutdown_connections 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 --2- 172.21.15.144:0/4020892720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86affd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 -- 172.21.15.144:0/4020892720 >> 172.21.15.144:0/4020892720 conn(0x564de82ff460 msgr2=0x564de86adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f7841600640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f78240b3dc0 0x7f78240b62d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x564de86aec50 tx=0x7f7830009360 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 -- 172.21.15.144:0/4020892720 shutdown_connections 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.481+0000 7f784ae31740 1 -- 172.21.15.144:0/4020892720 wait complete. 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 -- start start 2024-07-26T11:08:35.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 Processor -- start 2024-07-26T11:08:35.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82e21b0 0x564de86bb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 0x564de86bba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86bbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564de8479700 con 0x564de82ea390 2024-07-26T11:08:35.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.482+0000 7f784ae31740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564de85b2660 con 0x564de82e21b0 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f784ae31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564de86be7b0 con 0x564de830b070 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f784ae31740 10 client.0 _ll_register_callbacks cb 0x564de85c2110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 0x564de86bba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 0x564de86bba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42272/0 (socket says 172.21.15.144:42272) 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 -- 172.21.15.144:0/4118930049 learned_addr learned my addr 172.21.15.144:0/4118930049 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7841600640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82e21b0 0x564de86bb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7842000640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86bbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 -- 172.21.15.144:0/4118930049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 msgr2=0x564de86bbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86bbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 -- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82e21b0 msgr2=0x564de86bb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82e21b0 0x564de86bb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.483+0000 7f7840c00640 1 -- 172.21.15.144:0/4118930049 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564de85c9800 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f7840c00640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 0x564de86bba50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f782800c6a0 tx=0x7f7828005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2fb22a61f602df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f7822a00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f7822a00640 1 -- 172.21.15.144:0/4118930049 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280aa750 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f7822a00640 1 -- 172.21.15.144:0/4118930049 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 10254+0+0 (secure 0 0 0) 0x7f78280b3e10 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564de8628ae0 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f7822a00640 1 -- 172.21.15.144:0/4118930049 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f78280aaaf0 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f784ae31740 10 client.16044 fetch_fsmap learned FSMap version 95 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.484+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x564de86ca280 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f7822a00640 1 -- 172.21.15.144:0/4118930049 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564de8628ae0 con 0x564de82ea390 2024-07-26T11:08:35.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f7822a00640 1 -- 172.21.15.144:0/4118930049 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 95) ==== 8358+0+0 (secure 0 0 0) 0x7f78280a6950 con 0x564de82ea390 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f784ae31740 10 client.16044 fetch_fsmap finished waiting for FSMap version 95 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f784ae31740 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f784ae31740 20 client.16044 populate_metadata read hostname 'smithi144' 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f784ae31740 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de82c99f0 0x564de86cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f784ae31740 4 client.16044 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564de8624580 con 0x564de82c99f0 2024-07-26T11:08:35.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.485+0000 7f7841600640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de82c99f0 0x564de86cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.486+0000 7f7841600640 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de82c99f0 0x564de86cd7c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ff5e3ffd0f5c23 server_cookie=67083b7efed4e1a4 in_seq=0 out_seq=0 2024-07-26T11:08:35.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.486+0000 7f7822a00640 10 client.16044 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:35.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f7822a00640 1 -- 172.21.15.144:0/4118930049 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564de8624580 con 0x564de82c99f0 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f7822a00640 10 client.16044 handle_command_reply: tid=0 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 client.16044 shutdown 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 msgr2=0x564de86bba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 0x564de86bba50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f782800c6a0 tx=0x7f7828005750 comp rx=0 tx=0).stop 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 shutdown_connections 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86bbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82ea390 0x564de86bba50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de82c99f0 0x564de86cd7c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 --2- 172.21.15.144:0/4118930049 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82e21b0 0x564de86bb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 >> 172.21.15.144:0/4118930049 conn(0x564de82ff460 msgr2=0x564de86b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f7822a00640 0 client.16044 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 shutdown_connections 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.487+0000 7f784ae31740 1 -- 172.21.15.144:0/4118930049 wait complete. 2024-07-26T11:08:35.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.488+0000 7f784ae31740 20 client.16044 trim_cache size 0 max 16384 2024-07-26T11:08:35.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.506+0000 7f784ae31740 1 Processor -- start 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.506+0000 7f784ae31740 1 -- start start 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f784ae31740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86bd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f784ae31740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f784ae31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86c7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f784ae31740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b9d50 con 0x564de82e21b0 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f784ae31740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b7620 con 0x564de82ea390 2024-07-26T11:08:35.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f784ae31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b5470 con 0x564de830b070 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f7841600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86bd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f7841600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86bd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42276/0 (socket says 172.21.15.144:42276) 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f7841600640 1 -- 172.21.15.144:0/1612819735 learned_addr learned my addr 172.21.15.144:0/1612819735 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f7840c00640 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.507+0000 7f7842000640 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7840c00640 1 -- 172.21.15.144:0/1612819735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 msgr2=0x564de86c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7840c00640 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86c7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7840c00640 1 -- 172.21.15.144:0/1612819735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 msgr2=0x564de86bd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7840c00640 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86bd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7840c00640 1 -- 172.21.15.144:0/1612819735 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564de86bc7d0 con 0x564de82ea390 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7841600640 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86bd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f7840c00640 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86c6fc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f78280047e0 tx=0x7f78280e4d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b72087ba981c0a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f782dc00640 1 -- 172.21.15.144:0/1612819735 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280aa750 con 0x564de82ea390 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f782dc00640 1 -- 172.21.15.144:0/1612819735 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78280ab6e0 con 0x564de82ea390 2024-07-26T11:08:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.508+0000 7f782dc00640 1 -- 172.21.15.144:0/1612819735 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280aadb0 con 0x564de82ea390 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 -- 172.21.15.144:0/1612819735 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 msgr2=0x564de86c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86c6fc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f78280047e0 tx=0x7f78280e4d70 comp rx=0 tx=0).stop 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 -- 172.21.15.144:0/1612819735 shutdown_connections 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de830b070 0x564de86c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de82ea390 0x564de86c6fc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 --2- 172.21.15.144:0/1612819735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86bd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.509+0000 7f784ae31740 1 -- 172.21.15.144:0/1612819735 >> 172.21.15.144:0/1612819735 conn(0x564de82ff460 msgr2=0x564de86c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.510+0000 7f784ae31740 1 -- 172.21.15.144:0/1612819735 shutdown_connections 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.510+0000 7f784ae31740 1 -- 172.21.15.144:0/1612819735 wait complete. 2024-07-26T11:08:35.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.510+0000 7f784ae31740 1 -- start start 2024-07-26T11:08:35.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.510+0000 7f784ae31740 1 Processor -- start 2024-07-26T11:08:35.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de82ea390 0x564de86c1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 0x564de86c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b5ae0 con 0x564de82e21b0 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b6f50 con 0x564de830b070 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b61b0 con 0x564de82ea390 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f784ae31740 10 client.0 _ll_register_callbacks cb 0x7f7828005680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f7841600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f7842000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 0x564de86c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f7842000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 0x564de86c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:47064/0 (socket says 172.21.15.144:47064) 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f7842000640 1 -- 172.21.15.144:0/1716563651 learned_addr learned my addr 172.21.15.144:0/1716563651 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:08:35.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.511+0000 7f7840c00640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de82ea390 0x564de86c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7841600640 1 -- 172.21.15.144:0/1716563651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de82ea390 msgr2=0x564de86c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7841600640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de82ea390 0x564de86c1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7841600640 1 -- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 msgr2=0x564de86c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7841600640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 0x564de86c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7841600640 1 -- 172.21.15.144:0/1716563651 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564de86c1f20 con 0x564de82e21b0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7842000640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 0x564de86c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7841600640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86c3f00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7830032bb0 tx=0x7f7830041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7ee30480dcb07f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.512+0000 7f7822a00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.513+0000 7f7822a00640 1 -- 172.21.15.144:0/1716563651 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300318e0 con 0x564de82e21b0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.513+0000 7f7822a00640 1 -- 172.21.15.144:0/1716563651 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 10254+0+0 (secure 0 0 0) 0x7f7830003970 con 0x564de82e21b0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.513+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78280ab6e0 con 0x564de82e21b0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.513+0000 7f7822a00640 1 -- 172.21.15.144:0/1716563651 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7830063050 con 0x564de82e21b0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.513+0000 7f784ae31740 10 client.16050 fetch_fsmap learned FSMap version 95 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.513+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x564de8735d30 con 0x564de82e21b0 2024-07-26T11:08:35.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f7822a00640 1 -- 172.21.15.144:0/1716563651 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78280ab6e0 con 0x564de82e21b0 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f7822a00640 1 -- 172.21.15.144:0/1716563651 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 95) ==== 8358+0+0 (secure 0 0 0) 0x7f783002f660 con 0x564de82e21b0 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f784ae31740 10 client.16050 fetch_fsmap finished waiting for FSMap version 95 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f784ae31740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f784ae31740 20 client.16050 populate_metadata read hostname 'smithi144' 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f784ae31740 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de829eda0 0x564de8738520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f784ae31740 4 client.16050 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564de8621cc0 con 0x564de829eda0 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.514+0000 7f7840c00640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de829eda0 0x564de8738520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f7840c00640 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de829eda0 0x564de8738520 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e06e1879366de4d server_cookie=561f84db061b613 in_seq=0 out_seq=0 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f7822a00640 10 client.16050 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:35.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f7822a00640 1 -- 172.21.15.144:0/1716563651 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564de8621cc0 con 0x564de829eda0 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f7822a00640 10 client.16050 handle_command_reply: tid=0 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f784ae31740 1 client.16050 shutdown 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 msgr2=0x564de86c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.515+0000 7f784ae31740 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86c3f00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7830032bb0 tx=0x7f7830041000 comp rx=0 tx=0).stop 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 shutdown_connections 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564de829eda0 0x564de8738520 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564de830b070 0x564de86c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564de82ea390 0x564de86c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 --2- 172.21.15.144:0/1716563651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564de82e21b0 0x564de86c3f00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f7822a00640 0 client.16050 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 >> 172.21.15.144:0/1716563651 conn(0x564de82ff460 msgr2=0x564de86bd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 shutdown_connections 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 1 -- 172.21.15.144:0/1716563651 wait complete. 2024-07-26T11:08:35.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.516+0000 7f784ae31740 20 client.16050 trim_cache size 0 max 16384 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:08:35.520 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:08:35.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:08:35.521 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:08:35.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f78240b3dc0 msgr2=0x7f78240b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f78240b3dc0 0x7f78240b62d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x564de86aec50 tx=0x7f7830009360 comp rx=0 tx=0).stop 2024-07-26T11:08:35.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 msgr2=0x7f783c1a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:35.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 0x7f783c1a08f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f782800e9c0 tx=0x7f782800edc0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 shutdown_connections 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f78240b3dc0 0x7f78240b62d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783c10fc80 0x7f783c1a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f783c10f330 0x7f783c1a08f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 --2- 172.21.15.144:0/2745945902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783c1099f0 0x7f783c1a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 >> 172.21.15.144:0/2745945902 conn(0x7f783c078ff0 msgr2=0x7f783c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 shutdown_connections 2024-07-26T11:08:35.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:08:35.517+0000 7f7842a00640 1 -- 172.21.15.144:0/2745945902 wait complete. 2024-07-26T11:08:36.399 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:08:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:36 smithi144 ceph-mon[32129]: pgmap v654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:36 smithi183 ceph-mon[37273]: pgmap v654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:36 smithi160 ceph-mon[36715]: pgmap v654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:38 smithi183 ceph-mon[37273]: pgmap v655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-07-26T11:08:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:38 smithi144 ceph-mon[32129]: pgmap v655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-07-26T11:08:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:38 smithi160 ceph-mon[36715]: pgmap v655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-07-26T11:08:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:40.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: pgmap v656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: Replacing daemon mds.i as rank 1 with standby daemon mds.c 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T11:08:41.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:40 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: pgmap v656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: Replacing daemon mds.i as rank 1 with standby daemon mds.c 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T11:08:41.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:40 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: pgmap v656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: Replacing daemon mds.i as rank 1 with standby daemon mds.c 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T11:08:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:40 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:41.694+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:41.695+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:41.695+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:41.695+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:41.695+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:43.059 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:42 smithi144 ceph-mon[32129]: pgmap v658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:08:43.059 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:42 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:resolve 2024-07-26T11:08:43.059 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:42 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:42 smithi160 ceph-mon[36715]: pgmap v658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:08:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:42 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:resolve 2024-07-26T11:08:43.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:42 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:42 smithi183 ceph-mon[37273]: pgmap v658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:08:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:42 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:resolve 2024-07-26T11:08:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:42 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:43 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:reconnect 2024-07-26T11:08:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:43 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:43 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00799978 2024-07-26T11:08:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:43 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:reconnect 2024-07-26T11:08:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:43 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:44.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:43 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00799978 2024-07-26T11:08:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:43 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:reconnect 2024-07-26T11:08:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:43 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:43 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00799978 2024-07-26T11:08:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-mon[37273]: pgmap v659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:08:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:rejoin 2024-07-26T11:08:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-mon[37273]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: *** Caught signal (Aborted) ** 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: in thread 7f48878e2640 thread_name:ms_dispatch 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f488ccb96f0] 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f488cd0694c] 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 3: raise() 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 4: abort() 2024-07-26T11:08:45.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 5: /lib64/libc.so.6(+0x2871b) [0x7f488cca371b] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 6: /lib64/libc.so.6(+0x37386) [0x7f488ccb2386] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56299255e6b9] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56299271c80c] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56299265c2ca] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56299262c753] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56299262da1b] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562992611752] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56299255fa56] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562992563637] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562992563f5e] 2024-07-26T11:08:45.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56299254703a] 2024-07-26T11:08:45.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f488d586407] 2024-07-26T11:08:45.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 18: (DispatchQueue::entry()+0x60f) [0x7f488d586cf9] 2024-07-26T11:08:45.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f488d6635a1] 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 20: (Thread::entry_wrapper()+0x43) [0x7f488d410cdd] 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 21: (Thread::_entry_func(void*)+0xd) [0x7f488d410cf9] 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 22: /lib64/libc.so.6(+0x89c02) [0x7f488cd04c02] 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f488cd89c40] 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:44.859+0000 7f48878e2640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: in thread 7f48878e2640 thread_name:ms_dispatch 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:45.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f488ccb96f0] 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f488cd0694c] 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 3: raise() 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 4: abort() 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 5: /lib64/libc.so.6(+0x2871b) [0x7f488cca371b] 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 6: /lib64/libc.so.6(+0x37386) [0x7f488ccb2386] 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56299255e6b9] 2024-07-26T11:08:45.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56299271c80c] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56299265c2ca] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56299262c753] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56299262da1b] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562992611752] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56299255fa56] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562992563637] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562992563f5e] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56299254703a] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f488d586407] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 18: (DispatchQueue::entry()+0x60f) [0x7f488d586cf9] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f488d6635a1] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 20: (Thread::entry_wrapper()+0x43) [0x7f488d410cdd] 2024-07-26T11:08:45.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 21: (Thread::_entry_func(void*)+0xd) [0x7f488d410cf9] 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 22: /lib64/libc.so.6(+0x89c02) [0x7f488cd04c02] 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f488cd89c40] 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 0> 2024-07-26T11:08:44.859+0000 7f48878e2640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: in thread 7f48878e2640 thread_name:ms_dispatch 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f488ccb96f0] 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f488cd0694c] 2024-07-26T11:08:45.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 3: raise() 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 4: abort() 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 5: /lib64/libc.so.6(+0x2871b) [0x7f488cca371b] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 6: /lib64/libc.so.6(+0x37386) [0x7f488ccb2386] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56299255e6b9] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56299271c80c] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56299265c2ca] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56299262c753] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56299262da1b] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562992611752] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56299255fa56] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562992563637] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562992563f5e] 2024-07-26T11:08:45.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56299254703a] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f488d586407] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 18: (DispatchQueue::entry()+0x60f) [0x7f488d586cf9] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f488d6635a1] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 20: (Thread::entry_wrapper()+0x43) [0x7f488d410cdd] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 21: (Thread::_entry_func(void*)+0xd) [0x7f488d410cf9] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 22: /lib64/libc.so.6(+0x89c02) [0x7f488cd04c02] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f488cd89c40] 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 0> 2024-07-26T11:08:44.859+0000 7f48878e2640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: in thread 7f48878e2640 thread_name:ms_dispatch 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:45.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f488ccb96f0] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f488cd0694c] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 3: raise() 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 4: abort() 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 5: /lib64/libc.so.6(+0x2871b) [0x7f488cca371b] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 6: /lib64/libc.so.6(+0x37386) [0x7f488ccb2386] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56299255e6b9] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56299271c80c] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56299265c2ca] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56299262c753] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56299262da1b] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562992611752] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56299255fa56] 2024-07-26T11:08:45.145 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562992563637] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562992563f5e] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56299254703a] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f488d586407] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 18: (DispatchQueue::entry()+0x60f) [0x7f488d586cf9] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f488d6635a1] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 20: (Thread::entry_wrapper()+0x43) [0x7f488d410cdd] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 21: (Thread::_entry_func(void*)+0xd) [0x7f488d410cf9] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 22: /lib64/libc.so.6(+0x89c02) [0x7f488cd04c02] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f488cd89c40] 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[72895]: 2024-07-26T11:08:45.146 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:44 smithi183 podman[73120]: 2024-07-26 11:08:44.92001995 +0000 UTC m=+0.018595495 container died ac43f6fd2cc2445a3a420f3e3e6f5d5fe653e605394bf652a5070f0f4dbd25f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T11:08:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:44 smithi144 ceph-mon[32129]: pgmap v659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:08:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:44 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:rejoin 2024-07-26T11:08:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:44 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:44 smithi144 ceph-mon[32129]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:08:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:44 smithi160 ceph-mon[36715]: pgmap v659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:08:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:44 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:rejoin 2024-07-26T11:08:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:44 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:44 smithi160 ceph-mon[36715]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:08:45.822 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:45 smithi183 podman[73120]: 2024-07-26 11:08:45.530949718 +0000 UTC m=+0.629525255 container remove ac43f6fd2cc2445a3a420f3e3e6f5d5fe653e605394bf652a5070f0f4dbd25f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T11:08:45.822 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:45 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:08:45.822 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:45 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:08:45.822 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:45 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.938s CPU time. 2024-07-26T11:08:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:46.113+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800034c0 con 0x7f74b0109990 2024-07-26T11:08:46.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:46.113+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800034c0 con 0x7f74b0109990 2024-07-26T11:08:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:46.126+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780034c0 con 0x563c6c4182f0 2024-07-26T11:08:46.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:46.127+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74780034c0 con 0x563c6c4182f0 2024-07-26T11:08:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:45 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:45 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:08:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:45 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:active 2024-07-26T11:08:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:45 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:45 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:08:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:45 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:45 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:08:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:45 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:active 2024-07-26T11:08:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:45 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:45 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:08:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:45 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:08:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:45 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:08:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:45 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/2008194334,v1:172.21.15.183:6837/2008194334] up:active 2024-07-26T11:08:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:45 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:45 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:08:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:46 smithi183 ceph-mon[37273]: pgmap v660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:46 smithi144 ceph-mon[32129]: pgmap v660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:46 smithi160 ceph-mon[36715]: pgmap v660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:08:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:48 smithi183 ceph-mon[37273]: pgmap v661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:48 smithi144 ceph-mon[32129]: pgmap v661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:48 smithi160 ceph-mon[36715]: pgmap v661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:50 smithi183 ceph-mon[37273]: pgmap v662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:50 smithi144 ceph-mon[32129]: pgmap v662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:50 smithi160 ceph-mon[36715]: pgmap v662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:08:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:52 smithi183 ceph-mon[37273]: pgmap v663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 189 B/s wr, 7 op/s 2024-07-26T11:08:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:52 smithi144 ceph-mon[32129]: pgmap v663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 189 B/s wr, 7 op/s 2024-07-26T11:08:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:52 smithi160 ceph-mon[36715]: pgmap v663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 189 B/s wr, 7 op/s 2024-07-26T11:08:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:54 smithi183 ceph-mon[37273]: pgmap v664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:54 smithi144 ceph-mon[32129]: pgmap v664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:54 smithi160 ceph-mon[36715]: pgmap v664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:08:56.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:55 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:08:56.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:55 smithi183 systemd[1]: Stopped Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:56.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:55 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.938s CPU time. 2024-07-26T11:08:56.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:55 smithi183 systemd[1]: Starting Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:08:56.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 podman[73222]: 2024-07-26 11:08:56.021914628 +0000 UTC m=+0.017634512 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:08:56.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 podman[73222]: 2024-07-26 11:08:56.139793746 +0000 UTC m=+0.135513617 container create 5c1436b6972800693da286774815e02ca82923ba12e8d2c8810c015b049aa4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724) 2024-07-26T11:08:56.420 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 podman[73222]: 2024-07-26 11:08:56.384576567 +0000 UTC m=+0.380296455 container init 5c1436b6972800693da286774815e02ca82923ba12e8d2c8810c015b049aa4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T11:08:56.420 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 podman[73222]: 2024-07-26 11:08:56.387138701 +0000 UTC m=+0.382858572 container start 5c1436b6972800693da286774815e02ca82923ba12e8d2c8810c015b049aa4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T11:08:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:56.700+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:56.701+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:56.701+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:56.701+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:08:56.701+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:56.874 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mds[73241]: -- 172.21.15.183:0/2594170665 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b84e15f860 con 0x55b852301180 2024-07-26T11:08:56.874 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: starting mds.c at 2024-07-26T11:08:56.874 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 bash[73222]: 5c1436b6972800693da286774815e02ca82923ba12e8d2c8810c015b049aa4c6 2024-07-26T11:08:56.874 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:08:56 smithi183 systemd[1]: Started Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: pgmap v665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: daemon mds.c restarted 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:boot 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:08:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:08:57.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:56 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: pgmap v665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: daemon mds.c restarted 2024-07-26T11:08:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:57.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:08:57.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T11:08:57.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:boot 2024-07-26T11:08:57.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:08:57.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:08:57.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:56 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: pgmap v665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: daemon mds.c restarted 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:boot 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:08:57.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:56 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:59.152 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:58 smithi144 ceph-mon[32129]: pgmap v667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:08:59.152 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:58 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:resolve 2024-07-26T11:08:59.152 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:58 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:58 smithi160 ceph-mon[36715]: pgmap v667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:08:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:58 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:resolve 2024-07-26T11:08:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:58 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:08:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:58 smithi183 ceph-mon[37273]: pgmap v667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:08:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:58 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:resolve 2024-07-26T11:08:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:58 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:00.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:59 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:reconnect 2024-07-26T11:09:00.200 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:59 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:00.200 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:08:59 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:59 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:reconnect 2024-07-26T11:09:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:59 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:00.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:08:59 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:59 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:reconnect 2024-07-26T11:09:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:59 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:08:59 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:00.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: *** Caught signal (Aborted) ** 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: in thread 7f9155fa1640 thread_name:ms_dispatch 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f915b3786f0] 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f915b3c594c] 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 3: raise() 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 4: abort() 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 5: /lib64/libc.so.6(+0x2871b) [0x7f915b36271b] 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 6: /lib64/libc.so.6(+0x37386) [0x7f915b371386] 2024-07-26T11:09:00.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf46e646b9] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf4702280c] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bf46f622ca] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bf46f32753] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf46f33a1b] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf46f17752] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf46e65a56] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf46e69637] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf46e69f5e] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf46e4d03a] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f915bc45407] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 18: (DispatchQueue::entry()+0x60f) [0x7f915bc45cf9] 2024-07-26T11:09:00.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f915bd225a1] 2024-07-26T11:09:00.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 20: (Thread::entry_wrapper()+0x43) [0x7f915bacfcdd] 2024-07-26T11:09:00.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 21: (Thread::_entry_func(void*)+0xd) [0x7f915bacfcf9] 2024-07-26T11:09:00.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 22: /lib64/libc.so.6(+0x89c02) [0x7f915b3c3c02] 2024-07-26T11:09:00.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f915b448c40] 2024-07-26T11:09:00.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.556+0000 7f9155fa1640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:00.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: in thread 7f9155fa1640 thread_name:ms_dispatch 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f915b3786f0] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f915b3c594c] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 3: raise() 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 4: abort() 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 5: /lib64/libc.so.6(+0x2871b) [0x7f915b36271b] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 6: /lib64/libc.so.6(+0x37386) [0x7f915b371386] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf46e646b9] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf4702280c] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bf46f622ca] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bf46f32753] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf46f33a1b] 2024-07-26T11:09:00.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf46f17752] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf46e65a56] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf46e69637] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf46e69f5e] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf46e4d03a] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f915bc45407] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 18: (DispatchQueue::entry()+0x60f) [0x7f915bc45cf9] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f915bd225a1] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 20: (Thread::entry_wrapper()+0x43) [0x7f915bacfcdd] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 21: (Thread::_entry_func(void*)+0xd) [0x7f915bacfcf9] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 22: /lib64/libc.so.6(+0x89c02) [0x7f915b3c3c02] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f915b448c40] 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 0> 2024-07-26T11:09:00.556+0000 7f9155fa1640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: in thread 7f9155fa1640 thread_name:ms_dispatch 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f915b3786f0] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f915b3c594c] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 3: raise() 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 4: abort() 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 5: /lib64/libc.so.6(+0x2871b) [0x7f915b36271b] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 6: /lib64/libc.so.6(+0x37386) [0x7f915b371386] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf46e646b9] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf4702280c] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bf46f622ca] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bf46f32753] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf46f33a1b] 2024-07-26T11:09:00.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf46f17752] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf46e65a56] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf46e69637] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf46e69f5e] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf46e4d03a] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f915bc45407] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 18: (DispatchQueue::entry()+0x60f) [0x7f915bc45cf9] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f915bd225a1] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 20: (Thread::entry_wrapper()+0x43) [0x7f915bacfcdd] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 21: (Thread::_entry_func(void*)+0xd) [0x7f915bacfcf9] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 22: /lib64/libc.so.6(+0x89c02) [0x7f915b3c3c02] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f915b448c40] 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 0> 2024-07-26T11:09:00.556+0000 7f9155fa1640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:00.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: in thread 7f9155fa1640 thread_name:ms_dispatch 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f915b3786f0] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f915b3c594c] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 3: raise() 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 4: abort() 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 5: /lib64/libc.so.6(+0x2871b) [0x7f915b36271b] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 6: /lib64/libc.so.6(+0x37386) [0x7f915b371386] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf46e646b9] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf4702280c] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bf46f622ca] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bf46f32753] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf46f33a1b] 2024-07-26T11:09:00.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf46f17752] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf46e65a56] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf46e69637] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf46e69f5e] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf46e4d03a] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f915bc45407] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 18: (DispatchQueue::entry()+0x60f) [0x7f915bc45cf9] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f915bd225a1] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 20: (Thread::entry_wrapper()+0x43) [0x7f915bacfcdd] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 21: (Thread::_entry_func(void*)+0xd) [0x7f915bacfcf9] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 22: /lib64/libc.so.6(+0x89c02) [0x7f915b3c3c02] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f915b448c40] 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:00.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[68329]: 2024-07-26T11:09:00.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:00 smithi183 podman[73275]: 2024-07-26 11:09:00.616832039 +0000 UTC m=+0.021851566 container died 1f1bd016736e6340599f54ac8fd835123bf13f4660757e184c3978cadabe1c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T11:09:01.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: pgmap v668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:09:01.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:rejoin 2024-07-26T11:09:01.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:01.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:09:01.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:01.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: Cluster is now healthy 2024-07-26T11:09:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:active 2024-07-26T11:09:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: 1 slow requests, 1 included below; oldest blocked for > 184.528681 secs 2024-07-26T11:09:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:00 smithi144 ceph-mon[32129]: slow request 184.528681 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:09:01.241 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: pgmap v668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:09:01.241 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:rejoin 2024-07-26T11:09:01.241 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:01.241 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:09:01.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:01.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: Cluster is now healthy 2024-07-26T11:09:01.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:active 2024-07-26T11:09:01.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:01.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: 1 slow requests, 1 included below; oldest blocked for > 184.528681 secs 2024-07-26T11:09:01.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:00 smithi183 ceph-mon[37273]: slow request 184.528681 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:09:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: pgmap v668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:09:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:rejoin 2024-07-26T11:09:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: Cluster is now healthy 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/15323047,v1:172.21.15.183:6835/15323047] up:active 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: 1 slow requests, 1 included below; oldest blocked for > 184.528681 secs 2024-07-26T11:09:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:00 smithi160 ceph-mon[36715]: slow request 184.528681 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:09:01.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:01 smithi183 podman[73275]: 2024-07-26 11:09:01.240200874 +0000 UTC m=+0.645220385 container remove 1f1bd016736e6340599f54ac8fd835123bf13f4660757e184c3978cadabe1c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-07-26T11:09:01.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:01 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:09:01.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:01 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:09:01.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:01 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.154s CPU time. 2024-07-26T11:09:02.309 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:09:02 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:09:02.164+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:09:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:01 smithi160 ceph-mon[36715]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:09:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:01 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:09:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:01 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:01 smithi183 ceph-mon[37273]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:09:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:01 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:09:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:01 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:01 smithi144 ceph-mon[32129]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:09:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:01 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:09:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:01 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:02 smithi144 ceph-mon[32129]: pgmap v669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:02 smithi160 ceph-mon[36715]: pgmap v669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:02 smithi183 ceph-mon[37273]: pgmap v669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:04 smithi144 ceph-mon[32129]: pgmap v670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:09:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:04 smithi160 ceph-mon[36715]: pgmap v670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:09:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:04 smithi183 ceph-mon[37273]: pgmap v670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:09:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:05 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 189.528798 secs 2024-07-26T11:09:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:05 smithi160 ceph-mon[36715]: slow request 184.667986 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:09:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:05 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 189.528798 secs 2024-07-26T11:09:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:05 smithi183 ceph-mon[37273]: slow request 184.667986 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:09:06.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:09:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:05 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 189.528798 secs 2024-07-26T11:09:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:05 smithi144 ceph-mon[32129]: slow request 184.667986 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:09:06.867 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:06 smithi160 ceph-mon[36715]: pgmap v671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:06 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:09:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:06 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:07.370 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:06 smithi144 ceph-mon[32129]: pgmap v671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:07.370 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:06 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:09:07.370 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:06 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:06 smithi183 ceph-mon[37273]: pgmap v671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:06 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] up:active 2024-07-26T11:09:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:06 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:09.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:09 smithi144 ceph-mon[32129]: pgmap v672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 187 B/s wr, 7 op/s 2024-07-26T11:09:09.124 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:09 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:09.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 -- 172.21.15.144:0/238537746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10f330 msgr2=0x7f646c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 --2- 172.21.15.144:0/238537746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10f330 0x7f646c10f740 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6458009900 tx=0x7f645802a400 comp rx=0 tx=0).stop 2024-07-26T11:09:09.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 -- 172.21.15.144:0/238537746 shutdown_connections 2024-07-26T11:09:09.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 --2- 172.21.15.144:0/238537746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f646c10fc80 0x7f646c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 --2- 172.21.15.144:0/238537746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10f330 0x7f646c10f740 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 --2- 172.21.15.144:0/238537746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.120+0000 7f646be00640 1 -- 172.21.15.144:0/238537746 >> 172.21.15.144:0/238537746 conn(0x7f646c078ff0 msgr2=0x7f646c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:09.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.121+0000 7f646be00640 1 -- 172.21.15.144:0/238537746 shutdown_connections 2024-07-26T11:09:09.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.121+0000 7f646be00640 1 -- 172.21.15.144:0/238537746 wait complete. 2024-07-26T11:09:09.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.122+0000 7f646be00640 1 Processor -- start 2024-07-26T11:09:09.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 -- start start 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c1a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f646c10f330 0x7f646c1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10fc80 0x7f646c1a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f646c119940 con 0x7f646c10f330 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f646c1197c0 con 0x7f646c10fc80 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.123+0000 7f646be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f646c119ac0 con 0x7f646c1099f0 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c1a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c1a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48072/0 (socket says 172.21.15.144:48072) 2024-07-26T11:09:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 -- 172.21.15.144:0/3578570357 learned_addr learned my addr 172.21.15.144:0/3578570357 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646a000640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f646c10f330 0x7f646c1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646b400640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10fc80 0x7f646c1a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 -- 172.21.15.144:0/3578570357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10fc80 msgr2=0x7f646c1a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10fc80 0x7f646c1a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 -- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f646c10f330 msgr2=0x7f646c1a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f646c10f330 0x7f646c1a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.124+0000 7f646aa00640 1 -- 172.21.15.144:0/3578570357 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c1a62a0 con 0x7f646c1099f0 2024-07-26T11:09:09.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.125+0000 7f646aa00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c1a0330 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f645c00d950 tx=0x7f645c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:09.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.125+0000 7f6463400640 1 -- 172.21.15.144:0/3578570357 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c017030 con 0x7f646c1099f0 2024-07-26T11:09:09.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.125+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646c1a6590 con 0x7f646c1099f0 2024-07-26T11:09:09.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.125+0000 7f6463400640 1 -- 172.21.15.144:0/3578570357 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f645c004020 con 0x7f646c1099f0 2024-07-26T11:09:09.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.125+0000 7f6463400640 1 -- 172.21.15.144:0/3578570357 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c0047a0 con 0x7f646c1099f0 2024-07-26T11:09:09.128 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.125+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646c1accf0 con 0x7f646c1099f0 2024-07-26T11:09:09.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 Processor -- start 2024-07-26T11:09:09.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 -- start start 2024-07-26T11:09:09.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f6463400640 1 -- 172.21.15.144:0/3578570357 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f645c0041d0 con 0x7f646c1099f0 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 0x5617e037b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e037b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffb5390 0x5617e037ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5617e02f1bc0 con 0x5617dffb5390 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5617e02e10c0 con 0x5617dffd6070 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f647825a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5617e02e5100 con 0x5617dffad1b0 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f646b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 0x5617e037b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f646b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 0x5617e037b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36244/0 (socket says 172.21.15.144:36244) 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.129+0000 7f646b400640 1 -- 172.21.15.144:0/2687955863 learned_addr learned my addr 172.21.15.144:0/2687955863 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646aa00640 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e037b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646a000640 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffb5390 0x5617e037ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646b400640 1 -- 172.21.15.144:0/2687955863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 msgr2=0x5617e037b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646b400640 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e037b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646b400640 1 -- 172.21.15.144:0/2687955863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffb5390 msgr2=0x5617e037ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646b400640 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffb5390 0x5617e037ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646b400640 1 -- 172.21.15.144:0/2687955863 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617e0041650 con 0x5617dffd6070 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.130+0000 7f646b400640 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 0x5617e037b000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6464009f50 tx=0x7f64640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b16e04f097aa88 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.131+0000 7f6457400640 1 -- 172.21.15.144:0/2687955863 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464004d20 con 0x5617dffd6070 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.131+0000 7f6457400640 1 -- 172.21.15.144:0/2687955863 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6464004050 con 0x5617dffd6070 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.131+0000 7f6457400640 1 -- 172.21.15.144:0/2687955863 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464005280 con 0x5617dffd6070 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.131+0000 7f6463400640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f64500b3dc0 0x7f64500b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.131+0000 7f6463400640 1 -- 172.21.15.144:0/3578570357 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 10426+0+0 (secure 0 0 0) 0x7f645c0e3f70 con 0x7f646c1099f0 2024-07-26T11:09:09.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.131+0000 7f646aa00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f64500b3dc0 0x7f64500b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 -- 172.21.15.144:0/2687955863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 msgr2=0x5617e037b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 0x5617e037b000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6464009f50 tx=0x7f64640118d0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 -- 172.21.15.144:0/2687955863 shutdown_connections 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffb5390 0x5617e037ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e037b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 --2- 172.21.15.144:0/2687955863 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffd6070 0x5617e037b000 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 -- 172.21.15.144:0/2687955863 >> 172.21.15.144:0/2687955863 conn(0x5617dffca460 msgr2=0x5617e0378ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f646aa00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f64500b3dc0 0x7f64500b62d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x5617e0379c80 tx=0x7f645c0052e0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 -- 172.21.15.144:0/2687955863 shutdown_connections 2024-07-26T11:09:09.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.132+0000 7f647825a740 1 -- 172.21.15.144:0/2687955863 wait complete. 2024-07-26T11:09:09.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.133+0000 7f647825a740 1 -- start start 2024-07-26T11:09:09.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.133+0000 7f647825a740 1 Processor -- start 2024-07-26T11:09:09.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0386a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e0386fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5617e0144700 con 0x5617dffd6070 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5617e027d690 con 0x5617dffad1b0 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5617e03897e0 con 0x5617dffb5390 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f647825a740 10 client.0 _ll_register_callbacks cb 0x5617e028d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f646b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e0386fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f646b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e0386fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43410/0 (socket says 172.21.15.144:43410) 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f646b400640 1 -- 172.21.15.144:0/1389423074 learned_addr learned my addr 172.21.15.144:0/1389423074 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f646a000640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0386a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.134+0000 7f646b400640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f646b400640 1 -- 172.21.15.144:0/1389423074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 msgr2=0x5617e0386a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f646b400640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0386a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f646b400640 1 -- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 msgr2=0x5617e0386520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f646b400640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f646b400640 1 -- 172.21.15.144:0/1389423074 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617e0294830 con 0x5617dffd6070 2024-07-26T11:09:09.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f646b400640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e0386fa0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6464003fd0 tx=0x7f6464004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9fb3524aee1ed8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:09.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f644be00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:09:09.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f644be00640 1 -- 172.21.15.144:0/1389423074 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646402d060 con 0x5617dffd6070 2024-07-26T11:09:09.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.135+0000 7f644be00640 1 -- 172.21.15.144:0/1389423074 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 10426+0+0 (secure 0 0 0) 0x7f6464004c10 con 0x5617dffd6070 2024-07-26T11:09:09.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6464004050 con 0x5617dffd6070 2024-07-26T11:09:09.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f644be00640 1 -- 172.21.15.144:0/1389423074 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f6464034050 con 0x5617dffd6070 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f647825a740 10 client.16062 fetch_fsmap learned FSMap version 110 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x5617e0395390 con 0x5617dffd6070 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f644be00640 1 -- 172.21.15.144:0/1389423074 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6464004050 con 0x5617dffd6070 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f644be00640 1 -- 172.21.15.144:0/1389423074 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 110) ==== 7749+0+0 (secure 0 0 0) 0x7f6464019430 con 0x5617dffd6070 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.136+0000 7f647825a740 10 client.16062 fetch_fsmap finished waiting for FSMap version 110 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.137+0000 7f647825a740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.137+0000 7f647825a740 20 client.16062 populate_metadata read hostname 'smithi144' 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.137+0000 7f647825a740 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff949f0 0x5617e0398800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.137+0000 7f647825a740 4 client.16062 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.137+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617e02f3b10 con 0x5617dff949f0 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.137+0000 7f646a000640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff949f0 0x5617e0398800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.138+0000 7f646a000640 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff949f0 0x5617e0398800 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=392cb116658a71f3 server_cookie=df93d2dac5a47a21 in_seq=0 out_seq=0 2024-07-26T11:09:09.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.138+0000 7f644be00640 10 client.16062 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:09.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.138+0000 7f644be00640 1 -- 172.21.15.144:0/1389423074 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5617e02f3b10 con 0x5617dff949f0 2024-07-26T11:09:09.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.138+0000 7f644be00640 10 client.16062 handle_command_reply: tid=0 2024-07-26T11:09:09.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.138+0000 7f647825a740 1 client.16062 shutdown 2024-07-26T11:09:09.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 msgr2=0x5617e0386fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e0386fa0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6464003fd0 tx=0x7f6464004950 comp rx=0 tx=0).stop 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 shutdown_connections 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e0386fa0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0386a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff949f0 0x5617e0398800 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 --2- 172.21.15.144:0/1389423074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 >> 172.21.15.144:0/1389423074 conn(0x5617dffca460 msgr2=0x5617e037e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f644be00640 0 client.16062 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 shutdown_connections 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 1 -- 172.21.15.144:0/1389423074 wait complete. 2024-07-26T11:09:09.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.139+0000 7f647825a740 20 client.16062 trim_cache size 0 max 16384 2024-07-26T11:09:09.160 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 Processor -- start 2024-07-26T11:09:09.160 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 -- start start 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0377530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e037c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f644c0122c0 con 0x5617dffd6070 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f644c012990 con 0x5617dffad1b0 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.158+0000 7f647825a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f644c0102a0 con 0x5617dffb5390 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646b400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e037c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0377530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0377530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48090/0 (socket says 172.21.15.144:48090) 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 -- 172.21.15.144:0/441836689 learned_addr learned my addr 172.21.15.144:0/441836689 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646b400640 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 -- 172.21.15.144:0/441836689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 msgr2=0x5617e0386e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 -- 172.21.15.144:0/441836689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 msgr2=0x5617e037c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e037c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.159+0000 7f646a000640 1 -- 172.21.15.144:0/441836689 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617e03877e0 con 0x5617dffb5390 2024-07-26T11:09:09.162 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.160+0000 7f646a000640 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0377530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f646c06c270 tx=0x7f6458031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42675e3617182852 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:09.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.160+0000 7f6456a00640 1 -- 172.21.15.144:0/441836689 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645804a070 con 0x5617dffb5390 2024-07-26T11:09:09.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.160+0000 7f6456a00640 1 -- 172.21.15.144:0/441836689 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6458004380 con 0x5617dffb5390 2024-07-26T11:09:09.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.160+0000 7f6456a00640 1 -- 172.21.15.144:0/441836689 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458035040 con 0x5617dffb5390 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 -- 172.21.15.144:0/441836689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 msgr2=0x5617e0377530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0377530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f646c06c270 tx=0x7f6458031400 comp rx=0 tx=0).stop 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 -- 172.21.15.144:0/441836689 shutdown_connections 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e037c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffb5390 0x5617e0377530 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 --2- 172.21.15.144:0/441836689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffad1b0 0x5617e0386e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 -- 172.21.15.144:0/441836689 >> 172.21.15.144:0/441836689 conn(0x5617dffca460 msgr2=0x5617e037c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 -- 172.21.15.144:0/441836689 shutdown_connections 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.161+0000 7f647825a740 1 -- 172.21.15.144:0/441836689 wait complete. 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.162+0000 7f647825a740 1 -- start start 2024-07-26T11:09:09.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.162+0000 7f647825a740 1 Processor -- start 2024-07-26T11:09:09.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e038c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffb5390 0x5617e038ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e038d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f644c00fbd0 con 0x5617dffd6070 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f644c00e7c0 con 0x5617dffb5390 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f644c00ee30 con 0x5617dffad1b0 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f647825a740 10 client.0 _ll_register_callbacks cb 0x7f6464004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f646b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e038c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f646b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e038c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48092/0 (socket says 172.21.15.144:48092) 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f646b400640 1 -- 172.21.15.144:0/463426467 learned_addr learned my addr 172.21.15.144:0/463426467 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f646a000640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffb5390 0x5617e038ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.163+0000 7f646b400640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e038d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646a000640 1 -- 172.21.15.144:0/463426467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 msgr2=0x5617e038c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646a000640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e038c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646a000640 1 -- 172.21.15.144:0/463426467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 msgr2=0x5617e038d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:09.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646a000640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e038d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646a000640 1 -- 172.21.15.144:0/463426467 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617e03ffe10 con 0x5617dffb5390 2024-07-26T11:09:09.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646b400640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e038c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f646a000640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffb5390 0x5617e038ce10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6458048040 tx=0x7f645802a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0bb8fae70c3583 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f644be00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f644be00640 1 -- 172.21.15.144:0/463426467 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645804a040 con 0x5617dffb5390 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6458004380 con 0x5617dffb5390 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f644be00640 1 -- 172.21.15.144:0/463426467 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 10426+0+0 (secure 0 0 0) 0x7f64580035f0 con 0x5617dffb5390 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.164+0000 7f644be00640 1 -- 172.21.15.144:0/463426467 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f6458035040 con 0x5617dffb5390 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 10 client.26252 fetch_fsmap learned FSMap version 110 2024-07-26T11:09:09.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x5617e0400fc0 con 0x5617dffb5390 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f644be00640 1 -- 172.21.15.144:0/463426467 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6458004380 con 0x5617dffb5390 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f644be00640 1 -- 172.21.15.144:0/463426467 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 110) ==== 7749+0+0 (secure 0 0 0) 0x7f645804d070 con 0x5617dffb5390 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 10 client.26252 fetch_fsmap finished waiting for FSMap version 110 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 20 client.26252 populate_metadata read hostname 'smithi144' 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff69da0 0x5617e0403710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 4 client.26252 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5617e02ef5b0 con 0x5617dff69da0 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.165+0000 7f646b400640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff69da0 0x5617e0403710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f646b400640 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff69da0 0x5617e0403710 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34441e41772cfc58 server_cookie=bd0bf144c1ecdc0d in_seq=0 out_seq=0 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f644be00640 10 client.26252 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f644be00640 1 -- 172.21.15.144:0/463426467 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5617e02ef5b0 con 0x5617dff69da0 2024-07-26T11:09:09.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f644be00640 10 client.26252 handle_command_reply: tid=0 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 client.26252 shutdown 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffb5390 msgr2=0x5617e038ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffb5390 0x5617e038ce10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6458048040 tx=0x7f645802a400 comp rx=0 tx=0).stop 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 shutdown_connections 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5617dffd6070 0x5617e038d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5617dff69da0 0x5617e0403710 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5617dffb5390 0x5617e038ce10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 --2- 172.21.15.144:0/463426467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5617dffad1b0 0x5617e038c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f644be00640 0 client.26252 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.166+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 >> 172.21.15.144:0/463426467 conn(0x5617dffca460 msgr2=0x5617e0386c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 shutdown_connections 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f647825a740 1 -- 172.21.15.144:0/463426467 wait complete. 2024-07-26T11:09:09.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f647825a740 20 client.26252 trim_cache size 0 max 16384 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:09:09.172 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:09:09.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f64500b3dc0 msgr2=0x7f64500b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f646be00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f64500b3dc0 0x7f64500b62d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x5617e0379c80 tx=0x7f645c0052e0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 msgr2=0x7f646c1a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:09.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.167+0000 7f646be00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c1a0330 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f645c00d950 tx=0x7f645c00de10 comp rx=0 tx=0).stop 2024-07-26T11:09:09.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 shutdown_connections 2024-07-26T11:09:09.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f64500b3dc0 0x7f64500b62d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f646c10fc80 0x7f646c1a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f646c10f330 0x7f646c1a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 --2- 172.21.15.144:0/3578570357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c1099f0 0x7f646c1a0330 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:09.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 >> 172.21.15.144:0/3578570357 conn(0x7f646c078ff0 msgr2=0x7f646c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:09.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 shutdown_connections 2024-07-26T11:09:09.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:09.168+0000 7f646be00640 1 -- 172.21.15.144:0/3578570357 wait complete. 2024-07-26T11:09:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:09 smithi160 ceph-mon[36715]: pgmap v672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 187 B/s wr, 7 op/s 2024-07-26T11:09:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:09 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:09 smithi183 ceph-mon[37273]: pgmap v672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 187 B/s wr, 7 op/s 2024-07-26T11:09:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:09 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:10.182 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:09:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:11 smithi160 ceph-mon[36715]: pgmap v673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:09:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 194.528857 secs 2024-07-26T11:09:11.373 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:11 smithi144 ceph-mon[32129]: pgmap v673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:09:11.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 194.528857 secs 2024-07-26T11:09:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:11 smithi183 ceph-mon[37273]: pgmap v673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:09:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 194.528857 secs 2024-07-26T11:09:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:11.717+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:11.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:11.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:11.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:11.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:11.988 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:11 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:09:11.988 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:11 smithi183 systemd[1]: Stopped Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:09:11.988 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:11 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.154s CPU time. 2024-07-26T11:09:11.988 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:11 smithi183 systemd[1]: Starting Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:09:11.988 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:11 smithi183 podman[73366]: 2024-07-26 11:09:11.767758575 +0000 UTC m=+0.017750752 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:11.988 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:11 smithi183 podman[73366]: 2024-07-26 11:09:11.86563211 +0000 UTC m=+0.115624288 container create fd883de6cf8a141a120f6e0ae755381f62a77e82e5abe431bbd26fc37224e8b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T11:09:12.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:12 smithi183 podman[73366]: 2024-07-26 11:09:12.101620556 +0000 UTC m=+0.351612751 container init fd883de6cf8a141a120f6e0ae755381f62a77e82e5abe431bbd26fc37224e8b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T11:09:12.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:12 smithi183 podman[73366]: 2024-07-26 11:09:12.10421751 +0000 UTC m=+0.354209685 container start fd883de6cf8a141a120f6e0ae755381f62a77e82e5abe431bbd26fc37224e8b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:09:12.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:12 smithi183 ceph-mds[73386]: -- 172.21.15.183:0/1299023498 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ca0e21860 con 0x560ca4fc3180 2024-07-26T11:09:12.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:12 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: starting mds.f at 2024-07-26T11:09:12.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:12 smithi183 bash[73366]: fd883de6cf8a141a120f6e0ae755381f62a77e82e5abe431bbd26fc37224e8b3 2024-07-26T11:09:12.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:12 smithi183 systemd[1]: Started Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:09:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: pgmap v674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: daemon mds.f restarted 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:boot 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:13.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:13 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: pgmap v674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: daemon mds.f restarted 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:boot 2024-07-26T11:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:09:13.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:13.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:13 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: pgmap v674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: daemon mds.f restarted 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:boot 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:13 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:15.290 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:15 smithi144 ceph-mon[32129]: pgmap v676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:15.290 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:resolve 2024-07-26T11:09:15.290 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:15 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:15 smithi160 ceph-mon[36715]: pgmap v676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:resolve 2024-07-26T11:09:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:15 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:15 smithi183 ceph-mon[37273]: pgmap v676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:resolve 2024-07-26T11:09:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:15 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:16 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:reconnect 2024-07-26T11:09:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:16 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:16 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 199.528986 secs 2024-07-26T11:09:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:16 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:reconnect 2024-07-26T11:09:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:16 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:16 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 199.528986 secs 2024-07-26T11:09:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:16 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:reconnect 2024-07-26T11:09:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:16 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:16 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 199.528986 secs 2024-07-26T11:09:17.287 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-mon[37273]: pgmap v677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:09:17.287 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:rejoin 2024-07-26T11:09:17.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:17.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-mon[37273]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:09:17.288 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:09:17.288 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: *** Caught signal (Aborted) ** 2024-07-26T11:09:17.288 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: in thread 7f8c65b68640 thread_name:ms_dispatch 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c6af3f6f0] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c6af8c94c] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 3: raise() 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 4: abort() 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c6af2971b] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c6af38386] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b84cd0c6b9] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b84ceca80c] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b84ce0a2ca] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b84cdda753] 2024-07-26T11:09:17.289 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b84cddba1b] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b84cdbf752] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b84cd0da56] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b84cd11637] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b84cd11f5e] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b84ccf503a] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8c6b80c407] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 18: (DispatchQueue::entry()+0x60f) [0x7f8c6b80ccf9] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8c6b8e95a1] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 20: (Thread::entry_wrapper()+0x43) [0x7f8c6b696cdd] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8c6b696cf9] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c6af8ac02] 2024-07-26T11:09:17.290 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c6b00fc40] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.102+0000 7f8c65b68640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: in thread 7f8c65b68640 thread_name:ms_dispatch 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c6af3f6f0] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c6af8c94c] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 3: raise() 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 4: abort() 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c6af2971b] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c6af38386] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b84cd0c6b9] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b84ceca80c] 2024-07-26T11:09:17.291 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b84ce0a2ca] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b84cdda753] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b84cddba1b] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b84cdbf752] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b84cd0da56] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b84cd11637] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b84cd11f5e] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b84ccf503a] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8c6b80c407] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 18: (DispatchQueue::entry()+0x60f) [0x7f8c6b80ccf9] 2024-07-26T11:09:17.292 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8c6b8e95a1] 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 20: (Thread::entry_wrapper()+0x43) [0x7f8c6b696cdd] 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8c6b696cf9] 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c6af8ac02] 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c6b00fc40] 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 0> 2024-07-26T11:09:17.102+0000 7f8c65b68640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: in thread 7f8c65b68640 thread_name:ms_dispatch 2024-07-26T11:09:17.293 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c6af3f6f0] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c6af8c94c] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 3: raise() 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 4: abort() 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c6af2971b] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c6af38386] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b84cd0c6b9] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b84ceca80c] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b84ce0a2ca] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b84cdda753] 2024-07-26T11:09:17.294 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b84cddba1b] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b84cdbf752] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b84cd0da56] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b84cd11637] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b84cd11f5e] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b84ccf503a] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8c6b80c407] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 18: (DispatchQueue::entry()+0x60f) [0x7f8c6b80ccf9] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8c6b8e95a1] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 20: (Thread::entry_wrapper()+0x43) [0x7f8c6b696cdd] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8c6b696cf9] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c6af8ac02] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c6b00fc40] 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.295 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 0> 2024-07-26T11:09:17.102+0000 7f8c65b68640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: in thread 7f8c65b68640 thread_name:ms_dispatch 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c6af3f6f0] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c6af8c94c] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 3: raise() 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 4: abort() 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c6af2971b] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c6af38386] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b84cd0c6b9] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b84ceca80c] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b84ce0a2ca] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b84cdda753] 2024-07-26T11:09:17.296 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b84cddba1b] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b84cdbf752] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b84cd0da56] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b84cd11637] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b84cd11f5e] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b84ccf503a] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8c6b80c407] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 18: (DispatchQueue::entry()+0x60f) [0x7f8c6b80ccf9] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8c6b8e95a1] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 20: (Thread::entry_wrapper()+0x43) [0x7f8c6b696cdd] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8c6b696cf9] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c6af8ac02] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c6b00fc40] 2024-07-26T11:09:17.297 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:17.298 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73237]: 2024-07-26T11:09:17.298 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 podman[73420]: 2024-07-26 11:09:17.1619864 +0000 UTC m=+0.019018152 container died 5c1436b6972800693da286774815e02ca82923ba12e8d2c8810c015b049aa4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:09:17.309 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:09:17 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:09:17.165+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:09:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:17 smithi160 ceph-mon[36715]: pgmap v677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:09:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:17 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:rejoin 2024-07-26T11:09:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:17 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:17 smithi160 ceph-mon[36715]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:09:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:17 smithi144 ceph-mon[32129]: pgmap v677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:09:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:17 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:rejoin 2024-07-26T11:09:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:17 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:17 smithi144 ceph-mon[32129]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:09:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:18 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:18 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:active 2024-07-26T11:09:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:18 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:18 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:18.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 podman[73420]: 2024-07-26 11:09:17.831996782 +0000 UTC m=+0.689028578 container remove 5c1436b6972800693da286774815e02ca82923ba12e8d2c8810c015b049aa4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724) 2024-07-26T11:09:18.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:17 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:09:18.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:18 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:09:18.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:18 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.980s CPU time. 2024-07-26T11:09:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:18 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:18 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:active 2024-07-26T11:09:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:18 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:18 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:18 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:18 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/1813700401,v1:172.21.15.183:6837/1813700401] up:active 2024-07-26T11:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:18 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:18 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:19 smithi183 ceph-mon[37273]: pgmap v678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:09:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:19 smithi144 ceph-mon[32129]: pgmap v678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:09:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:19 smithi160 ceph-mon[36715]: pgmap v678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:09:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:21 smithi183 ceph-mon[37273]: pgmap v679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 204.529100 secs 2024-07-26T11:09:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:21 smithi144 ceph-mon[32129]: pgmap v679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 204.529100 secs 2024-07-26T11:09:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:21 smithi160 ceph-mon[36715]: pgmap v679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 204.529100 secs 2024-07-26T11:09:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:23 smithi183 ceph-mon[37273]: pgmap v680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:23 smithi144 ceph-mon[32129]: pgmap v680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:23 smithi160 ceph-mon[36715]: pgmap v680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:25 smithi183 ceph-mon[37273]: pgmap v681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:09:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:25 smithi144 ceph-mon[32129]: pgmap v681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:09:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:25 smithi160 ceph-mon[36715]: pgmap v681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:09:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 209.529248 secs 2024-07-26T11:09:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 209.529248 secs 2024-07-26T11:09:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 209.529248 secs 2024-07-26T11:09:26.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:26.725+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:26.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:26.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:26.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:26.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:26.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:26.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:26.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:26.727+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:27 smithi144 ceph-mon[32129]: pgmap v682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:09:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:27 smithi183 ceph-mon[37273]: pgmap v682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:09:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:27 smithi160 ceph-mon[36715]: pgmap v682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:09:28.492 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:09:28.492 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 systemd[1]: Stopped Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:09:28.492 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.980s CPU time. 2024-07-26T11:09:28.492 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 systemd[1]: Starting Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:09:28.492 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 podman[73511]: 2024-07-26 11:09:28.27489244 +0000 UTC m=+0.017418071 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:28.492 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 podman[73511]: 2024-07-26 11:09:28.373798713 +0000 UTC m=+0.116324325 container create 318f2e570fa95a4e0c31d05a41046d20e390a9620af82b5859e4eead29095142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T11:09:28.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 podman[73511]: 2024-07-26 11:09:28.610047415 +0000 UTC m=+0.352573041 container init 318f2e570fa95a4e0c31d05a41046d20e390a9620af82b5859e4eead29095142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, ceph=True) 2024-07-26T11:09:28.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 podman[73511]: 2024-07-26 11:09:28.612768306 +0000 UTC m=+0.355293926 container start 318f2e570fa95a4e0c31d05a41046d20e390a9620af82b5859e4eead29095142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T11:09:28.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 ceph-mds[73531]: -- 172.21.15.183:0/3931560984 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55895e4c3860 con 0x558962665180 2024-07-26T11:09:28.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: starting mds.c at 2024-07-26T11:09:28.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 bash[73511]: 318f2e570fa95a4e0c31d05a41046d20e390a9620af82b5859e4eead29095142 2024-07-26T11:09:28.885 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:28 smithi183 systemd[1]: Started Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:09:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: pgmap v683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: daemon mds.c restarted 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:boot 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:29.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:29 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: pgmap v683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: daemon mds.c restarted 2024-07-26T11:09:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:09:29.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T11:09:29.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:boot 2024-07-26T11:09:29.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:09:29.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:29.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:29 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: pgmap v683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: daemon mds.c restarted 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:boot 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:29 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:31.393 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:31 smithi144 ceph-mon[32129]: pgmap v685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:31.393 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:resolve 2024-07-26T11:09:31.393 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:31 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:31.393 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 214.529373 secs 2024-07-26T11:09:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:31 smithi160 ceph-mon[36715]: pgmap v685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:resolve 2024-07-26T11:09:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:31 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 214.529373 secs 2024-07-26T11:09:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:31 smithi183 ceph-mon[37273]: pgmap v685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:09:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:resolve 2024-07-26T11:09:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:31 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 214.529373 secs 2024-07-26T11:09:32.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:32 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:reconnect 2024-07-26T11:09:32.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:32 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:32.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:32 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:09:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:32 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:reconnect 2024-07-26T11:09:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:32 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:32 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:09:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:32 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:reconnect 2024-07-26T11:09:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:32 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:32 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:09:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:33 smithi144 ceph-mon[32129]: pgmap v686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6 op/s 2024-07-26T11:09:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:33 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:rejoin 2024-07-26T11:09:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:33 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:33 smithi144 ceph-mon[32129]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:09:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:33 smithi160 ceph-mon[36715]: pgmap v686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6 op/s 2024-07-26T11:09:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:33 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:rejoin 2024-07-26T11:09:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:33 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:33 smithi160 ceph-mon[36715]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:09:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-mon[37273]: pgmap v686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6 op/s 2024-07-26T11:09:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:rejoin 2024-07-26T11:09:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-mon[37273]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: *** Caught signal (Aborted) ** 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: in thread 7f5de5fd4640 thread_name:ms_dispatch 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5deb3ab6f0] 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5deb3f894c] 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 3: raise() 2024-07-26T11:09:33.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 4: abort() 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5deb39571b] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 6: /lib64/libc.so.6(+0x37386) [0x7f5deb3a4386] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560c9fb416b9] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560c9fcff80c] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560c9fc3f2ca] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560c9fc0f753] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560c9fc10a1b] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560c9fbf4752] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560c9fb42a56] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560c9fb46637] 2024-07-26T11:09:33.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560c9fb46f5e] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560c9fb2a03a] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5debc78407] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 18: (DispatchQueue::entry()+0x60f) [0x7f5debc78cf9] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5debd555a1] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 20: (Thread::entry_wrapper()+0x43) [0x7f5debb02cdd] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5debb02cf9] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5deb3f6c02] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5deb47bc40] 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.208+0000 7f5de5fd4640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: in thread 7f5de5fd4640 thread_name:ms_dispatch 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:33.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5deb3ab6f0] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5deb3f894c] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 3: raise() 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 4: abort() 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5deb39571b] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 6: /lib64/libc.so.6(+0x37386) [0x7f5deb3a4386] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560c9fb416b9] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560c9fcff80c] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560c9fc3f2ca] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560c9fc0f753] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560c9fc10a1b] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560c9fbf4752] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560c9fb42a56] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560c9fb46637] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560c9fb46f5e] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560c9fb2a03a] 2024-07-26T11:09:33.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5debc78407] 2024-07-26T11:09:33.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 18: (DispatchQueue::entry()+0x60f) [0x7f5debc78cf9] 2024-07-26T11:09:33.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5debd555a1] 2024-07-26T11:09:33.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 20: (Thread::entry_wrapper()+0x43) [0x7f5debb02cdd] 2024-07-26T11:09:33.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5debb02cf9] 2024-07-26T11:09:33.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5deb3f6c02] 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5deb47bc40] 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 0> 2024-07-26T11:09:33.208+0000 7f5de5fd4640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: in thread 7f5de5fd4640 thread_name:ms_dispatch 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5deb3ab6f0] 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5deb3f894c] 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 3: raise() 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 4: abort() 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5deb39571b] 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 6: /lib64/libc.so.6(+0x37386) [0x7f5deb3a4386] 2024-07-26T11:09:33.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560c9fb416b9] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560c9fcff80c] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560c9fc3f2ca] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560c9fc0f753] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560c9fc10a1b] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560c9fbf4752] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560c9fb42a56] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560c9fb46637] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560c9fb46f5e] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560c9fb2a03a] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5debc78407] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 18: (DispatchQueue::entry()+0x60f) [0x7f5debc78cf9] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5debd555a1] 2024-07-26T11:09:33.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 20: (Thread::entry_wrapper()+0x43) [0x7f5debb02cdd] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5debb02cf9] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5deb3f6c02] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5deb47bc40] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 0> 2024-07-26T11:09:33.208+0000 7f5de5fd4640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: in thread 7f5de5fd4640 thread_name:ms_dispatch 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5deb3ab6f0] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5deb3f894c] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 3: raise() 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 4: abort() 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5deb39571b] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 6: /lib64/libc.so.6(+0x37386) [0x7f5deb3a4386] 2024-07-26T11:09:33.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560c9fb416b9] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560c9fcff80c] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560c9fc3f2ca] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560c9fc0f753] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560c9fc10a1b] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560c9fbf4752] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560c9fb42a56] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560c9fb46637] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560c9fb46f5e] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560c9fb2a03a] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5debc78407] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 18: (DispatchQueue::entry()+0x60f) [0x7f5debc78cf9] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5debd555a1] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 20: (Thread::entry_wrapper()+0x43) [0x7f5debb02cdd] 2024-07-26T11:09:33.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5debb02cf9] 2024-07-26T11:09:33.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5deb3f6c02] 2024-07-26T11:09:33.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5deb47bc40] 2024-07-26T11:09:33.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:33.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[73382]: 2024-07-26T11:09:33.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 podman[73565]: 2024-07-26 11:09:33.269576476 +0000 UTC m=+0.019249817 container died fd883de6cf8a141a120f6e0ae755381f62a77e82e5abe431bbd26fc37224e8b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-07-26T11:09:34.249 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:34 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:34.249 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:34 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:active 2024-07-26T11:09:34.249 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:34 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:34.249 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:34 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:34 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:34 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:active 2024-07-26T11:09:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:34 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:34 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:34.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:33 smithi183 podman[73565]: 2024-07-26 11:09:33.998945959 +0000 UTC m=+0.748619300 container remove fd883de6cf8a141a120f6e0ae755381f62a77e82e5abe431bbd26fc37224e8b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T11:09:34.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:34 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:09:34.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:34 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:09:34.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:34 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 1.959s CPU time. 2024-07-26T11:09:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:34 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:34 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2601842162,v1:172.21.15.183:6835/2601842162] up:active 2024-07-26T11:09:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:34 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:34.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:34 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:35 smithi144 ceph-mon[32129]: pgmap v687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:09:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:35 smithi160 ceph-mon[36715]: pgmap v687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:09:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:35 smithi183 ceph-mon[37273]: pgmap v687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:09:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 219.529455 secs 2024-07-26T11:09:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 219.529455 secs 2024-07-26T11:09:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 219.529455 secs 2024-07-26T11:09:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:37 smithi144 ceph-mon[32129]: pgmap v688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:09:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:37 smithi160 ceph-mon[36715]: pgmap v688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:09:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:37 smithi183 ceph-mon[37273]: pgmap v688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:09:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:39 smithi144 ceph-mon[32129]: pgmap v689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:39 smithi160 ceph-mon[36715]: pgmap v689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:39 smithi183 ceph-mon[37273]: pgmap v689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:09:40.183 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:09:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:40.633 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:41 smithi160 ceph-mon[36715]: pgmap v690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 191 B/s wr, 8 op/s 2024-07-26T11:09:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 224.529521 secs 2024-07-26T11:09:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:41 smithi183 ceph-mon[37273]: pgmap v690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 191 B/s wr, 8 op/s 2024-07-26T11:09:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 224.529521 secs 2024-07-26T11:09:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:41 smithi144 ceph-mon[32129]: pgmap v690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 191 B/s wr, 8 op/s 2024-07-26T11:09:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 224.529521 secs 2024-07-26T11:09:41.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:41.741+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:41.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:41.742+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:41.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:41.742+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:41.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:41.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:41.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:41.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:42 smithi144 ceph-mon[32129]: pgmap v691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:09:42.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:42.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:42.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:42.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:42 smithi160 ceph-mon[36715]: pgmap v691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:09:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:42 smithi183 ceph-mon[37273]: pgmap v691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:09:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 -- 172.21.15.144:0/2964543410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f962810f940 msgr2=0x7f9628111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 --2- 172.21.15.144:0/2964543410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f962810f940 0x7f9628111d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9620009860 tx=0x7f962002a340 comp rx=0 tx=0).stop 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 -- 172.21.15.144:0/2964543410 shutdown_connections 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 --2- 172.21.15.144:0/2964543410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f962810f940 0x7f9628111d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 --2- 172.21.15.144:0/2964543410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9628075b30 0x7f962810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 --2- 172.21.15.144:0/2964543410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f96280751e0 0x7f96280755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 -- 172.21.15.144:0/2964543410 >> 172.21.15.144:0/2964543410 conn(0x7f96280fee70 msgr2=0x7f96281012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.547+0000 7f962ee00640 1 -- 172.21.15.144:0/2964543410 shutdown_connections 2024-07-26T11:09:43.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.548+0000 7f962ee00640 1 -- 172.21.15.144:0/2964543410 wait complete. 2024-07-26T11:09:43.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.549+0000 7f962ee00640 1 Processor -- start 2024-07-26T11:09:43.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.549+0000 7f962ee00640 1 -- start start 2024-07-26T11:09:43.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962ee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 0x7f962819bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962ee00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9628075b30 0x7f962819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962ee00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f962810f940 0x7f96281a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962ee00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f96281154d0 con 0x7f9628075b30 2024-07-26T11:09:43.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962ee00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9628115350 con 0x7f962810f940 2024-07-26T11:09:43.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962ee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9628115650 con 0x7f96280751e0 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962da00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 0x7f962819bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962da00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 0x7f962819bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58644/0 (socket says 172.21.15.144:58644) 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962da00640 1 -- 172.21.15.144:0/4119215673 learned_addr learned my addr 172.21.15.144:0/4119215673 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962d000640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9628075b30 0x7f962819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.550+0000 7f962e400640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f962810f940 0x7f96281a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962da00640 1 -- 172.21.15.144:0/4119215673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f962810f940 msgr2=0x7f96281a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962da00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f962810f940 0x7f96281a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962da00640 1 -- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9628075b30 msgr2=0x7f962819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962da00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9628075b30 0x7f962819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962da00640 1 -- 172.21.15.144:0/4119215673 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96281a1e10 con 0x7f96280751e0 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962e400640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f962810f940 0x7f96281a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962da00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 0x7f962819bea0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f961c00efb0 tx=0x7f961c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f9626000640 1 -- 172.21.15.144:0/4119215673 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c01b070 con 0x7f96280751e0 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f9626000640 1 -- 172.21.15.144:0/4119215673 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f961c00ba20 con 0x7f96280751e0 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.551+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96281a2100 con 0x7f96280751e0 2024-07-26T11:09:43.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.552+0000 7f9626000640 1 -- 172.21.15.144:0/4119215673 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c016b30 con 0x7f96280751e0 2024-07-26T11:09:43.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.552+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96281a8860 con 0x7f96280751e0 2024-07-26T11:09:43.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.556+0000 7f96371ef740 1 Processor -- start 2024-07-26T11:09:43.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.556+0000 7f96371ef740 1 -- start start 2024-07-26T11:09:43.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f9626000640 1 -- 172.21.15.144:0/4119215673 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f961c01e070 con 0x7f96280751e0 2024-07-26T11:09:43.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f96371ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3315fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f96371ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3316510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f96371ef740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3316a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f96371ef740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5638b328cb90 con 0x5638b2f50390 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f96371ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5638b327c090 con 0x5638b2f481b0 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.557+0000 7f96371ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5638b32800d0 con 0x5638b2f71070 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3316510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3315fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3315fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58660/0 (socket says 172.21.15.144:58660) 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 -- 172.21.15.144:0/3049971964 learned_addr learned my addr 172.21.15.144:0/3049971964 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:43.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962da00640 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3316510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:57096/0 (socket says 172.21.15.144:57096) 2024-07-26T11:09:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 -- 172.21.15.144:0/3049971964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 msgr2=0x5638b3316510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3316510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 -- 172.21.15.144:0/3049971964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 msgr2=0x5638b3316a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962d000640 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3316a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3316a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962e400640 1 -- 172.21.15.144:0/3049971964 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638b2fdc650 con 0x5638b2f71070 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.558+0000 7f962d000640 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3316a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f962e400640 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3315fd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9620009860 tx=0x7f9620005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2758df4a59f4150c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f961aa00640 1 -- 172.21.15.144:0/3049971964 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620037040 con 0x5638b2f71070 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f9626000640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96100b3e90 0x7f96100b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f961aa00640 1 -- 172.21.15.144:0/3049971964 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9620030450 con 0x5638b2f71070 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f961aa00640 1 -- 172.21.15.144:0/3049971964 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620005490 con 0x5638b2f71070 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f9626000640 1 -- 172.21.15.144:0/4119215673 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 10598+0+0 (secure 0 0 0) 0x7f961c0e4700 con 0x7f96280751e0 2024-07-26T11:09:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.559+0000 7f962da00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96100b3e90 0x7f96100b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f962da00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96100b3e90 0x7f96100b63a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x5638b3314c50 tx=0x7f961c0ea530 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:43.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 -- 172.21.15.144:0/3049971964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 msgr2=0x5638b3315fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3315fd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9620009860 tx=0x7f9620005e40 comp rx=0 tx=0).stop 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 -- 172.21.15.144:0/3049971964 shutdown_connections 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3316a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3316510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 --2- 172.21.15.144:0/3049971964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3315fd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 -- 172.21.15.144:0/3049971964 >> 172.21.15.144:0/3049971964 conn(0x5638b2f65460 msgr2=0x5638b3313ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.560+0000 7f96371ef740 1 -- 172.21.15.144:0/3049971964 shutdown_connections 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.561+0000 7f96371ef740 1 -- 172.21.15.144:0/3049971964 wait complete. 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.561+0000 7f96371ef740 1 -- start start 2024-07-26T11:09:43.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.561+0000 7f96371ef740 1 Processor -- start 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3321510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3321a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3321f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5638b30df700 con 0x5638b2f50390 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5638b3218660 con 0x5638b2f481b0 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5638b3324750 con 0x5638b2f71070 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f96371ef740 10 client.0 _ll_register_callbacks cb 0x5638b3228110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f962e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3321510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f962e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3321510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:57108/0 (socket says 172.21.15.144:57108) 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f962e400640 1 -- 172.21.15.144:0/595330393 learned_addr learned my addr 172.21.15.144:0/595330393 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f962d000640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3321a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.562+0000 7f962e400640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3321f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f962e400640 1 -- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 msgr2=0x5638b3321510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f962e400640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3321510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f962e400640 1 -- 172.21.15.144:0/595330393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 msgr2=0x5638b3321a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f962e400640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3321a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f962e400640 1 -- 172.21.15.144:0/595330393 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638b322f800 con 0x5638b2f71070 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f962e400640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3321f90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9620005fd0 tx=0x7f962002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3be2142fffaf2699 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.563+0000 7f960f400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f960f400640 1 -- 172.21.15.144:0/595330393 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962004b020 con 0x5638b2f71070 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9620030450 con 0x5638b2f71070 2024-07-26T11:09:43.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f960f400640 1 -- 172.21.15.144:0/595330393 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 10598+0+0 (secure 0 0 0) 0x7f9620004240 con 0x5638b2f71070 2024-07-26T11:09:43.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f960f400640 1 -- 172.21.15.144:0/595330393 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f9620059050 con 0x5638b2f71070 2024-07-26T11:09:43.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f96371ef740 10 client.26275 fetch_fsmap learned FSMap version 124 2024-07-26T11:09:43.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x5638b33302b0 con 0x5638b2f71070 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.564+0000 7f960f400640 1 -- 172.21.15.144:0/595330393 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9620030450 con 0x5638b2f71070 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f960f400640 1 -- 172.21.15.144:0/595330393 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 124) ==== 7749+0+0 (secure 0 0 0) 0x7f96200046c0 con 0x5638b2f71070 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f96371ef740 10 client.26275 fetch_fsmap finished waiting for FSMap version 124 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f96371ef740 10 client.26275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f96371ef740 20 client.26275 populate_metadata read hostname 'smithi144' 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f96371ef740 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f2f9f0 0x5638b3333610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f96371ef740 4 client.26275 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638b328eae0 con 0x5638b2f2f9f0 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.565+0000 7f962d000640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f2f9f0 0x5638b3333610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.566+0000 7f962d000640 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f2f9f0 0x5638b3333610 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4c0d4f9ccbcf966 server_cookie=6511de515bef6654 in_seq=0 out_seq=0 2024-07-26T11:09:43.571 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.566+0000 7f960f400640 10 client.26275 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:43.571 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.567+0000 7f960f400640 1 -- 172.21.15.144:0/595330393 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638b328eae0 con 0x5638b2f2f9f0 2024-07-26T11:09:43.571 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.567+0000 7f960f400640 10 client.26275 handle_command_reply: tid=0 2024-07-26T11:09:43.571 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.567+0000 7f96371ef740 1 client.26275 shutdown 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 msgr2=0x5638b3321f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3321f90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9620005fd0 tx=0x7f962002a340 comp rx=0 tx=0).stop 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 shutdown_connections 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3321f90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3321a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f2f9f0 0x5638b3333610 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 --2- 172.21.15.144:0/595330393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3321510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 >> 172.21.15.144:0/595330393 conn(0x5638b2f65460 msgr2=0x5638b3319160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f960f400640 0 client.26275 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.568+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 shutdown_connections 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.569+0000 7f96371ef740 1 -- 172.21.15.144:0/595330393 wait complete. 2024-07-26T11:09:43.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.569+0000 7f96371ef740 20 client.26275 trim_cache size 0 max 16384 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 Processor -- start 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 -- start start 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b33233f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3312500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3317710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f96281b4c50 con 0x5638b2f50390 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f96281b5320 con 0x5638b2f481b0 2024-07-26T11:09:43.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.595+0000 7f96371ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96281b2c30 con 0x5638b2f71070 2024-07-26T11:09:43.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b33233f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962d000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3312500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3317710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3317710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58670/0 (socket says 172.21.15.144:58670) 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962d000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3312500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45910/0 (socket says 172.21.15.144:45910) 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 -- 172.21.15.144:0/1643716123 learned_addr learned my addr 172.21.15.144:0/1643716123 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 -- 172.21.15.144:0/1643716123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 msgr2=0x5638b33233f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b33233f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 -- 172.21.15.144:0/1643716123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 msgr2=0x5638b3312500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3312500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.596+0000 7f962e400640 1 -- 172.21.15.144:0/1643716123 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638b33184f0 con 0x5638b2f71070 2024-07-26T11:09:43.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.597+0000 7f962e400640 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3317710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9620004a90 tx=0x7f9620000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3710f2b19717b797 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.597+0000 7f961a000640 1 -- 172.21.15.144:0/1643716123 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620062070 con 0x5638b2f71070 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.597+0000 7f961a000640 1 -- 172.21.15.144:0/1643716123 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9620030450 con 0x5638b2f71070 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.597+0000 7f961a000640 1 -- 172.21.15.144:0/1643716123 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962004b040 con 0x5638b2f71070 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 -- 172.21.15.144:0/1643716123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 msgr2=0x5638b3317710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3317710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9620004a90 tx=0x7f9620000bc0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 -- 172.21.15.144:0/1643716123 shutdown_connections 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f71070 0x5638b3317710 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f50390 0x5638b3312500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.601 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 --2- 172.21.15.144:0/1643716123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b33233f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 -- 172.21.15.144:0/1643716123 >> 172.21.15.144:0/1643716123 conn(0x5638b2f65460 msgr2=0x5638b3317000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:43.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 -- 172.21.15.144:0/1643716123 shutdown_connections 2024-07-26T11:09:43.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.598+0000 7f96371ef740 1 -- 172.21.15.144:0/1643716123 wait complete. 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 -- start start 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 Processor -- start 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3335980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 0x5638b3335f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f71070 0x5638b3399b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f96281b2560 con 0x5638b2f71070 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f96281b1150 con 0x5638b2f481b0 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96281b17c0 con 0x5638b2f50390 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.599+0000 7f96371ef740 10 client.0 _ll_register_callbacks cb 0x7f96200096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 0x5638b3335f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3335980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 0x5638b3335f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:58678/0 (socket says 172.21.15.144:58678) 2024-07-26T11:09:43.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 -- 172.21.15.144:0/441039940 learned_addr learned my addr 172.21.15.144:0/441039940 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:09:43.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962e400640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f71070 0x5638b3399b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 -- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 msgr2=0x5638b3335980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3335980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 -- 172.21.15.144:0/441039940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f71070 msgr2=0x5638b3399b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f71070 0x5638b3399b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962d000640 1 -- 172.21.15.144:0/441039940 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638b339a0a0 con 0x5638b2f50390 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.600+0000 7f962e400640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3335980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:09:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f962d000640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 0x5638b3335f40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5638b32187b0 tx=0x7f96140039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=859a12c6f627971c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f960f400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f960f400640 1 -- 172.21.15.144:0/441039940 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961401e790 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f960f400640 1 -- 172.21.15.144:0/441039940 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 10598+0+0 (secure 0 0 0) 0x7f961401e930 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9620030450 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f960f400640 1 -- 172.21.15.144:0/441039940 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f961402e050 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f96371ef740 10 client.26281 fetch_fsmap learned FSMap version 124 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x5638b339b2a0 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.601+0000 7f960f400640 1 -- 172.21.15.144:0/441039940 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9620030450 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f960f400640 1 -- 172.21.15.144:0/441039940 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 124) ==== 7749+0+0 (secure 0 0 0) 0x7f9614027430 con 0x5638b2f50390 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f96371ef740 10 client.26281 fetch_fsmap finished waiting for FSMap version 124 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f96371ef740 10 client.26281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:09:43.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f96371ef740 20 client.26281 populate_metadata read hostname 'smithi144' 2024-07-26T11:09:43.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f96371ef740 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f04da0 0x5638b339da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:43.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f96371ef740 4 client.26281 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:09:43.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638b328a580 con 0x5638b2f04da0 2024-07-26T11:09:43.607 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.602+0000 7f962e400640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f04da0 0x5638b339da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f960f400640 10 client.26281 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f962e400640 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f04da0 0x5638b339da60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1929ac2d99548b4 server_cookie=77947d8a8d2bec70 in_seq=0 out_seq=0 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f960f400640 1 -- 172.21.15.144:0/441039940 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5638b328a580 con 0x5638b2f04da0 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f960f400640 10 client.26281 handle_command_reply: tid=0 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f96371ef740 1 client.26281 shutdown 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 msgr2=0x5638b3335f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f96371ef740 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 0x5638b3335f40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5638b32187b0 tx=0x7f96140039c0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.603+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 shutdown_connections 2024-07-26T11:09:43.608 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5638b2f71070 0x5638b3399b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5638b2f50390 0x5638b3335f40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5638b2f04da0 0x5638b339da60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 --2- 172.21.15.144:0/441039940 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5638b2f481b0 0x5638b3335980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f960f400640 0 client.26281 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 >> 172.21.15.144:0/441039940 conn(0x5638b2f65460 msgr2=0x5638b332d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 shutdown_connections 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 1 -- 172.21.15.144:0/441039940 wait complete. 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.604+0000 7f96371ef740 20 client.26281 trim_cache size 0 max 16384 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:09:43.609 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:09:43.610 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96100b3e90 msgr2=0x7f96100b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96100b3e90 0x7f96100b63a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x5638b3314c50 tx=0x7f961c0ea530 comp rx=0 tx=0).stop 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 msgr2=0x7f962819bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 0x7f962819bea0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f961c00efb0 tx=0x7f961c00c800 comp rx=0 tx=0).stop 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 shutdown_connections 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f96100b3e90 0x7f96100b63a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f962810f940 0x7f96281a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9628075b30 0x7f962819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 --2- 172.21.15.144:0/4119215673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96280751e0 0x7f962819bea0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 >> 172.21.15.144:0/4119215673 conn(0x7f96280fee70 msgr2=0x7f96281094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:43.611 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 shutdown_connections 2024-07-26T11:09:43.612 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:09:43.605+0000 7f962ee00640 1 -- 172.21.15.144:0/4119215673 wait complete. 2024-07-26T11:09:44.439 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:09:44.830 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:44 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:09:44.830 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:44 smithi183 systemd[1]: Stopped Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:09:44.830 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:44 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 1.959s CPU time. 2024-07-26T11:09:44.830 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:44 smithi183 systemd[1]: Starting Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:09:44.830 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:44 smithi183 podman[74205]: 2024-07-26 11:09:44.524565576 +0000 UTC m=+0.017266614 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:44.830 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:44 smithi183 podman[74205]: 2024-07-26 11:09:44.828768328 +0000 UTC m=+0.321469421 container create 53841ccf3f247bd5a0bcf0a783f9f1f6898d995b2aefd4de1f1d8fa7e1f6abf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-26T11:09:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:44 smithi183 ceph-mon[37273]: pgmap v692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:45.105 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:45 smithi183 podman[74205]: 2024-07-26 11:09:45.068443145 +0000 UTC m=+0.561144176 container init 53841ccf3f247bd5a0bcf0a783f9f1f6898d995b2aefd4de1f1d8fa7e1f6abf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:09:45.105 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:45 smithi183 podman[74205]: 2024-07-26 11:09:45.071149824 +0000 UTC m=+0.563850855 container start 53841ccf3f247bd5a0bcf0a783f9f1f6898d995b2aefd4de1f1d8fa7e1f6abf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:09:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:44 smithi144 ceph-mon[32129]: pgmap v692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:44 smithi160 ceph-mon[36715]: pgmap v692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:45.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mds[74224]: -- 172.21.15.183:0/980157357 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56533bd1f860 con 0x56533fec1600 2024-07-26T11:09:45.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: starting mds.f at 2024-07-26T11:09:45.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:45 smithi183 bash[74205]: 53841ccf3f247bd5a0bcf0a783f9f1f6898d995b2aefd4de1f1d8fa7e1f6abf3 2024-07-26T11:09:45.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:09:45 smithi183 systemd[1]: Started Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:09:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: daemon mds.f restarted 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:boot 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 229.529647 secs 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:46.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: daemon mds.f restarted 2024-07-26T11:09:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:boot 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 229.529647 secs 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: daemon mds.f restarted 2024-07-26T11:09:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:09:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:boot 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:replay,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 229.529647 secs 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:09:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:46 smithi144 ceph-mon[32129]: pgmap v694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:09:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:46 smithi144 ceph-mon[32129]: pgmap v695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:09:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:46 smithi144 ceph-mon[32129]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:09:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:46 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:resolve 2024-07-26T11:09:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:46 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:47.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:46 smithi183 ceph-mon[37273]: pgmap v694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:09:47.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:46 smithi183 ceph-mon[37273]: pgmap v695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:09:47.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:46 smithi183 ceph-mon[37273]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:09:47.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:46 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:resolve 2024-07-26T11:09:47.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:46 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:46 smithi160 ceph-mon[36715]: pgmap v694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:09:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:46 smithi160 ceph-mon[36715]: pgmap v695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:09:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:46 smithi160 ceph-mon[36715]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:09:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:46 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:resolve 2024-07-26T11:09:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:46 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:resolve,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:48 smithi183 ceph-mon[37273]: pgmap v696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4 op/s 2024-07-26T11:09:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:reconnect 2024-07-26T11:09:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:48 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:48 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:48 smithi160 ceph-mon[36715]: pgmap v696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4 op/s 2024-07-26T11:09:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:reconnect 2024-07-26T11:09:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:48 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:49.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:48 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:48 smithi144 ceph-mon[32129]: pgmap v696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4 op/s 2024-07-26T11:09:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:reconnect 2024-07-26T11:09:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:48 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:reconnect,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:48 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:09:49.770 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:rejoin 2024-07-26T11:09:49.770 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:49.770 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-mon[37273]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:09:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:rejoin 2024-07-26T11:09:50.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:49 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:50.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:49 smithi160 ceph-mon[36715]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: *** Caught signal (Aborted) ** 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: in thread 7fe621136640 thread_name:ms_dispatch 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe62650d6f0] 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe62655a94c] 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 3: raise() 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 4: abort() 2024-07-26T11:09:50.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe6264f771b] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 6: /lib64/libc.so.6(+0x37386) [0x7fe626506386] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55895c5696b9] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55895c72780c] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55895c6672ca] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55895c637753] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55895c638a1b] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55895c61c752] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55895c56aa56] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55895c56e637] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55895c56ef5e] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55895c55203a] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe626dda407] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 18: (DispatchQueue::entry()+0x60f) [0x7fe626ddacf9] 2024-07-26T11:09:50.137 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe626eb75a1] 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 20: (Thread::entry_wrapper()+0x43) [0x7fe626c64cdd] 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe626c64cf9] 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe626558c02] 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe6265ddc40] 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:49.768+0000 7fe621136640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: in thread 7fe621136640 thread_name:ms_dispatch 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:50.138 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe62650d6f0] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe62655a94c] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 3: raise() 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 4: abort() 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe6264f771b] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 6: /lib64/libc.so.6(+0x37386) [0x7fe626506386] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55895c5696b9] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55895c72780c] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55895c6672ca] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55895c637753] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55895c638a1b] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55895c61c752] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55895c56aa56] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55895c56e637] 2024-07-26T11:09:50.139 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55895c56ef5e] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55895c55203a] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe626dda407] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 18: (DispatchQueue::entry()+0x60f) [0x7fe626ddacf9] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe626eb75a1] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 20: (Thread::entry_wrapper()+0x43) [0x7fe626c64cdd] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe626c64cf9] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe626558c02] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe6265ddc40] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 0> 2024-07-26T11:09:49.768+0000 7fe621136640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: in thread 7fe621136640 thread_name:ms_dispatch 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe62650d6f0] 2024-07-26T11:09:50.140 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe62655a94c] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 3: raise() 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 4: abort() 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe6264f771b] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 6: /lib64/libc.so.6(+0x37386) [0x7fe626506386] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55895c5696b9] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55895c72780c] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55895c6672ca] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55895c637753] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55895c638a1b] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55895c61c752] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55895c56aa56] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55895c56e637] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55895c56ef5e] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55895c55203a] 2024-07-26T11:09:50.141 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe626dda407] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 18: (DispatchQueue::entry()+0x60f) [0x7fe626ddacf9] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe626eb75a1] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 20: (Thread::entry_wrapper()+0x43) [0x7fe626c64cdd] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe626c64cf9] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe626558c02] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe6265ddc40] 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:50.142 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 0> 2024-07-26T11:09:49.768+0000 7fe621136640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: in thread 7fe621136640 thread_name:ms_dispatch 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe62650d6f0] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe62655a94c] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 3: raise() 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 4: abort() 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe6264f771b] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 6: /lib64/libc.so.6(+0x37386) [0x7fe626506386] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55895c5696b9] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55895c72780c] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55895c6672ca] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55895c637753] 2024-07-26T11:09:50.143 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55895c638a1b] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55895c61c752] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55895c56aa56] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55895c56e637] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55895c56ef5e] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55895c55203a] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe626dda407] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 18: (DispatchQueue::entry()+0x60f) [0x7fe626ddacf9] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe626eb75a1] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 20: (Thread::entry_wrapper()+0x43) [0x7fe626c64cdd] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe626c64cf9] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe626558c02] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe6265ddc40] 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c[73527]: 2024-07-26T11:09:50.144 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:49 smithi183 podman[74302]: 2024-07-26 11:09:49.838848079 +0000 UTC m=+0.029119955 container died 318f2e570fa95a4e0c31d05a41046d20e390a9620af82b5859e4eead29095142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T11:09:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:rejoin 2024-07-26T11:09:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:49 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=c=up:rejoin,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:49 smithi144 ceph-mon[32129]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-26T11:09:50.753 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:50 smithi183 podman[74302]: 2024-07-26 11:09:50.481638912 +0000 UTC m=+0.671910782 container remove 318f2e570fa95a4e0c31d05a41046d20e390a9620af82b5859e4eead29095142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:09:50.753 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:50 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:09:50.754 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:50 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:09:50.754 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:09:50 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.961s CPU time. 2024-07-26T11:09:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:50 smithi160 ceph-mon[36715]: pgmap v697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 122 B/s wr, 6 op/s 2024-07-26T11:09:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:50 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:51.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:50 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:active 2024-07-26T11:09:51.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:50 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:51.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:50 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:51.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 234.529773 secs 2024-07-26T11:09:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:50 smithi183 ceph-mon[37273]: pgmap v697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 122 B/s wr, 6 op/s 2024-07-26T11:09:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:50 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:50 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:active 2024-07-26T11:09:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:50 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:50 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 234.529773 secs 2024-07-26T11:09:51.145 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:50 smithi144 ceph-mon[32129]: pgmap v697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 122 B/s wr, 6 op/s 2024-07-26T11:09:51.145 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:50 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:09:51.145 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:50 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6836/3769917474,v1:172.21.15.183:6837/3769917474] up:active 2024-07-26T11:09:51.145 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:50 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 2 up:standby-replay 6 up:standby 2024-07-26T11:09:51.145 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:50 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=c=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby 2024-07-26T11:09:51.145 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 234.529773 secs 2024-07-26T11:09:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:52 smithi160 ceph-mon[36715]: pgmap v698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 245 B/s wr, 7 op/s 2024-07-26T11:09:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:52 smithi183 ceph-mon[37273]: pgmap v698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 245 B/s wr, 7 op/s 2024-07-26T11:09:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:52 smithi144 ceph-mon[32129]: pgmap v698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 245 B/s wr, 7 op/s 2024-07-26T11:09:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:54 smithi160 ceph-mon[36715]: pgmap v699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 238 B/s wr, 7 op/s 2024-07-26T11:09:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:54 smithi183 ceph-mon[37273]: pgmap v699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 238 B/s wr, 7 op/s 2024-07-26T11:09:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:54 smithi144 ceph-mon[32129]: pgmap v699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 238 B/s wr, 7 op/s 2024-07-26T11:09:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 239.529897 secs 2024-07-26T11:09:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 239.529897 secs 2024-07-26T11:09:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 239.529897 secs 2024-07-26T11:09:56.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:56.759+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:56.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:56.760+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:56.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:56.760+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:56.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:56.760+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:09:56.760+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:56 smithi160 ceph-mon[36715]: pgmap v700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 197 B/s wr, 7 op/s 2024-07-26T11:09:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:56 smithi183 ceph-mon[37273]: pgmap v700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 197 B/s wr, 7 op/s 2024-07-26T11:09:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:56 smithi144 ceph-mon[32129]: pgmap v700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 197 B/s wr, 7 op/s 2024-07-26T11:09:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:09:58 smithi160 ceph-mon[36715]: pgmap v701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:09:58 smithi183 ceph-mon[37273]: pgmap v701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:09:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:09:58 smithi144 ceph-mon[32129]: pgmap v701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:10:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: pgmap v702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: daemon mds.f on smithi183 is in unknown state 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: daemon mds.i on smithi183 is in error state 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: mds.h(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:10:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 244.530016 secs 2024-07-26T11:10:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: pgmap v702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-07-26T11:10:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: daemon mds.f on smithi183 is in unknown state 2024-07-26T11:10:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: daemon mds.i on smithi183 is in error state 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: mds.h(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 244.530016 secs 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:10:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:10:00 smithi183 systemd[1]: Stopped Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:10:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Consumed 1.961s CPU time. 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:10:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Start request repeated too quickly. 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:10:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T11:10:01.136 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 26 11:10:00 smithi183 systemd[1]: Failed to start Ceph mds.c for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: pgmap v702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-07-26T11:10:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: daemon mds.f on smithi183 is in unknown state 2024-07-26T11:10:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: daemon mds.i on smithi183 is in error state 2024-07-26T11:10:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T11:10:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: mds.h(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:10:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 244.530016 secs 2024-07-26T11:10:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:02 smithi160 ceph-mon[36715]: pgmap v703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:10:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:02 smithi183 ceph-mon[37273]: pgmap v703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:10:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:02 smithi144 ceph-mon[32129]: pgmap v703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:10:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:04 smithi160 ceph-mon[36715]: pgmap v704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:10:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:04 smithi183 ceph-mon[37273]: pgmap v704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:10:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:04 smithi144 ceph-mon[32129]: pgmap v704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:10:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 249.530121 secs 2024-07-26T11:10:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:05 smithi160 ceph-mon[36715]: Replacing daemon mds.c as rank 1 with standby daemon mds.f 2024-07-26T11:10:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:05 smithi160 ceph-mon[36715]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T11:10:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:05 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:05 smithi160 ceph-mon[36715]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T11:10:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:05 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 249.530121 secs 2024-07-26T11:10:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:05 smithi183 ceph-mon[37273]: Replacing daemon mds.c as rank 1 with standby daemon mds.f 2024-07-26T11:10:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:05 smithi183 ceph-mon[37273]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T11:10:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:05 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:05 smithi183 ceph-mon[37273]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T11:10:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:05 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 249.530121 secs 2024-07-26T11:10:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:05 smithi144 ceph-mon[32129]: Replacing daemon mds.c as rank 1 with standby daemon mds.f 2024-07-26T11:10:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:05 smithi144 ceph-mon[32129]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T11:10:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:05 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:05 smithi144 ceph-mon[32129]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T11:10:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:05 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:06 smithi160 ceph-mon[36715]: pgmap v706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:06 smithi183 ceph-mon[37273]: pgmap v706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:06 smithi144 ceph-mon[32129]: pgmap v706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:08.084 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:07 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:resolve 2024-07-26T11:10:08.084 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:07 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:07 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:resolve 2024-07-26T11:10:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:07 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:07 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:resolve 2024-07-26T11:10:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:07 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:08 smithi183 ceph-mon[37273]: pgmap v707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:08 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:reconnect 2024-07-26T11:10:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:08 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:08 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:10:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:08 smithi144 ceph-mon[32129]: pgmap v707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:08 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:reconnect 2024-07-26T11:10:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:08 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:08 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:10:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:08 smithi160 ceph-mon[36715]: pgmap v707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:08 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:reconnect 2024-07-26T11:10:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:08 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:08 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:10:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:rejoin 2024-07-26T11:10:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-mon[37273]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: *** Caught signal (Aborted) ** 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: in thread 7f82b5762640 thread_name:ms_dispatch 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f82bab396f0] 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f82bab8694c] 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 3: raise() 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 4: abort() 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 5: /lib64/libc.so.6(+0x2871b) [0x7f82bab2371b] 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 6: /lib64/libc.so.6(+0x37386) [0x7f82bab32386] 2024-07-26T11:10:10.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5653391c56b9] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56533938380c] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5653392c32ca] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565339293753] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565339294a1b] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565339278752] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5653391c6a56] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5653391ca637] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5653391caf5e] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5653391ae03a] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f82bb406407] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 18: (DispatchQueue::entry()+0x60f) [0x7f82bb406cf9] 2024-07-26T11:10:10.137 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f82bb4e35a1] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 20: (Thread::entry_wrapper()+0x43) [0x7f82bb290cdd] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 21: (Thread::_entry_func(void*)+0xd) [0x7f82bb290cf9] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 22: /lib64/libc.so.6(+0x89c02) [0x7f82bab84c02] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f82bac09c40] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:09.883+0000 7f82b5762640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: in thread 7f82b5762640 thread_name:ms_dispatch 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f82bab396f0] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f82bab8694c] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 3: raise() 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 4: abort() 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 5: /lib64/libc.so.6(+0x2871b) [0x7f82bab2371b] 2024-07-26T11:10:10.138 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 6: /lib64/libc.so.6(+0x37386) [0x7f82bab32386] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5653391c56b9] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56533938380c] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5653392c32ca] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565339293753] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565339294a1b] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565339278752] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5653391c6a56] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5653391ca637] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5653391caf5e] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5653391ae03a] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f82bb406407] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 18: (DispatchQueue::entry()+0x60f) [0x7f82bb406cf9] 2024-07-26T11:10:10.139 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f82bb4e35a1] 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 20: (Thread::entry_wrapper()+0x43) [0x7f82bb290cdd] 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 21: (Thread::_entry_func(void*)+0xd) [0x7f82bb290cf9] 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 22: /lib64/libc.so.6(+0x89c02) [0x7f82bab84c02] 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f82bac09c40] 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 0> 2024-07-26T11:10:09.883+0000 7f82b5762640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: in thread 7f82b5762640 thread_name:ms_dispatch 2024-07-26T11:10:10.140 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f82bab396f0] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f82bab8694c] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 3: raise() 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 4: abort() 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 5: /lib64/libc.so.6(+0x2871b) [0x7f82bab2371b] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 6: /lib64/libc.so.6(+0x37386) [0x7f82bab32386] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5653391c56b9] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56533938380c] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5653392c32ca] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565339293753] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565339294a1b] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565339278752] 2024-07-26T11:10:10.141 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5653391c6a56] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5653391ca637] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5653391caf5e] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5653391ae03a] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f82bb406407] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 18: (DispatchQueue::entry()+0x60f) [0x7f82bb406cf9] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f82bb4e35a1] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 20: (Thread::entry_wrapper()+0x43) [0x7f82bb290cdd] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 21: (Thread::_entry_func(void*)+0xd) [0x7f82bb290cf9] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 22: /lib64/libc.so.6(+0x89c02) [0x7f82bab84c02] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f82bac09c40] 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:10.142 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 0> 2024-07-26T11:10:09.883+0000 7f82b5762640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: in thread 7f82b5762640 thread_name:ms_dispatch 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f82bab396f0] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f82bab8694c] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 3: raise() 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 4: abort() 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 5: /lib64/libc.so.6(+0x2871b) [0x7f82bab2371b] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 6: /lib64/libc.so.6(+0x37386) [0x7f82bab32386] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5653391c56b9] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56533938380c] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5653392c32ca] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565339293753] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565339294a1b] 2024-07-26T11:10:10.143 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565339278752] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5653391c6a56] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5653391ca637] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5653391caf5e] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5653391ae03a] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f82bb406407] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 18: (DispatchQueue::entry()+0x60f) [0x7f82bb406cf9] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f82bb4e35a1] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 20: (Thread::entry_wrapper()+0x43) [0x7f82bb290cdd] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 21: (Thread::_entry_func(void*)+0xd) [0x7f82bb290cf9] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 22: /lib64/libc.so.6(+0x89c02) [0x7f82bab84c02] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f82bac09c40] 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74220]: 2024-07-26T11:10:10.144 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:09 smithi183 podman[74350]: 2024-07-26 11:10:09.945443349 +0000 UTC m=+0.019231711 container died 53841ccf3f247bd5a0bcf0a783f9f1f6898d995b2aefd4de1f1d8fa7e1f6abf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724) 2024-07-26T11:10:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:09 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:rejoin 2024-07-26T11:10:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:09 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:09 smithi144 ceph-mon[32129]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:10:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:09 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:rejoin 2024-07-26T11:10:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:09 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:10.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:09 smithi160 ceph-mon[36715]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:10:10.858 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:10 smithi183 podman[74350]: 2024-07-26 11:10:10.526791025 +0000 UTC m=+0.600579450 container remove 53841ccf3f247bd5a0bcf0a783f9f1f6898d995b2aefd4de1f1d8fa7e1f6abf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-26T11:10:10.858 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:10 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:10:10.858 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:10 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:10:10.859 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:10 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.002s CPU time. 2024-07-26T11:10:11.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:10 smithi144 ceph-mon[32129]: pgmap v708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:10:11.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:10 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:11.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:10 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:active 2024-07-26T11:10:11.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:10 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:11.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:10 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:11.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 254.530250 secs 2024-07-26T11:10:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:10 smithi183 ceph-mon[37273]: pgmap v708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:10:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:10 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:10 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:active 2024-07-26T11:10:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:10 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:10 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 254.530250 secs 2024-07-26T11:10:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:10 smithi160 ceph-mon[36715]: pgmap v708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:10:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:10 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:10 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/4207656406,v1:172.21.15.183:6835/4207656406] up:active 2024-07-26T11:10:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:10 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:10 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 254.530250 secs 2024-07-26T11:10:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:11.764+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:11.765+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:11.765+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:11.765+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:11.765+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:12 smithi183 ceph-mon[37273]: pgmap v709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:10:13.160 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:12 smithi144 ceph-mon[32129]: pgmap v709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:10:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:12 smithi160 ceph-mon[36715]: pgmap v709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:10:14.440 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:10:14.906 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:14 smithi183 ceph-mon[37273]: pgmap v710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:15.136 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:14 smithi144 ceph-mon[32129]: pgmap v710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:15.153 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:14 smithi160 ceph-mon[36715]: pgmap v710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 259.530360 secs 2024-07-26T11:10:16.147 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 259.530360 secs 2024-07-26T11:10:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 259.530360 secs 2024-07-26T11:10:17.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:16 smithi144 ceph-mon[32129]: pgmap v711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 7 op/s 2024-07-26T11:10:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:16 smithi183 ceph-mon[37273]: pgmap v711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 7 op/s 2024-07-26T11:10:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:16 smithi160 ceph-mon[36715]: pgmap v711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 7 op/s 2024-07-26T11:10:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 -- 172.21.15.144:0/2620215599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 msgr2=0x7f98e010fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 --2- 172.21.15.144:0/2620215599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e010fec0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009ca0 tx=0x7f98d802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 -- 172.21.15.144:0/2620215599 shutdown_connections 2024-07-26T11:10:17.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 --2- 172.21.15.144:0/2620215599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e010fec0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 --2- 172.21.15.144:0/2620215599 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 --2- 172.21.15.144:0/2620215599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e0103400 0x7f98e0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 -- 172.21.15.144:0/2620215599 >> 172.21.15.144:0/2620215599 conn(0x7f98e00fee90 msgr2=0x7f98e01012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 -- 172.21.15.144:0/2620215599 shutdown_connections 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.327+0000 7f98df200640 1 -- 172.21.15.144:0/2620215599 wait complete. 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.328+0000 7f98df200640 1 Processor -- start 2024-07-26T11:10:17.330 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 -- start start 2024-07-26T11:10:17.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e0103400 0x7f98e0075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e0075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0113360 con 0x7f98e0103400 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01131e0 con 0x7f98e0108d40 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.329+0000 7f98df200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01134e0 con 0x7f98e0109690 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dde00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e0103400 0x7f98e0075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:33240/0 (socket says 172.21.15.144:33240) 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 -- 172.21.15.144:0/3368901803 learned_addr learned my addr 172.21.15.144:0/3368901803 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:17.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98de800640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e0075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 -- 172.21.15.144:0/3368901803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 msgr2=0x7f98e0075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e0075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 -- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e0103400 msgr2=0x7f98e0075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e0103400 0x7f98e0075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98dd400640 1 -- 172.21.15.144:0/3368901803 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e01a7f90 con 0x7f98e0108d40 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.330+0000 7f98de800640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.331+0000 7f98dd400640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0075680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00e9c0 tx=0x7f98cc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.331+0000 7f98d6a00640 1 -- 172.21.15.144:0/3368901803 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00ccc0 con 0x7f98e0108d40 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.331+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e01a8280 con 0x7f98e0108d40 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.331+0000 7f98d6a00640 1 -- 172.21.15.144:0/3368901803 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98cc0054d0 con 0x7f98e0108d40 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.331+0000 7f98d6a00640 1 -- 172.21.15.144:0/3368901803 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc010430 con 0x7f98e0108d40 2024-07-26T11:10:17.334 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.331+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e01a87c0 con 0x7f98e0108d40 2024-07-26T11:10:17.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.334+0000 7f98ebbcb740 1 Processor -- start 2024-07-26T11:10:17.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 -- start start 2024-07-26T11:10:17.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09aafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09ab510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09aba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0921b90 con 0x55ebc0606070 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0911090 con 0x55ebc05e5390 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ebc09150d0 con 0x55ebc05dd1b0 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09ab510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:52850/0 (socket says 172.21.15.144:52850) 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98dde00640 1 -- 172.21.15.144:0/2720364153 learned_addr learned my addr 172.21.15.144:0/2720364153 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.335+0000 7f98de800640 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09aafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98d6a00640 1 -- 172.21.15.144:0/3368901803 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f98cc005020 con 0x7f98e0108d40 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98dde00640 1 -- 172.21.15.144:0/2720364153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 msgr2=0x55ebc09aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98dde00640 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98dde00640 1 -- 172.21.15.144:0/2720364153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 msgr2=0x55ebc09aafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98dde00640 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09aafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98dde00640 1 -- 172.21.15.144:0/2720364153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc0671650 con 0x55ebc05dd1b0 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.336+0000 7f98dde00640 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09ab510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009c90 tx=0x7f98d0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=869323d431e06e9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.337+0000 7f98cb400640 1 -- 172.21.15.144:0/2720364153 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0005220 con 0x55ebc05dd1b0 2024-07-26T11:10:17.339 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.337+0000 7f98cb400640 1 -- 172.21.15.144:0/2720364153 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98d0004c50 con 0x55ebc05dd1b0 2024-07-26T11:10:17.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.337+0000 7f98cb400640 1 -- 172.21.15.144:0/2720364153 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d00196d0 con 0x55ebc05dd1b0 2024-07-26T11:10:17.340 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.337+0000 7f98d6a00640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98c40b3dc0 0x7f98c40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.337+0000 7f98d6a00640 1 -- 172.21.15.144:0/3368901803 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10770+0+0 (secure 0 0 0) 0x7f98cc0e4b70 con 0x7f98e0108d40 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2720364153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 msgr2=0x55ebc09ab510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09ab510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009c90 tx=0x7f98d0011810 comp rx=0 tx=0).stop 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98de800640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98c40b3dc0 0x7f98c40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2720364153 shutdown_connections 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09ab510 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2720364153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09aafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2720364153 >> 172.21.15.144:0/2720364153 conn(0x55ebc05fa460 msgr2=0x55ebc09a8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2720364153 shutdown_connections 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.338+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2720364153 wait complete. 2024-07-26T11:10:17.341 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.339+0000 7f98de800640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98c40b3dc0 0x7f98c40b62d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006010 tx=0x7f98d8009800 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:17.342 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.339+0000 7f98ebbcb740 1 -- start start 2024-07-26T11:10:17.342 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.339+0000 7f98ebbcb740 1 Processor -- start 2024-07-26T11:10:17.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09b6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc0606070 0x55ebc09b7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55ebc0774700 con 0x55ebc05e5390 2024-07-26T11:10:17.343 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ebc08ad660 con 0x55ebc0606070 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ebc09b97b0 con 0x55ebc05dd1b0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98ebbcb740 10 client.0 _ll_register_callbacks cb 0x55ebc08bd110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09b6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:52858/0 (socket says 172.21.15.144:52858) 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 -- 172.21.15.144:0/2251781994 learned_addr learned my addr 172.21.15.144:0/2251781994 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dd400640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98de800640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc0606070 0x55ebc09b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 -- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc0606070 msgr2=0x55ebc09b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc0606070 0x55ebc09b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 -- 172.21.15.144:0/2251781994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 msgr2=0x55ebc09b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.340+0000 7f98dde00640 1 -- 172.21.15.144:0/2251781994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc08c4800 con 0x55ebc05dd1b0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98dde00640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09b6600 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f98d0000c00 tx=0x7f98d0005030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d04015517d724429 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:10:17.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98bfe00640 1 -- 172.21.15.144:0/2251781994 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0029070 con 0x55ebc05dd1b0 2024-07-26T11:10:17.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98d0004c50 con 0x55ebc05dd1b0 2024-07-26T11:10:17.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98bfe00640 1 -- 172.21.15.144:0/2251781994 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 10770+0+0 (secure 0 0 0) 0x7f98d0023630 con 0x55ebc05dd1b0 2024-07-26T11:10:17.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98bfe00640 1 -- 172.21.15.144:0/2251781994 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7f98d0035050 con 0x55ebc05dd1b0 2024-07-26T11:10:17.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.341+0000 7f98ebbcb740 10 client.26299 fetch_fsmap learned FSMap version 137 2024-07-26T11:10:17.345 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x55ebc09c5410 con 0x55ebc05dd1b0 2024-07-26T11:10:17.346 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98bfe00640 1 -- 172.21.15.144:0/2251781994 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98d0004c50 con 0x55ebc05dd1b0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98bfe00640 1 -- 172.21.15.144:0/2251781994 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 137) ==== 7144+0+0 (secure 0 0 0) 0x7f98d00194c0 con 0x55ebc05dd1b0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 10 client.26299 fetch_fsmap finished waiting for FSMap version 137 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 10 client.26299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 20 client.26299 populate_metadata read hostname 'smithi144' 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc05c49f0 0x55ebc09c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 4 client.26299 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc0923ae0 con 0x55ebc05c49f0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.342+0000 7f98dd400640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc05c49f0 0x55ebc09c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.343+0000 7f98dd400640 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc05c49f0 0x55ebc09c87d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3346eeb41eda9a6 server_cookie=aa235cbfc6b18990 in_seq=0 out_seq=0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.343+0000 7f98bfe00640 10 client.26299 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98bfe00640 1 -- 172.21.15.144:0/2251781994 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ebc0923ae0 con 0x55ebc05c49f0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98bfe00640 10 client.26299 handle_command_reply: tid=0 2024-07-26T11:10:17.347 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 client.26299 shutdown 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 msgr2=0x55ebc09b6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09b6600 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f98d0000c00 tx=0x7f98d0005030 comp rx=0 tx=0).stop 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 shutdown_connections 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc05c49f0 0x55ebc09c87d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc0606070 0x55ebc09b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/2251781994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09b6600 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98bfe00640 0 client.26299 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 >> 172.21.15.144:0/2251781994 conn(0x55ebc05fa460 msgr2=0x55ebc09ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 shutdown_connections 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.344+0000 7f98ebbcb740 1 -- 172.21.15.144:0/2251781994 wait complete. 2024-07-26T11:10:17.348 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.345+0000 7f98ebbcb740 20 client.26299 trim_cache size 0 max 16384 2024-07-26T11:10:17.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.363+0000 7f98ebbcb740 1 Processor -- start 2024-07-26T11:10:17.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.363+0000 7f98ebbcb740 1 -- start start 2024-07-26T11:10:17.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09acad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01b40b0 con 0x55ebc0606070 2024-07-26T11:10:17.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01b4780 con 0x55ebc05e5390 2024-07-26T11:10:17.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01b4e50 con 0x55ebc05dd1b0 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98de800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98dde00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98de800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45306/0 (socket says 172.21.15.144:45306) 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98de800640 1 -- 172.21.15.144:0/1006042721 learned_addr learned my addr 172.21.15.144:0/1006042721 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.364+0000 7f98dd400640 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98dde00640 1 -- 172.21.15.144:0/1006042721 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 msgr2=0x55ebc09b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98dde00640 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98dde00640 1 -- 172.21.15.144:0/1006042721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 msgr2=0x55ebc09cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98dde00640 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98dde00640 1 -- 172.21.15.144:0/1006042721 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc09c8b40 con 0x55ebc05dd1b0 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98dde00640 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09acad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f98d0011850 tx=0x7f98d0005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67dc802fdb570f02 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98caa00640 1 -- 172.21.15.144:0/1006042721 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0029040 con 0x55ebc05dd1b0 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98caa00640 1 -- 172.21.15.144:0/1006042721 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98d0005430 con 0x55ebc05dd1b0 2024-07-26T11:10:17.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.365+0000 7f98caa00640 1 -- 172.21.15.144:0/1006042721 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0019430 con 0x55ebc05dd1b0 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 -- 172.21.15.144:0/1006042721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 msgr2=0x55ebc09acad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09acad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f98d0011850 tx=0x7f98d0005b90 comp rx=0 tx=0).stop 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 -- 172.21.15.144:0/1006042721 shutdown_connections 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc0606070 0x55ebc09cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05e5390 0x55ebc09b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/1006042721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc05dd1b0 0x55ebc09acad0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 -- 172.21.15.144:0/1006042721 >> 172.21.15.144:0/1006042721 conn(0x55ebc05fa460 msgr2=0x55ebc09b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.369 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 -- 172.21.15.144:0/1006042721 shutdown_connections 2024-07-26T11:10:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.366+0000 7f98ebbcb740 1 -- 172.21.15.144:0/1006042721 wait complete. 2024-07-26T11:10:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.367+0000 7f98ebbcb740 1 -- start start 2024-07-26T11:10:17.370 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.367+0000 7f98ebbcb740 1 Processor -- start 2024-07-26T11:10:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05dd1b0 0x55ebc09bba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09bbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.371 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 0x55ebc09bc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01b1390 con 0x55ebc05e5390 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01b27a0 con 0x55ebc05dd1b0 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01b1a00 con 0x55ebc0606070 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98ebbcb740 10 client.0 _ll_register_callbacks cb 0x7f98d00049d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 0x55ebc09bc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 0x55ebc09bc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:52872/0 (socket says 172.21.15.144:52872) 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 -- 172.21.15.144:0/3127640414 learned_addr learned my addr 172.21.15.144:0/3127640414 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98dd400640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09bbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98dde00640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05dd1b0 0x55ebc09bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 -- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05dd1b0 msgr2=0x55ebc09bba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05dd1b0 0x55ebc09bba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 -- 172.21.15.144:0/3127640414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 msgr2=0x55ebc09bbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:17.372 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09bbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.368+0000 7f98de800640 1 -- 172.21.15.144:0/3127640414 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc0a2f090 con 0x55ebc0606070 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98de800640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 0x55ebc09bc500 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f98d8003bf0 tx=0x7f98d80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8618943ad6f71d80 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98bfe00640 1 -- 172.21.15.144:0/3127640414 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8059060 con 0x55ebc0606070 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98bfe00640 1 -- 172.21.15.144:0/3127640414 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 10770+0+0 (secure 0 0 0) 0x7f98d80046d0 con 0x55ebc0606070 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98d0005430 con 0x55ebc0606070 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98bfe00640 1 -- 172.21.15.144:0/3127640414 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7f98d8060050 con 0x55ebc0606070 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98ebbcb740 10 client.26305 fetch_fsmap learned FSMap version 137 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.369+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x55ebc0a30350 con 0x55ebc0606070 2024-07-26T11:10:17.373 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98bfe00640 1 -- 172.21.15.144:0/3127640414 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98d0005430 con 0x55ebc0606070 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98bfe00640 1 -- 172.21.15.144:0/3127640414 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 137) ==== 7144+0+0 (secure 0 0 0) 0x7f98d802f820 con 0x55ebc0606070 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98ebbcb740 10 client.26305 fetch_fsmap finished waiting for FSMap version 137 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98ebbcb740 10 client.26305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98ebbcb740 20 client.26305 populate_metadata read hostname 'smithi144' 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc0599da0 0x55ebc0a32b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98ebbcb740 4 client.26305 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebc09a8360 con 0x55ebc0599da0 2024-07-26T11:10:17.375 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.370+0000 7f98dde00640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc0599da0 0x55ebc0a32b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98dde00640 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc0599da0 0x55ebc0a32b40 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f95886ef60a761a2 server_cookie=1a9c25989860862d in_seq=0 out_seq=0 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98bfe00640 10 client.26305 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98bfe00640 1 -- 172.21.15.144:0/3127640414 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f98d0032110 con 0x55ebc0599da0 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98bfe00640 10 client.26305 handle_command_reply: tid=0 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98ebbcb740 1 client.26305 shutdown 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 msgr2=0x55ebc09bc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 0x55ebc09bc500 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f98d8003bf0 tx=0x7f98d80095c0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 shutdown_connections 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.371+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebc0606070 0x55ebc09bc500 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc05e5390 0x55ebc09bbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55ebc0599da0 0x55ebc0a32b40 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 1 --2- 172.21.15.144:0/3127640414 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ebc05dd1b0 0x55ebc09bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 >> 172.21.15.144:0/3127640414 conn(0x55ebc05fa460 msgr2=0x55ebc09b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98bfe00640 0 client.26305 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 shutdown_connections 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 1 -- 172.21.15.144:0/3127640414 wait complete. 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.372+0000 7f98ebbcb740 20 client.26305 trim_cache size 0 max 16384 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:10:17.377 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:10:17.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:10:17.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:10:17.378 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:10:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98c40b3dc0 msgr2=0x7f98c40b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98c40b3dc0 0x7f98c40b62d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006010 tx=0x7f98d8009800 comp rx=0 tx=0).stop 2024-07-26T11:10:17.378 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 msgr2=0x7f98e0075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0075680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00e9c0 tx=0x7f98cc00ee80 comp rx=0 tx=0).stop 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 shutdown_connections 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f98c40b3dc0 0x7f98c40b62d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98e0109690 0x7f98e0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f98e0108d40 0x7f98e0075680 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 --2- 172.21.15.144:0/3368901803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e0103400 0x7f98e0075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 >> 172.21.15.144:0/3368901803 conn(0x7f98e00fee90 msgr2=0x7f98e01083b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 shutdown_connections 2024-07-26T11:10:17.379 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:17.373+0000 7f98df200640 1 -- 172.21.15.144:0/3368901803 wait complete. 2024-07-26T11:10:18.243 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:10:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:18 smithi144 ceph-mon[32129]: pgmap v712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:10:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:18 smithi160 ceph-mon[36715]: pgmap v712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:10:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:18 smithi183 ceph-mon[37273]: pgmap v712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:10:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:20 smithi144 ceph-mon[32129]: pgmap v713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:10:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 264.530481 secs 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:20 smithi183 ceph-mon[37273]: pgmap v713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 264.530481 secs 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:20 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:20 smithi183 systemd[1]: Stopped Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:20 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.002s CPU time. 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:20 smithi183 systemd[1]: Starting Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 podman[74454]: 2024-07-26 11:10:21.020577869 +0000 UTC m=+0.017483590 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:10:21.256 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 podman[74454]: 2024-07-26 11:10:21.126919176 +0000 UTC m=+0.123824880 container create 9d78e2b6e793b2c3d5f1dded885ddce7ca6251184818d75c8a68b3d181cff912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T11:10:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:20 smithi160 ceph-mon[36715]: pgmap v713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:10:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 264.530481 secs 2024-07-26T11:10:21.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 podman[74454]: 2024-07-26 11:10:21.380034338 +0000 UTC m=+0.376940043 container init 9d78e2b6e793b2c3d5f1dded885ddce7ca6251184818d75c8a68b3d181cff912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:10:21.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 podman[74454]: 2024-07-26 11:10:21.382522024 +0000 UTC m=+0.379427733 container start 9d78e2b6e793b2c3d5f1dded885ddce7ca6251184818d75c8a68b3d181cff912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T11:10:21.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mds[74473]: -- 172.21.15.183:0/918322401 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606a107b860 con 0x5606a521da80 2024-07-26T11:10:21.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: starting mds.f at 2024-07-26T11:10:21.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 bash[74454]: 9d78e2b6e793b2c3d5f1dded885ddce7ca6251184818d75c8a68b3d181cff912 2024-07-26T11:10:21.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:21 smithi183 systemd[1]: Started Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: daemon mds.f restarted 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:boot 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:22.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:21 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: daemon mds.f restarted 2024-07-26T11:10:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:10:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T11:10:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:boot 2024-07-26T11:10:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:10:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:22.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:21 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: daemon mds.f restarted 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:boot 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:21 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:22 smithi144 ceph-mon[32129]: pgmap v715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:10:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:22 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:resolve 2024-07-26T11:10:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:22 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:22 smithi160 ceph-mon[36715]: pgmap v715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:10:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:22 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:resolve 2024-07-26T11:10:23.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:22 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:22 smithi183 ceph-mon[37273]: pgmap v715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:10:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:22 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:resolve 2024-07-26T11:10:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:22 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:24.765 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:24 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:reconnect 2024-07-26T11:10:24.765 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:24 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:24.765 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:24 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00199997 2024-07-26T11:10:24.765 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:24 smithi144 ceph-mon[32129]: pgmap v716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6 op/s 2024-07-26T11:10:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:24 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:reconnect 2024-07-26T11:10:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:24 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:24 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00199997 2024-07-26T11:10:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:24 smithi160 ceph-mon[36715]: pgmap v716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6 op/s 2024-07-26T11:10:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:24 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:reconnect 2024-07-26T11:10:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:24 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:24 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00199997 2024-07-26T11:10:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:24 smithi183 ceph-mon[37273]: pgmap v716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6 op/s 2024-07-26T11:10:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:rejoin 2024-07-26T11:10:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-mon[32129]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: *** Caught signal (Aborted) ** 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: in thread 7f9084a00640 thread_name:ms_dispatch 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f908ac3e6f0] 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f908ac8b94c] 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 3: raise() 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 4: abort() 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 5: /lib64/libc.so.6(+0x2871b) [0x7f908ac2871b] 2024-07-26T11:10:25.721 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 6: /lib64/libc.so.6(+0x37386) [0x7f908ac37386] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5651f6a606b9] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5651f6c1e80c] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5651f6b5e2ca] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5651f6b2e753] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5651f6b2fa1b] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5651f6b13752] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5651f6a61a56] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5651f6a65637] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5651f6a65f5e] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5651f6a4903a] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f908b79b407] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 18: (DispatchQueue::entry()+0x60f) [0x7f908b79bcf9] 2024-07-26T11:10:25.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f908b8785a1] 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 20: (Thread::entry_wrapper()+0x43) [0x7f908b625cdd] 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 21: (Thread::_entry_func(void*)+0xd) [0x7f908b625cf9] 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 22: /lib64/libc.so.6(+0x89c02) [0x7f908ac89c02] 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f908ad0ec40] 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.521+0000 7f9084a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: in thread 7f9084a00640 thread_name:ms_dispatch 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f908ac3e6f0] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f908ac8b94c] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 3: raise() 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 4: abort() 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 5: /lib64/libc.so.6(+0x2871b) [0x7f908ac2871b] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 6: /lib64/libc.so.6(+0x37386) [0x7f908ac37386] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5651f6a606b9] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5651f6c1e80c] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5651f6b5e2ca] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5651f6b2e753] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5651f6b2fa1b] 2024-07-26T11:10:25.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5651f6b13752] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5651f6a61a56] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5651f6a65637] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5651f6a65f5e] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5651f6a4903a] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f908b79b407] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 18: (DispatchQueue::entry()+0x60f) [0x7f908b79bcf9] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f908b8785a1] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 20: (Thread::entry_wrapper()+0x43) [0x7f908b625cdd] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 21: (Thread::_entry_func(void*)+0xd) [0x7f908b625cf9] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 22: /lib64/libc.so.6(+0x89c02) [0x7f908ac89c02] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f908ad0ec40] 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 0> 2024-07-26T11:10:25.521+0000 7f9084a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: in thread 7f9084a00640 thread_name:ms_dispatch 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f908ac3e6f0] 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f908ac8b94c] 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 3: raise() 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 4: abort() 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 5: /lib64/libc.so.6(+0x2871b) [0x7f908ac2871b] 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 6: /lib64/libc.so.6(+0x37386) [0x7f908ac37386] 2024-07-26T11:10:25.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5651f6a606b9] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5651f6c1e80c] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5651f6b5e2ca] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5651f6b2e753] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5651f6b2fa1b] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5651f6b13752] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5651f6a61a56] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5651f6a65637] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5651f6a65f5e] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5651f6a4903a] 2024-07-26T11:10:25.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f908b79b407] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 18: (DispatchQueue::entry()+0x60f) [0x7f908b79bcf9] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f908b8785a1] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 20: (Thread::entry_wrapper()+0x43) [0x7f908b625cdd] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 21: (Thread::_entry_func(void*)+0xd) [0x7f908b625cf9] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 22: /lib64/libc.so.6(+0x89c02) [0x7f908ac89c02] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f908ad0ec40] 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 0> 2024-07-26T11:10:25.521+0000 7f9084a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: in thread 7f9084a00640 thread_name:ms_dispatch 2024-07-26T11:10:25.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f908ac3e6f0] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f908ac8b94c] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 3: raise() 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 4: abort() 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 5: /lib64/libc.so.6(+0x2871b) [0x7f908ac2871b] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 6: /lib64/libc.so.6(+0x37386) [0x7f908ac37386] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5651f6a606b9] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5651f6c1e80c] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5651f6b5e2ca] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5651f6b2e753] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5651f6b2fa1b] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5651f6b13752] 2024-07-26T11:10:25.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5651f6a61a56] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5651f6a65637] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5651f6a65f5e] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5651f6a4903a] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f908b79b407] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 18: (DispatchQueue::entry()+0x60f) [0x7f908b79bcf9] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f908b8785a1] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 20: (Thread::entry_wrapper()+0x43) [0x7f908b625cdd] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 21: (Thread::_entry_func(void*)+0xd) [0x7f908b625cf9] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 22: /lib64/libc.so.6(+0x89c02) [0x7f908ac89c02] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f908ad0ec40] 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[76963]: 2024-07-26T11:10:25.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:25 smithi144 podman[105413]: 2024-07-26 11:10:25.671185194 +0000 UTC m=+0.036419344 container died f17952baa7f056250f0c392b5d038f75160695505bfa29a9ea979e6686c53194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T11:10:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:25 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:rejoin 2024-07-26T11:10:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:25 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:25.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:25 smithi160 ceph-mon[36715]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:10:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:25 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:rejoin 2024-07-26T11:10:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:25 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:25 smithi183 ceph-mon[37273]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:26 smithi144 podman[105413]: 2024-07-26 11:10:26.463295491 +0000 UTC m=+0.828529674 container remove f17952baa7f056250f0c392b5d038f75160695505bfa29a9ea979e6686c53194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:26 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:26 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:active 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:26 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:26 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 269.530563 secs 2024-07-26T11:10:26.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:26 smithi144 ceph-mon[32129]: pgmap v717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:10:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:26.782+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:26.782+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:26.782+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:26.782+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:26.782+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:26 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:26 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:active 2024-07-26T11:10:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:26 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:26 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 269.530563 secs 2024-07-26T11:10:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:26 smithi160 ceph-mon[36715]: pgmap v717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:10:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:26 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:26 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2179784040,v1:172.21.15.144:6835/2179784040] up:active 2024-07-26T11:10:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:26 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:26 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:26.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 269.530563 secs 2024-07-26T11:10:26.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:26 smithi183 ceph-mon[37273]: pgmap v717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:10:27.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:10:27.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.745s CPU time. 2024-07-26T11:10:27.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:10:27 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:10:27.169+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:10:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:28 smithi160 ceph-mon[36715]: pgmap v718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:10:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:28 smithi183 ceph-mon[37273]: pgmap v718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:10:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:28 smithi144 ceph-mon[32129]: pgmap v718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:10:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:30 smithi160 ceph-mon[36715]: pgmap v719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 274.530684 secs 2024-07-26T11:10:31.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:30 smithi183 ceph-mon[37273]: pgmap v719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:31.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 274.530684 secs 2024-07-26T11:10:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:30 smithi144 ceph-mon[32129]: pgmap v719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 274.530684 secs 2024-07-26T11:10:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:32 smithi160 ceph-mon[36715]: pgmap v720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 198 B/s wr, 7 op/s 2024-07-26T11:10:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:32 smithi183 ceph-mon[37273]: pgmap v720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 198 B/s wr, 7 op/s 2024-07-26T11:10:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:32 smithi144 ceph-mon[32129]: pgmap v720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 198 B/s wr, 7 op/s 2024-07-26T11:10:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:34 smithi160 ceph-mon[36715]: pgmap v721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:34 smithi183 ceph-mon[37273]: pgmap v721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:34 smithi144 ceph-mon[32129]: pgmap v721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 279.530784 secs 2024-07-26T11:10:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 279.530784 secs 2024-07-26T11:10:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 279.530784 secs 2024-07-26T11:10:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:36 smithi183 ceph-mon[37273]: pgmap v722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:36 smithi144 ceph-mon[32129]: pgmap v722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:37.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:10:37.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:36 smithi144 systemd[1]: Stopped Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:37.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.745s CPU time. 2024-07-26T11:10:37.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:36 smithi144 systemd[1]: Starting Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:10:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:36 smithi160 ceph-mon[36715]: pgmap v722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:10:37.720 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 podman[105511]: 2024-07-26 11:10:37.274449021 +0000 UTC m=+0.034436033 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:10:37.720 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 podman[105511]: 2024-07-26 11:10:37.397261757 +0000 UTC m=+0.157248788 container create 22d8ee358c9393e8459d109d9b652f65cd19f03d35af3640b00b8d4ab78c8f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T11:10:38.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 podman[105511]: 2024-07-26 11:10:37.725921066 +0000 UTC m=+0.485908100 container init 22d8ee358c9393e8459d109d9b652f65cd19f03d35af3640b00b8d4ab78c8f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS) 2024-07-26T11:10:38.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 podman[105511]: 2024-07-26 11:10:37.735911401 +0000 UTC m=+0.495898417 container start 22d8ee358c9393e8459d109d9b652f65cd19f03d35af3640b00b8d4ab78c8f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:10:38.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 bash[105511]: 22d8ee358c9393e8459d109d9b652f65cd19f03d35af3640b00b8d4ab78c8f21 2024-07-26T11:10:38.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 systemd[1]: Started Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:38.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 ceph-mds[105530]: -- 172.21.15.144:0/3562487942 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a0f3399860 con 0x55a0f753b180 2024-07-26T11:10:38.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:10:37 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: starting mds.j at 2024-07-26T11:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: pgmap v723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: daemon mds.j restarted 2024-07-26T11:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T11:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:boot 2024-07-26T11:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:10:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:10:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:38 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: pgmap v723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:10:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: daemon mds.j restarted 2024-07-26T11:10:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T11:10:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:boot 2024-07-26T11:10:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:10:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:10:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:38 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: pgmap v723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: daemon mds.j restarted 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:boot 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:10:39.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:38 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:40.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:39 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:resolve 2024-07-26T11:10:40.140 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:39 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:39 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:resolve 2024-07-26T11:10:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:39 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:39 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:resolve 2024-07-26T11:10:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:39 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:40 smithi144 ceph-mon[32129]: pgmap v725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:40 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:reconnect 2024-07-26T11:10:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:40 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:40 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:10:41.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 284.530922 secs 2024-07-26T11:10:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:40 smithi160 ceph-mon[36715]: pgmap v725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:40 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:reconnect 2024-07-26T11:10:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:40 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:40 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:10:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 284.530922 secs 2024-07-26T11:10:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:40 smithi183 ceph-mon[37273]: pgmap v725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:10:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:40 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:reconnect 2024-07-26T11:10:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:40 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:40 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:10:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 284.530922 secs 2024-07-26T11:10:41.800 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:41.797+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:41.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:41.798+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:41.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:41.798+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:41.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:41.798+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:41.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:41.798+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:42.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:41 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:rejoin 2024-07-26T11:10:42.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:41 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:42.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:41 smithi160 ceph-mon[36715]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:10:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:41 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:rejoin 2024-07-26T11:10:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:41 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:41 smithi144 ceph-mon[32129]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:10:42.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:rejoin 2024-07-26T11:10:42.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:42.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-mon[37273]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: *** Caught signal (Aborted) ** 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: in thread 7efeeefce640 thread_name:ms_dispatch 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efef43a56f0] 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efef43f294c] 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 3: raise() 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 4: abort() 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 5: /lib64/libc.so.6(+0x2871b) [0x7efef438f71b] 2024-07-26T11:10:42.344 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 6: /lib64/libc.so.6(+0x37386) [0x7efef439e386] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56069e89f6b9] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56069ea5d80c] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56069e99d2ca] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56069e96d753] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56069e96ea1b] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56069e952752] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56069e8a0a56] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56069e8a4637] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56069e8a4f5e] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56069e88803a] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efef4c72407] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 18: (DispatchQueue::entry()+0x60f) [0x7efef4c72cf9] 2024-07-26T11:10:42.345 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efef4d4f5a1] 2024-07-26T11:10:42.346 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 20: (Thread::entry_wrapper()+0x43) [0x7efef4afccdd] 2024-07-26T11:10:42.346 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 21: (Thread::_entry_func(void*)+0xd) [0x7efef4afccf9] 2024-07-26T11:10:42.346 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 22: /lib64/libc.so.6(+0x89c02) [0x7efef43f0c02] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efef4475c40] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:41.939+0000 7efeeefce640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: in thread 7efeeefce640 thread_name:ms_dispatch 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efef43a56f0] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efef43f294c] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 3: raise() 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 4: abort() 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 5: /lib64/libc.so.6(+0x2871b) [0x7efef438f71b] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 6: /lib64/libc.so.6(+0x37386) [0x7efef439e386] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56069e89f6b9] 2024-07-26T11:10:42.347 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56069ea5d80c] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56069e99d2ca] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56069e96d753] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56069e96ea1b] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56069e952752] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56069e8a0a56] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56069e8a4637] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56069e8a4f5e] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56069e88803a] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efef4c72407] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 18: (DispatchQueue::entry()+0x60f) [0x7efef4c72cf9] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efef4d4f5a1] 2024-07-26T11:10:42.348 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 20: (Thread::entry_wrapper()+0x43) [0x7efef4afccdd] 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 21: (Thread::_entry_func(void*)+0xd) [0x7efef4afccf9] 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 22: /lib64/libc.so.6(+0x89c02) [0x7efef43f0c02] 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efef4475c40] 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 0> 2024-07-26T11:10:41.939+0000 7efeeefce640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: in thread 7efeeefce640 thread_name:ms_dispatch 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efef43a56f0] 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efef43f294c] 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 3: raise() 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 4: abort() 2024-07-26T11:10:42.349 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 5: /lib64/libc.so.6(+0x2871b) [0x7efef438f71b] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 6: /lib64/libc.so.6(+0x37386) [0x7efef439e386] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56069e89f6b9] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56069ea5d80c] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56069e99d2ca] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56069e96d753] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56069e96ea1b] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56069e952752] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56069e8a0a56] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56069e8a4637] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56069e8a4f5e] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56069e88803a] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efef4c72407] 2024-07-26T11:10:42.350 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 18: (DispatchQueue::entry()+0x60f) [0x7efef4c72cf9] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efef4d4f5a1] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 20: (Thread::entry_wrapper()+0x43) [0x7efef4afccdd] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 21: (Thread::_entry_func(void*)+0xd) [0x7efef4afccf9] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 22: /lib64/libc.so.6(+0x89c02) [0x7efef43f0c02] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efef4475c40] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 0> 2024-07-26T11:10:41.939+0000 7efeeefce640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: in thread 7efeeefce640 thread_name:ms_dispatch 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efef43a56f0] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efef43f294c] 2024-07-26T11:10:42.351 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 3: raise() 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 4: abort() 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 5: /lib64/libc.so.6(+0x2871b) [0x7efef438f71b] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 6: /lib64/libc.so.6(+0x37386) [0x7efef439e386] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56069e89f6b9] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56069ea5d80c] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56069e99d2ca] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56069e96d753] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56069e96ea1b] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56069e952752] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56069e8a0a56] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56069e8a4637] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56069e8a4f5e] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56069e88803a] 2024-07-26T11:10:42.352 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efef4c72407] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 18: (DispatchQueue::entry()+0x60f) [0x7efef4c72cf9] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efef4d4f5a1] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 20: (Thread::entry_wrapper()+0x43) [0x7efef4afccdd] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 21: (Thread::_entry_func(void*)+0xd) [0x7efef4afccf9] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 22: /lib64/libc.so.6(+0x89c02) [0x7efef43f0c02] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efef4475c40] 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74469]: 2024-07-26T11:10:42.353 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:41 smithi183 podman[74508]: 2024-07-26 11:10:41.997572681 +0000 UTC m=+0.018364694 container died 9d78e2b6e793b2c3d5f1dded885ddce7ca6251184818d75c8a68b3d181cff912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T11:10:42.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:10:42 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:10:42.170+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:10:42.912 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:42 smithi183 podman[74508]: 2024-07-26 11:10:42.659063509 +0000 UTC m=+0.679855519 container remove 9d78e2b6e793b2c3d5f1dded885ddce7ca6251184818d75c8a68b3d181cff912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T11:10:42.912 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:42 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:10:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:42 smithi144 ceph-mon[32129]: pgmap v726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:10:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:42 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:42 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:active 2024-07-26T11:10:43.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:42 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:43.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:42 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:42 smithi160 ceph-mon[36715]: pgmap v726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:10:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:42 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:42 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:active 2024-07-26T11:10:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:42 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:42 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:43.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:42 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:10:43.385 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:42 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 1.946s CPU time. 2024-07-26T11:10:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:42 smithi183 ceph-mon[37273]: pgmap v726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:10:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:42 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:42 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2904346377,v1:172.21.15.183:6835/2904346377] up:active 2024-07-26T11:10:43.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:42 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:43.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:42 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:44 smithi144 ceph-mon[32129]: pgmap v727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 6 op/s 2024-07-26T11:10:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:44 smithi160 ceph-mon[36715]: pgmap v727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 6 op/s 2024-07-26T11:10:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:44 smithi183 ceph-mon[37273]: pgmap v727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 6 op/s 2024-07-26T11:10:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 289.531045 secs 2024-07-26T11:10:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 289.531045 secs 2024-07-26T11:10:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 289.531045 secs 2024-07-26T11:10:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:46 smithi144 ceph-mon[32129]: pgmap v728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:47.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:46 smithi183 ceph-mon[37273]: pgmap v728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:47.287 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:46 smithi160 ceph-mon[36715]: pgmap v728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:10:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:47 smithi144 ceph-mon[32129]: pgmap v729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:10:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:47 smithi144 ceph-mon[32129]: pgmap v730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 8 op/s 2024-07-26T11:10:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:10:48.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:10:48.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:48.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:48.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:47 smithi183 ceph-mon[37273]: pgmap v729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:10:48.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:47 smithi183 ceph-mon[37273]: pgmap v730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 8 op/s 2024-07-26T11:10:48.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:10:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:47 smithi160 ceph-mon[36715]: pgmap v729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:10:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:47 smithi160 ceph-mon[36715]: pgmap v730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 8 op/s 2024-07-26T11:10:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:10:48.700 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:50.152 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:50 smithi144 ceph-mon[32129]: pgmap v731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 6 op/s 2024-07-26T11:10:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:50 smithi160 ceph-mon[36715]: pgmap v731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 6 op/s 2024-07-26T11:10:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:50 smithi183 ceph-mon[37273]: pgmap v731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 6 op/s 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 -- 172.21.15.144:0/3165266273 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077cc0 msgr2=0x7f1f30113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 --2- 172.21.15.144:0/3165266273 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077cc0 0x7f1f30113a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009900 tx=0x7f1f1c02a410 comp rx=0 tx=0).stop 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 -- 172.21.15.144:0/3165266273 shutdown_connections 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 --2- 172.21.15.144:0/3165266273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30113fb0 0x7f1f301163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 --2- 172.21.15.144:0/3165266273 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077cc0 0x7f1f30113a70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 --2- 172.21.15.144:0/3165266273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30077370 0x7f1f30077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.092+0000 7f1f2f200640 1 -- 172.21.15.144:0/3165266273 >> 172.21.15.144:0/3165266273 conn(0x7f1f30101060 msgr2=0x7f1f301034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.093+0000 7f1f2f200640 1 -- 172.21.15.144:0/3165266273 shutdown_connections 2024-07-26T11:10:51.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.093+0000 7f1f2f200640 1 -- 172.21.15.144:0/3165266273 wait complete. 2024-07-26T11:10:51.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 Processor -- start 2024-07-26T11:10:51.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 -- start start 2024-07-26T11:10:51.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077370 0x7f1f301a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 0x7f1f301a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30113fb0 0x7f1f301a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1f30119910 con 0x7f1f30077cc0 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f30119790 con 0x7f1f30077370 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.094+0000 7f1f2f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1f30119a90 con 0x7f1f30113fb0 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 0x7f1f301a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 0x7f1f301a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49306/0 (socket says 172.21.15.144:49306) 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 -- 172.21.15.144:0/1522903426 learned_addr learned my addr 172.21.15.144:0/1522903426 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077370 0x7f1f301a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2e800640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30113fb0 0x7f1f301a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 -- 172.21.15.144:0/1522903426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30113fb0 msgr2=0x7f1f301a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30113fb0 0x7f1f301a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 -- 172.21.15.144:0/1522903426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077370 msgr2=0x7f1f301a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077370 0x7f1f301a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2d400640 1 -- 172.21.15.144:0/1522903426 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f301a62b0 con 0x7f1f30077cc0 2024-07-26T11:10:51.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2e800640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30113fb0 0x7f1f301a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:51.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.095+0000 7f1f2de00640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077370 0x7f1f301a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:51.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.096+0000 7f1f2d400640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 0x7f1f301a08e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c02a930 tx=0x7f1f1c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.096+0000 7f1f26a00640 1 -- 172.21.15.144:0/1522903426 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c03f030 con 0x7f1f30077cc0 2024-07-26T11:10:51.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.096+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f301a6540 con 0x7f1f30077cc0 2024-07-26T11:10:51.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.096+0000 7f1f26a00640 1 -- 172.21.15.144:0/1522903426 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f1c004490 con 0x7f1f30077cc0 2024-07-26T11:10:51.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.096+0000 7f1f26a00640 1 -- 172.21.15.144:0/1522903426 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c035070 con 0x7f1f30077cc0 2024-07-26T11:10:51.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.096+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f301acd00 con 0x7f1f30077cc0 2024-07-26T11:10:51.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.100+0000 7f1f3b7fa740 1 Processor -- start 2024-07-26T11:10:51.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.100+0000 7f1f26a00640 1 -- 172.21.15.144:0/1522903426 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f1f1c042030 con 0x7f1f30077cc0 2024-07-26T11:10:51.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.100+0000 7f1f3b7fa740 1 -- start start 2024-07-26T11:10:51.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57b9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57b9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5616f572fb90 con 0x5616f53eb1b0 2024-07-26T11:10:51.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5616f571f090 con 0x5616f53f3390 2024-07-26T11:10:51.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5616f57230d0 con 0x5616f5414070 2024-07-26T11:10:51.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57b9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49320/0 (socket says 172.21.15.144:49320) 2024-07-26T11:10:51.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f2de00640 1 -- 172.21.15.144:0/3982274399 learned_addr learned my addr 172.21.15.144:0/3982274399 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:51.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f2e800640 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.101+0000 7f1f2d400640 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57b9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2e800640 1 -- 172.21.15.144:0/3982274399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 msgr2=0x5616f57b9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2e800640 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57b9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2e800640 1 -- 172.21.15.144:0/3982274399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 msgr2=0x5616f57b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2e800640 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2e800640 1 -- 172.21.15.144:0/3982274399 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616f547f650 con 0x5616f5414070 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2de00640 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f26a00640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f140b3d40 0x7f1f140b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f2e800640 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57b8fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1f28005fd0 tx=0x7f1f28011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff15d70b695ee39d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.102+0000 7f1f26a00640 1 -- 172.21.15.144:0/1522903426 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 10942+0+0 (secure 0 0 0) 0x7f1f1c104c50 con 0x7f1f30077cc0 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.103+0000 7f1f1b000640 1 -- 172.21.15.144:0/3982274399 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28004d20 con 0x5616f5414070 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.103+0000 7f1f1b000640 1 -- 172.21.15.144:0/3982274399 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f2800f9a0 con 0x5616f5414070 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.103+0000 7f1f1b000640 1 -- 172.21.15.144:0/3982274399 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28004470 con 0x5616f5414070 2024-07-26T11:10:51.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.103+0000 7f1f2de00640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f140b3d40 0x7f1f140b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3982274399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 msgr2=0x5616f57b8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f2de00640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f140b3d40 0x7f1f140b6250 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009fd0 tx=0x7f1f20005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57b8fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1f28005fd0 tx=0x7f1f28011bf0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3982274399 shutdown_connections 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57b9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3982274399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57b8fd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3982274399 >> 172.21.15.144:0/3982274399 conn(0x5616f5408460 msgr2=0x5616f57b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3982274399 shutdown_connections 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.104+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3982274399 wait complete. 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 -- start start 2024-07-26T11:10:51.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 Processor -- start 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5616f5582700 con 0x5616f53eb1b0 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5616f56bb660 con 0x5616f53f3390 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.105+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5616f57c77b0 con 0x5616f5414070 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f3b7fa740 10 client.0 _ll_register_callbacks cb 0x5616f56cb110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49330/0 (socket says 172.21.15.144:49330) 2024-07-26T11:10:51.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2d400640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 -- 172.21.15.144:0/2750034770 learned_addr learned my addr 172.21.15.144:0/2750034770 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2e800640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 -- 172.21.15.144:0/2750034770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 msgr2=0x5616f57c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 -- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 msgr2=0x5616f57c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2de00640 1 -- 172.21.15.144:0/2750034770 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616f56d2800 con 0x5616f53eb1b0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.106+0000 7f1f2d400640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f2de00640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c4590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004620 tx=0x7f1f20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916efe7956501c58 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f0f400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f0f400640 1 -- 172.21.15.144:0/2750034770 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20009ce0 con 0x5616f53eb1b0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f0f400640 1 -- 172.21.15.144:0/2750034770 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 10942+0+0 (secure 0 0 0) 0x7f1f20003af0 con 0x5616f53eb1b0 2024-07-26T11:10:51.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f2800f9a0 con 0x5616f53eb1b0 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f0f400640 1 -- 172.21.15.144:0/2750034770 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) ==== 24+0+0 (secure 0 0 0) 0x7f1f20003cf0 con 0x5616f53eb1b0 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.107+0000 7f1f3b7fa740 10 client.16119 fetch_fsmap learned FSMap version 151 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=4+}) -- 0x5616f57d3420 con 0x5616f53eb1b0 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f0f400640 1 -- 172.21.15.144:0/2750034770 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f2800f9a0 con 0x5616f53eb1b0 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f0f400640 1 -- 172.21.15.144:0/2750034770 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 151) ==== 7144+0+0 (secure 0 0 0) 0x7f1f20027070 con 0x5616f53eb1b0 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 10 client.16119 fetch_fsmap finished waiting for FSMap version 151 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:10:51.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 20 client.16119 populate_metadata read hostname 'smithi144' 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53d29f0 0x5616f57d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 4 client.16119 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.108+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616f5731ae0 con 0x5616f53d29f0 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.109+0000 7f1f2d400640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53d29f0 0x5616f57d6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.109+0000 7f1f2d400640 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53d29f0 0x5616f57d6d00 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1a810c29b2bc744 server_cookie=15e98362d3bbfbbf in_seq=0 out_seq=0 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.109+0000 7f1f0f400640 10 client.16119 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f0f400640 1 -- 172.21.15.144:0/2750034770 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5616f5731ae0 con 0x5616f53d29f0 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f0f400640 10 client.16119 handle_command_reply: tid=0 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 client.16119 shutdown 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 msgr2=0x5616f57c4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c4590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004620 tx=0x7f1f20004730 comp rx=0 tx=0).stop 2024-07-26T11:10:51.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 shutdown_connections 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53d29f0 0x5616f57d6d00 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2750034770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c4590 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f0f400640 0 client.16119 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 >> 172.21.15.144:0/2750034770 conn(0x5616f5408460 msgr2=0x5616f57bc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 shutdown_connections 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.110+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2750034770 wait complete. 2024-07-26T11:10:51.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.111+0000 7f1f3b7fa740 20 client.16119 trim_cache size 0 max 16384 2024-07-26T11:10:51.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.129+0000 7f1f3b7fa740 1 Processor -- start 2024-07-26T11:10:51.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.129+0000 7f1f3b7fa740 1 -- start start 2024-07-26T11:10:51.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.129+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57ba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1f301b8600 con 0x5616f53eb1b0 2024-07-26T11:10:51.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f301b8cd0 con 0x5616f53f3390 2024-07-26T11:10:51.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1f301b93a0 con 0x5616f5414070 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49346/0 (socket says 172.21.15.144:49346) 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 -- 172.21.15.144:0/2981891681 learned_addr learned my addr 172.21.15.144:0/2981891681 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2d400640 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2e800640 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 -- 172.21.15.144:0/2981891681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 msgr2=0x5616f57d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 -- 172.21.15.144:0/2981891681 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 msgr2=0x5616f57ba710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57ba710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2de00640 1 -- 172.21.15.144:0/2981891681 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616f57c2a90 con 0x5616f53eb1b0 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.130+0000 7f1f2d400640 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57ba710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.131+0000 7f1f2de00640 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c6470 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1f20019040 tx=0x7f1f200092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6335514cf63d3b0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.131+0000 7f1f1a600640 1 -- 172.21.15.144:0/2981891681 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20003e90 con 0x5616f53eb1b0 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.131+0000 7f1f1a600640 1 -- 172.21.15.144:0/2981891681 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f200039c0 con 0x5616f53eb1b0 2024-07-26T11:10:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.131+0000 7f1f1a600640 1 -- 172.21.15.144:0/2981891681 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2001f430 con 0x5616f53eb1b0 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2981891681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 msgr2=0x5616f57c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c6470 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1f20019040 tx=0x7f1f200092a0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2981891681 shutdown_connections 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f5414070 0x5616f57d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53f3390 0x5616f57ba710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/2981891681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f53eb1b0 0x5616f57c6470 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2981891681 >> 172.21.15.144:0/2981891681 conn(0x5616f5408460 msgr2=0x5616f57ba000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2981891681 shutdown_connections 2024-07-26T11:10:51.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.132+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/2981891681 wait complete. 2024-07-26T11:10:51.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.133+0000 7f1f3b7fa740 1 -- start start 2024-07-26T11:10:51.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.133+0000 7f1f3b7fa740 1 Processor -- start 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.133+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53eb1b0 0x5616f57cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 0x5616f57c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f3b7fa740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f5414070 0x5616f57c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f1f301b58e0 con 0x5616f5414070 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f301b6cf0 con 0x5616f53eb1b0 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f3b7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1f301b5f50 con 0x5616f53f3390 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f3b7fa740 10 client.0 _ll_register_callbacks cb 0x7f1f200044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 0x5616f57c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 0x5616f57c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:46510/0 (socket says 172.21.15.144:46510) 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 -- 172.21.15.144:0/3997505587 learned_addr learned my addr 172.21.15.144:0/3997505587 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:10:51.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2e800640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f5414070 0x5616f57c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2de00640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53eb1b0 0x5616f57cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 -- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53eb1b0 msgr2=0x5616f57cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53eb1b0 0x5616f57cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 -- 172.21.15.144:0/3997505587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f5414070 msgr2=0x5616f57c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f5414070 0x5616f57c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.134+0000 7f1f2d400640 1 -- 172.21.15.144:0/3997505587 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616f57c9140 con 0x5616f53f3390 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f2d400640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 0x5616f57c8510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c006010 tx=0x7f1f1c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6ccc3ad1f55e64c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f0f400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f0f400640 1 -- 172.21.15.144:0/3997505587 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c049070 con 0x5616f53f3390 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f0f400640 1 -- 172.21.15.144:0/3997505587 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 10942+0+0 (secure 0 0 0) 0x7f1f1c003d30 con 0x5616f53f3390 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f200039c0 con 0x5616f53f3390 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f0f400640 1 -- 172.21.15.144:0/3997505587 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) ==== 24+0+0 (secure 0 0 0) 0x7f1f1c0cf2d0 con 0x5616f53f3390 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f3b7fa740 10 client.26335 fetch_fsmap learned FSMap version 151 2024-07-26T11:10:51.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=4+}) -- 0x5616f583f360 con 0x5616f53f3390 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.135+0000 7f1f0f400640 1 -- 172.21.15.144:0/3997505587 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f200039c0 con 0x5616f53f3390 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f0f400640 1 -- 172.21.15.144:0/3997505587 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 151) ==== 7144+0+0 (secure 0 0 0) 0x7f1f1c101cb0 con 0x5616f53f3390 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f3b7fa740 10 client.26335 fetch_fsmap finished waiting for FSMap version 151 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f3b7fa740 10 client.26335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f3b7fa740 20 client.26335 populate_metadata read hostname 'smithi144' 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53a7da0 0x5616f5841b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f3b7fa740 4 client.26335 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616f572d580 con 0x5616f53a7da0 2024-07-26T11:10:51.141 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 294.531167 secs 2024-07-26T11:10:51.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.136+0000 7f1f2de00640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53a7da0 0x5616f5841b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.137+0000 7f1f2de00640 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53a7da0 0x5616f5841b20 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d74c0c7e266c3bd server_cookie=43e64d507a23d704 in_seq=0 out_seq=0 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.137+0000 7f1f0f400640 10 client.26335 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.137+0000 7f1f0f400640 1 -- 172.21.15.144:0/3997505587 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5616f572d580 con 0x5616f53a7da0 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.137+0000 7f1f0f400640 10 client.26335 handle_command_reply: tid=0 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.137+0000 7f1f3b7fa740 1 client.26335 shutdown 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 msgr2=0x5616f57c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 0x5616f57c8510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c006010 tx=0x7f1f1c000bc0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 shutdown_connections 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616f5414070 0x5616f57c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616f53f3390 0x5616f57c8510 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616f53a7da0 0x5616f5841b20 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 --2- 172.21.15.144:0/3997505587 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616f53eb1b0 0x5616f57cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 >> 172.21.15.144:0/3997505587 conn(0x5616f5408460 msgr2=0x5616f57c61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f0f400640 0 client.26335 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 shutdown_connections 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 1 -- 172.21.15.144:0/3997505587 wait complete. 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.138+0000 7f1f3b7fa740 20 client.26335 trim_cache size 0 max 16384 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:10:51.144 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f140b3d40 msgr2=0x7f1f140b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f140b3d40 0x7f1f140b6250 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009fd0 tx=0x7f1f20005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:51.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 msgr2=0x7f1f301a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 0x7f1f301a08e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c02a930 tx=0x7f1f1c031a20 comp rx=0 tx=0).stop 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 shutdown_connections 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f1f140b3d40 0x7f1f140b6250 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f30113fb0 0x7f1f301a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f30077cc0 0x7f1f301a08e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 --2- 172.21.15.144:0/1522903426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f30077370 0x7f1f301a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 >> 172.21.15.144:0/1522903426 conn(0x7f1f30101060 msgr2=0x7f1f301160e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.139+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 shutdown_connections 2024-07-26T11:10:51.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:10:51.140+0000 7f1f2f200640 1 -- 172.21.15.144:0/1522903426 wait complete. 2024-07-26T11:10:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 294.531167 secs 2024-07-26T11:10:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 294.531167 secs 2024-07-26T11:10:51.954 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:10:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:52 smithi144 ceph-mon[32129]: pgmap v732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 7 op/s 2024-07-26T11:10:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:52 smithi160 ceph-mon[36715]: pgmap v732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 7 op/s 2024-07-26T11:10:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:52 smithi183 ceph-mon[37273]: pgmap v732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 7 op/s 2024-07-26T11:10:53.451 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:10:53.451 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 systemd[1]: Stopped Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:53.451 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 1.946s CPU time. 2024-07-26T11:10:53.451 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 systemd[1]: Starting Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:10:53.451 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 podman[74740]: 2024-07-26 11:10:53.367875333 +0000 UTC m=+0.110018266 container create 39d4f1e958bcecee81ecd0ab2e7b68b6622392a94c35eba0e0863be91269d206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T11:10:53.451 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 podman[74740]: 2024-07-26 11:10:53.275432784 +0000 UTC m=+0.017575718 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:10:53.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 podman[74740]: 2024-07-26 11:10:53.562167602 +0000 UTC m=+0.304310549 container init 39d4f1e958bcecee81ecd0ab2e7b68b6622392a94c35eba0e0863be91269d206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T11:10:53.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 podman[74740]: 2024-07-26 11:10:53.56476322 +0000 UTC m=+0.306906150 container start 39d4f1e958bcecee81ecd0ab2e7b68b6622392a94c35eba0e0863be91269d206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T11:10:53.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 ceph-mds[74759]: -- 172.21.15.183:0/2256201126 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55815ac45860 con 0x55815ede7180 2024-07-26T11:10:53.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: starting mds.f at 2024-07-26T11:10:53.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 bash[74740]: 39d4f1e958bcecee81ecd0ab2e7b68b6622392a94c35eba0e0863be91269d206 2024-07-26T11:10:53.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:10:53 smithi183 systemd[1]: Started Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:10:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: pgmap v733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 222 B/s wr, 5 op/s 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: daemon mds.f restarted 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:boot 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:54 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: pgmap v733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 222 B/s wr, 5 op/s 2024-07-26T11:10:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: daemon mds.f restarted 2024-07-26T11:10:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:10:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T11:10:54.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:boot 2024-07-26T11:10:54.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:10:54.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:54.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:54 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: pgmap v733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 222 B/s wr, 5 op/s 2024-07-26T11:10:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: daemon mds.f restarted 2024-07-26T11:10:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:10:54.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:10:54.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T11:10:54.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:boot 2024-07-26T11:10:54.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:10:54.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:10:54.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:54 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:56.307 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:56 smithi144 ceph-mon[32129]: pgmap v735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T11:10:56.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:56 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:resolve 2024-07-26T11:10:56.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:56 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:56.308 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 299.531313 secs 2024-07-26T11:10:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:56 smithi160 ceph-mon[36715]: pgmap v735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T11:10:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:56 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:resolve 2024-07-26T11:10:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:56 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 299.531313 secs 2024-07-26T11:10:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:56 smithi183 ceph-mon[37273]: pgmap v735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T11:10:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:56 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:resolve 2024-07-26T11:10:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:56 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:56.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 299.531313 secs 2024-07-26T11:10:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:56.811+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:56.812+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:56.812+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:56.812+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:10:56.812+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:57 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:reconnect 2024-07-26T11:10:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:57 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:57 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:10:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:57 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:reconnect 2024-07-26T11:10:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:57 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:57.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:57 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:10:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:57 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:reconnect 2024-07-26T11:10:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:57 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:57 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:10:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:58 smithi183 ceph-mon[37273]: pgmap v736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:10:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:58 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:rejoin 2024-07-26T11:10:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:58 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:58 smithi183 ceph-mon[37273]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:10:58.468 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-mon[36715]: pgmap v736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:10:58.468 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:rejoin 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-mon[36715]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: *** Caught signal (Aborted) ** 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: in thread 7f9605376640 thread_name:ms_dispatch 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f960a74d6f0] 2024-07-26T11:10:58.469 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f960a79a94c] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 3: raise() 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 4: abort() 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 5: /lib64/libc.so.6(+0x2871b) [0x7f960a73771b] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 6: /lib64/libc.so.6(+0x37386) [0x7f960a746386] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564151d696b9] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564151f2780c] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564151e672ca] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564151e37753] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564151e38a1b] 2024-07-26T11:10:58.470 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564151e1c752] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564151d6aa56] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564151d6e637] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564151d6ef5e] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564151d5203a] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f960b01a407] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 18: (DispatchQueue::entry()+0x60f) [0x7f960b01acf9] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f960b0f75a1] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 20: (Thread::entry_wrapper()+0x43) [0x7f960aea4cdd] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 21: (Thread::_entry_func(void*)+0xd) [0x7f960aea4cf9] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 22: /lib64/libc.so.6(+0x89c02) [0x7f960a798c02] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f960a81dc40] 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.104+0000 7f9605376640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:58.471 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: in thread 7f9605376640 thread_name:ms_dispatch 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f960a74d6f0] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f960a79a94c] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 3: raise() 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 4: abort() 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 5: /lib64/libc.so.6(+0x2871b) [0x7f960a73771b] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 6: /lib64/libc.so.6(+0x37386) [0x7f960a746386] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564151d696b9] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564151f2780c] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564151e672ca] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564151e37753] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564151e38a1b] 2024-07-26T11:10:58.472 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564151e1c752] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564151d6aa56] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564151d6e637] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564151d6ef5e] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564151d5203a] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f960b01a407] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 18: (DispatchQueue::entry()+0x60f) [0x7f960b01acf9] 2024-07-26T11:10:58.473 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f960b0f75a1] 2024-07-26T11:10:58.474 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:58 smithi144 ceph-mon[32129]: pgmap v736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:10:58.474 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:58 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:rejoin 2024-07-26T11:10:58.474 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:58 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:58.474 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:58 smithi144 ceph-mon[32129]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 20: (Thread::entry_wrapper()+0x43) [0x7f960aea4cdd] 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 21: (Thread::_entry_func(void*)+0xd) [0x7f960aea4cf9] 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 22: /lib64/libc.so.6(+0x89c02) [0x7f960a798c02] 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f960a81dc40] 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 0> 2024-07-26T11:10:58.104+0000 7f9605376640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: in thread 7f9605376640 thread_name:ms_dispatch 2024-07-26T11:10:58.475 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f960a74d6f0] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f960a79a94c] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 3: raise() 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 4: abort() 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 5: /lib64/libc.so.6(+0x2871b) [0x7f960a73771b] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 6: /lib64/libc.so.6(+0x37386) [0x7f960a746386] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564151d696b9] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564151f2780c] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564151e672ca] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564151e37753] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564151e38a1b] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564151e1c752] 2024-07-26T11:10:58.476 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564151d6aa56] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564151d6e637] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564151d6ef5e] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564151d5203a] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f960b01a407] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 18: (DispatchQueue::entry()+0x60f) [0x7f960b01acf9] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f960b0f75a1] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 20: (Thread::entry_wrapper()+0x43) [0x7f960aea4cdd] 2024-07-26T11:10:58.477 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 21: (Thread::_entry_func(void*)+0xd) [0x7f960aea4cf9] 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 22: /lib64/libc.so.6(+0x89c02) [0x7f960a798c02] 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f960a81dc40] 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 0> 2024-07-26T11:10:58.104+0000 7f9605376640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: in thread 7f9605376640 thread_name:ms_dispatch 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f960a74d6f0] 2024-07-26T11:10:58.478 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f960a79a94c] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 3: raise() 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 4: abort() 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 5: /lib64/libc.so.6(+0x2871b) [0x7f960a73771b] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 6: /lib64/libc.so.6(+0x37386) [0x7f960a746386] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564151d696b9] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564151f2780c] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564151e672ca] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564151e37753] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564151e38a1b] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564151e1c752] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564151d6aa56] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564151d6e637] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564151d6ef5e] 2024-07-26T11:10:58.479 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564151d5203a] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f960b01a407] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 18: (DispatchQueue::entry()+0x60f) [0x7f960b01acf9] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f960b0f75a1] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 20: (Thread::entry_wrapper()+0x43) [0x7f960aea4cdd] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 21: (Thread::_entry_func(void*)+0xd) [0x7f960aea4cf9] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 22: /lib64/libc.so.6(+0x89c02) [0x7f960a798c02] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f960a81dc40] 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[66942]: 2024-07-26T11:10:58.480 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 podman[72003]: 2024-07-26 11:10:58.171048255 +0000 UTC m=+0.021850226 container died b5a2932745cb17c2f235dbf175655c5e0b6b1ec77a98146225f3454366b42571 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:10:59.113 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 podman[72003]: 2024-07-26 11:10:58.860583778 +0000 UTC m=+0.711385739 container remove b5a2932745cb17c2f235dbf175655c5e0b6b1ec77a98146225f3454366b42571 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T11:10:59.113 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:58 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:10:59.114 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:59 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:59.114 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:59 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:active 2024-07-26T11:10:59.114 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:59 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:59.114 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:10:59 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:59 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:59 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:active 2024-07-26T11:10:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:59 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:10:59 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:59 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:10:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:59 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1864776442,v1:172.21.15.160:6833/1864776442] up:active 2024-07-26T11:10:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:59 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:10:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:10:59 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:10:59.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:59 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:10:59.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:10:59 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 2.207s CPU time. 2024-07-26T11:11:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:00 smithi183 ceph-mon[37273]: pgmap v737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:11:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:00 smithi144 ceph-mon[32129]: pgmap v737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:11:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:00 smithi160 ceph-mon[36715]: pgmap v737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:11:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 304.531401 secs 2024-07-26T11:11:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 304.531401 secs 2024-07-26T11:11:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 304.531401 secs 2024-07-26T11:11:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:02 smithi183 ceph-mon[37273]: pgmap v738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:02.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:02 smithi160 ceph-mon[36715]: pgmap v738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:02 smithi144 ceph-mon[32129]: pgmap v738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:04 smithi183 ceph-mon[37273]: pgmap v739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:04 smithi144 ceph-mon[32129]: pgmap v739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:04.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:04 smithi160 ceph-mon[36715]: pgmap v739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:06 smithi183 ceph-mon[37273]: pgmap v740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 180 B/s wr, 7 op/s 2024-07-26T11:11:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 309.531541 secs 2024-07-26T11:11:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:06 smithi144 ceph-mon[32129]: pgmap v740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 180 B/s wr, 7 op/s 2024-07-26T11:11:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 309.531541 secs 2024-07-26T11:11:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:06 smithi160 ceph-mon[36715]: pgmap v740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 180 B/s wr, 7 op/s 2024-07-26T11:11:06.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 309.531541 secs 2024-07-26T11:11:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:08 smithi144 ceph-mon[32129]: pgmap v741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:08 smithi160 ceph-mon[36715]: pgmap v741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:08 smithi183 ceph-mon[37273]: pgmap v741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:09.579 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:11:09.580 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 systemd[1]: Stopped Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:09.580 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 2.207s CPU time. 2024-07-26T11:11:09.580 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 systemd[1]: Starting Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:11:09.580 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 podman[72108]: 2024-07-26 11:11:09.447522679 +0000 UTC m=+0.018229737 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:11:09.580 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 podman[72108]: 2024-07-26 11:11:09.578167784 +0000 UTC m=+0.148874876 container create 13535f51d5514a00be00ab7e226934f8c6b4aa28b892d99b6ca33f67b0a82599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:11:10.145 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 podman[72108]: 2024-07-26 11:11:09.830955961 +0000 UTC m=+0.401663013 container init 13535f51d5514a00be00ab7e226934f8c6b4aa28b892d99b6ca33f67b0a82599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0) 2024-07-26T11:11:10.146 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 podman[72108]: 2024-07-26 11:11:09.833946617 +0000 UTC m=+0.404653670 container start 13535f51d5514a00be00ab7e226934f8c6b4aa28b892d99b6ca33f67b0a82599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T11:11:10.146 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 ceph-mds[72127]: -- 172.21.15.160:0/4260740263 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e27b73860 con 0x563e2bd15600 2024-07-26T11:11:10.146 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: starting mds.k at 2024-07-26T11:11:10.146 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 bash[72108]: 13535f51d5514a00be00ab7e226934f8c6b4aa28b892d99b6ca33f67b0a82599 2024-07-26T11:11:10.146 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:09 smithi160 systemd[1]: Started Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: pgmap v742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: daemon mds.k restarted 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:boot 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:10.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:10 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: pgmap v742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:11:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: daemon mds.k restarted 2024-07-26T11:11:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:11:10.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T11:11:10.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:boot 2024-07-26T11:11:10.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:11:10.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:10.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:10 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: pgmap v742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: daemon mds.k restarted 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:boot 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:10 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 314.531651 secs 2024-07-26T11:11:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 314.531651 secs 2024-07-26T11:11:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 314.531651 secs 2024-07-26T11:11:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:11.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:11.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:11.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:11.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:11.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:12.423 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:12 smithi144 ceph-mon[32129]: pgmap v744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:11:12.423 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:12 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:resolve 2024-07-26T11:11:12.423 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:12 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:12 smithi160 ceph-mon[36715]: pgmap v744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:11:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:12 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:resolve 2024-07-26T11:11:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:12 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:12 smithi183 ceph-mon[37273]: pgmap v744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:11:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:12 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:resolve 2024-07-26T11:11:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:12 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:13 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:reconnect 2024-07-26T11:11:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:13 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:13 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:11:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:13 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:reconnect 2024-07-26T11:11:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:13 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:13 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:11:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:13 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:reconnect 2024-07-26T11:11:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:13 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:13 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:11:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:14 smithi144 ceph-mon[32129]: pgmap v745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:11:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:14 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:rejoin 2024-07-26T11:11:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:14 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:14 smithi144 ceph-mon[32129]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:11:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:14 smithi160 ceph-mon[36715]: pgmap v745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:11:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:14 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:rejoin 2024-07-26T11:11:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:14 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:14 smithi160 ceph-mon[36715]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:11:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-mon[37273]: pgmap v745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:11:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:rejoin 2024-07-26T11:11:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-mon[37273]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: *** Caught signal (Aborted) ** 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: in thread 7f2133d12640 thread_name:ms_dispatch 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21390e96f0] 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f213913694c] 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 3: raise() 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 4: abort() 2024-07-26T11:11:14.636 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21390d371b] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 6: /lib64/libc.so.6(+0x37386) [0x7f21390e2386] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5581592126b9] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5581593d080c] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5581593102ca] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5581592e0753] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5581592e1a1b] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5581592c5752] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558159213a56] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558159217637] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558159217f5e] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5581591fb03a] 2024-07-26T11:11:14.637 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21399b6407] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 18: (DispatchQueue::entry()+0x60f) [0x7f21399b6cf9] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2139a935a1] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 20: (Thread::entry_wrapper()+0x43) [0x7f2139840cdd] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2139840cf9] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2139134c02] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21391b9c40] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.234+0000 7f2133d12640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: in thread 7f2133d12640 thread_name:ms_dispatch 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21390e96f0] 2024-07-26T11:11:14.638 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f213913694c] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 3: raise() 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 4: abort() 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21390d371b] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 6: /lib64/libc.so.6(+0x37386) [0x7f21390e2386] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5581592126b9] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5581593d080c] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5581593102ca] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5581592e0753] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5581592e1a1b] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5581592c5752] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558159213a56] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558159217637] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558159217f5e] 2024-07-26T11:11:14.639 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5581591fb03a] 2024-07-26T11:11:14.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21399b6407] 2024-07-26T11:11:14.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 18: (DispatchQueue::entry()+0x60f) [0x7f21399b6cf9] 2024-07-26T11:11:14.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2139a935a1] 2024-07-26T11:11:14.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 20: (Thread::entry_wrapper()+0x43) [0x7f2139840cdd] 2024-07-26T11:11:14.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2139840cf9] 2024-07-26T11:11:14.640 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2139134c02] 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21391b9c40] 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 0> 2024-07-26T11:11:14.234+0000 7f2133d12640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: in thread 7f2133d12640 thread_name:ms_dispatch 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21390e96f0] 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f213913694c] 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 3: raise() 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 4: abort() 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21390d371b] 2024-07-26T11:11:14.641 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 6: /lib64/libc.so.6(+0x37386) [0x7f21390e2386] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5581592126b9] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5581593d080c] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5581593102ca] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5581592e0753] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5581592e1a1b] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5581592c5752] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558159213a56] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558159217637] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558159217f5e] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5581591fb03a] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21399b6407] 2024-07-26T11:11:14.642 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 18: (DispatchQueue::entry()+0x60f) [0x7f21399b6cf9] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2139a935a1] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 20: (Thread::entry_wrapper()+0x43) [0x7f2139840cdd] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2139840cf9] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2139134c02] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21391b9c40] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 0> 2024-07-26T11:11:14.234+0000 7f2133d12640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: in thread 7f2133d12640 thread_name:ms_dispatch 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21390e96f0] 2024-07-26T11:11:14.643 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f213913694c] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 3: raise() 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 4: abort() 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21390d371b] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 6: /lib64/libc.so.6(+0x37386) [0x7f21390e2386] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5581592126b9] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5581593d080c] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5581593102ca] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5581592e0753] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5581592e1a1b] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5581592c5752] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558159213a56] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558159217637] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558159217f5e] 2024-07-26T11:11:14.644 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5581591fb03a] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21399b6407] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 18: (DispatchQueue::entry()+0x60f) [0x7f21399b6cf9] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2139a935a1] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 20: (Thread::entry_wrapper()+0x43) [0x7f2139840cdd] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2139840cf9] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2139134c02] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21391b9c40] 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74755]: 2024-07-26T11:11:14.645 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:14 smithi183 podman[74793]: 2024-07-26 11:11:14.291677963 +0000 UTC m=+0.018492056 container died 39d4f1e958bcecee81ecd0ab2e7b68b6622392a94c35eba0e0863be91269d206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T11:11:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:15 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:active 2024-07-26T11:11:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:15 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:15 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:15.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:15 smithi183 podman[74793]: 2024-07-26 11:11:15.032766769 +0000 UTC m=+0.759580856 container remove 39d4f1e958bcecee81ecd0ab2e7b68b6622392a94c35eba0e0863be91269d206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T11:11:15.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:11:15.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:11:15.386 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:15 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.005s CPU time. 2024-07-26T11:11:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:15 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:active 2024-07-26T11:11:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:15 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:15 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:15 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2607893039,v1:172.21.15.183:6835/2607893039] up:active 2024-07-26T11:11:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:15 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:15 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:16 smithi144 ceph-mon[32129]: pgmap v746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 319.531765 secs 2024-07-26T11:11:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:16 smithi160 ceph-mon[36715]: pgmap v746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 319.531765 secs 2024-07-26T11:11:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:16 smithi183 ceph-mon[37273]: pgmap v746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 319.531765 secs 2024-07-26T11:11:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:18 smithi160 ceph-mon[36715]: pgmap v747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:18 smithi183 ceph-mon[37273]: pgmap v747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:18 smithi144 ceph-mon[32129]: pgmap v747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:20 smithi160 ceph-mon[36715]: pgmap v748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:20 smithi183 ceph-mon[37273]: pgmap v748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:20 smithi144 ceph-mon[32129]: pgmap v748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 324.531880 secs 2024-07-26T11:11:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:21 smithi183 ceph-mon[37273]: pgmap v749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:11:21.955 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:11:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 324.531880 secs 2024-07-26T11:11:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:21 smithi144 ceph-mon[32129]: pgmap v749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:11:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 324.531880 secs 2024-07-26T11:11:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:21 smithi160 ceph-mon[36715]: pgmap v749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:11:22.435 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:24 smithi144 ceph-mon[32129]: pgmap v750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:11:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:24 smithi160 ceph-mon[36715]: pgmap v750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:11:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:24 smithi183 ceph-mon[37273]: pgmap v750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:11:24.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 -- 172.21.15.144:0/1063927612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 msgr2=0x7f34ec10b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 --2- 172.21.15.144:0/1063927612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 0x7f34ec10b500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009900 tx=0x7f34d402a410 comp rx=0 tx=0).stop 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 -- 172.21.15.144:0/1063927612 shutdown_connections 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 --2- 172.21.15.144:0/1063927612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec10ba40 0x7f34ec112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 --2- 172.21.15.144:0/1063927612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 0x7f34ec10b500 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 --2- 172.21.15.144:0/1063927612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34ec1057b0 0x7f34ec105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 -- 172.21.15.144:0/1063927612 >> 172.21.15.144:0/1063927612 conn(0x7f34ec101200 msgr2=0x7f34ec103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 -- 172.21.15.144:0/1063927612 shutdown_connections 2024-07-26T11:11:24.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.885+0000 7f34f2000640 1 -- 172.21.15.144:0/1063927612 wait complete. 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 Processor -- start 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 -- start start 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 0x7f34ec19c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 0x7f34ec19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34ec10ba40 0x7f34ec1a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1156d0 con 0x7f34ec10b0f0 2024-07-26T11:11:24.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec115550 con 0x7f34ec1057b0 2024-07-26T11:11:24.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.887+0000 7f34f2000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec115850 con 0x7f34ec10ba40 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34e2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 0x7f34ec19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 0x7f34ec19c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 0x7f34ec19c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:58342/0 (socket says 172.21.15.144:58342) 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 -- 172.21.15.144:0/1258923170 learned_addr learned my addr 172.21.15.144:0/1258923170 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34ebe00640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34ec10ba40 0x7f34ec1a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 -- 172.21.15.144:0/1258923170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34ec10ba40 msgr2=0x7f34ec1a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34ec10ba40 0x7f34ec1a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 -- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 msgr2=0x7f34ec19c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 0x7f34ec19c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.888+0000 7f34eb400640 1 -- 172.21.15.144:0/1258923170 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ec1a2090 con 0x7f34ec1057b0 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.889+0000 7f34eb400640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 0x7f34ec19c120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00e9f0 tx=0x7f34dc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:24.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.889+0000 7f34e8c00640 1 -- 172.21.15.144:0/1258923170 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc00ccc0 con 0x7f34ec1057b0 2024-07-26T11:11:24.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.889+0000 7f34e8c00640 1 -- 172.21.15.144:0/1258923170 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34dc00ce60 con 0x7f34ec1057b0 2024-07-26T11:11:24.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.889+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ec1a2380 con 0x7f34ec1057b0 2024-07-26T11:11:24.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.889+0000 7f34e8c00640 1 -- 172.21.15.144:0/1258923170 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc010430 con 0x7f34ec1057b0 2024-07-26T11:11:24.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.889+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ec1a8ae0 con 0x7f34ec1057b0 2024-07-26T11:11:24.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.892+0000 7f34f8c73740 1 Processor -- start 2024-07-26T11:11:24.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 -- start start 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f23e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f20151b0 0x5567f23e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34e8c00640 1 -- 172.21.15.144:0/1258923170 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f34dc0046c0 con 0x7f34ec1057b0 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5567f2359b90 con 0x5567f203e070 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5567f2349090 con 0x5567f201d390 2024-07-26T11:11:24.895 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34f8c73740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5567f234d0d0 con 0x5567f20151b0 2024-07-26T11:11:24.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.893+0000 7f34ebe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f23e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34ebe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f23e2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33762/0 (socket says 172.21.15.144:33762) 2024-07-26T11:11:24.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34ebe00640 1 -- 172.21.15.144:0/2274884392 learned_addr learned my addr 172.21.15.144:0/2274884392 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:24.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34e2a00640 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34eb400640 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f20151b0 0x5567f23e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34e2a00640 1 -- 172.21.15.144:0/2274884392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f20151b0 msgr2=0x5567f23e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34e2a00640 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f20151b0 0x5567f23e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34e2a00640 1 -- 172.21.15.144:0/2274884392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 msgr2=0x5567f23e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34e2a00640 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f23e2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34e2a00640 1 -- 172.21.15.144:0/2274884392 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f20a9650 con 0x5567f201d390 2024-07-26T11:11:24.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.894+0000 7f34ebe00640 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f23e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:24.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.895+0000 7f34e2a00640 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23e3870 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f34d402a930 tx=0x7f34d4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=112b5c2226ae5b17 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:24.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.895+0000 7f34dbe00640 1 -- 172.21.15.144:0/2274884392 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4040070 con 0x5567f201d390 2024-07-26T11:11:24.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.895+0000 7f34dbe00640 1 -- 172.21.15.144:0/2274884392 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d4004030 con 0x5567f201d390 2024-07-26T11:11:24.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.895+0000 7f34dbe00640 1 -- 172.21.15.144:0/2274884392 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4035040 con 0x5567f201d390 2024-07-26T11:11:24.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.895+0000 7f34e8c00640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34d00b3dc0 0x7f34d00b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34e8c00640 1 -- 172.21.15.144:0/1258923170 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 11114+0+0 (secure 0 0 0) 0x7f34dc0e4dd0 con 0x7f34ec1057b0 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34ebe00640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34d00b3dc0 0x7f34d00b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 -- 172.21.15.144:0/2274884392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 msgr2=0x5567f23e3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23e3870 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f34d402a930 tx=0x7f34d4000bc0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 -- 172.21.15.144:0/2274884392 shutdown_connections 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23e3870 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f20151b0 0x5567f23e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 --2- 172.21.15.144:0/2274884392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f23e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 -- 172.21.15.144:0/2274884392 >> 172.21.15.144:0/2274884392 conn(0x5567f2032460 msgr2=0x5567f23e0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 -- 172.21.15.144:0/2274884392 shutdown_connections 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34f8c73740 1 -- 172.21.15.144:0/2274884392 wait complete. 2024-07-26T11:11:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.896+0000 7f34ebe00640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34d00b3dc0 0x7f34d00b62d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f34e4012760 tx=0x7f34e4009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.897+0000 7f34f8c73740 1 -- start start 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.897+0000 7f34f8c73740 1 Processor -- start 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 0x5567f23ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23ee930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23eee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5567f21ac700 con 0x5567f20151b0 2024-07-26T11:11:24.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5567f22e5660 con 0x5567f201d390 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5567f23f1570 con 0x5567f203e070 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34f8c73740 10 client.0 _ll_register_callbacks cb 0x5567f22f5110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34eb400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 0x5567f23ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34eb400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 0x5567f23ee3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33778/0 (socket says 172.21.15.144:33778) 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34e2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34eb400640 1 -- 172.21.15.144:0/3140483576 learned_addr learned my addr 172.21.15.144:0/3140483576 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.898+0000 7f34ebe00640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34e2a00640 1 -- 172.21.15.144:0/3140483576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 msgr2=0x5567f23eee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34e2a00640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23eee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34e2a00640 1 -- 172.21.15.144:0/3140483576 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 msgr2=0x5567f23ee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34e2a00640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 0x5567f23ee3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.902 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34e2a00640 1 -- 172.21.15.144:0/3140483576 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f22fc800 con 0x5567f201d390 2024-07-26T11:11:24.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34eb400640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 0x5567f23ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:24.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34e2a00640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23ee930 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f34d4006010 tx=0x7f34d4032ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6c37c28bd40a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:24.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:11:24.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34d8c00640 1 -- 172.21.15.144:0/3140483576 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4035070 con 0x5567f201d390 2024-07-26T11:11:24.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34d8c00640 1 -- 172.21.15.144:0/3140483576 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 11114+0+0 (secure 0 0 0) 0x7f34d40314a0 con 0x5567f201d390 2024-07-26T11:11:24.903 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.899+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34d4004030 con 0x5567f201d390 2024-07-26T11:11:24.904 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34d8c00640 1 -- 172.21.15.144:0/3140483576 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) ==== 24+0+0 (secure 0 0 0) 0x7f34d4058050 con 0x5567f201d390 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34f8c73740 10 client.26336 fetch_fsmap learned FSMap version 165 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=4+}) -- 0x5567f23fd1a0 con 0x5567f201d390 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34d8c00640 1 -- 172.21.15.144:0/3140483576 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d4004030 con 0x5567f201d390 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34d8c00640 1 -- 172.21.15.144:0/3140483576 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 165) ==== 7144+0+0 (secure 0 0 0) 0x7f34d4004a60 con 0x5567f201d390 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34f8c73740 10 client.26336 fetch_fsmap finished waiting for FSMap version 165 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.900+0000 7f34f8c73740 10 client.26336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34f8c73740 20 client.26336 populate_metadata read hostname 'smithi144' 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34f8c73740 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1ffc9f0 0x5567f2400500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34f8c73740 4 client.26336 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567f235bae0 con 0x5567f1ffc9f0 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34eb400640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1ffc9f0 0x5567f2400500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34eb400640 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1ffc9f0 0x5567f2400500 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbbd0358fd5c507d server_cookie=76041401b36c8fe1 in_seq=0 out_seq=0 2024-07-26T11:11:24.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.901+0000 7f34d8c00640 10 client.26336 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.902+0000 7f34d8c00640 1 -- 172.21.15.144:0/3140483576 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5567f235bae0 con 0x5567f1ffc9f0 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.902+0000 7f34d8c00640 10 client.26336 handle_command_reply: tid=0 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.902+0000 7f34f8c73740 1 client.26336 shutdown 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.902+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 msgr2=0x5567f23ee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23ee930 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f34d4006010 tx=0x7f34d4032ed0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 shutdown_connections 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23eee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1ffc9f0 0x5567f2400500 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f201d390 0x5567f23ee930 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 --2- 172.21.15.144:0/3140483576 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f20151b0 0x5567f23ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34d8c00640 0 client.26336 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 >> 172.21.15.144:0/3140483576 conn(0x5567f2032460 msgr2=0x5567f23e5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 shutdown_connections 2024-07-26T11:11:24.906 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 1 -- 172.21.15.144:0/3140483576 wait complete. 2024-07-26T11:11:24.907 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.903+0000 7f34f8c73740 20 client.26336 trim_cache size 0 max 16384 2024-07-26T11:11:24.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.921+0000 7f34f8c73740 1 Processor -- start 2024-07-26T11:11:24.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 -- start start 2024-07-26T11:11:24.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23eed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f201d390 0x5567f23e9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f2405bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b44a0 con 0x5567f203e070 2024-07-26T11:11:24.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b4b70 con 0x5567f20151b0 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34f8c73740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b5240 con 0x5567f201d390 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34ebe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f2405bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34ebe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f2405bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:33780/0 (socket says 172.21.15.144:33780) 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34ebe00640 1 -- 172.21.15.144:0/53416848 learned_addr learned my addr 172.21.15.144:0/53416848 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.922+0000 7f34eb400640 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34e2a00640 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f201d390 0x5567f23e9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34ebe00640 1 -- 172.21.15.144:0/53416848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f201d390 msgr2=0x5567f23e9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34ebe00640 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f201d390 0x5567f23e9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34ebe00640 1 -- 172.21.15.144:0/53416848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 msgr2=0x5567f23eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34ebe00640 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34ebe00640 1 -- 172.21.15.144:0/53416848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f23ef6b0 con 0x5567f203e070 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34eb400640 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.923+0000 7f34ebe00640 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f2405bd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f34e4003e50 tx=0x7f34e401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0c9baf635198c5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.924+0000 7f34db400640 1 -- 172.21.15.144:0/53416848 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4004b20 con 0x5567f203e070 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.924+0000 7f34db400640 1 -- 172.21.15.144:0/53416848 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e4004cc0 con 0x5567f203e070 2024-07-26T11:11:24.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.924+0000 7f34db400640 1 -- 172.21.15.144:0/53416848 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4005420 con 0x5567f203e070 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 -- 172.21.15.144:0/53416848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 msgr2=0x5567f2405bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f2405bd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f34e4003e50 tx=0x7f34e401a000 comp rx=0 tx=0).stop 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 -- 172.21.15.144:0/53416848 shutdown_connections 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f203e070 0x5567f2405bd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f201d390 0x5567f23e9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 --2- 172.21.15.144:0/53416848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 -- 172.21.15.144:0/53416848 >> 172.21.15.144:0/53416848 conn(0x5567f2032460 msgr2=0x5567f23e9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 -- 172.21.15.144:0/53416848 shutdown_connections 2024-07-26T11:11:24.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 -- 172.21.15.144:0/53416848 wait complete. 2024-07-26T11:11:24.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.925+0000 7f34f8c73740 1 -- start start 2024-07-26T11:11:24.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 Processor -- start 2024-07-26T11:11:24.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f201d390 0x5567f23f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23f2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b1780 con 0x5567f201d390 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b2b90 con 0x5567f20151b0 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b1df0 con 0x5567f203e070 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.926+0000 7f34f8c73740 10 client.0 _ll_register_callbacks cb 0x7f34d4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23f18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:58382/0 (socket says 172.21.15.144:58382) 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 -- 172.21.15.144:0/1762684497 learned_addr learned my addr 172.21.15.144:0/1762684497 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34ebe00640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34e2a00640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f201d390 0x5567f23f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 -- 172.21.15.144:0/1762684497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 msgr2=0x5567f23f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 -- 172.21.15.144:0/1762684497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f201d390 msgr2=0x5567f23f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f201d390 0x5567f23f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 -- 172.21.15.144:0/1762684497 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f2466d90 con 0x5567f20151b0 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34e2a00640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f201d390 0x5567f23f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34eb400640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23f18a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0a6fe0 tx=0x7f34dc0e2010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e4a0ba6858fe37 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.927+0000 7f34d8c00640 1 -- 172.21.15.144:0/1762684497 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0aa6d0 con 0x5567f20151b0 2024-07-26T11:11:24.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34d8c00640 1 -- 172.21.15.144:0/1762684497 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 11114+0+0 (secure 0 0 0) 0x7f34dc0b3d90 con 0x5567f20151b0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e4004cc0 con 0x5567f20151b0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34d8c00640 1 -- 172.21.15.144:0/1762684497 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) ==== 24+0+0 (secure 0 0 0) 0x7f34dc0aaa70 con 0x5567f20151b0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 10 client.26342 fetch_fsmap learned FSMap version 165 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=4+}) -- 0x5567f2468080 con 0x5567f20151b0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34d8c00640 1 -- 172.21.15.144:0/1762684497 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e4004cc0 con 0x5567f20151b0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34d8c00640 1 -- 172.21.15.144:0/1762684497 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 165) ==== 7144+0+0 (secure 0 0 0) 0x7f34dc0a68d0 con 0x5567f20151b0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 10 client.26342 fetch_fsmap finished waiting for FSMap version 165 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 10 client.26342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 20 client.26342 populate_metadata read hostname 'smithi144' 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1fd1da0 0x5567f246a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 4 client.26342 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.928+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5567f2357580 con 0x5567f1fd1da0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.929+0000 7f34e2a00640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1fd1da0 0x5567f246a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:24.932 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.929+0000 7f34e2a00640 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1fd1da0 0x5567f246a840 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac0ab022333b0d7d server_cookie=9f4be601e2a5f185 in_seq=0 out_seq=0 2024-07-26T11:11:24.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.929+0000 7f34d8c00640 10 client.26342 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:24.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34d8c00640 1 -- 172.21.15.144:0/1762684497 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5567f2357580 con 0x5567f1fd1da0 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34d8c00640 10 client.26342 handle_command_reply: tid=0 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 client.26342 shutdown 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 msgr2=0x5567f23f18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23f18a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0a6fe0 tx=0x7f34dc0e2010 comp rx=0 tx=0).stop 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 shutdown_connections 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567f203e070 0x5567f23f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5567f201d390 0x5567f23f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5567f1fd1da0 0x5567f246a840 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 --2- 172.21.15.144:0/1762684497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5567f20151b0 0x5567f23f18a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 >> 172.21.15.144:0/1762684497 conn(0x5567f2032460 msgr2=0x5567f23eeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34d8c00640 0 client.26342 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 shutdown_connections 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.930+0000 7f34f8c73740 1 -- 172.21.15.144:0/1762684497 wait complete. 2024-07-26T11:11:24.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f8c73740 20 client.26342 trim_cache size 0 max 16384 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:11:24.935 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:11:24.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34d00b3dc0 msgr2=0x7f34d00b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34d00b3dc0 0x7f34d00b62d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f34e4012760 tx=0x7f34e4009500 comp rx=0 tx=0).stop 2024-07-26T11:11:24.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 msgr2=0x7f34ec19c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 0x7f34ec19c120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00e9f0 tx=0x7f34dc00edf0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 shutdown_connections 2024-07-26T11:11:24.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34d00b3dc0 0x7f34d00b62d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34ec10ba40 0x7f34ec1a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ec10b0f0 0x7f34ec19c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 --2- 172.21.15.144:0/1258923170 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34ec1057b0 0x7f34ec19c120 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.931+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 >> 172.21.15.144:0/1258923170 conn(0x7f34ec101200 msgr2=0x7f34ec103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.932+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 shutdown_connections 2024-07-26T11:11:24.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:24.932+0000 7f34f2000640 1 -- 172.21.15.144:0/1258923170 wait complete. 2024-07-26T11:11:25.710 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:11:25.710 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 systemd[1]: Stopped Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:25.710 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.005s CPU time. 2024-07-26T11:11:25.710 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 systemd[1]: Starting Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:11:25.710 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 podman[74885]: 2024-07-26 11:11:25.616397463 +0000 UTC m=+0.110336044 container create 71ea3aa15bada2658264d54f41e48637f229f6ac9e9d1589757388eed9730ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:11:25.710 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 podman[74885]: 2024-07-26 11:11:25.524451175 +0000 UTC m=+0.018389770 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:11:25.798 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:11:26.038 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 podman[74885]: 2024-07-26 11:11:25.819289537 +0000 UTC m=+0.313228125 container init 71ea3aa15bada2658264d54f41e48637f229f6ac9e9d1589757388eed9730ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T11:11:26.039 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 podman[74885]: 2024-07-26 11:11:25.821996727 +0000 UTC m=+0.315935327 container start 71ea3aa15bada2658264d54f41e48637f229f6ac9e9d1589757388eed9730ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:11:26.039 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 ceph-mds[74904]: -- 172.21.15.183:0/1017737501 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560141b1860 con 0x556018353180 2024-07-26T11:11:26.039 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: starting mds.f at 2024-07-26T11:11:26.039 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 bash[74885]: 71ea3aa15bada2658264d54f41e48637f229f6ac9e9d1589757388eed9730ce8 2024-07-26T11:11:26.039 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:25 smithi183 systemd[1]: Started Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: pgmap v751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 329.531998 secs 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: daemon mds.f restarted 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:boot 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:26 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: pgmap v751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 329.531998 secs 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: daemon mds.f restarted 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:boot 2024-07-26T11:11:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:11:26.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:26.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:26 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: pgmap v751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 329.531998 secs 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: daemon mds.f restarted 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:boot 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:26.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:26 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:26.839+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:26.840+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:26.840+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:26.840+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:26.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:28.306 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:28 smithi144 ceph-mon[32129]: pgmap v753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:11:28.306 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:28 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:resolve 2024-07-26T11:11:28.306 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:28 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:28 smithi160 ceph-mon[36715]: pgmap v753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:11:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:28 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:resolve 2024-07-26T11:11:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:28 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:28 smithi183 ceph-mon[37273]: pgmap v753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:11:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:28 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:resolve 2024-07-26T11:11:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:28 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:29 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:reconnect 2024-07-26T11:11:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:29 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:29 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:11:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:29 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:reconnect 2024-07-26T11:11:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:29 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:29 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:11:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:29 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:reconnect 2024-07-26T11:11:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:29 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:29 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:11:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:30 smithi183 ceph-mon[37273]: pgmap v754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6 op/s 2024-07-26T11:11:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:30 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:rejoin 2024-07-26T11:11:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:30 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:30 smithi183 ceph-mon[37273]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:11:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:30 smithi144 ceph-mon[32129]: pgmap v754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6 op/s 2024-07-26T11:11:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:30 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:rejoin 2024-07-26T11:11:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:30 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:30 smithi144 ceph-mon[32129]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:11:30.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-mon[36715]: pgmap v754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6 op/s 2024-07-26T11:11:30.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:rejoin 2024-07-26T11:11:30.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-mon[36715]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: *** Caught signal (Aborted) ** 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: in thread 7f50b47ad640 thread_name:ms_dispatch 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50b9b846f0] 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50b9bd194c] 2024-07-26T11:11:30.486 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 3: raise() 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 4: abort() 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50b9b6e71b] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 6: /lib64/libc.so.6(+0x37386) [0x7f50b9b7d386] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563e258b66b9] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563e25a7480c] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563e259b42ca] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563e25984753] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563e25985a1b] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563e25969752] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563e258b7a56] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563e258bb637] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563e258bbf5e] 2024-07-26T11:11:30.487 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563e2589f03a] 2024-07-26T11:11:30.488 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50ba451407] 2024-07-26T11:11:30.488 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 18: (DispatchQueue::entry()+0x60f) [0x7f50ba451cf9] 2024-07-26T11:11:30.488 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50ba52e5a1] 2024-07-26T11:11:30.488 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 20: (Thread::entry_wrapper()+0x43) [0x7f50ba2dbcdd] 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50ba2dbcf9] 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50b9bcfc02] 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50b9c54c40] 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.113+0000 7f50b47ad640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: in thread 7f50b47ad640 thread_name:ms_dispatch 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50b9b846f0] 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50b9bd194c] 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 3: raise() 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 4: abort() 2024-07-26T11:11:30.489 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50b9b6e71b] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 6: /lib64/libc.so.6(+0x37386) [0x7f50b9b7d386] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563e258b66b9] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563e25a7480c] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563e259b42ca] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563e25984753] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563e25985a1b] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563e25969752] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563e258b7a56] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563e258bb637] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563e258bbf5e] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563e2589f03a] 2024-07-26T11:11:30.490 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50ba451407] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 18: (DispatchQueue::entry()+0x60f) [0x7f50ba451cf9] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50ba52e5a1] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 20: (Thread::entry_wrapper()+0x43) [0x7f50ba2dbcdd] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50ba2dbcf9] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50b9bcfc02] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50b9c54c40] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 0> 2024-07-26T11:11:30.113+0000 7f50b47ad640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: in thread 7f50b47ad640 thread_name:ms_dispatch 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50b9b846f0] 2024-07-26T11:11:30.491 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50b9bd194c] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 3: raise() 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 4: abort() 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50b9b6e71b] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 6: /lib64/libc.so.6(+0x37386) [0x7f50b9b7d386] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563e258b66b9] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563e25a7480c] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563e259b42ca] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563e25984753] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563e25985a1b] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563e25969752] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563e258b7a56] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563e258bb637] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563e258bbf5e] 2024-07-26T11:11:30.492 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563e2589f03a] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50ba451407] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 18: (DispatchQueue::entry()+0x60f) [0x7f50ba451cf9] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50ba52e5a1] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 20: (Thread::entry_wrapper()+0x43) [0x7f50ba2dbcdd] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50ba2dbcf9] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50b9bcfc02] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50b9c54c40] 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 0> 2024-07-26T11:11:30.113+0000 7f50b47ad640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: in thread 7f50b47ad640 thread_name:ms_dispatch 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:30.493 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50b9b846f0] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50b9bd194c] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 3: raise() 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 4: abort() 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50b9b6e71b] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 6: /lib64/libc.so.6(+0x37386) [0x7f50b9b7d386] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563e258b66b9] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563e25a7480c] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563e259b42ca] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563e25984753] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563e25985a1b] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563e25969752] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563e258b7a56] 2024-07-26T11:11:30.494 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563e258bb637] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563e258bbf5e] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563e2589f03a] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50ba451407] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 18: (DispatchQueue::entry()+0x60f) [0x7f50ba451cf9] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50ba52e5a1] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 20: (Thread::entry_wrapper()+0x43) [0x7f50ba2dbcdd] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50ba2dbcf9] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50b9bcfc02] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50b9c54c40] 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72123]: 2024-07-26T11:11:30.495 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 podman[72161]: 2024-07-26 11:11:30.178862712 +0000 UTC m=+0.019436697 container died 13535f51d5514a00be00ab7e226934f8c6b4aa28b892d99b6ca33f67b0a82599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:11:31.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 podman[72161]: 2024-07-26 11:11:30.802632887 +0000 UTC m=+0.643206904 container remove 13535f51d5514a00be00ab7e226934f8c6b4aa28b892d99b6ca33f67b0a82599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T11:11:31.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:30 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:11:31.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:31 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:31.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:active 2024-07-26T11:11:31.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:31 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:31.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:31 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:31.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 334.532111 secs 2024-07-26T11:11:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:31 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:active 2024-07-26T11:11:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:31 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:31 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 334.532111 secs 2024-07-26T11:11:31.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:31 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:11:31.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:31 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 2.001s CPU time. 2024-07-26T11:11:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:31 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2413300160,v1:172.21.15.160:6833/2413300160] up:active 2024-07-26T11:11:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:31 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:31 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 334.532111 secs 2024-07-26T11:11:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:32 smithi183 ceph-mon[37273]: pgmap v755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:32.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:32 smithi160 ceph-mon[36715]: pgmap v755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:32 smithi144 ceph-mon[32129]: pgmap v755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:11:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:34 smithi183 ceph-mon[37273]: pgmap v756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:34 smithi144 ceph-mon[32129]: pgmap v756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:34 smithi160 ceph-mon[36715]: pgmap v756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:36 smithi183 ceph-mon[37273]: pgmap v757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 339.532257 secs 2024-07-26T11:11:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:36 smithi144 ceph-mon[32129]: pgmap v757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 339.532257 secs 2024-07-26T11:11:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:36 smithi160 ceph-mon[36715]: pgmap v757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:11:36.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 339.532257 secs 2024-07-26T11:11:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:38 smithi183 ceph-mon[37273]: pgmap v758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:11:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:38 smithi144 ceph-mon[32129]: pgmap v758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:11:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:38 smithi160 ceph-mon[36715]: pgmap v758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:11:40.376 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:40 smithi144 ceph-mon[32129]: pgmap v759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:40 smithi183 ceph-mon[37273]: pgmap v759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:40 smithi160 ceph-mon[36715]: pgmap v759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:11:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 344.532345 secs 2024-07-26T11:11:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 344.532345 secs 2024-07-26T11:11:41.542 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 344.532345 secs 2024-07-26T11:11:41.543 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:11:41.543 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 systemd[1]: Stopped Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:41.543 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 2.001s CPU time. 2024-07-26T11:11:41.543 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 systemd[1]: Starting Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:11:41.809 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 podman[72254]: 2024-07-26 11:11:41.444284683 +0000 UTC m=+0.017654157 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:11:41.809 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 podman[72254]: 2024-07-26 11:11:41.570201571 +0000 UTC m=+0.143571074 container create bc9da31a8332eb225fd37ca3691c4d19e307dff5b3fe2203407fd883432ef1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T11:11:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:41.845+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:41.846+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:41.846+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:41.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:41.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:42.134 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 podman[72254]: 2024-07-26 11:11:41.847529701 +0000 UTC m=+0.420899170 container init bc9da31a8332eb225fd37ca3691c4d19e307dff5b3fe2203407fd883432ef1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T11:11:42.134 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 podman[72254]: 2024-07-26 11:11:41.850035157 +0000 UTC m=+0.423404614 container start bc9da31a8332eb225fd37ca3691c4d19e307dff5b3fe2203407fd883432ef1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:11:42.134 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 ceph-mds[72274]: -- 172.21.15.160:0/4132064440 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b60de9b860 con 0x55b61203d600 2024-07-26T11:11:42.134 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: starting mds.k at 2024-07-26T11:11:42.134 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 bash[72254]: bc9da31a8332eb225fd37ca3691c4d19e307dff5b3fe2203407fd883432ef1c9 2024-07-26T11:11:42.134 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:11:41 smithi160 systemd[1]: Started Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: pgmap v760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: daemon mds.k restarted 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:boot 2024-07-26T11:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:11:42.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:42.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:42 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: pgmap v760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: daemon mds.k restarted 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:boot 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:42.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:42 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: pgmap v760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: daemon mds.k restarted 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:boot 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T11:11:42.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:42 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:replay,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:44.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:44 smithi144 ceph-mon[32129]: pgmap v762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:11:44.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:44 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:resolve 2024-07-26T11:11:44.405 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:44 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:44 smithi160 ceph-mon[36715]: pgmap v762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:11:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:44 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:resolve 2024-07-26T11:11:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:44 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:44 smithi183 ceph-mon[37273]: pgmap v762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:11:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:44 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:resolve 2024-07-26T11:11:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:44 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:resolve,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:45 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:reconnect 2024-07-26T11:11:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:45 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:45 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:11:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:45 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:reconnect 2024-07-26T11:11:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:45 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:45 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:11:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:reconnect 2024-07-26T11:11:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:reconnect,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:11:45.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:11:45.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: *** Caught signal (Aborted) ** 2024-07-26T11:11:45.885 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: in thread 7f544f07e640 thread_name:ms_dispatch 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f54544556f0] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f54544a294c] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 3: raise() 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 4: abort() 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f545443f71b] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 6: /lib64/libc.so.6(+0x37386) [0x7f545444e386] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55601164e6b9] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55601180c80c] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55601174c2ca] 2024-07-26T11:11:45.886 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55601171c753] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55601171da1b] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556011701752] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55601164fa56] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x556011653637] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x556011653f5e] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55601163703a] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5454d22407] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 18: (DispatchQueue::entry()+0x60f) [0x7f5454d22cf9] 2024-07-26T11:11:45.887 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5454dff5a1] 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 20: (Thread::entry_wrapper()+0x43) [0x7f5454baccdd] 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5454baccf9] 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f54544a0c02] 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5454525c40] 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.634+0000 7f544f07e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: in thread 7f544f07e640 thread_name:ms_dispatch 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:45.888 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f54544556f0] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f54544a294c] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 3: raise() 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 4: abort() 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f545443f71b] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 6: /lib64/libc.so.6(+0x37386) [0x7f545444e386] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55601164e6b9] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55601180c80c] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55601174c2ca] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55601171c753] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55601171da1b] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556011701752] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55601164fa56] 2024-07-26T11:11:45.889 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x556011653637] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x556011653f5e] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55601163703a] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5454d22407] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 18: (DispatchQueue::entry()+0x60f) [0x7f5454d22cf9] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5454dff5a1] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 20: (Thread::entry_wrapper()+0x43) [0x7f5454baccdd] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5454baccf9] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f54544a0c02] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5454525c40] 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 0> 2024-07-26T11:11:45.634+0000 7f544f07e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:45.890 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: in thread 7f544f07e640 thread_name:ms_dispatch 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f54544556f0] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f54544a294c] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 3: raise() 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 4: abort() 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f545443f71b] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 6: /lib64/libc.so.6(+0x37386) [0x7f545444e386] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55601164e6b9] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55601180c80c] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55601174c2ca] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55601171c753] 2024-07-26T11:11:45.891 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55601171da1b] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556011701752] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55601164fa56] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x556011653637] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x556011653f5e] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55601163703a] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5454d22407] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 18: (DispatchQueue::entry()+0x60f) [0x7f5454d22cf9] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5454dff5a1] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 20: (Thread::entry_wrapper()+0x43) [0x7f5454baccdd] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5454baccf9] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f54544a0c02] 2024-07-26T11:11:45.892 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5454525c40] 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 0> 2024-07-26T11:11:45.634+0000 7f544f07e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: in thread 7f544f07e640 thread_name:ms_dispatch 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f54544556f0] 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f54544a294c] 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 3: raise() 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 4: abort() 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f545443f71b] 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 6: /lib64/libc.so.6(+0x37386) [0x7f545444e386] 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55601164e6b9] 2024-07-26T11:11:45.893 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55601180c80c] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55601174c2ca] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55601171c753] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55601171da1b] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556011701752] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55601164fa56] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x556011653637] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x556011653f5e] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55601163703a] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5454d22407] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 18: (DispatchQueue::entry()+0x60f) [0x7f5454d22cf9] 2024-07-26T11:11:45.894 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5454dff5a1] 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 20: (Thread::entry_wrapper()+0x43) [0x7f5454baccdd] 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5454baccf9] 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f54544a0c02] 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5454525c40] 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f[74900]: 2024-07-26T11:11:45.895 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:45 smithi183 podman[74938]: 2024-07-26 11:11:45.690042209 +0000 UTC m=+0.018739723 container died 71ea3aa15bada2658264d54f41e48637f229f6ac9e9d1589757388eed9730ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T11:11:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:46.142+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480005850 con 0x7f74b0109990 2024-07-26T11:11:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:46.143+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480005850 con 0x7f74b0109990 2024-07-26T11:11:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:46.158+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478005820 con 0x563c6c4182f0 2024-07-26T11:11:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:46.159+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478005820 con 0x563c6c4182f0 2024-07-26T11:11:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: pgmap v763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:11:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:rejoin 2024-07-26T11:11:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:46.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:11:46.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 349.532469 secs 2024-07-26T11:11:46.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:46.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:active 2024-07-26T11:11:46.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:46 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: pgmap v763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:11:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:rejoin 2024-07-26T11:11:46.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:46.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:11:46.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 349.532469 secs 2024-07-26T11:11:46.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:46.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:active 2024-07-26T11:11:46.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:46 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: pgmap v763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:11:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:rejoin 2024-07-26T11:11:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=f=up:rejoin,2=g=up:active} 2 up:standby-replay 5 up:standby 2024-07-26T11:11:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T11:11:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 349.532469 secs 2024-07-26T11:11:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:11:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6834/2920134281,v1:172.21.15.183:6835/2920134281] up:active 2024-07-26T11:11:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:46 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=f=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby 2024-07-26T11:11:46.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:46 smithi183 podman[74938]: 2024-07-26 11:11:46.289350753 +0000 UTC m=+0.618048260 container remove 71ea3aa15bada2658264d54f41e48637f229f6ac9e9d1589757388eed9730ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-f, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T11:11:46.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:46 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:11:46.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:46 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:11:46.635 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:46 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 1.912s CPU time. 2024-07-26T11:11:47.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:11:47 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:11:47.175+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:11:48.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:48 smithi183 ceph-mon[37273]: pgmap v764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:48.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:48 smithi144 ceph-mon[32129]: pgmap v764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:48 smithi160 ceph-mon[36715]: pgmap v764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:49 smithi144 ceph-mon[32129]: pgmap v765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 111 B/s wr, 6 op/s 2024-07-26T11:11:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:49 smithi144 ceph-mon[32129]: pgmap v766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 141 B/s wr, 5 op/s 2024-07-26T11:11:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:11:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:49.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:49 smithi160 ceph-mon[36715]: pgmap v765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 111 B/s wr, 6 op/s 2024-07-26T11:11:49.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:49 smithi160 ceph-mon[36715]: pgmap v766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 141 B/s wr, 5 op/s 2024-07-26T11:11:49.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:11:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:49 smithi183 ceph-mon[37273]: pgmap v765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 111 B/s wr, 6 op/s 2024-07-26T11:11:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:49 smithi183 ceph-mon[37273]: pgmap v766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 141 B/s wr, 5 op/s 2024-07-26T11:11:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:11:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:51 smithi160 ceph-mon[36715]: pgmap v767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 246 B/s wr, 7 op/s 2024-07-26T11:11:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 354.532595 secs 2024-07-26T11:11:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:51 smithi183 ceph-mon[37273]: pgmap v767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 246 B/s wr, 7 op/s 2024-07-26T11:11:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 354.532595 secs 2024-07-26T11:11:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:51 smithi144 ceph-mon[32129]: pgmap v767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 246 B/s wr, 7 op/s 2024-07-26T11:11:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 354.532595 secs 2024-07-26T11:11:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:53 smithi160 ceph-mon[36715]: pgmap v768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:11:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:53 smithi183 ceph-mon[37273]: pgmap v768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:11:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:53 smithi144 ceph-mon[32129]: pgmap v768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:11:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:55 smithi160 ceph-mon[36715]: pgmap v769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:11:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:55 smithi183 ceph-mon[37273]: pgmap v769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:11:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:55 smithi144 ceph-mon[32129]: pgmap v769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 222 B/s wr, 8 op/s 2024-07-26T11:11:55.799 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:11:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 359.532750 secs 2024-07-26T11:11:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 359.532750 secs 2024-07-26T11:11:56.647 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 359.532750 secs 2024-07-26T11:11:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:56.861+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:56.862+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:56.862+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:56.862+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:11:56.862+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:57.135 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:56 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:11:57.135 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:56 smithi183 systemd[1]: Stopped Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:57.135 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:56 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Consumed 1.912s CPU time. 2024-07-26T11:11:57.135 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:56 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-07-26T11:11:57.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:56 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T11:11:57.136 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 26 11:11:56 smithi183 systemd[1]: Failed to start Ceph mds.f for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:11:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:57 smithi160 ceph-mon[36715]: pgmap v770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 111 B/s wr, 7 op/s 2024-07-26T11:11:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:57 smithi183 ceph-mon[37273]: pgmap v770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 111 B/s wr, 7 op/s 2024-07-26T11:11:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:57 smithi144 ceph-mon[32129]: pgmap v770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 111 B/s wr, 7 op/s 2024-07-26T11:11:58.629 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.627+0000 7fd8a7e00640 1 -- 172.21.15.144:0/659730384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 msgr2=0x7fd8a8077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.627+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/659730384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a8077780 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd898009900 tx=0x7fd89802a410 comp rx=0 tx=0).stop 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.627+0000 7fd8a7e00640 1 -- 172.21.15.144:0/659730384 shutdown_connections 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.628+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/659730384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 0x7fd8a81163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.628+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/659730384 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 0x7fd8a8113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.628+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/659730384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a8077780 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.628+0000 7fd8a7e00640 1 -- 172.21.15.144:0/659730384 >> 172.21.15.144:0/659730384 conn(0x7fd8a81010b0 msgr2=0x7fd8a8103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.628+0000 7fd8a7e00640 1 -- 172.21.15.144:0/659730384 shutdown_connections 2024-07-26T11:11:58.630 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.628+0000 7fd8a7e00640 1 -- 172.21.15.144:0/659730384 wait complete. 2024-07-26T11:11:58.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.629+0000 7fd8a7e00640 1 Processor -- start 2024-07-26T11:11:58.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.629+0000 7fd8a7e00640 1 -- start start 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a81a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 0x7fd8a81a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 0x7fd8a81a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8119900 con 0x7fd8a8077370 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8119780 con 0x7fd8a8077cc0 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8119a80 con 0x7fd8a8113fa0 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a81a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.632 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a81a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59184/0 (socket says 172.21.15.144:59184) 2024-07-26T11:11:58.633 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a6a00640 1 -- 172.21.15.144:0/1702690814 learned_addr learned my addr 172.21.15.144:0/1702690814 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:58.633 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd89fe00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 0x7fd8a81a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.633 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a7400640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 0x7fd8a81a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a6a00640 1 -- 172.21.15.144:0/1702690814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 msgr2=0x7fd8a81a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.630+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 0x7fd8a81a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd8a6a00640 1 -- 172.21.15.144:0/1702690814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 msgr2=0x7fd8a81a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 0x7fd8a81a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd8a6a00640 1 -- 172.21.15.144:0/1702690814 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a81a6360 con 0x7fd8a8077370 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd89fe00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 0x7fd8a81a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:58.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd8a7400640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 0x7fd8a81a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:58.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a81a03f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd898005940 tx=0x7fd8980040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:58.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd89f400640 1 -- 172.21.15.144:0/1702690814 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89803f030 con 0x7fd8a8077370 2024-07-26T11:11:58.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd89f400640 1 -- 172.21.15.144:0/1702690814 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8980044c0 con 0x7fd8a8077370 2024-07-26T11:11:58.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd89f400640 1 -- 172.21.15.144:0/1702690814 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898035070 con 0x7fd8a8077370 2024-07-26T11:11:58.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.631+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a81a65f0 con 0x7fd8a8077370 2024-07-26T11:11:58.635 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.632+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a81acdb0 con 0x7fd8a8077370 2024-07-26T11:11:58.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 Processor -- start 2024-07-26T11:11:58.637 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 -- start start 2024-07-26T11:11:58.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 0x561dc7dbb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.638 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dbb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dbbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x561dc7d31bc0 con 0x561dc79f5390 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x561dc7d210c0 con 0x561dc7a16070 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd8b461b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x561dc7d25100 con 0x561dc79ed1b0 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.635+0000 7fd89f400640 1 -- 172.21.15.144:0/1702690814 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd898042030 con 0x7fd8a8077370 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dbbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 0x561dc7dbb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 0x561dc7dbb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:48766/0 (socket says 172.21.15.144:48766) 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 -- 172.21.15.144:0/1000438797 learned_addr learned my addr 172.21.15.144:0/1000438797 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dbb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.639 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 -- 172.21.15.144:0/1000438797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 msgr2=0x561dc7dbb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dbb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 -- 172.21.15.144:0/1000438797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 msgr2=0x561dc7dbbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dbbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a7400640 1 -- 172.21.15.144:0/1000438797 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dc7a81650 con 0x561dc7a16070 2024-07-26T11:11:58.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.636+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dbb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:11:58.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.637+0000 7fd8a7400640 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 0x561dc7dbb1f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0011790 tx=0x7fd8a0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b08ac45c8a9ada84 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:58.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.637+0000 7fd897e00640 1 -- 172.21.15.144:0/1000438797 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0004d20 con 0x561dc7a16070 2024-07-26T11:11:58.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.637+0000 7fd897e00640 1 -- 172.21.15.144:0/1000438797 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8a0003970 con 0x561dc7a16070 2024-07-26T11:11:58.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.637+0000 7fd897e00640 1 -- 172.21.15.144:0/1000438797 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00044c0 con 0x561dc7a16070 2024-07-26T11:11:58.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd89f400640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd8880b3d40 0x7fd8880b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd89f400640 1 -- 172.21.15.144:0/1702690814 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 11286+0+0 (secure 0 0 0) 0x7fd898104e00 con 0x7fd8a8077370 2024-07-26T11:11:58.642 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd8880b3d40 0x7fd8880b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 -- 172.21.15.144:0/1000438797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 msgr2=0x561dc7dbb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 0x561dc7dbb1f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0011790 tx=0x7fd8a0011c50 comp rx=0 tx=0).stop 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 -- 172.21.15.144:0/1000438797 shutdown_connections 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dbbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dbb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 --2- 172.21.15.144:0/1000438797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc7a16070 0x561dc7dbb1f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 -- 172.21.15.144:0/1000438797 >> 172.21.15.144:0/1000438797 conn(0x561dc7a0a460 msgr2=0x561dc7db90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 -- 172.21.15.144:0/1000438797 shutdown_connections 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.638+0000 7fd8b461b740 1 -- 172.21.15.144:0/1000438797 wait complete. 2024-07-26T11:11:58.643 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.639+0000 7fd8a6a00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd8880b3d40 0x7fd8880b6250 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd898101b30 tx=0x7fd89802a410 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.639+0000 7fd8b461b740 1 -- start start 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.639+0000 7fd8b461b740 1 Processor -- start 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79ed1b0 0x561dc7dc66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dc6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x561dc7b84700 con 0x561dc79f5390 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x561dc7cbd690 con 0x561dc79ed1b0 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x561dc7dc99d0 con 0x561dc7a16070 2024-07-26T11:11:58.644 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8b461b740 10 client.0 _ll_register_callbacks cb 0x561dc7ccd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:11:58.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dc6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dc6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59206/0 (socket says 172.21.15.144:59206) 2024-07-26T11:11:58.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd89fe00640 1 -- 172.21.15.144:0/3743382990 learned_addr learned my addr 172.21.15.144:0/3743382990 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:58.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.640+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79ed1b0 0x561dc7dc66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd8a7400640 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd89fe00640 1 -- 172.21.15.144:0/3743382990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 msgr2=0x561dc7dc7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd89fe00640 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd89fe00640 1 -- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79ed1b0 msgr2=0x561dc7dc66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd89fe00640 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79ed1b0 0x561dc7dc66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd89fe00640 1 -- 172.21.15.144:0/3743382990 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dc7cd4830 con 0x561dc79f5390 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd89fe00640 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dc6c30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd890012d20 tx=0x7fd89000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25cb7f8a0f4af30 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.641+0000 7fd894c00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd894c00640 1 -- 172.21.15.144:0/3743382990 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890012110 con 0x561dc79f5390 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd894c00640 1 -- 172.21.15.144:0/3743382990 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 11286+0+0 (secure 0 0 0) 0x7fd890004050 con 0x561dc79f5390 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8a0003970 con 0x561dc79f5390 2024-07-26T11:11:58.648 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd894c00640 1 -- 172.21.15.144:0/3743382990 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=178) ==== 24+0+0 (secure 0 0 0) 0x7fd890025050 con 0x561dc79f5390 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd8b461b740 10 client.16155 fetch_fsmap learned FSMap version 178 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=178,monmap=4+}) -- 0x561dc7dd5580 con 0x561dc79f5390 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd894c00640 1 -- 172.21.15.144:0/3743382990 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8a0003970 con 0x561dc79f5390 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.642+0000 7fd894c00640 1 -- 172.21.15.144:0/3743382990 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 178) ==== 7144+0+0 (secure 0 0 0) 0x7fd890020ce0 con 0x561dc79f5390 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8b461b740 10 client.16155 fetch_fsmap finished waiting for FSMap version 178 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8b461b740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8b461b740 20 client.16155 populate_metadata read hostname 'smithi144' 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8b461b740 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79d49f0 0x561dc7dd8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8b461b740 4 client.16155 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dc7d33b10 con 0x561dc79d49f0 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.643+0000 7fd8a7400640 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79d49f0 0x561dc7dd8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.644+0000 7fd8a7400640 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79d49f0 0x561dc7dd8a90 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78e5414fc2b29561 server_cookie=74b40441da38c108 in_seq=0 out_seq=0 2024-07-26T11:11:58.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.644+0000 7fd894c00640 10 client.16155 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd894c00640 1 -- 172.21.15.144:0/3743382990 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561dc7d33b10 con 0x561dc79d49f0 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd894c00640 10 client.16155 handle_command_reply: tid=0 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 client.16155 shutdown 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 msgr2=0x561dc7dc6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dc6c30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd890012d20 tx=0x7fd89000f790 comp rx=0 tx=0).stop 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 shutdown_connections 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79f5390 0x561dc7dc6c30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79d49f0 0x561dc7dd8a90 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 --2- 172.21.15.144:0/3743382990 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79ed1b0 0x561dc7dc66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 >> 172.21.15.144:0/3743382990 conn(0x561dc7a0a460 msgr2=0x561dc7dbe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd894c00640 0 client.16155 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:58.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 shutdown_connections 2024-07-26T11:11:58.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 1 -- 172.21.15.144:0/3743382990 wait complete. 2024-07-26T11:11:58.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.645+0000 7fd8b461b740 20 client.16155 trim_cache size 0 max 16384 2024-07-26T11:11:58.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.664+0000 7fd8b461b740 1 Processor -- start 2024-07-26T11:11:58.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.664+0000 7fd8b461b740 1 -- start start 2024-07-26T11:11:58.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dd2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 0x561dc7dde250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8b461b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b89b0 con 0x561dc7a16070 2024-07-26T11:11:58.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8b461b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b9080 con 0x561dc79f5390 2024-07-26T11:11:58.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8b461b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b9750 con 0x561dc79ed1b0 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 0x561dc7dde250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 0x561dc7dde250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59220/0 (socket says 172.21.15.144:59220) 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8a7400640 1 -- 172.21.15.144:0/2129317832 learned_addr learned my addr 172.21.15.144:0/2129317832 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.665+0000 7fd8a7400640 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dd2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd89fe00640 1 -- 172.21.15.144:0/2129317832 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 msgr2=0x561dc7dd2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd89fe00640 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dd2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd89fe00640 1 -- 172.21.15.144:0/2129317832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 msgr2=0x561dc7dde250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd89fe00640 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 0x561dc7dde250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd89fe00640 1 -- 172.21.15.144:0/2129317832 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dc7db80e0 con 0x561dc79f5390 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd8a7400640 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 0x561dc7dde250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd89fe00640 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc6d70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd890012cf0 tx=0x7fd890020370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a6f576f3e04530 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:58.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd897400640 1 -- 172.21.15.144:0/2129317832 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890012110 con 0x561dc79f5390 2024-07-26T11:11:58.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd897400640 1 -- 172.21.15.144:0/2129317832 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89002e070 con 0x561dc79f5390 2024-07-26T11:11:58.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.666+0000 7fd897400640 1 -- 172.21.15.144:0/2129317832 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890005580 con 0x561dc79f5390 2024-07-26T11:11:58.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 -- 172.21.15.144:0/2129317832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 msgr2=0x561dc7dc6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc6d70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd890012cf0 tx=0x7fd890020370 comp rx=0 tx=0).stop 2024-07-26T11:11:58.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 -- 172.21.15.144:0/2129317832 shutdown_connections 2024-07-26T11:11:58.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc7a16070 0x561dc7dde250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc6d70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 --2- 172.21.15.144:0/2129317832 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc79ed1b0 0x561dc7dd2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 -- 172.21.15.144:0/2129317832 >> 172.21.15.144:0/2129317832 conn(0x561dc7a0a460 msgr2=0x561dc7dc6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 -- 172.21.15.144:0/2129317832 shutdown_connections 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.667+0000 7fd8b461b740 1 -- 172.21.15.144:0/2129317832 wait complete. 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.668+0000 7fd8b461b740 1 -- start start 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.668+0000 7fd8b461b740 1 Processor -- start 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79ed1b0 0x561dc7dc3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b5c90 con 0x561dc79ed1b0 2024-07-26T11:11:58.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b70a0 con 0x561dc79f5390 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b6300 con 0x561dc7a16070 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8b461b740 10 client.0 _ll_register_callbacks cb 0x7fd890012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79ed1b0 0x561dc7dc3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:48794/0 (socket says 172.21.15.144:48794) 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd89fe00640 1 -- 172.21.15.144:0/4183161598 learned_addr learned my addr 172.21.15.144:0/4183161598 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:11:58.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.669+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79ed1b0 0x561dc7dc3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:59228/0 (socket says 172.21.15.144:59228) 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd8a7400640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd89fe00640 1 -- 172.21.15.144:0/4183161598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 msgr2=0x561dc7dc41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd89fe00640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd89fe00640 1 -- 172.21.15.144:0/4183161598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79ed1b0 msgr2=0x561dc7dc3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd89fe00640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79ed1b0 0x561dc7dc3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd89fe00640 1 -- 172.21.15.144:0/4183161598 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dc7e40220 con 0x561dc79f5390 2024-07-26T11:11:58.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd8a7400640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd89fe00640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc3c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd890012870 tx=0x7fd89000fd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11e89eac367035bd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.670+0000 7fd894c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd894c00640 1 -- 172.21.15.144:0/4183161598 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890036030 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd89002e070 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd894c00640 1 -- 172.21.15.144:0/4183161598 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 11286+0+0 (secure 0 0 0) 0x7fd890018040 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd894c00640 1 -- 172.21.15.144:0/4183161598 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=178) ==== 24+0+0 (secure 0 0 0) 0x7fd890040050 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd8b461b740 10 client.26369 fetch_fsmap learned FSMap version 178 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=178,monmap=4+}) -- 0x561dc7e41410 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd894c00640 1 -- 172.21.15.144:0/4183161598 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89002e070 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.671+0000 7fd894c00640 1 -- 172.21.15.144:0/4183161598 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 178) ==== 7144+0+0 (secure 0 0 0) 0x7fd8900205e0 con 0x561dc79f5390 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8b461b740 10 client.26369 fetch_fsmap finished waiting for FSMap version 178 2024-07-26T11:11:58.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8b461b740 10 client.26369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8b461b740 20 client.26369 populate_metadata read hostname 'smithi144' 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8b461b740 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79a9da0 0x561dc7e43b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8b461b740 4 client.26369 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561dc7d2f5b0 con 0x561dc79a9da0 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.672+0000 7fd8a7400640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79a9da0 0x561dc7e43b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8a7400640 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79a9da0 0x561dc7e43b60 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d522e919f9631aa1 server_cookie=749b9e36d6f3a0c1 in_seq=0 out_seq=0 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd894c00640 10 client.26369 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd894c00640 1 -- 172.21.15.144:0/4183161598 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561dc7d2f5b0 con 0x561dc79a9da0 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd894c00640 10 client.26369 handle_command_reply: tid=0 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 client.26369 shutdown 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 msgr2=0x561dc7dc3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc3c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd890012870 tx=0x7fd89000fd50 comp rx=0 tx=0).stop 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 shutdown_connections 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561dc7a16070 0x561dc7dc41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x561dc79a9da0 0x561dc7e43b60 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x561dc79f5390 0x561dc7dc3c70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 --2- 172.21.15.144:0/4183161598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x561dc79ed1b0 0x561dc7dc3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 >> 172.21.15.144:0/4183161598 conn(0x561dc7a0a460 msgr2=0x561dc7dbc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:58.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.673+0000 7fd894c00640 0 client.26369 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.674+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 shutdown_connections 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.674+0000 7fd8b461b740 1 -- 172.21.15.144:0/4183161598 wait complete. 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.674+0000 7fd8b461b740 20 client.26369 trim_cache size 0 max 16384 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:11:58.679 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd8880b3d40 msgr2=0x7fd8880b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd8880b3d40 0x7fd8880b6250 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd898101b30 tx=0x7fd89802a410 comp rx=0 tx=0).stop 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 msgr2=0x7fd8a81a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a81a03f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd898005940 tx=0x7fd8980040c0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 shutdown_connections 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd8880b3d40 0x7fd8880b6250 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8113fa0 0x7fd8a81a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8077cc0 0x7fd8a81a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 --2- 172.21.15.144:0/1702690814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8a8077370 0x7fd8a81a03f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 >> 172.21.15.144:0/1702690814 conn(0x7fd8a81010b0 msgr2=0x7fd8a81160b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 shutdown_connections 2024-07-26T11:11:58.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:11:58.675+0000 7fd8a7e00640 1 -- 172.21.15.144:0/1702690814 wait complete. 2024-07-26T11:11:59.365 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:11:59 smithi144 ceph-mon[32129]: pgmap v771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:59.443 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:11:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:11:59 smithi160 ceph-mon[36715]: pgmap v771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:11:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:11:59 smithi183 ceph-mon[37273]: pgmap v771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: pgmap v772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 364.532872 secs 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: slow request 364.532871 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: Replacing daemon mds.f as rank 1 with standby daemon mds.k 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T11:12:01.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:01 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: pgmap v772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:12:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 364.532872 secs 2024-07-26T11:12:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: slow request 364.532871 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:12:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: Replacing daemon mds.f as rank 1 with standby daemon mds.k 2024-07-26T11:12:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T11:12:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T11:12:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:01 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: pgmap v772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 364.532872 secs 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: slow request 364.532871 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: Replacing daemon mds.f as rank 1 with standby daemon mds.k 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T11:12:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:01 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:02 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:resolve 2024-07-26T11:12:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:02 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:02 smithi183 ceph-mon[37273]: pgmap v774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:12:02.888 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:02 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:resolve 2024-07-26T11:12:02.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:02 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:02.889 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:02 smithi144 ceph-mon[32129]: pgmap v774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:12:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:02 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:resolve 2024-07-26T11:12:03.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:02 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:03.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:02 smithi160 ceph-mon[36715]: pgmap v774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:12:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:03 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:reconnect 2024-07-26T11:12:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:03 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:03 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:12:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:03 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:reconnect 2024-07-26T11:12:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:03 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:03 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:12:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:03 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:reconnect 2024-07-26T11:12:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:03 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:03 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:12:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:04 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:rejoin 2024-07-26T11:12:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:04 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:04 smithi144 ceph-mon[32129]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:12:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:04 smithi144 ceph-mon[32129]: pgmap v775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:05.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:rejoin 2024-07-26T11:12:05.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:05.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-mon[36715]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:12:05.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-mon[36715]: pgmap v775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: *** Caught signal (Aborted) ** 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: in thread 7fcbd8674640 thread_name:ms_dispatch 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcbdda4b6f0] 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcbdda9894c] 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 3: raise() 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 4: abort() 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcbdda3571b] 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 6: /lib64/libc.so.6(+0x37386) [0x7fcbdda44386] 2024-07-26T11:12:05.037 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b60cf0b6b9] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b60d0c980c] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b60d0092ca] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b60cfd9753] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b60cfdaa1b] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b60cfbe752] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b60cf0ca56] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b60cf10637] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b60cf10f5e] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b60cef403a] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcbde318407] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 18: (DispatchQueue::entry()+0x60f) [0x7fcbde318cf9] 2024-07-26T11:12:05.038 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcbde3f55a1] 2024-07-26T11:12:05.039 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 20: (Thread::entry_wrapper()+0x43) [0x7fcbde1a2cdd] 2024-07-26T11:12:05.039 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcbde1a2cf9] 2024-07-26T11:12:05.039 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcbdda96c02] 2024-07-26T11:12:05.039 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcbddb1bc40] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:04.699+0000 7fcbd8674640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: in thread 7fcbd8674640 thread_name:ms_dispatch 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcbdda4b6f0] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcbdda9894c] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 3: raise() 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 4: abort() 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcbdda3571b] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 6: /lib64/libc.so.6(+0x37386) [0x7fcbdda44386] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b60cf0b6b9] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b60d0c980c] 2024-07-26T11:12:05.040 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b60d0092ca] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b60cfd9753] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b60cfdaa1b] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b60cfbe752] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b60cf0ca56] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b60cf10637] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b60cf10f5e] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b60cef403a] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcbde318407] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 18: (DispatchQueue::entry()+0x60f) [0x7fcbde318cf9] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcbde3f55a1] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 20: (Thread::entry_wrapper()+0x43) [0x7fcbde1a2cdd] 2024-07-26T11:12:05.041 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcbde1a2cf9] 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcbdda96c02] 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcbddb1bc40] 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 0> 2024-07-26T11:12:04.699+0000 7fcbd8674640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: in thread 7fcbd8674640 thread_name:ms_dispatch 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcbdda4b6f0] 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcbdda9894c] 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 3: raise() 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 4: abort() 2024-07-26T11:12:05.042 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcbdda3571b] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 6: /lib64/libc.so.6(+0x37386) [0x7fcbdda44386] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b60cf0b6b9] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b60d0c980c] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b60d0092ca] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b60cfd9753] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b60cfdaa1b] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b60cfbe752] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b60cf0ca56] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b60cf10637] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b60cf10f5e] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b60cef403a] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcbde318407] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 18: (DispatchQueue::entry()+0x60f) [0x7fcbde318cf9] 2024-07-26T11:12:05.043 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcbde3f55a1] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 20: (Thread::entry_wrapper()+0x43) [0x7fcbde1a2cdd] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcbde1a2cf9] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcbdda96c02] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcbddb1bc40] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 0> 2024-07-26T11:12:04.699+0000 7fcbd8674640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: in thread 7fcbd8674640 thread_name:ms_dispatch 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcbdda4b6f0] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcbdda9894c] 2024-07-26T11:12:05.044 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 3: raise() 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 4: abort() 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcbdda3571b] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 6: /lib64/libc.so.6(+0x37386) [0x7fcbdda44386] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b60cf0b6b9] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b60d0c980c] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b60d0092ca] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b60cfd9753] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b60cfdaa1b] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b60cfbe752] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b60cf0ca56] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b60cf10637] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b60cf10f5e] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b60cef403a] 2024-07-26T11:12:05.045 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcbde318407] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 18: (DispatchQueue::entry()+0x60f) [0x7fcbde318cf9] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcbde3f55a1] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 20: (Thread::entry_wrapper()+0x43) [0x7fcbde1a2cdd] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcbde1a2cf9] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcbdda96c02] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcbddb1bc40] 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72270]: 2024-07-26T11:12:05.046 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:04 smithi160 podman[72437]: 2024-07-26 11:12:04.757692623 +0000 UTC m=+0.019161438 container died bc9da31a8332eb225fd37ca3691c4d19e307dff5b3fe2203407fd883432ef1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T11:12:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:04 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:rejoin 2024-07-26T11:12:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:04 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:04 smithi183 ceph-mon[37273]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:12:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:04 smithi183 ceph-mon[37273]: pgmap v775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:05.616 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:05 smithi160 podman[72437]: 2024-07-26 11:12:05.360680717 +0000 UTC m=+0.622149527 container remove bc9da31a8332eb225fd37ca3691c4d19e307dff5b3fe2203407fd883432ef1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:12:05.616 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:05 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:12:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:05 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:05 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:active 2024-07-26T11:12:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:05 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:05 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:06.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:05 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:12:06.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:05 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 1.983s CPU time. 2024-07-26T11:12:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:05 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:05 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:active 2024-07-26T11:12:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:05 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:05 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:05 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:05 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/392473501,v1:172.21.15.160:6833/392473501] up:active 2024-07-26T11:12:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:05 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:05 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:06 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 369.532978 secs 2024-07-26T11:12:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:06 smithi144 ceph-mon[32129]: slow request 364.672167 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:12:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:06 smithi144 ceph-mon[32129]: pgmap v776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:12:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:06 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 369.532978 secs 2024-07-26T11:12:07.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:06 smithi160 ceph-mon[36715]: slow request 364.672167 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:12:07.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:06 smithi160 ceph-mon[36715]: pgmap v776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:12:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:06 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 369.532978 secs 2024-07-26T11:12:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:06 smithi183 ceph-mon[37273]: slow request 364.672167 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:12:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:06 smithi183 ceph-mon[37273]: pgmap v776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:12:09.384 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:09 smithi144 ceph-mon[32129]: pgmap v777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:09 smithi160 ceph-mon[36715]: pgmap v777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:09 smithi183 ceph-mon[37273]: pgmap v777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:11 smithi160 ceph-mon[36715]: pgmap v778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 374.533120 secs 2024-07-26T11:12:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:11 smithi183 ceph-mon[37273]: pgmap v778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 374.533120 secs 2024-07-26T11:12:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:11 smithi144 ceph-mon[32129]: pgmap v778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 374.533120 secs 2024-07-26T11:12:11.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:11.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:11.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:11.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:11.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:11.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:13 smithi160 ceph-mon[36715]: pgmap v779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 176 B/s wr, 6 op/s 2024-07-26T11:12:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:13 smithi183 ceph-mon[37273]: pgmap v779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 176 B/s wr, 6 op/s 2024-07-26T11:12:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:13 smithi144 ceph-mon[32129]: pgmap v779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 176 B/s wr, 6 op/s 2024-07-26T11:12:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:15 smithi160 ceph-mon[36715]: pgmap v780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:12:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:15 smithi183 ceph-mon[37273]: pgmap v780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:12:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:15 smithi144 ceph-mon[32129]: pgmap v780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:12:16.078 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:15 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:12:16.079 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:15 smithi160 systemd[1]: Stopped Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:12:16.079 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:15 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 1.983s CPU time. 2024-07-26T11:12:16.079 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:15 smithi160 systemd[1]: Starting Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:12:16.079 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 podman[72532]: 2024-07-26 11:12:15.945800591 +0000 UTC m=+0.017562322 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:12:16.079 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 podman[72532]: 2024-07-26 11:12:16.077771854 +0000 UTC m=+0.149533574 container create fd430f5d24fe547487b4af4051b37d45f6a67c05a1ae05f7d251cf649bbf48a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T11:12:16.350 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 379.533241 secs 2024-07-26T11:12:16.350 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 podman[72532]: 2024-07-26 11:12:16.313808026 +0000 UTC m=+0.385569746 container init fd430f5d24fe547487b4af4051b37d45f6a67c05a1ae05f7d251cf649bbf48a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-26T11:12:16.350 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 podman[72532]: 2024-07-26 11:12:16.316596126 +0000 UTC m=+0.388357850 container start fd430f5d24fe547487b4af4051b37d45f6a67c05a1ae05f7d251cf649bbf48a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T11:12:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 379.533241 secs 2024-07-26T11:12:16.675 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 ceph-mds[72550]: -- 172.21.15.160:0/2743658686 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5597d6a13860 con 0x5597dabb5600 2024-07-26T11:12:16.675 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: starting mds.k at 2024-07-26T11:12:16.675 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 bash[72532]: fd430f5d24fe547487b4af4051b37d45f6a67c05a1ae05f7d251cf649bbf48a6 2024-07-26T11:12:16.675 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:16 smithi160 systemd[1]: Started Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:12:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 379.533241 secs 2024-07-26T11:12:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: pgmap v781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:12:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: daemon mds.k restarted 2024-07-26T11:12:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:17.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:12:17.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T11:12:17.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:boot 2024-07-26T11:12:17.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:12:17.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:17.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:17 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: pgmap v781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: daemon mds.k restarted 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:boot 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:17 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: pgmap v781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:12:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: daemon mds.k restarted 2024-07-26T11:12:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:12:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T11:12:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:boot 2024-07-26T11:12:17.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:12:17.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:17.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:17 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:18.642 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:18 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:resolve 2024-07-26T11:12:18.642 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:18 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:18.642 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:18 smithi144 ceph-mon[32129]: pgmap v783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:12:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:18 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:resolve 2024-07-26T11:12:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:18 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:18 smithi160 ceph-mon[36715]: pgmap v783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:12:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:18 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:resolve 2024-07-26T11:12:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:18 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:18 smithi183 ceph-mon[37273]: pgmap v783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:12:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:19 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:reconnect 2024-07-26T11:12:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:19 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:19 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:12:19.799 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:19 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:reconnect 2024-07-26T11:12:19.799 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:19 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:19.799 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:19 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:12:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:19 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:reconnect 2024-07-26T11:12:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:19 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:19 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:12:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:20 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:rejoin 2024-07-26T11:12:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:20 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:20 smithi144 ceph-mon[32129]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:12:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:20 smithi144 ceph-mon[32129]: pgmap v784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:20.745 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:rejoin 2024-07-26T11:12:20.745 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:20.746 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-mon[37273]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:12:20.746 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-mon[37273]: pgmap v784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:20.746 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:12:20.746 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: *** Caught signal (Aborted) ** 2024-07-26T11:12:20.746 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: in thread 7fbb3e4bb640 thread_name:ms_dispatch 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb438926f0] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb438df94c] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 3: raise() 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 4: abort() 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb4387c71b] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb4388b386] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56250c3a16b9] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56250c55f80c] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56250c49f2ca] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56250c46f753] 2024-07-26T11:12:20.747 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56250c470a1b] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56250c454752] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56250c3a2a56] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56250c3a6637] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56250c3a6f5e] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56250c38a03a] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb4415f407] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb4415fcf9] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb4423c5a1] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb43fe9cdd] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb43fe9cf9] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb438ddc02] 2024-07-26T11:12:20.748 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb43962c40] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.455+0000 7fbb3e4bb640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: in thread 7fbb3e4bb640 thread_name:ms_dispatch 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb438926f0] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb438df94c] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 3: raise() 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 4: abort() 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb4387c71b] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb4388b386] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56250c3a16b9] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56250c55f80c] 2024-07-26T11:12:20.749 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56250c49f2ca] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56250c46f753] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56250c470a1b] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56250c454752] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56250c3a2a56] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56250c3a6637] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56250c3a6f5e] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56250c38a03a] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb4415f407] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb4415fcf9] 2024-07-26T11:12:20.750 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb4423c5a1] 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb43fe9cdd] 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb43fe9cf9] 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb438ddc02] 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb43962c40] 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 0> 2024-07-26T11:12:20.455+0000 7fbb3e4bb640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: in thread 7fbb3e4bb640 thread_name:ms_dispatch 2024-07-26T11:12:20.751 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb438926f0] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb438df94c] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 3: raise() 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 4: abort() 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb4387c71b] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb4388b386] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56250c3a16b9] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56250c55f80c] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56250c49f2ca] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56250c46f753] 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56250c470a1b] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56250c454752] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56250c3a2a56] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56250c3a6637] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56250c3a6f5e] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56250c38a03a] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb4415f407] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb4415fcf9] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb4423c5a1] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb43fe9cdd] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb43fe9cf9] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb438ddc02] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb43962c40] 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:20.753 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 0> 2024-07-26T11:12:20.455+0000 7fbb3e4bb640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: in thread 7fbb3e4bb640 thread_name:ms_dispatch 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb438926f0] 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb438df94c] 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 3: raise() 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 4: abort() 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb4387c71b] 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb4388b386] 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56250c3a16b9] 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56250c55f80c] 2024-07-26T11:12:20.754 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56250c49f2ca] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56250c46f753] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56250c470a1b] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56250c454752] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56250c3a2a56] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56250c3a6637] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56250c3a6f5e] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56250c38a03a] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb4415f407] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb4415fcf9] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb4423c5a1] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb43fe9cdd] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb43fe9cf9] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb438ddc02] 2024-07-26T11:12:20.755 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb43962c40] 2024-07-26T11:12:20.756 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:20.756 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[68015]: 2024-07-26T11:12:20.756 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:20 smithi183 podman[75135]: 2024-07-26 11:12:20.517071629 +0000 UTC m=+0.019515405 container died 44e1580f9f8c8e46d7c7e73e39f1d57926ec04d03844f54906967e118bcffeb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T11:12:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:20 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:rejoin 2024-07-26T11:12:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:20 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:20 smithi160 ceph-mon[36715]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:12:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:20 smithi160 ceph-mon[36715]: pgmap v784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:21.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:21 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:21.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:21 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:active 2024-07-26T11:12:21.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:21 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:21.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:21 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:21.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 384.533324 secs 2024-07-26T11:12:21.466 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:21 smithi183 podman[75135]: 2024-07-26 11:12:21.208778853 +0000 UTC m=+0.711222609 container remove 44e1580f9f8c8e46d7c7e73e39f1d57926ec04d03844f54906967e118bcffeb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T11:12:21.467 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:21 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:12:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:21 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:21 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:active 2024-07-26T11:12:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:21 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:21 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 384.533324 secs 2024-07-26T11:12:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:21 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:21 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/4222472223,v1:172.21.15.183:6833/4222472223] up:active 2024-07-26T11:12:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:21 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:21 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 384.533324 secs 2024-07-26T11:12:21.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:21 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:12:21.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:21 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 2.168s CPU time. 2024-07-26T11:12:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:22 smithi144 ceph-mon[32129]: pgmap v785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:12:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:22 smithi160 ceph-mon[36715]: pgmap v785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:12:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:22 smithi183 ceph-mon[37273]: pgmap v785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:12:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:25 smithi160 ceph-mon[36715]: pgmap v786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:25 smithi183 ceph-mon[37273]: pgmap v786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:25 smithi144 ceph-mon[32129]: pgmap v786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 389.533440 secs 2024-07-26T11:12:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 389.533440 secs 2024-07-26T11:12:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 389.533440 secs 2024-07-26T11:12:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:26.886+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:26.887+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:26.887+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:26.887+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:26.887+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:27 smithi160 ceph-mon[36715]: pgmap v787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:27 smithi183 ceph-mon[37273]: pgmap v787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:27 smithi144 ceph-mon[32129]: pgmap v787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:29.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:12:29.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:29 smithi144 ceph-mon[32129]: pgmap v788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 172 B/s wr, 6 op/s 2024-07-26T11:12:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:29 smithi160 ceph-mon[36715]: pgmap v788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 172 B/s wr, 6 op/s 2024-07-26T11:12:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:29 smithi183 ceph-mon[37273]: pgmap v788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 172 B/s wr, 6 op/s 2024-07-26T11:12:29.909 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:31 smithi160 ceph-mon[36715]: pgmap v789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 394.533557 secs 2024-07-26T11:12:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:31 smithi183 ceph-mon[37273]: pgmap v789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 394.533557 secs 2024-07-26T11:12:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:31 smithi144 ceph-mon[32129]: pgmap v789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:31.756 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 394.533557 secs 2024-07-26T11:12:32.002 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:31 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:12:32.002 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:31 smithi183 systemd[1]: Stopped Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:12:32.002 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:31 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 2.168s CPU time. 2024-07-26T11:12:32.002 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:31 smithi183 systemd[1]: Starting Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:12:32.002 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:31 smithi183 podman[75225]: 2024-07-26 11:12:31.776394401 +0000 UTC m=+0.017668799 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:12:32.003 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:31 smithi183 podman[75225]: 2024-07-26 11:12:31.875670496 +0000 UTC m=+0.116944897 container create a307ec7760ef66254c36acc5a6a8ece27c36906f4246cb4bf58bebce99918456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.270+0000 7efeb0e00640 1 -- 172.21.15.144:0/1364742410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1099f0 msgr2=0x7efeac109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.270+0000 7efeb0e00640 1 --2- 172.21.15.144:0/1364742410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1099f0 0x7efeac109e00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009900 tx=0x7efe9c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 -- 172.21.15.144:0/1364742410 shutdown_connections 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 --2- 172.21.15.144:0/1364742410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac10fc80 0x7efeac1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 --2- 172.21.15.144:0/1364742410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac10f330 0x7efeac10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 --2- 172.21.15.144:0/1364742410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1099f0 0x7efeac109e00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 -- 172.21.15.144:0/1364742410 >> 172.21.15.144:0/1364742410 conn(0x7efeac078ff0 msgr2=0x7efeac107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:32.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 -- 172.21.15.144:0/1364742410 shutdown_connections 2024-07-26T11:12:32.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.271+0000 7efeb0e00640 1 -- 172.21.15.144:0/1364742410 wait complete. 2024-07-26T11:12:32.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.272+0000 7efeb0e00640 1 Processor -- start 2024-07-26T11:12:32.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.272+0000 7efeb0e00640 1 -- start start 2024-07-26T11:12:32.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeb0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 0x7efeac1a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeb0e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac10f330 0x7efeac1a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeb0e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac10fc80 0x7efeac1a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeb0e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efeac119990 con 0x7efeac10f330 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeb0e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efeac119810 con 0x7efeac1099f0 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeb0e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efeac119b10 con 0x7efeac10fc80 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 0x7efeac1a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efea2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac10f330 0x7efeac1a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 0x7efeac1a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55184/0 (socket says 172.21.15.144:55184) 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeab400640 1 -- 172.21.15.144:0/331700691 learned_addr learned my addr 172.21.15.144:0/331700691 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:12:32.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.273+0000 7efeabe00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac10fc80 0x7efeac1a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeabe00640 1 -- 172.21.15.144:0/331700691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 msgr2=0x7efeac1a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeabe00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 0x7efeac1a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeabe00640 1 -- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac10f330 msgr2=0x7efeac1a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeabe00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac10f330 0x7efeac1a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeabe00640 1 -- 172.21.15.144:0/331700691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac1a62d0 con 0x7efeac10fc80 2024-07-26T11:12:32.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeab400640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 0x7efeac1a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:32.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efeabe00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac10fc80 0x7efeac1a5ba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efea400e940 tx=0x7efea400ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:32.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.274+0000 7efea8c00640 1 -- 172.21.15.144:0/331700691 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4004d20 con 0x7efeac10fc80 2024-07-26T11:12:32.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.275+0000 7efea8c00640 1 -- 172.21.15.144:0/331700691 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efea4005020 con 0x7efeac10fc80 2024-07-26T11:12:32.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.275+0000 7efea8c00640 1 -- 172.21.15.144:0/331700691 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea40116d0 con 0x7efeac10fc80 2024-07-26T11:12:32.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.275+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeac1a65c0 con 0x7efeac10fc80 2024-07-26T11:12:32.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.275+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeac1acd20 con 0x7efeac10fc80 2024-07-26T11:12:32.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.278+0000 7efeb91c5740 1 Processor -- start 2024-07-26T11:12:32.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.278+0000 7efea8c00640 1 -- 172.21.15.144:0/331700691 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7efea4005460 con 0x7efeac10fc80 2024-07-26T11:12:32.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.278+0000 7efeb91c5740 1 -- start start 2024-07-26T11:12:32.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 0x564ab31c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeb91c5740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564ab313abc0 con 0x564ab2e1f070 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeb91c5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564ab312a0c0 con 0x564ab2df61b0 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeb91c5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564ab312e100 con 0x564ab2dfe390 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efea2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 0x564ab31c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efea2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 0x564ab31c4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41466/0 (socket says 172.21.15.144:41466) 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.279+0000 7efea2a00640 1 -- 172.21.15.144:0/723772361 learned_addr learned my addr 172.21.15.144:0/723772361 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:12:32.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efea2a00640 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efeab400640 1 -- 172.21.15.144:0/723772361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 msgr2=0x564ab31c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efeab400640 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efeab400640 1 -- 172.21.15.144:0/723772361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 msgr2=0x564ab31c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efeab400640 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 0x564ab31c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efeab400640 1 -- 172.21.15.144:0/723772361 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab2e8a650 con 0x564ab2df61b0 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efea2a00640 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 0x564ab31c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efeab400640 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c4540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe9c000c00 tx=0x7efe9c0035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34e819b21040cd9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:32.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.280+0000 7efea8c00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efe900b3d40 0x7efe900b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.281+0000 7efea8c00640 1 -- 172.21.15.144:0/331700691 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 11544+0+0 (secure 0 0 0) 0x7efea40e4c90 con 0x7efeac10fc80 2024-07-26T11:12:32.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.281+0000 7efea2a00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efe900b3d40 0x7efe900b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.281+0000 7efea0c00640 1 -- 172.21.15.144:0/723772361 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c040070 con 0x564ab2df61b0 2024-07-26T11:12:32.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.281+0000 7efea0c00640 1 -- 172.21.15.144:0/723772361 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe9c0047a0 con 0x564ab2df61b0 2024-07-26T11:12:32.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.281+0000 7efea0c00640 1 -- 172.21.15.144:0/723772361 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c035040 con 0x564ab2df61b0 2024-07-26T11:12:32.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 -- 172.21.15.144:0/723772361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 msgr2=0x564ab31c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.285 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efea2a00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efe900b3d40 0x7efe900b6250 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efe9400bfd0 tx=0x7efe9400b770 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c4540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe9c000c00 tx=0x7efe9c0035f0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 -- 172.21.15.144:0/723772361 shutdown_connections 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c4540 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 --2- 172.21.15.144:0/723772361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2e1f070 0x564ab31c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 -- 172.21.15.144:0/723772361 >> 172.21.15.144:0/723772361 conn(0x564ab2e13460 msgr2=0x564ab31c1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 -- 172.21.15.144:0/723772361 shutdown_connections 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.282+0000 7efeb91c5740 1 -- 172.21.15.144:0/723772361 wait complete. 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.283+0000 7efeb91c5740 1 -- start start 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.283+0000 7efeb91c5740 1 Processor -- start 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.286 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564ab2f8d700 con 0x564ab2dfe390 2024-07-26T11:12:32.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564ab30c6690 con 0x564ab2df61b0 2024-07-26T11:12:32.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564ab31d2780 con 0x564ab2e1f070 2024-07-26T11:12:32.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeb91c5740 10 client.0 _ll_register_callbacks cb 0x564ab30d6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:12:32.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efea2a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeabe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55206/0 (socket says 172.21.15.144:55206) 2024-07-26T11:12:32.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 -- 172.21.15.144:0/2735163124 learned_addr learned my addr 172.21.15.144:0/2735163124 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 -- 172.21.15.144:0/2735163124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 msgr2=0x564ab31d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 -- 172.21.15.144:0/2735163124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 msgr2=0x564ab31cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.284+0000 7efeab400640 1 -- 172.21.15.144:0/2735163124 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab30dd830 con 0x564ab2df61b0 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.285+0000 7efeabe00640 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.285+0000 7efeab400640 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31cf620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009a20 tx=0x7efe9c003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39cb3991b33518c9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.285+0000 7efe99600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.285+0000 7efe99600640 1 -- 172.21.15.144:0/2735163124 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c035070 con 0x564ab2df61b0 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.285+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe9c0047a0 con 0x564ab2df61b0 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.285+0000 7efe99600640 1 -- 172.21.15.144:0/2735163124 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 11544+0+0 (secure 0 0 0) 0x7efe9c0316a0 con 0x564ab2df61b0 2024-07-26T11:12:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efe99600640 1 -- 172.21.15.144:0/2735163124 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7efe9c040040 con 0x564ab2df61b0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 10 client.26390 fetch_fsmap learned FSMap version 193 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=4+}) -- 0x564ab31de3c0 con 0x564ab2df61b0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efe99600640 1 -- 172.21.15.144:0/2735163124 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe9c0047a0 con 0x564ab2df61b0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efe99600640 1 -- 172.21.15.144:0/2735163124 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 193) ==== 6543+0+0 (secure 0 0 0) 0x7efe9c051b40 con 0x564ab2df61b0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 10 client.26390 fetch_fsmap finished waiting for FSMap version 193 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 10 client.26390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 20 client.26390 populate_metadata read hostname 'smithi144' 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2ddd9f0 0x564ab31e1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 4 client.26390 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.286+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ab313cb10 con 0x564ab2ddd9f0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.287+0000 7efea2a00640 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2ddd9f0 0x564ab31e1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.287+0000 7efea2a00640 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2ddd9f0 0x564ab31e1720 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=770cb23a68e136c5 server_cookie=bc06e7cce29fb651 in_seq=0 out_seq=0 2024-07-26T11:12:32.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.287+0000 7efe99600640 10 client.26390 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:12:32.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efe99600640 1 -- 172.21.15.144:0/2735163124 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ab313cb10 con 0x564ab2ddd9f0 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efe99600640 10 client.26390 handle_command_reply: tid=0 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 client.26390 shutdown 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 msgr2=0x564ab31cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31cf620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009a20 tx=0x7efe9c003a80 comp rx=0 tx=0).stop 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 shutdown_connections 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2ddd9f0 0x564ab31e1720 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 --2- 172.21.15.144:0/2735163124 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31cf620 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 >> 172.21.15.144:0/2735163124 conn(0x564ab2e13460 msgr2=0x564ab31c71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.288+0000 7efe99600640 0 client.26390 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:12:32.293 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.289+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 shutdown_connections 2024-07-26T11:12:32.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.289+0000 7efeb91c5740 1 -- 172.21.15.144:0/2735163124 wait complete. 2024-07-26T11:12:32.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.289+0000 7efeb91c5740 20 client.26390 trim_cache size 0 max 16384 2024-07-26T11:12:32.307 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:32 smithi183 podman[75225]: 2024-07-26 11:12:32.08651522 +0000 UTC m=+0.327789628 container init a307ec7760ef66254c36acc5a6a8ece27c36906f4246cb4bf58bebce99918456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:12:32.307 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:32 smithi183 podman[75225]: 2024-07-26 11:12:32.089302716 +0000 UTC m=+0.330577110 container start a307ec7760ef66254c36acc5a6a8ece27c36906f4246cb4bf58bebce99918456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:12:32.307 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mds[75245]: -- 172.21.15.183:0/4036165773 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557714a39860 con 0x557718bdb600 2024-07-26T11:12:32.307 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: starting mds.l at 2024-07-26T11:12:32.307 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:32 smithi183 bash[75225]: a307ec7760ef66254c36acc5a6a8ece27c36906f4246cb4bf58bebce99918456 2024-07-26T11:12:32.307 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:32 smithi183 systemd[1]: Started Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:12:32.313 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.311+0000 7efeb91c5740 1 Processor -- start 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: daemon mds.l restarted 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:boot 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:12:32.314 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:32.315 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:32.315 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:32 smithi144 ceph-mon[32129]: pgmap v791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.311+0000 7efeb91c5740 1 -- start start 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31e72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeb91c5740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b9b00 con 0x564ab2dfe390 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeb91c5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b8d60 con 0x564ab2df61b0 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeb91c5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b9430 con 0x564ab2e1f070 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeabe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31e72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.312+0000 7efeab400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55218/0 (socket says 172.21.15.144:55218) 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeab400640 1 -- 172.21.15.144:0/69496196 learned_addr learned my addr 172.21.15.144:0/69496196 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efea2a00640 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31e6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeabe00640 1 -- 172.21.15.144:0/69496196 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 msgr2=0x564ab31c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeabe00640 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeabe00640 1 -- 172.21.15.144:0/69496196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 msgr2=0x564ab31e6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeabe00640 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31e6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeabe00640 1 -- 172.21.15.144:0/69496196 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab31d08e0 con 0x564ab2e1f070 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeab400640 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.313+0000 7efeabe00640 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31e72f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efea40e17b0 tx=0x7efea40aa830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=886352373fca4057 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.314+0000 7efe9be00640 1 -- 172.21.15.144:0/69496196 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea40cd070 con 0x564ab2e1f070 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.314+0000 7efe9be00640 1 -- 172.21.15.144:0/69496196 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efea40b3d20 con 0x564ab2e1f070 2024-07-26T11:12:32.317 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.314+0000 7efe9be00640 1 -- 172.21.15.144:0/69496196 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea401f040 con 0x564ab2e1f070 2024-07-26T11:12:32.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 -- 172.21.15.144:0/69496196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 msgr2=0x564ab31e72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31e72f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efea40e17b0 tx=0x7efea40aa830 comp rx=0 tx=0).stop 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 -- 172.21.15.144:0/69496196 shutdown_connections 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2e1f070 0x564ab31e72f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2dfe390 0x564ab31e6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 --2- 172.21.15.144:0/69496196 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2df61b0 0x564ab31c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 -- 172.21.15.144:0/69496196 >> 172.21.15.144:0/69496196 conn(0x564ab2e13460 msgr2=0x564ab31d1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 -- 172.21.15.144:0/69496196 shutdown_connections 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 -- 172.21.15.144:0/69496196 wait complete. 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.315+0000 7efeb91c5740 1 -- start start 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 Processor -- start 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2df61b0 0x564ab31d5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31d5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2e1f070 0x564ab3240db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b6060 con 0x564ab2df61b0 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b7470 con 0x564ab2e1f070 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b66d0 con 0x564ab2dfe390 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.316+0000 7efeb91c5740 10 client.0 _ll_register_callbacks cb 0x7efe9c009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efeab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2df61b0 0x564ab31d5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31d5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31d5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:50266/0 (socket says 172.21.15.144:50266) 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efeab400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2df61b0 0x564ab31d5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41488/0 (socket says 172.21.15.144:41488) 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 -- 172.21.15.144:0/1700554881 learned_addr learned my addr 172.21.15.144:0/1700554881 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efeabe00640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2e1f070 0x564ab3240db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 -- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2e1f070 msgr2=0x564ab3240db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2e1f070 0x564ab3240db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 -- 172.21.15.144:0/1700554881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2df61b0 msgr2=0x564ab31d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2df61b0 0x564ab31d5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efea2a00640 1 -- 172.21.15.144:0/1700554881 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab3241420 con 0x564ab2dfe390 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.317+0000 7efeabe00640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2e1f070 0x564ab3240db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efea2a00640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31d5a30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x564ab31e1d20 tx=0x7efe94004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bd18ec486a3998d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efe99600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efe99600640 1 -- 172.21.15.144:0/1700554881 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe94030070 con 0x564ab2dfe390 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efe99600640 1 -- 172.21.15.144:0/1700554881 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 11544+0+0 (secure 0 0 0) 0x7efe9400edd0 con 0x564ab2dfe390 2024-07-26T11:12:32.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea40b3d20 con 0x564ab2dfe390 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efe99600640 1 -- 172.21.15.144:0/1700554881 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7efe94038050 con 0x564ab2dfe390 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efeb91c5740 10 client.26422 fetch_fsmap learned FSMap version 193 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=4+}) -- 0x564ab3249960 con 0x564ab2dfe390 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.318+0000 7efe99600640 1 -- 172.21.15.144:0/1700554881 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efea40b3d20 con 0x564ab2dfe390 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efe99600640 1 -- 172.21.15.144:0/1700554881 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 193) ==== 6543+0+0 (secure 0 0 0) 0x7efe94033030 con 0x564ab2dfe390 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeb91c5740 10 client.26422 fetch_fsmap finished waiting for FSMap version 193 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeb91c5740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeb91c5740 20 client.26422 populate_metadata read hostname 'smithi144' 2024-07-26T11:12:32.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeb91c5740 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2db2da0 0x564ab324c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeb91c5740 4 client.26422 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564ab31385b0 con 0x564ab2db2da0 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.319+0000 7efeab400640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2db2da0 0x564ab324c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efe99600640 10 client.26422 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeab400640 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2db2da0 0x564ab324c130 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f8d664d7a399ceb server_cookie=2a14e9ccddb02ab9 in_seq=0 out_seq=0 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efe99600640 1 -- 172.21.15.144:0/1700554881 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564ab31385b0 con 0x564ab2db2da0 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efe99600640 10 client.26422 handle_command_reply: tid=0 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 client.26422 shutdown 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 msgr2=0x564ab31d5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31d5a30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x564ab31e1d20 tx=0x7efe94004730 comp rx=0 tx=0).stop 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 shutdown_connections 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564ab2db2da0 0x564ab324c130 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564ab2e1f070 0x564ab3240db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564ab2dfe390 0x564ab31d5a30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 --2- 172.21.15.144:0/1700554881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564ab2df61b0 0x564ab31d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 >> 172.21.15.144:0/1700554881 conn(0x564ab2e13460 msgr2=0x564ab31da880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.320+0000 7efe99600640 0 client.26422 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.321+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 shutdown_connections 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.321+0000 7efeb91c5740 1 -- 172.21.15.144:0/1700554881 wait complete. 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.321+0000 7efeb91c5740 20 client.26422 trim_cache size 0 max 16384 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:12:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:12:32.327 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efe900b3d40 msgr2=0x7efe900b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efe900b3d40 0x7efe900b6250 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efe9400bfd0 tx=0x7efe9400b770 comp rx=0 tx=0).stop 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac10fc80 msgr2=0x7efeac1a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac10fc80 0x7efeac1a5ba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efea400e940 tx=0x7efea400ee00 comp rx=0 tx=0).stop 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 shutdown_connections 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7efe900b3d40 0x7efe900b6250 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac10fc80 0x7efeac1a5ba0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac10f330 0x7efeac1a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 --2- 172.21.15.144:0/331700691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efeac1099f0 0x7efeac1a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 >> 172.21.15.144:0/331700691 conn(0x7efeac078ff0 msgr2=0x7efeac0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:32.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 shutdown_connections 2024-07-26T11:12:32.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:12:32.322+0000 7efeb0e00640 1 -- 172.21.15.144:0/331700691 wait complete. 2024-07-26T11:12:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: daemon mds.l restarted 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:boot 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:32 smithi160 ceph-mon[36715]: pgmap v791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: daemon mds.l restarted 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:boot 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:32 smithi183 ceph-mon[37273]: pgmap v791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:12:33.113 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:12:34.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:34 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:resolve 2024-07-26T11:12:34.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:34 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:34.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:34 smithi144 ceph-mon[32129]: pgmap v792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:12:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:34 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:resolve 2024-07-26T11:12:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:34 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:34 smithi183 ceph-mon[37273]: pgmap v792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:12:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:34 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:resolve 2024-07-26T11:12:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:34 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:34 smithi160 ceph-mon[36715]: pgmap v792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:12:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:35 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:reconnect 2024-07-26T11:12:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:35 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:35 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:12:35.640 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:35 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:reconnect 2024-07-26T11:12:35.640 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:35 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:35.640 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:35 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:12:35.670 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:reconnect 2024-07-26T11:12:35.670 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:35.670 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: *** Caught signal (Aborted) ** 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: in thread 7f115434f640 thread_name:ms_dispatch 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f11597266f0] 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f115977394c] 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 3: raise() 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 4: abort() 2024-07-26T11:12:36.053 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 5: /lib64/libc.so.6(+0x2871b) [0x7f115971071b] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 6: /lib64/libc.so.6(+0x37386) [0x7f115971f386] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5597d4b7a6b9] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5597d4d3880c] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5597d4c782ca] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5597d4c48753] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5597d4c49a1b] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5597d4c2d752] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5597d4b7ba56] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5597d4b7f637] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5597d4b7ff5e] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5597d4b6303a] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1159ff3407] 2024-07-26T11:12:36.054 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 18: (DispatchQueue::entry()+0x60f) [0x7f1159ff3cf9] 2024-07-26T11:12:36.055 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f115a0d05a1] 2024-07-26T11:12:36.055 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 20: (Thread::entry_wrapper()+0x43) [0x7f1159e7dcdd] 2024-07-26T11:12:36.055 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1159e7dcf9] 2024-07-26T11:12:36.055 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1159771c02] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f11597f6c40] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:35.668+0000 7f115434f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: in thread 7f115434f640 thread_name:ms_dispatch 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f11597266f0] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f115977394c] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 3: raise() 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 4: abort() 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 5: /lib64/libc.so.6(+0x2871b) [0x7f115971071b] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 6: /lib64/libc.so.6(+0x37386) [0x7f115971f386] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5597d4b7a6b9] 2024-07-26T11:12:36.056 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5597d4d3880c] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5597d4c782ca] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5597d4c48753] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5597d4c49a1b] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5597d4c2d752] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5597d4b7ba56] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5597d4b7f637] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5597d4b7ff5e] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5597d4b6303a] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1159ff3407] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 18: (DispatchQueue::entry()+0x60f) [0x7f1159ff3cf9] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f115a0d05a1] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 20: (Thread::entry_wrapper()+0x43) [0x7f1159e7dcdd] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1159e7dcf9] 2024-07-26T11:12:36.057 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1159771c02] 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f11597f6c40] 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 0> 2024-07-26T11:12:35.668+0000 7f115434f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: in thread 7f115434f640 thread_name:ms_dispatch 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f11597266f0] 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f115977394c] 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 3: raise() 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 4: abort() 2024-07-26T11:12:36.058 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 5: /lib64/libc.so.6(+0x2871b) [0x7f115971071b] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 6: /lib64/libc.so.6(+0x37386) [0x7f115971f386] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5597d4b7a6b9] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5597d4d3880c] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5597d4c782ca] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5597d4c48753] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5597d4c49a1b] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5597d4c2d752] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5597d4b7ba56] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5597d4b7f637] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5597d4b7ff5e] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5597d4b6303a] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1159ff3407] 2024-07-26T11:12:36.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 18: (DispatchQueue::entry()+0x60f) [0x7f1159ff3cf9] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f115a0d05a1] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 20: (Thread::entry_wrapper()+0x43) [0x7f1159e7dcdd] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1159e7dcf9] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1159771c02] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f11597f6c40] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 0> 2024-07-26T11:12:35.668+0000 7f115434f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: in thread 7f115434f640 thread_name:ms_dispatch 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f11597266f0] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f115977394c] 2024-07-26T11:12:36.060 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 3: raise() 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 4: abort() 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 5: /lib64/libc.so.6(+0x2871b) [0x7f115971071b] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 6: /lib64/libc.so.6(+0x37386) [0x7f115971f386] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5597d4b7a6b9] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5597d4d3880c] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5597d4c782ca] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5597d4c48753] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5597d4c49a1b] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5597d4c2d752] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5597d4b7ba56] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5597d4b7f637] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5597d4b7ff5e] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5597d4b6303a] 2024-07-26T11:12:36.061 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1159ff3407] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 18: (DispatchQueue::entry()+0x60f) [0x7f1159ff3cf9] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f115a0d05a1] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 20: (Thread::entry_wrapper()+0x43) [0x7f1159e7dcdd] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1159e7dcf9] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1159771c02] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f11597f6c40] 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72546]: 2024-07-26T11:12:36.062 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:35 smithi160 podman[72596]: 2024-07-26 11:12:35.724594491 +0000 UTC m=+0.018685462 container died fd430f5d24fe547487b4af4051b37d45f6a67c05a1ae05f7d251cf649bbf48a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:36 smithi160 podman[72596]: 2024-07-26 11:12:36.368934854 +0000 UTC m=+0.663025825 container remove fd430f5d24fe547487b4af4051b37d45f6a67c05a1ae05f7d251cf649bbf48a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:36 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:rejoin 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 399.533676 secs 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:36.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:active 2024-07-26T11:12:36.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:36.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:36 smithi160 ceph-mon[36715]: pgmap v793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:rejoin 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 399.533676 secs 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:active 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:36 smithi183 ceph-mon[37273]: pgmap v793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:rejoin 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 399.533676 secs 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/1824839248,v1:172.21.15.160:6833/1824839248] up:active 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:36.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:36 smithi144 ceph-mon[32129]: pgmap v793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:12:37.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:36 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:12:37.059 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:36 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 2.022s CPU time. 2024-07-26T11:12:37.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:12:37 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:12:37.179+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:12:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:39 smithi160 ceph-mon[36715]: pgmap v794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:12:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:39 smithi183 ceph-mon[37273]: pgmap v794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:12:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:39 smithi144 ceph-mon[32129]: pgmap v794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:12:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:41 smithi160 ceph-mon[36715]: pgmap v795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 404.533816 secs 2024-07-26T11:12:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:41 smithi183 ceph-mon[37273]: pgmap v795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 404.533816 secs 2024-07-26T11:12:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:41 smithi144 ceph-mon[32129]: pgmap v795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:12:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 404.533816 secs 2024-07-26T11:12:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:41.900+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:41.901+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:41.901+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:41.901+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:41.901+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:43.407 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:43 smithi144 ceph-mon[32129]: pgmap v796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 202 B/s wr, 7 op/s 2024-07-26T11:12:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:43 smithi160 ceph-mon[36715]: pgmap v796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 202 B/s wr, 7 op/s 2024-07-26T11:12:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:43 smithi183 ceph-mon[37273]: pgmap v796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 202 B/s wr, 7 op/s 2024-07-26T11:12:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:45 smithi160 ceph-mon[36715]: pgmap v797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:12:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:45 smithi183 ceph-mon[37273]: pgmap v797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:12:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:45 smithi144 ceph-mon[32129]: pgmap v797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:12:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 409.533940 secs 2024-07-26T11:12:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 409.533940 secs 2024-07-26T11:12:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 409.533940 secs 2024-07-26T11:12:47.071 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:46 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:12:47.120 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:46 smithi160 systemd[1]: Stopped Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:12:47.121 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:46 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 2.022s CPU time. 2024-07-26T11:12:47.121 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:46 smithi160 systemd[1]: Starting Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:12:47.121 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 podman[72687]: 2024-07-26 11:12:46.951436454 +0000 UTC m=+0.017640033 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:12:47.121 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 podman[72687]: 2024-07-26 11:12:47.069799472 +0000 UTC m=+0.136003089 container create 0234712af09a9b49b5420b657de03dcbc7a8923c047da823b56d549fef454d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T11:12:47.331 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:47 smithi160 ceph-mon[36715]: pgmap v798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:12:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:47 smithi183 ceph-mon[37273]: pgmap v798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:12:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:47 smithi144 ceph-mon[32129]: pgmap v798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:12:47.809 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 podman[72687]: 2024-07-26 11:12:47.330923176 +0000 UTC m=+0.397126755 container init 0234712af09a9b49b5420b657de03dcbc7a8923c047da823b56d549fef454d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux ) 2024-07-26T11:12:47.810 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 podman[72687]: 2024-07-26 11:12:47.333629872 +0000 UTC m=+0.399833453 container start 0234712af09a9b49b5420b657de03dcbc7a8923c047da823b56d549fef454d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-07-26T11:12:47.810 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 ceph-mds[72706]: -- 172.21.15.160:0/3234989513 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a7dc3f860 con 0x557a81de1180 2024-07-26T11:12:47.810 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: starting mds.k at 2024-07-26T11:12:47.810 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 bash[72687]: 0234712af09a9b49b5420b657de03dcbc7a8923c047da823b56d549fef454d51 2024-07-26T11:12:47.810 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:12:47 smithi160 systemd[1]: Started Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:12:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: daemon mds.k restarted 2024-07-26T11:12:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:12:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T11:12:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:boot 2024-07-26T11:12:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:12:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:48 smithi183 ceph-mon[37273]: pgmap v800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:12:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: daemon mds.k restarted 2024-07-26T11:12:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:12:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T11:12:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:boot 2024-07-26T11:12:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:12:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:48 smithi144 ceph-mon[32129]: pgmap v800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:12:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: daemon mds.k restarted 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:boot 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:48 smithi160 ceph-mon[36715]: pgmap v800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:12:49.669 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:resolve 2024-07-26T11:12:49.669 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:49.669 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:49 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:49.669 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:49.669 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:resolve 2024-07-26T11:12:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:49 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:49.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:49.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:resolve 2024-07-26T11:12:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:49 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:50.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-mon[37273]: pgmap v801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:12:50.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-mon[37273]: pgmap v802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-26T11:12:50.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:12:50.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:reconnect 2024-07-26T11:12:50.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:50.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: *** Caught signal (Aborted) ** 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: in thread 7f70989cf640 thread_name:ms_dispatch 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709dda66f0] 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709ddf394c] 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 3: raise() 2024-07-26T11:12:50.688 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 4: abort() 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709dd9071b] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 6: /lib64/libc.so.6(+0x37386) [0x7f709dd9f386] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5577129246b9] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557712ae280c] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557712a222ca] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5577129f2753] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5577129f3a1b] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5577129d7752] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557712925a56] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557712929637] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557712929f5e] 2024-07-26T11:12:50.689 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55771290d03a] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e673407] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e673cf9] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709e7505a1] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e4fdcdd] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e4fdcf9] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709ddf1c02] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709de76c40] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.670+0000 7f70989cf640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: in thread 7f70989cf640 thread_name:ms_dispatch 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709dda66f0] 2024-07-26T11:12:50.690 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709ddf394c] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 3: raise() 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 4: abort() 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709dd9071b] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 6: /lib64/libc.so.6(+0x37386) [0x7f709dd9f386] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5577129246b9] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557712ae280c] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557712a222ca] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5577129f2753] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5577129f3a1b] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5577129d7752] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557712925a56] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557712929637] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557712929f5e] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55771290d03a] 2024-07-26T11:12:50.691 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e673407] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e673cf9] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709e7505a1] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e4fdcdd] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e4fdcf9] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709ddf1c02] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709de76c40] 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.692 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 0> 2024-07-26T11:12:50.670+0000 7f70989cf640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: in thread 7f70989cf640 thread_name:ms_dispatch 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709dda66f0] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709ddf394c] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 3: raise() 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 4: abort() 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709dd9071b] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 6: /lib64/libc.so.6(+0x37386) [0x7f709dd9f386] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5577129246b9] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557712ae280c] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557712a222ca] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5577129f2753] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5577129f3a1b] 2024-07-26T11:12:50.693 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5577129d7752] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557712925a56] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557712929637] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557712929f5e] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55771290d03a] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e673407] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e673cf9] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709e7505a1] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e4fdcdd] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e4fdcf9] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709ddf1c02] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709de76c40] 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:50.694 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.711 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:50 smithi144 ceph-mon[32129]: pgmap v801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:12:50.711 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:50 smithi144 ceph-mon[32129]: pgmap v802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-26T11:12:50.711 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:12:50.711 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:50 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:reconnect 2024-07-26T11:12:50.711 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:50 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:50.712 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:50 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:12:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:50 smithi160 ceph-mon[36715]: pgmap v801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:12:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:50 smithi160 ceph-mon[36715]: pgmap v802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-26T11:12:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:12:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:50 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:reconnect 2024-07-26T11:12:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:50 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:50.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:50 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:12:50.991 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 0> 2024-07-26T11:12:50.670+0000 7f70989cf640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:12:50.991 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: in thread 7f70989cf640 thread_name:ms_dispatch 2024-07-26T11:12:50.991 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.991 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709dda66f0] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709ddf394c] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 3: raise() 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 4: abort() 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709dd9071b] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 6: /lib64/libc.so.6(+0x37386) [0x7f709dd9f386] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5577129246b9] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557712ae280c] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557712a222ca] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5577129f2753] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5577129f3a1b] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5577129d7752] 2024-07-26T11:12:50.992 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557712925a56] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557712929637] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557712929f5e] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55771290d03a] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e673407] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e673cf9] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709e7505a1] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e4fdcdd] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e4fdcf9] 2024-07-26T11:12:50.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709ddf1c02] 2024-07-26T11:12:50.994 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709de76c40] 2024-07-26T11:12:50.994 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:12:50.994 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75241]: 2024-07-26T11:12:50.994 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:50 smithi183 podman[75408]: 2024-07-26 11:12:50.728292763 +0000 UTC m=+0.018964907 container died a307ec7760ef66254c36acc5a6a8ece27c36906f4246cb4bf58bebce99918456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:12:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:rejoin 2024-07-26T11:12:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:12:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 414.534055 secs 2024-07-26T11:12:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:51.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:active 2024-07-26T11:12:51.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:51 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:rejoin 2024-07-26T11:12:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:12:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 414.534055 secs 2024-07-26T11:12:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:active 2024-07-26T11:12:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:51 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:rejoin 2024-07-26T11:12:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:12:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:12:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 414.534055 secs 2024-07-26T11:12:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:12:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/13208120,v1:172.21.15.183:6833/13208120] up:active 2024-07-26T11:12:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:51 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:12:52.193 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:51 smithi183 podman[75408]: 2024-07-26 11:12:51.940893913 +0000 UTC m=+1.231566047 container remove a307ec7760ef66254c36acc5a6a8ece27c36906f4246cb4bf58bebce99918456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:12:52.193 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:51 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:12:52.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:12:52 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:12:52.180+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:12:52.635 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:52 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:12:52.635 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:12:52 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.950s CPU time. 2024-07-26T11:12:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:52 smithi144 ceph-mon[32129]: pgmap v803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-26T11:12:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:52 smithi160 ceph-mon[36715]: pgmap v803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-26T11:12:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:52 smithi183 ceph-mon[37273]: pgmap v803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-26T11:12:54.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:54 smithi144 ceph-mon[32129]: pgmap v804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 5 op/s 2024-07-26T11:12:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:54 smithi160 ceph-mon[36715]: pgmap v804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 5 op/s 2024-07-26T11:12:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:54 smithi183 ceph-mon[37273]: pgmap v804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 5 op/s 2024-07-26T11:12:56.915 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:56.913+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:56.914+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:56.914+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:56.914+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:12:56.915+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:56 smithi144 ceph-mon[32129]: pgmap v805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T11:12:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 419.534198 secs 2024-07-26T11:12:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:56 smithi160 ceph-mon[36715]: pgmap v805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T11:12:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 419.534198 secs 2024-07-26T11:12:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:56 smithi183 ceph-mon[37273]: pgmap v805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T11:12:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 419.534198 secs 2024-07-26T11:12:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:12:58 smithi144 ceph-mon[32129]: pgmap v806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 8 op/s 2024-07-26T11:12:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:12:58 smithi160 ceph-mon[36715]: pgmap v806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 8 op/s 2024-07-26T11:12:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:12:58 smithi183 ceph-mon[37273]: pgmap v806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 8 op/s 2024-07-26T11:13:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:00 smithi144 ceph-mon[32129]: pgmap v807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 424.534361 secs 2024-07-26T11:13:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:00 smithi160 ceph-mon[36715]: pgmap v807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 424.534361 secs 2024-07-26T11:13:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:00 smithi183 ceph-mon[37273]: pgmap v807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 424.534361 secs 2024-07-26T11:13:02.676 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:13:02.676 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 systemd[1]: Stopped Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:02.676 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.950s CPU time. 2024-07-26T11:13:02.676 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 systemd[1]: Starting Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:13:02.676 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 podman[75501]: 2024-07-26 11:13:02.521643415 +0000 UTC m=+0.018134297 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:13:02.676 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 podman[75501]: 2024-07-26 11:13:02.632636641 +0000 UTC m=+0.129127516 container create 87d3c075ce9921e175eeb7b4870bdc9cd7a9494bbaba6a8bd41a8e3e9bc0ed6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True) 2024-07-26T11:13:02.945 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:02 smithi183 ceph-mon[37273]: pgmap v808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:02.945 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 podman[75501]: 2024-07-26 11:13:02.85206786 +0000 UTC m=+0.348558755 container init 87d3c075ce9921e175eeb7b4870bdc9cd7a9494bbaba6a8bd41a8e3e9bc0ed6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-07-26T11:13:02.945 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 podman[75501]: 2024-07-26 11:13:02.854724733 +0000 UTC m=+0.351215609 container start 87d3c075ce9921e175eeb7b4870bdc9cd7a9494bbaba6a8bd41a8e3e9bc0ed6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:13:02.945 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 ceph-mds[75520]: -- 172.21.15.183:0/1444314697 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564674ff9860 con 0x56467919b180 2024-07-26T11:13:02.945 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: starting mds.l at 2024-07-26T11:13:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:02 smithi144 ceph-mon[32129]: pgmap v808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:02 smithi160 ceph-mon[36715]: pgmap v808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:03.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:13:03.206 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 bash[75501]: 87d3c075ce9921e175eeb7b4870bdc9cd7a9494bbaba6a8bd41a8e3e9bc0ed6e 2024-07-26T11:13:03.207 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:02 smithi183 systemd[1]: Started Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:03.568 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: daemon mds.l restarted 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: osdmap e117: 12 total, 12 up, 12 in 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:boot 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:03.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:03 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: daemon mds.l restarted 2024-07-26T11:13:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:13:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: osdmap e117: 12 total, 12 up, 12 in 2024-07-26T11:13:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:boot 2024-07-26T11:13:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:13:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:03 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:04.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: daemon mds.l restarted 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: osdmap e117: 12 total, 12 up, 12 in 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:boot 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:03 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:05.188 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:04 smithi144 ceph-mon[32129]: pgmap v810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:05.188 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:04 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:resolve 2024-07-26T11:13:05.188 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:04 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:05.246 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:04 smithi160 ceph-mon[36715]: pgmap v810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:05.246 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:04 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:resolve 2024-07-26T11:13:05.246 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:04 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:04 smithi183 ceph-mon[37273]: pgmap v810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:04 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:resolve 2024-07-26T11:13:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:04 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:06.061 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.058+0000 7f18eea00640 1 -- 172.21.15.144:0/4104978406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 msgr2=0x7f18e81026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.061 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.058+0000 7f18eea00640 1 --2- 172.21.15.144:0/4104978406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e81026b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009900 tx=0x7f18d802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.061 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.058+0000 7f18eea00640 1 -- 172.21.15.144:0/4104978406 shutdown_connections 2024-07-26T11:13:06.061 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.059+0000 7f18eea00640 1 --2- 172.21.15.144:0/4104978406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 0x7f18e810ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.059+0000 7f18eea00640 1 --2- 172.21.15.144:0/4104978406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 0x7f18e8107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.059+0000 7f18eea00640 1 --2- 172.21.15.144:0/4104978406 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e81026b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.059+0000 7f18eea00640 1 -- 172.21.15.144:0/4104978406 >> 172.21.15.144:0/4104978406 conn(0x7f18e80fdcf0 msgr2=0x7f18e8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:06.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.059+0000 7f18eea00640 1 -- 172.21.15.144:0/4104978406 shutdown_connections 2024-07-26T11:13:06.062 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.059+0000 7f18eea00640 1 -- 172.21.15.144:0/4104978406 wait complete. 2024-07-26T11:13:06.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.060+0000 7f18eea00640 1 Processor -- start 2024-07-26T11:13:06.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.060+0000 7f18eea00640 1 -- start start 2024-07-26T11:13:06.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18eea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e80fe220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18eea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 0x7f18e80fe760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18eea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 0x7f18e80feca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18eea00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f18e8112200 con 0x7f18e8108530 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18eea00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f18e8112080 con 0x7f18e81022a0 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18eea00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18e8112380 con 0x7f18e8107be0 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e80fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ecc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 0x7f18e80fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e80fe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:41930/0 (socket says 172.21.15.144:41930) 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 -- 172.21.15.144:0/2151821690 learned_addr learned my addr 172.21.15.144:0/2151821690 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ee000640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 0x7f18e80feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 -- 172.21.15.144:0/2151821690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 msgr2=0x7f18e80fe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 0x7f18e80fe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 -- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 msgr2=0x7f18e80feca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 0x7f18e80feca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ed600640 1 -- 172.21.15.144:0/2151821690 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e81a8080 con 0x7f18e81022a0 2024-07-26T11:13:06.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ecc00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 0x7f18e80fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:06.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.061+0000 7f18ee000640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 0x7f18e80feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:06.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.062+0000 7f18ed600640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e80fe220 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f18d8005940 tx=0x7f18d80040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.062+0000 7f18e6000640 1 -- 172.21.15.144:0/2151821690 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d803f030 con 0x7f18e81022a0 2024-07-26T11:13:06.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.062+0000 7f18e6000640 1 -- 172.21.15.144:0/2151821690 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18d80044c0 con 0x7f18e81022a0 2024-07-26T11:13:06.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.062+0000 7f18e6000640 1 -- 172.21.15.144:0/2151821690 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d8035070 con 0x7f18e81022a0 2024-07-26T11:13:06.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.062+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e81a8310 con 0x7f18e81022a0 2024-07-26T11:13:06.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.062+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e81a87a0 con 0x7f18e81022a0 2024-07-26T11:13:06.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 Processor -- start 2024-07-26T11:13:06.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18e6000640 1 -- 172.21.15.144:0/2151821690 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f18d8042030 con 0x7f18e81022a0 2024-07-26T11:13:06.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 -- start start 2024-07-26T11:13:06.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.068 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb51700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb51c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564f8cac7b90 con 0x564f8c78b390 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564f8cab7090 con 0x564f8c7ac070 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.066+0000 7f18f6dea740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564f8cabb0d0 con 0x564f8c7831b0 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ed600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb51700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:41942/0 (socket says 172.21.15.144:41942) 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 -- 172.21.15.144:0/592171 learned_addr learned my addr 172.21.15.144:0/592171 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:06.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ecc00640 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb51c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 -- 172.21.15.144:0/592171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 msgr2=0x564f8cb51700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb51700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 -- 172.21.15.144:0/592171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 msgr2=0x564f8cb51c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb51c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 -- 172.21.15.144:0/592171 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f8c817650 con 0x564f8c7ac070 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.067+0000 7f18ee000640 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb511c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18e0011790 tx=0x7f18e0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa20c852f3ff1cb1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.068+0000 7f18de600640 1 -- 172.21.15.144:0/592171 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0004d20 con 0x564f8c7ac070 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.068+0000 7f18de600640 1 -- 172.21.15.144:0/592171 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0003970 con 0x564f8c7ac070 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.068+0000 7f18de600640 1 -- 172.21.15.144:0/592171 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0004540 con 0x564f8c7ac070 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.068+0000 7f18e6000640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18cc0b3dc0 0x7f18cc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.068+0000 7f18ed600640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18cc0b3dc0 0x7f18cc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18e6000640 1 -- 172.21.15.144:0/2151821690 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 11716+0+0 (secure 0 0 0) 0x7f18d8104fe0 con 0x7f18e81022a0 2024-07-26T11:13:06.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 -- 172.21.15.144:0/592171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 msgr2=0x564f8cb511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.072 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb511c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18e0011790 tx=0x7f18e0011c50 comp rx=0 tx=0).stop 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 -- 172.21.15.144:0/592171 shutdown_connections 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18ed600640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18cc0b3dc0 0x7f18cc0b62d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x564f8cb4fe40 tx=0x7f18d8108ab0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb51c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb51700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 --2- 172.21.15.144:0/592171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb511c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 -- 172.21.15.144:0/592171 >> 172.21.15.144:0/592171 conn(0x564f8c7a0460 msgr2=0x564f8cb4f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 -- 172.21.15.144:0/592171 shutdown_connections 2024-07-26T11:13:06.073 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.069+0000 7f18f6dea740 1 -- 172.21.15.144:0/592171 wait complete. 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.070+0000 7f18f6dea740 1 -- start start 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.070+0000 7f18f6dea740 1 Processor -- start 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb5c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb5cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb5d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564f8c91a700 con 0x564f8c78b390 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564f8ca53660 con 0x564f8c7ac070 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564f8cb5f9a0 con 0x564f8c7831b0 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18f6dea740 10 client.0 _ll_register_callbacks cb 0x564f8ca63110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb5c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18ecc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb5cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18ee000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb5c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42098/0 (socket says 172.21.15.144:42098) 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18ee000640 1 -- 172.21.15.144:0/819848100 learned_addr learned my addr 172.21.15.144:0/819848100 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:06.074 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.071+0000 7f18ee000640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb5d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ecc00640 1 -- 172.21.15.144:0/819848100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 msgr2=0x564f8cb5c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ecc00640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb5c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ecc00640 1 -- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 msgr2=0x564f8cb5d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ecc00640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb5d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ecc00640 1 -- 172.21.15.144:0/819848100 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f8ca6a800 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ee000640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb5c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18ecc00640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb5cc10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18e806a230 tx=0x7f18d400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8423c4acfffdb418 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18d2a00640 1 -- 172.21.15.144:0/819848100 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4009e30 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18e0003970 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.072+0000 7f18d2a00640 1 -- 172.21.15.144:0/819848100 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 11716+0+0 (secure 0 0 0) 0x7f18d4012110 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18d2a00640 1 -- 172.21.15.144:0/819848100 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=208) ==== 24+0+0 (secure 0 0 0) 0x7f18d4025050 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18f6dea740 10 client.16191 fetch_fsmap learned FSMap version 208 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=208,monmap=4+}) -- 0x564f8cb6b610 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18d2a00640 1 -- 172.21.15.144:0/819848100 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0003970 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18d2a00640 1 -- 172.21.15.144:0/819848100 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 208) ==== 6543+0+0 (secure 0 0 0) 0x7f18d4003ba0 con 0x564f8c78b390 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18f6dea740 10 client.16191 fetch_fsmap finished waiting for FSMap version 208 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18f6dea740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.073+0000 7f18f6dea740 20 client.16191 populate_metadata read hostname 'smithi144' 2024-07-26T11:13:06.077 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.074+0000 7f18f6dea740 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c76a9f0 0x564f8cb6ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.074+0000 7f18f6dea740 4 client.16191 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:13:06.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.074+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f8cac9ae0 con 0x564f8c76a9f0 2024-07-26T11:13:06.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.074+0000 7f18ee000640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c76a9f0 0x564f8cb6ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.074+0000 7f18ee000640 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c76a9f0 0x564f8cb6ec40 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6382a76a9f71929a server_cookie=441ecbbe732aad0a in_seq=0 out_seq=0 2024-07-26T11:13:06.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.074+0000 7f18d2a00640 10 client.16191 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18d2a00640 1 -- 172.21.15.144:0/819848100 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f8cac9ae0 con 0x564f8c76a9f0 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18d2a00640 10 client.16191 handle_command_reply: tid=0 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18f6dea740 1 client.16191 shutdown 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 msgr2=0x564f8cb5cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18f6dea740 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb5cc10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18e806a230 tx=0x7f18d400f790 comp rx=0 tx=0).stop 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 shutdown_connections 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.075+0000 7f18f6dea740 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c76a9f0 0x564f8cb6ec40 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7ac070 0x564f8cb5d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c78b390 0x564f8cb5cc10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 1 --2- 172.21.15.144:0/819848100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7831b0 0x564f8cb5c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 >> 172.21.15.144:0/819848100 conn(0x564f8c7a0460 msgr2=0x564f8cb54350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:06.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18d2a00640 0 client.16191 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:06.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 shutdown_connections 2024-07-26T11:13:06.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 1 -- 172.21.15.144:0/819848100 wait complete. 2024-07-26T11:13:06.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.076+0000 7f18f6dea740 20 client.16191 trim_cache size 0 max 16384 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 Processor -- start 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 -- start start 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7a0460 0x564f8cb5e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb52900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c78b390 0x564f8cb74420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f18e81b5980 con 0x564f8c7831b0 2024-07-26T11:13:06.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f18e81b4be0 con 0x564f8c78b390 2024-07-26T11:13:06.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.095+0000 7f18f6dea740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18e81b52b0 con 0x564f8c7a0460 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb52900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb52900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54772/0 (socket says 172.21.15.144:54772) 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 -- 172.21.15.144:0/2519022608 learned_addr learned my addr 172.21.15.144:0/2519022608 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ee000640 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7a0460 0x564f8cb5e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ee000640 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c78b390 0x564f8cb74420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 -- 172.21.15.144:0/2519022608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7a0460 msgr2=0x564f8cb5e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7a0460 0x564f8cb5e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 -- 172.21.15.144:0/2519022608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c78b390 msgr2=0x564f8cb74420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c78b390 0x564f8cb74420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 -- 172.21.15.144:0/2519022608 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f8c76ac20 con 0x564f8c7831b0 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.096+0000 7f18ecc00640 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb52900 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f18d4012870 tx=0x7f18d4009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fe5611ad9194c8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.097+0000 7f18ddc00640 1 -- 172.21.15.144:0/2519022608 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4000e60 con 0x564f8c7831b0 2024-07-26T11:13:06.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.097+0000 7f18ddc00640 1 -- 172.21.15.144:0/2519022608 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18d4012250 con 0x564f8c7831b0 2024-07-26T11:13:06.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.097+0000 7f18ddc00640 1 -- 172.21.15.144:0/2519022608 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d40051e0 con 0x564f8c7831b0 2024-07-26T11:13:06.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 -- 172.21.15.144:0/2519022608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 msgr2=0x564f8cb52900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb52900 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f18d4012870 tx=0x7f18d4009fc0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 -- 172.21.15.144:0/2519022608 shutdown_connections 2024-07-26T11:13:06.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c78b390 0x564f8cb74420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb52900 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 --2- 172.21.15.144:0/2519022608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c7a0460 0x564f8cb5e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 -- 172.21.15.144:0/2519022608 >> 172.21.15.144:0/2519022608 conn(0x564f8cb6ef80 msgr2=0x564f8cb521f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:06.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 -- 172.21.15.144:0/2519022608 shutdown_connections 2024-07-26T11:13:06.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.098+0000 7f18f6dea740 1 -- 172.21.15.144:0/2519022608 wait complete. 2024-07-26T11:13:06.101 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:05 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:reconnect 2024-07-26T11:13:06.101 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:05 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:06.101 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:05 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:13:06.101 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 429.534457 secs 2024-07-26T11:13:06.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.099+0000 7f18f6dea740 1 -- start start 2024-07-26T11:13:06.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.099+0000 7f18f6dea740 1 Processor -- start 2024-07-26T11:13:06.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb65390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 0x564f8cb62860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7a0460 0x564f8cb62e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f18e81b1ee0 con 0x564f8c7831b0 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f18e81b32f0 con 0x564f8c7a0460 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18e81b2550 con 0x564f8c78b390 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18f6dea740 10 client.0 _ll_register_callbacks cb 0x7f18d4012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ee000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb65390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 0x564f8cb62860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 0x564f8cb62860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:42110/0 (socket says 172.21.15.144:42110) 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 -- 172.21.15.144:0/3915297011 learned_addr learned my addr 172.21.15.144:0/3915297011 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ee000640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7a0460 0x564f8cb62e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 -- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7a0460 msgr2=0x564f8cb62e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7a0460 0x564f8cb62e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 -- 172.21.15.144:0/3915297011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 msgr2=0x564f8cb65390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb65390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.100+0000 7f18ecc00640 1 -- 172.21.15.144:0/3915297011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f8cb63590 con 0x564f8c78b390 2024-07-26T11:13:06.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.101+0000 7f18ee000640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7a0460 0x564f8cb62e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:13:06.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.101+0000 7f18ecc00640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 0x564f8cb62860 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f18d4000c00 tx=0x7f18d4009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e028bbd5a40ba7d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.101+0000 7f18d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.101+0000 7f18d2a00640 1 -- 172.21.15.144:0/3915297011 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4037060 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.101+0000 7f18d2a00640 1 -- 172.21.15.144:0/3915297011 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 11716+0+0 (secure 0 0 0) 0x7f18d4018040 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.101+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18d4012250 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18d2a00640 1 -- 172.21.15.144:0/3915297011 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=208) ==== 24+0+0 (secure 0 0 0) 0x7f18d403e050 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 10 client.26449 fetch_fsmap learned FSMap version 208 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=208,monmap=4+}) -- 0x564f8cbdfbd0 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18d2a00640 1 -- 172.21.15.144:0/3915297011 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18d4012250 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18d2a00640 1 -- 172.21.15.144:0/3915297011 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 208) ==== 6543+0+0 (secure 0 0 0) 0x7f18d40035f0 con 0x564f8c78b390 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 10 client.26449 fetch_fsmap finished waiting for FSMap version 208 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 20 client.26449 populate_metadata read hostname 'smithi144' 2024-07-26T11:13:06.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c7ac070 0x564f8cbe22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 4 client.26449 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:13:06.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.102+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f8cac5580 con 0x564f8c7ac070 2024-07-26T11:13:06.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.103+0000 7f18ee000640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c7ac070 0x564f8cbe22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.103+0000 7f18ee000640 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c7ac070 0x564f8cbe22d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7fec39e8dc1713d server_cookie=e16d64eba44767f1 in_seq=0 out_seq=0 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.103+0000 7f18d2a00640 10 client.26449 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.103+0000 7f18d2a00640 1 -- 172.21.15.144:0/3915297011 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564f8cac5580 con 0x564f8c7ac070 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.103+0000 7f18d2a00640 10 client.26449 handle_command_reply: tid=0 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.103+0000 7f18f6dea740 1 client.26449 shutdown 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 msgr2=0x564f8cb62860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 0x564f8cb62860 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f18d4000c00 tx=0x7f18d4009280 comp rx=0 tx=0).stop 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 shutdown_connections 2024-07-26T11:13:06.107 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564f8c7ac070 0x564f8cbe22d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564f8c7a0460 0x564f8cb62e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564f8c78b390 0x564f8cb62860 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 --2- 172.21.15.144:0/3915297011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564f8c7831b0 0x564f8cb65390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18d2a00640 0 client.26449 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 >> 172.21.15.144:0/3915297011 conn(0x564f8cb6ef80 msgr2=0x564f8cb5e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 shutdown_connections 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 1 -- 172.21.15.144:0/3915297011 wait complete. 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.104+0000 7f18f6dea740 20 client.26449 trim_cache size 0 max 16384 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:13:06.108 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:13:06.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18cc0b3dc0 msgr2=0x7f18cc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18cc0b3dc0 0x7f18cc0b62d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x564f8cb4fe40 tx=0x7f18d8108ab0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 msgr2=0x7f18e80fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e80fe220 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f18d8005940 tx=0x7f18d80040c0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 shutdown_connections 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f18cc0b3dc0 0x7f18cc0b62d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18e8108530 0x7f18e80feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e8107be0 0x7f18e80fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 --2- 172.21.15.144:0/2151821690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f18e81022a0 0x7f18e80fe220 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 >> 172.21.15.144:0/2151821690 conn(0x7f18e80fdcf0 msgr2=0x7f18e8100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 shutdown_connections 2024-07-26T11:13:06.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:06.105+0000 7f18eea00640 1 -- 172.21.15.144:0/2151821690 wait complete. 2024-07-26T11:13:06.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:05 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:reconnect 2024-07-26T11:13:06.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:05 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:06.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:05 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:13:06.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 429.534457 secs 2024-07-26T11:13:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:05 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:reconnect 2024-07-26T11:13:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:05 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:05 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899986 2024-07-26T11:13:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 429.534457 secs 2024-07-26T11:13:06.917 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:13:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:06 smithi144 ceph-mon[32129]: pgmap v811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:13:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:06 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:rejoin 2024-07-26T11:13:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:06 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:06 smithi144 ceph-mon[32129]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:13:07.309 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:13:07.182+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:13:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-mon[36715]: pgmap v811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:13:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:rejoin 2024-07-26T11:13:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-mon[36715]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: *** Caught signal (Aborted) ** 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: in thread 7f814304e640 thread_name:ms_dispatch 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81484256f0] 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f814847294c] 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 3: raise() 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 4: abort() 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 5: /lib64/libc.so.6(+0x2871b) [0x7f814840f71b] 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 6: /lib64/libc.so.6(+0x37386) [0x7f814841e386] 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557a7b9816b9] 2024-07-26T11:13:07.311 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557a7bb3f80c] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557a7ba7f2ca] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557a7ba4f753] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557a7ba50a1b] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557a7ba34752] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557a7b982a56] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557a7b986637] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557a7b986f5e] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557a7b96a03a] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8148cf2407] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 18: (DispatchQueue::entry()+0x60f) [0x7f8148cf2cf9] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8148dcf5a1] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 20: (Thread::entry_wrapper()+0x43) [0x7f8148b7ccdd] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8148b7ccf9] 2024-07-26T11:13:07.312 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8148470c02] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81484f5c40] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:06.980+0000 7f814304e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: in thread 7f814304e640 thread_name:ms_dispatch 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81484256f0] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f814847294c] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 3: raise() 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 4: abort() 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 5: /lib64/libc.so.6(+0x2871b) [0x7f814840f71b] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 6: /lib64/libc.so.6(+0x37386) [0x7f814841e386] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557a7b9816b9] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557a7bb3f80c] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557a7ba7f2ca] 2024-07-26T11:13:07.313 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557a7ba4f753] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557a7ba50a1b] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557a7ba34752] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557a7b982a56] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557a7b986637] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557a7b986f5e] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557a7b96a03a] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8148cf2407] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 18: (DispatchQueue::entry()+0x60f) [0x7f8148cf2cf9] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8148dcf5a1] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 20: (Thread::entry_wrapper()+0x43) [0x7f8148b7ccdd] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8148b7ccf9] 2024-07-26T11:13:07.314 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8148470c02] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81484f5c40] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 0> 2024-07-26T11:13:06.980+0000 7f814304e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: in thread 7f814304e640 thread_name:ms_dispatch 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81484256f0] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f814847294c] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 3: raise() 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 4: abort() 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 5: /lib64/libc.so.6(+0x2871b) [0x7f814840f71b] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 6: /lib64/libc.so.6(+0x37386) [0x7f814841e386] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557a7b9816b9] 2024-07-26T11:13:07.315 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557a7bb3f80c] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557a7ba7f2ca] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557a7ba4f753] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557a7ba50a1b] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557a7ba34752] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557a7b982a56] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557a7b986637] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557a7b986f5e] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557a7b96a03a] 2024-07-26T11:13:07.316 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8148cf2407] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 18: (DispatchQueue::entry()+0x60f) [0x7f8148cf2cf9] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8148dcf5a1] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 20: (Thread::entry_wrapper()+0x43) [0x7f8148b7ccdd] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8148b7ccf9] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8148470c02] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81484f5c40] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:06 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 0> 2024-07-26T11:13:06.980+0000 7f814304e640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: in thread 7f814304e640 thread_name:ms_dispatch 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81484256f0] 2024-07-26T11:13:07.317 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f814847294c] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 3: raise() 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 4: abort() 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 5: /lib64/libc.so.6(+0x2871b) [0x7f814840f71b] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 6: /lib64/libc.so.6(+0x37386) [0x7f814841e386] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557a7b9816b9] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557a7bb3f80c] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557a7ba7f2ca] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557a7ba4f753] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557a7ba50a1b] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557a7ba34752] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557a7b982a56] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557a7b986637] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557a7b986f5e] 2024-07-26T11:13:07.318 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557a7b96a03a] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8148cf2407] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 18: (DispatchQueue::entry()+0x60f) [0x7f8148cf2cf9] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8148dcf5a1] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 20: (Thread::entry_wrapper()+0x43) [0x7f8148b7ccdd] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8148b7ccf9] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8148470c02] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81484f5c40] 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72702]: 2024-07-26T11:13:07.319 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 podman[72882]: 2024-07-26 11:13:07.037369633 +0000 UTC m=+0.018303872 container died 0234712af09a9b49b5420b657de03dcbc7a8923c047da823b56d549fef454d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:13:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:06 smithi183 ceph-mon[37273]: pgmap v811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:13:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:06 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:rejoin 2024-07-26T11:13:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:06 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:06 smithi183 ceph-mon[37273]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:13:07.980 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 podman[72882]: 2024-07-26 11:13:07.67693755 +0000 UTC m=+0.657871834 container remove 0234712af09a9b49b5420b657de03dcbc7a8923c047da823b56d549fef454d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-07-26T11:13:07.981 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:13:07.981 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:13:07.981 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:07 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 1.964s CPU time. 2024-07-26T11:13:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:active 2024-07-26T11:13:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:07 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:07 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:07 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:active 2024-07-26T11:13:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:07 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:07 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:07 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:07 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/3772501675,v1:172.21.15.160:6833/3772501675] up:active 2024-07-26T11:13:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:07 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:07 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:08 smithi160 ceph-mon[36715]: pgmap v812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:13:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:08 smithi183 ceph-mon[37273]: pgmap v812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:13:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:08 smithi144 ceph-mon[32129]: pgmap v812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:13:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:10 smithi144 ceph-mon[32129]: pgmap v813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 434.534555 secs 2024-07-26T11:13:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:10 smithi160 ceph-mon[36715]: pgmap v813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 434.534555 secs 2024-07-26T11:13:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:10 smithi183 ceph-mon[37273]: pgmap v813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 434.534555 secs 2024-07-26T11:13:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:11.929+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:11.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:11.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:11.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:11.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:12 smithi160 ceph-mon[36715]: pgmap v814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:12 smithi183 ceph-mon[37273]: pgmap v814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:12 smithi144 ceph-mon[32129]: pgmap v814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:14 smithi160 ceph-mon[36715]: pgmap v815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 7 op/s 2024-07-26T11:13:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:14 smithi183 ceph-mon[37273]: pgmap v815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 7 op/s 2024-07-26T11:13:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:14 smithi144 ceph-mon[32129]: pgmap v815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 7 op/s 2024-07-26T11:13:16.267 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 439.534670 secs 2024-07-26T11:13:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 439.534670 secs 2024-07-26T11:13:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 439.534670 secs 2024-07-26T11:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:17 smithi160 ceph-mon[36715]: pgmap v816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:13:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:17 smithi183 ceph-mon[37273]: pgmap v816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:13:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:17 smithi144 ceph-mon[32129]: pgmap v816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:13:18.320 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:13:18.320 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 systemd[1]: Stopped Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:18.320 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 1.964s CPU time. 2024-07-26T11:13:18.320 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 systemd[1]: Starting Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:13:18.320 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 podman[72974]: 2024-07-26 11:13:18.20165669 +0000 UTC m=+0.017718131 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:13:18.703 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 podman[72974]: 2024-07-26 11:13:18.319232941 +0000 UTC m=+0.135294381 container create 3caad0a4cdb4e92693245b917752ba5fb813813aaa0437fae8616fba63bf7992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T11:13:18.965 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 podman[72974]: 2024-07-26 11:13:18.864285327 +0000 UTC m=+0.680346782 container init 3caad0a4cdb4e92693245b917752ba5fb813813aaa0437fae8616fba63bf7992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T11:13:18.965 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 podman[72974]: 2024-07-26 11:13:18.867327209 +0000 UTC m=+0.683388652 container start 3caad0a4cdb4e92693245b917752ba5fb813813aaa0437fae8616fba63bf7992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T11:13:18.965 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 ceph-mds[72992]: -- 172.21.15.160:0/3587496776 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab99d9b860 con 0x55ab9df3d180 2024-07-26T11:13:18.966 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: starting mds.k at 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 bash[72974]: 3caad0a4cdb4e92693245b917752ba5fb813813aaa0437fae8616fba63bf7992 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:18 smithi160 systemd[1]: Started Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: pgmap v817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: daemon mds.k restarted 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:13:19.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: osdmap e118: 12 total, 12 up, 12 in 2024-07-26T11:13:19.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:boot 2024-07-26T11:13:19.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:13:19.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:19.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:19 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: pgmap v817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: daemon mds.k restarted 2024-07-26T11:13:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:13:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: osdmap e118: 12 total, 12 up, 12 in 2024-07-26T11:13:19.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:boot 2024-07-26T11:13:19.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:13:19.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:19.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:19 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: pgmap v817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: daemon mds.k restarted 2024-07-26T11:13:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:19.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:13:19.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: osdmap e118: 12 total, 12 up, 12 in 2024-07-26T11:13:19.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:boot 2024-07-26T11:13:19.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:13:19.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:19.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:19 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:21.298 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:21 smithi144 ceph-mon[32129]: pgmap v819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:21.298 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:21 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:resolve 2024-07-26T11:13:21.298 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:21 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:21.298 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 444.534780 secs 2024-07-26T11:13:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:21 smithi160 ceph-mon[36715]: pgmap v819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:21 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:resolve 2024-07-26T11:13:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:21 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 444.534780 secs 2024-07-26T11:13:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:21 smithi183 ceph-mon[37273]: pgmap v819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:21 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:resolve 2024-07-26T11:13:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:21 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 444.534780 secs 2024-07-26T11:13:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:22 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:reconnect 2024-07-26T11:13:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:22 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:22 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:13:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:22 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:reconnect 2024-07-26T11:13:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:22 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:22 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:22 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:reconnect 2024-07-26T11:13:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:22 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:22 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:13:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-mon[37273]: pgmap v820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6 op/s 2024-07-26T11:13:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:rejoin 2024-07-26T11:13:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-mon[37273]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:13:23.386 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:13:23.386 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: *** Caught signal (Aborted) ** 2024-07-26T11:13:23.386 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: in thread 7f4b7f08f640 thread_name:ms_dispatch 2024-07-26T11:13:23.386 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:23.386 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4b844666f0] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4b844b394c] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 3: raise() 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 4: abort() 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4b8445071b] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 6: /lib64/libc.so.6(+0x37386) [0x7f4b8445f386] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564673d5d6b9] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564673f1b80c] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564673e5b2ca] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564673e2b753] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564673e2ca1b] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564673e10752] 2024-07-26T11:13:23.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564673d5ea56] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564673d62637] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564673d62f5e] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564673d4603a] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4b84d33407] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 18: (DispatchQueue::entry()+0x60f) [0x7f4b84d33cf9] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4b84e105a1] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 20: (Thread::entry_wrapper()+0x43) [0x7f4b84bbdcdd] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4b84bbdcf9] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4b844b1c02] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4b84536c40] 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.090+0000 7f4b7f08f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: in thread 7f4b7f08f640 thread_name:ms_dispatch 2024-07-26T11:13:23.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4b844666f0] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4b844b394c] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 3: raise() 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 4: abort() 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4b8445071b] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 6: /lib64/libc.so.6(+0x37386) [0x7f4b8445f386] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564673d5d6b9] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564673f1b80c] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564673e5b2ca] 2024-07-26T11:13:23.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564673e2b753] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564673e2ca1b] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564673e10752] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564673d5ea56] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564673d62637] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564673d62f5e] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564673d4603a] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4b84d33407] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 18: (DispatchQueue::entry()+0x60f) [0x7f4b84d33cf9] 2024-07-26T11:13:23.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4b84e105a1] 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 20: (Thread::entry_wrapper()+0x43) [0x7f4b84bbdcdd] 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4b84bbdcf9] 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4b844b1c02] 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4b84536c40] 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 0> 2024-07-26T11:13:23.090+0000 7f4b7f08f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: in thread 7f4b7f08f640 thread_name:ms_dispatch 2024-07-26T11:13:23.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4b844666f0] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4b844b394c] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 3: raise() 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 4: abort() 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4b8445071b] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 6: /lib64/libc.so.6(+0x37386) [0x7f4b8445f386] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564673d5d6b9] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564673f1b80c] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564673e5b2ca] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564673e2b753] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564673e2ca1b] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564673e10752] 2024-07-26T11:13:23.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564673d5ea56] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564673d62637] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564673d62f5e] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564673d4603a] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4b84d33407] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 18: (DispatchQueue::entry()+0x60f) [0x7f4b84d33cf9] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4b84e105a1] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 20: (Thread::entry_wrapper()+0x43) [0x7f4b84bbdcdd] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4b84bbdcf9] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4b844b1c02] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4b84536c40] 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 0> 2024-07-26T11:13:23.090+0000 7f4b7f08f640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:23.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: in thread 7f4b7f08f640 thread_name:ms_dispatch 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4b844666f0] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4b844b394c] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 3: raise() 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 4: abort() 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4b8445071b] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 6: /lib64/libc.so.6(+0x37386) [0x7f4b8445f386] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564673d5d6b9] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564673f1b80c] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564673e5b2ca] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564673e2b753] 2024-07-26T11:13:23.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564673e2ca1b] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564673e10752] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564673d5ea56] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564673d62637] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564673d62f5e] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564673d4603a] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4b84d33407] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 18: (DispatchQueue::entry()+0x60f) [0x7f4b84d33cf9] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4b84e105a1] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 20: (Thread::entry_wrapper()+0x43) [0x7f4b84bbdcdd] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4b84bbdcf9] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4b844b1c02] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4b84536c40] 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:23.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75516]: 2024-07-26T11:13:23.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 podman[75568]: 2024-07-26 11:13:23.155486697 +0000 UTC m=+0.018821995 container died 87d3c075ce9921e175eeb7b4870bdc9cd7a9494bbaba6a8bd41a8e3e9bc0ed6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T11:13:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:23 smithi144 ceph-mon[32129]: pgmap v820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6 op/s 2024-07-26T11:13:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:23 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:rejoin 2024-07-26T11:13:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:23 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:23 smithi144 ceph-mon[32129]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:23 smithi160 ceph-mon[36715]: pgmap v820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6 op/s 2024-07-26T11:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:23 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:rejoin 2024-07-26T11:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:23 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:23 smithi160 ceph-mon[36715]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:13:24.179 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 podman[75568]: 2024-07-26 11:13:23.926057131 +0000 UTC m=+0.789392492 container remove 87d3c075ce9921e175eeb7b4870bdc9cd7a9494bbaba6a8bd41a8e3e9bc0ed6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-26T11:13:24.179 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:23 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:13:24.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:24 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:24.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:24 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:active 2024-07-26T11:13:24.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:24 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:24.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:24 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:24 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:24 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:active 2024-07-26T11:13:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:24 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:24 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:24 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:24 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1430327724,v1:172.21.15.183:6833/1430327724] up:active 2024-07-26T11:13:24.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:24 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:24.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:24 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:24.635 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:24 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:13:24.635 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:24 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.928s CPU time. 2024-07-26T11:13:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:25 smithi183 ceph-mon[37273]: pgmap v821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:13:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:25 smithi144 ceph-mon[32129]: pgmap v821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:13:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:25 smithi160 ceph-mon[36715]: pgmap v821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:13:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 449.534904 secs 2024-07-26T11:13:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 449.534904 secs 2024-07-26T11:13:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 449.534904 secs 2024-07-26T11:13:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:26.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:26.942+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:26.942+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:26.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:26.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:27.208 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:27 smithi144 ceph-mon[32129]: pgmap v822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:27 smithi183 ceph-mon[37273]: pgmap v822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:27 smithi160 ceph-mon[36715]: pgmap v822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:29 smithi144 ceph-mon[32129]: pgmap v823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:29 smithi183 ceph-mon[37273]: pgmap v823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:29 smithi160 ceph-mon[36715]: pgmap v823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:31 smithi183 ceph-mon[37273]: pgmap v824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:13:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 454.535042 secs 2024-07-26T11:13:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:31 smithi144 ceph-mon[32129]: pgmap v824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:13:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 454.535042 secs 2024-07-26T11:13:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:31 smithi160 ceph-mon[36715]: pgmap v824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:13:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 454.535042 secs 2024-07-26T11:13:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:33 smithi183 ceph-mon[37273]: pgmap v825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:13:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:33 smithi144 ceph-mon[32129]: pgmap v825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:13:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:33 smithi160 ceph-mon[36715]: pgmap v825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:13:34.757 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:13:34.757 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 systemd[1]: Stopped Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:34.757 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.928s CPU time. 2024-07-26T11:13:34.757 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 systemd[1]: Starting Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:13:34.757 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 podman[75658]: 2024-07-26 11:13:34.520264724 +0000 UTC m=+0.017823996 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:13:34.757 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 podman[75658]: 2024-07-26 11:13:34.634375604 +0000 UTC m=+0.131934935 container create 171ea4b907da7170590c69528a5ed576e1a1f71b4deb42725873ff1240127c7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:13:35.128 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 podman[75658]: 2024-07-26 11:13:34.845480817 +0000 UTC m=+0.343040078 container init 171ea4b907da7170590c69528a5ed576e1a1f71b4deb42725873ff1240127c7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:13:35.128 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 podman[75658]: 2024-07-26 11:13:34.848072367 +0000 UTC m=+0.345631630 container start 171ea4b907da7170590c69528a5ed576e1a1f71b4deb42725873ff1240127c7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724) 2024-07-26T11:13:35.128 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 ceph-mds[75677]: -- 172.21.15.183:0/234562843 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f06f7a3860 con 0x55f073945a80 2024-07-26T11:13:35.128 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: starting mds.l at 2024-07-26T11:13:35.128 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 bash[75658]: 171ea4b907da7170590c69528a5ed576e1a1f71b4deb42725873ff1240127c7f 2024-07-26T11:13:35.128 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:34 smithi183 systemd[1]: Started Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: pgmap v826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: daemon mds.l restarted 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: osdmap e119: 12 total, 12 up, 12 in 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:boot 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:13:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:35.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:35 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: pgmap v826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: daemon mds.l restarted 2024-07-26T11:13:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:13:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: osdmap e119: 12 total, 12 up, 12 in 2024-07-26T11:13:35.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:boot 2024-07-26T11:13:35.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:13:35.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:35.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:35 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: pgmap v826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: daemon mds.l restarted 2024-07-26T11:13:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:13:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: osdmap e119: 12 total, 12 up, 12 in 2024-07-26T11:13:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:boot 2024-07-26T11:13:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-07-26T11:13:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T11:13:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:35 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:replay,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 459.535163 secs 2024-07-26T11:13:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 459.535163 secs 2024-07-26T11:13:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 459.535163 secs 2024-07-26T11:13:36.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:13:37.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:37 smithi144 ceph-mon[32129]: pgmap v828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:13:37.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:37 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:resolve 2024-07-26T11:13:37.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:37 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:37.379 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:37 smithi160 ceph-mon[36715]: pgmap v828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:13:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:37 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:resolve 2024-07-26T11:13:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:37 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:37 smithi183 ceph-mon[37273]: pgmap v828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:13:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:37 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:resolve 2024-07-26T11:13:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:37 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:resolve,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:38.442 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:38 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:reconnect 2024-07-26T11:13:38.442 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:38 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:38.442 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:38 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:13:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:38 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:reconnect 2024-07-26T11:13:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:38 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:38 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:13:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:38 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:reconnect 2024-07-26T11:13:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:38 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:reconnect,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:38 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999985 2024-07-26T11:13:39.430 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:39 smithi144 ceph-mon[32129]: pgmap v829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:13:39.430 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:39 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:rejoin 2024-07-26T11:13:39.430 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:39 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:39.430 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:39 smithi144 ceph-mon[32129]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:13:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-mon[36715]: pgmap v829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:13:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:rejoin 2024-07-26T11:13:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:39.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-mon[36715]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:13:39.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:13:39.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: *** Caught signal (Aborted) ** 2024-07-26T11:13:39.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: in thread 7f1d60818640 thread_name:ms_dispatch 2024-07-26T11:13:39.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d65bef6f0] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d65c3c94c] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 3: raise() 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 4: abort() 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d65bd971b] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d65be8386] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab9700b6b9] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab971c980c] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab971092ca] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab970d9753] 2024-07-26T11:13:39.561 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab970daa1b] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab970be752] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab9700ca56] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab97010637] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab97010f5e] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab96ff403a] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d664bc407] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d664bccf9] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d665995a1] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d66346cdd] 2024-07-26T11:13:39.562 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d66346cf9] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d65c3ac02] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d65cbfc40] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.217+0000 7f1d60818640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: in thread 7f1d60818640 thread_name:ms_dispatch 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d65bef6f0] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d65c3c94c] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 3: raise() 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 4: abort() 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d65bd971b] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d65be8386] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab9700b6b9] 2024-07-26T11:13:39.563 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab971c980c] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab971092ca] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab970d9753] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab970daa1b] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab970be752] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab9700ca56] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab97010637] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab97010f5e] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab96ff403a] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d664bc407] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d664bccf9] 2024-07-26T11:13:39.564 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d665995a1] 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d66346cdd] 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d66346cf9] 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d65c3ac02] 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d65cbfc40] 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 0> 2024-07-26T11:13:39.217+0000 7f1d60818640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:39.565 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: in thread 7f1d60818640 thread_name:ms_dispatch 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d65bef6f0] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d65c3c94c] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 3: raise() 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 4: abort() 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d65bd971b] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d65be8386] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab9700b6b9] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab971c980c] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab971092ca] 2024-07-26T11:13:39.566 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab970d9753] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab970daa1b] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab970be752] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab9700ca56] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab97010637] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab97010f5e] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab96ff403a] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d664bc407] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d664bccf9] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d665995a1] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d66346cdd] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d66346cf9] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d65c3ac02] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d65cbfc40] 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:39.567 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 0> 2024-07-26T11:13:39.217+0000 7f1d60818640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: in thread 7f1d60818640 thread_name:ms_dispatch 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1d65bef6f0] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1d65c3c94c] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 3: raise() 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 4: abort() 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1d65bd971b] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 6: /lib64/libc.so.6(+0x37386) [0x7f1d65be8386] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab9700b6b9] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab971c980c] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab971092ca] 2024-07-26T11:13:39.568 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab970d9753] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab970daa1b] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab970be752] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab9700ca56] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab97010637] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab97010f5e] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab96ff403a] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1d664bc407] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 18: (DispatchQueue::entry()+0x60f) [0x7f1d664bccf9] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1d665995a1] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 20: (Thread::entry_wrapper()+0x43) [0x7f1d66346cdd] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1d66346cf9] 2024-07-26T11:13:39.569 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1d65c3ac02] 2024-07-26T11:13:39.570 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1d65cbfc40] 2024-07-26T11:13:39.570 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:13:39.570 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k[72988]: 2024-07-26T11:13:39.570 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 podman[73026]: 2024-07-26 11:13:39.27517609 +0000 UTC m=+0.019071414 container died 3caad0a4cdb4e92693245b917752ba5fb813813aaa0437fae8616fba63bf7992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-07-26T11:13:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:39 smithi183 ceph-mon[37273]: pgmap v829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:13:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:39 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:rejoin 2024-07-26T11:13:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:39 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=k=up:rejoin,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:39 smithi183 ceph-mon[37273]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-07-26T11:13:39.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 -- 172.21.15.144:0/1578078682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc40751e0 msgr2=0x7f9bc40755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 --2- 172.21.15.144:0/1578078682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc40751e0 0x7f9bc40755f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009ca0 tx=0x7f9bb402a8b0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 -- 172.21.15.144:0/1578078682 shutdown_connections 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 --2- 172.21.15.144:0/1578078682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc410f940 0x7f9bc4111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 --2- 172.21.15.144:0/1578078682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc4075b30 0x7f9bc410f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 --2- 172.21.15.144:0/1578078682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc40751e0 0x7f9bc40755f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 -- 172.21.15.144:0/1578078682 >> 172.21.15.144:0/1578078682 conn(0x7f9bc40fee30 msgr2=0x7f9bc4101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 -- 172.21.15.144:0/1578078682 shutdown_connections 2024-07-26T11:13:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.795+0000 7f9bcac00640 1 -- 172.21.15.144:0/1578078682 wait complete. 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.796+0000 7f9bcac00640 1 Processor -- start 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.796+0000 7f9bcac00640 1 -- start start 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bcac00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 0x7f9bc419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bcac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc4075b30 0x7f9bc419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bcac00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc410f940 0x7f9bc41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bcac00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4115540 con 0x7f9bc410f940 2024-07-26T11:13:39.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bcac00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41153c0 con 0x7f9bc40751e0 2024-07-26T11:13:39.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bcac00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41156c0 con 0x7f9bc4075b30 2024-07-26T11:13:39.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bc9800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 0x7f9bc419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bca200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc410f940 0x7f9bc41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bc9800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 0x7f9bc419be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56092/0 (socket says 172.21.15.144:56092) 2024-07-26T11:13:39.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bc8e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc4075b30 0x7f9bc419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.797+0000 7f9bc9800640 1 -- 172.21.15.144:0/212185203 learned_addr learned my addr 172.21.15.144:0/212185203 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc9800640 1 -- 172.21.15.144:0/212185203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc4075b30 msgr2=0x7f9bc419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc9800640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc4075b30 0x7f9bc419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc9800640 1 -- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc410f940 msgr2=0x7f9bc41a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc9800640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc410f940 0x7f9bc41a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc9800640 1 -- 172.21.15.144:0/212185203 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc41a1de0 con 0x7f9bc40751e0 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bca200640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc410f940 0x7f9bc41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:39.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc4075b30 0x7f9bc419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:39.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc9800640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 0x7f9bc419be70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4005e30 tx=0x7f9bb4003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:39.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc2000640 1 -- 172.21.15.144:0/212185203 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4004040 con 0x7f9bc40751e0 2024-07-26T11:13:39.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc41a2070 con 0x7f9bc40751e0 2024-07-26T11:13:39.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc2000640 1 -- 172.21.15.144:0/212185203 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bb4040070 con 0x7f9bc40751e0 2024-07-26T11:13:39.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.798+0000 7f9bc2000640 1 -- 172.21.15.144:0/212185203 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4031b30 con 0x7f9bc40751e0 2024-07-26T11:13:39.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.799+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc41a8830 con 0x7f9bc40751e0 2024-07-26T11:13:39.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.802+0000 7f9bd2e3f740 1 Processor -- start 2024-07-26T11:13:39.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.802+0000 7f9bd2e3f740 1 -- start start 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfd91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfd9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfd9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5573fdf4fbc0 con 0x5573fdc13390 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5573fdf3f0c0 con 0x5573fdc34070 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5573fdf43100 con 0x5573fdc0b1b0 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bc2000640 1 -- 172.21.15.144:0/212185203 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f9bb4031cf0 con 0x7f9bc40751e0 2024-07-26T11:13:39.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfd91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfd91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56100/0 (socket says 172.21.15.144:56100) 2024-07-26T11:13:39.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 -- 172.21.15.144:0/2010448970 learned_addr learned my addr 172.21.15.144:0/2010448970 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:39.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfd9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bc9800640 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfd9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.807 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 -- 172.21.15.144:0/2010448970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 msgr2=0x5573fdfd9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfd9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 -- 172.21.15.144:0/2010448970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 msgr2=0x5573fdfd9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfd9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.803+0000 7f9bca200640 1 -- 172.21.15.144:0/2010448970 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573fdc9f650 con 0x5573fdc34070 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.804+0000 7f9bca200640 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfd91f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc011540 tx=0x7f9bbc011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b510234398d997a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.804+0000 7f9bbaa00640 1 -- 172.21.15.144:0/2010448970 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00f5c0 con 0x5573fdc34070 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.804+0000 7f9bbaa00640 1 -- 172.21.15.144:0/2010448970 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc00fc10 con 0x5573fdc34070 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.804+0000 7f9bbaa00640 1 -- 172.21.15.144:0/2010448970 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc019c50 con 0x5573fdc34070 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bc2000640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9ba80b3d40 0x7f9ba80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2010448970 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 msgr2=0x5573fdfd91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfd91f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc011540 tx=0x7f9bbc011a00 comp rx=0 tx=0).stop 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bc2000640 1 -- 172.21.15.144:0/212185203 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 11888+0+0 (secure 0 0 0) 0x7f9bb4105dd0 con 0x7f9bc40751e0 2024-07-26T11:13:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2010448970 shutdown_connections 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfd9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfd9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2010448970 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfd91f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2010448970 >> 172.21.15.144:0/2010448970 conn(0x5573fdc28460 msgr2=0x5573fdfd70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.805+0000 7f9bc9800640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9ba80b3d40 0x7f9ba80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.806+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2010448970 shutdown_connections 2024-07-26T11:13:39.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.806+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2010448970 wait complete. 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.806+0000 7f9bc9800640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9ba80b3d40 0x7f9ba80b6250 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5573fdfd7e70 tx=0x7f9bb4109880 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.806+0000 7f9bd2e3f740 1 -- start start 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 Processor -- start 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 0x5573fdfe4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc34070 0x5573fdfe5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5573fdda2700 con 0x5573fdc0b1b0 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5573fdedb690 con 0x5573fdc13390 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5573fdfe79d0 con 0x5573fdc34070 2024-07-26T11:13:39.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bd2e3f740 10 client.0 _ll_register_callbacks cb 0x5573fdeeb140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.807+0000 7f9bc8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 0x5573fdfe4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bca200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 0x5573fdfe4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56110/0 (socket says 172.21.15.144:56110) 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 -- 172.21.15.144:0/477411489 learned_addr learned my addr 172.21.15.144:0/477411489 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 -- 172.21.15.144:0/477411489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc34070 msgr2=0x5573fdfe5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc34070 0x5573fdfe5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 -- 172.21.15.144:0/477411489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc0b1b0 msgr2=0x5573fdfe4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 -- 172.21.15.144:0/477411489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573fdef2830 con 0x5573fdc13390 2024-07-26T11:13:39.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 0x5573fdfe4c50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00043c0 tx=0x7f9bb00039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90046bb06464d1ca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9baf400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.808+0000 7f9baf400640 1 -- 172.21.15.144:0/477411489 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0005020 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bbc00fc10 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9baf400640 1 -- 172.21.15.144:0/477411489 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 11888+0+0 (secure 0 0 0) 0x7f9bb00051c0 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9baf400640 1 -- 172.21.15.144:0/477411489 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=224) ==== 24+0+0 (secure 0 0 0) 0x7f9bb0024050 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 10 client.26441 fetch_fsmap learned FSMap version 224 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=224,monmap=4+}) -- 0x5573fdff3630 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9baf400640 1 -- 172.21.15.144:0/477411489 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc00fc10 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9baf400640 1 -- 172.21.15.144:0/477411489 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 224) ==== 6539+0+0 (secure 0 0 0) 0x7f9bb0011710 con 0x5573fdc13390 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 10 client.26441 fetch_fsmap finished waiting for FSMap version 224 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 10 client.26441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 20 client.26441 populate_metadata read hostname 'smithi144' 2024-07-26T11:13:39.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.809+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbf29f0 0x5573fdff6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.810+0000 7f9bd2e3f740 4 client.26441 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:13:39.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.810+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573fdf51b10 con 0x5573fdbf29f0 2024-07-26T11:13:39.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.810+0000 7f9bca200640 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbf29f0 0x5573fdff6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.810+0000 7f9bca200640 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbf29f0 0x5573fdff6d40 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90fb2cbf70834e1d server_cookie=f9646a6e79812164 in_seq=0 out_seq=0 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.810+0000 7f9baf400640 10 client.26441 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.811+0000 7f9baf400640 1 -- 172.21.15.144:0/477411489 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5573fdf51b10 con 0x5573fdbf29f0 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.811+0000 7f9baf400640 10 client.26441 handle_command_reply: tid=0 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.811+0000 7f9bd2e3f740 1 client.26441 shutdown 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.811+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 msgr2=0x5573fdfe4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.811+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 0x5573fdfe4c50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00043c0 tx=0x7f9bb00039c0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 shutdown_connections 2024-07-26T11:13:39.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc34070 0x5573fdfe5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbf29f0 0x5573fdff6d40 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc13390 0x5573fdfe4c50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/477411489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 >> 172.21.15.144:0/477411489 conn(0x5573fdc28460 msgr2=0x5573fdfdc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9baf400640 0 client.26441 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 shutdown_connections 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/477411489 wait complete. 2024-07-26T11:13:39.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.812+0000 7f9bd2e3f740 20 client.26441 trim_cache size 0 max 16384 2024-07-26T11:13:39.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 Processor -- start 2024-07-26T11:13:39.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 -- start start 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 0x5573fdfd5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 0x5573fdfda930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41b5920 con 0x5573fdc34070 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41b44b0 con 0x5573fdc0b1b0 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.831+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41b4b80 con 0x5573fdc13390 2024-07-26T11:13:39.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bc8e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 0x5573fdfd5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 0x5573fdfda930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bc8e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 0x5573fdfd5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:36284/0 (socket says 172.21.15.144:36284) 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 0x5573fdfda930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:32776/0 (socket says 172.21.15.144:32776) 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bc8e00640 1 -- 172.21.15.144:0/2549984264 learned_addr learned my addr 172.21.15.144:0/2549984264 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 -- 172.21.15.144:0/2549984264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 msgr2=0x5573fdfd5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 0x5573fdfd5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 -- 172.21.15.144:0/2549984264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc0b1b0 msgr2=0x5573fdfe65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 -- 172.21.15.144:0/2549984264 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573fdfdb4e0 con 0x5573fdc34070 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 0x5573fdfd5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:39.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.832+0000 7f9bca200640 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 0x5573fdfda930 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x5573fdfe6f60 tx=0x7f9bbc011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b089b8768136b807 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:39.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.833+0000 7f9bba000640 1 -- 172.21.15.144:0/2549984264 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0050c0 con 0x5573fdc34070 2024-07-26T11:13:39.836 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.833+0000 7f9bba000640 1 -- 172.21.15.144:0/2549984264 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc00f5c0 con 0x5573fdc34070 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.833+0000 7f9bba000640 1 -- 172.21.15.144:0/2549984264 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0196c0 con 0x5573fdc34070 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.833+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2549984264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 msgr2=0x5573fdfda930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.833+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 0x5573fdfda930 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x5573fdfe6f60 tx=0x7f9bbc011a00 comp rx=0 tx=0).stop 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2549984264 shutdown_connections 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc34070 0x5573fdfda930 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc13390 0x5573fdfd5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2549984264 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc0b1b0 0x5573fdfe65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.837 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2549984264 >> 172.21.15.144:0/2549984264 conn(0x5573fdc28460 msgr2=0x5573fdfda220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:39.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2549984264 shutdown_connections 2024-07-26T11:13:39.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2549984264 wait complete. 2024-07-26T11:13:39.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.834+0000 7f9bd2e3f740 1 -- start start 2024-07-26T11:13:39.838 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 Processor -- start 2024-07-26T11:13:39.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfed550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfeaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.840 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfeb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41b2be0 con 0x5573fdc13390 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41b5250 con 0x5573fdc34070 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41b1e80 con 0x5573fdc0b1b0 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bd2e3f740 10 client.0 _ll_register_callbacks cb 0x7f9bb0004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bca200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bca200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfed550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:36292/0 (socket says 172.21.15.144:36292) 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.835+0000 7f9bc8e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfeaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 -- 172.21.15.144:0/2319982691 learned_addr learned my addr 172.21.15.144:0/2319982691 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfeb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 -- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 msgr2=0x5573fdfeb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfeb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.841 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 -- 172.21.15.144:0/2319982691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 msgr2=0x5573fdfeaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfeaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 -- 172.21.15.144:0/2319982691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573fe066cf0 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bca200640 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfed550 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009ea0 tx=0x7f9bbc009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b81d34202a207cf1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9baf400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9baf400640 1 -- 172.21.15.144:0/2319982691 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc005750 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9baf400640 1 -- 172.21.15.144:0/2319982691 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 11888+0+0 (secure 0 0 0) 0x7f9bbc00fa40 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.836+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bbc00f5c0 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9baf400640 1 -- 172.21.15.144:0/2319982691 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=224) ==== 24+0+0 (secure 0 0 0) 0x7f9bbc03b070 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 10 client.26473 fetch_fsmap learned FSMap version 224 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=224,monmap=4+}) -- 0x5573fe067e80 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9baf400640 1 -- 172.21.15.144:0/2319982691 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc00f5c0 con 0x5573fdc0b1b0 2024-07-26T11:13:39.842 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9baf400640 1 -- 172.21.15.144:0/2319982691 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 224) ==== 6539+0+0 (secure 0 0 0) 0x7f9bbc030070 con 0x5573fdc0b1b0 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 10 client.26473 fetch_fsmap finished waiting for FSMap version 224 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 10 client.26473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 20 client.26473 populate_metadata read hostname 'smithi144' 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbc7da0 0x5573fe06a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 4 client.26473 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.837+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5573fdf4d5b0 con 0x5573fdbc7da0 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.838+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbc7da0 0x5573fe06a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.838+0000 7f9bc8e00640 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbc7da0 0x5573fe06a510 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f45f08bb45042bc server_cookie=b3015a7cefb4e3f0 in_seq=0 out_seq=0 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.838+0000 7f9baf400640 10 client.26473 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9baf400640 1 -- 172.21.15.144:0/2319982691 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5573fdf4d5b0 con 0x5573fdbc7da0 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9baf400640 10 client.26473 handle_command_reply: tid=0 2024-07-26T11:13:39.843 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 client.26473 shutdown 2024-07-26T11:13:39.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 msgr2=0x5573fdfed550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfed550 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009ea0 tx=0x7f9bbc009b30 comp rx=0 tx=0).stop 2024-07-26T11:13:39.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 shutdown_connections 2024-07-26T11:13:39.844 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5573fdbc7da0 0x5573fe06a510 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5573fdc34070 0x5573fdfeb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5573fdc13390 0x5573fdfeaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 --2- 172.21.15.144:0/2319982691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5573fdc0b1b0 0x5573fdfed550 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9baf400640 0 client.26473 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 >> 172.21.15.144:0/2319982691 conn(0x5573fdc28460 msgr2=0x5573fdfe6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 shutdown_connections 2024-07-26T11:13:39.845 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 1 -- 172.21.15.144:0/2319982691 wait complete. 2024-07-26T11:13:39.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.839+0000 7f9bd2e3f740 20 client.26473 trim_cache size 0 max 16384 2024-07-26T11:13:39.846 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:13:39.846 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:13:39.846 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:13:39.846 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:13:39.846 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9ba80b3d40 msgr2=0x7f9ba80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9ba80b3d40 0x7f9ba80b6250 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5573fdfd7e70 tx=0x7f9bb4109880 comp rx=0 tx=0).stop 2024-07-26T11:13:39.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 msgr2=0x7f9bc419be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 0x7f9bc419be70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4005e30 tx=0x7f9bb4003970 comp rx=0 tx=0).stop 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 shutdown_connections 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f9ba80b3d40 0x7f9ba80b6250 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bc410f940 0x7f9bc41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bc4075b30 0x7f9bc419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 --2- 172.21.15.144:0/212185203 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9bc40751e0 0x7f9bc419be70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 >> 172.21.15.144:0/212185203 conn(0x7f9bc40fee30 msgr2=0x7f9bc4101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.840+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 shutdown_connections 2024-07-26T11:13:39.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:13:39.841+0000 7f9bcac00640 1 -- 172.21.15.144:0/212185203 wait complete. 2024-07-26T11:13:40.203 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 podman[73026]: 2024-07-26 11:13:39.951533439 +0000 UTC m=+0.695428794 container remove 3caad0a4cdb4e92693245b917752ba5fb813813aaa0437fae8616fba63bf7992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:13:40.204 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:39 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:13:40.204 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:40 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:40.204 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:40 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:active 2024-07-26T11:13:40.204 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:40 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:40.204 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:40 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:40 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:40 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:active 2024-07-26T11:13:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:40 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:40 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:40.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:40 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:13:40.559 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:40 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 1.967s CPU time. 2024-07-26T11:13:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:40 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:13:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:40 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.160:6832/2177302951,v1:172.21.15.160:6833/2177302951] up:active 2024-07-26T11:13:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:40 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 2 up:standby-replay 4 up:standby 2024-07-26T11:13:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:40 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=k=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby 2024-07-26T11:13:40.799 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:13:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:41 smithi144 ceph-mon[32129]: pgmap v830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 464.535280 secs 2024-07-26T11:13:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:41 smithi160 ceph-mon[36715]: pgmap v830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 464.535280 secs 2024-07-26T11:13:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:41 smithi183 ceph-mon[37273]: pgmap v830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:13:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 464.535280 secs 2024-07-26T11:13:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:41.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:41.959+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:41.959+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:41.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:41.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:43 smithi144 ceph-mon[32129]: pgmap v831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:43 smithi160 ceph-mon[36715]: pgmap v831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:43 smithi183 ceph-mon[37273]: pgmap v831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:13:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:45 smithi144 ceph-mon[32129]: pgmap v832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:45 smithi160 ceph-mon[36715]: pgmap v832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:45 smithi183 ceph-mon[37273]: pgmap v832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:13:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 469.535336 secs 2024-07-26T11:13:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 469.535336 secs 2024-07-26T11:13:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 469.535336 secs 2024-07-26T11:13:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:47 smithi160 ceph-mon[36715]: pgmap v833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:13:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:47 smithi183 ceph-mon[37273]: pgmap v833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:13:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:47 smithi144 ceph-mon[32129]: pgmap v833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:13:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:49 smithi160 ceph-mon[36715]: pgmap v834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:49 smithi183 ceph-mon[37273]: pgmap v834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:49 smithi144 ceph-mon[32129]: pgmap v834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:13:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:50.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:50 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:13:50.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:50 smithi160 systemd[1]: Stopped Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:50.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:50 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Consumed 1.967s CPU time. 2024-07-26T11:13:50.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:50 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Start request repeated too quickly. 2024-07-26T11:13:50.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:50 smithi160 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T11:13:50.560 INFO:journalctl@ceph.mds.k.smithi160.stdout:Jul 26 11:13:50 smithi160 systemd[1]: Failed to start Ceph mds.k for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:13:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:51 smithi160 ceph-mon[36715]: pgmap v835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:13:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 474.535396 secs 2024-07-26T11:13:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:13:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:51 smithi183 ceph-mon[37273]: pgmap v835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:13:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 474.535396 secs 2024-07-26T11:13:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:13:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:51 smithi144 ceph-mon[32129]: pgmap v835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:13:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 474.535396 secs 2024-07-26T11:13:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:13:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:52 smithi160 ceph-mon[36715]: pgmap v836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 92 B/s wr, 7 op/s 2024-07-26T11:13:52.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:52 smithi160 ceph-mon[36715]: pgmap v837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7 op/s 2024-07-26T11:13:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:52 smithi183 ceph-mon[37273]: pgmap v836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 92 B/s wr, 7 op/s 2024-07-26T11:13:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:52 smithi183 ceph-mon[37273]: pgmap v837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7 op/s 2024-07-26T11:13:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:52 smithi144 ceph-mon[32129]: pgmap v836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 92 B/s wr, 7 op/s 2024-07-26T11:13:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:52 smithi144 ceph-mon[32129]: pgmap v837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7 op/s 2024-07-26T11:13:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:54 smithi160 ceph-mon[36715]: pgmap v838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7 op/s 2024-07-26T11:13:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:54 smithi183 ceph-mon[37273]: pgmap v838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7 op/s 2024-07-26T11:13:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:54 smithi144 ceph-mon[32129]: pgmap v838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7 op/s 2024-07-26T11:13:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: pgmap v839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:13:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 479.535513 secs 2024-07-26T11:13:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: Replacing daemon mds.k as rank 1 with standby daemon mds.l 2024-07-26T11:13:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-07-26T11:13:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: osdmap e120: 12 total, 12 up, 12 in 2024-07-26T11:13:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:56 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: pgmap v839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 479.535513 secs 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: Replacing daemon mds.k as rank 1 with standby daemon mds.l 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: osdmap e120: 12 total, 12 up, 12 in 2024-07-26T11:13:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:56 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: pgmap v839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:13:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 479.535513 secs 2024-07-26T11:13:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: Replacing daemon mds.k as rank 1 with standby daemon mds.l 2024-07-26T11:13:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-07-26T11:13:56.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:13:56.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: osdmap e120: 12 total, 12 up, 12 in 2024-07-26T11:13:56.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:56 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:56.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:56.970+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:56.970+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:56.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:13:56.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:57.932 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:57 smithi144 ceph-mon[32129]: pgmap v841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s 2024-07-26T11:13:57.932 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:57 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:resolve 2024-07-26T11:13:57.932 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:57 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:57 smithi160 ceph-mon[36715]: pgmap v841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s 2024-07-26T11:13:58.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:57 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:resolve 2024-07-26T11:13:58.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:57 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:57 smithi183 ceph-mon[37273]: pgmap v841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s 2024-07-26T11:13:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:57 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:resolve 2024-07-26T11:13:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:57 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:58 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:reconnect 2024-07-26T11:13:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:58 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:58 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:13:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:58 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:reconnect 2024-07-26T11:13:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:58 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:59.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:58 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:13:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:58 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:reconnect 2024-07-26T11:13:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:58 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:58 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00899975 2024-07-26T11:13:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:59 smithi144 ceph-mon[32129]: pgmap v842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T11:13:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:59 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:rejoin 2024-07-26T11:13:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:59 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:13:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:13:59 smithi144 ceph-mon[32129]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:14:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:59 smithi160 ceph-mon[36715]: pgmap v842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T11:14:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:59 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:rejoin 2024-07-26T11:14:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:59 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:13:59 smithi160 ceph-mon[36715]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:14:00.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-mon[37273]: pgmap v842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T11:14:00.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:rejoin 2024-07-26T11:14:00.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:00.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-mon[37273]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:14:00.098 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: *** Caught signal (Aborted) ** 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: in thread 7fd259afa640 thread_name:ms_dispatch 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd25eed16f0] 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd25ef1e94c] 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 3: raise() 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 4: abort() 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd25eebb71b] 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 6: /lib64/libc.so.6(+0x37386) [0x7fd25eeca386] 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f06d6186b9] 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f06d7d680c] 2024-07-26T11:14:00.099 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f06d7162ca] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f06d6e6753] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f06d6e7a1b] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f06d6cb752] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f06d619a56] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f06d61d637] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f06d61df5e] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f06d60103a] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd25f79e407] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 18: (DispatchQueue::entry()+0x60f) [0x7fd25f79ecf9] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd25f87b5a1] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 20: (Thread::entry_wrapper()+0x43) [0x7fd25f628cdd] 2024-07-26T11:14:00.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd25f628cf9] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd25ef1cc02] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd25efa1c40] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:13:59.743+0000 7fd259afa640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: in thread 7fd259afa640 thread_name:ms_dispatch 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd25eed16f0] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd25ef1e94c] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 3: raise() 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 4: abort() 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd25eebb71b] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 6: /lib64/libc.so.6(+0x37386) [0x7fd25eeca386] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f06d6186b9] 2024-07-26T11:14:00.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f06d7d680c] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f06d7162ca] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f06d6e6753] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f06d6e7a1b] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f06d6cb752] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f06d619a56] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f06d61d637] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f06d61df5e] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f06d60103a] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd25f79e407] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 18: (DispatchQueue::entry()+0x60f) [0x7fd25f79ecf9] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd25f87b5a1] 2024-07-26T11:14:00.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 20: (Thread::entry_wrapper()+0x43) [0x7fd25f628cdd] 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd25f628cf9] 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd25ef1cc02] 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd25efa1c40] 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 0> 2024-07-26T11:13:59.743+0000 7fd259afa640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: in thread 7fd259afa640 thread_name:ms_dispatch 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd25eed16f0] 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd25ef1e94c] 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 3: raise() 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 4: abort() 2024-07-26T11:14:00.103 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd25eebb71b] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 6: /lib64/libc.so.6(+0x37386) [0x7fd25eeca386] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f06d6186b9] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f06d7d680c] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f06d7162ca] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f06d6e6753] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f06d6e7a1b] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f06d6cb752] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f06d619a56] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f06d61d637] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f06d61df5e] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f06d60103a] 2024-07-26T11:14:00.104 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd25f79e407] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 18: (DispatchQueue::entry()+0x60f) [0x7fd25f79ecf9] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd25f87b5a1] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 20: (Thread::entry_wrapper()+0x43) [0x7fd25f628cdd] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd25f628cf9] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd25ef1cc02] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd25efa1c40] 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:00.105 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 0> 2024-07-26T11:13:59.743+0000 7fd259afa640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: in thread 7fd259afa640 thread_name:ms_dispatch 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd25eed16f0] 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd25ef1e94c] 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 3: raise() 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 4: abort() 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd25eebb71b] 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 6: /lib64/libc.so.6(+0x37386) [0x7fd25eeca386] 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f06d6186b9] 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f06d7d680c] 2024-07-26T11:14:00.106 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f06d7162ca] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f06d6e6753] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f06d6e7a1b] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f06d6cb752] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f06d619a56] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f06d61d637] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f06d61df5e] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f06d60103a] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd25f79e407] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 18: (DispatchQueue::entry()+0x60f) [0x7fd25f79ecf9] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd25f87b5a1] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 20: (Thread::entry_wrapper()+0x43) [0x7fd25f628cdd] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd25f628cf9] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd25ef1cc02] 2024-07-26T11:14:00.107 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd25efa1c40] 2024-07-26T11:14:00.108 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:00.108 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75672]: 2024-07-26T11:14:00.108 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:13:59 smithi183 podman[75840]: 2024-07-26 11:13:59.802626114 +0000 UTC m=+0.019079704 container died 171ea4b907da7170590c69528a5ed576e1a1f71b4deb42725873ff1240127c7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:14:00.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:00 smithi183 podman[75840]: 2024-07-26 11:14:00.472128959 +0000 UTC m=+0.688582549 container remove 171ea4b907da7170590c69528a5ed576e1a1f71b4deb42725873ff1240127c7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-07-26T11:14:00.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:14:00.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:14:00.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:00 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.919s CPU time. 2024-07-26T11:14:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:00 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:00 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:active 2024-07-26T11:14:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:00 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:00 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 484.535639 secs 2024-07-26T11:14:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:00 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:00 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:active 2024-07-26T11:14:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:00 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:00 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 484.535639 secs 2024-07-26T11:14:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:00 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:00 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/1582972062,v1:172.21.15.183:6833/1582972062] up:active 2024-07-26T11:14:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:00 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:00 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 484.535639 secs 2024-07-26T11:14:01.171 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:01.169+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480005b90 con 0x7f74b0109990 2024-07-26T11:14:01.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:01.170+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480005b90 con 0x7f74b0109990 2024-07-26T11:14:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:01.185+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478005b00 con 0x563c6c4182f0 2024-07-26T11:14:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:01.185+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478005b00 con 0x563c6c4182f0 2024-07-26T11:14:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:01 smithi160 ceph-mon[36715]: pgmap v843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:14:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:01 smithi183 ceph-mon[37273]: pgmap v843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:14:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:01 smithi144 ceph-mon[32129]: pgmap v843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:14:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:03 smithi160 ceph-mon[36715]: pgmap v844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:03 smithi183 ceph-mon[37273]: pgmap v844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:03 smithi144 ceph-mon[32129]: pgmap v844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:05 smithi160 ceph-mon[36715]: pgmap v845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 489.535762 secs 2024-07-26T11:14:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:05 smithi183 ceph-mon[37273]: pgmap v845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 489.535762 secs 2024-07-26T11:14:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:05 smithi144 ceph-mon[32129]: pgmap v845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 489.535762 secs 2024-07-26T11:14:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:07 smithi160 ceph-mon[36715]: pgmap v846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 188 B/s wr, 6 op/s 2024-07-26T11:14:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:07 smithi183 ceph-mon[37273]: pgmap v846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 188 B/s wr, 6 op/s 2024-07-26T11:14:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:07 smithi144 ceph-mon[32129]: pgmap v846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 188 B/s wr, 6 op/s 2024-07-26T11:14:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:09 smithi160 ceph-mon[36715]: pgmap v847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:09 smithi183 ceph-mon[37273]: pgmap v847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:09 smithi144 ceph-mon[32129]: pgmap v847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:10.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:14:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 494.535878 secs 2024-07-26T11:14:11.121 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 494.535878 secs 2024-07-26T11:14:11.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 494.535878 secs 2024-07-26T11:14:11.122 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:10 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:14:11.122 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:10 smithi183 systemd[1]: Stopped Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:14:11.122 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:10 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.919s CPU time. 2024-07-26T11:14:11.122 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:10 smithi183 systemd[1]: Starting Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:14:11.262 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:11.385 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 podman[75932]: 2024-07-26 11:14:11.022744067 +0000 UTC m=+0.017746923 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:14:11.385 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 podman[75932]: 2024-07-26 11:14:11.147453757 +0000 UTC m=+0.142456655 container create 65a3d7b7e4fe3862939e2f8962dbbbfb21d75a086de8dda63e929639c7d8b46a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:14:11.664 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 podman[75932]: 2024-07-26 11:14:11.418229926 +0000 UTC m=+0.413232795 container init 65a3d7b7e4fe3862939e2f8962dbbbfb21d75a086de8dda63e929639c7d8b46a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-07-26T11:14:11.664 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 podman[75932]: 2024-07-26 11:14:11.420878181 +0000 UTC m=+0.415881049 container start 65a3d7b7e4fe3862939e2f8962dbbbfb21d75a086de8dda63e929639c7d8b46a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-26T11:14:11.664 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mds[75952]: -- 172.21.15.183:0/3750741848 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5626b7c7d860 con 0x5626bbe1fa80 2024-07-26T11:14:11.664 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: starting mds.l at 2024-07-26T11:14:11.664 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 bash[75932]: 65a3d7b7e4fe3862939e2f8962dbbbfb21d75a086de8dda63e929639c7d8b46a 2024-07-26T11:14:11.664 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:11 smithi183 systemd[1]: Started Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:14:11.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:11.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:11.989 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:11.987+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:11.989 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:11.987+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:11.989 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:11.987+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:11.989 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:11.987+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: pgmap v848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: daemon mds.l restarted 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: osdmap e121: 12 total, 12 up, 12 in 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:boot 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:11 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: pgmap v848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: daemon mds.l restarted 2024-07-26T11:14:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:14:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: osdmap e121: 12 total, 12 up, 12 in 2024-07-26T11:14:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:boot 2024-07-26T11:14:12.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:14:12.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:12.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:11 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: pgmap v848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: daemon mds.l restarted 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: osdmap e121: 12 total, 12 up, 12 in 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:boot 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:11 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:13.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 -- 172.21.15.144:0/3144457041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010af10 msgr2=0x7f2fb010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 --2- 172.21.15.144:0/3144457041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010af10 0x7f2fb010b320 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009900 tx=0x7f2f9802a410 comp rx=0 tx=0).stop 2024-07-26T11:14:13.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 -- 172.21.15.144:0/3144457041 shutdown_connections 2024-07-26T11:14:13.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 --2- 172.21.15.144:0/3144457041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fb010b860 0x7f2fb0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 --2- 172.21.15.144:0/3144457041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010af10 0x7f2fb010b320 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 --2- 172.21.15.144:0/3144457041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.511+0000 7f2fb5600640 1 -- 172.21.15.144:0/3144457041 >> 172.21.15.144:0/3144457041 conn(0x7f2fb0101040 msgr2=0x7f2fb0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.512+0000 7f2fb5600640 1 -- 172.21.15.144:0/3144457041 shutdown_connections 2024-07-26T11:14:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.512+0000 7f2fb5600640 1 -- 172.21.15.144:0/3144457041 wait complete. 2024-07-26T11:14:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.513+0000 7f2fb5600640 1 Processor -- start 2024-07-26T11:14:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.513+0000 7f2fb5600640 1 -- start start 2024-07-26T11:14:13.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb5600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fb010af10 0x7f2fb019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb5600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010b860 0x7f2fb01a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb5600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01154d0 con 0x7f2fb01055d0 2024-07-26T11:14:13.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb5600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0115350 con 0x7f2fb010b860 2024-07-26T11:14:13.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb5600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0115650 con 0x7f2fb010af10 2024-07-26T11:14:13.517 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.517 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb019bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51146/0 (socket says 172.21.15.144:51146) 2024-07-26T11:14:13.517 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fafe00640 1 -- 172.21.15.144:0/79870473 learned_addr learned my addr 172.21.15.144:0/79870473 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:13.517 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010b860 0x7f2fb01a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.517 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.514+0000 7f2faf400640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fb010af10 0x7f2fb019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fafe00640 1 -- 172.21.15.144:0/79870473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fb010af10 msgr2=0x7f2fb019c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fafe00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fb010af10 0x7f2fb019c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fafe00640 1 -- 172.21.15.144:0/79870473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010b860 msgr2=0x7f2fb01a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fafe00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010b860 0x7f2fb01a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fafe00640 1 -- 172.21.15.144:0/79870473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb01a1e90 con 0x7f2fb01055d0 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010b860 0x7f2fb01a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2fafe00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb019bf20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000e9f0 tx=0x7f2fa000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.515+0000 7f2facc00640 1 -- 172.21.15.144:0/79870473 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa000cd80 con 0x7f2fb01055d0 2024-07-26T11:14:13.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.516+0000 7f2facc00640 1 -- 172.21.15.144:0/79870473 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa00108e0 con 0x7f2fb01055d0 2024-07-26T11:14:13.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.516+0000 7f2facc00640 1 -- 172.21.15.144:0/79870473 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0018980 con 0x7f2fb01055d0 2024-07-26T11:14:13.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.516+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb01a2180 con 0x7f2fb01055d0 2024-07-26T11:14:13.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.516+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb01a88e0 con 0x7f2fb01055d0 2024-07-26T11:14:13.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.519+0000 7f2fbd868740 1 Processor -- start 2024-07-26T11:14:13.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.519+0000 7f2fbd868740 1 -- start start 2024-07-26T11:14:13.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.519+0000 7f2facc00640 1 -- 172.21.15.144:0/79870473 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f2fa0010430 con 0x7f2fb01055d0 2024-07-26T11:14:13.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135572070 0x5561359171c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x556135917700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x556135917c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fbd868740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55613588db90 con 0x556135551390 2024-07-26T11:14:13.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fbd868740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55613587d090 con 0x5561355491b0 2024-07-26T11:14:13.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fbd868740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5561358810d0 con 0x556135572070 2024-07-26T11:14:13.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2faf400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x556135917c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x556135917700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2faf400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x556135917c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51148/0 (socket says 172.21.15.144:51148) 2024-07-26T11:14:13.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fb4c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135572070 0x5561359171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2faf400640 1 -- 172.21.15.144:0/1884910849 learned_addr learned my addr 172.21.15.144:0/1884910849 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.520+0000 7f2fafe00640 1 -- 172.21.15.144:0/1884910849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135572070 msgr2=0x5561359171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135572070 0x5561359171c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fafe00640 1 -- 172.21.15.144:0/1884910849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 msgr2=0x556135917c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x556135917c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fafe00640 1 -- 172.21.15.144:0/1884910849 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561355dd650 con 0x5561355491b0 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135572070 0x5561359171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x556135917700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2fa00e19f0 tx=0x7f2fa00ed780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d088607410caa3ec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2facc00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2f940b3dc0 0x7f2f940b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.521+0000 7f2fa5200640 1 -- 172.21.15.144:0/1884910849 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00f8070 con 0x5561355491b0 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.522+0000 7f2facc00640 1 -- 172.21.15.144:0/79870473 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 12060+0+0 (secure 0 0 0) 0x7f2fa00e5070 con 0x7f2fb01055d0 2024-07-26T11:14:13.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.522+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2f940b3dc0 0x7f2f940b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.522+0000 7f2fa5200640 1 -- 172.21.15.144:0/1884910849 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa0010720 con 0x5561355491b0 2024-07-26T11:14:13.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.522+0000 7f2fa5200640 1 -- 172.21.15.144:0/1884910849 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00aa8b0 con 0x5561355491b0 2024-07-26T11:14:13.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 -- 172.21.15.144:0/1884910849 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 msgr2=0x556135917700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x556135917700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2fa00e19f0 tx=0x7f2fa00ed780 comp rx=0 tx=0).stop 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2f940b3dc0 0x7f2f940b62d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009fd0 tx=0x7f2fa8009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 -- 172.21.15.144:0/1884910849 shutdown_connections 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x556135917c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x556135917700 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1884910849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135572070 0x5561359171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 -- 172.21.15.144:0/1884910849 >> 172.21.15.144:0/1884910849 conn(0x556135566460 msgr2=0x5561359150b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 -- 172.21.15.144:0/1884910849 shutdown_connections 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 -- 172.21.15.144:0/1884910849 wait complete. 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.523+0000 7f2fbd868740 1 -- start start 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 Processor -- start 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x55613592b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x55613592be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135572070 0x556135930ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5561356e0700 con 0x556135551390 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x556135819660 con 0x556135572070 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5561359259a0 con 0x5561355491b0 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.524+0000 7f2fbd868740 10 client.0 _ll_register_callbacks cb 0x556135829110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x55613592be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x55613592be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51164/0 (socket says 172.21.15.144:51164) 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 -- 172.21.15.144:0/2363826859 learned_addr learned my addr 172.21.15.144:0/2363826859 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2fafe00640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x55613592b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135572070 0x556135930ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 -- 172.21.15.144:0/2363826859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 msgr2=0x55613592b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x55613592b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 -- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135572070 msgr2=0x556135930ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135572070 0x556135930ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.525+0000 7f2faf400640 1 -- 172.21.15.144:0/2363826859 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556135830800 con 0x556135551390 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2faf400640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x55613592be80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2f98032e90 tx=0x7f2f98032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9917cef971377e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:13.530 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2f9d600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2f9d600640 1 -- 172.21.15.144:0/2363826859 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98035040 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2f9d600640 1 -- 172.21.15.144:0/2363826859 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 12060+0+0 (secure 0 0 0) 0x7f2f980317e0 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55613588fae0 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2f9d600640 1 -- 172.21.15.144:0/2363826859 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=233) ==== 24+0+0 (secure 0 0 0) 0x7f2f9804f050 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2fbd868740 10 client.16227 fetch_fsmap learned FSMap version 233 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.526+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=233,monmap=4+}) -- 0x556135939df0 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2f9d600640 1 -- 172.21.15.144:0/2363826859 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55613588fae0 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2f9d600640 1 -- 172.21.15.144:0/2363826859 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 233) ==== 5938+0+0 (secure 0 0 0) 0x7f2f98004750 con 0x556135551390 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2fbd868740 10 client.16227 fetch_fsmap finished waiting for FSMap version 233 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2fbd868740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2fbd868740 20 client.16227 populate_metadata read hostname 'smithi144' 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2fbd868740 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5561355309f0 0x55613593d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2fbd868740 4 client.16227 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:13.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.527+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55613588b580 con 0x5561355309f0 2024-07-26T11:14:13.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.528+0000 7f2fafe00640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5561355309f0 0x55613593d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.528+0000 7f2fafe00640 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5561355309f0 0x55613593d5c0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f931686dd0bc4b8b server_cookie=761fb92be0eecc21 in_seq=0 out_seq=0 2024-07-26T11:14:13.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.528+0000 7f2f9d600640 10 client.16227 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:13.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2f9d600640 1 -- 172.21.15.144:0/2363826859 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55613588b580 con 0x5561355309f0 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2f9d600640 10 client.16227 handle_command_reply: tid=0 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 client.16227 shutdown 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 msgr2=0x55613592be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x55613592be80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2f98032e90 tx=0x7f2f98032ec0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 shutdown_connections 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5561355309f0 0x55613593d5c0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135572070 0x556135930ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135551390 0x55613592be80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 --2- 172.21.15.144:0/2363826859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x55613592b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 >> 172.21.15.144:0/2363826859 conn(0x556135566460 msgr2=0x55613591a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2f9d600640 0 client.16227 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 shutdown_connections 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.529+0000 7f2fbd868740 1 -- 172.21.15.144:0/2363826859 wait complete. 2024-07-26T11:14:13.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.530+0000 7f2fbd868740 20 client.16227 trim_cache size 0 max 16384 2024-07-26T11:14:13.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.548+0000 7f2fbd868740 1 Processor -- start 2024-07-26T11:14:13.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.548+0000 7f2fbd868740 1 -- start start 2024-07-26T11:14:13.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x5561359140b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135551390 0x556135926bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135924cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fbd868740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01afb60 con 0x556135572070 2024-07-26T11:14:13.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fbd868740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01b0fd0 con 0x5561355491b0 2024-07-26T11:14:13.551 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fbd868740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01b0230 con 0x556135551390 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2faf400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135551390 0x556135926bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x5561359140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2faf400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135551390 0x556135926bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39454/0 (socket says 172.21.15.144:39454) 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2faf400640 1 -- 172.21.15.144:0/1700406646 learned_addr learned my addr 172.21.15.144:0/1700406646 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.549+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135924cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2fafe00640 1 -- 172.21.15.144:0/1700406646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135551390 msgr2=0x556135926bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135551390 0x556135926bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2fafe00640 1 -- 172.21.15.144:0/1700406646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 msgr2=0x556135924cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135924cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2fafe00640 1 -- 172.21.15.144:0/1700406646 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556135915e40 con 0x5561355491b0 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x5561359140b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2fa00a81d0 tx=0x7f2fa00aad30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd11c51b5f13973 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2f9fe00640 1 -- 172.21.15.144:0/1700406646 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00aa6d0 con 0x5561355491b0 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2f9fe00640 1 -- 172.21.15.144:0/1700406646 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa00aaa70 con 0x5561355491b0 2024-07-26T11:14:13.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.550+0000 7f2f9fe00640 1 -- 172.21.15.144:0/1700406646 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00b3f60 con 0x5561355491b0 2024-07-26T11:14:13.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 -- 172.21.15.144:0/1700406646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 msgr2=0x5561359140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x5561359140b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2fa00a81d0 tx=0x7f2fa00aad30 comp rx=0 tx=0).stop 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 -- 172.21.15.144:0/1700406646 shutdown_connections 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135924cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556135551390 0x556135926bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1700406646 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5561355491b0 0x5561359140b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 -- 172.21.15.144:0/1700406646 >> 172.21.15.144:0/1700406646 conn(0x556135566460 msgr2=0x5561359181f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 -- 172.21.15.144:0/1700406646 shutdown_connections 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.551+0000 7f2fbd868740 1 -- 172.21.15.144:0/1700406646 wait complete. 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.552+0000 7f2fbd868740 1 -- start start 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.552+0000 7f2fbd868740 1 Processor -- start 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x5561359305a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135551390 0x556135930b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135935ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01ae980 con 0x556135572070 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01b0900 con 0x556135551390 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01adc40 con 0x5561355491b0 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fbd868740 10 client.0 _ll_register_callbacks cb 0x7f2f98032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x5561359305a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x5561359305a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:39468/0 (socket says 172.21.15.144:39468) 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 -- 172.21.15.144:0/1153932568 learned_addr learned my addr 172.21.15.144:0/1153932568 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fb4c00640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135935ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2faf400640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135551390 0x556135930b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 -- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135551390 msgr2=0x556135930b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135551390 0x556135930b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 -- 172.21.15.144:0/1153932568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 msgr2=0x556135935ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135935ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.553+0000 7f2fafe00640 1 -- 172.21.15.144:0/1153932568 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556135936210 con 0x5561355491b0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2fafe00640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x5561359305a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c6a0 tx=0x7f2fa00e87e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324417865fb9573d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2f9d600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2f9d600640 1 -- 172.21.15.144:0/1153932568 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00aa6d0 con 0x5561355491b0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2f9d600640 1 -- 172.21.15.144:0/1153932568 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 12060+0+0 (secure 0 0 0) 0x7f2fa00a7550 con 0x5561355491b0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fa00aaa70 con 0x5561355491b0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2f9d600640 1 -- 172.21.15.144:0/1153932568 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=233) ==== 24+0+0 (secure 0 0 0) 0x7f2fa00c5050 con 0x5561355491b0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2fbd868740 10 client.26497 fetch_fsmap learned FSMap version 233 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.554+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=233,monmap=4+}) -- 0x5561359375e0 con 0x5561355491b0 2024-07-26T11:14:13.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2f9d600640 1 -- 172.21.15.144:0/1153932568 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa00aaa70 con 0x5561355491b0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2f9d600640 1 -- 172.21.15.144:0/1153932568 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 233) ==== 5938+0+0 (secure 0 0 0) 0x7f2fa00a71f0 con 0x5561355491b0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2fbd868740 10 client.26497 fetch_fsmap finished waiting for FSMap version 233 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2fbd868740 10 client.26497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2fbd868740 20 client.26497 populate_metadata read hostname 'smithi144' 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x556135505da0 0x556135937970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2fbd868740 4 client.26497 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556135888cc0 con 0x556135505da0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.555+0000 7f2faf400640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x556135505da0 0x556135937970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2faf400640 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x556135505da0 0x556135937970 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb681a3b918768d5 server_cookie=dace0224aaa8fea0 in_seq=0 out_seq=0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2f9d600640 10 client.26497 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2f9d600640 1 -- 172.21.15.144:0/1153932568 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556135888cc0 con 0x556135505da0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2f9d600640 10 client.26497 handle_command_reply: tid=0 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2fbd868740 1 client.26497 shutdown 2024-07-26T11:14:13.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 msgr2=0x5561359305a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x5561359305a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c6a0 tx=0x7f2fa00e87e0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 shutdown_connections 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556135572070 0x556135935ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.556+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x556135505da0 0x556135937970 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556135551390 0x556135930b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2fbd868740 1 --2- 172.21.15.144:0/1153932568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5561355491b0 0x5561359305a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2f9d600640 0 client.26497 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 >> 172.21.15.144:0/1153932568 conn(0x556135566460 msgr2=0x556135913e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 shutdown_connections 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2fbd868740 1 -- 172.21.15.144:0/1153932568 wait complete. 2024-07-26T11:14:13.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.557+0000 7f2fbd868740 20 client.26497 trim_cache size 0 max 16384 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:14:13.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2f940b3dc0 msgr2=0x7f2f940b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2f940b3dc0 0x7f2f940b62d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009fd0 tx=0x7f2fa8009500 comp rx=0 tx=0).stop 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 msgr2=0x7f2fb019bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb019bf20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000e9f0 tx=0x7f2fa000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 shutdown_connections 2024-07-26T11:14:13.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f2f940b3dc0 0x7f2f940b62d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2fb010b860 0x7f2fb01a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fb010af10 0x7f2fb019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 --2- 172.21.15.144:0/79870473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb01055d0 0x7f2fb019bf20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 >> 172.21.15.144:0/79870473 conn(0x7f2fb0101040 msgr2=0x7f2fb0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 shutdown_connections 2024-07-26T11:14:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:13.558+0000 7f2fb5600640 1 -- 172.21.15.144:0/79870473 wait complete. 2024-07-26T11:14:13.822 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:13 smithi144 ceph-mon[32129]: pgmap v850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:14:13.822 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:13 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:resolve 2024-07-26T11:14:13.822 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:13 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:13 smithi160 ceph-mon[36715]: pgmap v850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:14:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:13 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:resolve 2024-07-26T11:14:14.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:13 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:13 smithi183 ceph-mon[37273]: pgmap v850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:14:14.156 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:13 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:resolve 2024-07-26T11:14:14.156 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:13 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:14.654 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:14:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:14 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:reconnect 2024-07-26T11:14:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:14 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:14 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:14:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:14 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:reconnect 2024-07-26T11:14:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:14 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:14 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:14:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:14 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:reconnect 2024-07-26T11:14:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:14 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:14 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:14:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: pgmap v851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:14:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:rejoin 2024-07-26T11:14:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 499.536018 secs 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:active 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: *** Caught signal (Aborted) ** 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: in thread 7f9b7ce00640 thread_name:ms_dispatch 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:15.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b8303e6f0] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b8308b94c] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 3: raise() 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 4: abort() 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b8302871b] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b83037386] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a0e21916b9] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a0e234f80c] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a0e228f2ca] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a0e225f753] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a0e2260a1b] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a0e2244752] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a0e2192a56] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a0e2196637] 2024-07-26T11:14:15.972 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a0e2196f5e] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a0e217a03a] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b83b9b407] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b83b9bcf9] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b83c785a1] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b83a25cdd] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b83a25cf9] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b83089c02] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b8310ec40] 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.742+0000 7f9b7ce00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: in thread 7f9b7ce00640 thread_name:ms_dispatch 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b8303e6f0] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b8308b94c] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 3: raise() 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 4: abort() 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b8302871b] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b83037386] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a0e21916b9] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a0e234f80c] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a0e228f2ca] 2024-07-26T11:14:15.974 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a0e225f753] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a0e2260a1b] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a0e2244752] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a0e2192a56] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a0e2196637] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a0e2196f5e] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a0e217a03a] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b83b9b407] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b83b9bcf9] 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b83c785a1] 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b83a25cdd] 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b83a25cf9] 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b83089c02] 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b8310ec40] 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 0> 2024-07-26T11:14:15.742+0000 7f9b7ce00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: in thread 7f9b7ce00640 thread_name:ms_dispatch 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b8303e6f0] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b8308b94c] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 3: raise() 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 4: abort() 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b8302871b] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b83037386] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a0e21916b9] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a0e234f80c] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a0e228f2ca] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a0e225f753] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a0e2260a1b] 2024-07-26T11:14:15.977 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a0e2244752] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a0e2192a56] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a0e2196637] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a0e2196f5e] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a0e217a03a] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b83b9b407] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b83b9bcf9] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b83c785a1] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b83a25cdd] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b83a25cf9] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b83089c02] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b8310ec40] 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 0> 2024-07-26T11:14:15.742+0000 7f9b7ce00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:15.978 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: in thread 7f9b7ce00640 thread_name:ms_dispatch 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b8303e6f0] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b8308b94c] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 3: raise() 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 4: abort() 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b8302871b] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b83037386] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a0e21916b9] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a0e234f80c] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a0e228f2ca] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a0e225f753] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a0e2260a1b] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a0e2244752] 2024-07-26T11:14:15.979 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a0e2192a56] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a0e2196637] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a0e2196f5e] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a0e217a03a] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b83b9b407] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b83b9bcf9] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b83c785a1] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b83a25cdd] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b83a25cf9] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b83089c02] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b8310ec40] 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[105526]: 2024-07-26T11:14:15.980 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:15 smithi144 podman[107439]: 2024-07-26 11:14:15.923894252 +0000 UTC m=+0.036052154 container died 22d8ee358c9393e8459d109d9b652f65cd19f03d35af3640b00b8d4ab78c8f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: pgmap v851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:rejoin 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 499.536018 secs 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:active 2024-07-26T11:14:16.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:15 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: pgmap v851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:14:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:rejoin 2024-07-26T11:14:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:14:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 499.536018 secs 2024-07-26T11:14:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1925758146,v1:172.21.15.144:6835/1925758146] up:active 2024-07-26T11:14:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:15 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:16.625 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:16 smithi144 podman[107439]: 2024-07-26 11:14:16.5900577 +0000 UTC m=+0.702215663 container remove 22d8ee358c9393e8459d109d9b652f65cd19f03d35af3640b00b8d4ab78c8f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T11:14:16.626 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:16 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:14:17.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:17 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:14:17.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:17 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.506s CPU time. 2024-07-26T11:14:17.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:14:17 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:14:17.187+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:14:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:17 smithi183 ceph-mon[37273]: pgmap v852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:17 smithi144 ceph-mon[32129]: pgmap v852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:17 smithi160 ceph-mon[36715]: pgmap v852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:19 smithi183 ceph-mon[37273]: pgmap v853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:14:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:19 smithi144 ceph-mon[32129]: pgmap v853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:14:20.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:19 smithi160 ceph-mon[36715]: pgmap v853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:14:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 504.536133 secs 2024-07-26T11:14:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 504.536133 secs 2024-07-26T11:14:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 504.536133 secs 2024-07-26T11:14:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:21 smithi183 ceph-mon[37273]: pgmap v854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:22.158 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:21 smithi144 ceph-mon[32129]: pgmap v854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:21 smithi160 ceph-mon[36715]: pgmap v854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:23 smithi183 ceph-mon[37273]: pgmap v855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:14:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:23 smithi144 ceph-mon[32129]: pgmap v855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:14:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:23 smithi160 ceph-mon[36715]: pgmap v855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 184 B/s wr, 7 op/s 2024-07-26T11:14:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:25 smithi183 ceph-mon[37273]: pgmap v856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:14:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 509.536248 secs 2024-07-26T11:14:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:25 smithi144 ceph-mon[32129]: pgmap v856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:14:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 509.536248 secs 2024-07-26T11:14:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:25 smithi160 ceph-mon[36715]: pgmap v856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:14:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 509.536248 secs 2024-07-26T11:14:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:26.996+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:26.997+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:26.997+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:26.997+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:26.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:27.627 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:14:27.627 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 systemd[1]: Stopped Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:14:27.627 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.506s CPU time. 2024-07-26T11:14:27.627 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 systemd[1]: Starting Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:14:27.913 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:27 smithi144 ceph-mon[32129]: pgmap v857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:27.914 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 podman[107529]: 2024-07-26 11:14:27.529406368 +0000 UTC m=+0.033933678 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:14:27.914 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 podman[107529]: 2024-07-26 11:14:27.657239258 +0000 UTC m=+0.161766603 container create b8877d88957df3b8f23aa9e797a2cbb0cea8158e8d2ec4f632dad07402a0a89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T11:14:27.914 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 podman[107529]: 2024-07-26 11:14:27.911322047 +0000 UTC m=+0.415849426 container init b8877d88957df3b8f23aa9e797a2cbb0cea8158e8d2ec4f632dad07402a0a89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-07-26T11:14:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:27 smithi183 ceph-mon[37273]: pgmap v857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:28.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:27 smithi144 podman[107529]: 2024-07-26 11:14:27.922629871 +0000 UTC m=+0.427157202 container start b8877d88957df3b8f23aa9e797a2cbb0cea8158e8d2ec4f632dad07402a0a89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T11:14:28.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:28 smithi144 bash[107529]: b8877d88957df3b8f23aa9e797a2cbb0cea8158e8d2ec4f632dad07402a0a89f 2024-07-26T11:14:28.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:28 smithi144 systemd[1]: Started Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:14:28.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mds[107548]: -- 172.21.15.144:0/4033438028 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571025fd860 con 0x55710679f180 2024-07-26T11:14:28.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: starting mds.j at 2024-07-26T11:14:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:27 smithi160 ceph-mon[36715]: pgmap v857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: daemon mds.j restarted 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: osdmap e122: 12 total, 12 up, 12 in 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:boot 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:14:29.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:28 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: daemon mds.j restarted 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: osdmap e122: 12 total, 12 up, 12 in 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:boot 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:14:29.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:28 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: daemon mds.j restarted 2024-07-26T11:14:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: osdmap e122: 12 total, 12 up, 12 in 2024-07-26T11:14:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:boot 2024-07-26T11:14:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:14:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:14:29.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:28 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:30.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:30 smithi144 ceph-mon[32129]: pgmap v859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:30.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:30 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:resolve 2024-07-26T11:14:30.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:30 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:30 smithi183 ceph-mon[37273]: pgmap v859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:30 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:resolve 2024-07-26T11:14:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:30 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:30 smithi160 ceph-mon[36715]: pgmap v859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:30 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:resolve 2024-07-26T11:14:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:30 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:reconnect 2024-07-26T11:14:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:31 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:31 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:14:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 514.536376 secs 2024-07-26T11:14:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:reconnect 2024-07-26T11:14:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:31 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:31 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:14:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 514.536376 secs 2024-07-26T11:14:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:reconnect 2024-07-26T11:14:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:31 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:31 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109997 2024-07-26T11:14:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 514.536376 secs 2024-07-26T11:14:32.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:32 smithi144 ceph-mon[32129]: pgmap v860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:14:32.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:32 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:rejoin 2024-07-26T11:14:32.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:32 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:32.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:32 smithi144 ceph-mon[32129]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:14:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-mon[37273]: pgmap v860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:14:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:rejoin 2024-07-26T11:14:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-mon[37273]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:14:32.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:14:32.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: *** Caught signal (Aborted) ** 2024-07-26T11:14:32.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: in thread 7f3ebbf0b640 thread_name:ms_dispatch 2024-07-26T11:14:32.387 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3ec12e26f0] 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3ec132f94c] 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 3: raise() 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 4: abort() 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3ec12cc71b] 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 6: /lib64/libc.so.6(+0x37386) [0x7f3ec12db386] 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5626b59716b9] 2024-07-26T11:14:32.388 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5626b5b2f80c] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5626b5a6f2ca] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5626b5a3f753] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5626b5a40a1b] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5626b5a24752] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5626b5972a56] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5626b5976637] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5626b5976f5e] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5626b595a03a] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3ec1baf407] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 18: (DispatchQueue::entry()+0x60f) [0x7f3ec1bafcf9] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3ec1c8c5a1] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 20: (Thread::entry_wrapper()+0x43) [0x7f3ec1a39cdd] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3ec1a39cf9] 2024-07-26T11:14:32.389 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3ec132dc02] 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3ec13b2c40] 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.147+0000 7f3ebbf0b640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: in thread 7f3ebbf0b640 thread_name:ms_dispatch 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3ec12e26f0] 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3ec132f94c] 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 3: raise() 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 4: abort() 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3ec12cc71b] 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 6: /lib64/libc.so.6(+0x37386) [0x7f3ec12db386] 2024-07-26T11:14:32.390 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5626b59716b9] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5626b5b2f80c] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5626b5a6f2ca] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5626b5a3f753] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5626b5a40a1b] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5626b5a24752] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5626b5972a56] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5626b5976637] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5626b5976f5e] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5626b595a03a] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3ec1baf407] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 18: (DispatchQueue::entry()+0x60f) [0x7f3ec1bafcf9] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3ec1c8c5a1] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 20: (Thread::entry_wrapper()+0x43) [0x7f3ec1a39cdd] 2024-07-26T11:14:32.391 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3ec1a39cf9] 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3ec132dc02] 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3ec13b2c40] 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 0> 2024-07-26T11:14:32.147+0000 7f3ebbf0b640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: in thread 7f3ebbf0b640 thread_name:ms_dispatch 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3ec12e26f0] 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3ec132f94c] 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 3: raise() 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 4: abort() 2024-07-26T11:14:32.392 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3ec12cc71b] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 6: /lib64/libc.so.6(+0x37386) [0x7f3ec12db386] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5626b59716b9] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5626b5b2f80c] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5626b5a6f2ca] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5626b5a3f753] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5626b5a40a1b] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5626b5a24752] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5626b5972a56] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5626b5976637] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5626b5976f5e] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5626b595a03a] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3ec1baf407] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 18: (DispatchQueue::entry()+0x60f) [0x7f3ec1bafcf9] 2024-07-26T11:14:32.393 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3ec1c8c5a1] 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 20: (Thread::entry_wrapper()+0x43) [0x7f3ec1a39cdd] 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3ec1a39cf9] 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3ec132dc02] 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3ec13b2c40] 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 0> 2024-07-26T11:14:32.147+0000 7f3ebbf0b640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: in thread 7f3ebbf0b640 thread_name:ms_dispatch 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3ec12e26f0] 2024-07-26T11:14:32.394 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3ec132f94c] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 3: raise() 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 4: abort() 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3ec12cc71b] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 6: /lib64/libc.so.6(+0x37386) [0x7f3ec12db386] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5626b59716b9] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5626b5b2f80c] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5626b5a6f2ca] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5626b5a3f753] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5626b5a40a1b] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5626b5a24752] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5626b5972a56] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5626b5976637] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5626b5976f5e] 2024-07-26T11:14:32.395 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5626b595a03a] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3ec1baf407] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 18: (DispatchQueue::entry()+0x60f) [0x7f3ec1bafcf9] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3ec1c8c5a1] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 20: (Thread::entry_wrapper()+0x43) [0x7f3ec1a39cdd] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3ec1a39cf9] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3ec132dc02] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3ec13b2c40] 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[75947]: 2024-07-26T11:14:32.396 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 podman[75998]: 2024-07-26 11:14:32.207510233 +0000 UTC m=+0.018867646 container died 65a3d7b7e4fe3862939e2f8962dbbbfb21d75a086de8dda63e929639c7d8b46a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T11:14:32.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:14:32 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:14:32.189+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:14:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:32 smithi160 ceph-mon[36715]: pgmap v860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:14:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:32 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:rejoin 2024-07-26T11:14:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:32 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:32 smithi160 ceph-mon[36715]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:14:33.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:33 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:33.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:33 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:active 2024-07-26T11:14:33.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:33 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:33.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:33 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:33.159 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 podman[75998]: 2024-07-26 11:14:32.904266766 +0000 UTC m=+0.715624245 container remove 65a3d7b7e4fe3862939e2f8962dbbbfb21d75a086de8dda63e929639c7d8b46a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T11:14:33.159 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:32 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:14:33.256 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:33 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:33.256 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:33 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:active 2024-07-26T11:14:33.256 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:33 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:33.257 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:33 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:33 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:33 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/2613006145,v1:172.21.15.183:6833/2613006145] up:active 2024-07-26T11:14:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:33 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:33 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:33.635 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:33 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:14:33.635 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:33 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.924s CPU time. 2024-07-26T11:14:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:34 smithi183 ceph-mon[37273]: pgmap v861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:14:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:34 smithi144 ceph-mon[32129]: pgmap v861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:14:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:34 smithi160 ceph-mon[36715]: pgmap v861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:14:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:36 smithi144 ceph-mon[32129]: pgmap v862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 519.536500 secs 2024-07-26T11:14:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:36 smithi160 ceph-mon[36715]: pgmap v862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 519.536500 secs 2024-07-26T11:14:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:36 smithi183 ceph-mon[37273]: pgmap v862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 519.536500 secs 2024-07-26T11:14:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:38 smithi144 ceph-mon[32129]: pgmap v863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:38 smithi160 ceph-mon[36715]: pgmap v863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:38 smithi183 ceph-mon[37273]: pgmap v863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:14:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:40 smithi144 ceph-mon[32129]: pgmap v864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:14:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:40 smithi160 ceph-mon[36715]: pgmap v864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:14:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:40 smithi183 ceph-mon[37273]: pgmap v864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 B/s wr, 8 op/s 2024-07-26T11:14:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 524.536578 secs 2024-07-26T11:14:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 524.536578 secs 2024-07-26T11:14:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 524.536578 secs 2024-07-26T11:14:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:42.013+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:42.014+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:42.014+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:42.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:42.015+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:42 smithi144 ceph-mon[32129]: pgmap v865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:14:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:42 smithi160 ceph-mon[36715]: pgmap v865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:14:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:42 smithi183 ceph-mon[37273]: pgmap v865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:14:43.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:14:43.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 systemd[1]: Stopped Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:14:43.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.924s CPU time. 2024-07-26T11:14:43.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 systemd[1]: Starting Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:14:43.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 podman[76091]: 2024-07-26 11:14:43.522914064 +0000 UTC m=+0.017345192 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:14:43.899 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 podman[76091]: 2024-07-26 11:14:43.637913632 +0000 UTC m=+0.132344796 container create 3abb6a37b9a5a3acf695fa7d31aa10bf244bc95334ded257d32dce06c5de39e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-26T11:14:43.900 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 podman[76091]: 2024-07-26 11:14:43.898810965 +0000 UTC m=+0.393242091 container init 3abb6a37b9a5a3acf695fa7d31aa10bf244bc95334ded257d32dce06c5de39e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T11:14:44.172 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 podman[76091]: 2024-07-26 11:14:43.901400371 +0000 UTC m=+0.395831499 container start 3abb6a37b9a5a3acf695fa7d31aa10bf244bc95334ded257d32dce06c5de39e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-07-26T11:14:44.172 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 ceph-mds[76110]: -- 172.21.15.183:0/3415911891 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a960b41860 con 0x55a964ce3180 2024-07-26T11:14:44.172 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: starting mds.l at 2024-07-26T11:14:44.172 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 bash[76091]: 3abb6a37b9a5a3acf695fa7d31aa10bf244bc95334ded257d32dce06c5de39e4 2024-07-26T11:14:44.172 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:14:43 smithi183 systemd[1]: Started Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:14:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: pgmap v866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:14:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: daemon mds.l restarted 2024-07-26T11:14:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:14:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: osdmap e123: 12 total, 12 up, 12 in 2024-07-26T11:14:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:boot 2024-07-26T11:14:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:14:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:44 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: pgmap v866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: daemon mds.l restarted 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: osdmap e123: 12 total, 12 up, 12 in 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:boot 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:44 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: pgmap v866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:14:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: daemon mds.l restarted 2024-07-26T11:14:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:14:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:14:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: osdmap e123: 12 total, 12 up, 12 in 2024-07-26T11:14:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:boot 2024-07-26T11:14:44.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:14:44.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:14:44.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:44 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:44.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:14:45.145 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:46.182+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480005ec0 con 0x7f74b0109990 2024-07-26T11:14:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:46.191+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478005d20 con 0x563c6c4182f0 2024-07-26T11:14:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:46.196+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480005ec0 con 0x7f74b0109990 2024-07-26T11:14:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:46.197+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x563c6c7ff400 con 0x563c6c4182f0 2024-07-26T11:14:46.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:46 smithi144 ceph-mon[32129]: pgmap v868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:46.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:46 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:resolve 2024-07-26T11:14:46.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:46 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:46.436 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 529.536702 secs 2024-07-26T11:14:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:46 smithi160 ceph-mon[36715]: pgmap v868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:46 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:resolve 2024-07-26T11:14:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:46 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 529.536702 secs 2024-07-26T11:14:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:46 smithi183 ceph-mon[37273]: pgmap v868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:14:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:46 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:resolve 2024-07-26T11:14:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:46 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 529.536702 secs 2024-07-26T11:14:47.345 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:47 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:reconnect 2024-07-26T11:14:47.345 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:47 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:47.345 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:47 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0139998 2024-07-26T11:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:47 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:reconnect 2024-07-26T11:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:47 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:47 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0139998 2024-07-26T11:14:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:47 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:reconnect 2024-07-26T11:14:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:47 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:47 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0139998 2024-07-26T11:14:47.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 -- 172.21.15.144:0/266559503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc109670 msgr2=0x7f85fc10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 --2- 172.21.15.144:0/266559503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc109670 0x7f85fc10fea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009ca0 tx=0x7f85f402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 -- 172.21.15.144:0/266559503 shutdown_connections 2024-07-26T11:14:47.917 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 --2- 172.21.15.144:0/266559503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc109670 0x7f85fc10fea0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 --2- 172.21.15.144:0/266559503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc108d20 0x7f85fc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 --2- 172.21.15.144:0/266559503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 0x7f85fc1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 -- 172.21.15.144:0/266559503 >> 172.21.15.144:0/266559503 conn(0x7f85fc0fee70 msgr2=0x7f85fc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 -- 172.21.15.144:0/266559503 shutdown_connections 2024-07-26T11:14:47.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.914+0000 7f85fb200640 1 -- 172.21.15.144:0/266559503 wait complete. 2024-07-26T11:14:47.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.915+0000 7f85fb200640 1 Processor -- start 2024-07-26T11:14:47.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 -- start start 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 0x7f85fc075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc108d20 0x7f85fc075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc109670 0x7f85fc075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc113390 con 0x7f85fc108d20 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc113210 con 0x7f85fc1033e0 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85fb200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc113510 con 0x7f85fc109670 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 0x7f85fc075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.916+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 0x7f85fc075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55614/0 (socket says 172.21.15.144:55614) 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85f9e00640 1 -- 172.21.15.144:0/3351211614 learned_addr learned my addr 172.21.15.144:0/3351211614 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85f1400640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc108d20 0x7f85fc075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85fa800640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc109670 0x7f85fc075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85fa800640 1 -- 172.21.15.144:0/3351211614 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 msgr2=0x7f85fc075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85fa800640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 0x7f85fc075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85fa800640 1 -- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc108d20 msgr2=0x7f85fc075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85fa800640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc108d20 0x7f85fc075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85fa800640 1 -- 172.21.15.144:0/3351211614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc1a7fc0 con 0x7f85fc109670 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.917+0000 7f85f1400640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc108d20 0x7f85fc075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.918+0000 7f85fa800640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc109670 0x7f85fc075bc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f85f4005e30 tx=0x7f85f40317c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.918+0000 7f85f3400640 1 -- 172.21.15.144:0/3351211614 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4003b70 con 0x7f85fc109670 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.918+0000 7f85f3400640 1 -- 172.21.15.144:0/3351211614 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4003d10 con 0x7f85fc109670 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.918+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85fc1a8250 con 0x7f85fc109670 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.918+0000 7f85f3400640 1 -- 172.21.15.144:0/3351211614 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f402f970 con 0x7f85fc109670 2024-07-26T11:14:47.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.918+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85fc1a8790 con 0x7f85fc109670 2024-07-26T11:14:47.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.922+0000 7f8607838740 1 Processor -- start 2024-07-26T11:14:47.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.922+0000 7f85f3400640 1 -- 172.21.15.144:0/3351211614 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f85f402fb10 con 0x7f85fc109670 2024-07-26T11:14:47.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.922+0000 7f8607838740 1 -- start start 2024-07-26T11:14:47.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f8607838740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x56265255b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f8607838740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265255b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f8607838740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562652195390 0x56265255ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f8607838740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5626524d1bc0 con 0x56265218d1b0 2024-07-26T11:14:47.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f8607838740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5626524c10c0 con 0x5626521b6070 2024-07-26T11:14:47.926 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f8607838740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5626524c5100 con 0x562652195390 2024-07-26T11:14:47.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265255b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265255b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40404/0 (socket says 172.21.15.144:40404) 2024-07-26T11:14:47.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f85f9e00640 1 -- 172.21.15.144:0/4045580258 learned_addr learned my addr 172.21.15.144:0/4045580258 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:47.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f85f1400640 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x56265255b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.923+0000 7f85f1400640 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562652195390 0x56265255ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f9e00640 1 -- 172.21.15.144:0/4045580258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562652195390 msgr2=0x56265255ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f9e00640 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562652195390 0x56265255ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f9e00640 1 -- 172.21.15.144:0/4045580258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 msgr2=0x56265255b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f9e00640 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x56265255b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f9e00640 1 -- 172.21.15.144:0/4045580258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562652221650 con 0x56265218d1b0 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f3400640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f85e00b3d40 0x7f85e00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f9e00640 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265255b540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009d70 tx=0x7f85ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46033ddfe59de50 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f3400640 1 -- 172.21.15.144:0/3351211614 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 12232+0+0 (secure 0 0 0) 0x7f85f4105fd0 con 0x7f85fc109670 2024-07-26T11:14:47.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.924+0000 7f85f1400640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f85e00b3d40 0x7f85e00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.925+0000 7f85eb000640 1 -- 172.21.15.144:0/4045580258 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec01e070 con 0x56265218d1b0 2024-07-26T11:14:47.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.925+0000 7f85eb000640 1 -- 172.21.15.144:0/4045580258 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85ec019ae0 con 0x56265218d1b0 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.925+0000 7f85eb000640 1 -- 172.21.15.144:0/4045580258 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec00eab0 con 0x56265218d1b0 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 -- 172.21.15.144:0/4045580258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 msgr2=0x56265255b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265255b540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009d70 tx=0x7f85ec011810 comp rx=0 tx=0).stop 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f85f1400640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f85e00b3d40 0x7f85e00b6250 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f85e40058c0 tx=0x7f85e4005850 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 -- 172.21.15.144:0/4045580258 shutdown_connections 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562652195390 0x56265255ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265255b540 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 --2- 172.21.15.144:0/4045580258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x56265255b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 -- 172.21.15.144:0/4045580258 >> 172.21.15.144:0/4045580258 conn(0x5626521aa460 msgr2=0x562652558ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 -- 172.21.15.144:0/4045580258 shutdown_connections 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.926+0000 7f8607838740 1 -- 172.21.15.144:0/4045580258 wait complete. 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 -- start start 2024-07-26T11:14:47.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 Processor -- start 2024-07-26T11:14:47.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x562652566500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652566a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x562652566f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x562652324700 con 0x56265218d1b0 2024-07-26T11:14:47.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56265245d690 con 0x562652195390 2024-07-26T11:14:47.933 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.927+0000 7f8607838740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5626525697e0 con 0x5626521b6070 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f8607838740 10 client.0 _ll_register_callbacks cb 0x56265246d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x562652566500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x562652566500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40418/0 (socket says 172.21.15.144:40418) 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 -- 172.21.15.144:0/621770307 learned_addr learned my addr 172.21.15.144:0/621770307 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85fa800640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x562652566f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f1400640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652566a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 -- 172.21.15.144:0/621770307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 msgr2=0x562652566f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x562652566f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 -- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 msgr2=0x562652566a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652566a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85f9e00640 1 -- 172.21.15.144:0/621770307 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562652474830 con 0x56265218d1b0 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.928+0000 7f85fa800640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x562652566f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:47.934 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.929+0000 7f85f9e00640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x562652566500 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00e6d0 tx=0x7f85ec009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258046287afbaaf9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.929+0000 7f85df400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.929+0000 7f85df400640 1 -- 172.21.15.144:0/621770307 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec029020 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.929+0000 7f85df400640 1 -- 172.21.15.144:0/621770307 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 12232+0+0 (secure 0 0 0) 0x7f85ec004240 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.929+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85ec019ae0 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.929+0000 7f85df400640 1 -- 172.21.15.144:0/621770307 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=248) ==== 24+0+0 (secure 0 0 0) 0x7f85ec01e040 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 10 client.16251 fetch_fsmap learned FSMap version 248 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=248,monmap=4+}) -- 0x5626525752e0 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f85df400640 1 -- 172.21.15.144:0/621770307 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85ec019ae0 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f85df400640 1 -- 172.21.15.144:0/621770307 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 248) ==== 5938+0+0 (secure 0 0 0) 0x7f85ec02fba0 con 0x56265218d1b0 2024-07-26T11:14:47.935 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 10 client.16251 fetch_fsmap finished waiting for FSMap version 248 2024-07-26T11:14:47.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:14:47.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 20 client.16251 populate_metadata read hostname 'smithi144' 2024-07-26T11:14:47.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5626521749f0 0x562652578940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 4 client.16251 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:47.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.930+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626524d3b10 con 0x5626521749f0 2024-07-26T11:14:47.936 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.931+0000 7f85f1400640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5626521749f0 0x562652578940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.931+0000 7f85f1400640 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5626521749f0 0x562652578940 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb0cf1ae916deb2e server_cookie=66b8531803cefe8c in_seq=0 out_seq=0 2024-07-26T11:14:47.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.931+0000 7f85df400640 10 client.16251 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:47.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.932+0000 7f85df400640 1 -- 172.21.15.144:0/621770307 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5626524d3b10 con 0x5626521749f0 2024-07-26T11:14:47.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.932+0000 7f85df400640 10 client.16251 handle_command_reply: tid=0 2024-07-26T11:14:47.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.932+0000 7f8607838740 1 client.16251 shutdown 2024-07-26T11:14:47.937 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.932+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 msgr2=0x562652566500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.932+0000 7f8607838740 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x562652566500 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00e6d0 tx=0x7f85ec009a90 comp rx=0 tx=0).stop 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 shutdown_connections 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x562652566f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5626521749f0 0x562652578940 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652566a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 --2- 172.21.15.144:0/621770307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x562652566500 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 >> 172.21.15.144:0/621770307 conn(0x5626521aa460 msgr2=0x56265255e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f85df400640 0 client.16251 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 shutdown_connections 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 1 -- 172.21.15.144:0/621770307 wait complete. 2024-07-26T11:14:47.938 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.933+0000 7f8607838740 20 client.16251 trim_cache size 0 max 16384 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.951+0000 7f8607838740 1 Processor -- start 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 -- start start 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56265218d1b0 0x562652557ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562652195390 0x5626525728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x562652566b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc1b40e0 con 0x562652195390 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc1b5550 con 0x5626521b6070 2024-07-26T11:14:47.955 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f8607838740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc1b47b0 con 0x56265218d1b0 2024-07-26T11:14:47.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f85fa800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x562652566b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.956 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f85f1400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562652195390 0x5626525728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.952+0000 7f85fa800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x562652566b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55652/0 (socket says 172.21.15.144:55652) 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 -- 172.21.15.144:0/1376169974 learned_addr learned my addr 172.21.15.144:0/1376169974 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85f9e00640 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56265218d1b0 0x562652557ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 -- 172.21.15.144:0/1376169974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56265218d1b0 msgr2=0x562652557ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56265218d1b0 0x562652557ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 -- 172.21.15.144:0/1376169974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562652195390 msgr2=0x5626525728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562652195390 0x5626525728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 -- 172.21.15.144:0/1376169974 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56265255d2f0 con 0x5626521b6070 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85fa800640 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x562652566b80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f85f41029b0 tx=0x7f85f4004890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8504948dcb59ad63 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.953+0000 7f85ea600640 1 -- 172.21.15.144:0/1376169974 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f40cc510 con 0x5626521b6070 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f85ea600640 1 -- 172.21.15.144:0/1376169974 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4036040 con 0x5626521b6070 2024-07-26T11:14:47.957 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f85ea600640 1 -- 172.21.15.144:0/1376169974 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f40c81f0 con 0x5626521b6070 2024-07-26T11:14:47.958 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f8607838740 1 -- 172.21.15.144:0/1376169974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 msgr2=0x562652566b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f8607838740 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x562652566b80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f85f41029b0 tx=0x7f85f4004890 comp rx=0 tx=0).stop 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f8607838740 1 -- 172.21.15.144:0/1376169974 shutdown_connections 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f8607838740 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626521b6070 0x562652566b80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f8607838740 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562652195390 0x5626525728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.954+0000 7f8607838740 1 --2- 172.21.15.144:0/1376169974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56265218d1b0 0x562652557ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.955+0000 7f8607838740 1 -- 172.21.15.144:0/1376169974 >> 172.21.15.144:0/1376169974 conn(0x5626521aa460 msgr2=0x562652566470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.955+0000 7f8607838740 1 -- 172.21.15.144:0/1376169974 shutdown_connections 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.955+0000 7f8607838740 1 -- 172.21.15.144:0/1376169974 wait complete. 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.955+0000 7f8607838740 1 -- start start 2024-07-26T11:14:47.959 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 Processor -- start 2024-07-26T11:14:47.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265256f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652569a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x56265256a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc1b2f00 con 0x56265218d1b0 2024-07-26T11:14:47.960 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc1b4e80 con 0x562652195390 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc1b21c0 con 0x5626521b6070 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f8607838740 10 client.0 _ll_register_callbacks cb 0x7f85e4009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.956+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265256f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f9e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265256f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40424/0 (socket says 172.21.15.144:40424) 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f9e00640 1 -- 172.21.15.144:0/4274483861 learned_addr learned my addr 172.21.15.144:0/4274483861 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652569a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85fa800640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x56265256a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 -- 172.21.15.144:0/4274483861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 msgr2=0x56265256a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x56265256a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 -- 172.21.15.144:0/4274483861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 msgr2=0x56265256f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265256f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 -- 172.21.15.144:0/4274483861 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626525df0c0 con 0x562652195390 2024-07-26T11:14:47.961 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f9e00640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265256f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:47.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85f1400640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652569a50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x562652579040 tx=0x7f85e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12ced5d6c7372a18 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.962 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.957+0000 7f85df400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:14:47.963 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.958+0000 7f85df400640 1 -- 172.21.15.144:0/4274483861 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e40209f0 con 0x562652195390 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.958+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f4036040 con 0x562652195390 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.958+0000 7f85df400640 1 -- 172.21.15.144:0/4274483861 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 12232+0+0 (secure 0 0 0) 0x7f85e401a070 con 0x562652195390 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.958+0000 7f85df400640 1 -- 172.21.15.144:0/4274483861 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=248) ==== 24+0+0 (secure 0 0 0) 0x7f85e4020b90 con 0x562652195390 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.958+0000 7f8607838740 10 client.26495 fetch_fsmap learned FSMap version 248 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.958+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=248,monmap=4+}) -- 0x5626525e02b0 con 0x562652195390 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f85df400640 1 -- 172.21.15.144:0/4274483861 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4036040 con 0x562652195390 2024-07-26T11:14:47.964 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f85df400640 1 -- 172.21.15.144:0/4274483861 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 248) ==== 5938+0+0 (secure 0 0 0) 0x7f85e401d070 con 0x562652195390 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f8607838740 10 client.26495 fetch_fsmap finished waiting for FSMap version 248 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f8607838740 10 client.26495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f8607838740 20 client.26495 populate_metadata read hostname 'smithi144' 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f8607838740 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x562652149da0 0x5626525e29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f8607838740 4 client.26495 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5626524cf5b0 con 0x562652149da0 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.959+0000 7f85f9e00640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x562652149da0 0x5626525e29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f85df400640 10 client.26495 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f85f9e00640 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x562652149da0 0x5626525e29f0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=151e9de4c5a772d server_cookie=8fa36403216ad0bf in_seq=0 out_seq=0 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f85df400640 1 -- 172.21.15.144:0/4274483861 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5626524cf5b0 con 0x562652149da0 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f85df400640 10 client.26495 handle_command_reply: tid=0 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f8607838740 1 client.26495 shutdown 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 msgr2=0x562652569a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.965 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.960+0000 7f8607838740 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652569a50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x562652579040 tx=0x7f85e4004730 comp rx=0 tx=0).stop 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 shutdown_connections 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5626521b6070 0x56265256a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x562652149da0 0x5626525e29f0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562652195390 0x562652569a50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 --2- 172.21.15.144:0/4274483861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56265218d1b0 0x56265256f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 >> 172.21.15.144:0/4274483861 conn(0x5626521aa460 msgr2=0x562652557cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f85df400640 0 client.26495 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 shutdown_connections 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 1 -- 172.21.15.144:0/4274483861 wait complete. 2024-07-26T11:14:47.966 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.961+0000 7f8607838740 20 client.26495 trim_cache size 0 max 16384 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:14:47.967 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:14:47.968 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:14:47.968 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:14:47.968 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f85e00b3d40 msgr2=0x7f85e00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f85e00b3d40 0x7f85e00b6250 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f85e40058c0 tx=0x7f85e4005850 comp rx=0 tx=0).stop 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc109670 msgr2=0x7f85fc075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc109670 0x7f85fc075bc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f85f4005e30 tx=0x7f85f40317c0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 shutdown_connections 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f85e00b3d40 0x7f85e00b6250 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc109670 0x7f85fc075bc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85fc108d20 0x7f85fc075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 --2- 172.21.15.144:0/3351211614 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85fc1033e0 0x7f85fc075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 >> 172.21.15.144:0/3351211614 conn(0x7f85fc0fee70 msgr2=0x7f85fc108390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.969 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 shutdown_connections 2024-07-26T11:14:47.970 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:14:47.962+0000 7f85fb200640 1 -- 172.21.15.144:0/3351211614 wait complete. 2024-07-26T11:14:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-mon[32129]: pgmap v869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:14:48.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:rejoin 2024-07-26T11:14:48.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:48.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-mon[32129]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:14:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:48 smithi160 ceph-mon[36715]: pgmap v869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:14:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:rejoin 2024-07-26T11:14:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:48 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:48.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:48 smithi160 ceph-mon[36715]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:14:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:48 smithi183 ceph-mon[37273]: pgmap v869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s 2024-07-26T11:14:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:rejoin 2024-07-26T11:14:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:48 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:48 smithi183 ceph-mon[37273]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:14:48.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:14:48.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: *** Caught signal (Aborted) ** 2024-07-26T11:14:48.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: in thread 7f771c200640 thread_name:ms_dispatch 2024-07-26T11:14:48.722 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f772243e6f0] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f772248b94c] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 3: raise() 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 4: abort() 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 5: /lib64/libc.so.6(+0x2871b) [0x7f772242871b] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 6: /lib64/libc.so.6(+0x37386) [0x7f7722437386] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5570c997f6b9] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5570c9b3d80c] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5570c9a7d2ca] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5570c9a4d753] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5570c9a4ea1b] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5570c9a32752] 2024-07-26T11:14:48.723 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5570c9980a56] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5570c9984637] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5570c9984f5e] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5570c996803a] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7722f9b407] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 18: (DispatchQueue::entry()+0x60f) [0x7f7722f9bcf9] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f77230785a1] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 20: (Thread::entry_wrapper()+0x43) [0x7f7722e25cdd] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7722e25cf9] 2024-07-26T11:14:48.724 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7722489c02] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f772250ec40] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.252+0000 7f771c200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: in thread 7f771c200640 thread_name:ms_dispatch 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f772243e6f0] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f772248b94c] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 3: raise() 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 4: abort() 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 5: /lib64/libc.so.6(+0x2871b) [0x7f772242871b] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 6: /lib64/libc.so.6(+0x37386) [0x7f7722437386] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5570c997f6b9] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5570c9b3d80c] 2024-07-26T11:14:48.725 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5570c9a7d2ca] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5570c9a4d753] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5570c9a4ea1b] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5570c9a32752] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5570c9980a56] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5570c9984637] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5570c9984f5e] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5570c996803a] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7722f9b407] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 18: (DispatchQueue::entry()+0x60f) [0x7f7722f9bcf9] 2024-07-26T11:14:48.726 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f77230785a1] 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 20: (Thread::entry_wrapper()+0x43) [0x7f7722e25cdd] 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7722e25cf9] 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7722489c02] 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f772250ec40] 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 0> 2024-07-26T11:14:48.252+0000 7f771c200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: in thread 7f771c200640 thread_name:ms_dispatch 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f772243e6f0] 2024-07-26T11:14:48.727 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f772248b94c] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 3: raise() 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 4: abort() 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 5: /lib64/libc.so.6(+0x2871b) [0x7f772242871b] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 6: /lib64/libc.so.6(+0x37386) [0x7f7722437386] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5570c997f6b9] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5570c9b3d80c] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5570c9a7d2ca] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5570c9a4d753] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5570c9a4ea1b] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5570c9a32752] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5570c9980a56] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5570c9984637] 2024-07-26T11:14:48.728 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5570c9984f5e] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5570c996803a] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7722f9b407] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 18: (DispatchQueue::entry()+0x60f) [0x7f7722f9bcf9] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f77230785a1] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 20: (Thread::entry_wrapper()+0x43) [0x7f7722e25cdd] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7722e25cf9] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7722489c02] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f772250ec40] 2024-07-26T11:14:48.729 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 0> 2024-07-26T11:14:48.252+0000 7f771c200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: in thread 7f771c200640 thread_name:ms_dispatch 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f772243e6f0] 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f772248b94c] 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 3: raise() 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 4: abort() 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 5: /lib64/libc.so.6(+0x2871b) [0x7f772242871b] 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 6: /lib64/libc.so.6(+0x37386) [0x7f7722437386] 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5570c997f6b9] 2024-07-26T11:14:48.730 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5570c9b3d80c] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5570c9a7d2ca] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5570c9a4d753] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5570c9a4ea1b] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5570c9a32752] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5570c9980a56] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5570c9984637] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5570c9984f5e] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5570c996803a] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7722f9b407] 2024-07-26T11:14:48.731 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 18: (DispatchQueue::entry()+0x60f) [0x7f7722f9bcf9] 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f77230785a1] 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 20: (Thread::entry_wrapper()+0x43) [0x7f7722e25cdd] 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7722e25cf9] 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7722489c02] 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f772250ec40] 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:14:48.732 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[107544]: 2024-07-26T11:14:48.940 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:14:49.208 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:48 smithi144 podman[107775]: 2024-07-26 11:14:48.945119467 +0000 UTC m=+0.572639002 container died b8877d88957df3b8f23aa9e797a2cbb0cea8158e8d2ec4f632dad07402a0a89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T11:14:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:49 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:active 2024-07-26T11:14:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:49 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:49 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:49.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:49 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:49.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:active 2024-07-26T11:14:49.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:49 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:49.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:49 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:49 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:14:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2974722636,v1:172.21.15.144:6835/2974722636] up:active 2024-07-26T11:14:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:49 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:14:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:49 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:14:49.886 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:49 smithi144 podman[107775]: 2024-07-26 11:14:49.622894115 +0000 UTC m=+1.250413698 container remove b8877d88957df3b8f23aa9e797a2cbb0cea8158e8d2ec4f632dad07402a0a89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T11:14:49.886 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:49 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:14:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:50 smithi144 ceph-mon[32129]: pgmap v870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:14:50.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:50 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:14:50.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:14:50 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.227s CPU time. 2024-07-26T11:14:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:50 smithi160 ceph-mon[36715]: pgmap v870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:14:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:50 smithi183 ceph-mon[37273]: pgmap v870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:14:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 534.536853 secs 2024-07-26T11:14:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 534.536853 secs 2024-07-26T11:14:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 534.536853 secs 2024-07-26T11:14:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:52 smithi160 ceph-mon[36715]: pgmap v871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:14:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:52.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:52.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:14:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:52 smithi183 ceph-mon[37273]: pgmap v871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:14:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:14:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:52 smithi144 ceph-mon[32129]: pgmap v871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:14:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:14:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:53 smithi160 ceph-mon[36715]: pgmap v872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 7 op/s 2024-07-26T11:14:53.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:53 smithi160 ceph-mon[36715]: pgmap v873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 262 B/s wr, 8 op/s 2024-07-26T11:14:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:53 smithi183 ceph-mon[37273]: pgmap v872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 7 op/s 2024-07-26T11:14:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:53 smithi183 ceph-mon[37273]: pgmap v873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 262 B/s wr, 8 op/s 2024-07-26T11:14:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:53 smithi144 ceph-mon[32129]: pgmap v872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 7 op/s 2024-07-26T11:14:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:53 smithi144 ceph-mon[32129]: pgmap v873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 262 B/s wr, 8 op/s 2024-07-26T11:14:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:55 smithi160 ceph-mon[36715]: pgmap v874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 7 op/s 2024-07-26T11:14:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:55 smithi183 ceph-mon[37273]: pgmap v874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 7 op/s 2024-07-26T11:14:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:55 smithi144 ceph-mon[32129]: pgmap v874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 7 op/s 2024-07-26T11:14:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 539.536970 secs 2024-07-26T11:14:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 539.536970 secs 2024-07-26T11:14:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 539.536970 secs 2024-07-26T11:14:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:57.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:57.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:57.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:57.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:14:57.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:57 smithi160 ceph-mon[36715]: pgmap v875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 8 op/s 2024-07-26T11:14:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:57 smithi183 ceph-mon[37273]: pgmap v875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 8 op/s 2024-07-26T11:14:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:57 smithi144 ceph-mon[32129]: pgmap v875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 8 op/s 2024-07-26T11:14:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:14:59 smithi160 ceph-mon[36715]: pgmap v876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:14:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:14:59 smithi183 ceph-mon[37273]: pgmap v876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:14:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:14:59 smithi144 ceph-mon[32129]: pgmap v876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:15:00.615 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:15:00.615 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 systemd[1]: Stopped Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:00.615 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.227s CPU time. 2024-07-26T11:15:00.615 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 systemd[1]: Starting Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:15:00.913 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 podman[107993]: 2024-07-26 11:15:00.517053238 +0000 UTC m=+0.032639876 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:15:00.914 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 podman[107993]: 2024-07-26 11:15:00.656618351 +0000 UTC m=+0.172205006 container create b55442061a9160228b1274aa7f8667567dab58379322731844bedbf330c5a155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-07-26T11:15:01.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 podman[107993]: 2024-07-26 11:15:00.911750319 +0000 UTC m=+0.427337015 container init b55442061a9160228b1274aa7f8667567dab58379322731844bedbf330c5a155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-07-26T11:15:01.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:00 smithi144 podman[107993]: 2024-07-26 11:15:00.922018376 +0000 UTC m=+0.437605038 container start b55442061a9160228b1274aa7f8667567dab58379322731844bedbf330c5a155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-07-26T11:15:01.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:01 smithi144 bash[107993]: b55442061a9160228b1274aa7f8667567dab58379322731844bedbf330c5a155 2024-07-26T11:15:01.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mds[108011]: -- 172.21.15.144:0/2373629930 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55646528f860 con 0x556469431a80 2024-07-26T11:15:01.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:01 smithi144 systemd[1]: Started Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:01.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: starting mds.j at 2024-07-26T11:15:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: pgmap v877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 544.537087 secs 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: daemon mds.j restarted 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: osdmap e124: 12 total, 12 up, 12 in 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:boot 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:15:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:01 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: pgmap v877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 544.537087 secs 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: daemon mds.j restarted 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: osdmap e124: 12 total, 12 up, 12 in 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:boot 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:01 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: pgmap v877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:15:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 544.537087 secs 2024-07-26T11:15:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: daemon mds.j restarted 2024-07-26T11:15:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: osdmap e124: 12 total, 12 up, 12 in 2024-07-26T11:15:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:boot 2024-07-26T11:15:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-07-26T11:15:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:15:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:15:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:01 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:replay,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:03 smithi160 ceph-mon[36715]: pgmap v879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-07-26T11:15:03.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:03 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:resolve 2024-07-26T11:15:03.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:03 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:03.560 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:03 smithi144 ceph-mon[32129]: pgmap v879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-07-26T11:15:03.561 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:03 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:resolve 2024-07-26T11:15:03.561 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:03 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:03 smithi183 ceph-mon[37273]: pgmap v879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-07-26T11:15:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:03 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:resolve 2024-07-26T11:15:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:03 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:resolve,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:04 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:reconnect 2024-07-26T11:15:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:04 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:04 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:15:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:04 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:reconnect 2024-07-26T11:15:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:04 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:04 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:15:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:04 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:reconnect 2024-07-26T11:15:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:04 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:reconnect,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:04 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999972 2024-07-26T11:15:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-mon[37273]: pgmap v880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:15:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:rejoin 2024-07-26T11:15:05.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:05.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-mon[37273]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:15:05.636 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:15:05.636 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: *** Caught signal (Aborted) ** 2024-07-26T11:15:05.636 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: in thread 7f95706fe640 thread_name:ms_dispatch 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9575ad56f0] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9575b2294c] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 3: raise() 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 4: abort() 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9575abf71b] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 6: /lib64/libc.so.6(+0x37386) [0x7f9575ace386] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a95e4016b9] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a95e5bf80c] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a95e4ff2ca] 2024-07-26T11:15:05.637 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a95e4cf753] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a95e4d0a1b] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a95e4b4752] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a95e402a56] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a95e406637] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a95e406f5e] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a95e3ea03a] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f95763a2407] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 18: (DispatchQueue::entry()+0x60f) [0x7f95763a2cf9] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f957647f5a1] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 20: (Thread::entry_wrapper()+0x43) [0x7f957622ccdd] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 21: (Thread::_entry_func(void*)+0xd) [0x7f957622ccf9] 2024-07-26T11:15:05.638 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9575b20c02] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9575ba5c40] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.375+0000 7f95706fe640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: in thread 7f95706fe640 thread_name:ms_dispatch 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9575ad56f0] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9575b2294c] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 3: raise() 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 4: abort() 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9575abf71b] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 6: /lib64/libc.so.6(+0x37386) [0x7f9575ace386] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a95e4016b9] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a95e5bf80c] 2024-07-26T11:15:05.639 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a95e4ff2ca] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a95e4cf753] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a95e4d0a1b] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a95e4b4752] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a95e402a56] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a95e406637] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a95e406f5e] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a95e3ea03a] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f95763a2407] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 18: (DispatchQueue::entry()+0x60f) [0x7f95763a2cf9] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f957647f5a1] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 20: (Thread::entry_wrapper()+0x43) [0x7f957622ccdd] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 21: (Thread::_entry_func(void*)+0xd) [0x7f957622ccf9] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9575b20c02] 2024-07-26T11:15:05.640 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9575ba5c40] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 0> 2024-07-26T11:15:05.375+0000 7f95706fe640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: in thread 7f95706fe640 thread_name:ms_dispatch 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9575ad56f0] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9575b2294c] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 3: raise() 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 4: abort() 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9575abf71b] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 6: /lib64/libc.so.6(+0x37386) [0x7f9575ace386] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a95e4016b9] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a95e5bf80c] 2024-07-26T11:15:05.641 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a95e4ff2ca] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a95e4cf753] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a95e4d0a1b] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a95e4b4752] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a95e402a56] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a95e406637] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a95e406f5e] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a95e3ea03a] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f95763a2407] 2024-07-26T11:15:05.642 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 18: (DispatchQueue::entry()+0x60f) [0x7f95763a2cf9] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f957647f5a1] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 20: (Thread::entry_wrapper()+0x43) [0x7f957622ccdd] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 21: (Thread::_entry_func(void*)+0xd) [0x7f957622ccf9] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9575b20c02] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9575ba5c40] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 0> 2024-07-26T11:15:05.375+0000 7f95706fe640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: in thread 7f95706fe640 thread_name:ms_dispatch 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9575ad56f0] 2024-07-26T11:15:05.643 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9575b2294c] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 3: raise() 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 4: abort() 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9575abf71b] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 6: /lib64/libc.so.6(+0x37386) [0x7f9575ace386] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a95e4016b9] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a95e5bf80c] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a95e4ff2ca] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a95e4cf753] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a95e4d0a1b] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a95e4b4752] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a95e402a56] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a95e406637] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a95e406f5e] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a95e3ea03a] 2024-07-26T11:15:05.644 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f95763a2407] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 18: (DispatchQueue::entry()+0x60f) [0x7f95763a2cf9] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f957647f5a1] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 20: (Thread::entry_wrapper()+0x43) [0x7f957622ccdd] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 21: (Thread::_entry_func(void*)+0xd) [0x7f957622ccf9] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9575b20c02] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9575ba5c40] 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l[76106]: 2024-07-26T11:15:05.645 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:05 smithi183 podman[76273]: 2024-07-26 11:15:05.433948784 +0000 UTC m=+0.018376917 container died 3abb6a37b9a5a3acf695fa7d31aa10bf244bc95334ded257d32dce06c5de39e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T11:15:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:05 smithi144 ceph-mon[32129]: pgmap v880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:15:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:05 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:rejoin 2024-07-26T11:15:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:05 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:05 smithi144 ceph-mon[32129]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:15:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:05 smithi160 ceph-mon[36715]: pgmap v880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:15:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:05 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:rejoin 2024-07-26T11:15:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:05 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=l=up:rejoin,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:05 smithi160 ceph-mon[36715]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-07-26T11:15:06.446 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:06 smithi183 podman[76273]: 2024-07-26 11:15:06.194547385 +0000 UTC m=+0.778975510 container remove 3abb6a37b9a5a3acf695fa7d31aa10bf244bc95334ded257d32dce06c5de39e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T11:15:06.446 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:06 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:15:06.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:06 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:06.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:06 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:active 2024-07-26T11:15:06.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:06 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:06.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:06 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:15:06.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 549.537222 secs 2024-07-26T11:15:06.536 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:06 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:06.536 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:06 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:active 2024-07-26T11:15:06.536 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:06 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:06.536 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:06 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:15:06.537 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 549.537222 secs 2024-07-26T11:15:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:06 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:06 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.183:6832/3028866165,v1:172.21.15.183:6833/3028866165] up:active 2024-07-26T11:15:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:06 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-07-26T11:15:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:06 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=l=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-07-26T11:15:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 549.537222 secs 2024-07-26T11:15:06.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:06 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:15:06.885 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:06 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.948s CPU time. 2024-07-26T11:15:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:07 smithi183 ceph-mon[37273]: pgmap v881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:15:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:07 smithi144 ceph-mon[32129]: pgmap v881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:15:07.722 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:07 smithi160 ceph-mon[36715]: pgmap v881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:15:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:08 smithi183 ceph-mon[37273]: pgmap v882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:15:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:08 smithi144 ceph-mon[32129]: pgmap v882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:15:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:08 smithi160 ceph-mon[36715]: pgmap v882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 7 op/s 2024-07-26T11:15:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:10 smithi160 ceph-mon[36715]: pgmap v883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 554.537362 secs 2024-07-26T11:15:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:10 smithi183 ceph-mon[37273]: pgmap v883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 554.537362 secs 2024-07-26T11:15:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:10 smithi144 ceph-mon[32129]: pgmap v883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 554.537362 secs 2024-07-26T11:15:12.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:12.036+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:12.037+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:12.037+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:12.037+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:12.037+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:12 smithi183 ceph-mon[37273]: pgmap v884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 191 B/s wr, 7 op/s 2024-07-26T11:15:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:12 smithi144 ceph-mon[32129]: pgmap v884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 191 B/s wr, 7 op/s 2024-07-26T11:15:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:12 smithi160 ceph-mon[36715]: pgmap v884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 191 B/s wr, 7 op/s 2024-07-26T11:15:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:14 smithi183 ceph-mon[37273]: pgmap v885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:14 smithi144 ceph-mon[32129]: pgmap v885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:14 smithi160 ceph-mon[36715]: pgmap v885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 559.537509 secs 2024-07-26T11:15:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 559.537509 secs 2024-07-26T11:15:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 559.537509 secs 2024-07-26T11:15:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:16 smithi183 ceph-mon[37273]: pgmap v886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:15:17.135 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:16 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:15:17.135 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:16 smithi183 systemd[1]: Stopped Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:17.135 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:16 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Consumed 1.948s CPU time. 2024-07-26T11:15:17.136 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:16 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-07-26T11:15:17.136 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:16 smithi183 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T11:15:17.136 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 26 11:15:16 smithi183 systemd[1]: Failed to start Ceph mds.l for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:16 smithi144 ceph-mon[32129]: pgmap v886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:15:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:16 smithi160 ceph-mon[36715]: pgmap v886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:15:18.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:15:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:18 smithi183 ceph-mon[37273]: pgmap v887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:15:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:18 smithi144 ceph-mon[32129]: pgmap v887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:15:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:18 smithi160 ceph-mon[36715]: pgmap v887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:15:19.411 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: pgmap v888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 564.537625 secs 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: Replacing daemon mds.l as rank 1 with standby daemon mds.j 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: osdmap e125: 12 total, 12 up, 12 in 2024-07-26T11:15:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:20 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: pgmap v888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:15:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 564.537625 secs 2024-07-26T11:15:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: Replacing daemon mds.l as rank 1 with standby daemon mds.j 2024-07-26T11:15:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-07-26T11:15:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:21.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: osdmap e125: 12 total, 12 up, 12 in 2024-07-26T11:15:21.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:20 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: pgmap v888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:15:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 564.537625 secs 2024-07-26T11:15:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: Replacing daemon mds.l as rank 1 with standby daemon mds.j 2024-07-26T11:15:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-07-26T11:15:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: osdmap e125: 12 total, 12 up, 12 in 2024-07-26T11:15:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:20 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:21.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.976+0000 7f3f5ce00640 1 -- 172.21.15.144:0/4147860173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 msgr2=0x7f3f58109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:21.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.976+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/4147860173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 0x7f3f58109ff0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009900 tx=0x7f3f4802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 -- 172.21.15.144:0/4147860173 shutdown_connections 2024-07-26T11:15:21.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/4147860173 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810fe70 0x7f3f581166a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/4147860173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810f520 0x7f3f5810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.979 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/4147860173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 0x7f3f58109ff0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.980 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 -- 172.21.15.144:0/4147860173 >> 172.21.15.144:0/4147860173 conn(0x7f3f58077b10 msgr2=0x7f3f58107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:21.980 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 -- 172.21.15.144:0/4147860173 shutdown_connections 2024-07-26T11:15:21.980 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.977+0000 7f3f5ce00640 1 -- 172.21.15.144:0/4147860173 wait complete. 2024-07-26T11:15:21.980 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.979+0000 7f3f5ce00640 1 Processor -- start 2024-07-26T11:15:21.980 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.979+0000 7f3f5ce00640 1 -- start start 2024-07-26T11:15:21.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.979+0000 7f3f5ce00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 0x7f3f581a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.979+0000 7f3f5ce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810f520 0x7f3f581a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f5ce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 0x7f3f581a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f5ce00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58119b30 con 0x7f3f58109be0 2024-07-26T11:15:21.981 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f5ce00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581199b0 con 0x7f3f5810f520 2024-07-26T11:15:21.982 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f5ce00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58119cb0 con 0x7f3f5810fe70 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 0x7f3f581a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 0x7f3f581a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 0x7f3f581a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48300/0 (socket says 172.21.15.144:48300) 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 -- 172.21.15.144:0/3043392294 learned_addr learned my addr 172.21.15.144:0/3043392294 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810f520 0x7f3f581a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 -- 172.21.15.144:0/3043392294 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810f520 msgr2=0x7f3f581a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810f520 0x7f3f581a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 -- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 msgr2=0x7f3f581a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 0x7f3f581a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.980+0000 7f3f56800640 1 -- 172.21.15.144:0/3043392294 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f581a6490 con 0x7f3f5810fe70 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810f520 0x7f3f581a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f56800640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 0x7f3f581a5d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3f5000eb10 tx=0x7f3f5000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f4ea00640 1 -- 172.21.15.144:0/3043392294 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f50004b20 con 0x7f3f5810fe70 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f4ea00640 1 -- 172.21.15.144:0/3043392294 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f50004cc0 con 0x7f3f5810fe70 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f581a6780 con 0x7f3f5810fe70 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f4ea00640 1 -- 172.21.15.144:0/3043392294 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f50010670 con 0x7f3f5810fe70 2024-07-26T11:15:21.984 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.981+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f581acee0 con 0x7f3f5810fe70 2024-07-26T11:15:21.986 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.985+0000 7f3f63beb740 1 Processor -- start 2024-07-26T11:15:21.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.985+0000 7f3f4ea00640 1 -- 172.21.15.144:0/3043392294 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3f50005020 con 0x7f3f5810fe70 2024-07-26T11:15:21.987 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.985+0000 7f3f63beb740 1 -- start start 2024-07-26T11:15:21.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49925fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e499260330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e499260870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f63beb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e4991d6b90 con 0x55e498e921b0 2024-07-26T11:15:21.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f63beb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e4991c6090 con 0x55e498ebb070 2024-07-26T11:15:21.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f63beb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e4991ca0d0 con 0x55e498e9a390 2024-07-26T11:15:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e499260870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e499260870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48306/0 (socket says 172.21.15.144:48306) 2024-07-26T11:15:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f4fe00640 1 -- 172.21.15.144:0/2016437822 learned_addr learned my addr 172.21.15.144:0/2016437822 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e499260330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.986+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49925fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4fe00640 1 -- 172.21.15.144:0/2016437822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 msgr2=0x55e49925fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49925fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4fe00640 1 -- 172.21.15.144:0/2016437822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 msgr2=0x55e499260330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e499260330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4fe00640 1 -- 172.21.15.144:0/2016437822 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e498f26650 con 0x55e498e9a390 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e499260870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3f40011b60 tx=0x7f3f40005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f804b9a41c85950 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4ea00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3f3c0b3d40 0x7f3f3c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f47400640 1 -- 172.21.15.144:0/2016437822 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000fc50 con 0x55e498e9a390 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f47400640 1 -- 172.21.15.144:0/2016437822 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4000fdf0 con 0x55e498e9a390 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f47400640 1 -- 172.21.15.144:0/2016437822 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40003970 con 0x55e498e9a390 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.987+0000 7f3f4ea00640 1 -- 172.21.15.144:0/3043392294 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 12404+0+0 (secure 0 0 0) 0x7f3f500e5550 con 0x7f3f5810fe70 2024-07-26T11:15:21.990 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.988+0000 7f3f55e00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3f3c0b3d40 0x7f3f3c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.988+0000 7f3f63beb740 1 -- 172.21.15.144:0/2016437822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 msgr2=0x55e499260870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:21.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.988+0000 7f3f55e00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3f3c0b3d40 0x7f3f3c0b6250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3f48031440 tx=0x7f3f48009560 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:21.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.988+0000 7f3f63beb740 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e499260870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3f40011b60 tx=0x7f3f40005850 comp rx=0 tx=0).stop 2024-07-26T11:15:21.991 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 -- 172.21.15.144:0/2016437822 shutdown_connections 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e499260870 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e499260330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 --2- 172.21.15.144:0/2016437822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49925fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 -- 172.21.15.144:0/2016437822 >> 172.21.15.144:0/2016437822 conn(0x55e498eaf460 msgr2=0x55e49925dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 -- 172.21.15.144:0/2016437822 shutdown_connections 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 -- 172.21.15.144:0/2016437822 wait complete. 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.989+0000 7f3f63beb740 1 -- start start 2024-07-26T11:15:21.992 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 Processor -- start 2024-07-26T11:15:21.994 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e49926b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e49926b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55e499029700 con 0x55e498e921b0 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e499162660 con 0x55e498ebb070 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e49926e5d0 con 0x55e498e9a390 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.990+0000 7f3f63beb740 10 client.0 _ll_register_callbacks cb 0x55e499172110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e49926b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f56800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e49926b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48318/0 (socket says 172.21.15.144:48318) 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 -- 172.21.15.144:0/3960993283 learned_addr learned my addr 172.21.15.144:0/3960993283 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f55e00640 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e49926b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 -- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 msgr2=0x55e49926bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:21.995 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 -- 172.21.15.144:0/3960993283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 msgr2=0x55e49926b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e49926b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 -- 172.21.15.144:0/3960993283 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e499179800 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f4fe00640 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e49926b880 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3f40005460 tx=0x7f3f400096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7312de8d6b587baf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.991+0000 7f3f3be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f3be00640 1 -- 172.21.15.144:0/3960993283 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40005640 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4000fdf0 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f3be00640 1 -- 172.21.15.144:0/3960993283 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 12404+0+0 (secure 0 0 0) 0x7f3f400046c0 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f3be00640 1 -- 172.21.15.144:0/3960993283 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (secure 0 0 0) 0x7f3f4000fc20 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f63beb740 10 client.26545 fetch_fsmap learned FSMap version 259 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=259,monmap=4+}) -- 0x55e49927a180 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.992+0000 7f3f3be00640 1 -- 172.21.15.144:0/3960993283 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4000fdf0 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f3be00640 1 -- 172.21.15.144:0/3960993283 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 259) ==== 5333+0+0 (secure 0 0 0) 0x7f3f4001f900 con 0x55e498e9a390 2024-07-26T11:15:21.996 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f63beb740 10 client.26545 fetch_fsmap finished waiting for FSMap version 259 2024-07-26T11:15:21.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f63beb740 10 client.26545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:15:21.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f63beb740 20 client.26545 populate_metadata read hostname 'smithi144' 2024-07-26T11:15:21.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f63beb740 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e799f0 0x55e49927d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:21.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f63beb740 4 client.26545 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:21.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4991d8ae0 con 0x55e498e799f0 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.993+0000 7f3f55e00640 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e799f0 0x55e49927d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.994+0000 7f3f55e00640 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e799f0 0x55e49927d790 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=716e54efbf5d19f9 server_cookie=a77365b3f4693afc in_seq=0 out_seq=0 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.994+0000 7f3f3be00640 10 client.26545 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f3be00640 1 -- 172.21.15.144:0/3960993283 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e4991d8ae0 con 0x55e498e799f0 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f3be00640 10 client.26545 handle_command_reply: tid=0 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 client.26545 shutdown 2024-07-26T11:15:21.998 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 msgr2=0x55e49926b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e49926b880 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3f40005460 tx=0x7f3f400096a0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 shutdown_connections 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e799f0 0x55e49927d790 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e9a390 0x55e49926b880 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 --2- 172.21.15.144:0/3960993283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e921b0 0x55e49926b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f3be00640 0 client.26545 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 >> 172.21.15.144:0/3960993283 conn(0x55e498eaf460 msgr2=0x55e499262fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 shutdown_connections 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 1 -- 172.21.15.144:0/3960993283 wait complete. 2024-07-26T11:15:21.999 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:21.995+0000 7f3f63beb740 20 client.26545 trim_cache size 0 max 16384 2024-07-26T11:15:22.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.014+0000 7f3f63beb740 1 Processor -- start 2024-07-26T11:15:22.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.014+0000 7f3f63beb740 1 -- start start 2024-07-26T11:15:22.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e9a390 0x55e4992617e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f63beb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b9bd0 con 0x55e498e9a390 2024-07-26T11:15:22.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f63beb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b8090 con 0x55e498ebb070 2024-07-26T11:15:22.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f63beb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b9500 con 0x55e498e921b0 2024-07-26T11:15:22.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e9a390 0x55e4992617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48328/0 (socket says 172.21.15.144:48328) 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f55e00640 1 -- 172.21.15.144:0/1077862603 learned_addr learned my addr 172.21.15.144:0/1077862603 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.015+0000 7f3f56800640 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f55e00640 1 -- 172.21.15.144:0/1077862603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 msgr2=0x55e49926cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f55e00640 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f55e00640 1 -- 172.21.15.144:0/1077862603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e9a390 msgr2=0x55e4992617e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f55e00640 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e9a390 0x55e4992617e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f55e00640 1 -- 172.21.15.144:0/1077862603 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e49926de70 con 0x55e498e921b0 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f55e00640 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926bd80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009900 tx=0x7f3f4803f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af0e1410ce5940ab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f46a00640 1 -- 172.21.15.144:0/1077862603 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48056070 con 0x55e498e921b0 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f46a00640 1 -- 172.21.15.144:0/1077862603 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f48004750 con 0x55e498e921b0 2024-07-26T11:15:22.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.016+0000 7f3f46a00640 1 -- 172.21.15.144:0/1077862603 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48035040 con 0x55e498e921b0 2024-07-26T11:15:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 -- 172.21.15.144:0/1077862603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 msgr2=0x55e49926bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926bd80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009900 tx=0x7f3f4803f000 comp rx=0 tx=0).stop 2024-07-26T11:15:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 -- 172.21.15.144:0/1077862603 shutdown_connections 2024-07-26T11:15:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498ebb070 0x55e49926cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498e9a390 0x55e4992617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 --2- 172.21.15.144:0/1077862603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926bd80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 -- 172.21.15.144:0/1077862603 >> 172.21.15.144:0/1077862603 conn(0x55e498eaf460 msgr2=0x55e49926c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:22.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 -- 172.21.15.144:0/1077862603 shutdown_connections 2024-07-26T11:15:22.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.017+0000 7f3f63beb740 1 -- 172.21.15.144:0/1077862603 wait complete. 2024-07-26T11:15:22.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.018+0000 7f3f63beb740 1 -- start start 2024-07-26T11:15:22.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.018+0000 7f3f63beb740 1 Processor -- start 2024-07-26T11:15:22.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498e9a390 0x55e49926f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 0x55e4992e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b8e30 con 0x55e498ebb070 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b8760 con 0x55e498e9a390 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b6880 con 0x55e498e921b0 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f63beb740 10 client.0 _ll_register_callbacks cb 0x7f3f40011a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f56800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 0x55e4992e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498e9a390 0x55e49926f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f56800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 0x55e4992e3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51956/0 (socket says 172.21.15.144:51956) 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:48342/0 (socket says 172.21.15.144:48342) 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.019+0000 7f3f56800640 1 -- 172.21.15.144:0/4054793273 learned_addr learned my addr 172.21.15.144:0/4054793273 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f56800640 1 -- 172.21.15.144:0/4054793273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 msgr2=0x55e49926ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f56800640 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f56800640 1 -- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498e9a390 msgr2=0x55e49926f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:22.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f56800640 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498e9a390 0x55e49926f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f56800640 1 -- 172.21.15.144:0/4054793273 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4992e4450 con 0x55e498ebb070 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f55e00640 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f56800640 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 0x55e4992e3f10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3f500e1cb0 tx=0x7f3f50005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166524ab3b00b4b3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f3be00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f3be00640 1 -- 172.21.15.144:0/4054793273 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500ab8d0 con 0x55e498ebb070 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f3be00640 1 -- 172.21.15.144:0/4054793273 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 12404+0+0 (secure 0 0 0) 0x7f3f500b4000 con 0x55e498ebb070 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.020+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f48004750 con 0x55e498ebb070 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f3be00640 1 -- 172.21.15.144:0/4054793273 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (secure 0 0 0) 0x7f3f500ab1d0 con 0x55e498ebb070 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 10 client.16281 fetch_fsmap learned FSMap version 259 2024-07-26T11:15:22.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=259,monmap=4+}) -- 0x55e4992e5640 con 0x55e498ebb070 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f3be00640 1 -- 172.21.15.144:0/4054793273 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f48004750 con 0x55e498ebb070 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f3be00640 1 -- 172.21.15.144:0/4054793273 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 259) ==== 5333+0+0 (secure 0 0 0) 0x7f3f500b2be0 con 0x55e498ebb070 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 10 client.16281 fetch_fsmap finished waiting for FSMap version 259 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 20 client.16281 populate_metadata read hostname 'smithi144' 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e4eda0 0x55e4992e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 4 client.16281 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.021+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e4991d4580 con 0x55e498e4eda0 2024-07-26T11:15:22.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.022+0000 7f3f55e00640 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e4eda0 0x55e4992e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.022+0000 7f3f55e00640 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e4eda0 0x55e4992e7e50 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea4c5c284d3ede07 server_cookie=ee37ae55da522d59 in_seq=0 out_seq=0 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.022+0000 7f3f3be00640 10 client.16281 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f3be00640 1 -- 172.21.15.144:0/4054793273 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e4991d4580 con 0x55e498e4eda0 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f3be00640 10 client.16281 handle_command_reply: tid=0 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 client.16281 shutdown 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 msgr2=0x55e4992e3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 0x55e4992e3f10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3f500e1cb0 tx=0x7f3f50005650 comp rx=0 tx=0).stop 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 shutdown_connections 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e498ebb070 0x55e4992e3f10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55e498e4eda0 0x55e4992e7e50 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e498e9a390 0x55e49926f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 --2- 172.21.15.144:0/4054793273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e498e921b0 0x55e49926ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 >> 172.21.15.144:0/4054793273 conn(0x55e498eaf460 msgr2=0x55e499266890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f3be00640 0 client.16281 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 shutdown_connections 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 1 -- 172.21.15.144:0/4054793273 wait complete. 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.023+0000 7f3f63beb740 20 client.16281 trim_cache size 0 max 16384 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:15:22.028 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:15:22.029 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:15:22.029 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:15:22.029 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:15:22.029 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:15:22.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3f3c0b3d40 msgr2=0x7f3f3c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:22.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3f3c0b3d40 0x7f3f3c0b6250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3f48031440 tx=0x7f3f48009560 comp rx=0 tx=0).stop 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 msgr2=0x7f3f581a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 0x7f3f581a5d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3f5000eb10 tx=0x7f3f5000c2d0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 shutdown_connections 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f3f3c0b3d40 0x7f3f3c0b6250 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f5810fe70 0x7f3f581a5d60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f5810f520 0x7f3f581a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 --2- 172.21.15.144:0/3043392294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f58109be0 0x7f3f581a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 >> 172.21.15.144:0/3043392294 conn(0x7f3f58077b10 msgr2=0x7f3f58107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 shutdown_connections 2024-07-26T11:15:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:22.024+0000 7f3f5ce00640 1 -- 172.21.15.144:0/3043392294 wait complete. 2024-07-26T11:15:22.856 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:15:23.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:22 smithi144 ceph-mon[32129]: pgmap v890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:15:23.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:22 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:resolve 2024-07-26T11:15:23.050 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:22 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:22 smithi183 ceph-mon[37273]: pgmap v890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:15:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:22 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:resolve 2024-07-26T11:15:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:22 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:22 smithi160 ceph-mon[36715]: pgmap v890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:15:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:22 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:resolve 2024-07-26T11:15:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:22 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:23 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:reconnect 2024-07-26T11:15:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:23 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:23 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999984 2024-07-26T11:15:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:23 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:reconnect 2024-07-26T11:15:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:23 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:23 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999984 2024-07-26T11:15:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:23 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:reconnect 2024-07-26T11:15:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:23 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:23 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999984 2024-07-26T11:15:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:24 smithi183 ceph-mon[37273]: pgmap v891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:15:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:24 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:rejoin 2024-07-26T11:15:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:24 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:24 smithi183 ceph-mon[37273]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:15:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-mon[32129]: pgmap v891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:15:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:rejoin 2024-07-26T11:15:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-mon[32129]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:15:25.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:15:25.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: *** Caught signal (Aborted) ** 2024-07-26T11:15:25.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: in thread 7f323e400640 thread_name:ms_dispatch 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f324483e6f0] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f324488b94c] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 3: raise() 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 4: abort() 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 5: /lib64/libc.so.6(+0x2871b) [0x7f324482871b] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 6: /lib64/libc.so.6(+0x37386) [0x7f3244837386] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55644e24b6b9] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55644e40980c] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55644e3492ca] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55644e319753] 2024-07-26T11:15:25.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55644e31aa1b] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55644e2fe752] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55644e24ca56] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55644e250637] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55644e250f5e] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55644e23403a] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f324539b407] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 18: (DispatchQueue::entry()+0x60f) [0x7f324539bcf9] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32454785a1] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 20: (Thread::entry_wrapper()+0x43) [0x7f3245225cdd] 2024-07-26T11:15:25.223 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3245225cf9] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3244889c02] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f324490ec40] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:24.929+0000 7f323e400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: in thread 7f323e400640 thread_name:ms_dispatch 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f324483e6f0] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f324488b94c] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 3: raise() 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 4: abort() 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 5: /lib64/libc.so.6(+0x2871b) [0x7f324482871b] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 6: /lib64/libc.so.6(+0x37386) [0x7f3244837386] 2024-07-26T11:15:25.224 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55644e24b6b9] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55644e40980c] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55644e3492ca] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55644e319753] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55644e31aa1b] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55644e2fe752] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55644e24ca56] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55644e250637] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55644e250f5e] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55644e23403a] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f324539b407] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 18: (DispatchQueue::entry()+0x60f) [0x7f324539bcf9] 2024-07-26T11:15:25.225 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32454785a1] 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 20: (Thread::entry_wrapper()+0x43) [0x7f3245225cdd] 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3245225cf9] 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3244889c02] 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f324490ec40] 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 0> 2024-07-26T11:15:24.929+0000 7f323e400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: in thread 7f323e400640 thread_name:ms_dispatch 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:25.226 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f324483e6f0] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f324488b94c] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 3: raise() 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 4: abort() 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 5: /lib64/libc.so.6(+0x2871b) [0x7f324482871b] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 6: /lib64/libc.so.6(+0x37386) [0x7f3244837386] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55644e24b6b9] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55644e40980c] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55644e3492ca] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55644e319753] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55644e31aa1b] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55644e2fe752] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55644e24ca56] 2024-07-26T11:15:25.227 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55644e250637] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55644e250f5e] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55644e23403a] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f324539b407] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 18: (DispatchQueue::entry()+0x60f) [0x7f324539bcf9] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32454785a1] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 20: (Thread::entry_wrapper()+0x43) [0x7f3245225cdd] 2024-07-26T11:15:25.228 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3245225cf9] 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3244889c02] 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f324490ec40] 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 0> 2024-07-26T11:15:24.929+0000 7f323e400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: in thread 7f323e400640 thread_name:ms_dispatch 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f324483e6f0] 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f324488b94c] 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 3: raise() 2024-07-26T11:15:25.229 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 4: abort() 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 5: /lib64/libc.so.6(+0x2871b) [0x7f324482871b] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 6: /lib64/libc.so.6(+0x37386) [0x7f3244837386] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55644e24b6b9] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55644e40980c] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55644e3492ca] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55644e319753] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55644e31aa1b] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55644e2fe752] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55644e24ca56] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55644e250637] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55644e250f5e] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55644e23403a] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f324539b407] 2024-07-26T11:15:25.230 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 18: (DispatchQueue::entry()+0x60f) [0x7f324539bcf9] 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32454785a1] 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 20: (Thread::entry_wrapper()+0x43) [0x7f3245225cdd] 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3245225cf9] 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3244889c02] 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f324490ec40] 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:24 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108007]: 2024-07-26T11:15:25.231 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:25 smithi144 podman[108248]: 2024-07-26 11:15:25.059223218 +0000 UTC m=+0.033623502 container died b55442061a9160228b1274aa7f8667567dab58379322731844bedbf330c5a155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-26T11:15:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:24 smithi160 ceph-mon[36715]: pgmap v891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:15:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:24 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:rejoin 2024-07-26T11:15:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:24 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:24 smithi160 ceph-mon[36715]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:15:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:25 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:25 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:active 2024-07-26T11:15:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:25 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:25 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:15:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 569.537735 secs 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:25 smithi144 podman[108248]: 2024-07-26 11:15:25.948316987 +0000 UTC m=+0.922717292 container remove b55442061a9160228b1274aa7f8667567dab58379322731844bedbf330c5a155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:25 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:25 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:25 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:active 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:25 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:25 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:15:26.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 569.537735 secs 2024-07-26T11:15:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:25 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:25 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3785172289,v1:172.21.15.144:6835/3785172289] up:active 2024-07-26T11:15:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:25 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:25 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:15:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 569.537735 secs 2024-07-26T11:15:26.911 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:15:26.911 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.351s CPU time. 2024-07-26T11:15:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:27.053+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:27.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:27.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:27.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:27.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:26 smithi144 ceph-mon[32129]: pgmap v892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:15:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:26 smithi160 ceph-mon[36715]: pgmap v892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:15:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:26 smithi183 ceph-mon[37273]: pgmap v892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:15:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:28 smithi144 ceph-mon[32129]: pgmap v893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:28 smithi160 ceph-mon[36715]: pgmap v893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:28 smithi183 ceph-mon[37273]: pgmap v893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:30 smithi144 ceph-mon[32129]: pgmap v894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 574.537818 secs 2024-07-26T11:15:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:30 smithi160 ceph-mon[36715]: pgmap v894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 574.537818 secs 2024-07-26T11:15:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:30 smithi183 ceph-mon[37273]: pgmap v894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:15:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 574.537818 secs 2024-07-26T11:15:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:32 smithi144 ceph-mon[32129]: pgmap v895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 185 B/s wr, 6 op/s 2024-07-26T11:15:33.224 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:32 smithi183 ceph-mon[37273]: pgmap v895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 185 B/s wr, 6 op/s 2024-07-26T11:15:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:32 smithi160 ceph-mon[36715]: pgmap v895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 185 B/s wr, 6 op/s 2024-07-26T11:15:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:34 smithi144 ceph-mon[32129]: pgmap v896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:34 smithi160 ceph-mon[36715]: pgmap v896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:34 smithi183 ceph-mon[37273]: pgmap v896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:36.211 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 579.537960 secs 2024-07-26T11:15:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 579.537960 secs 2024-07-26T11:15:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 579.537960 secs 2024-07-26T11:15:37.130 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:36 smithi144 ceph-mon[32129]: pgmap v897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:37.130 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:15:37.130 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:36 smithi144 systemd[1]: Stopped Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:37.130 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.351s CPU time. 2024-07-26T11:15:37.130 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:36 smithi144 systemd[1]: Starting Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:15:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:36 smithi160 ceph-mon[36715]: pgmap v897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:36 smithi183 ceph-mon[37273]: pgmap v897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:37.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 podman[108349]: 2024-07-26 11:15:37.032355792 +0000 UTC m=+0.031629011 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:15:37.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 podman[108349]: 2024-07-26 11:15:37.190768005 +0000 UTC m=+0.190041261 container create d1bf2e0b51ea46ef9166fce988a361df6e888aaddda57711edb2ec662ad1d855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:15:37.952 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 podman[108349]: 2024-07-26 11:15:37.469779683 +0000 UTC m=+0.469052985 container init d1bf2e0b51ea46ef9166fce988a361df6e888aaddda57711edb2ec662ad1d855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:15:37.952 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 podman[108349]: 2024-07-26 11:15:37.480198327 +0000 UTC m=+0.479471606 container start d1bf2e0b51ea46ef9166fce988a361df6e888aaddda57711edb2ec662ad1d855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724) 2024-07-26T11:15:37.952 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 bash[108349]: d1bf2e0b51ea46ef9166fce988a361df6e888aaddda57711edb2ec662ad1d855 2024-07-26T11:15:37.952 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 systemd[1]: Started Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:37.953 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mds[108368]: -- 172.21.15.144:0/1300293461 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d3cbe7860 con 0x560d40d89600 2024-07-26T11:15:37.953 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: starting mds.j at 2024-07-26T11:15:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: daemon mds.j restarted 2024-07-26T11:15:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:15:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: osdmap e126: 12 total, 12 up, 12 in 2024-07-26T11:15:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:boot 2024-07-26T11:15:38.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:15:38.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:15:38.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:37 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: daemon mds.j restarted 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: osdmap e126: 12 total, 12 up, 12 in 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:boot 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:15:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:37 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: daemon mds.j restarted 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: osdmap e126: 12 total, 12 up, 12 in 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:boot 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:15:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:37 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:38 smithi144 ceph-mon[32129]: pgmap v899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:15:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:38 smithi160 ceph-mon[36715]: pgmap v899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:15:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:38 smithi183 ceph-mon[37273]: pgmap v899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:15:40.228 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:39 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:resolve 2024-07-26T11:15:40.229 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:39 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:39 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:resolve 2024-07-26T11:15:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:39 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:39 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:resolve 2024-07-26T11:15:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:39 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:40 smithi160 ceph-mon[36715]: pgmap v900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:15:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:40 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:reconnect 2024-07-26T11:15:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:40 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:40 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:15:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 584.538064 secs 2024-07-26T11:15:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:40 smithi183 ceph-mon[37273]: pgmap v900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:15:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:40 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:reconnect 2024-07-26T11:15:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:40 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:40 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:15:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 584.538064 secs 2024-07-26T11:15:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:40 smithi144 ceph-mon[32129]: pgmap v900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:15:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:40 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:reconnect 2024-07-26T11:15:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:40 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:40 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:15:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 584.538064 secs 2024-07-26T11:15:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:42.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:42.073 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:42.071+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:42.073 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:42.071+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:42.073 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:42.071+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:42.073 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:42.071+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:42.267 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:41 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:rejoin 2024-07-26T11:15:42.267 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:41 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:41 smithi144 ceph-mon[32129]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: *** Caught signal (Aborted) ** 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: in thread 7f8c99a00640 thread_name:ms_dispatch 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c9fc3e6f0] 2024-07-26T11:15:42.268 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c9fc8b94c] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 3: raise() 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 4: abort() 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c9fc2871b] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c9fc37386] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55738b6d46b9] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55738b89280c] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55738b7d22ca] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55738b7a2753] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55738b7a3a1b] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55738b787752] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55738b6d5a56] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55738b6d9637] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55738b6d9f5e] 2024-07-26T11:15:42.269 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55738b6bd03a] 2024-07-26T11:15:42.270 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ca079b407] 2024-07-26T11:15:42.270 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ca079bcf9] 2024-07-26T11:15:42.270 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ca08785a1] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ca0625cdd] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ca0625cf9] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c9fc89c02] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c9fd0ec40] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.076+0000 7f8c99a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: in thread 7f8c99a00640 thread_name:ms_dispatch 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c9fc3e6f0] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c9fc8b94c] 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 3: raise() 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 4: abort() 2024-07-26T11:15:42.271 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c9fc2871b] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c9fc37386] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55738b6d46b9] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55738b89280c] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55738b7d22ca] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55738b7a2753] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55738b7a3a1b] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55738b787752] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55738b6d5a56] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55738b6d9637] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55738b6d9f5e] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55738b6bd03a] 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ca079b407] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ca079bcf9] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ca08785a1] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ca0625cdd] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ca0625cf9] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c9fc89c02] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c9fd0ec40] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 0> 2024-07-26T11:15:42.076+0000 7f8c99a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: in thread 7f8c99a00640 thread_name:ms_dispatch 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c9fc3e6f0] 2024-07-26T11:15:42.273 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c9fc8b94c] 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 3: raise() 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 4: abort() 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c9fc2871b] 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c9fc37386] 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55738b6d46b9] 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55738b89280c] 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55738b7d22ca] 2024-07-26T11:15:42.274 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55738b7a2753] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55738b7a3a1b] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55738b787752] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55738b6d5a56] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55738b6d9637] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55738b6d9f5e] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55738b6bd03a] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ca079b407] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ca079bcf9] 2024-07-26T11:15:42.275 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ca08785a1] 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ca0625cdd] 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ca0625cf9] 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c9fc89c02] 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c9fd0ec40] 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 0> 2024-07-26T11:15:42.076+0000 7f8c99a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: in thread 7f8c99a00640 thread_name:ms_dispatch 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.276 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8c9fc3e6f0] 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8c9fc8b94c] 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 3: raise() 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 4: abort() 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8c9fc2871b] 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 6: /lib64/libc.so.6(+0x37386) [0x7f8c9fc37386] 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55738b6d46b9] 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55738b89280c] 2024-07-26T11:15:42.277 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55738b7d22ca] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55738b7a2753] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55738b7a3a1b] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55738b787752] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55738b6d5a56] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55738b6d9637] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55738b6d9f5e] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55738b6bd03a] 2024-07-26T11:15:42.278 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ca079b407] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ca079bcf9] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ca08785a1] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ca0625cdd] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ca0625cf9] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8c9fc89c02] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8c9fd0ec40] 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[103227]: 2024-07-26T11:15:42.279 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:42 smithi144 podman[108402]: 2024-07-26 11:15:42.266276239 +0000 UTC m=+0.033685768 container died ddd473e7ebdb817c101faa7ab65f3e8331518e3751f92f7a71c3a9b17672108f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T11:15:42.309 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:15:42 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:15:42.195+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:15:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:41 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:rejoin 2024-07-26T11:15:42.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:41 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:42.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:41 smithi160 ceph-mon[36715]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:15:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:41 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:rejoin 2024-07-26T11:15:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:41 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:41 smithi183 ceph-mon[37273]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:15:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:42 smithi160 ceph-mon[36715]: pgmap v901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:15:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:42 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:42 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:active 2024-07-26T11:15:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:42 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:43.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:42 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:15:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:42 smithi183 ceph-mon[37273]: pgmap v901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:15:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:42 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:42 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:active 2024-07-26T11:15:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:42 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:42 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:15:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-mon[32129]: pgmap v901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:15:43.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:15:43.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/1062414314,v1:172.21.15.144:6841/1062414314] up:active 2024-07-26T11:15:43.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:43.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:42 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:15:43.471 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:43 smithi144 podman[108402]: 2024-07-26 11:15:43.047527233 +0000 UTC m=+0.814936771 container remove ddd473e7ebdb817c101faa7ab65f3e8331518e3751f92f7a71c3a9b17672108f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-07-26T11:15:43.471 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:43 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:15:43.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:43 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:15:43.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:43 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.677s CPU time. 2024-07-26T11:15:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:45 smithi144 ceph-mon[32129]: pgmap v902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:15:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:45 smithi160 ceph-mon[36715]: pgmap v902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:15:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:45 smithi183 ceph-mon[37273]: pgmap v902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:15:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 589.538204 secs 2024-07-26T11:15:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 589.538204 secs 2024-07-26T11:15:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 589.538204 secs 2024-07-26T11:15:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:47 smithi160 ceph-mon[36715]: pgmap v903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:15:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:47 smithi183 ceph-mon[37273]: pgmap v903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:15:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:47 smithi144 ceph-mon[32129]: pgmap v903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:15:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:49 smithi160 ceph-mon[36715]: pgmap v904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 7 op/s 2024-07-26T11:15:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:49 smithi183 ceph-mon[37273]: pgmap v904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 7 op/s 2024-07-26T11:15:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:49 smithi144 ceph-mon[32129]: pgmap v904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 7 op/s 2024-07-26T11:15:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:51 smithi160 ceph-mon[36715]: pgmap v905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:15:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 594.538332 secs 2024-07-26T11:15:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:51 smithi183 ceph-mon[37273]: pgmap v905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:15:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 594.538332 secs 2024-07-26T11:15:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:51 smithi144 ceph-mon[32129]: pgmap v905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:15:51.497 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 594.538332 secs 2024-07-26T11:15:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:52.857 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: pgmap v906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:53.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:53.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: pgmap v906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:15:53.333 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: pgmap v906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:15:53.879 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:53 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:15:53.879 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:53 smithi144 systemd[1]: Stopped Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:53.879 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:53 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.677s CPU time. 2024-07-26T11:15:53.879 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:53 smithi144 systemd[1]: Starting Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:15:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:54 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:15:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:54 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:15:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:54 smithi144 ceph-mon[32129]: pgmap v907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:15:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:54 smithi144 ceph-mon[32129]: pgmap v908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 222 B/s wr, 6 op/s 2024-07-26T11:15:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:54 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:15:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:54 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:15:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:54 smithi160 ceph-mon[36715]: pgmap v907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:15:54.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:54 smithi160 ceph-mon[36715]: pgmap v908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 222 B/s wr, 6 op/s 2024-07-26T11:15:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:54 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:15:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:54 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:15:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:54 smithi183 ceph-mon[37273]: pgmap v907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:15:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:54 smithi183 ceph-mon[37273]: pgmap v908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 222 B/s wr, 6 op/s 2024-07-26T11:15:55.319 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 podman[108718]: 2024-07-26 11:15:54.956724516 +0000 UTC m=+0.032614321 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:15:55.320 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 podman[108718]: 2024-07-26 11:15:55.072583921 +0000 UTC m=+0.148473732 container create 117e53fbe5a808bc78bd5d89ffd2b1ce73638de522659e9442a1775df2466d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T11:15:55.592 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 podman[108718]: 2024-07-26 11:15:55.588671784 +0000 UTC m=+0.664561621 container init 117e53fbe5a808bc78bd5d89ffd2b1ce73638de522659e9442a1775df2466d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:15:55.866 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 podman[108718]: 2024-07-26 11:15:55.600506823 +0000 UTC m=+0.676396706 container start 117e53fbe5a808bc78bd5d89ffd2b1ce73638de522659e9442a1775df2466d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T11:15:55.867 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 bash[108718]: 117e53fbe5a808bc78bd5d89ffd2b1ce73638de522659e9442a1775df2466d53 2024-07-26T11:15:55.867 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 systemd[1]: Started Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:15:55.867 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 ceph-mds[108760]: -- 172.21.15.144:0/841101100 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff2ff61860 con 0x55ff34103600 2024-07-26T11:15:55.867 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:15:55 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: starting mds.a at 2024-07-26T11:15:56.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: pgmap v909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:15:56.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 599.538421 secs 2024-07-26T11:15:56.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: daemon mds.a restarted 2024-07-26T11:15:56.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:56.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: osdmap e127: 12 total, 12 up, 12 in 2024-07-26T11:15:56.116 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:boot 2024-07-26T11:15:56.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:15:56.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:15:56.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:15:56.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:56 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 -- 172.21.15.144:0/401158239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 msgr2=0x7f79d8107900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 --2- 172.21.15.144:0/401158239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 0x7f79d8107900 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009860 tx=0x7f79d002a340 comp rx=0 tx=0).stop 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 -- 172.21.15.144:0/401158239 shutdown_connections 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 --2- 172.21.15.144:0/401158239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 0x7f79d8107900 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 --2- 172.21.15.144:0/401158239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 --2- 172.21.15.144:0/401158239 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79d8101fe0 0x7f79d81023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 -- 172.21.15.144:0/401158239 >> 172.21.15.144:0/401158239 conn(0x7f79d80fdcf0 msgr2=0x7f79d8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.072+0000 7f79dca00640 1 -- 172.21.15.144:0/401158239 shutdown_connections 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.073+0000 7f79dca00640 1 -- 172.21.15.144:0/401158239 wait complete. 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.074+0000 7f79dca00640 1 Processor -- start 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.074+0000 7f79dca00640 1 -- start start 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79dca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79d8101fe0 0x7f79d81a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79dca00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d81a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.268 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79dca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 0x7f79d81a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79dca00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81198c0 con 0x7f79d8102930 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79dca00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f79d8119740 con 0x7f79d8101fe0 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79dca00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f79d8119a40 con 0x7f79d8107420 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79d5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d81a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79d5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d81a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51492/0 (socket says 172.21.15.144:51492) 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79d5e00640 1 -- 172.21.15.144:0/141136469 learned_addr learned my addr 172.21.15.144:0/141136469 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79d6800640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79d8101fe0 0x7f79d81a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.075+0000 7f79d7200640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 0x7f79d81a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79d5e00640 1 -- 172.21.15.144:0/141136469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 msgr2=0x7f79d81a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79d5e00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 0x7f79d81a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79d5e00640 1 -- 172.21.15.144:0/141136469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79d8101fe0 msgr2=0x7f79d81a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79d5e00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79d8101fe0 0x7f79d81a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79d5e00640 1 -- 172.21.15.144:0/141136469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d81a6200 con 0x7f79d8102930 2024-07-26T11:15:56.269 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79d5e00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d81a0830 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f79c000efc0 tx=0x7f79c000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79cf400640 1 -- 172.21.15.144:0/141136469 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0004b20 con 0x7f79d8102930 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d81a64f0 con 0x7f79d8102930 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79cf400640 1 -- 172.21.15.144:0/141136469 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c0004cc0 con 0x7f79d8102930 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79cf400640 1 -- 172.21.15.144:0/141136469 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0005480 con 0x7f79d8102930 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.076+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d81acc50 con 0x7f79d8102930 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 Processor -- start 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 -- start start 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb10fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb11510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb11a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564bbea87b90 con 0x564bbe7431b0 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564bbea77090 con 0x564bbe76c070 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.080+0000 7f79e4089740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564bbea7b0d0 con 0x564bbe74b390 2024-07-26T11:15:56.270 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79cf400640 1 -- 172.21.15.144:0/141136469 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f79c0005620 con 0x7f79d8102930 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb11510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb10fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb11510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51508/0 (socket says 172.21.15.144:51508) 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d6800640 1 -- 172.21.15.144:0/3898894026 learned_addr learned my addr 172.21.15.144:0/3898894026 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d7200640 1 -- 172.21.15.144:0/3898894026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 msgr2=0x564bbeb11a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d7200640 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb11a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d7200640 1 -- 172.21.15.144:0/3898894026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 msgr2=0x564bbeb11510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d7200640 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb11510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.081+0000 7f79d7200640 1 -- 172.21.15.144:0/3898894026 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bbe7d7650 con 0x564bbe76c070 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.082+0000 7f79d7200640 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb10fd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f79d002a930 tx=0x7f79d0005cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a45bba6e72ed2539 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.082+0000 7f79c7400640 1 -- 172.21.15.144:0/3898894026 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0037040 con 0x564bbe76c070 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.082+0000 7f79c7400640 1 -- 172.21.15.144:0/3898894026 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d0030e00 con 0x564bbe76c070 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.082+0000 7f79c7400640 1 -- 172.21.15.144:0/3898894026 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0003830 con 0x564bbe76c070 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79cf400640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79b80b3d40 0x7f79b80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.271 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79cf400640 1 -- 172.21.15.144:0/141136469 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 12576+0+0 (secure 0 0 0) 0x7f79c00e5170 con 0x7f79d8102930 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79d6800640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79b80b3d40 0x7f79b80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 -- 172.21.15.144:0/3898894026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 msgr2=0x564bbeb10fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb10fd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f79d002a930 tx=0x7f79d0005cb0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 -- 172.21.15.144:0/3898894026 shutdown_connections 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb11a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb11510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 --2- 172.21.15.144:0/3898894026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb10fd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.083+0000 7f79e4089740 1 -- 172.21.15.144:0/3898894026 >> 172.21.15.144:0/3898894026 conn(0x564bbe760460 msgr2=0x564bbeb0eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.084+0000 7f79e4089740 1 -- 172.21.15.144:0/3898894026 shutdown_connections 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.084+0000 7f79e4089740 1 -- 172.21.15.144:0/3898894026 wait complete. 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.084+0000 7f79d6800640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79b80b3d40 0x7f79b80b6250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x564bbeb0fc50 tx=0x7f79c8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.084+0000 7f79e4089740 1 -- start start 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 Processor -- start 2024-07-26T11:15:56.272 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb1c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb1cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb1d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x564bbe8da700 con 0x564bbe7431b0 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564bbea13660 con 0x564bbe76c070 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564bbeb1f750 con 0x564bbe74b390 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79e4089740 10 client.0 _ll_register_callbacks cb 0x564bbea23110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79d6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb1c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79d7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb1d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79d6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb1c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51520/0 (socket says 172.21.15.144:51520) 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79d6800640 1 -- 172.21.15.144:0/3024433748 learned_addr learned my addr 172.21.15.144:0/3024433748 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.085+0000 7f79d5e00640 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb1cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79d7200640 1 -- 172.21.15.144:0/3024433748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 msgr2=0x564bbeb1cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79d7200640 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb1cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.273 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79d7200640 1 -- 172.21.15.144:0/3024433748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 msgr2=0x564bbeb1c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79d7200640 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb1c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79d7200640 1 -- 172.21.15.144:0/3024433748 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bbea2a800 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79d7200640 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb1d080 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f79d0004650 tx=0x7f79d00309f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb913e498adf71ed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.086+0000 7f79bfe00640 1 -- 172.21.15.144:0/3024433748 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0037030 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79d0030e00 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79bfe00640 1 -- 172.21.15.144:0/3024433748 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 12576+0+0 (secure 0 0 0) 0x7f79d0005460 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79bfe00640 1 -- 172.21.15.144:0/3024433748 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=272) ==== 24+0+0 (secure 0 0 0) 0x7f79d0058050 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79e4089740 10 client.26540 fetch_fsmap learned FSMap version 272 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=272,monmap=4+}) -- 0x564bbeb2b330 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79bfe00640 1 -- 172.21.15.144:0/3024433748 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d0030e00 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.087+0000 7f79bfe00640 1 -- 172.21.15.144:0/3024433748 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 272) ==== 5333+0+0 (secure 0 0 0) 0x7f79d0052970 con 0x564bbe76c070 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79e4089740 10 client.26540 fetch_fsmap finished waiting for FSMap version 272 2024-07-26T11:15:56.274 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79e4089740 10 client.26540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79e4089740 20 client.26540 populate_metadata read hostname 'smithi144' 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79e4089740 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe72a9f0 0x564bbeb2e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79e4089740 4 client.26540 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bbea89ae0 con 0x564bbe72a9f0 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79d6800640 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe72a9f0 0x564bbeb2e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79d6800640 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe72a9f0 0x564bbeb2e940 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdfad988c7d6ccaa server_cookie=df3fe2f1b99ce571 in_seq=0 out_seq=0 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.088+0000 7f79bfe00640 10 client.26540 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.089+0000 7f79bfe00640 1 -- 172.21.15.144:0/3024433748 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564bbea89ae0 con 0x564bbe72a9f0 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.089+0000 7f79bfe00640 10 client.26540 handle_command_reply: tid=0 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.089+0000 7f79e4089740 1 client.26540 shutdown 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.089+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 msgr2=0x564bbeb1d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb1d080 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f79d0004650 tx=0x7f79d00309f0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 shutdown_connections 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe72a9f0 0x564bbeb2e940 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.275 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb1d080 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe74b390 0x564bbeb1cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79bfe00640 0 client.26540 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 --2- 172.21.15.144:0/3024433748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb1c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 >> 172.21.15.144:0/3024433748 conn(0x564bbe760460 msgr2=0x564bbeb14160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 shutdown_connections 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 1 -- 172.21.15.144:0/3024433748 wait complete. 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.090+0000 7f79e4089740 20 client.26540 trim_cache size 0 max 16384 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 Processor -- start 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 -- start start 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb0dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe74b390 0x564bbeb1e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe76c070 0x564bbeb34140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81b9ba0 con 0x564bbe7431b0 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81b8060 con 0x564bbe74b390 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.110+0000 7f79e4089740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81b94d0 con 0x564bbe76c070 2024-07-26T11:15:56.276 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb0dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d6800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb0dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51526/0 (socket says 172.21.15.144:51526) 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d6800640 1 -- 172.21.15.144:0/223040420 learned_addr learned my addr 172.21.15.144:0/223040420 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe74b390 0x564bbeb1e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d7200640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe76c070 0x564bbeb34140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 -- 172.21.15.144:0/223040420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe76c070 msgr2=0x564bbeb34140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe76c070 0x564bbeb34140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 -- 172.21.15.144:0/223040420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 msgr2=0x564bbeb0dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb0dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 -- 172.21.15.144:0/223040420 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bbeb1d8c0 con 0x564bbe74b390 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d7200640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe76c070 0x564bbeb34140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d6800640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb0dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.111+0000 7f79d5e00640 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe74b390 0x564bbeb1e230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f79c00e18f0 tx=0x7f79c00aa8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=610b6f80675e66c9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.112+0000 7f79c6a00640 1 -- 172.21.15.144:0/223040420 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c00af3b0 con 0x564bbe74b390 2024-07-26T11:15:56.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.112+0000 7f79c6a00640 1 -- 172.21.15.144:0/223040420 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c00b3e60 con 0x564bbe74b390 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.112+0000 7f79c6a00640 1 -- 172.21.15.144:0/223040420 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c00e7340 con 0x564bbe74b390 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.112+0000 7f79e4089740 1 -- 172.21.15.144:0/223040420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe74b390 msgr2=0x564bbeb1e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.112+0000 7f79e4089740 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe74b390 0x564bbeb1e230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f79c00e18f0 tx=0x7f79c00aa8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 -- 172.21.15.144:0/223040420 shutdown_connections 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe76c070 0x564bbeb34140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe74b390 0x564bbeb1e230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 --2- 172.21.15.144:0/223040420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe7431b0 0x564bbeb0dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 -- 172.21.15.144:0/223040420 >> 172.21.15.144:0/223040420 conn(0x564bbe760460 msgr2=0x564bbeb1db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 -- 172.21.15.144:0/223040420 shutdown_connections 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 -- 172.21.15.144:0/223040420 wait complete. 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.113+0000 7f79e4089740 1 -- start start 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 Processor -- start 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe7431b0 0x564bbeb21cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe74b390 0x564bbeb22230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.278 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb95110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81b8e00 con 0x564bbe74b390 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81b8730 con 0x564bbe76c070 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f79d81b6830 con 0x564bbe7431b0 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.114+0000 7f79e4089740 10 client.0 _ll_register_callbacks cb 0x7f79d00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe74b390 0x564bbeb22230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb95110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d6800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe7431b0 0x564bbeb21cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d5e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe74b390 0x564bbeb22230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51542/0 (socket says 172.21.15.144:51542) 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d5e00640 1 -- 172.21.15.144:0/2421026431 learned_addr learned my addr 172.21.15.144:0/2421026431 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb95110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:41330/0 (socket says 172.21.15.144:41330) 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 -- 172.21.15.144:0/2421026431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe7431b0 msgr2=0x564bbeb21cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe7431b0 0x564bbeb21cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 -- 172.21.15.144:0/2421026431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe74b390 msgr2=0x564bbeb22230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe74b390 0x564bbeb22230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 -- 172.21.15.144:0/2421026431 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bbeb95650 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d6800640 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe7431b0 0x564bbeb21cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79d7200640 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb95110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009780 tx=0x7f79d0030820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db5458240f7f57df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.115+0000 7f79bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.116+0000 7f79bfe00640 1 -- 172.21.15.144:0/2421026431 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0037040 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.116+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79c00b3e60 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.116+0000 7f79bfe00640 1 -- 172.21.15.144:0/2421026431 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 12576+0+0 (secure 0 0 0) 0x7f79d0005460 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.116+0000 7f79bfe00640 1 -- 172.21.15.144:0/2421026431 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=272) ==== 24+0+0 (secure 0 0 0) 0x7f79d003d7b0 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.116+0000 7f79e4089740 10 client.26546 fetch_fsmap learned FSMap version 272 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.116+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=272,monmap=4+}) -- 0x564bbeb967b0 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79bfe00640 1 -- 172.21.15.144:0/2421026431 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c00b3e60 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79bfe00640 1 -- 172.21.15.144:0/2421026431 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 272) ==== 5333+0+0 (secure 0 0 0) 0x7f79d005b020 con 0x564bbe76c070 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79e4089740 10 client.26546 fetch_fsmap finished waiting for FSMap version 272 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79e4089740 10 client.26546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79e4089740 20 client.26546 populate_metadata read hostname 'smithi144' 2024-07-26T11:15:56.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79e4089740 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe6ffda0 0x564bbeb98e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79e4089740 4 client.26546 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564bbea85580 con 0x564bbe6ffda0 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.117+0000 7f79d6800640 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe6ffda0 0x564bbeb98e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79d6800640 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe6ffda0 0x564bbeb98e00 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6167492202836c3 server_cookie=f0e69c6124b2a047 in_seq=0 out_seq=0 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79bfe00640 10 client.26546 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79bfe00640 1 -- 172.21.15.144:0/2421026431 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564bbea85580 con 0x564bbe6ffda0 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79bfe00640 10 client.26546 handle_command_reply: tid=0 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 client.26546 shutdown 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 msgr2=0x564bbeb95110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb95110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009780 tx=0x7f79d0030820 comp rx=0 tx=0).stop 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 shutdown_connections 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x564bbe6ffda0 0x564bbeb98e00 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x564bbe76c070 0x564bbeb95110 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.118+0000 7f79e4089740 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564bbe74b390 0x564bbeb22230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.119+0000 7f79bfe00640 0 client.26546 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.119+0000 7f79e4089740 1 --2- 172.21.15.144:0/2421026431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564bbe7431b0 0x564bbeb21cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.119+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 >> 172.21.15.144:0/2421026431 conn(0x564bbe760460 msgr2=0x564bbeb0dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.119+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 shutdown_connections 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.119+0000 7f79e4089740 1 -- 172.21.15.144:0/2421026431 wait complete. 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.119+0000 7f79e4089740 20 client.26546 trim_cache size 0 max 16384 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:15:56.282 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79b80b3d40 msgr2=0x7f79b80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79b80b3d40 0x7f79b80b6250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x564bbeb0fc50 tx=0x7f79c8005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 msgr2=0x7f79d81a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d81a0830 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f79c000efc0 tx=0x7f79c000c800 comp rx=0 tx=0).stop 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 shutdown_connections 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f79b80b3d40 0x7f79b80b6250 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d8107420 0x7f79d81a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79d8102930 0x7f79d81a0830 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 --2- 172.21.15.144:0/141136469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f79d8101fe0 0x7f79d81a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 >> 172.21.15.144:0/141136469 conn(0x7f79d80fdcf0 msgr2=0x7f79d8104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 shutdown_connections 2024-07-26T11:15:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:15:56.120+0000 7f79dca00640 1 -- 172.21.15.144:0/141136469 wait complete. 2024-07-26T11:15:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: pgmap v909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:15:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 599.538421 secs 2024-07-26T11:15:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: daemon mds.a restarted 2024-07-26T11:15:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: osdmap e127: 12 total, 12 up, 12 in 2024-07-26T11:15:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:boot 2024-07-26T11:15:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:15:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:15:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:15:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:56 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: pgmap v909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 599.538421 secs 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: daemon mds.a restarted 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: osdmap e127: 12 total, 12 up, 12 in 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:boot 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:15:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:15:56.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:15:56.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:56 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:57.090 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:57.087+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:57.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:57.088+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:57.091 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:57.088+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:57.089+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:15:57.089+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:57.196 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:15:58.327 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:58 smithi144 ceph-mon[32129]: pgmap v911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s 2024-07-26T11:15:58.328 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:58 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:resolve 2024-07-26T11:15:58.328 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:58 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:58 smithi183 ceph-mon[37273]: pgmap v911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s 2024-07-26T11:15:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:58 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:resolve 2024-07-26T11:15:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:58 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:58 smithi160 ceph-mon[36715]: pgmap v911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s 2024-07-26T11:15:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:58 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:resolve 2024-07-26T11:15:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:58 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:59 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:reconnect 2024-07-26T11:15:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:59 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:15:59 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:15:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:59 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:reconnect 2024-07-26T11:15:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:59 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:15:59 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:15:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:59 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:reconnect 2024-07-26T11:15:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:59 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:15:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:15:59 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:16:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:00 smithi183 ceph-mon[37273]: pgmap v912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T11:16:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:00 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:rejoin 2024-07-26T11:16:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:00 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:00 smithi183 ceph-mon[37273]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:16:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-mon[32129]: pgmap v912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T11:16:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:rejoin 2024-07-26T11:16:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-mon[32129]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: *** Caught signal (Aborted) ** 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: in thread 7fcb80000640 thread_name:ms_dispatch 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcb8643e6f0] 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcb8648b94c] 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 3: raise() 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 4: abort() 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcb8642871b] 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 6: /lib64/libc.so.6(+0x37386) [0x7fcb86437386] 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d31bd86b9] 2024-07-26T11:16:00.471 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d31d9680c] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d31cd62ca] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d31ca6753] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d31ca7a1b] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d31c8b752] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d31bd9a56] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d31bdd637] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d31bddf5e] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d31bc103a] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcb86f9b407] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 18: (DispatchQueue::entry()+0x60f) [0x7fcb86f9bcf9] 2024-07-26T11:16:00.472 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcb870785a1] 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 20: (Thread::entry_wrapper()+0x43) [0x7fcb86e25cdd] 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcb86e25cf9] 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcb86489c02] 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcb8650ec40] 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.175+0000 7fcb80000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: in thread 7fcb80000640 thread_name:ms_dispatch 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcb8643e6f0] 2024-07-26T11:16:00.473 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcb8648b94c] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 3: raise() 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 4: abort() 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcb8642871b] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 6: /lib64/libc.so.6(+0x37386) [0x7fcb86437386] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d31bd86b9] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d31d9680c] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d31cd62ca] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d31ca6753] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d31ca7a1b] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d31c8b752] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d31bd9a56] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d31bdd637] 2024-07-26T11:16:00.474 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d31bddf5e] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d31bc103a] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcb86f9b407] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 18: (DispatchQueue::entry()+0x60f) [0x7fcb86f9bcf9] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcb870785a1] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 20: (Thread::entry_wrapper()+0x43) [0x7fcb86e25cdd] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcb86e25cf9] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcb86489c02] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcb8650ec40] 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 0> 2024-07-26T11:16:00.175+0000 7fcb80000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: in thread 7fcb80000640 thread_name:ms_dispatch 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.475 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcb8643e6f0] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcb8648b94c] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 3: raise() 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 4: abort() 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcb8642871b] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 6: /lib64/libc.so.6(+0x37386) [0x7fcb86437386] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d31bd86b9] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d31d9680c] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d31cd62ca] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d31ca6753] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d31ca7a1b] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d31c8b752] 2024-07-26T11:16:00.476 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d31bd9a56] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d31bdd637] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d31bddf5e] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d31bc103a] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcb86f9b407] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 18: (DispatchQueue::entry()+0x60f) [0x7fcb86f9bcf9] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcb870785a1] 2024-07-26T11:16:00.477 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 20: (Thread::entry_wrapper()+0x43) [0x7fcb86e25cdd] 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcb86e25cf9] 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcb86489c02] 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcb8650ec40] 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 0> 2024-07-26T11:16:00.175+0000 7fcb80000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: in thread 7fcb80000640 thread_name:ms_dispatch 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fcb8643e6f0] 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fcb8648b94c] 2024-07-26T11:16:00.478 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 3: raise() 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 4: abort() 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fcb8642871b] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 6: /lib64/libc.so.6(+0x37386) [0x7fcb86437386] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d31bd86b9] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d31d9680c] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d31cd62ca] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d31ca6753] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d31ca7a1b] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d31c8b752] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d31bd9a56] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d31bdd637] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d31bddf5e] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d31bc103a] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fcb86f9b407] 2024-07-26T11:16:00.479 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 18: (DispatchQueue::entry()+0x60f) [0x7fcb86f9bcf9] 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fcb870785a1] 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 20: (Thread::entry_wrapper()+0x43) [0x7fcb86e25cdd] 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fcb86e25cf9] 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fcb86489c02] 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fcb8650ec40] 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108364]: 2024-07-26T11:16:00.480 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:00 smithi144 podman[108855]: 2024-07-26 11:16:00.330551835 +0000 UTC m=+0.037023225 container died d1bf2e0b51ea46ef9166fce988a361df6e888aaddda57711edb2ec662ad1d855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T11:16:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:00 smithi160 ceph-mon[36715]: pgmap v912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T11:16:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:00 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:rejoin 2024-07-26T11:16:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:00 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:00 smithi160 ceph-mon[36715]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:16:01.320 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:01 smithi144 podman[108855]: 2024-07-26 11:16:01.029343551 +0000 UTC m=+0.735814977 container remove d1bf2e0b51ea46ef9166fce988a361df6e888aaddda57711edb2ec662ad1d855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:16:01.320 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:01 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:16:01.320 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:01 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:01.320 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:01 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:active 2024-07-26T11:16:01.320 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:01 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:01.321 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:01 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:01.321 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 604.538535 secs 2024-07-26T11:16:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:01 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:01 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:active 2024-07-26T11:16:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:01 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:01 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 604.538535 secs 2024-07-26T11:16:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:01 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:01 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/2749174005,v1:172.21.15.144:6835/2749174005] up:active 2024-07-26T11:16:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:01 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:01 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:01.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 604.538535 secs 2024-07-26T11:16:01.970 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:01 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:16:01.970 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:01 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.483s CPU time. 2024-07-26T11:16:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:02 smithi183 ceph-mon[37273]: pgmap v913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:16:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:02 smithi144 ceph-mon[32129]: pgmap v913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:16:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:02 smithi160 ceph-mon[36715]: pgmap v913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-07-26T11:16:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:04 smithi144 ceph-mon[32129]: pgmap v914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:04 smithi183 ceph-mon[37273]: pgmap v914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:04 smithi160 ceph-mon[36715]: pgmap v914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:06 smithi183 ceph-mon[37273]: pgmap v915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 609.538636 secs 2024-07-26T11:16:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:06 smithi144 ceph-mon[32129]: pgmap v915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 609.538636 secs 2024-07-26T11:16:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:06 smithi160 ceph-mon[36715]: pgmap v915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 609.538636 secs 2024-07-26T11:16:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:08 smithi183 ceph-mon[37273]: pgmap v916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:16:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:08 smithi144 ceph-mon[32129]: pgmap v916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:16:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:08 smithi160 ceph-mon[36715]: pgmap v916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 182 B/s wr, 6 op/s 2024-07-26T11:16:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:10 smithi144 ceph-mon[32129]: pgmap v917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:10 smithi160 ceph-mon[36715]: pgmap v917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:10 smithi183 ceph-mon[37273]: pgmap v917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 614.538760 secs 2024-07-26T11:16:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 614.538760 secs 2024-07-26T11:16:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 614.538760 secs 2024-07-26T11:16:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:12.091+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:12.092+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:12.092+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:12.092+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:12.092+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:12.129 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:11 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:16:12.129 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:11 smithi144 systemd[1]: Stopped Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:16:12.129 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:11 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.483s CPU time. 2024-07-26T11:16:12.129 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:11 smithi144 systemd[1]: Starting Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:16:12.391 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:12 smithi144 ceph-mon[32129]: pgmap v918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:16:12.391 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 podman[108957]: 2024-07-26 11:16:12.032118911 +0000 UTC m=+0.033383423 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:16:12.391 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 podman[108957]: 2024-07-26 11:16:12.165419084 +0000 UTC m=+0.166683616 container create d7e3168446e5b350b91a02c1b8df83aed0dcb7701735c206fb41a3076d179449 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T11:16:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:12 smithi160 ceph-mon[36715]: pgmap v918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:16:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:12 smithi183 ceph-mon[37273]: pgmap v918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:16:12.970 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 podman[108957]: 2024-07-26 11:16:12.48607004 +0000 UTC m=+0.487334578 container init d7e3168446e5b350b91a02c1b8df83aed0dcb7701735c206fb41a3076d179449 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-26T11:16:12.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 podman[108957]: 2024-07-26 11:16:12.496407211 +0000 UTC m=+0.497671750 container start d7e3168446e5b350b91a02c1b8df83aed0dcb7701735c206fb41a3076d179449 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T11:16:12.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 bash[108957]: d7e3168446e5b350b91a02c1b8df83aed0dcb7701735c206fb41a3076d179449 2024-07-26T11:16:12.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 systemd[1]: Started Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:16:12.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 ceph-mds[108975]: -- 172.21.15.144:0/2960536970 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559216f8f860 con 0x55921b131180 2024-07-26T11:16:12.971 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:12 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: starting mds.j at 2024-07-26T11:16:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: daemon mds.j restarted 2024-07-26T11:16:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: osdmap e128: 12 total, 12 up, 12 in 2024-07-26T11:16:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:boot 2024-07-26T11:16:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:16:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:16:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:16:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:13 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: daemon mds.j restarted 2024-07-26T11:16:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: osdmap e128: 12 total, 12 up, 12 in 2024-07-26T11:16:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:boot 2024-07-26T11:16:13.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:16:13.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:16:13.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:16:13.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:13 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: daemon mds.j restarted 2024-07-26T11:16:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: osdmap e128: 12 total, 12 up, 12 in 2024-07-26T11:16:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:boot 2024-07-26T11:16:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:16:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:16:13.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:16:13.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:13 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:14 smithi144 ceph-mon[32129]: pgmap v920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:16:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:14 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:resolve 2024-07-26T11:16:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:14 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:14 smithi160 ceph-mon[36715]: pgmap v920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:16:14.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:14 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:resolve 2024-07-26T11:16:14.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:14 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:14 smithi183 ceph-mon[37273]: pgmap v920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6 op/s 2024-07-26T11:16:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:14 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:resolve 2024-07-26T11:16:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:14 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:reconnect 2024-07-26T11:16:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:15 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:15 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999984 2024-07-26T11:16:15.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:15 smithi144 ceph-mon[32129]: pgmap v921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:16:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:reconnect 2024-07-26T11:16:16.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:15 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:16.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:15 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999984 2024-07-26T11:16:16.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:15 smithi160 ceph-mon[36715]: pgmap v921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:16:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:reconnect 2024-07-26T11:16:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:15 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:15 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999984 2024-07-26T11:16:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:15 smithi183 ceph-mon[37273]: pgmap v921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 619.538882 secs 2024-07-26T11:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:rejoin 2024-07-26T11:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-mon[32129]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:16:16.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:16:16.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: *** Caught signal (Aborted) ** 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: in thread 7f215f600640 thread_name:ms_dispatch 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2165a3e6f0] 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2165a8b94c] 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 3: raise() 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 4: abort() 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2165a2871b] 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 6: /lib64/libc.so.6(+0x37386) [0x7f2165a37386] 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fef43106b9] 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fef44ce80c] 2024-07-26T11:16:16.972 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fef440e2ca] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fef43de753] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fef43dfa1b] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fef43c3752] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fef4311a56] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fef4315637] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fef4315f5e] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fef42f903a] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f216659b407] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 18: (DispatchQueue::entry()+0x60f) [0x7f216659bcf9] 2024-07-26T11:16:16.973 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21666785a1] 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 20: (Thread::entry_wrapper()+0x43) [0x7f2166425cdd] 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2166425cf9] 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2165a89c02] 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2165b0ec40] 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.709+0000 7f215f600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: in thread 7f215f600640 thread_name:ms_dispatch 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:16.974 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2165a3e6f0] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2165a8b94c] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 3: raise() 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 4: abort() 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2165a2871b] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 6: /lib64/libc.so.6(+0x37386) [0x7f2165a37386] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fef43106b9] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fef44ce80c] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fef440e2ca] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fef43de753] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fef43dfa1b] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fef43c3752] 2024-07-26T11:16:16.975 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fef4311a56] 2024-07-26T11:16:16.976 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fef4315637] 2024-07-26T11:16:16.976 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fef4315f5e] 2024-07-26T11:16:16.976 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fef42f903a] 2024-07-26T11:16:16.976 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f216659b407] 2024-07-26T11:16:16.976 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 18: (DispatchQueue::entry()+0x60f) [0x7f216659bcf9] 2024-07-26T11:16:16.976 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21666785a1] 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 20: (Thread::entry_wrapper()+0x43) [0x7f2166425cdd] 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2166425cf9] 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2165a89c02] 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2165b0ec40] 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 0> 2024-07-26T11:16:16.709+0000 7f215f600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: in thread 7f215f600640 thread_name:ms_dispatch 2024-07-26T11:16:16.977 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2165a3e6f0] 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2165a8b94c] 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 3: raise() 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 4: abort() 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2165a2871b] 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 6: /lib64/libc.so.6(+0x37386) [0x7f2165a37386] 2024-07-26T11:16:16.978 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fef43106b9] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fef44ce80c] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fef440e2ca] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fef43de753] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fef43dfa1b] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fef43c3752] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fef4311a56] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fef4315637] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fef4315f5e] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fef42f903a] 2024-07-26T11:16:16.979 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f216659b407] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 18: (DispatchQueue::entry()+0x60f) [0x7f216659bcf9] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21666785a1] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 20: (Thread::entry_wrapper()+0x43) [0x7f2166425cdd] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2166425cf9] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2165a89c02] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2165b0ec40] 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: -3> 2024-07-26T11:16:16.709+0000 7f215f600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: in thread 7f215f600640 thread_name:ms_dispatch 2024-07-26T11:16:16.980 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2165a3e6f0] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2165a8b94c] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 3: raise() 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 4: abort() 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2165a2871b] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 6: /lib64/libc.so.6(+0x37386) [0x7f2165a37386] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fef43106b9] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fef44ce80c] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fef440e2ca] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fef43de753] 2024-07-26T11:16:16.981 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fef43dfa1b] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fef43c3752] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fef4311a56] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fef4315637] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fef4315f5e] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fef42f903a] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f216659b407] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 18: (DispatchQueue::entry()+0x60f) [0x7f216659bcf9] 2024-07-26T11:16:16.982 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21666785a1] 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 20: (Thread::entry_wrapper()+0x43) [0x7f2166425cdd] 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2166425cf9] 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2165a89c02] 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2165b0ec40] 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[108749]: 2024-07-26T11:16:16.983 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:16 smithi144 podman[109009]: 2024-07-26 11:16:16.863842916 +0000 UTC m=+0.044714914 container died 117e53fbe5a808bc78bd5d89ffd2b1ce73638de522659e9442a1775df2466d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 619.538882 secs 2024-07-26T11:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:16 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:rejoin 2024-07-26T11:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:16 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:17.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:16 smithi160 ceph-mon[36715]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:16:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 619.538882 secs 2024-07-26T11:16:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:16 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:rejoin 2024-07-26T11:16:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:16 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:16 smithi183 ceph-mon[37273]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:16:17.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:16:17 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:16:17.197+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:16:17.954 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:17 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:17.954 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:17 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:active 2024-07-26T11:16:17.955 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:17 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:17.955 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:17 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:17.955 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:17 smithi144 ceph-mon[32129]: pgmap v922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:17.955 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:17 smithi144 podman[109009]: 2024-07-26 11:16:17.723062212 +0000 UTC m=+0.903934243 container remove 117e53fbe5a808bc78bd5d89ffd2b1ce73638de522659e9442a1775df2466d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T11:16:17.955 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:17 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:16:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:17 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:17 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:active 2024-07-26T11:16:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:17 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:17 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:18.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:17 smithi160 ceph-mon[36715]: pgmap v922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:17 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:17 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3840683228,v1:172.21.15.144:6841/3840683228] up:active 2024-07-26T11:16:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:17 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:17 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:17 smithi183 ceph-mon[37273]: pgmap v922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:18.720 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:18 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:16:18.720 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:18 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.457s CPU time. 2024-07-26T11:16:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:20 smithi144 ceph-mon[32129]: pgmap v923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:16:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:20 smithi160 ceph-mon[36715]: pgmap v923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:16:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:20 smithi183 ceph-mon[37273]: pgmap v923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 6 op/s 2024-07-26T11:16:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 624.539023 secs 2024-07-26T11:16:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 624.539023 secs 2024-07-26T11:16:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 624.539023 secs 2024-07-26T11:16:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:22 smithi144 ceph-mon[32129]: pgmap v924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:22 smithi160 ceph-mon[36715]: pgmap v924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:22 smithi183 ceph-mon[37273]: pgmap v924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:24 smithi183 ceph-mon[37273]: pgmap v925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 197 B/s wr, 7 op/s 2024-07-26T11:16:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:24 smithi144 ceph-mon[32129]: pgmap v925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 197 B/s wr, 7 op/s 2024-07-26T11:16:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:24 smithi160 ceph-mon[36715]: pgmap v925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 197 B/s wr, 7 op/s 2024-07-26T11:16:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:26 smithi144 ceph-mon[32129]: pgmap v926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 629.539136 secs 2024-07-26T11:16:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:26 smithi183 ceph-mon[37273]: pgmap v926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 629.539136 secs 2024-07-26T11:16:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:26 smithi160 ceph-mon[36715]: pgmap v926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 629.539136 secs 2024-07-26T11:16:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:27.108+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:27.109+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:27.109+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:27.109+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:27.110+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:27.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:16:27.681 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:28 smithi183 ceph-mon[37273]: pgmap v927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:28 smithi144 ceph-mon[32129]: pgmap v927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:28 smithi160 ceph-mon[36715]: pgmap v927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:16:28.833 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:28 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:16:28.834 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:28 smithi144 systemd[1]: Stopped Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:16:28.834 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:28 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.457s CPU time. 2024-07-26T11:16:28.834 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:28 smithi144 systemd[1]: Starting Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:16:29.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:29 smithi144 podman[109211]: 2024-07-26 11:16:29.621217343 +0000 UTC m=+0.272649160 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:16:29.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:29 smithi144 podman[109211]: 2024-07-26 11:16:29.783568685 +0000 UTC m=+0.435000498 container create afef44a4a186c240ec0e0166a629d55d53cc26b09b72ea3a6fcf6af41dcedf7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, CEPH_POINT_RELEASE=) 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 -- 172.21.15.144:0/3260550787 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884108f20 msgr2=0x7f5884109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 --2- 172.21.15.144:0/3260550787 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884108f20 0x7f5884109330 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5870009900 tx=0x7f587002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 -- 172.21.15.144:0/3260550787 shutdown_connections 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 --2- 172.21.15.144:0/3260550787 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5884109870 0x7f58841100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 --2- 172.21.15.144:0/3260550787 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884108f20 0x7f5884109330 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 --2- 172.21.15.144:0/3260550787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f58841039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.096+0000 7f5888e00640 1 -- 172.21.15.144:0/3260550787 >> 172.21.15.144:0/3260550787 conn(0x7f58840ff030 msgr2=0x7f5884101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:30.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.097+0000 7f5888e00640 1 -- 172.21.15.144:0/3260550787 shutdown_connections 2024-07-26T11:16:30.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.097+0000 7f5888e00640 1 -- 172.21.15.144:0/3260550787 wait complete. 2024-07-26T11:16:30.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.098+0000 7f5888e00640 1 Processor -- start 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.098+0000 7f5888e00640 1 -- start start 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5888e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f588419c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5888e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5884108f20 0x7f588419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5888e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884109870 0x7f58841a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5888e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f5884113500 con 0x7f58841035e0 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5888e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5884113380 con 0x7f5884108f20 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5888e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5884113680 con 0x7f5884109870 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5881e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f588419c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5881e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f588419c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45112/0 (socket says 172.21.15.144:45112) 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5881e00640 1 -- 172.21.15.144:0/516589297 learned_addr learned my addr 172.21.15.144:0/516589297 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:16:30.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5882800640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884109870 0x7f58841a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5881400640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5884108f20 0x7f588419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.099+0000 7f5881e00640 1 -- 172.21.15.144:0/516589297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884109870 msgr2=0x7f58841a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5881e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884109870 0x7f58841a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5881e00640 1 -- 172.21.15.144:0/516589297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5884108f20 msgr2=0x7f588419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5881e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5884108f20 0x7f588419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5881e00640 1 -- 172.21.15.144:0/516589297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58841a2080 con 0x7f58841035e0 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5882800640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884109870 0x7f58841a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:30.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5881e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f588419c110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f587400e9f0 tx=0x7f587400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:30.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f587aa00640 1 -- 172.21.15.144:0/516589297 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587400cd80 con 0x7f58841035e0 2024-07-26T11:16:30.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f587aa00640 1 -- 172.21.15.144:0/516589297 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58740108e0 con 0x7f58841035e0 2024-07-26T11:16:30.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58841a2370 con 0x7f58841035e0 2024-07-26T11:16:30.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f587aa00640 1 -- 172.21.15.144:0/516589297 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874018980 con 0x7f58841035e0 2024-07-26T11:16:30.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.100+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58841a8ad0 con 0x7f58841035e0 2024-07-26T11:16:30.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.104+0000 7f588fa39740 1 Processor -- start 2024-07-26T11:16:30.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.104+0000 7f587aa00640 1 -- 172.21.15.144:0/516589297 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5874010430 con 0x7f58841035e0 2024-07-26T11:16:30.106 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.104+0000 7f588fa39740 1 -- start start 2024-07-26T11:16:30.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f588fa39740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77256010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.108 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f588fa39740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77256550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f588fa39740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c77256a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.109 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f588fa39740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563c771ccbc0 con 0x563c76e90390 2024-07-26T11:16:30.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f588fa39740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563c771bc0c0 con 0x563c76eb1070 2024-07-26T11:16:30.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f588fa39740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c771c0100 con 0x563c76e881b0 2024-07-26T11:16:30.110 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5881400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c77256a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5881400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c77256a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45122/0 (socket says 172.21.15.144:45122) 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5881400640 1 -- 172.21.15.144:0/2748179596 learned_addr learned my addr 172.21.15.144:0/2748179596 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5881e00640 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77256550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5882800640 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77256010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5881400640 1 -- 172.21.15.144:0/2748179596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 msgr2=0x563c77256550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.105+0000 7f5881400640 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77256550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f5881400640 1 -- 172.21.15.144:0/2748179596 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 msgr2=0x563c77256010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f5881400640 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77256010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f5881400640 1 -- 172.21.15.144:0/2748179596 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c76f1c650 con 0x563c76e90390 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f5881e00640 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77256550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f5881400640 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c77256a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5870009900 tx=0x7f5870000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6b7665d2ac928 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f586f400640 1 -- 172.21.15.144:0/2748179596 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870040070 con 0x563c76e90390 2024-07-26T11:16:30.111 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f586f400640 1 -- 172.21.15.144:0/2748179596 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5870004030 con 0x563c76e90390 2024-07-26T11:16:30.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f586f400640 1 -- 172.21.15.144:0/2748179596 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870035040 con 0x563c76e90390 2024-07-26T11:16:30.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.106+0000 7f587aa00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58680b3d40 0x7f58680b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.107+0000 7f587aa00640 1 -- 172.21.15.144:0/516589297 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 12662+0+0 (secure 0 0 0) 0x7f58740e53a0 con 0x7f58841035e0 2024-07-26T11:16:30.112 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.107+0000 7f5882800640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58680b3d40 0x7f58680b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.107+0000 7f588fa39740 1 -- 172.21.15.144:0/2748179596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 msgr2=0x563c77256a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.107+0000 7f588fa39740 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c77256a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5870009900 tx=0x7f5870000bc0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f5882800640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58680b3d40 0x7f58680b6250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f587c009fd0 tx=0x7f587c009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:30.115 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 -- 172.21.15.144:0/2748179596 shutdown_connections 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c77256a90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77256550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 --2- 172.21.15.144:0/2748179596 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77256010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 -- 172.21.15.144:0/2748179596 >> 172.21.15.144:0/2748179596 conn(0x563c76ea5460 msgr2=0x563c77253f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 -- 172.21.15.144:0/2748179596 shutdown_connections 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 -- 172.21.15.144:0/2748179596 wait complete. 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.108+0000 7f588fa39740 1 -- start start 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 Processor -- start 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77269db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.116 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c7726a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x563c7701f700 con 0x563c76e90390 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563c77158690 con 0x563c76eb1070 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c7726cec0 con 0x563c76e881b0 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.109+0000 7f588fa39740 10 client.0 _ll_register_callbacks cb 0x563c77168140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c7726a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5881400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c7726a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36848/0 (socket says 172.21.15.144:36848) 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 -- 172.21.15.144:0/3857222079 learned_addr learned my addr 172.21.15.144:0/3857222079 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5881e00640 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77269db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 -- 172.21.15.144:0/3857222079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 msgr2=0x563c77269db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77269db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 -- 172.21.15.144:0/3857222079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 msgr2=0x563c7726a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.117 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 -- 172.21.15.144:0/3857222079 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c7716f830 con 0x563c76eb1070 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5882800640 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c7726a830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f587c003e20 tx=0x7f587c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ad43a67fb3dcfe3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.110+0000 7f5863e00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.111+0000 7f5863e00640 1 -- 172.21.15.144:0/3857222079 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c004550 con 0x563c76eb1070 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.111+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5870004030 con 0x563c76eb1070 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.111+0000 7f5863e00640 1 -- 172.21.15.144:0/3857222079 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 12662+0+0 (secure 0 0 0) 0x7f587c018030 con 0x563c76eb1070 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.111+0000 7f5863e00640 1 -- 172.21.15.144:0/3857222079 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=284) ==== 24+0+0 (secure 0 0 0) 0x7f587c0046f0 con 0x563c76eb1070 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.111+0000 7f588fa39740 10 client.26564 fetch_fsmap learned FSMap version 284 2024-07-26T11:16:30.118 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.111+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=284,monmap=4+}) -- 0x563c77278bf0 con 0x563c76eb1070 2024-07-26T11:16:30.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f5863e00640 1 -- 172.21.15.144:0/3857222079 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5870004030 con 0x563c76eb1070 2024-07-26T11:16:30.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f5863e00640 1 -- 172.21.15.144:0/3857222079 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 284) ==== 5329+0+0 (secure 0 0 0) 0x7f587c031430 con 0x563c76eb1070 2024-07-26T11:16:30.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f588fa39740 10 client.26564 fetch_fsmap finished waiting for FSMap version 284 2024-07-26T11:16:30.119 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f588fa39740 10 client.26564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:16:30.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f588fa39740 20 client.26564 populate_metadata read hostname 'smithi144' 2024-07-26T11:16:30.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f588fa39740 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e6f9f0 0x563c7727c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f588fa39740 4 client.26564 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:30.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c771ceb10 con 0x563c76e6f9f0 2024-07-26T11:16:30.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.112+0000 7f5881e00640 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e6f9f0 0x563c7727c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.120 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.113+0000 7f5881e00640 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e6f9f0 0x563c7727c4d0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7fa62558eeab08 server_cookie=b500ad9cab0c0b97 in_seq=0 out_seq=0 2024-07-26T11:16:30.121 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.113+0000 7f5863e00640 10 client.26564 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:16:30.122 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f5863e00640 1 -- 172.21.15.144:0/3857222079 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c771ceb10 con 0x563c76e6f9f0 2024-07-26T11:16:30.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f5863e00640 10 client.26564 handle_command_reply: tid=0 2024-07-26T11:16:30.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 client.26564 shutdown 2024-07-26T11:16:30.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 msgr2=0x563c7726a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c7726a830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f587c003e20 tx=0x7f587c004ab0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 shutdown_connections 2024-07-26T11:16:30.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e6f9f0 0x563c7727c4d0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c7726a830 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 --2- 172.21.15.144:0/3857222079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e881b0 0x563c77269db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 >> 172.21.15.144:0/3857222079 conn(0x563c76ea5460 msgr2=0x563c772525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:30.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.114+0000 7f5863e00640 0 client.26564 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:16:30.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.115+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 shutdown_connections 2024-07-26T11:16:30.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.115+0000 7f588fa39740 1 -- 172.21.15.144:0/3857222079 wait complete. 2024-07-26T11:16:30.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.115+0000 7f588fa39740 20 client.26564 trim_cache size 0 max 16384 2024-07-26T11:16:30.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.133+0000 7f588fa39740 1 Processor -- start 2024-07-26T11:16:30.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.133+0000 7f588fa39740 1 -- start start 2024-07-26T11:16:30.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f588fa39740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 0x563c77265510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f588fa39740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e90390 0x563c7726b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f588fa39740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77281cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f588fa39740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f58841b15b0 con 0x563c76e881b0 2024-07-26T11:16:30.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f588fa39740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f58841af3a0 con 0x563c76eb1070 2024-07-26T11:16:30.136 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f588fa39740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f58841b0ee0 con 0x563c76e90390 2024-07-26T11:16:30.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f5881e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 0x563c77265510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f5881e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 0x563c77265510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:45136/0 (socket says 172.21.15.144:45136) 2024-07-26T11:16:30.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f5881e00640 1 -- 172.21.15.144:0/3631192469 learned_addr learned my addr 172.21.15.144:0/3631192469 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:16:30.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f5882800640 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77281cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.134+0000 7f5881400640 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e90390 0x563c7726b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5882800640 1 -- 172.21.15.144:0/3631192469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e90390 msgr2=0x563c7726b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5882800640 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e90390 0x563c7726b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5882800640 1 -- 172.21.15.144:0/3631192469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 msgr2=0x563c77265510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5882800640 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 0x563c77265510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5882800640 1 -- 172.21.15.144:0/3631192469 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c77257f50 con 0x563c76eb1070 2024-07-26T11:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5881e00640 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 0x563c77265510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f5882800640 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77281cd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f587c031a90 tx=0x7f587c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb6f797fd969002a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f586ea00640 1 -- 172.21.15.144:0/3631192469 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c004420 con 0x563c76eb1070 2024-07-26T11:16:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f586ea00640 1 -- 172.21.15.144:0/3631192469 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f587c031b10 con 0x563c76eb1070 2024-07-26T11:16:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.135+0000 7f586ea00640 1 -- 172.21.15.144:0/3631192469 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c005280 con 0x563c76eb1070 2024-07-26T11:16:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.136+0000 7f588fa39740 1 -- 172.21.15.144:0/3631192469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 msgr2=0x563c77281cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.136+0000 7f588fa39740 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77281cd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f587c031a90 tx=0x7f587c004a40 comp rx=0 tx=0).stop 2024-07-26T11:16:30.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.136+0000 7f588fa39740 1 -- 172.21.15.144:0/3631192469 shutdown_connections 2024-07-26T11:16:30.141 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.136+0000 7f588fa39740 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76eb1070 0x563c77281cd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.136+0000 7f588fa39740 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76e90390 0x563c7726b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.136+0000 7f588fa39740 1 --2- 172.21.15.144:0/3631192469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e881b0 0x563c77265510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.137+0000 7f588fa39740 1 -- 172.21.15.144:0/3631192469 >> 172.21.15.144:0/3631192469 conn(0x563c76ea5460 msgr2=0x563c7726b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.137+0000 7f588fa39740 1 -- 172.21.15.144:0/3631192469 shutdown_connections 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.137+0000 7f588fa39740 1 -- 172.21.15.144:0/3631192469 wait complete. 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.137+0000 7f588fa39740 1 -- start start 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.137+0000 7f588fa39740 1 Processor -- start 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 0x563c77272750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76eb1070 0x563c7726e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.142 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f58841b0140 con 0x563c76e90390 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f58841afa70 con 0x563c76e881b0 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f58841b0810 con 0x563c76eb1070 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f588fa39740 10 client.0 _ll_register_callbacks cb 0x7f587c003ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f5881e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 0x563c77272750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f5881400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f5881e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 0x563c77272750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36860/0 (socket says 172.21.15.144:36860) 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f5882800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76eb1070 0x563c7726e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.138+0000 7f5881e00640 1 -- 172.21.15.144:0/1496212263 learned_addr learned my addr 172.21.15.144:0/1496212263 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:16:30.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5881e00640 1 -- 172.21.15.144:0/1496212263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76eb1070 msgr2=0x563c7726e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5881e00640 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76eb1070 0x563c7726e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5881e00640 1 -- 172.21.15.144:0/1496212263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 msgr2=0x563c7726dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5881e00640 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5881e00640 1 -- 172.21.15.144:0/1496212263 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c7726e7b0 con 0x563c76e881b0 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5881e00640 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 0x563c77272750 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f58740aacd0 tx=0x7f58740e8dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb86061c49ec6fd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5863e00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5863e00640 1 -- 172.21.15.144:0/1496212263 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874010720 con 0x563c76e881b0 2024-07-26T11:16:30.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f5863e00640 1 -- 172.21.15.144:0/1496212263 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 12662+0+0 (secure 0 0 0) 0x7f58740e1ce0 con 0x563c76e881b0 2024-07-26T11:16:30.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.139+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f587c031b10 con 0x563c76e881b0 2024-07-26T11:16:30.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f5863e00640 1 -- 172.21.15.144:0/1496212263 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=284) ==== 24+0+0 (secure 0 0 0) 0x7f58740e1ee0 con 0x563c76e881b0 2024-07-26T11:16:30.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 10 client.26570 fetch_fsmap learned FSMap version 284 2024-07-26T11:16:30.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=284,monmap=4+}) -- 0x563c772e4b40 con 0x563c76e881b0 2024-07-26T11:16:30.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f5863e00640 1 -- 172.21.15.144:0/1496212263 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58740be070 con 0x563c76e881b0 2024-07-26T11:16:30.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f5863e00640 1 -- 172.21.15.144:0/1496212263 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 284) ==== 5329+0+0 (secure 0 0 0) 0x7f5874018b20 con 0x563c76e881b0 2024-07-26T11:16:30.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 10 client.26570 fetch_fsmap finished waiting for FSMap version 284 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 10 client.26570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 20 client.26570 populate_metadata read hostname 'smithi144' 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e44da0 0x563c772e7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 4 client.26570 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c771ca5b0 con 0x563c76e44da0 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.140+0000 7f5881400640 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e44da0 0x563c772e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.141+0000 7f5881400640 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e44da0 0x563c772e7330 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b44f394caccc27 server_cookie=78bac0a67692a7b6 in_seq=0 out_seq=0 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.141+0000 7f5863e00640 10 client.26570 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.141+0000 7f5863e00640 1 -- 172.21.15.144:0/1496212263 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563c771ca5b0 con 0x563c76e44da0 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.141+0000 7f5863e00640 10 client.26570 handle_command_reply: tid=0 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.141+0000 7f588fa39740 1 client.26570 shutdown 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 msgr2=0x563c77272750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 0x563c77272750 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f58740aacd0 tx=0x7f58740e8dd0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 shutdown_connections 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c76eb1070 0x563c7726e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563c76e90390 0x563c7726dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x563c76e44da0 0x563c772e7330 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 --2- 172.21.15.144:0/1496212263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563c76e881b0 0x563c77272750 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 >> 172.21.15.144:0/1496212263 conn(0x563c76ea5460 msgr2=0x563c772652e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f5863e00640 0 client.26570 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 shutdown_connections 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 1 -- 172.21.15.144:0/1496212263 wait complete. 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.142+0000 7f588fa39740 20 client.26570 trim_cache size 0 max 16384 2024-07-26T11:16:30.148 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:16:30.149 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58680b3d40 msgr2=0x7f58680b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58680b3d40 0x7f58680b6250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f587c009fd0 tx=0x7f587c009500 comp rx=0 tx=0).stop 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 msgr2=0x7f588419c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f588419c110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f587400e9f0 tx=0x7f587400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 shutdown_connections 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f58680b3d40 0x7f58680b6250 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5884109870 0x7f58841a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5884108f20 0x7f588419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 --2- 172.21.15.144:0/516589297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58841035e0 0x7f588419c110 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:30.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 >> 172.21.15.144:0/516589297 conn(0x7f58840ff030 msgr2=0x7f588410e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:30.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 shutdown_connections 2024-07-26T11:16:30.151 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:16:30.143+0000 7f5888e00640 1 -- 172.21.15.144:0/516589297 wait complete. 2024-07-26T11:16:30.347 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:30 smithi144 ceph-mon[32129]: pgmap v928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:16:30.347 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:30 smithi144 podman[109211]: 2024-07-26 11:16:30.204641156 +0000 UTC m=+0.856072934 container init afef44a4a186c240ec0e0166a629d55d53cc26b09b72ea3a6fcf6af41dcedf7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:16:30.347 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:30 smithi144 podman[109211]: 2024-07-26 11:16:30.212879543 +0000 UTC m=+0.864311328 container start afef44a4a186c240ec0e0166a629d55d53cc26b09b72ea3a6fcf6af41dcedf7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T11:16:30.347 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:30 smithi144 bash[109211]: afef44a4a186c240ec0e0166a629d55d53cc26b09b72ea3a6fcf6af41dcedf7d 2024-07-26T11:16:30.347 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:30 smithi144 systemd[1]: Started Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:16:30.348 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:30 smithi144 ceph-mds[109294]: -- 172.21.15.144:0/457116112 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565256631860 con 0x56525a7d3600 2024-07-26T11:16:30.348 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:30 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: starting mds.a at 2024-07-26T11:16:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:30 smithi183 ceph-mon[37273]: pgmap v928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:16:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:30 smithi160 ceph-mon[36715]: pgmap v928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:16:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: daemon mds.a restarted 2024-07-26T11:16:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: osdmap e129: 12 total, 12 up, 12 in 2024-07-26T11:16:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:boot 2024-07-26T11:16:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:16:31.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:16:31.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:16:31.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:31.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 634.539275 secs 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: daemon mds.a restarted 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: osdmap e129: 12 total, 12 up, 12 in 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:boot 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:16:31.397 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:16:31.398 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:31.398 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 634.539275 secs 2024-07-26T11:16:31.455 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:16:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: daemon mds.a restarted 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: osdmap e129: 12 total, 12 up, 12 in 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:boot 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:replay,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 634.539275 secs 2024-07-26T11:16:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:32 smithi183 ceph-mon[37273]: pgmap v930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:16:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:32 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:resolve 2024-07-26T11:16:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:32 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:32 smithi144 ceph-mon[32129]: pgmap v930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:16:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:32 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:resolve 2024-07-26T11:16:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:32 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:32 smithi160 ceph-mon[36715]: pgmap v930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:16:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:32 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:resolve 2024-07-26T11:16:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:32 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:resolve,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:34.383 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:34 smithi144 ceph-mon[32129]: pgmap v931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:34.383 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:34 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:reconnect 2024-07-26T11:16:34.383 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:34 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:34.383 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:34 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00199997 2024-07-26T11:16:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:34 smithi183 ceph-mon[37273]: pgmap v931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:34 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:reconnect 2024-07-26T11:16:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:34 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:34 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00199997 2024-07-26T11:16:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:34 smithi160 ceph-mon[36715]: pgmap v931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:34 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:reconnect 2024-07-26T11:16:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:34 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:reconnect,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:34.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:34 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00199997 2024-07-26T11:16:35.347 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:rejoin 2024-07-26T11:16:35.347 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:35.347 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-mon[32129]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: *** Caught signal (Aborted) ** 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: in thread 7f9346000640 thread_name:ms_dispatch 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f934c23e6f0] 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f934c28b94c] 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 3: raise() 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 4: abort() 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 5: /lib64/libc.so.6(+0x2871b) [0x7f934c22871b] 2024-07-26T11:16:35.348 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 6: /lib64/libc.so.6(+0x37386) [0x7f934c237386] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5591f5dda6b9] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5591f5f9880c] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5591f5ed82ca] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5591f5ea8753] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5591f5ea9a1b] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5591f5e8d752] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5591f5ddba56] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5591f5ddf637] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5591f5ddff5e] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5591f5dc303a] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f934cd9b407] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 18: (DispatchQueue::entry()+0x60f) [0x7f934cd9bcf9] 2024-07-26T11:16:35.349 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f934ce785a1] 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 20: (Thread::entry_wrapper()+0x43) [0x7f934cc25cdd] 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 21: (Thread::_entry_func(void*)+0xd) [0x7f934cc25cf9] 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 22: /lib64/libc.so.6(+0x89c02) [0x7f934c289c02] 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f934c30ec40] 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.165+0000 7f9346000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: in thread 7f9346000640 thread_name:ms_dispatch 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.350 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f934c23e6f0] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f934c28b94c] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 3: raise() 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 4: abort() 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 5: /lib64/libc.so.6(+0x2871b) [0x7f934c22871b] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 6: /lib64/libc.so.6(+0x37386) [0x7f934c237386] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5591f5dda6b9] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5591f5f9880c] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5591f5ed82ca] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5591f5ea8753] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5591f5ea9a1b] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5591f5e8d752] 2024-07-26T11:16:35.351 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5591f5ddba56] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5591f5ddf637] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5591f5ddff5e] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5591f5dc303a] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f934cd9b407] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 18: (DispatchQueue::entry()+0x60f) [0x7f934cd9bcf9] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f934ce785a1] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 20: (Thread::entry_wrapper()+0x43) [0x7f934cc25cdd] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 21: (Thread::_entry_func(void*)+0xd) [0x7f934cc25cf9] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 22: /lib64/libc.so.6(+0x89c02) [0x7f934c289c02] 2024-07-26T11:16:35.352 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f934c30ec40] 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 0> 2024-07-26T11:16:35.165+0000 7f9346000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: in thread 7f9346000640 thread_name:ms_dispatch 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f934c23e6f0] 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f934c28b94c] 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 3: raise() 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 4: abort() 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 5: /lib64/libc.so.6(+0x2871b) [0x7f934c22871b] 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 6: /lib64/libc.so.6(+0x37386) [0x7f934c237386] 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5591f5dda6b9] 2024-07-26T11:16:35.353 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5591f5f9880c] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5591f5ed82ca] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5591f5ea8753] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5591f5ea9a1b] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5591f5e8d752] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5591f5ddba56] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5591f5ddf637] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5591f5ddff5e] 2024-07-26T11:16:35.354 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5591f5dc303a] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f934cd9b407] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 18: (DispatchQueue::entry()+0x60f) [0x7f934cd9bcf9] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f934ce785a1] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 20: (Thread::entry_wrapper()+0x43) [0x7f934cc25cdd] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 21: (Thread::_entry_func(void*)+0xd) [0x7f934cc25cf9] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 22: /lib64/libc.so.6(+0x89c02) [0x7f934c289c02] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f934c30ec40] 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.355 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 0> 2024-07-26T11:16:35.165+0000 7f9346000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: in thread 7f9346000640 thread_name:ms_dispatch 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f934c23e6f0] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f934c28b94c] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 3: raise() 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 4: abort() 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 5: /lib64/libc.so.6(+0x2871b) [0x7f934c22871b] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 6: /lib64/libc.so.6(+0x37386) [0x7f934c237386] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5591f5dda6b9] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5591f5f9880c] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5591f5ed82ca] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5591f5ea8753] 2024-07-26T11:16:35.356 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5591f5ea9a1b] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5591f5e8d752] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5591f5ddba56] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5591f5ddf637] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5591f5ddff5e] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5591f5dc303a] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f934cd9b407] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 18: (DispatchQueue::entry()+0x60f) [0x7f934cd9bcf9] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f934ce785a1] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 20: (Thread::entry_wrapper()+0x43) [0x7f934cc25cdd] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 21: (Thread::_entry_func(void*)+0xd) [0x7f934cc25cf9] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 22: /lib64/libc.so.6(+0x89c02) [0x7f934c289c02] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f934c30ec40] 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:35.357 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j[108971]: 2024-07-26T11:16:35.358 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:35 smithi144 podman[109339]: 2024-07-26 11:16:35.345721334 +0000 UTC m=+0.050971827 container died d7e3168446e5b350b91a02c1b8df83aed0dcb7701735c206fb41a3076d179449 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724) 2024-07-26T11:16:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:35 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:rejoin 2024-07-26T11:16:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:35 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:35 smithi183 ceph-mon[37273]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:16:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:35 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:rejoin 2024-07-26T11:16:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:35 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=j=up:rejoin,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:35 smithi160 ceph-mon[36715]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T11:16:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:36 smithi183 ceph-mon[37273]: pgmap v932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:16:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:36 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:36 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:active 2024-07-26T11:16:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:36 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:36 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 639.539397 secs 2024-07-26T11:16:36.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:36 smithi160 ceph-mon[36715]: pgmap v932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:16:36.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:36 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:36.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:36 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:active 2024-07-26T11:16:36.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:36 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:36.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:36 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:36.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 639.539397 secs 2024-07-26T11:16:36.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:36 smithi144 podman[109339]: 2024-07-26 11:16:36.056436648 +0000 UTC m=+0.761687173 container remove d7e3168446e5b350b91a02c1b8df83aed0dcb7701735c206fb41a3076d179449 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-j, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T11:16:36.470 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:16:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:36 smithi144 ceph-mon[32129]: pgmap v932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:16:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:36 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:36 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1761669865,v1:172.21.15.144:6835/1761669865] up:active 2024-07-26T11:16:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:36 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-07-26T11:16:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:36 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=j=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-07-26T11:16:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 639.539397 secs 2024-07-26T11:16:36.970 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:16:36.970 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:36 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.407s CPU time. 2024-07-26T11:16:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:38 smithi183 ceph-mon[37273]: pgmap v933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:38 smithi144 ceph-mon[32129]: pgmap v933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:38 smithi160 ceph-mon[36715]: pgmap v933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:40 smithi144 ceph-mon[32129]: pgmap v934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:40 smithi183 ceph-mon[37273]: pgmap v934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:40 smithi160 ceph-mon[36715]: pgmap v934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:16:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 644.539509 secs 2024-07-26T11:16:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 644.539509 secs 2024-07-26T11:16:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 644.539509 secs 2024-07-26T11:16:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:42.125+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:42.127+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:42.127+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:42.127+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:42.127+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:42 smithi144 ceph-mon[32129]: pgmap v935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 192 B/s wr, 7 op/s 2024-07-26T11:16:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:42 smithi160 ceph-mon[36715]: pgmap v935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 192 B/s wr, 7 op/s 2024-07-26T11:16:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:42 smithi183 ceph-mon[37273]: pgmap v935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 192 B/s wr, 7 op/s 2024-07-26T11:16:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:44 smithi144 ceph-mon[32129]: pgmap v936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:16:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:44 smithi160 ceph-mon[36715]: pgmap v936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:16:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:44 smithi183 ceph-mon[37273]: pgmap v936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:16:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:46 smithi144 ceph-mon[32129]: pgmap v937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:16:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 649.539632 secs 2024-07-26T11:16:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:46 smithi160 ceph-mon[36715]: pgmap v937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:16:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 649.539632 secs 2024-07-26T11:16:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:46 smithi183 ceph-mon[37273]: pgmap v937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:16:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 649.539632 secs 2024-07-26T11:16:47.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:46 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:16:47.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:46 smithi144 systemd[1]: Stopped Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:16:47.220 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:46 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.407s CPU time. 2024-07-26T11:16:47.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:46 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-07-26T11:16:47.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:46 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T11:16:47.221 INFO:journalctl@ceph.mds.j.smithi144.stdout:Jul 26 11:16:46 smithi144 systemd[1]: Failed to start Ceph mds.j for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:16:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:48 smithi144 ceph-mon[32129]: pgmap v938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:16:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:48 smithi160 ceph-mon[36715]: pgmap v938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:16:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:48 smithi183 ceph-mon[37273]: pgmap v938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 85 B/s wr, 7 op/s 2024-07-26T11:16:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:50 smithi144 ceph-mon[32129]: pgmap v939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:16:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:50 smithi160 ceph-mon[36715]: pgmap v939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:16:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:50 smithi183 ceph-mon[37273]: pgmap v939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:16:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 654.539745 secs 2024-07-26T11:16:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:51 smithi144 ceph-mon[32129]: Replacing daemon mds.j as rank 1 with standby daemon mds.a 2024-07-26T11:16:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:51 smithi144 ceph-mon[32129]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-07-26T11:16:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:51 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:51 smithi144 ceph-mon[32129]: osdmap e130: 12 total, 12 up, 12 in 2024-07-26T11:16:51.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:51 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:51.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 654.539745 secs 2024-07-26T11:16:51.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:51 smithi183 ceph-mon[37273]: Replacing daemon mds.j as rank 1 with standby daemon mds.a 2024-07-26T11:16:51.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:51 smithi183 ceph-mon[37273]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-07-26T11:16:51.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:51 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:51.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:51 smithi183 ceph-mon[37273]: osdmap e130: 12 total, 12 up, 12 in 2024-07-26T11:16:51.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:51 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 654.539745 secs 2024-07-26T11:16:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:51 smithi160 ceph-mon[36715]: Replacing daemon mds.j as rank 1 with standby daemon mds.a 2024-07-26T11:16:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:51 smithi160 ceph-mon[36715]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-07-26T11:16:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:51 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:16:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:51 smithi160 ceph-mon[36715]: osdmap e130: 12 total, 12 up, 12 in 2024-07-26T11:16:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:51 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:replay,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:52 smithi144 ceph-mon[32129]: pgmap v941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:52 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:resolve 2024-07-26T11:16:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:52 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:52 smithi160 ceph-mon[36715]: pgmap v941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:52 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:resolve 2024-07-26T11:16:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:52 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:52 smithi183 ceph-mon[37273]: pgmap v941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:52 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:resolve 2024-07-26T11:16:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:52 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:resolve,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:reconnect 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: pgmap v942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:reconnect 2024-07-26T11:16:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: pgmap v942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:reconnect 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:reconnect,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: pgmap v942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:54.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:16:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:16:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:rejoin 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: pgmap v943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: pgmap v944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:55.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:16:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:16:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:16:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:16:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:16:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:rejoin 2024-07-26T11:16:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:16:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: pgmap v943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-07-26T11:16:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: pgmap v944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:55.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:16:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:16:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:16:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:16:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:rejoin 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=a=up:rejoin,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: pgmap v943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: pgmap v944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:16:55.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: *** Caught signal (Aborted) ** 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: in thread 7f2a8cc00640 thread_name:ms_dispatch 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2a92e3e6f0] 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2a92e8b94c] 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 3: raise() 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 4: abort() 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2a92e2871b] 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 6: /lib64/libc.so.6(+0x37386) [0x7f2a92e37386] 2024-07-26T11:16:55.222 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x565248b0c6b9] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x565248cca80c] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x565248c0a2ca] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565248bda753] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565248bdba1b] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565248bbf752] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x565248b0da56] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565248b11637] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565248b11f5e] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x565248af503a] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2a9399b407] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 18: (DispatchQueue::entry()+0x60f) [0x7f2a9399bcf9] 2024-07-26T11:16:55.223 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2a93a785a1] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 20: (Thread::entry_wrapper()+0x43) [0x7f2a93825cdd] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2a93825cf9] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2a92e89c02] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2a92f0ec40] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:54.834+0000 7f2a8cc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: in thread 7f2a8cc00640 thread_name:ms_dispatch 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2a92e3e6f0] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2a92e8b94c] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 3: raise() 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 4: abort() 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2a92e2871b] 2024-07-26T11:16:55.224 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 6: /lib64/libc.so.6(+0x37386) [0x7f2a92e37386] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x565248b0c6b9] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x565248cca80c] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x565248c0a2ca] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565248bda753] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565248bdba1b] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565248bbf752] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x565248b0da56] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565248b11637] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565248b11f5e] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x565248af503a] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2a9399b407] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 18: (DispatchQueue::entry()+0x60f) [0x7f2a9399bcf9] 2024-07-26T11:16:55.225 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2a93a785a1] 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 20: (Thread::entry_wrapper()+0x43) [0x7f2a93825cdd] 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2a93825cf9] 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2a92e89c02] 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2a92f0ec40] 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:55.226 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 0> 2024-07-26T11:16:54.834+0000 7f2a8cc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: in thread 7f2a8cc00640 thread_name:ms_dispatch 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2a92e3e6f0] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2a92e8b94c] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 3: raise() 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 4: abort() 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2a92e2871b] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 6: /lib64/libc.so.6(+0x37386) [0x7f2a92e37386] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x565248b0c6b9] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x565248cca80c] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x565248c0a2ca] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565248bda753] 2024-07-26T11:16:55.227 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565248bdba1b] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565248bbf752] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x565248b0da56] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565248b11637] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565248b11f5e] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x565248af503a] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2a9399b407] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 18: (DispatchQueue::entry()+0x60f) [0x7f2a9399bcf9] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2a93a785a1] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 20: (Thread::entry_wrapper()+0x43) [0x7f2a93825cdd] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2a93825cf9] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2a92e89c02] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2a92f0ec40] 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:55.228 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 0> 2024-07-26T11:16:54.834+0000 7f2a8cc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: in thread 7f2a8cc00640 thread_name:ms_dispatch 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2a92e3e6f0] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2a92e8b94c] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 3: raise() 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 4: abort() 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2a92e2871b] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 6: /lib64/libc.so.6(+0x37386) [0x7f2a92e37386] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x565248b0c6b9] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x565248cca80c] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x565248c0a2ca] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x565248bda753] 2024-07-26T11:16:55.229 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x565248bdba1b] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x565248bbf752] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x565248b0da56] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565248b11637] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565248b11f5e] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x565248af503a] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2a9399b407] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 18: (DispatchQueue::entry()+0x60f) [0x7f2a9399bcf9] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2a93a785a1] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 20: (Thread::entry_wrapper()+0x43) [0x7f2a93825cdd] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2a93825cf9] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2a92e89c02] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2a92f0ec40] 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:16:55.230 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a[109234]: 2024-07-26T11:16:55.231 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:54 smithi144 podman[109529]: 2024-07-26 11:16:54.975405028 +0000 UTC m=+0.035618308 container died afef44a4a186c240ec0e0166a629d55d53cc26b09b72ea3a6fcf6af41dcedf7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T11:16:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:55 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:55 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:active 2024-07-26T11:16:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:55 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:55 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 2024-07-26T11:16:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 659.539865 secs 2024-07-26T11:16:55.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:55 smithi144 podman[109529]: 2024-07-26 11:16:55.694320753 +0000 UTC m=+0.754534053 container remove afef44a4a186c240ec0e0166a629d55d53cc26b09b72ea3a6fcf6af41dcedf7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-07-26T11:16:55.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:55 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:16:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:55 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:55 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:active 2024-07-26T11:16:56.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:55 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:56.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:55 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 2024-07-26T11:16:56.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 659.539865 secs 2024-07-26T11:16:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:55 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:16:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:55 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6840/3815310409,v1:172.21.15.144:6841/3815310409] up:active 2024-07-26T11:16:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:55 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-07-26T11:16:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:55 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 2024-07-26T11:16:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 659.539865 secs 2024-07-26T11:16:56.720 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:56 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:16:56.720 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:16:56 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.392s CPU time. 2024-07-26T11:16:57.126 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:56 smithi144 ceph-mon[32129]: pgmap v945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 289 B/s wr, 7 op/s 2024-07-26T11:16:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:56 smithi183 ceph-mon[37273]: pgmap v945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 289 B/s wr, 7 op/s 2024-07-26T11:16:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:57.142+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:57.143+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:57.143+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:57.143+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:16:57.143+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:56 smithi160 ceph-mon[36715]: pgmap v945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 289 B/s wr, 7 op/s 2024-07-26T11:16:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:16:59 smithi183 ceph-mon[37273]: pgmap v946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 280 B/s wr, 5 op/s 2024-07-26T11:16:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:16:59 smithi144 ceph-mon[32129]: pgmap v946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 280 B/s wr, 5 op/s 2024-07-26T11:16:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:16:59 smithi160 ceph-mon[36715]: pgmap v946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 280 B/s wr, 5 op/s 2024-07-26T11:17:01.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:17:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:01 smithi144 ceph-mon[32129]: pgmap v947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 7 op/s 2024-07-26T11:17:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 664.539985 secs 2024-07-26T11:17:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:01 smithi160 ceph-mon[36715]: pgmap v947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 7 op/s 2024-07-26T11:17:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 664.539985 secs 2024-07-26T11:17:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:01 smithi183 ceph-mon[37273]: pgmap v947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 7 op/s 2024-07-26T11:17:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 664.539985 secs 2024-07-26T11:17:01.934 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:03 smithi144 ceph-mon[32129]: pgmap v948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 8 op/s 2024-07-26T11:17:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:03 smithi160 ceph-mon[36715]: pgmap v948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 8 op/s 2024-07-26T11:17:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:03 smithi183 ceph-mon[37273]: pgmap v948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 225 B/s wr, 8 op/s 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 -- 172.21.15.144:0/84617816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0075800 msgr2=0x7f21e0075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 --2- 172.21.15.144:0/84617816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0075800 0x7f21e0075c10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009900 tx=0x7f21cc02a410 comp rx=0 tx=0).stop 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 -- 172.21.15.144:0/84617816 shutdown_connections 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 --2- 172.21.15.144:0/84617816 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0113cb0 0x7f21e0116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 --2- 172.21.15.144:0/84617816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0075800 0x7f21e0075c10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 --2- 172.21.15.144:0/84617816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21e0076db0 0x7f21e00751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 -- 172.21.15.144:0/84617816 >> 172.21.15.144:0/84617816 conn(0x7f21e00fee70 msgr2=0x7f21e01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 -- 172.21.15.144:0/84617816 shutdown_connections 2024-07-26T11:17:04.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.238+0000 7f21dfe00640 1 -- 172.21.15.144:0/84617816 wait complete. 2024-07-26T11:17:04.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 Processor -- start 2024-07-26T11:17:04.241 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 -- start start 2024-07-26T11:17:04.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 0x7f21e01a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0076db0 0x7f21e01a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21e0113cb0 0x7f21e01a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01198f0 con 0x7f21e0076db0 2024-07-26T11:17:04.242 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f21e0119770 con 0x7f21e0075800 2024-07-26T11:17:04.243 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.240+0000 7f21dfe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f21e0119a70 con 0x7f21e0113cb0 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 0x7f21e01a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21de000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0076db0 0x7f21e01a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 0x7f21e01a02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:39932/0 (socket says 172.21.15.144:39932) 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 -- 172.21.15.144:0/1215619244 learned_addr learned my addr 172.21.15.144:0/1215619244 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21df400640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21e0113cb0 0x7f21e01a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 -- 172.21.15.144:0/1215619244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21e0113cb0 msgr2=0x7f21e01a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21e0113cb0 0x7f21e01a5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 -- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0076db0 msgr2=0x7f21e01a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0076db0 0x7f21e01a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 -- 172.21.15.144:0/1215619244 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e01a6250 con 0x7f21e0075800 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.241+0000 7f21dea00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 0x7f21e01a02e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f21d000e9f0 tx=0x7f21d000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.242+0000 7f21e4a00640 1 -- 172.21.15.144:0/1215619244 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d000cd80 con 0x7f21e0075800 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.242+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e01a6540 con 0x7f21e0075800 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.242+0000 7f21e4a00640 1 -- 172.21.15.144:0/1215619244 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d00054d0 con 0x7f21e0075800 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.242+0000 7f21e4a00640 1 -- 172.21.15.144:0/1215619244 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0010690 con 0x7f21e0075800 2024-07-26T11:17:04.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.242+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e01acca0 con 0x7f21e0075800 2024-07-26T11:17:04.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 Processor -- start 2024-07-26T11:17:04.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21e4a00640 1 -- 172.21.15.144:0/1215619244 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f21d0004e90 con 0x7f21e0075800 2024-07-26T11:17:04.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 -- start start 2024-07-26T11:17:04.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5d41b0 0x55a49c9a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 0x55a49c9a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55a49c918bc0 con 0x55a49c5fd070 2024-07-26T11:17:04.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a49c9080c0 con 0x55a49c5dc390 2024-07-26T11:17:04.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.246+0000 7f21eca03740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a49c90c100 con 0x55a49c5d41b0 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21dea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5d41b0 0x55a49c9a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 0x55a49c9a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21df400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 0x55a49c9a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:39944/0 (socket says 172.21.15.144:39944) 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 -- 172.21.15.144:0/4107300550 learned_addr learned my addr 172.21.15.144:0/4107300550 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 -- 172.21.15.144:0/4107300550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5d41b0 msgr2=0x55a49c9a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5d41b0 0x55a49c9a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 -- 172.21.15.144:0/4107300550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 msgr2=0x55a49c9a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21de000640 1 -- 172.21.15.144:0/4107300550 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a49c668650 con 0x55a49c5dc390 2024-07-26T11:17:04.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.247+0000 7f21dea00640 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5d41b0 0x55a49c9a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:04.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21de000640 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 0x55a49c9a2a80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009900 tx=0x7f21cc032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed97cdd15c58cf4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21cbe00640 1 -- 172.21.15.144:0/4107300550 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc040070 con 0x55a49c5dc390 2024-07-26T11:17:04.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21cbe00640 1 -- 172.21.15.144:0/4107300550 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21cc004030 con 0x55a49c5dc390 2024-07-26T11:17:04.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21cbe00640 1 -- 172.21.15.144:0/4107300550 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc035040 con 0x55a49c5dc390 2024-07-26T11:17:04.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21e4a00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21c40b3d40 0x7f21c40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21e4a00640 1 -- 172.21.15.144:0/1215619244 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 12834+0+0 (secure 0 0 0) 0x7f21d00e55d0 con 0x7f21e0075800 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.248+0000 7f21df400640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21c40b3d40 0x7f21c40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 -- 172.21.15.144:0/4107300550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 msgr2=0x55a49c9a2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 0x55a49c9a2a80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009900 tx=0x7f21cc032b20 comp rx=0 tx=0).stop 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 -- 172.21.15.144:0/4107300550 shutdown_connections 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5dc390 0x55a49c9a2a80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.252 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5d41b0 0x55a49c9a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 --2- 172.21.15.144:0/4107300550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 -- 172.21.15.144:0/4107300550 >> 172.21.15.144:0/4107300550 conn(0x55a49c5f1460 msgr2=0x55a49c99fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 -- 172.21.15.144:0/4107300550 shutdown_connections 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21df400640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21c40b3d40 0x7f21c40b6250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f21d8005fd0 tx=0x7f21d8009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.249+0000 7f21eca03740 1 -- 172.21.15.144:0/4107300550 wait complete. 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.250+0000 7f21eca03740 1 -- start start 2024-07-26T11:17:04.253 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.250+0000 7f21eca03740 1 Processor -- start 2024-07-26T11:17:04.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9ad600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.255 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9adb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9ae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55a49c76b700 con 0x55a49c5fd070 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a49c8a4690 con 0x55a49c5d41b0 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a49c9b0780 con 0x55a49c5dc390 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21eca03740 10 client.0 _ll_register_callbacks cb 0x55a49c8b4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21df400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21df400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35690/0 (socket says 172.21.15.144:35690) 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21df400640 1 -- 172.21.15.144:0/2507769602 learned_addr learned my addr 172.21.15.144:0/2507769602 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21de000640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.256 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.251+0000 7f21dea00640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21df400640 1 -- 172.21.15.144:0/2507769602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 msgr2=0x55a49c9adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21df400640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21df400640 1 -- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 msgr2=0x55a49c9ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21df400640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21df400640 1 -- 172.21.15.144:0/2507769602 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a49c8bb830 con 0x55a49c5fd070 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21df400640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9ae080 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009fd0 tx=0x7f21d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba82b4954c338038 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.257 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.252+0000 7f21c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21c8c00640 1 -- 172.21.15.144:0/2507769602 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d802c060 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21c8c00640 1 -- 172.21.15.144:0/2507769602 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 12834+0+0 (secure 0 0 0) 0x7f21d8017030 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21cc004030 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21c8c00640 1 -- 172.21.15.144:0/2507769602 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=297) ==== 24+0+0 (secure 0 0 0) 0x7f21d8034050 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21eca03740 10 client.16353 fetch_fsmap learned FSMap version 297 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=297,monmap=4+}) -- 0x55a49c9bc430 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21c8c00640 1 -- 172.21.15.144:0/2507769602 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21cc004030 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.253+0000 7f21c8c00640 1 -- 172.21.15.144:0/2507769602 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 297) ==== 4724+0+0 (secure 0 0 0) 0x7f21d80278f0 con 0x55a49c5fd070 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21eca03740 10 client.16353 fetch_fsmap finished waiting for FSMap version 297 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21eca03740 10 client.16353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21eca03740 20 client.16353 populate_metadata read hostname 'smithi144' 2024-07-26T11:17:04.258 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21eca03740 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c5bb9f0 0x55a49c9bfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21eca03740 4 client.16353 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:04.259 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a49c91ab10 con 0x55a49c5bb9f0 2024-07-26T11:17:04.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.254+0000 7f21dea00640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c5bb9f0 0x55a49c9bfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.255+0000 7f21dea00640 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c5bb9f0 0x55a49c9bfdf0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=915e5539cae3b16c server_cookie=1c83fd7fc8e025e7 in_seq=0 out_seq=0 2024-07-26T11:17:04.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.255+0000 7f21c8c00640 10 client.16353 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:04.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21c8c00640 1 -- 172.21.15.144:0/2507769602 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a49c91ab10 con 0x55a49c5bb9f0 2024-07-26T11:17:04.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21c8c00640 10 client.16353 handle_command_reply: tid=0 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 client.16353 shutdown 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 msgr2=0x55a49c9ae080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9ae080 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009fd0 tx=0x7f21d8004ab0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 shutdown_connections 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5fd070 0x55a49c9ae080 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c5bb9f0 0x55a49c9bfdf0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 --2- 172.21.15.144:0/2507769602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21c8c00640 0 client.16353 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 >> 172.21.15.144:0/2507769602 conn(0x55a49c5f1460 msgr2=0x55a49c9a51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 shutdown_connections 2024-07-26T11:17:04.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 1 -- 172.21.15.144:0/2507769602 wait complete. 2024-07-26T11:17:04.265 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.256+0000 7f21eca03740 20 client.16353 trim_cache size 0 max 16384 2024-07-26T11:17:04.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.275+0000 7f21eca03740 1 Processor -- start 2024-07-26T11:17:04.277 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.275+0000 7f21eca03740 1 -- start start 2024-07-26T11:17:04.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.275+0000 7f21eca03740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21eca03740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5dc390 0x55a49c9a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.279 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21eca03740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5fd070 0x55a49c9c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21eca03740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01b9510 con 0x55a49c5dc390 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21eca03740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01b7300 con 0x55a49c5d41b0 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21eca03740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01b8e40 con 0x55a49c5fd070 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21de000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5dc390 0x55a49c9a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21df400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5fd070 0x55a49c9c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:39964/0 (socket says 172.21.15.144:39964) 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 -- 172.21.15.144:0/3814017748 learned_addr learned my addr 172.21.15.144:0/3814017748 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:04.280 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 -- 172.21.15.144:0/3814017748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5fd070 msgr2=0x55a49c9c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5fd070 0x55a49c9c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 -- 172.21.15.144:0/3814017748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5dc390 msgr2=0x55a49c9a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5dc390 0x55a49c9a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.281 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.276+0000 7f21dea00640 1 -- 172.21.15.144:0/3814017748 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a49c9abb00 con 0x55a49c5d41b0 2024-07-26T11:17:04.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.277+0000 7f21dea00640 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9af4e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f21d000c6a0 tx=0x7f21d00e7a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=984abbe3c3c263aa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.277+0000 7f21cb400640 1 -- 172.21.15.144:0/3814017748 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0005180 con 0x55a49c5d41b0 2024-07-26T11:17:04.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.277+0000 7f21cb400640 1 -- 172.21.15.144:0/3814017748 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d00e53c0 con 0x55a49c5d41b0 2024-07-26T11:17:04.282 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.277+0000 7f21cb400640 1 -- 172.21.15.144:0/3814017748 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d00af6a0 con 0x55a49c5d41b0 2024-07-26T11:17:04.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 -- 172.21.15.144:0/3814017748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 msgr2=0x55a49c9af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9af4e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f21d000c6a0 tx=0x7f21d00e7a20 comp rx=0 tx=0).stop 2024-07-26T11:17:04.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 -- 172.21.15.144:0/3814017748 shutdown_connections 2024-07-26T11:17:04.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5fd070 0x55a49c9c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5dc390 0x55a49c9a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.287 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 --2- 172.21.15.144:0/3814017748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5d41b0 0x55a49c9af4e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 -- 172.21.15.144:0/3814017748 >> 172.21.15.144:0/3814017748 conn(0x55a49c5f1460 msgr2=0x55a49c9a3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 -- 172.21.15.144:0/3814017748 shutdown_connections 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.278+0000 7f21eca03740 1 -- 172.21.15.144:0/3814017748 wait complete. 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 -- start start 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 Processor -- start 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 0x55a49c9b3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.288 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5fd070 0x55a49c9b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01b80a0 con 0x55a49c5d41b0 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01b79d0 con 0x55a49c5fd070 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.279+0000 7f21eca03740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01b8770 con 0x55a49c5dc390 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21eca03740 10 client.0 _ll_register_callbacks cb 0x7f21d801f5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 0x55a49c9b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21de000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 0x55a49c9b3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35702/0 (socket says 172.21.15.144:35702) 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 -- 172.21.15.144:0/2722689682 learned_addr learned my addr 172.21.15.144:0/2722689682 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21df400640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5fd070 0x55a49c9b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.289 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 -- 172.21.15.144:0/2722689682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 msgr2=0x55a49c9b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 -- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5fd070 msgr2=0x55a49c9b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5fd070 0x55a49c9b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21dea00640 1 -- 172.21.15.144:0/2722689682 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a49ca276a0 con 0x55a49c5d41b0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.280+0000 7f21df400640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5fd070 0x55a49c9b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21dea00640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 0x55a49c9b3990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f21d00aada0 tx=0x7f21d00ab110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392b70c3414360f4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21c8c00640 1 -- 172.21.15.144:0/2722689682 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d00aabb0 con 0x55a49c5d41b0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21c8c00640 1 -- 172.21.15.144:0/2722689682 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 12834+0+0 (secure 0 0 0) 0x7f21d00a7330 con 0x55a49c5d41b0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21d00e53c0 con 0x55a49c5d41b0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21c8c00640 1 -- 172.21.15.144:0/2722689682 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=297) ==== 24+0+0 (secure 0 0 0) 0x7f21d00ab7a0 con 0x55a49c5d41b0 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21eca03740 10 client.16359 fetch_fsmap learned FSMap version 297 2024-07-26T11:17:04.290 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.281+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=297,monmap=4+}) -- 0x55a49ca28930 con 0x55a49c5d41b0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21c8c00640 1 -- 172.21.15.144:0/2722689682 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d00e53c0 con 0x55a49c5d41b0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21c8c00640 1 -- 172.21.15.144:0/2722689682 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 297) ==== 4724+0+0 (secure 0 0 0) 0x7f21d00bd030 con 0x55a49c5d41b0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21eca03740 10 client.16359 fetch_fsmap finished waiting for FSMap version 297 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21eca03740 10 client.16359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21eca03740 20 client.16359 populate_metadata read hostname 'smithi144' 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21eca03740 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c590da0 0x55a49ca2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21eca03740 4 client.16359 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a49c9165b0 con 0x55a49c590da0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.282+0000 7f21de000640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c590da0 0x55a49ca2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.283+0000 7f21c8c00640 10 client.16359 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.283+0000 7f21de000640 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c590da0 0x55a49ca2b090 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5173b1d1bfaaa82 server_cookie=c0748d5b4609c34 in_seq=0 out_seq=0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21c8c00640 1 -- 172.21.15.144:0/2722689682 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a49c9165b0 con 0x55a49c590da0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21c8c00640 10 client.16359 handle_command_reply: tid=0 2024-07-26T11:17:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 client.16359 shutdown 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 msgr2=0x55a49c9b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 0x55a49c9b3990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f21d00aada0 tx=0x7f21d00ab110 comp rx=0 tx=0).stop 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 shutdown_connections 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55a49c590da0 0x55a49ca2b090 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49c5fd070 0x55a49c9b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a49c5dc390 0x55a49c9b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 --2- 172.21.15.144:0/2722689682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a49c5d41b0 0x55a49c9b3990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21c8c00640 0 client.16359 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 >> 172.21.15.144:0/2722689682 conn(0x55a49c5f1460 msgr2=0x55a49c9af280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 shutdown_connections 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 1 -- 172.21.15.144:0/2722689682 wait complete. 2024-07-26T11:17:04.292 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.284+0000 7f21eca03740 20 client.16359 trim_cache size 0 max 16384 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:17:04.293 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:17:04.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21c40b3d40 msgr2=0x7f21c40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21c40b3d40 0x7f21c40b6250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f21d8005fd0 tx=0x7f21d8009500 comp rx=0 tx=0).stop 2024-07-26T11:17:04.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 msgr2=0x7f21e01a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 0x7f21e01a02e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f21d000e9f0 tx=0x7f21d000edf0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.294 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 shutdown_connections 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f21c40b3d40 0x7f21c40b6250 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21e0113cb0 0x7f21e01a5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e0076db0 0x7f21e01a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 --2- 172.21.15.144:0/1215619244 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21e0075800 0x7f21e01a02e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.285+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 >> 172.21.15.144:0/1215619244 conn(0x7f21e00fee70 msgr2=0x7f21e010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.286+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 shutdown_connections 2024-07-26T11:17:04.295 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:04.286+0000 7f21dfe00640 1 -- 172.21.15.144:0/1215619244 wait complete. 2024-07-26T11:17:05.117 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:17:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:05 smithi144 ceph-mon[32129]: pgmap v949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:17:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:05 smithi160 ceph-mon[36715]: pgmap v949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:17:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:05 smithi183 ceph-mon[37273]: pgmap v949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 7 op/s 2024-07-26T11:17:06.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 669.540090 secs 2024-07-26T11:17:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 669.540090 secs 2024-07-26T11:17:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 669.540090 secs 2024-07-26T11:17:06.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:17:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:17:06.970 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:17:06 smithi144 systemd[1]: Stopped Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:17:06.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:17:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.392s CPU time. 2024-07-26T11:17:06.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:17:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-07-26T11:17:06.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:17:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:17:06.971 INFO:journalctl@ceph.mds.a.smithi144.stdout:Jul 26 11:17:06 smithi144 systemd[1]: Failed to start Ceph mds.a for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:17:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:07 smithi144 ceph-mon[32129]: pgmap v950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:17:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:07 smithi160 ceph-mon[36715]: pgmap v950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:17:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:07 smithi183 ceph-mon[37273]: pgmap v950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 7 op/s 2024-07-26T11:17:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:09 smithi144 ceph-mon[32129]: pgmap v951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:17:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:09 smithi160 ceph-mon[36715]: pgmap v951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:17:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:09 smithi183 ceph-mon[37273]: pgmap v951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:17:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: pgmap v952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:17:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 674.540228 secs 2024-07-26T11:17:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: Replacing daemon mds.a as rank 1 with standby daemon mds.d 2024-07-26T11:17:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:17:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: osdmap e131: 12 total, 12 up, 12 in 2024-07-26T11:17:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:11 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: pgmap v952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:17:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 674.540228 secs 2024-07-26T11:17:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: Replacing daemon mds.a as rank 1 with standby daemon mds.d 2024-07-26T11:17:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:17:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: osdmap e131: 12 total, 12 up, 12 in 2024-07-26T11:17:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:11 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: pgmap v952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6 op/s 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 674.540228 secs 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: Replacing daemon mds.a as rank 1 with standby daemon mds.d 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: osdmap e131: 12 total, 12 up, 12 in 2024-07-26T11:17:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:11 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:12.160+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:12.161+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:12.161+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:12.161+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:12.161+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:13.458 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:13 smithi144 ceph-mon[32129]: pgmap v954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:17:13.459 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:13 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:resolve 2024-07-26T11:17:13.459 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:13 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:13 smithi160 ceph-mon[36715]: pgmap v954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:17:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:13 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:resolve 2024-07-26T11:17:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:13 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:13 smithi183 ceph-mon[37273]: pgmap v954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-07-26T11:17:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:13 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:resolve 2024-07-26T11:17:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:13 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:14 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:reconnect 2024-07-26T11:17:14.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:14 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:14.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:14 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:14 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:reconnect 2024-07-26T11:17:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:14 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:14 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:14 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:reconnect 2024-07-26T11:17:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:14 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:14 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:15 smithi160 ceph-mon[36715]: pgmap v955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:17:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:15 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:rejoin 2024-07-26T11:17:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:15 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:15 smithi160 ceph-mon[36715]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:15 smithi183 ceph-mon[37273]: pgmap v955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:17:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:15 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:rejoin 2024-07-26T11:17:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:15 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:15 smithi183 ceph-mon[37273]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-mon[32129]: pgmap v955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-07-26T11:17:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:rejoin 2024-07-26T11:17:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:15.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-mon[32129]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: *** Caught signal (Aborted) ** 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: in thread 7ff61e800640 thread_name:ms_dispatch 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff624a3e6f0] 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff624a8b94c] 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 3: raise() 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 4: abort() 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff624a2871b] 2024-07-26T11:17:15.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 6: /lib64/libc.so.6(+0x37386) [0x7ff624a37386] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564c65add6b9] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564c65c9b80c] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564c65bdb2ca] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564c65bab753] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564c65baca1b] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564c65b90752] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564c65adea56] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564c65ae2637] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564c65ae2f5e] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564c65ac603a] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff62559b407] 2024-07-26T11:17:15.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 18: (DispatchQueue::entry()+0x60f) [0x7ff62559bcf9] 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6256785a1] 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 20: (Thread::entry_wrapper()+0x43) [0x7ff625425cdd] 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff625425cf9] 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff624a89c02] 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff624b0ec40] 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.258+0000 7ff61e800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: in thread 7ff61e800640 thread_name:ms_dispatch 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:15.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff624a3e6f0] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff624a8b94c] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 3: raise() 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 4: abort() 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff624a2871b] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 6: /lib64/libc.so.6(+0x37386) [0x7ff624a37386] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564c65add6b9] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564c65c9b80c] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564c65bdb2ca] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564c65bab753] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564c65baca1b] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564c65b90752] 2024-07-26T11:17:15.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564c65adea56] 2024-07-26T11:17:15.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564c65ae2637] 2024-07-26T11:17:15.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564c65ae2f5e] 2024-07-26T11:17:15.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564c65ac603a] 2024-07-26T11:17:15.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff62559b407] 2024-07-26T11:17:15.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 18: (DispatchQueue::entry()+0x60f) [0x7ff62559bcf9] 2024-07-26T11:17:15.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6256785a1] 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 20: (Thread::entry_wrapper()+0x43) [0x7ff625425cdd] 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff625425cf9] 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff624a89c02] 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff624b0ec40] 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 0> 2024-07-26T11:17:15.258+0000 7ff61e800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: in thread 7ff61e800640 thread_name:ms_dispatch 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff624a3e6f0] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff624a8b94c] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 3: raise() 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 4: abort() 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff624a2871b] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 6: /lib64/libc.so.6(+0x37386) [0x7ff624a37386] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564c65add6b9] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564c65c9b80c] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564c65bdb2ca] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564c65bab753] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564c65baca1b] 2024-07-26T11:17:15.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564c65b90752] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564c65adea56] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564c65ae2637] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564c65ae2f5e] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564c65ac603a] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff62559b407] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 18: (DispatchQueue::entry()+0x60f) [0x7ff62559bcf9] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6256785a1] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 20: (Thread::entry_wrapper()+0x43) [0x7ff625425cdd] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff625425cf9] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff624a89c02] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff624b0ec40] 2024-07-26T11:17:15.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 0> 2024-07-26T11:17:15.258+0000 7ff61e800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: in thread 7ff61e800640 thread_name:ms_dispatch 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff624a3e6f0] 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff624a8b94c] 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 3: raise() 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 4: abort() 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff624a2871b] 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 6: /lib64/libc.so.6(+0x37386) [0x7ff624a37386] 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564c65add6b9] 2024-07-26T11:17:15.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564c65c9b80c] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564c65bdb2ca] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564c65bab753] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564c65baca1b] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564c65b90752] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564c65adea56] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564c65ae2637] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564c65ae2f5e] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564c65ac603a] 2024-07-26T11:17:15.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff62559b407] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 18: (DispatchQueue::entry()+0x60f) [0x7ff62559bcf9] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6256785a1] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 20: (Thread::entry_wrapper()+0x43) [0x7ff625425cdd] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff625425cf9] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff624a89c02] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff624b0ec40] 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[77348]: 2024-07-26T11:17:15.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:15 smithi144 podman[109764]: 2024-07-26 11:17:15.435518292 +0000 UTC m=+0.034625320 container died 6f685a8b539069587345dda2a9e5e78669dfe7aa1a7ed28515ac2cc1a5fc3d4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:17:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:16 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:16 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:active 2024-07-26T11:17:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:16 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 679.540354 secs 2024-07-26T11:17:16.470 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:16 smithi144 podman[109764]: 2024-07-26 11:17:16.14779359 +0000 UTC m=+0.746900646 container remove 6f685a8b539069587345dda2a9e5e78669dfe7aa1a7ed28515ac2cc1a5fc3d4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:17:16.470 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:16 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:17:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:16 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:16 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:active 2024-07-26T11:17:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:16 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 679.540354 secs 2024-07-26T11:17:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:16 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:16 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6836/3205660369,v1:172.21.15.144:6837/3205660369] up:active 2024-07-26T11:17:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:16 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 679.540354 secs 2024-07-26T11:17:16.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:16 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:17:16.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:16 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 4.118s CPU time. 2024-07-26T11:17:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:17 smithi160 ceph-mon[36715]: pgmap v956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:17:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:17 smithi183 ceph-mon[37273]: pgmap v956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:17:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:17 smithi144 ceph-mon[32129]: pgmap v956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:17:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:19 smithi144 ceph-mon[32129]: pgmap v957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:17:19.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:19 smithi160 ceph-mon[36715]: pgmap v957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:17:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:19 smithi183 ceph-mon[37273]: pgmap v957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:17:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:21 smithi160 ceph-mon[36715]: pgmap v958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:21.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 684.540480 secs 2024-07-26T11:17:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:21 smithi183 ceph-mon[37273]: pgmap v958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 684.540480 secs 2024-07-26T11:17:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:21 smithi144 ceph-mon[32129]: pgmap v958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 684.540480 secs 2024-07-26T11:17:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:23 smithi160 ceph-mon[36715]: pgmap v959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 181 B/s wr, 5 op/s 2024-07-26T11:17:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:23 smithi183 ceph-mon[37273]: pgmap v959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 181 B/s wr, 5 op/s 2024-07-26T11:17:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:23 smithi144 ceph-mon[32129]: pgmap v959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 181 B/s wr, 5 op/s 2024-07-26T11:17:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:25 smithi160 ceph-mon[36715]: pgmap v960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:17:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:25 smithi183 ceph-mon[37273]: pgmap v960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:17:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:25 smithi144 ceph-mon[32129]: pgmap v960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:17:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 689.540565 secs 2024-07-26T11:17:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 689.540565 secs 2024-07-26T11:17:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 689.540565 secs 2024-07-26T11:17:27.124 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:17:27.124 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:26 smithi144 systemd[1]: Stopped Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:17:27.124 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:26 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 4.118s CPU time. 2024-07-26T11:17:27.124 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:26 smithi144 systemd[1]: Starting Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:17:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:27.177+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:27.178+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:27.178+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:27.178+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:27.178+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:27.402 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:27 smithi144 ceph-mon[32129]: pgmap v961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:17:27.403 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 podman[109857]: 2024-07-26 11:17:27.026785204 +0000 UTC m=+0.035304907 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:17:27.403 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 podman[109857]: 2024-07-26 11:17:27.215217729 +0000 UTC m=+0.223737462 container create e09a05fe18417cf2d833bd72043204d8181519ee2c925ef6272d2a5bb8c27737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T11:17:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:27 smithi160 ceph-mon[36715]: pgmap v961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:17:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:27 smithi183 ceph-mon[37273]: pgmap v961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:17:27.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 podman[109857]: 2024-07-26 11:17:27.502855627 +0000 UTC m=+0.511375363 container init e09a05fe18417cf2d833bd72043204d8181519ee2c925ef6272d2a5bb8c27737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T11:17:27.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 podman[109857]: 2024-07-26 11:17:27.515251541 +0000 UTC m=+0.523771244 container start e09a05fe18417cf2d833bd72043204d8181519ee2c925ef6272d2a5bb8c27737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T11:17:27.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 bash[109857]: e09a05fe18417cf2d833bd72043204d8181519ee2c925ef6272d2a5bb8c27737 2024-07-26T11:17:27.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 systemd[1]: Started Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:17:27.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 ceph-mds[109875]: -- 172.21.15.144:0/3287995646 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f36d2d860 con 0x561f3aecf180 2024-07-26T11:17:27.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:27 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: starting mds.d at 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: daemon mds.d restarted 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: osdmap e132: 12 total, 12 up, 12 in 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:boot 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:17:28.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:28 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: daemon mds.d restarted 2024-07-26T11:17:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: osdmap e132: 12 total, 12 up, 12 in 2024-07-26T11:17:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:boot 2024-07-26T11:17:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:17:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:17:28.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:17:28.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:28 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: daemon mds.d restarted 2024-07-26T11:17:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:28.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: osdmap e132: 12 total, 12 up, 12 in 2024-07-26T11:17:28.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:boot 2024-07-26T11:17:28.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:17:28.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:17:28.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:17:28.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:28 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:29 smithi160 ceph-mon[36715]: pgmap v963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2024-07-26T11:17:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:29 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:resolve 2024-07-26T11:17:29.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:29 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:29 smithi183 ceph-mon[37273]: pgmap v963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2024-07-26T11:17:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:29 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:resolve 2024-07-26T11:17:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:29 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:29.679 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:29 smithi144 ceph-mon[32129]: pgmap v963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2024-07-26T11:17:29.679 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:29 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:resolve 2024-07-26T11:17:29.679 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:29 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:30 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:reconnect 2024-07-26T11:17:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:30 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:30 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:30 smithi144 ceph-mon[32129]: pgmap v964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4 op/s 2024-07-26T11:17:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:30 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:reconnect 2024-07-26T11:17:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:30 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:31.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:30 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:31.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:30 smithi160 ceph-mon[36715]: pgmap v964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4 op/s 2024-07-26T11:17:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:30 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:reconnect 2024-07-26T11:17:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:30 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:30 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:30 smithi183 ceph-mon[37273]: pgmap v964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4 op/s 2024-07-26T11:17:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 694.540701 secs 2024-07-26T11:17:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:rejoin 2024-07-26T11:17:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-mon[32129]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: *** Caught signal (Aborted) ** 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: in thread 7fe658600640 thread_name:ms_dispatch 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe65e83e6f0] 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe65e88b94c] 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 3: raise() 2024-07-26T11:17:31.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 4: abort() 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe65e82871b] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 6: /lib64/libc.so.6(+0x37386) [0x7fe65e837386] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561ef73a56b9] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561ef756380c] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561ef74a32ca] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561ef7473753] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561ef7474a1b] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561ef7458752] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561ef73a6a56] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561ef73aa637] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561ef73aaf5e] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561ef738e03a] 2024-07-26T11:17:31.972 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe65f39b407] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 18: (DispatchQueue::entry()+0x60f) [0x7fe65f39bcf9] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe65f4785a1] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 20: (Thread::entry_wrapper()+0x43) [0x7fe65f225cdd] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe65f225cf9] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe65e889c02] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe65e90ec40] 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.755+0000 7fe658600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: in thread 7fe658600640 thread_name:ms_dispatch 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe65e83e6f0] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe65e88b94c] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 3: raise() 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 4: abort() 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe65e82871b] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 6: /lib64/libc.so.6(+0x37386) [0x7fe65e837386] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561ef73a56b9] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561ef756380c] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561ef74a32ca] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561ef7473753] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561ef7474a1b] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561ef7458752] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561ef73a6a56] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561ef73aa637] 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561ef73aaf5e] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561ef738e03a] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe65f39b407] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 18: (DispatchQueue::entry()+0x60f) [0x7fe65f39bcf9] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe65f4785a1] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 20: (Thread::entry_wrapper()+0x43) [0x7fe65f225cdd] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe65f225cf9] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe65e889c02] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe65e90ec40] 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:31.975 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 0> 2024-07-26T11:17:31.755+0000 7fe658600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: in thread 7fe658600640 thread_name:ms_dispatch 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe65e83e6f0] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe65e88b94c] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 3: raise() 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 4: abort() 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe65e82871b] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 6: /lib64/libc.so.6(+0x37386) [0x7fe65e837386] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561ef73a56b9] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561ef756380c] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561ef74a32ca] 2024-07-26T11:17:31.976 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561ef7473753] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561ef7474a1b] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561ef7458752] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561ef73a6a56] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561ef73aa637] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561ef73aaf5e] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561ef738e03a] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe65f39b407] 2024-07-26T11:17:31.977 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 18: (DispatchQueue::entry()+0x60f) [0x7fe65f39bcf9] 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe65f4785a1] 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 20: (Thread::entry_wrapper()+0x43) [0x7fe65f225cdd] 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe65f225cf9] 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe65e889c02] 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe65e90ec40] 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 0> 2024-07-26T11:17:31.755+0000 7fe658600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: in thread 7fe658600640 thread_name:ms_dispatch 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:31.978 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe65e83e6f0] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe65e88b94c] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 3: raise() 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 4: abort() 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe65e82871b] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 6: /lib64/libc.so.6(+0x37386) [0x7fe65e837386] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561ef73a56b9] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561ef756380c] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561ef74a32ca] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561ef7473753] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561ef7474a1b] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561ef7458752] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561ef73a6a56] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561ef73aa637] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561ef73aaf5e] 2024-07-26T11:17:31.979 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561ef738e03a] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe65f39b407] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 18: (DispatchQueue::entry()+0x60f) [0x7fe65f39bcf9] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe65f4785a1] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 20: (Thread::entry_wrapper()+0x43) [0x7fe65f225cdd] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe65f225cf9] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe65e889c02] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe65e90ec40] 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[109871]: 2024-07-26T11:17:31.980 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:31 smithi144 podman[109909]: 2024-07-26 11:17:31.895482777 +0000 UTC m=+0.033913221 container died e09a05fe18417cf2d833bd72043204d8181519ee2c925ef6272d2a5bb8c27737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T11:17:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 694.540701 secs 2024-07-26T11:17:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:31 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:rejoin 2024-07-26T11:17:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:31 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:31 smithi160 ceph-mon[36715]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 694.540701 secs 2024-07-26T11:17:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:31 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:rejoin 2024-07-26T11:17:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:31 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:31 smithi183 ceph-mon[37273]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:32.559 INFO:journalctl@ceph.mds.h.smithi160.stdout:Jul 26 11:17:32 smithi160 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-h[67562]: 2024-07-26T11:17:32.203+0000 7f017ce3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:17:32.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:32 smithi144 podman[109909]: 2024-07-26 11:17:32.788933421 +0000 UTC m=+0.927363907 container remove e09a05fe18417cf2d833bd72043204d8181519ee2c925ef6272d2a5bb8c27737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-07-26T11:17:32.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:32 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:17:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:32 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:32 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:active 2024-07-26T11:17:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:32 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:32 smithi144 ceph-mon[32129]: pgmap v965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:17:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:32 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:32 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:active 2024-07-26T11:17:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:32 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:32 smithi160 ceph-mon[36715]: pgmap v965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:17:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:32 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:32 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/391828061,v1:172.21.15.144:6835/391828061] up:active 2024-07-26T11:17:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:32 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:32 smithi183 ceph-mon[37273]: pgmap v965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:17:33.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:33 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:17:33.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:33 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.122s CPU time. 2024-07-26T11:17:35.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:17:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:35 smithi144 ceph-mon[32129]: pgmap v966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:17:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:35 smithi160 ceph-mon[36715]: pgmap v966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:17:35.601 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:35 smithi183 ceph-mon[37273]: pgmap v966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:17:36.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 699.540785 secs 2024-07-26T11:17:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 699.540785 secs 2024-07-26T11:17:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 699.540785 secs 2024-07-26T11:17:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:37 smithi144 ceph-mon[32129]: pgmap v967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:37 smithi160 ceph-mon[36715]: pgmap v967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:37 smithi183 ceph-mon[37273]: pgmap v967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:37.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 -- 172.21.15.144:0/968822036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010af10 msgr2=0x7f820010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 --2- 172.21.15.144:0/968822036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010af10 0x7f820010b320 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f81ec009900 tx=0x7f81ec02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 -- 172.21.15.144:0/968822036 shutdown_connections 2024-07-26T11:17:37.882 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 --2- 172.21.15.144:0/968822036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f820010b860 0x7f8200112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 --2- 172.21.15.144:0/968822036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010af10 0x7f820010b320 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 --2- 172.21.15.144:0/968822036 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82001055d0 0x7f82001059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.879+0000 7f8206e00640 1 -- 172.21.15.144:0/968822036 >> 172.21.15.144:0/968822036 conn(0x7f8200101040 msgr2=0x7f8200103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:37.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.880+0000 7f8206e00640 1 -- 172.21.15.144:0/968822036 shutdown_connections 2024-07-26T11:17:37.883 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.880+0000 7f8206e00640 1 -- 172.21.15.144:0/968822036 wait complete. 2024-07-26T11:17:37.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 Processor -- start 2024-07-26T11:17:37.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 -- start start 2024-07-26T11:17:37.885 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82001055d0 0x7f820019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 0x7f820019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010b860 0x7f82001a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f82001154d0 con 0x7f820010b860 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8200115350 con 0x7f820010af10 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.881+0000 7f8206e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8200115650 con 0x7f82001055d0 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 0x7f820019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8206400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010b860 0x7f82001a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 0x7f820019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55534/0 (socket says 172.21.15.144:55534) 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 -- 172.21.15.144:0/322589497 learned_addr learned my addr 172.21.15.144:0/322589497 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205a00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82001055d0 0x7f820019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 -- 172.21.15.144:0/322589497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82001055d0 msgr2=0x7f820019bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82001055d0 0x7f820019bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.886 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 -- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010b860 msgr2=0x7f82001a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010b860 0x7f82001a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.882+0000 7f8205000640 1 -- 172.21.15.144:0/322589497 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82001a1ed0 con 0x7f820010af10 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.883+0000 7f8205000640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 0x7f820019c500 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006010 tx=0x7f81ec004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.883+0000 7f81fe000640 1 -- 172.21.15.144:0/322589497 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec03f030 con 0x7f820010af10 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.883+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82001a2160 con 0x7f820010af10 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.883+0000 7f81fe000640 1 -- 172.21.15.144:0/322589497 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81ec003aa0 con 0x7f820010af10 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.883+0000 7f81fe000640 1 -- 172.21.15.144:0/322589497 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec035070 con 0x7f820010af10 2024-07-26T11:17:37.887 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.883+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82001a8920 con 0x7f820010af10 2024-07-26T11:17:37.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.886+0000 7f820f05a740 1 Processor -- start 2024-07-26T11:17:37.888 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 -- start start 2024-07-26T11:17:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce81fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 0x560c2ce82510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cabc390 0x560c2ce82a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x560c2cdf8b90 con 0x560c2cadd070 2024-07-26T11:17:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560c2cde8090 con 0x560c2cab41b0 2024-07-26T11:17:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f820f05a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560c2cdec0d0 con 0x560c2cabc390 2024-07-26T11:17:37.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f81fe000640 1 -- 172.21.15.144:0/322589497 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f81ec042030 con 0x7f820010af10 2024-07-26T11:17:37.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f8205a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 0x560c2ce82510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f8206400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce81fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f8205a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 0x560c2ce82510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55540/0 (socket says 172.21.15.144:55540) 2024-07-26T11:17:37.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.887+0000 7f8205a00640 1 -- 172.21.15.144:0/774747151 learned_addr learned my addr 172.21.15.144:0/774747151 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:37.890 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205000640 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cabc390 0x560c2ce82a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.891 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205a00640 1 -- 172.21.15.144:0/774747151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cabc390 msgr2=0x560c2ce82a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205a00640 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cabc390 0x560c2ce82a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205a00640 1 -- 172.21.15.144:0/774747151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 msgr2=0x560c2ce81fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205a00640 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce81fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205a00640 1 -- 172.21.15.144:0/774747151 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2cb48650 con 0x560c2cab41b0 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.888+0000 7f8205a00640 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 0x560c2ce82510 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f81f400bd10 tx=0x7f81f40107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28f871f27a20b4ec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.889+0000 7f81f2a00640 1 -- 172.21.15.144:0/774747151 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4004030 con 0x560c2cab41b0 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.889+0000 7f81f2a00640 1 -- 172.21.15.144:0/774747151 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81f40035f0 con 0x560c2cab41b0 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.889+0000 7f81f2a00640 1 -- 172.21.15.144:0/774747151 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f40117b0 con 0x560c2cab41b0 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.889+0000 7f81fe000640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f81e80b3d40 0x7f81e80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.889+0000 7f81fe000640 1 -- 172.21.15.144:0/322589497 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 13006+0+0 (secure 0 0 0) 0x7f81ec105640 con 0x7f820010af10 2024-07-26T11:17:37.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.889+0000 7f8206400640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f81e80b3d40 0x7f81e80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 -- 172.21.15.144:0/774747151 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 msgr2=0x560c2ce82510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 0x560c2ce82510 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f81f400bd10 tx=0x7f81f40107a0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 -- 172.21.15.144:0/774747151 shutdown_connections 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cabc390 0x560c2ce82a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cab41b0 0x560c2ce82510 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 --2- 172.21.15.144:0/774747151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce81fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 -- 172.21.15.144:0/774747151 >> 172.21.15.144:0/774747151 conn(0x560c2cad1460 msgr2=0x560c2ce7fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 -- 172.21.15.144:0/774747151 shutdown_connections 2024-07-26T11:17:37.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f820f05a740 1 -- 172.21.15.144:0/774747151 wait complete. 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.890+0000 7f8206400640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f81e80b3d40 0x7f81e80b6250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f81f8005fd0 tx=0x7f81f8009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 -- start start 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 Processor -- start 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 0x560c2ce8d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce8db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cadd070 0x560c2ce8e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x560c2cc4b700 con 0x560c2cab41b0 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560c2cd84660 con 0x560c2cabc390 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.891+0000 7f820f05a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560c2ce907b0 con 0x560c2cadd070 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f820f05a740 10 client.0 _ll_register_callbacks cb 0x560c2cd94110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 0x560c2ce8d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.896 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 0x560c2ce8d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37794/0 (socket says 172.21.15.144:37794) 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 -- 172.21.15.144:0/1874779965 learned_addr learned my addr 172.21.15.144:0/1874779965 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8206400640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cadd070 0x560c2ce8e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205000640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce8db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 -- 172.21.15.144:0/1874779965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cadd070 msgr2=0x560c2ce8e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cadd070 0x560c2ce8e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 -- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 msgr2=0x560c2ce8db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce8db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.892+0000 7f8205a00640 1 -- 172.21.15.144:0/1874779965 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2cd9b800 con 0x560c2cab41b0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f8205a00640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 0x560c2ce8d610 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f81f4005920 tx=0x7f81f400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a245b53e94027d0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f81e7400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f81e7400640 1 -- 172.21.15.144:0/1874779965 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4029060 con 0x560c2cab41b0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81f40035f0 con 0x560c2cab41b0 2024-07-26T11:17:37.897 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f81e7400640 1 -- 172.21.15.144:0/1874779965 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 13006+0+0 (secure 0 0 0) 0x7f81f4004000 con 0x560c2cab41b0 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f81e7400640 1 -- 172.21.15.144:0/1874779965 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=308) ==== 24+0+0 (secure 0 0 0) 0x7f81f4031050 con 0x560c2cab41b0 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f820f05a740 10 client.16377 fetch_fsmap learned FSMap version 308 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.893+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=308,monmap=4+}) -- 0x560c2ce9c410 con 0x560c2cab41b0 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f81e7400640 1 -- 172.21.15.144:0/1874779965 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81f40035f0 con 0x560c2cab41b0 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f81e7400640 1 -- 172.21.15.144:0/1874779965 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 308) ==== 4131+0+0 (secure 0 0 0) 0x7f81f4023030 con 0x560c2cab41b0 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f820f05a740 10 client.16377 fetch_fsmap finished waiting for FSMap version 308 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f820f05a740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f820f05a740 20 client.16377 populate_metadata read hostname 'smithi144' 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f820f05a740 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca9b9f0 0x560c2ce9fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f820f05a740 4 client.16377 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:37.898 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c2cdfaae0 con 0x560c2ca9b9f0 2024-07-26T11:17:37.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.894+0000 7f8205000640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca9b9f0 0x560c2ce9fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.899 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.895+0000 7f8205000640 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca9b9f0 0x560c2ce9fc10 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae332a32899cf30 server_cookie=f4933882087ca896 in_seq=0 out_seq=0 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.895+0000 7f81e7400640 10 client.16377 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f81e7400640 1 -- 172.21.15.144:0/1874779965 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560c2cdfaae0 con 0x560c2ca9b9f0 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f81e7400640 10 client.16377 handle_command_reply: tid=0 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 client.16377 shutdown 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 msgr2=0x560c2ce8d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 0x560c2ce8d610 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f81f4005920 tx=0x7f81f400b8e0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 shutdown_connections 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cadd070 0x560c2ce8e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca9b9f0 0x560c2ce9fc10 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.900 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce8db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 --2- 172.21.15.144:0/1874779965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cab41b0 0x560c2ce8d610 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 >> 172.21.15.144:0/1874779965 conn(0x560c2cad1460 msgr2=0x560c2ce851b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:37.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f81e7400640 0 client.16377 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:37.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 shutdown_connections 2024-07-26T11:17:37.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.896+0000 7f820f05a740 1 -- 172.21.15.144:0/1874779965 wait complete. 2024-07-26T11:17:37.901 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.897+0000 7f820f05a740 20 client.16377 trim_cache size 0 max 16384 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.915+0000 7f820f05a740 1 Processor -- start 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.915+0000 7f820f05a740 1 -- start start 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f820f05a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce7eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f820f05a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cabc390 0x560c2ce83710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f820f05a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cadd070 0x560c2cea5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f820f05a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f82001b4be0 con 0x560c2cabc390 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f820f05a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f82001b30a0 con 0x560c2cadd070 2024-07-26T11:17:37.918 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f820f05a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82001b4510 con 0x560c2cab41b0 2024-07-26T11:17:37.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f8205a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce7eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f8205000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cabc390 0x560c2ce83710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f8205a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce7eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:37524/0 (socket says 172.21.15.144:37524) 2024-07-26T11:17:37.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f8205a00640 1 -- 172.21.15.144:0/1499309135 learned_addr learned my addr 172.21.15.144:0/1499309135 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:37.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.916+0000 7f8206400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cadd070 0x560c2cea5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.919 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8206400640 1 -- 172.21.15.144:0/1499309135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 msgr2=0x560c2ce7eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8206400640 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce7eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8206400640 1 -- 172.21.15.144:0/1499309135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cabc390 msgr2=0x560c2ce83710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8206400640 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cabc390 0x560c2ce83710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8206400640 1 -- 172.21.15.144:0/1499309135 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2cea01e0 con 0x560c2cadd070 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8205a00640 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce7eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f8206400640 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cadd070 0x560c2cea5410 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f81f8011e90 tx=0x7f81f801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee94248cc58e576 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f81f2000640 1 -- 172.21.15.144:0/1499309135 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f8004b20 con 0x560c2cadd070 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f81f2000640 1 -- 172.21.15.144:0/1499309135 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81f8004cc0 con 0x560c2cadd070 2024-07-26T11:17:37.920 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.917+0000 7f81f2000640 1 -- 172.21.15.144:0/1499309135 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f80053a0 con 0x560c2cadd070 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.918+0000 7f820f05a740 1 -- 172.21.15.144:0/1499309135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cadd070 msgr2=0x560c2cea5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.918+0000 7f820f05a740 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cadd070 0x560c2cea5410 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f81f8011e90 tx=0x7f81f801a000 comp rx=0 tx=0).stop 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 -- 172.21.15.144:0/1499309135 shutdown_connections 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cadd070 0x560c2cea5410 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cabc390 0x560c2ce83710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 --2- 172.21.15.144:0/1499309135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce7eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.921 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 -- 172.21.15.144:0/1499309135 >> 172.21.15.144:0/1499309135 conn(0x560c2cad1460 msgr2=0x560c2ce83000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:37.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 -- 172.21.15.144:0/1499309135 shutdown_connections 2024-07-26T11:17:37.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 -- 172.21.15.144:0/1499309135 wait complete. 2024-07-26T11:17:37.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.919+0000 7f820f05a740 1 -- start start 2024-07-26T11:17:37.922 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 Processor -- start 2024-07-26T11:17:37.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce92a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce92fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.923 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce93500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f82001b3e40 con 0x560c2cadd070 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f82001b3770 con 0x560c2cabc390 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560c2cf064e0 con 0x560c2cab41b0 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f820f05a740 10 client.0 _ll_register_callbacks cb 0x7f81f4003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f8206400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce93500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f8206400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce93500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37814/0 (socket says 172.21.15.144:37814) 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.920+0000 7f8206400640 1 -- 172.21.15.144:0/1524655463 learned_addr learned my addr 172.21.15.144:0/1524655463 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce92a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205000640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce92fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 -- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 msgr2=0x560c2ce92fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce92fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 -- 172.21.15.144:0/1524655463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 msgr2=0x560c2ce93500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce93500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 -- 172.21.15.144:0/1524655463 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2cf066e0 con 0x560c2cab41b0 2024-07-26T11:17:37.924 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8206400640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce93500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:37.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205000640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce92fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:37.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f8205a00640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce92a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c010 tx=0x7f81f4024a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf397409acff7a3b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:37.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.921+0000 7f81e7400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:17:37.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f81e7400640 1 -- 172.21.15.144:0/1524655463 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4039030 con 0x560c2cab41b0 2024-07-26T11:17:37.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81f8004cc0 con 0x560c2cab41b0 2024-07-26T11:17:37.925 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f81e7400640 1 -- 172.21.15.144:0/1524655463 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 13006+0+0 (secure 0 0 0) 0x7f81f40035f0 con 0x560c2cab41b0 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f81e7400640 1 -- 172.21.15.144:0/1524655463 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=308) ==== 24+0+0 (secure 0 0 0) 0x7f81f4044050 con 0x560c2cab41b0 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f820f05a740 10 client.26650 fetch_fsmap learned FSMap version 308 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=308,monmap=4+}) -- 0x560c2cf07970 con 0x560c2cab41b0 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.922+0000 7f81e7400640 1 -- 172.21.15.144:0/1524655463 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81f8004cc0 con 0x560c2cab41b0 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f81e7400640 1 -- 172.21.15.144:0/1524655463 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 308) ==== 4131+0+0 (secure 0 0 0) 0x7f81f4034050 con 0x560c2cab41b0 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f820f05a740 10 client.26650 fetch_fsmap finished waiting for FSMap version 308 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f820f05a740 10 client.26650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f820f05a740 20 client.26650 populate_metadata read hostname 'smithi144' 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f820f05a740 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca70da0 0x560c2cf0a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f820f05a740 4 client.26650 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560c2cdf6580 con 0x560c2ca70da0 2024-07-26T11:17:37.927 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f8205000640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca70da0 0x560c2cf0a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f8205000640 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca70da0 0x560c2cf0a030 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=404399ae2b2fbed0 server_cookie=ec6fd3fa91d03a92 in_seq=0 out_seq=0 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.923+0000 7f81e7400640 10 client.26650 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f81e7400640 1 -- 172.21.15.144:0/1524655463 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560c2cdf6580 con 0x560c2ca70da0 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f81e7400640 10 client.26650 handle_command_reply: tid=0 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 client.26650 shutdown 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 msgr2=0x560c2ce92a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce92a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c010 tx=0x7f81f4024a90 comp rx=0 tx=0).stop 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 shutdown_connections 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560c2cadd070 0x560c2ce93500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560c2ca70da0 0x560c2cf0a030 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560c2cabc390 0x560c2ce92fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 --2- 172.21.15.144:0/1524655463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c2cab41b0 0x560c2ce92a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f81e7400640 0 client.26650 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 >> 172.21.15.144:0/1524655463 conn(0x560c2cad1460 msgr2=0x560c2ce7ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:37.928 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 shutdown_connections 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.924+0000 7f820f05a740 1 -- 172.21.15.144:0/1524655463 wait complete. 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f820f05a740 20 client.26650 trim_cache size 0 max 16384 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:17:37.929 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:17:37.930 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:17:37.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f81e80b3d40 msgr2=0x7f81e80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f81e80b3d40 0x7f81e80b6250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f81f8005fd0 tx=0x7f81f8009500 comp rx=0 tx=0).stop 2024-07-26T11:17:37.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 msgr2=0x7f820019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:37.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 0x7f820019c500 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006010 tx=0x7f81ec004060 comp rx=0 tx=0).stop 2024-07-26T11:17:37.930 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 shutdown_connections 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f81e80b3d40 0x7f81e80b6250 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f820010b860 0x7f82001a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f820010af10 0x7f820019c500 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 --2- 172.21.15.144:0/322589497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82001055d0 0x7f820019bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.925+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 >> 172.21.15.144:0/322589497 conn(0x7f8200101040 msgr2=0x7f8200103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.926+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 shutdown_connections 2024-07-26T11:17:37.931 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:17:37.926+0000 7f8206e00640 1 -- 172.21.15.144:0/322589497 wait complete. 2024-07-26T11:17:38.771 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:17:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:39 smithi144 ceph-mon[32129]: pgmap v968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 195 B/s wr, 6 op/s 2024-07-26T11:17:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:39 smithi160 ceph-mon[36715]: pgmap v968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 195 B/s wr, 6 op/s 2024-07-26T11:17:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:39 smithi183 ceph-mon[37273]: pgmap v968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 195 B/s wr, 6 op/s 2024-07-26T11:17:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:41 smithi144 ceph-mon[32129]: pgmap v969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:17:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 704.540912 secs 2024-07-26T11:17:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:41 smithi160 ceph-mon[36715]: pgmap v969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:17:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 704.540912 secs 2024-07-26T11:17:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:41 smithi183 ceph-mon[37273]: pgmap v969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:17:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 704.540912 secs 2024-07-26T11:17:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:42.188+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:42.189+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:42.189+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:42.189+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:42.190+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:43 smithi144 ceph-mon[32129]: pgmap v970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:17:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:43 smithi160 ceph-mon[36715]: pgmap v970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:17:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:43 smithi183 ceph-mon[37273]: pgmap v970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:17:43.889 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:43 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:17:43.889 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:43 smithi144 systemd[1]: Stopped Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:17:43.889 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:43 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.122s CPU time. 2024-07-26T11:17:43.890 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:43 smithi144 systemd[1]: Starting Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:17:44.187 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:43 smithi144 podman[110191]: 2024-07-26 11:17:43.790971826 +0000 UTC m=+0.030831219 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:17:44.187 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:43 smithi144 podman[110191]: 2024-07-26 11:17:43.923962806 +0000 UTC m=+0.163822224 container create 92fe937e085d75c9bdc05e8de9cc34ab3ba3c3398a8ed0c8e1a84638e8a0fc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T11:17:44.470 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:44 smithi144 podman[110191]: 2024-07-26 11:17:44.18518223 +0000 UTC m=+0.425041665 container init 92fe937e085d75c9bdc05e8de9cc34ab3ba3c3398a8ed0c8e1a84638e8a0fc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:17:44.470 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:44 smithi144 podman[110191]: 2024-07-26 11:17:44.196224211 +0000 UTC m=+0.436083610 container start 92fe937e085d75c9bdc05e8de9cc34ab3ba3c3398a8ed0c8e1a84638e8a0fc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:17:44.471 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:44 smithi144 bash[110191]: 92fe937e085d75c9bdc05e8de9cc34ab3ba3c3398a8ed0c8e1a84638e8a0fc20 2024-07-26T11:17:44.471 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:44 smithi144 systemd[1]: Started Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:17:44.471 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:44 smithi144 ceph-mds[110210]: -- 172.21.15.144:0/3882401231 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560dacf43860 con 0x560db10e5180 2024-07-26T11:17:44.471 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:44 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: starting mds.d at 2024-07-26T11:17:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: pgmap v971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:17:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: daemon mds.d restarted 2024-07-26T11:17:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: osdmap e133: 12 total, 12 up, 12 in 2024-07-26T11:17:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:boot 2024-07-26T11:17:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:17:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:17:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:17:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:45 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: pgmap v971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:17:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: daemon mds.d restarted 2024-07-26T11:17:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: osdmap e133: 12 total, 12 up, 12 in 2024-07-26T11:17:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:boot 2024-07-26T11:17:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:17:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:17:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:17:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:45 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: pgmap v971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:17:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: daemon mds.d restarted 2024-07-26T11:17:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:17:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: osdmap e133: 12 total, 12 up, 12 in 2024-07-26T11:17:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:boot 2024-07-26T11:17:45.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:17:45.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:17:45.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:17:45.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:45 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:46.214+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480006140 con 0x7f74b0109990 2024-07-26T11:17:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:46.215+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480006140 con 0x7f74b0109990 2024-07-26T11:17:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:46.218+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478005d20 con 0x563c6c4182f0 2024-07-26T11:17:46.220 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:46.219+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478005d20 con 0x563c6c4182f0 2024-07-26T11:17:46.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:46 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:resolve 2024-07-26T11:17:46.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:46 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:46.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 709.541032 secs 2024-07-26T11:17:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:46 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:resolve 2024-07-26T11:17:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:46 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 709.541032 secs 2024-07-26T11:17:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:46 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:resolve 2024-07-26T11:17:46.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:46 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:46.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 709.541032 secs 2024-07-26T11:17:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:47 smithi183 ceph-mon[37273]: pgmap v973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-07-26T11:17:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:47 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:reconnect 2024-07-26T11:17:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:47 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:47 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:47 smithi144 ceph-mon[32129]: pgmap v973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-07-26T11:17:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:47 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:reconnect 2024-07-26T11:17:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:47 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:47 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:47 smithi160 ceph-mon[36715]: pgmap v973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-07-26T11:17:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:47 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:reconnect 2024-07-26T11:17:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:47 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:47 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:17:48.635 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:rejoin 2024-07-26T11:17:48.635 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:48.635 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-mon[32129]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:48 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:rejoin 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:48 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:48 smithi183 ceph-mon[37273]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: *** Caught signal (Aborted) ** 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: in thread 7f8695600640 thread_name:ms_dispatch 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f869b83e6f0] 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f869b88b94c] 2024-07-26T11:17:48.636 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 3: raise() 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 4: abort() 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 5: /lib64/libc.so.6(+0x2871b) [0x7f869b82871b] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 6: /lib64/libc.so.6(+0x37386) [0x7f869b837386] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d9146d6b9] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d9162b80c] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d9156b2ca] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d9153b753] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d9153ca1b] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d91520752] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d9146ea56] 2024-07-26T11:17:48.637 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d91472637] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d91472f5e] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d9145603a] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f869c39b407] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 18: (DispatchQueue::entry()+0x60f) [0x7f869c39bcf9] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f869c4785a1] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 20: (Thread::entry_wrapper()+0x43) [0x7f869c225cdd] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 21: (Thread::_entry_func(void*)+0xd) [0x7f869c225cf9] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 22: /lib64/libc.so.6(+0x89c02) [0x7f869b889c02] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f869b90ec40] 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.435+0000 7f8695600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: in thread 7f8695600640 thread_name:ms_dispatch 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.638 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f869b83e6f0] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f869b88b94c] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 3: raise() 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 4: abort() 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 5: /lib64/libc.so.6(+0x2871b) [0x7f869b82871b] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 6: /lib64/libc.so.6(+0x37386) [0x7f869b837386] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d9146d6b9] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d9162b80c] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d9156b2ca] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d9153b753] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d9153ca1b] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d91520752] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d9146ea56] 2024-07-26T11:17:48.639 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d91472637] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d91472f5e] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d9145603a] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f869c39b407] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 18: (DispatchQueue::entry()+0x60f) [0x7f869c39bcf9] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f869c4785a1] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 20: (Thread::entry_wrapper()+0x43) [0x7f869c225cdd] 2024-07-26T11:17:48.640 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 21: (Thread::_entry_func(void*)+0xd) [0x7f869c225cf9] 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 22: /lib64/libc.so.6(+0x89c02) [0x7f869b889c02] 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f869b90ec40] 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 0> 2024-07-26T11:17:48.435+0000 7f8695600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: in thread 7f8695600640 thread_name:ms_dispatch 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f869b83e6f0] 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f869b88b94c] 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 3: raise() 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 4: abort() 2024-07-26T11:17:48.641 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 5: /lib64/libc.so.6(+0x2871b) [0x7f869b82871b] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 6: /lib64/libc.so.6(+0x37386) [0x7f869b837386] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d9146d6b9] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d9162b80c] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d9156b2ca] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d9153b753] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d9153ca1b] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d91520752] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d9146ea56] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d91472637] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d91472f5e] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d9145603a] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f869c39b407] 2024-07-26T11:17:48.642 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 18: (DispatchQueue::entry()+0x60f) [0x7f869c39bcf9] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f869c4785a1] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 20: (Thread::entry_wrapper()+0x43) [0x7f869c225cdd] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 21: (Thread::_entry_func(void*)+0xd) [0x7f869c225cf9] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 22: /lib64/libc.so.6(+0x89c02) [0x7f869b889c02] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f869b90ec40] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 0> 2024-07-26T11:17:48.435+0000 7f8695600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: in thread 7f8695600640 thread_name:ms_dispatch 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f869b83e6f0] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f869b88b94c] 2024-07-26T11:17:48.643 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 3: raise() 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 4: abort() 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 5: /lib64/libc.so.6(+0x2871b) [0x7f869b82871b] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 6: /lib64/libc.so.6(+0x37386) [0x7f869b837386] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d9146d6b9] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d9162b80c] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560d9156b2ca] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560d9153b753] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d9153ca1b] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d91520752] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d9146ea56] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d91472637] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d91472f5e] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d9145603a] 2024-07-26T11:17:48.644 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f869c39b407] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 18: (DispatchQueue::entry()+0x60f) [0x7f869c39bcf9] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f869c4785a1] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 20: (Thread::entry_wrapper()+0x43) [0x7f869c225cdd] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 21: (Thread::_entry_func(void*)+0xd) [0x7f869c225cf9] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 22: /lib64/libc.so.6(+0x89c02) [0x7f869b889c02] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f869b90ec40] 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110206]: 2024-07-26T11:17:48.645 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:48 smithi144 podman[110244]: 2024-07-26 11:17:48.633660709 +0000 UTC m=+0.040556513 container died 92fe937e085d75c9bdc05e8de9cc34ab3ba3c3398a8ed0c8e1a84638e8a0fc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724) 2024-07-26T11:17:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:48 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:rejoin 2024-07-26T11:17:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:48 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:48 smithi160 ceph-mon[36715]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:17:49.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:49 smithi144 ceph-mon[32129]: pgmap v974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T11:17:49.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:49 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:49.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:49 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:active 2024-07-26T11:17:49.581 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:49 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:49 smithi183 ceph-mon[37273]: pgmap v974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T11:17:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:49 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:49 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:active 2024-07-26T11:17:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:49 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:49 smithi160 ceph-mon[36715]: pgmap v974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T11:17:49.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:49 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:17:49.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:49 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/3711904182,v1:172.21.15.144:6835/3711904182] up:active 2024-07-26T11:17:49.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:49 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:17:49.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:49 smithi144 podman[110244]: 2024-07-26 11:17:49.5786461 +0000 UTC m=+0.985541943 container remove 92fe937e085d75c9bdc05e8de9cc34ab3ba3c3398a8ed0c8e1a84638e8a0fc20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T11:17:49.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:49 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:17:50.392 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:50 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:17:50.392 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:17:50 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.285s CPU time. 2024-07-26T11:17:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:50 smithi144 ceph-mon[32129]: pgmap v975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:17:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:50 smithi160 ceph-mon[36715]: pgmap v975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:17:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:50 smithi183 ceph-mon[37273]: pgmap v975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:17:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 714.541100 secs 2024-07-26T11:17:51.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 714.541100 secs 2024-07-26T11:17:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 714.541100 secs 2024-07-26T11:17:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:52 smithi144 ceph-mon[32129]: pgmap v976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:52 smithi160 ceph-mon[36715]: pgmap v976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:52 smithi183 ceph-mon[37273]: pgmap v976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:55 smithi144 ceph-mon[32129]: pgmap v977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:55 smithi160 ceph-mon[36715]: pgmap v977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:55 smithi183 ceph-mon[37273]: pgmap v977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:17:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:56 smithi144 ceph-mon[32129]: pgmap v978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 187 B/s wr, 6 op/s 2024-07-26T11:17:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:56 smithi144 ceph-mon[32129]: pgmap v979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:17:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:17:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 719.541184 secs 2024-07-26T11:17:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:56 smithi160 ceph-mon[36715]: pgmap v978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 187 B/s wr, 6 op/s 2024-07-26T11:17:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:56 smithi160 ceph-mon[36715]: pgmap v979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:17:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:17:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 719.541184 secs 2024-07-26T11:17:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:56 smithi183 ceph-mon[37273]: pgmap v978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 187 B/s wr, 6 op/s 2024-07-26T11:17:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:56 smithi183 ceph-mon[37273]: pgmap v979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:17:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:17:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 719.541184 secs 2024-07-26T11:17:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:57.205+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:57.206+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:57.206+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:57.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:17:57.207+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:17:58 smithi183 ceph-mon[37273]: pgmap v980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:17:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:17:58 smithi144 ceph-mon[32129]: pgmap v980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:17:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:17:58 smithi160 ceph-mon[36715]: pgmap v980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 223 B/s wr, 6 op/s 2024-07-26T11:18:00.626 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:00 smithi144 ceph-mon[32129]: pgmap v981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 111 B/s wr, 5 op/s 2024-07-26T11:18:00.626 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:18:00.626 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 systemd[1]: Stopped Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:00.626 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.285s CPU time. 2024-07-26T11:18:00.626 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 systemd[1]: Starting Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:18:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:00 smithi183 ceph-mon[37273]: pgmap v981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 111 B/s wr, 5 op/s 2024-07-26T11:18:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:00 smithi160 ceph-mon[36715]: pgmap v981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 111 B/s wr, 5 op/s 2024-07-26T11:18:00.959 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 podman[110464]: 2024-07-26 11:18:00.529103149 +0000 UTC m=+0.032318009 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:18:00.960 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 podman[110464]: 2024-07-26 11:18:00.654538267 +0000 UTC m=+0.157753138 container create 2767a78b64a9c4b44068fc4a7ebf7006ad663e06929d08b1e634728ea9045094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-07-26T11:18:01.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 podman[110464]: 2024-07-26 11:18:00.957793305 +0000 UTC m=+0.461008234 container init 2767a78b64a9c4b44068fc4a7ebf7006ad663e06929d08b1e634728ea9045094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:18:01.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:00 smithi144 podman[110464]: 2024-07-26 11:18:00.968582115 +0000 UTC m=+0.471796993 container start 2767a78b64a9c4b44068fc4a7ebf7006ad663e06929d08b1e634728ea9045094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:18:01.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:01 smithi144 bash[110464]: 2767a78b64a9c4b44068fc4a7ebf7006ad663e06929d08b1e634728ea9045094 2024-07-26T11:18:01.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mds[110483]: -- 172.21.15.144:0/2084266473 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adb9f8d860 con 0x55adbe12fa80 2024-07-26T11:18:01.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: starting mds.d at 2024-07-26T11:18:01.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:01 smithi144 systemd[1]: Started Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 724.541321 secs 2024-07-26T11:18:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: slow request 724.541320 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:18:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: daemon mds.d restarted 2024-07-26T11:18:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: osdmap e134: 12 total, 12 up, 12 in 2024-07-26T11:18:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:boot 2024-07-26T11:18:01.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:01.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:01.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:01.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:01 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 724.541321 secs 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: slow request 724.541320 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: daemon mds.d restarted 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: osdmap e134: 12 total, 12 up, 12 in 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:boot 2024-07-26T11:18:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:01.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:01 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 724.541321 secs 2024-07-26T11:18:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: slow request 724.541320 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: daemon mds.d restarted 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: osdmap e134: 12 total, 12 up, 12 in 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:boot 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:01 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:02 smithi183 ceph-mon[37273]: pgmap v983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:18:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:02 smithi144 ceph-mon[32129]: pgmap v983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:18:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:02 smithi160 ceph-mon[36715]: pgmap v983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:18:03.594 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:03 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:resolve 2024-07-26T11:18:03.595 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:03 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:03 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:resolve 2024-07-26T11:18:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:03 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:03 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:resolve 2024-07-26T11:18:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:03 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:04 smithi183 ceph-mon[37273]: pgmap v984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T11:18:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:04 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:reconnect 2024-07-26T11:18:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:04 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:04 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:18:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:04 smithi144 ceph-mon[32129]: pgmap v984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T11:18:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:04 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:reconnect 2024-07-26T11:18:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:04 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:04 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:18:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:04 smithi160 ceph-mon[36715]: pgmap v984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T11:18:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:04 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:reconnect 2024-07-26T11:18:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:04 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:04 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:18:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:rejoin 2024-07-26T11:18:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-mon[32129]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:05.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:18:05.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: *** Caught signal (Aborted) ** 2024-07-26T11:18:05.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: in thread 7f545f200640 thread_name:ms_dispatch 2024-07-26T11:18:05.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:05.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f546543e6f0] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f546548b94c] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 3: raise() 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 4: abort() 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 5: /lib64/libc.so.6(+0x2871b) [0x7f546542871b] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 6: /lib64/libc.so.6(+0x37386) [0x7f5465437386] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ad873fb6b9] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ad875b980c] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ad874f92ca] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ad874c9753] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ad874caa1b] 2024-07-26T11:18:05.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ad874ae752] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ad873fca56] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ad87400637] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ad87400f5e] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ad873e403a] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5465f9b407] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 18: (DispatchQueue::entry()+0x60f) [0x7f5465f9bcf9] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f54660785a1] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 20: (Thread::entry_wrapper()+0x43) [0x7f5465e25cdd] 2024-07-26T11:18:05.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5465e25cf9] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5465489c02] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f546550ec40] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.443+0000 7f545f200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: in thread 7f545f200640 thread_name:ms_dispatch 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f546543e6f0] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f546548b94c] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 3: raise() 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 4: abort() 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 5: /lib64/libc.so.6(+0x2871b) [0x7f546542871b] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 6: /lib64/libc.so.6(+0x37386) [0x7f5465437386] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ad873fb6b9] 2024-07-26T11:18:05.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ad875b980c] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ad874f92ca] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ad874c9753] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ad874caa1b] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ad874ae752] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ad873fca56] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ad87400637] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ad87400f5e] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ad873e403a] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5465f9b407] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 18: (DispatchQueue::entry()+0x60f) [0x7f5465f9bcf9] 2024-07-26T11:18:05.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f54660785a1] 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 20: (Thread::entry_wrapper()+0x43) [0x7f5465e25cdd] 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5465e25cf9] 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5465489c02] 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f546550ec40] 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 0> 2024-07-26T11:18:05.443+0000 7f545f200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: in thread 7f545f200640 thread_name:ms_dispatch 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f546543e6f0] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f546548b94c] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 3: raise() 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 4: abort() 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 5: /lib64/libc.so.6(+0x2871b) [0x7f546542871b] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 6: /lib64/libc.so.6(+0x37386) [0x7f5465437386] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ad873fb6b9] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ad875b980c] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ad874f92ca] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ad874c9753] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ad874caa1b] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ad874ae752] 2024-07-26T11:18:05.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ad873fca56] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ad87400637] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ad87400f5e] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ad873e403a] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5465f9b407] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 18: (DispatchQueue::entry()+0x60f) [0x7f5465f9bcf9] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f54660785a1] 2024-07-26T11:18:05.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 20: (Thread::entry_wrapper()+0x43) [0x7f5465e25cdd] 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5465e25cf9] 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5465489c02] 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f546550ec40] 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: -2> 2024-07-26T11:18:05.443+0000 7f545f200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: in thread 7f545f200640 thread_name:ms_dispatch 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f546543e6f0] 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f546548b94c] 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 3: raise() 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 4: abort() 2024-07-26T11:18:05.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 5: /lib64/libc.so.6(+0x2871b) [0x7f546542871b] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 6: /lib64/libc.so.6(+0x37386) [0x7f5465437386] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ad873fb6b9] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ad875b980c] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ad874f92ca] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ad874c9753] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ad874caa1b] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ad874ae752] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ad873fca56] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ad87400637] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ad87400f5e] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ad873e403a] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5465f9b407] 2024-07-26T11:18:05.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 18: (DispatchQueue::entry()+0x60f) [0x7f5465f9bcf9] 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f54660785a1] 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 20: (Thread::entry_wrapper()+0x43) [0x7f5465e25cdd] 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5465e25cf9] 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5465489c02] 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f546550ec40] 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110479]: 2024-07-26T11:18:05.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:05 smithi144 podman[110517]: 2024-07-26 11:18:05.595618833 +0000 UTC m=+0.034226707 container died 2767a78b64a9c4b44068fc4a7ebf7006ad663e06929d08b1e634728ea9045094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T11:18:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:05 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:rejoin 2024-07-26T11:18:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:05 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:05 smithi160 ceph-mon[36715]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:05 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:rejoin 2024-07-26T11:18:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:05 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:05 smithi183 ceph-mon[37273]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:06 smithi160 ceph-mon[36715]: pgmap v985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:18:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:06 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:06 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:active 2024-07-26T11:18:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:06 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:06 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 729.541442 secs 2024-07-26T11:18:06.712 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:06 smithi160 ceph-mon[36715]: slow request 724.680631 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:06 smithi144 podman[110517]: 2024-07-26 11:18:06.386909035 +0000 UTC m=+0.825516939 container remove 2767a78b64a9c4b44068fc4a7ebf7006ad663e06929d08b1e634728ea9045094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:06 smithi144 ceph-mon[32129]: pgmap v985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:06 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:06 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:active 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:06 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:06 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 729.541442 secs 2024-07-26T11:18:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:06 smithi144 ceph-mon[32129]: slow request 724.680631 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:18:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:06 smithi183 ceph-mon[37273]: pgmap v985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:18:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:06 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:06 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1686807856,v1:172.21.15.144:6835/1686807856] up:active 2024-07-26T11:18:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:06 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:06 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 729.541442 secs 2024-07-26T11:18:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:06 smithi183 ceph-mon[37273]: slow request 724.680631 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:18:07.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:18:07.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:06 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.226s CPU time. 2024-07-26T11:18:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:07 smithi144 ceph-mon[32129]: pgmap v986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:18:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:07 smithi160 ceph-mon[36715]: pgmap v986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:18:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:07 smithi183 ceph-mon[37273]: pgmap v986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 6 op/s 2024-07-26T11:18:08.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:18:09.243 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:10.537 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:10 smithi144 ceph-mon[32129]: pgmap v987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:10 smithi183 ceph-mon[37273]: pgmap v987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:10 smithi160 ceph-mon[36715]: pgmap v987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 -- 172.21.15.144:0/1796977935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf410f330 msgr2=0x7fadf410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 --2- 172.21.15.144:0/1796977935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf410f330 0x7fadf410f740 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fade4009900 tx=0x7fade402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 -- 172.21.15.144:0/1796977935 shutdown_connections 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 --2- 172.21.15.144:0/1796977935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410fc80 0x7fadf41164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 --2- 172.21.15.144:0/1796977935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf410f330 0x7fadf410f740 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 --2- 172.21.15.144:0/1796977935 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf41099f0 0x7fadf4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 -- 172.21.15.144:0/1796977935 >> 172.21.15.144:0/1796977935 conn(0x7fadf4078ff0 msgr2=0x7fadf4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:11.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 -- 172.21.15.144:0/1796977935 shutdown_connections 2024-07-26T11:18:11.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.472+0000 7fadfae00640 1 -- 172.21.15.144:0/1796977935 wait complete. 2024-07-26T11:18:11.476 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 734.541563 secs 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.474+0000 7fadfae00640 1 Processor -- start 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.474+0000 7fadfae00640 1 -- start start 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfae00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf41099f0 0x7fadf41a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfae00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410f330 0x7fadf41a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfae00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf410fc80 0x7fadf41a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfae00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fadf4119990 con 0x7fadf41099f0 2024-07-26T11:18:11.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfae00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fadf4119810 con 0x7fadf410fc80 2024-07-26T11:18:11.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfae00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fadf4119b10 con 0x7fadf410f330 2024-07-26T11:18:11.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadf9a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf41099f0 0x7fadf41a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadf9000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410f330 0x7fadf41a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadf9a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf41099f0 0x7fadf41a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55922/0 (socket says 172.21.15.144:55922) 2024-07-26T11:18:11.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadf9a00640 1 -- 172.21.15.144:0/3568759774 learned_addr learned my addr 172.21.15.144:0/3568759774 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:11.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.475+0000 7fadfa400640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf410fc80 0x7fadf41a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf9000640 1 -- 172.21.15.144:0/3568759774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf410fc80 msgr2=0x7fadf41a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf9000640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf410fc80 0x7fadf41a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf9000640 1 -- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf41099f0 msgr2=0x7fadf41a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf9000640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf41099f0 0x7fadf41a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf9000640 1 -- 172.21.15.144:0/3568759774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf41a6330 con 0x7fadf410f330 2024-07-26T11:18:11.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadfa400640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf410fc80 0x7fadf41a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:11.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf9000640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410f330 0x7fadf41a0960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fade4009230 tx=0x7fade4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:11.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.476+0000 7fadf2000640 1 -- 172.21.15.144:0/3568759774 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade403f070 con 0x7fadf410f330 2024-07-26T11:18:11.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.477+0000 7fadf2000640 1 -- 172.21.15.144:0/3568759774 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade4031b60 con 0x7fadf410f330 2024-07-26T11:18:11.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.477+0000 7fadf2000640 1 -- 172.21.15.144:0/3568759774 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4035030 con 0x7fadf410f330 2024-07-26T11:18:11.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.477+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf41a65c0 con 0x7fadf410f330 2024-07-26T11:18:11.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.477+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf41acd80 con 0x7fadf410f330 2024-07-26T11:18:11.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.481+0000 7fadf2000640 1 -- 172.21.15.144:0/3568759774 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fade4042070 con 0x7fadf410f330 2024-07-26T11:18:11.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.481+0000 7fae0305c740 1 Processor -- start 2024-07-26T11:18:11.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.481+0000 7fae0305c740 1 -- start start 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fae0305c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a3470000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fae0305c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 0x5645a3470540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fae0305c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a3470a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fae0305c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5645a33e6bc0 con 0x5645a30aa390 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fae0305c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5645a33d60c0 con 0x5645a30cb070 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fae0305c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5645a33da100 con 0x5645a30a21b0 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 0x5645a3470540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 0x5645a3470540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:44488/0 (socket says 172.21.15.144:44488) 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 -- 172.21.15.144:0/142350387 learned_addr learned my addr 172.21.15.144:0/142350387 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9000640 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a3470a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadfa400640 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a3470000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 -- 172.21.15.144:0/142350387 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 msgr2=0x5645a3470000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a3470000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 -- 172.21.15.144:0/142350387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 msgr2=0x5645a3470a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a3470a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9a00640 1 -- 172.21.15.144:0/142350387 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645a3136650 con 0x5645a30a21b0 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.482+0000 7fadf9000640 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a3470a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fadf2000640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faddc0b3dc0 0x7faddc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fadf9a00640 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 0x5645a3470540 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fade8009dc0 tx=0x7fade8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10b4676825106ed9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fadf2000640 1 -- 172.21.15.144:0/3568759774 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 13178+0+0 (secure 0 0 0) 0x7fade4105460 con 0x7fadf410f330 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fadfa400640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faddc0b3dc0 0x7faddc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fade2a00640 1 -- 172.21.15.144:0/142350387 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8005220 con 0x5645a30a21b0 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fade2a00640 1 -- 172.21.15.144:0/142350387 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade8004c50 con 0x5645a30a21b0 2024-07-26T11:18:11.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.483+0000 7fade2a00640 1 -- 172.21.15.144:0/142350387 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8019810 con 0x5645a30a21b0 2024-07-26T11:18:11.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 -- 172.21.15.144:0/142350387 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 msgr2=0x5645a3470540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 0x5645a3470540 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fade8009dc0 tx=0x7fade8011810 comp rx=0 tx=0).stop 2024-07-26T11:18:11.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fadfa400640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faddc0b3dc0 0x7faddc0b62d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fadec005fd0 tx=0x7fadec0096a0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:11.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 -- 172.21.15.144:0/142350387 shutdown_connections 2024-07-26T11:18:11.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a3470a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30a21b0 0x5645a3470540 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 --2- 172.21.15.144:0/142350387 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a3470000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 -- 172.21.15.144:0/142350387 >> 172.21.15.144:0/142350387 conn(0x5645a30bf460 msgr2=0x5645a346def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 -- 172.21.15.144:0/142350387 shutdown_connections 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.484+0000 7fae0305c740 1 -- 172.21.15.144:0/142350387 wait complete. 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.485+0000 7fae0305c740 1 -- start start 2024-07-26T11:18:11.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.485+0000 7fae0305c740 1 Processor -- start 2024-07-26T11:18:11.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30a21b0 0x5645a347b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a347bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a347c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5645a3239700 con 0x5645a30aa390 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5645a3372690 con 0x5645a30a21b0 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5645a347e7e0 con 0x5645a30cb070 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fae0305c740 10 client.0 _ll_register_callbacks cb 0x5645a3382140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fadf9000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a347bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fadf9000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a347bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55938/0 (socket says 172.21.15.144:55938) 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fadf9000640 1 -- 172.21.15.144:0/500953759 learned_addr learned my addr 172.21.15.144:0/500953759 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fadf9a00640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30a21b0 0x5645a347b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.486+0000 7fadfa400640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a347c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9000640 1 -- 172.21.15.144:0/500953759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 msgr2=0x5645a347c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9000640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a347c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9000640 1 -- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30a21b0 msgr2=0x5645a347b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9000640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30a21b0 0x5645a347b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9000640 1 -- 172.21.15.144:0/500953759 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645a3389830 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9a00640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30a21b0 0x5645a347b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadf9000640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a347bb20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fade4105700 tx=0x7fade402a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec99fc62b0ad20cd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadd7400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadd7400640 1 -- 172.21.15.144:0/500953759 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4049070 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fadd7400640 1 -- 172.21.15.144:0/500953759 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 13178+0+0 (secure 0 0 0) 0x7fade4101ba0 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.487+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fade8004c50 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fadd7400640 1 -- 172.21.15.144:0/500953759 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=320) ==== 24+0+0 (secure 0 0 0) 0x7fade40ca600 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 10 client.16407 fetch_fsmap learned FSMap version 320 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=320,monmap=4+}) -- 0x5645a348a410 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fadd7400640 1 -- 172.21.15.144:0/500953759 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade8004c50 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fadd7400640 1 -- 172.21.15.144:0/500953759 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 320) ==== 4131+0+0 (secure 0 0 0) 0x7fade404c060 con 0x5645a30aa390 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 10 client.16407 fetch_fsmap finished waiting for FSMap version 320 2024-07-26T11:18:11.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:18:11.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 20 client.16407 populate_metadata read hostname 'smithi144' 2024-07-26T11:18:11.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a30899f0 0x5645a348db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 4 client.16407 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:11.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.488+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645a33e8b10 con 0x5645a30899f0 2024-07-26T11:18:11.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.489+0000 7fadf9a00640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a30899f0 0x5645a348db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.489+0000 7fadf9a00640 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a30899f0 0x5645a348db30 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1b224498614258 server_cookie=a1a08912de8ec9d3 in_seq=0 out_seq=0 2024-07-26T11:18:11.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.489+0000 7fadd7400640 10 client.16407 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:11.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.490+0000 7fadd7400640 1 -- 172.21.15.144:0/500953759 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5645a33e8b10 con 0x5645a30899f0 2024-07-26T11:18:11.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.490+0000 7fadd7400640 10 client.16407 handle_command_reply: tid=0 2024-07-26T11:18:11.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.490+0000 7fae0305c740 1 client.16407 shutdown 2024-07-26T11:18:11.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 msgr2=0x5645a347bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a347bb20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fade4105700 tx=0x7fade402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 shutdown_connections 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a347c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30aa390 0x5645a347bb20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a30899f0 0x5645a348db30 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 --2- 172.21.15.144:0/500953759 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30a21b0 0x5645a347b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fadd7400640 0 client.16407 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 >> 172.21.15.144:0/500953759 conn(0x5645a30bf460 msgr2=0x5645a34731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 shutdown_connections 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 1 -- 172.21.15.144:0/500953759 wait complete. 2024-07-26T11:18:11.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.491+0000 7fae0305c740 20 client.16407 trim_cache size 0 max 16384 2024-07-26T11:18:11.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 Processor -- start 2024-07-26T11:18:11.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 -- start start 2024-07-26T11:18:11.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a346cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 0x5645a3471740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a34932f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fadf41b9120 con 0x5645a30a21b0 2024-07-26T11:18:11.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fadf41b75e0 con 0x5645a30cb070 2024-07-26T11:18:11.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fae0305c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fadf41b8a50 con 0x5645a30aa390 2024-07-26T11:18:11.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.510+0000 7fadf9000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 0x5645a3471740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 0x5645a3471740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:44502/0 (socket says 172.21.15.144:44502) 2024-07-26T11:18:11.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 -- 172.21.15.144:0/2019442285 learned_addr learned my addr 172.21.15.144:0/2019442285 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:11.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9a00640 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a346cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.513 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadfa400640 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a34932f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 -- 172.21.15.144:0/2019442285 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 msgr2=0x5645a34932f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a34932f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 -- 172.21.15.144:0/2019442285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 msgr2=0x5645a346cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a346cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 -- 172.21.15.144:0/2019442285 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645a3479ae0 con 0x5645a30aa390 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.511+0000 7fadf9000640 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 0x5645a3471740 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fade40c8100 tx=0x7fade4031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38e395b2e78eaac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.512+0000 7fade2000640 1 -- 172.21.15.144:0/2019442285 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4049030 con 0x5645a30aa390 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.512+0000 7fade2000640 1 -- 172.21.15.144:0/2019442285 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade40ca9a0 con 0x5645a30aa390 2024-07-26T11:18:11.514 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.512+0000 7fade2000640 1 -- 172.21.15.144:0/2019442285 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade40cb0f0 con 0x5645a30aa390 2024-07-26T11:18:11.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.512+0000 7fae0305c740 1 -- 172.21.15.144:0/2019442285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 msgr2=0x5645a3471740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.512+0000 7fae0305c740 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 0x5645a3471740 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fade40c8100 tx=0x7fade4031400 comp rx=0 tx=0).stop 2024-07-26T11:18:11.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 -- 172.21.15.144:0/2019442285 shutdown_connections 2024-07-26T11:18:11.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30cb070 0x5645a34932f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.515 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30aa390 0x5645a3471740 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 --2- 172.21.15.144:0/2019442285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a346cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 -- 172.21.15.144:0/2019442285 >> 172.21.15.144:0/2019442285 conn(0x5645a30bf460 msgr2=0x5645a3471030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:11.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 -- 172.21.15.144:0/2019442285 shutdown_connections 2024-07-26T11:18:11.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 -- 172.21.15.144:0/2019442285 wait complete. 2024-07-26T11:18:11.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 -- start start 2024-07-26T11:18:11.516 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.513+0000 7fae0305c740 1 Processor -- start 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a34841d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30aa390 0x5645a34816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a3481ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fadf41b8380 con 0x5645a30a21b0 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fadf41b7cb0 con 0x5645a30aa390 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5645a33e8b10 con 0x5645a30cb070 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fae0305c740 10 client.0 _ll_register_callbacks cb 0x7fade40c6d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fadf9a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a34841d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fadfa400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a3481ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fadfa400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a3481ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:44514/0 (socket says 172.21.15.144:44514) 2024-07-26T11:18:11.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fadfa400640 1 -- 172.21.15.144:0/2458617848 learned_addr learned my addr 172.21.15.144:0/2458617848 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.514+0000 7fadf9000640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30aa390 0x5645a34816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9a00640 1 -- 172.21.15.144:0/2458617848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 msgr2=0x5645a3481ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9a00640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a3481ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9a00640 1 -- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30aa390 msgr2=0x5645a34816a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9a00640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30aa390 0x5645a34816a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9a00640 1 -- 172.21.15.144:0/2458617848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645a3482430 con 0x5645a30a21b0 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadfa400640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a3481ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9000640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30aa390 0x5645a34816a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadf9a00640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a34841d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fade80097b0 tx=0x7fade80056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606ef7af28d3d0bf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadd7400640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.515+0000 7fadd7400640 1 -- 172.21.15.144:0/2458617848 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8004450 con 0x5645a30a21b0 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fadd7400640 1 -- 172.21.15.144:0/2458617848 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 13178+0+0 (secure 0 0 0) 0x7fade8004b70 con 0x5645a30a21b0 2024-07-26T11:18:11.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fade40ca9a0 con 0x5645a30a21b0 2024-07-26T11:18:11.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fadd7400640 1 -- 172.21.15.144:0/2458617848 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=320) ==== 24+0+0 (secure 0 0 0) 0x7fade803b050 con 0x5645a30a21b0 2024-07-26T11:18:11.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 10 client.16413 fetch_fsmap learned FSMap version 320 2024-07-26T11:18:11.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=320,monmap=4+}) -- 0x5645a34f62f0 con 0x5645a30a21b0 2024-07-26T11:18:11.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fadd7400640 1 -- 172.21.15.144:0/2458617848 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade40ca9a0 con 0x5645a30a21b0 2024-07-26T11:18:11.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fadd7400640 1 -- 172.21.15.144:0/2458617848 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 320) ==== 4131+0+0 (secure 0 0 0) 0x7fade8003900 con 0x5645a30a21b0 2024-07-26T11:18:11.521 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 10 client.16413 fetch_fsmap finished waiting for FSMap version 320 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 20 client.16413 populate_metadata read hostname 'smithi144' 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a305eda0 0x5645a34f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 4 client.16413 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.516+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5645a33e45b0 con 0x5645a305eda0 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.517+0000 7fadf9000640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a305eda0 0x5645a34f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.517+0000 7fadf9000640 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a305eda0 0x5645a34f8970 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d011c44f3c9da59 server_cookie=866526a59c0631bd in_seq=0 out_seq=0 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.517+0000 7fadd7400640 10 client.16413 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.517+0000 7fadd7400640 1 -- 172.21.15.144:0/2458617848 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5645a33e45b0 con 0x5645a305eda0 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.517+0000 7fadd7400640 10 client.16413 handle_command_reply: tid=0 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 client.16413 shutdown 2024-07-26T11:18:11.522 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 msgr2=0x5645a34841d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a34841d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fade80097b0 tx=0x7fade80056c0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 shutdown_connections 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5645a30cb070 0x5645a3481ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5645a305eda0 0x5645a34f8970 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5645a30aa390 0x5645a34816a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 --2- 172.21.15.144:0/2458617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5645a30a21b0 0x5645a34841d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 >> 172.21.15.144:0/2458617848 conn(0x5645a30bf460 msgr2=0x5645a346ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fadd7400640 0 client.16413 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 shutdown_connections 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 1 -- 172.21.15.144:0/2458617848 wait complete. 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.518+0000 7fae0305c740 20 client.16413 trim_cache size 0 max 16384 2024-07-26T11:18:11.523 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:18:11.524 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faddc0b3dc0 msgr2=0x7faddc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faddc0b3dc0 0x7faddc0b62d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fadec005fd0 tx=0x7fadec0096a0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410f330 msgr2=0x7fadf41a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410f330 0x7fadf41a0960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fade4009230 tx=0x7fade4003c10 comp rx=0 tx=0).stop 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 shutdown_connections 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7faddc0b3dc0 0x7faddc0b62d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadf410fc80 0x7fadf41a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf410f330 0x7fadf41a0960 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 --2- 172.21.15.144:0/3568759774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadf41099f0 0x7fadf41a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 >> 172.21.15.144:0/3568759774 conn(0x7fadf4078ff0 msgr2=0x7fadf40798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:11.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 shutdown_connections 2024-07-26T11:18:11.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:11.519+0000 7fadfae00640 1 -- 172.21.15.144:0/3568759774 wait complete. 2024-07-26T11:18:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 734.541563 secs 2024-07-26T11:18:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 734.541563 secs 2024-07-26T11:18:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:12.224+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:12.225+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:12.225+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:12.225+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:12.225+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:12.445 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:18:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:12 smithi183 ceph-mon[37273]: pgmap v988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 6 op/s 2024-07-26T11:18:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:12 smithi144 ceph-mon[32129]: pgmap v988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 6 op/s 2024-07-26T11:18:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:12 smithi160 ceph-mon[36715]: pgmap v988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 201 B/s wr, 6 op/s 2024-07-26T11:18:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:14 smithi183 ceph-mon[37273]: pgmap v989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:14 smithi144 ceph-mon[32129]: pgmap v989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:14 smithi160 ceph-mon[36715]: pgmap v989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:16 smithi183 ceph-mon[37273]: pgmap v990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:18:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 739.541656 secs 2024-07-26T11:18:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:16 smithi144 ceph-mon[32129]: pgmap v990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:18:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 739.541656 secs 2024-07-26T11:18:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:16 smithi160 ceph-mon[36715]: pgmap v990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 6 op/s 2024-07-26T11:18:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 739.541656 secs 2024-07-26T11:18:17.362 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:16 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:18:17.362 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:16 smithi144 systemd[1]: Stopped Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:17.363 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:16 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.226s CPU time. 2024-07-26T11:18:17.363 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:16 smithi144 systemd[1]: Starting Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:18:17.659 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 podman[110797]: 2024-07-26 11:18:17.26685938 +0000 UTC m=+0.035459159 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:18:17.659 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 podman[110797]: 2024-07-26 11:18:17.403548158 +0000 UTC m=+0.172147911 container create 561a730e240eba3c9b1351563f3246ca205dd530106c1c8452e7cd26976a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-26T11:18:17.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 podman[110797]: 2024-07-26 11:18:17.657519341 +0000 UTC m=+0.426119151 container init 561a730e240eba3c9b1351563f3246ca205dd530106c1c8452e7cd26976a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T11:18:17.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 podman[110797]: 2024-07-26 11:18:17.667756295 +0000 UTC m=+0.436356028 container start 561a730e240eba3c9b1351563f3246ca205dd530106c1c8452e7cd26976a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:18:17.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 bash[110797]: 561a730e240eba3c9b1351563f3246ca205dd530106c1c8452e7cd26976a4478 2024-07-26T11:18:17.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 systemd[1]: Started Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:17.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 ceph-mds[110816]: -- 172.21.15.144:0/1782030152 <== mon.0 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5614f7547860 con 0x5614fb6e9180 2024-07-26T11:18:17.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:17 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: starting mds.d at 2024-07-26T11:18:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: pgmap v991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 3 op/s 2024-07-26T11:18:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: daemon mds.d restarted 2024-07-26T11:18:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: osdmap e135: 12 total, 12 up, 12 in 2024-07-26T11:18:18.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:boot 2024-07-26T11:18:18.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:18.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:18.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:18 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: pgmap v991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 3 op/s 2024-07-26T11:18:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: daemon mds.d restarted 2024-07-26T11:18:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: osdmap e135: 12 total, 12 up, 12 in 2024-07-26T11:18:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:boot 2024-07-26T11:18:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:18.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:18 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: pgmap v991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 3 op/s 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: daemon mds.d restarted 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: osdmap e135: 12 total, 12 up, 12 in 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:boot 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:18.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:18 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:20.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:20 smithi144 ceph-mon[32129]: pgmap v993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4 op/s 2024-07-26T11:18:20.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:20 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:resolve 2024-07-26T11:18:20.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:20 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:20 smithi183 ceph-mon[37273]: pgmap v993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4 op/s 2024-07-26T11:18:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:20 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:resolve 2024-07-26T11:18:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:20 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:20 smithi160 ceph-mon[36715]: pgmap v993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4 op/s 2024-07-26T11:18:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:20 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:resolve 2024-07-26T11:18:20.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:20 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:21.694 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:21 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:reconnect 2024-07-26T11:18:21.694 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:21 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:21.694 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:21 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:18:21.694 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 744.541728 secs 2024-07-26T11:18:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:21 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:reconnect 2024-07-26T11:18:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:21 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:21 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:18:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 744.541728 secs 2024-07-26T11:18:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:21 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:reconnect 2024-07-26T11:18:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:21 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:21 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.00999983 2024-07-26T11:18:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 744.541728 secs 2024-07-26T11:18:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-mon[32129]: pgmap v994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:18:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:rejoin 2024-07-26T11:18:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-mon[32129]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:22.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:18:22.721 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: *** Caught signal (Aborted) ** 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: in thread 7fd81b400640 thread_name:ms_dispatch 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd82163e6f0] 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd82168b94c] 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 3: raise() 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 4: abort() 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd82162871b] 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 6: /lib64/libc.so.6(+0x37386) [0x7fd821637386] 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614ceaa96b9] 2024-07-26T11:18:22.722 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614cec6780c] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5614ceba72ca] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614ceb77753] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5614ceb78a1b] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614ceb5c752] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5614ceaaaa56] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5614ceaae637] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5614ceaaef5e] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5614cea9203a] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd82219b407] 2024-07-26T11:18:22.723 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 18: (DispatchQueue::entry()+0x60f) [0x7fd82219bcf9] 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8222785a1] 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 20: (Thread::entry_wrapper()+0x43) [0x7fd822025cdd] 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd822025cf9] 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd821689c02] 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd82170ec40] 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.454+0000 7fd81b400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:22.724 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: in thread 7fd81b400640 thread_name:ms_dispatch 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd82163e6f0] 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd82168b94c] 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 3: raise() 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 4: abort() 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd82162871b] 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 6: /lib64/libc.so.6(+0x37386) [0x7fd821637386] 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614ceaa96b9] 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614cec6780c] 2024-07-26T11:18:22.725 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5614ceba72ca] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614ceb77753] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5614ceb78a1b] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614ceb5c752] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5614ceaaaa56] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5614ceaae637] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5614ceaaef5e] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5614cea9203a] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd82219b407] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 18: (DispatchQueue::entry()+0x60f) [0x7fd82219bcf9] 2024-07-26T11:18:22.726 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8222785a1] 2024-07-26T11:18:22.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 20: (Thread::entry_wrapper()+0x43) [0x7fd822025cdd] 2024-07-26T11:18:22.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd822025cf9] 2024-07-26T11:18:22.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd821689c02] 2024-07-26T11:18:22.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd82170ec40] 2024-07-26T11:18:22.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:22.727 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 0> 2024-07-26T11:18:22.454+0000 7fd81b400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: in thread 7fd81b400640 thread_name:ms_dispatch 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd82163e6f0] 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd82168b94c] 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 3: raise() 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 4: abort() 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd82162871b] 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 6: /lib64/libc.so.6(+0x37386) [0x7fd821637386] 2024-07-26T11:18:22.728 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614ceaa96b9] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614cec6780c] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5614ceba72ca] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614ceb77753] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5614ceb78a1b] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614ceb5c752] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5614ceaaaa56] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5614ceaae637] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5614ceaaef5e] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5614cea9203a] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd82219b407] 2024-07-26T11:18:22.729 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 18: (DispatchQueue::entry()+0x60f) [0x7fd82219bcf9] 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8222785a1] 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 20: (Thread::entry_wrapper()+0x43) [0x7fd822025cdd] 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd822025cf9] 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd821689c02] 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd82170ec40] 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 0> 2024-07-26T11:18:22.454+0000 7fd81b400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: in thread 7fd81b400640 thread_name:ms_dispatch 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:22.730 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd82163e6f0] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd82168b94c] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 3: raise() 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 4: abort() 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd82162871b] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 6: /lib64/libc.so.6(+0x37386) [0x7fd821637386] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614ceaa96b9] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614cec6780c] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5614ceba72ca] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614ceb77753] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5614ceb78a1b] 2024-07-26T11:18:22.731 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614ceb5c752] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5614ceaaaa56] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5614ceaae637] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5614ceaaef5e] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5614cea9203a] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd82219b407] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 18: (DispatchQueue::entry()+0x60f) [0x7fd82219bcf9] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8222785a1] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 20: (Thread::entry_wrapper()+0x43) [0x7fd822025cdd] 2024-07-26T11:18:22.732 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd822025cf9] 2024-07-26T11:18:22.733 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd821689c02] 2024-07-26T11:18:22.733 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd82170ec40] 2024-07-26T11:18:22.733 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:22.733 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110812]: 2024-07-26T11:18:22.733 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:22 smithi144 podman[110851]: 2024-07-26 11:18:22.621073183 +0000 UTC m=+0.035128530 container died 561a730e240eba3c9b1351563f3246ca205dd530106c1c8452e7cd26976a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True) 2024-07-26T11:18:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:22 smithi160 ceph-mon[36715]: pgmap v994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:18:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:22 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:rejoin 2024-07-26T11:18:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:22 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:22 smithi160 ceph-mon[36715]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:22 smithi183 ceph-mon[37273]: pgmap v994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:18:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:22 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:rejoin 2024-07-26T11:18:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:22 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:22 smithi183 ceph-mon[37273]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:23.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:23 smithi144 podman[110851]: 2024-07-26 11:18:23.326804543 +0000 UTC m=+0.740859905 container remove 561a730e240eba3c9b1351563f3246ca205dd530106c1c8452e7cd26976a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T11:18:23.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:23 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:18:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:23 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:23 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:active 2024-07-26T11:18:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:23 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:23 smithi144 ceph-mon[32129]: pgmap v995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:18:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:23 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:23 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:active 2024-07-26T11:18:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:23 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:23 smithi160 ceph-mon[36715]: pgmap v995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:18:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:23 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:23 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/1780573619,v1:172.21.15.144:6835/1780573619] up:active 2024-07-26T11:18:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:23 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:23 smithi183 ceph-mon[37273]: pgmap v995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5 op/s 2024-07-26T11:18:24.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:23 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:18:24.220 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:23 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.183s CPU time. 2024-07-26T11:18:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:26 smithi183 ceph-mon[37273]: pgmap v996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 5 op/s 2024-07-26T11:18:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 749.541827 secs 2024-07-26T11:18:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:26 smithi144 ceph-mon[32129]: pgmap v996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 5 op/s 2024-07-26T11:18:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 749.541827 secs 2024-07-26T11:18:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:26 smithi160 ceph-mon[36715]: pgmap v996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 5 op/s 2024-07-26T11:18:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 749.541827 secs 2024-07-26T11:18:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:27.240+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:27.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:27.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:27.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:27.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:28 smithi183 ceph-mon[37273]: pgmap v997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:28 smithi144 ceph-mon[32129]: pgmap v997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:28 smithi160 ceph-mon[36715]: pgmap v997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:30 smithi183 ceph-mon[37273]: pgmap v998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 178 B/s wr, 6 op/s 2024-07-26T11:18:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:30 smithi144 ceph-mon[32129]: pgmap v998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 178 B/s wr, 6 op/s 2024-07-26T11:18:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:30 smithi160 ceph-mon[36715]: pgmap v998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 178 B/s wr, 6 op/s 2024-07-26T11:18:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 754.541899 secs 2024-07-26T11:18:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 754.541899 secs 2024-07-26T11:18:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 754.541899 secs 2024-07-26T11:18:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:32 smithi183 ceph-mon[37273]: pgmap v999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:18:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:32 smithi144 ceph-mon[32129]: pgmap v999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:18:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:32 smithi160 ceph-mon[36715]: pgmap v999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-07-26T11:18:34.368 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:33 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:18:34.369 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:33 smithi144 systemd[1]: Stopped Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:34.369 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:33 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.183s CPU time. 2024-07-26T11:18:34.369 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:33 smithi144 systemd[1]: Starting Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968... 2024-07-26T11:18:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:34 smithi183 ceph-mon[37273]: pgmap v1000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 3 op/s 2024-07-26T11:18:34.699 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 podman[110939]: 2024-07-26 11:18:34.297239293 +0000 UTC m=+0.034204891 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:18:34.699 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 podman[110939]: 2024-07-26 11:18:34.419165308 +0000 UTC m=+0.156130932 container create 52d61c851b0fff6daa647377b515e343714d15bc2589dfac76856a302cdd5dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:18:34.700 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:34 smithi144 ceph-mon[32129]: pgmap v1000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 3 op/s 2024-07-26T11:18:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:34 smithi160 ceph-mon[36715]: pgmap v1000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 3 op/s 2024-07-26T11:18:34.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 podman[110939]: 2024-07-26 11:18:34.697552987 +0000 UTC m=+0.434518644 container init 52d61c851b0fff6daa647377b515e343714d15bc2589dfac76856a302cdd5dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-07-26T11:18:34.970 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 podman[110939]: 2024-07-26 11:18:34.707492746 +0000 UTC m=+0.444458343 container start 52d61c851b0fff6daa647377b515e343714d15bc2589dfac76856a302cdd5dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:18:34.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 bash[110939]: 52d61c851b0fff6daa647377b515e343714d15bc2589dfac76856a302cdd5dfa 2024-07-26T11:18:34.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 systemd[1]: Started Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:34.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 ceph-mds[110958]: -- 172.21.15.144:0/1846885083 <== mon.2 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab39911860 con 0x55ab3dab3a80 2024-07-26T11:18:34.971 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:34 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: starting mds.d at 2024-07-26T11:18:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: daemon mds.d restarted 2024-07-26T11:18:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: osdmap e136: 12 total, 12 up, 12 in 2024-07-26T11:18:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:boot 2024-07-26T11:18:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:35.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:35.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:35 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: daemon mds.d restarted 2024-07-26T11:18:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:35.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: osdmap e136: 12 total, 12 up, 12 in 2024-07-26T11:18:35.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:boot 2024-07-26T11:18:35.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:35.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:35.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:35 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: daemon mds.d restarted 2024-07-26T11:18:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:18:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:18:35.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: osdmap e136: 12 total, 12 up, 12 in 2024-07-26T11:18:35.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:boot 2024-07-26T11:18:35.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:18:35.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: fsmap cephfs:2/3 {0=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:18:35.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:35 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:replay,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:36 smithi183 ceph-mon[37273]: pgmap v1002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-07-26T11:18:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 759.542021 secs 2024-07-26T11:18:36.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:36 smithi160 ceph-mon[36715]: pgmap v1002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-07-26T11:18:36.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 759.542021 secs 2024-07-26T11:18:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:36 smithi144 ceph-mon[32129]: pgmap v1002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-07-26T11:18:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 759.542021 secs 2024-07-26T11:18:37.646 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:37 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:resolve 2024-07-26T11:18:37.646 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:37 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:37.646 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:37 smithi144 ceph-mon[32129]: pgmap v1003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T11:18:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:37 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:resolve 2024-07-26T11:18:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:37 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:37.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:37 smithi160 ceph-mon[36715]: pgmap v1003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T11:18:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:37 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:resolve 2024-07-26T11:18:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:37 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:resolve,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:37 smithi183 ceph-mon[37273]: pgmap v1003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T11:18:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:38 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:reconnect 2024-07-26T11:18:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:38 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:38 smithi144 ceph-mon[32129]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:18:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:38 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:reconnect 2024-07-26T11:18:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:38 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:38 smithi160 ceph-mon[36715]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:18:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:38 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:reconnect 2024-07-26T11:18:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:38 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:reconnect,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:38 smithi183 ceph-mon[37273]: reconnect by client.15201 192.168.0.1:0/3243981714 after 0.0109998 2024-07-26T11:18:39.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:rejoin 2024-07-26T11:18:39.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-mon[32129]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:39.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-mon[32129]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:39.600 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-mon[32129]: pgmap v1004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:18:39.600 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: *** Caught signal (Aborted) ** 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: in thread 7f0dbd000640 thread_name:ms_dispatch 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0dc323e6f0] 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0dc328b94c] 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 3: raise() 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 4: abort() 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0dc322871b] 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 6: /lib64/libc.so.6(+0x37386) [0x7f0dc3237386] 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab131526b9] 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab1331080c] 2024-07-26T11:18:39.601 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab132502ca] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab13220753] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab13221a1b] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab13205752] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab13153a56] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab13157637] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab13157f5e] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab1313b03a] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0dc3d9b407] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 18: (DispatchQueue::entry()+0x60f) [0x7f0dc3d9bcf9] 2024-07-26T11:18:39.602 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0dc3e785a1] 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 20: (Thread::entry_wrapper()+0x43) [0x7f0dc3c25cdd] 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0dc3c25cf9] 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0dc3289c02] 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0dc330ec40] 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.449+0000 7f0dbd000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: in thread 7f0dbd000640 thread_name:ms_dispatch 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0dc323e6f0] 2024-07-26T11:18:39.606 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0dc328b94c] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 3: raise() 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 4: abort() 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0dc322871b] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 6: /lib64/libc.so.6(+0x37386) [0x7f0dc3237386] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab131526b9] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab1331080c] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab132502ca] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab13220753] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab13221a1b] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab13205752] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab13153a56] 2024-07-26T11:18:39.607 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab13157637] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab13157f5e] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab1313b03a] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0dc3d9b407] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 18: (DispatchQueue::entry()+0x60f) [0x7f0dc3d9bcf9] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0dc3e785a1] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 20: (Thread::entry_wrapper()+0x43) [0x7f0dc3c25cdd] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0dc3c25cf9] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0dc3289c02] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0dc330ec40] 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.608 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 0> 2024-07-26T11:18:39.449+0000 7f0dbd000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: in thread 7f0dbd000640 thread_name:ms_dispatch 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0dc323e6f0] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0dc328b94c] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 3: raise() 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 4: abort() 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0dc322871b] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 6: /lib64/libc.so.6(+0x37386) [0x7f0dc3237386] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab131526b9] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab1331080c] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab132502ca] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab13220753] 2024-07-26T11:18:39.609 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab13221a1b] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab13205752] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab13153a56] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab13157637] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab13157f5e] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab1313b03a] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0dc3d9b407] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 18: (DispatchQueue::entry()+0x60f) [0x7f0dc3d9bcf9] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0dc3e785a1] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 20: (Thread::entry_wrapper()+0x43) [0x7f0dc3c25cdd] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0dc3c25cf9] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0dc3289c02] 2024-07-26T11:18:39.610 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0dc330ec40] 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 0> 2024-07-26T11:18:39.449+0000 7f0dbd000640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: in thread 7f0dbd000640 thread_name:ms_dispatch 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0dc323e6f0] 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0dc328b94c] 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 3: raise() 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 4: abort() 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0dc322871b] 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 6: /lib64/libc.so.6(+0x37386) [0x7f0dc3237386] 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ab131526b9] 2024-07-26T11:18:39.611 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ab1331080c] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ab132502ca] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ab13220753] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ab13221a1b] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ab13205752] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ab13153a56] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ab13157637] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ab13157f5e] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ab1313b03a] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0dc3d9b407] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 18: (DispatchQueue::entry()+0x60f) [0x7f0dc3d9bcf9] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0dc3e785a1] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 20: (Thread::entry_wrapper()+0x43) [0x7f0dc3c25cdd] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0dc3c25cf9] 2024-07-26T11:18:39.612 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0dc3289c02] 2024-07-26T11:18:39.613 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0dc330ec40] 2024-07-26T11:18:39.613 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:18:39.613 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d[110954]: 2024-07-26T11:18:39.613 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:39 smithi144 podman[111004]: 2024-07-26 11:18:39.598281652 +0000 UTC m=+0.034864148 container died 52d61c851b0fff6daa647377b515e343714d15bc2589dfac76856a302cdd5dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T11:18:39.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:39 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:rejoin 2024-07-26T11:18:39.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:39 smithi160 ceph-mon[36715]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:39.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:39 smithi160 ceph-mon[36715]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:39.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:39 smithi160 ceph-mon[36715]: pgmap v1004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:18:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:39 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:rejoin 2024-07-26T11:18:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:39 smithi183 ceph-mon[37273]: fsmap cephfs:3/3 {0=h=up:active,1=d=up:rejoin,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:39 smithi183 ceph-mon[37273]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:18:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:39 smithi183 ceph-mon[37273]: pgmap v1004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 102 B/s wr, 5 op/s 2024-07-26T11:18:40.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:40 smithi144 podman[111004]: 2024-07-26 11:18:40.576752457 +0000 UTC m=+1.013334982 container remove 52d61c851b0fff6daa647377b515e343714d15bc2589dfac76856a302cdd5dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:18:40.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:40 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:18:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:40 smithi144 ceph-mon[32129]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:40 smithi144 ceph-mon[32129]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:active 2024-07-26T11:18:40.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:40 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:40 smithi160 ceph-mon[36715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:40 smithi160 ceph-mon[36715]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:active 2024-07-26T11:18:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:40 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:40 smithi183 ceph-mon[37273]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:18:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:40 smithi183 ceph-mon[37273]: mds.? [v2:172.21.15.144:6834/4145931333,v1:172.21.15.144:6835/4145931333] up:active 2024-07-26T11:18:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:40 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active,2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:41.428 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:41 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:18:41.428 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:41 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.116s CPU time. 2024-07-26T11:18:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 764.542092 secs 2024-07-26T11:18:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:41 smithi144 ceph-mon[32129]: pgmap v1005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 5 op/s 2024-07-26T11:18:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 764.542092 secs 2024-07-26T11:18:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:41 smithi160 ceph-mon[36715]: pgmap v1005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 5 op/s 2024-07-26T11:18:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 764.542092 secs 2024-07-26T11:18:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:41 smithi183 ceph-mon[37273]: pgmap v1005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 5 op/s 2024-07-26T11:18:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:42.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:42.257+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:42.257+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:42.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:42.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:42.447 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:18:42.920 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:44.550 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:44 smithi144 ceph-mon[32129]: pgmap v1006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:44 smithi183 ceph-mon[37273]: pgmap v1006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:44 smithi160 ceph-mon[36715]: pgmap v1006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 204 B/s wr, 6 op/s 2024-07-26T11:18:45.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 -- 172.21.15.144:0/3475735779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa810af10 msgr2=0x7fcfa810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 --2- 172.21.15.144:0/3475735779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa810af10 0x7fcfa810b320 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009860 tx=0x7fcf9402a340 comp rx=0 tx=0).stop 2024-07-26T11:18:45.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 -- 172.21.15.144:0/3475735779 shutdown_connections 2024-07-26T11:18:45.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 --2- 172.21.15.144:0/3475735779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 0x7fcfa8112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.085 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 --2- 172.21.15.144:0/3475735779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa810af10 0x7fcfa810b320 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 --2- 172.21.15.144:0/3475735779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfa81055d0 0x7fcfa81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.082+0000 7fcfae200640 1 -- 172.21.15.144:0/3475735779 >> 172.21.15.144:0/3475735779 conn(0x7fcfa8101040 msgr2=0x7fcfa8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:45.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.083+0000 7fcfae200640 1 -- 172.21.15.144:0/3475735779 shutdown_connections 2024-07-26T11:18:45.086 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.083+0000 7fcfae200640 1 -- 172.21.15.144:0/3475735779 wait complete. 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.084+0000 7fcfae200640 1 Processor -- start 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.084+0000 7fcfae200640 1 -- start start 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.084+0000 7fcfae200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 0x7fcfa819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.084+0000 7fcfae200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfa810af10 0x7fcfa819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfae200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 0x7fcfa81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfae200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81154d0 con 0x7fcfa810b860 2024-07-26T11:18:45.088 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfae200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa8115350 con 0x7fcfa81055d0 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfae200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa8115650 con 0x7fcfa810af10 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 0x7fcfa819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 0x7fcfa819bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:55546/0 (socket says 172.21.15.144:55546) 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 0x7fcfa81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 -- 172.21.15.144:0/1813423806 learned_addr learned my addr 172.21.15.144:0/1813423806 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfa810af10 0x7fcfa819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 -- 172.21.15.144:0/1813423806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfa810af10 msgr2=0x7fcfa819c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfa810af10 0x7fcfa819c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 -- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 msgr2=0x7fcfa81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 0x7fcfa81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcface00640 1 -- 172.21.15.144:0/1813423806 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa81a1e90 con 0x7fcfa81055d0 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.085+0000 7fcfad800640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 0x7fcfa81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.086+0000 7fcface00640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 0x7fcfa819bf20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00d950 tx=0x7fcf9c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:45.089 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.086+0000 7fcfa5600640 1 -- 172.21.15.144:0/1813423806 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c017030 con 0x7fcfa81055d0 2024-07-26T11:18:45.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.086+0000 7fcfa5600640 1 -- 172.21.15.144:0/1813423806 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf9c004020 con 0x7fcfa81055d0 2024-07-26T11:18:45.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.086+0000 7fcfa5600640 1 -- 172.21.15.144:0/1813423806 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0047a0 con 0x7fcfa81055d0 2024-07-26T11:18:45.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.086+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa81a2180 con 0x7fcfa81055d0 2024-07-26T11:18:45.090 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.086+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa81a88e0 con 0x7fcfa81055d0 2024-07-26T11:18:45.092 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.090+0000 7fcfb649e740 1 Processor -- start 2024-07-26T11:18:45.092 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 -- start start 2024-07-26T11:18:45.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac75c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac75c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac75cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x560eac6d2b90 con 0x560eac3b7070 2024-07-26T11:18:45.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560eac6c2090 con 0x560eac38e1b0 2024-07-26T11:18:45.093 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.091+0000 7fcfb649e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560eac6c60d0 con 0x560eac396390 2024-07-26T11:18:45.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac75c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac75c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48320/0 (socket says 172.21.15.144:48320) 2024-07-26T11:18:45.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 -- 172.21.15.144:0/67252156 learned_addr learned my addr 172.21.15.144:0/67252156 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:45.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac75cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.094 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 -- 172.21.15.144:0/67252156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 msgr2=0x560eac75cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac75cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfa5600640 1 -- 172.21.15.144:0/1813423806 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fcf9c009a30 con 0x7fcfa81055d0 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 -- 172.21.15.144:0/67252156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 msgr2=0x560eac75c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac75c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfad800640 1 -- 172.21.15.144:0/67252156 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eac422650 con 0x560eac3b7070 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.092+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac75cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:45.095 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.093+0000 7fcfad800640 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac75c1c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009fc0 tx=0x7fcfa000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6341f5097338ae server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.093+0000 7fcf9a000640 1 -- 172.21.15.144:0/67252156 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa001d070 con 0x560eac3b7070 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.093+0000 7fcf9a000640 1 -- 172.21.15.144:0/67252156 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfa0003e20 con 0x560eac3b7070 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.093+0000 7fcf9a000640 1 -- 172.21.15.144:0/67252156 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0004670 con 0x560eac3b7070 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 -- 172.21.15.144:0/67252156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 msgr2=0x560eac75c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac75c1c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009fc0 tx=0x7fcfa000ec30 comp rx=0 tx=0).stop 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfa5600640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcf900b3dc0 0x7fcf900b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 -- 172.21.15.144:0/67252156 shutdown_connections 2024-07-26T11:18:45.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac75cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac75c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 --2- 172.21.15.144:0/67252156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac75c1c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfb649e740 1 -- 172.21.15.144:0/67252156 >> 172.21.15.144:0/67252156 conn(0x560eac3ab460 msgr2=0x560eac75a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:45.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcfa5600640 1 -- 172.21.15.144:0/1813423806 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fcf9c0e4c90 con 0x7fcfa81055d0 2024-07-26T11:18:45.097 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.094+0000 7fcface00640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcf900b3dc0 0x7fcf900b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.095+0000 7fcfb649e740 1 -- 172.21.15.144:0/67252156 shutdown_connections 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.095+0000 7fcfb649e740 1 -- 172.21.15.144:0/67252156 wait complete. 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.095+0000 7fcface00640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcf900b3dc0 0x7fcf900b62d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x560eac75ae40 tx=0x7fcf9c004730 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.095+0000 7fcfb649e740 1 -- start start 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.095+0000 7fcfb649e740 1 Processor -- start 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7677a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac396390 0x560eac767ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac3b7070 0x560eac768220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x560eac525700 con 0x560eac38e1b0 2024-07-26T11:18:45.099 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560eac65e660 con 0x560eac396390 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560eac76a9a0 con 0x560eac3b7070 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfb649e740 10 client.0 _ll_register_callbacks cb 0x560eac66e110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7677a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48330/0 (socket says 172.21.15.144:48330) 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.096+0000 7fcfad800640 1 -- 172.21.15.144:0/3058480853 learned_addr learned my addr 172.21.15.144:0/3058480853 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac396390 0x560eac767ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac3b7070 0x560eac768220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 -- 172.21.15.144:0/3058480853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac3b7070 msgr2=0x560eac768220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac3b7070 0x560eac768220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 -- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac396390 msgr2=0x560eac767ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac396390 0x560eac767ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.100 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 -- 172.21.15.144:0/3058480853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eac675800 con 0x560eac38e1b0 2024-07-26T11:18:45.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac396390 0x560eac767ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:45.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcfad800640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7677a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009c20 tx=0x7fcfa000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8823a825195be0e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:45.101 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.097+0000 7fcf8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:18:45.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcf8ea00640 1 -- 172.21.15.144:0/3058480853 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa001d070 con 0x560eac38e1b0 2024-07-26T11:18:45.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcf8ea00640 1 -- 172.21.15.144:0/3058480853 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fcfa0004c50 con 0x560eac38e1b0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa0003e20 con 0x560eac38e1b0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcf8ea00640 1 -- 172.21.15.144:0/3058480853 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=332) ==== 24+0+0 (secure 0 0 0) 0x7fcfa0032050 con 0x560eac38e1b0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcfb649e740 10 client.16437 fetch_fsmap learned FSMap version 332 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=332,monmap=4+}) -- 0x560eac7765d0 con 0x560eac38e1b0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcf8ea00640 1 -- 172.21.15.144:0/3058480853 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfa0003e20 con 0x560eac38e1b0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcf8ea00640 1 -- 172.21.15.144:0/3058480853 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 332) ==== 4131+0+0 (secure 0 0 0) 0x7fcfa000b040 con 0x560eac38e1b0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcfb649e740 10 client.16437 fetch_fsmap finished waiting for FSMap version 332 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.098+0000 7fcfb649e740 10 client.16437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.099+0000 7fcfb649e740 20 client.16437 populate_metadata read hostname 'smithi144' 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.099+0000 7fcfb649e740 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac3759f0 0x560eac779dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.099+0000 7fcfb649e740 4 client.16437 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.099+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eac6d4ae0 con 0x560eac3759f0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.099+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac3759f0 0x560eac779dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.103 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.100+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac3759f0 0x560eac779dd0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44fb7c632159eaf9 server_cookie=6550299b4e7807f8 in_seq=0 out_seq=0 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.100+0000 7fcf8ea00640 10 client.16437 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.100+0000 7fcf8ea00640 1 -- 172.21.15.144:0/3058480853 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560eac6d4ae0 con 0x560eac3759f0 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.100+0000 7fcf8ea00640 10 client.16437 handle_command_reply: tid=0 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.100+0000 7fcfb649e740 1 client.16437 shutdown 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 msgr2=0x560eac7677a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7677a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009c20 tx=0x7fcfa000eee0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 shutdown_connections 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac3b7070 0x560eac768220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac3759f0 0x560eac779dd0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac396390 0x560eac767ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 --2- 172.21.15.144:0/3058480853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7677a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 >> 172.21.15.144:0/3058480853 conn(0x560eac3ab460 msgr2=0x560eac75f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcf8ea00640 0 client.16437 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 shutdown_connections 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 1 -- 172.21.15.144:0/3058480853 wait complete. 2024-07-26T11:18:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.101+0000 7fcfb649e740 20 client.16437 trim_cache size 0 max 16384 2024-07-26T11:18:45.123 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.120+0000 7fcfb649e740 1 Processor -- start 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.120+0000 7fcfb649e740 1 -- start start 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac7680d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac3b7070 0x560eac7693d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfb649e740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81b4e70 con 0x560eac38e1b0 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfb649e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81b3330 con 0x560eac3b7070 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfb649e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81b47a0 con 0x560eac396390 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac7680d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfad800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48338/0 (socket says 172.21.15.144:48338) 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.121+0000 7fcfad800640 1 -- 172.21.15.144:0/2772704595 learned_addr learned my addr 172.21.15.144:0/2772704595 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:45.124 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfad800640 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac3b7070 0x560eac7693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfa7e00640 1 -- 172.21.15.144:0/2772704595 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac3b7070 msgr2=0x560eac7693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac3b7070 0x560eac7693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfa7e00640 1 -- 172.21.15.144:0/2772704595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 msgr2=0x560eac7591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfa7e00640 1 -- 172.21.15.144:0/2772704595 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eac75f020 con 0x560eac396390 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.122+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac7680d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009780 tx=0x7fcf94005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51dfe5cd3d51f07c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.123+0000 7fcf99600640 1 -- 172.21.15.144:0/2772704595 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9404b070 con 0x560eac396390 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.123+0000 7fcf99600640 1 -- 172.21.15.144:0/2772704595 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf94030900 con 0x560eac396390 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.123+0000 7fcf99600640 1 -- 172.21.15.144:0/2772704595 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf94037040 con 0x560eac396390 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 -- 172.21.15.144:0/2772704595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 msgr2=0x560eac7680d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac7680d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009780 tx=0x7fcf94005e20 comp rx=0 tx=0).stop 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 -- 172.21.15.144:0/2772704595 shutdown_connections 2024-07-26T11:18:45.126 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac3b7070 0x560eac7693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac7680d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 --2- 172.21.15.144:0/2772704595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac38e1b0 0x560eac7591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 -- 172.21.15.144:0/2772704595 >> 172.21.15.144:0/2772704595 conn(0x560eac3ab460 msgr2=0x560eac768cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:45.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 -- 172.21.15.144:0/2772704595 shutdown_connections 2024-07-26T11:18:45.127 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.124+0000 7fcfb649e740 1 -- 172.21.15.144:0/2772704595 wait complete. 2024-07-26T11:18:45.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 -- start start 2024-07-26T11:18:45.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 Processor -- start 2024-07-26T11:18:45.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac76d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac76d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac7e05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.129 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81b40d0 con 0x560eac3b7070 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.125+0000 7fcfb649e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81b3a00 con 0x560eac38e1b0 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfb649e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560eac7e0ae0 con 0x560eac396390 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfb649e740 10 client.0 _ll_register_callbacks cb 0x7fcfa0009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfad800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac76d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac76d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac76d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57038/0 (socket says 172.21.15.144:57038) 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 -- 172.21.15.144:0/370748095 learned_addr learned my addr 172.21.15.144:0/370748095 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:18:45.130 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfad800640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac7e05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 -- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 msgr2=0x560eac76d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac76d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 -- 172.21.15.144:0/370748095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 msgr2=0x560eac7e05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac7e05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfa7e00640 1 -- 172.21.15.144:0/370748095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eac7e0c60 con 0x560eac396390 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.126+0000 7fcfad800640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac7e05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcfa7e00640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac76d7d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf94005df0 tx=0x7fcf94003870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c76899cb2af2b2b6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcf8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcf8ea00640 1 -- 172.21.15.144:0/370748095 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9404b070 con 0x560eac396390 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf94030900 con 0x560eac396390 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcf8ea00640 1 -- 172.21.15.144:0/370748095 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fcf94004340 con 0x560eac396390 2024-07-26T11:18:45.131 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcf8ea00640 1 -- 172.21.15.144:0/370748095 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=332) ==== 24+0+0 (secure 0 0 0) 0x7fcf94037070 con 0x560eac396390 2024-07-26T11:18:45.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcfb649e740 10 client.26710 fetch_fsmap learned FSMap version 332 2024-07-26T11:18:45.132 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.127+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=332,monmap=4+}) -- 0x560eac7e1ec0 con 0x560eac396390 2024-07-26T11:18:45.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcf8ea00640 1 -- 172.21.15.144:0/370748095 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf94030900 con 0x560eac396390 2024-07-26T11:18:45.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcf8ea00640 1 -- 172.21.15.144:0/370748095 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 332) ==== 4131+0+0 (secure 0 0 0) 0x7fcf94050580 con 0x560eac396390 2024-07-26T11:18:45.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfb649e740 10 client.26710 fetch_fsmap finished waiting for FSMap version 332 2024-07-26T11:18:45.133 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfb649e740 10 client.26710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfb649e740 20 client.26710 populate_metadata read hostname 'smithi144' 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfb649e740 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac34ada0 0x560eac7e44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfb649e740 4 client.26710 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560eac6d0580 con 0x560eac34ada0 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.128+0000 7fcfad800640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac34ada0 0x560eac7e44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.129+0000 7fcfad800640 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac34ada0 0x560eac7e44a0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=693492d5d057665f server_cookie=c394288059908e1a in_seq=0 out_seq=0 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.129+0000 7fcf8ea00640 10 client.26710 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.129+0000 7fcf8ea00640 1 -- 172.21.15.144:0/370748095 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560eac6d0580 con 0x560eac34ada0 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.129+0000 7fcf8ea00640 10 client.26710 handle_command_reply: tid=0 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.129+0000 7fcfb649e740 1 client.26710 shutdown 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 msgr2=0x560eac76d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.134 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac76d7d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf94005df0 tx=0x7fcf94003870 comp rx=0 tx=0).stop 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 shutdown_connections 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560eac3b7070 0x560eac7e05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560eac396390 0x560eac76d7d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x560eac34ada0 0x560eac7e44a0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 --2- 172.21.15.144:0/370748095 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560eac38e1b0 0x560eac76d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 >> 172.21.15.144:0/370748095 conn(0x560eac3ab460 msgr2=0x560eac7694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcf8ea00640 0 client.26710 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 shutdown_connections 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 1 -- 172.21.15.144:0/370748095 wait complete. 2024-07-26T11:18:45.135 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.130+0000 7fcfb649e740 20 client.26710 trim_cache size 0 max 16384 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:18:45.136 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcf900b3dc0 msgr2=0x7fcf900b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcf900b3dc0 0x7fcf900b62d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x560eac75ae40 tx=0x7fcf9c004730 comp rx=0 tx=0).stop 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 msgr2=0x7fcfa819bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 0x7fcfa819bf20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00d950 tx=0x7fcf9c00de10 comp rx=0 tx=0).stop 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 shutdown_connections 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fcf900b3dc0 0x7fcf900b62d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa810b860 0x7fcfa81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfa810af10 0x7fcfa819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 --2- 172.21.15.144:0/1813423806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfa81055d0 0x7fcfa819bf20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:45.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.131+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 >> 172.21.15.144:0/1813423806 conn(0x7fcfa8101040 msgr2=0x7fcfa8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:45.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.132+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 shutdown_connections 2024-07-26T11:18:45.138 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:18:45.132+0000 7fcfae200640 1 -- 172.21.15.144:0/1813423806 wait complete. 2024-07-26T11:18:45.974 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:18:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:46 smithi183 ceph-mon[37273]: pgmap v1007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 195 B/s wr, 6 op/s 2024-07-26T11:18:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 769.542212 secs 2024-07-26T11:18:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:46 smithi144 ceph-mon[32129]: pgmap v1007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 195 B/s wr, 6 op/s 2024-07-26T11:18:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 769.542212 secs 2024-07-26T11:18:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:46 smithi160 ceph-mon[36715]: pgmap v1007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 195 B/s wr, 6 op/s 2024-07-26T11:18:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 769.542212 secs 2024-07-26T11:18:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:48 smithi183 ceph-mon[37273]: pgmap v1008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:48 smithi144 ceph-mon[32129]: pgmap v1008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:48 smithi160 ceph-mon[36715]: pgmap v1008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:50 smithi183 ceph-mon[37273]: pgmap v1009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:50 smithi144 ceph-mon[32129]: pgmap v1009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:50 smithi160 ceph-mon[36715]: pgmap v1009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 5 op/s 2024-07-26T11:18:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 774.542342 secs 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 774.542342 secs 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:51 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:51 smithi144 systemd[1]: Stopped Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:51 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Consumed 3.116s CPU time. 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:51 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:51 smithi144 systemd[1]: ceph-493f92c6-4b3c-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:18:51.720 INFO:journalctl@ceph.mds.d.smithi144.stdout:Jul 26 11:18:51 smithi144 systemd[1]: Failed to start Ceph mds.d for 493f92c6-4b3c-11ef-bcb7-c7b262605968. 2024-07-26T11:18:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 774.542342 secs 2024-07-26T11:18:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:52 smithi183 ceph-mon[37273]: pgmap v1010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 4 op/s 2024-07-26T11:18:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:52 smithi144 ceph-mon[32129]: pgmap v1010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 4 op/s 2024-07-26T11:18:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:52 smithi160 ceph-mon[36715]: pgmap v1010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 4 op/s 2024-07-26T11:18:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:54 smithi183 ceph-mon[37273]: pgmap v1011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T11:18:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:54 smithi144 ceph-mon[32129]: pgmap v1011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T11:18:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:54 smithi160 ceph-mon[36715]: pgmap v1011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T11:18:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:55 smithi144 ceph-mon[32129]: pgmap v1012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:55 smithi160 ceph-mon[36715]: pgmap v1012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:55 smithi183 ceph-mon[37273]: pgmap v1012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 779.542411 secs 2024-07-26T11:18:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:56 smithi144 ceph-mon[32129]: fsmap cephfs:3 {0=h=up:active,1=d=up:active(laggy or crashed),2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 779.542411 secs 2024-07-26T11:18:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:56 smithi160 ceph-mon[36715]: fsmap cephfs:3 {0=h=up:active,1=d=up:active(laggy or crashed),2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:56.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:56.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 779.542411 secs 2024-07-26T11:18:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:56 smithi183 ceph-mon[37273]: fsmap cephfs:3 {0=h=up:active,1=d=up:active(laggy or crashed),2=g=up:active} 2 up:standby-replay 2024-07-26T11:18:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:57.271+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:57.272+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:57.272+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:57.272+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:18:57.272+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:57 smithi144 ceph-mon[32129]: pgmap v1013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:57 smithi144 ceph-mon[32129]: pgmap v1014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:18:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:18:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:57 smithi160 ceph-mon[36715]: pgmap v1013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:57.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:57 smithi160 ceph-mon[36715]: pgmap v1014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:18:57.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:18:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:18:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:57 smithi183 ceph-mon[37273]: pgmap v1013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:57 smithi183 ceph-mon[37273]: pgmap v1014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:18:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:18:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:00 smithi144 ceph-mon[32129]: pgmap v1015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:00 smithi160 ceph-mon[36715]: pgmap v1015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:00 smithi183 ceph-mon[37273]: pgmap v1015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 784.542473 secs 2024-07-26T11:19:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 784.542473 secs 2024-07-26T11:19:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 784.542473 secs 2024-07-26T11:19:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:02 smithi144 ceph-mon[32129]: pgmap v1016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:02 smithi160 ceph-mon[36715]: pgmap v1016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:02 smithi183 ceph-mon[37273]: pgmap v1016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:04 smithi144 ceph-mon[32129]: pgmap v1017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:04 smithi160 ceph-mon[36715]: pgmap v1017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:04 smithi183 ceph-mon[37273]: pgmap v1017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:06 smithi144 ceph-mon[32129]: pgmap v1018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:19:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 789.542599 secs 2024-07-26T11:19:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:06 smithi160 ceph-mon[36715]: pgmap v1018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:19:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 789.542599 secs 2024-07-26T11:19:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:06 smithi183 ceph-mon[37273]: pgmap v1018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:19:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 789.542599 secs 2024-07-26T11:19:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:08 smithi160 ceph-mon[36715]: pgmap v1019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:08 smithi183 ceph-mon[37273]: pgmap v1019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:08 smithi144 ceph-mon[32129]: pgmap v1019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:10 smithi160 ceph-mon[36715]: pgmap v1020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:10 smithi183 ceph-mon[37273]: pgmap v1020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:10 smithi144 ceph-mon[32129]: pgmap v1020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 794.542673 secs 2024-07-26T11:19:11.622 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 794.542673 secs 2024-07-26T11:19:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 794.542673 secs 2024-07-26T11:19:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:12.289+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:12.290+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:12.290+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:12.290+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:12.290+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:12 smithi160 ceph-mon[36715]: pgmap v1021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:12 smithi183 ceph-mon[37273]: pgmap v1021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:12.657 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:12 smithi144 ceph-mon[32129]: pgmap v1021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:14 smithi144 ceph-mon[32129]: pgmap v1022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:14 smithi160 ceph-mon[36715]: pgmap v1022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:14 smithi183 ceph-mon[37273]: pgmap v1022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:15.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:19:16.461 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:16 smithi144 ceph-mon[32129]: pgmap v1023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 799.542766 secs 2024-07-26T11:19:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:16 smithi160 ceph-mon[36715]: pgmap v1023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 799.542766 secs 2024-07-26T11:19:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:16 smithi183 ceph-mon[37273]: pgmap v1023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 799.542766 secs 2024-07-26T11:19:18.495 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:18 smithi144 ceph-mon[32129]: pgmap v1024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:18 smithi160 ceph-mon[36715]: pgmap v1024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:18 smithi183 ceph-mon[37273]: pgmap v1024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:18.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 -- 172.21.15.144:0/1430338448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 msgr2=0x7fd35410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 --2- 172.21.15.144:0/1430338448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 0x7fd35410fea0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009860 tx=0x7fd34c02a340 comp rx=0 tx=0).stop 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 -- 172.21.15.144:0/1430338448 shutdown_connections 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 --2- 172.21.15.144:0/1430338448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 0x7fd35410fea0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 --2- 172.21.15.144:0/1430338448 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd354108d20 0x7fd354109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 --2- 172.21.15.144:0/1430338448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd3541037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 -- 172.21.15.144:0/1430338448 >> 172.21.15.144:0/1430338448 conn(0x7fd3540fee70 msgr2=0x7fd3541012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 -- 172.21.15.144:0/1430338448 shutdown_connections 2024-07-26T11:19:18.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.813+0000 7fd358c00640 1 -- 172.21.15.144:0/1430338448 wait complete. 2024-07-26T11:19:18.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 Processor -- start 2024-07-26T11:19:18.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 -- start start 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd35419bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd354108d20 0x7fd35419c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 0x7fd3541a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd354113320 con 0x7fd3541033e0 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541131a0 con 0x7fd354108d20 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.815+0000 7fd358c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541134a0 con 0x7fd354109670 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd35419bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd35419bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53360/0 (socket says 172.21.15.144:53360) 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 0x7fd3541a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 -- 172.21.15.144:0/1250195873 learned_addr learned my addr 172.21.15.144:0/1250195873 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd352a00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd354108d20 0x7fd35419c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 -- 172.21.15.144:0/1250195873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 msgr2=0x7fd3541a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 0x7fd3541a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 -- 172.21.15.144:0/1250195873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd354108d20 msgr2=0x7fd35419c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd354108d20 0x7fd35419c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.816+0000 7fd353400640 1 -- 172.21.15.144:0/1250195873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3541a1e60 con 0x7fd3541033e0 2024-07-26T11:19:18.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.817+0000 7fd353400640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd35419bef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd34400c420 tx=0x7fd344009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.817+0000 7fd34be00640 1 -- 172.21.15.144:0/1250195873 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344003970 con 0x7fd3541033e0 2024-07-26T11:19:18.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.817+0000 7fd34be00640 1 -- 172.21.15.144:0/1250195873 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd344004020 con 0x7fd3541033e0 2024-07-26T11:19:18.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.817+0000 7fd34be00640 1 -- 172.21.15.144:0/1250195873 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34400b980 con 0x7fd3541033e0 2024-07-26T11:19:18.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.817+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3541a2150 con 0x7fd3541033e0 2024-07-26T11:19:18.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.817+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3541a88b0 con 0x7fd3541033e0 2024-07-26T11:19:18.822 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 Processor -- start 2024-07-26T11:19:18.822 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 -- start start 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd34be00640 1 -- 172.21.15.144:0/1250195873 <== mon.0 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd34401d020 con 0x7fd3541033e0 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01813000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 0x55dc01813540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0144d390 0x55dc01813a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55dc01789bc0 con 0x55dc0146e070 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55dc017790c0 con 0x55dc014451b0 2024-07-26T11:19:18.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.821+0000 7fd360e87740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dc0177d100 con 0x55dc0144d390 2024-07-26T11:19:18.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 0x55dc01813540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 0x55dc01813540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36418/0 (socket says 172.21.15.144:36418) 2024-07-26T11:19:18.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 -- 172.21.15.144:0/2294548121 learned_addr learned my addr 172.21.15.144:0/2294548121 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:18.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353e00640 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01813000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd352a00640 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0144d390 0x55dc01813a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 -- 172.21.15.144:0/2294548121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0144d390 msgr2=0x55dc01813a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0144d390 0x55dc01813a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 -- 172.21.15.144:0/2294548121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 msgr2=0x55dc01813000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01813000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.822+0000 7fd353400640 1 -- 172.21.15.144:0/2294548121 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc014d9650 con 0x55dc014451b0 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.823+0000 7fd353400640 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 0x55dc01813540 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd3440e14d0 tx=0x7fd3440ed540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f65e8c99996c561 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.823+0000 7fd343e00640 1 -- 172.21.15.144:0/2294548121 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3440f8070 con 0x55dc014451b0 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.823+0000 7fd343e00640 1 -- 172.21.15.144:0/2294548121 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3440e72d0 con 0x55dc014451b0 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.823+0000 7fd343e00640 1 -- 172.21.15.144:0/2294548121 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3440f3660 con 0x55dc014451b0 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.823+0000 7fd34be00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3380b3d40 0x7fd3380b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd34be00640 1 -- 172.21.15.144:0/1250195873 <== mon.0 v2:172.21.15.144:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fd3440e5180 con 0x7fd3541033e0 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd353e00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3380b3d40 0x7fd3380b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 -- 172.21.15.144:0/2294548121 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 msgr2=0x55dc01813540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 0x55dc01813540 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd3440e14d0 tx=0x7fd3440ed540 comp rx=0 tx=0).stop 2024-07-26T11:19:18.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 -- 172.21.15.144:0/2294548121 shutdown_connections 2024-07-26T11:19:18.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0144d390 0x55dc01813a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc014451b0 0x55dc01813540 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 --2- 172.21.15.144:0/2294548121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01813000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.824+0000 7fd360e87740 1 -- 172.21.15.144:0/2294548121 >> 172.21.15.144:0/2294548121 conn(0x55dc01462460 msgr2=0x55dc01810ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.825+0000 7fd360e87740 1 -- 172.21.15.144:0/2294548121 shutdown_connections 2024-07-26T11:19:18.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.825+0000 7fd353e00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3380b3d40 0x7fd3380b6250 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0303f0 tx=0x7fd34c02c040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.825+0000 7fd360e87740 1 -- 172.21.15.144:0/2294548121 wait complete. 2024-07-26T11:19:18.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.825+0000 7fd360e87740 1 -- start start 2024-07-26T11:19:18.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 Processor -- start 2024-07-26T11:19:18.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc0181e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc0181ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc0181f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55dc015dc700 con 0x55dc0146e070 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55dc01715690 con 0x55dc0144d390 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dc01821660 con 0x55dc014451b0 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd360e87740 10 client.0 _ll_register_callbacks cb 0x55dc01725140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd352a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc0181ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd352a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc0181ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:36430/0 (socket says 172.21.15.144:36430) 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd352a00640 1 -- 172.21.15.144:0/3707804800 learned_addr learned my addr 172.21.15.144:0/3707804800 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.826+0000 7fd353e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc0181f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd353400640 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc0181e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd352a00640 1 -- 172.21.15.144:0/3707804800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 msgr2=0x55dc0181e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd352a00640 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc0181e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd352a00640 1 -- 172.21.15.144:0/3707804800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 msgr2=0x55dc0181f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd352a00640 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc0181f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd352a00640 1 -- 172.21.15.144:0/3707804800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc0172c830 con 0x55dc0144d390 2024-07-26T11:19:18.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd352a00640 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc0181ebd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd33c0126e0 tx=0x7fd33c012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e64b320d0fadf0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.827+0000 7fd340c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:19:18.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd340c00640 1 -- 172.21.15.144:0/3707804800 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33c012190 con 0x55dc0144d390 2024-07-26T11:19:18.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd340c00640 1 -- 172.21.15.144:0/3707804800 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fd33c00f920 con 0x55dc0144d390 2024-07-26T11:19:18.831 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3440e72d0 con 0x55dc0144d390 2024-07-26T11:19:18.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd340c00640 1 -- 172.21.15.144:0/3707804800 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7fd33c00fb20 con 0x55dc0144d390 2024-07-26T11:19:18.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd360e87740 10 client.26687 fetch_fsmap learned FSMap version 333 2024-07-26T11:19:18.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55dc0182d360 con 0x55dc0144d390 2024-07-26T11:19:18.832 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.828+0000 7fd340c00640 1 -- 172.21.15.144:0/3707804800 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3440e72d0 con 0x55dc0144d390 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd340c00640 1 -- 172.21.15.144:0/3707804800 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7fd33c01c420 con 0x55dc0144d390 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd360e87740 10 client.26687 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd360e87740 10 client.26687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd360e87740 20 client.26687 populate_metadata read hostname 'smithi144' 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd360e87740 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc0142c9f0 0x55dc01830d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd360e87740 4 client.26687 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc0178bb10 con 0x55dc0142c9f0 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.829+0000 7fd353400640 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc0142c9f0 0x55dc01830d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.830+0000 7fd353400640 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc0142c9f0 0x55dc01830d10 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cb145b2c1ea86e4 server_cookie=f7db5325e0f64b5c in_seq=0 out_seq=0 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.830+0000 7fd340c00640 10 client.26687 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.830+0000 7fd340c00640 1 -- 172.21.15.144:0/3707804800 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc0178bb10 con 0x55dc0142c9f0 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.830+0000 7fd340c00640 10 client.26687 handle_command_reply: tid=0 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 client.26687 shutdown 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 msgr2=0x55dc0181ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.833 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc0181ebd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd33c0126e0 tx=0x7fd33c012ba0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 shutdown_connections 2024-07-26T11:19:18.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc0181f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc0142c9f0 0x55dc01830d10 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc0181ebd0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.834 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 --2- 172.21.15.144:0/3707804800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc0181e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 >> 172.21.15.144:0/3707804800 conn(0x55dc01462460 msgr2=0x55dc018161e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd340c00640 0 client.26687 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:18.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 shutdown_connections 2024-07-26T11:19:18.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 1 -- 172.21.15.144:0/3707804800 wait complete. 2024-07-26T11:19:18.835 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.831+0000 7fd360e87740 20 client.26687 trim_cache size 0 max 16384 2024-07-26T11:19:18.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 Processor -- start 2024-07-26T11:19:18.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 -- start start 2024-07-26T11:19:18.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc01810f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01836510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01836a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541b4fd0 con 0x55dc0146e070 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541b3490 con 0x55dc0144d390 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.850+0000 7fd360e87740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541b4900 con 0x55dc014451b0 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc01810f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01836a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc01810f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:35778/0 (socket says 172.21.15.144:35778) 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 -- 172.21.15.144:0/539133246 learned_addr learned my addr 172.21.15.144:0/539133246 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:18.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd352a00640 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01836510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 -- 172.21.15.144:0/539133246 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 msgr2=0x55dc01836510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01836510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 -- 172.21.15.144:0/539133246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 msgr2=0x55dc01836a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01836a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.851+0000 7fd353400640 1 -- 172.21.15.144:0/539133246 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc0181d5f0 con 0x55dc014451b0 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.852+0000 7fd353400640 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc01810f20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd3440a6ad0 tx=0x7fd3440e8c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5845475cfb0fad server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.852+0000 7fd343400640 1 -- 172.21.15.144:0/539133246 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a6040 con 0x55dc014451b0 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.852+0000 7fd343400640 1 -- 172.21.15.144:0/539133246 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3440b3440 con 0x55dc014451b0 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.852+0000 7fd343400640 1 -- 172.21.15.144:0/539133246 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a61e0 con 0x55dc014451b0 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 -- 172.21.15.144:0/539133246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 msgr2=0x55dc01810f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc01810f20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd3440a6ad0 tx=0x7fd3440e8c30 comp rx=0 tx=0).stop 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 -- 172.21.15.144:0/539133246 shutdown_connections 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc0146e070 0x55dc01836a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01836510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 --2- 172.21.15.144:0/539133246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc014451b0 0x55dc01810f20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 -- 172.21.15.144:0/539133246 >> 172.21.15.144:0/539133246 conn(0x55dc01462460 msgr2=0x55dc01810810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 -- 172.21.15.144:0/539133246 shutdown_connections 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 -- 172.21.15.144:0/539133246 wait complete. 2024-07-26T11:19:18.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.853+0000 7fd360e87740 1 -- start start 2024-07-26T11:19:18.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 Processor -- start 2024-07-26T11:19:18.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 0x55dc01823f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01824470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0146e070 0x55dc018974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541b4230 con 0x55dc014451b0 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd3541b3b60 con 0x55dc0144d390 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dc018152f0 con 0x55dc0146e070 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd360e87740 10 client.0 _ll_register_callbacks cb 0x7fd33c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd353400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 0x55dc01823f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd353400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 0x55dc01823f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53402/0 (socket says 172.21.15.144:53402) 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.854+0000 7fd353400640 1 -- 172.21.15.144:0/1156792471 learned_addr learned my addr 172.21.15.144:0/1156792471 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01824470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd353e00640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0146e070 0x55dc018974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 -- 172.21.15.144:0/1156792471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0146e070 msgr2=0x55dc018974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0146e070 0x55dc018974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 -- 172.21.15.144:0/1156792471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 msgr2=0x55dc01823f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.859 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 0x55dc01823f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 -- 172.21.15.144:0/1156792471 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc01897a50 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd353400640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 0x55dc01823f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd352a00640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01824470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd33c01ca50 tx=0x7fd33c0123c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=470468f2c8fbbd0a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd340c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.855+0000 7fd340c00640 1 -- 172.21.15.144:0/1156792471 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33c01c630 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dc017875b0 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd340c00640 1 -- 172.21.15.144:0/1156792471 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fd33c01cad0 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd340c00640 1 -- 172.21.15.144:0/1156792471 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7fd33c01ccd0 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 10 client.26693 fetch_fsmap learned FSMap version 333 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55dc01898ce0 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd340c00640 1 -- 172.21.15.144:0/1156792471 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55dc017875b0 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd340c00640 1 -- 172.21.15.144:0/1156792471 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7fd33c02a060 con 0x55dc0144d390 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 10 client.26693 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:19:18.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 10 client.26693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:19:18.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 20 client.26693 populate_metadata read hostname 'smithi144' 2024-07-26T11:19:18.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc01401da0 0x55dc0189b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 4 client.26693 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:19:18.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.856+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc01784cf0 con 0x55dc01401da0 2024-07-26T11:19:18.861 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.857+0000 7fd353400640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc01401da0 0x55dc0189b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.857+0000 7fd353400640 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc01401da0 0x55dc0189b3f0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e0364ad2980a590 server_cookie=b42b627fc7f4bf00 in_seq=0 out_seq=0 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.857+0000 7fd340c00640 10 client.26693 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.857+0000 7fd340c00640 1 -- 172.21.15.144:0/1156792471 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dc01784cf0 con 0x55dc01401da0 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.857+0000 7fd340c00640 10 client.26693 handle_command_reply: tid=0 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.857+0000 7fd360e87740 1 client.26693 shutdown 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 msgr2=0x55dc01824470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01824470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd33c01ca50 tx=0x7fd33c0123c0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 shutdown_connections 2024-07-26T11:19:18.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dc0146e070 0x55dc018974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55dc01401da0 0x55dc0189b3f0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55dc0144d390 0x55dc01824470 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 --2- 172.21.15.144:0/1156792471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55dc014451b0 0x55dc01823f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 >> 172.21.15.144:0/1156792471 conn(0x55dc01462460 msgr2=0x55dc0180f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd340c00640 0 client.26693 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 shutdown_connections 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 1 -- 172.21.15.144:0/1156792471 wait complete. 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.858+0000 7fd360e87740 20 client.26693 trim_cache size 0 max 16384 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:19:18.863 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:19:18.864 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3380b3d40 msgr2=0x7fd3380b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3380b3d40 0x7fd3380b6250 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0303f0 tx=0x7fd34c02c040 comp rx=0 tx=0).stop 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 msgr2=0x7fd35419bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd35419bef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd34400c420 tx=0x7fd344009f90 comp rx=0 tx=0).stop 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 shutdown_connections 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fd3380b3d40 0x7fd3380b6250 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd354109670 0x7fd3541a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd354108d20 0x7fd35419c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 --2- 172.21.15.144:0/1250195873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3541033e0 0x7fd35419bef0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 >> 172.21.15.144:0/1250195873 conn(0x7fd3540fee70 msgr2=0x7fd354108340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 shutdown_connections 2024-07-26T11:19:18.865 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:18.859+0000 7fd358c00640 1 -- 172.21.15.144:0/1250195873 wait complete. 2024-07-26T11:19:19.721 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:19:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:20 smithi160 ceph-mon[36715]: pgmap v1025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:20 smithi183 ceph-mon[37273]: pgmap v1025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:20 smithi144 ceph-mon[32129]: pgmap v1025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:21.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 804.542876 secs 2024-07-26T11:19:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 804.542876 secs 2024-07-26T11:19:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 804.542876 secs 2024-07-26T11:19:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:22 smithi160 ceph-mon[36715]: pgmap v1026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:22 smithi183 ceph-mon[37273]: pgmap v1026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:22 smithi144 ceph-mon[32129]: pgmap v1026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:24 smithi160 ceph-mon[36715]: pgmap v1027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:24 smithi183 ceph-mon[37273]: pgmap v1027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:24 smithi144 ceph-mon[32129]: pgmap v1027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:26 smithi160 ceph-mon[36715]: pgmap v1028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 809.543025 secs 2024-07-26T11:19:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:26 smithi183 ceph-mon[37273]: pgmap v1028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 809.543025 secs 2024-07-26T11:19:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:26 smithi144 ceph-mon[32129]: pgmap v1028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 809.543025 secs 2024-07-26T11:19:27.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:27.303+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:27.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:27.304+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:27.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:27.304+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:27.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:27.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:27.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:27.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:28 smithi160 ceph-mon[36715]: pgmap v1029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:28 smithi183 ceph-mon[37273]: pgmap v1029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:28 smithi144 ceph-mon[32129]: pgmap v1029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:30 smithi160 ceph-mon[36715]: pgmap v1030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:30 smithi183 ceph-mon[37273]: pgmap v1030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:30 smithi144 ceph-mon[32129]: pgmap v1030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 814.543162 secs 2024-07-26T11:19:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 814.543162 secs 2024-07-26T11:19:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 814.543162 secs 2024-07-26T11:19:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:32 smithi160 ceph-mon[36715]: pgmap v1031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:32 smithi183 ceph-mon[37273]: pgmap v1031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:32 smithi144 ceph-mon[32129]: pgmap v1031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:34 smithi183 ceph-mon[37273]: pgmap v1032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:34 smithi144 ceph-mon[32129]: pgmap v1032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:34 smithi160 ceph-mon[36715]: pgmap v1032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:36.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:36 smithi160 ceph-mon[36715]: pgmap v1033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:36.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 819.543276 secs 2024-07-26T11:19:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:36 smithi183 ceph-mon[37273]: pgmap v1033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 819.543276 secs 2024-07-26T11:19:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:36 smithi144 ceph-mon[32129]: pgmap v1033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 819.543276 secs 2024-07-26T11:19:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:38 smithi183 ceph-mon[37273]: pgmap v1034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:38 smithi144 ceph-mon[32129]: pgmap v1034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:38 smithi160 ceph-mon[36715]: pgmap v1034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:40 smithi183 ceph-mon[37273]: pgmap v1035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:40 smithi144 ceph-mon[32129]: pgmap v1035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:40 smithi160 ceph-mon[36715]: pgmap v1035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:41.622 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 824.543423 secs 2024-07-26T11:19:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 824.543423 secs 2024-07-26T11:19:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 824.543423 secs 2024-07-26T11:19:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:42.307+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:42.308+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:42.308+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:42.308+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:42.309+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:42 smithi183 ceph-mon[37273]: pgmap v1036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:42.659 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:42 smithi144 ceph-mon[32129]: pgmap v1036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:42 smithi160 ceph-mon[36715]: pgmap v1036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:44 smithi183 ceph-mon[37273]: pgmap v1037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:44 smithi144 ceph-mon[32129]: pgmap v1037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:44 smithi160 ceph-mon[36715]: pgmap v1037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:46 smithi183 ceph-mon[37273]: pgmap v1038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 829.543489 secs 2024-07-26T11:19:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:46 smithi144 ceph-mon[32129]: pgmap v1038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 829.543489 secs 2024-07-26T11:19:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:46 smithi160 ceph-mon[36715]: pgmap v1038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:46.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 829.543489 secs 2024-07-26T11:19:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:48 smithi183 ceph-mon[37273]: pgmap v1039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:48 smithi144 ceph-mon[32129]: pgmap v1039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:48 smithi160 ceph-mon[36715]: pgmap v1039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:49.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:19:50.183 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:50 smithi183 ceph-mon[37273]: pgmap v1040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:50.680 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:50 smithi144 ceph-mon[32129]: pgmap v1040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:50 smithi160 ceph-mon[36715]: pgmap v1040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 834.543631 secs 2024-07-26T11:19:51.642 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 834.543631 secs 2024-07-26T11:19:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 834.543631 secs 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 -- 172.21.15.144:0/3633093891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149410af10 msgr2=0x7f149410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 --2- 172.21.15.144:0/3633093891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149410af10 0x7f149410b320 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1480009900 tx=0x7f148002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 -- 172.21.15.144:0/3633093891 shutdown_connections 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 --2- 172.21.15.144:0/3633093891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410b860 0x7f1494112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 --2- 172.21.15.144:0/3633093891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f149410af10 0x7f149410b320 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 --2- 172.21.15.144:0/3633093891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14941055d0 0x7f14941059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.549+0000 7f1499600640 1 -- 172.21.15.144:0/3633093891 >> 172.21.15.144:0/3633093891 conn(0x7f1494101040 msgr2=0x7f1494103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.550+0000 7f1499600640 1 -- 172.21.15.144:0/3633093891 shutdown_connections 2024-07-26T11:19:52.553 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.550+0000 7f1499600640 1 -- 172.21.15.144:0/3633093891 wait complete. 2024-07-26T11:19:52.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 Processor -- start 2024-07-26T11:19:52.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 -- start start 2024-07-26T11:19:52.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14941055d0 0x7f149419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410af10 0x7f149419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149410b860 0x7f14941a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f14941154f0 con 0x7f149410af10 2024-07-26T11:19:52.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1494115370 con 0x7f149410b860 2024-07-26T11:19:52.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.552+0000 7f1499600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1494115670 con 0x7f14941055d0 2024-07-26T11:19:52.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410af10 0x7f149419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410af10 0x7f149419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37824/0 (socket says 172.21.15.144:37824) 2024-07-26T11:19:52.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493400640 1 -- 172.21.15.144:0/852405325 learned_addr learned my addr 172.21.15.144:0/852405325 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:52.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493e00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14941055d0 0x7f149419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1498c00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149410b860 0x7f14941a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493e00640 1 -- 172.21.15.144:0/852405325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149410b860 msgr2=0x7f14941a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493e00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149410b860 0x7f14941a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493e00640 1 -- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410af10 msgr2=0x7f149419c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493e00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410af10 0x7f149419c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1493e00640 1 -- 172.21.15.144:0/852405325 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14941a1eb0 con 0x7f14941055d0 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.553+0000 7f1498c00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149410b860 0x7f14941a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:19:52.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.554+0000 7f1493e00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14941055d0 0x7f149419bf40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f148400e9f0 tx=0x7f148400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:52.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.554+0000 7f1490c00640 1 -- 172.21.15.144:0/852405325 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148400ccc0 con 0x7f14941055d0 2024-07-26T11:19:52.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.554+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14941a21a0 con 0x7f14941055d0 2024-07-26T11:19:52.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.554+0000 7f1490c00640 1 -- 172.21.15.144:0/852405325 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1484005020 con 0x7f14941055d0 2024-07-26T11:19:52.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.554+0000 7f1490c00640 1 -- 172.21.15.144:0/852405325 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484004a00 con 0x7f14941055d0 2024-07-26T11:19:52.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.554+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14941a8900 con 0x7f14941055d0 2024-07-26T11:19:52.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.557+0000 7f14a17e1740 1 Processor -- start 2024-07-26T11:19:52.560 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 -- start start 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1490c00640 1 -- 172.21.15.144:0/852405325 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f1484004ba0 con 0x7f14941055d0 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b365fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b366510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b366a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56282b2dcb90 con 0x56282af981b0 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56282b2cc090 con 0x56282afc1070 2024-07-26T11:19:52.561 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f14a17e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56282b2d00d0 con 0x56282afa0390 2024-07-26T11:19:52.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1493e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b366510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1498c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b365fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1493400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b366a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1498c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b365fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:35470/0 (socket says 172.21.15.144:35470) 2024-07-26T11:19:52.562 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1498c00640 1 -- 172.21.15.144:0/3061976507 learned_addr learned my addr 172.21.15.144:0/3061976507 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.558+0000 7f1493e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b366510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37828/0 (socket says 172.21.15.144:37828) 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.559+0000 7f1493400640 1 -- 172.21.15.144:0/3061976507 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 msgr2=0x56282b365fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.559+0000 7f1493400640 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b365fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.559+0000 7f1493400640 1 -- 172.21.15.144:0/3061976507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 msgr2=0x56282b366510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.559+0000 7f1493400640 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b366510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.559+0000 7f1493400640 1 -- 172.21.15.144:0/3061976507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282b02c650 con 0x56282afa0390 2024-07-26T11:19:52.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.559+0000 7f1493400640 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b366a50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1480031440 tx=0x7f1480031b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4950addd7a83921 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:52.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.560+0000 7f1490c00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14780b3d40 0x7f14780b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.560+0000 7f1489200640 1 -- 172.21.15.144:0/3061976507 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480040070 con 0x56282afa0390 2024-07-26T11:19:52.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.560+0000 7f1489200640 1 -- 172.21.15.144:0/3061976507 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1480004030 con 0x56282afa0390 2024-07-26T11:19:52.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.560+0000 7f1489200640 1 -- 172.21.15.144:0/3061976507 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480035040 con 0x56282afa0390 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.560+0000 7f1490c00640 1 -- 172.21.15.144:0/852405325 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f14840e5370 con 0x7f14941055d0 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.560+0000 7f1498c00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14780b3d40 0x7f14780b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 -- 172.21.15.144:0/3061976507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 msgr2=0x56282b366a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b366a50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1480031440 tx=0x7f1480031b10 comp rx=0 tx=0).stop 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f1498c00640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14780b3d40 0x7f14780b6250 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f148c005c40 tx=0x7f148c009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 -- 172.21.15.144:0/3061976507 shutdown_connections 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b366a50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b366510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3061976507 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b365fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 -- 172.21.15.144:0/3061976507 >> 172.21.15.144:0/3061976507 conn(0x56282afb5460 msgr2=0x56282b363ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 -- 172.21.15.144:0/3061976507 shutdown_connections 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.561+0000 7f14a17e1740 1 -- 172.21.15.144:0/3061976507 wait complete. 2024-07-26T11:19:52.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.562+0000 7f14a17e1740 1 -- start start 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.562+0000 7f14a17e1740 1 Processor -- start 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b371ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afc1070 0x56282b372020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x56282b12f700 con 0x56282afc1070 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56282b268660 con 0x56282af981b0 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56282b374750 con 0x56282afa0390 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f14a17e1740 10 client.0 _ll_register_callbacks cb 0x56282b278110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:52.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f1498c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afc1070 0x56282b372020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f1493400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b371ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f1493400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b371ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:53678/0 (socket says 172.21.15.144:53678) 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f1493400640 1 -- 172.21.15.144:0/3370055093 learned_addr learned my addr 172.21.15.144:0/3370055093 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.563+0000 7f1493e00640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493e00640 1 -- 172.21.15.144:0/3370055093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 msgr2=0x56282b371ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493e00640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b371ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493e00640 1 -- 172.21.15.144:0/3370055093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afc1070 msgr2=0x56282b372020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493e00640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afc1070 0x56282b372020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493e00640 1 -- 172.21.15.144:0/3370055093 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282b27f800 con 0x56282af981b0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493400640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b371ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f1493e00640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3715a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f14840e1780 tx=0x7f1484009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae5648468c0e3d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f147d600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f147d600640 1 -- 172.21.15.144:0/3370055093 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148401c070 con 0x56282af981b0 2024-07-26T11:19:52.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1480004030 con 0x56282af981b0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f147d600640 1 -- 172.21.15.144:0/3370055093 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f14840e1a00 con 0x56282af981b0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f147d600640 1 -- 172.21.15.144:0/3370055093 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f14840aa3b0 con 0x56282af981b0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f14a17e1740 10 client.26705 fetch_fsmap learned FSMap version 333 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.564+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x56282b380380 con 0x56282af981b0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f147d600640 1 -- 172.21.15.144:0/3370055093 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1480004030 con 0x56282af981b0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f147d600640 1 -- 172.21.15.144:0/3370055093 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f1484090ce0 con 0x56282af981b0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f14a17e1740 10 client.26705 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f14a17e1740 10 client.26705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f14a17e1740 20 client.26705 populate_metadata read hostname 'smithi144' 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af7f9f0 0x56282b383cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f14a17e1740 4 client.26705 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56282b2deae0 con 0x56282af7f9f0 2024-07-26T11:19:52.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.565+0000 7f1493400640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af7f9f0 0x56282b383cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.571 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.566+0000 7f1493400640 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af7f9f0 0x56282b383cd0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37ef0043786220d server_cookie=93764516258c9d29 in_seq=0 out_seq=0 2024-07-26T11:19:52.571 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.566+0000 7f147d600640 10 client.26705 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:52.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f147d600640 1 -- 172.21.15.144:0/3370055093 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56282b2deae0 con 0x56282af7f9f0 2024-07-26T11:19:52.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f147d600640 10 client.26705 handle_command_reply: tid=0 2024-07-26T11:19:52.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 client.26705 shutdown 2024-07-26T11:19:52.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 msgr2=0x56282b3715a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3715a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f14840e1780 tx=0x7f1484009f60 comp rx=0 tx=0).stop 2024-07-26T11:19:52.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 shutdown_connections 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afc1070 0x56282b372020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b371ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af7f9f0 0x56282b383cd0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 --2- 172.21.15.144:0/3370055093 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3715a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 >> 172.21.15.144:0/3370055093 conn(0x56282afb5460 msgr2=0x56282b3691b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.567+0000 7f147d600640 0 client.26705 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.568+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 shutdown_connections 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.568+0000 7f14a17e1740 1 -- 172.21.15.144:0/3370055093 wait complete. 2024-07-26T11:19:52.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.568+0000 7f14a17e1740 20 client.26705 trim_cache size 0 max 16384 2024-07-26T11:19:52.589 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.586+0000 7f14a17e1740 1 Processor -- start 2024-07-26T11:19:52.589 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:52 smithi144 ceph-mon[32129]: pgmap v1041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:52.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 -- start start 2024-07-26T11:19:52.590 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b373480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b362500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b367710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f14941b4e20 con 0x56282af981b0 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f14941b32e0 con 0x56282afc1070 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.587+0000 7f14a17e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f14941b4750 con 0x56282afa0390 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b373480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b362500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b362500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:53690/0 (socket says 172.21.15.144:53690) 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 -- 172.21.15.144:0/2875172180 learned_addr learned my addr 172.21.15.144:0/2875172180 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1498c00640 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b367710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 -- 172.21.15.144:0/2875172180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 msgr2=0x56282b367710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b367710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 -- 172.21.15.144:0/2875172180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 msgr2=0x56282b373480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b373480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493400640 1 -- 172.21.15.144:0/2875172180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282b3682c0 con 0x56282afa0390 2024-07-26T11:19:52.592 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.588+0000 7f1493e00640 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b373480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:52.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.589+0000 7f1493400640 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b362500 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1480031440 tx=0x7f1480005890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f78edc2129f49641 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:52.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.589+0000 7f147fe00640 1 -- 172.21.15.144:0/2875172180 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480040040 con 0x56282afa0390 2024-07-26T11:19:52.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.589+0000 7f147fe00640 1 -- 172.21.15.144:0/2875172180 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1480054070 con 0x56282afa0390 2024-07-26T11:19:52.593 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.589+0000 7f147fe00640 1 -- 172.21.15.144:0/2875172180 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480035040 con 0x56282afa0390 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 -- 172.21.15.144:0/2875172180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 msgr2=0x56282b362500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b362500 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1480031440 tx=0x7f1480005890 comp rx=0 tx=0).stop 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 -- 172.21.15.144:0/2875172180 shutdown_connections 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282afc1070 0x56282b367710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afa0390 0x56282b362500 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 --2- 172.21.15.144:0/2875172180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282af981b0 0x56282b373480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 -- 172.21.15.144:0/2875172180 >> 172.21.15.144:0/2875172180 conn(0x56282afb5460 msgr2=0x56282b367000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 -- 172.21.15.144:0/2875172180 shutdown_connections 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.590+0000 7f14a17e1740 1 -- 172.21.15.144:0/2875172180 wait complete. 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.591+0000 7f14a17e1740 1 -- start start 2024-07-26T11:19:52.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.591+0000 7f14a17e1740 1 Processor -- start 2024-07-26T11:19:52.597 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3773e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.597 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afa0390 0x56282b3779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.597 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afc1070 0x56282b3e45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.597 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f14941b4080 con 0x56282afa0390 2024-07-26T11:19:52.597 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f14941b39b0 con 0x56282af981b0 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56282b362a40 con 0x56282afc1070 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f14a17e1740 10 client.0 _ll_register_callbacks cb 0x7f1484004e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f1493400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afa0390 0x56282b3779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f1493e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3773e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f1493400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afa0390 0x56282b3779c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37870/0 (socket says 172.21.15.144:37870) 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f1493400640 1 -- 172.21.15.144:0/361378778 learned_addr learned my addr 172.21.15.144:0/361378778 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:19:52.598 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.592+0000 7f1498c00640 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afc1070 0x56282b3e45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f1493e00640 1 -- 172.21.15.144:0/361378778 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afc1070 msgr2=0x56282b3e45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f1493e00640 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afc1070 0x56282b3e45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f1493e00640 1 -- 172.21.15.144:0/361378778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afa0390 msgr2=0x56282b3779c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f1493e00640 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afa0390 0x56282b3779c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f1493e00640 1 -- 172.21.15.144:0/361378778 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282b3e4c50 con 0x56282af981b0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f1493e00640 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3773e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f14840a6cc0 tx=0x7f14840e1c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cc984c23131aff8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f147d600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f147d600640 1 -- 172.21.15.144:0/361378778 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148401c040 con 0x56282af981b0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1480054070 con 0x56282af981b0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f147d600640 1 -- 172.21.15.144:0/361378778 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f14840b3a70 con 0x56282af981b0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.593+0000 7f147d600640 1 -- 172.21.15.144:0/361378778 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f14840a63d0 con 0x56282af981b0 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 10 client.26711 fetch_fsmap learned FSMap version 333 2024-07-26T11:19:52.599 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x56282b3ec050 con 0x56282af981b0 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f147d600640 1 -- 172.21.15.144:0/361378778 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1480054070 con 0x56282af981b0 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f147d600640 1 -- 172.21.15.144:0/361378778 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f1484090ce0 con 0x56282af981b0 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 10 client.26711 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 10 client.26711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 20 client.26711 populate_metadata read hostname 'smithi144' 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af54da0 0x56282b3ee720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 4 client.26711 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:19:52.600 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56282b2da580 con 0x56282af54da0 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.594+0000 7f1493400640 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af54da0 0x56282b3ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.595+0000 7f1493400640 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af54da0 0x56282b3ee720 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7233b671f950d057 server_cookie=1a5482a85b4a5ade in_seq=0 out_seq=0 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.595+0000 7f147d600640 10 client.26711 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.595+0000 7f147d600640 1 -- 172.21.15.144:0/361378778 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56282b2da580 con 0x56282af54da0 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.595+0000 7f147d600640 10 client.26711 handle_command_reply: tid=0 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 client.26711 shutdown 2024-07-26T11:19:52.602 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 msgr2=0x56282b3773e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3773e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f14840a6cc0 tx=0x7f14840e1c30 comp rx=0 tx=0).stop 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 shutdown_connections 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56282afc1070 0x56282b3e45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56282afa0390 0x56282b3779c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x56282af54da0 0x56282b3ee720 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 --2- 172.21.15.144:0/361378778 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56282af981b0 0x56282b3773e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f147d600640 0 client.26711 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 >> 172.21.15.144:0/361378778 conn(0x56282afb5460 msgr2=0x56282b3679c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 shutdown_connections 2024-07-26T11:19:52.603 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 1 -- 172.21.15.144:0/361378778 wait complete. 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.596+0000 7f14a17e1740 20 client.26711 trim_cache size 0 max 16384 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:19:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14780b3d40 msgr2=0x7f14780b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14780b3d40 0x7f14780b6250 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f148c005c40 tx=0x7f148c009500 comp rx=0 tx=0).stop 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14941055d0 msgr2=0x7f149419bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14941055d0 0x7f149419bf40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f148400e9f0 tx=0x7f148400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.605 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 shutdown_connections 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f14780b3d40 0x7f14780b6250 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f149410b860 0x7f14941a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f149410af10 0x7f149419c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 --2- 172.21.15.144:0/852405325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14941055d0 0x7f149419bf40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.597+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 >> 172.21.15.144:0/852405325 conn(0x7f1494101040 msgr2=0x7f1494103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.598+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 shutdown_connections 2024-07-26T11:19:52.606 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:19:52.598+0000 7f1499600640 1 -- 172.21.15.144:0/852405325 wait complete. 2024-07-26T11:19:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:52 smithi160 ceph-mon[36715]: pgmap v1041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:52 smithi183 ceph-mon[37273]: pgmap v1041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:53.415 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:19:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:54 smithi144 ceph-mon[32129]: pgmap v1042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:54 smithi160 ceph-mon[36715]: pgmap v1042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:54 smithi183 ceph-mon[37273]: pgmap v1042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:55 smithi144 ceph-mon[32129]: pgmap v1043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:55 smithi160 ceph-mon[36715]: pgmap v1043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:55 smithi183 ceph-mon[37273]: pgmap v1043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:19:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 839.543768 secs 2024-07-26T11:19:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 839.543768 secs 2024-07-26T11:19:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 839.543768 secs 2024-07-26T11:19:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:57.324+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:57.325+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:57.325+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:57.325+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:19:57.325+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:57 smithi144 ceph-mon[32129]: pgmap v1044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:57 smithi160 ceph-mon[36715]: pgmap v1044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:57 smithi183 ceph-mon[37273]: pgmap v1044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:19:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:19:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:59 smithi160 ceph-mon[36715]: pgmap v1045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:19:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:00.082 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:59 smithi144 ceph-mon[32129]: pgmap v1045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:00.082 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:00.082 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:19:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:59 smithi183 ceph-mon[37273]: pgmap v1045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:19:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 844.543909 secs 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 844.543909 secs 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 844.543909 secs 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:02 smithi160 ceph-mon[36715]: pgmap v1046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:02.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:02.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:02 smithi183 ceph-mon[37273]: pgmap v1046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:02 smithi144 ceph-mon[32129]: pgmap v1046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: pgmap v1047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: pgmap v1048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: pgmap v1047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: pgmap v1048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: pgmap v1047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:04.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: pgmap v1048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:04.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:20:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:04 smithi183 ceph-mon[37273]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:20:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:04 smithi144 ceph-mon[32129]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:20:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:04 smithi160 ceph-mon[36715]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:20:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:05 smithi183 ceph-mon[37273]: pgmap v1049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 849.544056 secs 2024-07-26T11:20:06.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:05 smithi160 ceph-mon[36715]: pgmap v1049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:06.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 849.544056 secs 2024-07-26T11:20:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:05 smithi144 ceph-mon[32129]: pgmap v1049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 849.544056 secs 2024-07-26T11:20:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:07 smithi144 ceph-mon[32129]: pgmap v1050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:07 smithi160 ceph-mon[36715]: pgmap v1050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:07 smithi183 ceph-mon[37273]: pgmap v1050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:09 smithi144 ceph-mon[32129]: pgmap v1051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:09 smithi160 ceph-mon[36715]: pgmap v1051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:09 smithi183 ceph-mon[37273]: pgmap v1051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 854.544177 secs 2024-07-26T11:20:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 854.544177 secs 2024-07-26T11:20:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 854.544177 secs 2024-07-26T11:20:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:11 smithi144 ceph-mon[32129]: pgmap v1052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:11 smithi160 ceph-mon[36715]: pgmap v1052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:12.343 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:12.341+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:12.342+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:12.342+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:12.342+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:12.342+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:11 smithi183 ceph-mon[37273]: pgmap v1052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:14.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:13 smithi144 ceph-mon[32129]: pgmap v1053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:13 smithi160 ceph-mon[36715]: pgmap v1053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:13 smithi183 ceph-mon[37273]: pgmap v1053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:15 smithi144 ceph-mon[32129]: pgmap v1054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 859.544315 secs 2024-07-26T11:20:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:15 smithi160 ceph-mon[36715]: pgmap v1054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 859.544315 secs 2024-07-26T11:20:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:15 smithi183 ceph-mon[37273]: pgmap v1054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 859.544315 secs 2024-07-26T11:20:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:17 smithi144 ceph-mon[32129]: pgmap v1055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:17 smithi160 ceph-mon[36715]: pgmap v1055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:17 smithi183 ceph-mon[37273]: pgmap v1055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:19 smithi144 ceph-mon[32129]: pgmap v1056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:19 smithi160 ceph-mon[36715]: pgmap v1056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:19 smithi183 ceph-mon[37273]: pgmap v1056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 864.544428 secs 2024-07-26T11:20:21.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 864.544428 secs 2024-07-26T11:20:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 864.544428 secs 2024-07-26T11:20:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:21 smithi144 ceph-mon[32129]: pgmap v1057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:21 smithi160 ceph-mon[36715]: pgmap v1057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:21 smithi183 ceph-mon[37273]: pgmap v1057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:23.418 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:20:23.887 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:23 smithi144 ceph-mon[32129]: pgmap v1058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:23 smithi160 ceph-mon[36715]: pgmap v1058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:23 smithi183 ceph-mon[37273]: pgmap v1058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:25 smithi144 ceph-mon[32129]: pgmap v1059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 869.544564 secs 2024-07-26T11:20:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:25 smithi160 ceph-mon[36715]: pgmap v1059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 869.544564 secs 2024-07-26T11:20:26.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 -- 172.21.15.144:0/1577935289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 msgr2=0x7fc03c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 --2- 172.21.15.144:0/1577935289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c10b320 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc028009900 tx=0x7fc02802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 -- 172.21.15.144:0/1577935289 shutdown_connections 2024-07-26T11:20:26.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 --2- 172.21.15.144:0/1577935289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c10b860 0x7fc03c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 --2- 172.21.15.144:0/1577935289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c10b320 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.318 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 --2- 172.21.15.144:0/1577935289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc03c1055d0 0x7fc03c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 -- 172.21.15.144:0/1577935289 >> 172.21.15.144:0/1577935289 conn(0x7fc03c101040 msgr2=0x7fc03c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:26.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 -- 172.21.15.144:0/1577935289 shutdown_connections 2024-07-26T11:20:26.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.316+0000 7fc042000640 1 -- 172.21.15.144:0/1577935289 wait complete. 2024-07-26T11:20:26.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.317+0000 7fc042000640 1 Processor -- start 2024-07-26T11:20:26.319 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 -- start start 2024-07-26T11:20:26.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 0x7fc03c19bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.320 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc03c10b860 0x7fc03c1a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1154f0 con 0x7fc03c10af10 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c115370 con 0x7fc03c1055d0 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc042000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c115670 con 0x7fc03c10b860 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.318+0000 7fc040c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 0x7fc03c19bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 0x7fc03c19bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51624/0 (socket says 172.21.15.144:51624) 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c19c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:47362/0 (socket says 172.21.15.144:47362) 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 -- 172.21.15.144:0/388569788 learned_addr learned my addr 172.21.15.144:0/388569788 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:20:26.321 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc041600640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc03c10b860 0x7fc03c1a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 -- 172.21.15.144:0/388569788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc03c10b860 msgr2=0x7fc03c1a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc03c10b860 0x7fc03c1a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 -- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 msgr2=0x7fc03c19c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c19c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 -- 172.21.15.144:0/388569788 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03c1a1f50 con 0x7fc03c1055d0 2024-07-26T11:20:26.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.319+0000 7fc040c00640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 0x7fc03c19bfe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc03000e9f0 tx=0x7fc03000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:26.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.320+0000 7fc039600640 1 -- 172.21.15.144:0/388569788 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03000ccc0 con 0x7fc03c1055d0 2024-07-26T11:20:26.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.320+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03c1a2240 con 0x7fc03c1055d0 2024-07-26T11:20:26.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.320+0000 7fc039600640 1 -- 172.21.15.144:0/388569788 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc03000ce60 con 0x7fc03c1055d0 2024-07-26T11:20:26.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.320+0000 7fc039600640 1 -- 172.21.15.144:0/388569788 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030010430 con 0x7fc03c1055d0 2024-07-26T11:20:26.323 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.320+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03c1a89a0 con 0x7fc03c1055d0 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.323+0000 7fc04a21f740 1 Processor -- start 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.323+0000 7fc04a21f740 1 -- start start 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.323+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b60000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.323+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b60540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.323+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b60a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc04a21f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55eb06ad6bc0 con 0x55eb0679a390 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc04a21f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eb06ac60c0 con 0x55eb067921b0 2024-07-26T11:20:26.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc04a21f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55eb06aca100 con 0x55eb067bb070 2024-07-26T11:20:26.326 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b60a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b60a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:47364/0 (socket says 172.21.15.144:47364) 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc041600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b60000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 -- 172.21.15.144:0/1247210430 learned_addr learned my addr 172.21.15.144:0/1247210430 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc039600640 1 -- 172.21.15.144:0/388569788 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fc0300046c0 con 0x7fc03c1055d0 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 -- 172.21.15.144:0/1247210430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 msgr2=0x55eb06b60000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b60000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 -- 172.21.15.144:0/1247210430 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 msgr2=0x55eb06b60540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.324+0000 7fc03be00640 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b60540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.325+0000 7fc03be00640 1 -- 172.21.15.144:0/1247210430 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb06826650 con 0x55eb0679a390 2024-07-26T11:20:26.327 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.325+0000 7fc03be00640 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b60a80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc028009900 tx=0x7fc028000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba64b6af6a9db88 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:26.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.325+0000 7fc02dc00640 1 -- 172.21.15.144:0/1247210430 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028040070 con 0x55eb0679a390 2024-07-26T11:20:26.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.325+0000 7fc02dc00640 1 -- 172.21.15.144:0/1247210430 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc028004030 con 0x55eb0679a390 2024-07-26T11:20:26.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.325+0000 7fc02dc00640 1 -- 172.21.15.144:0/1247210430 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028035040 con 0x55eb0679a390 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 -- 172.21.15.144:0/1247210430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 msgr2=0x55eb06b60a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b60a80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc028009900 tx=0x7fc028000bc0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 -- 172.21.15.144:0/1247210430 shutdown_connections 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b60a80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b60540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1247210430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b60000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 -- 172.21.15.144:0/1247210430 >> 172.21.15.144:0/1247210430 conn(0x55eb067af460 msgr2=0x55eb06b5def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 -- 172.21.15.144:0/1247210430 shutdown_connections 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc04a21f740 1 -- 172.21.15.144:0/1247210430 wait complete. 2024-07-26T11:20:26.329 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.327+0000 7fc039600640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc0240b3c50 0x7fc0240b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.328+0000 7fc039600640 1 -- 172.21.15.144:0/388569788 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fc0300e56f0 con 0x7fc03c1055d0 2024-07-26T11:20:26.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.328+0000 7fc04a21f740 1 -- start start 2024-07-26T11:20:26.331 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.328+0000 7fc03be00640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc0240b3c50 0x7fc0240b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.328+0000 7fc04a21f740 1 Processor -- start 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b6b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b6bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc03be00640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc0240b3c50 0x7fc0240b6160 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc028005f10 tx=0x7fc028005fd0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b6bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55eb06929700 con 0x55eb0679a390 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eb06a62690 con 0x55eb067921b0 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55eb06b6e780 con 0x55eb067bb070 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc04a21f740 10 client.0 _ll_register_callbacks cb 0x55eb06a72140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b6bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.332 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b6bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:47372/0 (socket says 172.21.15.144:47372) 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc03be00640 1 -- 172.21.15.144:0/3207261231 learned_addr learned my addr 172.21.15.144:0/3207261231 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc040c00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b6b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc041600640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b6bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc040c00640 1 -- 172.21.15.144:0/3207261231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 msgr2=0x55eb06b6bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc040c00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b6bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc040c00640 1 -- 172.21.15.144:0/3207261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 msgr2=0x55eb06b6bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc040c00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b6bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.329+0000 7fc040c00640 1 -- 172.21.15.144:0/3207261231 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb06a79830 con 0x55eb067921b0 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc03be00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b6bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc040c00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b6b570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc0300e1b00 tx=0x7fc0300aa7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=750db93e5bc447e1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:26.333 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc022000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc022000640 1 -- 172.21.15.144:0/3207261231 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300af380 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc028004030 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc022000640 1 -- 172.21.15.144:0/3207261231 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fc0300b3df0 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc022000640 1 -- 172.21.15.144:0/3207261231 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7fc0300aafc0 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc04a21f740 10 client.26720 fetch_fsmap learned FSMap version 333 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.330+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55eb06b7a3b0 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc022000640 1 -- 172.21.15.144:0/3207261231 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc028004030 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc022000640 1 -- 172.21.15.144:0/3207261231 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7fc0300e4030 con 0x55eb067921b0 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc04a21f740 10 client.26720 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc04a21f740 10 client.26720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc04a21f740 20 client.26720 populate_metadata read hostname 'smithi144' 2024-07-26T11:20:26.336 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc04a21f740 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067799f0 0x55eb06b7dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc04a21f740 4 client.26720 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.331+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb06ad8b10 con 0x55eb067799f0 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.332+0000 7fc03be00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067799f0 0x55eb06b7dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.332+0000 7fc022000640 10 client.26720 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.332+0000 7fc03be00640 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067799f0 0x55eb06b7dbf0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f36772ca4bc44be server_cookie=9c52176231b495dd in_seq=0 out_seq=0 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc022000640 1 -- 172.21.15.144:0/3207261231 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eb06ad8b10 con 0x55eb067799f0 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc022000640 10 client.26720 handle_command_reply: tid=0 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 client.26720 shutdown 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 msgr2=0x55eb06b6b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b6b570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc0300e1b00 tx=0x7fc0300aa7f0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 shutdown_connections 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067bb070 0x55eb06b6bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.337 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb0679a390 0x55eb06b6bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067799f0 0x55eb06b7dbf0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 --2- 172.21.15.144:0/3207261231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb067921b0 0x55eb06b6b570 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc022000640 0 client.26720 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.333+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 >> 172.21.15.144:0/3207261231 conn(0x55eb067af460 msgr2=0x55eb06b63190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.334+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 shutdown_connections 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.334+0000 7fc04a21f740 1 -- 172.21.15.144:0/3207261231 wait complete. 2024-07-26T11:20:26.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.334+0000 7fc04a21f740 20 client.26720 trim_cache size 0 max 16384 2024-07-26T11:20:26.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.352+0000 7fc04a21f740 1 Processor -- start 2024-07-26T11:20:26.354 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.352+0000 7fc04a21f740 1 -- start start 2024-07-26T11:20:26.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b5d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b66990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.355 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc04a21f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1b4e70 con 0x55eb067921b0 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc04a21f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1b3330 con 0x55eb0679a390 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc04a21f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1b47a0 con 0x55eb067af460 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc03be00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b619f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:47376/0 (socket says 172.21.15.144:47376) 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc03be00640 1 -- 172.21.15.144:0/2082312780 learned_addr learned my addr 172.21.15.144:0/2082312780 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc040c00640 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b5d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.353+0000 7fc041600640 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b66990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc03be00640 1 -- 172.21.15.144:0/2082312780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 msgr2=0x55eb06b5d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc03be00640 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b5d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc03be00640 1 -- 172.21.15.144:0/2082312780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 msgr2=0x55eb06b66990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc03be00640 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b66990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.356 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc03be00640 1 -- 172.21.15.144:0/2082312780 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb06b622f0 con 0x55eb067921b0 2024-07-26T11:20:26.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc03be00640 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b619f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc028031a60 tx=0x7fc028031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14a614c4f4cc71 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:26.357 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc02d200640 1 -- 172.21.15.144:0/2082312780 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028040050 con 0x55eb067921b0 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc02d200640 1 -- 172.21.15.144:0/2082312780 <== mon.0 v2:172.21.15.144:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0280054f0 con 0x55eb067921b0 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.354+0000 7fc02d200640 1 -- 172.21.15.144:0/2082312780 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028035040 con 0x55eb067921b0 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.355+0000 7fc04a21f740 1 -- 172.21.15.144:0/2082312780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 msgr2=0x55eb06b619f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.355+0000 7fc04a21f740 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b619f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc028031a60 tx=0x7fc028031400 comp rx=0 tx=0).stop 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.355+0000 7fc04a21f740 1 -- 172.21.15.144:0/2082312780 shutdown_connections 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b66990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b619f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 --2- 172.21.15.144:0/2082312780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b5d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 -- 172.21.15.144:0/2082312780 >> 172.21.15.144:0/2082312780 conn(0x55eb06b5e050 msgr2=0x55eb06b66280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 -- 172.21.15.144:0/2082312780 shutdown_connections 2024-07-26T11:20:26.358 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 -- 172.21.15.144:0/2082312780 wait complete. 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 -- start start 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.356+0000 7fc04a21f740 1 Processor -- start 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b73fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b6f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b6f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1b40d0 con 0x55eb067921b0 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1b3a00 con 0x55eb0679a390 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55eb06ad8b10 con 0x55eb067af460 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc04a21f740 10 client.0 _ll_register_callbacks cb 0x7fc0300aa910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc040c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b73fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.360 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc03be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b6f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc040c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b73fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:47378/0 (socket says 172.21.15.144:47378) 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc040c00640 1 -- 172.21.15.144:0/1618124828 learned_addr learned my addr 172.21.15.144:0/1618124828 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.357+0000 7fc041600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b6f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc03be00640 1 -- 172.21.15.144:0/1618124828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 msgr2=0x55eb06b6f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc03be00640 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b6f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc03be00640 1 -- 172.21.15.144:0/1618124828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 msgr2=0x55eb06b73fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc03be00640 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b73fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc03be00640 1 -- 172.21.15.144:0/1618124828 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb06b70070 con 0x55eb0679a390 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc040c00640 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b73fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc03be00640 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b6f490 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc028009900 tx=0x7fc028005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307abea353097aca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:26.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc022000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:20:26.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.358+0000 7fc022000640 1 -- 172.21.15.144:0/1618124828 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028068060 con 0x55eb0679a390 2024-07-26T11:20:26.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0280054f0 con 0x55eb0679a390 2024-07-26T11:20:26.363 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc022000640 1 -- 172.21.15.144:0/1618124828 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fc028005040 con 0x55eb0679a390 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc022000640 1 -- 172.21.15.144:0/1618124828 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7fc028040040 con 0x55eb0679a390 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 10 client.26726 fetch_fsmap learned FSMap version 333 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55eb06be6220 con 0x55eb0679a390 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc022000640 1 -- 172.21.15.144:0/1618124828 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0280054f0 con 0x55eb0679a390 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc022000640 1 -- 172.21.15.144:0/1618124828 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7fc0280528a0 con 0x55eb0679a390 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 10 client.26726 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 10 client.26726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 20 client.26726 populate_metadata read hostname 'smithi144' 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067bb070 0x55eb06be87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 4 client.26726 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.359+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eb06ad45b0 con 0x55eb067bb070 2024-07-26T11:20:26.364 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.360+0000 7fc040c00640 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067bb070 0x55eb06be87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.360+0000 7fc040c00640 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067bb070 0x55eb06be87d0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52695be23c7ad21b server_cookie=19563f777568daaf in_seq=0 out_seq=0 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.360+0000 7fc022000640 10 client.26726 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.360+0000 7fc022000640 1 -- 172.21.15.144:0/1618124828 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55eb06ad45b0 con 0x55eb067bb070 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.360+0000 7fc022000640 10 client.26726 handle_command_reply: tid=0 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.360+0000 7fc04a21f740 1 client.26726 shutdown 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 msgr2=0x55eb06b6f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b6f490 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc028009900 tx=0x7fc028005c90 comp rx=0 tx=0).stop 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 shutdown_connections 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55eb067af460 0x55eb06b6f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55eb067bb070 0x55eb06be87d0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb0679a390 0x55eb06b6f490 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 --2- 172.21.15.144:0/1618124828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55eb067921b0 0x55eb06b73fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc022000640 0 client.26726 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:20:26.365 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 >> 172.21.15.144:0/1618124828 conn(0x55eb06b5e050 msgr2=0x55eb06b66a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 shutdown_connections 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 1 -- 172.21.15.144:0/1618124828 wait complete. 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.361+0000 7fc04a21f740 20 client.26726 trim_cache size 0 max 16384 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:20:26.366 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:20:26.367 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:20:26.367 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:20:26.367 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:20:26.367 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:20:26.367 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc0240b3c50 msgr2=0x7fc0240b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc0240b3c50 0x7fc0240b6160 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc028005f10 tx=0x7fc028005fd0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 msgr2=0x7fc03c19bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 0x7fc03c19bfe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc03000e9f0 tx=0x7fc03000edf0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 shutdown_connections 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fc0240b3c50 0x7fc0240b6160 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc03c10b860 0x7fc03c1a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc03c10af10 0x7fc03c19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 --2- 172.21.15.144:0/388569788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc03c1055d0 0x7fc03c19bfe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 >> 172.21.15.144:0/388569788 conn(0x7fc03c101040 msgr2=0x7fc03c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 shutdown_connections 2024-07-26T11:20:26.368 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:20:26.362+0000 7fc042000640 1 -- 172.21.15.144:0/388569788 wait complete. 2024-07-26T11:20:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:25 smithi183 ceph-mon[37273]: pgmap v1059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 869.544564 secs 2024-07-26T11:20:27.218 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:20:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:27.358+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:27.359+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:27.359+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:27.359+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:27.359+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:28.299 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:27 smithi144 ceph-mon[32129]: pgmap v1060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:27 smithi160 ceph-mon[36715]: pgmap v1060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:27 smithi183 ceph-mon[37273]: pgmap v1060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:29 smithi160 ceph-mon[36715]: pgmap v1061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:29 smithi183 ceph-mon[37273]: pgmap v1061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:29 smithi144 ceph-mon[32129]: pgmap v1061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 874.544699 secs 2024-07-26T11:20:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 874.544699 secs 2024-07-26T11:20:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 874.544699 secs 2024-07-26T11:20:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:31 smithi160 ceph-mon[36715]: pgmap v1062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:31 smithi183 ceph-mon[37273]: pgmap v1062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:31 smithi144 ceph-mon[32129]: pgmap v1062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:33 smithi160 ceph-mon[36715]: pgmap v1063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:33 smithi183 ceph-mon[37273]: pgmap v1063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:33 smithi144 ceph-mon[32129]: pgmap v1063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:36 smithi160 ceph-mon[36715]: pgmap v1064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 879.544781 secs 2024-07-26T11:20:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:36 smithi183 ceph-mon[37273]: pgmap v1064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 879.544781 secs 2024-07-26T11:20:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:36 smithi144 ceph-mon[32129]: pgmap v1064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 879.544781 secs 2024-07-26T11:20:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:38 smithi160 ceph-mon[36715]: pgmap v1065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:38 smithi183 ceph-mon[37273]: pgmap v1065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:38 smithi144 ceph-mon[32129]: pgmap v1065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:40 smithi160 ceph-mon[36715]: pgmap v1066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:40 smithi183 ceph-mon[37273]: pgmap v1066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:40 smithi144 ceph-mon[32129]: pgmap v1066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 884.544894 secs 2024-07-26T11:20:41.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 884.544894 secs 2024-07-26T11:20:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 884.544894 secs 2024-07-26T11:20:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:42 smithi160 ceph-mon[36715]: pgmap v1067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:42.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:42.377+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:42.377+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:42.377+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:42.377+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:42 smithi183 ceph-mon[37273]: pgmap v1067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:42 smithi144 ceph-mon[32129]: pgmap v1067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:44 smithi160 ceph-mon[36715]: pgmap v1068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:44 smithi183 ceph-mon[37273]: pgmap v1068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:44 smithi144 ceph-mon[32129]: pgmap v1068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900b6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005e30 tx=0x7f74a8005c50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900b6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005e30 tx=0x7f74a8005c50 comp rx=0 tx=0).stop 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900b6250 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900b6250 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.092+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.094+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c17b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a8046000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.245+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480006400 con 0x7f74b0109990 2024-07-26T11:20:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.245+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006080 con 0x563c6c4182f0 2024-07-26T11:20:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.245+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480006400 con 0x7f74b0109990 2024-07-26T11:20:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:46.246+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006080 con 0x563c6c4182f0 2024-07-26T11:20:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:46 smithi160 ceph-mon[36715]: pgmap v1069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 889.545034 secs 2024-07-26T11:20:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:46 smithi183 ceph-mon[37273]: pgmap v1069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 889.545034 secs 2024-07-26T11:20:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:46 smithi144 ceph-mon[32129]: pgmap v1069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 889.545034 secs 2024-07-26T11:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:48 smithi160 ceph-mon[36715]: pgmap v1070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:48 smithi183 ceph-mon[37273]: pgmap v1070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:48.527 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:48 smithi144 ceph-mon[32129]: pgmap v1070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:50 smithi160 ceph-mon[36715]: pgmap v1071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:50 smithi183 ceph-mon[37273]: pgmap v1071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:50 smithi144 ceph-mon[32129]: pgmap v1071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 894.545150 secs 2024-07-26T11:20:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 894.545150 secs 2024-07-26T11:20:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 894.545150 secs 2024-07-26T11:20:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:52 smithi183 ceph-mon[37273]: pgmap v1072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:52.441 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:52 smithi144 ceph-mon[32129]: pgmap v1072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:52 smithi160 ceph-mon[36715]: pgmap v1072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:54 smithi183 ceph-mon[37273]: pgmap v1073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:54 smithi144 ceph-mon[32129]: pgmap v1073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:54 smithi160 ceph-mon[36715]: pgmap v1073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:56 smithi183 ceph-mon[37273]: pgmap v1074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 899.545321 secs 2024-07-26T11:20:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:56 smithi144 ceph-mon[32129]: pgmap v1074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 899.545321 secs 2024-07-26T11:20:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:56 smithi160 ceph-mon[36715]: pgmap v1074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:20:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 899.545321 secs 2024-07-26T11:20:57.219 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:20:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:57.391+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:57.392+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:57.392+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:57.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:20:57.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:57.682 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:20:58 smithi183 ceph-mon[37273]: pgmap v1075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:20:58 smithi144 ceph-mon[32129]: pgmap v1075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:20:58.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:20:58 smithi160 ceph-mon[36715]: pgmap v1075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:00.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 -- 172.21.15.144:0/74141860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c102760 msgr2=0x7f434c10f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 --2- 172.21.15.144:0/74141860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c102760 0x7f434c10f480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4334009900 tx=0x7f433402a480 comp rx=0 tx=0).stop 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 -- 172.21.15.144:0/74141860 shutdown_connections 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 --2- 172.21.15.144:0/74141860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c10f9c0 0x7f434c111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 --2- 172.21.15.144:0/74141860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c102760 0x7f434c10f480 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 --2- 172.21.15.144:0/74141860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 -- 172.21.15.144:0/74141860 >> 172.21.15.144:0/74141860 conn(0x7f434c0fdcf0 msgr2=0x7f434c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.016+0000 7f434b200640 1 -- 172.21.15.144:0/74141860 shutdown_connections 2024-07-26T11:21:00.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.017+0000 7f434b200640 1 -- 172.21.15.144:0/74141860 wait complete. 2024-07-26T11:21:00.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 Processor -- start 2024-07-26T11:21:00.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 -- start start 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c102760 0x7f434c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c10f9c0 0x7f434c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1154e0 con 0x7f434c102760 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f434c115360 con 0x7f434c10f9c0 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.018+0000 7f434b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f434c115660 con 0x7f434c101e10 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4343e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c102760 0x7f434c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:51988/0 (socket says 172.21.15.144:51988) 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 -- 172.21.15.144:0/2653028006 learned_addr learned my addr 172.21.15.144:0/2653028006 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4343e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c102760 0x7f434c19c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51488/0 (socket says 172.21.15.144:51488) 2024-07-26T11:21:00.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f434a800640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c10f9c0 0x7f434c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 -- 172.21.15.144:0/2653028006 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c10f9c0 msgr2=0x7f434c1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c10f9c0 0x7f434c1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 -- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c102760 msgr2=0x7f434c19c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c102760 0x7f434c19c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.019+0000 7f4349e00640 1 -- 172.21.15.144:0/2653028006 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c1a1e80 con 0x7f434c101e10 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.020+0000 7f4349e00640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c19bf10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e9f0 tx=0x7f433c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.020+0000 7f4342a00640 1 -- 172.21.15.144:0/2653028006 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c00cd80 con 0x7f434c101e10 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.020+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c1a2170 con 0x7f434c101e10 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.020+0000 7f4342a00640 1 -- 172.21.15.144:0/2653028006 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c0054d0 con 0x7f434c101e10 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.020+0000 7f4342a00640 1 -- 172.21.15.144:0/2653028006 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c010690 con 0x7f434c101e10 2024-07-26T11:21:00.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.020+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434c1a88d0 con 0x7f434c101e10 2024-07-26T11:21:00.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.023+0000 7f4357aa1740 1 Processor -- start 2024-07-26T11:21:00.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 -- start start 2024-07-26T11:21:00.025 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4342a00640 1 -- 172.21.15.144:0/2653028006 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f433c004e90 con 0x7f434c101e10 2024-07-26T11:21:00.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c12761ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127620510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c127620a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55c127596b90 con 0x55c12725a390 2024-07-26T11:21:00.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c127586090 con 0x55c1272521b0 2024-07-26T11:21:00.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f4357aa1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c12758a0d0 con 0x55c12727b070 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.024+0000 7f434a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c12761ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4349e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127620510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f434a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c12761ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:51992/0 (socket says 172.21.15.144:51992) 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f434a800640 1 -- 172.21.15.144:0/1550317924 learned_addr learned my addr 172.21.15.144:0/1550317924 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4343e00640 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c127620a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4349e00640 1 -- 172.21.15.144:0/1550317924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 msgr2=0x55c12761ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4349e00640 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c12761ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4349e00640 1 -- 172.21.15.144:0/1550317924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 msgr2=0x55c127620a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4349e00640 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c127620a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f4349e00640 1 -- 172.21.15.144:0/1550317924 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1272e6650 con 0x55c1272521b0 2024-07-26T11:21:00.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.025+0000 7f434a800640 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c12761ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:00.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.026+0000 7f4349e00640 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127620510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f433c0e1be0 tx=0x7f433c0ebb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d500e2df865d5e15 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:00.028 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.026+0000 7f4342a00640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3d40 0x7f43300b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.026+0000 7f433b400640 1 -- 172.21.15.144:0/1550317924 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c0f4070 con 0x55c1272521b0 2024-07-26T11:21:00.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.026+0000 7f433b400640 1 -- 172.21.15.144:0/1550317924 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c0aabb0 con 0x55c1272521b0 2024-07-26T11:21:00.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.026+0000 7f433b400640 1 -- 172.21.15.144:0/1550317924 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c0d8ba0 con 0x55c1272521b0 2024-07-26T11:21:00.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.027+0000 7f4342a00640 1 -- 172.21.15.144:0/2653028006 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f433c0e57d0 con 0x7f434c101e10 2024-07-26T11:21:00.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.027+0000 7f434a800640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3d40 0x7f43300b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.027+0000 7f4357aa1740 1 -- 172.21.15.144:0/1550317924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 msgr2=0x55c127620510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.027+0000 7f4357aa1740 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127620510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f433c0e1be0 tx=0x7f433c0ebb20 comp rx=0 tx=0).stop 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.027+0000 7f434a800640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3d40 0x7f43300b6250 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f43440099a0 tx=0x7f4344009500 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 -- 172.21.15.144:0/1550317924 shutdown_connections 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c127620a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127620510 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 --2- 172.21.15.144:0/1550317924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c12761ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 -- 172.21.15.144:0/1550317924 >> 172.21.15.144:0/1550317924 conn(0x55c12726f460 msgr2=0x55c12761dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 -- 172.21.15.144:0/1550317924 shutdown_connections 2024-07-26T11:21:00.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.028+0000 7f4357aa1740 1 -- 172.21.15.144:0/1550317924 wait complete. 2024-07-26T11:21:00.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 -- start start 2024-07-26T11:21:00.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 Processor -- start 2024-07-26T11:21:00.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c1272521b0 0x55c12762b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c12762ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.031 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12727b070 0x55c12762bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55c1273e9700 con 0x55c12725a390 2024-07-26T11:21:00.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c127522660 con 0x55c12727b070 2024-07-26T11:21:00.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.029+0000 7f4357aa1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c12762e7b0 con 0x55c1272521b0 2024-07-26T11:21:00.032 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4357aa1740 10 client.0 _ll_register_callbacks cb 0x55c127532110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c12762ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4349e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c1272521b0 0x55c12762b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c12762ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51508/0 (socket says 172.21.15.144:51508) 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 -- 172.21.15.144:0/289212806 learned_addr learned my addr 172.21.15.144:0/289212806 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f434a800640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12727b070 0x55c12762bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 -- 172.21.15.144:0/289212806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c1272521b0 msgr2=0x55c12762b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c1272521b0 0x55c12762b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 -- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12727b070 msgr2=0x55c12762bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12727b070 0x55c12762bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4343e00640 1 -- 172.21.15.144:0/289212806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c127539800 con 0x55c12725a390 2024-07-26T11:21:00.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f4349e00640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c1272521b0 0x55c12762b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:00.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.030+0000 7f434a800640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12727b070 0x55c12762bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:00.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.031+0000 7f4343e00640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c12762ba80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f4334004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcd6a46bac36dbf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:00.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.031+0000 7f432fe00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:21:00.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.031+0000 7f432fe00640 1 -- 172.21.15.144:0/289212806 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334035040 con 0x55c12725a390 2024-07-26T11:21:00.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.031+0000 7f432fe00640 1 -- 172.21.15.144:0/289212806 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f4334005450 con 0x55c12725a390 2024-07-26T11:21:00.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.031+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f433c0aabb0 con 0x55c12725a390 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f432fe00640 1 -- 172.21.15.144:0/289212806 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f4334050050 con 0x55c12725a390 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 10 client.16509 fetch_fsmap learned FSMap version 333 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55c12763a380 con 0x55c12725a390 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f432fe00640 1 -- 172.21.15.144:0/289212806 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c0aabb0 con 0x55c12725a390 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f432fe00640 1 -- 172.21.15.144:0/289212806 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f4334003a30 con 0x55c12725a390 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 10 client.16509 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 20 client.16509 populate_metadata read hostname 'smithi144' 2024-07-26T11:21:00.035 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c1272399f0 0x55c12763dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.036 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 4 client.16509 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:00.036 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.032+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c127598ae0 con 0x55c1272399f0 2024-07-26T11:21:00.036 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.033+0000 7f4349e00640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c1272399f0 0x55c12763dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.036 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.033+0000 7f4349e00640 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c1272399f0 0x55c12763dcf0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7e6713df4922f56 server_cookie=58a2abfc5d094e01 in_seq=0 out_seq=0 2024-07-26T11:21:00.036 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.033+0000 7f432fe00640 10 client.16509 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.034+0000 7f432fe00640 1 -- 172.21.15.144:0/289212806 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c127598ae0 con 0x55c1272399f0 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.034+0000 7f432fe00640 10 client.16509 handle_command_reply: tid=0 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.034+0000 7f4357aa1740 1 client.16509 shutdown 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 msgr2=0x55c12762ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c12762ba80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f4334004440 comp rx=0 tx=0).stop 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 shutdown_connections 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c1272399f0 0x55c12763dcf0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12727b070 0x55c12762bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12725a390 0x55c12762ba80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 --2- 172.21.15.144:0/289212806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c1272521b0 0x55c12762b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 >> 172.21.15.144:0/289212806 conn(0x55c12726f460 msgr2=0x55c127621000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:00.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f432fe00640 0 client.16509 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:00.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 shutdown_connections 2024-07-26T11:21:00.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 1 -- 172.21.15.144:0/289212806 wait complete. 2024-07-26T11:21:00.038 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.035+0000 7f4357aa1740 20 client.16509 trim_cache size 0 max 16384 2024-07-26T11:21:00.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 Processor -- start 2024-07-26T11:21:00.055 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 -- start start 2024-07-26T11:21:00.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c1272521b0 0x55c12761cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12725a390 0x55c12762d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c127637890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1b4e20 con 0x55c1272521b0 2024-07-26T11:21:00.056 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1b32e0 con 0x55c12725a390 2024-07-26T11:21:00.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4357aa1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1b4750 con 0x55c12727b070 2024-07-26T11:21:00.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.054+0000 7f4349e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c1272521b0 0x55c12761cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4349e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c1272521b0 0x55c12761cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51518/0 (socket says 172.21.15.144:51518) 2024-07-26T11:21:00.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12725a390 0x55c12762d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4349e00640 1 -- 172.21.15.144:0/749835300 learned_addr learned my addr 172.21.15.144:0/749835300 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:00.057 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f434a800640 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c127637890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 -- 172.21.15.144:0/749835300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 msgr2=0x55c127637890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c127637890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 -- 172.21.15.144:0/749835300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c1272521b0 msgr2=0x55c12761cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c1272521b0 0x55c12761cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 -- 172.21.15.144:0/749835300 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c12762c800 con 0x55c12725a390 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.055+0000 7f4343e00640 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12725a390 0x55c12762d170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4334009900 tx=0x7f4334001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2f72a88365b885 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.056+0000 7f433aa00640 1 -- 172.21.15.144:0/749835300 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334035070 con 0x55c12725a390 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.056+0000 7f433aa00640 1 -- 172.21.15.144:0/749835300 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4334041070 con 0x55c12725a390 2024-07-26T11:21:00.058 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.056+0000 7f433aa00640 1 -- 172.21.15.144:0/749835300 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43340319b0 con 0x55c12725a390 2024-07-26T11:21:00.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 -- 172.21.15.144:0/749835300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12725a390 msgr2=0x55c12762d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12725a390 0x55c12762d170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4334009900 tx=0x7f4334001e70 comp rx=0 tx=0).stop 2024-07-26T11:21:00.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 -- 172.21.15.144:0/749835300 shutdown_connections 2024-07-26T11:21:00.059 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12727b070 0x55c127637890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c12725a390 0x55c12762d170 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 --2- 172.21.15.144:0/749835300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c1272521b0 0x55c12761cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 -- 172.21.15.144:0/749835300 >> 172.21.15.144:0/749835300 conn(0x55c12726f460 msgr2=0x55c12762ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 -- 172.21.15.144:0/749835300 shutdown_connections 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 -- 172.21.15.144:0/749835300 wait complete. 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.057+0000 7f4357aa1740 1 -- start start 2024-07-26T11:21:00.060 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 Processor -- start 2024-07-26T11:21:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127634160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12725a390 0x55c127630620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12727b070 0x55c127630be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1b4080 con 0x55c12727b070 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1b39b0 con 0x55c1272521b0 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c127598ae0 con 0x55c12725a390 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4357aa1740 10 client.0 _ll_register_callbacks cb 0x7f43340040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4349e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127634160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4349e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127634160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:52956/0 (socket says 172.21.15.144:52956) 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.058+0000 7f4349e00640 1 -- 172.21.15.144:0/4018446324 learned_addr learned my addr 172.21.15.144:0/4018446324 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f434a800640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12727b070 0x55c127630be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4343e00640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12725a390 0x55c127630620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4349e00640 1 -- 172.21.15.144:0/4018446324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12725a390 msgr2=0x55c127630620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4349e00640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12725a390 0x55c127630620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4349e00640 1 -- 172.21.15.144:0/4018446324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12727b070 msgr2=0x55c127630be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4349e00640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12727b070 0x55c127630be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4349e00640 1 -- 172.21.15.144:0/4018446324 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c127631370 con 0x55c1272521b0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4349e00640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127634160 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f433c0dcf20 tx=0x7f433c009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fce93d3e75ea6bf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f432fe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f432fe00640 1 -- 172.21.15.144:0/4018446324 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c0f4030 con 0x55c1272521b0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4334041070 con 0x55c1272521b0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.059+0000 7f432fe00640 1 -- 172.21.15.144:0/4018446324 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f433c0eb300 con 0x55c1272521b0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f432fe00640 1 -- 172.21.15.144:0/4018446324 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f433c0dba10 con 0x55c1272521b0 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 10 client.26744 fetch_fsmap learned FSMap version 333 2024-07-26T11:21:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55c1276a54c0 con 0x55c1272521b0 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f432fe00640 1 -- 172.21.15.144:0/4018446324 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4334041070 con 0x55c1272521b0 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f432fe00640 1 -- 172.21.15.144:0/4018446324 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f433c0e1e60 con 0x55c1272521b0 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 10 client.26744 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 10 client.26744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 20 client.26744 populate_metadata read hostname 'smithi144' 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c12720eda0 0x55c1276a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 4 client.26744 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.060+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c127594580 con 0x55c12720eda0 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.061+0000 7f4343e00640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c12720eda0 0x55c1276a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.061+0000 7f4343e00640 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c12720eda0 0x55c1276a7c60 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e74c1714cbec944 server_cookie=135979fbe1498c58 in_seq=0 out_seq=0 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.061+0000 7f432fe00640 10 client.26744 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.061+0000 7f432fe00640 1 -- 172.21.15.144:0/4018446324 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c127594580 con 0x55c12720eda0 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.061+0000 7f432fe00640 10 client.26744 handle_command_reply: tid=0 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.061+0000 7f4357aa1740 1 client.26744 shutdown 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 msgr2=0x55c127634160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127634160 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f433c0dcf20 tx=0x7f433c009720 comp rx=0 tx=0).stop 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 shutdown_connections 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c12727b070 0x55c127630be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c12725a390 0x55c127630620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55c12720eda0 0x55c1276a7c60 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 --2- 172.21.15.144:0/4018446324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c1272521b0 0x55c127634160 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 >> 172.21.15.144:0/4018446324 conn(0x55c12726f460 msgr2=0x55c12761cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f432fe00640 0 client.26744 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 shutdown_connections 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 1 -- 172.21.15.144:0/4018446324 wait complete. 2024-07-26T11:21:00.067 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.062+0000 7f4357aa1740 20 client.26744 trim_cache size 0 max 16384 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:21:00.068 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:21:00.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:21:00.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:21:00.069 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:21:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3d40 msgr2=0x7f43300b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3d40 0x7f43300b6250 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f43440099a0 tx=0x7f4344009500 comp rx=0 tx=0).stop 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 msgr2=0x7f434c19bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c19bf10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e9f0 tx=0x7f433c00edf0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 shutdown_connections 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f43300b3d40 0x7f43300b6250 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f434c10f9c0 0x7f434c1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434c102760 0x7f434c19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 --2- 172.21.15.144:0/2653028006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f434c101e10 0x7f434c19bf10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 >> 172.21.15.144:0/2653028006 conn(0x7f434c0fdcf0 msgr2=0x7f434c111b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 shutdown_connections 2024-07-26T11:21:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:00.063+0000 7f434b200640 1 -- 172.21.15.144:0/2653028006 wait complete. 2024-07-26T11:21:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:00 smithi183 ceph-mon[37273]: pgmap v1076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:00 smithi144 ceph-mon[32129]: pgmap v1076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:00 smithi160 ceph-mon[36715]: pgmap v1076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:01.045 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:21:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 904.545387 secs 2024-07-26T11:21:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 904.545387 secs 2024-07-26T11:21:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 904.545387 secs 2024-07-26T11:21:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:02 smithi183 ceph-mon[37273]: pgmap v1077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:02 smithi144 ceph-mon[32129]: pgmap v1077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:02 smithi160 ceph-mon[36715]: pgmap v1077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:04 smithi183 ceph-mon[37273]: pgmap v1078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:21:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:04 smithi144 ceph-mon[32129]: pgmap v1078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:21:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:04 smithi160 ceph-mon[36715]: pgmap v1078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:21:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:05 smithi183 ceph-mon[37273]: pgmap v1079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:05 smithi183 ceph-mon[37273]: pgmap v1080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:21:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:05 smithi144 ceph-mon[32129]: pgmap v1079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:05 smithi144 ceph-mon[32129]: pgmap v1080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:21:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:05 smithi160 ceph-mon[36715]: pgmap v1079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:05 smithi160 ceph-mon[36715]: pgmap v1080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:21:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 909.545453 secs 2024-07-26T11:21:06.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 909.545453 secs 2024-07-26T11:21:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 909.545453 secs 2024-07-26T11:21:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:07 smithi183 ceph-mon[37273]: pgmap v1081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:07 smithi144 ceph-mon[32129]: pgmap v1081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:07 smithi160 ceph-mon[36715]: pgmap v1081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:09 smithi144 ceph-mon[32129]: pgmap v1082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:09 smithi160 ceph-mon[36715]: pgmap v1082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:09 smithi183 ceph-mon[37273]: pgmap v1082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:11 smithi144 ceph-mon[32129]: pgmap v1083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 914.545532 secs 2024-07-26T11:21:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:11 smithi160 ceph-mon[36715]: pgmap v1083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 914.545532 secs 2024-07-26T11:21:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:11 smithi183 ceph-mon[37273]: pgmap v1083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 914.545532 secs 2024-07-26T11:21:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:12.405+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:12.407+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:12.407+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:12.407+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:12.407+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:13 smithi144 ceph-mon[32129]: pgmap v1084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:13 smithi160 ceph-mon[36715]: pgmap v1084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:13 smithi183 ceph-mon[37273]: pgmap v1084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:21:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:15 smithi144 ceph-mon[32129]: pgmap v1085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:15 smithi160 ceph-mon[36715]: pgmap v1085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:15 smithi183 ceph-mon[37273]: pgmap v1085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 919.545614 secs 2024-07-26T11:21:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 919.545614 secs 2024-07-26T11:21:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 919.545614 secs 2024-07-26T11:21:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:17 smithi144 ceph-mon[32129]: pgmap v1086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:17 smithi160 ceph-mon[36715]: pgmap v1086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:17 smithi183 ceph-mon[37273]: pgmap v1086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:19 smithi144 ceph-mon[32129]: pgmap v1087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:19 smithi160 ceph-mon[36715]: pgmap v1087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:19 smithi183 ceph-mon[37273]: pgmap v1087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:21.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:21 smithi183 ceph-mon[37273]: pgmap v1088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:21.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 924.545750 secs 2024-07-26T11:21:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:21 smithi144 ceph-mon[32129]: pgmap v1088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 924.545750 secs 2024-07-26T11:21:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:21 smithi160 ceph-mon[36715]: pgmap v1088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 924.545750 secs 2024-07-26T11:21:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:23 smithi144 ceph-mon[32129]: pgmap v1089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:23 smithi160 ceph-mon[36715]: pgmap v1089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:23 smithi183 ceph-mon[37273]: pgmap v1089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:25 smithi144 ceph-mon[32129]: pgmap v1090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:25 smithi160 ceph-mon[36715]: pgmap v1090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:25 smithi183 ceph-mon[37273]: pgmap v1090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 929.545830 secs 2024-07-26T11:21:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 929.545830 secs 2024-07-26T11:21:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 929.545830 secs 2024-07-26T11:21:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:27.419+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:27.420+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:27.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:27.420+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:27.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:27.420+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:27.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:27.421+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:27 smithi144 ceph-mon[32129]: pgmap v1091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:27 smithi160 ceph-mon[36715]: pgmap v1091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:27 smithi183 ceph-mon[37273]: pgmap v1091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:29 smithi144 ceph-mon[32129]: pgmap v1092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:29 smithi160 ceph-mon[36715]: pgmap v1092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:29 smithi183 ceph-mon[37273]: pgmap v1092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:31.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:21:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:31 smithi144 ceph-mon[32129]: pgmap v1093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 934.545970 secs 2024-07-26T11:21:31.505 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:21:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:31 smithi160 ceph-mon[36715]: pgmap v1093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 934.545970 secs 2024-07-26T11:21:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:31 smithi183 ceph-mon[37273]: pgmap v1093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 934.545970 secs 2024-07-26T11:21:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:33 smithi144 ceph-mon[32129]: pgmap v1094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:33 smithi160 ceph-mon[36715]: pgmap v1094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:33 smithi183 ceph-mon[37273]: pgmap v1094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:33.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 -- 172.21.15.144:0/771847201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98108d00 msgr2=0x7f4d98109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/771847201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98108d00 0x7f4d98109110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009900 tx=0x7f4d8402a410 comp rx=0 tx=0).stop 2024-07-26T11:21:33.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 -- 172.21.15.144:0/771847201 shutdown_connections 2024-07-26T11:21:33.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/771847201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d98109650 0x7f4d9810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/771847201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98108d00 0x7f4d98109110 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.809 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/771847201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d981037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 -- 172.21.15.144:0/771847201 >> 172.21.15.144:0/771847201 conn(0x7f4d980fee10 msgr2=0x7f4d98101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:33.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 -- 172.21.15.144:0/771847201 shutdown_connections 2024-07-26T11:21:33.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.807+0000 7f4d9ca00640 1 -- 172.21.15.144:0/771847201 wait complete. 2024-07-26T11:21:33.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.808+0000 7f4d9ca00640 1 Processor -- start 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 -- start start 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d9819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d98108d00 0x7f4d9819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98109650 0x7f4d981a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981132e0 con 0x7f4d98109650 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4d98113160 con 0x7f4d981033c0 2024-07-26T11:21:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d9ca00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4d98113460 con 0x7f4d98108d00 2024-07-26T11:21:33.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98109650 0x7f4d981a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d97400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d9819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d97400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d9819bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:59988/0 (socket says 172.21.15.144:59988) 2024-07-26T11:21:33.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.809+0000 7f4d97400640 1 -- 172.21.15.144:0/3570964100 learned_addr learned my addr 172.21.15.144:0/3570964100 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d96a00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d98108d00 0x7f4d9819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97400640 1 -- 172.21.15.144:0/3570964100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d98108d00 msgr2=0x7f4d9819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97400640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d98108d00 0x7f4d9819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97400640 1 -- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98109650 msgr2=0x7f4d981a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97400640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98109650 0x7f4d981a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97400640 1 -- 172.21.15.144:0/3570964100 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d981a1ec0 con 0x7f4d981033c0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97e00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98109650 0x7f4d981a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.810+0000 7f4d97400640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d9819bf50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800e9f0 tx=0x7f4d8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.811+0000 7f4d8fe00640 1 -- 172.21.15.144:0/3570964100 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8800cd80 con 0x7f4d981033c0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.811+0000 7f4d8fe00640 1 -- 172.21.15.144:0/3570964100 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d880108e0 con 0x7f4d981033c0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.811+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d981a21b0 con 0x7f4d981033c0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.811+0000 7f4d8fe00640 1 -- 172.21.15.144:0/3570964100 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88018980 con 0x7f4d981033c0 2024-07-26T11:21:33.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.811+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d981a8910 con 0x7f4d981033c0 2024-07-26T11:21:33.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.814+0000 7f4da4c56740 1 Processor -- start 2024-07-26T11:21:33.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.814+0000 7f4da4c56740 1 -- start start 2024-07-26T11:21:33.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.814+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758336070 0x5627586dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.816 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586db510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 0x5627586dba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4da4c56740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x562758651b90 con 0x562758315390 2024-07-26T11:21:33.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4da4c56740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x562758641090 con 0x562758336070 2024-07-26T11:21:33.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4da4c56740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5627586450d0 con 0x56275830d1b0 2024-07-26T11:21:33.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4d96a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 0x5627586dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4d8fe00640 1 -- 172.21.15.144:0/3570964100 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f4d88010430 con 0x7f4d981033c0 2024-07-26T11:21:33.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758336070 0x5627586dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4d96a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 0x5627586dba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60274/0 (socket says 172.21.15.144:60274) 2024-07-26T11:21:33.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4d96a00640 1 -- 172.21.15.144:0/2307326546 learned_addr learned my addr 172.21.15.144:0/2307326546 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:33.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.815+0000 7f4d97400640 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586db510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d97e00640 1 -- 172.21.15.144:0/2307326546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 msgr2=0x5627586db510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586db510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d97e00640 1 -- 172.21.15.144:0/2307326546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 msgr2=0x5627586dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 0x5627586dba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d97e00640 1 -- 172.21.15.144:0/2307326546 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627583a1650 con 0x562758336070 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d96a00640 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 0x5627586dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758336070 0x5627586dafd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4d90005fd0 tx=0x7f4d90011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25419c6dd803e9aa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d83e00640 1 -- 172.21.15.144:0/2307326546 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90004d20 con 0x562758336070 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d83e00640 1 -- 172.21.15.144:0/2307326546 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d9000f9a0 con 0x562758336070 2024-07-26T11:21:33.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.816+0000 7f4d83e00640 1 -- 172.21.15.144:0/2307326546 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90004470 con 0x562758336070 2024-07-26T11:21:33.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.817+0000 7f4d8fe00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d7c0b3d40 0x7f4d7c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.817+0000 7f4d8fe00640 1 -- 172.21.15.144:0/3570964100 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f4d880e5650 con 0x7f4d981033c0 2024-07-26T11:21:33.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.817+0000 7f4d97400640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d7c0b3d40 0x7f4d7c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.817+0000 7f4da4c56740 1 -- 172.21.15.144:0/2307326546 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758336070 msgr2=0x5627586dafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.817+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758336070 0x5627586dafd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4d90005fd0 tx=0x7f4d90011bf0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 -- 172.21.15.144:0/2307326546 shutdown_connections 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758315390 0x5627586dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586db510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2307326546 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758336070 0x5627586dafd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 -- 172.21.15.144:0/2307326546 >> 172.21.15.144:0/2307326546 conn(0x56275832a460 msgr2=0x5627586d8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 -- 172.21.15.144:0/2307326546 shutdown_connections 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4da4c56740 1 -- 172.21.15.144:0/2307326546 wait complete. 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.818+0000 7f4d97400640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d7c0b3d40 0x7f4d7c0b6250 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5627586d9c50 tx=0x7f4d88000bc0 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 -- start start 2024-07-26T11:21:33.821 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 Processor -- start 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586e6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5627584a4700 con 0x562758336070 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5627585dd660 con 0x562758315390 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.819+0000 7f4da4c56740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5627586e97b0 con 0x56275830d1b0 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4da4c56740 10 client.0 _ll_register_callbacks cb 0x5627585ed110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586e6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:60006/0 (socket says 172.21.15.144:60006) 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 -- 172.21.15.144:0/3271860917 learned_addr learned my addr 172.21.15.144:0/3271860917 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d97e00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 -- 172.21.15.144:0/3271860917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 msgr2=0x5627586e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 -- 172.21.15.144:0/3271860917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 msgr2=0x5627586e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d96a00640 1 -- 172.21.15.144:0/3271860917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627585f4800 con 0x562758315390 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.820+0000 7f4d97e00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.821+0000 7f4d96a00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586e6a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4d84032e20 tx=0x7f4d84032e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=692c4f069340df81 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.821+0000 7f4d80c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:21:33.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.821+0000 7f4d80c00640 1 -- 172.21.15.144:0/3271860917 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84035040 con 0x562758315390 2024-07-26T11:21:33.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.821+0000 7f4d80c00640 1 -- 172.21.15.144:0/3271860917 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f4d84031760 con 0x562758315390 2024-07-26T11:21:33.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.821+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d9000f9a0 con 0x562758315390 2024-07-26T11:21:33.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4d80c00640 1 -- 172.21.15.144:0/3271860917 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f4d84004040 con 0x562758315390 2024-07-26T11:21:33.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 10 client.26756 fetch_fsmap learned FSMap version 333 2024-07-26T11:21:33.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x5627586f53a0 con 0x562758315390 2024-07-26T11:21:33.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4d80c00640 1 -- 172.21.15.144:0/3271860917 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d9000f9a0 con 0x562758315390 2024-07-26T11:21:33.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4d80c00640 1 -- 172.21.15.144:0/3271860917 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f4d840046c0 con 0x562758315390 2024-07-26T11:21:33.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 10 client.26756 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 20 client.26756 populate_metadata read hostname 'smithi144' 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582f49f0 0x5627586f8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 4 client.26756 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.822+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562758653ae0 con 0x5627582f49f0 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.823+0000 7f4d97e00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582f49f0 0x5627586f8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.823+0000 7f4d97e00640 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582f49f0 0x5627586f8d10 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c337a84769365241 server_cookie=36ec6c91dbe6d3d7 in_seq=0 out_seq=0 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.823+0000 7f4d80c00640 10 client.26756 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:33.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.824+0000 7f4d80c00640 1 -- 172.21.15.144:0/3271860917 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562758653ae0 con 0x5627582f49f0 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.824+0000 7f4d80c00640 10 client.26756 handle_command_reply: tid=0 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.824+0000 7f4da4c56740 1 client.26756 shutdown 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 msgr2=0x5627586e6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586e6a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4d84032e20 tx=0x7f4d84032e50 comp rx=0 tx=0).stop 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 shutdown_connections 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.828 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582f49f0 0x5627586f8d10 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586e6a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 --2- 172.21.15.144:0/3271860917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 >> 172.21.15.144:0/3271860917 conn(0x56275832a460 msgr2=0x5627586de160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4d80c00640 0 client.26756 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 shutdown_connections 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 1 -- 172.21.15.144:0/3271860917 wait complete. 2024-07-26T11:21:33.829 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.825+0000 7f4da4c56740 20 client.26756 trim_cache size 0 max 16384 2024-07-26T11:21:33.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 Processor -- start 2024-07-26T11:21:33.846 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 -- start start 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 0x5627586e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562758315390 0x5627586d7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586dc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981b4e00 con 0x562758336070 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981b32c0 con 0x56275830d1b0 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.844+0000 7f4da4c56740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981b4730 con 0x562758315390 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.845+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 0x5627586e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.845+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 0x5627586e83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:60012/0 (socket says 172.21.15.144:60012) 2024-07-26T11:21:33.847 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.845+0000 7f4d97e00640 1 -- 172.21.15.144:0/2791423804 learned_addr learned my addr 172.21.15.144:0/2791423804 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:33.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.845+0000 7f4d96a00640 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562758315390 0x5627586d7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.848 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.845+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586dc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.845+0000 7f4d97e00640 1 -- 172.21.15.144:0/2791423804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562758315390 msgr2=0x5627586d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562758315390 0x5627586d7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d97e00640 1 -- 172.21.15.144:0/2791423804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 msgr2=0x5627586dc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586dc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d97e00640 1 -- 172.21.15.144:0/2791423804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627586e77d0 con 0x56275830d1b0 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d96a00640 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562758315390 0x5627586d7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d97e00640 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 0x5627586e83f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4d90005c40 tx=0x7f4d90009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c25ed1d234ca7491 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d83400640 1 -- 172.21.15.144:0/2791423804 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9001e040 con 0x56275830d1b0 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d83400640 1 -- 172.21.15.144:0/2791423804 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d9000f9a0 con 0x56275830d1b0 2024-07-26T11:21:33.849 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.846+0000 7f4d83400640 1 -- 172.21.15.144:0/2791423804 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90004180 con 0x56275830d1b0 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 -- 172.21.15.144:0/2791423804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 msgr2=0x5627586e83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 0x5627586e83f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4d90005c40 tx=0x7f4d90009500 comp rx=0 tx=0).stop 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 -- 172.21.15.144:0/2791423804 shutdown_connections 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586dc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562758315390 0x5627586d7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 --2- 172.21.15.144:0/2791423804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56275830d1b0 0x5627586e83f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 -- 172.21.15.144:0/2791423804 >> 172.21.15.144:0/2791423804 conn(0x56275832a460 msgr2=0x5627586dc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 -- 172.21.15.144:0/2791423804 shutdown_connections 2024-07-26T11:21:33.850 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.847+0000 7f4da4c56740 1 -- 172.21.15.144:0/2791423804 wait complete. 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.848+0000 7f4da4c56740 1 -- start start 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.848+0000 7f4da4c56740 1 Processor -- start 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586ef300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586ec7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586ecee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981b4060 con 0x562758336070 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981b3990 con 0x562758315390 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5627586e8930 con 0x56275830d1b0 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4da4c56740 10 client.0 _ll_register_callbacks cb 0x7f4d84032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586ecee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4d96a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586ec7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4d96a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586ec7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:60020/0 (socket says 172.21.15.144:60020) 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4d97e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586ecee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60300/0 (socket says 172.21.15.144:60300) 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4d96a00640 1 -- 172.21.15.144:0/1154249523 learned_addr learned my addr 172.21.15.144:0/1154249523 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.849+0000 7f4d97e00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d96a00640 1 -- 172.21.15.144:0/1154249523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 msgr2=0x5627586ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d96a00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586ef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d96a00640 1 -- 172.21.15.144:0/1154249523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 msgr2=0x5627586ecee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d96a00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586ecee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d96a00640 1 -- 172.21.15.144:0/1154249523 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562758768cc0 con 0x562758315390 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d97e00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586ecee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d96a00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586ec7d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d84004a60 tx=0x7f4d840043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d57cc9789135c1a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:33.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d80c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d80c00640 1 -- 172.21.15.144:0/1154249523 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84058020 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4d80c00640 1 -- 172.21.15.144:0/1154249523 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f4d84003b40 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.850+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d9000f9a0 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4d80c00640 1 -- 172.21.15.144:0/1154249523 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f4d84035070 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 10 client.26762 fetch_fsmap learned FSMap version 333 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x562758769df0 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4d80c00640 1 -- 172.21.15.144:0/1154249523 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d9000f9a0 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4d80c00640 1 -- 172.21.15.144:0/1154249523 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f4d8403d970 con 0x562758315390 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 10 client.26762 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 10 client.26762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 20 client.26762 populate_metadata read hostname 'smithi144' 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582c9da0 0x56275876c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:33.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 4 client.26762 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:21:33.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56275864f580 con 0x5627582c9da0 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.851+0000 7f4d97e00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582c9da0 0x56275876c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4d80c00640 10 client.26762 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4d97e00640 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582c9da0 0x56275876c5a0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaeb5aa2d76d5f39 server_cookie=17122692f4fcb8cf in_seq=0 out_seq=0 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4d80c00640 1 -- 172.21.15.144:0/1154249523 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56275864f580 con 0x5627582c9da0 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4d80c00640 10 client.26762 handle_command_reply: tid=0 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4da4c56740 1 client.26762 shutdown 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 msgr2=0x5627586ec7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4da4c56740 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586ec7d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d84004a60 tx=0x7f4d840043f0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 shutdown_connections 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4da4c56740 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562758336070 0x5627586ecee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.852+0000 7f4da4c56740 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5627582c9da0 0x56275876c5a0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4da4c56740 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562758315390 0x5627586ec7d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4da4c56740 1 --2- 172.21.15.144:0/1154249523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56275830d1b0 0x5627586ef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4d80c00640 0 client.26762 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 >> 172.21.15.144:0/1154249523 conn(0x56275832a460 msgr2=0x5627586e8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 shutdown_connections 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4da4c56740 1 -- 172.21.15.144:0/1154249523 wait complete. 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.853+0000 7f4da4c56740 20 client.26762 trim_cache size 0 max 16384 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:21:33.858 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:21:33.859 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d7c0b3d40 msgr2=0x7f4d7c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d7c0b3d40 0x7f4d7c0b6250 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5627586d9c50 tx=0x7f4d88000bc0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 msgr2=0x7f4d9819bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d9819bf50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800e9f0 tx=0x7f4d8800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 shutdown_connections 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f4d7c0b3d40 0x7f4d7c0b6250 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d98109650 0x7f4d981a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4d98108d00 0x7f4d9819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 --2- 172.21.15.144:0/3570964100 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d981033c0 0x7f4d9819bf50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 >> 172.21.15.144:0/3570964100 conn(0x7f4d980fee10 msgr2=0x7f4d9810e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 shutdown_connections 2024-07-26T11:21:33.860 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:21:33.854+0000 7f4d9ca00640 1 -- 172.21.15.144:0/3570964100 wait complete. 2024-07-26T11:21:34.652 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:21:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:35 smithi160 ceph-mon[36715]: pgmap v1095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:35 smithi183 ceph-mon[37273]: pgmap v1095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:35 smithi144 ceph-mon[32129]: pgmap v1095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 939.546110 secs 2024-07-26T11:21:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 939.546110 secs 2024-07-26T11:21:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 939.546110 secs 2024-07-26T11:21:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:37 smithi160 ceph-mon[36715]: pgmap v1096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:37 smithi183 ceph-mon[37273]: pgmap v1096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:37 smithi144 ceph-mon[32129]: pgmap v1096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:39 smithi160 ceph-mon[36715]: pgmap v1097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:39 smithi183 ceph-mon[37273]: pgmap v1097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:39 smithi144 ceph-mon[32129]: pgmap v1097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:41.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:41 smithi144 ceph-mon[32129]: pgmap v1098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:41.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 944.546253 secs 2024-07-26T11:21:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:41 smithi160 ceph-mon[36715]: pgmap v1098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 944.546253 secs 2024-07-26T11:21:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:41 smithi183 ceph-mon[37273]: pgmap v1098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 944.546253 secs 2024-07-26T11:21:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:42.423+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:42.424+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:42.424+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:42.424+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:42.424+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:43 smithi160 ceph-mon[36715]: pgmap v1099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:43 smithi183 ceph-mon[37273]: pgmap v1099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:43 smithi144 ceph-mon[32129]: pgmap v1099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:21:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:45 smithi160 ceph-mon[36715]: pgmap v1100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:45 smithi183 ceph-mon[37273]: pgmap v1100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:45 smithi144 ceph-mon[32129]: pgmap v1100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:21:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 949.546408 secs 2024-07-26T11:21:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 949.546408 secs 2024-07-26T11:21:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 949.546408 secs 2024-07-26T11:21:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:47 smithi160 ceph-mon[36715]: pgmap v1101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:47 smithi183 ceph-mon[37273]: pgmap v1101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:47 smithi144 ceph-mon[32129]: pgmap v1101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:49 smithi160 ceph-mon[36715]: pgmap v1102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:49 smithi183 ceph-mon[37273]: pgmap v1102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:49 smithi144 ceph-mon[32129]: pgmap v1102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:21:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:51 smithi160 ceph-mon[36715]: pgmap v1103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 954.546550 secs 2024-07-26T11:21:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:51 smithi183 ceph-mon[37273]: pgmap v1103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 954.546550 secs 2024-07-26T11:21:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:51 smithi144 ceph-mon[32129]: pgmap v1103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 954.546550 secs 2024-07-26T11:21:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:53 smithi160 ceph-mon[36715]: pgmap v1104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:53 smithi183 ceph-mon[37273]: pgmap v1104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:53 smithi144 ceph-mon[32129]: pgmap v1104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:55 smithi160 ceph-mon[36715]: pgmap v1105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:55 smithi183 ceph-mon[37273]: pgmap v1105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:55 smithi144 ceph-mon[32129]: pgmap v1105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 959.546629 secs 2024-07-26T11:21:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 959.546629 secs 2024-07-26T11:21:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 959.546629 secs 2024-07-26T11:21:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:57.431+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:57.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:57.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:57.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:21:57.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:57 smithi183 ceph-mon[37273]: pgmap v1106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:57 smithi144 ceph-mon[32129]: pgmap v1106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:57 smithi160 ceph-mon[36715]: pgmap v1106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:21:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:21:59 smithi183 ceph-mon[37273]: pgmap v1107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:21:59 smithi144 ceph-mon[32129]: pgmap v1107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:21:59 smithi160 ceph-mon[36715]: pgmap v1107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:01 smithi183 ceph-mon[37273]: pgmap v1108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 964.546740 secs 2024-07-26T11:22:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:01 smithi144 ceph-mon[32129]: pgmap v1108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 964.546740 secs 2024-07-26T11:22:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:01 smithi160 ceph-mon[36715]: pgmap v1108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 964.546740 secs 2024-07-26T11:22:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:03 smithi183 ceph-mon[37273]: pgmap v1109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:03 smithi144 ceph-mon[32129]: pgmap v1109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:03 smithi160 ceph-mon[36715]: pgmap v1109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:04.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:22:04.712 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:05.095 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:22:05.608 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:05 smithi144 ceph-mon[32129]: pgmap v1110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:05.608 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:05.608 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:05.608 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:22:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:05 smithi183 ceph-mon[37273]: pgmap v1110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:22:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:05 smithi160 ceph-mon[36715]: pgmap v1110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:05.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:22:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:06 smithi183 ceph-mon[37273]: pgmap v1111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:22:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:06 smithi183 ceph-mon[37273]: pgmap v1112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:22:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 969.546894 secs 2024-07-26T11:22:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:06 smithi160 ceph-mon[36715]: pgmap v1111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:22:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:06 smithi160 ceph-mon[36715]: pgmap v1112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:22:06.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 969.546894 secs 2024-07-26T11:22:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:06 smithi144 ceph-mon[32129]: pgmap v1111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:22:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:06 smithi144 ceph-mon[32129]: pgmap v1112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:22:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 969.546894 secs 2024-07-26T11:22:07.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 -- 172.21.15.144:0/1621068047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 msgr2=0x7fbfc8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 --2- 172.21.15.144:0/1621068047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc8109130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009900 tx=0x7fbfb002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 -- 172.21.15.144:0/1621068047 shutdown_connections 2024-07-26T11:22:07.442 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 --2- 172.21.15.144:0/1621068047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbfc8109670 0x7fbfc810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.442 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 --2- 172.21.15.144:0/1621068047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc8109130 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.442 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 --2- 172.21.15.144:0/1621068047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 0x7fbfc81037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.442 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 -- 172.21.15.144:0/1621068047 >> 172.21.15.144:0/1621068047 conn(0x7fbfc80fee70 msgr2=0x7fbfc81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:07.442 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 -- 172.21.15.144:0/1621068047 shutdown_connections 2024-07-26T11:22:07.442 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.439+0000 7fbfccc00640 1 -- 172.21.15.144:0/1621068047 wait complete. 2024-07-26T11:22:07.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.440+0000 7fbfccc00640 1 Processor -- start 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.440+0000 7fbfccc00640 1 -- start start 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfccc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 0x7fbfc8199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfccc00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfccc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbfc8109670 0x7fbfc819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfccc00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc8113350 con 0x7fbfc8108d20 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfccc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81131d0 con 0x7fbfc8109670 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfccc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81134d0 con 0x7fbfc81033e0 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfc7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 0x7fbfc8199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41128/0 (socket says 172.21.15.144:41128) 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfc6a00640 1 -- 172.21.15.144:0/805378860 learned_addr learned my addr 172.21.15.144:0/805378860 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.441+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbfc8109670 0x7fbfc819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfc7400640 1 -- 172.21.15.144:0/805378860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbfc8109670 msgr2=0x7fbfc819f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfc7400640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbfc8109670 0x7fbfc819f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfc7400640 1 -- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 msgr2=0x7fbfc819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfc7400640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfc7400640 1 -- 172.21.15.144:0/805378860 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc819fd10 con 0x7fbfc81033e0 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfc7400640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 0x7fbfc8199da0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800e9f0 tx=0x7fbfb800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfbfe00640 1 -- 172.21.15.144:0/805378860 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb800ccc0 con 0x7fbfc81033e0 2024-07-26T11:22:07.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfbfe00640 1 -- 172.21.15.144:0/805378860 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb8005020 con 0x7fbfc81033e0 2024-07-26T11:22:07.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc81a0000 con 0x7fbfc81033e0 2024-07-26T11:22:07.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfbfe00640 1 -- 172.21.15.144:0/805378860 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb800ce60 con 0x7fbfc81033e0 2024-07-26T11:22:07.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.442+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc81a6760 con 0x7fbfc81033e0 2024-07-26T11:22:07.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.446+0000 7fbfd5034740 1 Processor -- start 2024-07-26T11:22:07.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.446+0000 7fbfd5034740 1 -- start start 2024-07-26T11:22:07.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.446+0000 7fbfbfe00640 1 -- 172.21.15.144:0/805378860 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fbfb80051d0 con 0x7fbfc81033e0 2024-07-26T11:22:07.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.446+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 0x55d6f0775000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.448 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.446+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f0775540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f0775a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfd5034740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55d6f06ebbc0 con 0x55d6f03a71b0 2024-07-26T11:22:07.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfd5034740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d6f06db0c0 con 0x55d6f03af390 2024-07-26T11:22:07.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfd5034740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d6f06df100 con 0x55d6f03d0070 2024-07-26T11:22:07.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f0775540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 0x55d6f0775000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f0775540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41144/0 (socket says 172.21.15.144:41144) 2024-07-26T11:22:07.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 0x55d6f0775000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57502/0 (socket says 172.21.15.144:57502) 2024-07-26T11:22:07.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7400640 1 -- 172.21.15.144:0/3635221569 learned_addr learned my addr 172.21.15.144:0/3635221569 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:07.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f0775a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7e00640 1 -- 172.21.15.144:0/3635221569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 msgr2=0x55d6f0775a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f0775a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.447+0000 7fbfc7e00640 1 -- 172.21.15.144:0/3635221569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 msgr2=0x55d6f0775540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f0775540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfc7e00640 1 -- 172.21.15.144:0/3635221569 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6f043b650 con 0x55d6f03d0070 2024-07-26T11:22:07.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfc7400640 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f0775540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:07.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 0x55d6f0775000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009f10 tx=0x7fbfc0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=249552fbfca199ab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:07.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfb7e00640 1 -- 172.21.15.144:0/3635221569 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0004d20 con 0x55d6f03d0070 2024-07-26T11:22:07.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfb7e00640 1 -- 172.21.15.144:0/3635221569 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc0004030 con 0x55d6f03d0070 2024-07-26T11:22:07.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.448+0000 7fbfb7e00640 1 -- 172.21.15.144:0/3635221569 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0005280 con 0x55d6f03d0070 2024-07-26T11:22:07.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfbfe00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbfac0b3d40 0x7fbfac0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.456 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfbfe00640 1 -- 172.21.15.144:0/805378860 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fbfb80e52b0 con 0x7fbfc81033e0 2024-07-26T11:22:07.456 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfc7400640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbfac0b3d40 0x7fbfac0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.456 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 -- 172.21.15.144:0/3635221569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 msgr2=0x55d6f0775000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.456 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 0x55d6f0775000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009f10 tx=0x7fbfc0011810 comp rx=0 tx=0).stop 2024-07-26T11:22:07.456 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 -- 172.21.15.144:0/3635221569 shutdown_connections 2024-07-26T11:22:07.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f0775a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f0775540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3635221569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03d0070 0x55d6f0775000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.449+0000 7fbfd5034740 1 -- 172.21.15.144:0/3635221569 >> 172.21.15.144:0/3635221569 conn(0x55d6f03c4460 msgr2=0x55d6f0772ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:07.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.450+0000 7fbfd5034740 1 -- 172.21.15.144:0/3635221569 shutdown_connections 2024-07-26T11:22:07.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.450+0000 7fbfd5034740 1 -- 172.21.15.144:0/3635221569 wait complete. 2024-07-26T11:22:07.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.450+0000 7fbfc7400640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbfac0b3d40 0x7fbfac0b6250 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80e1720 tx=0x7fbfb80e8d60 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:07.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.450+0000 7fbfd5034740 1 -- start start 2024-07-26T11:22:07.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.450+0000 7fbfd5034740 1 Processor -- start 2024-07-26T11:22:07.462 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.462 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 0x55d6f077cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.462 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f077d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.462 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x55d6f053e700 con 0x55d6f03af390 2024-07-26T11:22:07.462 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d6f0677690 con 0x55d6f03d0070 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d6f078bf70 con 0x55d6f03a71b0 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfd5034740 10 client.0 _ll_register_callbacks cb 0x55d6f0687140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 0x55d6f077cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 0x55d6f077cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41154/0 (socket says 172.21.15.144:41154) 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.451+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2826313326 learned_addr learned my addr 172.21.15.144:0/2826313326 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f077d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2826313326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 msgr2=0x55d6f077c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 msgr2=0x55d6f077d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.463 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f077d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2826313326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6f068e830 con 0x55d6f03af390 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f077d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 0x55d6f077cac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002a910 tx=0x7fbfb00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e668c1c23582b1c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.452+0000 7fbfb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2826313326 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0035040 con 0x55d6f03af390 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2826313326 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fbfb0005220 con 0x55d6f03af390 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfc0004030 con 0x55d6f03af390 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2826313326 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7fbfb004f050 con 0x55d6f03af390 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfd5034740 10 client.16545 fetch_fsmap learned FSMap version 333 2024-07-26T11:22:07.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55d6f0797ae0 con 0x55d6f03af390 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.453+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2826313326 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc0004030 con 0x55d6f03af390 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2826313326 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7fbfb0003780 con 0x55d6f03af390 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfd5034740 10 client.16545 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfd5034740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfd5034740 20 client.16545 populate_metadata read hostname 'smithi144' 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f038e9f0 0x55d6f079b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfd5034740 4 client.16545 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6f06edb10 con 0x55d6f038e9f0 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.454+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f038e9f0 0x55d6f079b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.455+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f038e9f0 0x55d6f079b470 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c1189295643aa01 server_cookie=752d050d689b1ed7 in_seq=0 out_seq=0 2024-07-26T11:22:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.455+0000 7fbfb4c00640 10 client.16545 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2826313326 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d6f06edb10 con 0x55d6f038e9f0 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfb4c00640 10 client.16545 handle_command_reply: tid=0 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 client.16545 shutdown 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 msgr2=0x55d6f077cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 0x55d6f077cac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002a910 tx=0x7fbfb00043b0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 shutdown_connections 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f038e9f0 0x55d6f079b470 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f077d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03af390 0x55d6f077cac0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2826313326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfb4c00640 0 client.16545 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:07.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.456+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 >> 172.21.15.144:0/2826313326 conn(0x55d6f03c4460 msgr2=0x55d6f07769f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:07.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.457+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 shutdown_connections 2024-07-26T11:22:07.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.457+0000 7fbfd5034740 1 -- 172.21.15.144:0/2826313326 wait complete. 2024-07-26T11:22:07.467 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.457+0000 7fbfd5034740 20 client.16545 trim_cache size 0 max 16384 2024-07-26T11:22:07.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.475+0000 7fbfd5034740 1 Processor -- start 2024-07-26T11:22:07.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 -- start start 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f078baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03d0070 0x55d6f0795090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81b0a60 con 0x55d6f03d0070 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81aef20 con 0x55d6f03af390 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.476+0000 7fbfd5034740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81b0390 con 0x55d6f03a71b0 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f078baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f078baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:51882/0 (socket says 172.21.15.144:51882) 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 -- 172.21.15.144:0/3782423856 learned_addr learned my addr 172.21.15.144:0/3782423856 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03d0070 0x55d6f0795090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 -- 172.21.15.144:0/3782423856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 msgr2=0x55d6f077e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 -- 172.21.15.144:0/3782423856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03d0070 msgr2=0x55d6f0795090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03d0070 0x55d6f0795090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc6a00640 1 -- 172.21.15.144:0/3782423856 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6f078a830 con 0x55d6f03af390 2024-07-26T11:22:07.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.477+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03d0070 0x55d6f0795090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.478+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f078baa0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005ab0 tx=0x7fbfb0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=754da4e323d3932f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.478+0000 7fbfb7400640 1 -- 172.21.15.144:0/3782423856 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0000e60 con 0x55d6f03af390 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.478+0000 7fbfb7400640 1 -- 172.21.15.144:0/3782423856 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb0046e50 con 0x55d6f03af390 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.478+0000 7fbfb7400640 1 -- 172.21.15.144:0/3782423856 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0035030 con 0x55d6f03af390 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 -- 172.21.15.144:0/3782423856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 msgr2=0x55d6f078baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f078baa0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005ab0 tx=0x7fbfb0000bc0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 -- 172.21.15.144:0/3782423856 shutdown_connections 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03d0070 0x55d6f0795090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03af390 0x55d6f078baa0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 --2- 172.21.15.144:0/3782423856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03a71b0 0x55d6f077e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 -- 172.21.15.144:0/3782423856 >> 172.21.15.144:0/3782423856 conn(0x55d6f03c4460 msgr2=0x55d6f0776030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 -- 172.21.15.144:0/3782423856 shutdown_connections 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.479+0000 7fbfd5034740 1 -- 172.21.15.144:0/3782423856 wait complete. 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 -- start start 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 Processor -- start 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f07915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 0x55d6f078caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f078d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81afcc0 con 0x55d6f03a71b0 2024-07-26T11:22:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81af5f0 con 0x55d6f03d0070 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d6f0775e70 con 0x55d6f03af390 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.480+0000 7fbfd5034740 10 client.0 _ll_register_callbacks cb 0x7fbfb0004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 0x55d6f078caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc7e00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f07915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 0x55d6f078caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:57528/0 (socket says 172.21.15.144:57528) 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2317996902 learned_addr learned my addr 172.21.15.144:0/2317996902 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f078d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 msgr2=0x55d6f078d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f078d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2317996902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 msgr2=0x55d6f07915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f07915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc6a00640 1 -- 172.21.15.144:0/2317996902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6f078d570 con 0x55d6f03af390 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.481+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f078d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.482+0000 7fbfc6a00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 0x55d6f078caf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002a430 tx=0x7fbfb0003a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5424867f03633a9e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.482+0000 7fbfb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.482+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2317996902 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0035040 con 0x55d6f03af390 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.482+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2317996902 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7fbfb00314a0 con 0x55d6f03af390 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.482+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfb0046e50 con 0x55d6f03af390 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2317996902 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7fbfb00316a0 con 0x55d6f03af390 2024-07-26T11:22:07.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 10 client.26818 fetch_fsmap learned FSMap version 333 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x55d6f0803860 con 0x55d6f03af390 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2317996902 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb0046e50 con 0x55d6f03af390 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2317996902 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7fbfb0052070 con 0x55d6f03af390 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 10 client.26818 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 10 client.26818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 20 client.26818 populate_metadata read hostname 'smithi144' 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f0363da0 0x55d6f0805f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 4 client.26818 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.483+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d6f06e95b0 con 0x55d6f0363da0 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.484+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f0363da0 0x55d6f0805f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.484+0000 7fbfc7e00640 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f0363da0 0x55d6f0805f20 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9382f894a93e56d server_cookie=4989f2f2d933a3a0 in_seq=0 out_seq=0 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.484+0000 7fbfb4c00640 10 client.26818 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfb4c00640 1 -- 172.21.15.144:0/2317996902 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d6f06e95b0 con 0x55d6f0363da0 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfb4c00640 10 client.26818 handle_command_reply: tid=0 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 client.26818 shutdown 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 msgr2=0x55d6f078caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 0x55d6f078caf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002a430 tx=0x7fbfb0003a30 comp rx=0 tx=0).stop 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 shutdown_connections 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x55d6f0363da0 0x55d6f0805f20 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d6f03d0070 0x55d6f078d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfb4c00640 0 client.26818 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:07.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d6f03af390 0x55d6f078caf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 --2- 172.21.15.144:0/2317996902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d6f03a71b0 0x55d6f07915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 >> 172.21.15.144:0/2317996902 conn(0x55d6f03c4460 msgr2=0x55d6f077e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 shutdown_connections 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 1 -- 172.21.15.144:0/2317996902 wait complete. 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.485+0000 7fbfd5034740 20 client.26818 trim_cache size 0 max 16384 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:22:07.491 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:22:07.492 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:22:07.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbfac0b3d40 msgr2=0x7fbfac0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbfac0b3d40 0x7fbfac0b6250 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80e1720 tx=0x7fbfb80e8d60 comp rx=0 tx=0).stop 2024-07-26T11:22:07.493 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 msgr2=0x7fbfc8199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 0x7fbfc8199da0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800e9f0 tx=0x7fbfb800edf0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 shutdown_connections 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7fbfac0b3d40 0x7fbfac0b6250 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbfc8109670 0x7fbfc819f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc8108d20 0x7fbfc819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 --2- 172.21.15.144:0/805378860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc81033e0 0x7fbfc8199da0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.486+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 >> 172.21.15.144:0/805378860 conn(0x7fbfc80fee70 msgr2=0x7fbfc8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.487+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 shutdown_connections 2024-07-26T11:22:07.494 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:07.487+0000 7fbfccc00640 1 -- 172.21.15.144:0/805378860 wait complete. 2024-07-26T11:22:08.348 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:22:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:08 smithi183 ceph-mon[37273]: pgmap v1113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:08 smithi144 ceph-mon[32129]: pgmap v1113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:08 smithi160 ceph-mon[36715]: pgmap v1113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:10 smithi183 ceph-mon[37273]: pgmap v1114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:10 smithi144 ceph-mon[32129]: pgmap v1114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:10 smithi160 ceph-mon[36715]: pgmap v1114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 974.546975 secs 2024-07-26T11:22:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 974.546975 secs 2024-07-26T11:22:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 974.546975 secs 2024-07-26T11:22:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:12.437+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:12.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:12.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:12.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:12.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:12 smithi144 ceph-mon[32129]: pgmap v1115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:12 smithi160 ceph-mon[36715]: pgmap v1115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:12 smithi183 ceph-mon[37273]: pgmap v1115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:22:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:14 smithi144 ceph-mon[32129]: pgmap v1116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:22:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:14 smithi160 ceph-mon[36715]: pgmap v1116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:22:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:14 smithi183 ceph-mon[37273]: pgmap v1116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:22:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:16.260+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:22:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:16.260+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006340 con 0x563c6c4182f0 2024-07-26T11:22:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:16.260+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:22:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:16.261+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T11:22:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:16 smithi144 ceph-mon[32129]: pgmap v1117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:22:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 979.547092 secs 2024-07-26T11:22:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:16 smithi160 ceph-mon[36715]: pgmap v1117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:22:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 979.547092 secs 2024-07-26T11:22:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:16 smithi183 ceph-mon[37273]: pgmap v1117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:22:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 979.547092 secs 2024-07-26T11:22:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:18 smithi144 ceph-mon[32129]: pgmap v1118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:18 smithi160 ceph-mon[36715]: pgmap v1118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:18 smithi183 ceph-mon[37273]: pgmap v1118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:20 smithi144 ceph-mon[32129]: pgmap v1119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:20 smithi160 ceph-mon[36715]: pgmap v1119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:20 smithi183 ceph-mon[37273]: pgmap v1119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 984.547208 secs 2024-07-26T11:22:21.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 984.547208 secs 2024-07-26T11:22:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 984.547208 secs 2024-07-26T11:22:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:22 smithi144 ceph-mon[32129]: pgmap v1120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:22 smithi160 ceph-mon[36715]: pgmap v1120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:22 smithi183 ceph-mon[37273]: pgmap v1120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:23 smithi160 ceph-mon[36715]: pgmap v1121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:23 smithi183 ceph-mon[37273]: pgmap v1121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:23 smithi144 ceph-mon[32129]: pgmap v1121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:26 smithi160 ceph-mon[36715]: pgmap v1122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 989.547335 secs 2024-07-26T11:22:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:26 smithi183 ceph-mon[37273]: pgmap v1122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 989.547335 secs 2024-07-26T11:22:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:26 smithi144 ceph-mon[32129]: pgmap v1122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 989.547335 secs 2024-07-26T11:22:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:27.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:27.450+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:27.450+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:27.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:27.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:28 smithi160 ceph-mon[36715]: pgmap v1123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:28 smithi183 ceph-mon[37273]: pgmap v1123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:28 smithi144 ceph-mon[32129]: pgmap v1123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:30 smithi160 ceph-mon[36715]: pgmap v1124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:30 smithi183 ceph-mon[37273]: pgmap v1124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:30 smithi144 ceph-mon[32129]: pgmap v1124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 994.547402 secs 2024-07-26T11:22:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 994.547402 secs 2024-07-26T11:22:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 994.547402 secs 2024-07-26T11:22:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:32 smithi183 ceph-mon[37273]: pgmap v1125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:32 smithi144 ceph-mon[32129]: pgmap v1125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:32 smithi160 ceph-mon[36715]: pgmap v1125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:34 smithi183 ceph-mon[37273]: pgmap v1126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:34 smithi144 ceph-mon[32129]: pgmap v1126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:34 smithi160 ceph-mon[36715]: pgmap v1126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:36.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:36 smithi160 ceph-mon[36715]: pgmap v1127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:36.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 999.547485 secs 2024-07-26T11:22:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:36 smithi183 ceph-mon[37273]: pgmap v1127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 999.547485 secs 2024-07-26T11:22:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:36 smithi144 ceph-mon[32129]: pgmap v1127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 999.547485 secs 2024-07-26T11:22:38.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 493f92c6-4b3c-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:22:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:38 smithi183 ceph-mon[37273]: pgmap v1128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:38 smithi144 ceph-mon[32129]: pgmap v1128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:38.802 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/493f92c6-4b3c-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:22:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:38 smithi160 ceph-mon[36715]: pgmap v1128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:40.610 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:40 smithi144 ceph-mon[32129]: pgmap v1129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:40 smithi183 ceph-mon[37273]: pgmap v1129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:40 smithi160 ceph-mon[36715]: pgmap v1129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:41.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 -- 172.21.15.144:0/3704562673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c01033c0 msgr2=0x7f34c01037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 --2- 172.21.15.144:0/3704562673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c01033c0 0x7f34c01037d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009900 tx=0x7f34b002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 -- 172.21.15.144:0/3704562673 shutdown_connections 2024-07-26T11:22:41.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 --2- 172.21.15.144:0/3704562673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34c0109650 0x7f34c010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 --2- 172.21.15.144:0/3704562673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c0108d00 0x7f34c0109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 --2- 172.21.15.144:0/3704562673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c01033c0 0x7f34c01037d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 -- 172.21.15.144:0/3704562673 >> 172.21.15.144:0/3704562673 conn(0x7f34c00fee10 msgr2=0x7f34c0101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:41.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 -- 172.21.15.144:0/3704562673 shutdown_connections 2024-07-26T11:22:41.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.187+0000 7f34c5600640 1 -- 172.21.15.144:0/3704562673 wait complete. 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.188+0000 7f34c5600640 1 Processor -- start 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.188+0000 7f34c5600640 1 -- start start 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c01033c0 0x7f34c0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c5600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 0x7f34c019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c5600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34c0109650 0x7f34c019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c5600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0113350 con 0x7f34c0109650 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c5600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01131d0 con 0x7f34c0108d00 2024-07-26T11:22:41.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c5600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01134d0 con 0x7f34c01033c0 2024-07-26T11:22:41.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34b7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 0x7f34c019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34c0109650 0x7f34c019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34b7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 0x7f34c019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56780/0 (socket says 172.21.15.144:56780) 2024-07-26T11:22:41.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c01033c0 0x7f34c0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.189+0000 7f34b7400640 1 -- 172.21.15.144:0/217834799 learned_addr learned my addr 172.21.15.144:0/217834799 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34b7400640 1 -- 172.21.15.144:0/217834799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c01033c0 msgr2=0x7f34c0199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34b7400640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c01033c0 0x7f34c0199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34b7400640 1 -- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34c0109650 msgr2=0x7f34c019f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34b7400640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34c0109650 0x7f34c019f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34b7400640 1 -- 172.21.15.144:0/217834799 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c019fcb0 con 0x7f34c0108d00 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34b7400640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 0x7f34c019a2e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f34a800ed00 tx=0x7f34a800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34bd600640 1 -- 172.21.15.144:0/217834799 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003b70 con 0x7f34c0108d00 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34bd600640 1 -- 172.21.15.144:0/217834799 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34a8003d10 con 0x7f34c0108d00 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c019ffa0 con 0x7f34c0108d00 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34bd600640 1 -- 172.21.15.144:0/217834799 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8018860 con 0x7f34c0108d00 2024-07-26T11:22:41.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.190+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c01a6700 con 0x7f34c0108d00 2024-07-26T11:22:41.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 Processor -- start 2024-07-26T11:22:41.195 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 -- start start 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6dd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34bd600640 1 -- 172.21.15.144:0/217834799 <== mon.2 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f34a8018a00 con 0x7f34c0108d00 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6dd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6dde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5616dd64fb90 con 0x5616dd313390 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5616dd63f090 con 0x5616dd30b1b0 2024-07-26T11:22:41.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.194+0000 7f34cd873740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5616dd6430d0 con 0x5616dd334070 2024-07-26T11:22:41.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6dd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34b7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6dde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6dd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6dd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56784/0 (socket says 172.21.15.144:56784) 2024-07-26T11:22:41.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 -- 172.21.15.144:0/3966006892 learned_addr learned my addr 172.21.15.144:0/3966006892 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:41.197 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 -- 172.21.15.144:0/3966006892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 msgr2=0x5616dd6dd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6dd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 -- 172.21.15.144:0/3966006892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 msgr2=0x5616dd6dde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6dde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34bfe00640 1 -- 172.21.15.144:0/3966006892 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616dd39f650 con 0x5616dd30b1b0 2024-07-26T11:22:41.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.195+0000 7f34c4c00640 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6dd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:41.198 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34bfe00640 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6dd920 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009230 tx=0x7f34b0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6b8163164953bf5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:41.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34b5200640 1 -- 172.21.15.144:0/3966006892 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0035040 con 0x5616dd30b1b0 2024-07-26T11:22:41.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34b5200640 1 -- 172.21.15.144:0/3966006892 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00054f0 con 0x5616dd30b1b0 2024-07-26T11:22:41.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34b5200640 1 -- 172.21.15.144:0/3966006892 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00035f0 con 0x5616dd30b1b0 2024-07-26T11:22:41.199 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34bd600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34a40b3dc0 0x7f34a40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34bd600640 1 -- 172.21.15.144:0/217834799 <== mon.2 v2:172.21.15.160:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f34a80e5a30 con 0x7f34c0108d00 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.196+0000 7f34c4c00640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34a40b3dc0 0x7f34a40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 -- 172.21.15.144:0/3966006892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 msgr2=0x5616dd6dd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6dd920 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009230 tx=0x7f34b0004040 comp rx=0 tx=0).stop 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34c4c00640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34a40b3dc0 0x7f34a40b62d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009f80 tx=0x7f34b8005f90 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 -- 172.21.15.144:0/3966006892 shutdown_connections 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6dde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6dd920 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 --2- 172.21.15.144:0/3966006892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6dd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.197+0000 7f34cd873740 1 -- 172.21.15.144:0/3966006892 >> 172.21.15.144:0/3966006892 conn(0x5616dd328460 msgr2=0x5616dd6db2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:41.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.198+0000 7f34cd873740 1 -- 172.21.15.144:0/3966006892 shutdown_connections 2024-07-26T11:22:41.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.198+0000 7f34cd873740 1 -- 172.21.15.144:0/3966006892 wait complete. 2024-07-26T11:22:41.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.198+0000 7f34cd873740 1 -- start start 2024-07-26T11:22:41.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.198+0000 7f34cd873740 1 Processor -- start 2024-07-26T11:22:41.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6e8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6e8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.203 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x5616dd4a2700 con 0x5616dd313390 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5616dd5db660 con 0x5616dd30b1b0 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5616dd6ebb60 con 0x5616dd334070 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34cd873740 10 client.0 _ll_register_callbacks cb 0x5616dd5eb110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6e8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56798/0 (socket says 172.21.15.144:56798) 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34bfe00640 1 -- 172.21.15.144:0/361088837 learned_addr learned my addr 172.21.15.144:0/361088837 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34b7400640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6e8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.199+0000 7f34c4c00640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34bfe00640 1 -- 172.21.15.144:0/361088837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 msgr2=0x5616dd6e93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34bfe00640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6e93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34bfe00640 1 -- 172.21.15.144:0/361088837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 msgr2=0x5616dd6e8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34bfe00640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6e8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.204 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34bfe00640 1 -- 172.21.15.144:0/361088837 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616dd5f2800 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34bfe00640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6e8950 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f34b0005ab0 tx=0x7f34b002a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dcefada2832c34d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34ad600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34ad600640 1 -- 172.21.15.144:0/361088837 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0035040 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.200+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b00054f0 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34ad600640 1 -- 172.21.15.144:0/361088837 <== mon.2 v2:172.21.15.160:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f34b0004ee0 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34ad600640 1 -- 172.21.15.144:0/361088837 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f34b0009520 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 10 client.26792 fetch_fsmap learned FSMap version 333 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x5616dd6f7750 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34ad600640 1 -- 172.21.15.144:0/361088837 <== mon.2 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00054f0 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34ad600640 1 -- 172.21.15.144:0/361088837 <== mon.2 v2:172.21.15.160:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f34b0055030 con 0x5616dd30b1b0 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 10 client.26792 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:22:41.205 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 10 client.26792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:22:41.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 20 client.26792 populate_metadata read hostname 'smithi144' 2024-07-26T11:22:41.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2f29f0 0x5616dd6fad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 4 client.26792 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:41.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.201+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616dd651ae0 con 0x5616dd2f29f0 2024-07-26T11:22:41.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.202+0000 7f34b7400640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2f29f0 0x5616dd6fad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.202+0000 7f34b7400640 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2f29f0 0x5616dd6fad40 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45eae370e8499e55 server_cookie=9983b3fac52de091 in_seq=0 out_seq=0 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.202+0000 7f34ad600640 10 client.26792 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.203+0000 7f34ad600640 1 -- 172.21.15.144:0/361088837 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5616dd651ae0 con 0x5616dd2f29f0 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.203+0000 7f34ad600640 10 client.26792 handle_command_reply: tid=0 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.203+0000 7f34cd873740 1 client.26792 shutdown 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 msgr2=0x5616dd6e8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6e8950 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f34b0005ab0 tx=0x7f34b002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 shutdown_connections 2024-07-26T11:22:41.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd334070 0x5616dd6e93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6e8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2f29f0 0x5616dd6fad40 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 --2- 172.21.15.144:0/361088837 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd30b1b0 0x5616dd6e8950 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 >> 172.21.15.144:0/361088837 conn(0x5616dd328460 msgr2=0x5616dd6e0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34ad600640 0 client.26792 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 shutdown_connections 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 1 -- 172.21.15.144:0/361088837 wait complete. 2024-07-26T11:22:41.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.204+0000 7f34cd873740 20 client.26792 trim_cache size 0 max 16384 2024-07-26T11:22:41.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 Processor -- start 2024-07-26T11:22:41.224 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 -- start start 2024-07-26T11:22:41.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6e9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 0x5616dd6f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd334070 0x5616dd6deb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01b09e0 con 0x5616dd334070 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01aeea0 con 0x5616dd313390 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.223+0000 7f34cd873740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01b0310 con 0x5616dd30b1b0 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd334070 0x5616dd6deb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 0x5616dd6f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 0x5616dd6f4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.144:56800/0 (socket says 172.21.15.144:56800) 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 -- 172.21.15.144:0/3245286299 learned_addr learned my addr 172.21.15.144:0/3245286299 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:41.226 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6e9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.144:49190/0 (socket says 172.21.15.144:49190) 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 -- 172.21.15.144:0/3245286299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 msgr2=0x5616dd6e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6e9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 -- 172.21.15.144:0/3245286299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd334070 msgr2=0x5616dd6deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd334070 0x5616dd6deb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 -- 172.21.15.144:0/3245286299 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616dd6e9c10 con 0x5616dd313390 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.224+0000 7f34b7400640 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 0x5616dd6f4cd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f34a800cba0 tx=0x7f34a80e94e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2f52b89b02a40f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.225+0000 7f34afe00640 1 -- 172.21.15.144:0/3245286299 <== mon.2 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a80aac50 con 0x5616dd313390 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.225+0000 7f34afe00640 1 -- 172.21.15.144:0/3245286299 <== mon.2 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34a8009820 con 0x5616dd313390 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.225+0000 7f34afe00640 1 -- 172.21.15.144:0/3245286299 <== mon.2 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a80aaa70 con 0x5616dd313390 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.225+0000 7f34cd873740 1 -- 172.21.15.144:0/3245286299 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 msgr2=0x5616dd6f4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.228 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.225+0000 7f34cd873740 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 0x5616dd6f4cd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f34a800cba0 tx=0x7f34a80e94e0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 -- 172.21.15.144:0/3245286299 shutdown_connections 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd334070 0x5616dd6deb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd313390 0x5616dd6f4cd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 --2- 172.21.15.144:0/3245286299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 -- 172.21.15.144:0/3245286299 >> 172.21.15.144:0/3245286299 conn(0x5616dd328460 msgr2=0x5616dd6de410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 -- 172.21.15.144:0/3245286299 shutdown_connections 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 -- 172.21.15.144:0/3245286299 wait complete. 2024-07-26T11:22:41.229 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 -- start start 2024-07-26T11:22:41.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.226+0000 7f34cd873740 1 Processor -- start 2024-07-26T11:22:41.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.230 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd334070 0x5616dd7614c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01afc40 con 0x5616dd313390 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01af570 con 0x5616dd334070 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5616dd651ae0 con 0x5616dd30b1b0 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34cd873740 10 client.0 _ll_register_callbacks cb 0x7f34b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34b7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd334070 0x5616dd7614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34b7400640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6ecbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:52788/0 (socket says 172.21.15.144:52788) 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34b7400640 1 -- 172.21.15.144:0/3229295871 learned_addr learned my addr 172.21.15.144:0/3229295871 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.227+0000 7f34bfe00640 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34b7400640 1 -- 172.21.15.144:0/3229295871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 msgr2=0x5616dd6ec5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34b7400640 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6ec5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34b7400640 1 -- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd334070 msgr2=0x5616dd7614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34b7400640 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd334070 0x5616dd7614c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34b7400640 1 -- 172.21.15.144:0/3229295871 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616dd761b30 con 0x5616dd313390 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34b7400640 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6ecbb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f34a8000f80 tx=0x7f34a80aaa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa88bfa945a54f7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.228+0000 7f34ad600640 10 client.0 ms_handle_connect on v2:172.21.15.144:3300/0 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.229+0000 7f34ad600640 1 -- 172.21.15.144:0/3229295871 <== mon.0 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a80ab300 con 0x5616dd313390 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.229+0000 7f34ad600640 1 -- 172.21.15.144:0/3229295871 <== mon.0 v2:172.21.15.144:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 13350+0+0 (secure 0 0 0) 0x7f34a80e20c0 con 0x5616dd313390 2024-07-26T11:22:41.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.229+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34a8009820 con 0x5616dd313390 2024-07-26T11:22:41.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.229+0000 7f34ad600640 1 -- 172.21.15.144:0/3229295871 <== mon.0 v2:172.21.15.144:3300/0 3 ==== mon_get_version_reply(handle=1 version=333) ==== 24+0+0 (secure 0 0 0) 0x7f34a80e22c0 con 0x5616dd313390 2024-07-26T11:22:41.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.229+0000 7f34cd873740 10 client.16569 fetch_fsmap learned FSMap version 333 2024-07-26T11:22:41.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.229+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,fsmap=333,monmap=4+}) -- 0x5616dd762dc0 con 0x5616dd313390 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34ad600640 1 -- 172.21.15.144:0/3229295871 <== mon.0 v2:172.21.15.144:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34a8009820 con 0x5616dd313390 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34ad600640 1 -- 172.21.15.144:0/3229295871 <== mon.0 v2:172.21.15.144:3300/0 5 ==== fsmap(e 333) ==== 4131+0+0 (secure 0 0 0) 0x7f34a80ab560 con 0x5616dd313390 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34cd873740 10 client.16569 fetch_fsmap finished waiting for FSMap version 333 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34cd873740 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34cd873740 20 client.16569 populate_metadata read hostname 'smithi144' 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34cd873740 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2c7da0 0x5616dd7654f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34cd873740 4 client.16569 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 --> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616dd64d580 con 0x5616dd2c7da0 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.230+0000 7f34bfe00640 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2c7da0 0x5616dd7654f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.231+0000 7f34bfe00640 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2c7da0 0x5616dd7654f0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c38f4b42db02cd0 server_cookie=6c7f214f6acd0704 in_seq=0 out_seq=0 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.231+0000 7f34ad600640 10 client.16569 ms_handle_connect on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:41.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.231+0000 7f34ad600640 1 -- 172.21.15.144:0/3229295871 <== mds.0 v2:172.21.15.160:6836/4027087859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5616dd64d580 con 0x5616dd2c7da0 2024-07-26T11:22:41.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.231+0000 7f34ad600640 10 client.16569 handle_command_reply: tid=0 2024-07-26T11:22:41.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.231+0000 7f34cd873740 1 client.16569 shutdown 2024-07-26T11:22:41.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 msgr2=0x5616dd6ecbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6ecbb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f34a8000f80 tx=0x7f34a80aaa70 comp rx=0 tx=0).stop 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 shutdown_connections 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:6836/4027087859,v1:172.21.15.160:6837/4027087859] conn(0x5616dd2c7da0 0x5616dd7654f0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5616dd334070 0x5616dd7614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5616dd313390 0x5616dd6ecbb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 --2- 172.21.15.144:0/3229295871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5616dd30b1b0 0x5616dd6ec5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34ad600640 0 client.16569 ms_handle_reset on v2:172.21.15.160:6836/4027087859 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 >> 172.21.15.144:0/3229295871 conn(0x5616dd328460 msgr2=0x5616dd6e9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 shutdown_connections 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.232+0000 7f34cd873740 1 -- 172.21.15.144:0/3229295871 wait complete. 2024-07-26T11:22:41.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.233+0000 7f34cd873740 20 client.16569 trim_cache size 0 max 16384 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: "scrubs": { 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: "1b5476fb-bc3d-459a-a271-a09ca57684f6": { 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: "path": "/", 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "1b5476fb-bc3d-459a-a271-a09ca57684f6", 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: "options": "recursive,force" 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-07-26T11:22:41.237 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.233+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34a40b3dc0 msgr2=0x7f34a40b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.233+0000 7f34c5600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34a40b3dc0 0x7f34a40b62d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009f80 tx=0x7f34b8005f90 comp rx=0 tx=0).stop 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.233+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 msgr2=0x7f34c019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.233+0000 7f34c5600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 0x7f34c019a2e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f34a800ed00 tx=0x7f34a800c6a0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 shutdown_connections 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f34a40b3dc0 0x7f34a40b62d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34c0109650 0x7f34c019f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0108d00 0x7f34c019a2e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 --2- 172.21.15.144:0/217834799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c01033c0 0x7f34c0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:41.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 >> 172.21.15.144:0/217834799 conn(0x7f34c00fee10 msgr2=0x7f34c0101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:41.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 shutdown_connections 2024-07-26T11:22:41.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-07-26T11:22:41.234+0000 7f34c5600640 1 -- 172.21.15.144:0/217834799 wait complete. 2024-07-26T11:22:41.596 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1004.547557 secs 2024-07-26T11:22:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1004.547557 secs 2024-07-26T11:22:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1004.547557 secs 2024-07-26T11:22:42.046 INFO:tasks.cephfs.filesystem:scrub status for tag:1b5476fb-bc3d-459a-a271-a09ca57684f6 - {'path': '/', 'tag': '1b5476fb-bc3d-459a-a271-a09ca57684f6', 'options': 'recursive,force'} 2024-07-26T11:22:42.046 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/fwd_scrub.py", line 48, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/fwd_scrub.py", line 67, in _scrub done = self.fs.wait_until_scrub_complete(tag=tag, sleep=30, timeout=self.scrub_timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/cephfs/filesystem.py", line 1825, in wait_until_scrub_complete while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (31) after waiting for 900 seconds 2024-07-26T11:22:42.458 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:42.457+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:42.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:42.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:42.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:42.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:42 smithi183 ceph-mon[37273]: pgmap v1130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:42 smithi144 ceph-mon[32129]: pgmap v1130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:42 smithi160 ceph-mon[36715]: pgmap v1130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:44 smithi183 ceph-mon[37273]: pgmap v1131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:44 smithi144 ceph-mon[32129]: pgmap v1131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:44 smithi160 ceph-mon[36715]: pgmap v1131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:46 smithi183 ceph-mon[37273]: pgmap v1132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1009.547618 secs 2024-07-26T11:22:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:46 smithi144 ceph-mon[32129]: pgmap v1132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1009.547618 secs 2024-07-26T11:22:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:46 smithi160 ceph-mon[36715]: pgmap v1132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1009.547618 secs 2024-07-26T11:22:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:48 smithi183 ceph-mon[37273]: pgmap v1133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:48 smithi144 ceph-mon[32129]: pgmap v1133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:48 smithi160 ceph-mon[36715]: pgmap v1133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:50 smithi183 ceph-mon[37273]: pgmap v1134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:50 smithi144 ceph-mon[32129]: pgmap v1134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:50 smithi160 ceph-mon[36715]: pgmap v1134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1014.547698 secs 2024-07-26T11:22:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1014.547698 secs 2024-07-26T11:22:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1014.547698 secs 2024-07-26T11:22:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:52 smithi144 ceph-mon[32129]: pgmap v1135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:52 smithi160 ceph-mon[36715]: pgmap v1135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:52 smithi183 ceph-mon[37273]: pgmap v1135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:54 smithi144 ceph-mon[32129]: pgmap v1136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:54 smithi160 ceph-mon[36715]: pgmap v1136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:54 smithi183 ceph-mon[37273]: pgmap v1136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:56 smithi144 ceph-mon[32129]: pgmap v1137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1019.547772 secs 2024-07-26T11:22:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:56 smithi160 ceph-mon[36715]: pgmap v1137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1019.547772 secs 2024-07-26T11:22:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:56 smithi183 ceph-mon[37273]: pgmap v1137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:22:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1019.547772 secs 2024-07-26T11:22:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:57.471+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:57.472+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:57.472+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:57.472+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:22:57.472+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:22:58 smithi144 ceph-mon[32129]: pgmap v1138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:22:58 smithi160 ceph-mon[36715]: pgmap v1138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:22:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:22:58 smithi183 ceph-mon[37273]: pgmap v1138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:00 smithi144 ceph-mon[32129]: pgmap v1139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:00 smithi160 ceph-mon[36715]: pgmap v1139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:00 smithi183 ceph-mon[37273]: pgmap v1139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1024.547844 secs 2024-07-26T11:23:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1024.547844 secs 2024-07-26T11:23:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1024.547844 secs 2024-07-26T11:23:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:02 smithi144 ceph-mon[32129]: pgmap v1140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:02 smithi160 ceph-mon[36715]: pgmap v1140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:02 smithi183 ceph-mon[37273]: pgmap v1140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:04 smithi144 ceph-mon[32129]: pgmap v1141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:04 smithi160 ceph-mon[36715]: pgmap v1141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:04 smithi183 ceph-mon[37273]: pgmap v1141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:06 smithi144 ceph-mon[32129]: pgmap v1142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1029.547915 secs 2024-07-26T11:23:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:23:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:06 smithi160 ceph-mon[36715]: pgmap v1142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1029.547915 secs 2024-07-26T11:23:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:23:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:06 smithi183 ceph-mon[37273]: pgmap v1142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1029.547915 secs 2024-07-26T11:23:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:23:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:07 smithi144 ceph-mon[32129]: pgmap v1143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:07 smithi144 ceph-mon[32129]: pgmap v1144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:07 smithi160 ceph-mon[36715]: pgmap v1143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:07 smithi160 ceph-mon[36715]: pgmap v1144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:07 smithi183 ceph-mon[37273]: pgmap v1143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:07 smithi183 ceph-mon[37273]: pgmap v1144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:08 smithi160 ceph-mon[36715]: pgmap v1145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T11:23:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:08 smithi183 ceph-mon[37273]: pgmap v1145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T11:23:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:08 smithi144 ceph-mon[32129]: pgmap v1145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T11:23:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:10 smithi160 ceph-mon[36715]: pgmap v1146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:10 smithi183 ceph-mon[37273]: pgmap v1146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:10 smithi144 ceph-mon[32129]: pgmap v1146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1034.547998 secs 2024-07-26T11:23:11.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1034.547998 secs 2024-07-26T11:23:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1034.547998 secs 2024-07-26T11:23:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:12.478+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:12.479+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:12.479+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:12.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:12.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:12 smithi160 ceph-mon[36715]: pgmap v1147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:12 smithi183 ceph-mon[37273]: pgmap v1147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:12 smithi144 ceph-mon[32129]: pgmap v1147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:14 smithi160 ceph-mon[36715]: pgmap v1148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:14 smithi183 ceph-mon[37273]: pgmap v1148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:14 smithi144 ceph-mon[32129]: pgmap v1148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:23:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1039.548091 secs 2024-07-26T11:23:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1039.548091 secs 2024-07-26T11:23:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1039.548091 secs 2024-07-26T11:23:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:16 smithi183 ceph-mon[37273]: pgmap v1149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:16 smithi144 ceph-mon[32129]: pgmap v1149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:16 smithi160 ceph-mon[36715]: pgmap v1149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:18 smithi160 ceph-mon[36715]: pgmap v1150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:18 smithi183 ceph-mon[37273]: pgmap v1150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:18 smithi144 ceph-mon[32129]: pgmap v1150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:20 smithi160 ceph-mon[36715]: pgmap v1151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:20 smithi183 ceph-mon[37273]: pgmap v1151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:20 smithi144 ceph-mon[32129]: pgmap v1151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:21.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1044.548181 secs 2024-07-26T11:23:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1044.548181 secs 2024-07-26T11:23:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1044.548181 secs 2024-07-26T11:23:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:22 smithi160 ceph-mon[36715]: pgmap v1152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:22 smithi183 ceph-mon[37273]: pgmap v1152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:22 smithi144 ceph-mon[32129]: pgmap v1152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:24 smithi160 ceph-mon[36715]: pgmap v1153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:24 smithi183 ceph-mon[37273]: pgmap v1153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:24 smithi144 ceph-mon[32129]: pgmap v1153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1049.548338 secs 2024-07-26T11:23:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1049.548338 secs 2024-07-26T11:23:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1049.548338 secs 2024-07-26T11:23:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:26 smithi183 ceph-mon[37273]: pgmap v1154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:26 smithi144 ceph-mon[32129]: pgmap v1154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:26 smithi160 ceph-mon[36715]: pgmap v1154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:27.495+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:27.496+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:27.496+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:27.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:27.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:28 smithi160 ceph-mon[36715]: pgmap v1155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:28 smithi183 ceph-mon[37273]: pgmap v1155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:28 smithi144 ceph-mon[32129]: pgmap v1155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:30 smithi160 ceph-mon[36715]: pgmap v1156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:30 smithi183 ceph-mon[37273]: pgmap v1156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:30 smithi144 ceph-mon[32129]: pgmap v1156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1054.548411 secs 2024-07-26T11:23:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1054.548411 secs 2024-07-26T11:23:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1054.548411 secs 2024-07-26T11:23:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:32 smithi160 ceph-mon[36715]: pgmap v1157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:32 smithi183 ceph-mon[37273]: pgmap v1157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:32 smithi144 ceph-mon[32129]: pgmap v1157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:34 smithi160 ceph-mon[36715]: pgmap v1158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:34 smithi183 ceph-mon[37273]: pgmap v1158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:34 smithi144 ceph-mon[32129]: pgmap v1158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1059.548473 secs 2024-07-26T11:23:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1059.548473 secs 2024-07-26T11:23:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1059.548473 secs 2024-07-26T11:23:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:36 smithi183 ceph-mon[37273]: pgmap v1159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:36 smithi144 ceph-mon[32129]: pgmap v1159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:36 smithi160 ceph-mon[36715]: pgmap v1159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:38 smithi160 ceph-mon[36715]: pgmap v1160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:38 smithi183 ceph-mon[37273]: pgmap v1160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:38 smithi144 ceph-mon[32129]: pgmap v1160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:40 smithi160 ceph-mon[36715]: pgmap v1161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:40 smithi183 ceph-mon[37273]: pgmap v1161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:40 smithi144 ceph-mon[32129]: pgmap v1161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1064.548534 secs 2024-07-26T11:23:41.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1064.548534 secs 2024-07-26T11:23:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1064.548534 secs 2024-07-26T11:23:42.513 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:42.511+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:42.512+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:42.512+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:42.512+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:42.512+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:42 smithi160 ceph-mon[36715]: pgmap v1162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:42 smithi183 ceph-mon[37273]: pgmap v1162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:42 smithi144 ceph-mon[32129]: pgmap v1162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:44 smithi160 ceph-mon[36715]: pgmap v1163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:23:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:44 smithi183 ceph-mon[37273]: pgmap v1163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:23:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:44 smithi144 ceph-mon[32129]: pgmap v1163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:23:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1069.548584 secs 2024-07-26T11:23:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1069.548584 secs 2024-07-26T11:23:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:46.276+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:23:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:46.276+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:23:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:46.277+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T11:23:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:46.277+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T11:23:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1069.548584 secs 2024-07-26T11:23:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:46 smithi183 ceph-mon[37273]: pgmap v1164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:46 smithi144 ceph-mon[32129]: pgmap v1164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:46 smithi160 ceph-mon[36715]: pgmap v1164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:48 smithi160 ceph-mon[36715]: pgmap v1165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:23:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:48 smithi183 ceph-mon[37273]: pgmap v1165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:23:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:48 smithi144 ceph-mon[32129]: pgmap v1165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:23:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:50 smithi160 ceph-mon[36715]: pgmap v1166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:50 smithi183 ceph-mon[37273]: pgmap v1166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:50 smithi144 ceph-mon[32129]: pgmap v1166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:51.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1074.548701 secs 2024-07-26T11:23:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1074.548701 secs 2024-07-26T11:23:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1074.548701 secs 2024-07-26T11:23:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:52 smithi160 ceph-mon[36715]: pgmap v1167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:23:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:52 smithi183 ceph-mon[37273]: pgmap v1167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:23:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:52 smithi144 ceph-mon[32129]: pgmap v1167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:23:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:54 smithi160 ceph-mon[36715]: pgmap v1168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:23:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:54 smithi183 ceph-mon[37273]: pgmap v1168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:23:54.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:54 smithi144 ceph-mon[32129]: pgmap v1168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:23:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1079.548818 secs 2024-07-26T11:23:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1079.548818 secs 2024-07-26T11:23:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1079.548818 secs 2024-07-26T11:23:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:56 smithi183 ceph-mon[37273]: pgmap v1169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:56 smithi144 ceph-mon[32129]: pgmap v1169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:56 smithi160 ceph-mon[36715]: pgmap v1169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:23:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:57.516+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:57.517+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:57.517+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:57.517+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:23:57.517+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:23:58 smithi160 ceph-mon[36715]: pgmap v1170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:23:58 smithi183 ceph-mon[37273]: pgmap v1170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:23:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:23:58 smithi144 ceph-mon[32129]: pgmap v1170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:01 smithi160 ceph-mon[36715]: pgmap v1171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1084.548961 secs 2024-07-26T11:24:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:01 smithi183 ceph-mon[37273]: pgmap v1171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1084.548961 secs 2024-07-26T11:24:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:01 smithi144 ceph-mon[32129]: pgmap v1171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1084.548961 secs 2024-07-26T11:24:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:03 smithi160 ceph-mon[36715]: pgmap v1172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:03 smithi183 ceph-mon[37273]: pgmap v1172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:03 smithi144 ceph-mon[32129]: pgmap v1172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:05 smithi160 ceph-mon[36715]: pgmap v1173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:05 smithi183 ceph-mon[37273]: pgmap v1173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:05 smithi144 ceph-mon[32129]: pgmap v1173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1089.549101 secs 2024-07-26T11:24:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1089.549101 secs 2024-07-26T11:24:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1089.549101 secs 2024-07-26T11:24:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:07 smithi160 ceph-mon[36715]: pgmap v1174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:07 smithi183 ceph-mon[37273]: pgmap v1174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:07 smithi144 ceph-mon[32129]: pgmap v1174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:08 smithi160 ceph-mon[36715]: pgmap v1175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:24:08.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:08 smithi160 ceph-mon[36715]: pgmap v1176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:24:08.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:24:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:08 smithi183 ceph-mon[37273]: pgmap v1175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:24:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:08 smithi183 ceph-mon[37273]: pgmap v1176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:24:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:24:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:08 smithi144 ceph-mon[32129]: pgmap v1175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:24:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:08 smithi144 ceph-mon[32129]: pgmap v1176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:24:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:24:11.038 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:10 smithi144 ceph-mon[32129]: pgmap v1177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:24:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:10 smithi160 ceph-mon[36715]: pgmap v1177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:24:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:10 smithi183 ceph-mon[37273]: pgmap v1177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:24:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1094.549219 secs 2024-07-26T11:24:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1094.549219 secs 2024-07-26T11:24:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1094.549219 secs 2024-07-26T11:24:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:12.534+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:12.535+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:12.535+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:12.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:12.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:12 smithi160 ceph-mon[36715]: pgmap v1178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:24:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:12 smithi183 ceph-mon[37273]: pgmap v1178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:24:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:12 smithi144 ceph-mon[32129]: pgmap v1178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:24:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:14 smithi160 ceph-mon[36715]: pgmap v1179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:24:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:14 smithi183 ceph-mon[37273]: pgmap v1179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:24:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:14 smithi144 ceph-mon[32129]: pgmap v1179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:24:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1099.549362 secs 2024-07-26T11:24:17.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:16 smithi160 ceph-mon[36715]: pgmap v1180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:24:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1099.549362 secs 2024-07-26T11:24:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:16 smithi183 ceph-mon[37273]: pgmap v1180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:24:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1099.549362 secs 2024-07-26T11:24:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:16 smithi144 ceph-mon[32129]: pgmap v1180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:24:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:18 smithi160 ceph-mon[36715]: pgmap v1181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:24:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:18 smithi183 ceph-mon[37273]: pgmap v1181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:24:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:18 smithi144 ceph-mon[32129]: pgmap v1181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:24:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:20 smithi160 ceph-mon[36715]: pgmap v1182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:20 smithi183 ceph-mon[37273]: pgmap v1182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:20 smithi144 ceph-mon[32129]: pgmap v1182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:22.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1104.549498 secs 2024-07-26T11:24:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1104.549498 secs 2024-07-26T11:24:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1104.549498 secs 2024-07-26T11:24:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:22 smithi160 ceph-mon[36715]: pgmap v1183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:22 smithi183 ceph-mon[37273]: pgmap v1183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:22 smithi144 ceph-mon[32129]: pgmap v1183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:24 smithi160 ceph-mon[36715]: pgmap v1184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:24 smithi183 ceph-mon[37273]: pgmap v1184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:24 smithi144 ceph-mon[32129]: pgmap v1184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1109.549630 secs 2024-07-26T11:24:26.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1109.549630 secs 2024-07-26T11:24:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1109.549630 secs 2024-07-26T11:24:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:26 smithi160 ceph-mon[36715]: pgmap v1185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:26 smithi183 ceph-mon[37273]: pgmap v1185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:26 smithi144 ceph-mon[32129]: pgmap v1185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:27.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:27.550+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:27.550+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:27.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:27.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:28 smithi183 ceph-mon[37273]: pgmap v1186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:28 smithi144 ceph-mon[32129]: pgmap v1186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:28 smithi160 ceph-mon[36715]: pgmap v1186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:30 smithi183 ceph-mon[37273]: pgmap v1187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1114.549765 secs 2024-07-26T11:24:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:30 smithi144 ceph-mon[32129]: pgmap v1187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1114.549765 secs 2024-07-26T11:24:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:30 smithi160 ceph-mon[36715]: pgmap v1187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1114.549765 secs 2024-07-26T11:24:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:32 smithi183 ceph-mon[37273]: pgmap v1188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:32 smithi144 ceph-mon[32129]: pgmap v1188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:32 smithi160 ceph-mon[36715]: pgmap v1188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:34 smithi183 ceph-mon[37273]: pgmap v1189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:34 smithi144 ceph-mon[32129]: pgmap v1189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:34 smithi160 ceph-mon[36715]: pgmap v1189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1119.549892 secs 2024-07-26T11:24:36.211 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1119.549892 secs 2024-07-26T11:24:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1119.549892 secs 2024-07-26T11:24:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:36 smithi183 ceph-mon[37273]: pgmap v1190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:36 smithi144 ceph-mon[32129]: pgmap v1190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:36 smithi160 ceph-mon[36715]: pgmap v1190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:38 smithi183 ceph-mon[37273]: pgmap v1191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:38 smithi144 ceph-mon[32129]: pgmap v1191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:38 smithi160 ceph-mon[36715]: pgmap v1191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:40 smithi183 ceph-mon[37273]: pgmap v1192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1124.549954 secs 2024-07-26T11:24:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:40 smithi144 ceph-mon[32129]: pgmap v1192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1124.549954 secs 2024-07-26T11:24:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:40 smithi160 ceph-mon[36715]: pgmap v1192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1124.549954 secs 2024-07-26T11:24:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:42.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:42.567+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:42.567+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:42.567+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:42.567+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:42 smithi183 ceph-mon[37273]: pgmap v1193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:42 smithi144 ceph-mon[32129]: pgmap v1193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:42 smithi160 ceph-mon[36715]: pgmap v1193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:44 smithi183 ceph-mon[37273]: pgmap v1194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:44 smithi144 ceph-mon[32129]: pgmap v1194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:44 smithi160 ceph-mon[36715]: pgmap v1194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1129.550087 secs 2024-07-26T11:24:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1129.550087 secs 2024-07-26T11:24:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1129.550087 secs 2024-07-26T11:24:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:46 smithi183 ceph-mon[37273]: pgmap v1195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:46 smithi144 ceph-mon[32129]: pgmap v1195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:46 smithi160 ceph-mon[36715]: pgmap v1195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:48 smithi183 ceph-mon[37273]: pgmap v1196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:48 smithi144 ceph-mon[32129]: pgmap v1196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:48 smithi160 ceph-mon[36715]: pgmap v1196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:50 smithi144 ceph-mon[32129]: pgmap v1197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1134.550155 secs 2024-07-26T11:24:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:50 smithi160 ceph-mon[36715]: pgmap v1197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1134.550155 secs 2024-07-26T11:24:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:50 smithi183 ceph-mon[37273]: pgmap v1197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1134.550155 secs 2024-07-26T11:24:53.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:52 smithi183 ceph-mon[37273]: pgmap v1198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:52 smithi144 ceph-mon[32129]: pgmap v1198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:52 smithi160 ceph-mon[36715]: pgmap v1198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:24:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:54 smithi144 ceph-mon[32129]: pgmap v1199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:54 smithi160 ceph-mon[36715]: pgmap v1199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:54 smithi183 ceph-mon[37273]: pgmap v1199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1139.550280 secs 2024-07-26T11:24:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1139.550280 secs 2024-07-26T11:24:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1139.550280 secs 2024-07-26T11:24:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:56 smithi144 ceph-mon[32129]: pgmap v1200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:56 smithi160 ceph-mon[36715]: pgmap v1200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:56 smithi183 ceph-mon[37273]: pgmap v1200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:24:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:57.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:57.583+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:57.583+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:57.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:24:57.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:24:58 smithi144 ceph-mon[32129]: pgmap v1201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:24:58 smithi160 ceph-mon[36715]: pgmap v1201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:24:58 smithi183 ceph-mon[37273]: pgmap v1201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:00 smithi144 ceph-mon[32129]: pgmap v1202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1144.550393 secs 2024-07-26T11:25:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:00 smithi160 ceph-mon[36715]: pgmap v1202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1144.550393 secs 2024-07-26T11:25:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:00 smithi183 ceph-mon[37273]: pgmap v1202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1144.550393 secs 2024-07-26T11:25:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:02 smithi144 ceph-mon[32129]: pgmap v1203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:02 smithi160 ceph-mon[36715]: pgmap v1203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:02 smithi183 ceph-mon[37273]: pgmap v1203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:04 smithi144 ceph-mon[32129]: pgmap v1204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:04 smithi160 ceph-mon[36715]: pgmap v1204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:04 smithi183 ceph-mon[37273]: pgmap v1204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:06.211 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1149.550540 secs 2024-07-26T11:25:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1149.550540 secs 2024-07-26T11:25:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1149.550540 secs 2024-07-26T11:25:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:06 smithi144 ceph-mon[32129]: pgmap v1205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:06 smithi160 ceph-mon[36715]: pgmap v1205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:06 smithi183 ceph-mon[37273]: pgmap v1205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:08 smithi160 ceph-mon[36715]: pgmap v1206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:25:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:08 smithi183 ceph-mon[37273]: pgmap v1206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:25:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:08 smithi144 ceph-mon[32129]: pgmap v1206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:25:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:09 smithi160 ceph-mon[36715]: pgmap v1207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:09 smithi160 ceph-mon[36715]: pgmap v1208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:25:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:09 smithi183 ceph-mon[37273]: pgmap v1207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:09 smithi183 ceph-mon[37273]: pgmap v1208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:25:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:09 smithi144 ceph-mon[32129]: pgmap v1207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:09 smithi144 ceph-mon[32129]: pgmap v1208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:25:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1154.550650 secs 2024-07-26T11:25:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1154.550650 secs 2024-07-26T11:25:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1154.550650 secs 2024-07-26T11:25:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:11 smithi160 ceph-mon[36715]: pgmap v1209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:11 smithi183 ceph-mon[37273]: pgmap v1209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:11 smithi144 ceph-mon[32129]: pgmap v1209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:12.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:12.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:12.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:12.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:12.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:13 smithi160 ceph-mon[36715]: pgmap v1210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:25:14.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:13 smithi144 ceph-mon[32129]: pgmap v1210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:25:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:13 smithi183 ceph-mon[37273]: pgmap v1210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:25:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:16 smithi160 ceph-mon[36715]: pgmap v1211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:25:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1159.550770 secs 2024-07-26T11:25:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:16 smithi183 ceph-mon[37273]: pgmap v1211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:25:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1159.550770 secs 2024-07-26T11:25:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:16 smithi144 ceph-mon[32129]: pgmap v1211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:25:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1159.550770 secs 2024-07-26T11:25:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:18 smithi160 ceph-mon[36715]: pgmap v1212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:18 smithi183 ceph-mon[37273]: pgmap v1212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:18 smithi144 ceph-mon[32129]: pgmap v1212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:25:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:20 smithi160 ceph-mon[36715]: pgmap v1213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:25:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:20 smithi183 ceph-mon[37273]: pgmap v1213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:25:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:20 smithi144 ceph-mon[32129]: pgmap v1213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:25:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1164.550884 secs 2024-07-26T11:25:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1164.550884 secs 2024-07-26T11:25:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1164.550884 secs 2024-07-26T11:25:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:22 smithi160 ceph-mon[36715]: pgmap v1214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:22 smithi183 ceph-mon[37273]: pgmap v1214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:22 smithi144 ceph-mon[32129]: pgmap v1214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:24 smithi160 ceph-mon[36715]: pgmap v1215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:24 smithi183 ceph-mon[37273]: pgmap v1215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:24 smithi144 ceph-mon[32129]: pgmap v1215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:26 smithi160 ceph-mon[36715]: pgmap v1216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1169.551014 secs 2024-07-26T11:25:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:26 smithi183 ceph-mon[37273]: pgmap v1216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1169.551014 secs 2024-07-26T11:25:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:26 smithi144 ceph-mon[32129]: pgmap v1216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1169.551014 secs 2024-07-26T11:25:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:27.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:27.615+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:27.615+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:27.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:27.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:28 smithi144 ceph-mon[32129]: pgmap v1217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:28 smithi183 ceph-mon[37273]: pgmap v1217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:28 smithi160 ceph-mon[36715]: pgmap v1217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:30 smithi183 ceph-mon[37273]: pgmap v1218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:30 smithi144 ceph-mon[32129]: pgmap v1218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:30 smithi160 ceph-mon[36715]: pgmap v1218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1174.551164 secs 2024-07-26T11:25:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1174.551164 secs 2024-07-26T11:25:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1174.551164 secs 2024-07-26T11:25:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:32 smithi183 ceph-mon[37273]: pgmap v1219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:32 smithi144 ceph-mon[32129]: pgmap v1219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:32 smithi160 ceph-mon[36715]: pgmap v1219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:34 smithi183 ceph-mon[37273]: pgmap v1220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:34 smithi144 ceph-mon[32129]: pgmap v1220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:34 smithi160 ceph-mon[36715]: pgmap v1220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:36.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:36 smithi160 ceph-mon[36715]: pgmap v1221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:36.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1179.551318 secs 2024-07-26T11:25:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:36 smithi183 ceph-mon[37273]: pgmap v1221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1179.551318 secs 2024-07-26T11:25:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:36 smithi144 ceph-mon[32129]: pgmap v1221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1179.551318 secs 2024-07-26T11:25:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:38 smithi183 ceph-mon[37273]: pgmap v1222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:38 smithi144 ceph-mon[32129]: pgmap v1222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:38 smithi160 ceph-mon[36715]: pgmap v1222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:40 smithi183 ceph-mon[37273]: pgmap v1223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:40 smithi144 ceph-mon[32129]: pgmap v1223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:40 smithi160 ceph-mon[36715]: pgmap v1223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1184.551392 secs 2024-07-26T11:25:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1184.551392 secs 2024-07-26T11:25:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1184.551392 secs 2024-07-26T11:25:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:42 smithi183 ceph-mon[37273]: pgmap v1224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:42 smithi144 ceph-mon[32129]: pgmap v1224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:42 smithi160 ceph-mon[36715]: pgmap v1224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:42.631+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:42.632+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:42.632+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:42.632+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:42.632+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:44 smithi183 ceph-mon[37273]: pgmap v1225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:44 smithi144 ceph-mon[32129]: pgmap v1225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:44 smithi160 ceph-mon[36715]: pgmap v1225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:46 smithi183 ceph-mon[37273]: pgmap v1226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1189.551527 secs 2024-07-26T11:25:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:46 smithi144 ceph-mon[32129]: pgmap v1226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1189.551527 secs 2024-07-26T11:25:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:46 smithi160 ceph-mon[36715]: pgmap v1226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1189.551527 secs 2024-07-26T11:25:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:48 smithi144 ceph-mon[32129]: pgmap v1227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:48 smithi160 ceph-mon[36715]: pgmap v1227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:48.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:48 smithi183 ceph-mon[37273]: pgmap v1227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:50 smithi144 ceph-mon[32129]: pgmap v1228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:50 smithi160 ceph-mon[36715]: pgmap v1228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:50 smithi183 ceph-mon[37273]: pgmap v1228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1194.551611 secs 2024-07-26T11:25:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1194.551611 secs 2024-07-26T11:25:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1194.551611 secs 2024-07-26T11:25:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:52 smithi144 ceph-mon[32129]: pgmap v1229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:52 smithi160 ceph-mon[36715]: pgmap v1229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:52 smithi183 ceph-mon[37273]: pgmap v1229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:54.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:54 smithi183 ceph-mon[37273]: pgmap v1230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:54 smithi144 ceph-mon[32129]: pgmap v1230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:54 smithi160 ceph-mon[36715]: pgmap v1230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:56 smithi144 ceph-mon[32129]: pgmap v1231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1199.551727 secs 2024-07-26T11:25:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:56 smithi160 ceph-mon[36715]: pgmap v1231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1199.551727 secs 2024-07-26T11:25:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:56 smithi183 ceph-mon[37273]: pgmap v1231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:25:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1199.551727 secs 2024-07-26T11:25:57.650 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:57.647+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:57.648+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:57.648+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:57.648+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:25:57.648+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:25:58 smithi144 ceph-mon[32129]: pgmap v1232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:25:58 smithi160 ceph-mon[36715]: pgmap v1232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:25:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:25:58 smithi183 ceph-mon[37273]: pgmap v1232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:00 smithi144 ceph-mon[32129]: pgmap v1233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:00 smithi160 ceph-mon[36715]: pgmap v1233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:00 smithi183 ceph-mon[37273]: pgmap v1233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1204.551807 secs 2024-07-26T11:26:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1204.551807 secs 2024-07-26T11:26:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1204.551807 secs 2024-07-26T11:26:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:02 smithi144 ceph-mon[32129]: pgmap v1234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:02 smithi160 ceph-mon[36715]: pgmap v1234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:02 smithi183 ceph-mon[37273]: pgmap v1234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:04 smithi144 ceph-mon[32129]: pgmap v1235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:04 smithi160 ceph-mon[36715]: pgmap v1235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:04 smithi183 ceph-mon[37273]: pgmap v1235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:06 smithi144 ceph-mon[32129]: pgmap v1236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1209.551945 secs 2024-07-26T11:26:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:06 smithi160 ceph-mon[36715]: pgmap v1236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1209.551945 secs 2024-07-26T11:26:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:06 smithi183 ceph-mon[37273]: pgmap v1236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1209.551945 secs 2024-07-26T11:26:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:08 smithi144 ceph-mon[32129]: pgmap v1237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:08 smithi160 ceph-mon[36715]: pgmap v1237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:08 smithi183 ceph-mon[37273]: pgmap v1237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:10.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:10 smithi160 ceph-mon[36715]: pgmap v1238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:10 smithi183 ceph-mon[37273]: pgmap v1238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:10 smithi144 ceph-mon[32129]: pgmap v1238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1214.552062 secs 2024-07-26T11:26:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1214.552062 secs 2024-07-26T11:26:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1214.552062 secs 2024-07-26T11:26:12.501 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:12 smithi160 ceph-mon[36715]: pgmap v1239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:12 smithi183 ceph-mon[37273]: pgmap v1239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:12.645 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:12 smithi144 ceph-mon[32129]: pgmap v1239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:12.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:12.665+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:12.666+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:12.666+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:12.666+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:12.666+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:14 smithi160 ceph-mon[36715]: pgmap v1240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:14.310 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:14 smithi144 ceph-mon[32129]: pgmap v1240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:14.311 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:14.311 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:14 smithi183 ceph-mon[37273]: pgmap v1240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: pgmap v1241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: pgmap v1242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: pgmap v1243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1219.552191 secs 2024-07-26T11:26:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: pgmap v1241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:16.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:16.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: pgmap v1242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:16.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: pgmap v1243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:16.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1219.552191 secs 2024-07-26T11:26:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: pgmap v1241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:16.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: pgmap v1242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:16.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: pgmap v1243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:16.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:26:16.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1219.552191 secs 2024-07-26T11:26:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:18 smithi183 ceph-mon[37273]: pgmap v1244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:26:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:18 smithi144 ceph-mon[32129]: pgmap v1244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:26:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:18 smithi160 ceph-mon[36715]: pgmap v1244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:26:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:20 smithi183 ceph-mon[37273]: pgmap v1245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:26:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:20 smithi144 ceph-mon[32129]: pgmap v1245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:26:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:20 smithi160 ceph-mon[36715]: pgmap v1245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:26:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1224.552331 secs 2024-07-26T11:26:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1224.552331 secs 2024-07-26T11:26:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1224.552331 secs 2024-07-26T11:26:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:22 smithi183 ceph-mon[37273]: pgmap v1246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:22 smithi144 ceph-mon[32129]: pgmap v1246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:22 smithi160 ceph-mon[36715]: pgmap v1246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:24 smithi183 ceph-mon[37273]: pgmap v1247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:24 smithi144 ceph-mon[32129]: pgmap v1247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:24 smithi160 ceph-mon[36715]: pgmap v1247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:26:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:26 smithi183 ceph-mon[37273]: pgmap v1248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1229.552450 secs 2024-07-26T11:26:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:26 smithi144 ceph-mon[32129]: pgmap v1248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1229.552450 secs 2024-07-26T11:26:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:26 smithi160 ceph-mon[36715]: pgmap v1248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1229.552450 secs 2024-07-26T11:26:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:27.677+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:27.679+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:27.679+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:27.679+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:27.679+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:28 smithi183 ceph-mon[37273]: pgmap v1249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:28 smithi144 ceph-mon[32129]: pgmap v1249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:28 smithi160 ceph-mon[36715]: pgmap v1249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:30 smithi183 ceph-mon[37273]: pgmap v1250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:30 smithi144 ceph-mon[32129]: pgmap v1250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:30 smithi160 ceph-mon[36715]: pgmap v1250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1234.552592 secs 2024-07-26T11:26:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1234.552592 secs 2024-07-26T11:26:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1234.552592 secs 2024-07-26T11:26:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:32 smithi183 ceph-mon[37273]: pgmap v1251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:32 smithi144 ceph-mon[32129]: pgmap v1251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:32 smithi160 ceph-mon[36715]: pgmap v1251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:34 smithi183 ceph-mon[37273]: pgmap v1252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:34 smithi144 ceph-mon[32129]: pgmap v1252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:34 smithi160 ceph-mon[36715]: pgmap v1252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:36 smithi183 ceph-mon[37273]: pgmap v1253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1239.552722 secs 2024-07-26T11:26:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:36 smithi144 ceph-mon[32129]: pgmap v1253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1239.552722 secs 2024-07-26T11:26:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:36 smithi160 ceph-mon[36715]: pgmap v1253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1239.552722 secs 2024-07-26T11:26:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:38 smithi144 ceph-mon[32129]: pgmap v1254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:38 smithi160 ceph-mon[36715]: pgmap v1254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:38 smithi183 ceph-mon[37273]: pgmap v1254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:40 smithi144 ceph-mon[32129]: pgmap v1255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:26:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:40 smithi160 ceph-mon[36715]: pgmap v1255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:26:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:40 smithi183 ceph-mon[37273]: pgmap v1255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:26:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1244.552845 secs 2024-07-26T11:26:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1244.552845 secs 2024-07-26T11:26:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1244.552845 secs 2024-07-26T11:26:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:42.695+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:42.696+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:42.696+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:42.696+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:42.696+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:42 smithi144 ceph-mon[32129]: pgmap v1256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:42 smithi160 ceph-mon[36715]: pgmap v1256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:42 smithi183 ceph-mon[37273]: pgmap v1256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:44 smithi144 ceph-mon[32129]: pgmap v1257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:44 smithi160 ceph-mon[36715]: pgmap v1257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:44 smithi183 ceph-mon[37273]: pgmap v1257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:46.308+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:26:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:46.308+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:26:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:46.308+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T11:26:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:46.309+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T11:26:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:46 smithi144 ceph-mon[32129]: pgmap v1258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1249.552973 secs 2024-07-26T11:26:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:46 smithi160 ceph-mon[36715]: pgmap v1258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1249.552973 secs 2024-07-26T11:26:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:46 smithi183 ceph-mon[37273]: pgmap v1258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1249.552973 secs 2024-07-26T11:26:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:48 smithi144 ceph-mon[32129]: pgmap v1259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:26:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:48 smithi160 ceph-mon[36715]: pgmap v1259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:26:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:48 smithi183 ceph-mon[37273]: pgmap v1259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:26:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:50 smithi144 ceph-mon[32129]: pgmap v1260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:26:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:50 smithi160 ceph-mon[36715]: pgmap v1260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:26:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:50 smithi183 ceph-mon[37273]: pgmap v1260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:26:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1254.553098 secs 2024-07-26T11:26:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1254.553098 secs 2024-07-26T11:26:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1254.553098 secs 2024-07-26T11:26:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:52 smithi144 ceph-mon[32129]: pgmap v1261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:52 smithi160 ceph-mon[36715]: pgmap v1261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:52 smithi183 ceph-mon[37273]: pgmap v1261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:26:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:54 smithi144 ceph-mon[32129]: pgmap v1262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:54 smithi160 ceph-mon[36715]: pgmap v1262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:54 smithi183 ceph-mon[37273]: pgmap v1262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:56 smithi144 ceph-mon[32129]: pgmap v1263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1259.553220 secs 2024-07-26T11:26:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:56 smithi160 ceph-mon[36715]: pgmap v1263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1259.553220 secs 2024-07-26T11:26:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:56 smithi183 ceph-mon[37273]: pgmap v1263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:26:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1259.553220 secs 2024-07-26T11:26:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:57.702+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:57.702+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:57.702+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:57.702+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:26:57.703+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:26:58 smithi160 ceph-mon[36715]: pgmap v1264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:26:58 smithi183 ceph-mon[37273]: pgmap v1264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:26:58 smithi144 ceph-mon[32129]: pgmap v1264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:00 smithi160 ceph-mon[36715]: pgmap v1265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:00 smithi183 ceph-mon[37273]: pgmap v1265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:00 smithi144 ceph-mon[32129]: pgmap v1265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1264.553347 secs 2024-07-26T11:27:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1264.553347 secs 2024-07-26T11:27:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1264.553347 secs 2024-07-26T11:27:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:02 smithi160 ceph-mon[36715]: pgmap v1266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:02 smithi183 ceph-mon[37273]: pgmap v1266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:02 smithi144 ceph-mon[32129]: pgmap v1266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:03 smithi160 ceph-mon[36715]: pgmap v1267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:03 smithi183 ceph-mon[37273]: pgmap v1267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:03 smithi144 ceph-mon[32129]: pgmap v1267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:06 smithi183 ceph-mon[37273]: pgmap v1268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1269.553390 secs 2024-07-26T11:27:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:06 smithi144 ceph-mon[32129]: pgmap v1268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1269.553390 secs 2024-07-26T11:27:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:06 smithi160 ceph-mon[36715]: pgmap v1268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1269.553390 secs 2024-07-26T11:27:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:08 smithi183 ceph-mon[37273]: pgmap v1269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:08 smithi144 ceph-mon[32129]: pgmap v1269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:08 smithi160 ceph-mon[36715]: pgmap v1269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:10 smithi183 ceph-mon[37273]: pgmap v1270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:10 smithi144 ceph-mon[32129]: pgmap v1270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:10 smithi160 ceph-mon[36715]: pgmap v1270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1274.553451 secs 2024-07-26T11:27:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1274.553451 secs 2024-07-26T11:27:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1274.553451 secs 2024-07-26T11:27:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:12 smithi183 ceph-mon[37273]: pgmap v1271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:12.717+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:12 smithi144 ceph-mon[32129]: pgmap v1271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:12.718+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:12.718+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:12.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:12.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:12 smithi160 ceph-mon[36715]: pgmap v1271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:14 smithi183 ceph-mon[37273]: pgmap v1272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:14 smithi144 ceph-mon[32129]: pgmap v1272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:14 smithi160 ceph-mon[36715]: pgmap v1272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: pgmap v1273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1279.553518 secs 2024-07-26T11:27:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: pgmap v1273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1279.553518 secs 2024-07-26T11:27:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: pgmap v1273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1279.553518 secs 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:16.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:17.712 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:17 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:27:17.712 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:17 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:27:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:17 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:27:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:17 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:27:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:17 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:27:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:17 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:27:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:18 smithi144 ceph-mon[32129]: pgmap v1274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:18 smithi160 ceph-mon[36715]: pgmap v1274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:18.811 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:18 smithi183 ceph-mon[37273]: pgmap v1274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:20.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: pgmap v1275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:20.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: pgmap v1276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:20.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: pgmap v1275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: pgmap v1276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:20.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:27:20.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:27:20.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:20.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:20.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: pgmap v1275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:20.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: pgmap v1276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:20.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:27:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1284.553583 secs 2024-07-26T11:27:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1284.553583 secs 2024-07-26T11:27:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1284.553583 secs 2024-07-26T11:27:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:22 smithi144 ceph-mon[32129]: pgmap v1277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:22 smithi160 ceph-mon[36715]: pgmap v1277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:22 smithi183 ceph-mon[37273]: pgmap v1277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:24 smithi160 ceph-mon[36715]: pgmap v1278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:24 smithi183 ceph-mon[37273]: pgmap v1278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:24 smithi144 ceph-mon[32129]: pgmap v1278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:26 smithi160 ceph-mon[36715]: pgmap v1279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1289.553697 secs 2024-07-26T11:27:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:26 smithi183 ceph-mon[37273]: pgmap v1279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1289.553697 secs 2024-07-26T11:27:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:26 smithi144 ceph-mon[32129]: pgmap v1279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1289.553697 secs 2024-07-26T11:27:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:27.727+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:27.728+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:27.728+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:27.728+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:27.728+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:28 smithi160 ceph-mon[36715]: pgmap v1280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:28 smithi183 ceph-mon[37273]: pgmap v1280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:28 smithi144 ceph-mon[32129]: pgmap v1280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:27:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:30 smithi160 ceph-mon[36715]: pgmap v1281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:27:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:30 smithi183 ceph-mon[37273]: pgmap v1281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:27:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:30 smithi144 ceph-mon[32129]: pgmap v1281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:27:31.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1294.553777 secs 2024-07-26T11:27:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1294.553777 secs 2024-07-26T11:27:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1294.553777 secs 2024-07-26T11:27:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:32 smithi160 ceph-mon[36715]: pgmap v1282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:32.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:32 smithi144 ceph-mon[32129]: pgmap v1282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:32 smithi183 ceph-mon[37273]: pgmap v1282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:34 smithi160 ceph-mon[36715]: pgmap v1283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:34 smithi183 ceph-mon[37273]: pgmap v1283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:34 smithi144 ceph-mon[32129]: pgmap v1283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:36 smithi183 ceph-mon[37273]: pgmap v1284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1299.553899 secs 2024-07-26T11:27:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:36 smithi144 ceph-mon[32129]: pgmap v1284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1299.553899 secs 2024-07-26T11:27:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:36 smithi160 ceph-mon[36715]: pgmap v1284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1299.553899 secs 2024-07-26T11:27:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:38 smithi183 ceph-mon[37273]: pgmap v1285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:38 smithi144 ceph-mon[32129]: pgmap v1285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:38 smithi160 ceph-mon[36715]: pgmap v1285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:40 smithi183 ceph-mon[37273]: pgmap v1286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:40 smithi144 ceph-mon[32129]: pgmap v1286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:40 smithi160 ceph-mon[36715]: pgmap v1286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1304.554032 secs 2024-07-26T11:27:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1304.554032 secs 2024-07-26T11:27:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1304.554032 secs 2024-07-26T11:27:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:42 smithi183 ceph-mon[37273]: pgmap v1287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:42 smithi144 ceph-mon[32129]: pgmap v1287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:42.744+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:42.745+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:42.745+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:42.745+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:42.745+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:42 smithi160 ceph-mon[36715]: pgmap v1287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:44 smithi183 ceph-mon[37273]: pgmap v1288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:44.695 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:44 smithi144 ceph-mon[32129]: pgmap v1288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:44 smithi160 ceph-mon[36715]: pgmap v1288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:46 smithi183 ceph-mon[37273]: pgmap v1289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1309.554153 secs 2024-07-26T11:27:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:46 smithi144 ceph-mon[32129]: pgmap v1289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1309.554153 secs 2024-07-26T11:27:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:46 smithi160 ceph-mon[36715]: pgmap v1289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1309.554153 secs 2024-07-26T11:27:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:48 smithi144 ceph-mon[32129]: pgmap v1290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:48 smithi183 ceph-mon[37273]: pgmap v1290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:48 smithi160 ceph-mon[36715]: pgmap v1290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:50 smithi183 ceph-mon[37273]: pgmap v1291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:50 smithi144 ceph-mon[32129]: pgmap v1291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:50 smithi160 ceph-mon[36715]: pgmap v1291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1314.554268 secs 2024-07-26T11:27:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1314.554268 secs 2024-07-26T11:27:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1314.554268 secs 2024-07-26T11:27:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:52 smithi183 ceph-mon[37273]: pgmap v1292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:52 smithi144 ceph-mon[32129]: pgmap v1292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:52 smithi160 ceph-mon[36715]: pgmap v1292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:27:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:54 smithi144 ceph-mon[32129]: pgmap v1293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:54 smithi160 ceph-mon[36715]: pgmap v1293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:54 smithi183 ceph-mon[37273]: pgmap v1293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:56 smithi144 ceph-mon[32129]: pgmap v1294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1319.554403 secs 2024-07-26T11:27:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:56 smithi160 ceph-mon[36715]: pgmap v1294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1319.554403 secs 2024-07-26T11:27:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:56 smithi183 ceph-mon[37273]: pgmap v1294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:27:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1319.554403 secs 2024-07-26T11:27:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:57.760+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:57.761+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:57.761+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:57.761+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:27:57.761+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:27:58 smithi144 ceph-mon[32129]: pgmap v1295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:27:58 smithi160 ceph-mon[36715]: pgmap v1295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:27:58 smithi183 ceph-mon[37273]: pgmap v1295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:00 smithi144 ceph-mon[32129]: pgmap v1296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:00 smithi160 ceph-mon[36715]: pgmap v1296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:00 smithi183 ceph-mon[37273]: pgmap v1296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1324.554503 secs 2024-07-26T11:28:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1324.554503 secs 2024-07-26T11:28:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1324.554503 secs 2024-07-26T11:28:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:02 smithi144 ceph-mon[32129]: pgmap v1297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:02 smithi160 ceph-mon[36715]: pgmap v1297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:02 smithi183 ceph-mon[37273]: pgmap v1297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:04 smithi144 ceph-mon[32129]: pgmap v1298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:04 smithi160 ceph-mon[36715]: pgmap v1298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:04 smithi183 ceph-mon[37273]: pgmap v1298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:06 smithi144 ceph-mon[32129]: pgmap v1299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1329.554598 secs 2024-07-26T11:28:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:06 smithi160 ceph-mon[36715]: pgmap v1299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1329.554598 secs 2024-07-26T11:28:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:06 smithi183 ceph-mon[37273]: pgmap v1299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1329.554598 secs 2024-07-26T11:28:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:08 smithi144 ceph-mon[32129]: pgmap v1300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:08 smithi160 ceph-mon[36715]: pgmap v1300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:08 smithi183 ceph-mon[37273]: pgmap v1300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:10 smithi144 ceph-mon[32129]: pgmap v1301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:10 smithi160 ceph-mon[36715]: pgmap v1301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:10 smithi183 ceph-mon[37273]: pgmap v1301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1334.554726 secs 2024-07-26T11:28:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1334.554726 secs 2024-07-26T11:28:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1334.554726 secs 2024-07-26T11:28:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:12 smithi144 ceph-mon[32129]: pgmap v1302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:12.780 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:12.778+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:12.779+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:12.779+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:12.779+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:12.779+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:12 smithi160 ceph-mon[36715]: pgmap v1302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:12 smithi183 ceph-mon[37273]: pgmap v1302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:14 smithi144 ceph-mon[32129]: pgmap v1303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:14 smithi160 ceph-mon[36715]: pgmap v1303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:14 smithi183 ceph-mon[37273]: pgmap v1303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:16 smithi144 ceph-mon[32129]: pgmap v1304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1339.554847 secs 2024-07-26T11:28:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:16 smithi160 ceph-mon[36715]: pgmap v1304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1339.554847 secs 2024-07-26T11:28:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:16 smithi183 ceph-mon[37273]: pgmap v1304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1339.554847 secs 2024-07-26T11:28:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:18 smithi160 ceph-mon[36715]: pgmap v1305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:18.846 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:18 smithi144 ceph-mon[32129]: pgmap v1305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:18 smithi183 ceph-mon[37273]: pgmap v1305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:20 smithi160 ceph-mon[36715]: pgmap v1306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:28:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:20 smithi183 ceph-mon[37273]: pgmap v1306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:28:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:20 smithi144 ceph-mon[32129]: pgmap v1306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:28:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:21 smithi160 ceph-mon[36715]: pgmap v1307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:21 smithi160 ceph-mon[36715]: pgmap v1308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1344.554914 secs 2024-07-26T11:28:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:21 smithi183 ceph-mon[37273]: pgmap v1307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:21 smithi183 ceph-mon[37273]: pgmap v1308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1344.554914 secs 2024-07-26T11:28:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:21 smithi144 ceph-mon[32129]: pgmap v1307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:21 smithi144 ceph-mon[32129]: pgmap v1308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1344.554914 secs 2024-07-26T11:28:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:23 smithi160 ceph-mon[36715]: pgmap v1309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:28:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:23 smithi183 ceph-mon[37273]: pgmap v1309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:28:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:23 smithi144 ceph-mon[32129]: pgmap v1309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:28:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:24 smithi160 ceph-mon[36715]: pgmap v1310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:28:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:24 smithi183 ceph-mon[37273]: pgmap v1310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:28:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:24 smithi144 ceph-mon[32129]: pgmap v1310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:28:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1349.555049 secs 2024-07-26T11:28:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1349.555049 secs 2024-07-26T11:28:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1349.555049 secs 2024-07-26T11:28:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:26 smithi183 ceph-mon[37273]: pgmap v1311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:26 smithi144 ceph-mon[32129]: pgmap v1311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:26 smithi160 ceph-mon[36715]: pgmap v1311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:28:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:27.795+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:27.796+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:27.796+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:27.796+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:27.797+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:29 smithi160 ceph-mon[36715]: pgmap v1312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:28:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:29 smithi183 ceph-mon[37273]: pgmap v1312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:28:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:29 smithi144 ceph-mon[32129]: pgmap v1312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:28:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:31 smithi160 ceph-mon[36715]: pgmap v1313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:28:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1354.555173 secs 2024-07-26T11:28:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:31 smithi183 ceph-mon[37273]: pgmap v1313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:28:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1354.555173 secs 2024-07-26T11:28:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:31 smithi144 ceph-mon[32129]: pgmap v1313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:28:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1354.555173 secs 2024-07-26T11:28:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:33 smithi160 ceph-mon[36715]: pgmap v1314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:33 smithi183 ceph-mon[37273]: pgmap v1314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:33 smithi144 ceph-mon[32129]: pgmap v1314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:35 smithi160 ceph-mon[36715]: pgmap v1315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:35 smithi183 ceph-mon[37273]: pgmap v1315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:35 smithi144 ceph-mon[32129]: pgmap v1315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1359.555326 secs 2024-07-26T11:28:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1359.555326 secs 2024-07-26T11:28:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1359.555326 secs 2024-07-26T11:28:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:37 smithi160 ceph-mon[36715]: pgmap v1316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:37 smithi183 ceph-mon[37273]: pgmap v1316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:37 smithi144 ceph-mon[32129]: pgmap v1316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:38 smithi160 ceph-mon[36715]: pgmap v1317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:38 smithi183 ceph-mon[37273]: pgmap v1317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:38 smithi144 ceph-mon[32129]: pgmap v1317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:41 smithi160 ceph-mon[36715]: pgmap v1318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1364.555445 secs 2024-07-26T11:28:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:41 smithi183 ceph-mon[37273]: pgmap v1318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1364.555445 secs 2024-07-26T11:28:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:41 smithi144 ceph-mon[32129]: pgmap v1318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1364.555445 secs 2024-07-26T11:28:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:42.813+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:42.814+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:42.814+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:42.814+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:42.814+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:43 smithi160 ceph-mon[36715]: pgmap v1319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:43 smithi183 ceph-mon[37273]: pgmap v1319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:43 smithi144 ceph-mon[32129]: pgmap v1319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:45.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:45 smithi144 ceph-mon[32129]: pgmap v1320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:45 smithi160 ceph-mon[36715]: pgmap v1320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:45 smithi183 ceph-mon[37273]: pgmap v1320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1369.555592 secs 2024-07-26T11:28:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1369.555592 secs 2024-07-26T11:28:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1369.555592 secs 2024-07-26T11:28:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:47 smithi160 ceph-mon[36715]: pgmap v1321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:47 smithi183 ceph-mon[37273]: pgmap v1321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:47 smithi144 ceph-mon[32129]: pgmap v1321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:48 smithi160 ceph-mon[36715]: pgmap v1322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:48 smithi183 ceph-mon[37273]: pgmap v1322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:48 smithi144 ceph-mon[32129]: pgmap v1322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:51 smithi160 ceph-mon[36715]: pgmap v1323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1374.555711 secs 2024-07-26T11:28:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:51 smithi183 ceph-mon[37273]: pgmap v1323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1374.555711 secs 2024-07-26T11:28:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:51 smithi144 ceph-mon[32129]: pgmap v1323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1374.555711 secs 2024-07-26T11:28:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:53 smithi160 ceph-mon[36715]: pgmap v1324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:53 smithi183 ceph-mon[37273]: pgmap v1324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:53 smithi144 ceph-mon[32129]: pgmap v1324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:55 smithi160 ceph-mon[36715]: pgmap v1325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:55 smithi183 ceph-mon[37273]: pgmap v1325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:55 smithi144 ceph-mon[32129]: pgmap v1325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1379.555812 secs 2024-07-26T11:28:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1379.555812 secs 2024-07-26T11:28:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1379.555812 secs 2024-07-26T11:28:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:57 smithi160 ceph-mon[36715]: pgmap v1326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:57.829+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:57.830+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:57.830+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:57.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:28:57.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:57 smithi183 ceph-mon[37273]: pgmap v1326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:57 smithi144 ceph-mon[32129]: pgmap v1326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:28:58 smithi160 ceph-mon[36715]: pgmap v1327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:28:58 smithi183 ceph-mon[37273]: pgmap v1327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:28:58 smithi144 ceph-mon[32129]: pgmap v1327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:01 smithi160 ceph-mon[36715]: pgmap v1328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1384.555967 secs 2024-07-26T11:29:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:01 smithi183 ceph-mon[37273]: pgmap v1328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1384.555967 secs 2024-07-26T11:29:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:01 smithi144 ceph-mon[32129]: pgmap v1328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1384.555967 secs 2024-07-26T11:29:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:03 smithi160 ceph-mon[36715]: pgmap v1329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:03 smithi183 ceph-mon[37273]: pgmap v1329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:03 smithi144 ceph-mon[32129]: pgmap v1329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:05 smithi160 ceph-mon[36715]: pgmap v1330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:05 smithi183 ceph-mon[37273]: pgmap v1330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:05.950 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:05 smithi144 ceph-mon[32129]: pgmap v1330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1389.556067 secs 2024-07-26T11:29:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:06 smithi160 ceph-mon[36715]: pgmap v1331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1389.556067 secs 2024-07-26T11:29:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:06 smithi183 ceph-mon[37273]: pgmap v1331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1389.556067 secs 2024-07-26T11:29:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:06 smithi144 ceph-mon[32129]: pgmap v1331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:09 smithi160 ceph-mon[36715]: pgmap v1332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:09 smithi183 ceph-mon[37273]: pgmap v1332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:09 smithi144 ceph-mon[32129]: pgmap v1332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:11 smithi160 ceph-mon[36715]: pgmap v1333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1394.556192 secs 2024-07-26T11:29:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:11 smithi183 ceph-mon[37273]: pgmap v1333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1394.556192 secs 2024-07-26T11:29:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:11 smithi144 ceph-mon[32129]: pgmap v1333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1394.556192 secs 2024-07-26T11:29:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:12.832+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:12.833+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:12.833+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:12.833+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:12.835 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:12.833+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:12 smithi183 ceph-mon[37273]: pgmap v1334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:12 smithi144 ceph-mon[32129]: pgmap v1334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:12 smithi160 ceph-mon[36715]: pgmap v1334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:15 smithi160 ceph-mon[36715]: pgmap v1335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:15 smithi183 ceph-mon[37273]: pgmap v1335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:15 smithi144 ceph-mon[32129]: pgmap v1335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1399.556327 secs 2024-07-26T11:29:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1399.556327 secs 2024-07-26T11:29:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1399.556327 secs 2024-07-26T11:29:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:17 smithi160 ceph-mon[36715]: pgmap v1336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:17 smithi183 ceph-mon[37273]: pgmap v1336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:17 smithi144 ceph-mon[32129]: pgmap v1336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:18 smithi160 ceph-mon[36715]: pgmap v1337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:18 smithi183 ceph-mon[37273]: pgmap v1337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:18 smithi144 ceph-mon[32129]: pgmap v1337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:21 smithi144 ceph-mon[32129]: pgmap v1338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1404.556418 secs 2024-07-26T11:29:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:21 smithi160 ceph-mon[36715]: pgmap v1338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1404.556418 secs 2024-07-26T11:29:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:21 smithi183 ceph-mon[37273]: pgmap v1338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1404.556418 secs 2024-07-26T11:29:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:22 smithi160 ceph-mon[36715]: pgmap v1339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:22 smithi160 ceph-mon[36715]: pgmap v1340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:22 smithi183 ceph-mon[37273]: pgmap v1339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:22 smithi183 ceph-mon[37273]: pgmap v1340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:22 smithi144 ceph-mon[32129]: pgmap v1339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:22 smithi144 ceph-mon[32129]: pgmap v1340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:29:25.588 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:25 smithi183 ceph-mon[37273]: pgmap v1341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:25 smithi144 ceph-mon[32129]: pgmap v1341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:25 smithi160 ceph-mon[36715]: pgmap v1341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1409.556566 secs 2024-07-26T11:29:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1409.556566 secs 2024-07-26T11:29:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1409.556566 secs 2024-07-26T11:29:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:27 smithi183 ceph-mon[37273]: pgmap v1342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:27 smithi144 ceph-mon[32129]: pgmap v1342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:27 smithi160 ceph-mon[36715]: pgmap v1342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:27.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:27.849+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:27.850+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:27.850+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:27.850+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:27.850+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:29 smithi183 ceph-mon[37273]: pgmap v1343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:29 smithi144 ceph-mon[32129]: pgmap v1343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:29 smithi160 ceph-mon[36715]: pgmap v1343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:29:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:31 smithi183 ceph-mon[37273]: pgmap v1344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1414.556646 secs 2024-07-26T11:29:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:31 smithi144 ceph-mon[32129]: pgmap v1344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1414.556646 secs 2024-07-26T11:29:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:31 smithi160 ceph-mon[36715]: pgmap v1344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1414.556646 secs 2024-07-26T11:29:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:33 smithi183 ceph-mon[37273]: pgmap v1345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:33 smithi144 ceph-mon[32129]: pgmap v1345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:33 smithi160 ceph-mon[36715]: pgmap v1345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:35 smithi183 ceph-mon[37273]: pgmap v1346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:35 smithi144 ceph-mon[32129]: pgmap v1346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:35 smithi160 ceph-mon[36715]: pgmap v1346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1419.556719 secs 2024-07-26T11:29:36.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1419.556719 secs 2024-07-26T11:29:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1419.556719 secs 2024-07-26T11:29:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:37 smithi144 ceph-mon[32129]: pgmap v1347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:37 smithi183 ceph-mon[37273]: pgmap v1347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:37 smithi160 ceph-mon[36715]: pgmap v1347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:39 smithi183 ceph-mon[37273]: pgmap v1348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:39 smithi144 ceph-mon[32129]: pgmap v1348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:39 smithi160 ceph-mon[36715]: pgmap v1348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:41 smithi144 ceph-mon[32129]: pgmap v1349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1424.556816 secs 2024-07-26T11:29:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:41 smithi160 ceph-mon[36715]: pgmap v1349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1424.556816 secs 2024-07-26T11:29:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:41 smithi183 ceph-mon[37273]: pgmap v1349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1424.556816 secs 2024-07-26T11:29:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:42.856+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:42.857+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:42.857+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:42.857+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:42.857+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:43 smithi144 ceph-mon[32129]: pgmap v1350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:43 smithi160 ceph-mon[36715]: pgmap v1350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:43 smithi183 ceph-mon[37273]: pgmap v1350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:45 smithi144 ceph-mon[32129]: pgmap v1351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:45 smithi160 ceph-mon[36715]: pgmap v1351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:45 smithi183 ceph-mon[37273]: pgmap v1351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:46.340+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:29:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:46.340+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:29:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:46.340+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T11:29:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:46.340+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T11:29:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1429.556935 secs 2024-07-26T11:29:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1429.556935 secs 2024-07-26T11:29:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1429.556935 secs 2024-07-26T11:29:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:47 smithi144 ceph-mon[32129]: pgmap v1352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:47 smithi160 ceph-mon[36715]: pgmap v1352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:47 smithi183 ceph-mon[37273]: pgmap v1352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:49 smithi144 ceph-mon[32129]: pgmap v1353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:49 smithi160 ceph-mon[36715]: pgmap v1353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:49 smithi183 ceph-mon[37273]: pgmap v1353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:51 smithi144 ceph-mon[32129]: pgmap v1354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1434.557061 secs 2024-07-26T11:29:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:51 smithi160 ceph-mon[36715]: pgmap v1354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1434.557061 secs 2024-07-26T11:29:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:51 smithi183 ceph-mon[37273]: pgmap v1354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1434.557061 secs 2024-07-26T11:29:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:53 smithi144 ceph-mon[32129]: pgmap v1355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:53 smithi160 ceph-mon[36715]: pgmap v1355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:53 smithi183 ceph-mon[37273]: pgmap v1355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:55 smithi144 ceph-mon[32129]: pgmap v1356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:55 smithi160 ceph-mon[36715]: pgmap v1356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:55 smithi183 ceph-mon[37273]: pgmap v1356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1439.557161 secs 2024-07-26T11:29:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1439.557161 secs 2024-07-26T11:29:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1439.557161 secs 2024-07-26T11:29:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:57 smithi144 ceph-mon[32129]: pgmap v1357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:57 smithi160 ceph-mon[36715]: pgmap v1357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:57.872+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:57.872+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:57.873+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:57.873+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:29:57.873+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:57 smithi183 ceph-mon[37273]: pgmap v1357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:29:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:29:59 smithi144 ceph-mon[32129]: pgmap v1358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:29:59 smithi160 ceph-mon[36715]: pgmap v1358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:29:59.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:29:59 smithi183 ceph-mon[37273]: pgmap v1358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-07-26T11:30:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.j on smithi144 is in error state 2024-07-26T11:30:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.d on smithi144 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.a on smithi144 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.k on smithi160 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.l on smithi183 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.f on smithi183 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.c on smithi183 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: daemon mds.i on smithi183 is in error state 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T11:30:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:00 smithi160 ceph-mon[36715]: mds.h(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.j on smithi144 is in error state 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.d on smithi144 is in error state 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.a on smithi144 is in error state 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.k on smithi160 is in error state 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.l on smithi183 is in error state 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.f on smithi183 is in error state 2024-07-26T11:30:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.c on smithi183 is in error state 2024-07-26T11:30:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: daemon mds.i on smithi183 is in error state 2024-07-26T11:30:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T11:30:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:00 smithi183 ceph-mon[37273]: mds.h(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:30:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-07-26T11:30:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.j on smithi144 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.d on smithi144 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.a on smithi144 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.k on smithi160 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.l on smithi183 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.f on smithi183 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.c on smithi183 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: daemon mds.i on smithi183 is in error state 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T11:30:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:00 smithi144 ceph-mon[32129]: mds.h(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:30:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:01 smithi160 ceph-mon[36715]: pgmap v1359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:01 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 1444.557305 secs 2024-07-26T11:30:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:01 smithi160 ceph-mon[36715]: slow request 1444.557304 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:30:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:01 smithi183 ceph-mon[37273]: pgmap v1359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:01 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 1444.557305 secs 2024-07-26T11:30:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:01 smithi183 ceph-mon[37273]: slow request 1444.557304 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:30:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:01 smithi144 ceph-mon[32129]: pgmap v1359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:01 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 1444.557305 secs 2024-07-26T11:30:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:01 smithi144 ceph-mon[32129]: slow request 1444.557304 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:30:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:03 smithi160 ceph-mon[36715]: pgmap v1360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:03 smithi183 ceph-mon[37273]: pgmap v1360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:03 smithi144 ceph-mon[32129]: pgmap v1360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:05 smithi160 ceph-mon[36715]: pgmap v1361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:05 smithi183 ceph-mon[37273]: pgmap v1361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:05 smithi144 ceph-mon[32129]: pgmap v1361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:06 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 1449.557416 secs 2024-07-26T11:30:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:06 smithi160 ceph-mon[36715]: slow request 1444.696605 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:30:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:06 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 1449.557416 secs 2024-07-26T11:30:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:06 smithi183 ceph-mon[37273]: slow request 1444.696605 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:30:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:06 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 1449.557416 secs 2024-07-26T11:30:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:06 smithi144 ceph-mon[32129]: slow request 1444.696605 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:30:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:07 smithi160 ceph-mon[36715]: pgmap v1362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:07 smithi183 ceph-mon[37273]: pgmap v1362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:07 smithi144 ceph-mon[32129]: pgmap v1362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:09 smithi160 ceph-mon[36715]: pgmap v1363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:09 smithi183 ceph-mon[37273]: pgmap v1363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:09 smithi144 ceph-mon[32129]: pgmap v1363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:11 smithi160 ceph-mon[36715]: pgmap v1364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1454.557565 secs 2024-07-26T11:30:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:11 smithi183 ceph-mon[37273]: pgmap v1364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1454.557565 secs 2024-07-26T11:30:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:11 smithi144 ceph-mon[32129]: pgmap v1364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1454.557565 secs 2024-07-26T11:30:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:12.889+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:12.890+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:12.890+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:12.890+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:12.891+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:13 smithi160 ceph-mon[36715]: pgmap v1365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:13 smithi183 ceph-mon[37273]: pgmap v1365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:13 smithi144 ceph-mon[32129]: pgmap v1365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:15.775 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:15 smithi144 ceph-mon[32129]: pgmap v1366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:15 smithi160 ceph-mon[36715]: pgmap v1366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:15 smithi183 ceph-mon[37273]: pgmap v1366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1459.557681 secs 2024-07-26T11:30:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:16 smithi160 ceph-mon[36715]: pgmap v1367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1459.557681 secs 2024-07-26T11:30:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:16 smithi183 ceph-mon[37273]: pgmap v1367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1459.557681 secs 2024-07-26T11:30:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:16 smithi144 ceph-mon[32129]: pgmap v1367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:19 smithi183 ceph-mon[37273]: pgmap v1368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:19 smithi144 ceph-mon[32129]: pgmap v1368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:19 smithi160 ceph-mon[36715]: pgmap v1368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:21 smithi183 ceph-mon[37273]: pgmap v1369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1464.557781 secs 2024-07-26T11:30:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:21 smithi144 ceph-mon[32129]: pgmap v1369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1464.557781 secs 2024-07-26T11:30:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:21 smithi160 ceph-mon[36715]: pgmap v1369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1464.557781 secs 2024-07-26T11:30:23.631 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:23 smithi160 ceph-mon[36715]: pgmap v1370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:30:23.631 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:23.631 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.631 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.631 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.631 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:23 smithi183 ceph-mon[37273]: pgmap v1370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:23 smithi144 ceph-mon[32129]: pgmap v1370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:30:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:25 smithi160 ceph-mon[36715]: pgmap v1371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:25 smithi183 ceph-mon[37273]: pgmap v1371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:25 smithi144 ceph-mon[32129]: pgmap v1371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:30:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1469.557855 secs 2024-07-26T11:30:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1469.557855 secs 2024-07-26T11:30:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1469.557855 secs 2024-07-26T11:30:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:27 smithi160 ceph-mon[36715]: pgmap v1372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:27.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:27.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:27 smithi183 ceph-mon[37273]: pgmap v1372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:27.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:27.896+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:27.896+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:27.896+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:27.896+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:27 smithi144 ceph-mon[32129]: pgmap v1372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:30:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:28 smithi160 ceph-mon[36715]: pgmap v1373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:30:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:28 smithi160 ceph-mon[36715]: pgmap v1374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:28 smithi183 ceph-mon[37273]: pgmap v1373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:30:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:28 smithi183 ceph-mon[37273]: pgmap v1374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:28 smithi144 ceph-mon[32129]: pgmap v1373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:30:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:28 smithi144 ceph-mon[32129]: pgmap v1374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:29 smithi160 ceph-mon[36715]: pgmap v1375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:29 smithi183 ceph-mon[37273]: pgmap v1375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:29 smithi144 ceph-mon[32129]: pgmap v1375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1474.557971 secs 2024-07-26T11:30:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1474.557971 secs 2024-07-26T11:30:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1474.557971 secs 2024-07-26T11:30:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:31.347+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:30:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:31.347+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:30:31.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:31.348+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T11:30:31.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:31.348+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T11:30:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:31 smithi144 ceph-mon[32129]: pgmap v1376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:30:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:31 smithi160 ceph-mon[36715]: pgmap v1376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:30:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:31 smithi183 ceph-mon[37273]: pgmap v1376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:30:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:34 smithi144 ceph-mon[32129]: pgmap v1377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:30:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:34 smithi160 ceph-mon[36715]: pgmap v1377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:30:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:34 smithi183 ceph-mon[37273]: pgmap v1377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:30:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:36 smithi144 ceph-mon[32129]: pgmap v1378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:30:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1479.558087 secs 2024-07-26T11:30:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:36 smithi160 ceph-mon[36715]: pgmap v1378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:30:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1479.558087 secs 2024-07-26T11:30:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:36 smithi183 ceph-mon[37273]: pgmap v1378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:30:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1479.558087 secs 2024-07-26T11:30:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:38 smithi144 ceph-mon[32129]: pgmap v1379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:38 smithi160 ceph-mon[36715]: pgmap v1379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:38 smithi183 ceph-mon[37273]: pgmap v1379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:40 smithi144 ceph-mon[32129]: pgmap v1380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:40 smithi160 ceph-mon[36715]: pgmap v1380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:40.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:40 smithi183 ceph-mon[37273]: pgmap v1380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1484.558225 secs 2024-07-26T11:30:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1484.558225 secs 2024-07-26T11:30:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1484.558225 secs 2024-07-26T11:30:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:42 smithi144 ceph-mon[32129]: pgmap v1381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:42 smithi160 ceph-mon[36715]: pgmap v1381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:42 smithi183 ceph-mon[37273]: pgmap v1381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:42.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:42.914+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:42.914+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:42.914+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:42.914+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:44 smithi160 ceph-mon[36715]: pgmap v1382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:44 smithi183 ceph-mon[37273]: pgmap v1382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:44 smithi144 ceph-mon[32129]: pgmap v1382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:46.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:46 smithi144 ceph-mon[32129]: pgmap v1383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:46.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1489.558347 secs 2024-07-26T11:30:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:46 smithi160 ceph-mon[36715]: pgmap v1383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1489.558347 secs 2024-07-26T11:30:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:46 smithi183 ceph-mon[37273]: pgmap v1383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1489.558347 secs 2024-07-26T11:30:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:48 smithi160 ceph-mon[36715]: pgmap v1384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:48 smithi183 ceph-mon[37273]: pgmap v1384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:48 smithi144 ceph-mon[32129]: pgmap v1384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:50 smithi160 ceph-mon[36715]: pgmap v1385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:50 smithi183 ceph-mon[37273]: pgmap v1385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:50 smithi144 ceph-mon[32129]: pgmap v1385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1494.558465 secs 2024-07-26T11:30:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1494.558465 secs 2024-07-26T11:30:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1494.558465 secs 2024-07-26T11:30:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:52 smithi160 ceph-mon[36715]: pgmap v1386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:52 smithi183 ceph-mon[37273]: pgmap v1386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:52 smithi144 ceph-mon[32129]: pgmap v1386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:30:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:54 smithi160 ceph-mon[36715]: pgmap v1387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:54 smithi183 ceph-mon[37273]: pgmap v1387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:54 smithi144 ceph-mon[32129]: pgmap v1387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:56 smithi160 ceph-mon[36715]: pgmap v1388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1499.558588 secs 2024-07-26T11:30:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:56 smithi183 ceph-mon[37273]: pgmap v1388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1499.558588 secs 2024-07-26T11:30:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:56 smithi144 ceph-mon[32129]: pgmap v1388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:30:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1499.558588 secs 2024-07-26T11:30:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:57.929+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:57.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:57.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:57.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:30:57.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:30:58 smithi160 ceph-mon[36715]: pgmap v1389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:30:58 smithi183 ceph-mon[37273]: pgmap v1389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:30:58 smithi144 ceph-mon[32129]: pgmap v1389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:00 smithi160 ceph-mon[36715]: pgmap v1390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:00 smithi183 ceph-mon[37273]: pgmap v1390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:00 smithi144 ceph-mon[32129]: pgmap v1390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1504.558679 secs 2024-07-26T11:31:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1504.558679 secs 2024-07-26T11:31:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1504.558679 secs 2024-07-26T11:31:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:02 smithi160 ceph-mon[36715]: pgmap v1391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:02 smithi183 ceph-mon[37273]: pgmap v1391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:02 smithi144 ceph-mon[32129]: pgmap v1391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:04 smithi160 ceph-mon[36715]: pgmap v1392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:04 smithi183 ceph-mon[37273]: pgmap v1392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:04 smithi144 ceph-mon[32129]: pgmap v1392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:06 smithi160 ceph-mon[36715]: pgmap v1393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1509.558801 secs 2024-07-26T11:31:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:06 smithi183 ceph-mon[37273]: pgmap v1393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1509.558801 secs 2024-07-26T11:31:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:06 smithi144 ceph-mon[32129]: pgmap v1393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1509.558801 secs 2024-07-26T11:31:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:07 smithi160 ceph-mon[36715]: pgmap v1394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:07 smithi183 ceph-mon[37273]: pgmap v1394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:07 smithi144 ceph-mon[32129]: pgmap v1394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:10 smithi144 ceph-mon[32129]: pgmap v1395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:10 smithi160 ceph-mon[36715]: pgmap v1395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:10 smithi183 ceph-mon[37273]: pgmap v1395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1514.558939 secs 2024-07-26T11:31:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1514.558939 secs 2024-07-26T11:31:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1514.558939 secs 2024-07-26T11:31:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:12 smithi160 ceph-mon[36715]: pgmap v1396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:12 smithi183 ceph-mon[37273]: pgmap v1396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:12.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:12.932+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:12.933+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:12.933+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:12.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:12.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:12 smithi144 ceph-mon[32129]: pgmap v1396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:14 smithi160 ceph-mon[36715]: pgmap v1397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:14 smithi183 ceph-mon[37273]: pgmap v1397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:14 smithi144 ceph-mon[32129]: pgmap v1397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:16.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:16 smithi144 ceph-mon[32129]: pgmap v1398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:16.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1519.559060 secs 2024-07-26T11:31:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:16 smithi160 ceph-mon[36715]: pgmap v1398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1519.559060 secs 2024-07-26T11:31:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:16 smithi183 ceph-mon[37273]: pgmap v1398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1519.559060 secs 2024-07-26T11:31:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:18 smithi160 ceph-mon[36715]: pgmap v1399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:18 smithi183 ceph-mon[37273]: pgmap v1399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:18 smithi144 ceph-mon[32129]: pgmap v1399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:20 smithi160 ceph-mon[36715]: pgmap v1400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:20 smithi183 ceph-mon[37273]: pgmap v1400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:20 smithi144 ceph-mon[32129]: pgmap v1400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1524.559152 secs 2024-07-26T11:31:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1524.559152 secs 2024-07-26T11:31:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1524.559152 secs 2024-07-26T11:31:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:22 smithi160 ceph-mon[36715]: pgmap v1401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:22 smithi183 ceph-mon[37273]: pgmap v1401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:22 smithi144 ceph-mon[32129]: pgmap v1401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:24 smithi160 ceph-mon[36715]: pgmap v1402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:24 smithi183 ceph-mon[37273]: pgmap v1402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:24 smithi144 ceph-mon[32129]: pgmap v1402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:26 smithi160 ceph-mon[36715]: pgmap v1403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1529.559278 secs 2024-07-26T11:31:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:26 smithi183 ceph-mon[37273]: pgmap v1403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1529.559278 secs 2024-07-26T11:31:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:26 smithi144 ceph-mon[32129]: pgmap v1403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1529.559278 secs 2024-07-26T11:31:27.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:27.949+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:27.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:27.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:27.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:27.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:27.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:27.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:28 smithi160 ceph-mon[36715]: pgmap v1404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:28 smithi183 ceph-mon[37273]: pgmap v1404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:28 smithi144 ceph-mon[32129]: pgmap v1404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:31:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:31:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:31:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:30 smithi160 ceph-mon[36715]: pgmap v1405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:30 smithi160 ceph-mon[36715]: pgmap v1406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:31:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:30 smithi183 ceph-mon[37273]: pgmap v1405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:30 smithi183 ceph-mon[37273]: pgmap v1406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:31:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:30 smithi144 ceph-mon[32129]: pgmap v1405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:30 smithi144 ceph-mon[32129]: pgmap v1406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:31:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1534.559396 secs 2024-07-26T11:31:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1534.559396 secs 2024-07-26T11:31:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1534.559396 secs 2024-07-26T11:31:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:32 smithi160 ceph-mon[36715]: pgmap v1407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:32 smithi183 ceph-mon[37273]: pgmap v1407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:32 smithi144 ceph-mon[32129]: pgmap v1407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:33 smithi183 ceph-mon[37273]: pgmap v1408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:31:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:33 smithi144 ceph-mon[32129]: pgmap v1408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:31:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:33 smithi160 ceph-mon[36715]: pgmap v1408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:31:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:35 smithi144 ceph-mon[32129]: pgmap v1409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:31:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:35 smithi160 ceph-mon[36715]: pgmap v1409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:31:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:35 smithi183 ceph-mon[37273]: pgmap v1409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:31:36.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1539.559519 secs 2024-07-26T11:31:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1539.559519 secs 2024-07-26T11:31:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1539.559519 secs 2024-07-26T11:31:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:37 smithi144 ceph-mon[32129]: pgmap v1410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:37 smithi160 ceph-mon[36715]: pgmap v1410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:37 smithi183 ceph-mon[37273]: pgmap v1410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:31:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:39 smithi144 ceph-mon[32129]: pgmap v1411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:31:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:39 smithi160 ceph-mon[36715]: pgmap v1411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:31:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:39 smithi183 ceph-mon[37273]: pgmap v1411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:31:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1544.559674 secs 2024-07-26T11:31:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:41 smithi144 ceph-mon[32129]: pgmap v1412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1544.559674 secs 2024-07-26T11:31:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:41 smithi160 ceph-mon[36715]: pgmap v1412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1544.559674 secs 2024-07-26T11:31:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:41 smithi183 ceph-mon[37273]: pgmap v1412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:42.965 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:42.963+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:42.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:42.964+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:42.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:42.964+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:42.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:42.964+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:42.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:42.964+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:43 smithi144 ceph-mon[32129]: pgmap v1413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:43 smithi160 ceph-mon[36715]: pgmap v1413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:43 smithi183 ceph-mon[37273]: pgmap v1413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:45 smithi144 ceph-mon[32129]: pgmap v1414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:45 smithi160 ceph-mon[36715]: pgmap v1414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:45 smithi183 ceph-mon[37273]: pgmap v1414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1549.559797 secs 2024-07-26T11:31:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1549.559797 secs 2024-07-26T11:31:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1549.559797 secs 2024-07-26T11:31:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:47 smithi144 ceph-mon[32129]: pgmap v1415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:47 smithi160 ceph-mon[36715]: pgmap v1415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:47 smithi183 ceph-mon[37273]: pgmap v1415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:50.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:49 smithi160 ceph-mon[36715]: pgmap v1416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:49 smithi183 ceph-mon[37273]: pgmap v1416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:49 smithi144 ceph-mon[32129]: pgmap v1416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1554.559930 secs 2024-07-26T11:31:52.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:51 smithi160 ceph-mon[36715]: pgmap v1417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1554.559930 secs 2024-07-26T11:31:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:51 smithi183 ceph-mon[37273]: pgmap v1417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1554.559930 secs 2024-07-26T11:31:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:51 smithi144 ceph-mon[32129]: pgmap v1417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:53 smithi160 ceph-mon[36715]: pgmap v1418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:53 smithi183 ceph-mon[37273]: pgmap v1418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:53 smithi144 ceph-mon[32129]: pgmap v1418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:55 smithi160 ceph-mon[36715]: pgmap v1419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:55 smithi183 ceph-mon[37273]: pgmap v1419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:55 smithi144 ceph-mon[32129]: pgmap v1419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:31:57.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1559.560036 secs 2024-07-26T11:31:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1559.560036 secs 2024-07-26T11:31:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1559.560036 secs 2024-07-26T11:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:57.976+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:57.977+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:57.977+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:57.977+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:31:57.977+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:57 smithi160 ceph-mon[36715]: pgmap v1420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:57 smithi183 ceph-mon[37273]: pgmap v1420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:31:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:57 smithi144 ceph-mon[32129]: pgmap v1420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:31:59 smithi160 ceph-mon[36715]: pgmap v1421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:31:59 smithi183 ceph-mon[37273]: pgmap v1421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:31:59 smithi144 ceph-mon[32129]: pgmap v1421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1564.560176 secs 2024-07-26T11:32:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:01 smithi183 ceph-mon[37273]: pgmap v1422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1564.560176 secs 2024-07-26T11:32:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:01 smithi160 ceph-mon[36715]: pgmap v1422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:02.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1564.560176 secs 2024-07-26T11:32:02.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:01 smithi144 ceph-mon[32129]: pgmap v1422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:03 smithi160 ceph-mon[36715]: pgmap v1423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:03 smithi183 ceph-mon[37273]: pgmap v1423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:03 smithi144 ceph-mon[32129]: pgmap v1423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:05 smithi160 ceph-mon[36715]: pgmap v1424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:05 smithi183 ceph-mon[37273]: pgmap v1424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:05 smithi144 ceph-mon[32129]: pgmap v1424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1569.560306 secs 2024-07-26T11:32:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1569.560306 secs 2024-07-26T11:32:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1569.560306 secs 2024-07-26T11:32:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:08 smithi144 ceph-mon[32129]: pgmap v1425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:08 smithi160 ceph-mon[36715]: pgmap v1425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:08 smithi183 ceph-mon[37273]: pgmap v1425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:10 smithi160 ceph-mon[36715]: pgmap v1426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:10 smithi183 ceph-mon[37273]: pgmap v1426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:10 smithi144 ceph-mon[32129]: pgmap v1426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1574.560404 secs 2024-07-26T11:32:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1574.560404 secs 2024-07-26T11:32:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1574.560404 secs 2024-07-26T11:32:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:12 smithi160 ceph-mon[36715]: pgmap v1427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:12 smithi183 ceph-mon[37273]: pgmap v1427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:12 smithi144 ceph-mon[32129]: pgmap v1427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:12.993+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:12.994+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:12.994+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:12.994+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:12.994+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:14 smithi160 ceph-mon[36715]: pgmap v1428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:14 smithi183 ceph-mon[37273]: pgmap v1428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:14 smithi144 ceph-mon[32129]: pgmap v1428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:16.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:16 smithi144 ceph-mon[32129]: pgmap v1429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:16.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1579.560524 secs 2024-07-26T11:32:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:16 smithi160 ceph-mon[36715]: pgmap v1429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1579.560524 secs 2024-07-26T11:32:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:16 smithi183 ceph-mon[37273]: pgmap v1429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1579.560524 secs 2024-07-26T11:32:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:18 smithi160 ceph-mon[36715]: pgmap v1430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:18 smithi183 ceph-mon[37273]: pgmap v1430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:18 smithi144 ceph-mon[32129]: pgmap v1430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:20 smithi160 ceph-mon[36715]: pgmap v1431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:20 smithi183 ceph-mon[37273]: pgmap v1431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:20 smithi144 ceph-mon[32129]: pgmap v1431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1584.560642 secs 2024-07-26T11:32:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1584.560642 secs 2024-07-26T11:32:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1584.560642 secs 2024-07-26T11:32:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:22 smithi160 ceph-mon[36715]: pgmap v1432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:22 smithi183 ceph-mon[37273]: pgmap v1432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:22 smithi144 ceph-mon[32129]: pgmap v1432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:24 smithi160 ceph-mon[36715]: pgmap v1433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:24 smithi183 ceph-mon[37273]: pgmap v1433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:24 smithi144 ceph-mon[32129]: pgmap v1433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:26 smithi160 ceph-mon[36715]: pgmap v1434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1589.560759 secs 2024-07-26T11:32:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:26 smithi183 ceph-mon[37273]: pgmap v1434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1589.560759 secs 2024-07-26T11:32:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:26 smithi144 ceph-mon[32129]: pgmap v1434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1589.560759 secs 2024-07-26T11:32:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:28.010+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:28.011+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:28.011+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:28.011+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:28.011+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:28 smithi144 ceph-mon[32129]: pgmap v1435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:28 smithi160 ceph-mon[36715]: pgmap v1435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:28 smithi183 ceph-mon[37273]: pgmap v1435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:30 smithi183 ceph-mon[37273]: pgmap v1436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:32:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:30 smithi144 ceph-mon[32129]: pgmap v1436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:32:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:30 smithi160 ceph-mon[36715]: pgmap v1436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:32:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:31 smithi183 ceph-mon[37273]: pgmap v1437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:32:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:31 smithi183 ceph-mon[37273]: pgmap v1438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:32:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1594.560884 secs 2024-07-26T11:32:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:31 smithi144 ceph-mon[32129]: pgmap v1437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:32:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:31 smithi144 ceph-mon[32129]: pgmap v1438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:32:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1594.560884 secs 2024-07-26T11:32:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:31 smithi160 ceph-mon[36715]: pgmap v1437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:32:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:31 smithi160 ceph-mon[36715]: pgmap v1438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:32:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1594.560884 secs 2024-07-26T11:32:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:33 smithi183 ceph-mon[37273]: pgmap v1439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:32:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:33 smithi144 ceph-mon[32129]: pgmap v1439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:32:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:33 smithi160 ceph-mon[36715]: pgmap v1439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:32:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:35 smithi183 ceph-mon[37273]: pgmap v1440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:32:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:35 smithi144 ceph-mon[32129]: pgmap v1440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:32:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:35 smithi160 ceph-mon[36715]: pgmap v1440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:32:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1599.561005 secs 2024-07-26T11:32:36.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1599.561005 secs 2024-07-26T11:32:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1599.561005 secs 2024-07-26T11:32:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:37 smithi183 ceph-mon[37273]: pgmap v1441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:32:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:37 smithi144 ceph-mon[32129]: pgmap v1441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:32:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:37 smithi160 ceph-mon[36715]: pgmap v1441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:32:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:39 smithi183 ceph-mon[37273]: pgmap v1442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:32:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:39 smithi144 ceph-mon[32129]: pgmap v1442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:32:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:39 smithi160 ceph-mon[36715]: pgmap v1442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:32:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:41 smithi183 ceph-mon[37273]: pgmap v1443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:32:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1604.561126 secs 2024-07-26T11:32:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:41 smithi144 ceph-mon[32129]: pgmap v1443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:32:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1604.561126 secs 2024-07-26T11:32:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:41 smithi160 ceph-mon[36715]: pgmap v1443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:32:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1604.561126 secs 2024-07-26T11:32:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:43.026+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:43.027+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:43.027+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:43.027+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:43.028+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:43 smithi183 ceph-mon[37273]: pgmap v1444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:43 smithi144 ceph-mon[32129]: pgmap v1444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:43 smithi160 ceph-mon[36715]: pgmap v1444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:45 smithi183 ceph-mon[37273]: pgmap v1445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:45 smithi144 ceph-mon[32129]: pgmap v1445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:45 smithi160 ceph-mon[36715]: pgmap v1445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:46.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:46.371+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:32:46.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:46.371+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:32:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:46.376+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:32:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:46.376+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:32:46.622 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1609.561203 secs 2024-07-26T11:32:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1609.561203 secs 2024-07-26T11:32:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1609.561203 secs 2024-07-26T11:32:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:47 smithi183 ceph-mon[37273]: pgmap v1446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:47 smithi144 ceph-mon[32129]: pgmap v1446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:47 smithi160 ceph-mon[36715]: pgmap v1446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:49 smithi183 ceph-mon[37273]: pgmap v1447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:49 smithi144 ceph-mon[32129]: pgmap v1447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:49 smithi160 ceph-mon[36715]: pgmap v1447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:51 smithi144 ceph-mon[32129]: pgmap v1448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1614.561347 secs 2024-07-26T11:32:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:51 smithi160 ceph-mon[36715]: pgmap v1448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1614.561347 secs 2024-07-26T11:32:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:51 smithi183 ceph-mon[37273]: pgmap v1448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1614.561347 secs 2024-07-26T11:32:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:53 smithi144 ceph-mon[32129]: pgmap v1449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:53 smithi160 ceph-mon[36715]: pgmap v1449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:53 smithi183 ceph-mon[37273]: pgmap v1449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:32:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:55 smithi144 ceph-mon[32129]: pgmap v1450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:55.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:55 smithi160 ceph-mon[36715]: pgmap v1450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:55 smithi183 ceph-mon[37273]: pgmap v1450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1619.561398 secs 2024-07-26T11:32:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1619.561398 secs 2024-07-26T11:32:56.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1619.561398 secs 2024-07-26T11:32:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:57 smithi144 ceph-mon[32129]: pgmap v1451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:57 smithi160 ceph-mon[36715]: pgmap v1451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:57 smithi183 ceph-mon[37273]: pgmap v1451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:32:58.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:58.043+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:58.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:58.043+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:58.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:58.043+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:58.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:58.043+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:58.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:32:58.044+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:59.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:32:59 smithi183 ceph-mon[37273]: pgmap v1452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:32:59 smithi144 ceph-mon[32129]: pgmap v1452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:32:59 smithi160 ceph-mon[36715]: pgmap v1452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:01 smithi144 ceph-mon[32129]: pgmap v1453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1624.561490 secs 2024-07-26T11:33:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:01 smithi160 ceph-mon[36715]: pgmap v1453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1624.561490 secs 2024-07-26T11:33:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:01 smithi183 ceph-mon[37273]: pgmap v1453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1624.561490 secs 2024-07-26T11:33:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:03 smithi144 ceph-mon[32129]: pgmap v1454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:03 smithi160 ceph-mon[36715]: pgmap v1454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:03 smithi183 ceph-mon[37273]: pgmap v1454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:05 smithi144 ceph-mon[32129]: pgmap v1455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:05 smithi160 ceph-mon[36715]: pgmap v1455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:05 smithi183 ceph-mon[37273]: pgmap v1455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:06.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1629.561583 secs 2024-07-26T11:33:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1629.561583 secs 2024-07-26T11:33:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1629.561583 secs 2024-07-26T11:33:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:07 smithi144 ceph-mon[32129]: pgmap v1456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:07 smithi160 ceph-mon[36715]: pgmap v1456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:07 smithi183 ceph-mon[37273]: pgmap v1456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:09 smithi144 ceph-mon[32129]: pgmap v1457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:09 smithi160 ceph-mon[36715]: pgmap v1457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:09 smithi183 ceph-mon[37273]: pgmap v1457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:11 smithi144 ceph-mon[32129]: pgmap v1458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1634.561666 secs 2024-07-26T11:33:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:11 smithi160 ceph-mon[36715]: pgmap v1458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1634.561666 secs 2024-07-26T11:33:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:11 smithi183 ceph-mon[37273]: pgmap v1458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1634.561666 secs 2024-07-26T11:33:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:13.060+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:13.061+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:13.061+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:13.061+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:13.062+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:13 smithi144 ceph-mon[32129]: pgmap v1459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:13 smithi160 ceph-mon[36715]: pgmap v1459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:13 smithi183 ceph-mon[37273]: pgmap v1459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:15 smithi160 ceph-mon[36715]: pgmap v1460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:15 smithi183 ceph-mon[37273]: pgmap v1460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:15 smithi144 ceph-mon[32129]: pgmap v1460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1639.561791 secs 2024-07-26T11:33:16.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1639.561791 secs 2024-07-26T11:33:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1639.561791 secs 2024-07-26T11:33:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:17 smithi160 ceph-mon[36715]: pgmap v1461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:17 smithi183 ceph-mon[37273]: pgmap v1461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:17 smithi144 ceph-mon[32129]: pgmap v1461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:19 smithi160 ceph-mon[36715]: pgmap v1462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:19 smithi183 ceph-mon[37273]: pgmap v1462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:19 smithi144 ceph-mon[32129]: pgmap v1462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:21 smithi160 ceph-mon[36715]: pgmap v1463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1644.561931 secs 2024-07-26T11:33:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:21 smithi183 ceph-mon[37273]: pgmap v1463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1644.561931 secs 2024-07-26T11:33:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:21 smithi144 ceph-mon[32129]: pgmap v1463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:33:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1644.561931 secs 2024-07-26T11:33:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:23 smithi160 ceph-mon[36715]: pgmap v1464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:23.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:23 smithi183 ceph-mon[37273]: pgmap v1464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:23 smithi144 ceph-mon[32129]: pgmap v1464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:25 smithi160 ceph-mon[36715]: pgmap v1465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:25 smithi183 ceph-mon[37273]: pgmap v1465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:25 smithi144 ceph-mon[32129]: pgmap v1465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1649.562066 secs 2024-07-26T11:33:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1649.562066 secs 2024-07-26T11:33:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1649.562066 secs 2024-07-26T11:33:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:27 smithi160 ceph-mon[36715]: pgmap v1466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:27 smithi183 ceph-mon[37273]: pgmap v1466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:27 smithi144 ceph-mon[32129]: pgmap v1466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:28.077+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:28.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:28.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:28.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:28.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:29 smithi160 ceph-mon[36715]: pgmap v1467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:29 smithi183 ceph-mon[37273]: pgmap v1467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:29 smithi144 ceph-mon[32129]: pgmap v1467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:31 smithi160 ceph-mon[36715]: pgmap v1468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1654.562185 secs 2024-07-26T11:33:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:33:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:31 smithi183 ceph-mon[37273]: pgmap v1468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1654.562185 secs 2024-07-26T11:33:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:33:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:31 smithi144 ceph-mon[32129]: pgmap v1468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1654.562185 secs 2024-07-26T11:33:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:33:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:32 smithi160 ceph-mon[36715]: pgmap v1469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:32 smithi160 ceph-mon[36715]: pgmap v1470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:32 smithi183 ceph-mon[37273]: pgmap v1469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:32 smithi183 ceph-mon[37273]: pgmap v1470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:32 smithi144 ceph-mon[32129]: pgmap v1469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:32 smithi144 ceph-mon[32129]: pgmap v1470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:34 smithi160 ceph-mon[36715]: pgmap v1471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:34 smithi183 ceph-mon[37273]: pgmap v1471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:34 smithi144 ceph-mon[32129]: pgmap v1471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:36 smithi183 ceph-mon[37273]: pgmap v1472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1659.562318 secs 2024-07-26T11:33:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:36 smithi144 ceph-mon[32129]: pgmap v1472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1659.562318 secs 2024-07-26T11:33:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:36 smithi160 ceph-mon[36715]: pgmap v1472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:33:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1659.562318 secs 2024-07-26T11:33:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:38 smithi183 ceph-mon[37273]: pgmap v1473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:38 smithi144 ceph-mon[32129]: pgmap v1473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:38 smithi160 ceph-mon[36715]: pgmap v1473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:33:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:39 smithi183 ceph-mon[37273]: pgmap v1474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:33:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:39 smithi144 ceph-mon[32129]: pgmap v1474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:33:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:39 smithi160 ceph-mon[36715]: pgmap v1474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:33:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1664.562418 secs 2024-07-26T11:33:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1664.562418 secs 2024-07-26T11:33:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1664.562418 secs 2024-07-26T11:33:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:41 smithi144 ceph-mon[32129]: pgmap v1475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:41 smithi160 ceph-mon[36715]: pgmap v1475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:41 smithi183 ceph-mon[37273]: pgmap v1475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:43.079+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:43.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:43.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:43.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:43.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:44 smithi183 ceph-mon[37273]: pgmap v1476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:44 smithi144 ceph-mon[32129]: pgmap v1476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:44 smithi160 ceph-mon[36715]: pgmap v1476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:46.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:46 smithi144 ceph-mon[32129]: pgmap v1477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:46.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1669.562487 secs 2024-07-26T11:33:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:46 smithi183 ceph-mon[37273]: pgmap v1477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1669.562487 secs 2024-07-26T11:33:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:46 smithi160 ceph-mon[36715]: pgmap v1477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1669.562487 secs 2024-07-26T11:33:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:48 smithi183 ceph-mon[37273]: pgmap v1478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:48 smithi144 ceph-mon[32129]: pgmap v1478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:48 smithi160 ceph-mon[36715]: pgmap v1478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:50 smithi183 ceph-mon[37273]: pgmap v1479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:50 smithi144 ceph-mon[32129]: pgmap v1479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:50 smithi160 ceph-mon[36715]: pgmap v1479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1674.562628 secs 2024-07-26T11:33:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1674.562628 secs 2024-07-26T11:33:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1674.562628 secs 2024-07-26T11:33:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:52 smithi144 ceph-mon[32129]: pgmap v1480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:52 smithi160 ceph-mon[36715]: pgmap v1480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:52 smithi183 ceph-mon[37273]: pgmap v1480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:54 smithi144 ceph-mon[32129]: pgmap v1481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:54 smithi160 ceph-mon[36715]: pgmap v1481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:54 smithi183 ceph-mon[37273]: pgmap v1481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:56 smithi144 ceph-mon[32129]: pgmap v1482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1679.562752 secs 2024-07-26T11:33:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:56 smithi160 ceph-mon[36715]: pgmap v1482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1679.562752 secs 2024-07-26T11:33:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:56 smithi183 ceph-mon[37273]: pgmap v1482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:33:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1679.562752 secs 2024-07-26T11:33:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:58.094+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:58.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:58.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:58.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:33:58.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:33:58 smithi144 ceph-mon[32129]: pgmap v1483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:33:58 smithi160 ceph-mon[36715]: pgmap v1483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:33:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:33:58 smithi183 ceph-mon[37273]: pgmap v1483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:00.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:00 smithi183 ceph-mon[37273]: pgmap v1484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:00 smithi144 ceph-mon[32129]: pgmap v1484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:00 smithi160 ceph-mon[36715]: pgmap v1484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1684.562869 secs 2024-07-26T11:34:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1684.562869 secs 2024-07-26T11:34:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1684.562869 secs 2024-07-26T11:34:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:02 smithi144 ceph-mon[32129]: pgmap v1485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:02 smithi160 ceph-mon[36715]: pgmap v1485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:02 smithi183 ceph-mon[37273]: pgmap v1485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:04 smithi144 ceph-mon[32129]: pgmap v1486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:04 smithi160 ceph-mon[36715]: pgmap v1486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:04 smithi183 ceph-mon[37273]: pgmap v1486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:06 smithi144 ceph-mon[32129]: pgmap v1487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1689.562990 secs 2024-07-26T11:34:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:06 smithi160 ceph-mon[36715]: pgmap v1487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1689.562990 secs 2024-07-26T11:34:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:06 smithi183 ceph-mon[37273]: pgmap v1487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1689.562990 secs 2024-07-26T11:34:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:08 smithi144 ceph-mon[32129]: pgmap v1488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:08 smithi160 ceph-mon[36715]: pgmap v1488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:08 smithi183 ceph-mon[37273]: pgmap v1488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:10 smithi144 ceph-mon[32129]: pgmap v1489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:10 smithi160 ceph-mon[36715]: pgmap v1489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:10 smithi183 ceph-mon[37273]: pgmap v1489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1694.563130 secs 2024-07-26T11:34:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1694.563130 secs 2024-07-26T11:34:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1694.563130 secs 2024-07-26T11:34:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:12 smithi160 ceph-mon[36715]: pgmap v1490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:12 smithi183 ceph-mon[37273]: pgmap v1490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:12 smithi144 ceph-mon[32129]: pgmap v1490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:13.112+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:13.113+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:13.113+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:13.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:13.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:14 smithi160 ceph-mon[36715]: pgmap v1491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:14 smithi183 ceph-mon[37273]: pgmap v1491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:14 smithi144 ceph-mon[32129]: pgmap v1491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:16.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:16 smithi144 ceph-mon[32129]: pgmap v1492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:16.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1699.563268 secs 2024-07-26T11:34:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:16 smithi160 ceph-mon[36715]: pgmap v1492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1699.563268 secs 2024-07-26T11:34:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:16 smithi183 ceph-mon[37273]: pgmap v1492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1699.563268 secs 2024-07-26T11:34:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:18 smithi160 ceph-mon[36715]: pgmap v1493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:18 smithi183 ceph-mon[37273]: pgmap v1493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:18 smithi144 ceph-mon[32129]: pgmap v1493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:20 smithi160 ceph-mon[36715]: pgmap v1494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:20 smithi183 ceph-mon[37273]: pgmap v1494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:20 smithi144 ceph-mon[32129]: pgmap v1494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1704.563394 secs 2024-07-26T11:34:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1704.563394 secs 2024-07-26T11:34:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1704.563394 secs 2024-07-26T11:34:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:22 smithi160 ceph-mon[36715]: pgmap v1495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:22 smithi183 ceph-mon[37273]: pgmap v1495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:22 smithi144 ceph-mon[32129]: pgmap v1495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:24 smithi160 ceph-mon[36715]: pgmap v1496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:24 smithi183 ceph-mon[37273]: pgmap v1496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:24 smithi144 ceph-mon[32129]: pgmap v1496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:26 smithi160 ceph-mon[36715]: pgmap v1497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1709.563483 secs 2024-07-26T11:34:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:26 smithi183 ceph-mon[37273]: pgmap v1497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1709.563483 secs 2024-07-26T11:34:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:26 smithi144 ceph-mon[32129]: pgmap v1497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1709.563483 secs 2024-07-26T11:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:28.117+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:28.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:28.119+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:28.119+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:28.119+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:28 smithi160 ceph-mon[36715]: pgmap v1498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:28 smithi183 ceph-mon[37273]: pgmap v1498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:28 smithi144 ceph-mon[32129]: pgmap v1498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:30.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:30 smithi160 ceph-mon[36715]: pgmap v1499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:30 smithi183 ceph-mon[37273]: pgmap v1499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:30 smithi144 ceph-mon[32129]: pgmap v1499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1714.563601 secs 2024-07-26T11:34:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1714.563601 secs 2024-07-26T11:34:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1714.563601 secs 2024-07-26T11:34:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:32 smithi144 ceph-mon[32129]: pgmap v1500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:34:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:32 smithi160 ceph-mon[36715]: pgmap v1500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:32.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:34:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:32 smithi183 ceph-mon[37273]: pgmap v1500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:34:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:33 smithi160 ceph-mon[36715]: pgmap v1501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:34:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:33 smithi160 ceph-mon[36715]: pgmap v1502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:33 smithi183 ceph-mon[37273]: pgmap v1501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:34:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:33 smithi183 ceph-mon[37273]: pgmap v1502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:33 smithi144 ceph-mon[32129]: pgmap v1501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:34:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:33 smithi144 ceph-mon[32129]: pgmap v1502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:35 smithi160 ceph-mon[36715]: pgmap v1503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:35 smithi183 ceph-mon[37273]: pgmap v1503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:35 smithi144 ceph-mon[32129]: pgmap v1503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1719.563748 secs 2024-07-26T11:34:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1719.563748 secs 2024-07-26T11:34:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1719.563748 secs 2024-07-26T11:34:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:37 smithi183 ceph-mon[37273]: pgmap v1504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:37 smithi144 ceph-mon[32129]: pgmap v1504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:37 smithi160 ceph-mon[36715]: pgmap v1504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:39 smithi183 ceph-mon[37273]: pgmap v1505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:39 smithi144 ceph-mon[32129]: pgmap v1505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:39 smithi160 ceph-mon[36715]: pgmap v1505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:41 smithi183 ceph-mon[37273]: pgmap v1506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1724.563875 secs 2024-07-26T11:34:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:41 smithi144 ceph-mon[32129]: pgmap v1506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1724.563875 secs 2024-07-26T11:34:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:41 smithi160 ceph-mon[36715]: pgmap v1506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:34:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1724.563875 secs 2024-07-26T11:34:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:43.134+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:43.135+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:43.135+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:43.135+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:43.135+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:43 smithi183 ceph-mon[37273]: pgmap v1507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:34:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:43 smithi144 ceph-mon[32129]: pgmap v1507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:34:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:43 smithi160 ceph-mon[36715]: pgmap v1507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:34:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:45 smithi183 ceph-mon[37273]: pgmap v1508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:45 smithi144 ceph-mon[32129]: pgmap v1508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:45 smithi160 ceph-mon[36715]: pgmap v1508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:46.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1729.563993 secs 2024-07-26T11:34:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1729.563993 secs 2024-07-26T11:34:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1729.563993 secs 2024-07-26T11:34:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:47 smithi183 ceph-mon[37273]: pgmap v1509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:47 smithi144 ceph-mon[32129]: pgmap v1509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:47 smithi160 ceph-mon[36715]: pgmap v1509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:49 smithi183 ceph-mon[37273]: pgmap v1510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:49 smithi144 ceph-mon[32129]: pgmap v1510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:49 smithi160 ceph-mon[36715]: pgmap v1510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:51 smithi183 ceph-mon[37273]: pgmap v1511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1734.564078 secs 2024-07-26T11:34:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:51 smithi144 ceph-mon[32129]: pgmap v1511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1734.564078 secs 2024-07-26T11:34:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:51 smithi160 ceph-mon[36715]: pgmap v1511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1734.564078 secs 2024-07-26T11:34:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:53 smithi183 ceph-mon[37273]: pgmap v1512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:53 smithi144 ceph-mon[32129]: pgmap v1512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:53 smithi160 ceph-mon[36715]: pgmap v1512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:55 smithi183 ceph-mon[37273]: pgmap v1513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:55 smithi144 ceph-mon[32129]: pgmap v1513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:55 smithi160 ceph-mon[36715]: pgmap v1513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1739.564208 secs 2024-07-26T11:34:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1739.564208 secs 2024-07-26T11:34:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1739.564208 secs 2024-07-26T11:34:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:57 smithi144 ceph-mon[32129]: pgmap v1514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:57 smithi160 ceph-mon[36715]: pgmap v1514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:57 smithi183 ceph-mon[37273]: pgmap v1514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:34:58.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:58.140+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:58.144 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:58.141+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:58.144 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:58.141+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:58.144 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:58.141+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:58.144 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:34:58.141+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:34:59 smithi144 ceph-mon[32129]: pgmap v1515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:34:59 smithi160 ceph-mon[36715]: pgmap v1515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:34:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:34:59 smithi183 ceph-mon[37273]: pgmap v1515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:01.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:01 smithi183 ceph-mon[37273]: pgmap v1516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:01.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1744.564333 secs 2024-07-26T11:35:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:01 smithi144 ceph-mon[32129]: pgmap v1516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1744.564333 secs 2024-07-26T11:35:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:01 smithi160 ceph-mon[36715]: pgmap v1516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1744.564333 secs 2024-07-26T11:35:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:03 smithi144 ceph-mon[32129]: pgmap v1517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:03 smithi160 ceph-mon[36715]: pgmap v1517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:03 smithi183 ceph-mon[37273]: pgmap v1517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:05 smithi144 ceph-mon[32129]: pgmap v1518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:05 smithi160 ceph-mon[36715]: pgmap v1518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:05 smithi183 ceph-mon[37273]: pgmap v1518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:06.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1749.564463 secs 2024-07-26T11:35:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1749.564463 secs 2024-07-26T11:35:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1749.564463 secs 2024-07-26T11:35:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:07 smithi144 ceph-mon[32129]: pgmap v1519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:07 smithi160 ceph-mon[36715]: pgmap v1519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:07 smithi183 ceph-mon[37273]: pgmap v1519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:09 smithi144 ceph-mon[32129]: pgmap v1520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:09 smithi160 ceph-mon[36715]: pgmap v1520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:09 smithi183 ceph-mon[37273]: pgmap v1520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:11 smithi144 ceph-mon[32129]: pgmap v1521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1754.564548 secs 2024-07-26T11:35:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:11 smithi160 ceph-mon[36715]: pgmap v1521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1754.564548 secs 2024-07-26T11:35:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:11 smithi183 ceph-mon[37273]: pgmap v1521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1754.564548 secs 2024-07-26T11:35:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:13.157+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:13.158+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:13.158+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:13.158+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:13.158+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:13 smithi144 ceph-mon[32129]: pgmap v1522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:13 smithi160 ceph-mon[36715]: pgmap v1522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:13 smithi183 ceph-mon[37273]: pgmap v1522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:15 smithi144 ceph-mon[32129]: pgmap v1523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:15 smithi160 ceph-mon[36715]: pgmap v1523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:15 smithi183 ceph-mon[37273]: pgmap v1523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1759.564650 secs 2024-07-26T11:35:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1759.564650 secs 2024-07-26T11:35:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1759.564650 secs 2024-07-26T11:35:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:17 smithi144 ceph-mon[32129]: pgmap v1524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:17 smithi160 ceph-mon[36715]: pgmap v1524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:17 smithi183 ceph-mon[37273]: pgmap v1524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:19 smithi160 ceph-mon[36715]: pgmap v1525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:19 smithi183 ceph-mon[37273]: pgmap v1525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:19 smithi144 ceph-mon[32129]: pgmap v1525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:21 smithi160 ceph-mon[36715]: pgmap v1526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1764.564726 secs 2024-07-26T11:35:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:21 smithi183 ceph-mon[37273]: pgmap v1526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1764.564726 secs 2024-07-26T11:35:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:21 smithi144 ceph-mon[32129]: pgmap v1526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1764.564726 secs 2024-07-26T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:23 smithi160 ceph-mon[36715]: pgmap v1527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:23 smithi183 ceph-mon[37273]: pgmap v1527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:23 smithi144 ceph-mon[32129]: pgmap v1527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:25 smithi160 ceph-mon[36715]: pgmap v1528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:25 smithi183 ceph-mon[37273]: pgmap v1528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:25 smithi144 ceph-mon[32129]: pgmap v1528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1769.564807 secs 2024-07-26T11:35:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1769.564807 secs 2024-07-26T11:35:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1769.564807 secs 2024-07-26T11:35:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:27 smithi160 ceph-mon[36715]: pgmap v1529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:27 smithi183 ceph-mon[37273]: pgmap v1529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:27 smithi144 ceph-mon[32129]: pgmap v1529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:28.170+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:28.171+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:28.171+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:28.171+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:28.171+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:29 smithi160 ceph-mon[36715]: pgmap v1530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:29 smithi183 ceph-mon[37273]: pgmap v1530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:29 smithi144 ceph-mon[32129]: pgmap v1530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:31 smithi160 ceph-mon[36715]: pgmap v1531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1774.564946 secs 2024-07-26T11:35:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:31 smithi183 ceph-mon[37273]: pgmap v1531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1774.564946 secs 2024-07-26T11:35:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:31 smithi144 ceph-mon[32129]: pgmap v1531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1774.564946 secs 2024-07-26T11:35:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:33 smithi160 ceph-mon[36715]: pgmap v1532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:35:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:33 smithi183 ceph-mon[37273]: pgmap v1532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:35:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:33 smithi144 ceph-mon[32129]: pgmap v1532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:35:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:34 smithi160 ceph-mon[36715]: pgmap v1533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:34 smithi160 ceph-mon[36715]: pgmap v1534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:35:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:34 smithi183 ceph-mon[37273]: pgmap v1533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:34 smithi183 ceph-mon[37273]: pgmap v1534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:35:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:34 smithi144 ceph-mon[32129]: pgmap v1533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:34 smithi144 ceph-mon[32129]: pgmap v1534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:35:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:36 smithi160 ceph-mon[36715]: pgmap v1535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1779.565079 secs 2024-07-26T11:35:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:36 smithi183 ceph-mon[37273]: pgmap v1535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1779.565079 secs 2024-07-26T11:35:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:36 smithi144 ceph-mon[32129]: pgmap v1535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1779.565079 secs 2024-07-26T11:35:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:38 smithi160 ceph-mon[36715]: pgmap v1536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:35:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:38 smithi183 ceph-mon[37273]: pgmap v1536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:35:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:38 smithi144 ceph-mon[32129]: pgmap v1536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:35:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:40 smithi160 ceph-mon[36715]: pgmap v1537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:35:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:40 smithi183 ceph-mon[37273]: pgmap v1537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:35:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:40 smithi144 ceph-mon[32129]: pgmap v1537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:35:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1784.565182 secs 2024-07-26T11:35:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1784.565182 secs 2024-07-26T11:35:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1784.565182 secs 2024-07-26T11:35:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:42 smithi183 ceph-mon[37273]: pgmap v1538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:42 smithi144 ceph-mon[32129]: pgmap v1538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:42 smithi160 ceph-mon[36715]: pgmap v1538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:35:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:43.187+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:43.188+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:43.188+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:43.188+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:43.188+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:44 smithi183 ceph-mon[37273]: pgmap v1539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:35:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:44 smithi144 ceph-mon[32129]: pgmap v1539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:35:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:44 smithi160 ceph-mon[36715]: pgmap v1539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c17b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a8046000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c17b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a8046000 comp rx=0 tx=0).stop 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c17b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c17b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c17b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.094+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.095+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.096+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900c1d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f74a8031470 tx=0x7f74a000e650 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:35:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.404+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:35:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.404+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:35:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.405+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T11:35:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:46.405+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T11:35:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:46 smithi183 ceph-mon[37273]: pgmap v1540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1789.565246 secs 2024-07-26T11:35:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:46 smithi144 ceph-mon[32129]: pgmap v1540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1789.565246 secs 2024-07-26T11:35:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:46 smithi160 ceph-mon[36715]: pgmap v1540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1789.565246 secs 2024-07-26T11:35:48.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:48 smithi183 ceph-mon[37273]: pgmap v1541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:48 smithi144 ceph-mon[32129]: pgmap v1541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:48 smithi160 ceph-mon[36715]: pgmap v1541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:50 smithi183 ceph-mon[37273]: pgmap v1542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:50.942 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:50 smithi144 ceph-mon[32129]: pgmap v1542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:50 smithi160 ceph-mon[36715]: pgmap v1542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1794.565369 secs 2024-07-26T11:35:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1794.565369 secs 2024-07-26T11:35:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1794.565369 secs 2024-07-26T11:35:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:52 smithi183 ceph-mon[37273]: pgmap v1543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:52 smithi144 ceph-mon[32129]: pgmap v1543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:52 smithi160 ceph-mon[36715]: pgmap v1543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:35:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:53 smithi183 ceph-mon[37273]: pgmap v1544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:53 smithi144 ceph-mon[32129]: pgmap v1544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:53 smithi160 ceph-mon[36715]: pgmap v1544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:56 smithi160 ceph-mon[36715]: pgmap v1545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1799.565502 secs 2024-07-26T11:35:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:56 smithi183 ceph-mon[37273]: pgmap v1545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1799.565502 secs 2024-07-26T11:35:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:56 smithi144 ceph-mon[32129]: pgmap v1545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:35:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1799.565502 secs 2024-07-26T11:35:58.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:58.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:58.191+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:58.191+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:58.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:35:58.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:35:58 smithi160 ceph-mon[36715]: pgmap v1546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:35:58 smithi183 ceph-mon[37273]: pgmap v1546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:35:58 smithi144 ceph-mon[32129]: pgmap v1546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:00 smithi160 ceph-mon[36715]: pgmap v1547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:00 smithi183 ceph-mon[37273]: pgmap v1547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:00 smithi144 ceph-mon[32129]: pgmap v1547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:01.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1804.565622 secs 2024-07-26T11:36:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1804.565622 secs 2024-07-26T11:36:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1804.565622 secs 2024-07-26T11:36:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:02 smithi160 ceph-mon[36715]: pgmap v1548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:02 smithi183 ceph-mon[37273]: pgmap v1548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:02 smithi144 ceph-mon[32129]: pgmap v1548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:04 smithi160 ceph-mon[36715]: pgmap v1549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:04 smithi183 ceph-mon[37273]: pgmap v1549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:04 smithi144 ceph-mon[32129]: pgmap v1549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:06 smithi144 ceph-mon[32129]: pgmap v1550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1809.565748 secs 2024-07-26T11:36:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:06 smithi160 ceph-mon[36715]: pgmap v1550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1809.565748 secs 2024-07-26T11:36:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:06 smithi183 ceph-mon[37273]: pgmap v1550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1809.565748 secs 2024-07-26T11:36:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:08 smithi160 ceph-mon[36715]: pgmap v1551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:08 smithi183 ceph-mon[37273]: pgmap v1551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:08 smithi144 ceph-mon[32129]: pgmap v1551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:10 smithi183 ceph-mon[37273]: pgmap v1552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:10 smithi144 ceph-mon[32129]: pgmap v1552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:10 smithi160 ceph-mon[36715]: pgmap v1552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1814.565866 secs 2024-07-26T11:36:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1814.565866 secs 2024-07-26T11:36:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1814.565866 secs 2024-07-26T11:36:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:12 smithi183 ceph-mon[37273]: pgmap v1553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:12 smithi144 ceph-mon[32129]: pgmap v1553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:12 smithi160 ceph-mon[36715]: pgmap v1553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:13.208+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:13.209+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:13.209+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:13.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:13.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:14 smithi183 ceph-mon[37273]: pgmap v1554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:14 smithi144 ceph-mon[32129]: pgmap v1554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:14 smithi160 ceph-mon[36715]: pgmap v1554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:16 smithi183 ceph-mon[37273]: pgmap v1555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1819.565935 secs 2024-07-26T11:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:16 smithi144 ceph-mon[32129]: pgmap v1555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1819.565935 secs 2024-07-26T11:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:16 smithi160 ceph-mon[36715]: pgmap v1555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1819.565935 secs 2024-07-26T11:36:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:18 smithi183 ceph-mon[37273]: pgmap v1556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:18 smithi144 ceph-mon[32129]: pgmap v1556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:18 smithi160 ceph-mon[36715]: pgmap v1556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:20 smithi183 ceph-mon[37273]: pgmap v1557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:20 smithi144 ceph-mon[32129]: pgmap v1557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:20 smithi160 ceph-mon[36715]: pgmap v1557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1824.566038 secs 2024-07-26T11:36:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1824.566038 secs 2024-07-26T11:36:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1824.566038 secs 2024-07-26T11:36:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:22 smithi183 ceph-mon[37273]: pgmap v1558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:22 smithi144 ceph-mon[32129]: pgmap v1558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:22 smithi160 ceph-mon[36715]: pgmap v1558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:23 smithi144 ceph-mon[32129]: pgmap v1559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:23 smithi160 ceph-mon[36715]: pgmap v1559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:23 smithi183 ceph-mon[37273]: pgmap v1559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:26 smithi160 ceph-mon[36715]: pgmap v1560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1829.566182 secs 2024-07-26T11:36:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:26 smithi183 ceph-mon[37273]: pgmap v1560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1829.566182 secs 2024-07-26T11:36:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:26 smithi144 ceph-mon[32129]: pgmap v1560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1829.566182 secs 2024-07-26T11:36:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:28.225+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:28.226+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:28.226+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:28.226+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:28.226+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:28 smithi160 ceph-mon[36715]: pgmap v1561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:28 smithi183 ceph-mon[37273]: pgmap v1561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:28 smithi144 ceph-mon[32129]: pgmap v1561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:30 smithi160 ceph-mon[36715]: pgmap v1562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:30 smithi183 ceph-mon[37273]: pgmap v1562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:30 smithi144 ceph-mon[32129]: pgmap v1562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1834.566272 secs 2024-07-26T11:36:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1834.566272 secs 2024-07-26T11:36:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1834.566272 secs 2024-07-26T11:36:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:32 smithi160 ceph-mon[36715]: pgmap v1563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:32 smithi183 ceph-mon[37273]: pgmap v1563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:32 smithi144 ceph-mon[32129]: pgmap v1563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:34 smithi183 ceph-mon[37273]: pgmap v1564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:34 smithi144 ceph-mon[32129]: pgmap v1564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:34 smithi160 ceph-mon[36715]: pgmap v1564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:36:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:35.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:35.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:35.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:36:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:36.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:36:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:36 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:36:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:36 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:36:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:36 smithi183 ceph-mon[37273]: pgmap v1565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:36:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:36 smithi183 ceph-mon[37273]: pgmap v1566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:36:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1839.566381 secs 2024-07-26T11:36:36.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:36 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:36:36.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:36 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:36:36.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:36 smithi160 ceph-mon[36715]: pgmap v1565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:36:36.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:36 smithi160 ceph-mon[36715]: pgmap v1566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:36:36.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1839.566381 secs 2024-07-26T11:36:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:36 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:36:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:36 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:36:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:36 smithi144 ceph-mon[32129]: pgmap v1565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:36:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:36 smithi144 ceph-mon[32129]: pgmap v1566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:36:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1839.566381 secs 2024-07-26T11:36:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:38 smithi183 ceph-mon[37273]: pgmap v1567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:36:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:38 smithi144 ceph-mon[32129]: pgmap v1567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:36:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:38 smithi160 ceph-mon[36715]: pgmap v1567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:36:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:40 smithi183 ceph-mon[37273]: pgmap v1568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:36:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:40 smithi144 ceph-mon[32129]: pgmap v1568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:36:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:40 smithi160 ceph-mon[36715]: pgmap v1568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:36:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1844.566502 secs 2024-07-26T11:36:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1844.566502 secs 2024-07-26T11:36:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1844.566502 secs 2024-07-26T11:36:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:42 smithi183 ceph-mon[37273]: pgmap v1569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:36:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:42 smithi144 ceph-mon[32129]: pgmap v1569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:36:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:42 smithi160 ceph-mon[36715]: pgmap v1569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:36:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:43.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:43.242+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:43.242+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:43.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:43.243+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:43 smithi144 ceph-mon[32129]: pgmap v1570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:36:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:43 smithi160 ceph-mon[36715]: pgmap v1570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:36:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:43 smithi183 ceph-mon[37273]: pgmap v1570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:36:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:45 smithi160 ceph-mon[36715]: pgmap v1571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:36:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:45 smithi183 ceph-mon[37273]: pgmap v1571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:36:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:45 smithi144 ceph-mon[32129]: pgmap v1571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:36:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1849.566624 secs 2024-07-26T11:36:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1849.566624 secs 2024-07-26T11:36:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1849.566624 secs 2024-07-26T11:36:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:47 smithi183 ceph-mon[37273]: pgmap v1572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:47 smithi144 ceph-mon[32129]: pgmap v1572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:47 smithi160 ceph-mon[36715]: pgmap v1572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:49 smithi183 ceph-mon[37273]: pgmap v1573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:49 smithi144 ceph-mon[32129]: pgmap v1573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:49 smithi160 ceph-mon[36715]: pgmap v1573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1854.566738 secs 2024-07-26T11:36:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:51 smithi183 ceph-mon[37273]: pgmap v1574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1854.566738 secs 2024-07-26T11:36:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:51 smithi144 ceph-mon[32129]: pgmap v1574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1854.566738 secs 2024-07-26T11:36:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:51 smithi160 ceph-mon[36715]: pgmap v1574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:53 smithi183 ceph-mon[37273]: pgmap v1575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:53 smithi144 ceph-mon[32129]: pgmap v1575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:53 smithi160 ceph-mon[36715]: pgmap v1575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:55 smithi183 ceph-mon[37273]: pgmap v1576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:55 smithi144 ceph-mon[32129]: pgmap v1576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:55 smithi160 ceph-mon[36715]: pgmap v1576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:36:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1859.566859 secs 2024-07-26T11:36:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1859.566859 secs 2024-07-26T11:36:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1859.566859 secs 2024-07-26T11:36:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:57 smithi183 ceph-mon[37273]: pgmap v1577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:57 smithi144 ceph-mon[32129]: pgmap v1577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:36:58.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:58.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:58.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:58.258+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:58.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:58.258+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:58.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:58.258+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:58.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:36:58.258+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:57 smithi160 ceph-mon[36715]: pgmap v1577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:36:59 smithi183 ceph-mon[37273]: pgmap v1578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:36:59 smithi144 ceph-mon[32129]: pgmap v1578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:36:59 smithi160 ceph-mon[36715]: pgmap v1578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1864.566979 secs 2024-07-26T11:37:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:01 smithi183 ceph-mon[37273]: pgmap v1579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1864.566979 secs 2024-07-26T11:37:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:01 smithi144 ceph-mon[32129]: pgmap v1579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1864.566979 secs 2024-07-26T11:37:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:01 smithi160 ceph-mon[36715]: pgmap v1579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:03 smithi183 ceph-mon[37273]: pgmap v1580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:03 smithi144 ceph-mon[32129]: pgmap v1580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:03 smithi160 ceph-mon[36715]: pgmap v1580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:05 smithi183 ceph-mon[37273]: pgmap v1581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:05 smithi144 ceph-mon[32129]: pgmap v1581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:05 smithi160 ceph-mon[36715]: pgmap v1581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1869.567140 secs 2024-07-26T11:37:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1869.567140 secs 2024-07-26T11:37:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1869.567140 secs 2024-07-26T11:37:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:07 smithi144 ceph-mon[32129]: pgmap v1582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:07 smithi160 ceph-mon[36715]: pgmap v1582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:07 smithi183 ceph-mon[37273]: pgmap v1582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:09 smithi144 ceph-mon[32129]: pgmap v1583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:09 smithi160 ceph-mon[36715]: pgmap v1583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:09 smithi183 ceph-mon[37273]: pgmap v1583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1874.567255 secs 2024-07-26T11:37:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1874.567255 secs 2024-07-26T11:37:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1874.567255 secs 2024-07-26T11:37:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:11 smithi144 ceph-mon[32129]: pgmap v1584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:11 smithi160 ceph-mon[36715]: pgmap v1584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:12.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:11 smithi183 ceph-mon[37273]: pgmap v1584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:13.275+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:13.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:13.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:13.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:13.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:14.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:13 smithi144 ceph-mon[32129]: pgmap v1585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:13 smithi160 ceph-mon[36715]: pgmap v1585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:13 smithi183 ceph-mon[37273]: pgmap v1585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:15 smithi144 ceph-mon[32129]: pgmap v1586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1879.567378 secs 2024-07-26T11:37:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:15 smithi160 ceph-mon[36715]: pgmap v1586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1879.567378 secs 2024-07-26T11:37:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:15 smithi183 ceph-mon[37273]: pgmap v1586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1879.567378 secs 2024-07-26T11:37:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:17 smithi144 ceph-mon[32129]: pgmap v1587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:17 smithi160 ceph-mon[36715]: pgmap v1587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:17 smithi183 ceph-mon[37273]: pgmap v1587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:19 smithi144 ceph-mon[32129]: pgmap v1588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:19 smithi160 ceph-mon[36715]: pgmap v1588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:19 smithi183 ceph-mon[37273]: pgmap v1588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1884.567494 secs 2024-07-26T11:37:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1884.567494 secs 2024-07-26T11:37:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1884.567494 secs 2024-07-26T11:37:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:21 smithi144 ceph-mon[32129]: pgmap v1589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:21 smithi160 ceph-mon[36715]: pgmap v1589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:21 smithi183 ceph-mon[37273]: pgmap v1589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:23 smithi144 ceph-mon[32129]: pgmap v1590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:23 smithi160 ceph-mon[36715]: pgmap v1590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:23 smithi183 ceph-mon[37273]: pgmap v1590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:25 smithi144 ceph-mon[32129]: pgmap v1591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1889.567616 secs 2024-07-26T11:37:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:25 smithi160 ceph-mon[36715]: pgmap v1591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1889.567616 secs 2024-07-26T11:37:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:25 smithi183 ceph-mon[37273]: pgmap v1591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1889.567616 secs 2024-07-26T11:37:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:28.292+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:28.293+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:28.293+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:28.293+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:28.293+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:27 smithi160 ceph-mon[36715]: pgmap v1592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:27 smithi183 ceph-mon[37273]: pgmap v1592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:27 smithi144 ceph-mon[32129]: pgmap v1592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:29 smithi160 ceph-mon[36715]: pgmap v1593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:29 smithi183 ceph-mon[37273]: pgmap v1593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:29 smithi144 ceph-mon[32129]: pgmap v1593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:31.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1894.567744 secs 2024-07-26T11:37:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1894.567744 secs 2024-07-26T11:37:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1894.567744 secs 2024-07-26T11:37:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:31 smithi160 ceph-mon[36715]: pgmap v1594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:31 smithi183 ceph-mon[37273]: pgmap v1594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:31 smithi144 ceph-mon[32129]: pgmap v1594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:34 smithi160 ceph-mon[36715]: pgmap v1595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:34 smithi183 ceph-mon[37273]: pgmap v1595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:34 smithi144 ceph-mon[32129]: pgmap v1595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: pgmap v1596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1899.567821 secs 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:36.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: pgmap v1596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1899.567821 secs 2024-07-26T11:37:36.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:36.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:36.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: pgmap v1596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1899.567821 secs 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:36.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:37:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:37 smithi160 ceph-mon[36715]: pgmap v1597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:37 smithi160 ceph-mon[36715]: pgmap v1598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:37 smithi183 ceph-mon[37273]: pgmap v1597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:37 smithi183 ceph-mon[37273]: pgmap v1598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:37 smithi144 ceph-mon[32129]: pgmap v1597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:37 smithi144 ceph-mon[32129]: pgmap v1598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:39 smithi160 ceph-mon[36715]: pgmap v1599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:37:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:39 smithi183 ceph-mon[37273]: pgmap v1599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:37:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:39 smithi144 ceph-mon[32129]: pgmap v1599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:37:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:41 smithi160 ceph-mon[36715]: pgmap v1600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:37:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1904.567943 secs 2024-07-26T11:37:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:41 smithi183 ceph-mon[37273]: pgmap v1600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:37:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1904.567943 secs 2024-07-26T11:37:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:41 smithi144 ceph-mon[32129]: pgmap v1600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:37:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1904.567943 secs 2024-07-26T11:37:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:43.298+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:43.299+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:43.299+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:43.299+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:43.299+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:43 smithi160 ceph-mon[36715]: pgmap v1601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:43 smithi183 ceph-mon[37273]: pgmap v1601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:43 smithi144 ceph-mon[32129]: pgmap v1601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:37:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:45 smithi160 ceph-mon[36715]: pgmap v1602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:37:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:45 smithi183 ceph-mon[37273]: pgmap v1602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:37:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:45 smithi144 ceph-mon[32129]: pgmap v1602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:37:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1909.568042 secs 2024-07-26T11:37:46.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1909.568042 secs 2024-07-26T11:37:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1909.568042 secs 2024-07-26T11:37:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:47 smithi160 ceph-mon[36715]: pgmap v1603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:37:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:47 smithi183 ceph-mon[37273]: pgmap v1603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:37:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:47 smithi144 ceph-mon[32129]: pgmap v1603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:37:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:49 smithi183 ceph-mon[37273]: pgmap v1604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:49 smithi144 ceph-mon[32129]: pgmap v1604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:49 smithi160 ceph-mon[36715]: pgmap v1604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:51 smithi183 ceph-mon[37273]: pgmap v1605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1914.568167 secs 2024-07-26T11:37:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:51 smithi144 ceph-mon[32129]: pgmap v1605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1914.568167 secs 2024-07-26T11:37:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:51 smithi160 ceph-mon[36715]: pgmap v1605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1914.568167 secs 2024-07-26T11:37:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:53 smithi183 ceph-mon[37273]: pgmap v1606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:53 smithi144 ceph-mon[32129]: pgmap v1606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:53 smithi160 ceph-mon[36715]: pgmap v1606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:37:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:55 smithi144 ceph-mon[32129]: pgmap v1607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:55 smithi183 ceph-mon[37273]: pgmap v1607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:55 smithi160 ceph-mon[36715]: pgmap v1607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1919.568307 secs 2024-07-26T11:37:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1919.568307 secs 2024-07-26T11:37:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1919.568307 secs 2024-07-26T11:37:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:57 smithi183 ceph-mon[37273]: pgmap v1608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:57 smithi144 ceph-mon[32129]: pgmap v1608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:57 smithi160 ceph-mon[36715]: pgmap v1608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:37:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:58.314+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:58.316+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:58.316+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:58.316+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:37:58.317+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:37:59 smithi183 ceph-mon[37273]: pgmap v1609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:37:59 smithi144 ceph-mon[32129]: pgmap v1609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:37:59 smithi160 ceph-mon[36715]: pgmap v1609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:01 smithi183 ceph-mon[37273]: pgmap v1610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1924.568430 secs 2024-07-26T11:38:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:01 smithi144 ceph-mon[32129]: pgmap v1610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1924.568430 secs 2024-07-26T11:38:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:01 smithi160 ceph-mon[36715]: pgmap v1610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1924.568430 secs 2024-07-26T11:38:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:03 smithi183 ceph-mon[37273]: pgmap v1611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:03 smithi144 ceph-mon[32129]: pgmap v1611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:03 smithi160 ceph-mon[36715]: pgmap v1611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:05 smithi183 ceph-mon[37273]: pgmap v1612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:05 smithi144 ceph-mon[32129]: pgmap v1612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:05 smithi160 ceph-mon[36715]: pgmap v1612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1929.568554 secs 2024-07-26T11:38:06.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1929.568554 secs 2024-07-26T11:38:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1929.568554 secs 2024-07-26T11:38:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:07 smithi183 ceph-mon[37273]: pgmap v1613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:07 smithi144 ceph-mon[32129]: pgmap v1613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:07 smithi160 ceph-mon[36715]: pgmap v1613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:09 smithi144 ceph-mon[32129]: pgmap v1614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:09 smithi160 ceph-mon[36715]: pgmap v1614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:09 smithi183 ceph-mon[37273]: pgmap v1614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:11 smithi144 ceph-mon[32129]: pgmap v1615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1934.568617 secs 2024-07-26T11:38:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:11 smithi160 ceph-mon[36715]: pgmap v1615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1934.568617 secs 2024-07-26T11:38:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:11 smithi183 ceph-mon[37273]: pgmap v1615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1934.568617 secs 2024-07-26T11:38:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:13.332+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:13.333+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:13.333+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:13.333+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:13.333+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:13 smithi144 ceph-mon[32129]: pgmap v1616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:13 smithi160 ceph-mon[36715]: pgmap v1616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:13 smithi183 ceph-mon[37273]: pgmap v1616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:15 smithi144 ceph-mon[32129]: pgmap v1617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:15 smithi160 ceph-mon[36715]: pgmap v1617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:15 smithi183 ceph-mon[37273]: pgmap v1617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1939.568685 secs 2024-07-26T11:38:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1939.568685 secs 2024-07-26T11:38:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1939.568685 secs 2024-07-26T11:38:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:17 smithi144 ceph-mon[32129]: pgmap v1618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:17 smithi160 ceph-mon[36715]: pgmap v1618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:17 smithi183 ceph-mon[37273]: pgmap v1618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:19 smithi144 ceph-mon[32129]: pgmap v1619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:19 smithi160 ceph-mon[36715]: pgmap v1619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:19 smithi183 ceph-mon[37273]: pgmap v1619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:21 smithi144 ceph-mon[32129]: pgmap v1620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1944.568767 secs 2024-07-26T11:38:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:21 smithi160 ceph-mon[36715]: pgmap v1620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1944.568767 secs 2024-07-26T11:38:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:21 smithi183 ceph-mon[37273]: pgmap v1620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1944.568767 secs 2024-07-26T11:38:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:23 smithi144 ceph-mon[32129]: pgmap v1621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:23 smithi160 ceph-mon[36715]: pgmap v1621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:23 smithi183 ceph-mon[37273]: pgmap v1621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:25 smithi144 ceph-mon[32129]: pgmap v1622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:25 smithi160 ceph-mon[36715]: pgmap v1622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:25 smithi183 ceph-mon[37273]: pgmap v1622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:26.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1949.568846 secs 2024-07-26T11:38:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1949.568846 secs 2024-07-26T11:38:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1949.568846 secs 2024-07-26T11:38:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:27 smithi144 ceph-mon[32129]: pgmap v1623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:27 smithi160 ceph-mon[36715]: pgmap v1623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:27 smithi183 ceph-mon[37273]: pgmap v1623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:28.341+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:28.342+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:28.342+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:28.342+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:28.343+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:29 smithi144 ceph-mon[32129]: pgmap v1624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:29 smithi160 ceph-mon[36715]: pgmap v1624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:29 smithi183 ceph-mon[37273]: pgmap v1624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:31 smithi144 ceph-mon[32129]: pgmap v1625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1954.568981 secs 2024-07-26T11:38:31.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:31 smithi160 ceph-mon[36715]: pgmap v1625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:31.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1954.568981 secs 2024-07-26T11:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:31 smithi183 ceph-mon[37273]: pgmap v1625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1954.568981 secs 2024-07-26T11:38:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:33 smithi144 ceph-mon[32129]: pgmap v1626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:33 smithi160 ceph-mon[36715]: pgmap v1626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:33 smithi183 ceph-mon[37273]: pgmap v1626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:35 smithi160 ceph-mon[36715]: pgmap v1627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:35 smithi183 ceph-mon[37273]: pgmap v1627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:35 smithi144 ceph-mon[32129]: pgmap v1627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1959.569045 secs 2024-07-26T11:38:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1959.569045 secs 2024-07-26T11:38:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1959.569045 secs 2024-07-26T11:38:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:37 smithi160 ceph-mon[36715]: pgmap v1628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:38:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:37 smithi183 ceph-mon[37273]: pgmap v1628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:38:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:37 smithi144 ceph-mon[32129]: pgmap v1628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:38:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:38 smithi160 ceph-mon[36715]: pgmap v1629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:38:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:38 smithi160 ceph-mon[36715]: pgmap v1630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:38 smithi183 ceph-mon[37273]: pgmap v1629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:38:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:38 smithi183 ceph-mon[37273]: pgmap v1630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:38 smithi144 ceph-mon[32129]: pgmap v1629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:38:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:38 smithi144 ceph-mon[32129]: pgmap v1630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:40 smithi160 ceph-mon[36715]: pgmap v1631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:40 smithi183 ceph-mon[37273]: pgmap v1631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:40 smithi144 ceph-mon[32129]: pgmap v1631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1964.569161 secs 2024-07-26T11:38:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1964.569161 secs 2024-07-26T11:38:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1964.569161 secs 2024-07-26T11:38:42.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:42 smithi160 ceph-mon[36715]: pgmap v1632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:42 smithi183 ceph-mon[37273]: pgmap v1632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:42 smithi144 ceph-mon[32129]: pgmap v1632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:38:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:43.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:43.351+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:43.351+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:43.351+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:43.351+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:44 smithi160 ceph-mon[36715]: pgmap v1633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:38:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:44 smithi183 ceph-mon[37273]: pgmap v1633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:38:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:44 smithi144 ceph-mon[32129]: pgmap v1633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:38:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.437+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T11:38:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.437+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780062d0 con 0x563c6c4182f0 2024-07-26T11:38:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.437+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T11:38:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.437+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:38:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.437+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T11:38:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.437+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T11:38:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.438+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T11:38:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:46.438+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T11:38:46.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:46 smithi144 ceph-mon[32129]: pgmap v1634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:38:46.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1969.569347 secs 2024-07-26T11:38:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:46 smithi160 ceph-mon[36715]: pgmap v1634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:38:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1969.569347 secs 2024-07-26T11:38:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:46 smithi183 ceph-mon[37273]: pgmap v1634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:38:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1969.569347 secs 2024-07-26T11:38:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:48 smithi160 ceph-mon[36715]: pgmap v1635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:38:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:48 smithi183 ceph-mon[37273]: pgmap v1635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:38:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:48 smithi144 ceph-mon[32129]: pgmap v1635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:38:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:50 smithi160 ceph-mon[36715]: pgmap v1636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:50 smithi183 ceph-mon[37273]: pgmap v1636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:50 smithi144 ceph-mon[32129]: pgmap v1636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1974.569405 secs 2024-07-26T11:38:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1974.569405 secs 2024-07-26T11:38:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1974.569405 secs 2024-07-26T11:38:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:52 smithi144 ceph-mon[32129]: pgmap v1637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:52 smithi160 ceph-mon[36715]: pgmap v1637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:52 smithi183 ceph-mon[37273]: pgmap v1637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:54 smithi144 ceph-mon[32129]: pgmap v1638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:54 smithi160 ceph-mon[36715]: pgmap v1638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:54 smithi183 ceph-mon[37273]: pgmap v1638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:56 smithi144 ceph-mon[32129]: pgmap v1639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1979.569471 secs 2024-07-26T11:38:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:56 smithi160 ceph-mon[36715]: pgmap v1639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1979.569471 secs 2024-07-26T11:38:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:56 smithi183 ceph-mon[37273]: pgmap v1639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:38:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1979.569471 secs 2024-07-26T11:38:58.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:58.367+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:58.367+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:58.367+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:58.367+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:38:58.368+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:38:58 smithi144 ceph-mon[32129]: pgmap v1640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:38:58 smithi160 ceph-mon[36715]: pgmap v1640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:38:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:38:58 smithi183 ceph-mon[37273]: pgmap v1640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:00 smithi144 ceph-mon[32129]: pgmap v1641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:00 smithi160 ceph-mon[36715]: pgmap v1641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:00 smithi183 ceph-mon[37273]: pgmap v1641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1984.569586 secs 2024-07-26T11:39:01.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1984.569586 secs 2024-07-26T11:39:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1984.569586 secs 2024-07-26T11:39:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:02 smithi144 ceph-mon[32129]: pgmap v1642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:02 smithi160 ceph-mon[36715]: pgmap v1642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:02 smithi183 ceph-mon[37273]: pgmap v1642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:04 smithi144 ceph-mon[32129]: pgmap v1643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:04 smithi160 ceph-mon[36715]: pgmap v1643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:04 smithi183 ceph-mon[37273]: pgmap v1643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:06 smithi144 ceph-mon[32129]: pgmap v1644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1989.569708 secs 2024-07-26T11:39:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:06 smithi160 ceph-mon[36715]: pgmap v1644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1989.569708 secs 2024-07-26T11:39:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:06 smithi183 ceph-mon[37273]: pgmap v1644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1989.569708 secs 2024-07-26T11:39:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:08 smithi144 ceph-mon[32129]: pgmap v1645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:08 smithi160 ceph-mon[36715]: pgmap v1645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:08 smithi183 ceph-mon[37273]: pgmap v1645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:10 smithi144 ceph-mon[32129]: pgmap v1646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:10 smithi160 ceph-mon[36715]: pgmap v1646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:10 smithi183 ceph-mon[37273]: pgmap v1646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1994.569827 secs 2024-07-26T11:39:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1994.569827 secs 2024-07-26T11:39:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1994.569827 secs 2024-07-26T11:39:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:12 smithi144 ceph-mon[32129]: pgmap v1647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:12 smithi160 ceph-mon[36715]: pgmap v1647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:12 smithi183 ceph-mon[37273]: pgmap v1647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:13.385 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:13.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:13.384+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:13.384+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:13.384+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:13.385+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:14 smithi144 ceph-mon[32129]: pgmap v1648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:14 smithi160 ceph-mon[36715]: pgmap v1648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:14 smithi183 ceph-mon[37273]: pgmap v1648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:16 smithi160 ceph-mon[36715]: pgmap v1649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 1999.569910 secs 2024-07-26T11:39:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:16 smithi183 ceph-mon[37273]: pgmap v1649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 1999.569910 secs 2024-07-26T11:39:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:16 smithi144 ceph-mon[32129]: pgmap v1649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 1999.569910 secs 2024-07-26T11:39:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:18 smithi160 ceph-mon[36715]: pgmap v1650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:18 smithi183 ceph-mon[37273]: pgmap v1650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:18 smithi144 ceph-mon[32129]: pgmap v1650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:20 smithi160 ceph-mon[36715]: pgmap v1651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:20 smithi183 ceph-mon[37273]: pgmap v1651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:20 smithi144 ceph-mon[32129]: pgmap v1651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2004.570029 secs 2024-07-26T11:39:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2004.570029 secs 2024-07-26T11:39:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2004.570029 secs 2024-07-26T11:39:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:22 smithi160 ceph-mon[36715]: pgmap v1652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:22 smithi183 ceph-mon[37273]: pgmap v1652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:22 smithi144 ceph-mon[32129]: pgmap v1652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:24 smithi160 ceph-mon[36715]: pgmap v1653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:24 smithi183 ceph-mon[37273]: pgmap v1653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:24 smithi144 ceph-mon[32129]: pgmap v1653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:26 smithi160 ceph-mon[36715]: pgmap v1654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2009.570154 secs 2024-07-26T11:39:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:26 smithi183 ceph-mon[37273]: pgmap v1654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2009.570154 secs 2024-07-26T11:39:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:26 smithi144 ceph-mon[32129]: pgmap v1654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2009.570154 secs 2024-07-26T11:39:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:28.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:28.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:28.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:28.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:28.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:28 smithi160 ceph-mon[36715]: pgmap v1655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:28 smithi183 ceph-mon[37273]: pgmap v1655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:28 smithi144 ceph-mon[32129]: pgmap v1655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:30 smithi160 ceph-mon[36715]: pgmap v1656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:30 smithi183 ceph-mon[37273]: pgmap v1656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:30 smithi144 ceph-mon[32129]: pgmap v1656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:31.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2014.570281 secs 2024-07-26T11:39:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2014.570281 secs 2024-07-26T11:39:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2014.570281 secs 2024-07-26T11:39:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:32 smithi160 ceph-mon[36715]: pgmap v1657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:32 smithi183 ceph-mon[37273]: pgmap v1657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:32 smithi144 ceph-mon[32129]: pgmap v1657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:34 smithi160 ceph-mon[36715]: pgmap v1658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:34 smithi183 ceph-mon[37273]: pgmap v1658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:34 smithi144 ceph-mon[32129]: pgmap v1658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:36 smithi160 ceph-mon[36715]: pgmap v1659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2019.570395 secs 2024-07-26T11:39:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:36 smithi183 ceph-mon[37273]: pgmap v1659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2019.570395 secs 2024-07-26T11:39:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:36 smithi144 ceph-mon[32129]: pgmap v1659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2019.570395 secs 2024-07-26T11:39:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:38 smithi160 ceph-mon[36715]: pgmap v1660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:38 smithi183 ceph-mon[37273]: pgmap v1660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:38 smithi144 ceph-mon[32129]: pgmap v1660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:39:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:40 smithi183 ceph-mon[37273]: pgmap v1661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:39:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:40 smithi183 ceph-mon[37273]: pgmap v1662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:40 smithi144 ceph-mon[32129]: pgmap v1661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:39:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:40 smithi144 ceph-mon[32129]: pgmap v1662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:40 smithi160 ceph-mon[36715]: pgmap v1661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:39:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:40 smithi160 ceph-mon[36715]: pgmap v1662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2024.570492 secs 2024-07-26T11:39:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2024.570492 secs 2024-07-26T11:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2024.570492 secs 2024-07-26T11:39:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:42 smithi183 ceph-mon[37273]: pgmap v1663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:42 smithi144 ceph-mon[32129]: pgmap v1663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:42 smithi160 ceph-mon[36715]: pgmap v1663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:43.399 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:43.397+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:43.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:43.398+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:43.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:43.398+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:43.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:43.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:43.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:43.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:44 smithi183 ceph-mon[37273]: pgmap v1664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:44.926 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:44 smithi144 ceph-mon[32129]: pgmap v1664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:44 smithi160 ceph-mon[36715]: pgmap v1664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:46 smithi183 ceph-mon[37273]: pgmap v1665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2029.570593 secs 2024-07-26T11:39:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:46 smithi144 ceph-mon[32129]: pgmap v1665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2029.570593 secs 2024-07-26T11:39:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:46 smithi160 ceph-mon[36715]: pgmap v1665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2029.570593 secs 2024-07-26T11:39:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:48 smithi183 ceph-mon[37273]: pgmap v1666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:48 smithi144 ceph-mon[32129]: pgmap v1666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:48 smithi160 ceph-mon[36715]: pgmap v1666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:39:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:50 smithi183 ceph-mon[37273]: pgmap v1667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:39:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:50 smithi144 ceph-mon[32129]: pgmap v1667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:39:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:50 smithi160 ceph-mon[36715]: pgmap v1667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:39:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2034.570675 secs 2024-07-26T11:39:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2034.570675 secs 2024-07-26T11:39:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2034.570675 secs 2024-07-26T11:39:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:52 smithi183 ceph-mon[37273]: pgmap v1668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:52 smithi144 ceph-mon[32129]: pgmap v1668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:52 smithi160 ceph-mon[36715]: pgmap v1668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:54 smithi183 ceph-mon[37273]: pgmap v1669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:54 smithi144 ceph-mon[32129]: pgmap v1669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:54 smithi160 ceph-mon[36715]: pgmap v1669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:56 smithi183 ceph-mon[37273]: pgmap v1670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2039.570799 secs 2024-07-26T11:39:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:56 smithi144 ceph-mon[32129]: pgmap v1670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2039.570799 secs 2024-07-26T11:39:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:56 smithi160 ceph-mon[36715]: pgmap v1670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:39:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2039.570799 secs 2024-07-26T11:39:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:58.414+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:58.415+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:58.415+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:58.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:39:58.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:39:58 smithi144 ceph-mon[32129]: pgmap v1671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:39:58 smithi160 ceph-mon[36715]: pgmap v1671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:39:58 smithi183 ceph-mon[37273]: pgmap v1671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:00 smithi144 ceph-mon[32129]: pgmap v1672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:00 smithi160 ceph-mon[36715]: pgmap v1672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:00 smithi183 ceph-mon[37273]: pgmap v1672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2044.570935 secs 2024-07-26T11:40:02.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2044.570935 secs 2024-07-26T11:40:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2044.570935 secs 2024-07-26T11:40:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:02 smithi144 ceph-mon[32129]: pgmap v1673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:02 smithi160 ceph-mon[36715]: pgmap v1673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:02 smithi183 ceph-mon[37273]: pgmap v1673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:03 smithi144 ceph-mon[32129]: pgmap v1674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:03 smithi160 ceph-mon[36715]: pgmap v1674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:03 smithi183 ceph-mon[37273]: pgmap v1674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:06 smithi183 ceph-mon[37273]: pgmap v1675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2049.571054 secs 2024-07-26T11:40:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:06 smithi144 ceph-mon[32129]: pgmap v1675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2049.571054 secs 2024-07-26T11:40:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:06 smithi160 ceph-mon[36715]: pgmap v1675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2049.571054 secs 2024-07-26T11:40:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:08 smithi183 ceph-mon[37273]: pgmap v1676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:08 smithi144 ceph-mon[32129]: pgmap v1676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:08 smithi160 ceph-mon[36715]: pgmap v1676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:10 smithi183 ceph-mon[37273]: pgmap v1677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:10 smithi144 ceph-mon[32129]: pgmap v1677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:10 smithi160 ceph-mon[36715]: pgmap v1677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2054.571192 secs 2024-07-26T11:40:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2054.571192 secs 2024-07-26T11:40:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2054.571192 secs 2024-07-26T11:40:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:12 smithi183 ceph-mon[37273]: pgmap v1678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:12 smithi144 ceph-mon[32129]: pgmap v1678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:12 smithi160 ceph-mon[36715]: pgmap v1678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:13.431+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:13.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:13.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:13.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:13.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:13.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:13.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:14 smithi144 ceph-mon[32129]: pgmap v1679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:14 smithi160 ceph-mon[36715]: pgmap v1679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:14 smithi183 ceph-mon[37273]: pgmap v1679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:16 smithi183 ceph-mon[37273]: pgmap v1680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2059.571342 secs 2024-07-26T11:40:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:16 smithi144 ceph-mon[32129]: pgmap v1680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2059.571342 secs 2024-07-26T11:40:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:16 smithi160 ceph-mon[36715]: pgmap v1680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2059.571342 secs 2024-07-26T11:40:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:18 smithi144 ceph-mon[32129]: pgmap v1681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:18 smithi160 ceph-mon[36715]: pgmap v1681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:18 smithi183 ceph-mon[37273]: pgmap v1681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:20 smithi144 ceph-mon[32129]: pgmap v1682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:20 smithi160 ceph-mon[36715]: pgmap v1682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:20 smithi183 ceph-mon[37273]: pgmap v1682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2064.571464 secs 2024-07-26T11:40:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2064.571464 secs 2024-07-26T11:40:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2064.571464 secs 2024-07-26T11:40:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:22 smithi144 ceph-mon[32129]: pgmap v1683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:22 smithi160 ceph-mon[36715]: pgmap v1683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:22 smithi183 ceph-mon[37273]: pgmap v1683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:24 smithi144 ceph-mon[32129]: pgmap v1684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:24 smithi160 ceph-mon[36715]: pgmap v1684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:24 smithi183 ceph-mon[37273]: pgmap v1684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:26 smithi144 ceph-mon[32129]: pgmap v1685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2069.571574 secs 2024-07-26T11:40:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:26 smithi160 ceph-mon[36715]: pgmap v1685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2069.571574 secs 2024-07-26T11:40:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:26 smithi183 ceph-mon[37273]: pgmap v1685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2069.571574 secs 2024-07-26T11:40:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:28.437+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:28.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:28.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:28.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:28.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:28 smithi144 ceph-mon[32129]: pgmap v1686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:28 smithi160 ceph-mon[36715]: pgmap v1686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:28 smithi183 ceph-mon[37273]: pgmap v1686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:30 smithi144 ceph-mon[32129]: pgmap v1687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:30 smithi160 ceph-mon[36715]: pgmap v1687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:30 smithi183 ceph-mon[37273]: pgmap v1687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2074.571701 secs 2024-07-26T11:40:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2074.571701 secs 2024-07-26T11:40:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2074.571701 secs 2024-07-26T11:40:32.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:32 smithi144 ceph-mon[32129]: pgmap v1688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:32.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:32 smithi183 ceph-mon[37273]: pgmap v1688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:32 smithi160 ceph-mon[36715]: pgmap v1688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:34 smithi144 ceph-mon[32129]: pgmap v1689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:34 smithi160 ceph-mon[36715]: pgmap v1689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:34 smithi183 ceph-mon[37273]: pgmap v1689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:36 smithi144 ceph-mon[32129]: pgmap v1690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2079.571845 secs 2024-07-26T11:40:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:36 smithi160 ceph-mon[36715]: pgmap v1690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2079.571845 secs 2024-07-26T11:40:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:36 smithi183 ceph-mon[37273]: pgmap v1690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2079.571845 secs 2024-07-26T11:40:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:38 smithi144 ceph-mon[32129]: pgmap v1691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:38 smithi160 ceph-mon[36715]: pgmap v1691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:38 smithi183 ceph-mon[37273]: pgmap v1691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:40.524 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:40 smithi183 ceph-mon[37273]: pgmap v1692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:40.525 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:40.525 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.525 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.525 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.525 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:40 smithi144 ceph-mon[32129]: pgmap v1692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:40.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:40.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.553 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:40 smithi160 ceph-mon[36715]: pgmap v1692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:40:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2084.571959 secs 2024-07-26T11:40:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2084.571959 secs 2024-07-26T11:40:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2084.571959 secs 2024-07-26T11:40:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:42 smithi160 ceph-mon[36715]: pgmap v1693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:42 smithi183 ceph-mon[37273]: pgmap v1693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:42 smithi144 ceph-mon[32129]: pgmap v1693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:43.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:43.455+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:43.455+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:43.455+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:43.455+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:44 smithi183 ceph-mon[37273]: pgmap v1694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:44 smithi144 ceph-mon[32129]: pgmap v1694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:44 smithi160 ceph-mon[36715]: pgmap v1694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:45 smithi144 ceph-mon[32129]: pgmap v1695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:45 smithi144 ceph-mon[32129]: pgmap v1696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:45 smithi160 ceph-mon[36715]: pgmap v1695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:45 smithi160 ceph-mon[36715]: pgmap v1696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:45 smithi183 ceph-mon[37273]: pgmap v1695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:45 smithi183 ceph-mon[37273]: pgmap v1696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:40:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2089.572096 secs 2024-07-26T11:40:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2089.572096 secs 2024-07-26T11:40:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2089.572096 secs 2024-07-26T11:40:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:47 smithi144 ceph-mon[32129]: pgmap v1697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:47 smithi160 ceph-mon[36715]: pgmap v1697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:47 smithi183 ceph-mon[37273]: pgmap v1697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:49 smithi160 ceph-mon[36715]: pgmap v1698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T11:40:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:49 smithi183 ceph-mon[37273]: pgmap v1698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T11:40:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:49 smithi144 ceph-mon[32129]: pgmap v1698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T11:40:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:51 smithi160 ceph-mon[36715]: pgmap v1699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:40:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2094.572214 secs 2024-07-26T11:40:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:51 smithi183 ceph-mon[37273]: pgmap v1699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:40:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2094.572214 secs 2024-07-26T11:40:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:51 smithi144 ceph-mon[32129]: pgmap v1699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:40:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2094.572214 secs 2024-07-26T11:40:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:53 smithi160 ceph-mon[36715]: pgmap v1700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:53 smithi183 ceph-mon[37273]: pgmap v1700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:53 smithi144 ceph-mon[32129]: pgmap v1700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:40:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:55 smithi160 ceph-mon[36715]: pgmap v1701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:40:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:55 smithi183 ceph-mon[37273]: pgmap v1701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:40:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:55 smithi144 ceph-mon[32129]: pgmap v1701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:40:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2099.572335 secs 2024-07-26T11:40:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2099.572335 secs 2024-07-26T11:40:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2099.572335 secs 2024-07-26T11:40:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:57 smithi160 ceph-mon[36715]: pgmap v1702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:57 smithi183 ceph-mon[37273]: pgmap v1702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:57 smithi144 ceph-mon[32129]: pgmap v1702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:40:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:58.470+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:58.471+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:58.471+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:58.471+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:40:58.471+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:40:59 smithi144 ceph-mon[32129]: pgmap v1703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:40:59 smithi160 ceph-mon[36715]: pgmap v1703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:40:59 smithi183 ceph-mon[37273]: pgmap v1703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:01 smithi160 ceph-mon[36715]: pgmap v1704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2104.572455 secs 2024-07-26T11:41:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:01 smithi183 ceph-mon[37273]: pgmap v1704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2104.572455 secs 2024-07-26T11:41:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:01 smithi144 ceph-mon[32129]: pgmap v1704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2104.572455 secs 2024-07-26T11:41:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:03 smithi160 ceph-mon[36715]: pgmap v1705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:03 smithi183 ceph-mon[37273]: pgmap v1705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:03 smithi144 ceph-mon[32129]: pgmap v1705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:05 smithi160 ceph-mon[36715]: pgmap v1706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:05 smithi183 ceph-mon[37273]: pgmap v1706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:05 smithi144 ceph-mon[32129]: pgmap v1706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2109.572557 secs 2024-07-26T11:41:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2109.572557 secs 2024-07-26T11:41:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2109.572557 secs 2024-07-26T11:41:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:07 smithi160 ceph-mon[36715]: pgmap v1707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:07 smithi183 ceph-mon[37273]: pgmap v1707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:07 smithi144 ceph-mon[32129]: pgmap v1707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:09 smithi183 ceph-mon[37273]: pgmap v1708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:09.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:09 smithi144 ceph-mon[32129]: pgmap v1708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:09 smithi160 ceph-mon[36715]: pgmap v1708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:11 smithi183 ceph-mon[37273]: pgmap v1709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:41:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2114.572636 secs 2024-07-26T11:41:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:11 smithi144 ceph-mon[32129]: pgmap v1709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:41:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2114.572636 secs 2024-07-26T11:41:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:11 smithi160 ceph-mon[36715]: pgmap v1709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:41:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2114.572636 secs 2024-07-26T11:41:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:13.487+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:13.488+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:13.488+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:13.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:13.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:13 smithi183 ceph-mon[37273]: pgmap v1710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:13 smithi144 ceph-mon[32129]: pgmap v1710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:13 smithi160 ceph-mon[36715]: pgmap v1710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:15 smithi183 ceph-mon[37273]: pgmap v1711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:41:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:15 smithi144 ceph-mon[32129]: pgmap v1711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:41:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:15 smithi160 ceph-mon[36715]: pgmap v1711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:41:16.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2119.572752 secs 2024-07-26T11:41:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2119.572752 secs 2024-07-26T11:41:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2119.572752 secs 2024-07-26T11:41:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:17 smithi160 ceph-mon[36715]: pgmap v1712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:17 smithi183 ceph-mon[37273]: pgmap v1712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:17 smithi144 ceph-mon[32129]: pgmap v1712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:19 smithi183 ceph-mon[37273]: pgmap v1713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:19 smithi144 ceph-mon[32129]: pgmap v1713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:19.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:19 smithi160 ceph-mon[36715]: pgmap v1713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:21 smithi183 ceph-mon[37273]: pgmap v1714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2124.572889 secs 2024-07-26T11:41:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:21 smithi144 ceph-mon[32129]: pgmap v1714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2124.572889 secs 2024-07-26T11:41:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:21 smithi160 ceph-mon[36715]: pgmap v1714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2124.572889 secs 2024-07-26T11:41:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:23 smithi183 ceph-mon[37273]: pgmap v1715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:23 smithi144 ceph-mon[32129]: pgmap v1715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:23 smithi160 ceph-mon[36715]: pgmap v1715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:25 smithi183 ceph-mon[37273]: pgmap v1716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:25 smithi144 ceph-mon[32129]: pgmap v1716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:25 smithi160 ceph-mon[36715]: pgmap v1716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2129.573006 secs 2024-07-26T11:41:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2129.573006 secs 2024-07-26T11:41:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2129.573006 secs 2024-07-26T11:41:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:27 smithi183 ceph-mon[37273]: pgmap v1717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:27 smithi144 ceph-mon[32129]: pgmap v1717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:27 smithi160 ceph-mon[36715]: pgmap v1717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:28.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:28.504+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:28.505+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:28.505+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:28.505+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:28.505+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:29 smithi144 ceph-mon[32129]: pgmap v1718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:29 smithi160 ceph-mon[36715]: pgmap v1718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:29 smithi183 ceph-mon[37273]: pgmap v1718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:31 smithi144 ceph-mon[32129]: pgmap v1719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2134.573125 secs 2024-07-26T11:41:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:31 smithi160 ceph-mon[36715]: pgmap v1719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2134.573125 secs 2024-07-26T11:41:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:31 smithi183 ceph-mon[37273]: pgmap v1719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2134.573125 secs 2024-07-26T11:41:33.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:33 smithi183 ceph-mon[37273]: pgmap v1720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:33 smithi144 ceph-mon[32129]: pgmap v1720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:33 smithi160 ceph-mon[36715]: pgmap v1720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:35 smithi144 ceph-mon[32129]: pgmap v1721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:35 smithi160 ceph-mon[36715]: pgmap v1721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:35 smithi183 ceph-mon[37273]: pgmap v1721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:36.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2139.573249 secs 2024-07-26T11:41:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2139.573249 secs 2024-07-26T11:41:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2139.573249 secs 2024-07-26T11:41:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:37 smithi144 ceph-mon[32129]: pgmap v1722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:37 smithi160 ceph-mon[36715]: pgmap v1722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:37 smithi183 ceph-mon[37273]: pgmap v1722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:41:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:39 smithi144 ceph-mon[32129]: pgmap v1723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:39 smithi160 ceph-mon[36715]: pgmap v1723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:39 smithi183 ceph-mon[37273]: pgmap v1723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:41 smithi144 ceph-mon[32129]: pgmap v1724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2144.573344 secs 2024-07-26T11:41:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:41 smithi160 ceph-mon[36715]: pgmap v1724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2144.573344 secs 2024-07-26T11:41:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:41 smithi183 ceph-mon[37273]: pgmap v1724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2144.573344 secs 2024-07-26T11:41:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:43.520+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:43.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:43.521+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:43.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:43.521+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:43.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:43.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:43.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:43.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:43 smithi144 ceph-mon[32129]: pgmap v1725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:43 smithi160 ceph-mon[36715]: pgmap v1725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:43 smithi183 ceph-mon[37273]: pgmap v1725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:45 smithi144 ceph-mon[32129]: pgmap v1726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:41:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:45 smithi160 ceph-mon[36715]: pgmap v1726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:41:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:45 smithi183 ceph-mon[37273]: pgmap v1726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:41:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:41:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:46.466+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:41:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:46.466+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:41:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:46.466+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:41:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:46.466+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x7f74b0109990 2024-07-26T11:41:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:46 smithi144 ceph-mon[32129]: pgmap v1727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:46 smithi144 ceph-mon[32129]: pgmap v1728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2149.573386 secs 2024-07-26T11:41:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:46 smithi160 ceph-mon[36715]: pgmap v1727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:46 smithi160 ceph-mon[36715]: pgmap v1728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2149.573386 secs 2024-07-26T11:41:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:46 smithi183 ceph-mon[37273]: pgmap v1727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:46 smithi183 ceph-mon[37273]: pgmap v1728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2149.573386 secs 2024-07-26T11:41:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:48 smithi144 ceph-mon[32129]: pgmap v1729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:48 smithi160 ceph-mon[36715]: pgmap v1729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:48 smithi183 ceph-mon[37273]: pgmap v1729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:50 smithi144 ceph-mon[32129]: pgmap v1730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:50 smithi160 ceph-mon[36715]: pgmap v1730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:50 smithi183 ceph-mon[37273]: pgmap v1730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:41:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2154.573470 secs 2024-07-26T11:41:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2154.573470 secs 2024-07-26T11:41:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2154.573470 secs 2024-07-26T11:41:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:52 smithi160 ceph-mon[36715]: pgmap v1731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:52 smithi183 ceph-mon[37273]: pgmap v1731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:52 smithi144 ceph-mon[32129]: pgmap v1731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:41:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:54 smithi160 ceph-mon[36715]: pgmap v1732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:41:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:54 smithi183 ceph-mon[37273]: pgmap v1732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:41:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:54 smithi144 ceph-mon[32129]: pgmap v1732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:41:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:56 smithi160 ceph-mon[36715]: pgmap v1733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2159.573553 secs 2024-07-26T11:41:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:56 smithi183 ceph-mon[37273]: pgmap v1733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2159.573553 secs 2024-07-26T11:41:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:56 smithi144 ceph-mon[32129]: pgmap v1733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2159.573553 secs 2024-07-26T11:41:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:58.529+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:58.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:58.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:58.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:41:58.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:41:58 smithi160 ceph-mon[36715]: pgmap v1734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:41:58 smithi183 ceph-mon[37273]: pgmap v1734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:41:58 smithi144 ceph-mon[32129]: pgmap v1734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:00 smithi160 ceph-mon[36715]: pgmap v1735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:00 smithi183 ceph-mon[37273]: pgmap v1735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:00 smithi144 ceph-mon[32129]: pgmap v1735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2164.573641 secs 2024-07-26T11:42:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2164.573641 secs 2024-07-26T11:42:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2164.573641 secs 2024-07-26T11:42:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:02 smithi160 ceph-mon[36715]: pgmap v1736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:02 smithi183 ceph-mon[37273]: pgmap v1736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:02 smithi144 ceph-mon[32129]: pgmap v1736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:04 smithi160 ceph-mon[36715]: pgmap v1737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:04 smithi183 ceph-mon[37273]: pgmap v1737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:04 smithi144 ceph-mon[32129]: pgmap v1737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:06 smithi160 ceph-mon[36715]: pgmap v1738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2169.573717 secs 2024-07-26T11:42:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:06 smithi183 ceph-mon[37273]: pgmap v1738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2169.573717 secs 2024-07-26T11:42:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:06 smithi144 ceph-mon[32129]: pgmap v1738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2169.573717 secs 2024-07-26T11:42:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:08 smithi160 ceph-mon[36715]: pgmap v1739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:08 smithi183 ceph-mon[37273]: pgmap v1739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:08 smithi144 ceph-mon[32129]: pgmap v1739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:10 smithi160 ceph-mon[36715]: pgmap v1740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:10 smithi183 ceph-mon[37273]: pgmap v1740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:10 smithi144 ceph-mon[32129]: pgmap v1740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2174.573798 secs 2024-07-26T11:42:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2174.573798 secs 2024-07-26T11:42:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2174.573798 secs 2024-07-26T11:42:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:12 smithi160 ceph-mon[36715]: pgmap v1741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:12 smithi183 ceph-mon[37273]: pgmap v1741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:12 smithi144 ceph-mon[32129]: pgmap v1741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:13.536 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:13.534+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:13.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:13.535+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:13.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:13.535+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:13.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:13.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:13.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:13.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:14 smithi183 ceph-mon[37273]: pgmap v1742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:14 smithi144 ceph-mon[32129]: pgmap v1742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:14 smithi160 ceph-mon[36715]: pgmap v1742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:16 smithi183 ceph-mon[37273]: pgmap v1743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2179.573895 secs 2024-07-26T11:42:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:16 smithi144 ceph-mon[32129]: pgmap v1743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2179.573895 secs 2024-07-26T11:42:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:16 smithi160 ceph-mon[36715]: pgmap v1743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2179.573895 secs 2024-07-26T11:42:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:18 smithi183 ceph-mon[37273]: pgmap v1744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:18 smithi144 ceph-mon[32129]: pgmap v1744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:18 smithi160 ceph-mon[36715]: pgmap v1744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:20 smithi183 ceph-mon[37273]: pgmap v1745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:20 smithi144 ceph-mon[32129]: pgmap v1745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:20 smithi160 ceph-mon[36715]: pgmap v1745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2184.574034 secs 2024-07-26T11:42:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2184.574034 secs 2024-07-26T11:42:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2184.574034 secs 2024-07-26T11:42:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:22 smithi183 ceph-mon[37273]: pgmap v1746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:22 smithi144 ceph-mon[32129]: pgmap v1746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:22 smithi160 ceph-mon[36715]: pgmap v1746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:24 smithi183 ceph-mon[37273]: pgmap v1747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:24 smithi144 ceph-mon[32129]: pgmap v1747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:24 smithi160 ceph-mon[36715]: pgmap v1747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:26 smithi183 ceph-mon[37273]: pgmap v1748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2189.574146 secs 2024-07-26T11:42:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:26 smithi144 ceph-mon[32129]: pgmap v1748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2189.574146 secs 2024-07-26T11:42:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:26 smithi160 ceph-mon[36715]: pgmap v1748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2189.574146 secs 2024-07-26T11:42:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:28.551+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:28.552+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:28.552+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:28.552+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:28.552+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:28 smithi183 ceph-mon[37273]: pgmap v1749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:28 smithi144 ceph-mon[32129]: pgmap v1749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:28 smithi160 ceph-mon[36715]: pgmap v1749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:30 smithi183 ceph-mon[37273]: pgmap v1750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:30 smithi144 ceph-mon[32129]: pgmap v1750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:30 smithi160 ceph-mon[36715]: pgmap v1750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2194.574261 secs 2024-07-26T11:42:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2194.574261 secs 2024-07-26T11:42:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2194.574261 secs 2024-07-26T11:42:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:32 smithi183 ceph-mon[37273]: pgmap v1751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:32 smithi144 ceph-mon[32129]: pgmap v1751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:32 smithi160 ceph-mon[36715]: pgmap v1751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:34 smithi183 ceph-mon[37273]: pgmap v1752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:34.910 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:34 smithi144 ceph-mon[32129]: pgmap v1752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:34 smithi160 ceph-mon[36715]: pgmap v1752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:36 smithi144 ceph-mon[32129]: pgmap v1753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2199.574359 secs 2024-07-26T11:42:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:36 smithi160 ceph-mon[36715]: pgmap v1753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2199.574359 secs 2024-07-26T11:42:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:36 smithi183 ceph-mon[37273]: pgmap v1753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2199.574359 secs 2024-07-26T11:42:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:38 smithi144 ceph-mon[32129]: pgmap v1754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:38 smithi160 ceph-mon[36715]: pgmap v1754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:38 smithi183 ceph-mon[37273]: pgmap v1754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:40 smithi144 ceph-mon[32129]: pgmap v1755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:40 smithi160 ceph-mon[36715]: pgmap v1755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:40 smithi183 ceph-mon[37273]: pgmap v1755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2204.574478 secs 2024-07-26T11:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2204.574478 secs 2024-07-26T11:42:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2204.574478 secs 2024-07-26T11:42:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:42 smithi144 ceph-mon[32129]: pgmap v1756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:42 smithi160 ceph-mon[36715]: pgmap v1756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:42 smithi183 ceph-mon[37273]: pgmap v1756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:42:43.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:43.567+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:43.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:43.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:43.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:43.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:44 smithi144 ceph-mon[32129]: pgmap v1757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:44 smithi160 ceph-mon[36715]: pgmap v1757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:44 smithi183 ceph-mon[37273]: pgmap v1757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:46 smithi144 ceph-mon[32129]: pgmap v1758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2209.574598 secs 2024-07-26T11:42:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:42:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:46 smithi160 ceph-mon[36715]: pgmap v1758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2209.574598 secs 2024-07-26T11:42:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:42:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:46 smithi183 ceph-mon[37273]: pgmap v1758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2209.574598 secs 2024-07-26T11:42:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:42:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:47 smithi144 ceph-mon[32129]: pgmap v1759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:42:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:47 smithi144 ceph-mon[32129]: pgmap v1760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:42:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:47 smithi160 ceph-mon[36715]: pgmap v1759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:42:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:47 smithi160 ceph-mon[36715]: pgmap v1760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:42:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:47 smithi183 ceph-mon[37273]: pgmap v1759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:42:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:47 smithi183 ceph-mon[37273]: pgmap v1760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:42:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:49 smithi144 ceph-mon[32129]: pgmap v1761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:42:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:49 smithi160 ceph-mon[36715]: pgmap v1761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:42:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:49 smithi183 ceph-mon[37273]: pgmap v1761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:42:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:50 smithi144 ceph-mon[32129]: pgmap v1762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:42:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:50 smithi160 ceph-mon[36715]: pgmap v1762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:42:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:50 smithi183 ceph-mon[37273]: pgmap v1762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:42:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2214.574677 secs 2024-07-26T11:42:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2214.574677 secs 2024-07-26T11:42:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2214.574677 secs 2024-07-26T11:42:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:52 smithi144 ceph-mon[32129]: pgmap v1763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:42:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:52 smithi160 ceph-mon[36715]: pgmap v1763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:42:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:52 smithi183 ceph-mon[37273]: pgmap v1763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:42:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:55 smithi144 ceph-mon[32129]: pgmap v1764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:42:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:55 smithi160 ceph-mon[36715]: pgmap v1764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:42:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:55 smithi183 ceph-mon[37273]: pgmap v1764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:42:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2219.574810 secs 2024-07-26T11:42:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2219.574810 secs 2024-07-26T11:42:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2219.574810 secs 2024-07-26T11:42:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:57 smithi144 ceph-mon[32129]: pgmap v1765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:42:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:57 smithi160 ceph-mon[36715]: pgmap v1765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:42:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:57 smithi183 ceph-mon[37273]: pgmap v1765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:42:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:58.585+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:58.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:58.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:58.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:42:58.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:42:58 smithi144 ceph-mon[32129]: pgmap v1766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:42:58 smithi160 ceph-mon[36715]: pgmap v1766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:42:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:42:58 smithi183 ceph-mon[37273]: pgmap v1766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2224.574936 secs 2024-07-26T11:43:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:01 smithi144 ceph-mon[32129]: pgmap v1767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2224.574936 secs 2024-07-26T11:43:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:01 smithi160 ceph-mon[36715]: pgmap v1767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2224.574936 secs 2024-07-26T11:43:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:01 smithi183 ceph-mon[37273]: pgmap v1767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:03 smithi144 ceph-mon[32129]: pgmap v1768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:03 smithi160 ceph-mon[36715]: pgmap v1768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:03 smithi183 ceph-mon[37273]: pgmap v1768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:05.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:05 smithi183 ceph-mon[37273]: pgmap v1769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:05 smithi144 ceph-mon[32129]: pgmap v1769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:05 smithi160 ceph-mon[36715]: pgmap v1769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:06.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2229.575066 secs 2024-07-26T11:43:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2229.575066 secs 2024-07-26T11:43:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2229.575066 secs 2024-07-26T11:43:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:07 smithi144 ceph-mon[32129]: pgmap v1770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:07 smithi160 ceph-mon[36715]: pgmap v1770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:07 smithi183 ceph-mon[37273]: pgmap v1770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:08 smithi144 ceph-mon[32129]: pgmap v1771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:08 smithi160 ceph-mon[36715]: pgmap v1771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:08 smithi183 ceph-mon[37273]: pgmap v1771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2234.575183 secs 2024-07-26T11:43:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:11 smithi144 ceph-mon[32129]: pgmap v1772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2234.575183 secs 2024-07-26T11:43:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:11 smithi160 ceph-mon[36715]: pgmap v1772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2234.575183 secs 2024-07-26T11:43:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:11 smithi183 ceph-mon[37273]: pgmap v1772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:13.602+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:13.603+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:13.603+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:13.603+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:13.603+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:13 smithi144 ceph-mon[32129]: pgmap v1773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:13 smithi160 ceph-mon[36715]: pgmap v1773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:13 smithi183 ceph-mon[37273]: pgmap v1773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:14 smithi160 ceph-mon[36715]: pgmap v1774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:14 smithi183 ceph-mon[37273]: pgmap v1774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:14 smithi144 ceph-mon[32129]: pgmap v1774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2239.575316 secs 2024-07-26T11:43:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2239.575316 secs 2024-07-26T11:43:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2239.575316 secs 2024-07-26T11:43:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:17 smithi160 ceph-mon[36715]: pgmap v1775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:17 smithi183 ceph-mon[37273]: pgmap v1775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:17 smithi144 ceph-mon[32129]: pgmap v1775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:19 smithi144 ceph-mon[32129]: pgmap v1776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:19 smithi160 ceph-mon[36715]: pgmap v1776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:19 smithi183 ceph-mon[37273]: pgmap v1776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2244.575435 secs 2024-07-26T11:43:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:21 smithi144 ceph-mon[32129]: pgmap v1777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2244.575435 secs 2024-07-26T11:43:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:21 smithi160 ceph-mon[36715]: pgmap v1777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2244.575435 secs 2024-07-26T11:43:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:21 smithi183 ceph-mon[37273]: pgmap v1777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:22 smithi144 ceph-mon[32129]: pgmap v1778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:22 smithi160 ceph-mon[36715]: pgmap v1778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:22 smithi183 ceph-mon[37273]: pgmap v1778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:25 smithi144 ceph-mon[32129]: pgmap v1779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:25 smithi160 ceph-mon[36715]: pgmap v1779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:25 smithi183 ceph-mon[37273]: pgmap v1779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2249.575558 secs 2024-07-26T11:43:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:26 smithi144 ceph-mon[32129]: pgmap v1780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2249.575558 secs 2024-07-26T11:43:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:26 smithi160 ceph-mon[36715]: pgmap v1780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2249.575558 secs 2024-07-26T11:43:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:26 smithi183 ceph-mon[37273]: pgmap v1780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:28.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:28.620+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:28.620+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:28.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:28.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:29 smithi144 ceph-mon[32129]: pgmap v1781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:29 smithi160 ceph-mon[36715]: pgmap v1781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:29 smithi183 ceph-mon[37273]: pgmap v1781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:30 smithi144 ceph-mon[32129]: pgmap v1782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:30 smithi160 ceph-mon[36715]: pgmap v1782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:30 smithi183 ceph-mon[37273]: pgmap v1782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2254.575674 secs 2024-07-26T11:43:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2254.575674 secs 2024-07-26T11:43:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2254.575674 secs 2024-07-26T11:43:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:32 smithi144 ceph-mon[32129]: pgmap v1783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:32 smithi160 ceph-mon[36715]: pgmap v1783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:32 smithi183 ceph-mon[37273]: pgmap v1783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:34 smithi160 ceph-mon[36715]: pgmap v1784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:34 smithi183 ceph-mon[37273]: pgmap v1784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:34 smithi144 ceph-mon[32129]: pgmap v1784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2259.575788 secs 2024-07-26T11:43:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2259.575788 secs 2024-07-26T11:43:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2259.575788 secs 2024-07-26T11:43:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:37 smithi144 ceph-mon[32129]: pgmap v1785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:37 smithi160 ceph-mon[36715]: pgmap v1785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:37 smithi183 ceph-mon[37273]: pgmap v1785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:38 smithi160 ceph-mon[36715]: pgmap v1786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:38 smithi183 ceph-mon[37273]: pgmap v1786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:38 smithi144 ceph-mon[32129]: pgmap v1786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:40 smithi160 ceph-mon[36715]: pgmap v1787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:40 smithi183 ceph-mon[37273]: pgmap v1787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:40 smithi144 ceph-mon[32129]: pgmap v1787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2264.575904 secs 2024-07-26T11:43:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2264.575904 secs 2024-07-26T11:43:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2264.575904 secs 2024-07-26T11:43:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:42 smithi160 ceph-mon[36715]: pgmap v1788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:42 smithi183 ceph-mon[37273]: pgmap v1788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:42 smithi144 ceph-mon[32129]: pgmap v1788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:43.635+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:43.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:43.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:43.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:43.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:44 smithi160 ceph-mon[36715]: pgmap v1789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:44 smithi183 ceph-mon[37273]: pgmap v1789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:44 smithi144 ceph-mon[32129]: pgmap v1789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:43:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2269.575969 secs 2024-07-26T11:43:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2269.575969 secs 2024-07-26T11:43:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2269.575969 secs 2024-07-26T11:43:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:47 smithi160 ceph-mon[36715]: pgmap v1790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:47 smithi183 ceph-mon[37273]: pgmap v1790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:47 smithi144 ceph-mon[32129]: pgmap v1790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:43:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:43:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:43:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:43:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:49 smithi160 ceph-mon[36715]: pgmap v1791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:43:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:49 smithi160 ceph-mon[36715]: pgmap v1792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:43:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:49 smithi183 ceph-mon[37273]: pgmap v1791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:43:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:49 smithi183 ceph-mon[37273]: pgmap v1792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:43:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:49 smithi144 ceph-mon[32129]: pgmap v1791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:43:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:49 smithi144 ceph-mon[32129]: pgmap v1792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:43:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:51 smithi160 ceph-mon[36715]: pgmap v1793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:43:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2274.576087 secs 2024-07-26T11:43:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:51 smithi183 ceph-mon[37273]: pgmap v1793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:43:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2274.576087 secs 2024-07-26T11:43:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:51 smithi144 ceph-mon[32129]: pgmap v1793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:43:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2274.576087 secs 2024-07-26T11:43:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:53 smithi160 ceph-mon[36715]: pgmap v1794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:43:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:53 smithi183 ceph-mon[37273]: pgmap v1794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:43:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:53 smithi144 ceph-mon[32129]: pgmap v1794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:43:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:55 smithi160 ceph-mon[36715]: pgmap v1795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:43:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:55 smithi183 ceph-mon[37273]: pgmap v1795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:43:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:55 smithi144 ceph-mon[32129]: pgmap v1795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:43:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2279.576203 secs 2024-07-26T11:43:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2279.576203 secs 2024-07-26T11:43:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2279.576203 secs 2024-07-26T11:43:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:57 smithi160 ceph-mon[36715]: pgmap v1796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:43:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:57 smithi183 ceph-mon[37273]: pgmap v1796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:43:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:57 smithi144 ceph-mon[32129]: pgmap v1796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:43:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:58.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:58.655+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:58.655+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:58.655+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:43:58.655+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:43:59 smithi160 ceph-mon[36715]: pgmap v1797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:43:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:43:59 smithi183 ceph-mon[37273]: pgmap v1797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:43:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:43:59 smithi144 ceph-mon[32129]: pgmap v1797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:44:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:01 smithi183 ceph-mon[37273]: pgmap v1798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2284.576305 secs 2024-07-26T11:44:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:01 smithi144 ceph-mon[32129]: pgmap v1798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2284.576305 secs 2024-07-26T11:44:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:01 smithi160 ceph-mon[36715]: pgmap v1798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2284.576305 secs 2024-07-26T11:44:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:03 smithi183 ceph-mon[37273]: pgmap v1799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:03 smithi144 ceph-mon[32129]: pgmap v1799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:03 smithi160 ceph-mon[36715]: pgmap v1799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:05 smithi183 ceph-mon[37273]: pgmap v1800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:05 smithi144 ceph-mon[32129]: pgmap v1800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:05 smithi160 ceph-mon[36715]: pgmap v1800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2289.576429 secs 2024-07-26T11:44:06.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2289.576429 secs 2024-07-26T11:44:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2289.576429 secs 2024-07-26T11:44:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:07 smithi183 ceph-mon[37273]: pgmap v1801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:07 smithi144 ceph-mon[32129]: pgmap v1801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:07 smithi160 ceph-mon[36715]: pgmap v1801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:09 smithi183 ceph-mon[37273]: pgmap v1802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:09 smithi144 ceph-mon[32129]: pgmap v1802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:09 smithi160 ceph-mon[36715]: pgmap v1802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:11 smithi183 ceph-mon[37273]: pgmap v1803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2294.576548 secs 2024-07-26T11:44:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:11 smithi144 ceph-mon[32129]: pgmap v1803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2294.576548 secs 2024-07-26T11:44:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:11 smithi160 ceph-mon[36715]: pgmap v1803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2294.576548 secs 2024-07-26T11:44:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:13 smithi183 ceph-mon[37273]: pgmap v1804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:13 smithi144 ceph-mon[32129]: pgmap v1804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:13 smithi160 ceph-mon[36715]: pgmap v1804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:13.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:13.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:13.672+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:13.672+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:13.672+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:13.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:13.672+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:15 smithi183 ceph-mon[37273]: pgmap v1805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:15 smithi144 ceph-mon[32129]: pgmap v1805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:15 smithi160 ceph-mon[36715]: pgmap v1805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2299.576670 secs 2024-07-26T11:44:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2299.576670 secs 2024-07-26T11:44:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2299.576670 secs 2024-07-26T11:44:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:17 smithi183 ceph-mon[37273]: pgmap v1806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:17 smithi144 ceph-mon[32129]: pgmap v1806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:17 smithi160 ceph-mon[36715]: pgmap v1806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:19 smithi183 ceph-mon[37273]: pgmap v1807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:19 smithi144 ceph-mon[32129]: pgmap v1807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:19 smithi160 ceph-mon[36715]: pgmap v1807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:21 smithi144 ceph-mon[32129]: pgmap v1808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2304.576809 secs 2024-07-26T11:44:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:21 smithi160 ceph-mon[36715]: pgmap v1808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2304.576809 secs 2024-07-26T11:44:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:21 smithi183 ceph-mon[37273]: pgmap v1808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2304.576809 secs 2024-07-26T11:44:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:23 smithi144 ceph-mon[32129]: pgmap v1809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:23 smithi160 ceph-mon[36715]: pgmap v1809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:23 smithi183 ceph-mon[37273]: pgmap v1809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:25 smithi144 ceph-mon[32129]: pgmap v1810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:25 smithi160 ceph-mon[36715]: pgmap v1810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:25 smithi183 ceph-mon[37273]: pgmap v1810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2309.576935 secs 2024-07-26T11:44:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2309.576935 secs 2024-07-26T11:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2309.576935 secs 2024-07-26T11:44:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:27 smithi144 ceph-mon[32129]: pgmap v1811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:27 smithi160 ceph-mon[36715]: pgmap v1811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:27 smithi183 ceph-mon[37273]: pgmap v1811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:28.687+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:28.688+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:28.688+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:28.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:28.690 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:28.689+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:29 smithi144 ceph-mon[32129]: pgmap v1812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:29 smithi160 ceph-mon[36715]: pgmap v1812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:29 smithi183 ceph-mon[37273]: pgmap v1812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:31 smithi144 ceph-mon[32129]: pgmap v1813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2314.577054 secs 2024-07-26T11:44:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:31 smithi160 ceph-mon[36715]: pgmap v1813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2314.577054 secs 2024-07-26T11:44:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:31 smithi183 ceph-mon[37273]: pgmap v1813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2314.577054 secs 2024-07-26T11:44:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:33 smithi144 ceph-mon[32129]: pgmap v1814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:33 smithi160 ceph-mon[36715]: pgmap v1814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:33 smithi183 ceph-mon[37273]: pgmap v1814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:35 smithi144 ceph-mon[32129]: pgmap v1815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:35 smithi160 ceph-mon[36715]: pgmap v1815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:35 smithi183 ceph-mon[37273]: pgmap v1815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:36.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2319.577177 secs 2024-07-26T11:44:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2319.577177 secs 2024-07-26T11:44:36.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2319.577177 secs 2024-07-26T11:44:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:37 smithi144 ceph-mon[32129]: pgmap v1816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:37 smithi160 ceph-mon[36715]: pgmap v1816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:37 smithi183 ceph-mon[37273]: pgmap v1816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:39 smithi144 ceph-mon[32129]: pgmap v1817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:39 smithi160 ceph-mon[36715]: pgmap v1817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:39 smithi183 ceph-mon[37273]: pgmap v1817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:41 smithi160 ceph-mon[36715]: pgmap v1818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2324.577325 secs 2024-07-26T11:44:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:41 smithi183 ceph-mon[37273]: pgmap v1818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2324.577325 secs 2024-07-26T11:44:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:41 smithi144 ceph-mon[32129]: pgmap v1818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2324.577325 secs 2024-07-26T11:44:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:43 smithi160 ceph-mon[36715]: pgmap v1819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:43 smithi183 ceph-mon[37273]: pgmap v1819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:43.703+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:43.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:43.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:43.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:43.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:43 smithi144 ceph-mon[32129]: pgmap v1819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:44:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:45 smithi160 ceph-mon[36715]: pgmap v1820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:45 smithi183 ceph-mon[37273]: pgmap v1820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:45 smithi144 ceph-mon[32129]: pgmap v1820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:46.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:46.499+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:44:46.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:46.499+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:44:46.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:46.499+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:44:46.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:46.499+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d020 con 0x563c6c4182f0 2024-07-26T11:44:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2329.577444 secs 2024-07-26T11:44:46.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2329.577444 secs 2024-07-26T11:44:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2329.577444 secs 2024-07-26T11:44:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:47 smithi160 ceph-mon[36715]: pgmap v1821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:47 smithi183 ceph-mon[37273]: pgmap v1821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:47 smithi144 ceph-mon[32129]: pgmap v1821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:44:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:49 smithi160 ceph-mon[36715]: pgmap v1822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:44:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:49 smithi183 ceph-mon[37273]: pgmap v1822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:44:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:49 smithi144 ceph-mon[32129]: pgmap v1822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:44:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:50 smithi160 ceph-mon[36715]: pgmap v1823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:50 smithi160 ceph-mon[36715]: pgmap v1824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:44:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:50 smithi183 ceph-mon[37273]: pgmap v1823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:50 smithi183 ceph-mon[37273]: pgmap v1824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:44:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:50 smithi144 ceph-mon[32129]: pgmap v1823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:50 smithi144 ceph-mon[32129]: pgmap v1824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:44:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2334.577589 secs 2024-07-26T11:44:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2334.577589 secs 2024-07-26T11:44:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2334.577589 secs 2024-07-26T11:44:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:52 smithi160 ceph-mon[36715]: pgmap v1825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:52 smithi183 ceph-mon[37273]: pgmap v1825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:52 smithi144 ceph-mon[32129]: pgmap v1825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:54 smithi144 ceph-mon[32129]: pgmap v1826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:44:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:54 smithi160 ceph-mon[36715]: pgmap v1826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:44:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:54 smithi183 ceph-mon[37273]: pgmap v1826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:44:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:56 smithi160 ceph-mon[36715]: pgmap v1827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:44:56.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2339.577672 secs 2024-07-26T11:44:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:56 smithi183 ceph-mon[37273]: pgmap v1827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:44:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2339.577672 secs 2024-07-26T11:44:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:56 smithi144 ceph-mon[32129]: pgmap v1827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:44:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2339.577672 secs 2024-07-26T11:44:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:44:58 smithi160 ceph-mon[36715]: pgmap v1828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:44:58 smithi183 ceph-mon[37273]: pgmap v1828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:44:58 smithi144 ceph-mon[32129]: pgmap v1828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:44:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:58.718+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:58.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:58.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:58.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:44:58.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:00 smithi160 ceph-mon[36715]: pgmap v1829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:45:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:00 smithi183 ceph-mon[37273]: pgmap v1829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:45:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:00 smithi144 ceph-mon[32129]: pgmap v1829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:45:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2344.577808 secs 2024-07-26T11:45:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2344.577808 secs 2024-07-26T11:45:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2344.577808 secs 2024-07-26T11:45:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:02 smithi183 ceph-mon[37273]: pgmap v1830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:02 smithi144 ceph-mon[32129]: pgmap v1830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:02 smithi160 ceph-mon[36715]: pgmap v1830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:04 smithi183 ceph-mon[37273]: pgmap v1831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:04 smithi144 ceph-mon[32129]: pgmap v1831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:04 smithi160 ceph-mon[36715]: pgmap v1831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:06.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:06 smithi160 ceph-mon[36715]: pgmap v1832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:06.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2349.577926 secs 2024-07-26T11:45:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:06 smithi183 ceph-mon[37273]: pgmap v1832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2349.577926 secs 2024-07-26T11:45:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:06 smithi144 ceph-mon[32129]: pgmap v1832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2349.577926 secs 2024-07-26T11:45:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:08 smithi183 ceph-mon[37273]: pgmap v1833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:08 smithi144 ceph-mon[32129]: pgmap v1833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:08 smithi160 ceph-mon[36715]: pgmap v1833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:10 smithi183 ceph-mon[37273]: pgmap v1834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:10 smithi144 ceph-mon[32129]: pgmap v1834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:10 smithi160 ceph-mon[36715]: pgmap v1834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2354.578064 secs 2024-07-26T11:45:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2354.578064 secs 2024-07-26T11:45:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2354.578064 secs 2024-07-26T11:45:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:12 smithi183 ceph-mon[37273]: pgmap v1835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:12 smithi144 ceph-mon[32129]: pgmap v1835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:12 smithi160 ceph-mon[36715]: pgmap v1835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:13.735+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:13.736+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:13.736+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:13.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:13.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:14 smithi183 ceph-mon[37273]: pgmap v1836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:14 smithi144 ceph-mon[32129]: pgmap v1836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:14 smithi160 ceph-mon[36715]: pgmap v1836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:16 smithi183 ceph-mon[37273]: pgmap v1837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2359.578202 secs 2024-07-26T11:45:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:16 smithi144 ceph-mon[32129]: pgmap v1837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2359.578202 secs 2024-07-26T11:45:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:16 smithi160 ceph-mon[36715]: pgmap v1837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2359.578202 secs 2024-07-26T11:45:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:18 smithi183 ceph-mon[37273]: pgmap v1838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:18 smithi144 ceph-mon[32129]: pgmap v1838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:18 smithi160 ceph-mon[36715]: pgmap v1838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:20 smithi183 ceph-mon[37273]: pgmap v1839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:20 smithi144 ceph-mon[32129]: pgmap v1839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:20 smithi160 ceph-mon[36715]: pgmap v1839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2364.578327 secs 2024-07-26T11:45:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2364.578327 secs 2024-07-26T11:45:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2364.578327 secs 2024-07-26T11:45:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:22 smithi144 ceph-mon[32129]: pgmap v1840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:22 smithi160 ceph-mon[36715]: pgmap v1840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:22 smithi183 ceph-mon[37273]: pgmap v1840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:24 smithi144 ceph-mon[32129]: pgmap v1841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:24 smithi160 ceph-mon[36715]: pgmap v1841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:24.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:24 smithi183 ceph-mon[37273]: pgmap v1841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:26 smithi144 ceph-mon[32129]: pgmap v1842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2369.578423 secs 2024-07-26T11:45:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:26 smithi160 ceph-mon[36715]: pgmap v1842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2369.578423 secs 2024-07-26T11:45:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:26 smithi183 ceph-mon[37273]: pgmap v1842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2369.578423 secs 2024-07-26T11:45:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:28 smithi144 ceph-mon[32129]: pgmap v1843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:28.752+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:28.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:28.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:28.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:28.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:28 smithi160 ceph-mon[36715]: pgmap v1843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:28 smithi183 ceph-mon[37273]: pgmap v1843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:30 smithi144 ceph-mon[32129]: pgmap v1844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:30 smithi160 ceph-mon[36715]: pgmap v1844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:30 smithi183 ceph-mon[37273]: pgmap v1844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2374.578509 secs 2024-07-26T11:45:31.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2374.578509 secs 2024-07-26T11:45:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2374.578509 secs 2024-07-26T11:45:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:32 smithi183 ceph-mon[37273]: pgmap v1845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:32 smithi144 ceph-mon[32129]: pgmap v1845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:32 smithi160 ceph-mon[36715]: pgmap v1845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:34 smithi183 ceph-mon[37273]: pgmap v1846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:34 smithi144 ceph-mon[32129]: pgmap v1846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:34 smithi160 ceph-mon[36715]: pgmap v1846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:36.860 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:36 smithi160 ceph-mon[36715]: pgmap v1847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:36.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2379.578591 secs 2024-07-26T11:45:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:36 smithi183 ceph-mon[37273]: pgmap v1847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2379.578591 secs 2024-07-26T11:45:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:36 smithi144 ceph-mon[32129]: pgmap v1847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2379.578591 secs 2024-07-26T11:45:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:38 smithi183 ceph-mon[37273]: pgmap v1848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:38 smithi144 ceph-mon[32129]: pgmap v1848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:38 smithi160 ceph-mon[36715]: pgmap v1848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:40 smithi183 ceph-mon[37273]: pgmap v1849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:40 smithi144 ceph-mon[32129]: pgmap v1849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:40 smithi160 ceph-mon[36715]: pgmap v1849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2384.578659 secs 2024-07-26T11:45:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2384.578659 secs 2024-07-26T11:45:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2384.578659 secs 2024-07-26T11:45:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:42 smithi183 ceph-mon[37273]: pgmap v1850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:42 smithi144 ceph-mon[32129]: pgmap v1850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:42 smithi160 ceph-mon[36715]: pgmap v1850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:43.767+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:43.768+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:43.768+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:43.768+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:43.768+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:44 smithi183 ceph-mon[37273]: pgmap v1851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:44 smithi144 ceph-mon[32129]: pgmap v1851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:44 smithi160 ceph-mon[36715]: pgmap v1851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:46 smithi183 ceph-mon[37273]: pgmap v1852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2389.578796 secs 2024-07-26T11:45:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:46 smithi144 ceph-mon[32129]: pgmap v1852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2389.578796 secs 2024-07-26T11:45:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:46 smithi160 ceph-mon[36715]: pgmap v1852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2389.578796 secs 2024-07-26T11:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:48 smithi183 ceph-mon[37273]: pgmap v1853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:48 smithi144 ceph-mon[32129]: pgmap v1853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:48 smithi160 ceph-mon[36715]: pgmap v1853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:45:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:50 smithi144 ceph-mon[32129]: pgmap v1854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:45:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:50 smithi160 ceph-mon[36715]: pgmap v1854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:51.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:51.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:45:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:50 smithi183 ceph-mon[37273]: pgmap v1854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:45:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:45:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:51 smithi144 ceph-mon[32129]: pgmap v1855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:51 smithi144 ceph-mon[32129]: pgmap v1856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2394.578880 secs 2024-07-26T11:45:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:51 smithi160 ceph-mon[36715]: pgmap v1855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:51 smithi160 ceph-mon[36715]: pgmap v1856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2394.578880 secs 2024-07-26T11:45:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:51 smithi183 ceph-mon[37273]: pgmap v1855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:51 smithi183 ceph-mon[37273]: pgmap v1856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2394.578880 secs 2024-07-26T11:45:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:53 smithi144 ceph-mon[32129]: pgmap v1857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:53 smithi160 ceph-mon[36715]: pgmap v1857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:53 smithi183 ceph-mon[37273]: pgmap v1857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:55 smithi144 ceph-mon[32129]: pgmap v1858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:55 smithi160 ceph-mon[36715]: pgmap v1858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:55 smithi183 ceph-mon[37273]: pgmap v1858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2399.579025 secs 2024-07-26T11:45:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2399.579025 secs 2024-07-26T11:45:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2399.579025 secs 2024-07-26T11:45:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:57 smithi144 ceph-mon[32129]: pgmap v1859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:58.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:57 smithi160 ceph-mon[36715]: pgmap v1859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:57 smithi183 ceph-mon[37273]: pgmap v1859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:58.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:58.784+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:58.786+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:58.786+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:58.786+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:45:58.786+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:45:59 smithi144 ceph-mon[32129]: pgmap v1860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:46:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:45:59 smithi160 ceph-mon[36715]: pgmap v1860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:46:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:45:59 smithi183 ceph-mon[37273]: pgmap v1860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:46:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:01 smithi144 ceph-mon[32129]: pgmap v1861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2404.579149 secs 2024-07-26T11:46:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:01 smithi160 ceph-mon[36715]: pgmap v1861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2404.579149 secs 2024-07-26T11:46:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:01 smithi183 ceph-mon[37273]: pgmap v1861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2404.579149 secs 2024-07-26T11:46:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:03 smithi144 ceph-mon[32129]: pgmap v1862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:03 smithi160 ceph-mon[36715]: pgmap v1862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:03 smithi183 ceph-mon[37273]: pgmap v1862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:05 smithi144 ceph-mon[32129]: pgmap v1863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:05 smithi160 ceph-mon[36715]: pgmap v1863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:05 smithi183 ceph-mon[37273]: pgmap v1863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2409.579269 secs 2024-07-26T11:46:07.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2409.579269 secs 2024-07-26T11:46:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2409.579269 secs 2024-07-26T11:46:07.782 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:07 smithi144 ceph-mon[32129]: pgmap v1864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:07 smithi160 ceph-mon[36715]: pgmap v1864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:07 smithi183 ceph-mon[37273]: pgmap v1864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:09 smithi160 ceph-mon[36715]: pgmap v1865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:09 smithi183 ceph-mon[37273]: pgmap v1865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:09 smithi144 ceph-mon[32129]: pgmap v1865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:11.999 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:11 smithi144 ceph-mon[32129]: pgmap v1866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:11.999 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2414.579391 secs 2024-07-26T11:46:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:11 smithi160 ceph-mon[36715]: pgmap v1866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2414.579391 secs 2024-07-26T11:46:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:11 smithi183 ceph-mon[37273]: pgmap v1866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2414.579391 secs 2024-07-26T11:46:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:12 smithi160 ceph-mon[36715]: pgmap v1867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:46:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:12 smithi183 ceph-mon[37273]: pgmap v1867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:46:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:12 smithi144 ceph-mon[32129]: pgmap v1867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:46:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:13.800+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:13.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:13.802+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:13.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:13.802+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:13.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:13.802+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:13.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:13.802+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:15 smithi183 ceph-mon[37273]: pgmap v1868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:46:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:15 smithi144 ceph-mon[32129]: pgmap v1868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:46:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:15 smithi160 ceph-mon[36715]: pgmap v1868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:46:16.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2419.579516 secs 2024-07-26T11:46:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2419.579516 secs 2024-07-26T11:46:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2419.579516 secs 2024-07-26T11:46:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:17 smithi183 ceph-mon[37273]: pgmap v1869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:17 smithi144 ceph-mon[32129]: pgmap v1869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:17 smithi160 ceph-mon[36715]: pgmap v1869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:19 smithi183 ceph-mon[37273]: pgmap v1870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:19 smithi144 ceph-mon[32129]: pgmap v1870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:19 smithi160 ceph-mon[36715]: pgmap v1870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:21 smithi183 ceph-mon[37273]: pgmap v1871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2424.579637 secs 2024-07-26T11:46:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:21 smithi144 ceph-mon[32129]: pgmap v1871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2424.579637 secs 2024-07-26T11:46:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:21 smithi160 ceph-mon[36715]: pgmap v1871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2424.579637 secs 2024-07-26T11:46:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:23 smithi183 ceph-mon[37273]: pgmap v1872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:23 smithi144 ceph-mon[32129]: pgmap v1872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:23 smithi160 ceph-mon[36715]: pgmap v1872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:25 smithi183 ceph-mon[37273]: pgmap v1873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:25 smithi144 ceph-mon[32129]: pgmap v1873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:25 smithi160 ceph-mon[36715]: pgmap v1873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2429.579758 secs 2024-07-26T11:46:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2429.579758 secs 2024-07-26T11:46:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2429.579758 secs 2024-07-26T11:46:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:27 smithi183 ceph-mon[37273]: pgmap v1874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:27 smithi144 ceph-mon[32129]: pgmap v1874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:27 smithi160 ceph-mon[36715]: pgmap v1874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:28.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:28.806+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:28.806+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:28.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:28.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:29 smithi144 ceph-mon[32129]: pgmap v1875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:29 smithi160 ceph-mon[36715]: pgmap v1875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:29 smithi183 ceph-mon[37273]: pgmap v1875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:31 smithi144 ceph-mon[32129]: pgmap v1876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2434.579853 secs 2024-07-26T11:46:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:31 smithi160 ceph-mon[36715]: pgmap v1876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2434.579853 secs 2024-07-26T11:46:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:31 smithi183 ceph-mon[37273]: pgmap v1876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2434.579853 secs 2024-07-26T11:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:33 smithi144 ceph-mon[32129]: pgmap v1877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:33 smithi160 ceph-mon[36715]: pgmap v1877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:33 smithi183 ceph-mon[37273]: pgmap v1877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:35 smithi144 ceph-mon[32129]: pgmap v1878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:35 smithi160 ceph-mon[36715]: pgmap v1878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:35 smithi183 ceph-mon[37273]: pgmap v1878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:36.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2439.579976 secs 2024-07-26T11:46:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2439.579976 secs 2024-07-26T11:46:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2439.579976 secs 2024-07-26T11:46:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:37 smithi144 ceph-mon[32129]: pgmap v1879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:37.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:37 smithi183 ceph-mon[37273]: pgmap v1879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:37 smithi160 ceph-mon[36715]: pgmap v1879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:39 smithi144 ceph-mon[32129]: pgmap v1880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:39 smithi160 ceph-mon[36715]: pgmap v1880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:39.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:39 smithi183 ceph-mon[37273]: pgmap v1880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:41 smithi144 ceph-mon[32129]: pgmap v1881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2444.580040 secs 2024-07-26T11:46:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:41 smithi160 ceph-mon[36715]: pgmap v1881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2444.580040 secs 2024-07-26T11:46:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:41 smithi183 ceph-mon[37273]: pgmap v1881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2444.580040 secs 2024-07-26T11:46:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:43 smithi144 ceph-mon[32129]: pgmap v1882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:43 smithi160 ceph-mon[36715]: pgmap v1882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:43.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:43.822+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:43.822+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:43.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:43.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:43 smithi183 ceph-mon[37273]: pgmap v1882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:45 smithi144 ceph-mon[32129]: pgmap v1883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:45 smithi160 ceph-mon[36715]: pgmap v1883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:45 smithi183 ceph-mon[37273]: pgmap v1883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2449.580182 secs 2024-07-26T11:46:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2449.580182 secs 2024-07-26T11:46:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2449.580182 secs 2024-07-26T11:46:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:47 smithi144 ceph-mon[32129]: pgmap v1884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:47 smithi160 ceph-mon[36715]: pgmap v1884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:47 smithi183 ceph-mon[37273]: pgmap v1884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:46:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:49 smithi144 ceph-mon[32129]: pgmap v1885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:49 smithi160 ceph-mon[36715]: pgmap v1885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:49 smithi183 ceph-mon[37273]: pgmap v1885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:51 smithi144 ceph-mon[32129]: pgmap v1886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2454.580281 secs 2024-07-26T11:46:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:51 smithi160 ceph-mon[36715]: pgmap v1886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2454.580281 secs 2024-07-26T11:46:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:51 smithi183 ceph-mon[37273]: pgmap v1886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:46:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2454.580281 secs 2024-07-26T11:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: pgmap v1887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: pgmap v1888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:46:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: pgmap v1887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: pgmap v1888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:46:52.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: pgmap v1887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: pgmap v1888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:46:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:46:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:54 smithi160 ceph-mon[36715]: pgmap v1889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:46:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:54 smithi183 ceph-mon[37273]: pgmap v1889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:46:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:54 smithi144 ceph-mon[32129]: pgmap v1889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:46:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:56 smithi160 ceph-mon[36715]: pgmap v1890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:46:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2459.580401 secs 2024-07-26T11:46:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:56 smithi183 ceph-mon[37273]: pgmap v1890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:46:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2459.580401 secs 2024-07-26T11:46:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:56 smithi144 ceph-mon[32129]: pgmap v1890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:46:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2459.580401 secs 2024-07-26T11:46:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:58.839+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:58.840+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:58.840+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:58.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:46:58.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:46:58 smithi183 ceph-mon[37273]: pgmap v1891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:46:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:46:58 smithi144 ceph-mon[32129]: pgmap v1891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:46:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:46:58 smithi160 ceph-mon[36715]: pgmap v1891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:00 smithi183 ceph-mon[37273]: pgmap v1892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:00 smithi144 ceph-mon[32129]: pgmap v1892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:00 smithi160 ceph-mon[36715]: pgmap v1892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:01.523+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:47:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:01.523+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:47:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:01.524+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:47:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:01.524+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:47:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2464.580532 secs 2024-07-26T11:47:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2464.580532 secs 2024-07-26T11:47:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2464.580532 secs 2024-07-26T11:47:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:02 smithi183 ceph-mon[37273]: pgmap v1893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:47:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:02 smithi144 ceph-mon[32129]: pgmap v1893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:47:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:02 smithi160 ceph-mon[36715]: pgmap v1893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:47:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:04 smithi183 ceph-mon[37273]: pgmap v1894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:04 smithi144 ceph-mon[32129]: pgmap v1894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:04 smithi160 ceph-mon[36715]: pgmap v1894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:06.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:06 smithi160 ceph-mon[36715]: pgmap v1895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:06.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2469.580617 secs 2024-07-26T11:47:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:06 smithi183 ceph-mon[37273]: pgmap v1895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2469.580617 secs 2024-07-26T11:47:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:06 smithi144 ceph-mon[32129]: pgmap v1895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2469.580617 secs 2024-07-26T11:47:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:08 smithi183 ceph-mon[37273]: pgmap v1896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:08 smithi144 ceph-mon[32129]: pgmap v1896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:08 smithi160 ceph-mon[36715]: pgmap v1896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:10 smithi183 ceph-mon[37273]: pgmap v1897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:10 smithi144 ceph-mon[32129]: pgmap v1897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:10 smithi160 ceph-mon[36715]: pgmap v1897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2474.580700 secs 2024-07-26T11:47:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2474.580700 secs 2024-07-26T11:47:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2474.580700 secs 2024-07-26T11:47:12.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:12 smithi183 ceph-mon[37273]: pgmap v1898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:12 smithi144 ceph-mon[32129]: pgmap v1898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:12 smithi160 ceph-mon[36715]: pgmap v1898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:13.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:13.856+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:13.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:13.857+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:13.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:13.857+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:13.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:13.857+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:13.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:13.858+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:14 smithi183 ceph-mon[37273]: pgmap v1899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:14 smithi144 ceph-mon[32129]: pgmap v1899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:14 smithi160 ceph-mon[36715]: pgmap v1899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:16 smithi183 ceph-mon[37273]: pgmap v1900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2479.580821 secs 2024-07-26T11:47:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:16 smithi144 ceph-mon[32129]: pgmap v1900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2479.580821 secs 2024-07-26T11:47:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:16 smithi160 ceph-mon[36715]: pgmap v1900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2479.580821 secs 2024-07-26T11:47:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:18 smithi144 ceph-mon[32129]: pgmap v1901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:18 smithi160 ceph-mon[36715]: pgmap v1901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:18 smithi183 ceph-mon[37273]: pgmap v1901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:20 smithi144 ceph-mon[32129]: pgmap v1902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:20 smithi160 ceph-mon[36715]: pgmap v1902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:20 smithi183 ceph-mon[37273]: pgmap v1902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2484.580936 secs 2024-07-26T11:47:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2484.580936 secs 2024-07-26T11:47:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2484.580936 secs 2024-07-26T11:47:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:22 smithi144 ceph-mon[32129]: pgmap v1903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:22 smithi160 ceph-mon[36715]: pgmap v1903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:22 smithi183 ceph-mon[37273]: pgmap v1903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:24 smithi144 ceph-mon[32129]: pgmap v1904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:24 smithi160 ceph-mon[36715]: pgmap v1904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:24 smithi183 ceph-mon[37273]: pgmap v1904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:26 smithi144 ceph-mon[32129]: pgmap v1905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2489.581056 secs 2024-07-26T11:47:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:26 smithi160 ceph-mon[36715]: pgmap v1905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2489.581056 secs 2024-07-26T11:47:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:26 smithi183 ceph-mon[37273]: pgmap v1905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2489.581056 secs 2024-07-26T11:47:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:28.873+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:28.875+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:28.875+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:28.875+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:28.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:28.875+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:28 smithi144 ceph-mon[32129]: pgmap v1906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:29.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:28 smithi160 ceph-mon[36715]: pgmap v1906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:28 smithi183 ceph-mon[37273]: pgmap v1906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:30 smithi144 ceph-mon[32129]: pgmap v1907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:30 smithi160 ceph-mon[36715]: pgmap v1907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:30 smithi183 ceph-mon[37273]: pgmap v1907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2494.581174 secs 2024-07-26T11:47:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2494.581174 secs 2024-07-26T11:47:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2494.581174 secs 2024-07-26T11:47:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:32 smithi144 ceph-mon[32129]: pgmap v1908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:32 smithi160 ceph-mon[36715]: pgmap v1908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:32 smithi183 ceph-mon[37273]: pgmap v1908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:34 smithi144 ceph-mon[32129]: pgmap v1909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:34 smithi160 ceph-mon[36715]: pgmap v1909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:34 smithi183 ceph-mon[37273]: pgmap v1909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:36 smithi144 ceph-mon[32129]: pgmap v1910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2499.581336 secs 2024-07-26T11:47:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:36 smithi160 ceph-mon[36715]: pgmap v1910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2499.581336 secs 2024-07-26T11:47:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:36 smithi183 ceph-mon[37273]: pgmap v1910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2499.581336 secs 2024-07-26T11:47:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:38 smithi144 ceph-mon[32129]: pgmap v1911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:38 smithi160 ceph-mon[36715]: pgmap v1911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:38 smithi183 ceph-mon[37273]: pgmap v1911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:40 smithi144 ceph-mon[32129]: pgmap v1912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:40 smithi160 ceph-mon[36715]: pgmap v1912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:40 smithi183 ceph-mon[37273]: pgmap v1912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2504.581409 secs 2024-07-26T11:47:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2504.581409 secs 2024-07-26T11:47:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2504.581409 secs 2024-07-26T11:47:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:42 smithi160 ceph-mon[36715]: pgmap v1913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:42 smithi183 ceph-mon[37273]: pgmap v1913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:42 smithi144 ceph-mon[32129]: pgmap v1913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:43.885+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:43.886+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:43.886+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:43.886+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:43.886+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:44 smithi160 ceph-mon[36715]: pgmap v1914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:44 smithi183 ceph-mon[37273]: pgmap v1914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:45.182 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:44 smithi144 ceph-mon[32129]: pgmap v1914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:46.531+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:47:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:46.531+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:47:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:46.531+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x7f74b0109990 2024-07-26T11:47:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:46.531+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T11:47:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:46 smithi160 ceph-mon[36715]: pgmap v1915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2509.581509 secs 2024-07-26T11:47:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:46 smithi183 ceph-mon[37273]: pgmap v1915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2509.581509 secs 2024-07-26T11:47:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:46 smithi144 ceph-mon[32129]: pgmap v1915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2509.581509 secs 2024-07-26T11:47:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:48 smithi160 ceph-mon[36715]: pgmap v1916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:48 smithi183 ceph-mon[37273]: pgmap v1916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:48 smithi144 ceph-mon[32129]: pgmap v1916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:49 smithi160 ceph-mon[36715]: pgmap v1917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:49 smithi183 ceph-mon[37273]: pgmap v1917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:49 smithi144 ceph-mon[32129]: pgmap v1917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2514.581581 secs 2024-07-26T11:47:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2514.581581 secs 2024-07-26T11:47:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2514.581581 secs 2024-07-26T11:47:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:52 smithi160 ceph-mon[36715]: pgmap v1918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:52 smithi183 ceph-mon[37273]: pgmap v1918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:52 smithi144 ceph-mon[32129]: pgmap v1918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: pgmap v1919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: pgmap v1920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:53.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:47:53.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: pgmap v1919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: pgmap v1920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:53.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:47:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: pgmap v1919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: pgmap v1920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:53.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:47:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:54 smithi160 ceph-mon[36715]: pgmap v1921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:54 smithi183 ceph-mon[37273]: pgmap v1921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:54 smithi144 ceph-mon[32129]: pgmap v1921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:47:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2519.581704 secs 2024-07-26T11:47:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2519.581704 secs 2024-07-26T11:47:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2519.581704 secs 2024-07-26T11:47:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:57 smithi160 ceph-mon[36715]: pgmap v1922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:47:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:57 smithi183 ceph-mon[37273]: pgmap v1922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:47:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:57 smithi144 ceph-mon[32129]: pgmap v1922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:47:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:58.903+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:58.904+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:58.904+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:58.904+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:47:58.904+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:47:58 smithi160 ceph-mon[36715]: pgmap v1923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:47:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:47:58 smithi183 ceph-mon[37273]: pgmap v1923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:47:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:47:58 smithi144 ceph-mon[32129]: pgmap v1923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:48:01.054 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:00 smithi144 ceph-mon[32129]: pgmap v1924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:00 smithi160 ceph-mon[36715]: pgmap v1924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:00 smithi183 ceph-mon[37273]: pgmap v1924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2524.581823 secs 2024-07-26T11:48:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2524.581823 secs 2024-07-26T11:48:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2524.581823 secs 2024-07-26T11:48:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:02 smithi183 ceph-mon[37273]: pgmap v1925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:48:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:02 smithi144 ceph-mon[32129]: pgmap v1925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:48:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:02 smithi160 ceph-mon[36715]: pgmap v1925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:48:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:04 smithi160 ceph-mon[36715]: pgmap v1926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:04 smithi183 ceph-mon[37273]: pgmap v1926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:04 smithi144 ceph-mon[32129]: pgmap v1926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2529.581961 secs 2024-07-26T11:48:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2529.581961 secs 2024-07-26T11:48:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2529.581961 secs 2024-07-26T11:48:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:07 smithi160 ceph-mon[36715]: pgmap v1927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:07 smithi183 ceph-mon[37273]: pgmap v1927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:07 smithi144 ceph-mon[32129]: pgmap v1927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:08 smithi160 ceph-mon[36715]: pgmap v1928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:08 smithi183 ceph-mon[37273]: pgmap v1928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:08 smithi144 ceph-mon[32129]: pgmap v1928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:10 smithi160 ceph-mon[36715]: pgmap v1929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:10 smithi183 ceph-mon[37273]: pgmap v1929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:10 smithi144 ceph-mon[32129]: pgmap v1929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2534.582099 secs 2024-07-26T11:48:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2534.582099 secs 2024-07-26T11:48:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2534.582099 secs 2024-07-26T11:48:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:12 smithi183 ceph-mon[37273]: pgmap v1930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:12 smithi144 ceph-mon[32129]: pgmap v1930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:12 smithi160 ceph-mon[36715]: pgmap v1930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:13.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:13.920+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:13.920+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:13.921+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:13.921+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:14 smithi160 ceph-mon[36715]: pgmap v1931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:14 smithi183 ceph-mon[37273]: pgmap v1931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:14 smithi144 ceph-mon[32129]: pgmap v1931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2539.582217 secs 2024-07-26T11:48:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2539.582217 secs 2024-07-26T11:48:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2539.582217 secs 2024-07-26T11:48:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:17 smithi160 ceph-mon[36715]: pgmap v1932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:17 smithi183 ceph-mon[37273]: pgmap v1932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:17 smithi144 ceph-mon[32129]: pgmap v1932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:18 smithi160 ceph-mon[36715]: pgmap v1933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:19.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:18 smithi183 ceph-mon[37273]: pgmap v1933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:18 smithi144 ceph-mon[32129]: pgmap v1933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:20 smithi160 ceph-mon[36715]: pgmap v1934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:20 smithi183 ceph-mon[37273]: pgmap v1934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:20 smithi144 ceph-mon[32129]: pgmap v1934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2544.582350 secs 2024-07-26T11:48:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2544.582350 secs 2024-07-26T11:48:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2544.582350 secs 2024-07-26T11:48:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:22 smithi160 ceph-mon[36715]: pgmap v1935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:22 smithi183 ceph-mon[37273]: pgmap v1935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:22 smithi144 ceph-mon[32129]: pgmap v1935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:24 smithi160 ceph-mon[36715]: pgmap v1936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:24 smithi183 ceph-mon[37273]: pgmap v1936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:24 smithi144 ceph-mon[32129]: pgmap v1936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2549.582449 secs 2024-07-26T11:48:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2549.582449 secs 2024-07-26T11:48:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2549.582449 secs 2024-07-26T11:48:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:27 smithi160 ceph-mon[36715]: pgmap v1937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:27 smithi183 ceph-mon[37273]: pgmap v1937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:27 smithi144 ceph-mon[32129]: pgmap v1937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:28.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:28.937+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:28.938+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:28.938+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:28.938+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:28 smithi160 ceph-mon[36715]: pgmap v1938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:28 smithi183 ceph-mon[37273]: pgmap v1938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:28 smithi144 ceph-mon[32129]: pgmap v1938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:30 smithi160 ceph-mon[36715]: pgmap v1939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:30 smithi183 ceph-mon[37273]: pgmap v1939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:30 smithi144 ceph-mon[32129]: pgmap v1939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2554.582545 secs 2024-07-26T11:48:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2554.582545 secs 2024-07-26T11:48:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2554.582545 secs 2024-07-26T11:48:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:32 smithi183 ceph-mon[37273]: pgmap v1940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:32 smithi144 ceph-mon[32129]: pgmap v1940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:32 smithi160 ceph-mon[36715]: pgmap v1940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:34 smithi160 ceph-mon[36715]: pgmap v1941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:34 smithi183 ceph-mon[37273]: pgmap v1941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:34 smithi144 ceph-mon[32129]: pgmap v1941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2559.582639 secs 2024-07-26T11:48:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2559.582639 secs 2024-07-26T11:48:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2559.582639 secs 2024-07-26T11:48:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:37 smithi160 ceph-mon[36715]: pgmap v1942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:37 smithi183 ceph-mon[37273]: pgmap v1942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:37 smithi144 ceph-mon[32129]: pgmap v1942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:38 smithi160 ceph-mon[36715]: pgmap v1943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:38 smithi183 ceph-mon[37273]: pgmap v1943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:38 smithi144 ceph-mon[32129]: pgmap v1943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:40 smithi160 ceph-mon[36715]: pgmap v1944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:40 smithi183 ceph-mon[37273]: pgmap v1944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:40 smithi144 ceph-mon[32129]: pgmap v1944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2564.582777 secs 2024-07-26T11:48:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2564.582777 secs 2024-07-26T11:48:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2564.582777 secs 2024-07-26T11:48:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:42 smithi183 ceph-mon[37273]: pgmap v1945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:42 smithi144 ceph-mon[32129]: pgmap v1945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:42 smithi160 ceph-mon[36715]: pgmap v1945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:43.949+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:43.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:43.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:43.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:43.951+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:44 smithi183 ceph-mon[37273]: pgmap v1946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:44 smithi144 ceph-mon[32129]: pgmap v1946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:44 smithi160 ceph-mon[36715]: pgmap v1946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2569.582921 secs 2024-07-26T11:48:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2569.582921 secs 2024-07-26T11:48:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2569.582921 secs 2024-07-26T11:48:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:47 smithi160 ceph-mon[36715]: pgmap v1947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:47.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:47 smithi144 ceph-mon[32129]: pgmap v1947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:47 smithi183 ceph-mon[37273]: pgmap v1947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:48:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:48 smithi183 ceph-mon[37273]: pgmap v1948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:48 smithi144 ceph-mon[32129]: pgmap v1948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:48 smithi160 ceph-mon[36715]: pgmap v1948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:50 smithi183 ceph-mon[37273]: pgmap v1949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:50 smithi144 ceph-mon[32129]: pgmap v1949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:50 smithi160 ceph-mon[36715]: pgmap v1949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:48:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2574.583042 secs 2024-07-26T11:48:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2574.583042 secs 2024-07-26T11:48:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2574.583042 secs 2024-07-26T11:48:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:52 smithi183 ceph-mon[37273]: pgmap v1950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:52 smithi144 ceph-mon[32129]: pgmap v1950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:52 smithi160 ceph-mon[36715]: pgmap v1950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:54 smithi183 ceph-mon[37273]: pgmap v1951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:54 smithi183 ceph-mon[37273]: pgmap v1952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:48:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:48:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:54 smithi144 ceph-mon[32129]: pgmap v1951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:54 smithi144 ceph-mon[32129]: pgmap v1952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:48:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:48:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:54 smithi160 ceph-mon[36715]: pgmap v1951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:54 smithi160 ceph-mon[36715]: pgmap v1952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:48:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:48:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2579.583161 secs 2024-07-26T11:48:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:56 smithi160 ceph-mon[36715]: pgmap v1953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2579.583161 secs 2024-07-26T11:48:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:56 smithi183 ceph-mon[37273]: pgmap v1953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2579.583161 secs 2024-07-26T11:48:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:56 smithi144 ceph-mon[32129]: pgmap v1953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:48:58.962 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:58.960+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:58.961+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:58.961+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:58.961+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:48:58.962+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:48:58 smithi160 ceph-mon[36715]: pgmap v1954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:48:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:48:58 smithi183 ceph-mon[37273]: pgmap v1954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:48:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:48:58 smithi144 ceph-mon[32129]: pgmap v1954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:49:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:00 smithi160 ceph-mon[36715]: pgmap v1955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:49:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:00 smithi183 ceph-mon[37273]: pgmap v1955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:49:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:00 smithi144 ceph-mon[32129]: pgmap v1955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:49:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2584.583304 secs 2024-07-26T11:49:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2584.583304 secs 2024-07-26T11:49:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2584.583304 secs 2024-07-26T11:49:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:02 smithi160 ceph-mon[36715]: pgmap v1956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:49:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:02 smithi183 ceph-mon[37273]: pgmap v1956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:49:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:02 smithi144 ceph-mon[32129]: pgmap v1956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:49:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:05 smithi160 ceph-mon[36715]: pgmap v1957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:49:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:05 smithi183 ceph-mon[37273]: pgmap v1957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:49:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:05 smithi144 ceph-mon[32129]: pgmap v1957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:49:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2589.583426 secs 2024-07-26T11:49:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2589.583426 secs 2024-07-26T11:49:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2589.583426 secs 2024-07-26T11:49:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:07 smithi160 ceph-mon[36715]: pgmap v1958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:07 smithi183 ceph-mon[37273]: pgmap v1958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:07 smithi144 ceph-mon[32129]: pgmap v1958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:09 smithi160 ceph-mon[36715]: pgmap v1959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:09 smithi183 ceph-mon[37273]: pgmap v1959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:09 smithi144 ceph-mon[32129]: pgmap v1959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:11 smithi160 ceph-mon[36715]: pgmap v1960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2594.583540 secs 2024-07-26T11:49:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:11 smithi183 ceph-mon[37273]: pgmap v1960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2594.583540 secs 2024-07-26T11:49:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:11 smithi144 ceph-mon[32129]: pgmap v1960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2594.583540 secs 2024-07-26T11:49:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:13 smithi160 ceph-mon[36715]: pgmap v1961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:13 smithi183 ceph-mon[37273]: pgmap v1961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:13 smithi144 ceph-mon[32129]: pgmap v1961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:13.978+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:13.979+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:13.979+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:13.979+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:13.979+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:15 smithi160 ceph-mon[36715]: pgmap v1962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:15 smithi183 ceph-mon[37273]: pgmap v1962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:15 smithi144 ceph-mon[32129]: pgmap v1962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2599.583669 secs 2024-07-26T11:49:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2599.583669 secs 2024-07-26T11:49:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2599.583669 secs 2024-07-26T11:49:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:17 smithi160 ceph-mon[36715]: pgmap v1963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:17.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:17 smithi144 ceph-mon[32129]: pgmap v1963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:17 smithi183 ceph-mon[37273]: pgmap v1963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:19 smithi183 ceph-mon[37273]: pgmap v1964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:19 smithi144 ceph-mon[32129]: pgmap v1964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:19 smithi160 ceph-mon[36715]: pgmap v1964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:21 smithi183 ceph-mon[37273]: pgmap v1965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2604.583785 secs 2024-07-26T11:49:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:21 smithi144 ceph-mon[32129]: pgmap v1965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2604.583785 secs 2024-07-26T11:49:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:21 smithi160 ceph-mon[36715]: pgmap v1965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2604.583785 secs 2024-07-26T11:49:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:23 smithi183 ceph-mon[37273]: pgmap v1966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:23 smithi144 ceph-mon[32129]: pgmap v1966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:23 smithi160 ceph-mon[36715]: pgmap v1966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:25 smithi183 ceph-mon[37273]: pgmap v1967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:25 smithi144 ceph-mon[32129]: pgmap v1967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:25 smithi160 ceph-mon[36715]: pgmap v1967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2609.583923 secs 2024-07-26T11:49:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2609.583923 secs 2024-07-26T11:49:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2609.583923 secs 2024-07-26T11:49:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:27 smithi183 ceph-mon[37273]: pgmap v1968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:27 smithi144 ceph-mon[32129]: pgmap v1968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:27 smithi160 ceph-mon[36715]: pgmap v1968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:28.996 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:28.995+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:28.997 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:28.996+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:28.996+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:28.996+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:28.998 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:28.996+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:29 smithi183 ceph-mon[37273]: pgmap v1969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:29 smithi144 ceph-mon[32129]: pgmap v1969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:29 smithi160 ceph-mon[36715]: pgmap v1969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:31 smithi183 ceph-mon[37273]: pgmap v1970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2614.584073 secs 2024-07-26T11:49:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:31 smithi144 ceph-mon[32129]: pgmap v1970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2614.584073 secs 2024-07-26T11:49:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:31 smithi160 ceph-mon[36715]: pgmap v1970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2614.584073 secs 2024-07-26T11:49:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:33 smithi183 ceph-mon[37273]: pgmap v1971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:33 smithi144 ceph-mon[32129]: pgmap v1971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:33 smithi160 ceph-mon[36715]: pgmap v1971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:35 smithi183 ceph-mon[37273]: pgmap v1972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:35 smithi144 ceph-mon[32129]: pgmap v1972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:35 smithi160 ceph-mon[36715]: pgmap v1972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2619.584204 secs 2024-07-26T11:49:36.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2619.584204 secs 2024-07-26T11:49:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2619.584204 secs 2024-07-26T11:49:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:37 smithi183 ceph-mon[37273]: pgmap v1973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:37 smithi144 ceph-mon[32129]: pgmap v1973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:37 smithi160 ceph-mon[36715]: pgmap v1973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:39 smithi144 ceph-mon[32129]: pgmap v1974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:39 smithi160 ceph-mon[36715]: pgmap v1974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:39 smithi183 ceph-mon[37273]: pgmap v1974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:41 smithi144 ceph-mon[32129]: pgmap v1975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2624.584330 secs 2024-07-26T11:49:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:41 smithi160 ceph-mon[36715]: pgmap v1975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2624.584330 secs 2024-07-26T11:49:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:41 smithi183 ceph-mon[37273]: pgmap v1975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2624.584330 secs 2024-07-26T11:49:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:43 smithi144 ceph-mon[32129]: pgmap v1976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:43 smithi160 ceph-mon[36715]: pgmap v1976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:43 smithi183 ceph-mon[37273]: pgmap v1976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:44.011+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:44.012+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:44.012+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:44.012+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:44.012+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:45 smithi144 ceph-mon[32129]: pgmap v1977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:45 smithi160 ceph-mon[36715]: pgmap v1977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:45 smithi183 ceph-mon[37273]: pgmap v1977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2629.584448 secs 2024-07-26T11:49:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2629.584448 secs 2024-07-26T11:49:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2629.584448 secs 2024-07-26T11:49:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:47 smithi144 ceph-mon[32129]: pgmap v1978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:47 smithi160 ceph-mon[36715]: pgmap v1978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:47 smithi183 ceph-mon[37273]: pgmap v1978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:49 smithi144 ceph-mon[32129]: pgmap v1979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:49 smithi160 ceph-mon[36715]: pgmap v1979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:49 smithi183 ceph-mon[37273]: pgmap v1979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:51 smithi144 ceph-mon[32129]: pgmap v1980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2634.584561 secs 2024-07-26T11:49:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:51 smithi160 ceph-mon[36715]: pgmap v1980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2634.584561 secs 2024-07-26T11:49:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:51 smithi183 ceph-mon[37273]: pgmap v1980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2634.584561 secs 2024-07-26T11:49:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:53 smithi144 ceph-mon[32129]: pgmap v1981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:53 smithi160 ceph-mon[36715]: pgmap v1981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:53 smithi183 ceph-mon[37273]: pgmap v1981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:49:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:55 smithi160 ceph-mon[36715]: pgmap v1982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:55 smithi183 ceph-mon[37273]: pgmap v1982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:55 smithi144 ceph-mon[32129]: pgmap v1982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2639.584675 secs 2024-07-26T11:49:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2639.584675 secs 2024-07-26T11:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2639.584675 secs 2024-07-26T11:49:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:49:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:57 smithi160 ceph-mon[36715]: pgmap v1983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:57 smithi160 ceph-mon[36715]: pgmap v1984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:49:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:57 smithi183 ceph-mon[37273]: pgmap v1983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:57 smithi183 ceph-mon[37273]: pgmap v1984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:49:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:57 smithi144 ceph-mon[32129]: pgmap v1983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:49:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:57 smithi144 ceph-mon[32129]: pgmap v1984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:49:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:59.028+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:59.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:59.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:59.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:49:59.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:49:59 smithi160 ceph-mon[36715]: pgmap v1985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:49:59 smithi183 ceph-mon[37273]: pgmap v1985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:49:59 smithi144 ceph-mon[32129]: pgmap v1985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:01 smithi160 ceph-mon[36715]: pgmap v1986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2644.584789 secs 2024-07-26T11:50:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:01 smithi183 ceph-mon[37273]: pgmap v1986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2644.584789 secs 2024-07-26T11:50:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:01 smithi144 ceph-mon[32129]: pgmap v1986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2644.584789 secs 2024-07-26T11:50:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:02 smithi183 ceph-mon[37273]: pgmap v1987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:02 smithi144 ceph-mon[32129]: pgmap v1987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:02 smithi160 ceph-mon[36715]: pgmap v1987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:04 smithi183 ceph-mon[37273]: pgmap v1988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:04 smithi144 ceph-mon[32129]: pgmap v1988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:04 smithi160 ceph-mon[36715]: pgmap v1988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:07.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2649.584931 secs 2024-07-26T11:50:07.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:06 smithi160 ceph-mon[36715]: pgmap v1989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2649.584931 secs 2024-07-26T11:50:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:06 smithi183 ceph-mon[37273]: pgmap v1989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2649.584931 secs 2024-07-26T11:50:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:06 smithi144 ceph-mon[32129]: pgmap v1989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:08 smithi183 ceph-mon[37273]: pgmap v1990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:08 smithi144 ceph-mon[32129]: pgmap v1990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:08 smithi160 ceph-mon[36715]: pgmap v1990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:10 smithi183 ceph-mon[37273]: pgmap v1991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:10 smithi144 ceph-mon[32129]: pgmap v1991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:10 smithi160 ceph-mon[36715]: pgmap v1991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2654.585069 secs 2024-07-26T11:50:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2654.585069 secs 2024-07-26T11:50:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2654.585069 secs 2024-07-26T11:50:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:12 smithi183 ceph-mon[37273]: pgmap v1992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:12 smithi144 ceph-mon[32129]: pgmap v1992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:12 smithi160 ceph-mon[36715]: pgmap v1992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:14.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:14.040+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:14.040+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:14.040+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:14.040+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:14 smithi183 ceph-mon[37273]: pgmap v1993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:14 smithi144 ceph-mon[32129]: pgmap v1993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:14 smithi160 ceph-mon[36715]: pgmap v1993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2659.585192 secs 2024-07-26T11:50:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:16 smithi183 ceph-mon[37273]: pgmap v1994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2659.585192 secs 2024-07-26T11:50:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:16 smithi144 ceph-mon[32129]: pgmap v1994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2659.585192 secs 2024-07-26T11:50:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:16 smithi160 ceph-mon[36715]: pgmap v1994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:18 smithi183 ceph-mon[37273]: pgmap v1995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:18 smithi144 ceph-mon[32129]: pgmap v1995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:18 smithi160 ceph-mon[36715]: pgmap v1995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:20 smithi183 ceph-mon[37273]: pgmap v1996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:20 smithi144 ceph-mon[32129]: pgmap v1996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:20 smithi160 ceph-mon[36715]: pgmap v1996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2664.585338 secs 2024-07-26T11:50:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2664.585338 secs 2024-07-26T11:50:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2664.585338 secs 2024-07-26T11:50:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:22 smithi144 ceph-mon[32129]: pgmap v1997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:22 smithi160 ceph-mon[36715]: pgmap v1997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:22 smithi183 ceph-mon[37273]: pgmap v1997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:24 smithi144 ceph-mon[32129]: pgmap v1998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:24 smithi160 ceph-mon[36715]: pgmap v1998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:24 smithi183 ceph-mon[37273]: pgmap v1998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2669.585445 secs 2024-07-26T11:50:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:26 smithi144 ceph-mon[32129]: pgmap v1999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2669.585445 secs 2024-07-26T11:50:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:26 smithi160 ceph-mon[36715]: pgmap v1999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2669.585445 secs 2024-07-26T11:50:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:26 smithi183 ceph-mon[37273]: pgmap v1999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:29.047 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:29.045+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:29.046+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:29.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:29.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:29.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:28 smithi144 ceph-mon[32129]: pgmap v2000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:28 smithi160 ceph-mon[36715]: pgmap v2000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:28 smithi183 ceph-mon[37273]: pgmap v2000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:30 smithi144 ceph-mon[32129]: pgmap v2001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:30 smithi160 ceph-mon[36715]: pgmap v2001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:30 smithi183 ceph-mon[37273]: pgmap v2001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2674.585565 secs 2024-07-26T11:50:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2674.585565 secs 2024-07-26T11:50:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2674.585565 secs 2024-07-26T11:50:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:32 smithi144 ceph-mon[32129]: pgmap v2002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:32 smithi160 ceph-mon[36715]: pgmap v2002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:32 smithi183 ceph-mon[37273]: pgmap v2002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:34 smithi144 ceph-mon[32129]: pgmap v2003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:34 smithi160 ceph-mon[36715]: pgmap v2003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:34 smithi183 ceph-mon[37273]: pgmap v2003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2679.585683 secs 2024-07-26T11:50:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:36 smithi144 ceph-mon[32129]: pgmap v2004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2679.585683 secs 2024-07-26T11:50:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:36 smithi160 ceph-mon[36715]: pgmap v2004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2679.585683 secs 2024-07-26T11:50:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:36 smithi183 ceph-mon[37273]: pgmap v2004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:38 smithi144 ceph-mon[32129]: pgmap v2005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:38 smithi160 ceph-mon[36715]: pgmap v2005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:38 smithi183 ceph-mon[37273]: pgmap v2005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:40 smithi144 ceph-mon[32129]: pgmap v2006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:40 smithi160 ceph-mon[36715]: pgmap v2006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:40 smithi183 ceph-mon[37273]: pgmap v2006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2684.585777 secs 2024-07-26T11:50:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2684.585777 secs 2024-07-26T11:50:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2684.585777 secs 2024-07-26T11:50:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:42 smithi160 ceph-mon[36715]: pgmap v2007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:42 smithi183 ceph-mon[37273]: pgmap v2007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:42 smithi144 ceph-mon[32129]: pgmap v2007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:44.062+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:44.063+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:44.063+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:44.063+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:44.063+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:44 smithi160 ceph-mon[36715]: pgmap v2008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:44 smithi183 ceph-mon[37273]: pgmap v2008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:44 smithi144 ceph-mon[32129]: pgmap v2008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900c1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900c1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900c1d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f74a8031470 tx=0x7f74a000e650 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900c1d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f74a8031470 tx=0x7f74a000e650 comp rx=0 tx=0).stop 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900c1d90 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 msgr2=0x7f74900c1d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b3d40 0x7f74900c1d90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:50:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.096+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:50:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.097+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.098+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c22d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a804d070 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:50:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.564+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:50:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.564+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:50:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.564+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:50:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:46.564+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x7f74b0109990 2024-07-26T11:50:47.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2689.585860 secs 2024-07-26T11:50:47.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:46 smithi144 ceph-mon[32129]: pgmap v2009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2689.585860 secs 2024-07-26T11:50:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:46 smithi160 ceph-mon[36715]: pgmap v2009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2689.585860 secs 2024-07-26T11:50:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:46 smithi183 ceph-mon[37273]: pgmap v2009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:48 smithi160 ceph-mon[36715]: pgmap v2010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:48 smithi183 ceph-mon[37273]: pgmap v2010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:48 smithi144 ceph-mon[32129]: pgmap v2010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:50 smithi160 ceph-mon[36715]: pgmap v2011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:50 smithi183 ceph-mon[37273]: pgmap v2011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:50 smithi144 ceph-mon[32129]: pgmap v2011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2694.585999 secs 2024-07-26T11:50:52.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2694.585999 secs 2024-07-26T11:50:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2694.585999 secs 2024-07-26T11:50:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:53 smithi160 ceph-mon[36715]: pgmap v2012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:53 smithi183 ceph-mon[37273]: pgmap v2012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:53 smithi144 ceph-mon[32129]: pgmap v2012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:50:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:55 smithi160 ceph-mon[36715]: pgmap v2013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:55 smithi183 ceph-mon[37273]: pgmap v2013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:55 smithi144 ceph-mon[32129]: pgmap v2013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2699.586125 secs 2024-07-26T11:50:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2699.586125 secs 2024-07-26T11:50:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2699.586125 secs 2024-07-26T11:50:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:57 smithi160 ceph-mon[36715]: pgmap v2014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:57 smithi183 ceph-mon[37273]: pgmap v2014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:57 smithi144 ceph-mon[32129]: pgmap v2014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:50:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:59.079+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:59.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:59.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:59.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:50:59.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: pgmap v2015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:50:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: pgmap v2015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:50:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: pgmap v2015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:50:59.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:50:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:01 smithi160 ceph-mon[36715]: pgmap v2016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2704.586205 secs 2024-07-26T11:51:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:01 smithi183 ceph-mon[37273]: pgmap v2016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2704.586205 secs 2024-07-26T11:51:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:01 smithi144 ceph-mon[32129]: pgmap v2016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2704.586205 secs 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: pgmap v2017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: pgmap v2018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: pgmap v2017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: pgmap v2018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: pgmap v2017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: pgmap v2018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:02.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:51:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:04 smithi144 ceph-mon[32129]: pgmap v2019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:04 smithi160 ceph-mon[36715]: pgmap v2019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:04 smithi183 ceph-mon[37273]: pgmap v2019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:06 smithi144 ceph-mon[32129]: pgmap v2020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2709.586341 secs 2024-07-26T11:51:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:06 smithi160 ceph-mon[36715]: pgmap v2020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2709.586341 secs 2024-07-26T11:51:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:06 smithi183 ceph-mon[37273]: pgmap v2020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2709.586341 secs 2024-07-26T11:51:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:08 smithi160 ceph-mon[36715]: pgmap v2021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:08 smithi183 ceph-mon[37273]: pgmap v2021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:08 smithi144 ceph-mon[32129]: pgmap v2021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:10 smithi160 ceph-mon[36715]: pgmap v2022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:10 smithi183 ceph-mon[37273]: pgmap v2022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:10 smithi144 ceph-mon[32129]: pgmap v2022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:11.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2714.586383 secs 2024-07-26T11:51:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2714.586383 secs 2024-07-26T11:51:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2714.586383 secs 2024-07-26T11:51:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:12 smithi160 ceph-mon[36715]: pgmap v2023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:51:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:12 smithi183 ceph-mon[37273]: pgmap v2023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:51:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:12 smithi144 ceph-mon[32129]: pgmap v2023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:51:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:14.096+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:14.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:14.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:14.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:14.098+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:14 smithi160 ceph-mon[36715]: pgmap v2024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:14 smithi183 ceph-mon[37273]: pgmap v2024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:14 smithi144 ceph-mon[32129]: pgmap v2024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:16 smithi160 ceph-mon[36715]: pgmap v2025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2719.586500 secs 2024-07-26T11:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:16 smithi183 ceph-mon[37273]: pgmap v2025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2719.586500 secs 2024-07-26T11:51:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:16 smithi144 ceph-mon[32129]: pgmap v2025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2719.586500 secs 2024-07-26T11:51:18.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:18 smithi144 ceph-mon[32129]: pgmap v2026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:18 smithi160 ceph-mon[36715]: pgmap v2026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:18 smithi183 ceph-mon[37273]: pgmap v2026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:20 smithi160 ceph-mon[36715]: pgmap v2027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:20 smithi183 ceph-mon[37273]: pgmap v2027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:20 smithi144 ceph-mon[32129]: pgmap v2027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2724.586618 secs 2024-07-26T11:51:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2724.586618 secs 2024-07-26T11:51:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2724.586618 secs 2024-07-26T11:51:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:22 smithi160 ceph-mon[36715]: pgmap v2028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:22 smithi183 ceph-mon[37273]: pgmap v2028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:22 smithi144 ceph-mon[32129]: pgmap v2028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:24 smithi160 ceph-mon[36715]: pgmap v2029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:24 smithi183 ceph-mon[37273]: pgmap v2029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:24 smithi144 ceph-mon[32129]: pgmap v2029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:26 smithi160 ceph-mon[36715]: pgmap v2030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2729.586744 secs 2024-07-26T11:51:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:26 smithi183 ceph-mon[37273]: pgmap v2030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2729.586744 secs 2024-07-26T11:51:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:26 smithi144 ceph-mon[32129]: pgmap v2030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2729.586744 secs 2024-07-26T11:51:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:28 smithi160 ceph-mon[36715]: pgmap v2031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:29.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:29.115+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:29.115+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:29.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:29.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:28 smithi183 ceph-mon[37273]: pgmap v2031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:28 smithi144 ceph-mon[32129]: pgmap v2031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:30 smithi160 ceph-mon[36715]: pgmap v2032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:30 smithi183 ceph-mon[37273]: pgmap v2032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:30 smithi144 ceph-mon[32129]: pgmap v2032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2734.586836 secs 2024-07-26T11:51:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2734.586836 secs 2024-07-26T11:51:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2734.586836 secs 2024-07-26T11:51:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:32 smithi160 ceph-mon[36715]: pgmap v2033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:32 smithi183 ceph-mon[37273]: pgmap v2033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:32 smithi144 ceph-mon[32129]: pgmap v2033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:34 smithi160 ceph-mon[36715]: pgmap v2034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:34 smithi183 ceph-mon[37273]: pgmap v2034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:34 smithi144 ceph-mon[32129]: pgmap v2034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:36 smithi160 ceph-mon[36715]: pgmap v2035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2739.586957 secs 2024-07-26T11:51:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:36 smithi183 ceph-mon[37273]: pgmap v2035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2739.586957 secs 2024-07-26T11:51:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:36 smithi144 ceph-mon[32129]: pgmap v2035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2739.586957 secs 2024-07-26T11:51:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:38 smithi160 ceph-mon[36715]: pgmap v2036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:38 smithi183 ceph-mon[37273]: pgmap v2036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:39.166 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:38 smithi144 ceph-mon[32129]: pgmap v2036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:40 smithi160 ceph-mon[36715]: pgmap v2037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:40 smithi183 ceph-mon[37273]: pgmap v2037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:40 smithi144 ceph-mon[32129]: pgmap v2037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2744.587099 secs 2024-07-26T11:51:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2744.587099 secs 2024-07-26T11:51:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2744.587099 secs 2024-07-26T11:51:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:42 smithi160 ceph-mon[36715]: pgmap v2038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:42 smithi183 ceph-mon[37273]: pgmap v2038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:42 smithi144 ceph-mon[32129]: pgmap v2038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:44.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:44.131+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:44.131+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:44.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:44.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:43 smithi183 ceph-mon[37273]: pgmap v2039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:43 smithi144 ceph-mon[32129]: pgmap v2039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:43 smithi160 ceph-mon[36715]: pgmap v2039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:46 smithi160 ceph-mon[36715]: pgmap v2040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2749.587232 secs 2024-07-26T11:51:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:46 smithi183 ceph-mon[37273]: pgmap v2040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2749.587232 secs 2024-07-26T11:51:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:46 smithi144 ceph-mon[32129]: pgmap v2040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2749.587232 secs 2024-07-26T11:51:48.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:48 smithi144 ceph-mon[32129]: pgmap v2041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:48 smithi160 ceph-mon[36715]: pgmap v2041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:48 smithi183 ceph-mon[37273]: pgmap v2041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:50 smithi160 ceph-mon[36715]: pgmap v2042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:50 smithi183 ceph-mon[37273]: pgmap v2042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:50 smithi144 ceph-mon[32129]: pgmap v2042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2754.587309 secs 2024-07-26T11:51:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2754.587309 secs 2024-07-26T11:51:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2754.587309 secs 2024-07-26T11:51:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:52 smithi160 ceph-mon[36715]: pgmap v2043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:52 smithi183 ceph-mon[37273]: pgmap v2043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:52 smithi144 ceph-mon[32129]: pgmap v2043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:51:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:54 smithi160 ceph-mon[36715]: pgmap v2044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:54 smithi183 ceph-mon[37273]: pgmap v2044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:54 smithi144 ceph-mon[32129]: pgmap v2044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:51:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:56 smithi160 ceph-mon[36715]: pgmap v2045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2759.587426 secs 2024-07-26T11:51:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:56 smithi183 ceph-mon[37273]: pgmap v2045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2759.587426 secs 2024-07-26T11:51:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:56 smithi144 ceph-mon[32129]: pgmap v2045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2759.587426 secs 2024-07-26T11:51:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:51:58 smithi160 ceph-mon[36715]: pgmap v2046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:51:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:51:58 smithi183 ceph-mon[37273]: pgmap v2046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:51:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:51:58 smithi144 ceph-mon[32129]: pgmap v2046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:51:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:59.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:59.148+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:59.148+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:59.148+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:59.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:51:59.148+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:00 smithi160 ceph-mon[36715]: pgmap v2047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:52:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:00 smithi183 ceph-mon[37273]: pgmap v2047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:52:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:00 smithi144 ceph-mon[32129]: pgmap v2047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:52:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2764.587548 secs 2024-07-26T11:52:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2764.587548 secs 2024-07-26T11:52:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2764.587548 secs 2024-07-26T11:52:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:02 smithi160 ceph-mon[36715]: pgmap v2048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:02 smithi183 ceph-mon[37273]: pgmap v2048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:02 smithi144 ceph-mon[32129]: pgmap v2048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:03 smithi183 ceph-mon[37273]: pgmap v2049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:52:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:03 smithi144 ceph-mon[32129]: pgmap v2049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:52:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:03 smithi160 ceph-mon[36715]: pgmap v2049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:52:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:05 smithi160 ceph-mon[36715]: pgmap v2050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:05 smithi183 ceph-mon[37273]: pgmap v2050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:05 smithi144 ceph-mon[32129]: pgmap v2050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2769.587673 secs 2024-07-26T11:52:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2769.587673 secs 2024-07-26T11:52:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2769.587673 secs 2024-07-26T11:52:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:07 smithi160 ceph-mon[36715]: pgmap v2051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:52:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:07 smithi183 ceph-mon[37273]: pgmap v2051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:52:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:07 smithi144 ceph-mon[32129]: pgmap v2051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:52:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:09 smithi160 ceph-mon[36715]: pgmap v2052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:52:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:09 smithi183 ceph-mon[37273]: pgmap v2052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:52:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:09 smithi144 ceph-mon[32129]: pgmap v2052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:52:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2774.587755 secs 2024-07-26T11:52:12.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:11 smithi160 ceph-mon[36715]: pgmap v2053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:52:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2774.587755 secs 2024-07-26T11:52:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:11 smithi183 ceph-mon[37273]: pgmap v2053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:52:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2774.587755 secs 2024-07-26T11:52:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:11 smithi144 ceph-mon[32129]: pgmap v2053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:52:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:13 smithi160 ceph-mon[36715]: pgmap v2054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:52:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:13 smithi183 ceph-mon[37273]: pgmap v2054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:52:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:14.164+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:14.165+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:14.165+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:14.165+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:14.165+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:13 smithi144 ceph-mon[32129]: pgmap v2054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:52:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:15 smithi160 ceph-mon[36715]: pgmap v2055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:15 smithi183 ceph-mon[37273]: pgmap v2055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:15 smithi144 ceph-mon[32129]: pgmap v2055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2779.587896 secs 2024-07-26T11:52:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2779.587896 secs 2024-07-26T11:52:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2779.587896 secs 2024-07-26T11:52:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:17 smithi160 ceph-mon[36715]: pgmap v2056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:18.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:17 smithi144 ceph-mon[32129]: pgmap v2056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:17 smithi183 ceph-mon[37273]: pgmap v2056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:19 smithi160 ceph-mon[36715]: pgmap v2057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:19 smithi183 ceph-mon[37273]: pgmap v2057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:19 smithi144 ceph-mon[32129]: pgmap v2057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:20 smithi183 ceph-mon[37273]: pgmap v2058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:21.155 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:20 smithi160 ceph-mon[36715]: pgmap v2058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:20 smithi144 ceph-mon[32129]: pgmap v2058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2784.588019 secs 2024-07-26T11:52:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2784.588019 secs 2024-07-26T11:52:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2784.588019 secs 2024-07-26T11:52:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:22 smithi183 ceph-mon[37273]: pgmap v2059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:22 smithi144 ceph-mon[32129]: pgmap v2059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:22 smithi160 ceph-mon[36715]: pgmap v2059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:25 smithi160 ceph-mon[36715]: pgmap v2060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:25 smithi183 ceph-mon[37273]: pgmap v2060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:25 smithi144 ceph-mon[32129]: pgmap v2060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2789.588144 secs 2024-07-26T11:52:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2789.588144 secs 2024-07-26T11:52:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2789.588144 secs 2024-07-26T11:52:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:27 smithi160 ceph-mon[36715]: pgmap v2061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:27 smithi183 ceph-mon[37273]: pgmap v2061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:27 smithi144 ceph-mon[32129]: pgmap v2061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:29.169+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:29.170+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:29.170+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:29.170+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:29.170+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:29 smithi160 ceph-mon[36715]: pgmap v2062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:29 smithi183 ceph-mon[37273]: pgmap v2062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:29 smithi144 ceph-mon[32129]: pgmap v2062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2794.588312 secs 2024-07-26T11:52:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:31 smithi160 ceph-mon[36715]: pgmap v2063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2794.588312 secs 2024-07-26T11:52:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:31 smithi183 ceph-mon[37273]: pgmap v2063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2794.588312 secs 2024-07-26T11:52:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:31 smithi144 ceph-mon[32129]: pgmap v2063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:33 smithi160 ceph-mon[36715]: pgmap v2064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:33 smithi183 ceph-mon[37273]: pgmap v2064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:33 smithi144 ceph-mon[32129]: pgmap v2064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:35 smithi160 ceph-mon[36715]: pgmap v2065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:35 smithi183 ceph-mon[37273]: pgmap v2065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:35 smithi144 ceph-mon[32129]: pgmap v2065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2799.588431 secs 2024-07-26T11:52:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2799.588431 secs 2024-07-26T11:52:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2799.588431 secs 2024-07-26T11:52:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:37 smithi160 ceph-mon[36715]: pgmap v2066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:37 smithi183 ceph-mon[37273]: pgmap v2066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:37 smithi144 ceph-mon[32129]: pgmap v2066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:39 smithi160 ceph-mon[36715]: pgmap v2067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:39 smithi183 ceph-mon[37273]: pgmap v2067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:39 smithi144 ceph-mon[32129]: pgmap v2067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2804.588544 secs 2024-07-26T11:52:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:41 smithi160 ceph-mon[36715]: pgmap v2068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2804.588544 secs 2024-07-26T11:52:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:41 smithi183 ceph-mon[37273]: pgmap v2068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2804.588544 secs 2024-07-26T11:52:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:41 smithi144 ceph-mon[32129]: pgmap v2068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:43 smithi160 ceph-mon[36715]: pgmap v2069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:52:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:43 smithi183 ceph-mon[37273]: pgmap v2069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:52:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:44.185+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:44.186+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:44.186+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:44.186+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:44.186+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:43 smithi144 ceph-mon[32129]: pgmap v2069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:52:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:44 smithi183 ceph-mon[37273]: pgmap v2070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:52:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:44 smithi144 ceph-mon[32129]: pgmap v2070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:52:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:44 smithi160 ceph-mon[36715]: pgmap v2070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:52:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2809.588664 secs 2024-07-26T11:52:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2809.588664 secs 2024-07-26T11:52:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2809.588664 secs 2024-07-26T11:52:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:47 smithi183 ceph-mon[37273]: pgmap v2071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:47 smithi144 ceph-mon[32129]: pgmap v2071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:47 smithi160 ceph-mon[36715]: pgmap v2071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:49 smithi160 ceph-mon[36715]: pgmap v2072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:49 smithi183 ceph-mon[37273]: pgmap v2072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:50.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:49 smithi144 ceph-mon[32129]: pgmap v2072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2814.588781 secs 2024-07-26T11:52:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:51 smithi160 ceph-mon[36715]: pgmap v2073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2814.588781 secs 2024-07-26T11:52:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:51 smithi183 ceph-mon[37273]: pgmap v2073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2814.588781 secs 2024-07-26T11:52:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:51 smithi144 ceph-mon[32129]: pgmap v2073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:53 smithi160 ceph-mon[36715]: pgmap v2074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:53 smithi183 ceph-mon[37273]: pgmap v2074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:53 smithi144 ceph-mon[32129]: pgmap v2074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:55 smithi160 ceph-mon[36715]: pgmap v2075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:55 smithi183 ceph-mon[37273]: pgmap v2075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:55 smithi144 ceph-mon[32129]: pgmap v2075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2819.588921 secs 2024-07-26T11:52:57.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2819.588921 secs 2024-07-26T11:52:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2819.588921 secs 2024-07-26T11:52:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:57 smithi160 ceph-mon[36715]: pgmap v2076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:57 smithi183 ceph-mon[37273]: pgmap v2076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:57 smithi144 ceph-mon[32129]: pgmap v2076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:52:59.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:59.202+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:59.203+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:59.203+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:59.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:52:59.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:52:59 smithi160 ceph-mon[36715]: pgmap v2077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:52:59 smithi183 ceph-mon[37273]: pgmap v2077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:52:59 smithi144 ceph-mon[32129]: pgmap v2077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2824.589040 secs 2024-07-26T11:53:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:01 smithi160 ceph-mon[36715]: pgmap v2078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2824.589040 secs 2024-07-26T11:53:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:01 smithi183 ceph-mon[37273]: pgmap v2078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2824.589040 secs 2024-07-26T11:53:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:01 smithi144 ceph-mon[32129]: pgmap v2078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:03 smithi160 ceph-mon[36715]: pgmap v2079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:03 smithi183 ceph-mon[37273]: pgmap v2079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:03 smithi144 ceph-mon[32129]: pgmap v2079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:53:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:53:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:53:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:05 smithi183 ceph-mon[37273]: pgmap v2080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:05 smithi144 ceph-mon[32129]: pgmap v2080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:05 smithi160 ceph-mon[36715]: pgmap v2080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2829.589169 secs 2024-07-26T11:53:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:06 smithi183 ceph-mon[37273]: pgmap v2081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2829.589169 secs 2024-07-26T11:53:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:06 smithi144 ceph-mon[32129]: pgmap v2081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2829.589169 secs 2024-07-26T11:53:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:06 smithi160 ceph-mon[36715]: pgmap v2081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:08 smithi144 ceph-mon[32129]: pgmap v2082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:53:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:08 smithi160 ceph-mon[36715]: pgmap v2082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:53:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:08 smithi183 ceph-mon[37273]: pgmap v2082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:53:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:10 smithi144 ceph-mon[32129]: pgmap v2083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:53:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:10 smithi160 ceph-mon[36715]: pgmap v2083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:53:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:10 smithi183 ceph-mon[37273]: pgmap v2083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:53:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2834.589286 secs 2024-07-26T11:53:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2834.589286 secs 2024-07-26T11:53:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2834.589286 secs 2024-07-26T11:53:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:12 smithi144 ceph-mon[32129]: pgmap v2084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:53:13.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:12 smithi183 ceph-mon[37273]: pgmap v2084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:53:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:12 smithi160 ceph-mon[36715]: pgmap v2084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:53:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:14.219+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:14.220+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:14.220+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:14.220+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:14.221+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:14 smithi144 ceph-mon[32129]: pgmap v2085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:14 smithi160 ceph-mon[36715]: pgmap v2085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:14 smithi183 ceph-mon[37273]: pgmap v2085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:53:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2839.589392 secs 2024-07-26T11:53:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:16 smithi144 ceph-mon[32129]: pgmap v2086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2839.589392 secs 2024-07-26T11:53:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:16 smithi160 ceph-mon[36715]: pgmap v2086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2839.589392 secs 2024-07-26T11:53:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:16 smithi183 ceph-mon[37273]: pgmap v2086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:18 smithi144 ceph-mon[32129]: pgmap v2087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:18 smithi160 ceph-mon[36715]: pgmap v2087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:18 smithi183 ceph-mon[37273]: pgmap v2087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:20 smithi144 ceph-mon[32129]: pgmap v2088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:20 smithi160 ceph-mon[36715]: pgmap v2088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:20 smithi183 ceph-mon[37273]: pgmap v2088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2844.589511 secs 2024-07-26T11:53:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2844.589511 secs 2024-07-26T11:53:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2844.589511 secs 2024-07-26T11:53:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:23 smithi160 ceph-mon[36715]: pgmap v2089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:23 smithi183 ceph-mon[37273]: pgmap v2089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:23 smithi144 ceph-mon[32129]: pgmap v2089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:25 smithi160 ceph-mon[36715]: pgmap v2090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:25 smithi183 ceph-mon[37273]: pgmap v2090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:25 smithi144 ceph-mon[32129]: pgmap v2090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2849.589592 secs 2024-07-26T11:53:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2849.589592 secs 2024-07-26T11:53:27.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2849.589592 secs 2024-07-26T11:53:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:27 smithi183 ceph-mon[37273]: pgmap v2091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:27 smithi144 ceph-mon[32129]: pgmap v2091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:27 smithi160 ceph-mon[36715]: pgmap v2091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:29.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:28 smithi183 ceph-mon[37273]: pgmap v2092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:28 smithi144 ceph-mon[32129]: pgmap v2092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:29.236+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:29.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:29.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:29.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:29.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:28 smithi160 ceph-mon[36715]: pgmap v2092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:30 smithi144 ceph-mon[32129]: pgmap v2093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:30 smithi160 ceph-mon[36715]: pgmap v2093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:31.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:30 smithi183 ceph-mon[37273]: pgmap v2093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2854.589715 secs 2024-07-26T11:53:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2854.589715 secs 2024-07-26T11:53:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2854.589715 secs 2024-07-26T11:53:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:32 smithi144 ceph-mon[32129]: pgmap v2094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:32 smithi160 ceph-mon[36715]: pgmap v2094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:32 smithi183 ceph-mon[37273]: pgmap v2094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:34 smithi144 ceph-mon[32129]: pgmap v2095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:34 smithi160 ceph-mon[36715]: pgmap v2095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:34 smithi183 ceph-mon[37273]: pgmap v2095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2859.589841 secs 2024-07-26T11:53:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:36 smithi144 ceph-mon[32129]: pgmap v2096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2859.589841 secs 2024-07-26T11:53:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:36 smithi160 ceph-mon[36715]: pgmap v2096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2859.589841 secs 2024-07-26T11:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:36 smithi183 ceph-mon[37273]: pgmap v2096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:38 smithi160 ceph-mon[36715]: pgmap v2097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:38 smithi183 ceph-mon[37273]: pgmap v2097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:38 smithi144 ceph-mon[32129]: pgmap v2097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:40 smithi160 ceph-mon[36715]: pgmap v2098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:40 smithi183 ceph-mon[37273]: pgmap v2098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:40 smithi144 ceph-mon[32129]: pgmap v2098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2864.589962 secs 2024-07-26T11:53:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2864.589962 secs 2024-07-26T11:53:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2864.589962 secs 2024-07-26T11:53:43.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:42 smithi183 ceph-mon[37273]: pgmap v2099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:42 smithi160 ceph-mon[36715]: pgmap v2099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:42 smithi144 ceph-mon[32129]: pgmap v2099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:44.253+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:44.254+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:44.254+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:44.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:44.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:44 smithi160 ceph-mon[36715]: pgmap v2100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:44 smithi183 ceph-mon[37273]: pgmap v2100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:44 smithi144 ceph-mon[32129]: pgmap v2100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:46.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:46.596+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:53:46.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:46.596+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:53:46.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:46.597+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:53:46.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:46.597+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T11:53:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2869.590085 secs 2024-07-26T11:53:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:47 smithi160 ceph-mon[36715]: pgmap v2101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2869.590085 secs 2024-07-26T11:53:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:46 smithi183 ceph-mon[37273]: pgmap v2101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2869.590085 secs 2024-07-26T11:53:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:46 smithi144 ceph-mon[32129]: pgmap v2101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:49 smithi160 ceph-mon[36715]: pgmap v2102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:49 smithi183 ceph-mon[37273]: pgmap v2102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:49 smithi144 ceph-mon[32129]: pgmap v2102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:51.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:51 smithi144 ceph-mon[32129]: pgmap v2103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:51 smithi160 ceph-mon[36715]: pgmap v2103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:51 smithi183 ceph-mon[37273]: pgmap v2103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2874.590205 secs 2024-07-26T11:53:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2874.590205 secs 2024-07-26T11:53:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2874.590205 secs 2024-07-26T11:53:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:53 smithi160 ceph-mon[36715]: pgmap v2104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:53 smithi183 ceph-mon[37273]: pgmap v2104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:53 smithi144 ceph-mon[32129]: pgmap v2104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:53:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:55 smithi160 ceph-mon[36715]: pgmap v2105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:55 smithi183 ceph-mon[37273]: pgmap v2105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:55 smithi144 ceph-mon[32129]: pgmap v2105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2879.590332 secs 2024-07-26T11:53:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:57 smithi160 ceph-mon[36715]: pgmap v2106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2879.590332 secs 2024-07-26T11:53:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:57 smithi183 ceph-mon[37273]: pgmap v2106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2879.590332 secs 2024-07-26T11:53:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:57 smithi144 ceph-mon[32129]: pgmap v2106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:53:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:53:59 smithi144 ceph-mon[32129]: pgmap v2107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:59.271+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:59.272+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:59.272+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:59.272+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:53:59.272+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:53:59 smithi160 ceph-mon[36715]: pgmap v2107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:53:59 smithi183 ceph-mon[37273]: pgmap v2107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:01 smithi160 ceph-mon[36715]: pgmap v2108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:01 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 2884.590397 secs 2024-07-26T11:54:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:01 smithi160 ceph-mon[36715]: slow request 2884.590397 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:54:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:01 smithi183 ceph-mon[37273]: pgmap v2108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:01 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 2884.590397 secs 2024-07-26T11:54:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:01 smithi183 ceph-mon[37273]: slow request 2884.590397 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:54:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:01 smithi144 ceph-mon[32129]: pgmap v2108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:01 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 2884.590397 secs 2024-07-26T11:54:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:01 smithi144 ceph-mon[32129]: slow request 2884.590397 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:54:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:03 smithi160 ceph-mon[36715]: pgmap v2109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:03 smithi183 ceph-mon[37273]: pgmap v2109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:03 smithi144 ceph-mon[32129]: pgmap v2109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:05 smithi160 ceph-mon[36715]: pgmap v2110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:05 smithi183 ceph-mon[37273]: pgmap v2110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:05 smithi144 ceph-mon[32129]: pgmap v2110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: pgmap v2111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: pgmap v2112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 2889.590480 secs 2024-07-26T11:54:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:06 smithi183 ceph-mon[37273]: slow request 2884.729669 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: pgmap v2111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: pgmap v2112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 2889.590480 secs 2024-07-26T11:54:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:06 smithi144 ceph-mon[32129]: slow request 2884.729669 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: pgmap v2111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: pgmap v2112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 2889.590480 secs 2024-07-26T11:54:06.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:06 smithi160 ceph-mon[36715]: slow request 2884.729669 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:54:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:08 smithi144 ceph-mon[32129]: pgmap v2113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:08 smithi160 ceph-mon[36715]: pgmap v2113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:08 smithi183 ceph-mon[37273]: pgmap v2113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:10 smithi144 ceph-mon[32129]: pgmap v2114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:10 smithi160 ceph-mon[36715]: pgmap v2114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:10 smithi183 ceph-mon[37273]: pgmap v2114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2894.590586 secs 2024-07-26T11:54:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2894.590586 secs 2024-07-26T11:54:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2894.590586 secs 2024-07-26T11:54:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:12 smithi144 ceph-mon[32129]: pgmap v2115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:12 smithi160 ceph-mon[36715]: pgmap v2115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:12 smithi183 ceph-mon[37273]: pgmap v2115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:14.284+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:14.285+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:14.285+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:14.285+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:14.285+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:14.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:14 smithi183 ceph-mon[37273]: pgmap v2116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:14 smithi144 ceph-mon[32129]: pgmap v2116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:14 smithi160 ceph-mon[36715]: pgmap v2116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:54:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:16 smithi144 ceph-mon[32129]: pgmap v2117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:54:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2899.590690 secs 2024-07-26T11:54:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:16 smithi160 ceph-mon[36715]: pgmap v2117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:54:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2899.590690 secs 2024-07-26T11:54:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:16 smithi183 ceph-mon[37273]: pgmap v2117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:54:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2899.590690 secs 2024-07-26T11:54:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:18 smithi144 ceph-mon[32129]: pgmap v2118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:18 smithi160 ceph-mon[36715]: pgmap v2118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:18 smithi183 ceph-mon[37273]: pgmap v2118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:20 smithi144 ceph-mon[32129]: pgmap v2119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:20 smithi160 ceph-mon[36715]: pgmap v2119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:20 smithi183 ceph-mon[37273]: pgmap v2119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2904.590803 secs 2024-07-26T11:54:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2904.590803 secs 2024-07-26T11:54:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2904.590803 secs 2024-07-26T11:54:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:22 smithi144 ceph-mon[32129]: pgmap v2120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:22 smithi160 ceph-mon[36715]: pgmap v2120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:22 smithi183 ceph-mon[37273]: pgmap v2120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:24 smithi144 ceph-mon[32129]: pgmap v2121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:24 smithi160 ceph-mon[36715]: pgmap v2121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:24 smithi183 ceph-mon[37273]: pgmap v2121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:26 smithi144 ceph-mon[32129]: pgmap v2122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2909.590917 secs 2024-07-26T11:54:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:26 smithi160 ceph-mon[36715]: pgmap v2122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2909.590917 secs 2024-07-26T11:54:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:26 smithi183 ceph-mon[37273]: pgmap v2122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2909.590917 secs 2024-07-26T11:54:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:28 smithi144 ceph-mon[32129]: pgmap v2123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:28 smithi160 ceph-mon[36715]: pgmap v2123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:28 smithi183 ceph-mon[37273]: pgmap v2123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:29.301+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:29.302+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:29.302+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:29.302+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:29.302+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:30 smithi144 ceph-mon[32129]: pgmap v2124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:30 smithi160 ceph-mon[36715]: pgmap v2124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:30 smithi183 ceph-mon[37273]: pgmap v2124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2914.591054 secs 2024-07-26T11:54:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2914.591054 secs 2024-07-26T11:54:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2914.591054 secs 2024-07-26T11:54:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:32 smithi160 ceph-mon[36715]: pgmap v2125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:32 smithi183 ceph-mon[37273]: pgmap v2125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:32 smithi144 ceph-mon[32129]: pgmap v2125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:34 smithi160 ceph-mon[36715]: pgmap v2126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:34 smithi183 ceph-mon[37273]: pgmap v2126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:34 smithi144 ceph-mon[32129]: pgmap v2126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:36 smithi160 ceph-mon[36715]: pgmap v2127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2919.591178 secs 2024-07-26T11:54:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:36 smithi183 ceph-mon[37273]: pgmap v2127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2919.591178 secs 2024-07-26T11:54:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:36 smithi144 ceph-mon[32129]: pgmap v2127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2919.591178 secs 2024-07-26T11:54:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:38 smithi160 ceph-mon[36715]: pgmap v2128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:38 smithi183 ceph-mon[37273]: pgmap v2128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:38 smithi144 ceph-mon[32129]: pgmap v2128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:40 smithi160 ceph-mon[36715]: pgmap v2129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:40 smithi183 ceph-mon[37273]: pgmap v2129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:40 smithi144 ceph-mon[32129]: pgmap v2129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2924.591322 secs 2024-07-26T11:54:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2924.591322 secs 2024-07-26T11:54:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2924.591322 secs 2024-07-26T11:54:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:42 smithi160 ceph-mon[36715]: pgmap v2130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:42 smithi183 ceph-mon[37273]: pgmap v2130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:42 smithi144 ceph-mon[32129]: pgmap v2130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:44.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:44.304+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:44.305+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:44.305+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:44.305+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:44.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:44 smithi160 ceph-mon[36715]: pgmap v2131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:44 smithi183 ceph-mon[37273]: pgmap v2131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:44 smithi144 ceph-mon[32129]: pgmap v2131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:46 smithi144 ceph-mon[32129]: pgmap v2132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2929.591450 secs 2024-07-26T11:54:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:46 smithi160 ceph-mon[36715]: pgmap v2132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2929.591450 secs 2024-07-26T11:54:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:46 smithi183 ceph-mon[37273]: pgmap v2132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2929.591450 secs 2024-07-26T11:54:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:48 smithi160 ceph-mon[36715]: pgmap v2133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:48 smithi183 ceph-mon[37273]: pgmap v2133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:48 smithi144 ceph-mon[32129]: pgmap v2133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:50 smithi160 ceph-mon[36715]: pgmap v2134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:50 smithi183 ceph-mon[37273]: pgmap v2134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:50 smithi144 ceph-mon[32129]: pgmap v2134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2934.591567 secs 2024-07-26T11:54:51.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2934.591567 secs 2024-07-26T11:54:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2934.591567 secs 2024-07-26T11:54:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:52 smithi183 ceph-mon[37273]: pgmap v2135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:52 smithi144 ceph-mon[32129]: pgmap v2135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:52 smithi160 ceph-mon[36715]: pgmap v2135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:54 smithi183 ceph-mon[37273]: pgmap v2136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:54 smithi144 ceph-mon[32129]: pgmap v2136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:54 smithi160 ceph-mon[36715]: pgmap v2136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:54:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:56 smithi183 ceph-mon[37273]: pgmap v2137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2939.591701 secs 2024-07-26T11:54:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:56 smithi144 ceph-mon[32129]: pgmap v2137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2939.591701 secs 2024-07-26T11:54:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:56 smithi160 ceph-mon[36715]: pgmap v2137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2939.591701 secs 2024-07-26T11:54:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:54:58 smithi183 ceph-mon[37273]: pgmap v2138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:54:58 smithi144 ceph-mon[32129]: pgmap v2138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:54:58 smithi160 ceph-mon[36715]: pgmap v2138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:54:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:59.321+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:59.322+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:59.322+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:59.322+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:54:59.322+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:00 smithi183 ceph-mon[37273]: pgmap v2139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:00 smithi144 ceph-mon[32129]: pgmap v2139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:00 smithi160 ceph-mon[36715]: pgmap v2139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:01.491 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2944.591826 secs 2024-07-26T11:55:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2944.591826 secs 2024-07-26T11:55:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2944.591826 secs 2024-07-26T11:55:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:02 smithi183 ceph-mon[37273]: pgmap v2140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:02 smithi144 ceph-mon[32129]: pgmap v2140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:02 smithi160 ceph-mon[36715]: pgmap v2140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:04 smithi183 ceph-mon[37273]: pgmap v2141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:04 smithi144 ceph-mon[32129]: pgmap v2141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:04 smithi160 ceph-mon[36715]: pgmap v2141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:06 smithi183 ceph-mon[37273]: pgmap v2142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2949.591913 secs 2024-07-26T11:55:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:55:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:06 smithi144 ceph-mon[32129]: pgmap v2142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2949.591913 secs 2024-07-26T11:55:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:06 smithi160 ceph-mon[36715]: pgmap v2142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2949.591913 secs 2024-07-26T11:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:55:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:07 smithi183 ceph-mon[37273]: pgmap v2143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:07 smithi183 ceph-mon[37273]: pgmap v2144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:07 smithi144 ceph-mon[32129]: pgmap v2143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:07 smithi144 ceph-mon[32129]: pgmap v2144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:07 smithi160 ceph-mon[36715]: pgmap v2143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:07 smithi160 ceph-mon[36715]: pgmap v2144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:09 smithi144 ceph-mon[32129]: pgmap v2145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:55:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:09 smithi160 ceph-mon[36715]: pgmap v2145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:55:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:09 smithi183 ceph-mon[37273]: pgmap v2145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:55:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:11 smithi144 ceph-mon[32129]: pgmap v2146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:55:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2954.592066 secs 2024-07-26T11:55:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:11 smithi160 ceph-mon[36715]: pgmap v2146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:55:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2954.592066 secs 2024-07-26T11:55:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:11 smithi183 ceph-mon[37273]: pgmap v2146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:55:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2954.592066 secs 2024-07-26T11:55:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:13 smithi144 ceph-mon[32129]: pgmap v2147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:55:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:13 smithi160 ceph-mon[36715]: pgmap v2147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:55:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:13 smithi183 ceph-mon[37273]: pgmap v2147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:55:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:14.338+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:14.339+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:14.339+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:14.339+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:14.339+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:15.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:15 smithi183 ceph-mon[37273]: pgmap v2148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:15 smithi144 ceph-mon[32129]: pgmap v2148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:15 smithi160 ceph-mon[36715]: pgmap v2148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:55:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:16.613+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:55:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:16.613+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:55:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:16.614+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:55:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:16.614+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T11:55:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2959.592208 secs 2024-07-26T11:55:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2959.592208 secs 2024-07-26T11:55:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2959.592208 secs 2024-07-26T11:55:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:17 smithi144 ceph-mon[32129]: pgmap v2149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:55:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:17 smithi160 ceph-mon[36715]: pgmap v2149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:55:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:17 smithi183 ceph-mon[37273]: pgmap v2149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:55:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:19 smithi144 ceph-mon[32129]: pgmap v2150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:19 smithi160 ceph-mon[36715]: pgmap v2150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:19 smithi183 ceph-mon[37273]: pgmap v2150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:21 smithi144 ceph-mon[32129]: pgmap v2151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2964.592365 secs 2024-07-26T11:55:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:21 smithi160 ceph-mon[36715]: pgmap v2151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2964.592365 secs 2024-07-26T11:55:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:21 smithi183 ceph-mon[37273]: pgmap v2151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2964.592365 secs 2024-07-26T11:55:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:22 smithi183 ceph-mon[37273]: pgmap v2152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:22 smithi144 ceph-mon[32129]: pgmap v2152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:22 smithi160 ceph-mon[36715]: pgmap v2152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:25 smithi183 ceph-mon[37273]: pgmap v2153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:25 smithi144 ceph-mon[32129]: pgmap v2153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:25 smithi160 ceph-mon[36715]: pgmap v2153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2969.592458 secs 2024-07-26T11:55:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2969.592458 secs 2024-07-26T11:55:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2969.592458 secs 2024-07-26T11:55:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:27 smithi183 ceph-mon[37273]: pgmap v2154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:27 smithi144 ceph-mon[32129]: pgmap v2154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:27 smithi160 ceph-mon[36715]: pgmap v2154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:29.353+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:29.354+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:29.354+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:29.354+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:29.354+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:29 smithi183 ceph-mon[37273]: pgmap v2155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:29 smithi144 ceph-mon[32129]: pgmap v2155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:29 smithi160 ceph-mon[36715]: pgmap v2155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:31 smithi183 ceph-mon[37273]: pgmap v2156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2974.592557 secs 2024-07-26T11:55:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:31 smithi144 ceph-mon[32129]: pgmap v2156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2974.592557 secs 2024-07-26T11:55:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:31 smithi160 ceph-mon[36715]: pgmap v2156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2974.592557 secs 2024-07-26T11:55:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:33 smithi183 ceph-mon[37273]: pgmap v2157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:33 smithi144 ceph-mon[32129]: pgmap v2157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:33 smithi160 ceph-mon[36715]: pgmap v2157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:35 smithi183 ceph-mon[37273]: pgmap v2158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:35 smithi144 ceph-mon[32129]: pgmap v2158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:35 smithi160 ceph-mon[36715]: pgmap v2158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2979.592683 secs 2024-07-26T11:55:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2979.592683 secs 2024-07-26T11:55:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2979.592683 secs 2024-07-26T11:55:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:37 smithi144 ceph-mon[32129]: pgmap v2159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:37 smithi160 ceph-mon[36715]: pgmap v2159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:37 smithi183 ceph-mon[37273]: pgmap v2159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:39 smithi144 ceph-mon[32129]: pgmap v2160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:39 smithi160 ceph-mon[36715]: pgmap v2160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:39 smithi183 ceph-mon[37273]: pgmap v2160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:41 smithi144 ceph-mon[32129]: pgmap v2161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2984.592799 secs 2024-07-26T11:55:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:41 smithi160 ceph-mon[36715]: pgmap v2161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2984.592799 secs 2024-07-26T11:55:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:41 smithi183 ceph-mon[37273]: pgmap v2161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2984.592799 secs 2024-07-26T11:55:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:43 smithi144 ceph-mon[32129]: pgmap v2162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:43 smithi160 ceph-mon[36715]: pgmap v2162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:43 smithi183 ceph-mon[37273]: pgmap v2162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:44.369+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:44.370+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:44.370+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:44.370+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:44.370+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:45.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:45 smithi183 ceph-mon[37273]: pgmap v2163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:45.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:45 smithi144 ceph-mon[32129]: pgmap v2163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:45 smithi160 ceph-mon[36715]: pgmap v2163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:46.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2989.592958 secs 2024-07-26T11:55:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2989.592958 secs 2024-07-26T11:55:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2989.592958 secs 2024-07-26T11:55:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:47 smithi144 ceph-mon[32129]: pgmap v2164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:47 smithi160 ceph-mon[36715]: pgmap v2164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:47.877 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:47 smithi183 ceph-mon[37273]: pgmap v2164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:49 smithi144 ceph-mon[32129]: pgmap v2165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:49 smithi160 ceph-mon[36715]: pgmap v2165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:49 smithi183 ceph-mon[37273]: pgmap v2165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:51 smithi144 ceph-mon[32129]: pgmap v2166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2994.593082 secs 2024-07-26T11:55:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:51 smithi160 ceph-mon[36715]: pgmap v2166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2994.593082 secs 2024-07-26T11:55:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:51 smithi183 ceph-mon[37273]: pgmap v2166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2994.593082 secs 2024-07-26T11:55:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:53 smithi144 ceph-mon[32129]: pgmap v2167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:53 smithi160 ceph-mon[36715]: pgmap v2167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:53 smithi183 ceph-mon[37273]: pgmap v2167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:55 smithi144 ceph-mon[32129]: pgmap v2168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:55 smithi160 ceph-mon[36715]: pgmap v2168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:55 smithi183 ceph-mon[37273]: pgmap v2168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 2999.593224 secs 2024-07-26T11:55:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 2999.593224 secs 2024-07-26T11:55:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 2999.593224 secs 2024-07-26T11:55:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:57 smithi144 ceph-mon[32129]: pgmap v2169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:57 smithi160 ceph-mon[36715]: pgmap v2169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:57 smithi183 ceph-mon[37273]: pgmap v2169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:55:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:59.387+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:59.388+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:59.388+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:59.388+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:55:59.388+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:55:59 smithi160 ceph-mon[36715]: pgmap v2170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:59.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:55:59 smithi183 ceph-mon[37273]: pgmap v2170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:55:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:55:59 smithi144 ceph-mon[32129]: pgmap v2170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:01 smithi160 ceph-mon[36715]: pgmap v2171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3004.593357 secs 2024-07-26T11:56:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:01 smithi183 ceph-mon[37273]: pgmap v2171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3004.593357 secs 2024-07-26T11:56:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:01 smithi144 ceph-mon[32129]: pgmap v2171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3004.593357 secs 2024-07-26T11:56:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:03 smithi160 ceph-mon[36715]: pgmap v2172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:03 smithi183 ceph-mon[37273]: pgmap v2172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:03 smithi144 ceph-mon[32129]: pgmap v2172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:05 smithi160 ceph-mon[36715]: pgmap v2173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:05 smithi183 ceph-mon[37273]: pgmap v2173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:05 smithi144 ceph-mon[32129]: pgmap v2173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3009.593509 secs 2024-07-26T11:56:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3009.593509 secs 2024-07-26T11:56:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3009.593509 secs 2024-07-26T11:56:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:07 smithi160 ceph-mon[36715]: pgmap v2174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:56:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:07 smithi183 ceph-mon[37273]: pgmap v2174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:56:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:07 smithi144 ceph-mon[32129]: pgmap v2174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:56:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:08 smithi160 ceph-mon[36715]: pgmap v2175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:56:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:08 smithi160 ceph-mon[36715]: pgmap v2176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:08 smithi183 ceph-mon[37273]: pgmap v2175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:56:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:08 smithi183 ceph-mon[37273]: pgmap v2176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:08 smithi144 ceph-mon[32129]: pgmap v2175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:56:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:08 smithi144 ceph-mon[32129]: pgmap v2176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:10 smithi160 ceph-mon[36715]: pgmap v2177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:10 smithi183 ceph-mon[37273]: pgmap v2177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:10 smithi144 ceph-mon[32129]: pgmap v2177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3014.593648 secs 2024-07-26T11:56:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3014.593648 secs 2024-07-26T11:56:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3014.593648 secs 2024-07-26T11:56:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:12 smithi160 ceph-mon[36715]: pgmap v2178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:56:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:12 smithi183 ceph-mon[37273]: pgmap v2178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:56:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:12 smithi144 ceph-mon[32129]: pgmap v2178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:56:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:14.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:14.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:14.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:14.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:14.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:14 smithi160 ceph-mon[36715]: pgmap v2179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:56:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:14 smithi183 ceph-mon[37273]: pgmap v2179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:56:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:14 smithi144 ceph-mon[32129]: pgmap v2179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:56:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:16 smithi160 ceph-mon[36715]: pgmap v2180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:56:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3019.593779 secs 2024-07-26T11:56:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:16 smithi183 ceph-mon[37273]: pgmap v2180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:56:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3019.593779 secs 2024-07-26T11:56:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:16 smithi144 ceph-mon[32129]: pgmap v2180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:56:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3019.593779 secs 2024-07-26T11:56:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:18 smithi160 ceph-mon[36715]: pgmap v2181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:18 smithi183 ceph-mon[37273]: pgmap v2181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:18 smithi144 ceph-mon[32129]: pgmap v2181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:20 smithi160 ceph-mon[36715]: pgmap v2182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:20 smithi183 ceph-mon[37273]: pgmap v2182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:20 smithi144 ceph-mon[32129]: pgmap v2182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:21.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3024.593859 secs 2024-07-26T11:56:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3024.593859 secs 2024-07-26T11:56:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3024.593859 secs 2024-07-26T11:56:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:22 smithi183 ceph-mon[37273]: pgmap v2183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:22 smithi144 ceph-mon[32129]: pgmap v2183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:22 smithi160 ceph-mon[36715]: pgmap v2183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:24 smithi183 ceph-mon[37273]: pgmap v2184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:24 smithi144 ceph-mon[32129]: pgmap v2184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:24 smithi160 ceph-mon[36715]: pgmap v2184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:26 smithi183 ceph-mon[37273]: pgmap v2185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3029.593965 secs 2024-07-26T11:56:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:26 smithi144 ceph-mon[32129]: pgmap v2185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3029.593965 secs 2024-07-26T11:56:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:26 smithi160 ceph-mon[36715]: pgmap v2185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3029.593965 secs 2024-07-26T11:56:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:28 smithi183 ceph-mon[37273]: pgmap v2186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:28 smithi144 ceph-mon[32129]: pgmap v2186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:28 smithi160 ceph-mon[36715]: pgmap v2186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:29.405 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:29.404+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:29.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:29.405+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:29.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:29.405+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:29.405+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:29.405+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:30 smithi183 ceph-mon[37273]: pgmap v2187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:30 smithi144 ceph-mon[32129]: pgmap v2187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:30 smithi160 ceph-mon[36715]: pgmap v2187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3034.594062 secs 2024-07-26T11:56:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3034.594062 secs 2024-07-26T11:56:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3034.594062 secs 2024-07-26T11:56:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:32 smithi183 ceph-mon[37273]: pgmap v2188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:32 smithi144 ceph-mon[32129]: pgmap v2188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:32 smithi160 ceph-mon[36715]: pgmap v2188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:34 smithi183 ceph-mon[37273]: pgmap v2189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:34 smithi144 ceph-mon[32129]: pgmap v2189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:34 smithi160 ceph-mon[36715]: pgmap v2189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:36 smithi183 ceph-mon[37273]: pgmap v2190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3039.594190 secs 2024-07-26T11:56:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:36 smithi144 ceph-mon[32129]: pgmap v2190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3039.594190 secs 2024-07-26T11:56:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:36 smithi160 ceph-mon[36715]: pgmap v2190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3039.594190 secs 2024-07-26T11:56:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:38 smithi183 ceph-mon[37273]: pgmap v2191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:38 smithi144 ceph-mon[32129]: pgmap v2191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:38 smithi160 ceph-mon[36715]: pgmap v2191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:40 smithi183 ceph-mon[37273]: pgmap v2192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:40 smithi144 ceph-mon[32129]: pgmap v2192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:40 smithi160 ceph-mon[36715]: pgmap v2192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:41.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3044.594285 secs 2024-07-26T11:56:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3044.594285 secs 2024-07-26T11:56:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3044.594285 secs 2024-07-26T11:56:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:42 smithi144 ceph-mon[32129]: pgmap v2193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:42 smithi160 ceph-mon[36715]: pgmap v2193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:42 smithi183 ceph-mon[37273]: pgmap v2193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:56:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:44.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:44.417+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:44.417+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:44.417+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:44.417+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:44 smithi144 ceph-mon[32129]: pgmap v2194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:44 smithi160 ceph-mon[36715]: pgmap v2194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:44 smithi183 ceph-mon[37273]: pgmap v2194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:46.629+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:56:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:46.629+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:56:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:46.630+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:56:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:46.631+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T11:56:46.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:46 smithi183 ceph-mon[37273]: pgmap v2195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:46.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3049.594399 secs 2024-07-26T11:56:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:46 smithi144 ceph-mon[32129]: pgmap v2195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3049.594399 secs 2024-07-26T11:56:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:46 smithi160 ceph-mon[36715]: pgmap v2195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3049.594399 secs 2024-07-26T11:56:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:48 smithi144 ceph-mon[32129]: pgmap v2196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:48 smithi160 ceph-mon[36715]: pgmap v2196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:48 smithi183 ceph-mon[37273]: pgmap v2196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:56:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:50 smithi144 ceph-mon[32129]: pgmap v2197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:56:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:50 smithi160 ceph-mon[36715]: pgmap v2197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:56:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:50 smithi183 ceph-mon[37273]: pgmap v2197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:56:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3054.594480 secs 2024-07-26T11:56:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3054.594480 secs 2024-07-26T11:56:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3054.594480 secs 2024-07-26T11:56:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:52 smithi144 ceph-mon[32129]: pgmap v2198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:52 smithi160 ceph-mon[36715]: pgmap v2198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:52 smithi183 ceph-mon[37273]: pgmap v2198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:54 smithi144 ceph-mon[32129]: pgmap v2199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:54 smithi160 ceph-mon[36715]: pgmap v2199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:54 smithi183 ceph-mon[37273]: pgmap v2199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:56 smithi144 ceph-mon[32129]: pgmap v2200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3059.594595 secs 2024-07-26T11:56:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:56 smithi160 ceph-mon[36715]: pgmap v2200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3059.594595 secs 2024-07-26T11:56:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:56 smithi183 ceph-mon[37273]: pgmap v2200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:56:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3059.594595 secs 2024-07-26T11:56:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:56:58 smithi144 ceph-mon[32129]: pgmap v2201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:56:58 smithi160 ceph-mon[36715]: pgmap v2201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:56:58 smithi183 ceph-mon[37273]: pgmap v2201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:59.431+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:59.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:59.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:59.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:56:59.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:00 smithi144 ceph-mon[32129]: pgmap v2202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:00 smithi160 ceph-mon[36715]: pgmap v2202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:00 smithi183 ceph-mon[37273]: pgmap v2202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3064.594707 secs 2024-07-26T11:57:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3064.594707 secs 2024-07-26T11:57:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3064.594707 secs 2024-07-26T11:57:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:02 smithi144 ceph-mon[32129]: pgmap v2203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:02 smithi160 ceph-mon[36715]: pgmap v2203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:02 smithi183 ceph-mon[37273]: pgmap v2203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:04 smithi160 ceph-mon[36715]: pgmap v2204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:04 smithi183 ceph-mon[37273]: pgmap v2204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:04 smithi144 ceph-mon[32129]: pgmap v2204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:06 smithi160 ceph-mon[36715]: pgmap v2205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3069.594837 secs 2024-07-26T11:57:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:06 smithi183 ceph-mon[37273]: pgmap v2205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3069.594837 secs 2024-07-26T11:57:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:06 smithi144 ceph-mon[32129]: pgmap v2205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3069.594837 secs 2024-07-26T11:57:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:08.930 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:08 smithi160 ceph-mon[36715]: pgmap v2206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:08 smithi183 ceph-mon[37273]: pgmap v2206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:08 smithi144 ceph-mon[32129]: pgmap v2206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:10 smithi160 ceph-mon[36715]: pgmap v2207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:11.068 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:10 smithi144 ceph-mon[32129]: pgmap v2207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:10 smithi183 ceph-mon[37273]: pgmap v2207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3074.594908 secs 2024-07-26T11:57:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3074.594908 secs 2024-07-26T11:57:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3074.594908 secs 2024-07-26T11:57:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:13.048 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:12 smithi144 ceph-mon[32129]: pgmap v2208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:12 smithi160 ceph-mon[36715]: pgmap v2208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:12 smithi183 ceph-mon[37273]: pgmap v2208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:14.450 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:14.448+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:14.450 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:14.449+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:14.449+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:14.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:14.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: pgmap v2209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:15.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: pgmap v2209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:15.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:15.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: pgmap v2209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:57:16.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:15 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:57:16.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:15 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:57:16.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:15 smithi183 ceph-mon[37273]: pgmap v2210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:57:16.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:15 smithi183 ceph-mon[37273]: pgmap v2211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:15 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:57:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:15 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:57:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:15 smithi160 ceph-mon[36715]: pgmap v2210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:57:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:15 smithi160 ceph-mon[36715]: pgmap v2211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:15 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T11:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:15 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T11:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:15 smithi144 ceph-mon[32129]: pgmap v2210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:57:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:15 smithi144 ceph-mon[32129]: pgmap v2211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3079.595027 secs 2024-07-26T11:57:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3079.595027 secs 2024-07-26T11:57:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3079.595027 secs 2024-07-26T11:57:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:17 smithi160 ceph-mon[36715]: pgmap v2212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:57:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:17 smithi183 ceph-mon[37273]: pgmap v2212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:57:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:17 smithi144 ceph-mon[32129]: pgmap v2212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:57:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:19 smithi160 ceph-mon[36715]: pgmap v2213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T11:57:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:19 smithi183 ceph-mon[37273]: pgmap v2213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T11:57:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:19 smithi144 ceph-mon[32129]: pgmap v2213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T11:57:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:21 smithi160 ceph-mon[36715]: pgmap v2214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:57:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3084.595149 secs 2024-07-26T11:57:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:21 smithi183 ceph-mon[37273]: pgmap v2214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:57:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3084.595149 secs 2024-07-26T11:57:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:21 smithi144 ceph-mon[32129]: pgmap v2214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:57:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3084.595149 secs 2024-07-26T11:57:23.918 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:23 smithi144 ceph-mon[32129]: pgmap v2215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:23 smithi160 ceph-mon[36715]: pgmap v2215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:23 smithi183 ceph-mon[37273]: pgmap v2215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:25 smithi160 ceph-mon[36715]: pgmap v2216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:57:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:25 smithi183 ceph-mon[37273]: pgmap v2216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:57:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:25 smithi144 ceph-mon[32129]: pgmap v2216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:57:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3089.595277 secs 2024-07-26T11:57:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3089.595277 secs 2024-07-26T11:57:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3089.595277 secs 2024-07-26T11:57:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:27 smithi183 ceph-mon[37273]: pgmap v2217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:27 smithi144 ceph-mon[32129]: pgmap v2217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:27 smithi160 ceph-mon[36715]: pgmap v2217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:29.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:29.457+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:29.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:29.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:29.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:29.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:29.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:29.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:29.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:29.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:29 smithi183 ceph-mon[37273]: pgmap v2218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:29 smithi144 ceph-mon[32129]: pgmap v2218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:29 smithi160 ceph-mon[36715]: pgmap v2218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:31 smithi183 ceph-mon[37273]: pgmap v2219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3094.595438 secs 2024-07-26T11:57:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:31 smithi144 ceph-mon[32129]: pgmap v2219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3094.595438 secs 2024-07-26T11:57:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:31 smithi160 ceph-mon[36715]: pgmap v2219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3094.595438 secs 2024-07-26T11:57:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:33 smithi183 ceph-mon[37273]: pgmap v2220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:33 smithi144 ceph-mon[32129]: pgmap v2220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:33 smithi160 ceph-mon[36715]: pgmap v2220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:35 smithi183 ceph-mon[37273]: pgmap v2221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:35 smithi144 ceph-mon[32129]: pgmap v2221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:35 smithi160 ceph-mon[36715]: pgmap v2221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:37.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3099.595575 secs 2024-07-26T11:57:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3099.595575 secs 2024-07-26T11:57:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3099.595575 secs 2024-07-26T11:57:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:37 smithi183 ceph-mon[37273]: pgmap v2222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:38.144 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:37 smithi144 ceph-mon[32129]: pgmap v2222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:37 smithi160 ceph-mon[36715]: pgmap v2222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:39 smithi144 ceph-mon[32129]: pgmap v2223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:39 smithi183 ceph-mon[37273]: pgmap v2223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:39 smithi160 ceph-mon[36715]: pgmap v2223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:41 smithi183 ceph-mon[37273]: pgmap v2224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3104.595701 secs 2024-07-26T11:57:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:41 smithi144 ceph-mon[32129]: pgmap v2224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3104.595701 secs 2024-07-26T11:57:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:41 smithi160 ceph-mon[36715]: pgmap v2224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3104.595701 secs 2024-07-26T11:57:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:42 smithi144 ceph-mon[32129]: pgmap v2225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:42 smithi160 ceph-mon[36715]: pgmap v2225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:42 smithi183 ceph-mon[37273]: pgmap v2225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:44.474+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:44.475+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:44.475+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:44.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:44.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:45 smithi160 ceph-mon[36715]: pgmap v2226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:45 smithi183 ceph-mon[37273]: pgmap v2226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:45 smithi144 ceph-mon[32129]: pgmap v2226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3109.595765 secs 2024-07-26T11:57:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:47 smithi160 ceph-mon[36715]: pgmap v2227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3109.595765 secs 2024-07-26T11:57:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:47 smithi183 ceph-mon[37273]: pgmap v2227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3109.595765 secs 2024-07-26T11:57:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:47 smithi144 ceph-mon[32129]: pgmap v2227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:49 smithi160 ceph-mon[36715]: pgmap v2228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:49 smithi183 ceph-mon[37273]: pgmap v2228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:49 smithi144 ceph-mon[32129]: pgmap v2228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:51 smithi160 ceph-mon[36715]: pgmap v2229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:51 smithi183 ceph-mon[37273]: pgmap v2229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:51 smithi144 ceph-mon[32129]: pgmap v2229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3114.595894 secs 2024-07-26T11:57:52.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3114.595894 secs 2024-07-26T11:57:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3114.595894 secs 2024-07-26T11:57:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:53 smithi144 ceph-mon[32129]: pgmap v2230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:53 smithi183 ceph-mon[37273]: pgmap v2230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:53 smithi160 ceph-mon[36715]: pgmap v2230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:57:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:55 smithi183 ceph-mon[37273]: pgmap v2231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:55 smithi144 ceph-mon[32129]: pgmap v2231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:55 smithi160 ceph-mon[36715]: pgmap v2231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3119.596034 secs 2024-07-26T11:57:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3119.596034 secs 2024-07-26T11:57:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3119.596034 secs 2024-07-26T11:57:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:57 smithi183 ceph-mon[37273]: pgmap v2232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:57 smithi144 ceph-mon[32129]: pgmap v2232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:57 smithi160 ceph-mon[36715]: pgmap v2232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:57:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:57:59 smithi183 ceph-mon[37273]: pgmap v2233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:57:59 smithi144 ceph-mon[32129]: pgmap v2233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:59.491+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:59.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:59.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:59.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:57:59.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:57:59 smithi160 ceph-mon[36715]: pgmap v2233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:01 smithi183 ceph-mon[37273]: pgmap v2234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3124.596157 secs 2024-07-26T11:58:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:01 smithi144 ceph-mon[32129]: pgmap v2234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3124.596157 secs 2024-07-26T11:58:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:01 smithi160 ceph-mon[36715]: pgmap v2234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3124.596157 secs 2024-07-26T11:58:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:03 smithi183 ceph-mon[37273]: pgmap v2235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:03 smithi144 ceph-mon[32129]: pgmap v2235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:03 smithi160 ceph-mon[36715]: pgmap v2235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:05 smithi183 ceph-mon[37273]: pgmap v2236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:05 smithi144 ceph-mon[32129]: pgmap v2236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:05 smithi160 ceph-mon[36715]: pgmap v2236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3129.596277 secs 2024-07-26T11:58:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3129.596277 secs 2024-07-26T11:58:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3129.596277 secs 2024-07-26T11:58:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:07 smithi183 ceph-mon[37273]: pgmap v2237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:07 smithi144 ceph-mon[32129]: pgmap v2237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:07 smithi160 ceph-mon[36715]: pgmap v2237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:09 smithi183 ceph-mon[37273]: pgmap v2238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:09 smithi144 ceph-mon[32129]: pgmap v2238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:09 smithi160 ceph-mon[36715]: pgmap v2238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:11 smithi144 ceph-mon[32129]: pgmap v2239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3134.596403 secs 2024-07-26T11:58:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:11 smithi160 ceph-mon[36715]: pgmap v2239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3134.596403 secs 2024-07-26T11:58:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:11 smithi183 ceph-mon[37273]: pgmap v2239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3134.596403 secs 2024-07-26T11:58:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:13 smithi144 ceph-mon[32129]: pgmap v2240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:13 smithi160 ceph-mon[36715]: pgmap v2240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:13 smithi183 ceph-mon[37273]: pgmap v2240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:14.507+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:14.508+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:14.508+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:14.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:14.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:15.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: pgmap v2241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:15.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:15.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:58:15.343 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:15 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:58:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: pgmap v2241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:58:15.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:15 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:58:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: pgmap v2241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T11:58:15.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:15 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:58:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3139.596528 secs 2024-07-26T11:58:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3139.596528 secs 2024-07-26T11:58:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3139.596528 secs 2024-07-26T11:58:17.441 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:17 smithi183 ceph-mon[37273]: pgmap v2242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:17 smithi144 ceph-mon[32129]: pgmap v2242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:17 smithi160 ceph-mon[36715]: pgmap v2242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: pgmap v2243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: pgmap v2244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:19.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: pgmap v2243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:19.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: pgmap v2244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:19.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:19.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:19.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: pgmap v2243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:19.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: pgmap v2244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:19.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:58:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:19 smithi144 ceph-mon[32129]: pgmap v2245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:19 smithi160 ceph-mon[36715]: pgmap v2245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:19 smithi183 ceph-mon[37273]: pgmap v2245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3144.596644 secs 2024-07-26T11:58:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3144.596644 secs 2024-07-26T11:58:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3144.596644 secs 2024-07-26T11:58:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:22 smithi144 ceph-mon[32129]: pgmap v2246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:22 smithi160 ceph-mon[36715]: pgmap v2246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:22 smithi183 ceph-mon[37273]: pgmap v2246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:23 smithi144 ceph-mon[32129]: pgmap v2247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:23 smithi160 ceph-mon[36715]: pgmap v2247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:23 smithi183 ceph-mon[37273]: pgmap v2247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:25 smithi144 ceph-mon[32129]: pgmap v2248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:25 smithi160 ceph-mon[36715]: pgmap v2248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:25 smithi183 ceph-mon[37273]: pgmap v2248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:58:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3149.596773 secs 2024-07-26T11:58:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3149.596773 secs 2024-07-26T11:58:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3149.596773 secs 2024-07-26T11:58:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:27 smithi144 ceph-mon[32129]: pgmap v2249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:58:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:27 smithi160 ceph-mon[36715]: pgmap v2249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:58:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:27 smithi183 ceph-mon[37273]: pgmap v2249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:58:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:29.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:29.525+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:29.525+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:29.525+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:29.525+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:30 smithi144 ceph-mon[32129]: pgmap v2250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:30 smithi160 ceph-mon[36715]: pgmap v2250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:30 smithi183 ceph-mon[37273]: pgmap v2250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3154.596890 secs 2024-07-26T11:58:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3154.596890 secs 2024-07-26T11:58:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3154.596890 secs 2024-07-26T11:58:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:32 smithi144 ceph-mon[32129]: pgmap v2251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:32 smithi160 ceph-mon[36715]: pgmap v2251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:32 smithi183 ceph-mon[37273]: pgmap v2251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:34 smithi144 ceph-mon[32129]: pgmap v2252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:34 smithi160 ceph-mon[36715]: pgmap v2252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:34 smithi183 ceph-mon[37273]: pgmap v2252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3159.597012 secs 2024-07-26T11:58:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:36 smithi144 ceph-mon[32129]: pgmap v2253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3159.597012 secs 2024-07-26T11:58:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:36 smithi160 ceph-mon[36715]: pgmap v2253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3159.597012 secs 2024-07-26T11:58:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:36 smithi183 ceph-mon[37273]: pgmap v2253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:38 smithi144 ceph-mon[32129]: pgmap v2254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:38 smithi160 ceph-mon[36715]: pgmap v2254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:38 smithi183 ceph-mon[37273]: pgmap v2254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:40 smithi160 ceph-mon[36715]: pgmap v2255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:40 smithi183 ceph-mon[37273]: pgmap v2255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:40 smithi144 ceph-mon[32129]: pgmap v2255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3164.597128 secs 2024-07-26T11:58:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3164.597128 secs 2024-07-26T11:58:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3164.597128 secs 2024-07-26T11:58:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:42 smithi160 ceph-mon[36715]: pgmap v2256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:42 smithi183 ceph-mon[37273]: pgmap v2256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:42 smithi144 ceph-mon[32129]: pgmap v2256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:44.526+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:44.527+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:44.527+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:44.527+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:44.527+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:44 smithi160 ceph-mon[36715]: pgmap v2257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:44 smithi183 ceph-mon[37273]: pgmap v2257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:44 smithi144 ceph-mon[32129]: pgmap v2257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3169.597245 secs 2024-07-26T11:58:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:46 smithi160 ceph-mon[36715]: pgmap v2258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3169.597245 secs 2024-07-26T11:58:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:46 smithi183 ceph-mon[37273]: pgmap v2258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3169.597245 secs 2024-07-26T11:58:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:46 smithi144 ceph-mon[32129]: pgmap v2258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:49 smithi160 ceph-mon[36715]: pgmap v2259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:49 smithi183 ceph-mon[37273]: pgmap v2259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:49 smithi144 ceph-mon[32129]: pgmap v2259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:51 smithi160 ceph-mon[36715]: pgmap v2260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:51 smithi183 ceph-mon[37273]: pgmap v2260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:51 smithi144 ceph-mon[32129]: pgmap v2260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3174.597363 secs 2024-07-26T11:58:52.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3174.597363 secs 2024-07-26T11:58:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3174.597363 secs 2024-07-26T11:58:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:53 smithi160 ceph-mon[36715]: pgmap v2261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:53 smithi183 ceph-mon[37273]: pgmap v2261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:53 smithi144 ceph-mon[32129]: pgmap v2261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:58:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:55 smithi144 ceph-mon[32129]: pgmap v2262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:55 smithi160 ceph-mon[36715]: pgmap v2262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:55 smithi183 ceph-mon[37273]: pgmap v2262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3179.597448 secs 2024-07-26T11:58:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:57 smithi160 ceph-mon[36715]: pgmap v2263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3179.597448 secs 2024-07-26T11:58:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:57 smithi183 ceph-mon[37273]: pgmap v2263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3179.597448 secs 2024-07-26T11:58:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:57 smithi144 ceph-mon[32129]: pgmap v2263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:58:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:58:59 smithi160 ceph-mon[36715]: pgmap v2264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:58:59 smithi183 ceph-mon[37273]: pgmap v2264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:58:59 smithi144 ceph-mon[32129]: pgmap v2264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:59.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:59.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:59.536+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:59.536+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:59.536+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:58:59.536+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:01 smithi160 ceph-mon[36715]: pgmap v2265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:01 smithi183 ceph-mon[37273]: pgmap v2265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:01 smithi144 ceph-mon[32129]: pgmap v2265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3184.597514 secs 2024-07-26T11:59:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3184.597514 secs 2024-07-26T11:59:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3184.597514 secs 2024-07-26T11:59:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:03 smithi160 ceph-mon[36715]: pgmap v2266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:03 smithi183 ceph-mon[37273]: pgmap v2266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:03 smithi144 ceph-mon[32129]: pgmap v2266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:05 smithi183 ceph-mon[37273]: pgmap v2267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:05 smithi144 ceph-mon[32129]: pgmap v2267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:05 smithi160 ceph-mon[36715]: pgmap v2267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3189.597637 secs 2024-07-26T11:59:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:07 smithi183 ceph-mon[37273]: pgmap v2268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3189.597637 secs 2024-07-26T11:59:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:07 smithi144 ceph-mon[32129]: pgmap v2268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3189.597637 secs 2024-07-26T11:59:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:07 smithi160 ceph-mon[36715]: pgmap v2268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:09 smithi183 ceph-mon[37273]: pgmap v2269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:09 smithi144 ceph-mon[32129]: pgmap v2269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:09 smithi160 ceph-mon[36715]: pgmap v2269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:11 smithi183 ceph-mon[37273]: pgmap v2270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:11 smithi144 ceph-mon[32129]: pgmap v2270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:11 smithi160 ceph-mon[36715]: pgmap v2270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3194.597716 secs 2024-07-26T11:59:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3194.597716 secs 2024-07-26T11:59:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3194.597716 secs 2024-07-26T11:59:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:13 smithi183 ceph-mon[37273]: pgmap v2271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:13.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:13 smithi160 ceph-mon[36715]: pgmap v2271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:13 smithi144 ceph-mon[32129]: pgmap v2271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:14.541+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:14.542+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:14.542+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:14.542+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:14.542+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:15 smithi183 ceph-mon[37273]: pgmap v2272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:15 smithi144 ceph-mon[32129]: pgmap v2272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:15 smithi160 ceph-mon[36715]: pgmap v2272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3199.597837 secs 2024-07-26T11:59:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3199.597837 secs 2024-07-26T11:59:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3199.597837 secs 2024-07-26T11:59:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:17 smithi183 ceph-mon[37273]: pgmap v2273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:17 smithi144 ceph-mon[32129]: pgmap v2273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:17 smithi160 ceph-mon[36715]: pgmap v2273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:19 smithi183 ceph-mon[37273]: pgmap v2274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:59:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:19 smithi144 ceph-mon[32129]: pgmap v2274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:59:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:19 smithi160 ceph-mon[36715]: pgmap v2274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T11:59:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:20 smithi144 ceph-mon[32129]: pgmap v2275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:20 smithi144 ceph-mon[32129]: pgmap v2276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:20 smithi160 ceph-mon[36715]: pgmap v2275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:20 smithi160 ceph-mon[36715]: pgmap v2276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:20 smithi183 ceph-mon[37273]: pgmap v2275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:20 smithi183 ceph-mon[37273]: pgmap v2276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3204.597954 secs 2024-07-26T11:59:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3204.597954 secs 2024-07-26T11:59:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3204.597954 secs 2024-07-26T11:59:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:22 smithi144 ceph-mon[32129]: pgmap v2277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:22 smithi160 ceph-mon[36715]: pgmap v2277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:22 smithi183 ceph-mon[37273]: pgmap v2277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:24 smithi144 ceph-mon[32129]: pgmap v2278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:24 smithi160 ceph-mon[36715]: pgmap v2278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:24 smithi183 ceph-mon[37273]: pgmap v2278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:26 smithi144 ceph-mon[32129]: pgmap v2279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:59:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3209.598100 secs 2024-07-26T11:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:26 smithi160 ceph-mon[36715]: pgmap v2279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3209.598100 secs 2024-07-26T11:59:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:26 smithi183 ceph-mon[37273]: pgmap v2279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T11:59:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3209.598100 secs 2024-07-26T11:59:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:28 smithi144 ceph-mon[32129]: pgmap v2280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:28 smithi160 ceph-mon[36715]: pgmap v2280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:28 smithi183 ceph-mon[37273]: pgmap v2280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:59:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:29.547+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:29.548+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:29.548+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:29.548+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:29.548+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:30 smithi144 ceph-mon[32129]: pgmap v2281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:59:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:30 smithi160 ceph-mon[36715]: pgmap v2281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:59:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:30 smithi183 ceph-mon[37273]: pgmap v2281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:59:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3214.598236 secs 2024-07-26T11:59:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3214.598236 secs 2024-07-26T11:59:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3214.598236 secs 2024-07-26T11:59:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:32 smithi144 ceph-mon[32129]: pgmap v2282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:32 smithi160 ceph-mon[36715]: pgmap v2282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:32 smithi183 ceph-mon[37273]: pgmap v2282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:34 smithi144 ceph-mon[32129]: pgmap v2283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:34 smithi160 ceph-mon[36715]: pgmap v2283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:34 smithi183 ceph-mon[37273]: pgmap v2283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:36 smithi144 ceph-mon[32129]: pgmap v2284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3219.598385 secs 2024-07-26T11:59:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:36 smithi160 ceph-mon[36715]: pgmap v2284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3219.598385 secs 2024-07-26T11:59:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:36 smithi183 ceph-mon[37273]: pgmap v2284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3219.598385 secs 2024-07-26T11:59:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:38 smithi144 ceph-mon[32129]: pgmap v2285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:38 smithi160 ceph-mon[36715]: pgmap v2285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:38 smithi183 ceph-mon[37273]: pgmap v2285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:40 smithi144 ceph-mon[32129]: pgmap v2286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:40 smithi160 ceph-mon[36715]: pgmap v2286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:40 smithi183 ceph-mon[37273]: pgmap v2286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3224.598507 secs 2024-07-26T11:59:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3224.598507 secs 2024-07-26T11:59:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3224.598507 secs 2024-07-26T11:59:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:42 smithi144 ceph-mon[32129]: pgmap v2287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:42 smithi160 ceph-mon[36715]: pgmap v2287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:42 smithi183 ceph-mon[37273]: pgmap v2287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:44 smithi160 ceph-mon[36715]: pgmap v2288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:44.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:44.564+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:44.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:44.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:44.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:44.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:44.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:44.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:44.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:44.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:44 smithi183 ceph-mon[37273]: pgmap v2288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:44 smithi144 ceph-mon[32129]: pgmap v2288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:46 smithi160 ceph-mon[36715]: pgmap v2289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3229.598648 secs 2024-07-26T11:59:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:46 smithi183 ceph-mon[37273]: pgmap v2289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3229.598648 secs 2024-07-26T11:59:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:46.661+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T11:59:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:46.661+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T11:59:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:46.662+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T11:59:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:46.662+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T11:59:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:46 smithi144 ceph-mon[32129]: pgmap v2289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3229.598648 secs 2024-07-26T11:59:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:48 smithi160 ceph-mon[36715]: pgmap v2290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:48 smithi183 ceph-mon[37273]: pgmap v2290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:48 smithi144 ceph-mon[32129]: pgmap v2290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:50 smithi160 ceph-mon[36715]: pgmap v2291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:50 smithi183 ceph-mon[37273]: pgmap v2291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:50 smithi144 ceph-mon[32129]: pgmap v2291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3234.598780 secs 2024-07-26T11:59:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3234.598780 secs 2024-07-26T11:59:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3234.598780 secs 2024-07-26T11:59:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:52 smithi160 ceph-mon[36715]: pgmap v2292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:52.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:52 smithi144 ceph-mon[32129]: pgmap v2292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:52 smithi183 ceph-mon[37273]: pgmap v2292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:54 smithi160 ceph-mon[36715]: pgmap v2293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:54 smithi183 ceph-mon[37273]: pgmap v2293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:54 smithi144 ceph-mon[32129]: pgmap v2293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:59:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:56 smithi160 ceph-mon[36715]: pgmap v2294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3239.598922 secs 2024-07-26T11:59:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:56 smithi183 ceph-mon[37273]: pgmap v2294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3239.598922 secs 2024-07-26T11:59:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:56 smithi144 ceph-mon[32129]: pgmap v2294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3239.598922 secs 2024-07-26T11:59:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 11:59:58 smithi160 ceph-mon[36715]: pgmap v2295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 11:59:58 smithi183 ceph-mon[37273]: pgmap v2295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 11:59:58 smithi144 ceph-mon[32129]: pgmap v2295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:59:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:59.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:59.582+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:59.582+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:59.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:59.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:59.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T11:59:59.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:00 smithi160 ceph-mon[36715]: pgmap v2296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:00 smithi183 ceph-mon[37273]: pgmap v2296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:00 smithi144 ceph-mon[32129]: pgmap v2296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3244.599044 secs 2024-07-26T12:00:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3244.599044 secs 2024-07-26T12:00:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3244.599044 secs 2024-07-26T12:00:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:02 smithi183 ceph-mon[37273]: pgmap v2297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:02 smithi144 ceph-mon[32129]: pgmap v2297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:02 smithi160 ceph-mon[36715]: pgmap v2297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:04 smithi183 ceph-mon[37273]: pgmap v2298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:04 smithi144 ceph-mon[32129]: pgmap v2298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:04 smithi160 ceph-mon[36715]: pgmap v2298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:06 smithi183 ceph-mon[37273]: pgmap v2299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3249.599168 secs 2024-07-26T12:00:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:06 smithi144 ceph-mon[32129]: pgmap v2299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3249.599168 secs 2024-07-26T12:00:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:06 smithi160 ceph-mon[36715]: pgmap v2299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3249.599168 secs 2024-07-26T12:00:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:08 smithi183 ceph-mon[37273]: pgmap v2300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:08 smithi144 ceph-mon[32129]: pgmap v2300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:08 smithi160 ceph-mon[36715]: pgmap v2300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:10 smithi183 ceph-mon[37273]: pgmap v2301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:10 smithi144 ceph-mon[32129]: pgmap v2301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:10 smithi160 ceph-mon[36715]: pgmap v2301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3254.599312 secs 2024-07-26T12:00:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3254.599312 secs 2024-07-26T12:00:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3254.599312 secs 2024-07-26T12:00:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:12 smithi144 ceph-mon[32129]: pgmap v2302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:12 smithi183 ceph-mon[37273]: pgmap v2302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:12 smithi160 ceph-mon[36715]: pgmap v2302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:14.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:14.599+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:14.599+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:14.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:14.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:14.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:14 smithi160 ceph-mon[36715]: pgmap v2303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:14 smithi183 ceph-mon[37273]: pgmap v2303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:14 smithi144 ceph-mon[32129]: pgmap v2303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:16 smithi183 ceph-mon[37273]: pgmap v2304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3259.599373 secs 2024-07-26T12:00:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:16 smithi144 ceph-mon[32129]: pgmap v2304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3259.599373 secs 2024-07-26T12:00:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:16 smithi160 ceph-mon[36715]: pgmap v2304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3259.599373 secs 2024-07-26T12:00:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:18 smithi183 ceph-mon[37273]: pgmap v2305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:18 smithi144 ceph-mon[32129]: pgmap v2305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:18 smithi160 ceph-mon[36715]: pgmap v2305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:20 smithi183 ceph-mon[37273]: pgmap v2306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:20 smithi144 ceph-mon[32129]: pgmap v2306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:20 smithi160 ceph-mon[36715]: pgmap v2306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3264.599493 secs 2024-07-26T12:00:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3264.599493 secs 2024-07-26T12:00:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3264.599493 secs 2024-07-26T12:00:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:00:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:22 smithi144 ceph-mon[32129]: pgmap v2307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:22 smithi144 ceph-mon[32129]: pgmap v2308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:22 smithi160 ceph-mon[36715]: pgmap v2307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:22 smithi160 ceph-mon[36715]: pgmap v2308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:22.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:22 smithi183 ceph-mon[37273]: pgmap v2307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:22.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:22 smithi183 ceph-mon[37273]: pgmap v2308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:24 smithi144 ceph-mon[32129]: pgmap v2309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:00:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:24 smithi160 ceph-mon[36715]: pgmap v2309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:00:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:24 smithi183 ceph-mon[37273]: pgmap v2309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:00:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:26 smithi144 ceph-mon[32129]: pgmap v2310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3269.599612 secs 2024-07-26T12:00:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:26 smithi160 ceph-mon[36715]: pgmap v2310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3269.599612 secs 2024-07-26T12:00:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:26 smithi183 ceph-mon[37273]: pgmap v2310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3269.599612 secs 2024-07-26T12:00:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:28 smithi144 ceph-mon[32129]: pgmap v2311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:28 smithi160 ceph-mon[36715]: pgmap v2311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:28 smithi183 ceph-mon[37273]: pgmap v2311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:29.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:29.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:29.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:29.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:29.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:30 smithi144 ceph-mon[32129]: pgmap v2312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:30 smithi160 ceph-mon[36715]: pgmap v2312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:30 smithi183 ceph-mon[37273]: pgmap v2312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3274.599732 secs 2024-07-26T12:00:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3274.599732 secs 2024-07-26T12:00:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3274.599732 secs 2024-07-26T12:00:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:32 smithi144 ceph-mon[32129]: pgmap v2313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:00:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:32 smithi160 ceph-mon[36715]: pgmap v2313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:00:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:32 smithi183 ceph-mon[37273]: pgmap v2313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:00:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:34 smithi144 ceph-mon[32129]: pgmap v2314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:00:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:34 smithi160 ceph-mon[36715]: pgmap v2314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:00:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:34 smithi183 ceph-mon[37273]: pgmap v2314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:00:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:36 smithi144 ceph-mon[32129]: pgmap v2315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3279.599885 secs 2024-07-26T12:00:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:36 smithi160 ceph-mon[36715]: pgmap v2315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3279.599885 secs 2024-07-26T12:00:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:36 smithi183 ceph-mon[37273]: pgmap v2315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3279.599885 secs 2024-07-26T12:00:38.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:38 smithi144 ceph-mon[32129]: pgmap v2316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:38 smithi160 ceph-mon[36715]: pgmap v2316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:38 smithi183 ceph-mon[37273]: pgmap v2316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:40 smithi144 ceph-mon[32129]: pgmap v2317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:40 smithi160 ceph-mon[36715]: pgmap v2317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:40 smithi183 ceph-mon[37273]: pgmap v2317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3284.600030 secs 2024-07-26T12:00:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3284.600030 secs 2024-07-26T12:00:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3284.600030 secs 2024-07-26T12:00:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:42 smithi144 ceph-mon[32129]: pgmap v2318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:42 smithi160 ceph-mon[36715]: pgmap v2318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:42 smithi183 ceph-mon[37273]: pgmap v2318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:00:44.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:44.632+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:44.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:44.633+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:44.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:44.633+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:44.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:44.633+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:44.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:44.633+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:44 smithi160 ceph-mon[36715]: pgmap v2319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:44 smithi183 ceph-mon[37273]: pgmap v2319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:44 smithi144 ceph-mon[32129]: pgmap v2319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:46 smithi160 ceph-mon[36715]: pgmap v2320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3289.600152 secs 2024-07-26T12:00:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:46 smithi183 ceph-mon[37273]: pgmap v2320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3289.600152 secs 2024-07-26T12:00:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:46 smithi144 ceph-mon[32129]: pgmap v2320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3289.600152 secs 2024-07-26T12:00:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:48 smithi160 ceph-mon[36715]: pgmap v2321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:48 smithi183 ceph-mon[37273]: pgmap v2321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:48 smithi144 ceph-mon[32129]: pgmap v2321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:50 smithi160 ceph-mon[36715]: pgmap v2322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:50 smithi183 ceph-mon[37273]: pgmap v2322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:50 smithi144 ceph-mon[32129]: pgmap v2322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3294.600242 secs 2024-07-26T12:00:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3294.600242 secs 2024-07-26T12:00:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3294.600242 secs 2024-07-26T12:00:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:52 smithi160 ceph-mon[36715]: pgmap v2323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:52.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:52 smithi144 ceph-mon[32129]: pgmap v2323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:52 smithi183 ceph-mon[37273]: pgmap v2323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:54 smithi160 ceph-mon[36715]: pgmap v2324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:54 smithi183 ceph-mon[37273]: pgmap v2324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:54 smithi144 ceph-mon[32129]: pgmap v2324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:56 smithi160 ceph-mon[36715]: pgmap v2325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3299.600364 secs 2024-07-26T12:00:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:56 smithi183 ceph-mon[37273]: pgmap v2325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3299.600364 secs 2024-07-26T12:00:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:56 smithi144 ceph-mon[32129]: pgmap v2325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:00:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3299.600364 secs 2024-07-26T12:00:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:00:58 smithi160 ceph-mon[36715]: pgmap v2326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:00:58 smithi183 ceph-mon[37273]: pgmap v2326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:00:58 smithi144 ceph-mon[32129]: pgmap v2326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:00:59.650 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:59.648+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:59.650+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:59.650+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:59.650+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:00:59.650+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:00 smithi160 ceph-mon[36715]: pgmap v2327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:00 smithi183 ceph-mon[37273]: pgmap v2327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:00 smithi144 ceph-mon[32129]: pgmap v2327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3304.600456 secs 2024-07-26T12:01:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3304.600456 secs 2024-07-26T12:01:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3304.600456 secs 2024-07-26T12:01:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:02 smithi160 ceph-mon[36715]: pgmap v2328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:02 smithi183 ceph-mon[37273]: pgmap v2328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:02 smithi144 ceph-mon[32129]: pgmap v2328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:04 smithi160 ceph-mon[36715]: pgmap v2329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:04 smithi183 ceph-mon[37273]: pgmap v2329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:04 smithi144 ceph-mon[32129]: pgmap v2329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:06 smithi160 ceph-mon[36715]: pgmap v2330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3309.600523 secs 2024-07-26T12:01:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:06 smithi183 ceph-mon[37273]: pgmap v2330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3309.600523 secs 2024-07-26T12:01:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:06 smithi144 ceph-mon[32129]: pgmap v2330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3309.600523 secs 2024-07-26T12:01:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:08 smithi160 ceph-mon[36715]: pgmap v2331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:08 smithi183 ceph-mon[37273]: pgmap v2331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:08 smithi144 ceph-mon[32129]: pgmap v2331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:10 smithi183 ceph-mon[37273]: pgmap v2332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:10 smithi144 ceph-mon[32129]: pgmap v2332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:10 smithi160 ceph-mon[36715]: pgmap v2332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3314.600652 secs 2024-07-26T12:01:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3314.600652 secs 2024-07-26T12:01:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3314.600652 secs 2024-07-26T12:01:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:12 smithi183 ceph-mon[37273]: pgmap v2333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:12 smithi144 ceph-mon[32129]: pgmap v2333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:12 smithi160 ceph-mon[36715]: pgmap v2333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:14.666+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:14.667+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:14.667+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:14.667+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:14.667+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:14.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:14 smithi160 ceph-mon[36715]: pgmap v2334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:14 smithi183 ceph-mon[37273]: pgmap v2334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:14 smithi144 ceph-mon[32129]: pgmap v2334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:16 smithi183 ceph-mon[37273]: pgmap v2335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3319.600780 secs 2024-07-26T12:01:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:16 smithi144 ceph-mon[32129]: pgmap v2335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3319.600780 secs 2024-07-26T12:01:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:16 smithi160 ceph-mon[36715]: pgmap v2335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3319.600780 secs 2024-07-26T12:01:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:18 smithi183 ceph-mon[37273]: pgmap v2336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:18 smithi144 ceph-mon[32129]: pgmap v2336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:18 smithi160 ceph-mon[36715]: pgmap v2336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:20 smithi183 ceph-mon[37273]: pgmap v2337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:20 smithi144 ceph-mon[32129]: pgmap v2337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:20 smithi160 ceph-mon[36715]: pgmap v2337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3324.600902 secs 2024-07-26T12:01:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3324.600902 secs 2024-07-26T12:01:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3324.600902 secs 2024-07-26T12:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:22 smithi183 ceph-mon[37273]: pgmap v2338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:22 smithi144 ceph-mon[32129]: pgmap v2338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:22 smithi160 ceph-mon[36715]: pgmap v2338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:24.327 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:24 smithi144 ceph-mon[32129]: pgmap v2339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:24.327 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.327 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.327 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.327 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:24 smithi160 ceph-mon[36715]: pgmap v2339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:24 smithi183 ceph-mon[37273]: pgmap v2339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:25 smithi160 ceph-mon[36715]: pgmap v2340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:25 smithi183 ceph-mon[37273]: pgmap v2340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:25 smithi144 ceph-mon[32129]: pgmap v2340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3329.601023 secs 2024-07-26T12:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:26 smithi160 ceph-mon[36715]: pgmap v2341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:26 smithi160 ceph-mon[36715]: pgmap v2342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3329.601023 secs 2024-07-26T12:01:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:26 smithi183 ceph-mon[37273]: pgmap v2341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:26 smithi183 ceph-mon[37273]: pgmap v2342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:01:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3329.601023 secs 2024-07-26T12:01:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:26 smithi144 ceph-mon[32129]: pgmap v2341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:26 smithi144 ceph-mon[32129]: pgmap v2342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:01:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:01:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:29 smithi183 ceph-mon[37273]: pgmap v2343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:29 smithi144 ceph-mon[32129]: pgmap v2343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:29 smithi160 ceph-mon[36715]: pgmap v2343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:29.682+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:29.683+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:29.683+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:29.683+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:29.683+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:31 smithi183 ceph-mon[37273]: pgmap v2344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:31 smithi144 ceph-mon[32129]: pgmap v2344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:31 smithi160 ceph-mon[36715]: pgmap v2344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3334.601148 secs 2024-07-26T12:01:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3334.601148 secs 2024-07-26T12:01:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3334.601148 secs 2024-07-26T12:01:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:33 smithi183 ceph-mon[37273]: pgmap v2345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:01:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:33 smithi144 ceph-mon[32129]: pgmap v2345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:01:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:33 smithi160 ceph-mon[36715]: pgmap v2345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:01:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:35 smithi183 ceph-mon[37273]: pgmap v2346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:01:35.393 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:35 smithi144 ceph-mon[32129]: pgmap v2346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:01:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:35 smithi160 ceph-mon[36715]: pgmap v2346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:01:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3339.601267 secs 2024-07-26T12:01:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:37 smithi183 ceph-mon[37273]: pgmap v2347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:01:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3339.601267 secs 2024-07-26T12:01:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:37 smithi144 ceph-mon[32129]: pgmap v2347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:01:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3339.601267 secs 2024-07-26T12:01:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:37 smithi160 ceph-mon[36715]: pgmap v2347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:01:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:39 smithi183 ceph-mon[37273]: pgmap v2348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:39 smithi144 ceph-mon[32129]: pgmap v2348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:39 smithi160 ceph-mon[36715]: pgmap v2348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:41 smithi183 ceph-mon[37273]: pgmap v2349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3344.601361 secs 2024-07-26T12:01:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:41 smithi144 ceph-mon[32129]: pgmap v2349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3344.601361 secs 2024-07-26T12:01:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:41 smithi160 ceph-mon[36715]: pgmap v2349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3344.601361 secs 2024-07-26T12:01:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:43 smithi183 ceph-mon[37273]: pgmap v2350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:43 smithi144 ceph-mon[32129]: pgmap v2350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:43 smithi160 ceph-mon[36715]: pgmap v2350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:44.699+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:44.701+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:44.701+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:44.701+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:44.701+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:45.379 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:45 smithi160 ceph-mon[36715]: pgmap v2351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:45 smithi183 ceph-mon[37273]: pgmap v2351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:45 smithi144 ceph-mon[32129]: pgmap v2351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3349.601424 secs 2024-07-26T12:01:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3349.601424 secs 2024-07-26T12:01:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3349.601424 secs 2024-07-26T12:01:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:47 smithi183 ceph-mon[37273]: pgmap v2352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:47 smithi144 ceph-mon[32129]: pgmap v2352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:47 smithi160 ceph-mon[36715]: pgmap v2352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:49 smithi144 ceph-mon[32129]: pgmap v2353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:49 smithi160 ceph-mon[36715]: pgmap v2353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:49 smithi183 ceph-mon[37273]: pgmap v2353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:51 smithi144 ceph-mon[32129]: pgmap v2354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3354.601518 secs 2024-07-26T12:01:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:51 smithi160 ceph-mon[36715]: pgmap v2354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3354.601518 secs 2024-07-26T12:01:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:51 smithi183 ceph-mon[37273]: pgmap v2354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3354.601518 secs 2024-07-26T12:01:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:53 smithi144 ceph-mon[32129]: pgmap v2355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:53 smithi160 ceph-mon[36715]: pgmap v2355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:53 smithi183 ceph-mon[37273]: pgmap v2355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:01:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:55 smithi144 ceph-mon[32129]: pgmap v2356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:55 smithi160 ceph-mon[36715]: pgmap v2356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:55 smithi183 ceph-mon[37273]: pgmap v2356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3359.601644 secs 2024-07-26T12:01:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3359.601644 secs 2024-07-26T12:01:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3359.601644 secs 2024-07-26T12:01:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:57 smithi144 ceph-mon[32129]: pgmap v2357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:57 smithi160 ceph-mon[36715]: pgmap v2357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:57 smithi183 ceph-mon[37273]: pgmap v2357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:01:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:01:59 smithi144 ceph-mon[32129]: pgmap v2358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:01:59 smithi160 ceph-mon[36715]: pgmap v2358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:01:59 smithi183 ceph-mon[37273]: pgmap v2358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:59.703+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:59.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:59.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:59.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:59.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:59.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:59.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:59.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:01:59.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:01 smithi144 ceph-mon[32129]: pgmap v2359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3364.601766 secs 2024-07-26T12:02:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:01 smithi160 ceph-mon[36715]: pgmap v2359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3364.601766 secs 2024-07-26T12:02:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:01 smithi183 ceph-mon[37273]: pgmap v2359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3364.601766 secs 2024-07-26T12:02:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:03 smithi144 ceph-mon[32129]: pgmap v2360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:03 smithi160 ceph-mon[36715]: pgmap v2360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:03 smithi183 ceph-mon[37273]: pgmap v2360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:05 smithi144 ceph-mon[32129]: pgmap v2361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:05 smithi160 ceph-mon[36715]: pgmap v2361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:05 smithi183 ceph-mon[37273]: pgmap v2361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3369.601843 secs 2024-07-26T12:02:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3369.601843 secs 2024-07-26T12:02:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3369.601843 secs 2024-07-26T12:02:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:07 smithi144 ceph-mon[32129]: pgmap v2362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:07 smithi160 ceph-mon[36715]: pgmap v2362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:07 smithi183 ceph-mon[37273]: pgmap v2362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:09 smithi160 ceph-mon[36715]: pgmap v2363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:09 smithi183 ceph-mon[37273]: pgmap v2363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:09 smithi144 ceph-mon[32129]: pgmap v2363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:11 smithi160 ceph-mon[36715]: pgmap v2364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3374.601963 secs 2024-07-26T12:02:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:11 smithi183 ceph-mon[37273]: pgmap v2364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3374.601963 secs 2024-07-26T12:02:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:11 smithi144 ceph-mon[32129]: pgmap v2364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3374.601963 secs 2024-07-26T12:02:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:13 smithi160 ceph-mon[36715]: pgmap v2365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:13 smithi183 ceph-mon[37273]: pgmap v2365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:13 smithi144 ceph-mon[32129]: pgmap v2365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:14.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:14.709+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:14.710+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:14.710+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:14.710+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:14.711+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:15 smithi160 ceph-mon[36715]: pgmap v2366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:15 smithi183 ceph-mon[37273]: pgmap v2366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:15 smithi144 ceph-mon[32129]: pgmap v2366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3379.602058 secs 2024-07-26T12:02:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3379.602058 secs 2024-07-26T12:02:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3379.602058 secs 2024-07-26T12:02:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:17 smithi144 ceph-mon[32129]: pgmap v2367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:17 smithi160 ceph-mon[36715]: pgmap v2367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:17 smithi183 ceph-mon[37273]: pgmap v2367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:19 smithi160 ceph-mon[36715]: pgmap v2368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:19 smithi183 ceph-mon[37273]: pgmap v2368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:19 smithi144 ceph-mon[32129]: pgmap v2368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:21 smithi160 ceph-mon[36715]: pgmap v2369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3384.602193 secs 2024-07-26T12:02:21.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:21 smithi183 ceph-mon[37273]: pgmap v2369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:21.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3384.602193 secs 2024-07-26T12:02:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:21 smithi144 ceph-mon[32129]: pgmap v2369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3384.602193 secs 2024-07-26T12:02:23.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:23 smithi144 ceph-mon[32129]: pgmap v2370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:23 smithi160 ceph-mon[36715]: pgmap v2370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:23 smithi183 ceph-mon[37273]: pgmap v2370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:25 smithi160 ceph-mon[36715]: pgmap v2371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:25 smithi183 ceph-mon[37273]: pgmap v2371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:25 smithi144 ceph-mon[32129]: pgmap v2371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3389.602358 secs 2024-07-26T12:02:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3389.602358 secs 2024-07-26T12:02:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3389.602358 secs 2024-07-26T12:02:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:27 smithi160 ceph-mon[36715]: pgmap v2372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:02:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:27 smithi183 ceph-mon[37273]: pgmap v2372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:02:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:27 smithi144 ceph-mon[32129]: pgmap v2372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:02:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:28 smithi160 ceph-mon[36715]: pgmap v2373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:02:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:28 smithi160 ceph-mon[36715]: pgmap v2374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:28 smithi183 ceph-mon[37273]: pgmap v2373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:02:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:28 smithi183 ceph-mon[37273]: pgmap v2374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:28 smithi144 ceph-mon[32129]: pgmap v2373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:02:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:28 smithi144 ceph-mon[32129]: pgmap v2374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:29.718+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:29.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:29.719+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:29.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:29.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:30 smithi160 ceph-mon[36715]: pgmap v2375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:30 smithi183 ceph-mon[37273]: pgmap v2375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:30 smithi144 ceph-mon[32129]: pgmap v2375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3394.602477 secs 2024-07-26T12:02:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3394.602477 secs 2024-07-26T12:02:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3394.602477 secs 2024-07-26T12:02:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:32 smithi183 ceph-mon[37273]: pgmap v2376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:32 smithi144 ceph-mon[32129]: pgmap v2376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:32 smithi160 ceph-mon[36715]: pgmap v2376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:34 smithi183 ceph-mon[37273]: pgmap v2377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:34 smithi144 ceph-mon[32129]: pgmap v2377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:34 smithi160 ceph-mon[36715]: pgmap v2377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:36 smithi183 ceph-mon[37273]: pgmap v2378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3399.602599 secs 2024-07-26T12:02:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:36 smithi144 ceph-mon[32129]: pgmap v2378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3399.602599 secs 2024-07-26T12:02:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:36 smithi160 ceph-mon[36715]: pgmap v2378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:02:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3399.602599 secs 2024-07-26T12:02:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:38 smithi183 ceph-mon[37273]: pgmap v2379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:02:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:38 smithi144 ceph-mon[32129]: pgmap v2379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:02:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:38 smithi160 ceph-mon[36715]: pgmap v2379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:02:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:40 smithi183 ceph-mon[37273]: pgmap v2380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:40 smithi144 ceph-mon[32129]: pgmap v2380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:40 smithi160 ceph-mon[36715]: pgmap v2380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3404.602682 secs 2024-07-26T12:02:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3404.602682 secs 2024-07-26T12:02:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3404.602682 secs 2024-07-26T12:02:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:42 smithi183 ceph-mon[37273]: pgmap v2381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:42 smithi144 ceph-mon[32129]: pgmap v2381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:42 smithi160 ceph-mon[36715]: pgmap v2381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:44 smithi183 ceph-mon[37273]: pgmap v2382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:44 smithi144 ceph-mon[32129]: pgmap v2382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:44.729+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:44.731+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:44.731+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:44.731+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:44.731+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:44 smithi160 ceph-mon[36715]: pgmap v2382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:46 smithi183 ceph-mon[37273]: pgmap v2383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3409.602824 secs 2024-07-26T12:02:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:46.694+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:02:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:46.694+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:02:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:46.695+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:02:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:46.695+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T12:02:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:46 smithi144 ceph-mon[32129]: pgmap v2383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3409.602824 secs 2024-07-26T12:02:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:46 smithi160 ceph-mon[36715]: pgmap v2383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3409.602824 secs 2024-07-26T12:02:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:48 smithi183 ceph-mon[37273]: pgmap v2384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:48 smithi144 ceph-mon[32129]: pgmap v2384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:48 smithi160 ceph-mon[36715]: pgmap v2384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:50 smithi183 ceph-mon[37273]: pgmap v2385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:50 smithi144 ceph-mon[32129]: pgmap v2385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:50 smithi160 ceph-mon[36715]: pgmap v2385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3414.602946 secs 2024-07-26T12:02:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3414.602946 secs 2024-07-26T12:02:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3414.602946 secs 2024-07-26T12:02:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:52 smithi144 ceph-mon[32129]: pgmap v2386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:52 smithi160 ceph-mon[36715]: pgmap v2386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:52 smithi183 ceph-mon[37273]: pgmap v2386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:02:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:54 smithi144 ceph-mon[32129]: pgmap v2387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:54 smithi160 ceph-mon[36715]: pgmap v2387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:54 smithi183 ceph-mon[37273]: pgmap v2387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:56 smithi144 ceph-mon[32129]: pgmap v2388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3419.603033 secs 2024-07-26T12:02:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:56 smithi160 ceph-mon[36715]: pgmap v2388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3419.603033 secs 2024-07-26T12:02:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:56 smithi183 ceph-mon[37273]: pgmap v2388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:02:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3419.603033 secs 2024-07-26T12:02:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:02:58 smithi144 ceph-mon[32129]: pgmap v2389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:02:58 smithi160 ceph-mon[36715]: pgmap v2389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:02:58 smithi183 ceph-mon[37273]: pgmap v2389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:59.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:59.739+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:59.739+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:59.739+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:02:59.739+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:00 smithi144 ceph-mon[32129]: pgmap v2390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:00 smithi160 ceph-mon[36715]: pgmap v2390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:00 smithi183 ceph-mon[37273]: pgmap v2390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3424.603163 secs 2024-07-26T12:03:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3424.603163 secs 2024-07-26T12:03:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3424.603163 secs 2024-07-26T12:03:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:02 smithi144 ceph-mon[32129]: pgmap v2391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:02 smithi160 ceph-mon[36715]: pgmap v2391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:02 smithi183 ceph-mon[37273]: pgmap v2391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:04 smithi144 ceph-mon[32129]: pgmap v2392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:04 smithi160 ceph-mon[36715]: pgmap v2392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:04 smithi183 ceph-mon[37273]: pgmap v2392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:06 smithi144 ceph-mon[32129]: pgmap v2393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3429.603323 secs 2024-07-26T12:03:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:06 smithi160 ceph-mon[36715]: pgmap v2393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3429.603323 secs 2024-07-26T12:03:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:06 smithi183 ceph-mon[37273]: pgmap v2393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3429.603323 secs 2024-07-26T12:03:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:08 smithi144 ceph-mon[32129]: pgmap v2394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:08 smithi160 ceph-mon[36715]: pgmap v2394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:08 smithi183 ceph-mon[37273]: pgmap v2394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:10 smithi144 ceph-mon[32129]: pgmap v2395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:10 smithi160 ceph-mon[36715]: pgmap v2395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:10 smithi183 ceph-mon[37273]: pgmap v2395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3434.603370 secs 2024-07-26T12:03:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3434.603370 secs 2024-07-26T12:03:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3434.603370 secs 2024-07-26T12:03:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:12 smithi144 ceph-mon[32129]: pgmap v2396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:12 smithi160 ceph-mon[36715]: pgmap v2396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:12 smithi183 ceph-mon[37273]: pgmap v2396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:14.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:14.754+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:14.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:14.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:14.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:14.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:14 smithi160 ceph-mon[36715]: pgmap v2397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:14 smithi183 ceph-mon[37273]: pgmap v2397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:14 smithi144 ceph-mon[32129]: pgmap v2397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:16 smithi160 ceph-mon[36715]: pgmap v2398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3439.603435 secs 2024-07-26T12:03:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:16 smithi183 ceph-mon[37273]: pgmap v2398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3439.603435 secs 2024-07-26T12:03:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:16 smithi144 ceph-mon[32129]: pgmap v2398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3439.603435 secs 2024-07-26T12:03:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:18 smithi160 ceph-mon[36715]: pgmap v2399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:18 smithi183 ceph-mon[37273]: pgmap v2399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:18 smithi144 ceph-mon[32129]: pgmap v2399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:20 smithi160 ceph-mon[36715]: pgmap v2400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:03:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:20 smithi183 ceph-mon[37273]: pgmap v2400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:03:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:20 smithi144 ceph-mon[32129]: pgmap v2400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:03:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3444.603541 secs 2024-07-26T12:03:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3444.603541 secs 2024-07-26T12:03:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3444.603541 secs 2024-07-26T12:03:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:22 smithi160 ceph-mon[36715]: pgmap v2401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:22 smithi183 ceph-mon[37273]: pgmap v2401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:22 smithi144 ceph-mon[32129]: pgmap v2401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:24.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:24 smithi144 ceph-mon[32129]: pgmap v2402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:24 smithi160 ceph-mon[36715]: pgmap v2402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:24 smithi183 ceph-mon[37273]: pgmap v2402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:26 smithi160 ceph-mon[36715]: pgmap v2403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3449.603604 secs 2024-07-26T12:03:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:26 smithi183 ceph-mon[37273]: pgmap v2403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3449.603604 secs 2024-07-26T12:03:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:26 smithi144 ceph-mon[32129]: pgmap v2403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3449.603604 secs 2024-07-26T12:03:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:28 smithi160 ceph-mon[36715]: pgmap v2404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:03:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:28 smithi183 ceph-mon[37273]: pgmap v2404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:03:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:28 smithi144 ceph-mon[32129]: pgmap v2404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:03:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:29.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:29.772+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:29.772+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:29.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:29.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:29 smithi160 ceph-mon[36715]: pgmap v2405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:29 smithi160 ceph-mon[36715]: pgmap v2406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:03:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:29 smithi183 ceph-mon[37273]: pgmap v2405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:29 smithi183 ceph-mon[37273]: pgmap v2406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:03:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:29 smithi144 ceph-mon[32129]: pgmap v2405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:29 smithi144 ceph-mon[32129]: pgmap v2406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:03:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:31.703+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:03:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:31.703+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:03:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:31.703+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:03:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:31.703+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:03:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:31 smithi160 ceph-mon[36715]: pgmap v2407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3454.603704 secs 2024-07-26T12:03:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:31 smithi183 ceph-mon[37273]: pgmap v2407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3454.603704 secs 2024-07-26T12:03:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:31 smithi144 ceph-mon[32129]: pgmap v2407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3454.603704 secs 2024-07-26T12:03:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:33 smithi160 ceph-mon[36715]: pgmap v2408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:33 smithi183 ceph-mon[37273]: pgmap v2408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:33 smithi144 ceph-mon[32129]: pgmap v2408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:35 smithi160 ceph-mon[36715]: pgmap v2409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:35 smithi183 ceph-mon[37273]: pgmap v2409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:35 smithi144 ceph-mon[32129]: pgmap v2409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:03:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3459.603841 secs 2024-07-26T12:03:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3459.603841 secs 2024-07-26T12:03:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3459.603841 secs 2024-07-26T12:03:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:37 smithi183 ceph-mon[37273]: pgmap v2410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:37 smithi144 ceph-mon[32129]: pgmap v2410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:37 smithi160 ceph-mon[36715]: pgmap v2410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:03:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:39 smithi183 ceph-mon[37273]: pgmap v2411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:39 smithi144 ceph-mon[32129]: pgmap v2411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:39 smithi160 ceph-mon[36715]: pgmap v2411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:41 smithi183 ceph-mon[37273]: pgmap v2412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3464.603962 secs 2024-07-26T12:03:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:41 smithi144 ceph-mon[32129]: pgmap v2412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3464.603962 secs 2024-07-26T12:03:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:41 smithi160 ceph-mon[36715]: pgmap v2412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3464.603962 secs 2024-07-26T12:03:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:43 smithi183 ceph-mon[37273]: pgmap v2413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:43 smithi144 ceph-mon[32129]: pgmap v2413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:43 smithi160 ceph-mon[36715]: pgmap v2413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:44.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:44.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:44.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:44.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:44.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:45 smithi183 ceph-mon[37273]: pgmap v2414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:45 smithi144 ceph-mon[32129]: pgmap v2414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:45 smithi160 ceph-mon[36715]: pgmap v2414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3469.604103 secs 2024-07-26T12:03:46.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3469.604103 secs 2024-07-26T12:03:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3469.604103 secs 2024-07-26T12:03:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:47 smithi183 ceph-mon[37273]: pgmap v2415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:47 smithi144 ceph-mon[32129]: pgmap v2415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:47 smithi160 ceph-mon[36715]: pgmap v2415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:49 smithi183 ceph-mon[37273]: pgmap v2416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:49 smithi144 ceph-mon[32129]: pgmap v2416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:49 smithi160 ceph-mon[36715]: pgmap v2416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:51 smithi183 ceph-mon[37273]: pgmap v2417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3474.604241 secs 2024-07-26T12:03:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:51 smithi144 ceph-mon[32129]: pgmap v2417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3474.604241 secs 2024-07-26T12:03:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:51 smithi160 ceph-mon[36715]: pgmap v2417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3474.604241 secs 2024-07-26T12:03:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:53 smithi183 ceph-mon[37273]: pgmap v2418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:53 smithi144 ceph-mon[32129]: pgmap v2418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:53 smithi160 ceph-mon[36715]: pgmap v2418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:55 smithi144 ceph-mon[32129]: pgmap v2419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:55 smithi160 ceph-mon[36715]: pgmap v2419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:55 smithi183 ceph-mon[37273]: pgmap v2419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3479.604380 secs 2024-07-26T12:03:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3479.604380 secs 2024-07-26T12:03:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3479.604380 secs 2024-07-26T12:03:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:57 smithi160 ceph-mon[36715]: pgmap v2420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:57.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:57 smithi183 ceph-mon[37273]: pgmap v2420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:57 smithi144 ceph-mon[32129]: pgmap v2420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:03:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:59.804+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:59.805+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:59.805+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:59.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:03:59.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:03:59 smithi160 ceph-mon[36715]: pgmap v2421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:59.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:03:59 smithi183 ceph-mon[37273]: pgmap v2421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:03:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:03:59 smithi144 ceph-mon[32129]: pgmap v2421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:01 smithi160 ceph-mon[36715]: pgmap v2422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3484.604491 secs 2024-07-26T12:04:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:01 smithi183 ceph-mon[37273]: pgmap v2422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3484.604491 secs 2024-07-26T12:04:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:01 smithi144 ceph-mon[32129]: pgmap v2422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3484.604491 secs 2024-07-26T12:04:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:03 smithi160 ceph-mon[36715]: pgmap v2423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:03 smithi183 ceph-mon[37273]: pgmap v2423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:03 smithi144 ceph-mon[32129]: pgmap v2423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:05 smithi160 ceph-mon[36715]: pgmap v2424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:05 smithi183 ceph-mon[37273]: pgmap v2424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:05 smithi144 ceph-mon[32129]: pgmap v2424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3489.604571 secs 2024-07-26T12:04:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3489.604571 secs 2024-07-26T12:04:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3489.604571 secs 2024-07-26T12:04:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:07 smithi160 ceph-mon[36715]: pgmap v2425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:07 smithi183 ceph-mon[37273]: pgmap v2425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:07 smithi144 ceph-mon[32129]: pgmap v2425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:09 smithi160 ceph-mon[36715]: pgmap v2426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:09 smithi183 ceph-mon[37273]: pgmap v2426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:09 smithi144 ceph-mon[32129]: pgmap v2426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:11 smithi160 ceph-mon[36715]: pgmap v2427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3494.604685 secs 2024-07-26T12:04:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:11 smithi183 ceph-mon[37273]: pgmap v2427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3494.604685 secs 2024-07-26T12:04:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:11 smithi144 ceph-mon[32129]: pgmap v2427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3494.604685 secs 2024-07-26T12:04:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:13 smithi160 ceph-mon[36715]: pgmap v2428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:13 smithi183 ceph-mon[37273]: pgmap v2428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:13 smithi144 ceph-mon[32129]: pgmap v2428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:14.821+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:14.822+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:14.822+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:14.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:14.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:15 smithi160 ceph-mon[36715]: pgmap v2429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:15 smithi183 ceph-mon[37273]: pgmap v2429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:15 smithi144 ceph-mon[32129]: pgmap v2429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3499.604809 secs 2024-07-26T12:04:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3499.604809 secs 2024-07-26T12:04:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3499.604809 secs 2024-07-26T12:04:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:17 smithi160 ceph-mon[36715]: pgmap v2430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:17 smithi183 ceph-mon[37273]: pgmap v2430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:17 smithi144 ceph-mon[32129]: pgmap v2430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:19 smithi183 ceph-mon[37273]: pgmap v2431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:19 smithi144 ceph-mon[32129]: pgmap v2431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:19 smithi160 ceph-mon[36715]: pgmap v2431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:21 smithi183 ceph-mon[37273]: pgmap v2432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3504.604927 secs 2024-07-26T12:04:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:21 smithi144 ceph-mon[32129]: pgmap v2432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3504.604927 secs 2024-07-26T12:04:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:21 smithi160 ceph-mon[36715]: pgmap v2432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3504.604927 secs 2024-07-26T12:04:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:23 smithi183 ceph-mon[37273]: pgmap v2433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:23 smithi144 ceph-mon[32129]: pgmap v2433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:23 smithi160 ceph-mon[36715]: pgmap v2433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:25 smithi160 ceph-mon[36715]: pgmap v2434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:25 smithi183 ceph-mon[37273]: pgmap v2434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:25 smithi144 ceph-mon[32129]: pgmap v2434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3509.605043 secs 2024-07-26T12:04:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3509.605043 secs 2024-07-26T12:04:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3509.605043 secs 2024-07-26T12:04:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:27 smithi160 ceph-mon[36715]: pgmap v2435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:27 smithi183 ceph-mon[37273]: pgmap v2435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:27 smithi144 ceph-mon[32129]: pgmap v2435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:29 smithi160 ceph-mon[36715]: pgmap v2436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:29.833+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:29.834+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:29.834+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:29.834+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:29.834+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:29 smithi183 ceph-mon[37273]: pgmap v2436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:29 smithi144 ceph-mon[32129]: pgmap v2436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:30 smithi160 ceph-mon[36715]: pgmap v2437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:04:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:30 smithi160 ceph-mon[36715]: pgmap v2438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:04:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:04:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:30 smithi183 ceph-mon[37273]: pgmap v2437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:04:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:30 smithi183 ceph-mon[37273]: pgmap v2438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:04:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:04:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:30 smithi144 ceph-mon[32129]: pgmap v2437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:04:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:30 smithi144 ceph-mon[32129]: pgmap v2438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:04:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:04:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3514.605135 secs 2024-07-26T12:04:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3514.605135 secs 2024-07-26T12:04:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3514.605135 secs 2024-07-26T12:04:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:32 smithi160 ceph-mon[36715]: pgmap v2439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:04:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:32 smithi183 ceph-mon[37273]: pgmap v2439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:04:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:32 smithi144 ceph-mon[32129]: pgmap v2439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:04:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:34 smithi144 ceph-mon[32129]: pgmap v2440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:04:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:34 smithi160 ceph-mon[36715]: pgmap v2440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:04:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:34 smithi183 ceph-mon[37273]: pgmap v2440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:04:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:36 smithi144 ceph-mon[32129]: pgmap v2441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:04:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3519.605257 secs 2024-07-26T12:04:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:36 smithi160 ceph-mon[36715]: pgmap v2441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:04:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3519.605257 secs 2024-07-26T12:04:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:36 smithi183 ceph-mon[37273]: pgmap v2441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:04:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3519.605257 secs 2024-07-26T12:04:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:38 smithi144 ceph-mon[32129]: pgmap v2442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:04:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:38 smithi160 ceph-mon[36715]: pgmap v2442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:04:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:38 smithi183 ceph-mon[37273]: pgmap v2442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:04:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:40 smithi144 ceph-mon[32129]: pgmap v2443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:04:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:40 smithi160 ceph-mon[36715]: pgmap v2443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:04:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:40 smithi183 ceph-mon[37273]: pgmap v2443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:04:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3524.605381 secs 2024-07-26T12:04:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3524.605381 secs 2024-07-26T12:04:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3524.605381 secs 2024-07-26T12:04:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:42 smithi144 ceph-mon[32129]: pgmap v2444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:42 smithi160 ceph-mon[36715]: pgmap v2444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:42 smithi183 ceph-mon[37273]: pgmap v2444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:44.851+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:44.853 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:44.852+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:44.853 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:44.852+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:44.853 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:44.852+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:44.853 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:44.852+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:44 smithi144 ceph-mon[32129]: pgmap v2445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:44 smithi160 ceph-mon[36715]: pgmap v2445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:44 smithi183 ceph-mon[37273]: pgmap v2445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:46 smithi144 ceph-mon[32129]: pgmap v2446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3529.605476 secs 2024-07-26T12:04:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:46 smithi160 ceph-mon[36715]: pgmap v2446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3529.605476 secs 2024-07-26T12:04:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:46 smithi183 ceph-mon[37273]: pgmap v2446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3529.605476 secs 2024-07-26T12:04:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:48 smithi144 ceph-mon[32129]: pgmap v2447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:48 smithi160 ceph-mon[36715]: pgmap v2447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:48 smithi183 ceph-mon[37273]: pgmap v2447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:50 smithi144 ceph-mon[32129]: pgmap v2448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:50 smithi160 ceph-mon[36715]: pgmap v2448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:50 smithi183 ceph-mon[37273]: pgmap v2448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3534.605544 secs 2024-07-26T12:04:52.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3534.605544 secs 2024-07-26T12:04:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3534.605544 secs 2024-07-26T12:04:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:52 smithi160 ceph-mon[36715]: pgmap v2449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:52 smithi183 ceph-mon[37273]: pgmap v2449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:52 smithi144 ceph-mon[32129]: pgmap v2449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:04:55.026 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:54 smithi144 ceph-mon[32129]: pgmap v2450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:54 smithi160 ceph-mon[36715]: pgmap v2450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:54 smithi183 ceph-mon[37273]: pgmap v2450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:57.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:56 smithi160 ceph-mon[36715]: pgmap v2451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:57.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3539.605673 secs 2024-07-26T12:04:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:56 smithi183 ceph-mon[37273]: pgmap v2451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3539.605673 secs 2024-07-26T12:04:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:56 smithi144 ceph-mon[32129]: pgmap v2451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:04:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3539.605673 secs 2024-07-26T12:04:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:04:58 smithi160 ceph-mon[36715]: pgmap v2452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:04:58 smithi183 ceph-mon[37273]: pgmap v2452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:04:58 smithi144 ceph-mon[32129]: pgmap v2452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:59.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:59.868+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:59.868+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:59.868+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:59.868+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:04:59.869+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:00 smithi160 ceph-mon[36715]: pgmap v2453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:00 smithi183 ceph-mon[37273]: pgmap v2453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:00 smithi144 ceph-mon[32129]: pgmap v2453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3544.605794 secs 2024-07-26T12:05:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3544.605794 secs 2024-07-26T12:05:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3544.605794 secs 2024-07-26T12:05:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:02 smithi160 ceph-mon[36715]: pgmap v2454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:02 smithi183 ceph-mon[37273]: pgmap v2454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:02 smithi144 ceph-mon[32129]: pgmap v2454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:04 smithi160 ceph-mon[36715]: pgmap v2455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:04 smithi183 ceph-mon[37273]: pgmap v2455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:04 smithi144 ceph-mon[32129]: pgmap v2455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:06 smithi160 ceph-mon[36715]: pgmap v2456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3549.605863 secs 2024-07-26T12:05:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:06 smithi183 ceph-mon[37273]: pgmap v2456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3549.605863 secs 2024-07-26T12:05:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:06 smithi144 ceph-mon[32129]: pgmap v2456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3549.605863 secs 2024-07-26T12:05:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:08 smithi160 ceph-mon[36715]: pgmap v2457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:08 smithi183 ceph-mon[37273]: pgmap v2457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:08 smithi144 ceph-mon[32129]: pgmap v2457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:10 smithi160 ceph-mon[36715]: pgmap v2458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:10 smithi183 ceph-mon[37273]: pgmap v2458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:10 smithi144 ceph-mon[32129]: pgmap v2458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3554.605956 secs 2024-07-26T12:05:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3554.605956 secs 2024-07-26T12:05:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3554.605956 secs 2024-07-26T12:05:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:12 smithi160 ceph-mon[36715]: pgmap v2459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:12 smithi183 ceph-mon[37273]: pgmap v2459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:12 smithi144 ceph-mon[32129]: pgmap v2459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:14.871+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:14.872+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:14.872+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:14.872+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:14.873+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:14 smithi183 ceph-mon[37273]: pgmap v2460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:14 smithi144 ceph-mon[32129]: pgmap v2460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:14 smithi160 ceph-mon[36715]: pgmap v2460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:17.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3559.606087 secs 2024-07-26T12:05:17.112 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:16 smithi160 ceph-mon[36715]: pgmap v2461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3559.606087 secs 2024-07-26T12:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:16 smithi183 ceph-mon[37273]: pgmap v2461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3559.606087 secs 2024-07-26T12:05:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:16 smithi144 ceph-mon[32129]: pgmap v2461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:18 smithi183 ceph-mon[37273]: pgmap v2462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:18 smithi144 ceph-mon[32129]: pgmap v2462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:18 smithi160 ceph-mon[36715]: pgmap v2462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:21.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:20 smithi183 ceph-mon[37273]: pgmap v2463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:20 smithi144 ceph-mon[32129]: pgmap v2463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:20 smithi160 ceph-mon[36715]: pgmap v2463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3564.606200 secs 2024-07-26T12:05:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3564.606200 secs 2024-07-26T12:05:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3564.606200 secs 2024-07-26T12:05:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:22 smithi183 ceph-mon[37273]: pgmap v2464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:22 smithi144 ceph-mon[32129]: pgmap v2464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:22 smithi160 ceph-mon[36715]: pgmap v2464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:24 smithi183 ceph-mon[37273]: pgmap v2465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:24 smithi144 ceph-mon[32129]: pgmap v2465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:24 smithi160 ceph-mon[36715]: pgmap v2465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3569.606270 secs 2024-07-26T12:05:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:26 smithi183 ceph-mon[37273]: pgmap v2466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3569.606270 secs 2024-07-26T12:05:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:26 smithi144 ceph-mon[32129]: pgmap v2466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3569.606270 secs 2024-07-26T12:05:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:26 smithi160 ceph-mon[36715]: pgmap v2466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:28 smithi183 ceph-mon[37273]: pgmap v2467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:28 smithi144 ceph-mon[32129]: pgmap v2467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:28 smithi160 ceph-mon[36715]: pgmap v2467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:29.887+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:29.888+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:29.888+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:29.888+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:29.888+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:30 smithi183 ceph-mon[37273]: pgmap v2468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:05:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:30 smithi144 ceph-mon[32129]: pgmap v2468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:05:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:30 smithi160 ceph-mon[36715]: pgmap v2468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:05:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3574.606339 secs 2024-07-26T12:05:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:31 smithi183 ceph-mon[37273]: pgmap v2469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:31 smithi183 ceph-mon[37273]: pgmap v2470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3574.606339 secs 2024-07-26T12:05:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:31 smithi144 ceph-mon[32129]: pgmap v2469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:31 smithi144 ceph-mon[32129]: pgmap v2470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3574.606339 secs 2024-07-26T12:05:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:31 smithi160 ceph-mon[36715]: pgmap v2469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:31 smithi160 ceph-mon[36715]: pgmap v2470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:33 smithi183 ceph-mon[37273]: pgmap v2471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:05:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:33 smithi144 ceph-mon[32129]: pgmap v2471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:05:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:33 smithi160 ceph-mon[36715]: pgmap v2471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:05:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:35 smithi144 ceph-mon[32129]: pgmap v2472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:05:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:35 smithi160 ceph-mon[36715]: pgmap v2472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:05:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:35 smithi183 ceph-mon[37273]: pgmap v2472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:05:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3579.606418 secs 2024-07-26T12:05:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3579.606418 secs 2024-07-26T12:05:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3579.606418 secs 2024-07-26T12:05:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:37 smithi144 ceph-mon[32129]: pgmap v2473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:37 smithi160 ceph-mon[36715]: pgmap v2473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:37 smithi183 ceph-mon[37273]: pgmap v2473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:05:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:39 smithi144 ceph-mon[32129]: pgmap v2474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:05:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:39 smithi160 ceph-mon[36715]: pgmap v2474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:05:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:39 smithi183 ceph-mon[37273]: pgmap v2474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:05:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3584.606529 secs 2024-07-26T12:05:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:41 smithi144 ceph-mon[32129]: pgmap v2475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:05:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3584.606529 secs 2024-07-26T12:05:42.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:41 smithi160 ceph-mon[36715]: pgmap v2475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:05:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3584.606529 secs 2024-07-26T12:05:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:41 smithi183 ceph-mon[37273]: pgmap v2475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:05:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:43 smithi144 ceph-mon[32129]: pgmap v2476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:43 smithi160 ceph-mon[36715]: pgmap v2476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:43 smithi183 ceph-mon[37273]: pgmap v2476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:44.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:44.905+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:44.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:44.906+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:44.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:44.906+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:44.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:44.906+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:44.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:44.906+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c22d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a804d070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c22d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a804d070 comp rx=0 tx=0).stop 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c22d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 msgr2=0x7f74900c22d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900bf390 0x7f74900c22d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:05:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.099+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 0x7f74900c2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:05:46.102 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.100+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 0x7f74900c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:46.103 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.102+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 0x7f74900c2810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f74a8031470 tx=0x7f74a8054000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:05:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:45 smithi144 ceph-mon[32129]: pgmap v2477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:45 smithi160 ceph-mon[36715]: pgmap v2477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:45 smithi183 ceph-mon[37273]: pgmap v2477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.727+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:05:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.727+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:05:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.728+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T12:05:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:46.728+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:05:47.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3589.606650 secs 2024-07-26T12:05:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3589.606650 secs 2024-07-26T12:05:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3589.606650 secs 2024-07-26T12:05:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:47 smithi144 ceph-mon[32129]: pgmap v2478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:47 smithi160 ceph-mon[36715]: pgmap v2478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:47 smithi183 ceph-mon[37273]: pgmap v2478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:49 smithi144 ceph-mon[32129]: pgmap v2479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:49 smithi160 ceph-mon[36715]: pgmap v2479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:49 smithi183 ceph-mon[37273]: pgmap v2479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3594.606760 secs 2024-07-26T12:05:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:51 smithi144 ceph-mon[32129]: pgmap v2480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3594.606760 secs 2024-07-26T12:05:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:51 smithi160 ceph-mon[36715]: pgmap v2480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3594.606760 secs 2024-07-26T12:05:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:51 smithi183 ceph-mon[37273]: pgmap v2480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:54.206 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:53 smithi144 ceph-mon[32129]: pgmap v2481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:53 smithi160 ceph-mon[36715]: pgmap v2481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:53 smithi183 ceph-mon[37273]: pgmap v2481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:55 smithi144 ceph-mon[32129]: pgmap v2482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:55 smithi160 ceph-mon[36715]: pgmap v2482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:55 smithi183 ceph-mon[37273]: pgmap v2482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3599.606893 secs 2024-07-26T12:05:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3599.606893 secs 2024-07-26T12:05:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3599.606893 secs 2024-07-26T12:05:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:57 smithi160 ceph-mon[36715]: pgmap v2483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:57 smithi183 ceph-mon[37273]: pgmap v2483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:57 smithi144 ceph-mon[32129]: pgmap v2483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:05:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:05:59 smithi160 ceph-mon[36715]: pgmap v2484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:05:59 smithi183 ceph-mon[37273]: pgmap v2484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:05:59 smithi144 ceph-mon[32129]: pgmap v2484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:05:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:59.918+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:59.919+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:59.919+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:59.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:05:59.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3604.607045 secs 2024-07-26T12:06:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:01 smithi183 ceph-mon[37273]: pgmap v2485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3604.607045 secs 2024-07-26T12:06:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:01 smithi144 ceph-mon[32129]: pgmap v2485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3604.607045 secs 2024-07-26T12:06:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:01 smithi160 ceph-mon[36715]: pgmap v2485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:03 smithi183 ceph-mon[37273]: pgmap v2486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:03 smithi144 ceph-mon[32129]: pgmap v2486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:03 smithi160 ceph-mon[36715]: pgmap v2486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:05 smithi144 ceph-mon[32129]: pgmap v2487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:05 smithi160 ceph-mon[36715]: pgmap v2487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:05 smithi183 ceph-mon[37273]: pgmap v2487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3609.607162 secs 2024-07-26T12:06:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3609.607162 secs 2024-07-26T12:06:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3609.607162 secs 2024-07-26T12:06:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:07 smithi144 ceph-mon[32129]: pgmap v2488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:07 smithi160 ceph-mon[36715]: pgmap v2488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:07 smithi183 ceph-mon[37273]: pgmap v2488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:09 smithi144 ceph-mon[32129]: pgmap v2489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:09 smithi160 ceph-mon[36715]: pgmap v2489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:09 smithi183 ceph-mon[37273]: pgmap v2489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3614.607274 secs 2024-07-26T12:06:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:11 smithi144 ceph-mon[32129]: pgmap v2490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3614.607274 secs 2024-07-26T12:06:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:11 smithi160 ceph-mon[36715]: pgmap v2490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3614.607274 secs 2024-07-26T12:06:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:11 smithi183 ceph-mon[37273]: pgmap v2490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:13 smithi144 ceph-mon[32129]: pgmap v2491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:13 smithi160 ceph-mon[36715]: pgmap v2491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:13 smithi183 ceph-mon[37273]: pgmap v2491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:14.935+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:14.936+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:14.936+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:14.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:14.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:16.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:15 smithi144 ceph-mon[32129]: pgmap v2492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:15 smithi160 ceph-mon[36715]: pgmap v2492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:15 smithi183 ceph-mon[37273]: pgmap v2492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3619.607395 secs 2024-07-26T12:06:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3619.607395 secs 2024-07-26T12:06:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3619.607395 secs 2024-07-26T12:06:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:17 smithi144 ceph-mon[32129]: pgmap v2493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:17 smithi160 ceph-mon[36715]: pgmap v2493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:17 smithi183 ceph-mon[37273]: pgmap v2493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:19 smithi144 ceph-mon[32129]: pgmap v2494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:19 smithi160 ceph-mon[36715]: pgmap v2494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:19 smithi183 ceph-mon[37273]: pgmap v2494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3624.607515 secs 2024-07-26T12:06:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:21 smithi144 ceph-mon[32129]: pgmap v2495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3624.607515 secs 2024-07-26T12:06:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:21 smithi160 ceph-mon[36715]: pgmap v2495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3624.607515 secs 2024-07-26T12:06:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:21 smithi183 ceph-mon[37273]: pgmap v2495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:23 smithi144 ceph-mon[32129]: pgmap v2496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:23 smithi160 ceph-mon[36715]: pgmap v2496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:23 smithi183 ceph-mon[37273]: pgmap v2496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:25 smithi144 ceph-mon[32129]: pgmap v2497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:25 smithi160 ceph-mon[36715]: pgmap v2497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:25 smithi183 ceph-mon[37273]: pgmap v2497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3629.607609 secs 2024-07-26T12:06:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3629.607609 secs 2024-07-26T12:06:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3629.607609 secs 2024-07-26T12:06:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:27 smithi160 ceph-mon[36715]: pgmap v2498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:27 smithi183 ceph-mon[37273]: pgmap v2498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:27 smithi144 ceph-mon[32129]: pgmap v2498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:29.940+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:29.941+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:29.941+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:29.941+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:29.941+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:29 smithi160 ceph-mon[36715]: pgmap v2499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:29 smithi183 ceph-mon[37273]: pgmap v2499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:29 smithi144 ceph-mon[32129]: pgmap v2499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3634.607729 secs 2024-07-26T12:06:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: pgmap v2500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: pgmap v2501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:06:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: pgmap v2502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:06:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3634.607729 secs 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: pgmap v2500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: pgmap v2501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: pgmap v2502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:06:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3634.607729 secs 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: pgmap v2500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: pgmap v2501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: pgmap v2502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:06:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:06:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:34 smithi160 ceph-mon[36715]: pgmap v2503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:06:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:34 smithi183 ceph-mon[37273]: pgmap v2503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:06:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:34 smithi144 ceph-mon[32129]: pgmap v2503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:06:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:36 smithi160 ceph-mon[36715]: pgmap v2504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:06:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:36 smithi183 ceph-mon[37273]: pgmap v2504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:06:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:36 smithi144 ceph-mon[32129]: pgmap v2504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:06:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3639.607846 secs 2024-07-26T12:06:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3639.607846 secs 2024-07-26T12:06:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3639.607846 secs 2024-07-26T12:06:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:38 smithi160 ceph-mon[36715]: pgmap v2505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:06:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:38 smithi183 ceph-mon[37273]: pgmap v2505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:06:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:38 smithi144 ceph-mon[32129]: pgmap v2505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:06:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:40 smithi160 ceph-mon[36715]: pgmap v2506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:06:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:40 smithi183 ceph-mon[37273]: pgmap v2506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:06:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:40 smithi144 ceph-mon[32129]: pgmap v2506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:06:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3644.607962 secs 2024-07-26T12:06:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3644.607962 secs 2024-07-26T12:06:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3644.607962 secs 2024-07-26T12:06:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:42 smithi144 ceph-mon[32129]: pgmap v2507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:06:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:42 smithi160 ceph-mon[36715]: pgmap v2507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:06:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:42 smithi183 ceph-mon[37273]: pgmap v2507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:06:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:44 smithi160 ceph-mon[36715]: pgmap v2508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:44 smithi183 ceph-mon[37273]: pgmap v2508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:44 smithi144 ceph-mon[32129]: pgmap v2508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:44.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:44.959+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:44.959+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:44.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:44.960+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:46 smithi160 ceph-mon[36715]: pgmap v2509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:46 smithi183 ceph-mon[37273]: pgmap v2509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:46 smithi144 ceph-mon[32129]: pgmap v2509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3649.608078 secs 2024-07-26T12:06:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3649.608078 secs 2024-07-26T12:06:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3649.608078 secs 2024-07-26T12:06:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:48 smithi160 ceph-mon[36715]: pgmap v2510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:48 smithi183 ceph-mon[37273]: pgmap v2510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:48 smithi144 ceph-mon[32129]: pgmap v2510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:50 smithi160 ceph-mon[36715]: pgmap v2511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:50 smithi183 ceph-mon[37273]: pgmap v2511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:50 smithi144 ceph-mon[32129]: pgmap v2511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3654.608218 secs 2024-07-26T12:06:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3654.608218 secs 2024-07-26T12:06:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3654.608218 secs 2024-07-26T12:06:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:52 smithi144 ceph-mon[32129]: pgmap v2512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:52.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:52 smithi183 ceph-mon[37273]: pgmap v2512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:52 smithi160 ceph-mon[36715]: pgmap v2512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:06:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:54 smithi160 ceph-mon[36715]: pgmap v2513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:54 smithi183 ceph-mon[37273]: pgmap v2513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:54 smithi144 ceph-mon[32129]: pgmap v2513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:56 smithi160 ceph-mon[36715]: pgmap v2514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:56 smithi183 ceph-mon[37273]: pgmap v2514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:56 smithi144 ceph-mon[32129]: pgmap v2514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:06:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3659.608353 secs 2024-07-26T12:06:57.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3659.608353 secs 2024-07-26T12:06:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3659.608353 secs 2024-07-26T12:06:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:06:59 smithi183 ceph-mon[37273]: pgmap v2515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:06:59 smithi144 ceph-mon[32129]: pgmap v2515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:06:59 smithi160 ceph-mon[36715]: pgmap v2515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:59.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:59.965+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:59.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:59.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:59.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:06:59.967+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:01 smithi183 ceph-mon[37273]: pgmap v2516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:01 smithi144 ceph-mon[32129]: pgmap v2516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:01 smithi160 ceph-mon[36715]: pgmap v2516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3664.608434 secs 2024-07-26T12:07:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3664.608434 secs 2024-07-26T12:07:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3664.608434 secs 2024-07-26T12:07:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:03 smithi183 ceph-mon[37273]: pgmap v2517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:03 smithi144 ceph-mon[32129]: pgmap v2517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:03 smithi160 ceph-mon[36715]: pgmap v2517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:05 smithi183 ceph-mon[37273]: pgmap v2518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:05 smithi144 ceph-mon[32129]: pgmap v2518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:05 smithi160 ceph-mon[36715]: pgmap v2518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3669.608554 secs 2024-07-26T12:07:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:07 smithi183 ceph-mon[37273]: pgmap v2519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3669.608554 secs 2024-07-26T12:07:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:07 smithi144 ceph-mon[32129]: pgmap v2519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3669.608554 secs 2024-07-26T12:07:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:07 smithi160 ceph-mon[36715]: pgmap v2519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:09 smithi183 ceph-mon[37273]: pgmap v2520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:09 smithi144 ceph-mon[32129]: pgmap v2520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:09 smithi160 ceph-mon[36715]: pgmap v2520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:11 smithi183 ceph-mon[37273]: pgmap v2521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:11 smithi144 ceph-mon[32129]: pgmap v2521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:11 smithi160 ceph-mon[36715]: pgmap v2521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3674.608614 secs 2024-07-26T12:07:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3674.608614 secs 2024-07-26T12:07:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3674.608614 secs 2024-07-26T12:07:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:13 smithi183 ceph-mon[37273]: pgmap v2522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:13 smithi144 ceph-mon[32129]: pgmap v2522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:13 smithi160 ceph-mon[36715]: pgmap v2522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:14.982+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:14.983+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:14.983+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:14.984+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:14.984+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:15 smithi183 ceph-mon[37273]: pgmap v2523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:15 smithi144 ceph-mon[32129]: pgmap v2523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:15 smithi160 ceph-mon[36715]: pgmap v2523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3679.608731 secs 2024-07-26T12:07:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:17 smithi183 ceph-mon[37273]: pgmap v2524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3679.608731 secs 2024-07-26T12:07:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:17 smithi144 ceph-mon[32129]: pgmap v2524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3679.608731 secs 2024-07-26T12:07:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:17 smithi160 ceph-mon[36715]: pgmap v2524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:19 smithi183 ceph-mon[37273]: pgmap v2525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:19 smithi144 ceph-mon[32129]: pgmap v2525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:19 smithi160 ceph-mon[36715]: pgmap v2525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:21 smithi144 ceph-mon[32129]: pgmap v2526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:21 smithi160 ceph-mon[36715]: pgmap v2526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:21 smithi183 ceph-mon[37273]: pgmap v2526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3684.608833 secs 2024-07-26T12:07:22.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3684.608833 secs 2024-07-26T12:07:22.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3684.608833 secs 2024-07-26T12:07:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:23 smithi144 ceph-mon[32129]: pgmap v2527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:23 smithi160 ceph-mon[36715]: pgmap v2527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:23 smithi183 ceph-mon[37273]: pgmap v2527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:25 smithi144 ceph-mon[32129]: pgmap v2528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:25 smithi160 ceph-mon[36715]: pgmap v2528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:25 smithi183 ceph-mon[37273]: pgmap v2528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3689.608962 secs 2024-07-26T12:07:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3689.608962 secs 2024-07-26T12:07:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3689.608962 secs 2024-07-26T12:07:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:27 smithi144 ceph-mon[32129]: pgmap v2529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:27 smithi160 ceph-mon[36715]: pgmap v2529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:27 smithi183 ceph-mon[37273]: pgmap v2529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:29 smithi144 ceph-mon[32129]: pgmap v2530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:29 smithi160 ceph-mon[36715]: pgmap v2530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:29 smithi183 ceph-mon[37273]: pgmap v2530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:29.990+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:29.991+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:29.991+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:29.991+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:29.991+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:31 smithi144 ceph-mon[32129]: pgmap v2531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3694.609083 secs 2024-07-26T12:07:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:31 smithi160 ceph-mon[36715]: pgmap v2531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3694.609083 secs 2024-07-26T12:07:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:31 smithi183 ceph-mon[37273]: pgmap v2531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3694.609083 secs 2024-07-26T12:07:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: pgmap v2532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: pgmap v2532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: pgmap v2532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:34 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:07:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:34 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:07:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:34 smithi160 ceph-mon[36715]: pgmap v2533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:34 smithi160 ceph-mon[36715]: pgmap v2534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:07:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:34 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:07:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:34 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:07:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:34 smithi183 ceph-mon[37273]: pgmap v2533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:34 smithi183 ceph-mon[37273]: pgmap v2534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:07:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:34 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:07:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:34 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:07:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:34 smithi144 ceph-mon[32129]: pgmap v2533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:34 smithi144 ceph-mon[32129]: pgmap v2534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:07:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:36 smithi160 ceph-mon[36715]: pgmap v2535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3699.609202 secs 2024-07-26T12:07:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:36 smithi183 ceph-mon[37273]: pgmap v2535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3699.609202 secs 2024-07-26T12:07:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:36 smithi144 ceph-mon[32129]: pgmap v2535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3699.609202 secs 2024-07-26T12:07:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:38 smithi160 ceph-mon[36715]: pgmap v2536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:38 smithi183 ceph-mon[37273]: pgmap v2536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:38 smithi144 ceph-mon[32129]: pgmap v2536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:40 smithi160 ceph-mon[36715]: pgmap v2537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:40 smithi183 ceph-mon[37273]: pgmap v2537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:40 smithi144 ceph-mon[32129]: pgmap v2537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3704.609322 secs 2024-07-26T12:07:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3704.609322 secs 2024-07-26T12:07:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3704.609322 secs 2024-07-26T12:07:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:42 smithi160 ceph-mon[36715]: pgmap v2538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:42 smithi183 ceph-mon[37273]: pgmap v2538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:42 smithi144 ceph-mon[32129]: pgmap v2538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:07:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:44 smithi160 ceph-mon[36715]: pgmap v2539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:07:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:44 smithi183 ceph-mon[37273]: pgmap v2539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:07:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:44 smithi144 ceph-mon[32129]: pgmap v2539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:07:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:45.008+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:45.009+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:45.009+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:45.009+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:07:45.009+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:46 smithi160 ceph-mon[36715]: pgmap v2540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3709.609463 secs 2024-07-26T12:07:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:46 smithi183 ceph-mon[37273]: pgmap v2540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3709.609463 secs 2024-07-26T12:07:46.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:46 smithi144 ceph-mon[32129]: pgmap v2540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:46.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3709.609463 secs 2024-07-26T12:07:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:48 smithi160 ceph-mon[36715]: pgmap v2541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:48 smithi183 ceph-mon[37273]: pgmap v2541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:48 smithi144 ceph-mon[32129]: pgmap v2541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:50 smithi160 ceph-mon[36715]: pgmap v2542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:50 smithi183 ceph-mon[37273]: pgmap v2542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:50 smithi144 ceph-mon[32129]: pgmap v2542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3714.609581 secs 2024-07-26T12:07:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3714.609581 secs 2024-07-26T12:07:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3714.609581 secs 2024-07-26T12:07:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:52 smithi183 ceph-mon[37273]: pgmap v2543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:52 smithi144 ceph-mon[32129]: pgmap v2543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:52 smithi160 ceph-mon[36715]: pgmap v2543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:07:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:54 smithi183 ceph-mon[37273]: pgmap v2544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:54 smithi144 ceph-mon[32129]: pgmap v2544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:54 smithi160 ceph-mon[36715]: pgmap v2544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:56 smithi183 ceph-mon[37273]: pgmap v2545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3719.609715 secs 2024-07-26T12:07:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:56 smithi144 ceph-mon[32129]: pgmap v2545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3719.609715 secs 2024-07-26T12:07:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:56 smithi160 ceph-mon[36715]: pgmap v2545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:07:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3719.609715 secs 2024-07-26T12:07:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:07:58 smithi183 ceph-mon[37273]: pgmap v2546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:07:58 smithi144 ceph-mon[32129]: pgmap v2546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:07:58 smithi160 ceph-mon[36715]: pgmap v2546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:00.013+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:00.014+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:00.014+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:00.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:00.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:00 smithi183 ceph-mon[37273]: pgmap v2547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:00 smithi144 ceph-mon[32129]: pgmap v2547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:00 smithi160 ceph-mon[36715]: pgmap v2547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3724.609835 secs 2024-07-26T12:08:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3724.609835 secs 2024-07-26T12:08:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3724.609835 secs 2024-07-26T12:08:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:02 smithi183 ceph-mon[37273]: pgmap v2548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:02 smithi144 ceph-mon[32129]: pgmap v2548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:02 smithi160 ceph-mon[36715]: pgmap v2548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:04 smithi183 ceph-mon[37273]: pgmap v2549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:04 smithi144 ceph-mon[32129]: pgmap v2549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:04 smithi160 ceph-mon[36715]: pgmap v2549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:06 smithi183 ceph-mon[37273]: pgmap v2550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3729.609955 secs 2024-07-26T12:08:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:06 smithi144 ceph-mon[32129]: pgmap v2550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3729.609955 secs 2024-07-26T12:08:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:06 smithi160 ceph-mon[36715]: pgmap v2550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3729.609955 secs 2024-07-26T12:08:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:08 smithi183 ceph-mon[37273]: pgmap v2551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:08 smithi144 ceph-mon[32129]: pgmap v2551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:08 smithi160 ceph-mon[36715]: pgmap v2551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:10 smithi183 ceph-mon[37273]: pgmap v2552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:10 smithi144 ceph-mon[32129]: pgmap v2552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:10 smithi160 ceph-mon[36715]: pgmap v2552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3734.610076 secs 2024-07-26T12:08:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3734.610076 secs 2024-07-26T12:08:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3734.610076 secs 2024-07-26T12:08:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:12 smithi144 ceph-mon[32129]: pgmap v2553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:12 smithi160 ceph-mon[36715]: pgmap v2553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:12 smithi183 ceph-mon[37273]: pgmap v2553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:14 smithi144 ceph-mon[32129]: pgmap v2554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:14 smithi160 ceph-mon[36715]: pgmap v2554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:14 smithi183 ceph-mon[37273]: pgmap v2554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:15.029+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:15.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:15.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:15.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:15.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:16 smithi144 ceph-mon[32129]: pgmap v2555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3739.610151 secs 2024-07-26T12:08:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:16 smithi160 ceph-mon[36715]: pgmap v2555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3739.610151 secs 2024-07-26T12:08:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:16 smithi183 ceph-mon[37273]: pgmap v2555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3739.610151 secs 2024-07-26T12:08:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:18 smithi144 ceph-mon[32129]: pgmap v2556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:18 smithi160 ceph-mon[36715]: pgmap v2556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:18 smithi183 ceph-mon[37273]: pgmap v2556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:20 smithi144 ceph-mon[32129]: pgmap v2557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:20 smithi160 ceph-mon[36715]: pgmap v2557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:20 smithi183 ceph-mon[37273]: pgmap v2557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3744.610306 secs 2024-07-26T12:08:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3744.610306 secs 2024-07-26T12:08:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3744.610306 secs 2024-07-26T12:08:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:22 smithi144 ceph-mon[32129]: pgmap v2558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:22.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:22 smithi183 ceph-mon[37273]: pgmap v2558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:22 smithi160 ceph-mon[36715]: pgmap v2558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:24 smithi144 ceph-mon[32129]: pgmap v2559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:24 smithi160 ceph-mon[36715]: pgmap v2559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:24 smithi183 ceph-mon[37273]: pgmap v2559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:26 smithi144 ceph-mon[32129]: pgmap v2560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3749.610429 secs 2024-07-26T12:08:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:26 smithi160 ceph-mon[36715]: pgmap v2560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3749.610429 secs 2024-07-26T12:08:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:26 smithi183 ceph-mon[37273]: pgmap v2560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3749.610429 secs 2024-07-26T12:08:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:28 smithi144 ceph-mon[32129]: pgmap v2561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:28 smithi160 ceph-mon[36715]: pgmap v2561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:28 smithi183 ceph-mon[37273]: pgmap v2561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:30.045+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:30.046+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:30.046+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:30.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:30.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:30 smithi144 ceph-mon[32129]: pgmap v2562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:30 smithi160 ceph-mon[36715]: pgmap v2562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:30 smithi183 ceph-mon[37273]: pgmap v2562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:31.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3754.610571 secs 2024-07-26T12:08:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3754.610571 secs 2024-07-26T12:08:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3754.610571 secs 2024-07-26T12:08:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:32 smithi144 ceph-mon[32129]: pgmap v2563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:32 smithi160 ceph-mon[36715]: pgmap v2563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:32 smithi183 ceph-mon[37273]: pgmap v2563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: pgmap v2564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:08:34.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:08:34.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:08:34.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:08:34.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:34.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:34.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:08:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: pgmap v2564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:34.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: pgmap v2564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:34.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:08:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:35 smithi160 ceph-mon[36715]: pgmap v2565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:08:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:35 smithi160 ceph-mon[36715]: pgmap v2566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:08:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:35 smithi183 ceph-mon[37273]: pgmap v2565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:08:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:35 smithi183 ceph-mon[37273]: pgmap v2566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:08:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:35 smithi144 ceph-mon[32129]: pgmap v2565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:08:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:35 smithi144 ceph-mon[32129]: pgmap v2566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:08:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3759.610694 secs 2024-07-26T12:08:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3759.610694 secs 2024-07-26T12:08:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3759.610694 secs 2024-07-26T12:08:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:37 smithi160 ceph-mon[36715]: pgmap v2567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:08:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:37 smithi183 ceph-mon[37273]: pgmap v2567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:08:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:37 smithi144 ceph-mon[32129]: pgmap v2567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:08:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:39 smithi160 ceph-mon[36715]: pgmap v2568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:08:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:39 smithi183 ceph-mon[37273]: pgmap v2568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:08:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:39 smithi144 ceph-mon[32129]: pgmap v2568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:08:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:41 smithi160 ceph-mon[36715]: pgmap v2569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:08:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3764.610793 secs 2024-07-26T12:08:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:41 smithi183 ceph-mon[37273]: pgmap v2569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:08:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3764.610793 secs 2024-07-26T12:08:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:41 smithi144 ceph-mon[32129]: pgmap v2569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:08:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3764.610793 secs 2024-07-26T12:08:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:43 smithi160 ceph-mon[36715]: pgmap v2570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:08:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:43 smithi183 ceph-mon[37273]: pgmap v2570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:08:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:43 smithi144 ceph-mon[32129]: pgmap v2570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:08:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:45.053+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:45.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:45.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:45.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:45.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:45 smithi160 ceph-mon[36715]: pgmap v2571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:08:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:45 smithi183 ceph-mon[37273]: pgmap v2571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:08:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:45 smithi144 ceph-mon[32129]: pgmap v2571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:08:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:46.760+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:08:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:46.760+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:08:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:46.761+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:08:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:08:46.761+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:08:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3769.610915 secs 2024-07-26T12:08:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3769.610915 secs 2024-07-26T12:08:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3769.610915 secs 2024-07-26T12:08:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:47 smithi160 ceph-mon[36715]: pgmap v2572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:47 smithi183 ceph-mon[37273]: pgmap v2572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:47 smithi144 ceph-mon[32129]: pgmap v2572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:49 smithi160 ceph-mon[36715]: pgmap v2573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:49 smithi183 ceph-mon[37273]: pgmap v2573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:49 smithi144 ceph-mon[32129]: pgmap v2573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:51 smithi160 ceph-mon[36715]: pgmap v2574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3774.611037 secs 2024-07-26T12:08:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:51 smithi183 ceph-mon[37273]: pgmap v2574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3774.611037 secs 2024-07-26T12:08:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:51 smithi144 ceph-mon[32129]: pgmap v2574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3774.611037 secs 2024-07-26T12:08:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:53 smithi160 ceph-mon[36715]: pgmap v2575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:53 smithi183 ceph-mon[37273]: pgmap v2575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:53 smithi144 ceph-mon[32129]: pgmap v2575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:55 smithi160 ceph-mon[36715]: pgmap v2576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:55 smithi183 ceph-mon[37273]: pgmap v2576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:55 smithi144 ceph-mon[32129]: pgmap v2576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:56.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3779.611158 secs 2024-07-26T12:08:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3779.611158 secs 2024-07-26T12:08:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3779.611158 secs 2024-07-26T12:08:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:57 smithi183 ceph-mon[37273]: pgmap v2577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:57 smithi144 ceph-mon[32129]: pgmap v2577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:57 smithi160 ceph-mon[36715]: pgmap v2577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:08:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:08:59 smithi183 ceph-mon[37273]: pgmap v2578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:08:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:08:59 smithi144 ceph-mon[32129]: pgmap v2578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:08:59 smithi160 ceph-mon[36715]: pgmap v2578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:00.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:00.071+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:00.071+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:00.071+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:00.071+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:01 smithi183 ceph-mon[37273]: pgmap v2579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3784.611276 secs 2024-07-26T12:09:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:01 smithi144 ceph-mon[32129]: pgmap v2579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3784.611276 secs 2024-07-26T12:09:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:01 smithi160 ceph-mon[36715]: pgmap v2579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3784.611276 secs 2024-07-26T12:09:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:03 smithi183 ceph-mon[37273]: pgmap v2580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:03 smithi144 ceph-mon[32129]: pgmap v2580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:03 smithi160 ceph-mon[36715]: pgmap v2580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:05 smithi183 ceph-mon[37273]: pgmap v2581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:05 smithi144 ceph-mon[32129]: pgmap v2581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:05 smithi160 ceph-mon[36715]: pgmap v2581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3789.611389 secs 2024-07-26T12:09:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3789.611389 secs 2024-07-26T12:09:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3789.611389 secs 2024-07-26T12:09:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:07 smithi183 ceph-mon[37273]: pgmap v2582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:07 smithi144 ceph-mon[32129]: pgmap v2582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:07 smithi160 ceph-mon[36715]: pgmap v2582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:09 smithi183 ceph-mon[37273]: pgmap v2583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:09 smithi144 ceph-mon[32129]: pgmap v2583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:09 smithi160 ceph-mon[36715]: pgmap v2583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:11 smithi183 ceph-mon[37273]: pgmap v2584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3794.611480 secs 2024-07-26T12:09:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:11 smithi144 ceph-mon[32129]: pgmap v2584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3794.611480 secs 2024-07-26T12:09:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:11 smithi160 ceph-mon[36715]: pgmap v2584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3794.611480 secs 2024-07-26T12:09:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:13 smithi144 ceph-mon[32129]: pgmap v2585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:13 smithi160 ceph-mon[36715]: pgmap v2585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:13 smithi183 ceph-mon[37273]: pgmap v2585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:15.079+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:15.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:15.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:15.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:15.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:15.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:15.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:15.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:15 smithi144 ceph-mon[32129]: pgmap v2586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:15 smithi160 ceph-mon[36715]: pgmap v2586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:15 smithi183 ceph-mon[37273]: pgmap v2586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:16.958 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3799.611592 secs 2024-07-26T12:09:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3799.611592 secs 2024-07-26T12:09:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3799.611592 secs 2024-07-26T12:09:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:17 smithi144 ceph-mon[32129]: pgmap v2587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:17 smithi160 ceph-mon[36715]: pgmap v2587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:17 smithi183 ceph-mon[37273]: pgmap v2587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:19 smithi144 ceph-mon[32129]: pgmap v2588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:19 smithi160 ceph-mon[36715]: pgmap v2588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:19 smithi183 ceph-mon[37273]: pgmap v2588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:21 smithi144 ceph-mon[32129]: pgmap v2589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3804.611734 secs 2024-07-26T12:09:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:21 smithi160 ceph-mon[36715]: pgmap v2589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3804.611734 secs 2024-07-26T12:09:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:21 smithi183 ceph-mon[37273]: pgmap v2589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3804.611734 secs 2024-07-26T12:09:23.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:23 smithi183 ceph-mon[37273]: pgmap v2590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:23 smithi144 ceph-mon[32129]: pgmap v2590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:23 smithi160 ceph-mon[36715]: pgmap v2590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:25 smithi144 ceph-mon[32129]: pgmap v2591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:25 smithi160 ceph-mon[36715]: pgmap v2591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:25 smithi183 ceph-mon[37273]: pgmap v2591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3809.611866 secs 2024-07-26T12:09:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3809.611866 secs 2024-07-26T12:09:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3809.611866 secs 2024-07-26T12:09:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:27 smithi144 ceph-mon[32129]: pgmap v2592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:27 smithi160 ceph-mon[36715]: pgmap v2592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:27 smithi183 ceph-mon[37273]: pgmap v2592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:09:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:29 smithi144 ceph-mon[32129]: pgmap v2593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:29 smithi160 ceph-mon[36715]: pgmap v2593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:30.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:30.096+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:30.096+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:30.096+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:30.096+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:29 smithi183 ceph-mon[37273]: pgmap v2593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:31 smithi144 ceph-mon[32129]: pgmap v2594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3814.611960 secs 2024-07-26T12:09:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:31 smithi160 ceph-mon[36715]: pgmap v2594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3814.611960 secs 2024-07-26T12:09:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:31 smithi183 ceph-mon[37273]: pgmap v2594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3814.611960 secs 2024-07-26T12:09:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:33 smithi144 ceph-mon[32129]: pgmap v2595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:33 smithi160 ceph-mon[36715]: pgmap v2595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:33 smithi183 ceph-mon[37273]: pgmap v2595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:35 smithi160 ceph-mon[36715]: pgmap v2596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:09:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:35 smithi183 ceph-mon[37273]: pgmap v2596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:09:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:35 smithi144 ceph-mon[32129]: pgmap v2596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:09:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:09:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:36 smithi160 ceph-mon[36715]: pgmap v2597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:36 smithi160 ceph-mon[36715]: pgmap v2598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3819.612034 secs 2024-07-26T12:09:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:36 smithi183 ceph-mon[37273]: pgmap v2597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:36 smithi183 ceph-mon[37273]: pgmap v2598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3819.612034 secs 2024-07-26T12:09:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:36 smithi144 ceph-mon[32129]: pgmap v2597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:36 smithi144 ceph-mon[32129]: pgmap v2598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3819.612034 secs 2024-07-26T12:09:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:38 smithi160 ceph-mon[36715]: pgmap v2599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:38 smithi183 ceph-mon[37273]: pgmap v2599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:38 smithi144 ceph-mon[32129]: pgmap v2599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:40 smithi160 ceph-mon[36715]: pgmap v2600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:40 smithi183 ceph-mon[37273]: pgmap v2600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:40 smithi144 ceph-mon[32129]: pgmap v2600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3824.612155 secs 2024-07-26T12:09:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3824.612155 secs 2024-07-26T12:09:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3824.612155 secs 2024-07-26T12:09:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:42 smithi160 ceph-mon[36715]: pgmap v2601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:42 smithi183 ceph-mon[37273]: pgmap v2601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:42 smithi144 ceph-mon[32129]: pgmap v2601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:09:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:44 smithi160 ceph-mon[36715]: pgmap v2602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:09:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:45.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:45.114+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:45.114+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:45.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:09:45.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:45.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:44 smithi183 ceph-mon[37273]: pgmap v2602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:09:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:44 smithi144 ceph-mon[32129]: pgmap v2602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:09:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:46 smithi160 ceph-mon[36715]: pgmap v2603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3829.612274 secs 2024-07-26T12:09:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:46 smithi183 ceph-mon[37273]: pgmap v2603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3829.612274 secs 2024-07-26T12:09:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:46 smithi144 ceph-mon[32129]: pgmap v2603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3829.612274 secs 2024-07-26T12:09:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:48 smithi160 ceph-mon[36715]: pgmap v2604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:48 smithi183 ceph-mon[37273]: pgmap v2604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:49.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:48 smithi144 ceph-mon[32129]: pgmap v2604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:50 smithi160 ceph-mon[36715]: pgmap v2605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:09:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:50 smithi183 ceph-mon[37273]: pgmap v2605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:09:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:50 smithi144 ceph-mon[32129]: pgmap v2605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:09:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3834.612400 secs 2024-07-26T12:09:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3834.612400 secs 2024-07-26T12:09:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3834.612400 secs 2024-07-26T12:09:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:52 smithi160 ceph-mon[36715]: pgmap v2606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:52 smithi183 ceph-mon[37273]: pgmap v2606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:52 smithi144 ceph-mon[32129]: pgmap v2606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:54 smithi160 ceph-mon[36715]: pgmap v2607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:54 smithi183 ceph-mon[37273]: pgmap v2607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:54 smithi144 ceph-mon[32129]: pgmap v2607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:56 smithi183 ceph-mon[37273]: pgmap v2608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3839.612521 secs 2024-07-26T12:09:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:56 smithi144 ceph-mon[32129]: pgmap v2608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3839.612521 secs 2024-07-26T12:09:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:56 smithi160 ceph-mon[36715]: pgmap v2608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3839.612521 secs 2024-07-26T12:09:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:09:58 smithi183 ceph-mon[37273]: pgmap v2609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:09:58 smithi144 ceph-mon[32129]: pgmap v2609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:09:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:09:58 smithi160 ceph-mon[36715]: pgmap v2609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:10:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:00.129+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:00.130+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:00.130+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:00.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:00.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:00 smithi183 ceph-mon[37273]: pgmap v2610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:00 smithi144 ceph-mon[32129]: pgmap v2610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:00 smithi160 ceph-mon[36715]: pgmap v2610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3844.612647 secs 2024-07-26T12:10:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3844.612647 secs 2024-07-26T12:10:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3844.612647 secs 2024-07-26T12:10:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:02 smithi183 ceph-mon[37273]: pgmap v2611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:02 smithi144 ceph-mon[32129]: pgmap v2611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:02 smithi160 ceph-mon[36715]: pgmap v2611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:04 smithi183 ceph-mon[37273]: pgmap v2612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:04 smithi144 ceph-mon[32129]: pgmap v2612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:04 smithi160 ceph-mon[36715]: pgmap v2612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:06 smithi183 ceph-mon[37273]: pgmap v2613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3849.612770 secs 2024-07-26T12:10:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:06 smithi144 ceph-mon[32129]: pgmap v2613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3849.612770 secs 2024-07-26T12:10:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:06 smithi160 ceph-mon[36715]: pgmap v2613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3849.612770 secs 2024-07-26T12:10:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:08 smithi183 ceph-mon[37273]: pgmap v2614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:08 smithi144 ceph-mon[32129]: pgmap v2614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:08 smithi160 ceph-mon[36715]: pgmap v2614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:10 smithi183 ceph-mon[37273]: pgmap v2615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:10 smithi144 ceph-mon[32129]: pgmap v2615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:10 smithi160 ceph-mon[36715]: pgmap v2615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3854.612905 secs 2024-07-26T12:10:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3854.612905 secs 2024-07-26T12:10:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3854.612905 secs 2024-07-26T12:10:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:12 smithi183 ceph-mon[37273]: pgmap v2616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:12 smithi144 ceph-mon[32129]: pgmap v2616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:12 smithi160 ceph-mon[36715]: pgmap v2616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:14 smithi183 ceph-mon[37273]: pgmap v2617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:15.146+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:15.147+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:15.147+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:15.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:15.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:14 smithi144 ceph-mon[32129]: pgmap v2617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:14 smithi160 ceph-mon[36715]: pgmap v2617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:16 smithi183 ceph-mon[37273]: pgmap v2618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3859.613033 secs 2024-07-26T12:10:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:16 smithi144 ceph-mon[32129]: pgmap v2618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3859.613033 secs 2024-07-26T12:10:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:16 smithi160 ceph-mon[36715]: pgmap v2618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3859.613033 secs 2024-07-26T12:10:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:18 smithi144 ceph-mon[32129]: pgmap v2619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:18 smithi160 ceph-mon[36715]: pgmap v2619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:18 smithi183 ceph-mon[37273]: pgmap v2619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:20 smithi144 ceph-mon[32129]: pgmap v2620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:20 smithi160 ceph-mon[36715]: pgmap v2620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:20 smithi183 ceph-mon[37273]: pgmap v2620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3864.613154 secs 2024-07-26T12:10:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3864.613154 secs 2024-07-26T12:10:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3864.613154 secs 2024-07-26T12:10:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:22 smithi144 ceph-mon[32129]: pgmap v2621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:23.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:22 smithi183 ceph-mon[37273]: pgmap v2621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:22 smithi160 ceph-mon[36715]: pgmap v2621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:24 smithi144 ceph-mon[32129]: pgmap v2622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:24 smithi160 ceph-mon[36715]: pgmap v2622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:24 smithi183 ceph-mon[37273]: pgmap v2622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:26 smithi144 ceph-mon[32129]: pgmap v2623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3869.613306 secs 2024-07-26T12:10:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:26 smithi160 ceph-mon[36715]: pgmap v2623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3869.613306 secs 2024-07-26T12:10:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:26 smithi183 ceph-mon[37273]: pgmap v2623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3869.613306 secs 2024-07-26T12:10:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:28 smithi144 ceph-mon[32129]: pgmap v2624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:28 smithi160 ceph-mon[36715]: pgmap v2624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:28 smithi183 ceph-mon[37273]: pgmap v2624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:30.162+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:30.162+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:30.162+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:30.162+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:30.162+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:30 smithi144 ceph-mon[32129]: pgmap v2625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:30 smithi160 ceph-mon[36715]: pgmap v2625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:30 smithi183 ceph-mon[37273]: pgmap v2625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3874.613426 secs 2024-07-26T12:10:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3874.613426 secs 2024-07-26T12:10:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3874.613426 secs 2024-07-26T12:10:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:32 smithi144 ceph-mon[32129]: pgmap v2626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:32 smithi160 ceph-mon[36715]: pgmap v2626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:32 smithi183 ceph-mon[37273]: pgmap v2626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:34 smithi144 ceph-mon[32129]: pgmap v2627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:34 smithi160 ceph-mon[36715]: pgmap v2627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:34 smithi183 ceph-mon[37273]: pgmap v2627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:36 smithi144 ceph-mon[32129]: pgmap v2628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3879.613546 secs 2024-07-26T12:10:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:36 smithi160 ceph-mon[36715]: pgmap v2628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3879.613546 secs 2024-07-26T12:10:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:36 smithi183 ceph-mon[37273]: pgmap v2628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3879.613546 secs 2024-07-26T12:10:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:10:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:38 smithi160 ceph-mon[36715]: pgmap v2629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:38 smithi160 ceph-mon[36715]: pgmap v2630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:38 smithi183 ceph-mon[37273]: pgmap v2629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:38 smithi183 ceph-mon[37273]: pgmap v2630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:38 smithi144 ceph-mon[32129]: pgmap v2629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:39.516 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:38 smithi144 ceph-mon[32129]: pgmap v2630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:40 smithi160 ceph-mon[36715]: pgmap v2631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:40 smithi183 ceph-mon[37273]: pgmap v2631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:40 smithi144 ceph-mon[32129]: pgmap v2631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3884.613658 secs 2024-07-26T12:10:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3884.613658 secs 2024-07-26T12:10:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3884.613658 secs 2024-07-26T12:10:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:43 smithi160 ceph-mon[36715]: pgmap v2632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:43 smithi183 ceph-mon[37273]: pgmap v2632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:43 smithi144 ceph-mon[32129]: pgmap v2632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:45.179+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:45.180+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:45.180+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:45.180+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:10:45.180+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:45 smithi160 ceph-mon[36715]: pgmap v2633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:45 smithi183 ceph-mon[37273]: pgmap v2633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:45 smithi144 ceph-mon[32129]: pgmap v2633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:47 smithi160 ceph-mon[36715]: pgmap v2634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3889.613777 secs 2024-07-26T12:10:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:47 smithi183 ceph-mon[37273]: pgmap v2634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3889.613777 secs 2024-07-26T12:10:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:47 smithi144 ceph-mon[32129]: pgmap v2634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:10:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3889.613777 secs 2024-07-26T12:10:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:49 smithi160 ceph-mon[36715]: pgmap v2635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:10:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:49 smithi183 ceph-mon[37273]: pgmap v2635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:10:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:49 smithi144 ceph-mon[32129]: pgmap v2635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:10:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:51 smithi160 ceph-mon[36715]: pgmap v2636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:51 smithi183 ceph-mon[37273]: pgmap v2636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:51 smithi144 ceph-mon[32129]: pgmap v2636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3894.613901 secs 2024-07-26T12:10:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3894.613901 secs 2024-07-26T12:10:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3894.613901 secs 2024-07-26T12:10:53.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:53 smithi183 ceph-mon[37273]: pgmap v2637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:53 smithi160 ceph-mon[36715]: pgmap v2637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:53 smithi144 ceph-mon[32129]: pgmap v2637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:10:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:55 smithi160 ceph-mon[36715]: pgmap v2638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:55 smithi183 ceph-mon[37273]: pgmap v2638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:55 smithi144 ceph-mon[32129]: pgmap v2638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:57 smithi160 ceph-mon[36715]: pgmap v2639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3899.614023 secs 2024-07-26T12:10:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:57 smithi183 ceph-mon[37273]: pgmap v2639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3899.614023 secs 2024-07-26T12:10:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:57 smithi144 ceph-mon[32129]: pgmap v2639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:10:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3899.614023 secs 2024-07-26T12:10:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:10:58 smithi160 ceph-mon[36715]: pgmap v2640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:10:58 smithi183 ceph-mon[37273]: pgmap v2640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:10:58 smithi144 ceph-mon[32129]: pgmap v2640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:00.196+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:00.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:00.197+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:00.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:00.197+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:00.197+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:00.197+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:00 smithi144 ceph-mon[32129]: pgmap v2641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:00 smithi160 ceph-mon[36715]: pgmap v2641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:00 smithi183 ceph-mon[37273]: pgmap v2641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3904.614165 secs 2024-07-26T12:11:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3904.614165 secs 2024-07-26T12:11:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3904.614165 secs 2024-07-26T12:11:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:02 smithi160 ceph-mon[36715]: pgmap v2642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:02 smithi183 ceph-mon[37273]: pgmap v2642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:02 smithi144 ceph-mon[32129]: pgmap v2642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:04 smithi160 ceph-mon[36715]: pgmap v2643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:04 smithi183 ceph-mon[37273]: pgmap v2643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:04 smithi144 ceph-mon[32129]: pgmap v2643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:06 smithi160 ceph-mon[36715]: pgmap v2644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3909.614286 secs 2024-07-26T12:11:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:06 smithi183 ceph-mon[37273]: pgmap v2644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3909.614286 secs 2024-07-26T12:11:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:06 smithi144 ceph-mon[32129]: pgmap v2644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3909.614286 secs 2024-07-26T12:11:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:08 smithi160 ceph-mon[36715]: pgmap v2645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:08 smithi183 ceph-mon[37273]: pgmap v2645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:08 smithi144 ceph-mon[32129]: pgmap v2645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:10 smithi160 ceph-mon[36715]: pgmap v2646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:10 smithi183 ceph-mon[37273]: pgmap v2646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:10 smithi144 ceph-mon[32129]: pgmap v2646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3914.614406 secs 2024-07-26T12:11:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3914.614406 secs 2024-07-26T12:11:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3914.614406 secs 2024-07-26T12:11:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:12 smithi160 ceph-mon[36715]: pgmap v2647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:12 smithi183 ceph-mon[37273]: pgmap v2647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:12 smithi144 ceph-mon[32129]: pgmap v2647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:14 smithi160 ceph-mon[36715]: pgmap v2648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:14 smithi183 ceph-mon[37273]: pgmap v2648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:14 smithi144 ceph-mon[32129]: pgmap v2648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:15.198+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:15.199+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:15.199+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:15.199+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:15.199+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:16 smithi160 ceph-mon[36715]: pgmap v2649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3919.614531 secs 2024-07-26T12:11:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:16 smithi183 ceph-mon[37273]: pgmap v2649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3919.614531 secs 2024-07-26T12:11:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:16 smithi144 ceph-mon[32129]: pgmap v2649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3919.614531 secs 2024-07-26T12:11:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:18 smithi160 ceph-mon[36715]: pgmap v2650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:18 smithi183 ceph-mon[37273]: pgmap v2650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:18 smithi144 ceph-mon[32129]: pgmap v2650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:20 smithi160 ceph-mon[36715]: pgmap v2651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:20 smithi183 ceph-mon[37273]: pgmap v2651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:20 smithi144 ceph-mon[32129]: pgmap v2651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3924.614635 secs 2024-07-26T12:11:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3924.614635 secs 2024-07-26T12:11:21.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3924.614635 secs 2024-07-26T12:11:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:22 smithi160 ceph-mon[36715]: pgmap v2652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:22 smithi183 ceph-mon[37273]: pgmap v2652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:22 smithi144 ceph-mon[32129]: pgmap v2652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:24 smithi183 ceph-mon[37273]: pgmap v2653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:24 smithi144 ceph-mon[32129]: pgmap v2653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:24 smithi160 ceph-mon[36715]: pgmap v2653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:26 smithi183 ceph-mon[37273]: pgmap v2654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3929.614762 secs 2024-07-26T12:11:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:26 smithi144 ceph-mon[32129]: pgmap v2654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3929.614762 secs 2024-07-26T12:11:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:26 smithi160 ceph-mon[36715]: pgmap v2654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3929.614762 secs 2024-07-26T12:11:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:28 smithi183 ceph-mon[37273]: pgmap v2655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:28 smithi144 ceph-mon[32129]: pgmap v2655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:28 smithi160 ceph-mon[36715]: pgmap v2655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:30.214+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:30.215+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:30.215+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:30.215+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:30.215+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:30 smithi183 ceph-mon[37273]: pgmap v2656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:30 smithi144 ceph-mon[32129]: pgmap v2656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:30 smithi160 ceph-mon[36715]: pgmap v2656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3934.614880 secs 2024-07-26T12:11:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3934.614880 secs 2024-07-26T12:11:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3934.614880 secs 2024-07-26T12:11:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:32 smithi183 ceph-mon[37273]: pgmap v2657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:32 smithi144 ceph-mon[32129]: pgmap v2657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:32 smithi160 ceph-mon[36715]: pgmap v2657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:34 smithi183 ceph-mon[37273]: pgmap v2658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:34 smithi144 ceph-mon[32129]: pgmap v2658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:34 smithi160 ceph-mon[36715]: pgmap v2658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:36 smithi183 ceph-mon[37273]: pgmap v2659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3939.615031 secs 2024-07-26T12:11:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:36 smithi144 ceph-mon[32129]: pgmap v2659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3939.615031 secs 2024-07-26T12:11:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:36 smithi160 ceph-mon[36715]: pgmap v2659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:11:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3939.615031 secs 2024-07-26T12:11:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:38 smithi183 ceph-mon[37273]: pgmap v2660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:38 smithi144 ceph-mon[32129]: pgmap v2660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:38.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:38.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:38.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:38 smithi160 ceph-mon[36715]: pgmap v2660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:40 smithi144 ceph-mon[32129]: pgmap v2661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:40 smithi160 ceph-mon[36715]: pgmap v2661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:40 smithi183 ceph-mon[37273]: pgmap v2661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3944.615144 secs 2024-07-26T12:11:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3944.615144 secs 2024-07-26T12:11:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3944.615144 secs 2024-07-26T12:11:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:42 smithi144 ceph-mon[32129]: pgmap v2662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:42 smithi160 ceph-mon[36715]: pgmap v2662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:42 smithi183 ceph-mon[37273]: pgmap v2662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:11:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:43 smithi144 ceph-mon[32129]: pgmap v2663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:11:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:43 smithi144 ceph-mon[32129]: pgmap v2664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:43 smithi160 ceph-mon[36715]: pgmap v2663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:11:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:43 smithi160 ceph-mon[36715]: pgmap v2664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:43 smithi183 ceph-mon[37273]: pgmap v2663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:11:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:43 smithi183 ceph-mon[37273]: pgmap v2664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:45.228+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:45.229+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:45.229+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:45.229+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:45.229+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:45 smithi144 ceph-mon[32129]: pgmap v2665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:45 smithi160 ceph-mon[36715]: pgmap v2665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:45 smithi183 ceph-mon[37273]: pgmap v2665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:11:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.793+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:11:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.793+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:11:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.794+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:11:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.794+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f74a009d020 con 0x7f74b0109990 2024-07-26T12:11:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.794+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T12:11:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.794+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T12:11:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.794+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780062d0 con 0x7f74b0109990 2024-07-26T12:11:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:11:46.795+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T12:11:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3949.615279 secs 2024-07-26T12:11:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3949.615279 secs 2024-07-26T12:11:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3949.615279 secs 2024-07-26T12:11:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:47 smithi144 ceph-mon[32129]: pgmap v2666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:11:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:47 smithi160 ceph-mon[36715]: pgmap v2666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:11:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:47 smithi183 ceph-mon[37273]: pgmap v2666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:11:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:49 smithi144 ceph-mon[32129]: pgmap v2667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:11:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:49 smithi160 ceph-mon[36715]: pgmap v2667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:11:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:49 smithi183 ceph-mon[37273]: pgmap v2667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:11:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:51 smithi144 ceph-mon[32129]: pgmap v2668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:11:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3954.615416 secs 2024-07-26T12:11:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:51 smithi160 ceph-mon[36715]: pgmap v2668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:11:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3954.615416 secs 2024-07-26T12:11:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:51 smithi183 ceph-mon[37273]: pgmap v2668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:11:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3954.615416 secs 2024-07-26T12:11:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:53 smithi144 ceph-mon[32129]: pgmap v2669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:53 smithi160 ceph-mon[36715]: pgmap v2669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:53 smithi183 ceph-mon[37273]: pgmap v2669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:55 smithi144 ceph-mon[32129]: pgmap v2670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:55 smithi160 ceph-mon[36715]: pgmap v2670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:55 smithi183 ceph-mon[37273]: pgmap v2670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:11:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3959.615530 secs 2024-07-26T12:11:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3959.615530 secs 2024-07-26T12:11:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3959.615530 secs 2024-07-26T12:11:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:57 smithi160 ceph-mon[36715]: pgmap v2671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:57 smithi183 ceph-mon[37273]: pgmap v2671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:11:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:57 smithi144 ceph-mon[32129]: pgmap v2671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:00.023 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:11:59 smithi144 ceph-mon[32129]: pgmap v2672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:11:59 smithi160 ceph-mon[36715]: pgmap v2672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:11:59 smithi183 ceph-mon[37273]: pgmap v2672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:00.245+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:00.246+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:00.246+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:00.246+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:00.246+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:01 smithi183 ceph-mon[37273]: pgmap v2673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3964.615653 secs 2024-07-26T12:12:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:01 smithi144 ceph-mon[32129]: pgmap v2673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3964.615653 secs 2024-07-26T12:12:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:01 smithi160 ceph-mon[36715]: pgmap v2673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3964.615653 secs 2024-07-26T12:12:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:03 smithi160 ceph-mon[36715]: pgmap v2674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:03 smithi183 ceph-mon[37273]: pgmap v2674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:03 smithi144 ceph-mon[32129]: pgmap v2674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:05 smithi160 ceph-mon[36715]: pgmap v2675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:05 smithi183 ceph-mon[37273]: pgmap v2675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:05 smithi144 ceph-mon[32129]: pgmap v2675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3969.615798 secs 2024-07-26T12:12:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3969.615798 secs 2024-07-26T12:12:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3969.615798 secs 2024-07-26T12:12:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:07 smithi160 ceph-mon[36715]: pgmap v2676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:07 smithi183 ceph-mon[37273]: pgmap v2676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:07 smithi144 ceph-mon[32129]: pgmap v2676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:09 smithi160 ceph-mon[36715]: pgmap v2677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:09 smithi183 ceph-mon[37273]: pgmap v2677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:09 smithi144 ceph-mon[32129]: pgmap v2677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:11 smithi160 ceph-mon[36715]: pgmap v2678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3974.615915 secs 2024-07-26T12:12:12.085 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:11 smithi144 ceph-mon[32129]: pgmap v2678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:12.085 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3974.615915 secs 2024-07-26T12:12:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:11 smithi183 ceph-mon[37273]: pgmap v2678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3974.615915 secs 2024-07-26T12:12:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:13 smithi160 ceph-mon[36715]: pgmap v2679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:13 smithi183 ceph-mon[37273]: pgmap v2679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:13 smithi144 ceph-mon[32129]: pgmap v2679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:15.253+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:15.254+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:15.254+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:15.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:15.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:15 smithi160 ceph-mon[36715]: pgmap v2680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:15 smithi183 ceph-mon[37273]: pgmap v2680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:15 smithi144 ceph-mon[32129]: pgmap v2680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3979.616050 secs 2024-07-26T12:12:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3979.616050 secs 2024-07-26T12:12:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3979.616050 secs 2024-07-26T12:12:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:17 smithi160 ceph-mon[36715]: pgmap v2681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:17 smithi183 ceph-mon[37273]: pgmap v2681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:17 smithi144 ceph-mon[32129]: pgmap v2681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:19 smithi160 ceph-mon[36715]: pgmap v2682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:19 smithi183 ceph-mon[37273]: pgmap v2682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:19 smithi144 ceph-mon[32129]: pgmap v2682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:21 smithi183 ceph-mon[37273]: pgmap v2683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3984.616185 secs 2024-07-26T12:12:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:21 smithi144 ceph-mon[32129]: pgmap v2683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3984.616185 secs 2024-07-26T12:12:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:21 smithi160 ceph-mon[36715]: pgmap v2683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3984.616185 secs 2024-07-26T12:12:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:23 smithi183 ceph-mon[37273]: pgmap v2684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:23 smithi144 ceph-mon[32129]: pgmap v2684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:23 smithi160 ceph-mon[36715]: pgmap v2684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:25 smithi183 ceph-mon[37273]: pgmap v2685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:25 smithi144 ceph-mon[32129]: pgmap v2685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:25 smithi160 ceph-mon[36715]: pgmap v2685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3989.616251 secs 2024-07-26T12:12:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3989.616251 secs 2024-07-26T12:12:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3989.616251 secs 2024-07-26T12:12:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:27 smithi144 ceph-mon[32129]: pgmap v2686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:27 smithi160 ceph-mon[36715]: pgmap v2686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:27 smithi183 ceph-mon[37273]: pgmap v2686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:29 smithi144 ceph-mon[32129]: pgmap v2687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:29 smithi160 ceph-mon[36715]: pgmap v2687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:29 smithi183 ceph-mon[37273]: pgmap v2687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:30.267 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:30.265+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:30.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:30.265+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:30.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:30.265+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:30.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:30.265+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:30.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:30.265+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:31 smithi144 ceph-mon[32129]: pgmap v2688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3994.616369 secs 2024-07-26T12:12:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:31 smithi160 ceph-mon[36715]: pgmap v2688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3994.616369 secs 2024-07-26T12:12:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:31 smithi183 ceph-mon[37273]: pgmap v2688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3994.616369 secs 2024-07-26T12:12:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:33 smithi144 ceph-mon[32129]: pgmap v2689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:33 smithi160 ceph-mon[36715]: pgmap v2689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:33 smithi183 ceph-mon[37273]: pgmap v2689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:35 smithi144 ceph-mon[32129]: pgmap v2690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:35 smithi160 ceph-mon[36715]: pgmap v2690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:35 smithi183 ceph-mon[37273]: pgmap v2690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 3999.616462 secs 2024-07-26T12:12:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 3999.616462 secs 2024-07-26T12:12:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 3999.616462 secs 2024-07-26T12:12:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:37 smithi144 ceph-mon[32129]: pgmap v2691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:37 smithi160 ceph-mon[36715]: pgmap v2691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:37 smithi183 ceph-mon[37273]: pgmap v2691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:12:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:39 smithi144 ceph-mon[32129]: pgmap v2692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:39 smithi160 ceph-mon[36715]: pgmap v2692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:39 smithi183 ceph-mon[37273]: pgmap v2692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:41 smithi144 ceph-mon[32129]: pgmap v2693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4004.616582 secs 2024-07-26T12:12:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:41 smithi160 ceph-mon[36715]: pgmap v2693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4004.616582 secs 2024-07-26T12:12:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:41 smithi183 ceph-mon[37273]: pgmap v2693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4004.616582 secs 2024-07-26T12:12:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:43 smithi144 ceph-mon[32129]: pgmap v2694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:43 smithi160 ceph-mon[36715]: pgmap v2694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:43 smithi183 ceph-mon[37273]: pgmap v2694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:44 smithi144 ceph-mon[32129]: pgmap v2695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:44 smithi144 ceph-mon[32129]: pgmap v2696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:12:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:12:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:44 smithi160 ceph-mon[36715]: pgmap v2695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:44 smithi160 ceph-mon[36715]: pgmap v2696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:12:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:12:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:44 smithi183 ceph-mon[37273]: pgmap v2695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:44 smithi183 ceph-mon[37273]: pgmap v2696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:12:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:12:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:45.282+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:45.283+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:45.283+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:45.283+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:12:45.283+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4009.616698 secs 2024-07-26T12:12:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:46 smithi183 ceph-mon[37273]: pgmap v2697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4009.616698 secs 2024-07-26T12:12:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:46 smithi144 ceph-mon[32129]: pgmap v2697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4009.616698 secs 2024-07-26T12:12:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:46 smithi160 ceph-mon[36715]: pgmap v2697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:48 smithi183 ceph-mon[37273]: pgmap v2698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:12:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:48 smithi144 ceph-mon[32129]: pgmap v2698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:12:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:48 smithi160 ceph-mon[36715]: pgmap v2698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:12:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:50 smithi183 ceph-mon[37273]: pgmap v2699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:12:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:50 smithi144 ceph-mon[32129]: pgmap v2699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:12:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:50 smithi160 ceph-mon[36715]: pgmap v2699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:12:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4014.616818 secs 2024-07-26T12:12:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4014.616818 secs 2024-07-26T12:12:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4014.616818 secs 2024-07-26T12:12:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:52 smithi183 ceph-mon[37273]: pgmap v2700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:52 smithi144 ceph-mon[32129]: pgmap v2700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:52 smithi160 ceph-mon[36715]: pgmap v2700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:12:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:54 smithi144 ceph-mon[32129]: pgmap v2701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:12:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:54 smithi160 ceph-mon[36715]: pgmap v2701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:12:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:54 smithi183 ceph-mon[37273]: pgmap v2701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:12:57.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4019.616938 secs 2024-07-26T12:12:57.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:56 smithi183 ceph-mon[37273]: pgmap v2702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4019.616938 secs 2024-07-26T12:12:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:56 smithi144 ceph-mon[32129]: pgmap v2702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4019.616938 secs 2024-07-26T12:12:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:56 smithi160 ceph-mon[36715]: pgmap v2702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:12:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:12:58 smithi144 ceph-mon[32129]: pgmap v2703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:12:58 smithi160 ceph-mon[36715]: pgmap v2703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:12:58 smithi183 ceph-mon[37273]: pgmap v2703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:00.299+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:00.300+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:00.300+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:00.300+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:00.300+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:00 smithi144 ceph-mon[32129]: pgmap v2704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:00 smithi160 ceph-mon[36715]: pgmap v2704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:00 smithi183 ceph-mon[37273]: pgmap v2704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4024.617051 secs 2024-07-26T12:13:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4024.617051 secs 2024-07-26T12:13:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4024.617051 secs 2024-07-26T12:13:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:02 smithi144 ceph-mon[32129]: pgmap v2705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:02 smithi160 ceph-mon[36715]: pgmap v2705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:02 smithi183 ceph-mon[37273]: pgmap v2705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:04 smithi144 ceph-mon[32129]: pgmap v2706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:04 smithi160 ceph-mon[36715]: pgmap v2706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:04 smithi183 ceph-mon[37273]: pgmap v2706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4029.617190 secs 2024-07-26T12:13:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:06 smithi144 ceph-mon[32129]: pgmap v2707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4029.617190 secs 2024-07-26T12:13:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:06 smithi160 ceph-mon[36715]: pgmap v2707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4029.617190 secs 2024-07-26T12:13:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:06 smithi183 ceph-mon[37273]: pgmap v2707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:08 smithi144 ceph-mon[32129]: pgmap v2708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:08 smithi160 ceph-mon[36715]: pgmap v2708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:08 smithi183 ceph-mon[37273]: pgmap v2708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:10 smithi144 ceph-mon[32129]: pgmap v2709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:10 smithi160 ceph-mon[36715]: pgmap v2709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:10 smithi183 ceph-mon[37273]: pgmap v2709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4034.617305 secs 2024-07-26T12:13:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4034.617305 secs 2024-07-26T12:13:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4034.617305 secs 2024-07-26T12:13:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:12 smithi144 ceph-mon[32129]: pgmap v2710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:12 smithi160 ceph-mon[36715]: pgmap v2710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:12 smithi183 ceph-mon[37273]: pgmap v2710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:14 smithi144 ceph-mon[32129]: pgmap v2711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:14 smithi160 ceph-mon[36715]: pgmap v2711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:15.316+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:15.317+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:15.317+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:15.317+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:15.317+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:14 smithi183 ceph-mon[37273]: pgmap v2711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4039.617430 secs 2024-07-26T12:13:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:16 smithi144 ceph-mon[32129]: pgmap v2712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4039.617430 secs 2024-07-26T12:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:16 smithi160 ceph-mon[36715]: pgmap v2712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4039.617430 secs 2024-07-26T12:13:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:16 smithi183 ceph-mon[37273]: pgmap v2712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:18 smithi160 ceph-mon[36715]: pgmap v2713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:18 smithi183 ceph-mon[37273]: pgmap v2713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:18 smithi144 ceph-mon[32129]: pgmap v2713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:20 smithi160 ceph-mon[36715]: pgmap v2714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:20 smithi183 ceph-mon[37273]: pgmap v2714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:20 smithi144 ceph-mon[32129]: pgmap v2714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4044.617557 secs 2024-07-26T12:13:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4044.617557 secs 2024-07-26T12:13:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4044.617557 secs 2024-07-26T12:13:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:22 smithi160 ceph-mon[36715]: pgmap v2715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:22 smithi183 ceph-mon[37273]: pgmap v2715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:22 smithi144 ceph-mon[32129]: pgmap v2715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:24 smithi160 ceph-mon[36715]: pgmap v2716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:24 smithi183 ceph-mon[37273]: pgmap v2716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:24 smithi144 ceph-mon[32129]: pgmap v2716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4049.617681 secs 2024-07-26T12:13:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:27 smithi160 ceph-mon[36715]: pgmap v2717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4049.617681 secs 2024-07-26T12:13:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:27 smithi183 ceph-mon[37273]: pgmap v2717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4049.617681 secs 2024-07-26T12:13:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:27 smithi144 ceph-mon[32129]: pgmap v2717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:29 smithi160 ceph-mon[36715]: pgmap v2718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:29 smithi183 ceph-mon[37273]: pgmap v2718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:29 smithi144 ceph-mon[32129]: pgmap v2718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:30.329+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:30.330+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:30.330+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:30.330+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:30.330+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:31.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:31 smithi144 ceph-mon[32129]: pgmap v2719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:31 smithi160 ceph-mon[36715]: pgmap v2719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:31 smithi183 ceph-mon[37273]: pgmap v2719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4054.617796 secs 2024-07-26T12:13:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4054.617796 secs 2024-07-26T12:13:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4054.617796 secs 2024-07-26T12:13:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:33 smithi160 ceph-mon[36715]: pgmap v2720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:33 smithi183 ceph-mon[37273]: pgmap v2720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:33 smithi144 ceph-mon[32129]: pgmap v2720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:35 smithi160 ceph-mon[36715]: pgmap v2721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:35 smithi183 ceph-mon[37273]: pgmap v2721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:35 smithi144 ceph-mon[32129]: pgmap v2721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4059.617908 secs 2024-07-26T12:13:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:37 smithi160 ceph-mon[36715]: pgmap v2722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4059.617908 secs 2024-07-26T12:13:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:37 smithi183 ceph-mon[37273]: pgmap v2722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4059.617908 secs 2024-07-26T12:13:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:37 smithi144 ceph-mon[32129]: pgmap v2722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:39 smithi160 ceph-mon[36715]: pgmap v2723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:39 smithi183 ceph-mon[37273]: pgmap v2723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:39 smithi144 ceph-mon[32129]: pgmap v2723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:41 smithi160 ceph-mon[36715]: pgmap v2724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:41 smithi183 ceph-mon[37273]: pgmap v2724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:41 smithi144 ceph-mon[32129]: pgmap v2724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:42.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4064.618027 secs 2024-07-26T12:13:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4064.618027 secs 2024-07-26T12:13:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4064.618027 secs 2024-07-26T12:13:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:43 smithi183 ceph-mon[37273]: pgmap v2725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:43 smithi144 ceph-mon[32129]: pgmap v2725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:43 smithi160 ceph-mon[36715]: pgmap v2725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:45.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:45.347+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:45.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:45.348+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:45.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:45.348+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:45.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:45.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:45.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:13:45.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:45 smithi183 ceph-mon[37273]: pgmap v2726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:13:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:45 smithi144 ceph-mon[32129]: pgmap v2726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:13:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:45 smithi160 ceph-mon[36715]: pgmap v2726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:13:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:13:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:46 smithi183 ceph-mon[37273]: pgmap v2727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:13:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:46 smithi183 ceph-mon[37273]: pgmap v2728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:13:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:46 smithi144 ceph-mon[32129]: pgmap v2727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:13:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:46 smithi144 ceph-mon[32129]: pgmap v2728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:13:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:46 smithi160 ceph-mon[36715]: pgmap v2727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:13:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:46 smithi160 ceph-mon[36715]: pgmap v2728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:13:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4069.618150 secs 2024-07-26T12:13:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4069.618150 secs 2024-07-26T12:13:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4069.618150 secs 2024-07-26T12:13:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:48 smithi183 ceph-mon[37273]: pgmap v2729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:13:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:48 smithi144 ceph-mon[32129]: pgmap v2729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:13:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:48 smithi160 ceph-mon[36715]: pgmap v2729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:13:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:49 smithi183 ceph-mon[37273]: pgmap v2730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:13:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:49 smithi144 ceph-mon[32129]: pgmap v2730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:13:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:49 smithi160 ceph-mon[36715]: pgmap v2730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:13:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4074.618266 secs 2024-07-26T12:13:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:52 smithi160 ceph-mon[36715]: pgmap v2731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:13:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4074.618266 secs 2024-07-26T12:13:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:52 smithi183 ceph-mon[37273]: pgmap v2731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:13:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4074.618266 secs 2024-07-26T12:13:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:52 smithi144 ceph-mon[32129]: pgmap v2731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:13:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:54 smithi160 ceph-mon[36715]: pgmap v2732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:13:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:54 smithi183 ceph-mon[37273]: pgmap v2732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:13:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:54 smithi144 ceph-mon[32129]: pgmap v2732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:13:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:56 smithi160 ceph-mon[36715]: pgmap v2733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:13:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:56 smithi183 ceph-mon[37273]: pgmap v2733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:13:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:56 smithi144 ceph-mon[32129]: pgmap v2733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:13:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4079.618380 secs 2024-07-26T12:13:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4079.618380 secs 2024-07-26T12:13:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4079.618380 secs 2024-07-26T12:13:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:13:58 smithi160 ceph-mon[36715]: pgmap v2734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:13:58 smithi183 ceph-mon[37273]: pgmap v2734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:13:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:13:58 smithi144 ceph-mon[32129]: pgmap v2734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:00.364+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:00.365+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:00.365+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:00.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:00.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:00 smithi183 ceph-mon[37273]: pgmap v2735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:00 smithi144 ceph-mon[32129]: pgmap v2735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:00 smithi160 ceph-mon[36715]: pgmap v2735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4084.618490 secs 2024-07-26T12:14:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:02 smithi183 ceph-mon[37273]: pgmap v2736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4084.618490 secs 2024-07-26T12:14:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:02 smithi144 ceph-mon[32129]: pgmap v2736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4084.618490 secs 2024-07-26T12:14:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:02 smithi160 ceph-mon[36715]: pgmap v2736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:04 smithi183 ceph-mon[37273]: pgmap v2737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:04 smithi144 ceph-mon[32129]: pgmap v2737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:04 smithi160 ceph-mon[36715]: pgmap v2737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:06 smithi183 ceph-mon[37273]: pgmap v2738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:06 smithi144 ceph-mon[32129]: pgmap v2738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:06 smithi160 ceph-mon[36715]: pgmap v2738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4089.618628 secs 2024-07-26T12:14:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4089.618628 secs 2024-07-26T12:14:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4089.618628 secs 2024-07-26T12:14:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:08 smithi183 ceph-mon[37273]: pgmap v2739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:08 smithi144 ceph-mon[32129]: pgmap v2739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:08 smithi160 ceph-mon[36715]: pgmap v2739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:09 smithi183 ceph-mon[37273]: pgmap v2740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:09 smithi144 ceph-mon[32129]: pgmap v2740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:09 smithi160 ceph-mon[36715]: pgmap v2740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4094.618759 secs 2024-07-26T12:14:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:12 smithi160 ceph-mon[36715]: pgmap v2741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4094.618759 secs 2024-07-26T12:14:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:12 smithi183 ceph-mon[37273]: pgmap v2741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4094.618759 secs 2024-07-26T12:14:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:12 smithi144 ceph-mon[32129]: pgmap v2741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:14 smithi144 ceph-mon[32129]: pgmap v2742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:14 smithi160 ceph-mon[36715]: pgmap v2742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:14 smithi183 ceph-mon[37273]: pgmap v2742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:15.382 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:15.381+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:15.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:15.382+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:15.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:15.382+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:15.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:15.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:15.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:15.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:16 smithi160 ceph-mon[36715]: pgmap v2743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:16 smithi183 ceph-mon[37273]: pgmap v2743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:16 smithi144 ceph-mon[32129]: pgmap v2743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4099.618815 secs 2024-07-26T12:14:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4099.618815 secs 2024-07-26T12:14:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4099.618815 secs 2024-07-26T12:14:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:18 smithi183 ceph-mon[37273]: pgmap v2744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:18 smithi144 ceph-mon[32129]: pgmap v2744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:18 smithi160 ceph-mon[36715]: pgmap v2744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:20 smithi183 ceph-mon[37273]: pgmap v2745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:20 smithi144 ceph-mon[32129]: pgmap v2745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:20 smithi160 ceph-mon[36715]: pgmap v2745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:22.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4104.618926 secs 2024-07-26T12:14:22.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:22 smithi160 ceph-mon[36715]: pgmap v2746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4104.618926 secs 2024-07-26T12:14:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:22 smithi183 ceph-mon[37273]: pgmap v2746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4104.618926 secs 2024-07-26T12:14:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:22 smithi144 ceph-mon[32129]: pgmap v2746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:24 smithi183 ceph-mon[37273]: pgmap v2747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:24 smithi144 ceph-mon[32129]: pgmap v2747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:24 smithi160 ceph-mon[36715]: pgmap v2747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:25 smithi183 ceph-mon[37273]: pgmap v2748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:25 smithi144 ceph-mon[32129]: pgmap v2748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:25 smithi160 ceph-mon[36715]: pgmap v2748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4109.619068 secs 2024-07-26T12:14:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4109.619068 secs 2024-07-26T12:14:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4109.619068 secs 2024-07-26T12:14:27.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:27 smithi183 ceph-mon[37273]: pgmap v2749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:27 smithi160 ceph-mon[36715]: pgmap v2749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:27 smithi144 ceph-mon[32129]: pgmap v2749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:30 smithi160 ceph-mon[36715]: pgmap v2750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:30 smithi183 ceph-mon[37273]: pgmap v2750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:30.397+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:30.398+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:30.398+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:30.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:30.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:30 smithi144 ceph-mon[32129]: pgmap v2750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4114.619191 secs 2024-07-26T12:14:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:32 smithi160 ceph-mon[36715]: pgmap v2751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4114.619191 secs 2024-07-26T12:14:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:32 smithi183 ceph-mon[37273]: pgmap v2751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4114.619191 secs 2024-07-26T12:14:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:32 smithi144 ceph-mon[32129]: pgmap v2751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:34 smithi183 ceph-mon[37273]: pgmap v2752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:34 smithi144 ceph-mon[32129]: pgmap v2752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:34 smithi160 ceph-mon[36715]: pgmap v2752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:36 smithi183 ceph-mon[37273]: pgmap v2753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:36 smithi144 ceph-mon[32129]: pgmap v2753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:36 smithi160 ceph-mon[36715]: pgmap v2753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4119.619325 secs 2024-07-26T12:14:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4119.619325 secs 2024-07-26T12:14:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4119.619325 secs 2024-07-26T12:14:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:38 smithi183 ceph-mon[37273]: pgmap v2754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:38 smithi144 ceph-mon[32129]: pgmap v2754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:38 smithi160 ceph-mon[36715]: pgmap v2754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:14:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:39 smithi144 ceph-mon[32129]: pgmap v2755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:39 smithi160 ceph-mon[36715]: pgmap v2755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:39 smithi183 ceph-mon[37273]: pgmap v2755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4124.619443 secs 2024-07-26T12:14:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:42 smithi183 ceph-mon[37273]: pgmap v2756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4124.619443 secs 2024-07-26T12:14:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:42 smithi144 ceph-mon[32129]: pgmap v2756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4124.619443 secs 2024-07-26T12:14:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:42 smithi160 ceph-mon[36715]: pgmap v2756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:44 smithi183 ceph-mon[37273]: pgmap v2757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:44 smithi144 ceph-mon[32129]: pgmap v2757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:44 smithi160 ceph-mon[36715]: pgmap v2757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:45.414+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:45.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:45.415+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:45.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:45.415+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:45.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:45.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:45.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:45.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:46 smithi183 ceph-mon[37273]: pgmap v2758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:46 smithi144 ceph-mon[32129]: pgmap v2758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:46 smithi160 ceph-mon[36715]: pgmap v2758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:14:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:46.825+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:14:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:46.825+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006290 con 0x563c6c4182f0 2024-07-26T12:14:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:46.826+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006290 con 0x7f74b0109990 2024-07-26T12:14:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:14:46.826+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:14:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4129.619566 secs 2024-07-26T12:14:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:14:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4129.619566 secs 2024-07-26T12:14:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4129.619566 secs 2024-07-26T12:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:14:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:48 smithi183 ceph-mon[37273]: pgmap v2759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:48 smithi183 ceph-mon[37273]: pgmap v2760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:48 smithi144 ceph-mon[32129]: pgmap v2759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:48 smithi144 ceph-mon[32129]: pgmap v2760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:48 smithi160 ceph-mon[36715]: pgmap v2759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:48 smithi160 ceph-mon[36715]: pgmap v2760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:49 smithi144 ceph-mon[32129]: pgmap v2761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:14:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:49 smithi160 ceph-mon[36715]: pgmap v2761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:14:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:49 smithi183 ceph-mon[37273]: pgmap v2761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:14:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:51 smithi160 ceph-mon[36715]: pgmap v2762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:14:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4134.619675 secs 2024-07-26T12:14:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:51 smithi183 ceph-mon[37273]: pgmap v2762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:14:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4134.619675 secs 2024-07-26T12:14:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:51 smithi144 ceph-mon[32129]: pgmap v2762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:14:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4134.619675 secs 2024-07-26T12:14:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:53 smithi160 ceph-mon[36715]: pgmap v2763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:53 smithi183 ceph-mon[37273]: pgmap v2763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:53 smithi144 ceph-mon[32129]: pgmap v2763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:55.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:55 smithi160 ceph-mon[36715]: pgmap v2764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:55 smithi183 ceph-mon[37273]: pgmap v2764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:55 smithi144 ceph-mon[32129]: pgmap v2764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:14:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4139.619801 secs 2024-07-26T12:14:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4139.619801 secs 2024-07-26T12:14:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4139.619801 secs 2024-07-26T12:14:57.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:57 smithi183 ceph-mon[37273]: pgmap v2765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:14:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:57 smithi160 ceph-mon[36715]: pgmap v2765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:14:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:57 smithi144 ceph-mon[32129]: pgmap v2765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:14:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:14:59 smithi160 ceph-mon[36715]: pgmap v2766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:14:59 smithi183 ceph-mon[37273]: pgmap v2766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:14:59 smithi144 ceph-mon[32129]: pgmap v2766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:00.423+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:00.424+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:00.424+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:00.424+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:00.424+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:01 smithi160 ceph-mon[36715]: pgmap v2767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4144.619925 secs 2024-07-26T12:15:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:01 smithi183 ceph-mon[37273]: pgmap v2767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4144.619925 secs 2024-07-26T12:15:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:01 smithi144 ceph-mon[32129]: pgmap v2767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4144.619925 secs 2024-07-26T12:15:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:03 smithi160 ceph-mon[36715]: pgmap v2768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:03 smithi183 ceph-mon[37273]: pgmap v2768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:03 smithi144 ceph-mon[32129]: pgmap v2768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:05 smithi160 ceph-mon[36715]: pgmap v2769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:05 smithi183 ceph-mon[37273]: pgmap v2769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:05 smithi144 ceph-mon[32129]: pgmap v2769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4149.620067 secs 2024-07-26T12:15:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4149.620067 secs 2024-07-26T12:15:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4149.620067 secs 2024-07-26T12:15:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:07 smithi183 ceph-mon[37273]: pgmap v2770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:07 smithi144 ceph-mon[32129]: pgmap v2770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:07 smithi160 ceph-mon[36715]: pgmap v2770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:09 smithi183 ceph-mon[37273]: pgmap v2771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:09 smithi144 ceph-mon[32129]: pgmap v2771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:09 smithi160 ceph-mon[36715]: pgmap v2771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:11 smithi183 ceph-mon[37273]: pgmap v2772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4154.620177 secs 2024-07-26T12:15:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:11 smithi144 ceph-mon[32129]: pgmap v2772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4154.620177 secs 2024-07-26T12:15:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:11 smithi160 ceph-mon[36715]: pgmap v2772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4154.620177 secs 2024-07-26T12:15:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:13 smithi183 ceph-mon[37273]: pgmap v2773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:13 smithi144 ceph-mon[32129]: pgmap v2773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:13 smithi160 ceph-mon[36715]: pgmap v2773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:15.440+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:15.441+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:15.441+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:15.441+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:15.441+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:15 smithi183 ceph-mon[37273]: pgmap v2774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:15 smithi144 ceph-mon[32129]: pgmap v2774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:15 smithi160 ceph-mon[36715]: pgmap v2774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4159.620320 secs 2024-07-26T12:15:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4159.620320 secs 2024-07-26T12:15:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4159.620320 secs 2024-07-26T12:15:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:17 smithi183 ceph-mon[37273]: pgmap v2775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:17 smithi144 ceph-mon[32129]: pgmap v2775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:17 smithi160 ceph-mon[36715]: pgmap v2775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:19 smithi183 ceph-mon[37273]: pgmap v2776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:19 smithi144 ceph-mon[32129]: pgmap v2776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:19 smithi160 ceph-mon[36715]: pgmap v2776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:21 smithi183 ceph-mon[37273]: pgmap v2777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4164.620401 secs 2024-07-26T12:15:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:21 smithi144 ceph-mon[32129]: pgmap v2777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4164.620401 secs 2024-07-26T12:15:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:21 smithi160 ceph-mon[36715]: pgmap v2777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4164.620401 secs 2024-07-26T12:15:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:23 smithi144 ceph-mon[32129]: pgmap v2778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:23 smithi160 ceph-mon[36715]: pgmap v2778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:23 smithi183 ceph-mon[37273]: pgmap v2778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:25 smithi144 ceph-mon[32129]: pgmap v2779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:25 smithi160 ceph-mon[36715]: pgmap v2779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:25 smithi183 ceph-mon[37273]: pgmap v2779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4169.620523 secs 2024-07-26T12:15:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4169.620523 secs 2024-07-26T12:15:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4169.620523 secs 2024-07-26T12:15:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:27 smithi144 ceph-mon[32129]: pgmap v2780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:27.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:27 smithi183 ceph-mon[37273]: pgmap v2780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:27 smithi160 ceph-mon[36715]: pgmap v2780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:29 smithi144 ceph-mon[32129]: pgmap v2781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:29 smithi160 ceph-mon[36715]: pgmap v2781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:29 smithi183 ceph-mon[37273]: pgmap v2781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:30.443+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:30.444+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:30.444+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:30.444+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:30.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:30.444+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:31 smithi144 ceph-mon[32129]: pgmap v2782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4174.620632 secs 2024-07-26T12:15:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:31 smithi160 ceph-mon[36715]: pgmap v2782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4174.620632 secs 2024-07-26T12:15:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:31 smithi183 ceph-mon[37273]: pgmap v2782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4174.620632 secs 2024-07-26T12:15:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:33 smithi144 ceph-mon[32129]: pgmap v2783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:33 smithi160 ceph-mon[36715]: pgmap v2783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:33 smithi183 ceph-mon[37273]: pgmap v2783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:35 smithi144 ceph-mon[32129]: pgmap v2784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:35 smithi160 ceph-mon[36715]: pgmap v2784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:35 smithi183 ceph-mon[37273]: pgmap v2784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4179.620760 secs 2024-07-26T12:15:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4179.620760 secs 2024-07-26T12:15:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4179.620760 secs 2024-07-26T12:15:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:37 smithi144 ceph-mon[32129]: pgmap v2785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:37 smithi160 ceph-mon[36715]: pgmap v2785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:37 smithi183 ceph-mon[37273]: pgmap v2785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:39 smithi144 ceph-mon[32129]: pgmap v2786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:39 smithi160 ceph-mon[36715]: pgmap v2786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:39 smithi183 ceph-mon[37273]: pgmap v2786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:41 smithi144 ceph-mon[32129]: pgmap v2787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4184.620819 secs 2024-07-26T12:15:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:41 smithi160 ceph-mon[36715]: pgmap v2787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4184.620819 secs 2024-07-26T12:15:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:41 smithi183 ceph-mon[37273]: pgmap v2787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4184.620819 secs 2024-07-26T12:15:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:43 smithi160 ceph-mon[36715]: pgmap v2788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:43 smithi183 ceph-mon[37273]: pgmap v2788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:43 smithi144 ceph-mon[32129]: pgmap v2788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:45.461+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:45.462+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:45.462+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:45.462+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:15:45.462+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:45 smithi160 ceph-mon[36715]: pgmap v2789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:45 smithi183 ceph-mon[37273]: pgmap v2789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:45 smithi144 ceph-mon[32129]: pgmap v2789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:15:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4189.620957 secs 2024-07-26T12:15:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4189.620957 secs 2024-07-26T12:15:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4189.620957 secs 2024-07-26T12:15:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:47 smithi160 ceph-mon[36715]: pgmap v2790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:15:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:47 smithi183 ceph-mon[37273]: pgmap v2790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:15:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:47 smithi144 ceph-mon[32129]: pgmap v2790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:15:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:15:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:48 smithi160 ceph-mon[36715]: pgmap v2791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:15:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:48 smithi160 ceph-mon[36715]: pgmap v2792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:48 smithi183 ceph-mon[37273]: pgmap v2791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:15:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:48 smithi183 ceph-mon[37273]: pgmap v2792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:48 smithi144 ceph-mon[32129]: pgmap v2791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:15:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:48 smithi144 ceph-mon[32129]: pgmap v2792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:50 smithi160 ceph-mon[36715]: pgmap v2793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:50 smithi183 ceph-mon[37273]: pgmap v2793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:50 smithi144 ceph-mon[32129]: pgmap v2793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4194.621045 secs 2024-07-26T12:15:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4194.621045 secs 2024-07-26T12:15:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4194.621045 secs 2024-07-26T12:15:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:52 smithi160 ceph-mon[36715]: pgmap v2794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:52 smithi183 ceph-mon[37273]: pgmap v2794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:52 smithi144 ceph-mon[32129]: pgmap v2794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:54 smithi160 ceph-mon[36715]: pgmap v2795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:54 smithi183 ceph-mon[37273]: pgmap v2795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:54 smithi144 ceph-mon[32129]: pgmap v2795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:56 smithi160 ceph-mon[36715]: pgmap v2796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4199.621165 secs 2024-07-26T12:15:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:56 smithi183 ceph-mon[37273]: pgmap v2796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4199.621165 secs 2024-07-26T12:15:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:56 smithi144 ceph-mon[32129]: pgmap v2796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4199.621165 secs 2024-07-26T12:15:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:15:58 smithi160 ceph-mon[36715]: pgmap v2797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:15:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:15:58 smithi183 ceph-mon[37273]: pgmap v2797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:15:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:15:58 smithi144 ceph-mon[32129]: pgmap v2797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:00.479 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:00.478+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:00.479+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:00.479+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:00.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:00.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:00 smithi160 ceph-mon[36715]: pgmap v2798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:00 smithi183 ceph-mon[37273]: pgmap v2798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:00 smithi144 ceph-mon[32129]: pgmap v2798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4204.621283 secs 2024-07-26T12:16:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4204.621283 secs 2024-07-26T12:16:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4204.621283 secs 2024-07-26T12:16:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:02 smithi183 ceph-mon[37273]: pgmap v2799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:02 smithi144 ceph-mon[32129]: pgmap v2799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:02 smithi160 ceph-mon[36715]: pgmap v2799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:04 smithi183 ceph-mon[37273]: pgmap v2800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:04 smithi144 ceph-mon[32129]: pgmap v2800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:04 smithi160 ceph-mon[36715]: pgmap v2800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:06 smithi183 ceph-mon[37273]: pgmap v2801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4209.621429 secs 2024-07-26T12:16:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:06 smithi144 ceph-mon[32129]: pgmap v2801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4209.621429 secs 2024-07-26T12:16:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:06 smithi160 ceph-mon[36715]: pgmap v2801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4209.621429 secs 2024-07-26T12:16:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:08 smithi183 ceph-mon[37273]: pgmap v2802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:08 smithi144 ceph-mon[32129]: pgmap v2802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:08 smithi160 ceph-mon[36715]: pgmap v2802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:10 smithi183 ceph-mon[37273]: pgmap v2803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:10 smithi144 ceph-mon[32129]: pgmap v2803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:10 smithi160 ceph-mon[36715]: pgmap v2803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4214.621531 secs 2024-07-26T12:16:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4214.621531 secs 2024-07-26T12:16:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4214.621531 secs 2024-07-26T12:16:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:12 smithi183 ceph-mon[37273]: pgmap v2804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:12 smithi144 ceph-mon[32129]: pgmap v2804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:12 smithi160 ceph-mon[36715]: pgmap v2804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:14 smithi183 ceph-mon[37273]: pgmap v2805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:14 smithi144 ceph-mon[32129]: pgmap v2805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:14 smithi160 ceph-mon[36715]: pgmap v2805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:15.490+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:15.491+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:15.491+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:15.491+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:15.491+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:16 smithi183 ceph-mon[37273]: pgmap v2806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4219.621601 secs 2024-07-26T12:16:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:16 smithi144 ceph-mon[32129]: pgmap v2806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4219.621601 secs 2024-07-26T12:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:16 smithi160 ceph-mon[36715]: pgmap v2806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4219.621601 secs 2024-07-26T12:16:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:18 smithi183 ceph-mon[37273]: pgmap v2807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:18 smithi144 ceph-mon[32129]: pgmap v2807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:18 smithi160 ceph-mon[36715]: pgmap v2807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:20 smithi144 ceph-mon[32129]: pgmap v2808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:16:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:20 smithi160 ceph-mon[36715]: pgmap v2808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:16:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:20 smithi183 ceph-mon[37273]: pgmap v2808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:16:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4224.621670 secs 2024-07-26T12:16:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4224.621670 secs 2024-07-26T12:16:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4224.621670 secs 2024-07-26T12:16:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:22 smithi144 ceph-mon[32129]: pgmap v2809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:22 smithi160 ceph-mon[36715]: pgmap v2809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:22 smithi183 ceph-mon[37273]: pgmap v2809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:24 smithi144 ceph-mon[32129]: pgmap v2810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:24 smithi160 ceph-mon[36715]: pgmap v2810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:24 smithi183 ceph-mon[37273]: pgmap v2810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:26 smithi144 ceph-mon[32129]: pgmap v2811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4229.621748 secs 2024-07-26T12:16:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:26 smithi160 ceph-mon[36715]: pgmap v2811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4229.621748 secs 2024-07-26T12:16:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:26 smithi183 ceph-mon[37273]: pgmap v2811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4229.621748 secs 2024-07-26T12:16:28.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:28 smithi183 ceph-mon[37273]: pgmap v2812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:28 smithi144 ceph-mon[32129]: pgmap v2812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:28 smithi160 ceph-mon[36715]: pgmap v2812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:30.506+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:30.507+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:30.507+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:30.507+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:30.507+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:30 smithi144 ceph-mon[32129]: pgmap v2813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:30 smithi160 ceph-mon[36715]: pgmap v2813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:30 smithi183 ceph-mon[37273]: pgmap v2813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4234.621861 secs 2024-07-26T12:16:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4234.621861 secs 2024-07-26T12:16:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4234.621861 secs 2024-07-26T12:16:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:32 smithi144 ceph-mon[32129]: pgmap v2814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:32 smithi160 ceph-mon[36715]: pgmap v2814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:32 smithi183 ceph-mon[37273]: pgmap v2814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:34 smithi144 ceph-mon[32129]: pgmap v2815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:34 smithi160 ceph-mon[36715]: pgmap v2815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:34 smithi183 ceph-mon[37273]: pgmap v2815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:36 smithi144 ceph-mon[32129]: pgmap v2816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4239.621976 secs 2024-07-26T12:16:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:36 smithi160 ceph-mon[36715]: pgmap v2816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4239.621976 secs 2024-07-26T12:16:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:36 smithi183 ceph-mon[37273]: pgmap v2816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4239.621976 secs 2024-07-26T12:16:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:38 smithi144 ceph-mon[32129]: pgmap v2817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:38 smithi160 ceph-mon[36715]: pgmap v2817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:38 smithi183 ceph-mon[37273]: pgmap v2817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:40 smithi144 ceph-mon[32129]: pgmap v2818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:40 smithi160 ceph-mon[36715]: pgmap v2818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:40 smithi183 ceph-mon[37273]: pgmap v2818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4244.622095 secs 2024-07-26T12:16:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4244.622095 secs 2024-07-26T12:16:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4244.622095 secs 2024-07-26T12:16:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:42 smithi160 ceph-mon[36715]: pgmap v2819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:42 smithi183 ceph-mon[37273]: pgmap v2819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:42 smithi144 ceph-mon[32129]: pgmap v2819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:16:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:44 smithi160 ceph-mon[36715]: pgmap v2820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:44 smithi183 ceph-mon[37273]: pgmap v2820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:44 smithi144 ceph-mon[32129]: pgmap v2820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:45.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:45.525+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:45.525+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:45.525+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:16:45.525+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:46 smithi160 ceph-mon[36715]: pgmap v2821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4249.622220 secs 2024-07-26T12:16:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:46 smithi183 ceph-mon[37273]: pgmap v2821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4249.622220 secs 2024-07-26T12:16:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:46 smithi144 ceph-mon[32129]: pgmap v2821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:16:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4249.622220 secs 2024-07-26T12:16:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:49.054 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:48 smithi144 ceph-mon[32129]: pgmap v2822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:49.054 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:49.054 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:49.054 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:16:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:48 smithi160 ceph-mon[36715]: pgmap v2822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:16:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:48 smithi183 ceph-mon[37273]: pgmap v2822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:16:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:49 smithi160 ceph-mon[36715]: pgmap v2823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:49 smithi160 ceph-mon[36715]: pgmap v2824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:16:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:49 smithi183 ceph-mon[37273]: pgmap v2823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:49 smithi183 ceph-mon[37273]: pgmap v2824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:16:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:49 smithi144 ceph-mon[32129]: pgmap v2823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:49 smithi144 ceph-mon[32129]: pgmap v2824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:16:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:51 smithi160 ceph-mon[36715]: pgmap v2825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4254.622340 secs 2024-07-26T12:16:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:51 smithi183 ceph-mon[37273]: pgmap v2825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4254.622340 secs 2024-07-26T12:16:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:51 smithi144 ceph-mon[32129]: pgmap v2825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4254.622340 secs 2024-07-26T12:16:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:53 smithi160 ceph-mon[36715]: pgmap v2826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:53 smithi183 ceph-mon[37273]: pgmap v2826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:53 smithi144 ceph-mon[32129]: pgmap v2826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:55 smithi160 ceph-mon[36715]: pgmap v2827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:55 smithi183 ceph-mon[37273]: pgmap v2827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:55 smithi144 ceph-mon[32129]: pgmap v2827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:16:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4259.622464 secs 2024-07-26T12:16:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4259.622464 secs 2024-07-26T12:16:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4259.622464 secs 2024-07-26T12:16:58.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:57 smithi183 ceph-mon[37273]: pgmap v2828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:57 smithi160 ceph-mon[36715]: pgmap v2828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:16:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:57 smithi144 ceph-mon[32129]: pgmap v2828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:16:59 smithi160 ceph-mon[36715]: pgmap v2829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:17:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:16:59 smithi183 ceph-mon[37273]: pgmap v2829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:17:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:16:59 smithi144 ceph-mon[32129]: pgmap v2829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:17:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:00.541+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:00.542+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:00.542+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:00.542+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:00.542+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:01 smithi160 ceph-mon[36715]: pgmap v2830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4264.622591 secs 2024-07-26T12:17:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:01 smithi183 ceph-mon[37273]: pgmap v2830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4264.622591 secs 2024-07-26T12:17:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:01 smithi144 ceph-mon[32129]: pgmap v2830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4264.622591 secs 2024-07-26T12:17:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:03 smithi160 ceph-mon[36715]: pgmap v2831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:03 smithi183 ceph-mon[37273]: pgmap v2831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:03 smithi144 ceph-mon[32129]: pgmap v2831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:05 smithi160 ceph-mon[36715]: pgmap v2832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:05 smithi183 ceph-mon[37273]: pgmap v2832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:05 smithi144 ceph-mon[32129]: pgmap v2832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4269.622708 secs 2024-07-26T12:17:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4269.622708 secs 2024-07-26T12:17:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4269.622708 secs 2024-07-26T12:17:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:07 smithi183 ceph-mon[37273]: pgmap v2833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:07 smithi144 ceph-mon[32129]: pgmap v2833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:07 smithi160 ceph-mon[36715]: pgmap v2833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:09 smithi183 ceph-mon[37273]: pgmap v2834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:09 smithi144 ceph-mon[32129]: pgmap v2834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:09 smithi160 ceph-mon[36715]: pgmap v2834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:11 smithi183 ceph-mon[37273]: pgmap v2835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4274.622831 secs 2024-07-26T12:17:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:11 smithi144 ceph-mon[32129]: pgmap v2835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4274.622831 secs 2024-07-26T12:17:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:11 smithi160 ceph-mon[36715]: pgmap v2835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4274.622831 secs 2024-07-26T12:17:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:13 smithi183 ceph-mon[37273]: pgmap v2836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:13 smithi144 ceph-mon[32129]: pgmap v2836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:13 smithi160 ceph-mon[36715]: pgmap v2836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:15.552+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:15.553+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:15.553+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:15.553+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:15.553+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:15 smithi183 ceph-mon[37273]: pgmap v2837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:15 smithi144 ceph-mon[32129]: pgmap v2837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:15 smithi160 ceph-mon[36715]: pgmap v2837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4279.622966 secs 2024-07-26T12:17:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4279.622966 secs 2024-07-26T12:17:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4279.622966 secs 2024-07-26T12:17:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:17 smithi183 ceph-mon[37273]: pgmap v2838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:17 smithi144 ceph-mon[32129]: pgmap v2838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:17 smithi160 ceph-mon[36715]: pgmap v2838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:19 smithi183 ceph-mon[37273]: pgmap v2839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:19 smithi144 ceph-mon[32129]: pgmap v2839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:19 smithi160 ceph-mon[36715]: pgmap v2839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4284.623044 secs 2024-07-26T12:17:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:21 smithi183 ceph-mon[37273]: pgmap v2840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4284.623044 secs 2024-07-26T12:17:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:21 smithi144 ceph-mon[32129]: pgmap v2840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4284.623044 secs 2024-07-26T12:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:21 smithi160 ceph-mon[36715]: pgmap v2840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:23 smithi183 ceph-mon[37273]: pgmap v2841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:23 smithi144 ceph-mon[32129]: pgmap v2841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:23 smithi160 ceph-mon[36715]: pgmap v2841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:25 smithi183 ceph-mon[37273]: pgmap v2842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:25 smithi144 ceph-mon[32129]: pgmap v2842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:25 smithi160 ceph-mon[36715]: pgmap v2842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4289.623170 secs 2024-07-26T12:17:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4289.623170 secs 2024-07-26T12:17:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4289.623170 secs 2024-07-26T12:17:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:27 smithi144 ceph-mon[32129]: pgmap v2843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:28.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:27 smithi183 ceph-mon[37273]: pgmap v2843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:27 smithi160 ceph-mon[36715]: pgmap v2843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:29 smithi144 ceph-mon[32129]: pgmap v2844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:29 smithi160 ceph-mon[36715]: pgmap v2844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:29 smithi183 ceph-mon[37273]: pgmap v2844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:30.562+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:30.563+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:30.563+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:30.563+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:30.563+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4294.623308 secs 2024-07-26T12:17:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:31 smithi144 ceph-mon[32129]: pgmap v2845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4294.623308 secs 2024-07-26T12:17:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:31 smithi160 ceph-mon[36715]: pgmap v2845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4294.623308 secs 2024-07-26T12:17:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:31 smithi183 ceph-mon[37273]: pgmap v2845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:33 smithi144 ceph-mon[32129]: pgmap v2846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:33 smithi160 ceph-mon[36715]: pgmap v2846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:33 smithi183 ceph-mon[37273]: pgmap v2846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:35 smithi144 ceph-mon[32129]: pgmap v2847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:35 smithi160 ceph-mon[36715]: pgmap v2847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:35 smithi183 ceph-mon[37273]: pgmap v2847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4299.623408 secs 2024-07-26T12:17:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4299.623408 secs 2024-07-26T12:17:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4299.623408 secs 2024-07-26T12:17:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:37 smithi144 ceph-mon[32129]: pgmap v2848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:37 smithi160 ceph-mon[36715]: pgmap v2848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:37 smithi183 ceph-mon[37273]: pgmap v2848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:39 smithi144 ceph-mon[32129]: pgmap v2849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:39 smithi160 ceph-mon[36715]: pgmap v2849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:39 smithi183 ceph-mon[37273]: pgmap v2849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4304.623550 secs 2024-07-26T12:17:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:41 smithi144 ceph-mon[32129]: pgmap v2850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4304.623550 secs 2024-07-26T12:17:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:41 smithi160 ceph-mon[36715]: pgmap v2850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4304.623550 secs 2024-07-26T12:17:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:41 smithi183 ceph-mon[37273]: pgmap v2850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:43 smithi144 ceph-mon[32129]: pgmap v2851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:43 smithi160 ceph-mon[36715]: pgmap v2851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:43 smithi183 ceph-mon[37273]: pgmap v2851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:45.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:45.569+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:45.569+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:45.569+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:45.569+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:45 smithi144 ceph-mon[32129]: pgmap v2852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:45 smithi160 ceph-mon[36715]: pgmap v2852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:45 smithi183 ceph-mon[37273]: pgmap v2852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:46.858+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:17:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:46.858+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006290 con 0x563c6c4182f0 2024-07-26T12:17:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:46.859+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006290 con 0x7f74b0109990 2024-07-26T12:17:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:17:46.859+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:17:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4309.623697 secs 2024-07-26T12:17:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4309.623697 secs 2024-07-26T12:17:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4309.623697 secs 2024-07-26T12:17:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:47 smithi160 ceph-mon[36715]: pgmap v2853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:47 smithi183 ceph-mon[37273]: pgmap v2853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:47 smithi144 ceph-mon[32129]: pgmap v2853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:17:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: pgmap v2854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:50.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:17:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: pgmap v2854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: pgmap v2854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:17:50.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:50.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:50.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:17:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:50 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:17:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:50 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:17:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:50 smithi160 ceph-mon[36715]: pgmap v2855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:17:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:50 smithi160 ceph-mon[36715]: pgmap v2856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:17:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:50 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:17:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:50 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:17:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:50 smithi183 ceph-mon[37273]: pgmap v2855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:17:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:50 smithi183 ceph-mon[37273]: pgmap v2856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:17:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:50 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:17:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:50 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:17:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:50 smithi144 ceph-mon[32129]: pgmap v2855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:17:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:50 smithi144 ceph-mon[32129]: pgmap v2856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:17:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4314.623839 secs 2024-07-26T12:17:52.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4314.623839 secs 2024-07-26T12:17:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4314.623839 secs 2024-07-26T12:17:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:53 smithi160 ceph-mon[36715]: pgmap v2857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:53 smithi183 ceph-mon[37273]: pgmap v2857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:53 smithi144 ceph-mon[32129]: pgmap v2857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:55 smithi160 ceph-mon[36715]: pgmap v2858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:17:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:55 smithi183 ceph-mon[37273]: pgmap v2858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:17:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:55 smithi144 ceph-mon[32129]: pgmap v2858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:17:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4319.623957 secs 2024-07-26T12:17:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:57 smithi160 ceph-mon[36715]: pgmap v2859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4319.623957 secs 2024-07-26T12:17:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:57 smithi183 ceph-mon[37273]: pgmap v2859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4319.623957 secs 2024-07-26T12:17:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:57 smithi144 ceph-mon[32129]: pgmap v2859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:17:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:17:59 smithi160 ceph-mon[36715]: pgmap v2860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:17:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:17:59 smithi183 ceph-mon[37273]: pgmap v2860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:17:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:17:59 smithi144 ceph-mon[32129]: pgmap v2860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:00.585+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:00.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:00.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:00.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:00.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:00.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:01 smithi160 ceph-mon[36715]: pgmap v2861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:18:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:01 smithi183 ceph-mon[37273]: pgmap v2861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:18:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:01 smithi144 ceph-mon[32129]: pgmap v2861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:18:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4324.624082 secs 2024-07-26T12:18:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4324.624082 secs 2024-07-26T12:18:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4324.624082 secs 2024-07-26T12:18:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:03 smithi160 ceph-mon[36715]: pgmap v2862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:03 smithi183 ceph-mon[37273]: pgmap v2862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:03 smithi144 ceph-mon[32129]: pgmap v2862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:05 smithi160 ceph-mon[36715]: pgmap v2863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:05 smithi183 ceph-mon[37273]: pgmap v2863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:05 smithi144 ceph-mon[32129]: pgmap v2863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4329.624212 secs 2024-07-26T12:18:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:07 smithi144 ceph-mon[32129]: pgmap v2864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4329.624212 secs 2024-07-26T12:18:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:07 smithi183 ceph-mon[37273]: pgmap v2864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4329.624212 secs 2024-07-26T12:18:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:07 smithi160 ceph-mon[36715]: pgmap v2864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:09 smithi183 ceph-mon[37273]: pgmap v2865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:09 smithi144 ceph-mon[32129]: pgmap v2865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:09 smithi160 ceph-mon[36715]: pgmap v2865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:11 smithi183 ceph-mon[37273]: pgmap v2866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:11 smithi144 ceph-mon[32129]: pgmap v2866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:11 smithi160 ceph-mon[36715]: pgmap v2866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4334.624330 secs 2024-07-26T12:18:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4334.624330 secs 2024-07-26T12:18:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4334.624330 secs 2024-07-26T12:18:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:13 smithi183 ceph-mon[37273]: pgmap v2867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:13 smithi144 ceph-mon[32129]: pgmap v2867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:13 smithi160 ceph-mon[36715]: pgmap v2867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:15 smithi183 ceph-mon[37273]: pgmap v2868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:15 smithi144 ceph-mon[32129]: pgmap v2868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:15 smithi160 ceph-mon[36715]: pgmap v2868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:15.602+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:15.603+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:15.603+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:15.603+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:15.603+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4339.624451 secs 2024-07-26T12:18:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:17 smithi183 ceph-mon[37273]: pgmap v2869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4339.624451 secs 2024-07-26T12:18:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:17 smithi144 ceph-mon[32129]: pgmap v2869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4339.624451 secs 2024-07-26T12:18:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:17 smithi160 ceph-mon[36715]: pgmap v2869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:19 smithi183 ceph-mon[37273]: pgmap v2870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:19 smithi144 ceph-mon[32129]: pgmap v2870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:19 smithi160 ceph-mon[36715]: pgmap v2870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:21 smithi183 ceph-mon[37273]: pgmap v2871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:21 smithi144 ceph-mon[32129]: pgmap v2871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:21 smithi160 ceph-mon[36715]: pgmap v2871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:22.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4344.624571 secs 2024-07-26T12:18:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4344.624571 secs 2024-07-26T12:18:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4344.624571 secs 2024-07-26T12:18:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:23 smithi183 ceph-mon[37273]: pgmap v2872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:23 smithi144 ceph-mon[32129]: pgmap v2872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:23 smithi160 ceph-mon[36715]: pgmap v2872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:25 smithi183 ceph-mon[37273]: pgmap v2873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:25 smithi144 ceph-mon[32129]: pgmap v2873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:25 smithi160 ceph-mon[36715]: pgmap v2873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4349.624658 secs 2024-07-26T12:18:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:27 smithi144 ceph-mon[32129]: pgmap v2874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4349.624658 secs 2024-07-26T12:18:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:27 smithi160 ceph-mon[36715]: pgmap v2874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4349.624658 secs 2024-07-26T12:18:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:27 smithi183 ceph-mon[37273]: pgmap v2874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:28 smithi144 ceph-mon[32129]: pgmap v2875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:28 smithi160 ceph-mon[36715]: pgmap v2875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:28 smithi183 ceph-mon[37273]: pgmap v2875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:30.612+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:30.613+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:30.613+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:30.613+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:30.613+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:30 smithi183 ceph-mon[37273]: pgmap v2876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:30 smithi144 ceph-mon[32129]: pgmap v2876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:30 smithi160 ceph-mon[36715]: pgmap v2876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4354.624782 secs 2024-07-26T12:18:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4354.624782 secs 2024-07-26T12:18:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4354.624782 secs 2024-07-26T12:18:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:32 smithi144 ceph-mon[32129]: pgmap v2877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:32 smithi160 ceph-mon[36715]: pgmap v2877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:32 smithi183 ceph-mon[37273]: pgmap v2877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:34 smithi144 ceph-mon[32129]: pgmap v2878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:35.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:34 smithi160 ceph-mon[36715]: pgmap v2878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:34 smithi183 ceph-mon[37273]: pgmap v2878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4359.624892 secs 2024-07-26T12:18:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:36 smithi144 ceph-mon[32129]: pgmap v2879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4359.624892 secs 2024-07-26T12:18:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:36 smithi160 ceph-mon[36715]: pgmap v2879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4359.624892 secs 2024-07-26T12:18:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:36 smithi183 ceph-mon[37273]: pgmap v2879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:38 smithi144 ceph-mon[32129]: pgmap v2880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:38 smithi160 ceph-mon[36715]: pgmap v2880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:38 smithi183 ceph-mon[37273]: pgmap v2880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:40 smithi144 ceph-mon[32129]: pgmap v2881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:40 smithi160 ceph-mon[36715]: pgmap v2881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:40 smithi183 ceph-mon[37273]: pgmap v2881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4364.624943 secs 2024-07-26T12:18:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4364.624943 secs 2024-07-26T12:18:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4364.624943 secs 2024-07-26T12:18:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:42 smithi144 ceph-mon[32129]: pgmap v2882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:42 smithi160 ceph-mon[36715]: pgmap v2882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:42 smithi183 ceph-mon[37273]: pgmap v2882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:18:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:44 smithi144 ceph-mon[32129]: pgmap v2883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:44 smithi160 ceph-mon[36715]: pgmap v2883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:44 smithi183 ceph-mon[37273]: pgmap v2883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:45.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:45.629+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:45.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:45.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:45.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:45.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:45.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:45.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:45.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:18:45.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4369.625099 secs 2024-07-26T12:18:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:46 smithi144 ceph-mon[32129]: pgmap v2884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4369.625099 secs 2024-07-26T12:18:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:46 smithi160 ceph-mon[36715]: pgmap v2884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4369.625099 secs 2024-07-26T12:18:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:46 smithi183 ceph-mon[37273]: pgmap v2884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:49.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:48 smithi144 ceph-mon[32129]: pgmap v2885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:48 smithi160 ceph-mon[36715]: pgmap v2885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:48 smithi183 ceph-mon[37273]: pgmap v2885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: pgmap v2886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: pgmap v2886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: pgmap v2886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4374.625218 secs 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: pgmap v2887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: pgmap v2888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:18:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:18:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:18:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:18:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:18:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4374.625218 secs 2024-07-26T12:18:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: pgmap v2887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: pgmap v2888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:18:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:18:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4374.625218 secs 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: pgmap v2887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: pgmap v2888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:18:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:54 smithi183 ceph-mon[37273]: pgmap v2889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:54 smithi144 ceph-mon[32129]: pgmap v2889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:54 smithi160 ceph-mon[36715]: pgmap v2889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:56 smithi183 ceph-mon[37273]: pgmap v2890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:56 smithi144 ceph-mon[32129]: pgmap v2890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:56 smithi160 ceph-mon[36715]: pgmap v2890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:18:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4379.625351 secs 2024-07-26T12:18:57.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4379.625351 secs 2024-07-26T12:18:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4379.625351 secs 2024-07-26T12:18:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:18:58 smithi183 ceph-mon[37273]: pgmap v2891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:18:58 smithi144 ceph-mon[32129]: pgmap v2891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:18:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:18:58 smithi160 ceph-mon[36715]: pgmap v2891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:00 smithi183 ceph-mon[37273]: pgmap v2892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:19:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:00 smithi144 ceph-mon[32129]: pgmap v2892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:19:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:00 smithi160 ceph-mon[36715]: pgmap v2892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:19:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:00.646+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:00.647+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:00.647+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:00.647+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:00.647+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4384.625473 secs 2024-07-26T12:19:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:02 smithi183 ceph-mon[37273]: pgmap v2893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:19:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4384.625473 secs 2024-07-26T12:19:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:02 smithi144 ceph-mon[32129]: pgmap v2893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:19:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4384.625473 secs 2024-07-26T12:19:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:02 smithi160 ceph-mon[36715]: pgmap v2893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:19:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:04 smithi183 ceph-mon[37273]: pgmap v2894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:04 smithi144 ceph-mon[32129]: pgmap v2894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:04 smithi160 ceph-mon[36715]: pgmap v2894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:06 smithi183 ceph-mon[37273]: pgmap v2895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:06 smithi144 ceph-mon[32129]: pgmap v2895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:06 smithi160 ceph-mon[36715]: pgmap v2895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4389.625595 secs 2024-07-26T12:19:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4389.625595 secs 2024-07-26T12:19:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4389.625595 secs 2024-07-26T12:19:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:08 smithi144 ceph-mon[32129]: pgmap v2896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:08 smithi160 ceph-mon[36715]: pgmap v2896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:08 smithi183 ceph-mon[37273]: pgmap v2896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:10 smithi144 ceph-mon[32129]: pgmap v2897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:10 smithi160 ceph-mon[36715]: pgmap v2897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:10 smithi183 ceph-mon[37273]: pgmap v2897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:11 smithi144 ceph-mon[32129]: pgmap v2898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:11 smithi160 ceph-mon[36715]: pgmap v2898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:11 smithi183 ceph-mon[37273]: pgmap v2898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4394.625719 secs 2024-07-26T12:19:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4394.625719 secs 2024-07-26T12:19:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4394.625719 secs 2024-07-26T12:19:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:13 smithi160 ceph-mon[36715]: pgmap v2899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:13 smithi183 ceph-mon[37273]: pgmap v2899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:13 smithi144 ceph-mon[32129]: pgmap v2899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:15.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:15.664+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:15.664+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:15.664+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:15.664+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:16 smithi183 ceph-mon[37273]: pgmap v2900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:16 smithi144 ceph-mon[32129]: pgmap v2900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:16 smithi160 ceph-mon[36715]: pgmap v2900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4399.625840 secs 2024-07-26T12:19:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4399.625840 secs 2024-07-26T12:19:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4399.625840 secs 2024-07-26T12:19:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:18 smithi183 ceph-mon[37273]: pgmap v2901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:18 smithi144 ceph-mon[32129]: pgmap v2901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:18 smithi160 ceph-mon[36715]: pgmap v2901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:20 smithi183 ceph-mon[37273]: pgmap v2902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:20 smithi144 ceph-mon[32129]: pgmap v2902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:20 smithi160 ceph-mon[36715]: pgmap v2902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4404.625962 secs 2024-07-26T12:19:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:22 smithi183 ceph-mon[37273]: pgmap v2903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4404.625962 secs 2024-07-26T12:19:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:22 smithi144 ceph-mon[32129]: pgmap v2903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4404.625962 secs 2024-07-26T12:19:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:22 smithi160 ceph-mon[36715]: pgmap v2903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:23 smithi160 ceph-mon[36715]: pgmap v2904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:23 smithi183 ceph-mon[37273]: pgmap v2904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:23 smithi144 ceph-mon[32129]: pgmap v2904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:26 smithi183 ceph-mon[37273]: pgmap v2905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:26 smithi144 ceph-mon[32129]: pgmap v2905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:26 smithi160 ceph-mon[36715]: pgmap v2905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4409.626084 secs 2024-07-26T12:19:27.462 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4409.626084 secs 2024-07-26T12:19:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4409.626084 secs 2024-07-26T12:19:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:28 smithi183 ceph-mon[37273]: pgmap v2906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:28 smithi144 ceph-mon[32129]: pgmap v2906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:28 smithi160 ceph-mon[36715]: pgmap v2906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:30 smithi183 ceph-mon[37273]: pgmap v2907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:30 smithi144 ceph-mon[32129]: pgmap v2907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:30 smithi160 ceph-mon[36715]: pgmap v2907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:30.679+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:30.680+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:30.680+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:30.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:30.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4414.626224 secs 2024-07-26T12:19:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:32 smithi183 ceph-mon[37273]: pgmap v2908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4414.626224 secs 2024-07-26T12:19:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:32 smithi144 ceph-mon[32129]: pgmap v2908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4414.626224 secs 2024-07-26T12:19:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:32 smithi160 ceph-mon[36715]: pgmap v2908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:34 smithi183 ceph-mon[37273]: pgmap v2909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:34 smithi144 ceph-mon[32129]: pgmap v2909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:34 smithi160 ceph-mon[36715]: pgmap v2909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:36 smithi144 ceph-mon[32129]: pgmap v2910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:36 smithi160 ceph-mon[36715]: pgmap v2910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:36 smithi183 ceph-mon[37273]: pgmap v2910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4419.626372 secs 2024-07-26T12:19:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4419.626372 secs 2024-07-26T12:19:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4419.626372 secs 2024-07-26T12:19:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:38 smithi144 ceph-mon[32129]: pgmap v2911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:38 smithi160 ceph-mon[36715]: pgmap v2911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:38 smithi183 ceph-mon[37273]: pgmap v2911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:39 smithi144 ceph-mon[32129]: pgmap v2912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:39 smithi160 ceph-mon[36715]: pgmap v2912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:39 smithi183 ceph-mon[37273]: pgmap v2912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4424.626495 secs 2024-07-26T12:19:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:42 smithi183 ceph-mon[37273]: pgmap v2913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4424.626495 secs 2024-07-26T12:19:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:42 smithi144 ceph-mon[32129]: pgmap v2913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4424.626495 secs 2024-07-26T12:19:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:42 smithi160 ceph-mon[36715]: pgmap v2913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:44 smithi183 ceph-mon[37273]: pgmap v2914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:44 smithi144 ceph-mon[32129]: pgmap v2914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:44 smithi160 ceph-mon[36715]: pgmap v2914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:45.686 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:45.684+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:45.687 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:45.685+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:45.687 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:45.685+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:45.687 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:45.685+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:45.687 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:19:45.685+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:46 smithi183 ceph-mon[37273]: pgmap v2915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:46 smithi144 ceph-mon[32129]: pgmap v2915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:46 smithi160 ceph-mon[36715]: pgmap v2915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4429.626611 secs 2024-07-26T12:19:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4429.626611 secs 2024-07-26T12:19:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4429.626611 secs 2024-07-26T12:19:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:48 smithi144 ceph-mon[32129]: pgmap v2916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:48 smithi160 ceph-mon[36715]: pgmap v2916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:48 smithi183 ceph-mon[37273]: pgmap v2916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:19:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:50 smithi144 ceph-mon[32129]: pgmap v2917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:50 smithi160 ceph-mon[36715]: pgmap v2917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:50 smithi183 ceph-mon[37273]: pgmap v2917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:51 smithi144 ceph-mon[32129]: pgmap v2918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:51 smithi160 ceph-mon[36715]: pgmap v2918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:51 smithi183 ceph-mon[37273]: pgmap v2918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:19:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4434.626735 secs 2024-07-26T12:19:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4434.626735 secs 2024-07-26T12:19:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4434.626735 secs 2024-07-26T12:19:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:53 smithi144 ceph-mon[32129]: pgmap v2919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:19:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:53 smithi144 ceph-mon[32129]: pgmap v2920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:19:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:53 smithi160 ceph-mon[36715]: pgmap v2919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:19:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:53 smithi160 ceph-mon[36715]: pgmap v2920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:19:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:53 smithi183 ceph-mon[37273]: pgmap v2919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:19:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:53 smithi183 ceph-mon[37273]: pgmap v2920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:19:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:55 smithi160 ceph-mon[36715]: pgmap v2921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:19:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:55 smithi183 ceph-mon[37273]: pgmap v2921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:19:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:55 smithi144 ceph-mon[32129]: pgmap v2921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:19:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4439.626818 secs 2024-07-26T12:19:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:57 smithi160 ceph-mon[36715]: pgmap v2922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4439.626818 secs 2024-07-26T12:19:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:57 smithi183 ceph-mon[37273]: pgmap v2922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4439.626818 secs 2024-07-26T12:19:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:57 smithi144 ceph-mon[32129]: pgmap v2922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:19:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:19:59 smithi160 ceph-mon[36715]: pgmap v2923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:19:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:19:59 smithi183 ceph-mon[37273]: pgmap v2923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:19:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:19:59 smithi144 ceph-mon[32129]: pgmap v2923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:20:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:00.695+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:00.696+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:00.696+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:00.696+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:00.696+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:01 smithi160 ceph-mon[36715]: pgmap v2924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:20:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4444.626899 secs 2024-07-26T12:20:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:01 smithi183 ceph-mon[37273]: pgmap v2924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:20:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4444.626899 secs 2024-07-26T12:20:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:01 smithi144 ceph-mon[32129]: pgmap v2924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:20:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4444.626899 secs 2024-07-26T12:20:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:01.883+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:20:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:01.883+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006290 con 0x563c6c4182f0 2024-07-26T12:20:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:01.883+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006290 con 0x7f74b0109990 2024-07-26T12:20:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:01.884+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:20:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:03 smithi160 ceph-mon[36715]: pgmap v2925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:03 smithi183 ceph-mon[37273]: pgmap v2925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:03 smithi144 ceph-mon[32129]: pgmap v2925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:05 smithi160 ceph-mon[36715]: pgmap v2926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:05 smithi183 ceph-mon[37273]: pgmap v2926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:05 smithi144 ceph-mon[32129]: pgmap v2926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4449.627037 secs 2024-07-26T12:20:06.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4449.627037 secs 2024-07-26T12:20:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4449.627037 secs 2024-07-26T12:20:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:07 smithi160 ceph-mon[36715]: pgmap v2927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:07 smithi183 ceph-mon[37273]: pgmap v2927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:07 smithi144 ceph-mon[32129]: pgmap v2927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:09 smithi183 ceph-mon[37273]: pgmap v2928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:09 smithi144 ceph-mon[32129]: pgmap v2928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:09 smithi160 ceph-mon[36715]: pgmap v2928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:11 smithi183 ceph-mon[37273]: pgmap v2929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4454.627189 secs 2024-07-26T12:20:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:11 smithi144 ceph-mon[32129]: pgmap v2929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4454.627189 secs 2024-07-26T12:20:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:11 smithi160 ceph-mon[36715]: pgmap v2929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4454.627189 secs 2024-07-26T12:20:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:13 smithi183 ceph-mon[37273]: pgmap v2930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:13 smithi144 ceph-mon[32129]: pgmap v2930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:13 smithi160 ceph-mon[36715]: pgmap v2930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:15 smithi183 ceph-mon[37273]: pgmap v2931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:15.712+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:15.713+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:15.713+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:15.713+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:15.713+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:15 smithi144 ceph-mon[32129]: pgmap v2931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:15 smithi160 ceph-mon[36715]: pgmap v2931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4459.627332 secs 2024-07-26T12:20:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4459.627332 secs 2024-07-26T12:20:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4459.627332 secs 2024-07-26T12:20:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:17 smithi183 ceph-mon[37273]: pgmap v2932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:17 smithi144 ceph-mon[32129]: pgmap v2932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:17 smithi160 ceph-mon[36715]: pgmap v2932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:19 smithi183 ceph-mon[37273]: pgmap v2933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:19 smithi144 ceph-mon[32129]: pgmap v2933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:19 smithi160 ceph-mon[36715]: pgmap v2933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:21 smithi183 ceph-mon[37273]: pgmap v2934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4464.627395 secs 2024-07-26T12:20:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:21 smithi144 ceph-mon[32129]: pgmap v2934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4464.627395 secs 2024-07-26T12:20:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:21 smithi160 ceph-mon[36715]: pgmap v2934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4464.627395 secs 2024-07-26T12:20:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:23 smithi160 ceph-mon[36715]: pgmap v2935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:23 smithi183 ceph-mon[37273]: pgmap v2935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:23 smithi144 ceph-mon[32129]: pgmap v2935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:25 smithi160 ceph-mon[36715]: pgmap v2936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:25 smithi183 ceph-mon[37273]: pgmap v2936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:25 smithi144 ceph-mon[32129]: pgmap v2936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4469.627488 secs 2024-07-26T12:20:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4469.627488 secs 2024-07-26T12:20:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4469.627488 secs 2024-07-26T12:20:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:27 smithi183 ceph-mon[37273]: pgmap v2937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:27.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:27 smithi160 ceph-mon[36715]: pgmap v2937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:27 smithi144 ceph-mon[32129]: pgmap v2937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:29 smithi183 ceph-mon[37273]: pgmap v2938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:29 smithi144 ceph-mon[32129]: pgmap v2938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:29 smithi160 ceph-mon[36715]: pgmap v2938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:30.724+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:30.725+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:30.725+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:30.725+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:30.725+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:31 smithi183 ceph-mon[37273]: pgmap v2939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4474.627612 secs 2024-07-26T12:20:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:31 smithi144 ceph-mon[32129]: pgmap v2939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4474.627612 secs 2024-07-26T12:20:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:31 smithi160 ceph-mon[36715]: pgmap v2939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4474.627612 secs 2024-07-26T12:20:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:33 smithi183 ceph-mon[37273]: pgmap v2940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:33 smithi144 ceph-mon[32129]: pgmap v2940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:33 smithi160 ceph-mon[36715]: pgmap v2940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:35 smithi183 ceph-mon[37273]: pgmap v2941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:35 smithi144 ceph-mon[32129]: pgmap v2941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:35 smithi160 ceph-mon[36715]: pgmap v2941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:36.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4479.627690 secs 2024-07-26T12:20:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4479.627690 secs 2024-07-26T12:20:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4479.627690 secs 2024-07-26T12:20:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:37 smithi183 ceph-mon[37273]: pgmap v2942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:37 smithi144 ceph-mon[32129]: pgmap v2942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:37 smithi160 ceph-mon[36715]: pgmap v2942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:39 smithi183 ceph-mon[37273]: pgmap v2943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:39 smithi144 ceph-mon[32129]: pgmap v2943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:39 smithi160 ceph-mon[36715]: pgmap v2943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:41 smithi183 ceph-mon[37273]: pgmap v2944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4484.627809 secs 2024-07-26T12:20:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:41 smithi144 ceph-mon[32129]: pgmap v2944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4484.627809 secs 2024-07-26T12:20:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:41 smithi160 ceph-mon[36715]: pgmap v2944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4484.627809 secs 2024-07-26T12:20:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:43 smithi183 ceph-mon[37273]: pgmap v2945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:43 smithi144 ceph-mon[32129]: pgmap v2945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:43 smithi160 ceph-mon[36715]: pgmap v2945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:45.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:45.732+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:45.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:45.733+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:45.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:45.733+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:45.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:45.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:45 smithi183 ceph-mon[37273]: pgmap v2946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:45 smithi144 ceph-mon[32129]: pgmap v2946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:45 smithi160 ceph-mon[36715]: pgmap v2946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 msgr2=0x7f74900c2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 msgr2=0x7f74900c2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 0x7f74900c2810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f74a8031470 tx=0x7f74a8054000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 0x7f74900c2810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f74a8031470 tx=0x7f74a8054000 comp rx=0 tx=0).stop 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 msgr2=0x7f74900c2810 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 msgr2=0x7f74900c2810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c1d90 0x7f74900c2810 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:20:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.104+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:20:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.105+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.107+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2d50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a805a040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:20:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.891+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:20:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.891+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:20:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.891+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:20:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:20:46.892+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:20:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4489.627935 secs 2024-07-26T12:20:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4489.627935 secs 2024-07-26T12:20:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4489.627935 secs 2024-07-26T12:20:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:47 smithi144 ceph-mon[32129]: pgmap v2947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:47 smithi160 ceph-mon[36715]: pgmap v2947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:47 smithi183 ceph-mon[37273]: pgmap v2947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:49 smithi144 ceph-mon[32129]: pgmap v2948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:49 smithi160 ceph-mon[36715]: pgmap v2948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:49 smithi183 ceph-mon[37273]: pgmap v2948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:51 smithi144 ceph-mon[32129]: pgmap v2949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4494.628027 secs 2024-07-26T12:20:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:51 smithi160 ceph-mon[36715]: pgmap v2949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4494.628027 secs 2024-07-26T12:20:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:51 smithi183 ceph-mon[37273]: pgmap v2949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4494.628027 secs 2024-07-26T12:20:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:54 smithi160 ceph-mon[36715]: pgmap v2950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:54 smithi183 ceph-mon[37273]: pgmap v2950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:54 smithi144 ceph-mon[32129]: pgmap v2950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:20:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:55 smithi160 ceph-mon[36715]: pgmap v2951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:55 smithi160 ceph-mon[36715]: pgmap v2952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:55 smithi183 ceph-mon[37273]: pgmap v2951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:55 smithi183 ceph-mon[37273]: pgmap v2952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:55 smithi144 ceph-mon[32129]: pgmap v2951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:20:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:55 smithi144 ceph-mon[32129]: pgmap v2952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:20:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4499.628155 secs 2024-07-26T12:20:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:57 smithi183 ceph-mon[37273]: pgmap v2953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:20:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4499.628155 secs 2024-07-26T12:20:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:57 smithi144 ceph-mon[32129]: pgmap v2953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:20:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4499.628155 secs 2024-07-26T12:20:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:57 smithi160 ceph-mon[36715]: pgmap v2953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:20:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:20:59 smithi183 ceph-mon[37273]: pgmap v2954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:20:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:20:59 smithi144 ceph-mon[32129]: pgmap v2954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:20:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:20:59 smithi160 ceph-mon[36715]: pgmap v2954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:00.741+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:00.742+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:00.742+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:00.744 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:00.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:00.744 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:00.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:01 smithi144 ceph-mon[32129]: pgmap v2955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:01 smithi160 ceph-mon[36715]: pgmap v2955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:01 smithi183 ceph-mon[37273]: pgmap v2955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4504.628256 secs 2024-07-26T12:21:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4504.628256 secs 2024-07-26T12:21:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4504.628256 secs 2024-07-26T12:21:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:03 smithi144 ceph-mon[32129]: pgmap v2956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:03.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:03 smithi183 ceph-mon[37273]: pgmap v2956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:03 smithi160 ceph-mon[36715]: pgmap v2956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:05 smithi144 ceph-mon[32129]: pgmap v2957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:05 smithi160 ceph-mon[36715]: pgmap v2957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:05 smithi183 ceph-mon[37273]: pgmap v2957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4509.628377 secs 2024-07-26T12:21:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:07 smithi144 ceph-mon[32129]: pgmap v2958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4509.628377 secs 2024-07-26T12:21:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:07 smithi160 ceph-mon[36715]: pgmap v2958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4509.628377 secs 2024-07-26T12:21:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:07 smithi183 ceph-mon[37273]: pgmap v2958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:09 smithi144 ceph-mon[32129]: pgmap v2959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:09 smithi160 ceph-mon[36715]: pgmap v2959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:09 smithi183 ceph-mon[37273]: pgmap v2959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:10 smithi144 ceph-mon[32129]: pgmap v2960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:10 smithi160 ceph-mon[36715]: pgmap v2960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:10 smithi183 ceph-mon[37273]: pgmap v2960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4514.628499 secs 2024-07-26T12:21:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4514.628499 secs 2024-07-26T12:21:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4514.628499 secs 2024-07-26T12:21:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:12 smithi183 ceph-mon[37273]: pgmap v2961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:12 smithi144 ceph-mon[32129]: pgmap v2961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:12 smithi160 ceph-mon[36715]: pgmap v2961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:15 smithi144 ceph-mon[32129]: pgmap v2962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:15 smithi160 ceph-mon[36715]: pgmap v2962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:15 smithi183 ceph-mon[37273]: pgmap v2962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:15.749+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:15.750+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:15.750+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:15.750+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:15.751+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4519.628622 secs 2024-07-26T12:21:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:17 smithi144 ceph-mon[32129]: pgmap v2963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4519.628622 secs 2024-07-26T12:21:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:17 smithi160 ceph-mon[36715]: pgmap v2963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4519.628622 secs 2024-07-26T12:21:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:17 smithi183 ceph-mon[37273]: pgmap v2963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:19 smithi144 ceph-mon[32129]: pgmap v2964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:19 smithi160 ceph-mon[36715]: pgmap v2964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:19 smithi183 ceph-mon[37273]: pgmap v2964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:21 smithi144 ceph-mon[32129]: pgmap v2965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:21 smithi160 ceph-mon[36715]: pgmap v2965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:21 smithi183 ceph-mon[37273]: pgmap v2965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4524.628764 secs 2024-07-26T12:21:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4524.628764 secs 2024-07-26T12:21:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4524.628764 secs 2024-07-26T12:21:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:23 smithi144 ceph-mon[32129]: pgmap v2966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:23 smithi160 ceph-mon[36715]: pgmap v2966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:23 smithi183 ceph-mon[37273]: pgmap v2966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:25 smithi144 ceph-mon[32129]: pgmap v2967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:25 smithi160 ceph-mon[36715]: pgmap v2967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:25 smithi183 ceph-mon[37273]: pgmap v2967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:26 smithi144 ceph-mon[32129]: pgmap v2968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:26 smithi160 ceph-mon[36715]: pgmap v2968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:26 smithi183 ceph-mon[37273]: pgmap v2968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:27.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4529.628921 secs 2024-07-26T12:21:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4529.628921 secs 2024-07-26T12:21:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4529.628921 secs 2024-07-26T12:21:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:28 smithi144 ceph-mon[32129]: pgmap v2969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:28 smithi160 ceph-mon[36715]: pgmap v2969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:28 smithi183 ceph-mon[37273]: pgmap v2969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:30.766+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:30.766+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:30.767+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:30.767+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:30.767+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:31 smithi144 ceph-mon[32129]: pgmap v2970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:31 smithi160 ceph-mon[36715]: pgmap v2970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:31 smithi183 ceph-mon[37273]: pgmap v2970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4534.629024 secs 2024-07-26T12:21:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4534.629024 secs 2024-07-26T12:21:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4534.629024 secs 2024-07-26T12:21:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:33 smithi144 ceph-mon[32129]: pgmap v2971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:33.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:33 smithi183 ceph-mon[37273]: pgmap v2971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:33 smithi160 ceph-mon[36715]: pgmap v2971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:35 smithi144 ceph-mon[32129]: pgmap v2972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:35 smithi160 ceph-mon[36715]: pgmap v2972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:35 smithi183 ceph-mon[37273]: pgmap v2972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4539.629128 secs 2024-07-26T12:21:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:37 smithi144 ceph-mon[32129]: pgmap v2973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4539.629128 secs 2024-07-26T12:21:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:37 smithi160 ceph-mon[36715]: pgmap v2973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4539.629128 secs 2024-07-26T12:21:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:37 smithi183 ceph-mon[37273]: pgmap v2973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:38 smithi144 ceph-mon[32129]: pgmap v2974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:38 smithi160 ceph-mon[36715]: pgmap v2974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:38 smithi183 ceph-mon[37273]: pgmap v2974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:41 smithi144 ceph-mon[32129]: pgmap v2975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:41 smithi160 ceph-mon[36715]: pgmap v2975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:41 smithi183 ceph-mon[37273]: pgmap v2975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4544.629260 secs 2024-07-26T12:21:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4544.629260 secs 2024-07-26T12:21:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4544.629260 secs 2024-07-26T12:21:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:43 smithi144 ceph-mon[32129]: pgmap v2976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:43 smithi160 ceph-mon[36715]: pgmap v2976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:43 smithi183 ceph-mon[37273]: pgmap v2976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:45 smithi144 ceph-mon[32129]: pgmap v2977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:45 smithi160 ceph-mon[36715]: pgmap v2977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:45 smithi183 ceph-mon[37273]: pgmap v2977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:45.783+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:45.785+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:45.785+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:45.785+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:21:45.785+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:46 smithi160 ceph-mon[36715]: pgmap v2978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:46 smithi183 ceph-mon[37273]: pgmap v2978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:46 smithi144 ceph-mon[32129]: pgmap v2978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4549.629382 secs 2024-07-26T12:21:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4549.629382 secs 2024-07-26T12:21:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4549.629382 secs 2024-07-26T12:21:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:48 smithi160 ceph-mon[36715]: pgmap v2979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:48 smithi183 ceph-mon[37273]: pgmap v2979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:48 smithi144 ceph-mon[32129]: pgmap v2979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:51 smithi144 ceph-mon[32129]: pgmap v2980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:51 smithi160 ceph-mon[36715]: pgmap v2980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:51 smithi183 ceph-mon[37273]: pgmap v2980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4554.629497 secs 2024-07-26T12:21:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4554.629497 secs 2024-07-26T12:21:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4554.629497 secs 2024-07-26T12:21:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:53 smithi144 ceph-mon[32129]: pgmap v2981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:53 smithi160 ceph-mon[36715]: pgmap v2981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:53 smithi183 ceph-mon[37273]: pgmap v2981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:55 smithi144 ceph-mon[32129]: pgmap v2982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.546 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:55 smithi183 ceph-mon[37273]: pgmap v2982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:55.546 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:55.546 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.546 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.546 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.546 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:55 smithi160 ceph-mon[36715]: pgmap v2982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:56 smithi144 ceph-mon[32129]: pgmap v2983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:56 smithi160 ceph-mon[36715]: pgmap v2983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:56 smithi183 ceph-mon[37273]: pgmap v2983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4559.629531 secs 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4559.629531 secs 2024-07-26T12:21:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4559.629531 secs 2024-07-26T12:21:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:58 smithi183 ceph-mon[37273]: pgmap v2984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:21:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:58 smithi144 ceph-mon[32129]: pgmap v2984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:21:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:58 smithi160 ceph-mon[36715]: pgmap v2984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:21:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:21:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:00.800+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:00.801+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:00.801+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:00.801+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:00.801+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: pgmap v2985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: pgmap v2986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: pgmap v2985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: pgmap v2986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: pgmap v2985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: pgmap v2986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:22:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4564.629621 secs 2024-07-26T12:22:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4564.629621 secs 2024-07-26T12:22:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4564.629621 secs 2024-07-26T12:22:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:02 smithi183 ceph-mon[37273]: pgmap v2987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:02 smithi144 ceph-mon[32129]: pgmap v2987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:02 smithi160 ceph-mon[36715]: pgmap v2987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:04 smithi183 ceph-mon[37273]: pgmap v2988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:04 smithi144 ceph-mon[32129]: pgmap v2988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:04 smithi160 ceph-mon[36715]: pgmap v2988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4569.629742 secs 2024-07-26T12:22:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:06 smithi144 ceph-mon[32129]: pgmap v2989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4569.629742 secs 2024-07-26T12:22:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:06 smithi160 ceph-mon[36715]: pgmap v2989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4569.629742 secs 2024-07-26T12:22:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:06 smithi183 ceph-mon[37273]: pgmap v2989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:08 smithi160 ceph-mon[36715]: pgmap v2990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:08 smithi144 ceph-mon[32129]: pgmap v2990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:08 smithi183 ceph-mon[37273]: pgmap v2990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:10 smithi144 ceph-mon[32129]: pgmap v2991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:10 smithi160 ceph-mon[36715]: pgmap v2991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:10 smithi183 ceph-mon[37273]: pgmap v2991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4574.629861 secs 2024-07-26T12:22:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4574.629861 secs 2024-07-26T12:22:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4574.629861 secs 2024-07-26T12:22:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:12 smithi144 ceph-mon[32129]: pgmap v2992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:22:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:12 smithi160 ceph-mon[36715]: pgmap v2992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:22:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:12 smithi183 ceph-mon[37273]: pgmap v2992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:22:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:14 smithi183 ceph-mon[37273]: pgmap v2993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:15.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:14 smithi144 ceph-mon[32129]: pgmap v2993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:14 smithi160 ceph-mon[36715]: pgmap v2993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:15.817+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:15.818+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:15.818+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:15.818+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:15.818+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:17.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4579.629978 secs 2024-07-26T12:22:17.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:16 smithi144 ceph-mon[32129]: pgmap v2994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4579.629978 secs 2024-07-26T12:22:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:16 smithi160 ceph-mon[36715]: pgmap v2994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4579.629978 secs 2024-07-26T12:22:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:16 smithi183 ceph-mon[37273]: pgmap v2994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:18 smithi144 ceph-mon[32129]: pgmap v2995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:18 smithi160 ceph-mon[36715]: pgmap v2995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:18 smithi183 ceph-mon[37273]: pgmap v2995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:20 smithi144 ceph-mon[32129]: pgmap v2996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:20 smithi160 ceph-mon[36715]: pgmap v2996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:20 smithi183 ceph-mon[37273]: pgmap v2996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4584.630113 secs 2024-07-26T12:22:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4584.630113 secs 2024-07-26T12:22:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4584.630113 secs 2024-07-26T12:22:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:22 smithi144 ceph-mon[32129]: pgmap v2997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:22 smithi160 ceph-mon[36715]: pgmap v2997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:22 smithi183 ceph-mon[37273]: pgmap v2997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:24 smithi144 ceph-mon[32129]: pgmap v2998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:24 smithi160 ceph-mon[36715]: pgmap v2998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:24 smithi183 ceph-mon[37273]: pgmap v2998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4589.630252 secs 2024-07-26T12:22:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:26 smithi144 ceph-mon[32129]: pgmap v2999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4589.630252 secs 2024-07-26T12:22:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:26 smithi160 ceph-mon[36715]: pgmap v2999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4589.630252 secs 2024-07-26T12:22:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:26 smithi183 ceph-mon[37273]: pgmap v2999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:28 smithi160 ceph-mon[36715]: pgmap v3000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:22:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:28 smithi183 ceph-mon[37273]: pgmap v3000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:22:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:28 smithi144 ceph-mon[32129]: pgmap v3000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:22:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:30.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:30.825+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:30.825+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:30.825+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:30.825+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:31.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:30 smithi144 ceph-mon[32129]: pgmap v3001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:30 smithi160 ceph-mon[36715]: pgmap v3001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:30 smithi183 ceph-mon[37273]: pgmap v3001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4594.630358 secs 2024-07-26T12:22:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4594.630358 secs 2024-07-26T12:22:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4594.630358 secs 2024-07-26T12:22:33.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:32 smithi183 ceph-mon[37273]: pgmap v3002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:32 smithi160 ceph-mon[36715]: pgmap v3002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:32 smithi144 ceph-mon[32129]: pgmap v3002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:34 smithi160 ceph-mon[36715]: pgmap v3003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:34 smithi183 ceph-mon[37273]: pgmap v3003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:34 smithi144 ceph-mon[32129]: pgmap v3003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:37.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4599.630425 secs 2024-07-26T12:22:37.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:37 smithi144 ceph-mon[32129]: pgmap v3004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4599.630425 secs 2024-07-26T12:22:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:37 smithi160 ceph-mon[36715]: pgmap v3004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4599.630425 secs 2024-07-26T12:22:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:37 smithi183 ceph-mon[37273]: pgmap v3004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:39 smithi160 ceph-mon[36715]: pgmap v3005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:39 smithi183 ceph-mon[37273]: pgmap v3005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:39 smithi144 ceph-mon[32129]: pgmap v3005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:41 smithi160 ceph-mon[36715]: pgmap v3006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:41 smithi183 ceph-mon[37273]: pgmap v3006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:41 smithi144 ceph-mon[32129]: pgmap v3006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4604.630487 secs 2024-07-26T12:22:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4604.630487 secs 2024-07-26T12:22:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4604.630487 secs 2024-07-26T12:22:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:43 smithi160 ceph-mon[36715]: pgmap v3007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:22:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:43 smithi183 ceph-mon[37273]: pgmap v3007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:22:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:43 smithi144 ceph-mon[32129]: pgmap v3007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:22:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:45 smithi160 ceph-mon[36715]: pgmap v3008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:45 smithi183 ceph-mon[37273]: pgmap v3008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:45 smithi144 ceph-mon[32129]: pgmap v3008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:45.842+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:45.843+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:45.843+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:45.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:45.843+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:45.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:22:45.843+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4609.630557 secs 2024-07-26T12:22:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:47 smithi144 ceph-mon[32129]: pgmap v3009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4609.630557 secs 2024-07-26T12:22:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:47 smithi160 ceph-mon[36715]: pgmap v3009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4609.630557 secs 2024-07-26T12:22:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:47 smithi183 ceph-mon[37273]: pgmap v3009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:49 smithi160 ceph-mon[36715]: pgmap v3010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:22:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:49 smithi183 ceph-mon[37273]: pgmap v3010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:22:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:49 smithi144 ceph-mon[32129]: pgmap v3010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:22:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:51 smithi160 ceph-mon[36715]: pgmap v3011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:51 smithi183 ceph-mon[37273]: pgmap v3011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:51 smithi144 ceph-mon[32129]: pgmap v3011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4614.630653 secs 2024-07-26T12:22:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4614.630653 secs 2024-07-26T12:22:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4614.630653 secs 2024-07-26T12:22:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:53 smithi183 ceph-mon[37273]: pgmap v3012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:53 smithi144 ceph-mon[32129]: pgmap v3012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:53 smithi160 ceph-mon[36715]: pgmap v3012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:55 smithi183 ceph-mon[37273]: pgmap v3013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:55 smithi144 ceph-mon[32129]: pgmap v3013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:55 smithi160 ceph-mon[36715]: pgmap v3013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:22:57.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4619.630786 secs 2024-07-26T12:22:57.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:57 smithi160 ceph-mon[36715]: pgmap v3014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4619.630786 secs 2024-07-26T12:22:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:57 smithi183 ceph-mon[37273]: pgmap v3014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4619.630786 secs 2024-07-26T12:22:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:57 smithi144 ceph-mon[32129]: pgmap v3014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:22:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:22:59 smithi183 ceph-mon[37273]: pgmap v3015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:22:59 smithi144 ceph-mon[32129]: pgmap v3015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:22:59 smithi160 ceph-mon[36715]: pgmap v3015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:00.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:00.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:00.859+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:00.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:00.859+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:00.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:00.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:00.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:00.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:01 smithi183 ceph-mon[37273]: pgmap v3016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:23:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:01 smithi144 ceph-mon[32129]: pgmap v3016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:23:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:01 smithi160 ceph-mon[36715]: pgmap v3016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:23:02.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4624.630895 secs 2024-07-26T12:23:02.486 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:02 smithi144 ceph-mon[32129]: pgmap v3017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:02.486 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:02 smithi144 ceph-mon[32129]: pgmap v3018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:02.487 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4624.630895 secs 2024-07-26T12:23:02.487 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:02 smithi183 ceph-mon[37273]: pgmap v3017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:02.487 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:02 smithi183 ceph-mon[37273]: pgmap v3018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4624.630895 secs 2024-07-26T12:23:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:02 smithi160 ceph-mon[36715]: pgmap v3017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:02 smithi160 ceph-mon[36715]: pgmap v3018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:04 smithi183 ceph-mon[37273]: pgmap v3019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:23:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:04 smithi144 ceph-mon[32129]: pgmap v3019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:23:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:04 smithi160 ceph-mon[36715]: pgmap v3019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:23:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:06 smithi183 ceph-mon[37273]: pgmap v3020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:23:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:06 smithi144 ceph-mon[32129]: pgmap v3020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:23:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:06 smithi160 ceph-mon[36715]: pgmap v3020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:23:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4629.631018 secs 2024-07-26T12:23:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4629.631018 secs 2024-07-26T12:23:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4629.631018 secs 2024-07-26T12:23:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:08 smithi183 ceph-mon[37273]: pgmap v3021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:08 smithi144 ceph-mon[32129]: pgmap v3021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:08 smithi160 ceph-mon[36715]: pgmap v3021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:23:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:10 smithi183 ceph-mon[37273]: pgmap v3022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:23:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:10 smithi144 ceph-mon[32129]: pgmap v3022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:23:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:10 smithi160 ceph-mon[36715]: pgmap v3022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:23:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4634.631175 secs 2024-07-26T12:23:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:12 smithi144 ceph-mon[32129]: pgmap v3023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:23:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4634.631175 secs 2024-07-26T12:23:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:12 smithi160 ceph-mon[36715]: pgmap v3023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:23:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4634.631175 secs 2024-07-26T12:23:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:12 smithi183 ceph-mon[37273]: pgmap v3023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:23:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:14 smithi144 ceph-mon[32129]: pgmap v3024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:14 smithi160 ceph-mon[36715]: pgmap v3024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:14 smithi183 ceph-mon[37273]: pgmap v3024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:15.863+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:15.863+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:15.863+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:15.863+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:15.864+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:16 smithi144 ceph-mon[32129]: pgmap v3025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:16 smithi160 ceph-mon[36715]: pgmap v3025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:16 smithi183 ceph-mon[37273]: pgmap v3025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4639.631319 secs 2024-07-26T12:23:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4639.631319 secs 2024-07-26T12:23:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4639.631319 secs 2024-07-26T12:23:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:18 smithi144 ceph-mon[32129]: pgmap v3026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:18 smithi160 ceph-mon[36715]: pgmap v3026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:18 smithi183 ceph-mon[37273]: pgmap v3026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:20 smithi144 ceph-mon[32129]: pgmap v3027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:20 smithi160 ceph-mon[36715]: pgmap v3027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:20 smithi183 ceph-mon[37273]: pgmap v3027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4644.631383 secs 2024-07-26T12:23:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:22 smithi144 ceph-mon[32129]: pgmap v3028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4644.631383 secs 2024-07-26T12:23:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:22 smithi160 ceph-mon[36715]: pgmap v3028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4644.631383 secs 2024-07-26T12:23:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:22 smithi183 ceph-mon[37273]: pgmap v3028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:23 smithi183 ceph-mon[37273]: pgmap v3029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:23 smithi144 ceph-mon[32129]: pgmap v3029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:23 smithi160 ceph-mon[36715]: pgmap v3029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:26 smithi183 ceph-mon[37273]: pgmap v3030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:26 smithi144 ceph-mon[32129]: pgmap v3030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:26 smithi160 ceph-mon[36715]: pgmap v3030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4649.631485 secs 2024-07-26T12:23:27.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4649.631485 secs 2024-07-26T12:23:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4649.631485 secs 2024-07-26T12:23:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:28 smithi183 ceph-mon[37273]: pgmap v3031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:28 smithi144 ceph-mon[32129]: pgmap v3031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:28 smithi160 ceph-mon[36715]: pgmap v3031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:30 smithi144 ceph-mon[32129]: pgmap v3032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:30 smithi160 ceph-mon[36715]: pgmap v3032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:30 smithi183 ceph-mon[37273]: pgmap v3032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:30.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:30.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:30.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:30.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:30.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4654.631605 secs 2024-07-26T12:23:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:32 smithi144 ceph-mon[32129]: pgmap v3033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4654.631605 secs 2024-07-26T12:23:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:32 smithi160 ceph-mon[36715]: pgmap v3033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4654.631605 secs 2024-07-26T12:23:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:32 smithi183 ceph-mon[37273]: pgmap v3033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:34.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:34 smithi183 ceph-mon[37273]: pgmap v3034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:34 smithi144 ceph-mon[32129]: pgmap v3034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:34 smithi160 ceph-mon[36715]: pgmap v3034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:36 smithi144 ceph-mon[32129]: pgmap v3035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:36 smithi160 ceph-mon[36715]: pgmap v3035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:36 smithi183 ceph-mon[37273]: pgmap v3035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4659.631729 secs 2024-07-26T12:23:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4659.631729 secs 2024-07-26T12:23:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4659.631729 secs 2024-07-26T12:23:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:38 smithi144 ceph-mon[32129]: pgmap v3036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:38 smithi160 ceph-mon[36715]: pgmap v3036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:38 smithi183 ceph-mon[37273]: pgmap v3036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:40 smithi144 ceph-mon[32129]: pgmap v3037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:40 smithi160 ceph-mon[36715]: pgmap v3037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:40 smithi183 ceph-mon[37273]: pgmap v3037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4664.631847 secs 2024-07-26T12:23:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:42 smithi144 ceph-mon[32129]: pgmap v3038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4664.631847 secs 2024-07-26T12:23:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:42 smithi160 ceph-mon[36715]: pgmap v3038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4664.631847 secs 2024-07-26T12:23:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:42 smithi183 ceph-mon[37273]: pgmap v3038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:44 smithi144 ceph-mon[32129]: pgmap v3039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:44 smithi160 ceph-mon[36715]: pgmap v3039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:44 smithi183 ceph-mon[37273]: pgmap v3039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:45.896+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:45.897+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:45.897+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:45.897+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:45.897+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:46 smithi144 ceph-mon[32129]: pgmap v3040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:46 smithi160 ceph-mon[36715]: pgmap v3040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:46 smithi183 ceph-mon[37273]: pgmap v3040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:46.923+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:23:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:46.923+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:23:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:46.924+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:23:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:23:46.924+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:23:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4669.631944 secs 2024-07-26T12:23:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4669.631944 secs 2024-07-26T12:23:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4669.631944 secs 2024-07-26T12:23:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:48 smithi144 ceph-mon[32129]: pgmap v3041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:48 smithi160 ceph-mon[36715]: pgmap v3041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:48 smithi183 ceph-mon[37273]: pgmap v3041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:49 smithi160 ceph-mon[36715]: pgmap v3042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:49 smithi183 ceph-mon[37273]: pgmap v3042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:49 smithi144 ceph-mon[32129]: pgmap v3042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4674.632096 secs 2024-07-26T12:23:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:52 smithi183 ceph-mon[37273]: pgmap v3043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4674.632096 secs 2024-07-26T12:23:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:52 smithi144 ceph-mon[32129]: pgmap v3043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4674.632096 secs 2024-07-26T12:23:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:52 smithi160 ceph-mon[36715]: pgmap v3043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:54 smithi144 ceph-mon[32129]: pgmap v3044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:54 smithi160 ceph-mon[36715]: pgmap v3044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:54 smithi183 ceph-mon[37273]: pgmap v3044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:56 smithi144 ceph-mon[32129]: pgmap v3045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:56 smithi160 ceph-mon[36715]: pgmap v3045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:56 smithi183 ceph-mon[37273]: pgmap v3045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:23:57.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4679.632219 secs 2024-07-26T12:23:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4679.632219 secs 2024-07-26T12:23:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4679.632219 secs 2024-07-26T12:23:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:23:58 smithi144 ceph-mon[32129]: pgmap v3046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:23:58 smithi160 ceph-mon[36715]: pgmap v3046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:23:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:23:58 smithi183 ceph-mon[37273]: pgmap v3046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:00 smithi144 ceph-mon[32129]: pgmap v3047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:00 smithi160 ceph-mon[36715]: pgmap v3047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:00 smithi183 ceph-mon[37273]: pgmap v3047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:00.913+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:00.914+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:00.914+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:00.914+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:00.914+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4684.632372 secs 2024-07-26T12:24:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:02 smithi144 ceph-mon[32129]: pgmap v3048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4684.632372 secs 2024-07-26T12:24:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:02 smithi160 ceph-mon[36715]: pgmap v3048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4684.632372 secs 2024-07-26T12:24:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:02 smithi183 ceph-mon[37273]: pgmap v3048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:24:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:24:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:24:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:04 smithi144 ceph-mon[32129]: pgmap v3049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:24:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:04 smithi144 ceph-mon[32129]: pgmap v3050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:04.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:04 smithi183 ceph-mon[37273]: pgmap v3049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:24:04.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:04 smithi183 ceph-mon[37273]: pgmap v3050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:04 smithi160 ceph-mon[36715]: pgmap v3049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:24:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:04 smithi160 ceph-mon[36715]: pgmap v3050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:06 smithi144 ceph-mon[32129]: pgmap v3051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:06 smithi160 ceph-mon[36715]: pgmap v3051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:06 smithi183 ceph-mon[37273]: pgmap v3051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4689.632451 secs 2024-07-26T12:24:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4689.632451 secs 2024-07-26T12:24:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4689.632451 secs 2024-07-26T12:24:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:08 smithi144 ceph-mon[32129]: pgmap v3052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:08 smithi160 ceph-mon[36715]: pgmap v3052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:08 smithi183 ceph-mon[37273]: pgmap v3052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:24:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:10 smithi144 ceph-mon[32129]: pgmap v3053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:24:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:10 smithi160 ceph-mon[36715]: pgmap v3053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:24:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:10 smithi183 ceph-mon[37273]: pgmap v3053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:24:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:11 smithi160 ceph-mon[36715]: pgmap v3054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:24:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:11 smithi183 ceph-mon[37273]: pgmap v3054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:24:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:11 smithi144 ceph-mon[32129]: pgmap v3054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:24:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4694.632523 secs 2024-07-26T12:24:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4694.632523 secs 2024-07-26T12:24:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4694.632523 secs 2024-07-26T12:24:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:13 smithi160 ceph-mon[36715]: pgmap v3055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:24:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:13 smithi183 ceph-mon[37273]: pgmap v3055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:24:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:13 smithi144 ceph-mon[32129]: pgmap v3055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:24:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:15 smithi160 ceph-mon[36715]: pgmap v3056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:15 smithi183 ceph-mon[37273]: pgmap v3056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:15 smithi144 ceph-mon[32129]: pgmap v3056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:15.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:15.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:15.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:15.931+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:15.931+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4699.632645 secs 2024-07-26T12:24:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:17 smithi183 ceph-mon[37273]: pgmap v3057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4699.632645 secs 2024-07-26T12:24:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:17 smithi144 ceph-mon[32129]: pgmap v3057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4699.632645 secs 2024-07-26T12:24:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:17 smithi160 ceph-mon[36715]: pgmap v3057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:19 smithi183 ceph-mon[37273]: pgmap v3058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:19.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:19 smithi144 ceph-mon[32129]: pgmap v3058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:19 smithi160 ceph-mon[36715]: pgmap v3058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:21 smithi183 ceph-mon[37273]: pgmap v3059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4704.632763 secs 2024-07-26T12:24:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:21 smithi144 ceph-mon[32129]: pgmap v3059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4704.632763 secs 2024-07-26T12:24:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:21 smithi160 ceph-mon[36715]: pgmap v3059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4704.632763 secs 2024-07-26T12:24:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:23 smithi183 ceph-mon[37273]: pgmap v3060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:23 smithi144 ceph-mon[32129]: pgmap v3060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:23 smithi160 ceph-mon[36715]: pgmap v3060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:25.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:25 smithi183 ceph-mon[37273]: pgmap v3061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:25 smithi144 ceph-mon[32129]: pgmap v3061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:25 smithi160 ceph-mon[36715]: pgmap v3061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4709.632885 secs 2024-07-26T12:24:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4709.632885 secs 2024-07-26T12:24:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4709.632885 secs 2024-07-26T12:24:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:27 smithi183 ceph-mon[37273]: pgmap v3062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:27.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:27 smithi160 ceph-mon[36715]: pgmap v3062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:27 smithi144 ceph-mon[32129]: pgmap v3062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:29 smithi183 ceph-mon[37273]: pgmap v3063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:29 smithi144 ceph-mon[32129]: pgmap v3063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:29 smithi160 ceph-mon[36715]: pgmap v3063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:30.946+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:30.946+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:30.946+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:30.946+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:30.946+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:31 smithi183 ceph-mon[37273]: pgmap v3064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4714.632974 secs 2024-07-26T12:24:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:31 smithi144 ceph-mon[32129]: pgmap v3064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4714.632974 secs 2024-07-26T12:24:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:31 smithi160 ceph-mon[36715]: pgmap v3064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4714.632974 secs 2024-07-26T12:24:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:33 smithi183 ceph-mon[37273]: pgmap v3065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:33 smithi144 ceph-mon[32129]: pgmap v3065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:33 smithi160 ceph-mon[36715]: pgmap v3065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:35 smithi183 ceph-mon[37273]: pgmap v3066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:35 smithi144 ceph-mon[32129]: pgmap v3066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:35 smithi160 ceph-mon[36715]: pgmap v3066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4719.633094 secs 2024-07-26T12:24:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4719.633094 secs 2024-07-26T12:24:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4719.633094 secs 2024-07-26T12:24:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:37 smithi144 ceph-mon[32129]: pgmap v3067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:37 smithi160 ceph-mon[36715]: pgmap v3067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:37 smithi183 ceph-mon[37273]: pgmap v3067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:39 smithi144 ceph-mon[32129]: pgmap v3068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:39 smithi160 ceph-mon[36715]: pgmap v3068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:39 smithi183 ceph-mon[37273]: pgmap v3068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:41 smithi144 ceph-mon[32129]: pgmap v3069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4724.633212 secs 2024-07-26T12:24:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:41 smithi160 ceph-mon[36715]: pgmap v3069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4724.633212 secs 2024-07-26T12:24:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:41 smithi183 ceph-mon[37273]: pgmap v3069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4724.633212 secs 2024-07-26T12:24:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:43 smithi144 ceph-mon[32129]: pgmap v3070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:43 smithi160 ceph-mon[36715]: pgmap v3070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:43 smithi183 ceph-mon[37273]: pgmap v3070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:45 smithi144 ceph-mon[32129]: pgmap v3071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:45 smithi160 ceph-mon[36715]: pgmap v3071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:45 smithi183 ceph-mon[37273]: pgmap v3071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:45.963+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:45.964+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:45.964+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:45.964+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:24:45.964+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4729.633339 secs 2024-07-26T12:24:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4729.633339 secs 2024-07-26T12:24:46.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4729.633339 secs 2024-07-26T12:24:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:47 smithi144 ceph-mon[32129]: pgmap v3072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:47 smithi160 ceph-mon[36715]: pgmap v3072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:47 smithi183 ceph-mon[37273]: pgmap v3072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:49 smithi144 ceph-mon[32129]: pgmap v3073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:49 smithi160 ceph-mon[36715]: pgmap v3073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:49 smithi183 ceph-mon[37273]: pgmap v3073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:51 smithi144 ceph-mon[32129]: pgmap v3074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4734.633452 secs 2024-07-26T12:24:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:51 smithi160 ceph-mon[36715]: pgmap v3074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4734.633452 secs 2024-07-26T12:24:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:51 smithi183 ceph-mon[37273]: pgmap v3074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4734.633452 secs 2024-07-26T12:24:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:53 smithi144 ceph-mon[32129]: pgmap v3075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:53 smithi160 ceph-mon[36715]: pgmap v3075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:53 smithi183 ceph-mon[37273]: pgmap v3075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:55.723 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:55 smithi144 ceph-mon[32129]: pgmap v3076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:55 smithi160 ceph-mon[36715]: pgmap v3076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:55 smithi183 ceph-mon[37273]: pgmap v3076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4739.633534 secs 2024-07-26T12:24:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4739.633534 secs 2024-07-26T12:24:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4739.633534 secs 2024-07-26T12:24:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:57 smithi144 ceph-mon[32129]: pgmap v3077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:57 smithi160 ceph-mon[36715]: pgmap v3077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:57 smithi183 ceph-mon[37273]: pgmap v3077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:24:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:24:59 smithi144 ceph-mon[32129]: pgmap v3078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:24:59 smithi160 ceph-mon[36715]: pgmap v3078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:24:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:24:59 smithi183 ceph-mon[37273]: pgmap v3078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:00.980+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:00.981+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:00.981+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:00.981+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:00.981+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:01 smithi160 ceph-mon[36715]: pgmap v3079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4744.633687 secs 2024-07-26T12:25:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:01 smithi183 ceph-mon[37273]: pgmap v3079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4744.633687 secs 2024-07-26T12:25:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:01 smithi144 ceph-mon[32129]: pgmap v3079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4744.633687 secs 2024-07-26T12:25:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:03 smithi160 ceph-mon[36715]: pgmap v3080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:03 smithi183 ceph-mon[37273]: pgmap v3080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:03 smithi144 ceph-mon[32129]: pgmap v3080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:04 smithi160 ceph-mon[36715]: pgmap v3081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:04 smithi160 ceph-mon[36715]: pgmap v3082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:25:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:25:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:04 smithi183 ceph-mon[37273]: pgmap v3081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:04 smithi183 ceph-mon[37273]: pgmap v3082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:25:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:25:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:04 smithi144 ceph-mon[32129]: pgmap v3081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:04 smithi144 ceph-mon[32129]: pgmap v3082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:25:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:25:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:06 smithi160 ceph-mon[36715]: pgmap v3083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4749.633819 secs 2024-07-26T12:25:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:06 smithi183 ceph-mon[37273]: pgmap v3083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4749.633819 secs 2024-07-26T12:25:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:06 smithi144 ceph-mon[32129]: pgmap v3083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4749.633819 secs 2024-07-26T12:25:08.776 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:08 smithi144 ceph-mon[32129]: pgmap v3084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:25:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:08 smithi160 ceph-mon[36715]: pgmap v3084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:25:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:08 smithi183 ceph-mon[37273]: pgmap v3084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:25:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:10 smithi160 ceph-mon[36715]: pgmap v3085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:10 smithi183 ceph-mon[37273]: pgmap v3085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:10 smithi144 ceph-mon[32129]: pgmap v3085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4754.633898 secs 2024-07-26T12:25:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4754.633898 secs 2024-07-26T12:25:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4754.633898 secs 2024-07-26T12:25:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:12 smithi160 ceph-mon[36715]: pgmap v3086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:12 smithi183 ceph-mon[37273]: pgmap v3086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:12 smithi144 ceph-mon[32129]: pgmap v3086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:25:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:14 smithi160 ceph-mon[36715]: pgmap v3087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:25:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:14 smithi183 ceph-mon[37273]: pgmap v3087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:25:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:14 smithi144 ceph-mon[32129]: pgmap v3087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:25:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:15.997+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:15.998+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:15.998+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:15.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:15.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:16 smithi160 ceph-mon[36715]: pgmap v3088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4759.633973 secs 2024-07-26T12:25:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:16 smithi183 ceph-mon[37273]: pgmap v3088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4759.633973 secs 2024-07-26T12:25:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:16 smithi144 ceph-mon[32129]: pgmap v3088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4759.633973 secs 2024-07-26T12:25:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:18 smithi160 ceph-mon[36715]: pgmap v3089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:18 smithi183 ceph-mon[37273]: pgmap v3089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:18 smithi144 ceph-mon[32129]: pgmap v3089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:20 smithi183 ceph-mon[37273]: pgmap v3090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:20 smithi144 ceph-mon[32129]: pgmap v3090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:20 smithi160 ceph-mon[36715]: pgmap v3090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4764.634095 secs 2024-07-26T12:25:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4764.634095 secs 2024-07-26T12:25:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4764.634095 secs 2024-07-26T12:25:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:22 smithi183 ceph-mon[37273]: pgmap v3091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:22 smithi144 ceph-mon[32129]: pgmap v3091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:22 smithi160 ceph-mon[36715]: pgmap v3091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:24 smithi183 ceph-mon[37273]: pgmap v3092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:24 smithi144 ceph-mon[32129]: pgmap v3092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:24 smithi160 ceph-mon[36715]: pgmap v3092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:26 smithi183 ceph-mon[37273]: pgmap v3093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4769.634225 secs 2024-07-26T12:25:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:26 smithi144 ceph-mon[32129]: pgmap v3093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4769.634225 secs 2024-07-26T12:25:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:26 smithi160 ceph-mon[36715]: pgmap v3093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4769.634225 secs 2024-07-26T12:25:28.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:28 smithi160 ceph-mon[36715]: pgmap v3094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:28 smithi183 ceph-mon[37273]: pgmap v3094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:28 smithi144 ceph-mon[32129]: pgmap v3094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:30 smithi183 ceph-mon[37273]: pgmap v3095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:30 smithi144 ceph-mon[32129]: pgmap v3095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:31.013+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:31.014+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:31.014+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:31.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:31.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:30 smithi160 ceph-mon[36715]: pgmap v3095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4774.634360 secs 2024-07-26T12:25:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4774.634360 secs 2024-07-26T12:25:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4774.634360 secs 2024-07-26T12:25:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:32 smithi183 ceph-mon[37273]: pgmap v3096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:32 smithi144 ceph-mon[32129]: pgmap v3096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:32 smithi160 ceph-mon[36715]: pgmap v3096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:34 smithi183 ceph-mon[37273]: pgmap v3097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:34 smithi144 ceph-mon[32129]: pgmap v3097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:34 smithi160 ceph-mon[36715]: pgmap v3097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:36 smithi183 ceph-mon[37273]: pgmap v3098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4779.634443 secs 2024-07-26T12:25:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:36 smithi144 ceph-mon[32129]: pgmap v3098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4779.634443 secs 2024-07-26T12:25:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:36 smithi160 ceph-mon[36715]: pgmap v3098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4779.634443 secs 2024-07-26T12:25:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:38 smithi183 ceph-mon[37273]: pgmap v3099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:38 smithi144 ceph-mon[32129]: pgmap v3099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:38 smithi160 ceph-mon[36715]: pgmap v3099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:40 smithi183 ceph-mon[37273]: pgmap v3100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:40 smithi144 ceph-mon[32129]: pgmap v3100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:40 smithi160 ceph-mon[36715]: pgmap v3100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4784.634562 secs 2024-07-26T12:25:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4784.634562 secs 2024-07-26T12:25:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4784.634562 secs 2024-07-26T12:25:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:42 smithi144 ceph-mon[32129]: pgmap v3101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:42 smithi160 ceph-mon[36715]: pgmap v3101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:42 smithi183 ceph-mon[37273]: pgmap v3101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:44 smithi144 ceph-mon[32129]: pgmap v3102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:44 smithi160 ceph-mon[36715]: pgmap v3102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:44 smithi183 ceph-mon[37273]: pgmap v3102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:46.031+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:46.032+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:46.032+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:46.032+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:25:46.032+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:46 smithi144 ceph-mon[32129]: pgmap v3103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4789.634701 secs 2024-07-26T12:25:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:46 smithi160 ceph-mon[36715]: pgmap v3103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4789.634701 secs 2024-07-26T12:25:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:46 smithi183 ceph-mon[37273]: pgmap v3103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4789.634701 secs 2024-07-26T12:25:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:48 smithi144 ceph-mon[32129]: pgmap v3104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:48 smithi160 ceph-mon[36715]: pgmap v3104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:48 smithi183 ceph-mon[37273]: pgmap v3104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:50 smithi144 ceph-mon[32129]: pgmap v3105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:50 smithi160 ceph-mon[36715]: pgmap v3105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:50 smithi183 ceph-mon[37273]: pgmap v3105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4794.634828 secs 2024-07-26T12:25:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4794.634828 secs 2024-07-26T12:25:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4794.634828 secs 2024-07-26T12:25:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:52 smithi144 ceph-mon[32129]: pgmap v3106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:52 smithi160 ceph-mon[36715]: pgmap v3106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:52 smithi183 ceph-mon[37273]: pgmap v3106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:25:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:54 smithi144 ceph-mon[32129]: pgmap v3107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:54 smithi160 ceph-mon[36715]: pgmap v3107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:54 smithi183 ceph-mon[37273]: pgmap v3107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:56 smithi144 ceph-mon[32129]: pgmap v3108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4799.634918 secs 2024-07-26T12:25:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:56 smithi160 ceph-mon[36715]: pgmap v3108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4799.634918 secs 2024-07-26T12:25:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:56 smithi183 ceph-mon[37273]: pgmap v3108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:25:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4799.634918 secs 2024-07-26T12:25:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:25:58 smithi144 ceph-mon[32129]: pgmap v3109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:25:58 smithi160 ceph-mon[36715]: pgmap v3109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:25:58 smithi183 ceph-mon[37273]: pgmap v3109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:00 smithi144 ceph-mon[32129]: pgmap v3110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:01.037+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:01.038+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:01.038+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:01.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:01.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:00 smithi160 ceph-mon[36715]: pgmap v3110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:00 smithi183 ceph-mon[37273]: pgmap v3110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4804.635012 secs 2024-07-26T12:26:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4804.635012 secs 2024-07-26T12:26:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4804.635012 secs 2024-07-26T12:26:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:02 smithi160 ceph-mon[36715]: pgmap v3111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:02 smithi183 ceph-mon[37273]: pgmap v3111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:02 smithi144 ceph-mon[32129]: pgmap v3111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:04 smithi160 ceph-mon[36715]: pgmap v3112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:26:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:04 smithi183 ceph-mon[37273]: pgmap v3112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:26:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:04 smithi144 ceph-mon[32129]: pgmap v3112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:26:06.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:05 smithi183 ceph-mon[37273]: pgmap v3113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:26:06.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:05 smithi183 ceph-mon[37273]: pgmap v3114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:05 smithi160 ceph-mon[36715]: pgmap v3113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:26:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:05 smithi160 ceph-mon[36715]: pgmap v3114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:05 smithi144 ceph-mon[32129]: pgmap v3113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:26:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:05 smithi144 ceph-mon[32129]: pgmap v3114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4809.635141 secs 2024-07-26T12:26:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4809.635141 secs 2024-07-26T12:26:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4809.635141 secs 2024-07-26T12:26:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:07 smithi160 ceph-mon[36715]: pgmap v3115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:26:08.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:07 smithi144 ceph-mon[32129]: pgmap v3115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:26:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:07 smithi183 ceph-mon[37273]: pgmap v3115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:26:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:09 smithi160 ceph-mon[36715]: pgmap v3116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:26:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:09 smithi183 ceph-mon[37273]: pgmap v3116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:26:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:09 smithi144 ceph-mon[32129]: pgmap v3116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:26:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4814.635277 secs 2024-07-26T12:26:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:11 smithi160 ceph-mon[36715]: pgmap v3117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:26:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4814.635277 secs 2024-07-26T12:26:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:11 smithi183 ceph-mon[37273]: pgmap v3117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:26:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4814.635277 secs 2024-07-26T12:26:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:11 smithi144 ceph-mon[32129]: pgmap v3117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:26:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:13 smithi160 ceph-mon[36715]: pgmap v3118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:13 smithi183 ceph-mon[37273]: pgmap v3118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:13 smithi144 ceph-mon[32129]: pgmap v3118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:16.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:16.055+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:16.055+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:16.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:16.055+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:16.057 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:16.055+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:15 smithi160 ceph-mon[36715]: pgmap v3119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:15 smithi183 ceph-mon[37273]: pgmap v3119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:15 smithi144 ceph-mon[32129]: pgmap v3119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4819.635374 secs 2024-07-26T12:26:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4819.635374 secs 2024-07-26T12:26:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4819.635374 secs 2024-07-26T12:26:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:17 smithi160 ceph-mon[36715]: pgmap v3120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:17 smithi183 ceph-mon[37273]: pgmap v3120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:17 smithi144 ceph-mon[32129]: pgmap v3120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:19 smithi160 ceph-mon[36715]: pgmap v3121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:19 smithi183 ceph-mon[37273]: pgmap v3121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:19 smithi144 ceph-mon[32129]: pgmap v3121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4824.635454 secs 2024-07-26T12:26:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:21 smithi160 ceph-mon[36715]: pgmap v3122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4824.635454 secs 2024-07-26T12:26:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:21 smithi183 ceph-mon[37273]: pgmap v3122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4824.635454 secs 2024-07-26T12:26:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:21 smithi144 ceph-mon[32129]: pgmap v3122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:23 smithi144 ceph-mon[32129]: pgmap v3123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:23 smithi160 ceph-mon[36715]: pgmap v3123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:23 smithi183 ceph-mon[37273]: pgmap v3123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:25 smithi160 ceph-mon[36715]: pgmap v3124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:25 smithi183 ceph-mon[37273]: pgmap v3124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:25 smithi144 ceph-mon[32129]: pgmap v3124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4829.635553 secs 2024-07-26T12:26:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4829.635553 secs 2024-07-26T12:26:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4829.635553 secs 2024-07-26T12:26:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:27 smithi160 ceph-mon[36715]: pgmap v3125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:27 smithi183 ceph-mon[37273]: pgmap v3125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:27 smithi144 ceph-mon[32129]: pgmap v3125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:29 smithi160 ceph-mon[36715]: pgmap v3126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:29 smithi183 ceph-mon[37273]: pgmap v3126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:29 smithi144 ceph-mon[32129]: pgmap v3126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:31.060+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:31.061+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:31.061+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:31.061+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:31.061+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4834.635661 secs 2024-07-26T12:26:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:31 smithi160 ceph-mon[36715]: pgmap v3127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4834.635661 secs 2024-07-26T12:26:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:31 smithi183 ceph-mon[37273]: pgmap v3127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4834.635661 secs 2024-07-26T12:26:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:31 smithi144 ceph-mon[32129]: pgmap v3127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:33 smithi160 ceph-mon[36715]: pgmap v3128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:33 smithi183 ceph-mon[37273]: pgmap v3128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:33 smithi144 ceph-mon[32129]: pgmap v3128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:35 smithi183 ceph-mon[37273]: pgmap v3129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:35 smithi144 ceph-mon[32129]: pgmap v3129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:35 smithi160 ceph-mon[36715]: pgmap v3129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4839.635774 secs 2024-07-26T12:26:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4839.635774 secs 2024-07-26T12:26:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4839.635774 secs 2024-07-26T12:26:38.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:37 smithi144 ceph-mon[32129]: pgmap v3130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:37 smithi183 ceph-mon[37273]: pgmap v3130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:37 smithi160 ceph-mon[36715]: pgmap v3130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:39 smithi183 ceph-mon[37273]: pgmap v3131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:39 smithi144 ceph-mon[32129]: pgmap v3131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:39 smithi160 ceph-mon[36715]: pgmap v3131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4844.635897 secs 2024-07-26T12:26:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:41 smithi183 ceph-mon[37273]: pgmap v3132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4844.635897 secs 2024-07-26T12:26:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:41 smithi144 ceph-mon[32129]: pgmap v3132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4844.635897 secs 2024-07-26T12:26:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:41 smithi160 ceph-mon[36715]: pgmap v3132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:44.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:43 smithi183 ceph-mon[37273]: pgmap v3133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:43 smithi144 ceph-mon[32129]: pgmap v3133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:43 smithi160 ceph-mon[36715]: pgmap v3133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.077+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:45 smithi183 ceph-mon[37273]: pgmap v3134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:45 smithi144 ceph-mon[32129]: pgmap v3134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:45 smithi160 ceph-mon[36715]: pgmap v3134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:26:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.956+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:26:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.956+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:26:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.957+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:26:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:26:46.957+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:26:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4849.635992 secs 2024-07-26T12:26:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4849.635992 secs 2024-07-26T12:26:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4849.635992 secs 2024-07-26T12:26:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:47 smithi183 ceph-mon[37273]: pgmap v3135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:47 smithi144 ceph-mon[32129]: pgmap v3135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:47 smithi160 ceph-mon[36715]: pgmap v3135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:26:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:49 smithi144 ceph-mon[32129]: pgmap v3136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:49 smithi160 ceph-mon[36715]: pgmap v3136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:49 smithi183 ceph-mon[37273]: pgmap v3136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4854.636089 secs 2024-07-26T12:26:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:51 smithi144 ceph-mon[32129]: pgmap v3137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4854.636089 secs 2024-07-26T12:26:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:51 smithi160 ceph-mon[36715]: pgmap v3137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4854.636089 secs 2024-07-26T12:26:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:51 smithi183 ceph-mon[37273]: pgmap v3137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:53 smithi144 ceph-mon[32129]: pgmap v3138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:53 smithi160 ceph-mon[36715]: pgmap v3138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:53 smithi183 ceph-mon[37273]: pgmap v3138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:26:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:55 smithi144 ceph-mon[32129]: pgmap v3139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:26:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:55 smithi160 ceph-mon[36715]: pgmap v3139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:26:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:55 smithi183 ceph-mon[37273]: pgmap v3139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:26:57.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4859.636203 secs 2024-07-26T12:26:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4859.636203 secs 2024-07-26T12:26:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4859.636203 secs 2024-07-26T12:26:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:57 smithi144 ceph-mon[32129]: pgmap v3140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:57 smithi160 ceph-mon[36715]: pgmap v3140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:57 smithi183 ceph-mon[37273]: pgmap v3140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:26:59 smithi144 ceph-mon[32129]: pgmap v3141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:26:59 smithi160 ceph-mon[36715]: pgmap v3141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:26:59 smithi183 ceph-mon[37273]: pgmap v3141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:01.094+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:01.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:01.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:01.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:01.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4864.636345 secs 2024-07-26T12:27:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:01 smithi144 ceph-mon[32129]: pgmap v3142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4864.636345 secs 2024-07-26T12:27:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:01 smithi160 ceph-mon[36715]: pgmap v3142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4864.636345 secs 2024-07-26T12:27:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:01 smithi183 ceph-mon[37273]: pgmap v3142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:03 smithi144 ceph-mon[32129]: pgmap v3143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:03 smithi160 ceph-mon[36715]: pgmap v3143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:03 smithi183 ceph-mon[37273]: pgmap v3143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:05.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:05 smithi144 ceph-mon[32129]: pgmap v3144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:27:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:05 smithi160 ceph-mon[36715]: pgmap v3144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:27:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:05 smithi183 ceph-mon[37273]: pgmap v3144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:27:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4869.636431 secs 2024-07-26T12:27:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:06 smithi144 ceph-mon[32129]: pgmap v3145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:06 smithi144 ceph-mon[32129]: pgmap v3146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:07.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4869.636431 secs 2024-07-26T12:27:07.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:06 smithi183 ceph-mon[37273]: pgmap v3145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:07.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:06 smithi183 ceph-mon[37273]: pgmap v3146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4869.636431 secs 2024-07-26T12:27:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:06 smithi160 ceph-mon[36715]: pgmap v3145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:06 smithi160 ceph-mon[36715]: pgmap v3146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:08 smithi144 ceph-mon[32129]: pgmap v3147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:27:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:08 smithi160 ceph-mon[36715]: pgmap v3147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:27:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:08 smithi183 ceph-mon[37273]: pgmap v3147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:27:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:10 smithi160 ceph-mon[36715]: pgmap v3148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:27:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:10 smithi183 ceph-mon[37273]: pgmap v3148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:27:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:10 smithi144 ceph-mon[32129]: pgmap v3148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:27:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4874.636550 secs 2024-07-26T12:27:12.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4874.636550 secs 2024-07-26T12:27:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4874.636550 secs 2024-07-26T12:27:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:12 smithi160 ceph-mon[36715]: pgmap v3149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:12 smithi183 ceph-mon[37273]: pgmap v3149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:12 smithi144 ceph-mon[32129]: pgmap v3149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:27:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:14 smithi160 ceph-mon[36715]: pgmap v3150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:27:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:14 smithi183 ceph-mon[37273]: pgmap v3150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:27:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:14 smithi144 ceph-mon[32129]: pgmap v3150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:27:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:16.111+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:16.111+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:16.112+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:16.112+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:16.112+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4879.636696 secs 2024-07-26T12:27:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:16 smithi160 ceph-mon[36715]: pgmap v3151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4879.636696 secs 2024-07-26T12:27:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:16 smithi183 ceph-mon[37273]: pgmap v3151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4879.636696 secs 2024-07-26T12:27:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:16 smithi144 ceph-mon[32129]: pgmap v3151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:19 smithi160 ceph-mon[36715]: pgmap v3152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:19 smithi183 ceph-mon[37273]: pgmap v3152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:19 smithi144 ceph-mon[32129]: pgmap v3152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:21 smithi160 ceph-mon[36715]: pgmap v3153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:21 smithi183 ceph-mon[37273]: pgmap v3153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:21 smithi144 ceph-mon[32129]: pgmap v3153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4884.636815 secs 2024-07-26T12:27:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4884.636815 secs 2024-07-26T12:27:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4884.636815 secs 2024-07-26T12:27:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:23 smithi160 ceph-mon[36715]: pgmap v3154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:23 smithi183 ceph-mon[37273]: pgmap v3154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:23 smithi144 ceph-mon[32129]: pgmap v3154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:25 smithi160 ceph-mon[36715]: pgmap v3155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:25 smithi183 ceph-mon[37273]: pgmap v3155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:25 smithi144 ceph-mon[32129]: pgmap v3155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4889.636936 secs 2024-07-26T12:27:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:27 smithi160 ceph-mon[36715]: pgmap v3156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4889.636936 secs 2024-07-26T12:27:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:27 smithi183 ceph-mon[37273]: pgmap v3156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4889.636936 secs 2024-07-26T12:27:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:27 smithi144 ceph-mon[32129]: pgmap v3156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:29 smithi160 ceph-mon[36715]: pgmap v3157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:29 smithi183 ceph-mon[37273]: pgmap v3157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:29 smithi144 ceph-mon[32129]: pgmap v3157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:31.127+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:31.127+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:31.127+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:31.127+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:31.128+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:31 smithi160 ceph-mon[36715]: pgmap v3158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:31 smithi183 ceph-mon[37273]: pgmap v3158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:31 smithi144 ceph-mon[32129]: pgmap v3158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4894.637058 secs 2024-07-26T12:27:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4894.637058 secs 2024-07-26T12:27:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4894.637058 secs 2024-07-26T12:27:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:33 smithi183 ceph-mon[37273]: pgmap v3159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:33 smithi144 ceph-mon[32129]: pgmap v3159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:33 smithi160 ceph-mon[36715]: pgmap v3159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:35 smithi183 ceph-mon[37273]: pgmap v3160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:35 smithi144 ceph-mon[32129]: pgmap v3160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:35 smithi160 ceph-mon[36715]: pgmap v3160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4899.637207 secs 2024-07-26T12:27:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:37 smithi183 ceph-mon[37273]: pgmap v3161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4899.637207 secs 2024-07-26T12:27:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:37 smithi144 ceph-mon[32129]: pgmap v3161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4899.637207 secs 2024-07-26T12:27:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:37 smithi160 ceph-mon[36715]: pgmap v3161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:39 smithi183 ceph-mon[37273]: pgmap v3162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:39 smithi144 ceph-mon[32129]: pgmap v3162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:39 smithi160 ceph-mon[36715]: pgmap v3162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:41 smithi183 ceph-mon[37273]: pgmap v3163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:41 smithi144 ceph-mon[32129]: pgmap v3163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:41 smithi160 ceph-mon[36715]: pgmap v3163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:42.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4904.637327 secs 2024-07-26T12:27:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4904.637327 secs 2024-07-26T12:27:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4904.637327 secs 2024-07-26T12:27:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:43 smithi144 ceph-mon[32129]: pgmap v3164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:43 smithi183 ceph-mon[37273]: pgmap v3164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:43 smithi160 ceph-mon[36715]: pgmap v3164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:45 smithi183 ceph-mon[37273]: pgmap v3165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:45 smithi144 ceph-mon[32129]: pgmap v3165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:45 smithi160 ceph-mon[36715]: pgmap v3165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:46.143+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:46.144+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:46.144+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:46.144+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:27:46.144+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4909.637449 secs 2024-07-26T12:27:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:47 smithi183 ceph-mon[37273]: pgmap v3166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4909.637449 secs 2024-07-26T12:27:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:47 smithi144 ceph-mon[32129]: pgmap v3166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4909.637449 secs 2024-07-26T12:27:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:47 smithi160 ceph-mon[36715]: pgmap v3166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:49 smithi183 ceph-mon[37273]: pgmap v3167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:49 smithi144 ceph-mon[32129]: pgmap v3167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:49 smithi160 ceph-mon[36715]: pgmap v3167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:51.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:51 smithi183 ceph-mon[37273]: pgmap v3168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:51 smithi144 ceph-mon[32129]: pgmap v3168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:51 smithi160 ceph-mon[36715]: pgmap v3168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4914.637532 secs 2024-07-26T12:27:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4914.637532 secs 2024-07-26T12:27:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4914.637532 secs 2024-07-26T12:27:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:53 smithi144 ceph-mon[32129]: pgmap v3169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:53 smithi160 ceph-mon[36715]: pgmap v3169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:53 smithi183 ceph-mon[37273]: pgmap v3169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:27:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:55 smithi144 ceph-mon[32129]: pgmap v3170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:55 smithi160 ceph-mon[36715]: pgmap v3170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:55 smithi183 ceph-mon[37273]: pgmap v3170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4919.637649 secs 2024-07-26T12:27:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:57 smithi144 ceph-mon[32129]: pgmap v3171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4919.637649 secs 2024-07-26T12:27:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:57 smithi160 ceph-mon[36715]: pgmap v3171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4919.637649 secs 2024-07-26T12:27:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:57 smithi183 ceph-mon[37273]: pgmap v3171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:27:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:27:59 smithi144 ceph-mon[32129]: pgmap v3172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:27:59 smithi160 ceph-mon[36715]: pgmap v3172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:27:59 smithi183 ceph-mon[37273]: pgmap v3172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:01.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:01.160+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:01.160+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:01.160+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:01.160+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:01 smithi144 ceph-mon[32129]: pgmap v3173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:01 smithi160 ceph-mon[36715]: pgmap v3173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:01 smithi183 ceph-mon[37273]: pgmap v3173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4924.637726 secs 2024-07-26T12:28:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4924.637726 secs 2024-07-26T12:28:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4924.637726 secs 2024-07-26T12:28:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:03 smithi144 ceph-mon[32129]: pgmap v3174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:03 smithi160 ceph-mon[36715]: pgmap v3174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:03 smithi183 ceph-mon[37273]: pgmap v3174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:05 smithi144 ceph-mon[32129]: pgmap v3175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:05 smithi160 ceph-mon[36715]: pgmap v3175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:05 smithi183 ceph-mon[37273]: pgmap v3175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4929.637847 secs 2024-07-26T12:28:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:07 smithi160 ceph-mon[36715]: pgmap v3176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4929.637847 secs 2024-07-26T12:28:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:07 smithi144 ceph-mon[32129]: pgmap v3176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:07.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4929.637847 secs 2024-07-26T12:28:07.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:07 smithi183 ceph-mon[37273]: pgmap v3176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:09.420 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:09 smithi144 ceph-mon[32129]: pgmap v3177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:09.428 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:09 smithi160 ceph-mon[36715]: pgmap v3177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:09 smithi183 ceph-mon[37273]: pgmap v3177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:11.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:11.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:11.149 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:10 smithi144 ceph-mon[32129]: pgmap v3178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:10 smithi160 ceph-mon[36715]: pgmap v3178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:10 smithi183 ceph-mon[37273]: pgmap v3178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:12.125 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4934.637918 secs 2024-07-26T12:28:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4934.637918 secs 2024-07-26T12:28:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4934.637918 secs 2024-07-26T12:28:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: pgmap v3179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:28:13.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:13.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:13.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: pgmap v3180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:28:13.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: pgmap v3181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:28:13.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: pgmap v3179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: pgmap v3180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: pgmap v3181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:28:13.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: pgmap v3179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:28:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:28:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: pgmap v3180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:28:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: pgmap v3181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:28:13.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:28:14.374 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:14 smithi144 ceph-mon[32129]: pgmap v3182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:28:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:14 smithi183 ceph-mon[37273]: pgmap v3182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:28:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:14 smithi160 ceph-mon[36715]: pgmap v3182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:28:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.177+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.177+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.177+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.177+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:16 smithi183 ceph-mon[37273]: pgmap v3183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:16 smithi144 ceph-mon[32129]: pgmap v3183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:16 smithi160 ceph-mon[36715]: pgmap v3183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.973+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:28:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.973+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:28:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.973+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:28:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:16.973+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:28:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4939.638034 secs 2024-07-26T12:28:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4939.638034 secs 2024-07-26T12:28:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4939.638034 secs 2024-07-26T12:28:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:18 smithi183 ceph-mon[37273]: pgmap v3184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:18 smithi144 ceph-mon[32129]: pgmap v3184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:18 smithi160 ceph-mon[36715]: pgmap v3184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:20 smithi183 ceph-mon[37273]: pgmap v3185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:20 smithi144 ceph-mon[32129]: pgmap v3185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:20 smithi160 ceph-mon[36715]: pgmap v3185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:28:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4944.638142 secs 2024-07-26T12:28:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4944.638142 secs 2024-07-26T12:28:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4944.638142 secs 2024-07-26T12:28:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:22 smithi144 ceph-mon[32129]: pgmap v3186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:28:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:22 smithi160 ceph-mon[36715]: pgmap v3186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:28:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:22 smithi183 ceph-mon[37273]: pgmap v3186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:28:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:24 smithi183 ceph-mon[37273]: pgmap v3187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:24 smithi144 ceph-mon[32129]: pgmap v3187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:24 smithi160 ceph-mon[36715]: pgmap v3187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:26 smithi183 ceph-mon[37273]: pgmap v3188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:26 smithi144 ceph-mon[32129]: pgmap v3188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:26 smithi160 ceph-mon[36715]: pgmap v3188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4949.638263 secs 2024-07-26T12:28:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4949.638263 secs 2024-07-26T12:28:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4949.638263 secs 2024-07-26T12:28:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:28 smithi183 ceph-mon[37273]: pgmap v3189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:28 smithi144 ceph-mon[32129]: pgmap v3189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:28 smithi160 ceph-mon[36715]: pgmap v3189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:30 smithi183 ceph-mon[37273]: pgmap v3190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:30 smithi144 ceph-mon[32129]: pgmap v3190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:30 smithi160 ceph-mon[36715]: pgmap v3190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:31.192+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:31.193+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:31.193+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:31.193+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:31.196 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:31.193+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:31.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4954.638381 secs 2024-07-26T12:28:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4954.638381 secs 2024-07-26T12:28:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4954.638381 secs 2024-07-26T12:28:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:32 smithi144 ceph-mon[32129]: pgmap v3191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:32 smithi160 ceph-mon[36715]: pgmap v3191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:32 smithi183 ceph-mon[37273]: pgmap v3191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:34 smithi183 ceph-mon[37273]: pgmap v3192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:34 smithi144 ceph-mon[32129]: pgmap v3192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:34 smithi160 ceph-mon[36715]: pgmap v3192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:36 smithi183 ceph-mon[37273]: pgmap v3193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:36 smithi144 ceph-mon[32129]: pgmap v3193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:36 smithi160 ceph-mon[36715]: pgmap v3193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4959.638502 secs 2024-07-26T12:28:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4959.638502 secs 2024-07-26T12:28:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4959.638502 secs 2024-07-26T12:28:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:38 smithi183 ceph-mon[37273]: pgmap v3194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:38 smithi144 ceph-mon[32129]: pgmap v3194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:38 smithi160 ceph-mon[36715]: pgmap v3194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:40 smithi183 ceph-mon[37273]: pgmap v3195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:40 smithi144 ceph-mon[32129]: pgmap v3195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:40 smithi160 ceph-mon[36715]: pgmap v3195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4964.638620 secs 2024-07-26T12:28:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4964.638620 secs 2024-07-26T12:28:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4964.638620 secs 2024-07-26T12:28:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:42 smithi144 ceph-mon[32129]: pgmap v3196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:42 smithi160 ceph-mon[36715]: pgmap v3196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:42 smithi183 ceph-mon[37273]: pgmap v3196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:44 smithi183 ceph-mon[37273]: pgmap v3197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:44 smithi144 ceph-mon[32129]: pgmap v3197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:44 smithi160 ceph-mon[36715]: pgmap v3197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:46.202+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:46.203+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:46.203+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:46.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:28:46.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:46 smithi183 ceph-mon[37273]: pgmap v3198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:46 smithi144 ceph-mon[32129]: pgmap v3198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:46 smithi160 ceph-mon[36715]: pgmap v3198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4969.638762 secs 2024-07-26T12:28:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4969.638762 secs 2024-07-26T12:28:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4969.638762 secs 2024-07-26T12:28:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:48 smithi183 ceph-mon[37273]: pgmap v3199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:48 smithi144 ceph-mon[32129]: pgmap v3199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:48 smithi160 ceph-mon[36715]: pgmap v3199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:50 smithi183 ceph-mon[37273]: pgmap v3200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:50 smithi144 ceph-mon[32129]: pgmap v3200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:50 smithi160 ceph-mon[36715]: pgmap v3200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4974.638905 secs 2024-07-26T12:28:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4974.638905 secs 2024-07-26T12:28:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4974.638905 secs 2024-07-26T12:28:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:52 smithi144 ceph-mon[32129]: pgmap v3201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:52 smithi160 ceph-mon[36715]: pgmap v3201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:52 smithi183 ceph-mon[37273]: pgmap v3201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:28:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:54 smithi183 ceph-mon[37273]: pgmap v3202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:54 smithi144 ceph-mon[32129]: pgmap v3202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:54 smithi160 ceph-mon[36715]: pgmap v3202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:56 smithi183 ceph-mon[37273]: pgmap v3203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:56 smithi144 ceph-mon[32129]: pgmap v3203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:56 smithi160 ceph-mon[36715]: pgmap v3203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:28:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4979.639029 secs 2024-07-26T12:28:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4979.639029 secs 2024-07-26T12:28:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4979.639029 secs 2024-07-26T12:28:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:28:58 smithi183 ceph-mon[37273]: pgmap v3204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:28:58 smithi144 ceph-mon[32129]: pgmap v3204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:28:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:28:58 smithi160 ceph-mon[36715]: pgmap v3204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:00 smithi144 ceph-mon[32129]: pgmap v3205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:00 smithi183 ceph-mon[37273]: pgmap v3205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:00 smithi160 ceph-mon[36715]: pgmap v3205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:01.204+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:01.205+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:01.205+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:01.205+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:01.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:01.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4984.639120 secs 2024-07-26T12:29:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4984.639120 secs 2024-07-26T12:29:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4984.639120 secs 2024-07-26T12:29:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:02 smithi144 ceph-mon[32129]: pgmap v3206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:02 smithi160 ceph-mon[36715]: pgmap v3206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:02 smithi183 ceph-mon[37273]: pgmap v3206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:04 smithi183 ceph-mon[37273]: pgmap v3207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:04 smithi144 ceph-mon[32129]: pgmap v3207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:04 smithi160 ceph-mon[36715]: pgmap v3207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:06 smithi183 ceph-mon[37273]: pgmap v3208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:06 smithi144 ceph-mon[32129]: pgmap v3208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:06 smithi160 ceph-mon[36715]: pgmap v3208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4989.639259 secs 2024-07-26T12:29:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4989.639259 secs 2024-07-26T12:29:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4989.639259 secs 2024-07-26T12:29:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:08 smithi183 ceph-mon[37273]: pgmap v3209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:08.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:08 smithi144 ceph-mon[32129]: pgmap v3209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:08 smithi160 ceph-mon[36715]: pgmap v3209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:10 smithi183 ceph-mon[37273]: pgmap v3210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:10 smithi144 ceph-mon[32129]: pgmap v3210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:10 smithi160 ceph-mon[36715]: pgmap v3210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4994.639366 secs 2024-07-26T12:29:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4994.639366 secs 2024-07-26T12:29:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4994.639366 secs 2024-07-26T12:29:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:12 smithi144 ceph-mon[32129]: pgmap v3211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:12 smithi160 ceph-mon[36715]: pgmap v3211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:12 smithi183 ceph-mon[37273]: pgmap v3211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.582 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.582 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:29:13.582 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:13 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:29:13.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:13 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:29:13.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:13 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:29:14.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:14 smithi183 ceph-mon[37273]: pgmap v3212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:14 smithi144 ceph-mon[32129]: pgmap v3212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:14 smithi160 ceph-mon[36715]: pgmap v3212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:16.216+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:16.217+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:16.217+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:16.217+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:16.217+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 4999.639489 secs 2024-07-26T12:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: pgmap v3213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: pgmap v3214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:29:17.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 4999.639489 secs 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: pgmap v3213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: pgmap v3214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:29:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 4999.639489 secs 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: pgmap v3213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: pgmap v3214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:29:17.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:29:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:19 smithi183 ceph-mon[37273]: pgmap v3215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:29:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:19 smithi144 ceph-mon[32129]: pgmap v3215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:29:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:19 smithi160 ceph-mon[36715]: pgmap v3215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:29:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:21 smithi183 ceph-mon[37273]: pgmap v3216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:29:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:21 smithi144 ceph-mon[32129]: pgmap v3216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:29:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:21 smithi160 ceph-mon[36715]: pgmap v3216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:29:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5004.639613 secs 2024-07-26T12:29:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5004.639613 secs 2024-07-26T12:29:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5004.639613 secs 2024-07-26T12:29:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:23 smithi183 ceph-mon[37273]: pgmap v3217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:29:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:23 smithi144 ceph-mon[32129]: pgmap v3217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:29:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:23 smithi160 ceph-mon[36715]: pgmap v3217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:29:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:25 smithi183 ceph-mon[37273]: pgmap v3218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:29:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:25 smithi144 ceph-mon[32129]: pgmap v3218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:29:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:25 smithi160 ceph-mon[36715]: pgmap v3218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:29:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5009.639699 secs 2024-07-26T12:29:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:27 smithi183 ceph-mon[37273]: pgmap v3219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:29:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5009.639699 secs 2024-07-26T12:29:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:27 smithi144 ceph-mon[32129]: pgmap v3219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:29:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5009.639699 secs 2024-07-26T12:29:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:27 smithi160 ceph-mon[36715]: pgmap v3219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:29:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:29 smithi183 ceph-mon[37273]: pgmap v3220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:29 smithi144 ceph-mon[32129]: pgmap v3220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:29 smithi160 ceph-mon[36715]: pgmap v3220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:31.234 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:31.232+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:31.235 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:31.233+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:31.235 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:31.233+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:31.235 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:31.233+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:31.235 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:31.233+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:31 smithi183 ceph-mon[37273]: pgmap v3221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:31 smithi144 ceph-mon[32129]: pgmap v3221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:31 smithi160 ceph-mon[36715]: pgmap v3221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5014.639822 secs 2024-07-26T12:29:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5014.639822 secs 2024-07-26T12:29:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5014.639822 secs 2024-07-26T12:29:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:33 smithi144 ceph-mon[32129]: pgmap v3222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:33 smithi160 ceph-mon[36715]: pgmap v3222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:33 smithi183 ceph-mon[37273]: pgmap v3222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:35 smithi144 ceph-mon[32129]: pgmap v3223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:35 smithi160 ceph-mon[36715]: pgmap v3223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:35 smithi183 ceph-mon[37273]: pgmap v3223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:37.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5019.639974 secs 2024-07-26T12:29:37.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:37 smithi183 ceph-mon[37273]: pgmap v3224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5019.639974 secs 2024-07-26T12:29:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:37 smithi144 ceph-mon[32129]: pgmap v3224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5019.639974 secs 2024-07-26T12:29:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:37 smithi160 ceph-mon[36715]: pgmap v3224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:39 smithi144 ceph-mon[32129]: pgmap v3225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:39 smithi160 ceph-mon[36715]: pgmap v3225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:39 smithi183 ceph-mon[37273]: pgmap v3225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:41 smithi144 ceph-mon[32129]: pgmap v3226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:41 smithi160 ceph-mon[36715]: pgmap v3226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:41 smithi183 ceph-mon[37273]: pgmap v3226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5024.640068 secs 2024-07-26T12:29:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5024.640068 secs 2024-07-26T12:29:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5024.640068 secs 2024-07-26T12:29:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:43 smithi144 ceph-mon[32129]: pgmap v3227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:43 smithi160 ceph-mon[36715]: pgmap v3227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:43 smithi183 ceph-mon[37273]: pgmap v3227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:45 smithi144 ceph-mon[32129]: pgmap v3228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:45 smithi160 ceph-mon[36715]: pgmap v3228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:45 smithi183 ceph-mon[37273]: pgmap v3228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.249+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.250+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:46.252 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.250+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:46.252 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.250+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:46.252 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.250+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:46.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.988+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:29:46.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.988+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:29:46.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.989+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:29:46.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:29:46.989+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:29:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5029.640213 secs 2024-07-26T12:29:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:47 smithi144 ceph-mon[32129]: pgmap v3229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5029.640213 secs 2024-07-26T12:29:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:47 smithi160 ceph-mon[36715]: pgmap v3229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5029.640213 secs 2024-07-26T12:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:47 smithi183 ceph-mon[37273]: pgmap v3229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:49 smithi144 ceph-mon[32129]: pgmap v3230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:49 smithi160 ceph-mon[36715]: pgmap v3230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:49 smithi183 ceph-mon[37273]: pgmap v3230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:51 smithi144 ceph-mon[32129]: pgmap v3231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:51 smithi160 ceph-mon[36715]: pgmap v3231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:51 smithi183 ceph-mon[37273]: pgmap v3231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5034.640343 secs 2024-07-26T12:29:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5034.640343 secs 2024-07-26T12:29:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5034.640343 secs 2024-07-26T12:29:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:53 smithi160 ceph-mon[36715]: pgmap v3232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:53 smithi183 ceph-mon[37273]: pgmap v3232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:53 smithi144 ceph-mon[32129]: pgmap v3232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:29:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:55 smithi160 ceph-mon[36715]: pgmap v3233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:55 smithi183 ceph-mon[37273]: pgmap v3233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:55 smithi144 ceph-mon[32129]: pgmap v3233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5039.640458 secs 2024-07-26T12:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:57 smithi160 ceph-mon[36715]: pgmap v3234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5039.640458 secs 2024-07-26T12:29:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:57 smithi183 ceph-mon[37273]: pgmap v3234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5039.640458 secs 2024-07-26T12:29:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:57 smithi144 ceph-mon[32129]: pgmap v3234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:29:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:29:59 smithi160 ceph-mon[36715]: pgmap v3235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:29:59 smithi183 ceph-mon[37273]: pgmap v3235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:29:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:29:59 smithi144 ceph-mon[32129]: pgmap v3235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:01.266+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:01.267+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:01.267+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:01.267+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:01.267+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:01 smithi160 ceph-mon[36715]: pgmap v3236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:01 smithi183 ceph-mon[37273]: pgmap v3236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:01 smithi144 ceph-mon[32129]: pgmap v3236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5044.640586 secs 2024-07-26T12:30:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5044.640586 secs 2024-07-26T12:30:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5044.640586 secs 2024-07-26T12:30:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:03 smithi144 ceph-mon[32129]: pgmap v3237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:03 smithi160 ceph-mon[36715]: pgmap v3237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:03 smithi183 ceph-mon[37273]: pgmap v3237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:05 smithi160 ceph-mon[36715]: pgmap v3238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:05 smithi183 ceph-mon[37273]: pgmap v3238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:05 smithi144 ceph-mon[32129]: pgmap v3238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5049.640652 secs 2024-07-26T12:30:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:07 smithi160 ceph-mon[36715]: pgmap v3239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5049.640652 secs 2024-07-26T12:30:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:07 smithi183 ceph-mon[37273]: pgmap v3239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5049.640652 secs 2024-07-26T12:30:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:07 smithi144 ceph-mon[32129]: pgmap v3239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:08 smithi183 ceph-mon[37273]: pgmap v3240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:08 smithi144 ceph-mon[32129]: pgmap v3240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:08 smithi160 ceph-mon[36715]: pgmap v3240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:11 smithi183 ceph-mon[37273]: pgmap v3241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:11.423 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:11 smithi144 ceph-mon[32129]: pgmap v3241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:11 smithi160 ceph-mon[36715]: pgmap v3241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5054.640772 secs 2024-07-26T12:30:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5054.640772 secs 2024-07-26T12:30:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5054.640772 secs 2024-07-26T12:30:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:13 smithi144 ceph-mon[32129]: pgmap v3242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:13 smithi160 ceph-mon[36715]: pgmap v3242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:13 smithi183 ceph-mon[37273]: pgmap v3242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:15 smithi144 ceph-mon[32129]: pgmap v3243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:15 smithi160 ceph-mon[36715]: pgmap v3243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:15 smithi183 ceph-mon[37273]: pgmap v3243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:16.277+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:16.278+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:16.278+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:16.278+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:16.278+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5059.640894 secs 2024-07-26T12:30:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:17 smithi144 ceph-mon[32129]: pgmap v3244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5059.640894 secs 2024-07-26T12:30:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:17 smithi160 ceph-mon[36715]: pgmap v3244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5059.640894 secs 2024-07-26T12:30:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:17 smithi183 ceph-mon[37273]: pgmap v3244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:18 smithi144 ceph-mon[32129]: pgmap v3245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:30:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:18 smithi144 ceph-mon[32129]: pgmap v3246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:30:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:18 smithi160 ceph-mon[36715]: pgmap v3245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:30:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:18 smithi160 ceph-mon[36715]: pgmap v3246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:30:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:18 smithi183 ceph-mon[37273]: pgmap v3245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:30:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:18 smithi183 ceph-mon[37273]: pgmap v3246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:30:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:19 smithi183 ceph-mon[37273]: pgmap v3247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:19 smithi144 ceph-mon[32129]: pgmap v3247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:19 smithi160 ceph-mon[36715]: pgmap v3247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:21 smithi183 ceph-mon[37273]: pgmap v3248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:21 smithi144 ceph-mon[32129]: pgmap v3248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:21 smithi160 ceph-mon[36715]: pgmap v3248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5064.641023 secs 2024-07-26T12:30:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5064.641023 secs 2024-07-26T12:30:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5064.641023 secs 2024-07-26T12:30:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:23 smithi183 ceph-mon[37273]: pgmap v3249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:23 smithi144 ceph-mon[32129]: pgmap v3249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:23 smithi160 ceph-mon[36715]: pgmap v3249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:25 smithi183 ceph-mon[37273]: pgmap v3250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:25 smithi144 ceph-mon[32129]: pgmap v3250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:25 smithi160 ceph-mon[36715]: pgmap v3250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:30:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5069.641148 secs 2024-07-26T12:30:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5069.641148 secs 2024-07-26T12:30:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5069.641148 secs 2024-07-26T12:30:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:27 smithi144 ceph-mon[32129]: pgmap v3251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:30:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:27 smithi160 ceph-mon[36715]: pgmap v3251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:30:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:27 smithi183 ceph-mon[37273]: pgmap v3251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:30:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:29 smithi183 ceph-mon[37273]: pgmap v3252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:29 smithi144 ceph-mon[32129]: pgmap v3252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:29 smithi160 ceph-mon[36715]: pgmap v3252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:31.293+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:31.294+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:31.294+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:31.294+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:31.294+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:31.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:31 smithi160 ceph-mon[36715]: pgmap v3253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:31 smithi183 ceph-mon[37273]: pgmap v3253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:31 smithi144 ceph-mon[32129]: pgmap v3253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5074.641308 secs 2024-07-26T12:30:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5074.641308 secs 2024-07-26T12:30:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5074.641308 secs 2024-07-26T12:30:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:33 smithi183 ceph-mon[37273]: pgmap v3254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:33 smithi144 ceph-mon[32129]: pgmap v3254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:33 smithi160 ceph-mon[36715]: pgmap v3254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:35 smithi183 ceph-mon[37273]: pgmap v3255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:35 smithi144 ceph-mon[32129]: pgmap v3255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:35 smithi160 ceph-mon[36715]: pgmap v3255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5079.641371 secs 2024-07-26T12:30:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5079.641371 secs 2024-07-26T12:30:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5079.641371 secs 2024-07-26T12:30:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:37 smithi144 ceph-mon[32129]: pgmap v3256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:37.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:37 smithi183 ceph-mon[37273]: pgmap v3256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:37 smithi160 ceph-mon[36715]: pgmap v3256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:39 smithi183 ceph-mon[37273]: pgmap v3257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:39 smithi144 ceph-mon[32129]: pgmap v3257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:39 smithi160 ceph-mon[36715]: pgmap v3257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:41 smithi183 ceph-mon[37273]: pgmap v3258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:41 smithi144 ceph-mon[32129]: pgmap v3258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:41 smithi160 ceph-mon[36715]: pgmap v3258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5084.641496 secs 2024-07-26T12:30:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5084.641496 secs 2024-07-26T12:30:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5084.641496 secs 2024-07-26T12:30:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:43 smithi183 ceph-mon[37273]: pgmap v3259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:43 smithi144 ceph-mon[32129]: pgmap v3259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:43 smithi160 ceph-mon[36715]: pgmap v3259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:45 smithi183 ceph-mon[37273]: pgmap v3260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:45 smithi144 ceph-mon[32129]: pgmap v3260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:45 smithi160 ceph-mon[36715]: pgmap v3260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:46.297+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:46.298+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:46.298+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:46.298+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:30:46.298+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5089.641563 secs 2024-07-26T12:30:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5089.641563 secs 2024-07-26T12:30:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5089.641563 secs 2024-07-26T12:30:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:47 smithi144 ceph-mon[32129]: pgmap v3261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:47 smithi160 ceph-mon[36715]: pgmap v3261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:47 smithi183 ceph-mon[37273]: pgmap v3261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:30:49.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:49 smithi144 ceph-mon[32129]: pgmap v3262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:49 smithi183 ceph-mon[37273]: pgmap v3262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:49 smithi160 ceph-mon[36715]: pgmap v3262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:51 smithi183 ceph-mon[37273]: pgmap v3263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:51 smithi144 ceph-mon[32129]: pgmap v3263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:51 smithi160 ceph-mon[36715]: pgmap v3263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:30:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5094.641682 secs 2024-07-26T12:30:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5094.641682 secs 2024-07-26T12:30:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5094.641682 secs 2024-07-26T12:30:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:53 smithi183 ceph-mon[37273]: pgmap v3264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:53 smithi144 ceph-mon[32129]: pgmap v3264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:53 smithi160 ceph-mon[36715]: pgmap v3264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:30:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:55 smithi183 ceph-mon[37273]: pgmap v3265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:30:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:55 smithi144 ceph-mon[32129]: pgmap v3265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:30:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:55 smithi160 ceph-mon[36715]: pgmap v3265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:30:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5099.641813 secs 2024-07-26T12:30:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5099.641813 secs 2024-07-26T12:30:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5099.641813 secs 2024-07-26T12:30:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:57 smithi144 ceph-mon[32129]: pgmap v3266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:30:57.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:57 smithi160 ceph-mon[36715]: pgmap v3266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:30:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:57 smithi183 ceph-mon[37273]: pgmap v3266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:30:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:30:59 smithi183 ceph-mon[37273]: pgmap v3267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:30:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:30:59 smithi144 ceph-mon[32129]: pgmap v3267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:30:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:30:59 smithi160 ceph-mon[36715]: pgmap v3267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:01.314+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:01.315+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:01.315+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:01.315+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:01.315+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:01 smithi160 ceph-mon[36715]: pgmap v3268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:31:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:01 smithi183 ceph-mon[37273]: pgmap v3268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:31:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:01 smithi144 ceph-mon[32129]: pgmap v3268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:31:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5104.641961 secs 2024-07-26T12:31:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5104.641961 secs 2024-07-26T12:31:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5104.641961 secs 2024-07-26T12:31:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:03 smithi183 ceph-mon[37273]: pgmap v3269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:03 smithi144 ceph-mon[32129]: pgmap v3269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:03 smithi160 ceph-mon[36715]: pgmap v3269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:05 smithi144 ceph-mon[32129]: pgmap v3270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:05 smithi183 ceph-mon[37273]: pgmap v3270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:05 smithi160 ceph-mon[36715]: pgmap v3270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5109.642098 secs 2024-07-26T12:31:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5109.642098 secs 2024-07-26T12:31:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5109.642098 secs 2024-07-26T12:31:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:07 smithi144 ceph-mon[32129]: pgmap v3271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:07.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:07 smithi183 ceph-mon[37273]: pgmap v3271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:07 smithi160 ceph-mon[36715]: pgmap v3271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:09 smithi183 ceph-mon[37273]: pgmap v3272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:09 smithi144 ceph-mon[32129]: pgmap v3272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:09 smithi160 ceph-mon[36715]: pgmap v3272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:11 smithi183 ceph-mon[37273]: pgmap v3273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:11 smithi144 ceph-mon[32129]: pgmap v3273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:11 smithi160 ceph-mon[36715]: pgmap v3273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5114.642231 secs 2024-07-26T12:31:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5114.642231 secs 2024-07-26T12:31:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5114.642231 secs 2024-07-26T12:31:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:13 smithi183 ceph-mon[37273]: pgmap v3274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:13 smithi144 ceph-mon[32129]: pgmap v3274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:13 smithi160 ceph-mon[36715]: pgmap v3274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:15 smithi183 ceph-mon[37273]: pgmap v3275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:15 smithi144 ceph-mon[32129]: pgmap v3275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:15 smithi160 ceph-mon[36715]: pgmap v3275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:16.319+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:16.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:16.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:16.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:16.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5119.642382 secs 2024-07-26T12:31:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5119.642382 secs 2024-07-26T12:31:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5119.642382 secs 2024-07-26T12:31:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:18 smithi144 ceph-mon[32129]: pgmap v3276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:18 smithi160 ceph-mon[36715]: pgmap v3276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:18 smithi183 ceph-mon[37273]: pgmap v3276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:19 smithi144 ceph-mon[32129]: pgmap v3277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:19 smithi144 ceph-mon[32129]: pgmap v3278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:19 smithi160 ceph-mon[36715]: pgmap v3277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:19 smithi160 ceph-mon[36715]: pgmap v3278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:19 smithi183 ceph-mon[37273]: pgmap v3277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:19 smithi183 ceph-mon[37273]: pgmap v3278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:31:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5124.642498 secs 2024-07-26T12:31:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:22 smithi144 ceph-mon[32129]: pgmap v3279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5124.642498 secs 2024-07-26T12:31:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:22 smithi160 ceph-mon[36715]: pgmap v3279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5124.642498 secs 2024-07-26T12:31:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:22 smithi183 ceph-mon[37273]: pgmap v3279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:23 smithi183 ceph-mon[37273]: pgmap v3280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:23 smithi144 ceph-mon[32129]: pgmap v3280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:23 smithi160 ceph-mon[36715]: pgmap v3280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:26 smithi144 ceph-mon[32129]: pgmap v3281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:31:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:26 smithi160 ceph-mon[36715]: pgmap v3281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:31:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:26 smithi183 ceph-mon[37273]: pgmap v3281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:31:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5129.642598 secs 2024-07-26T12:31:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5129.642598 secs 2024-07-26T12:31:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5129.642598 secs 2024-07-26T12:31:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:28 smithi144 ceph-mon[32129]: pgmap v3282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:28 smithi160 ceph-mon[36715]: pgmap v3282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:28 smithi183 ceph-mon[37273]: pgmap v3282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:31:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:30 smithi144 ceph-mon[32129]: pgmap v3283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:31:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:30 smithi160 ceph-mon[36715]: pgmap v3283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:31:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:30 smithi183 ceph-mon[37273]: pgmap v3283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:31:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:31.334+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:31.335+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:31.335+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:31.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:31.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5134.642702 secs 2024-07-26T12:31:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:32 smithi144 ceph-mon[32129]: pgmap v3284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5134.642702 secs 2024-07-26T12:31:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:32 smithi160 ceph-mon[36715]: pgmap v3284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5134.642702 secs 2024-07-26T12:31:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:32 smithi183 ceph-mon[37273]: pgmap v3284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:34 smithi144 ceph-mon[32129]: pgmap v3285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:34 smithi160 ceph-mon[36715]: pgmap v3285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:34 smithi183 ceph-mon[37273]: pgmap v3285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:36 smithi144 ceph-mon[32129]: pgmap v3286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:36 smithi160 ceph-mon[36715]: pgmap v3286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:36 smithi183 ceph-mon[37273]: pgmap v3286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5139.642786 secs 2024-07-26T12:31:37.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5139.642786 secs 2024-07-26T12:31:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5139.642786 secs 2024-07-26T12:31:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:38 smithi144 ceph-mon[32129]: pgmap v3287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:38 smithi160 ceph-mon[36715]: pgmap v3287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:38 smithi183 ceph-mon[37273]: pgmap v3287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:40 smithi144 ceph-mon[32129]: pgmap v3288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:40 smithi160 ceph-mon[36715]: pgmap v3288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:40 smithi183 ceph-mon[37273]: pgmap v3288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5144.642928 secs 2024-07-26T12:31:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:42 smithi160 ceph-mon[36715]: pgmap v3289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5144.642928 secs 2024-07-26T12:31:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:42 smithi183 ceph-mon[37273]: pgmap v3289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5144.642928 secs 2024-07-26T12:31:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:42 smithi144 ceph-mon[32129]: pgmap v3289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:44 smithi160 ceph-mon[36715]: pgmap v3290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:44 smithi183 ceph-mon[37273]: pgmap v3290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:44 smithi144 ceph-mon[32129]: pgmap v3290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:46.352+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:46.353+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:46.353+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:46.353+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:31:46.353+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:46 smithi160 ceph-mon[36715]: pgmap v3291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:46 smithi183 ceph-mon[37273]: pgmap v3291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:46 smithi144 ceph-mon[32129]: pgmap v3291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:47.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5149.643075 secs 2024-07-26T12:31:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5149.643075 secs 2024-07-26T12:31:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5149.643075 secs 2024-07-26T12:31:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:48 smithi160 ceph-mon[36715]: pgmap v3292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:48 smithi183 ceph-mon[37273]: pgmap v3292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:48 smithi144 ceph-mon[32129]: pgmap v3292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:50 smithi160 ceph-mon[36715]: pgmap v3293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:50 smithi183 ceph-mon[37273]: pgmap v3293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:50 smithi144 ceph-mon[32129]: pgmap v3293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5154.643218 secs 2024-07-26T12:31:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:52 smithi160 ceph-mon[36715]: pgmap v3294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5154.643218 secs 2024-07-26T12:31:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:52 smithi183 ceph-mon[37273]: pgmap v3294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5154.643218 secs 2024-07-26T12:31:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:52 smithi144 ceph-mon[32129]: pgmap v3294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:54 smithi160 ceph-mon[36715]: pgmap v3295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:54 smithi183 ceph-mon[37273]: pgmap v3295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:54 smithi144 ceph-mon[32129]: pgmap v3295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:31:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:56 smithi160 ceph-mon[36715]: pgmap v3296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:56 smithi183 ceph-mon[37273]: pgmap v3296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:56 smithi144 ceph-mon[32129]: pgmap v3296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:31:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5159.643353 secs 2024-07-26T12:31:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5159.643353 secs 2024-07-26T12:31:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5159.643353 secs 2024-07-26T12:31:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:31:58 smithi160 ceph-mon[36715]: pgmap v3297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:31:58 smithi183 ceph-mon[37273]: pgmap v3297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:31:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:31:58 smithi144 ceph-mon[32129]: pgmap v3297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:00 smithi160 ceph-mon[36715]: pgmap v3298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:00 smithi183 ceph-mon[37273]: pgmap v3298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:00 smithi144 ceph-mon[32129]: pgmap v3298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:01.368+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:01.369+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:01.369+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:01.369+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:01.369+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5164.643418 secs 2024-07-26T12:32:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:02 smithi160 ceph-mon[36715]: pgmap v3299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5164.643418 secs 2024-07-26T12:32:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:02 smithi183 ceph-mon[37273]: pgmap v3299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5164.643418 secs 2024-07-26T12:32:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:02 smithi144 ceph-mon[32129]: pgmap v3299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:04 smithi160 ceph-mon[36715]: pgmap v3300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:04 smithi183 ceph-mon[37273]: pgmap v3300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:04 smithi144 ceph-mon[32129]: pgmap v3300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:06 smithi160 ceph-mon[36715]: pgmap v3301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:06 smithi183 ceph-mon[37273]: pgmap v3301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:06 smithi144 ceph-mon[32129]: pgmap v3301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:07.320 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5169.643502 secs 2024-07-26T12:32:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5169.643502 secs 2024-07-26T12:32:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:07 smithi183 ceph-mon[37273]: pgmap v3302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:07 smithi144 ceph-mon[32129]: pgmap v3302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5169.643502 secs 2024-07-26T12:32:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:07 smithi160 ceph-mon[36715]: pgmap v3302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:10 smithi144 ceph-mon[32129]: pgmap v3303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:10 smithi160 ceph-mon[36715]: pgmap v3303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:10 smithi183 ceph-mon[37273]: pgmap v3303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5174.643572 secs 2024-07-26T12:32:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:12 smithi144 ceph-mon[32129]: pgmap v3304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5174.643572 secs 2024-07-26T12:32:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:12 smithi160 ceph-mon[36715]: pgmap v3304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5174.643572 secs 2024-07-26T12:32:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:12 smithi183 ceph-mon[37273]: pgmap v3304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:14 smithi144 ceph-mon[32129]: pgmap v3305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:14 smithi160 ceph-mon[36715]: pgmap v3305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:14 smithi183 ceph-mon[37273]: pgmap v3305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:16.381+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:16.382+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:16.382+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:16.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:16.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:16 smithi144 ceph-mon[32129]: pgmap v3306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:16 smithi160 ceph-mon[36715]: pgmap v3306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:16 smithi183 ceph-mon[37273]: pgmap v3306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5179.643645 secs 2024-07-26T12:32:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5179.643645 secs 2024-07-26T12:32:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5179.643645 secs 2024-07-26T12:32:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:18 smithi144 ceph-mon[32129]: pgmap v3307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:18 smithi160 ceph-mon[36715]: pgmap v3307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:18 smithi183 ceph-mon[37273]: pgmap v3307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:20.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:20 smithi160 ceph-mon[36715]: pgmap v3308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:20.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:20 smithi183 ceph-mon[37273]: pgmap v3308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.516 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.517 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.603 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:20 smithi144 ceph-mon[32129]: pgmap v3308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:20.603 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:20.604 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.604 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.604 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:20.604 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5184.643705 secs 2024-07-26T12:32:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:22 smithi160 ceph-mon[36715]: pgmap v3309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5184.643705 secs 2024-07-26T12:32:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:22 smithi183 ceph-mon[37273]: pgmap v3309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5184.643705 secs 2024-07-26T12:32:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:22 smithi144 ceph-mon[32129]: pgmap v3309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:24 smithi183 ceph-mon[37273]: pgmap v3310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:24 smithi144 ceph-mon[32129]: pgmap v3310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:24 smithi160 ceph-mon[36715]: pgmap v3310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:25 smithi183 ceph-mon[37273]: pgmap v3311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:32:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:25 smithi183 ceph-mon[37273]: pgmap v3312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:32:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:25 smithi144 ceph-mon[32129]: pgmap v3311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:32:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:25 smithi144 ceph-mon[32129]: pgmap v3312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:32:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:25 smithi160 ceph-mon[36715]: pgmap v3311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:32:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:25 smithi160 ceph-mon[36715]: pgmap v3312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:32:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:32:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5189.643788 secs 2024-07-26T12:32:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:27 smithi144 ceph-mon[32129]: pgmap v3313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:32:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5189.643788 secs 2024-07-26T12:32:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:27 smithi160 ceph-mon[36715]: pgmap v3313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:32:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5189.643788 secs 2024-07-26T12:32:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:27 smithi183 ceph-mon[37273]: pgmap v3313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:32:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:28 smithi183 ceph-mon[37273]: pgmap v3314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:32:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:28 smithi144 ceph-mon[32129]: pgmap v3314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:32:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:28 smithi160 ceph-mon[36715]: pgmap v3314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:32:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:31.397+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:31.398+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:31.398+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:31.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:31.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:31 smithi144 ceph-mon[32129]: pgmap v3315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:31 smithi160 ceph-mon[36715]: pgmap v3315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:31 smithi183 ceph-mon[37273]: pgmap v3315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:32.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5194.643868 secs 2024-07-26T12:32:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5194.643868 secs 2024-07-26T12:32:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5194.643868 secs 2024-07-26T12:32:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:33 smithi144 ceph-mon[32129]: pgmap v3316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:33 smithi160 ceph-mon[36715]: pgmap v3316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:33 smithi183 ceph-mon[37273]: pgmap v3316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:35 smithi144 ceph-mon[32129]: pgmap v3317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:32:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:35 smithi160 ceph-mon[36715]: pgmap v3317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:32:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:35 smithi183 ceph-mon[37273]: pgmap v3317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:32:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5199.643935 secs 2024-07-26T12:32:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:37 smithi160 ceph-mon[36715]: pgmap v3318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5199.643935 secs 2024-07-26T12:32:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:37 smithi183 ceph-mon[37273]: pgmap v3318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5199.643935 secs 2024-07-26T12:32:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:37 smithi144 ceph-mon[32129]: pgmap v3318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:39 smithi160 ceph-mon[36715]: pgmap v3319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:39 smithi183 ceph-mon[37273]: pgmap v3319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:39 smithi144 ceph-mon[32129]: pgmap v3319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:41 smithi160 ceph-mon[36715]: pgmap v3320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:41 smithi183 ceph-mon[37273]: pgmap v3320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:41 smithi144 ceph-mon[32129]: pgmap v3320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:32:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5204.644054 secs 2024-07-26T12:32:42.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5204.644054 secs 2024-07-26T12:32:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5204.644054 secs 2024-07-26T12:32:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:43 smithi160 ceph-mon[36715]: pgmap v3321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:43 smithi183 ceph-mon[37273]: pgmap v3321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:43 smithi144 ceph-mon[32129]: pgmap v3321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:45 smithi160 ceph-mon[36715]: pgmap v3322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:45 smithi183 ceph-mon[37273]: pgmap v3322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:45 smithi144 ceph-mon[32129]: pgmap v3322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:46.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:46.416+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:46.416+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:46.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:46.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:47.020+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:47.020+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:32:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:47.021+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:32:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:32:47.021+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:32:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5209.644118 secs 2024-07-26T12:32:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:47 smithi160 ceph-mon[36715]: pgmap v3323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5209.644118 secs 2024-07-26T12:32:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:47 smithi183 ceph-mon[37273]: pgmap v3323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5209.644118 secs 2024-07-26T12:32:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:47 smithi144 ceph-mon[32129]: pgmap v3323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:49 smithi160 ceph-mon[36715]: pgmap v3324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:49 smithi183 ceph-mon[37273]: pgmap v3324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:49 smithi144 ceph-mon[32129]: pgmap v3324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:51 smithi160 ceph-mon[36715]: pgmap v3325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:51 smithi183 ceph-mon[37273]: pgmap v3325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:51 smithi144 ceph-mon[32129]: pgmap v3325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5214.644238 secs 2024-07-26T12:32:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5214.644238 secs 2024-07-26T12:32:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5214.644238 secs 2024-07-26T12:32:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:53 smithi160 ceph-mon[36715]: pgmap v3326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:53 smithi183 ceph-mon[37273]: pgmap v3326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:53 smithi144 ceph-mon[32129]: pgmap v3326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:55 smithi160 ceph-mon[36715]: pgmap v3327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:55 smithi183 ceph-mon[37273]: pgmap v3327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:55 smithi144 ceph-mon[32129]: pgmap v3327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5219.644339 secs 2024-07-26T12:32:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:57 smithi160 ceph-mon[36715]: pgmap v3328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5219.644339 secs 2024-07-26T12:32:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:57 smithi183 ceph-mon[37273]: pgmap v3328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5219.644339 secs 2024-07-26T12:32:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:57 smithi144 ceph-mon[32129]: pgmap v3328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:32:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:32:59 smithi160 ceph-mon[36715]: pgmap v3329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:32:59 smithi183 ceph-mon[37273]: pgmap v3329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:32:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:32:59 smithi144 ceph-mon[32129]: pgmap v3329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:01.429+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:01.430+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:01.430+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:01.430+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:01.430+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:01 smithi183 ceph-mon[37273]: pgmap v3330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:01 smithi144 ceph-mon[32129]: pgmap v3330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:01 smithi160 ceph-mon[36715]: pgmap v3330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5224.644458 secs 2024-07-26T12:33:02.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5224.644458 secs 2024-07-26T12:33:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5224.644458 secs 2024-07-26T12:33:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:03 smithi183 ceph-mon[37273]: pgmap v3331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:03 smithi144 ceph-mon[32129]: pgmap v3331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:03 smithi160 ceph-mon[36715]: pgmap v3331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:04 smithi183 ceph-mon[37273]: pgmap v3332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:04 smithi144 ceph-mon[32129]: pgmap v3332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:04 smithi160 ceph-mon[36715]: pgmap v3332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5229.644579 secs 2024-07-26T12:33:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:07 smithi160 ceph-mon[36715]: pgmap v3333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5229.644579 secs 2024-07-26T12:33:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:07 smithi183 ceph-mon[37273]: pgmap v3333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5229.644579 secs 2024-07-26T12:33:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:07 smithi144 ceph-mon[32129]: pgmap v3333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:09 smithi160 ceph-mon[36715]: pgmap v3334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:09 smithi183 ceph-mon[37273]: pgmap v3334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:09 smithi144 ceph-mon[32129]: pgmap v3334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:11 smithi160 ceph-mon[36715]: pgmap v3335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:11 smithi183 ceph-mon[37273]: pgmap v3335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:11 smithi144 ceph-mon[32129]: pgmap v3335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5234.644700 secs 2024-07-26T12:33:12.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5234.644700 secs 2024-07-26T12:33:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5234.644700 secs 2024-07-26T12:33:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:13 smithi160 ceph-mon[36715]: pgmap v3336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:13 smithi183 ceph-mon[37273]: pgmap v3336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:13 smithi144 ceph-mon[32129]: pgmap v3336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:15 smithi160 ceph-mon[36715]: pgmap v3337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:15 smithi183 ceph-mon[37273]: pgmap v3337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:15 smithi144 ceph-mon[32129]: pgmap v3337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:16.434+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:16.435+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:16.435+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:16.435+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:16.435+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5239.644821 secs 2024-07-26T12:33:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:17 smithi160 ceph-mon[36715]: pgmap v3338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5239.644821 secs 2024-07-26T12:33:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:17 smithi183 ceph-mon[37273]: pgmap v3338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5239.644821 secs 2024-07-26T12:33:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:17 smithi144 ceph-mon[32129]: pgmap v3338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:19 smithi160 ceph-mon[36715]: pgmap v3339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:33:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:19 smithi183 ceph-mon[37273]: pgmap v3339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:33:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:19 smithi144 ceph-mon[32129]: pgmap v3339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:33:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:21 smithi160 ceph-mon[36715]: pgmap v3340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:33:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:21 smithi183 ceph-mon[37273]: pgmap v3340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:33:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:21 smithi144 ceph-mon[32129]: pgmap v3340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:33:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5244.644883 secs 2024-07-26T12:33:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5244.644883 secs 2024-07-26T12:33:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5244.644883 secs 2024-07-26T12:33:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:23 smithi160 ceph-mon[36715]: pgmap v3341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:33:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:23 smithi183 ceph-mon[37273]: pgmap v3341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:33:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:23 smithi144 ceph-mon[32129]: pgmap v3341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:33:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:25 smithi160 ceph-mon[36715]: pgmap v3342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:33:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:25 smithi183 ceph-mon[37273]: pgmap v3342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:33:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:25 smithi144 ceph-mon[32129]: pgmap v3342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:33:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:33:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:33:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:33:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:27 smithi160 ceph-mon[36715]: pgmap v3343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5249.644952 secs 2024-07-26T12:33:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:27 smithi183 ceph-mon[37273]: pgmap v3343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5249.644952 secs 2024-07-26T12:33:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:27 smithi144 ceph-mon[32129]: pgmap v3343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5249.644952 secs 2024-07-26T12:33:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:29 smithi160 ceph-mon[36715]: pgmap v3344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:29 smithi183 ceph-mon[37273]: pgmap v3344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:29 smithi144 ceph-mon[32129]: pgmap v3344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:31.448+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:31.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:31.448+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:31.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:31.448+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:31.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:31.448+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:31.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:31.448+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:31 smithi183 ceph-mon[37273]: pgmap v3345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:33:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:31 smithi144 ceph-mon[32129]: pgmap v3345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:33:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:31 smithi160 ceph-mon[36715]: pgmap v3345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:33:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5254.645069 secs 2024-07-26T12:33:32.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5254.645069 secs 2024-07-26T12:33:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5254.645069 secs 2024-07-26T12:33:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:33 smithi183 ceph-mon[37273]: pgmap v3346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:33:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:33 smithi144 ceph-mon[32129]: pgmap v3346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:33:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:33 smithi160 ceph-mon[36715]: pgmap v3346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:33:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:34 smithi183 ceph-mon[37273]: pgmap v3347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:33:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:34 smithi144 ceph-mon[32129]: pgmap v3347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:33:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:34 smithi160 ceph-mon[36715]: pgmap v3347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:33:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:36 smithi144 ceph-mon[32129]: pgmap v3348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:33:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5259.645198 secs 2024-07-26T12:33:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:36 smithi160 ceph-mon[36715]: pgmap v3348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:33:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5259.645198 secs 2024-07-26T12:33:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:36 smithi183 ceph-mon[37273]: pgmap v3348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:33:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5259.645198 secs 2024-07-26T12:33:38.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:38 smithi183 ceph-mon[37273]: pgmap v3349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:38.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:38 smithi144 ceph-mon[32129]: pgmap v3349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:38 smithi160 ceph-mon[36715]: pgmap v3349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:40 smithi144 ceph-mon[32129]: pgmap v3350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:40 smithi160 ceph-mon[36715]: pgmap v3350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:40 smithi183 ceph-mon[37273]: pgmap v3350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5264.645329 secs 2024-07-26T12:33:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5264.645329 secs 2024-07-26T12:33:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5264.645329 secs 2024-07-26T12:33:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:42 smithi144 ceph-mon[32129]: pgmap v3351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:42 smithi160 ceph-mon[36715]: pgmap v3351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:42 smithi183 ceph-mon[37273]: pgmap v3351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:44 smithi144 ceph-mon[32129]: pgmap v3352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:44 smithi160 ceph-mon[36715]: pgmap v3352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:44 smithi183 ceph-mon[37273]: pgmap v3352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:46.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:46.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:46.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:46.460+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:46.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:46.460+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:46.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:46.460+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:46.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:33:46.460+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:46 smithi144 ceph-mon[32129]: pgmap v3353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5269.645460 secs 2024-07-26T12:33:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:46 smithi160 ceph-mon[36715]: pgmap v3353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5269.645460 secs 2024-07-26T12:33:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:46 smithi183 ceph-mon[37273]: pgmap v3353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5269.645460 secs 2024-07-26T12:33:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:48 smithi144 ceph-mon[32129]: pgmap v3354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:48 smithi160 ceph-mon[36715]: pgmap v3354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:48 smithi183 ceph-mon[37273]: pgmap v3354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:50 smithi160 ceph-mon[36715]: pgmap v3355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:50 smithi183 ceph-mon[37273]: pgmap v3355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:50 smithi144 ceph-mon[32129]: pgmap v3355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5274.645603 secs 2024-07-26T12:33:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5274.645603 secs 2024-07-26T12:33:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5274.645603 secs 2024-07-26T12:33:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:52 smithi160 ceph-mon[36715]: pgmap v3356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:52 smithi183 ceph-mon[37273]: pgmap v3356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:52 smithi144 ceph-mon[32129]: pgmap v3356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:33:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:54 smithi160 ceph-mon[36715]: pgmap v3357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:54 smithi183 ceph-mon[37273]: pgmap v3357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:54 smithi144 ceph-mon[32129]: pgmap v3357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:56 smithi160 ceph-mon[36715]: pgmap v3358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5279.645734 secs 2024-07-26T12:33:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:56 smithi183 ceph-mon[37273]: pgmap v3358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5279.645734 secs 2024-07-26T12:33:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:56 smithi144 ceph-mon[32129]: pgmap v3358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:33:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5279.645734 secs 2024-07-26T12:33:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:33:58 smithi160 ceph-mon[36715]: pgmap v3359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:58.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:33:58 smithi183 ceph-mon[37273]: pgmap v3359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:33:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:33:58 smithi144 ceph-mon[32129]: pgmap v3359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:00 smithi160 ceph-mon[36715]: pgmap v3360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:00 smithi183 ceph-mon[37273]: pgmap v3360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:00 smithi144 ceph-mon[32129]: pgmap v3360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:01.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:01.477+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:01.477+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:01.477+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:01.477+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5284.645864 secs 2024-07-26T12:34:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5284.645864 secs 2024-07-26T12:34:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5284.645864 secs 2024-07-26T12:34:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:02 smithi160 ceph-mon[36715]: pgmap v3361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:02 smithi183 ceph-mon[37273]: pgmap v3361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:02 smithi144 ceph-mon[32129]: pgmap v3361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:04 smithi160 ceph-mon[36715]: pgmap v3362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:04 smithi183 ceph-mon[37273]: pgmap v3362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:04 smithi144 ceph-mon[32129]: pgmap v3362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:06 smithi160 ceph-mon[36715]: pgmap v3363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5289.645991 secs 2024-07-26T12:34:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:06 smithi183 ceph-mon[37273]: pgmap v3363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5289.645991 secs 2024-07-26T12:34:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:06 smithi144 ceph-mon[32129]: pgmap v3363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5289.645991 secs 2024-07-26T12:34:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:08 smithi160 ceph-mon[36715]: pgmap v3364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:08 smithi183 ceph-mon[37273]: pgmap v3364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:08 smithi144 ceph-mon[32129]: pgmap v3364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:10 smithi160 ceph-mon[36715]: pgmap v3365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:10 smithi183 ceph-mon[37273]: pgmap v3365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:10 smithi144 ceph-mon[32129]: pgmap v3365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5294.646112 secs 2024-07-26T12:34:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5294.646112 secs 2024-07-26T12:34:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5294.646112 secs 2024-07-26T12:34:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:12 smithi183 ceph-mon[37273]: pgmap v3366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:12 smithi144 ceph-mon[32129]: pgmap v3366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:12 smithi160 ceph-mon[36715]: pgmap v3366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:14 smithi183 ceph-mon[37273]: pgmap v3367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:14 smithi144 ceph-mon[32129]: pgmap v3367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:14 smithi160 ceph-mon[36715]: pgmap v3367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:16.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:16.493+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:16.493+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:16.493+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:16.494+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:16 smithi183 ceph-mon[37273]: pgmap v3368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5299.646237 secs 2024-07-26T12:34:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:16 smithi144 ceph-mon[32129]: pgmap v3368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5299.646237 secs 2024-07-26T12:34:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:16 smithi160 ceph-mon[36715]: pgmap v3368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5299.646237 secs 2024-07-26T12:34:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:18 smithi183 ceph-mon[37273]: pgmap v3369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:18 smithi144 ceph-mon[32129]: pgmap v3369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:18 smithi160 ceph-mon[36715]: pgmap v3369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:20 smithi183 ceph-mon[37273]: pgmap v3370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:20 smithi144 ceph-mon[32129]: pgmap v3370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:20 smithi160 ceph-mon[36715]: pgmap v3370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5304.646363 secs 2024-07-26T12:34:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5304.646363 secs 2024-07-26T12:34:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5304.646363 secs 2024-07-26T12:34:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:22 smithi183 ceph-mon[37273]: pgmap v3371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:22 smithi144 ceph-mon[32129]: pgmap v3371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:22 smithi160 ceph-mon[36715]: pgmap v3371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:24 smithi183 ceph-mon[37273]: pgmap v3372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:24 smithi144 ceph-mon[32129]: pgmap v3372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:24 smithi160 ceph-mon[36715]: pgmap v3372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:26.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:26 smithi144 ceph-mon[32129]: pgmap v3373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:26.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5309.646437 secs 2024-07-26T12:34:26.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:26.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:26.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:34:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:26 smithi183 ceph-mon[37273]: pgmap v3373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5309.646437 secs 2024-07-26T12:34:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:34:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:26 smithi160 ceph-mon[36715]: pgmap v3373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5309.646437 secs 2024-07-26T12:34:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:34:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:27 smithi183 ceph-mon[37273]: pgmap v3374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:34:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:27 smithi183 ceph-mon[37273]: pgmap v3375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:34:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:27 smithi144 ceph-mon[32129]: pgmap v3374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:34:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:27 smithi144 ceph-mon[32129]: pgmap v3375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:34:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:27 smithi160 ceph-mon[36715]: pgmap v3374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:34:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:27 smithi160 ceph-mon[36715]: pgmap v3375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:34:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:29 smithi183 ceph-mon[37273]: pgmap v3376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:34:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:29 smithi144 ceph-mon[32129]: pgmap v3376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:34:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:29 smithi160 ceph-mon[36715]: pgmap v3376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:34:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:31.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:31.509+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:31.509+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:31.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:31.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:31 smithi183 ceph-mon[37273]: pgmap v3377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:34:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5314.646506 secs 2024-07-26T12:34:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:31 smithi144 ceph-mon[32129]: pgmap v3377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:34:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5314.646506 secs 2024-07-26T12:34:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:31 smithi160 ceph-mon[36715]: pgmap v3377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:34:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5314.646506 secs 2024-07-26T12:34:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:33 smithi144 ceph-mon[32129]: pgmap v3378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:34:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:33 smithi160 ceph-mon[36715]: pgmap v3378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:34:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:33 smithi183 ceph-mon[37273]: pgmap v3378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:34:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:35 smithi144 ceph-mon[32129]: pgmap v3379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:34:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:35 smithi160 ceph-mon[36715]: pgmap v3379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:34:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:35 smithi183 ceph-mon[37273]: pgmap v3379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:34:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5319.646582 secs 2024-07-26T12:34:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5319.646582 secs 2024-07-26T12:34:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5319.646582 secs 2024-07-26T12:34:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:37 smithi144 ceph-mon[32129]: pgmap v3380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:34:38.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:37 smithi183 ceph-mon[37273]: pgmap v3380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:34:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:37 smithi160 ceph-mon[36715]: pgmap v3380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:34:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:39 smithi144 ceph-mon[32129]: pgmap v3381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:39 smithi160 ceph-mon[36715]: pgmap v3381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:39 smithi183 ceph-mon[37273]: pgmap v3381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:41 smithi144 ceph-mon[32129]: pgmap v3382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5324.646721 secs 2024-07-26T12:34:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:41 smithi160 ceph-mon[36715]: pgmap v3382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5324.646721 secs 2024-07-26T12:34:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:41 smithi183 ceph-mon[37273]: pgmap v3382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5324.646721 secs 2024-07-26T12:34:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:43 smithi144 ceph-mon[32129]: pgmap v3383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:43 smithi160 ceph-mon[36715]: pgmap v3383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:43 smithi183 ceph-mon[37273]: pgmap v3383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:45 smithi144 ceph-mon[32129]: pgmap v3384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:45 smithi160 ceph-mon[36715]: pgmap v3384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:45 smithi183 ceph-mon[37273]: pgmap v3384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:46.519+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:46.520+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:46.520+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:46.520+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:34:46.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5329.646846 secs 2024-07-26T12:34:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5329.646846 secs 2024-07-26T12:34:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5329.646846 secs 2024-07-26T12:34:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:47 smithi144 ceph-mon[32129]: pgmap v3385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:47 smithi160 ceph-mon[36715]: pgmap v3385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:47 smithi183 ceph-mon[37273]: pgmap v3385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:49 smithi144 ceph-mon[32129]: pgmap v3386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:49 smithi160 ceph-mon[36715]: pgmap v3386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:49 smithi183 ceph-mon[37273]: pgmap v3386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:51 smithi144 ceph-mon[32129]: pgmap v3387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5334.646966 secs 2024-07-26T12:34:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:51 smithi160 ceph-mon[36715]: pgmap v3387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5334.646966 secs 2024-07-26T12:34:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:51 smithi183 ceph-mon[37273]: pgmap v3387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5334.646966 secs 2024-07-26T12:34:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:53 smithi144 ceph-mon[32129]: pgmap v3388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:53 smithi160 ceph-mon[36715]: pgmap v3388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:53 smithi183 ceph-mon[37273]: pgmap v3388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:34:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:55 smithi144 ceph-mon[32129]: pgmap v3389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:55 smithi160 ceph-mon[36715]: pgmap v3389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:55 smithi183 ceph-mon[37273]: pgmap v3389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:34:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5339.647089 secs 2024-07-26T12:34:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5339.647089 secs 2024-07-26T12:34:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5339.647089 secs 2024-07-26T12:34:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:57 smithi160 ceph-mon[36715]: pgmap v3390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:58.078 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:57 smithi144 ceph-mon[32129]: pgmap v3390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:34:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:57 smithi183 ceph-mon[37273]: pgmap v3390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:34:59 smithi160 ceph-mon[36715]: pgmap v3391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:34:59 smithi183 ceph-mon[37273]: pgmap v3391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:34:59 smithi144 ceph-mon[32129]: pgmap v3391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:01.529+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:01.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:01.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:01.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:01.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:01 smithi160 ceph-mon[36715]: pgmap v3392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5344.647224 secs 2024-07-26T12:35:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:01 smithi183 ceph-mon[37273]: pgmap v3392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5344.647224 secs 2024-07-26T12:35:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:01 smithi144 ceph-mon[32129]: pgmap v3392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5344.647224 secs 2024-07-26T12:35:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:03 smithi160 ceph-mon[36715]: pgmap v3393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:03 smithi183 ceph-mon[37273]: pgmap v3393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:03 smithi144 ceph-mon[32129]: pgmap v3393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:05 smithi160 ceph-mon[36715]: pgmap v3394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:05 smithi183 ceph-mon[37273]: pgmap v3394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:05 smithi144 ceph-mon[32129]: pgmap v3394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5349.647345 secs 2024-07-26T12:35:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5349.647345 secs 2024-07-26T12:35:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5349.647345 secs 2024-07-26T12:35:08.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:07 smithi183 ceph-mon[37273]: pgmap v3395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:07 smithi160 ceph-mon[36715]: pgmap v3395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:07 smithi144 ceph-mon[32129]: pgmap v3395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:09 smithi160 ceph-mon[36715]: pgmap v3396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:09 smithi183 ceph-mon[37273]: pgmap v3396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:09 smithi144 ceph-mon[32129]: pgmap v3396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:11 smithi160 ceph-mon[36715]: pgmap v3397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5354.647457 secs 2024-07-26T12:35:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:11 smithi183 ceph-mon[37273]: pgmap v3397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5354.647457 secs 2024-07-26T12:35:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:11 smithi144 ceph-mon[32129]: pgmap v3397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5354.647457 secs 2024-07-26T12:35:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:13 smithi160 ceph-mon[36715]: pgmap v3398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:13 smithi183 ceph-mon[37273]: pgmap v3398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:13 smithi144 ceph-mon[32129]: pgmap v3398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:15 smithi160 ceph-mon[36715]: pgmap v3399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:15 smithi183 ceph-mon[37273]: pgmap v3399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:15 smithi144 ceph-mon[32129]: pgmap v3399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:16.533+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:16.534+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:16.534+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:16.534+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:16.534+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5359.647623 secs 2024-07-26T12:35:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5359.647623 secs 2024-07-26T12:35:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5359.647623 secs 2024-07-26T12:35:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:17 smithi160 ceph-mon[36715]: pgmap v3400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:17 smithi183 ceph-mon[37273]: pgmap v3400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:17 smithi144 ceph-mon[32129]: pgmap v3400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:19 smithi160 ceph-mon[36715]: pgmap v3401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:19 smithi183 ceph-mon[37273]: pgmap v3401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:19 smithi144 ceph-mon[32129]: pgmap v3401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:21 smithi183 ceph-mon[37273]: pgmap v3402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5364.647725 secs 2024-07-26T12:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:21 smithi144 ceph-mon[32129]: pgmap v3402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5364.647725 secs 2024-07-26T12:35:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:21 smithi160 ceph-mon[36715]: pgmap v3402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5364.647725 secs 2024-07-26T12:35:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:23 smithi144 ceph-mon[32129]: pgmap v3403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:23 smithi160 ceph-mon[36715]: pgmap v3403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:23 smithi183 ceph-mon[37273]: pgmap v3403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:25 smithi144 ceph-mon[32129]: pgmap v3404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:25 smithi160 ceph-mon[36715]: pgmap v3404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:25 smithi183 ceph-mon[37273]: pgmap v3404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5369.647843 secs 2024-07-26T12:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5369.647843 secs 2024-07-26T12:35:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5369.647843 secs 2024-07-26T12:35:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:27 smithi144 ceph-mon[32129]: pgmap v3405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:35:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:27 smithi160 ceph-mon[36715]: pgmap v3405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:35:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:27 smithi183 ceph-mon[37273]: pgmap v3405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:35:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:28 smithi160 ceph-mon[36715]: pgmap v3406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:35:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:28 smithi160 ceph-mon[36715]: pgmap v3407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:35:29.311 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:28 smithi144 ceph-mon[32129]: pgmap v3406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:35:29.311 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:28 smithi144 ceph-mon[32129]: pgmap v3407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:35:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:28 smithi183 ceph-mon[37273]: pgmap v3406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:35:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:28 smithi183 ceph-mon[37273]: pgmap v3407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:35:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:30 smithi160 ceph-mon[36715]: pgmap v3408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:35:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:30 smithi183 ceph-mon[37273]: pgmap v3408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:35:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:30 smithi144 ceph-mon[32129]: pgmap v3408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:35:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:31.547+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:31.547+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:31.549+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:31.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:31.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5374.647961 secs 2024-07-26T12:35:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5374.647961 secs 2024-07-26T12:35:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5374.647961 secs 2024-07-26T12:35:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:32 smithi160 ceph-mon[36715]: pgmap v3409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:35:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:32 smithi183 ceph-mon[37273]: pgmap v3409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:35:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:32 smithi144 ceph-mon[32129]: pgmap v3409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:35:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:35 smithi160 ceph-mon[36715]: pgmap v3410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:35:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:35 smithi183 ceph-mon[37273]: pgmap v3410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:35:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:34 smithi144 ceph-mon[32129]: pgmap v3410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:35:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5379.648079 secs 2024-07-26T12:35:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:37 smithi160 ceph-mon[36715]: pgmap v3411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:35:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5379.648079 secs 2024-07-26T12:35:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:37 smithi183 ceph-mon[37273]: pgmap v3411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:35:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5379.648079 secs 2024-07-26T12:35:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:37 smithi144 ceph-mon[32129]: pgmap v3411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:35:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:39 smithi160 ceph-mon[36715]: pgmap v3412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:35:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:39 smithi183 ceph-mon[37273]: pgmap v3412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:35:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:39 smithi144 ceph-mon[32129]: pgmap v3412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:35:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:41 smithi160 ceph-mon[36715]: pgmap v3413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:41 smithi183 ceph-mon[37273]: pgmap v3413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:41 smithi144 ceph-mon[32129]: pgmap v3413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5384.648196 secs 2024-07-26T12:35:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5384.648196 secs 2024-07-26T12:35:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5384.648196 secs 2024-07-26T12:35:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:43 smithi160 ceph-mon[36715]: pgmap v3414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:43 smithi183 ceph-mon[37273]: pgmap v3414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:43 smithi144 ceph-mon[32129]: pgmap v3414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:45 smithi160 ceph-mon[36715]: pgmap v3415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:45 smithi183 ceph-mon[37273]: pgmap v3415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:45 smithi144 ceph-mon[32129]: pgmap v3415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2d50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a805a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2d50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000f80 tx=0x7f74a805a040 comp rx=0 tx=0).stop 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2d50 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2d50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.109+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.110+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.111+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f74a80316d0 tx=0x7f74a8061000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.564+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:46.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:47.052+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:35:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:47.052+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:35:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:47.053+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:35:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:35:47.053+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:35:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5389.648343 secs 2024-07-26T12:35:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:47 smithi160 ceph-mon[36715]: pgmap v3416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5389.648343 secs 2024-07-26T12:35:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:47 smithi183 ceph-mon[37273]: pgmap v3416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5389.648343 secs 2024-07-26T12:35:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:47 smithi144 ceph-mon[32129]: pgmap v3416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:49 smithi160 ceph-mon[36715]: pgmap v3417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:49 smithi183 ceph-mon[37273]: pgmap v3417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:49 smithi144 ceph-mon[32129]: pgmap v3417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:51 smithi160 ceph-mon[36715]: pgmap v3418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:51 smithi183 ceph-mon[37273]: pgmap v3418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:51 smithi144 ceph-mon[32129]: pgmap v3418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5394.648465 secs 2024-07-26T12:35:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5394.648465 secs 2024-07-26T12:35:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5394.648465 secs 2024-07-26T12:35:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:53 smithi183 ceph-mon[37273]: pgmap v3419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:53 smithi144 ceph-mon[32129]: pgmap v3419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:53 smithi160 ceph-mon[36715]: pgmap v3419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:35:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:55 smithi183 ceph-mon[37273]: pgmap v3420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:55 smithi144 ceph-mon[32129]: pgmap v3420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:55 smithi160 ceph-mon[36715]: pgmap v3420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5399.648557 secs 2024-07-26T12:35:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:57 smithi183 ceph-mon[37273]: pgmap v3421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5399.648557 secs 2024-07-26T12:35:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:57 smithi144 ceph-mon[32129]: pgmap v3421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5399.648557 secs 2024-07-26T12:35:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:57 smithi160 ceph-mon[36715]: pgmap v3421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:35:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:35:59 smithi183 ceph-mon[37273]: pgmap v3422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:35:59 smithi144 ceph-mon[32129]: pgmap v3422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:35:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:35:59 smithi160 ceph-mon[36715]: pgmap v3422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:01 smithi183 ceph-mon[37273]: pgmap v3423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:01 smithi144 ceph-mon[32129]: pgmap v3423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:01 smithi160 ceph-mon[36715]: pgmap v3423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:01.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:01.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:01.582+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:01.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:01.582+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:01.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:01.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:01.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:01.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5404.648653 secs 2024-07-26T12:36:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5404.648653 secs 2024-07-26T12:36:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5404.648653 secs 2024-07-26T12:36:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:03 smithi183 ceph-mon[37273]: pgmap v3424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:03.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:03 smithi160 ceph-mon[36715]: pgmap v3424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:03 smithi144 ceph-mon[32129]: pgmap v3424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:05 smithi183 ceph-mon[37273]: pgmap v3425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:05 smithi144 ceph-mon[32129]: pgmap v3425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:05 smithi160 ceph-mon[36715]: pgmap v3425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5409.648794 secs 2024-07-26T12:36:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:07 smithi183 ceph-mon[37273]: pgmap v3426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5409.648794 secs 2024-07-26T12:36:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:07 smithi144 ceph-mon[32129]: pgmap v3426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5409.648794 secs 2024-07-26T12:36:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:07 smithi160 ceph-mon[36715]: pgmap v3426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:09 smithi183 ceph-mon[37273]: pgmap v3427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:09 smithi144 ceph-mon[32129]: pgmap v3427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:09 smithi160 ceph-mon[36715]: pgmap v3427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:11 smithi183 ceph-mon[37273]: pgmap v3428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:11 smithi144 ceph-mon[32129]: pgmap v3428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:11 smithi160 ceph-mon[36715]: pgmap v3428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5414.648910 secs 2024-07-26T12:36:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5414.648910 secs 2024-07-26T12:36:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5414.648910 secs 2024-07-26T12:36:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:13 smithi144 ceph-mon[32129]: pgmap v3429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:13 smithi160 ceph-mon[36715]: pgmap v3429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:13 smithi183 ceph-mon[37273]: pgmap v3429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:15 smithi144 ceph-mon[32129]: pgmap v3430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:15 smithi160 ceph-mon[36715]: pgmap v3430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:15 smithi183 ceph-mon[37273]: pgmap v3430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:16.585+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:16.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:16.586+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:16.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:16.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5419.649033 secs 2024-07-26T12:36:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:17 smithi144 ceph-mon[32129]: pgmap v3431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5419.649033 secs 2024-07-26T12:36:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:17 smithi160 ceph-mon[36715]: pgmap v3431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5419.649033 secs 2024-07-26T12:36:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:17 smithi183 ceph-mon[37273]: pgmap v3431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:19 smithi144 ceph-mon[32129]: pgmap v3432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:19 smithi160 ceph-mon[36715]: pgmap v3432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:19 smithi183 ceph-mon[37273]: pgmap v3432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:21 smithi144 ceph-mon[32129]: pgmap v3433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:21 smithi160 ceph-mon[36715]: pgmap v3433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:21 smithi183 ceph-mon[37273]: pgmap v3433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5424.649151 secs 2024-07-26T12:36:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5424.649151 secs 2024-07-26T12:36:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5424.649151 secs 2024-07-26T12:36:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:23 smithi144 ceph-mon[32129]: pgmap v3434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:23 smithi160 ceph-mon[36715]: pgmap v3434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:23 smithi183 ceph-mon[37273]: pgmap v3434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:25 smithi144 ceph-mon[32129]: pgmap v3435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:25 smithi160 ceph-mon[36715]: pgmap v3435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:25 smithi183 ceph-mon[37273]: pgmap v3435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5429.649226 secs 2024-07-26T12:36:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:27 smithi144 ceph-mon[32129]: pgmap v3436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5429.649226 secs 2024-07-26T12:36:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:27 smithi160 ceph-mon[36715]: pgmap v3436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5429.649226 secs 2024-07-26T12:36:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:27 smithi183 ceph-mon[37273]: pgmap v3436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:29 smithi144 ceph-mon[32129]: pgmap v3437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:36:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:29 smithi160 ceph-mon[36715]: pgmap v3437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:36:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:29 smithi183 ceph-mon[37273]: pgmap v3437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:36:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:30 smithi160 ceph-mon[36715]: pgmap v3438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:30 smithi160 ceph-mon[36715]: pgmap v3439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:36:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:30 smithi183 ceph-mon[37273]: pgmap v3438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:30 smithi183 ceph-mon[37273]: pgmap v3439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:36:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:30 smithi144 ceph-mon[32129]: pgmap v3438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:30 smithi144 ceph-mon[32129]: pgmap v3439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:36:31.603 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:31.602+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:31.603+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:31.603+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:31.603+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:31.603+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:32.060+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:36:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:32.060+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:36:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:32.061+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:36:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:32.061+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:36:32.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5434.649332 secs 2024-07-26T12:36:32.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:32 smithi144 ceph-mon[32129]: pgmap v3440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5434.649332 secs 2024-07-26T12:36:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:32 smithi160 ceph-mon[36715]: pgmap v3440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5434.649332 secs 2024-07-26T12:36:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:32 smithi183 ceph-mon[37273]: pgmap v3440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:34 smithi160 ceph-mon[36715]: pgmap v3441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:36:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:34 smithi183 ceph-mon[37273]: pgmap v3441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:36:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:34 smithi144 ceph-mon[32129]: pgmap v3441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:36:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:36 smithi160 ceph-mon[36715]: pgmap v3442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:36:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:36 smithi183 ceph-mon[37273]: pgmap v3442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:36:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:36 smithi144 ceph-mon[32129]: pgmap v3442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:36:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5439.649425 secs 2024-07-26T12:36:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5439.649425 secs 2024-07-26T12:36:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5439.649425 secs 2024-07-26T12:36:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:38 smithi160 ceph-mon[36715]: pgmap v3443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:38 smithi183 ceph-mon[37273]: pgmap v3443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:38 smithi144 ceph-mon[32129]: pgmap v3443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:36:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:40 smithi160 ceph-mon[36715]: pgmap v3444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:36:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:40 smithi183 ceph-mon[37273]: pgmap v3444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:36:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:40 smithi144 ceph-mon[32129]: pgmap v3444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:36:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5444.649538 secs 2024-07-26T12:36:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:42 smithi160 ceph-mon[36715]: pgmap v3445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5444.649538 secs 2024-07-26T12:36:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:42 smithi183 ceph-mon[37273]: pgmap v3445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5444.649538 secs 2024-07-26T12:36:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:42 smithi144 ceph-mon[32129]: pgmap v3445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:44 smithi160 ceph-mon[36715]: pgmap v3446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:44 smithi183 ceph-mon[37273]: pgmap v3446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:44 smithi144 ceph-mon[32129]: pgmap v3446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:46 smithi160 ceph-mon[36715]: pgmap v3447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:46.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:46.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:46.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:46.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:36:46.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:46 smithi183 ceph-mon[37273]: pgmap v3447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:46 smithi144 ceph-mon[32129]: pgmap v3447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5449.649654 secs 2024-07-26T12:36:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5449.649654 secs 2024-07-26T12:36:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5449.649654 secs 2024-07-26T12:36:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:48 smithi144 ceph-mon[32129]: pgmap v3448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:48 smithi160 ceph-mon[36715]: pgmap v3448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:48 smithi183 ceph-mon[37273]: pgmap v3448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:50 smithi160 ceph-mon[36715]: pgmap v3449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:50 smithi183 ceph-mon[37273]: pgmap v3449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:50 smithi144 ceph-mon[32129]: pgmap v3449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5454.649755 secs 2024-07-26T12:36:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:52 smithi160 ceph-mon[36715]: pgmap v3450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5454.649755 secs 2024-07-26T12:36:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:52 smithi183 ceph-mon[37273]: pgmap v3450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5454.649755 secs 2024-07-26T12:36:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:52 smithi144 ceph-mon[32129]: pgmap v3450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:54 smithi160 ceph-mon[36715]: pgmap v3451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:54 smithi183 ceph-mon[37273]: pgmap v3451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:54 smithi144 ceph-mon[32129]: pgmap v3451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:36:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:56 smithi183 ceph-mon[37273]: pgmap v3452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:56 smithi144 ceph-mon[32129]: pgmap v3452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:56 smithi160 ceph-mon[36715]: pgmap v3452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:36:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5459.649896 secs 2024-07-26T12:36:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5459.649896 secs 2024-07-26T12:36:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5459.649896 secs 2024-07-26T12:36:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:36:58 smithi183 ceph-mon[37273]: pgmap v3453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:36:58 smithi144 ceph-mon[32129]: pgmap v3453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:36:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:36:58 smithi160 ceph-mon[36715]: pgmap v3453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:00 smithi183 ceph-mon[37273]: pgmap v3454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:00 smithi144 ceph-mon[32129]: pgmap v3454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:00 smithi160 ceph-mon[36715]: pgmap v3454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:01.621+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:01.622+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:01.622+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:01.622+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:01.622+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5464.650015 secs 2024-07-26T12:37:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:02 smithi183 ceph-mon[37273]: pgmap v3455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5464.650015 secs 2024-07-26T12:37:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:02 smithi144 ceph-mon[32129]: pgmap v3455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5464.650015 secs 2024-07-26T12:37:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:02 smithi160 ceph-mon[36715]: pgmap v3455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:04.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:04 smithi160 ceph-mon[36715]: pgmap v3456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:04 smithi183 ceph-mon[37273]: pgmap v3456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:04 smithi144 ceph-mon[32129]: pgmap v3456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:06 smithi183 ceph-mon[37273]: pgmap v3457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:06 smithi144 ceph-mon[32129]: pgmap v3457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:06 smithi160 ceph-mon[36715]: pgmap v3457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5469.650071 secs 2024-07-26T12:37:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5469.650071 secs 2024-07-26T12:37:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5469.650071 secs 2024-07-26T12:37:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:08 smithi183 ceph-mon[37273]: pgmap v3458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:08 smithi144 ceph-mon[32129]: pgmap v3458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:08 smithi160 ceph-mon[36715]: pgmap v3458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:10 smithi183 ceph-mon[37273]: pgmap v3459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:10 smithi144 ceph-mon[32129]: pgmap v3459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:10 smithi160 ceph-mon[36715]: pgmap v3459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:11 smithi183 ceph-mon[37273]: pgmap v3460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:11 smithi144 ceph-mon[32129]: pgmap v3460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:11 smithi160 ceph-mon[36715]: pgmap v3460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5474.650206 secs 2024-07-26T12:37:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5474.650206 secs 2024-07-26T12:37:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5474.650206 secs 2024-07-26T12:37:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:13 smithi144 ceph-mon[32129]: pgmap v3461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:13 smithi160 ceph-mon[36715]: pgmap v3461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:13 smithi183 ceph-mon[37273]: pgmap v3461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:16 smithi160 ceph-mon[36715]: pgmap v3462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:16 smithi183 ceph-mon[37273]: pgmap v3462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:16.423 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:16 smithi144 ceph-mon[32129]: pgmap v3462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:16.629+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:16.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:16.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:16.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:16.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5479.650333 secs 2024-07-26T12:37:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5479.650333 secs 2024-07-26T12:37:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5479.650333 secs 2024-07-26T12:37:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:18 smithi160 ceph-mon[36715]: pgmap v3463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:18 smithi183 ceph-mon[37273]: pgmap v3463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:18 smithi144 ceph-mon[32129]: pgmap v3463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:20 smithi160 ceph-mon[36715]: pgmap v3464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:20 smithi183 ceph-mon[37273]: pgmap v3464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:20 smithi144 ceph-mon[32129]: pgmap v3464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5484.650425 secs 2024-07-26T12:37:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:22 smithi160 ceph-mon[36715]: pgmap v3465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5484.650425 secs 2024-07-26T12:37:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:22 smithi183 ceph-mon[37273]: pgmap v3465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5484.650425 secs 2024-07-26T12:37:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:22 smithi144 ceph-mon[32129]: pgmap v3465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:24 smithi160 ceph-mon[36715]: pgmap v3466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:24 smithi183 ceph-mon[37273]: pgmap v3466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:24 smithi144 ceph-mon[32129]: pgmap v3466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:26 smithi160 ceph-mon[36715]: pgmap v3467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:26 smithi183 ceph-mon[37273]: pgmap v3467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:26 smithi144 ceph-mon[32129]: pgmap v3467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5489.650543 secs 2024-07-26T12:37:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5489.650543 secs 2024-07-26T12:37:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5489.650543 secs 2024-07-26T12:37:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:28 smithi183 ceph-mon[37273]: pgmap v3468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:28 smithi144 ceph-mon[32129]: pgmap v3468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:28 smithi160 ceph-mon[36715]: pgmap v3468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:30 smithi183 ceph-mon[37273]: pgmap v3469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:30 smithi144 ceph-mon[32129]: pgmap v3469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:30.520 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:30 smithi160 ceph-mon[36715]: pgmap v3469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:31 smithi183 ceph-mon[37273]: pgmap v3470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:37:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:37:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:31 smithi144 ceph-mon[32129]: pgmap v3470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:37:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:37:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:31 smithi160 ceph-mon[36715]: pgmap v3470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:37:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:37:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:31.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:31.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:31.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:31.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:31.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5494.650633 secs 2024-07-26T12:37:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5494.650633 secs 2024-07-26T12:37:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5494.650633 secs 2024-07-26T12:37:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:33 smithi183 ceph-mon[37273]: pgmap v3471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:37:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:33 smithi144 ceph-mon[32129]: pgmap v3471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:37:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:33 smithi160 ceph-mon[36715]: pgmap v3471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:37:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:35 smithi144 ceph-mon[32129]: pgmap v3472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:37:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:35 smithi160 ceph-mon[36715]: pgmap v3472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:37:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:35 smithi183 ceph-mon[37273]: pgmap v3472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:37:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5499.650753 secs 2024-07-26T12:37:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:37 smithi144 ceph-mon[32129]: pgmap v3473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:37:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5499.650753 secs 2024-07-26T12:37:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:37 smithi160 ceph-mon[36715]: pgmap v3473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:37:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5499.650753 secs 2024-07-26T12:37:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:37 smithi183 ceph-mon[37273]: pgmap v3473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:37:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:39 smithi144 ceph-mon[32129]: pgmap v3474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:37:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:39 smithi160 ceph-mon[36715]: pgmap v3474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:37:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:39 smithi183 ceph-mon[37273]: pgmap v3474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:37:41.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:41 smithi183 ceph-mon[37273]: pgmap v3475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:37:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:41 smithi144 ceph-mon[32129]: pgmap v3475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:37:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:41 smithi160 ceph-mon[36715]: pgmap v3475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:37:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5504.650884 secs 2024-07-26T12:37:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5504.650884 secs 2024-07-26T12:37:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5504.650884 secs 2024-07-26T12:37:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:43 smithi144 ceph-mon[32129]: pgmap v3476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:43 smithi160 ceph-mon[36715]: pgmap v3476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:43 smithi183 ceph-mon[37273]: pgmap v3476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:45 smithi144 ceph-mon[32129]: pgmap v3477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:45 smithi160 ceph-mon[36715]: pgmap v3477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:45 smithi183 ceph-mon[37273]: pgmap v3477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:46.645+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:46.646+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:46.646+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:46.646+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:37:46.646+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5509.651028 secs 2024-07-26T12:37:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:47 smithi144 ceph-mon[32129]: pgmap v3478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5509.651028 secs 2024-07-26T12:37:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:47 smithi160 ceph-mon[36715]: pgmap v3478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5509.651028 secs 2024-07-26T12:37:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:47 smithi183 ceph-mon[37273]: pgmap v3478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:49 smithi144 ceph-mon[32129]: pgmap v3479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:49 smithi160 ceph-mon[36715]: pgmap v3479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:49 smithi183 ceph-mon[37273]: pgmap v3479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:51 smithi160 ceph-mon[36715]: pgmap v3480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:51 smithi183 ceph-mon[37273]: pgmap v3480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:51 smithi144 ceph-mon[32129]: pgmap v3480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5514.651145 secs 2024-07-26T12:37:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5514.651145 secs 2024-07-26T12:37:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5514.651145 secs 2024-07-26T12:37:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:53 smithi160 ceph-mon[36715]: pgmap v3481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:53 smithi183 ceph-mon[37273]: pgmap v3481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:53.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:53 smithi144 ceph-mon[32129]: pgmap v3481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:37:55.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:55 smithi160 ceph-mon[36715]: pgmap v3482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:55 smithi183 ceph-mon[37273]: pgmap v3482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:55 smithi144 ceph-mon[32129]: pgmap v3482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5519.651232 secs 2024-07-26T12:37:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:57 smithi160 ceph-mon[36715]: pgmap v3483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5519.651232 secs 2024-07-26T12:37:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:57 smithi183 ceph-mon[37273]: pgmap v3483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5519.651232 secs 2024-07-26T12:37:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:57 smithi144 ceph-mon[32129]: pgmap v3483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:37:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:37:59 smithi160 ceph-mon[36715]: pgmap v3484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:37:59 smithi183 ceph-mon[37273]: pgmap v3484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:37:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:37:59 smithi144 ceph-mon[32129]: pgmap v3484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:01 smithi160 ceph-mon[36715]: pgmap v3485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:01 smithi183 ceph-mon[37273]: pgmap v3485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:01.662+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:01.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:01.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:01.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:01.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:01 smithi144 ceph-mon[32129]: pgmap v3485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5524.651373 secs 2024-07-26T12:38:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5524.651373 secs 2024-07-26T12:38:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5524.651373 secs 2024-07-26T12:38:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:03 smithi160 ceph-mon[36715]: pgmap v3486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:03 smithi183 ceph-mon[37273]: pgmap v3486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:03 smithi144 ceph-mon[32129]: pgmap v3486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:05 smithi160 ceph-mon[36715]: pgmap v3487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:05 smithi183 ceph-mon[37273]: pgmap v3487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:05 smithi144 ceph-mon[32129]: pgmap v3487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5529.651492 secs 2024-07-26T12:38:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:07 smithi160 ceph-mon[36715]: pgmap v3488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5529.651492 secs 2024-07-26T12:38:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:07 smithi183 ceph-mon[37273]: pgmap v3488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5529.651492 secs 2024-07-26T12:38:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:07 smithi144 ceph-mon[32129]: pgmap v3488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:09 smithi160 ceph-mon[36715]: pgmap v3489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:09 smithi183 ceph-mon[37273]: pgmap v3489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:09 smithi144 ceph-mon[32129]: pgmap v3489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:11 smithi160 ceph-mon[36715]: pgmap v3490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:11 smithi183 ceph-mon[37273]: pgmap v3490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:11 smithi144 ceph-mon[32129]: pgmap v3490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5534.651568 secs 2024-07-26T12:38:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5534.651568 secs 2024-07-26T12:38:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5534.651568 secs 2024-07-26T12:38:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:13 smithi183 ceph-mon[37273]: pgmap v3491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:13 smithi144 ceph-mon[32129]: pgmap v3491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:13 smithi160 ceph-mon[36715]: pgmap v3491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:15 smithi183 ceph-mon[37273]: pgmap v3492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:15 smithi144 ceph-mon[32129]: pgmap v3492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:15 smithi160 ceph-mon[36715]: pgmap v3492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:16.677+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:16.679+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:16.679+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:16.679+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:16.679+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5539.651709 secs 2024-07-26T12:38:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:17 smithi183 ceph-mon[37273]: pgmap v3493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5539.651709 secs 2024-07-26T12:38:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:17 smithi144 ceph-mon[32129]: pgmap v3493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5539.651709 secs 2024-07-26T12:38:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:17 smithi160 ceph-mon[36715]: pgmap v3493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:19 smithi183 ceph-mon[37273]: pgmap v3494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:19 smithi144 ceph-mon[32129]: pgmap v3494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:19 smithi160 ceph-mon[36715]: pgmap v3494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:21 smithi183 ceph-mon[37273]: pgmap v3495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:21 smithi144 ceph-mon[32129]: pgmap v3495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:21 smithi160 ceph-mon[36715]: pgmap v3495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5544.651826 secs 2024-07-26T12:38:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5544.651826 secs 2024-07-26T12:38:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5544.651826 secs 2024-07-26T12:38:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:23 smithi183 ceph-mon[37273]: pgmap v3496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:23 smithi144 ceph-mon[32129]: pgmap v3496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:23 smithi160 ceph-mon[36715]: pgmap v3496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:25 smithi183 ceph-mon[37273]: pgmap v3497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:25 smithi144 ceph-mon[32129]: pgmap v3497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:25 smithi160 ceph-mon[36715]: pgmap v3497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5549.651947 secs 2024-07-26T12:38:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:27 smithi183 ceph-mon[37273]: pgmap v3498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5549.651947 secs 2024-07-26T12:38:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:27 smithi144 ceph-mon[32129]: pgmap v3498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5549.651947 secs 2024-07-26T12:38:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:27 smithi160 ceph-mon[36715]: pgmap v3498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:29 smithi183 ceph-mon[37273]: pgmap v3499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:29 smithi144 ceph-mon[32129]: pgmap v3499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:29 smithi160 ceph-mon[36715]: pgmap v3499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: pgmap v3500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:31.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:38:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:31.694+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:31.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:31.695+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:31.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:31.695+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:31.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:31.695+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:31.697 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:31.695+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: pgmap v3500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: pgmap v3500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:38:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5554.652014 secs 2024-07-26T12:38:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:32 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:38:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:32 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:38:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:32 smithi144 ceph-mon[32129]: pgmap v3501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:32 smithi144 ceph-mon[32129]: pgmap v3502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5554.652014 secs 2024-07-26T12:38:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:32 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:38:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:32 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:38:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:32 smithi160 ceph-mon[36715]: pgmap v3501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:32 smithi160 ceph-mon[36715]: pgmap v3502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5554.652014 secs 2024-07-26T12:38:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:32 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:38:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:32 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:38:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:32 smithi183 ceph-mon[37273]: pgmap v3501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:32 smithi183 ceph-mon[37273]: pgmap v3502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:33 smithi144 ceph-mon[32129]: pgmap v3503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:38:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:33 smithi160 ceph-mon[36715]: pgmap v3503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:38:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:33 smithi183 ceph-mon[37273]: pgmap v3503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:38:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:36 smithi144 ceph-mon[32129]: pgmap v3504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:36 smithi183 ceph-mon[37273]: pgmap v3504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:36 smithi160 ceph-mon[36715]: pgmap v3504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5559.652160 secs 2024-07-26T12:38:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5559.652160 secs 2024-07-26T12:38:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5559.652160 secs 2024-07-26T12:38:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:38 smithi183 ceph-mon[37273]: pgmap v3505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:38 smithi144 ceph-mon[32129]: pgmap v3505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:38 smithi160 ceph-mon[36715]: pgmap v3505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:40 smithi144 ceph-mon[32129]: pgmap v3506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:40 smithi160 ceph-mon[36715]: pgmap v3506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:40 smithi183 ceph-mon[37273]: pgmap v3506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:38:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:41 smithi144 ceph-mon[32129]: pgmap v3507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:38:41.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:41 smithi183 ceph-mon[37273]: pgmap v3507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:38:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:41 smithi160 ceph-mon[36715]: pgmap v3507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:38:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5564.652280 secs 2024-07-26T12:38:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5564.652280 secs 2024-07-26T12:38:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5564.652280 secs 2024-07-26T12:38:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:43 smithi144 ceph-mon[32129]: pgmap v3508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:43 smithi160 ceph-mon[36715]: pgmap v3508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:43 smithi183 ceph-mon[37273]: pgmap v3508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:46 smithi183 ceph-mon[37273]: pgmap v3509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:46.713 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:46.711+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:46.713 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:46.712+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:46.713 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:46.712+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:46.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:46.712+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:46.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:46.712+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:46 smithi144 ceph-mon[32129]: pgmap v3509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:46 smithi160 ceph-mon[36715]: pgmap v3509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:47.084+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:38:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:47.084+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:38:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:47.085+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T12:38:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:38:47.085+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T12:38:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5569.652396 secs 2024-07-26T12:38:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5569.652396 secs 2024-07-26T12:38:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5569.652396 secs 2024-07-26T12:38:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:48 smithi144 ceph-mon[32129]: pgmap v3510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:48 smithi160 ceph-mon[36715]: pgmap v3510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:48 smithi183 ceph-mon[37273]: pgmap v3510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:49 smithi144 ceph-mon[32129]: pgmap v3511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:49 smithi160 ceph-mon[36715]: pgmap v3511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:49 smithi183 ceph-mon[37273]: pgmap v3511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5574.652486 secs 2024-07-26T12:38:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:52 smithi183 ceph-mon[37273]: pgmap v3512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5574.652486 secs 2024-07-26T12:38:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:52 smithi144 ceph-mon[32129]: pgmap v3512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5574.652486 secs 2024-07-26T12:38:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:52 smithi160 ceph-mon[36715]: pgmap v3512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:38:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:54 smithi183 ceph-mon[37273]: pgmap v3513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:54 smithi144 ceph-mon[32129]: pgmap v3513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:54 smithi160 ceph-mon[36715]: pgmap v3513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:56 smithi144 ceph-mon[32129]: pgmap v3514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:56 smithi160 ceph-mon[36715]: pgmap v3514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:56 smithi183 ceph-mon[37273]: pgmap v3514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:38:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5579.652623 secs 2024-07-26T12:38:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5579.652623 secs 2024-07-26T12:38:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5579.652623 secs 2024-07-26T12:38:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:58 smithi144 ceph-mon[32129]: pgmap v3515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:58 smithi160 ceph-mon[36715]: pgmap v3515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:58 smithi183 ceph-mon[37273]: pgmap v3515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:38:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:38:59 smithi144 ceph-mon[32129]: pgmap v3516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:38:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:38:59 smithi160 ceph-mon[36715]: pgmap v3516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:38:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:38:59 smithi183 ceph-mon[37273]: pgmap v3516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:39:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:01.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:01.721+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:01.721+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:01.721+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:01.721+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5584.652746 secs 2024-07-26T12:39:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:02 smithi144 ceph-mon[32129]: pgmap v3517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5584.652746 secs 2024-07-26T12:39:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:02 smithi160 ceph-mon[36715]: pgmap v3517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5584.652746 secs 2024-07-26T12:39:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:02 smithi183 ceph-mon[37273]: pgmap v3517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:04 smithi144 ceph-mon[32129]: pgmap v3518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:04 smithi160 ceph-mon[36715]: pgmap v3518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:04 smithi183 ceph-mon[37273]: pgmap v3518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:06 smithi144 ceph-mon[32129]: pgmap v3519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:06 smithi160 ceph-mon[36715]: pgmap v3519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:06 smithi183 ceph-mon[37273]: pgmap v3519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5589.652864 secs 2024-07-26T12:39:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:07 smithi144 ceph-mon[32129]: pgmap v3520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5589.652864 secs 2024-07-26T12:39:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:07 smithi160 ceph-mon[36715]: pgmap v3520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5589.652864 secs 2024-07-26T12:39:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:07 smithi183 ceph-mon[37273]: pgmap v3520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:10 smithi144 ceph-mon[32129]: pgmap v3521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:39:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:10 smithi160 ceph-mon[36715]: pgmap v3521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:39:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:10 smithi183 ceph-mon[37273]: pgmap v3521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:39:12.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5594.652962 secs 2024-07-26T12:39:12.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:12 smithi183 ceph-mon[37273]: pgmap v3522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5594.652962 secs 2024-07-26T12:39:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:12 smithi144 ceph-mon[32129]: pgmap v3522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5594.652962 secs 2024-07-26T12:39:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:12 smithi160 ceph-mon[36715]: pgmap v3522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:14 smithi183 ceph-mon[37273]: pgmap v3523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:14 smithi144 ceph-mon[32129]: pgmap v3523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:14 smithi160 ceph-mon[36715]: pgmap v3523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:16 smithi144 ceph-mon[32129]: pgmap v3524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:16.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:16.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:16.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:16.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:16.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:16 smithi160 ceph-mon[36715]: pgmap v3524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:16 smithi183 ceph-mon[37273]: pgmap v3524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5599.653063 secs 2024-07-26T12:39:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5599.653063 secs 2024-07-26T12:39:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5599.653063 secs 2024-07-26T12:39:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:18 smithi144 ceph-mon[32129]: pgmap v3525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:18 smithi160 ceph-mon[36715]: pgmap v3525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:18 smithi183 ceph-mon[37273]: pgmap v3525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:19 smithi144 ceph-mon[32129]: pgmap v3526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:19 smithi160 ceph-mon[36715]: pgmap v3526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:19 smithi183 ceph-mon[37273]: pgmap v3526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5604.653178 secs 2024-07-26T12:39:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:22 smithi144 ceph-mon[32129]: pgmap v3527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5604.653178 secs 2024-07-26T12:39:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:22 smithi160 ceph-mon[36715]: pgmap v3527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5604.653178 secs 2024-07-26T12:39:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:22 smithi183 ceph-mon[37273]: pgmap v3527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:24 smithi183 ceph-mon[37273]: pgmap v3528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:24 smithi144 ceph-mon[32129]: pgmap v3528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:24 smithi160 ceph-mon[36715]: pgmap v3528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:26 smithi144 ceph-mon[32129]: pgmap v3529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:26 smithi160 ceph-mon[36715]: pgmap v3529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:26 smithi183 ceph-mon[37273]: pgmap v3529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5609.653342 secs 2024-07-26T12:39:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:27 smithi144 ceph-mon[32129]: pgmap v3530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5609.653342 secs 2024-07-26T12:39:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:27 smithi160 ceph-mon[36715]: pgmap v3530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5609.653342 secs 2024-07-26T12:39:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:27 smithi183 ceph-mon[37273]: pgmap v3530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:30 smithi144 ceph-mon[32129]: pgmap v3531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:30 smithi160 ceph-mon[36715]: pgmap v3531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:30 smithi183 ceph-mon[37273]: pgmap v3531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:31.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:31 smithi144 ceph-mon[32129]: pgmap v3532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:31.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:31.751+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:31.753 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:31.752+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:31.753 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:31.752+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:31.753 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:31.752+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:31.753 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:31.752+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:31 smithi160 ceph-mon[36715]: pgmap v3532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:31 smithi183 ceph-mon[37273]: pgmap v3532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5614.653483 secs 2024-07-26T12:39:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:39:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:39:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:39:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:32 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:39:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5614.653483 secs 2024-07-26T12:39:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:39:32.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:39:32.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:39:32.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:32 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:39:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5614.653483 secs 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:39:32.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:32 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:39:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:33 smithi144 ceph-mon[32129]: pgmap v3533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:39:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:33 smithi144 ceph-mon[32129]: pgmap v3534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:39:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:33 smithi160 ceph-mon[36715]: pgmap v3533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:39:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:33 smithi160 ceph-mon[36715]: pgmap v3534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:39:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:33 smithi183 ceph-mon[37273]: pgmap v3533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:39:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:33 smithi183 ceph-mon[37273]: pgmap v3534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:39:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:35 smithi160 ceph-mon[36715]: pgmap v3535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:35 smithi183 ceph-mon[37273]: pgmap v3535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:35 smithi144 ceph-mon[32129]: pgmap v3535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:39:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5619.653605 secs 2024-07-26T12:39:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5619.653605 secs 2024-07-26T12:39:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5619.653605 secs 2024-07-26T12:39:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:37 smithi183 ceph-mon[37273]: pgmap v3536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:39:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:37 smithi144 ceph-mon[32129]: pgmap v3536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:39:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:37 smithi160 ceph-mon[36715]: pgmap v3536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:39:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:39 smithi183 ceph-mon[37273]: pgmap v3537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:39:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:39 smithi144 ceph-mon[32129]: pgmap v3537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:39:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:39 smithi160 ceph-mon[36715]: pgmap v3537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:39:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:41 smithi183 ceph-mon[37273]: pgmap v3538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:39:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5624.653724 secs 2024-07-26T12:39:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:41 smithi144 ceph-mon[32129]: pgmap v3538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:39:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5624.653724 secs 2024-07-26T12:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:41 smithi160 ceph-mon[36715]: pgmap v3538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5624.653724 secs 2024-07-26T12:39:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:43 smithi183 ceph-mon[37273]: pgmap v3539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:43 smithi144 ceph-mon[32129]: pgmap v3539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:43 smithi160 ceph-mon[36715]: pgmap v3539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:39:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:45 smithi183 ceph-mon[37273]: pgmap v3540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:45 smithi144 ceph-mon[32129]: pgmap v3540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:45 smithi160 ceph-mon[36715]: pgmap v3540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:46.756+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:46.757+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:46.757+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:46.757+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:39:46.757+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5629.653866 secs 2024-07-26T12:39:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5629.653866 secs 2024-07-26T12:39:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5629.653866 secs 2024-07-26T12:39:47.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:47 smithi144 ceph-mon[32129]: pgmap v3541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:47 smithi183 ceph-mon[37273]: pgmap v3541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:47 smithi160 ceph-mon[36715]: pgmap v3541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:49 smithi183 ceph-mon[37273]: pgmap v3542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:49 smithi144 ceph-mon[32129]: pgmap v3542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:49 smithi160 ceph-mon[36715]: pgmap v3542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:51 smithi183 ceph-mon[37273]: pgmap v3543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5634.654001 secs 2024-07-26T12:39:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:51 smithi144 ceph-mon[32129]: pgmap v3543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5634.654001 secs 2024-07-26T12:39:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:51 smithi160 ceph-mon[36715]: pgmap v3543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5634.654001 secs 2024-07-26T12:39:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:53 smithi183 ceph-mon[37273]: pgmap v3544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:53 smithi144 ceph-mon[32129]: pgmap v3544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:53 smithi160 ceph-mon[36715]: pgmap v3544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:39:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:55 smithi183 ceph-mon[37273]: pgmap v3545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:55 smithi144 ceph-mon[32129]: pgmap v3545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:55 smithi160 ceph-mon[36715]: pgmap v3545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:39:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5639.654144 secs 2024-07-26T12:39:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5639.654144 secs 2024-07-26T12:39:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5639.654144 secs 2024-07-26T12:39:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:57 smithi144 ceph-mon[32129]: pgmap v3546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:57 smithi160 ceph-mon[36715]: pgmap v3546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:57 smithi183 ceph-mon[37273]: pgmap v3546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:39:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:39:59 smithi144 ceph-mon[32129]: pgmap v3547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:39:59 smithi160 ceph-mon[36715]: pgmap v3547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:39:59 smithi183 ceph-mon[37273]: pgmap v3547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:01.773+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:01.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:01.774+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:01.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:01.774+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:01.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:01.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:01.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:01.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:01 smithi144 ceph-mon[32129]: pgmap v3548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5644.654262 secs 2024-07-26T12:40:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:01 smithi160 ceph-mon[36715]: pgmap v3548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5644.654262 secs 2024-07-26T12:40:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:01 smithi183 ceph-mon[37273]: pgmap v3548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5644.654262 secs 2024-07-26T12:40:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:03 smithi144 ceph-mon[32129]: pgmap v3549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:03 smithi160 ceph-mon[36715]: pgmap v3549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:03 smithi183 ceph-mon[37273]: pgmap v3549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:05 smithi144 ceph-mon[32129]: pgmap v3550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:05 smithi160 ceph-mon[36715]: pgmap v3550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:05 smithi183 ceph-mon[37273]: pgmap v3550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:06.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5649.654314 secs 2024-07-26T12:40:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5649.654314 secs 2024-07-26T12:40:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5649.654314 secs 2024-07-26T12:40:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:07 smithi144 ceph-mon[32129]: pgmap v3551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:07 smithi160 ceph-mon[36715]: pgmap v3551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:07 smithi183 ceph-mon[37273]: pgmap v3551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:09 smithi144 ceph-mon[32129]: pgmap v3552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:09 smithi160 ceph-mon[36715]: pgmap v3552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:09 smithi183 ceph-mon[37273]: pgmap v3552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:11 smithi144 ceph-mon[32129]: pgmap v3553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5654.654430 secs 2024-07-26T12:40:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:11 smithi160 ceph-mon[36715]: pgmap v3553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5654.654430 secs 2024-07-26T12:40:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:11 smithi183 ceph-mon[37273]: pgmap v3553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5654.654430 secs 2024-07-26T12:40:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:13 smithi144 ceph-mon[32129]: pgmap v3554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:13 smithi160 ceph-mon[36715]: pgmap v3554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:13 smithi183 ceph-mon[37273]: pgmap v3554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:15 smithi144 ceph-mon[32129]: pgmap v3555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:15 smithi160 ceph-mon[36715]: pgmap v3555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:15 smithi183 ceph-mon[37273]: pgmap v3555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:16.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:16.790+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:16.790+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:16.790+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:16.790+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5659.654559 secs 2024-07-26T12:40:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5659.654559 secs 2024-07-26T12:40:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5659.654559 secs 2024-07-26T12:40:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:17 smithi160 ceph-mon[36715]: pgmap v3556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:18.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:17 smithi144 ceph-mon[32129]: pgmap v3556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:17 smithi183 ceph-mon[37273]: pgmap v3556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:19 smithi160 ceph-mon[36715]: pgmap v3557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:19 smithi183 ceph-mon[37273]: pgmap v3557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:19 smithi144 ceph-mon[32129]: pgmap v3557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:21 smithi160 ceph-mon[36715]: pgmap v3558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5664.654664 secs 2024-07-26T12:40:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:21 smithi183 ceph-mon[37273]: pgmap v3558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5664.654664 secs 2024-07-26T12:40:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:21 smithi144 ceph-mon[32129]: pgmap v3558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5664.654664 secs 2024-07-26T12:40:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:23 smithi160 ceph-mon[36715]: pgmap v3559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:23 smithi183 ceph-mon[37273]: pgmap v3559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:23 smithi144 ceph-mon[32129]: pgmap v3559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:25 smithi160 ceph-mon[36715]: pgmap v3560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:25 smithi183 ceph-mon[37273]: pgmap v3560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:25 smithi144 ceph-mon[32129]: pgmap v3560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5669.654780 secs 2024-07-26T12:40:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5669.654780 secs 2024-07-26T12:40:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5669.654780 secs 2024-07-26T12:40:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:27 smithi160 ceph-mon[36715]: pgmap v3561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:27 smithi183 ceph-mon[37273]: pgmap v3561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:27 smithi144 ceph-mon[32129]: pgmap v3561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:29 smithi160 ceph-mon[36715]: pgmap v3562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:29 smithi183 ceph-mon[37273]: pgmap v3562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:29 smithi144 ceph-mon[32129]: pgmap v3562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:31.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:31.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:31.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:31.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:31.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:31 smithi160 ceph-mon[36715]: pgmap v3563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5674.654882 secs 2024-07-26T12:40:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:31 smithi183 ceph-mon[37273]: pgmap v3563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5674.654882 secs 2024-07-26T12:40:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:31 smithi144 ceph-mon[32129]: pgmap v3563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5674.654882 secs 2024-07-26T12:40:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:33 smithi160 ceph-mon[36715]: pgmap v3564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:40:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:33 smithi183 ceph-mon[37273]: pgmap v3564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:40:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:33 smithi144 ceph-mon[32129]: pgmap v3564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:40:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:34 smithi160 ceph-mon[36715]: pgmap v3565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:34 smithi160 ceph-mon[36715]: pgmap v3566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:40:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:34 smithi183 ceph-mon[37273]: pgmap v3565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:34 smithi183 ceph-mon[37273]: pgmap v3566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:40:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:34 smithi144 ceph-mon[32129]: pgmap v3565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:34 smithi144 ceph-mon[32129]: pgmap v3566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:40:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:36 smithi160 ceph-mon[36715]: pgmap v3567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5679.655011 secs 2024-07-26T12:40:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:36 smithi183 ceph-mon[37273]: pgmap v3567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5679.655011 secs 2024-07-26T12:40:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:36 smithi144 ceph-mon[32129]: pgmap v3567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5679.655011 secs 2024-07-26T12:40:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:38 smithi160 ceph-mon[36715]: pgmap v3568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:40:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:38 smithi183 ceph-mon[37273]: pgmap v3568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:40:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:38 smithi144 ceph-mon[32129]: pgmap v3568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:40:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:40 smithi160 ceph-mon[36715]: pgmap v3569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:40:41.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:40 smithi144 ceph-mon[32129]: pgmap v3569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:40:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:40 smithi183 ceph-mon[37273]: pgmap v3569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:40:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5684.655149 secs 2024-07-26T12:40:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5684.655149 secs 2024-07-26T12:40:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5684.655149 secs 2024-07-26T12:40:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:42 smithi144 ceph-mon[32129]: pgmap v3570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:42 smithi183 ceph-mon[37273]: pgmap v3570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:42 smithi160 ceph-mon[36715]: pgmap v3570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:40:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:44 smithi183 ceph-mon[37273]: pgmap v3571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:40:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:44 smithi144 ceph-mon[32129]: pgmap v3571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:40:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:44 smithi160 ceph-mon[36715]: pgmap v3571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:40:46.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:46.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:46.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:46.824+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:46.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:46.824+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:46.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:46.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:46.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:40:46.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:46 smithi183 ceph-mon[37273]: pgmap v3572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5689.655244 secs 2024-07-26T12:40:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:46 smithi144 ceph-mon[32129]: pgmap v3572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5689.655244 secs 2024-07-26T12:40:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:46 smithi160 ceph-mon[36715]: pgmap v3572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5689.655244 secs 2024-07-26T12:40:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:48 smithi183 ceph-mon[37273]: pgmap v3573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:48 smithi144 ceph-mon[32129]: pgmap v3573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:48 smithi160 ceph-mon[36715]: pgmap v3573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:50 smithi183 ceph-mon[37273]: pgmap v3574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:50 smithi144 ceph-mon[32129]: pgmap v3574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:50 smithi160 ceph-mon[36715]: pgmap v3574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5694.655366 secs 2024-07-26T12:40:52.151 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5694.655366 secs 2024-07-26T12:40:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5694.655366 secs 2024-07-26T12:40:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:52 smithi183 ceph-mon[37273]: pgmap v3575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:52 smithi144 ceph-mon[32129]: pgmap v3575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:52 smithi160 ceph-mon[36715]: pgmap v3575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:40:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:54 smithi183 ceph-mon[37273]: pgmap v3576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:54 smithi144 ceph-mon[32129]: pgmap v3576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:54 smithi160 ceph-mon[36715]: pgmap v3576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5699.655491 secs 2024-07-26T12:40:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:56 smithi183 ceph-mon[37273]: pgmap v3577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5699.655491 secs 2024-07-26T12:40:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:56 smithi144 ceph-mon[32129]: pgmap v3577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5699.655491 secs 2024-07-26T12:40:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:56 smithi160 ceph-mon[36715]: pgmap v3577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:40:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:40:58 smithi183 ceph-mon[37273]: pgmap v3578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:40:58 smithi144 ceph-mon[32129]: pgmap v3578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:40:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:40:58 smithi160 ceph-mon[36715]: pgmap v3578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:00 smithi183 ceph-mon[37273]: pgmap v3579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:00 smithi144 ceph-mon[32129]: pgmap v3579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:00 smithi160 ceph-mon[36715]: pgmap v3579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:01.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:01.836+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:01.837+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:01.837+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:01.837+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:01.837+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5704.655570 secs 2024-07-26T12:41:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5704.655570 secs 2024-07-26T12:41:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5704.655570 secs 2024-07-26T12:41:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:02 smithi144 ceph-mon[32129]: pgmap v3580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:02 smithi160 ceph-mon[36715]: pgmap v3580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:02 smithi183 ceph-mon[37273]: pgmap v3580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:04 smithi144 ceph-mon[32129]: pgmap v3581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:04 smithi160 ceph-mon[36715]: pgmap v3581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:04 smithi183 ceph-mon[37273]: pgmap v3581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5709.655696 secs 2024-07-26T12:41:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:06 smithi144 ceph-mon[32129]: pgmap v3582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5709.655696 secs 2024-07-26T12:41:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:06 smithi160 ceph-mon[36715]: pgmap v3582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5709.655696 secs 2024-07-26T12:41:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:06 smithi183 ceph-mon[37273]: pgmap v3582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:08 smithi144 ceph-mon[32129]: pgmap v3583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:08 smithi160 ceph-mon[36715]: pgmap v3583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:08 smithi183 ceph-mon[37273]: pgmap v3583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:10 smithi144 ceph-mon[32129]: pgmap v3584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:10 smithi160 ceph-mon[36715]: pgmap v3584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:10 smithi183 ceph-mon[37273]: pgmap v3584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:12.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5714.655813 secs 2024-07-26T12:41:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5714.655813 secs 2024-07-26T12:41:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5714.655813 secs 2024-07-26T12:41:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:12 smithi144 ceph-mon[32129]: pgmap v3585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:13.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:12 smithi183 ceph-mon[37273]: pgmap v3585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:12 smithi160 ceph-mon[36715]: pgmap v3585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:14 smithi144 ceph-mon[32129]: pgmap v3586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:14 smithi160 ceph-mon[36715]: pgmap v3586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:14 smithi183 ceph-mon[37273]: pgmap v3586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:16.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:16.847+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:16.847+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:16.847+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:16.847+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5719.655953 secs 2024-07-26T12:41:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:16 smithi144 ceph-mon[32129]: pgmap v3587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5719.655953 secs 2024-07-26T12:41:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:16 smithi160 ceph-mon[36715]: pgmap v3587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5719.655953 secs 2024-07-26T12:41:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:16 smithi183 ceph-mon[37273]: pgmap v3587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:18 smithi144 ceph-mon[32129]: pgmap v3588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:18 smithi160 ceph-mon[36715]: pgmap v3588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:18 smithi183 ceph-mon[37273]: pgmap v3588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:20 smithi144 ceph-mon[32129]: pgmap v3589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:20 smithi160 ceph-mon[36715]: pgmap v3589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:20 smithi183 ceph-mon[37273]: pgmap v3589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5724.656070 secs 2024-07-26T12:41:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5724.656070 secs 2024-07-26T12:41:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5724.656070 secs 2024-07-26T12:41:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:22 smithi144 ceph-mon[32129]: pgmap v3590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:22 smithi160 ceph-mon[36715]: pgmap v3590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:22 smithi183 ceph-mon[37273]: pgmap v3590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:24 smithi160 ceph-mon[36715]: pgmap v3591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:24 smithi183 ceph-mon[37273]: pgmap v3591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:24 smithi144 ceph-mon[32129]: pgmap v3591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5729.656208 secs 2024-07-26T12:41:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:26 smithi160 ceph-mon[36715]: pgmap v3592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5729.656208 secs 2024-07-26T12:41:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:26 smithi183 ceph-mon[37273]: pgmap v3592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5729.656208 secs 2024-07-26T12:41:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:26 smithi144 ceph-mon[32129]: pgmap v3592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:28 smithi160 ceph-mon[36715]: pgmap v3593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:28 smithi183 ceph-mon[37273]: pgmap v3593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:28 smithi144 ceph-mon[32129]: pgmap v3593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:30 smithi160 ceph-mon[36715]: pgmap v3594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:30 smithi183 ceph-mon[37273]: pgmap v3594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:30 smithi144 ceph-mon[32129]: pgmap v3594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:31.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:31.860+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:31.860+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:31.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:31.860+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:31.860+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5734.656332 secs 2024-07-26T12:41:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5734.656332 secs 2024-07-26T12:41:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5734.656332 secs 2024-07-26T12:41:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:32 smithi160 ceph-mon[36715]: pgmap v3595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:32 smithi183 ceph-mon[37273]: pgmap v3595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:32 smithi144 ceph-mon[32129]: pgmap v3595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:41:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:35 smithi160 ceph-mon[36715]: pgmap v3596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:35 smithi183 ceph-mon[37273]: pgmap v3596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:35 smithi144 ceph-mon[32129]: pgmap v3596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:41:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:37 smithi160 ceph-mon[36715]: pgmap v3597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:37 smithi160 ceph-mon[36715]: pgmap v3598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5739.656435 secs 2024-07-26T12:41:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:37 smithi183 ceph-mon[37273]: pgmap v3597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:37 smithi183 ceph-mon[37273]: pgmap v3598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5739.656435 secs 2024-07-26T12:41:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:37 smithi144 ceph-mon[32129]: pgmap v3597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:37 smithi144 ceph-mon[32129]: pgmap v3598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5739.656435 secs 2024-07-26T12:41:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:39 smithi160 ceph-mon[36715]: pgmap v3599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:39 smithi183 ceph-mon[37273]: pgmap v3599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:39 smithi144 ceph-mon[32129]: pgmap v3599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:41 smithi160 ceph-mon[36715]: pgmap v3600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:41 smithi183 ceph-mon[37273]: pgmap v3600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:41 smithi144 ceph-mon[32129]: pgmap v3600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5744.656550 secs 2024-07-26T12:41:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5744.656550 secs 2024-07-26T12:41:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5744.656550 secs 2024-07-26T12:41:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:43 smithi160 ceph-mon[36715]: pgmap v3601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:43.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:43 smithi144 ceph-mon[32129]: pgmap v3601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:43 smithi183 ceph-mon[37273]: pgmap v3601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:45 smithi183 ceph-mon[37273]: pgmap v3602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:45 smithi144 ceph-mon[32129]: pgmap v3602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:45 smithi160 ceph-mon[36715]: pgmap v3602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:46.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:46.876+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:46.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:46.877+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:46.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:46.877+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:46.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:46.877+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:46.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:46.877+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:47.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:47.116+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:41:47.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:47.116+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:41:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:47.117+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:41:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:41:47.117+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:41:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:47 smithi183 ceph-mon[37273]: pgmap v3603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:41:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5749.656672 secs 2024-07-26T12:41:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:47 smithi144 ceph-mon[32129]: pgmap v3603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:41:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5749.656672 secs 2024-07-26T12:41:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:47 smithi160 ceph-mon[36715]: pgmap v3603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:41:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5749.656672 secs 2024-07-26T12:41:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:49 smithi183 ceph-mon[37273]: pgmap v3604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:49 smithi144 ceph-mon[32129]: pgmap v3604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:49 smithi160 ceph-mon[36715]: pgmap v3604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:41:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:51 smithi183 ceph-mon[37273]: pgmap v3605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:51 smithi144 ceph-mon[32129]: pgmap v3605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:51 smithi160 ceph-mon[36715]: pgmap v3605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:41:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5754.656790 secs 2024-07-26T12:41:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5754.656790 secs 2024-07-26T12:41:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5754.656790 secs 2024-07-26T12:41:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:53 smithi183 ceph-mon[37273]: pgmap v3606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:41:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:53 smithi144 ceph-mon[32129]: pgmap v3606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:41:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:53 smithi160 ceph-mon[36715]: pgmap v3606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:41:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:55 smithi183 ceph-mon[37273]: pgmap v3607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:55 smithi144 ceph-mon[32129]: pgmap v3607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:55 smithi160 ceph-mon[36715]: pgmap v3607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:57 smithi183 ceph-mon[37273]: pgmap v3608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5759.656912 secs 2024-07-26T12:41:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:57 smithi144 ceph-mon[32129]: pgmap v3608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5759.656912 secs 2024-07-26T12:41:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:57 smithi160 ceph-mon[36715]: pgmap v3608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:41:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5759.656912 secs 2024-07-26T12:41:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:41:59 smithi144 ceph-mon[32129]: pgmap v3609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:41:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:41:59 smithi183 ceph-mon[37273]: pgmap v3609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:41:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:41:59 smithi160 ceph-mon[36715]: pgmap v3609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:42:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:01 smithi183 ceph-mon[37273]: pgmap v3610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:42:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:01 smithi144 ceph-mon[32129]: pgmap v3610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:42:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:01 smithi160 ceph-mon[36715]: pgmap v3610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:42:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:01.893+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:01.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:01.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:01.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:01.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:01.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:02 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 5764.657009 secs 2024-07-26T12:42:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:02 smithi183 ceph-mon[37273]: slow request 5764.657008 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:42:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:02 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 5764.657009 secs 2024-07-26T12:42:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:02 smithi144 ceph-mon[32129]: slow request 5764.657008 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:42:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:02 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 5764.657009 secs 2024-07-26T12:42:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:02 smithi160 ceph-mon[36715]: slow request 5764.657008 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:42:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:03 smithi183 ceph-mon[37273]: pgmap v3611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:42:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:03 smithi144 ceph-mon[32129]: pgmap v3611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:42:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:03 smithi160 ceph-mon[36715]: pgmap v3611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:42:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:05 smithi183 ceph-mon[37273]: pgmap v3612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:05 smithi144 ceph-mon[32129]: pgmap v3612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:05 smithi160 ceph-mon[36715]: pgmap v3612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:07 smithi144 ceph-mon[32129]: pgmap v3613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:07 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 5769.657129 secs 2024-07-26T12:42:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:07 smithi144 ceph-mon[32129]: slow request 5764.796317 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T12:42:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:07 smithi160 ceph-mon[36715]: pgmap v3613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:07 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 5769.657129 secs 2024-07-26T12:42:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:07 smithi160 ceph-mon[36715]: slow request 5764.796317 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T12:42:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:07 smithi183 ceph-mon[37273]: pgmap v3613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:07 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 5769.657129 secs 2024-07-26T12:42:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:07 smithi183 ceph-mon[37273]: slow request 5764.796317 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T12:42:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:09 smithi144 ceph-mon[32129]: pgmap v3614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:09 smithi160 ceph-mon[36715]: pgmap v3614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:09 smithi183 ceph-mon[37273]: pgmap v3614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:11 smithi144 ceph-mon[32129]: pgmap v3615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:11 smithi160 ceph-mon[36715]: pgmap v3615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:11 smithi183 ceph-mon[37273]: pgmap v3615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5774.657253 secs 2024-07-26T12:42:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5774.657253 secs 2024-07-26T12:42:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5774.657253 secs 2024-07-26T12:42:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:13 smithi144 ceph-mon[32129]: pgmap v3616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:13.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:13 smithi183 ceph-mon[37273]: pgmap v3616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:13 smithi160 ceph-mon[36715]: pgmap v3616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:15 smithi144 ceph-mon[32129]: pgmap v3617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:15 smithi160 ceph-mon[36715]: pgmap v3617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:15 smithi183 ceph-mon[37273]: pgmap v3617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:16.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:16.895+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:16.895+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:16.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:16.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:17 smithi144 ceph-mon[32129]: pgmap v3618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5779.657352 secs 2024-07-26T12:42:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:17 smithi160 ceph-mon[36715]: pgmap v3618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5779.657352 secs 2024-07-26T12:42:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:17 smithi183 ceph-mon[37273]: pgmap v3618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5779.657352 secs 2024-07-26T12:42:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:19 smithi144 ceph-mon[32129]: pgmap v3619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:19 smithi160 ceph-mon[36715]: pgmap v3619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:19 smithi183 ceph-mon[37273]: pgmap v3619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:21 smithi144 ceph-mon[32129]: pgmap v3620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:21 smithi160 ceph-mon[36715]: pgmap v3620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:21 smithi183 ceph-mon[37273]: pgmap v3620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5784.657416 secs 2024-07-26T12:42:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5784.657416 secs 2024-07-26T12:42:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5784.657416 secs 2024-07-26T12:42:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:23 smithi144 ceph-mon[32129]: pgmap v3621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:23 smithi160 ceph-mon[36715]: pgmap v3621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:23 smithi183 ceph-mon[37273]: pgmap v3621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:25 smithi144 ceph-mon[32129]: pgmap v3622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:25 smithi160 ceph-mon[36715]: pgmap v3622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:25 smithi183 ceph-mon[37273]: pgmap v3622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:27 smithi144 ceph-mon[32129]: pgmap v3623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5789.657544 secs 2024-07-26T12:42:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:27 smithi160 ceph-mon[36715]: pgmap v3623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5789.657544 secs 2024-07-26T12:42:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:27 smithi183 ceph-mon[37273]: pgmap v3623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5789.657544 secs 2024-07-26T12:42:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:29 smithi160 ceph-mon[36715]: pgmap v3624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:29 smithi183 ceph-mon[37273]: pgmap v3624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:29 smithi144 ceph-mon[32129]: pgmap v3624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:31 smithi160 ceph-mon[36715]: pgmap v3625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:31 smithi183 ceph-mon[37273]: pgmap v3625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:31 smithi144 ceph-mon[32129]: pgmap v3625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:31.910+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:31.911+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:31.911+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:31.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:31.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5794.657646 secs 2024-07-26T12:42:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5794.657646 secs 2024-07-26T12:42:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5794.657646 secs 2024-07-26T12:42:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:33 smithi160 ceph-mon[36715]: pgmap v3626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:33 smithi183 ceph-mon[37273]: pgmap v3626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:33 smithi144 ceph-mon[32129]: pgmap v3626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:35 smithi160 ceph-mon[36715]: pgmap v3627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:35 smithi183 ceph-mon[37273]: pgmap v3627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:35 smithi144 ceph-mon[32129]: pgmap v3627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:37 smithi160 ceph-mon[36715]: pgmap v3628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5799.657791 secs 2024-07-26T12:42:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:37 smithi183 ceph-mon[37273]: pgmap v3628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5799.657791 secs 2024-07-26T12:42:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:37 smithi144 ceph-mon[32129]: pgmap v3628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5799.657791 secs 2024-07-26T12:42:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:38 smithi160 ceph-mon[36715]: pgmap v3629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:38 smithi183 ceph-mon[37273]: pgmap v3629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:38 smithi144 ceph-mon[32129]: pgmap v3629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:40 smithi183 ceph-mon[37273]: pgmap v3630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:40 smithi144 ceph-mon[32129]: pgmap v3630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:40 smithi160 ceph-mon[36715]: pgmap v3630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:41 smithi183 ceph-mon[37273]: pgmap v3631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:41 smithi183 ceph-mon[37273]: pgmap v3632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5804.657931 secs 2024-07-26T12:42:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:41 smithi144 ceph-mon[32129]: pgmap v3631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:42:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:41 smithi144 ceph-mon[32129]: pgmap v3632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:42:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5804.657931 secs 2024-07-26T12:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:41 smithi160 ceph-mon[36715]: pgmap v3631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:41 smithi160 ceph-mon[36715]: pgmap v3632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:42:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5804.657931 secs 2024-07-26T12:42:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:43 smithi144 ceph-mon[32129]: pgmap v3633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:43 smithi160 ceph-mon[36715]: pgmap v3633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:43 smithi183 ceph-mon[37273]: pgmap v3633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:45 smithi144 ceph-mon[32129]: pgmap v3634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:45 smithi160 ceph-mon[36715]: pgmap v3634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:45 smithi183 ceph-mon[37273]: pgmap v3634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:46.925+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:46.925+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:46.925+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:46.925+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:42:46.926+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5809.658073 secs 2024-07-26T12:42:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5809.658073 secs 2024-07-26T12:42:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5809.658073 secs 2024-07-26T12:42:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:47 smithi144 ceph-mon[32129]: pgmap v3635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:42:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:47 smithi160 ceph-mon[36715]: pgmap v3635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:42:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:47 smithi183 ceph-mon[37273]: pgmap v3635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:42:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:49 smithi160 ceph-mon[36715]: pgmap v3636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:42:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:49 smithi183 ceph-mon[37273]: pgmap v3636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:42:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:49 smithi144 ceph-mon[32129]: pgmap v3636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:42:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:51 smithi160 ceph-mon[36715]: pgmap v3637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:42:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5814.658200 secs 2024-07-26T12:42:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:51 smithi183 ceph-mon[37273]: pgmap v3637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:42:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5814.658200 secs 2024-07-26T12:42:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:51 smithi144 ceph-mon[32129]: pgmap v3637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:42:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5814.658200 secs 2024-07-26T12:42:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:53 smithi160 ceph-mon[36715]: pgmap v3638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:53 smithi183 ceph-mon[37273]: pgmap v3638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:53 smithi144 ceph-mon[32129]: pgmap v3638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:42:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:55 smithi160 ceph-mon[36715]: pgmap v3639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:55 smithi183 ceph-mon[37273]: pgmap v3639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:55 smithi144 ceph-mon[32129]: pgmap v3639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:42:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5819.658332 secs 2024-07-26T12:42:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5819.658332 secs 2024-07-26T12:42:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5819.658332 secs 2024-07-26T12:42:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:57 smithi160 ceph-mon[36715]: pgmap v3640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:57 smithi183 ceph-mon[37273]: pgmap v3640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:42:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:57 smithi144 ceph-mon[32129]: pgmap v3640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:42:59 smithi160 ceph-mon[36715]: pgmap v3641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:42:59 smithi183 ceph-mon[37273]: pgmap v3641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:42:59 smithi144 ceph-mon[32129]: pgmap v3641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:01.941+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:01.942+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:01.942+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:01.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:01.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:01 smithi160 ceph-mon[36715]: pgmap v3642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5824.658436 secs 2024-07-26T12:43:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:01 smithi183 ceph-mon[37273]: pgmap v3642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5824.658436 secs 2024-07-26T12:43:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:01 smithi144 ceph-mon[32129]: pgmap v3642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5824.658436 secs 2024-07-26T12:43:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:03 smithi160 ceph-mon[36715]: pgmap v3643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:03 smithi183 ceph-mon[37273]: pgmap v3643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:03 smithi144 ceph-mon[32129]: pgmap v3643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:05 smithi160 ceph-mon[36715]: pgmap v3644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:05 smithi183 ceph-mon[37273]: pgmap v3644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:05 smithi144 ceph-mon[32129]: pgmap v3644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5829.658552 secs 2024-07-26T12:43:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5829.658552 secs 2024-07-26T12:43:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5829.658552 secs 2024-07-26T12:43:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:07 smithi160 ceph-mon[36715]: pgmap v3645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:07 smithi183 ceph-mon[37273]: pgmap v3645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:07 smithi144 ceph-mon[32129]: pgmap v3645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:09 smithi160 ceph-mon[36715]: pgmap v3646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:09 smithi183 ceph-mon[37273]: pgmap v3646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:09 smithi144 ceph-mon[32129]: pgmap v3646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:11 smithi160 ceph-mon[36715]: pgmap v3647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5834.658673 secs 2024-07-26T12:43:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:11 smithi183 ceph-mon[37273]: pgmap v3647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5834.658673 secs 2024-07-26T12:43:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:11 smithi144 ceph-mon[32129]: pgmap v3647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5834.658673 secs 2024-07-26T12:43:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:13 smithi160 ceph-mon[36715]: pgmap v3648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:13 smithi183 ceph-mon[37273]: pgmap v3648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:13 smithi144 ceph-mon[32129]: pgmap v3648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:15 smithi183 ceph-mon[37273]: pgmap v3649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:15 smithi144 ceph-mon[32129]: pgmap v3649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:15 smithi160 ceph-mon[36715]: pgmap v3649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:16.957+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:16.958+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:16.958+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:16.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:16.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5839.658786 secs 2024-07-26T12:43:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5839.658786 secs 2024-07-26T12:43:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5839.658786 secs 2024-07-26T12:43:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:17 smithi183 ceph-mon[37273]: pgmap v3650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:17 smithi144 ceph-mon[32129]: pgmap v3650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:17 smithi160 ceph-mon[36715]: pgmap v3650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:19 smithi183 ceph-mon[37273]: pgmap v3651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:19 smithi144 ceph-mon[32129]: pgmap v3651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:19 smithi160 ceph-mon[36715]: pgmap v3651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:21 smithi183 ceph-mon[37273]: pgmap v3652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5844.658907 secs 2024-07-26T12:43:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:21 smithi144 ceph-mon[32129]: pgmap v3652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5844.658907 secs 2024-07-26T12:43:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:21 smithi160 ceph-mon[36715]: pgmap v3652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5844.658907 secs 2024-07-26T12:43:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:24 smithi144 ceph-mon[32129]: pgmap v3653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:24 smithi160 ceph-mon[36715]: pgmap v3653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:24 smithi183 ceph-mon[37273]: pgmap v3653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:26 smithi144 ceph-mon[32129]: pgmap v3654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:26 smithi160 ceph-mon[36715]: pgmap v3654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:26 smithi183 ceph-mon[37273]: pgmap v3654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5849.659024 secs 2024-07-26T12:43:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5849.659024 secs 2024-07-26T12:43:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5849.659024 secs 2024-07-26T12:43:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:28 smithi144 ceph-mon[32129]: pgmap v3655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:28 smithi160 ceph-mon[36715]: pgmap v3655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:28 smithi183 ceph-mon[37273]: pgmap v3655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:30 smithi144 ceph-mon[32129]: pgmap v3656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:30 smithi160 ceph-mon[36715]: pgmap v3656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:30 smithi183 ceph-mon[37273]: pgmap v3656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:31.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:31.975+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:31.975+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:31.976+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:31.976+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:31.976+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:32 smithi144 ceph-mon[32129]: pgmap v3657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5854.659112 secs 2024-07-26T12:43:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:32 smithi160 ceph-mon[36715]: pgmap v3657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5854.659112 secs 2024-07-26T12:43:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:32 smithi183 ceph-mon[37273]: pgmap v3657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5854.659112 secs 2024-07-26T12:43:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:34 smithi144 ceph-mon[32129]: pgmap v3658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:34 smithi160 ceph-mon[36715]: pgmap v3658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:34 smithi183 ceph-mon[37273]: pgmap v3658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:36 smithi144 ceph-mon[32129]: pgmap v3659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:36 smithi160 ceph-mon[36715]: pgmap v3659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:36 smithi183 ceph-mon[37273]: pgmap v3659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5859.659239 secs 2024-07-26T12:43:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5859.659239 secs 2024-07-26T12:43:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5859.659239 secs 2024-07-26T12:43:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:38 smithi160 ceph-mon[36715]: pgmap v3660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:38 smithi183 ceph-mon[37273]: pgmap v3660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:38.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:38 smithi144 ceph-mon[32129]: pgmap v3660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:43:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:40 smithi160 ceph-mon[36715]: pgmap v3661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:40 smithi183 ceph-mon[37273]: pgmap v3661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:40 smithi144 ceph-mon[32129]: pgmap v3661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:42 smithi160 ceph-mon[36715]: pgmap v3662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5864.659363 secs 2024-07-26T12:43:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:43:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:42 smithi183 ceph-mon[37273]: pgmap v3662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5864.659363 secs 2024-07-26T12:43:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:43:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:42 smithi144 ceph-mon[32129]: pgmap v3662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5864.659363 secs 2024-07-26T12:43:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:43:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:43 smithi160 ceph-mon[36715]: pgmap v3663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:43:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:43 smithi160 ceph-mon[36715]: pgmap v3664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:43:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:43 smithi183 ceph-mon[37273]: pgmap v3663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:43:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:43 smithi183 ceph-mon[37273]: pgmap v3664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:43:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:43 smithi144 ceph-mon[32129]: pgmap v3663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:43:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:43 smithi144 ceph-mon[32129]: pgmap v3664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:43:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:45 smithi160 ceph-mon[36715]: pgmap v3665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:43:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:45 smithi183 ceph-mon[37273]: pgmap v3665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:43:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:45 smithi144 ceph-mon[32129]: pgmap v3665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:43:46.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:46.981+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:46.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:46.982+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:46.982+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:46.982+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:43:46.982+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5869.659431 secs 2024-07-26T12:43:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:47 smithi160 ceph-mon[36715]: pgmap v3666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:43:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5869.659431 secs 2024-07-26T12:43:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:47 smithi183 ceph-mon[37273]: pgmap v3666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:43:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5869.659431 secs 2024-07-26T12:43:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:47 smithi144 ceph-mon[32129]: pgmap v3666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:43:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:49 smithi160 ceph-mon[36715]: pgmap v3667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:43:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:49 smithi183 ceph-mon[37273]: pgmap v3667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:43:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:49 smithi144 ceph-mon[32129]: pgmap v3667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:43:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:51 smithi160 ceph-mon[36715]: pgmap v3668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:43:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:51 smithi183 ceph-mon[37273]: pgmap v3668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:43:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:51 smithi144 ceph-mon[32129]: pgmap v3668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:43:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5874.659540 secs 2024-07-26T12:43:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5874.659540 secs 2024-07-26T12:43:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5874.659540 secs 2024-07-26T12:43:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:53 smithi160 ceph-mon[36715]: pgmap v3669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:43:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:53 smithi183 ceph-mon[37273]: pgmap v3669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:43:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:53 smithi144 ceph-mon[32129]: pgmap v3669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:43:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:55 smithi160 ceph-mon[36715]: pgmap v3670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:55 smithi183 ceph-mon[37273]: pgmap v3670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:55 smithi144 ceph-mon[32129]: pgmap v3670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5879.659629 secs 2024-07-26T12:43:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:57 smithi160 ceph-mon[36715]: pgmap v3671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5879.659629 secs 2024-07-26T12:43:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:57 smithi183 ceph-mon[37273]: pgmap v3671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5879.659629 secs 2024-07-26T12:43:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:57 smithi144 ceph-mon[32129]: pgmap v3671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:43:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:43:59 smithi183 ceph-mon[37273]: pgmap v3672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:43:59 smithi144 ceph-mon[32129]: pgmap v3672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:43:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:43:59 smithi160 ceph-mon[36715]: pgmap v3672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:01 smithi183 ceph-mon[37273]: pgmap v3673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:01 smithi144 ceph-mon[32129]: pgmap v3673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:01 smithi160 ceph-mon[36715]: pgmap v3673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:01.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:01.999+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:01.999+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:01.999+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:01.999+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5884.659756 secs 2024-07-26T12:44:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5884.659756 secs 2024-07-26T12:44:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5884.659756 secs 2024-07-26T12:44:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:03 smithi183 ceph-mon[37273]: pgmap v3674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:03 smithi144 ceph-mon[32129]: pgmap v3674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:03 smithi160 ceph-mon[36715]: pgmap v3674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:05 smithi183 ceph-mon[37273]: pgmap v3675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:05 smithi144 ceph-mon[32129]: pgmap v3675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:05 smithi160 ceph-mon[36715]: pgmap v3675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:07.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5889.659877 secs 2024-07-26T12:44:07.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:07 smithi160 ceph-mon[36715]: pgmap v3676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5889.659877 secs 2024-07-26T12:44:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:07 smithi183 ceph-mon[37273]: pgmap v3676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5889.659877 secs 2024-07-26T12:44:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:07 smithi144 ceph-mon[32129]: pgmap v3676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:09 smithi183 ceph-mon[37273]: pgmap v3677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:09 smithi144 ceph-mon[32129]: pgmap v3677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:09 smithi160 ceph-mon[36715]: pgmap v3677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:11 smithi183 ceph-mon[37273]: pgmap v3678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:11 smithi144 ceph-mon[32129]: pgmap v3678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:11 smithi160 ceph-mon[36715]: pgmap v3678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5894.660027 secs 2024-07-26T12:44:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5894.660027 secs 2024-07-26T12:44:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5894.660027 secs 2024-07-26T12:44:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:13 smithi183 ceph-mon[37273]: pgmap v3679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:13 smithi144 ceph-mon[32129]: pgmap v3679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:13 smithi160 ceph-mon[36715]: pgmap v3679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:15 smithi183 ceph-mon[37273]: pgmap v3680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:15 smithi144 ceph-mon[32129]: pgmap v3680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:15 smithi160 ceph-mon[36715]: pgmap v3680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:17.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:17.015+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:17.015+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:17.015+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:17.015+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5899.660156 secs 2024-07-26T12:44:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:17 smithi183 ceph-mon[37273]: pgmap v3681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5899.660156 secs 2024-07-26T12:44:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:17 smithi144 ceph-mon[32129]: pgmap v3681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5899.660156 secs 2024-07-26T12:44:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:17 smithi160 ceph-mon[36715]: pgmap v3681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:19.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:19 smithi144 ceph-mon[32129]: pgmap v3682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:19 smithi183 ceph-mon[37273]: pgmap v3682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:19.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:19 smithi160 ceph-mon[36715]: pgmap v3682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:21 smithi144 ceph-mon[32129]: pgmap v3683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:21 smithi160 ceph-mon[36715]: pgmap v3683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:21 smithi183 ceph-mon[37273]: pgmap v3683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5904.660356 secs 2024-07-26T12:44:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5904.660356 secs 2024-07-26T12:44:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5904.660356 secs 2024-07-26T12:44:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:23 smithi144 ceph-mon[32129]: pgmap v3684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:23 smithi160 ceph-mon[36715]: pgmap v3684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:23 smithi183 ceph-mon[37273]: pgmap v3684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:25 smithi144 ceph-mon[32129]: pgmap v3685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:25 smithi160 ceph-mon[36715]: pgmap v3685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:25 smithi183 ceph-mon[37273]: pgmap v3685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5909.660473 secs 2024-07-26T12:44:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:27 smithi144 ceph-mon[32129]: pgmap v3686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5909.660473 secs 2024-07-26T12:44:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:27 smithi160 ceph-mon[36715]: pgmap v3686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5909.660473 secs 2024-07-26T12:44:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:27 smithi183 ceph-mon[37273]: pgmap v3686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:29 smithi144 ceph-mon[32129]: pgmap v3687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:29 smithi160 ceph-mon[36715]: pgmap v3687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:29 smithi183 ceph-mon[37273]: pgmap v3687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:31 smithi144 ceph-mon[32129]: pgmap v3688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:31 smithi160 ceph-mon[36715]: pgmap v3688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:31 smithi183 ceph-mon[37273]: pgmap v3688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:32.021+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:32.022+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:32.022+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:32.022+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:32.022+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5914.660616 secs 2024-07-26T12:44:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5914.660616 secs 2024-07-26T12:44:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5914.660616 secs 2024-07-26T12:44:33.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:33 smithi144 ceph-mon[32129]: pgmap v3689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:33 smithi160 ceph-mon[36715]: pgmap v3689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:33 smithi183 ceph-mon[37273]: pgmap v3689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:35 smithi144 ceph-mon[32129]: pgmap v3690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:35 smithi160 ceph-mon[36715]: pgmap v3690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:35 smithi183 ceph-mon[37273]: pgmap v3690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5919.660772 secs 2024-07-26T12:44:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:37 smithi144 ceph-mon[32129]: pgmap v3691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5919.660772 secs 2024-07-26T12:44:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:37 smithi160 ceph-mon[36715]: pgmap v3691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5919.660772 secs 2024-07-26T12:44:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:37 smithi183 ceph-mon[37273]: pgmap v3691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:39 smithi144 ceph-mon[32129]: pgmap v3692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:39 smithi160 ceph-mon[36715]: pgmap v3692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:39 smithi183 ceph-mon[37273]: pgmap v3692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:44:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:41 smithi144 ceph-mon[32129]: pgmap v3693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:41 smithi160 ceph-mon[36715]: pgmap v3693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:41 smithi183 ceph-mon[37273]: pgmap v3693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5924.660894 secs 2024-07-26T12:44:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:42 smithi144 ceph-mon[32129]: pgmap v3694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5924.660894 secs 2024-07-26T12:44:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:42 smithi160 ceph-mon[36715]: pgmap v3694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5924.660894 secs 2024-07-26T12:44:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:42 smithi183 ceph-mon[37273]: pgmap v3694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:43 smithi160 ceph-mon[36715]: pgmap v3695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:44:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:43 smithi160 ceph-mon[36715]: pgmap v3696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:44:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:44:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:43 smithi183 ceph-mon[37273]: pgmap v3695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:44:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:43 smithi183 ceph-mon[37273]: pgmap v3696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:44:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:44:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:43 smithi144 ceph-mon[32129]: pgmap v3695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:44:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:43 smithi144 ceph-mon[32129]: pgmap v3696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:44:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:44:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:46 smithi183 ceph-mon[37273]: pgmap v3697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:44:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:46 smithi144 ceph-mon[32129]: pgmap v3697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:44:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:46 smithi160 ceph-mon[36715]: pgmap v3697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:44:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.039+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.039+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.039+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.039+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.147+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:44:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.147+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:44:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.148+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T12:44:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.148+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T12:44:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.148+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:44:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.148+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74a009d020 con 0x7f74b0109990 2024-07-26T12:44:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.149+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d020 con 0x563c6c4182f0 2024-07-26T12:44:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:44:47.149+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T12:44:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5929.661029 secs 2024-07-26T12:44:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5929.661029 secs 2024-07-26T12:44:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5929.661029 secs 2024-07-26T12:44:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:48 smithi183 ceph-mon[37273]: pgmap v3698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:44:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:48 smithi144 ceph-mon[32129]: pgmap v3698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:44:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:48 smithi160 ceph-mon[36715]: pgmap v3698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:44:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:50 smithi144 ceph-mon[32129]: pgmap v3699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:44:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:50 smithi160 ceph-mon[36715]: pgmap v3699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:44:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:50 smithi183 ceph-mon[37273]: pgmap v3699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:44:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5934.661158 secs 2024-07-26T12:44:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:52 smithi144 ceph-mon[32129]: pgmap v3700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:44:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5934.661158 secs 2024-07-26T12:44:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:52 smithi160 ceph-mon[36715]: pgmap v3700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:44:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5934.661158 secs 2024-07-26T12:44:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:52 smithi183 ceph-mon[37273]: pgmap v3700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:44:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:54 smithi144 ceph-mon[32129]: pgmap v3701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:44:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:54 smithi160 ceph-mon[36715]: pgmap v3701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:44:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:54 smithi183 ceph-mon[37273]: pgmap v3701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:44:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:56 smithi144 ceph-mon[32129]: pgmap v3702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:56 smithi160 ceph-mon[36715]: pgmap v3702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:56 smithi183 ceph-mon[37273]: pgmap v3702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:44:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5939.661308 secs 2024-07-26T12:44:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5939.661308 secs 2024-07-26T12:44:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5939.661308 secs 2024-07-26T12:44:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:44:58 smithi144 ceph-mon[32129]: pgmap v3703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:44:58 smithi160 ceph-mon[36715]: pgmap v3703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:44:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:44:58 smithi183 ceph-mon[37273]: pgmap v3703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:00 smithi144 ceph-mon[32129]: pgmap v3704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:00 smithi160 ceph-mon[36715]: pgmap v3704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:00 smithi183 ceph-mon[37273]: pgmap v3704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:02.053+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:02.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:02.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:02.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:02.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5944.661394 secs 2024-07-26T12:45:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:02 smithi144 ceph-mon[32129]: pgmap v3705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5944.661394 secs 2024-07-26T12:45:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:02 smithi160 ceph-mon[36715]: pgmap v3705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5944.661394 secs 2024-07-26T12:45:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:02 smithi183 ceph-mon[37273]: pgmap v3705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:04 smithi144 ceph-mon[32129]: pgmap v3706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:45:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:04 smithi160 ceph-mon[36715]: pgmap v3706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:45:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:04 smithi183 ceph-mon[37273]: pgmap v3706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:45:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:06 smithi144 ceph-mon[32129]: pgmap v3707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:45:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:06 smithi160 ceph-mon[36715]: pgmap v3707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:45:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:06 smithi183 ceph-mon[37273]: pgmap v3707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:45:07.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5949.661547 secs 2024-07-26T12:45:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5949.661547 secs 2024-07-26T12:45:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5949.661547 secs 2024-07-26T12:45:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:08 smithi144 ceph-mon[32129]: pgmap v3708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:08 smithi160 ceph-mon[36715]: pgmap v3708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:08 smithi183 ceph-mon[37273]: pgmap v3708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:10 smithi144 ceph-mon[32129]: pgmap v3709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:10 smithi160 ceph-mon[36715]: pgmap v3709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:10 smithi183 ceph-mon[37273]: pgmap v3709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5954.661667 secs 2024-07-26T12:45:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:12 smithi144 ceph-mon[32129]: pgmap v3710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5954.661667 secs 2024-07-26T12:45:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:12 smithi160 ceph-mon[36715]: pgmap v3710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5954.661667 secs 2024-07-26T12:45:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:12 smithi183 ceph-mon[37273]: pgmap v3710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:14 smithi160 ceph-mon[36715]: pgmap v3711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:14 smithi183 ceph-mon[37273]: pgmap v3711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:14 smithi144 ceph-mon[32129]: pgmap v3711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:16 smithi160 ceph-mon[36715]: pgmap v3712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:45:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:16 smithi183 ceph-mon[37273]: pgmap v3712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:45:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:16 smithi144 ceph-mon[32129]: pgmap v3712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:45:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:17.069+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:17.070+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:17.070+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:17.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:17.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5959.661783 secs 2024-07-26T12:45:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5959.661783 secs 2024-07-26T12:45:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5959.661783 secs 2024-07-26T12:45:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:18 smithi160 ceph-mon[36715]: pgmap v3713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:18 smithi183 ceph-mon[37273]: pgmap v3713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:18 smithi144 ceph-mon[32129]: pgmap v3713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:20 smithi160 ceph-mon[36715]: pgmap v3714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:20 smithi183 ceph-mon[37273]: pgmap v3714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:20 smithi144 ceph-mon[32129]: pgmap v3714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5964.661876 secs 2024-07-26T12:45:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:22 smithi160 ceph-mon[36715]: pgmap v3715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5964.661876 secs 2024-07-26T12:45:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:22 smithi183 ceph-mon[37273]: pgmap v3715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5964.661876 secs 2024-07-26T12:45:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:22 smithi144 ceph-mon[32129]: pgmap v3715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:24.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:24 smithi160 ceph-mon[36715]: pgmap v3716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:45:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:24 smithi183 ceph-mon[37273]: pgmap v3716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:45:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:24 smithi144 ceph-mon[32129]: pgmap v3716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:45:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:26 smithi160 ceph-mon[36715]: pgmap v3717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:26 smithi183 ceph-mon[37273]: pgmap v3717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:26 smithi144 ceph-mon[32129]: pgmap v3717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5969.661994 secs 2024-07-26T12:45:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5969.661994 secs 2024-07-26T12:45:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5969.661994 secs 2024-07-26T12:45:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:28 smithi160 ceph-mon[36715]: pgmap v3718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:28 smithi183 ceph-mon[37273]: pgmap v3718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:28 smithi144 ceph-mon[32129]: pgmap v3718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:30 smithi160 ceph-mon[36715]: pgmap v3719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:30 smithi183 ceph-mon[37273]: pgmap v3719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:30 smithi144 ceph-mon[32129]: pgmap v3719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:32.087+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:32.088+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:32.088+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:32.088+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:32.088+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5974.662108 secs 2024-07-26T12:45:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:32 smithi160 ceph-mon[36715]: pgmap v3720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5974.662108 secs 2024-07-26T12:45:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:32 smithi183 ceph-mon[37273]: pgmap v3720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5974.662108 secs 2024-07-26T12:45:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:32 smithi144 ceph-mon[32129]: pgmap v3720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:34 smithi160 ceph-mon[36715]: pgmap v3721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:34 smithi183 ceph-mon[37273]: pgmap v3721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:34 smithi144 ceph-mon[32129]: pgmap v3721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:36 smithi160 ceph-mon[36715]: pgmap v3722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:36 smithi183 ceph-mon[37273]: pgmap v3722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:36 smithi144 ceph-mon[32129]: pgmap v3722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5979.662226 secs 2024-07-26T12:45:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5979.662226 secs 2024-07-26T12:45:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5979.662226 secs 2024-07-26T12:45:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:38 smithi183 ceph-mon[37273]: pgmap v3723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:38 smithi144 ceph-mon[32129]: pgmap v3723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:38 smithi160 ceph-mon[36715]: pgmap v3723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:40 smithi183 ceph-mon[37273]: pgmap v3724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:40 smithi144 ceph-mon[32129]: pgmap v3724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:40 smithi160 ceph-mon[36715]: pgmap v3724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5984.662368 secs 2024-07-26T12:45:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:42 smithi183 ceph-mon[37273]: pgmap v3725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5984.662368 secs 2024-07-26T12:45:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:42 smithi144 ceph-mon[32129]: pgmap v3725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5984.662368 secs 2024-07-26T12:45:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:42 smithi160 ceph-mon[36715]: pgmap v3725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:44 smithi183 ceph-mon[37273]: pgmap v3726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:45:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:44 smithi144 ceph-mon[32129]: pgmap v3726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:45:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:44 smithi160 ceph-mon[36715]: pgmap v3726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:45:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:45 smithi183 ceph-mon[37273]: pgmap v3727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:45:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:45 smithi183 ceph-mon[37273]: pgmap v3728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:45 smithi144 ceph-mon[32129]: pgmap v3727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:45:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:45 smithi144 ceph-mon[32129]: pgmap v3728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:45 smithi160 ceph-mon[36715]: pgmap v3727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:45:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:45 smithi160 ceph-mon[36715]: pgmap v3728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:47.104+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:47.105+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:47.105+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:47.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:45:47.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5989.662488 secs 2024-07-26T12:45:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:47 smithi183 ceph-mon[37273]: pgmap v3729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5989.662488 secs 2024-07-26T12:45:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:47 smithi144 ceph-mon[32129]: pgmap v3729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5989.662488 secs 2024-07-26T12:45:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:47 smithi160 ceph-mon[36715]: pgmap v3729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:49 smithi183 ceph-mon[37273]: pgmap v3730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:49 smithi144 ceph-mon[32129]: pgmap v3730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:49 smithi160 ceph-mon[36715]: pgmap v3730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:45:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:51 smithi183 ceph-mon[37273]: pgmap v3731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:51 smithi144 ceph-mon[32129]: pgmap v3731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:51.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:51 smithi160 ceph-mon[36715]: pgmap v3731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5994.662603 secs 2024-07-26T12:45:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5994.662603 secs 2024-07-26T12:45:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5994.662603 secs 2024-07-26T12:45:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:53 smithi183 ceph-mon[37273]: pgmap v3732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:53 smithi144 ceph-mon[32129]: pgmap v3732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:53 smithi160 ceph-mon[36715]: pgmap v3732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:45:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:55 smithi183 ceph-mon[37273]: pgmap v3733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:55 smithi144 ceph-mon[32129]: pgmap v3733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:55 smithi160 ceph-mon[36715]: pgmap v3733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:45:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 5999.662722 secs 2024-07-26T12:45:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:57 smithi144 ceph-mon[32129]: pgmap v3734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 5999.662722 secs 2024-07-26T12:45:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:57 smithi160 ceph-mon[36715]: pgmap v3734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 5999.662722 secs 2024-07-26T12:45:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:57 smithi183 ceph-mon[37273]: pgmap v3734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:45:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:45:59 smithi160 ceph-mon[36715]: pgmap v3735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:45:59 smithi183 ceph-mon[37273]: pgmap v3735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:45:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:45:59 smithi144 ceph-mon[32129]: pgmap v3735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:01 smithi160 ceph-mon[36715]: pgmap v3736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:01 smithi183 ceph-mon[37273]: pgmap v3736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:01 smithi144 ceph-mon[32129]: pgmap v3736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:02.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:02.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:02.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:02.122+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:02.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:02.122+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:02.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:02.122+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:02.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:02.122+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6004.662856 secs 2024-07-26T12:46:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6004.662856 secs 2024-07-26T12:46:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6004.662856 secs 2024-07-26T12:46:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:03 smithi160 ceph-mon[36715]: pgmap v3737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:03 smithi183 ceph-mon[37273]: pgmap v3737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:03 smithi144 ceph-mon[32129]: pgmap v3737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:05 smithi160 ceph-mon[36715]: pgmap v3738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:05 smithi183 ceph-mon[37273]: pgmap v3738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:05 smithi144 ceph-mon[32129]: pgmap v3738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6009.662977 secs 2024-07-26T12:46:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:07 smithi160 ceph-mon[36715]: pgmap v3739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6009.662977 secs 2024-07-26T12:46:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:07 smithi183 ceph-mon[37273]: pgmap v3739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6009.662977 secs 2024-07-26T12:46:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:07 smithi144 ceph-mon[32129]: pgmap v3739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:09 smithi160 ceph-mon[36715]: pgmap v3740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:09 smithi183 ceph-mon[37273]: pgmap v3740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:09 smithi144 ceph-mon[32129]: pgmap v3740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:11 smithi160 ceph-mon[36715]: pgmap v3741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:11 smithi183 ceph-mon[37273]: pgmap v3741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:11 smithi144 ceph-mon[32129]: pgmap v3741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6014.663096 secs 2024-07-26T12:46:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6014.663096 secs 2024-07-26T12:46:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6014.663096 secs 2024-07-26T12:46:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:13 smithi160 ceph-mon[36715]: pgmap v3742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:13 smithi183 ceph-mon[37273]: pgmap v3742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:13 smithi144 ceph-mon[32129]: pgmap v3742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:15 smithi183 ceph-mon[37273]: pgmap v3743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:15 smithi144 ceph-mon[32129]: pgmap v3743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:15 smithi160 ceph-mon[36715]: pgmap v3743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:17.123+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:17.124+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:17.124+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:17.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:17.124+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:17.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:17.124+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6019.663221 secs 2024-07-26T12:46:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:17 smithi183 ceph-mon[37273]: pgmap v3744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6019.663221 secs 2024-07-26T12:46:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:17 smithi144 ceph-mon[32129]: pgmap v3744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6019.663221 secs 2024-07-26T12:46:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:17 smithi160 ceph-mon[36715]: pgmap v3744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:19.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:19 smithi183 ceph-mon[37273]: pgmap v3745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:19 smithi144 ceph-mon[32129]: pgmap v3745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:19 smithi160 ceph-mon[36715]: pgmap v3745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:21 smithi183 ceph-mon[37273]: pgmap v3746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:21 smithi144 ceph-mon[32129]: pgmap v3746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:21 smithi160 ceph-mon[36715]: pgmap v3746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6024.663338 secs 2024-07-26T12:46:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6024.663338 secs 2024-07-26T12:46:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6024.663338 secs 2024-07-26T12:46:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:23 smithi183 ceph-mon[37273]: pgmap v3747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:23 smithi144 ceph-mon[32129]: pgmap v3747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:23 smithi160 ceph-mon[36715]: pgmap v3747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:25 smithi183 ceph-mon[37273]: pgmap v3748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:25 smithi144 ceph-mon[32129]: pgmap v3748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:25 smithi160 ceph-mon[36715]: pgmap v3748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6029.663405 secs 2024-07-26T12:46:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:27 smithi183 ceph-mon[37273]: pgmap v3749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6029.663405 secs 2024-07-26T12:46:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:27 smithi144 ceph-mon[32129]: pgmap v3749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6029.663405 secs 2024-07-26T12:46:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:27 smithi160 ceph-mon[36715]: pgmap v3749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:29 smithi183 ceph-mon[37273]: pgmap v3750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:29 smithi144 ceph-mon[32129]: pgmap v3750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:29 smithi160 ceph-mon[36715]: pgmap v3750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:31 smithi183 ceph-mon[37273]: pgmap v3751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:31 smithi144 ceph-mon[32129]: pgmap v3751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:31 smithi160 ceph-mon[36715]: pgmap v3751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:32.141+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:32.142+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:32.142+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:32.142+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:32.142+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6034.663484 secs 2024-07-26T12:46:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6034.663484 secs 2024-07-26T12:46:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6034.663484 secs 2024-07-26T12:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:33 smithi144 ceph-mon[32129]: pgmap v3752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:33 smithi160 ceph-mon[36715]: pgmap v3752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:33 smithi183 ceph-mon[37273]: pgmap v3752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:35 smithi144 ceph-mon[32129]: pgmap v3753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:35 smithi160 ceph-mon[36715]: pgmap v3753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:35 smithi183 ceph-mon[37273]: pgmap v3753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6039.663603 secs 2024-07-26T12:46:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:37 smithi144 ceph-mon[32129]: pgmap v3754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6039.663603 secs 2024-07-26T12:46:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:37 smithi160 ceph-mon[36715]: pgmap v3754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6039.663603 secs 2024-07-26T12:46:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:37 smithi183 ceph-mon[37273]: pgmap v3754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:46:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:39 smithi144 ceph-mon[32129]: pgmap v3755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:39 smithi160 ceph-mon[36715]: pgmap v3755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:39 smithi183 ceph-mon[37273]: pgmap v3755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:41 smithi144 ceph-mon[32129]: pgmap v3756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:41 smithi160 ceph-mon[36715]: pgmap v3756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:41 smithi183 ceph-mon[37273]: pgmap v3756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6044.663718 secs 2024-07-26T12:46:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6044.663718 secs 2024-07-26T12:46:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6044.663718 secs 2024-07-26T12:46:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:43 smithi144 ceph-mon[32129]: pgmap v3757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:43 smithi160 ceph-mon[36715]: pgmap v3757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:43 smithi183 ceph-mon[37273]: pgmap v3757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:45 smithi144 ceph-mon[32129]: pgmap v3758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:45 smithi160 ceph-mon[36715]: pgmap v3758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:45 smithi183 ceph-mon[37273]: pgmap v3758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:46 smithi144 ceph-mon[32129]: pgmap v3759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:46 smithi144 ceph-mon[32129]: pgmap v3760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:46:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:46 smithi160 ceph-mon[36715]: pgmap v3759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:46 smithi160 ceph-mon[36715]: pgmap v3760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:46:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:46 smithi183 ceph-mon[37273]: pgmap v3759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:46 smithi183 ceph-mon[37273]: pgmap v3760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:46:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:47.144+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:47.145+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:47.145+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:47.145+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:46:47.145+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6049.663836 secs 2024-07-26T12:46:47.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6049.663836 secs 2024-07-26T12:46:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6049.663836 secs 2024-07-26T12:46:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:48 smithi144 ceph-mon[32129]: pgmap v3761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:46:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:48 smithi160 ceph-mon[36715]: pgmap v3761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:46:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:48 smithi183 ceph-mon[37273]: pgmap v3761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:46:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:50 smithi144 ceph-mon[32129]: pgmap v3762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:46:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:50 smithi160 ceph-mon[36715]: pgmap v3762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:46:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:50 smithi183 ceph-mon[37273]: pgmap v3762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6054.663953 secs 2024-07-26T12:46:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:52 smithi160 ceph-mon[36715]: pgmap v3763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6054.663953 secs 2024-07-26T12:46:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:52 smithi183 ceph-mon[37273]: pgmap v3763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6054.663953 secs 2024-07-26T12:46:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:52 smithi144 ceph-mon[32129]: pgmap v3763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:46:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:54 smithi160 ceph-mon[36715]: pgmap v3764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:46:54.877 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:54 smithi144 ceph-mon[32129]: pgmap v3764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:46:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:54 smithi183 ceph-mon[37273]: pgmap v3764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:46:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:56 smithi160 ceph-mon[36715]: pgmap v3765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:46:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:56 smithi183 ceph-mon[37273]: pgmap v3765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:46:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:56 smithi144 ceph-mon[32129]: pgmap v3765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:46:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6059.664102 secs 2024-07-26T12:46:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6059.664102 secs 2024-07-26T12:46:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6059.664102 secs 2024-07-26T12:46:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:58 smithi160 ceph-mon[36715]: pgmap v3766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:58 smithi183 ceph-mon[37273]: pgmap v3766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:58 smithi144 ceph-mon[32129]: pgmap v3766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:46:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:46:59 smithi160 ceph-mon[36715]: pgmap v3767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:46:59 smithi183 ceph-mon[37273]: pgmap v3767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:46:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:46:59 smithi144 ceph-mon[32129]: pgmap v3767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:02.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:02.161+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:02.161+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:02.162+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:02.162+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:02.162+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6064.664243 secs 2024-07-26T12:47:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:02 smithi160 ceph-mon[36715]: pgmap v3768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6064.664243 secs 2024-07-26T12:47:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:02 smithi183 ceph-mon[37273]: pgmap v3768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6064.664243 secs 2024-07-26T12:47:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:02 smithi144 ceph-mon[32129]: pgmap v3768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:03 smithi160 ceph-mon[36715]: pgmap v3769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:03 smithi183 ceph-mon[37273]: pgmap v3769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:03 smithi144 ceph-mon[32129]: pgmap v3769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:06 smithi160 ceph-mon[36715]: pgmap v3770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:06 smithi183 ceph-mon[37273]: pgmap v3770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:06 smithi144 ceph-mon[32129]: pgmap v3770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6069.664380 secs 2024-07-26T12:47:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:07 smithi160 ceph-mon[36715]: pgmap v3771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6069.664380 secs 2024-07-26T12:47:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:07 smithi183 ceph-mon[37273]: pgmap v3771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6069.664380 secs 2024-07-26T12:47:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:07 smithi144 ceph-mon[32129]: pgmap v3771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:09 smithi160 ceph-mon[36715]: pgmap v3772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:09 smithi183 ceph-mon[37273]: pgmap v3772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:09 smithi144 ceph-mon[32129]: pgmap v3772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6074.664498 secs 2024-07-26T12:47:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:11 smithi160 ceph-mon[36715]: pgmap v3773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6074.664498 secs 2024-07-26T12:47:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:11 smithi183 ceph-mon[37273]: pgmap v3773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6074.664498 secs 2024-07-26T12:47:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:11 smithi144 ceph-mon[32129]: pgmap v3773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:13 smithi160 ceph-mon[36715]: pgmap v3774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:13 smithi183 ceph-mon[37273]: pgmap v3774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:13 smithi144 ceph-mon[32129]: pgmap v3774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:15 smithi160 ceph-mon[36715]: pgmap v3775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:15 smithi183 ceph-mon[37273]: pgmap v3775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:15 smithi144 ceph-mon[32129]: pgmap v3775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6079.664585 secs 2024-07-26T12:47:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6079.664585 secs 2024-07-26T12:47:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6079.664585 secs 2024-07-26T12:47:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:17.173+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:17.174+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:17.174+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:17.174+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:17.174+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:17 smithi183 ceph-mon[37273]: pgmap v3776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:17 smithi144 ceph-mon[32129]: pgmap v3776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:17 smithi160 ceph-mon[36715]: pgmap v3776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:19 smithi160 ceph-mon[36715]: pgmap v3777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:19 smithi183 ceph-mon[37273]: pgmap v3777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:19 smithi144 ceph-mon[32129]: pgmap v3777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6084.664702 secs 2024-07-26T12:47:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:21 smithi160 ceph-mon[36715]: pgmap v3778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6084.664702 secs 2024-07-26T12:47:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:21 smithi183 ceph-mon[37273]: pgmap v3778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6084.664702 secs 2024-07-26T12:47:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:21 smithi144 ceph-mon[32129]: pgmap v3778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:24 smithi160 ceph-mon[36715]: pgmap v3779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:24 smithi183 ceph-mon[37273]: pgmap v3779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:24.673 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:24 smithi144 ceph-mon[32129]: pgmap v3779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:25 smithi160 ceph-mon[36715]: pgmap v3780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:25 smithi183 ceph-mon[37273]: pgmap v3780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:25 smithi144 ceph-mon[32129]: pgmap v3780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6089.664834 secs 2024-07-26T12:47:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6089.664834 secs 2024-07-26T12:47:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6089.664834 secs 2024-07-26T12:47:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:27 smithi183 ceph-mon[37273]: pgmap v3781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:27 smithi144 ceph-mon[32129]: pgmap v3781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:27 smithi160 ceph-mon[36715]: pgmap v3781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:29 smithi160 ceph-mon[36715]: pgmap v3782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:29 smithi183 ceph-mon[37273]: pgmap v3782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:29 smithi144 ceph-mon[32129]: pgmap v3782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6094.664969 secs 2024-07-26T12:47:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:31 smithi160 ceph-mon[36715]: pgmap v3783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6094.664969 secs 2024-07-26T12:47:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:31 smithi183 ceph-mon[37273]: pgmap v3783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6094.664969 secs 2024-07-26T12:47:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:31 smithi144 ceph-mon[32129]: pgmap v3783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:32.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:32.192+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:32.192+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:32.192+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:32.192+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:33 smithi160 ceph-mon[36715]: pgmap v3784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:33 smithi183 ceph-mon[37273]: pgmap v3784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:33 smithi144 ceph-mon[32129]: pgmap v3784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:35 smithi160 ceph-mon[36715]: pgmap v3785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:35 smithi183 ceph-mon[37273]: pgmap v3785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:35 smithi144 ceph-mon[32129]: pgmap v3785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6099.665071 secs 2024-07-26T12:47:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6099.665071 secs 2024-07-26T12:47:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6099.665071 secs 2024-07-26T12:47:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:37 smithi183 ceph-mon[37273]: pgmap v3786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:37 smithi144 ceph-mon[32129]: pgmap v3786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:37 smithi160 ceph-mon[36715]: pgmap v3786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:47:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:39 smithi160 ceph-mon[36715]: pgmap v3787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:39 smithi183 ceph-mon[37273]: pgmap v3787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:39 smithi144 ceph-mon[32129]: pgmap v3787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6104.665186 secs 2024-07-26T12:47:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:41 smithi160 ceph-mon[36715]: pgmap v3788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6104.665186 secs 2024-07-26T12:47:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:41 smithi183 ceph-mon[37273]: pgmap v3788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6104.665186 secs 2024-07-26T12:47:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:41 smithi144 ceph-mon[32129]: pgmap v3788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:47:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:43 smithi160 ceph-mon[36715]: pgmap v3789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:43 smithi183 ceph-mon[37273]: pgmap v3789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:43 smithi144 ceph-mon[32129]: pgmap v3789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:45 smithi160 ceph-mon[36715]: pgmap v3790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:45 smithi183 ceph-mon[37273]: pgmap v3790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:45 smithi144 ceph-mon[32129]: pgmap v3790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:47:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6109.665365 secs 2024-07-26T12:47:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6109.665365 secs 2024-07-26T12:47:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6109.665365 secs 2024-07-26T12:47:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.179+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:47:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.179+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:47:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.180+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:47:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.180+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T12:47:47.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.208+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.209+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.209+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:47:47.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:47 smithi183 ceph-mon[37273]: pgmap v3791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:47:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:47 smithi183 ceph-mon[37273]: pgmap v3792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:47:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:47:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:47 smithi144 ceph-mon[32129]: pgmap v3791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:47:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:47 smithi144 ceph-mon[32129]: pgmap v3792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:47:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:47:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:47 smithi160 ceph-mon[36715]: pgmap v3791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:47:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:47 smithi160 ceph-mon[36715]: pgmap v3792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:47:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:47:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:49 smithi144 ceph-mon[32129]: pgmap v3793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:47:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:49 smithi160 ceph-mon[36715]: pgmap v3793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:47:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:49 smithi183 ceph-mon[37273]: pgmap v3793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:47:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:51 smithi144 ceph-mon[32129]: pgmap v3794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:47:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6114.665481 secs 2024-07-26T12:47:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:51 smithi160 ceph-mon[36715]: pgmap v3794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:47:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6114.665481 secs 2024-07-26T12:47:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:51 smithi183 ceph-mon[37273]: pgmap v3794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:47:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6114.665481 secs 2024-07-26T12:47:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:53 smithi144 ceph-mon[32129]: pgmap v3795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:47:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:53 smithi160 ceph-mon[36715]: pgmap v3795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:47:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:53 smithi183 ceph-mon[37273]: pgmap v3795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:47:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:55 smithi160 ceph-mon[36715]: pgmap v3796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:47:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:55 smithi183 ceph-mon[37273]: pgmap v3796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:47:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:55 smithi144 ceph-mon[32129]: pgmap v3796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:47:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6119.665547 secs 2024-07-26T12:47:57.125 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6119.665547 secs 2024-07-26T12:47:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6119.665547 secs 2024-07-26T12:47:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:57 smithi160 ceph-mon[36715]: pgmap v3797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:47:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:57 smithi183 ceph-mon[37273]: pgmap v3797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:47:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:57 smithi144 ceph-mon[32129]: pgmap v3797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:48:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:47:59 smithi160 ceph-mon[36715]: pgmap v3798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:47:59 smithi183 ceph-mon[37273]: pgmap v3798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:47:59 smithi144 ceph-mon[32129]: pgmap v3798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:01 smithi160 ceph-mon[36715]: pgmap v3799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6124.665662 secs 2024-07-26T12:48:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:01 smithi183 ceph-mon[37273]: pgmap v3799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6124.665662 secs 2024-07-26T12:48:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:01 smithi144 ceph-mon[32129]: pgmap v3799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6124.665662 secs 2024-07-26T12:48:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:02.225+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:02.226+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:02.226+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:02.226+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:02.226+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:03 smithi160 ceph-mon[36715]: pgmap v3800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:03 smithi183 ceph-mon[37273]: pgmap v3800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:03 smithi144 ceph-mon[32129]: pgmap v3800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:05 smithi160 ceph-mon[36715]: pgmap v3801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:05 smithi183 ceph-mon[37273]: pgmap v3801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:05 smithi144 ceph-mon[32129]: pgmap v3801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6129.665787 secs 2024-07-26T12:48:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6129.665787 secs 2024-07-26T12:48:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6129.665787 secs 2024-07-26T12:48:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:07 smithi160 ceph-mon[36715]: pgmap v3802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:07 smithi183 ceph-mon[37273]: pgmap v3802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:07 smithi144 ceph-mon[32129]: pgmap v3802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:09 smithi160 ceph-mon[36715]: pgmap v3803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:09 smithi183 ceph-mon[37273]: pgmap v3803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:09 smithi144 ceph-mon[32129]: pgmap v3803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:11 smithi160 ceph-mon[36715]: pgmap v3804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6134.665883 secs 2024-07-26T12:48:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:11 smithi183 ceph-mon[37273]: pgmap v3804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6134.665883 secs 2024-07-26T12:48:12.189 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:11 smithi144 ceph-mon[32129]: pgmap v3804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:12.189 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6134.665883 secs 2024-07-26T12:48:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:13 smithi160 ceph-mon[36715]: pgmap v3805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:13 smithi183 ceph-mon[37273]: pgmap v3805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:13 smithi144 ceph-mon[32129]: pgmap v3805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:15 smithi160 ceph-mon[36715]: pgmap v3806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:15 smithi183 ceph-mon[37273]: pgmap v3806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:15 smithi144 ceph-mon[32129]: pgmap v3806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6139.666012 secs 2024-07-26T12:48:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6139.666012 secs 2024-07-26T12:48:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:17.241+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:17.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:17.242+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:17.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:17.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6139.666012 secs 2024-07-26T12:48:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:17 smithi183 ceph-mon[37273]: pgmap v3807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:18.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:17 smithi144 ceph-mon[32129]: pgmap v3807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:17 smithi160 ceph-mon[36715]: pgmap v3807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:19 smithi183 ceph-mon[37273]: pgmap v3808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:19 smithi144 ceph-mon[32129]: pgmap v3808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:19 smithi160 ceph-mon[36715]: pgmap v3808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:21 smithi183 ceph-mon[37273]: pgmap v3809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6144.666128 secs 2024-07-26T12:48:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:21 smithi144 ceph-mon[32129]: pgmap v3809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6144.666128 secs 2024-07-26T12:48:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:21 smithi160 ceph-mon[36715]: pgmap v3809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6144.666128 secs 2024-07-26T12:48:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:24 smithi160 ceph-mon[36715]: pgmap v3810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:24 smithi183 ceph-mon[37273]: pgmap v3810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:24 smithi144 ceph-mon[32129]: pgmap v3810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:26 smithi183 ceph-mon[37273]: pgmap v3811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:26 smithi144 ceph-mon[32129]: pgmap v3811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:26 smithi160 ceph-mon[36715]: pgmap v3811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6149.666248 secs 2024-07-26T12:48:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6149.666248 secs 2024-07-26T12:48:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6149.666248 secs 2024-07-26T12:48:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:28 smithi183 ceph-mon[37273]: pgmap v3812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:28 smithi144 ceph-mon[32129]: pgmap v3812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:28 smithi160 ceph-mon[36715]: pgmap v3812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:30 smithi183 ceph-mon[37273]: pgmap v3813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:30 smithi144 ceph-mon[32129]: pgmap v3813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:30 smithi160 ceph-mon[36715]: pgmap v3813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:32.258+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:32.259+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:32.259+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:32.259+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:32.260+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:32 smithi183 ceph-mon[37273]: pgmap v3814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6154.666373 secs 2024-07-26T12:48:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:32 smithi144 ceph-mon[32129]: pgmap v3814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6154.666373 secs 2024-07-26T12:48:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:32 smithi160 ceph-mon[36715]: pgmap v3814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6154.666373 secs 2024-07-26T12:48:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:34 smithi183 ceph-mon[37273]: pgmap v3815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:34 smithi144 ceph-mon[32129]: pgmap v3815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:34 smithi160 ceph-mon[36715]: pgmap v3815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:36 smithi183 ceph-mon[37273]: pgmap v3816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:36 smithi144 ceph-mon[32129]: pgmap v3816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:36 smithi160 ceph-mon[36715]: pgmap v3816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6159.666440 secs 2024-07-26T12:48:37.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6159.666440 secs 2024-07-26T12:48:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6159.666440 secs 2024-07-26T12:48:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:38 smithi183 ceph-mon[37273]: pgmap v3817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:38 smithi144 ceph-mon[32129]: pgmap v3817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:38 smithi160 ceph-mon[36715]: pgmap v3817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:40 smithi183 ceph-mon[37273]: pgmap v3818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:40 smithi144 ceph-mon[32129]: pgmap v3818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:40 smithi160 ceph-mon[36715]: pgmap v3818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:42 smithi183 ceph-mon[37273]: pgmap v3819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6164.666563 secs 2024-07-26T12:48:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:42 smithi144 ceph-mon[32129]: pgmap v3819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6164.666563 secs 2024-07-26T12:48:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:42 smithi160 ceph-mon[36715]: pgmap v3819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6164.666563 secs 2024-07-26T12:48:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:44 smithi183 ceph-mon[37273]: pgmap v3820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:44 smithi144 ceph-mon[32129]: pgmap v3820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:44 smithi160 ceph-mon[36715]: pgmap v3820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:48:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:46 smithi183 ceph-mon[37273]: pgmap v3821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:46 smithi144 ceph-mon[32129]: pgmap v3821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:46 smithi160 ceph-mon[36715]: pgmap v3821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:48:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:47.275+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:47.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:47.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:47.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:48:47.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6169.666642 secs 2024-07-26T12:48:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6169.666642 secs 2024-07-26T12:48:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6169.666642 secs 2024-07-26T12:48:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: pgmap v3822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:48.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: pgmap v3822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:48.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:48:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: pgmap v3822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:48:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:49 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:48:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:49 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:48:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:49 smithi144 ceph-mon[32129]: pgmap v3823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:48:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:49 smithi144 ceph-mon[32129]: pgmap v3824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:48:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:49 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:48:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:49 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:48:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:49 smithi160 ceph-mon[36715]: pgmap v3823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:48:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:49 smithi160 ceph-mon[36715]: pgmap v3824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:48:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:49 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:48:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:49 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:48:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:49 smithi183 ceph-mon[37273]: pgmap v3823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:48:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:49 smithi183 ceph-mon[37273]: pgmap v3824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:48:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:51 smithi144 ceph-mon[32129]: pgmap v3825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:48:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:51 smithi160 ceph-mon[36715]: pgmap v3825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:48:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:51 smithi183 ceph-mon[37273]: pgmap v3825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:48:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6174.666758 secs 2024-07-26T12:48:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6174.666758 secs 2024-07-26T12:48:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6174.666758 secs 2024-07-26T12:48:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:53 smithi144 ceph-mon[32129]: pgmap v3826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:48:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:53 smithi160 ceph-mon[36715]: pgmap v3826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:48:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:53 smithi183 ceph-mon[37273]: pgmap v3826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:48:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:55 smithi144 ceph-mon[32129]: pgmap v3827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:48:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:55 smithi160 ceph-mon[36715]: pgmap v3827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:48:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:55 smithi183 ceph-mon[37273]: pgmap v3827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:48:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6179.666871 secs 2024-07-26T12:48:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:57 smithi144 ceph-mon[32129]: pgmap v3828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:48:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6179.666871 secs 2024-07-26T12:48:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:57 smithi160 ceph-mon[36715]: pgmap v3828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:48:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6179.666871 secs 2024-07-26T12:48:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:57 smithi183 ceph-mon[37273]: pgmap v3828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:48:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:48:59 smithi144 ceph-mon[32129]: pgmap v3829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:48:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:48:59 smithi160 ceph-mon[36715]: pgmap v3829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:48:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:48:59 smithi183 ceph-mon[37273]: pgmap v3829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:49:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:01 smithi144 ceph-mon[32129]: pgmap v3830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:01 smithi160 ceph-mon[36715]: pgmap v3830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:01 smithi183 ceph-mon[37273]: pgmap v3830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:02.292+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:02.293+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:02.293+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:02.293+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:02.293+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6184.667004 secs 2024-07-26T12:49:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6184.667004 secs 2024-07-26T12:49:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6184.667004 secs 2024-07-26T12:49:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:03 smithi144 ceph-mon[32129]: pgmap v3831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:03 smithi160 ceph-mon[36715]: pgmap v3831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:03 smithi183 ceph-mon[37273]: pgmap v3831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:05 smithi160 ceph-mon[36715]: pgmap v3832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:05 smithi183 ceph-mon[37273]: pgmap v3832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:05 smithi144 ceph-mon[32129]: pgmap v3832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6189.667145 secs 2024-07-26T12:49:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:07 smithi160 ceph-mon[36715]: pgmap v3833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6189.667145 secs 2024-07-26T12:49:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:07 smithi183 ceph-mon[37273]: pgmap v3833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6189.667145 secs 2024-07-26T12:49:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:07 smithi144 ceph-mon[32129]: pgmap v3833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:09 smithi160 ceph-mon[36715]: pgmap v3834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:09 smithi183 ceph-mon[37273]: pgmap v3834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:09 smithi144 ceph-mon[32129]: pgmap v3834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:11 smithi160 ceph-mon[36715]: pgmap v3835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:11 smithi183 ceph-mon[37273]: pgmap v3835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:11 smithi144 ceph-mon[32129]: pgmap v3835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6194.667266 secs 2024-07-26T12:49:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6194.667266 secs 2024-07-26T12:49:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6194.667266 secs 2024-07-26T12:49:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:13 smithi160 ceph-mon[36715]: pgmap v3836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:13 smithi183 ceph-mon[37273]: pgmap v3836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:13 smithi144 ceph-mon[32129]: pgmap v3836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:15 smithi160 ceph-mon[36715]: pgmap v3837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:15 smithi183 ceph-mon[37273]: pgmap v3837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:15 smithi144 ceph-mon[32129]: pgmap v3837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:17.308+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:17.309+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:17.309+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:17.309+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:17.309+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6199.667395 secs 2024-07-26T12:49:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:17 smithi160 ceph-mon[36715]: pgmap v3838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6199.667395 secs 2024-07-26T12:49:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:17 smithi183 ceph-mon[37273]: pgmap v3838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6199.667395 secs 2024-07-26T12:49:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:17 smithi144 ceph-mon[32129]: pgmap v3838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:19 smithi160 ceph-mon[36715]: pgmap v3839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:19 smithi183 ceph-mon[37273]: pgmap v3839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:19 smithi144 ceph-mon[32129]: pgmap v3839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:21 smithi160 ceph-mon[36715]: pgmap v3840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:21 smithi183 ceph-mon[37273]: pgmap v3840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:21 smithi144 ceph-mon[32129]: pgmap v3840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6204.667514 secs 2024-07-26T12:49:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6204.667514 secs 2024-07-26T12:49:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6204.667514 secs 2024-07-26T12:49:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:23 smithi160 ceph-mon[36715]: pgmap v3841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:23 smithi183 ceph-mon[37273]: pgmap v3841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:23 smithi144 ceph-mon[32129]: pgmap v3841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:24 smithi160 ceph-mon[36715]: pgmap v3842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:24 smithi183 ceph-mon[37273]: pgmap v3842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:24 smithi144 ceph-mon[32129]: pgmap v3842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6209.667630 secs 2024-07-26T12:49:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:26 smithi144 ceph-mon[32129]: pgmap v3843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6209.667630 secs 2024-07-26T12:49:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:26 smithi160 ceph-mon[36715]: pgmap v3843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6209.667630 secs 2024-07-26T12:49:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:26 smithi183 ceph-mon[37273]: pgmap v3843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:28 smithi160 ceph-mon[36715]: pgmap v3844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:28 smithi183 ceph-mon[37273]: pgmap v3844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:28 smithi144 ceph-mon[32129]: pgmap v3844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:31.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:30 smithi144 ceph-mon[32129]: pgmap v3845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:30 smithi160 ceph-mon[36715]: pgmap v3845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:30 smithi183 ceph-mon[37273]: pgmap v3845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6214.667752 secs 2024-07-26T12:49:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:32.321+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:32.321+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:32.321+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:32.321+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:32.322+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6214.667752 secs 2024-07-26T12:49:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6214.667752 secs 2024-07-26T12:49:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:32 smithi160 ceph-mon[36715]: pgmap v3846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:32 smithi183 ceph-mon[37273]: pgmap v3846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:32 smithi144 ceph-mon[32129]: pgmap v3846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:34 smithi160 ceph-mon[36715]: pgmap v3847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:34 smithi183 ceph-mon[37273]: pgmap v3847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:34 smithi144 ceph-mon[32129]: pgmap v3847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6219.667898 secs 2024-07-26T12:49:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:37 smithi160 ceph-mon[36715]: pgmap v3848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6219.667898 secs 2024-07-26T12:49:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:37 smithi183 ceph-mon[37273]: pgmap v3848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6219.667898 secs 2024-07-26T12:49:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:37 smithi144 ceph-mon[32129]: pgmap v3848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:39 smithi160 ceph-mon[36715]: pgmap v3849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:39 smithi183 ceph-mon[37273]: pgmap v3849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:39 smithi144 ceph-mon[32129]: pgmap v3849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:41 smithi160 ceph-mon[36715]: pgmap v3850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:41 smithi183 ceph-mon[37273]: pgmap v3850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:41 smithi144 ceph-mon[32129]: pgmap v3850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6224.668020 secs 2024-07-26T12:49:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6224.668020 secs 2024-07-26T12:49:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6224.668020 secs 2024-07-26T12:49:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:43 smithi160 ceph-mon[36715]: pgmap v3851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:43 smithi183 ceph-mon[37273]: pgmap v3851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:43 smithi144 ceph-mon[32129]: pgmap v3851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:49:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:45 smithi160 ceph-mon[36715]: pgmap v3852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:45 smithi183 ceph-mon[37273]: pgmap v3852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:45 smithi144 ceph-mon[32129]: pgmap v3852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6229.668136 secs 2024-07-26T12:49:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:47 smithi160 ceph-mon[36715]: pgmap v3853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:47.326 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:47.325+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:47.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:47.326+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:47.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:47.326+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:47.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:47.326+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:47.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:49:47.326+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6229.668136 secs 2024-07-26T12:49:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:47 smithi183 ceph-mon[37273]: pgmap v3853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6229.668136 secs 2024-07-26T12:49:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:47 smithi144 ceph-mon[32129]: pgmap v3853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:49:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: pgmap v3854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:49.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: pgmap v3854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:49.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: pgmap v3854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:49:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:49.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:49.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: pgmap v3855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: pgmap v3856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:49:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:49:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: pgmap v3855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: pgmap v3856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:49:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:49:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T12:49:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T12:49:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T12:49:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:49:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: pgmap v3855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: pgmap v3856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:49:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:49:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6234.668262 secs 2024-07-26T12:49:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:52 smithi183 ceph-mon[37273]: pgmap v3857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6234.668262 secs 2024-07-26T12:49:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:52 smithi144 ceph-mon[32129]: pgmap v3857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6234.668262 secs 2024-07-26T12:49:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:52 smithi160 ceph-mon[36715]: pgmap v3857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:54 smithi183 ceph-mon[37273]: pgmap v3858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:49:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:54 smithi144 ceph-mon[32129]: pgmap v3858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:49:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:54 smithi160 ceph-mon[36715]: pgmap v3858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:49:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:56 smithi183 ceph-mon[37273]: pgmap v3859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:49:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:56 smithi144 ceph-mon[32129]: pgmap v3859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:49:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:56 smithi160 ceph-mon[36715]: pgmap v3859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:49:57.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6239.668378 secs 2024-07-26T12:49:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6239.668378 secs 2024-07-26T12:49:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6239.668378 secs 2024-07-26T12:49:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:49:58 smithi183 ceph-mon[37273]: pgmap v3860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:49:58 smithi144 ceph-mon[32129]: pgmap v3860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:49:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:49:58 smithi160 ceph-mon[36715]: pgmap v3860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:00 smithi183 ceph-mon[37273]: pgmap v3861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:50:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:00 smithi144 ceph-mon[32129]: pgmap v3861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:50:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:00 smithi160 ceph-mon[36715]: pgmap v3861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:50:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:02.334+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:02.335+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:02.335+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:02.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:02.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6244.668472 secs 2024-07-26T12:50:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:02 smithi183 ceph-mon[37273]: pgmap v3862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6244.668472 secs 2024-07-26T12:50:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:02 smithi144 ceph-mon[32129]: pgmap v3862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6244.668472 secs 2024-07-26T12:50:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:02 smithi160 ceph-mon[36715]: pgmap v3862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:04 smithi183 ceph-mon[37273]: pgmap v3863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:04 smithi144 ceph-mon[32129]: pgmap v3863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:04 smithi160 ceph-mon[36715]: pgmap v3863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:06 smithi183 ceph-mon[37273]: pgmap v3864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:06 smithi144 ceph-mon[32129]: pgmap v3864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:06 smithi160 ceph-mon[36715]: pgmap v3864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6249.668594 secs 2024-07-26T12:50:07.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6249.668594 secs 2024-07-26T12:50:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6249.668594 secs 2024-07-26T12:50:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:08 smithi183 ceph-mon[37273]: pgmap v3865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:08 smithi144 ceph-mon[32129]: pgmap v3865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:08 smithi160 ceph-mon[36715]: pgmap v3865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:10 smithi183 ceph-mon[37273]: pgmap v3866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:10 smithi144 ceph-mon[32129]: pgmap v3866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:10 smithi160 ceph-mon[36715]: pgmap v3866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6254.668672 secs 2024-07-26T12:50:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:12 smithi144 ceph-mon[32129]: pgmap v3867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6254.668672 secs 2024-07-26T12:50:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:12 smithi160 ceph-mon[36715]: pgmap v3867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6254.668672 secs 2024-07-26T12:50:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:12 smithi183 ceph-mon[37273]: pgmap v3867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:14 smithi144 ceph-mon[32129]: pgmap v3868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:14 smithi160 ceph-mon[36715]: pgmap v3868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:14 smithi183 ceph-mon[37273]: pgmap v3868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:16 smithi144 ceph-mon[32129]: pgmap v3869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:16 smithi160 ceph-mon[36715]: pgmap v3869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:16 smithi183 ceph-mon[37273]: pgmap v3869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:17.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:17.351+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:17.351+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:17.351+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:17.351+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6259.668820 secs 2024-07-26T12:50:17.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6259.668820 secs 2024-07-26T12:50:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6259.668820 secs 2024-07-26T12:50:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:18 smithi144 ceph-mon[32129]: pgmap v3870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:18 smithi160 ceph-mon[36715]: pgmap v3870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:18 smithi183 ceph-mon[37273]: pgmap v3870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:20 smithi144 ceph-mon[32129]: pgmap v3871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:20 smithi160 ceph-mon[36715]: pgmap v3871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:20 smithi183 ceph-mon[37273]: pgmap v3871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6264.668935 secs 2024-07-26T12:50:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:22 smithi144 ceph-mon[32129]: pgmap v3872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6264.668935 secs 2024-07-26T12:50:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:22 smithi160 ceph-mon[36715]: pgmap v3872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6264.668935 secs 2024-07-26T12:50:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:22 smithi183 ceph-mon[37273]: pgmap v3872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:24 smithi144 ceph-mon[32129]: pgmap v3873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:24 smithi160 ceph-mon[36715]: pgmap v3873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:24 smithi183 ceph-mon[37273]: pgmap v3873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:26 smithi144 ceph-mon[32129]: pgmap v3874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:26 smithi160 ceph-mon[36715]: pgmap v3874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:26 smithi183 ceph-mon[37273]: pgmap v3874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6269.669053 secs 2024-07-26T12:50:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6269.669053 secs 2024-07-26T12:50:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6269.669053 secs 2024-07-26T12:50:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:28 smithi144 ceph-mon[32129]: pgmap v3875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:28 smithi160 ceph-mon[36715]: pgmap v3875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:28 smithi183 ceph-mon[37273]: pgmap v3875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:30 smithi144 ceph-mon[32129]: pgmap v3876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:30 smithi160 ceph-mon[36715]: pgmap v3876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:30 smithi183 ceph-mon[37273]: pgmap v3876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:32.368+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:32.369+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:32.369+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:32.369+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:32.369+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6274.669115 secs 2024-07-26T12:50:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:32 smithi144 ceph-mon[32129]: pgmap v3877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6274.669115 secs 2024-07-26T12:50:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:32 smithi160 ceph-mon[36715]: pgmap v3877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6274.669115 secs 2024-07-26T12:50:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:32 smithi183 ceph-mon[37273]: pgmap v3877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:34 smithi144 ceph-mon[32129]: pgmap v3878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:34 smithi160 ceph-mon[36715]: pgmap v3878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:34 smithi183 ceph-mon[37273]: pgmap v3878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:36 smithi160 ceph-mon[36715]: pgmap v3879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:36 smithi183 ceph-mon[37273]: pgmap v3879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:36 smithi144 ceph-mon[32129]: pgmap v3879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6279.669204 secs 2024-07-26T12:50:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6279.669204 secs 2024-07-26T12:50:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6279.669204 secs 2024-07-26T12:50:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:38 smithi160 ceph-mon[36715]: pgmap v3880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:38 smithi183 ceph-mon[37273]: pgmap v3880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:38 smithi144 ceph-mon[32129]: pgmap v3880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:40 smithi160 ceph-mon[36715]: pgmap v3881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:40 smithi183 ceph-mon[37273]: pgmap v3881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:40 smithi144 ceph-mon[32129]: pgmap v3881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6284.669344 secs 2024-07-26T12:50:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:42 smithi160 ceph-mon[36715]: pgmap v3882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6284.669344 secs 2024-07-26T12:50:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:42 smithi183 ceph-mon[37273]: pgmap v3882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6284.669344 secs 2024-07-26T12:50:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:42 smithi144 ceph-mon[32129]: pgmap v3882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:44 smithi160 ceph-mon[36715]: pgmap v3883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:44 smithi183 ceph-mon[37273]: pgmap v3883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:44 smithi144 ceph-mon[32129]: pgmap v3883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f74a80316d0 tx=0x7f74a8061000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f74a80316d0 tx=0x7f74a8061000 comp rx=0 tx=0).stop 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.112+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.113+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:46.115+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a8063070 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:46 smithi160 ceph-mon[36715]: pgmap v3884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:46 smithi183 ceph-mon[37273]: pgmap v3884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:46.767 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:46 smithi144 ceph-mon[32129]: pgmap v3884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.211+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:50:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.211+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T12:50:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.212+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T12:50:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.212+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:50:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.385+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.386+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.386+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.386+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:50:47.386+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:47.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6289.669481 secs 2024-07-26T12:50:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6289.669481 secs 2024-07-26T12:50:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6289.669481 secs 2024-07-26T12:50:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:48 smithi160 ceph-mon[36715]: pgmap v3885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:48 smithi183 ceph-mon[37273]: pgmap v3885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:48 smithi144 ceph-mon[32129]: pgmap v3885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:50:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:50 smithi160 ceph-mon[36715]: pgmap v3886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:50 smithi183 ceph-mon[37273]: pgmap v3886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:50 smithi144 ceph-mon[32129]: pgmap v3886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:50:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:51 smithi160 ceph-mon[36715]: pgmap v3887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:51 smithi160 ceph-mon[36715]: pgmap v3888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:50:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:51 smithi183 ceph-mon[37273]: pgmap v3887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:51 smithi183 ceph-mon[37273]: pgmap v3888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:50:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:51 smithi144 ceph-mon[32129]: pgmap v3887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:51 smithi144 ceph-mon[32129]: pgmap v3888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:50:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6294.669595 secs 2024-07-26T12:50:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6294.669595 secs 2024-07-26T12:50:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6294.669595 secs 2024-07-26T12:50:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:53 smithi160 ceph-mon[36715]: pgmap v3889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:53 smithi183 ceph-mon[37273]: pgmap v3889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:53 smithi144 ceph-mon[32129]: pgmap v3889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:55 smithi183 ceph-mon[37273]: pgmap v3890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:55 smithi144 ceph-mon[32129]: pgmap v3890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:55 smithi160 ceph-mon[36715]: pgmap v3890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:57.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6299.669709 secs 2024-07-26T12:50:57.341 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:57 smithi144 ceph-mon[32129]: pgmap v3891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6299.669709 secs 2024-07-26T12:50:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:57 smithi183 ceph-mon[37273]: pgmap v3891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6299.669709 secs 2024-07-26T12:50:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:57 smithi160 ceph-mon[36715]: pgmap v3891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:50:59 smithi183 ceph-mon[37273]: pgmap v3892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:50:59 smithi144 ceph-mon[32129]: pgmap v3892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:50:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:50:59 smithi160 ceph-mon[36715]: pgmap v3892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:51:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:01 smithi183 ceph-mon[37273]: pgmap v3893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:01 smithi144 ceph-mon[32129]: pgmap v3893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:01 smithi160 ceph-mon[36715]: pgmap v3893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:02.403 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:02.402+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:02.403+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:02.403+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:02.403+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:02.403+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6304.669839 secs 2024-07-26T12:51:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6304.669839 secs 2024-07-26T12:51:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6304.669839 secs 2024-07-26T12:51:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:03 smithi144 ceph-mon[32129]: pgmap v3894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:03 smithi183 ceph-mon[37273]: pgmap v3894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:03 smithi160 ceph-mon[36715]: pgmap v3894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:05 smithi183 ceph-mon[37273]: pgmap v3895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:05 smithi144 ceph-mon[32129]: pgmap v3895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:05 smithi160 ceph-mon[36715]: pgmap v3895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:07.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6309.669956 secs 2024-07-26T12:51:07.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:07 smithi160 ceph-mon[36715]: pgmap v3896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6309.669956 secs 2024-07-26T12:51:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:07 smithi183 ceph-mon[37273]: pgmap v3896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6309.669956 secs 2024-07-26T12:51:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:07 smithi144 ceph-mon[32129]: pgmap v3896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:09 smithi183 ceph-mon[37273]: pgmap v3897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:09 smithi144 ceph-mon[32129]: pgmap v3897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:09 smithi160 ceph-mon[36715]: pgmap v3897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:11 smithi183 ceph-mon[37273]: pgmap v3898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:11 smithi144 ceph-mon[32129]: pgmap v3898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:11 smithi160 ceph-mon[36715]: pgmap v3898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6314.670080 secs 2024-07-26T12:51:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6314.670080 secs 2024-07-26T12:51:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6314.670080 secs 2024-07-26T12:51:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:13 smithi144 ceph-mon[32129]: pgmap v3899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:13 smithi160 ceph-mon[36715]: pgmap v3899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:13 smithi183 ceph-mon[37273]: pgmap v3899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:15 smithi144 ceph-mon[32129]: pgmap v3900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:15 smithi160 ceph-mon[36715]: pgmap v3900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:15 smithi183 ceph-mon[37273]: pgmap v3900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:17.418+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:17.418+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:17.418+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:17.418+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:17.419+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:17.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6319.670223 secs 2024-07-26T12:51:17.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:17 smithi183 ceph-mon[37273]: pgmap v3901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6319.670223 secs 2024-07-26T12:51:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:17 smithi144 ceph-mon[32129]: pgmap v3901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6319.670223 secs 2024-07-26T12:51:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:17 smithi160 ceph-mon[36715]: pgmap v3901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:19 smithi144 ceph-mon[32129]: pgmap v3902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:19 smithi160 ceph-mon[36715]: pgmap v3902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:19 smithi183 ceph-mon[37273]: pgmap v3902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:21 smithi144 ceph-mon[32129]: pgmap v3903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:21 smithi160 ceph-mon[36715]: pgmap v3903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:21 smithi183 ceph-mon[37273]: pgmap v3903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:22.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6324.670343 secs 2024-07-26T12:51:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6324.670343 secs 2024-07-26T12:51:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6324.670343 secs 2024-07-26T12:51:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:23 smithi144 ceph-mon[32129]: pgmap v3904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:23 smithi160 ceph-mon[36715]: pgmap v3904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:23 smithi183 ceph-mon[37273]: pgmap v3904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:25 smithi144 ceph-mon[32129]: pgmap v3905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:25 smithi160 ceph-mon[36715]: pgmap v3905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:25 smithi183 ceph-mon[37273]: pgmap v3905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6329.670438 secs 2024-07-26T12:51:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:27 smithi144 ceph-mon[32129]: pgmap v3906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6329.670438 secs 2024-07-26T12:51:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:27 smithi160 ceph-mon[36715]: pgmap v3906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6329.670438 secs 2024-07-26T12:51:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:27 smithi183 ceph-mon[37273]: pgmap v3906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:29 smithi144 ceph-mon[32129]: pgmap v3907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:29 smithi160 ceph-mon[36715]: pgmap v3907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:29 smithi183 ceph-mon[37273]: pgmap v3907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:31 smithi144 ceph-mon[32129]: pgmap v3908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:31 smithi160 ceph-mon[36715]: pgmap v3908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:31 smithi183 ceph-mon[37273]: pgmap v3908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:32.435+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:32.436+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:32.436+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:32.436+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:32.436+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6334.670535 secs 2024-07-26T12:51:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6334.670535 secs 2024-07-26T12:51:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6334.670535 secs 2024-07-26T12:51:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:33 smithi144 ceph-mon[32129]: pgmap v3909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:51:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:33 smithi160 ceph-mon[36715]: pgmap v3909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:51:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:33 smithi183 ceph-mon[37273]: pgmap v3909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:51:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:35 smithi160 ceph-mon[36715]: pgmap v3910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:35 smithi183 ceph-mon[37273]: pgmap v3910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:35 smithi144 ceph-mon[32129]: pgmap v3910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6339.670639 secs 2024-07-26T12:51:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:37 smithi160 ceph-mon[36715]: pgmap v3911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6339.670639 secs 2024-07-26T12:51:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:37 smithi183 ceph-mon[37273]: pgmap v3911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6339.670639 secs 2024-07-26T12:51:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:37 smithi144 ceph-mon[32129]: pgmap v3911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:39 smithi160 ceph-mon[36715]: pgmap v3912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:39 smithi183 ceph-mon[37273]: pgmap v3912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:39 smithi144 ceph-mon[32129]: pgmap v3912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:41 smithi160 ceph-mon[36715]: pgmap v3913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:41 smithi183 ceph-mon[37273]: pgmap v3913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:41 smithi144 ceph-mon[32129]: pgmap v3913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6344.670764 secs 2024-07-26T12:51:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6344.670764 secs 2024-07-26T12:51:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6344.670764 secs 2024-07-26T12:51:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:43 smithi160 ceph-mon[36715]: pgmap v3914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:43 smithi183 ceph-mon[37273]: pgmap v3914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:43 smithi144 ceph-mon[32129]: pgmap v3914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:45 smithi160 ceph-mon[36715]: pgmap v3915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:45 smithi183 ceph-mon[37273]: pgmap v3915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:45 smithi144 ceph-mon[32129]: pgmap v3915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:47.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:47.445+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:47.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:47.445+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:47.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:47.445+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:47.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:47.445+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:47.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:51:47.446+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6349.670891 secs 2024-07-26T12:51:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:47 smithi160 ceph-mon[36715]: pgmap v3916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6349.670891 secs 2024-07-26T12:51:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:47 smithi183 ceph-mon[37273]: pgmap v3916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6349.670891 secs 2024-07-26T12:51:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:47 smithi144 ceph-mon[32129]: pgmap v3916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:51:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:49 smithi160 ceph-mon[36715]: pgmap v3917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:49 smithi183 ceph-mon[37273]: pgmap v3917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:49 smithi144 ceph-mon[32129]: pgmap v3917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:50.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:51 smithi160 ceph-mon[36715]: pgmap v3918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:51 smithi183 ceph-mon[37273]: pgmap v3918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:51 smithi144 ceph-mon[32129]: pgmap v3918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:51:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6354.670975 secs 2024-07-26T12:51:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6354.670975 secs 2024-07-26T12:51:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6354.670975 secs 2024-07-26T12:51:52.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:52.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:52.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:51:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:53 smithi160 ceph-mon[36715]: pgmap v3919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:53 smithi160 ceph-mon[36715]: pgmap v3920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:53 smithi183 ceph-mon[37273]: pgmap v3919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:53 smithi183 ceph-mon[37273]: pgmap v3920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:53 smithi144 ceph-mon[32129]: pgmap v3919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:51:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:53 smithi144 ceph-mon[32129]: pgmap v3920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:55 smithi160 ceph-mon[36715]: pgmap v3921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:55 smithi183 ceph-mon[37273]: pgmap v3921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:55 smithi144 ceph-mon[32129]: pgmap v3921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:51:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:56 smithi183 ceph-mon[37273]: pgmap v3922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:56 smithi144 ceph-mon[32129]: pgmap v3922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:56 smithi160 ceph-mon[36715]: pgmap v3922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:51:57.874 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6359.671099 secs 2024-07-26T12:51:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6359.671099 secs 2024-07-26T12:51:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6359.671099 secs 2024-07-26T12:51:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:51:58 smithi183 ceph-mon[37273]: pgmap v3923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:51:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:51:58 smithi144 ceph-mon[32129]: pgmap v3923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:51:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:51:58 smithi160 ceph-mon[36715]: pgmap v3923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:52:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:01 smithi144 ceph-mon[32129]: pgmap v3924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:52:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:01 smithi160 ceph-mon[36715]: pgmap v3924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:52:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:01 smithi183 ceph-mon[37273]: pgmap v3924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:52:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:02.448+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:02.449+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:02.449+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:02.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:02.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6364.671218 secs 2024-07-26T12:52:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6364.671218 secs 2024-07-26T12:52:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6364.671218 secs 2024-07-26T12:52:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:03 smithi144 ceph-mon[32129]: pgmap v3925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:52:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:03 smithi160 ceph-mon[36715]: pgmap v3925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:52:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:03 smithi183 ceph-mon[37273]: pgmap v3925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:52:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:05 smithi144 ceph-mon[32129]: pgmap v3926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:05 smithi160 ceph-mon[36715]: pgmap v3926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:05 smithi183 ceph-mon[37273]: pgmap v3926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6369.671343 secs 2024-07-26T12:52:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:07 smithi144 ceph-mon[32129]: pgmap v3927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6369.671343 secs 2024-07-26T12:52:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:07 smithi160 ceph-mon[36715]: pgmap v3927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6369.671343 secs 2024-07-26T12:52:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:07 smithi183 ceph-mon[37273]: pgmap v3927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:09 smithi144 ceph-mon[32129]: pgmap v3928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:09 smithi160 ceph-mon[36715]: pgmap v3928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:09 smithi183 ceph-mon[37273]: pgmap v3928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:11 smithi144 ceph-mon[32129]: pgmap v3929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:11 smithi160 ceph-mon[36715]: pgmap v3929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:11 smithi183 ceph-mon[37273]: pgmap v3929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6374.671473 secs 2024-07-26T12:52:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6374.671473 secs 2024-07-26T12:52:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6374.671473 secs 2024-07-26T12:52:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:13 smithi144 ceph-mon[32129]: pgmap v3930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:13 smithi160 ceph-mon[36715]: pgmap v3930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:13 smithi183 ceph-mon[37273]: pgmap v3930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:15 smithi160 ceph-mon[36715]: pgmap v3931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:15 smithi183 ceph-mon[37273]: pgmap v3931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:15 smithi144 ceph-mon[32129]: pgmap v3931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:17.464+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:17.465+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:17.465+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:17.465+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:17.465+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6379.671599 secs 2024-07-26T12:52:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:17 smithi160 ceph-mon[36715]: pgmap v3932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6379.671599 secs 2024-07-26T12:52:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:17 smithi183 ceph-mon[37273]: pgmap v3932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6379.671599 secs 2024-07-26T12:52:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:17 smithi144 ceph-mon[32129]: pgmap v3932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:19 smithi160 ceph-mon[36715]: pgmap v3933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:19 smithi183 ceph-mon[37273]: pgmap v3933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:19 smithi144 ceph-mon[32129]: pgmap v3933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:21 smithi160 ceph-mon[36715]: pgmap v3934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:21 smithi183 ceph-mon[37273]: pgmap v3934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:21 smithi144 ceph-mon[32129]: pgmap v3934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6384.671723 secs 2024-07-26T12:52:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6384.671723 secs 2024-07-26T12:52:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6384.671723 secs 2024-07-26T12:52:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:23 smithi160 ceph-mon[36715]: pgmap v3935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:23 smithi183 ceph-mon[37273]: pgmap v3935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:23 smithi144 ceph-mon[32129]: pgmap v3935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:25 smithi160 ceph-mon[36715]: pgmap v3936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:25 smithi183 ceph-mon[37273]: pgmap v3936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:25 smithi144 ceph-mon[32129]: pgmap v3936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:27.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6389.671863 secs 2024-07-26T12:52:27.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:27 smithi144 ceph-mon[32129]: pgmap v3937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6389.671863 secs 2024-07-26T12:52:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:27 smithi160 ceph-mon[36715]: pgmap v3937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6389.671863 secs 2024-07-26T12:52:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:27 smithi183 ceph-mon[37273]: pgmap v3937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:29 smithi160 ceph-mon[36715]: pgmap v3938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:29 smithi183 ceph-mon[37273]: pgmap v3938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:29 smithi144 ceph-mon[32129]: pgmap v3938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:31 smithi160 ceph-mon[36715]: pgmap v3939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:31 smithi183 ceph-mon[37273]: pgmap v3939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:31 smithi144 ceph-mon[32129]: pgmap v3939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:32.472+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:32.473+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:32.473+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:32.473+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:32.473+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6394.671959 secs 2024-07-26T12:52:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6394.671959 secs 2024-07-26T12:52:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6394.671959 secs 2024-07-26T12:52:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:33 smithi160 ceph-mon[36715]: pgmap v3940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:33 smithi183 ceph-mon[37273]: pgmap v3940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:33 smithi144 ceph-mon[32129]: pgmap v3940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:35 smithi160 ceph-mon[36715]: pgmap v3941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:35 smithi183 ceph-mon[37273]: pgmap v3941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:35 smithi144 ceph-mon[32129]: pgmap v3941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6399.672080 secs 2024-07-26T12:52:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:37 smithi160 ceph-mon[36715]: pgmap v3942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6399.672080 secs 2024-07-26T12:52:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:37 smithi183 ceph-mon[37273]: pgmap v3942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6399.672080 secs 2024-07-26T12:52:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:37 smithi144 ceph-mon[32129]: pgmap v3942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:39 smithi160 ceph-mon[36715]: pgmap v3943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:39 smithi183 ceph-mon[37273]: pgmap v3943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:39 smithi144 ceph-mon[32129]: pgmap v3943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:41 smithi183 ceph-mon[37273]: pgmap v3944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:41 smithi144 ceph-mon[32129]: pgmap v3944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:41 smithi160 ceph-mon[36715]: pgmap v3944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6404.672215 secs 2024-07-26T12:52:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6404.672215 secs 2024-07-26T12:52:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6404.672215 secs 2024-07-26T12:52:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:43 smithi183 ceph-mon[37273]: pgmap v3945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:43 smithi144 ceph-mon[32129]: pgmap v3945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:43 smithi160 ceph-mon[36715]: pgmap v3945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:52:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:45 smithi183 ceph-mon[37273]: pgmap v3946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:45 smithi144 ceph-mon[32129]: pgmap v3946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:45 smithi160 ceph-mon[36715]: pgmap v3946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:47.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:47.489+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:47.490+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:47.490+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:47.490+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:52:47.490+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6409.672334 secs 2024-07-26T12:52:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:47 smithi183 ceph-mon[37273]: pgmap v3947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6409.672334 secs 2024-07-26T12:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:47 smithi144 ceph-mon[32129]: pgmap v3947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6409.672334 secs 2024-07-26T12:52:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:47 smithi160 ceph-mon[36715]: pgmap v3947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:49 smithi144 ceph-mon[32129]: pgmap v3948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:49 smithi160 ceph-mon[36715]: pgmap v3948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:49 smithi183 ceph-mon[37273]: pgmap v3948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:52:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:51 smithi144 ceph-mon[32129]: pgmap v3949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:51 smithi160 ceph-mon[36715]: pgmap v3949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:51 smithi183 ceph-mon[37273]: pgmap v3949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:52:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6414.672453 secs 2024-07-26T12:52:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6414.672453 secs 2024-07-26T12:52:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6414.672453 secs 2024-07-26T12:52:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:53.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:53 smithi183 ceph-mon[37273]: pgmap v3950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:52:53.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.619 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:53 smithi144 ceph-mon[32129]: pgmap v3950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:52:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:53 smithi160 ceph-mon[36715]: pgmap v3950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:52:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: pgmap v3951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: pgmap v3951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: pgmap v3951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6419.672519 secs 2024-07-26T12:52:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:56.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:56 smithi144 ceph-mon[32129]: pgmap v3952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6419.672519 secs 2024-07-26T12:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:56 smithi160 ceph-mon[36715]: pgmap v3952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6419.672519 secs 2024-07-26T12:52:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:56 smithi183 ceph-mon[37273]: pgmap v3952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: pgmap v3953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: pgmap v3954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:52:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: pgmap v3953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: pgmap v3954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:52:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:52:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: pgmap v3953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:52:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: pgmap v3954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:52:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:52:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:53:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:00 smithi160 ceph-mon[36715]: pgmap v3955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:53:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:00 smithi183 ceph-mon[37273]: pgmap v3955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:53:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:00 smithi144 ceph-mon[32129]: pgmap v3955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:53:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6424.672658 secs 2024-07-26T12:53:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6424.672658 secs 2024-07-26T12:53:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6424.672658 secs 2024-07-26T12:53:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.235+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:53:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.235+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:53:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.236+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:53:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.236+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:53:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.493+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.493+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.493+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:02.493+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:02 smithi160 ceph-mon[36715]: pgmap v3956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:02 smithi183 ceph-mon[37273]: pgmap v3956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:02 smithi144 ceph-mon[32129]: pgmap v3956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:04 smithi160 ceph-mon[36715]: pgmap v3957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:04 smithi183 ceph-mon[37273]: pgmap v3957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:04 smithi144 ceph-mon[32129]: pgmap v3957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:06 smithi160 ceph-mon[36715]: pgmap v3958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6429.672781 secs 2024-07-26T12:53:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:06 smithi183 ceph-mon[37273]: pgmap v3958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6429.672781 secs 2024-07-26T12:53:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:06 smithi144 ceph-mon[32129]: pgmap v3958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6429.672781 secs 2024-07-26T12:53:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:08 smithi160 ceph-mon[36715]: pgmap v3959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:53:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:08 smithi183 ceph-mon[37273]: pgmap v3959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:53:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:08 smithi144 ceph-mon[32129]: pgmap v3959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:53:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:10 smithi160 ceph-mon[36715]: pgmap v3960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:10 smithi183 ceph-mon[37273]: pgmap v3960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:10 smithi144 ceph-mon[32129]: pgmap v3960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6434.672902 secs 2024-07-26T12:53:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6434.672902 secs 2024-07-26T12:53:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6434.672902 secs 2024-07-26T12:53:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:12 smithi160 ceph-mon[36715]: pgmap v3961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:12 smithi183 ceph-mon[37273]: pgmap v3961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:12 smithi144 ceph-mon[32129]: pgmap v3961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:14 smithi160 ceph-mon[36715]: pgmap v3962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:14 smithi183 ceph-mon[37273]: pgmap v3962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:14 smithi144 ceph-mon[32129]: pgmap v3962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:16 smithi160 ceph-mon[36715]: pgmap v3963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6439.672971 secs 2024-07-26T12:53:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:16 smithi183 ceph-mon[37273]: pgmap v3963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6439.672971 secs 2024-07-26T12:53:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:16 smithi144 ceph-mon[32129]: pgmap v3963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6439.672971 secs 2024-07-26T12:53:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:17.494+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:17.495+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:17.495+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:17.495+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:17.495+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:18 smithi183 ceph-mon[37273]: pgmap v3964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:18 smithi144 ceph-mon[32129]: pgmap v3964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:18 smithi160 ceph-mon[36715]: pgmap v3964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:20 smithi183 ceph-mon[37273]: pgmap v3965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:20 smithi144 ceph-mon[32129]: pgmap v3965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:20 smithi160 ceph-mon[36715]: pgmap v3965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6444.673092 secs 2024-07-26T12:53:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6444.673092 secs 2024-07-26T12:53:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6444.673092 secs 2024-07-26T12:53:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:22 smithi183 ceph-mon[37273]: pgmap v3966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:22 smithi144 ceph-mon[32129]: pgmap v3966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:22 smithi160 ceph-mon[36715]: pgmap v3966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:24 smithi183 ceph-mon[37273]: pgmap v3967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:24 smithi144 ceph-mon[32129]: pgmap v3967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:24 smithi160 ceph-mon[36715]: pgmap v3967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6449.673208 secs 2024-07-26T12:53:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:26 smithi183 ceph-mon[37273]: pgmap v3968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6449.673208 secs 2024-07-26T12:53:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:26 smithi144 ceph-mon[32129]: pgmap v3968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6449.673208 secs 2024-07-26T12:53:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:26 smithi160 ceph-mon[36715]: pgmap v3968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:29 smithi144 ceph-mon[32129]: pgmap v3969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:29 smithi160 ceph-mon[36715]: pgmap v3969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:29 smithi183 ceph-mon[37273]: pgmap v3969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:31 smithi144 ceph-mon[32129]: pgmap v3970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:31 smithi160 ceph-mon[36715]: pgmap v3970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:31 smithi183 ceph-mon[37273]: pgmap v3970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:32.504+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:32.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:32.505+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:32.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:32.505+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:32.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:32.505+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:32.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:32.505+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6454.673311 secs 2024-07-26T12:53:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6454.673311 secs 2024-07-26T12:53:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6454.673311 secs 2024-07-26T12:53:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:33 smithi160 ceph-mon[36715]: pgmap v3971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:33 smithi183 ceph-mon[37273]: pgmap v3971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:33 smithi144 ceph-mon[32129]: pgmap v3971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:35 smithi160 ceph-mon[36715]: pgmap v3972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:35 smithi183 ceph-mon[37273]: pgmap v3972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:35 smithi144 ceph-mon[32129]: pgmap v3972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6459.673451 secs 2024-07-26T12:53:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:37 smithi160 ceph-mon[36715]: pgmap v3973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6459.673451 secs 2024-07-26T12:53:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:37 smithi183 ceph-mon[37273]: pgmap v3973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6459.673451 secs 2024-07-26T12:53:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:37 smithi144 ceph-mon[32129]: pgmap v3973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:39 smithi160 ceph-mon[36715]: pgmap v3974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:39 smithi183 ceph-mon[37273]: pgmap v3974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:39 smithi144 ceph-mon[32129]: pgmap v3974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:41 smithi160 ceph-mon[36715]: pgmap v3975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:41 smithi183 ceph-mon[37273]: pgmap v3975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:41 smithi144 ceph-mon[32129]: pgmap v3975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6464.673569 secs 2024-07-26T12:53:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6464.673569 secs 2024-07-26T12:53:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6464.673569 secs 2024-07-26T12:53:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:43 smithi160 ceph-mon[36715]: pgmap v3976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:43 smithi183 ceph-mon[37273]: pgmap v3976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:43 smithi144 ceph-mon[32129]: pgmap v3976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:45 smithi160 ceph-mon[36715]: pgmap v3977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:45 smithi183 ceph-mon[37273]: pgmap v3977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:45 smithi144 ceph-mon[32129]: pgmap v3977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.244+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:53:47.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.244+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:53:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.245+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x7f74b0109990 2024-07-26T12:53:47.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.245+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T12:53:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.520+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.521+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.521+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:53:47.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6469.673697 secs 2024-07-26T12:53:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:47 smithi160 ceph-mon[36715]: pgmap v3978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6469.673697 secs 2024-07-26T12:53:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:47 smithi183 ceph-mon[37273]: pgmap v3978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6469.673697 secs 2024-07-26T12:53:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:47 smithi144 ceph-mon[32129]: pgmap v3978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:49 smithi160 ceph-mon[36715]: pgmap v3979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:49 smithi183 ceph-mon[37273]: pgmap v3979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:49 smithi144 ceph-mon[32129]: pgmap v3979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:51 smithi160 ceph-mon[36715]: pgmap v3980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:51 smithi183 ceph-mon[37273]: pgmap v3980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:51 smithi144 ceph-mon[32129]: pgmap v3980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6474.673814 secs 2024-07-26T12:53:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6474.673814 secs 2024-07-26T12:53:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6474.673814 secs 2024-07-26T12:53:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:53 smithi160 ceph-mon[36715]: pgmap v3981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:53 smithi183 ceph-mon[37273]: pgmap v3981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:53 smithi144 ceph-mon[32129]: pgmap v3981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:53:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:55 smithi160 ceph-mon[36715]: pgmap v3982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:55 smithi183 ceph-mon[37273]: pgmap v3982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:55 smithi144 ceph-mon[32129]: pgmap v3982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6479.673928 secs 2024-07-26T12:53:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:57 smithi160 ceph-mon[36715]: pgmap v3983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6479.673928 secs 2024-07-26T12:53:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:57 smithi183 ceph-mon[37273]: pgmap v3983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6479.673928 secs 2024-07-26T12:53:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:57 smithi144 ceph-mon[32129]: pgmap v3983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:53:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:59.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:59 smithi144 ceph-mon[32129]: pgmap v3984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:59.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:59.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:59.873 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:53:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:53:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:59 smithi183 ceph-mon[37273]: pgmap v3984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:53:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:53:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:54:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:59 smithi160 ceph-mon[36715]: pgmap v3984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:53:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:54:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:00 smithi183 ceph-mon[37273]: pgmap v3985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:00 smithi144 ceph-mon[32129]: pgmap v3985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:00 smithi160 ceph-mon[36715]: pgmap v3985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:02.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:02.523+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:02.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:02.524+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:02.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:02.524+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:02.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:02.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:02.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:02.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6484.674042 secs 2024-07-26T12:54:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:02 smithi183 ceph-mon[37273]: pgmap v3986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6484.674042 secs 2024-07-26T12:54:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:02 smithi144 ceph-mon[32129]: pgmap v3986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6484.674042 secs 2024-07-26T12:54:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:02 smithi160 ceph-mon[36715]: pgmap v3986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:04 smithi183 ceph-mon[37273]: pgmap v3987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:54:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:04 smithi144 ceph-mon[32129]: pgmap v3987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:54:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:04 smithi160 ceph-mon[36715]: pgmap v3987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:54:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:05 smithi183 ceph-mon[37273]: pgmap v3988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:54:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:05 smithi144 ceph-mon[32129]: pgmap v3988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:54:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:05 smithi160 ceph-mon[36715]: pgmap v3988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:54:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6489.674112 secs 2024-07-26T12:54:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6489.674112 secs 2024-07-26T12:54:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6489.674112 secs 2024-07-26T12:54:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:07 smithi183 ceph-mon[37273]: pgmap v3989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:54:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:07 smithi144 ceph-mon[32129]: pgmap v3989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:54:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:07 smithi160 ceph-mon[36715]: pgmap v3989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:54:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:09 smithi144 ceph-mon[32129]: pgmap v3990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:09 smithi160 ceph-mon[36715]: pgmap v3990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:09 smithi183 ceph-mon[37273]: pgmap v3990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:54:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6494.674263 secs 2024-07-26T12:54:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:11 smithi160 ceph-mon[36715]: pgmap v3991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6494.674263 secs 2024-07-26T12:54:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:11 smithi183 ceph-mon[37273]: pgmap v3991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6494.674263 secs 2024-07-26T12:54:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:11 smithi144 ceph-mon[32129]: pgmap v3991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:13 smithi160 ceph-mon[36715]: pgmap v3992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:13 smithi183 ceph-mon[37273]: pgmap v3992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:13 smithi144 ceph-mon[32129]: pgmap v3992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:15 smithi160 ceph-mon[36715]: pgmap v3993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:15 smithi183 ceph-mon[37273]: pgmap v3993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:15 smithi144 ceph-mon[32129]: pgmap v3993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6499.674364 secs 2024-07-26T12:54:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6499.674364 secs 2024-07-26T12:54:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6499.674364 secs 2024-07-26T12:54:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:17.539+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:17.540+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:17.540+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:17.540+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:17.540+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:17 smithi160 ceph-mon[36715]: pgmap v3994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:17 smithi183 ceph-mon[37273]: pgmap v3994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:17 smithi144 ceph-mon[32129]: pgmap v3994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:20 smithi160 ceph-mon[36715]: pgmap v3995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:20 smithi183 ceph-mon[37273]: pgmap v3995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:20 smithi144 ceph-mon[32129]: pgmap v3995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6504.674506 secs 2024-07-26T12:54:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:22 smithi160 ceph-mon[36715]: pgmap v3996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6504.674506 secs 2024-07-26T12:54:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:22 smithi183 ceph-mon[37273]: pgmap v3996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6504.674506 secs 2024-07-26T12:54:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:22 smithi144 ceph-mon[32129]: pgmap v3996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:24 smithi160 ceph-mon[36715]: pgmap v3997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:24 smithi183 ceph-mon[37273]: pgmap v3997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:24 smithi144 ceph-mon[32129]: pgmap v3997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:26 smithi144 ceph-mon[32129]: pgmap v3998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:26 smithi160 ceph-mon[36715]: pgmap v3998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:26 smithi183 ceph-mon[37273]: pgmap v3998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6509.674587 secs 2024-07-26T12:54:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6509.674587 secs 2024-07-26T12:54:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6509.674587 secs 2024-07-26T12:54:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:28 smithi160 ceph-mon[36715]: pgmap v3999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:28 smithi183 ceph-mon[37273]: pgmap v3999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:28 smithi144 ceph-mon[32129]: pgmap v3999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:30 smithi160 ceph-mon[36715]: pgmap v4000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:30 smithi183 ceph-mon[37273]: pgmap v4000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:30 smithi144 ceph-mon[32129]: pgmap v4000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6514.674671 secs 2024-07-26T12:54:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:32 smithi160 ceph-mon[36715]: pgmap v4001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6514.674671 secs 2024-07-26T12:54:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:32 smithi183 ceph-mon[37273]: pgmap v4001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6514.674671 secs 2024-07-26T12:54:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:32 smithi144 ceph-mon[32129]: pgmap v4001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:32.557+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:32.558+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:32.558+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:32.558+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:32.558+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:34 smithi160 ceph-mon[36715]: pgmap v4002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:34 smithi183 ceph-mon[37273]: pgmap v4002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:34 smithi144 ceph-mon[32129]: pgmap v4002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:36 smithi160 ceph-mon[36715]: pgmap v4003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:36 smithi183 ceph-mon[37273]: pgmap v4003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:36 smithi144 ceph-mon[32129]: pgmap v4003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6519.674732 secs 2024-07-26T12:54:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6519.674732 secs 2024-07-26T12:54:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6519.674732 secs 2024-07-26T12:54:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:38 smithi183 ceph-mon[37273]: pgmap v4004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:38 smithi144 ceph-mon[32129]: pgmap v4004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:38 smithi160 ceph-mon[36715]: pgmap v4004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:40.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:40 smithi160 ceph-mon[36715]: pgmap v4005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:40 smithi183 ceph-mon[37273]: pgmap v4005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:40 smithi144 ceph-mon[32129]: pgmap v4005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6524.674850 secs 2024-07-26T12:54:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:42 smithi183 ceph-mon[37273]: pgmap v4006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6524.674850 secs 2024-07-26T12:54:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:42 smithi144 ceph-mon[32129]: pgmap v4006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6524.674850 secs 2024-07-26T12:54:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:42 smithi160 ceph-mon[36715]: pgmap v4006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:44 smithi183 ceph-mon[37273]: pgmap v4007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:44 smithi144 ceph-mon[32129]: pgmap v4007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:44 smithi160 ceph-mon[36715]: pgmap v4007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:46 smithi183 ceph-mon[37273]: pgmap v4008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:46 smithi144 ceph-mon[32129]: pgmap v4008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:46 smithi160 ceph-mon[36715]: pgmap v4008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6529.674993 secs 2024-07-26T12:54:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6529.674993 secs 2024-07-26T12:54:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6529.674993 secs 2024-07-26T12:54:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:47.573+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:47.574+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:47.574+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:47.574+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:54:47.574+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:48 smithi183 ceph-mon[37273]: pgmap v4009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:48 smithi144 ceph-mon[32129]: pgmap v4009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:48 smithi160 ceph-mon[36715]: pgmap v4009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:50 smithi183 ceph-mon[37273]: pgmap v4010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:50 smithi144 ceph-mon[32129]: pgmap v4010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:50 smithi160 ceph-mon[36715]: pgmap v4010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6534.675059 secs 2024-07-26T12:54:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:52 smithi183 ceph-mon[37273]: pgmap v4011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6534.675059 secs 2024-07-26T12:54:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:52 smithi144 ceph-mon[32129]: pgmap v4011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6534.675059 secs 2024-07-26T12:54:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:52 smithi160 ceph-mon[36715]: pgmap v4011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:54 smithi183 ceph-mon[37273]: pgmap v4012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:54 smithi144 ceph-mon[32129]: pgmap v4012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:54 smithi160 ceph-mon[36715]: pgmap v4012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:54:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:56 smithi183 ceph-mon[37273]: pgmap v4013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:56 smithi144 ceph-mon[32129]: pgmap v4013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:56 smithi160 ceph-mon[36715]: pgmap v4013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:54:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6539.675216 secs 2024-07-26T12:54:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6539.675216 secs 2024-07-26T12:54:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6539.675216 secs 2024-07-26T12:54:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:58 smithi144 ceph-mon[32129]: pgmap v4014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:58 smithi160 ceph-mon[36715]: pgmap v4014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:58 smithi183 ceph-mon[37273]: pgmap v4014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:54:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:54:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:54:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:54:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:00 smithi144 ceph-mon[32129]: pgmap v4015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:55:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:00 smithi160 ceph-mon[36715]: pgmap v4015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:55:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:00 smithi183 ceph-mon[37273]: pgmap v4015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:55:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:01 smithi144 ceph-mon[32129]: pgmap v4016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:55:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:01 smithi144 ceph-mon[32129]: pgmap v4017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:55:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:01 smithi160 ceph-mon[36715]: pgmap v4016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:55:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:01 smithi160 ceph-mon[36715]: pgmap v4017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:55:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:01 smithi183 ceph-mon[37273]: pgmap v4016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:55:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:01 smithi183 ceph-mon[37273]: pgmap v4017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:55:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6544.675311 secs 2024-07-26T12:55:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6544.675311 secs 2024-07-26T12:55:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:02.575+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:02.576+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:02.576+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:02.576+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:02.576+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6544.675311 secs 2024-07-26T12:55:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:03 smithi144 ceph-mon[32129]: pgmap v4018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:03 smithi160 ceph-mon[36715]: pgmap v4018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:03 smithi183 ceph-mon[37273]: pgmap v4018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:05 smithi144 ceph-mon[32129]: pgmap v4019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:05 smithi160 ceph-mon[36715]: pgmap v4019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:05 smithi183 ceph-mon[37273]: pgmap v4019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6549.675436 secs 2024-07-26T12:55:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:07 smithi144 ceph-mon[32129]: pgmap v4020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6549.675436 secs 2024-07-26T12:55:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:07 smithi160 ceph-mon[36715]: pgmap v4020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6549.675436 secs 2024-07-26T12:55:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:07 smithi183 ceph-mon[37273]: pgmap v4020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:55:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:09 smithi144 ceph-mon[32129]: pgmap v4021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:55:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:09 smithi160 ceph-mon[36715]: pgmap v4021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:55:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:09 smithi183 ceph-mon[37273]: pgmap v4021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:55:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:11 smithi144 ceph-mon[32129]: pgmap v4022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:55:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:11 smithi160 ceph-mon[36715]: pgmap v4022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:55:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:11 smithi183 ceph-mon[37273]: pgmap v4022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:55:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6554.675555 secs 2024-07-26T12:55:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6554.675555 secs 2024-07-26T12:55:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6554.675555 secs 2024-07-26T12:55:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:13 smithi144 ceph-mon[32129]: pgmap v4023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:13 smithi160 ceph-mon[36715]: pgmap v4023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:13 smithi183 ceph-mon[37273]: pgmap v4023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:15 smithi160 ceph-mon[36715]: pgmap v4024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:15 smithi183 ceph-mon[37273]: pgmap v4024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:15 smithi144 ceph-mon[32129]: pgmap v4024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6559.675672 secs 2024-07-26T12:55:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:17 smithi160 ceph-mon[36715]: pgmap v4025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:17.580+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:17.581+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:17.581+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:17.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:17.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6559.675672 secs 2024-07-26T12:55:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:17 smithi183 ceph-mon[37273]: pgmap v4025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6559.675672 secs 2024-07-26T12:55:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:17 smithi144 ceph-mon[32129]: pgmap v4025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:19 smithi160 ceph-mon[36715]: pgmap v4026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:19 smithi183 ceph-mon[37273]: pgmap v4026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:19 smithi144 ceph-mon[32129]: pgmap v4026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:21 smithi160 ceph-mon[36715]: pgmap v4027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:21 smithi183 ceph-mon[37273]: pgmap v4027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:21 smithi144 ceph-mon[32129]: pgmap v4027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6564.675754 secs 2024-07-26T12:55:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6564.675754 secs 2024-07-26T12:55:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6564.675754 secs 2024-07-26T12:55:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:23 smithi160 ceph-mon[36715]: pgmap v4028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:23 smithi183 ceph-mon[37273]: pgmap v4028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:23 smithi144 ceph-mon[32129]: pgmap v4028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:25 smithi160 ceph-mon[36715]: pgmap v4029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:25 smithi183 ceph-mon[37273]: pgmap v4029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:25 smithi144 ceph-mon[32129]: pgmap v4029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6569.675876 secs 2024-07-26T12:55:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:27 smithi160 ceph-mon[36715]: pgmap v4030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6569.675876 secs 2024-07-26T12:55:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:27 smithi183 ceph-mon[37273]: pgmap v4030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6569.675876 secs 2024-07-26T12:55:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:27 smithi144 ceph-mon[32129]: pgmap v4030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:29 smithi160 ceph-mon[36715]: pgmap v4031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:29 smithi183 ceph-mon[37273]: pgmap v4031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:29 smithi144 ceph-mon[32129]: pgmap v4031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:31.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:31 smithi144 ceph-mon[32129]: pgmap v4032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:31 smithi160 ceph-mon[36715]: pgmap v4032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:31 smithi183 ceph-mon[37273]: pgmap v4032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:32.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:32.585+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:32.585+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:32.585+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:32.585+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6574.676013 secs 2024-07-26T12:55:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6574.676013 secs 2024-07-26T12:55:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6574.676013 secs 2024-07-26T12:55:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:33 smithi160 ceph-mon[36715]: pgmap v4033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:33 smithi183 ceph-mon[37273]: pgmap v4033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:33 smithi144 ceph-mon[32129]: pgmap v4033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:35 smithi160 ceph-mon[36715]: pgmap v4034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:35 smithi183 ceph-mon[37273]: pgmap v4034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:35 smithi144 ceph-mon[32129]: pgmap v4034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6579.676141 secs 2024-07-26T12:55:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:37 smithi160 ceph-mon[36715]: pgmap v4035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6579.676141 secs 2024-07-26T12:55:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:37 smithi183 ceph-mon[37273]: pgmap v4035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6579.676141 secs 2024-07-26T12:55:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:37 smithi144 ceph-mon[32129]: pgmap v4035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:39 smithi160 ceph-mon[36715]: pgmap v4036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:39 smithi183 ceph-mon[37273]: pgmap v4036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:39 smithi144 ceph-mon[32129]: pgmap v4036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:41 smithi160 ceph-mon[36715]: pgmap v4037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:41 smithi183 ceph-mon[37273]: pgmap v4037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:41 smithi144 ceph-mon[32129]: pgmap v4037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6584.676279 secs 2024-07-26T12:55:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6584.676279 secs 2024-07-26T12:55:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6584.676279 secs 2024-07-26T12:55:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:43 smithi183 ceph-mon[37273]: pgmap v4038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:43 smithi144 ceph-mon[32129]: pgmap v4038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:43 smithi160 ceph-mon[36715]: pgmap v4038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:45 smithi183 ceph-mon[37273]: pgmap v4039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:45 smithi144 ceph-mon[32129]: pgmap v4039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:45 smithi160 ceph-mon[36715]: pgmap v4039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:47.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:47.601+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:47.601+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:47.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:55:47.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6589.676315 secs 2024-07-26T12:55:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:47 smithi183 ceph-mon[37273]: pgmap v4040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6589.676315 secs 2024-07-26T12:55:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:47 smithi144 ceph-mon[32129]: pgmap v4040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6589.676315 secs 2024-07-26T12:55:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:47 smithi160 ceph-mon[36715]: pgmap v4040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:49 smithi183 ceph-mon[37273]: pgmap v4041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:49 smithi144 ceph-mon[32129]: pgmap v4041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:49 smithi160 ceph-mon[36715]: pgmap v4041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:51 smithi183 ceph-mon[37273]: pgmap v4042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:55:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:51 smithi144 ceph-mon[32129]: pgmap v4042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:55:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:51 smithi160 ceph-mon[36715]: pgmap v4042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:55:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6594.676436 secs 2024-07-26T12:55:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6594.676436 secs 2024-07-26T12:55:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6594.676436 secs 2024-07-26T12:55:53.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:53 smithi183 ceph-mon[37273]: pgmap v4043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:53 smithi144 ceph-mon[32129]: pgmap v4043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:53 smithi160 ceph-mon[36715]: pgmap v4043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:54 smithi183 ceph-mon[37273]: pgmap v4044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:54 smithi144 ceph-mon[32129]: pgmap v4044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:54 smithi160 ceph-mon[36715]: pgmap v4044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:55:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6599.676534 secs 2024-07-26T12:55:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:57 smithi144 ceph-mon[32129]: pgmap v4045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6599.676534 secs 2024-07-26T12:55:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:57 smithi160 ceph-mon[36715]: pgmap v4045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6599.676534 secs 2024-07-26T12:55:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:57 smithi183 ceph-mon[37273]: pgmap v4045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:55:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:55:59 smithi144 ceph-mon[32129]: pgmap v4046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:55:59 smithi160 ceph-mon[36715]: pgmap v4046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:55:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:55:59 smithi183 ceph-mon[37273]: pgmap v4046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:01 smithi144 ceph-mon[32129]: pgmap v4047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:56:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:01 smithi160 ceph-mon[36715]: pgmap v4047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:56:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:01 smithi183 ceph-mon[37273]: pgmap v4047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:56:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6604.676639 secs 2024-07-26T12:56:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:02 smithi144 ceph-mon[32129]: pgmap v4048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:56:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:02 smithi144 ceph-mon[32129]: pgmap v4049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6604.676639 secs 2024-07-26T12:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:02 smithi160 ceph-mon[36715]: pgmap v4048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:02 smithi160 ceph-mon[36715]: pgmap v4049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:56:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:02.617+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:02.618+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:02.618+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:02.618+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:02.618+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6604.676639 secs 2024-07-26T12:56:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:02 smithi183 ceph-mon[37273]: pgmap v4048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:56:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:02 smithi183 ceph-mon[37273]: pgmap v4049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:56:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:04 smithi183 ceph-mon[37273]: pgmap v4050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:56:04.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:04 smithi144 ceph-mon[32129]: pgmap v4050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:56:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:04 smithi160 ceph-mon[36715]: pgmap v4050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:56:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:06 smithi183 ceph-mon[37273]: pgmap v4051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:06 smithi144 ceph-mon[32129]: pgmap v4051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:06 smithi160 ceph-mon[36715]: pgmap v4051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6609.676766 secs 2024-07-26T12:56:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6609.676766 secs 2024-07-26T12:56:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6609.676766 secs 2024-07-26T12:56:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:08 smithi183 ceph-mon[37273]: pgmap v4052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:08 smithi144 ceph-mon[32129]: pgmap v4052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:08 smithi160 ceph-mon[36715]: pgmap v4052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:56:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:10 smithi183 ceph-mon[37273]: pgmap v4053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:56:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:10 smithi144 ceph-mon[32129]: pgmap v4053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:56:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:10 smithi160 ceph-mon[36715]: pgmap v4053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:56:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6614.676895 secs 2024-07-26T12:56:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:12 smithi144 ceph-mon[32129]: pgmap v4054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:56:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6614.676895 secs 2024-07-26T12:56:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:12 smithi160 ceph-mon[36715]: pgmap v4054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:56:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6614.676895 secs 2024-07-26T12:56:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:12 smithi183 ceph-mon[37273]: pgmap v4054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:56:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:14 smithi144 ceph-mon[32129]: pgmap v4055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:14 smithi160 ceph-mon[36715]: pgmap v4055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:14 smithi183 ceph-mon[37273]: pgmap v4055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:16 smithi144 ceph-mon[32129]: pgmap v4056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:16 smithi160 ceph-mon[36715]: pgmap v4056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:16 smithi183 ceph-mon[37273]: pgmap v4056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:17.633+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:17.634+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:17.634+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:17.634+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:17.634+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6619.676969 secs 2024-07-26T12:56:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6619.676969 secs 2024-07-26T12:56:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6619.676969 secs 2024-07-26T12:56:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:18 smithi144 ceph-mon[32129]: pgmap v4057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:18 smithi160 ceph-mon[36715]: pgmap v4057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:18 smithi183 ceph-mon[37273]: pgmap v4057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:20 smithi144 ceph-mon[32129]: pgmap v4058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:20 smithi160 ceph-mon[36715]: pgmap v4058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:20 smithi183 ceph-mon[37273]: pgmap v4058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:22.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6624.677108 secs 2024-07-26T12:56:22.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:22 smithi183 ceph-mon[37273]: pgmap v4059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6624.677108 secs 2024-07-26T12:56:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:22 smithi144 ceph-mon[32129]: pgmap v4059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6624.677108 secs 2024-07-26T12:56:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:22 smithi160 ceph-mon[36715]: pgmap v4059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:24 smithi144 ceph-mon[32129]: pgmap v4060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:24 smithi160 ceph-mon[36715]: pgmap v4060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:24 smithi183 ceph-mon[37273]: pgmap v4060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:26 smithi144 ceph-mon[32129]: pgmap v4061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:26 smithi160 ceph-mon[36715]: pgmap v4061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:26 smithi183 ceph-mon[37273]: pgmap v4061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6629.677174 secs 2024-07-26T12:56:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6629.677174 secs 2024-07-26T12:56:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6629.677174 secs 2024-07-26T12:56:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:28 smithi144 ceph-mon[32129]: pgmap v4062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:28 smithi160 ceph-mon[36715]: pgmap v4062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:28 smithi183 ceph-mon[37273]: pgmap v4062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:30 smithi144 ceph-mon[32129]: pgmap v4063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:30 smithi160 ceph-mon[36715]: pgmap v4063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:30 smithi183 ceph-mon[37273]: pgmap v4063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:32.641+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:32.642+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:32.642+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:32.642+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:32.642+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6634.677351 secs 2024-07-26T12:56:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:32 smithi144 ceph-mon[32129]: pgmap v4064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6634.677351 secs 2024-07-26T12:56:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:32 smithi160 ceph-mon[36715]: pgmap v4064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6634.677351 secs 2024-07-26T12:56:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:32 smithi183 ceph-mon[37273]: pgmap v4064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:34 smithi144 ceph-mon[32129]: pgmap v4065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:34 smithi160 ceph-mon[36715]: pgmap v4065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:34 smithi183 ceph-mon[37273]: pgmap v4065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:36 smithi160 ceph-mon[36715]: pgmap v4066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:36 smithi183 ceph-mon[37273]: pgmap v4066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:36 smithi144 ceph-mon[32129]: pgmap v4066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6639.677471 secs 2024-07-26T12:56:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6639.677471 secs 2024-07-26T12:56:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6639.677471 secs 2024-07-26T12:56:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:38 smithi160 ceph-mon[36715]: pgmap v4067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:38 smithi183 ceph-mon[37273]: pgmap v4067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:38 smithi144 ceph-mon[32129]: pgmap v4067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:40 smithi160 ceph-mon[36715]: pgmap v4068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:40 smithi183 ceph-mon[37273]: pgmap v4068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:40 smithi144 ceph-mon[32129]: pgmap v4068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6644.677593 secs 2024-07-26T12:56:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:42 smithi160 ceph-mon[36715]: pgmap v4069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6644.677593 secs 2024-07-26T12:56:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:42 smithi183 ceph-mon[37273]: pgmap v4069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6644.677593 secs 2024-07-26T12:56:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:42 smithi144 ceph-mon[32129]: pgmap v4069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:44 smithi160 ceph-mon[36715]: pgmap v4070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:44 smithi183 ceph-mon[37273]: pgmap v4070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:44 smithi144 ceph-mon[32129]: pgmap v4070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:46 smithi160 ceph-mon[36715]: pgmap v4071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:46 smithi183 ceph-mon[37273]: pgmap v4071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:46 smithi144 ceph-mon[32129]: pgmap v4071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.276+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:56:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.276+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:56:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.277+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:56:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.277+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:56:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.658+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.659+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.659+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.659+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:56:47.659+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6649.677723 secs 2024-07-26T12:56:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6649.677723 secs 2024-07-26T12:56:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6649.677723 secs 2024-07-26T12:56:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:48 smithi160 ceph-mon[36715]: pgmap v4072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:48 smithi183 ceph-mon[37273]: pgmap v4072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:48 smithi144 ceph-mon[32129]: pgmap v4072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:50 smithi160 ceph-mon[36715]: pgmap v4073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:50 smithi183 ceph-mon[37273]: pgmap v4073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:50 smithi144 ceph-mon[32129]: pgmap v4073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6654.677821 secs 2024-07-26T12:56:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:52 smithi160 ceph-mon[36715]: pgmap v4074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6654.677821 secs 2024-07-26T12:56:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:52 smithi183 ceph-mon[37273]: pgmap v4074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6654.677821 secs 2024-07-26T12:56:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:52 smithi144 ceph-mon[32129]: pgmap v4074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:56:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:54 smithi160 ceph-mon[36715]: pgmap v4075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:54 smithi183 ceph-mon[37273]: pgmap v4075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:54 smithi144 ceph-mon[32129]: pgmap v4075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:56 smithi160 ceph-mon[36715]: pgmap v4076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:56 smithi183 ceph-mon[37273]: pgmap v4076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:56 smithi144 ceph-mon[32129]: pgmap v4076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:56:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6659.677955 secs 2024-07-26T12:56:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6659.677955 secs 2024-07-26T12:56:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6659.677955 secs 2024-07-26T12:56:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:56:58 smithi183 ceph-mon[37273]: pgmap v4077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:56:58 smithi144 ceph-mon[32129]: pgmap v4077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:56:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:56:58 smithi160 ceph-mon[36715]: pgmap v4077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:00 smithi183 ceph-mon[37273]: pgmap v4078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:00 smithi144 ceph-mon[32129]: pgmap v4078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:00 smithi160 ceph-mon[36715]: pgmap v4078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:02.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:02.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:02.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:02.672+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:02.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:02.672+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:02.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:02.672+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:02.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:02.672+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6664.678081 secs 2024-07-26T12:57:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:02 smithi183 ceph-mon[37273]: pgmap v4079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:57:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6664.678081 secs 2024-07-26T12:57:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:02 smithi144 ceph-mon[32129]: pgmap v4079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6664.678081 secs 2024-07-26T12:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:02 smithi160 ceph-mon[36715]: pgmap v4079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:57:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:03 smithi183 ceph-mon[37273]: pgmap v4080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:57:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:03 smithi183 ceph-mon[37273]: pgmap v4081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:57:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:03 smithi144 ceph-mon[32129]: pgmap v4080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:57:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:03 smithi144 ceph-mon[32129]: pgmap v4081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:57:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:03 smithi160 ceph-mon[36715]: pgmap v4080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:57:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:03 smithi160 ceph-mon[36715]: pgmap v4081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:57:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:05 smithi183 ceph-mon[37273]: pgmap v4082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:57:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:05 smithi144 ceph-mon[32129]: pgmap v4082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:57:06.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:05 smithi160 ceph-mon[36715]: pgmap v4082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:57:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6669.678203 secs 2024-07-26T12:57:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:07 smithi183 ceph-mon[37273]: pgmap v4083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:57:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6669.678203 secs 2024-07-26T12:57:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:07 smithi144 ceph-mon[32129]: pgmap v4083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:57:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6669.678203 secs 2024-07-26T12:57:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:07 smithi160 ceph-mon[36715]: pgmap v4083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:57:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:09 smithi183 ceph-mon[37273]: pgmap v4084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:57:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:09 smithi144 ceph-mon[32129]: pgmap v4084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:57:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:09 smithi160 ceph-mon[36715]: pgmap v4084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:57:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:11 smithi183 ceph-mon[37273]: pgmap v4085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:57:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:11 smithi144 ceph-mon[32129]: pgmap v4085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:57:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:11 smithi160 ceph-mon[36715]: pgmap v4085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:57:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6674.678330 secs 2024-07-26T12:57:12.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6674.678330 secs 2024-07-26T12:57:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6674.678330 secs 2024-07-26T12:57:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:13 smithi183 ceph-mon[37273]: pgmap v4086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:13 smithi144 ceph-mon[32129]: pgmap v4086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:13 smithi160 ceph-mon[36715]: pgmap v4086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:14 smithi144 ceph-mon[32129]: pgmap v4087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:15.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:14 smithi160 ceph-mon[36715]: pgmap v4087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:14 smithi183 ceph-mon[37273]: pgmap v4087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:17.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:17.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:17.680+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:17.680+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:17.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:17.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6679.678435 secs 2024-07-26T12:57:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:17 smithi160 ceph-mon[36715]: pgmap v4088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6679.678435 secs 2024-07-26T12:57:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:17 smithi183 ceph-mon[37273]: pgmap v4088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6679.678435 secs 2024-07-26T12:57:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:17 smithi144 ceph-mon[32129]: pgmap v4088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:19 smithi160 ceph-mon[36715]: pgmap v4089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:19 smithi183 ceph-mon[37273]: pgmap v4089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:19 smithi144 ceph-mon[32129]: pgmap v4089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:21 smithi183 ceph-mon[37273]: pgmap v4090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:21 smithi144 ceph-mon[32129]: pgmap v4090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:21 smithi160 ceph-mon[36715]: pgmap v4090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6684.678562 secs 2024-07-26T12:57:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6684.678562 secs 2024-07-26T12:57:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6684.678562 secs 2024-07-26T12:57:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:23 smithi183 ceph-mon[37273]: pgmap v4091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:23 smithi144 ceph-mon[32129]: pgmap v4091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:23 smithi160 ceph-mon[36715]: pgmap v4091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:25 smithi183 ceph-mon[37273]: pgmap v4092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:25 smithi144 ceph-mon[32129]: pgmap v4092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:25 smithi160 ceph-mon[36715]: pgmap v4092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6689.678682 secs 2024-07-26T12:57:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:27 smithi183 ceph-mon[37273]: pgmap v4093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6689.678682 secs 2024-07-26T12:57:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:27 smithi144 ceph-mon[32129]: pgmap v4093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6689.678682 secs 2024-07-26T12:57:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:27 smithi160 ceph-mon[36715]: pgmap v4093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:29 smithi183 ceph-mon[37273]: pgmap v4094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:29 smithi144 ceph-mon[32129]: pgmap v4094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:29 smithi160 ceph-mon[36715]: pgmap v4094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:31 smithi183 ceph-mon[37273]: pgmap v4095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:31 smithi144 ceph-mon[32129]: pgmap v4095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:31 smithi160 ceph-mon[36715]: pgmap v4095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:32.697+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:32.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:32.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:32.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:32.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:32.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6694.678782 secs 2024-07-26T12:57:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6694.678782 secs 2024-07-26T12:57:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6694.678782 secs 2024-07-26T12:57:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:33 smithi183 ceph-mon[37273]: pgmap v4096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:33 smithi144 ceph-mon[32129]: pgmap v4096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:33 smithi160 ceph-mon[36715]: pgmap v4096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:35 smithi183 ceph-mon[37273]: pgmap v4097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:35 smithi144 ceph-mon[32129]: pgmap v4097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:35 smithi160 ceph-mon[36715]: pgmap v4097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6699.678943 secs 2024-07-26T12:57:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6699.678943 secs 2024-07-26T12:57:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6699.678943 secs 2024-07-26T12:57:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:37 smithi183 ceph-mon[37273]: pgmap v4098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:37 smithi144 ceph-mon[32129]: pgmap v4098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:37 smithi160 ceph-mon[36715]: pgmap v4098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:38 smithi144 ceph-mon[32129]: pgmap v4099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:38 smithi160 ceph-mon[36715]: pgmap v4099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:38 smithi183 ceph-mon[37273]: pgmap v4099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:41 smithi183 ceph-mon[37273]: pgmap v4100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:41 smithi144 ceph-mon[32129]: pgmap v4100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:41 smithi160 ceph-mon[36715]: pgmap v4100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6704.679062 secs 2024-07-26T12:57:42.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6704.679062 secs 2024-07-26T12:57:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6704.679062 secs 2024-07-26T12:57:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:43 smithi183 ceph-mon[37273]: pgmap v4101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:43 smithi144 ceph-mon[32129]: pgmap v4101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:43 smithi160 ceph-mon[36715]: pgmap v4101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:45 smithi183 ceph-mon[37273]: pgmap v4102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:45 smithi144 ceph-mon[32129]: pgmap v4102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:45 smithi160 ceph-mon[36715]: pgmap v4102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:47.714+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:47.715+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:47.715+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:47.715+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:57:47.715+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6709.679188 secs 2024-07-26T12:57:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:47 smithi183 ceph-mon[37273]: pgmap v4103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6709.679188 secs 2024-07-26T12:57:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:47 smithi144 ceph-mon[32129]: pgmap v4103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6709.679188 secs 2024-07-26T12:57:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:47 smithi160 ceph-mon[36715]: pgmap v4103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:57:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:49 smithi183 ceph-mon[37273]: pgmap v4104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:49 smithi144 ceph-mon[32129]: pgmap v4104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:49 smithi160 ceph-mon[36715]: pgmap v4104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:51 smithi183 ceph-mon[37273]: pgmap v4105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:51 smithi144 ceph-mon[32129]: pgmap v4105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:51 smithi160 ceph-mon[36715]: pgmap v4105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:57:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6714.679339 secs 2024-07-26T12:57:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6714.679339 secs 2024-07-26T12:57:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6714.679339 secs 2024-07-26T12:57:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:53 smithi183 ceph-mon[37273]: pgmap v4106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:53 smithi144 ceph-mon[32129]: pgmap v4106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:53 smithi160 ceph-mon[36715]: pgmap v4106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:57:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:55 smithi183 ceph-mon[37273]: pgmap v4107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:57:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:55 smithi144 ceph-mon[32129]: pgmap v4107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:57:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:55 smithi160 ceph-mon[36715]: pgmap v4107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:57:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6719.679489 secs 2024-07-26T12:57:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6719.679489 secs 2024-07-26T12:57:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6719.679489 secs 2024-07-26T12:57:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:57 smithi144 ceph-mon[32129]: pgmap v4108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:57 smithi160 ceph-mon[36715]: pgmap v4108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:57 smithi183 ceph-mon[37273]: pgmap v4108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:57:58 smithi144 ceph-mon[32129]: pgmap v4109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:57:58 smithi160 ceph-mon[36715]: pgmap v4109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:57:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:57:58 smithi183 ceph-mon[37273]: pgmap v4109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:01 smithi183 ceph-mon[37273]: pgmap v4110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:01 smithi144 ceph-mon[32129]: pgmap v4110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:01 smithi160 ceph-mon[36715]: pgmap v4110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:02.718 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:02.716+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:02.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:02.717+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:02.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:02.717+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:02.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:02.717+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:02.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:02.718+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:02.870 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6724.679630 secs 2024-07-26T12:58:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6724.679630 secs 2024-07-26T12:58:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6724.679630 secs 2024-07-26T12:58:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:03 smithi183 ceph-mon[37273]: pgmap v4111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:03 smithi144 ceph-mon[32129]: pgmap v4111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:03 smithi160 ceph-mon[36715]: pgmap v4111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:58:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:58:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:58:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:05 smithi183 ceph-mon[37273]: pgmap v4112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:05 smithi183 ceph-mon[37273]: pgmap v4113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:58:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:05 smithi144 ceph-mon[32129]: pgmap v4112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:05 smithi144 ceph-mon[32129]: pgmap v4113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:58:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:05 smithi160 ceph-mon[36715]: pgmap v4112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:05 smithi160 ceph-mon[36715]: pgmap v4113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:58:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:07 smithi183 ceph-mon[37273]: pgmap v4114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:58:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6729.679715 secs 2024-07-26T12:58:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:07 smithi144 ceph-mon[32129]: pgmap v4114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:58:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6729.679715 secs 2024-07-26T12:58:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:07 smithi160 ceph-mon[36715]: pgmap v4114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:58:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6729.679715 secs 2024-07-26T12:58:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:09 smithi183 ceph-mon[37273]: pgmap v4115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:09 smithi144 ceph-mon[32129]: pgmap v4115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:09 smithi160 ceph-mon[36715]: pgmap v4115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:11 smithi183 ceph-mon[37273]: pgmap v4116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:11 smithi144 ceph-mon[32129]: pgmap v4116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:11 smithi160 ceph-mon[36715]: pgmap v4116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T12:58:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6734.679864 secs 2024-07-26T12:58:12.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6734.679864 secs 2024-07-26T12:58:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6734.679864 secs 2024-07-26T12:58:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:13 smithi144 ceph-mon[32129]: pgmap v4117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:58:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:13 smithi160 ceph-mon[36715]: pgmap v4117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:58:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:13 smithi183 ceph-mon[37273]: pgmap v4117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:58:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:14 smithi144 ceph-mon[32129]: pgmap v4118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:14 smithi160 ceph-mon[36715]: pgmap v4118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:14 smithi183 ceph-mon[37273]: pgmap v4118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:58:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6739.679969 secs 2024-07-26T12:58:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:16 smithi160 ceph-mon[36715]: pgmap v4119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6739.679969 secs 2024-07-26T12:58:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:16 smithi183 ceph-mon[37273]: pgmap v4119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6739.679969 secs 2024-07-26T12:58:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:16 smithi144 ceph-mon[32129]: pgmap v4119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:17.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:17.733+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:17.733+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:17.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:17.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:19.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:18 smithi144 ceph-mon[32129]: pgmap v4120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:18 smithi160 ceph-mon[36715]: pgmap v4120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:18 smithi183 ceph-mon[37273]: pgmap v4120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:20 smithi160 ceph-mon[36715]: pgmap v4121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:20 smithi183 ceph-mon[37273]: pgmap v4121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:20 smithi144 ceph-mon[32129]: pgmap v4121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:22.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6744.680098 secs 2024-07-26T12:58:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6744.680098 secs 2024-07-26T12:58:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6744.680098 secs 2024-07-26T12:58:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:22 smithi160 ceph-mon[36715]: pgmap v4122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:22 smithi183 ceph-mon[37273]: pgmap v4122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:22 smithi144 ceph-mon[32129]: pgmap v4122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:24 smithi160 ceph-mon[36715]: pgmap v4123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:24 smithi183 ceph-mon[37273]: pgmap v4123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:24 smithi144 ceph-mon[32129]: pgmap v4123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6749.680170 secs 2024-07-26T12:58:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:26 smithi160 ceph-mon[36715]: pgmap v4124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6749.680170 secs 2024-07-26T12:58:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:26 smithi183 ceph-mon[37273]: pgmap v4124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6749.680170 secs 2024-07-26T12:58:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:26 smithi144 ceph-mon[32129]: pgmap v4124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:28 smithi160 ceph-mon[36715]: pgmap v4125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:28 smithi183 ceph-mon[37273]: pgmap v4125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:28 smithi144 ceph-mon[32129]: pgmap v4125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:30 smithi160 ceph-mon[36715]: pgmap v4126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:30 smithi183 ceph-mon[37273]: pgmap v4126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:30 smithi144 ceph-mon[32129]: pgmap v4126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:32.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6754.680304 secs 2024-07-26T12:58:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6754.680304 secs 2024-07-26T12:58:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6754.680304 secs 2024-07-26T12:58:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:32.750+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:32.751+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:32.751+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:32.751+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:32.751+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:32 smithi183 ceph-mon[37273]: pgmap v4127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:32 smithi144 ceph-mon[32129]: pgmap v4127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:32 smithi160 ceph-mon[36715]: pgmap v4127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:34 smithi183 ceph-mon[37273]: pgmap v4128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:34 smithi144 ceph-mon[32129]: pgmap v4128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:34 smithi160 ceph-mon[36715]: pgmap v4128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6759.680434 secs 2024-07-26T12:58:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:36 smithi183 ceph-mon[37273]: pgmap v4129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6759.680434 secs 2024-07-26T12:58:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:36 smithi144 ceph-mon[32129]: pgmap v4129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6759.680434 secs 2024-07-26T12:58:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:36 smithi160 ceph-mon[36715]: pgmap v4129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:38 smithi183 ceph-mon[37273]: pgmap v4130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:38 smithi144 ceph-mon[32129]: pgmap v4130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:38 smithi160 ceph-mon[36715]: pgmap v4130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:40 smithi183 ceph-mon[37273]: pgmap v4131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:40 smithi144 ceph-mon[32129]: pgmap v4131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:40 smithi160 ceph-mon[36715]: pgmap v4131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6764.680525 secs 2024-07-26T12:58:42.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6764.680525 secs 2024-07-26T12:58:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6764.680525 secs 2024-07-26T12:58:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:42 smithi183 ceph-mon[37273]: pgmap v4132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:42 smithi144 ceph-mon[32129]: pgmap v4132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:42 smithi160 ceph-mon[36715]: pgmap v4132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:44 smithi183 ceph-mon[37273]: pgmap v4133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:44 smithi144 ceph-mon[32129]: pgmap v4133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:44 smithi160 ceph-mon[36715]: pgmap v4133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6769.680648 secs 2024-07-26T12:58:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:46 smithi183 ceph-mon[37273]: pgmap v4134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6769.680648 secs 2024-07-26T12:58:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:46 smithi144 ceph-mon[32129]: pgmap v4134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6769.680648 secs 2024-07-26T12:58:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:46 smithi160 ceph-mon[36715]: pgmap v4134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:47.767+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:47.768+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:47.768+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:47.768+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:58:47.768+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:48 smithi183 ceph-mon[37273]: pgmap v4135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:48 smithi144 ceph-mon[32129]: pgmap v4135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:48 smithi160 ceph-mon[36715]: pgmap v4135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:50 smithi144 ceph-mon[32129]: pgmap v4136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:50 smithi160 ceph-mon[36715]: pgmap v4136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:50 smithi183 ceph-mon[37273]: pgmap v4136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6774.680774 secs 2024-07-26T12:58:52.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6774.680774 secs 2024-07-26T12:58:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6774.680774 secs 2024-07-26T12:58:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:52 smithi144 ceph-mon[32129]: pgmap v4137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:52 smithi160 ceph-mon[36715]: pgmap v4137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:52 smithi183 ceph-mon[37273]: pgmap v4137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:58:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:54 smithi144 ceph-mon[32129]: pgmap v4138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:54 smithi160 ceph-mon[36715]: pgmap v4138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:54 smithi183 ceph-mon[37273]: pgmap v4138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6779.680893 secs 2024-07-26T12:58:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:56 smithi144 ceph-mon[32129]: pgmap v4139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6779.680893 secs 2024-07-26T12:58:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:56 smithi160 ceph-mon[36715]: pgmap v4139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6779.680893 secs 2024-07-26T12:58:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:56 smithi183 ceph-mon[37273]: pgmap v4139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:58:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:58:58 smithi144 ceph-mon[32129]: pgmap v4140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:58:58 smithi160 ceph-mon[36715]: pgmap v4140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:58:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:58:58 smithi183 ceph-mon[37273]: pgmap v4140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:00 smithi144 ceph-mon[32129]: pgmap v4141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:00 smithi160 ceph-mon[36715]: pgmap v4141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:00 smithi183 ceph-mon[37273]: pgmap v4141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:02.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6784.681011 secs 2024-07-26T12:59:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6784.681011 secs 2024-07-26T12:59:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6784.681011 secs 2024-07-26T12:59:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:02.773+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:02.774+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:02.774+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:02.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:02.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:02 smithi144 ceph-mon[32129]: pgmap v4142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:02 smithi160 ceph-mon[36715]: pgmap v4142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:02 smithi183 ceph-mon[37273]: pgmap v4142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:05.031 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:04 smithi160 ceph-mon[36715]: pgmap v4143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:04 smithi144 ceph-mon[32129]: pgmap v4143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:04 smithi183 ceph-mon[37273]: pgmap v4143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6789.681149 secs 2024-07-26T12:59:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:06 smithi144 ceph-mon[32129]: pgmap v4144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:07.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6789.681149 secs 2024-07-26T12:59:07.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:06 smithi160 ceph-mon[36715]: pgmap v4144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6789.681149 secs 2024-07-26T12:59:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:06 smithi183 ceph-mon[37273]: pgmap v4144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:09 smithi144 ceph-mon[32129]: pgmap v4145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:09 smithi183 ceph-mon[37273]: pgmap v4145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:09 smithi160 ceph-mon[36715]: pgmap v4145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: pgmap v4146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: pgmap v4147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: pgmap v4148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:59:11.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: pgmap v4146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: pgmap v4147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: pgmap v4148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:59:11.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: pgmap v4146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: pgmap v4147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: pgmap v4148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T12:59:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6794.681272 secs 2024-07-26T12:59:12.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6794.681272 secs 2024-07-26T12:59:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6794.681272 secs 2024-07-26T12:59:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:12 smithi183 ceph-mon[37273]: pgmap v4149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:12 smithi144 ceph-mon[32129]: pgmap v4149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:12 smithi160 ceph-mon[36715]: pgmap v4149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:14 smithi160 ceph-mon[36715]: pgmap v4150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:14 smithi183 ceph-mon[37273]: pgmap v4150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:14 smithi144 ceph-mon[32129]: pgmap v4150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6799.681402 secs 2024-07-26T12:59:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:16 smithi160 ceph-mon[36715]: pgmap v4151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:59:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6799.681402 secs 2024-07-26T12:59:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:16 smithi183 ceph-mon[37273]: pgmap v4151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:59:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6799.681402 secs 2024-07-26T12:59:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:16 smithi144 ceph-mon[32129]: pgmap v4151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:59:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:17.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:17.790+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:17.790+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:17.790+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:17.790+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:18 smithi160 ceph-mon[36715]: pgmap v4152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:18 smithi183 ceph-mon[37273]: pgmap v4152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:18 smithi144 ceph-mon[32129]: pgmap v4152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:59:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:20 smithi144 ceph-mon[32129]: pgmap v4153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:59:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:20 smithi160 ceph-mon[36715]: pgmap v4153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:59:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:20 smithi183 ceph-mon[37273]: pgmap v4153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:59:22.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6804.681500 secs 2024-07-26T12:59:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6804.681500 secs 2024-07-26T12:59:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6804.681500 secs 2024-07-26T12:59:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:23 smithi160 ceph-mon[36715]: pgmap v4154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:23 smithi183 ceph-mon[37273]: pgmap v4154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:23 smithi144 ceph-mon[32129]: pgmap v4154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:25 smithi183 ceph-mon[37273]: pgmap v4155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:25 smithi144 ceph-mon[32129]: pgmap v4155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:25 smithi160 ceph-mon[36715]: pgmap v4155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6809.681637 secs 2024-07-26T12:59:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:27 smithi183 ceph-mon[37273]: pgmap v4156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6809.681637 secs 2024-07-26T12:59:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:27 smithi144 ceph-mon[32129]: pgmap v4156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6809.681637 secs 2024-07-26T12:59:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:27 smithi160 ceph-mon[36715]: pgmap v4156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:29 smithi183 ceph-mon[37273]: pgmap v4157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:29 smithi144 ceph-mon[32129]: pgmap v4157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:29 smithi160 ceph-mon[36715]: pgmap v4157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:31 smithi183 ceph-mon[37273]: pgmap v4158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:31 smithi144 ceph-mon[32129]: pgmap v4158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:31 smithi160 ceph-mon[36715]: pgmap v4158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:32.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:32.808+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:32.808+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:32.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:32.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6814.681764 secs 2024-07-26T12:59:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6814.681764 secs 2024-07-26T12:59:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6814.681764 secs 2024-07-26T12:59:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:33 smithi144 ceph-mon[32129]: pgmap v4159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:33 smithi160 ceph-mon[36715]: pgmap v4159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:33 smithi183 ceph-mon[37273]: pgmap v4159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:34 smithi144 ceph-mon[32129]: pgmap v4160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:34 smithi160 ceph-mon[36715]: pgmap v4160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:34 smithi183 ceph-mon[37273]: pgmap v4160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6819.681895 secs 2024-07-26T12:59:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:36 smithi160 ceph-mon[36715]: pgmap v4161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6819.681895 secs 2024-07-26T12:59:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:36 smithi183 ceph-mon[37273]: pgmap v4161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6819.681895 secs 2024-07-26T12:59:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:36 smithi144 ceph-mon[32129]: pgmap v4161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:39 smithi160 ceph-mon[36715]: pgmap v4162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:39 smithi183 ceph-mon[37273]: pgmap v4162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:39.534 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:39 smithi144 ceph-mon[32129]: pgmap v4162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:41 smithi160 ceph-mon[36715]: pgmap v4163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:41 smithi183 ceph-mon[37273]: pgmap v4163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:41 smithi144 ceph-mon[32129]: pgmap v4163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6824.681973 secs 2024-07-26T12:59:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6824.681973 secs 2024-07-26T12:59:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6824.681973 secs 2024-07-26T12:59:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:43 smithi160 ceph-mon[36715]: pgmap v4164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:43 smithi183 ceph-mon[37273]: pgmap v4164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:43 smithi144 ceph-mon[32129]: pgmap v4164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:45 smithi160 ceph-mon[36715]: pgmap v4165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:45 smithi183 ceph-mon[37273]: pgmap v4165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:45 smithi144 ceph-mon[32129]: pgmap v4165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6829.682095 secs 2024-07-26T12:59:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:47 smithi160 ceph-mon[36715]: pgmap v4166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:47.309 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.308+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T12:59:47.309 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.308+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T12:59:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.309+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T12:59:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.309+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T12:59:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6829.682095 secs 2024-07-26T12:59:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:47 smithi183 ceph-mon[37273]: pgmap v4166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6829.682095 secs 2024-07-26T12:59:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:47 smithi144 ceph-mon[32129]: pgmap v4166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.820+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.821+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.821+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.821+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T12:59:47.821+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:49 smithi160 ceph-mon[36715]: pgmap v4167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:49 smithi183 ceph-mon[37273]: pgmap v4167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:49 smithi144 ceph-mon[32129]: pgmap v4167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:51 smithi160 ceph-mon[36715]: pgmap v4168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:51 smithi183 ceph-mon[37273]: pgmap v4168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:51 smithi144 ceph-mon[32129]: pgmap v4168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6834.682228 secs 2024-07-26T12:59:52.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6834.682228 secs 2024-07-26T12:59:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6834.682228 secs 2024-07-26T12:59:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:53 smithi183 ceph-mon[37273]: pgmap v4169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:53 smithi144 ceph-mon[32129]: pgmap v4169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:53 smithi160 ceph-mon[36715]: pgmap v4169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T12:59:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:55 smithi183 ceph-mon[37273]: pgmap v4170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:55 smithi144 ceph-mon[32129]: pgmap v4170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:55 smithi160 ceph-mon[36715]: pgmap v4170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6839.682356 secs 2024-07-26T12:59:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:57 smithi183 ceph-mon[37273]: pgmap v4171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6839.682356 secs 2024-07-26T12:59:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:57 smithi144 ceph-mon[32129]: pgmap v4171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6839.682356 secs 2024-07-26T12:59:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:57 smithi160 ceph-mon[36715]: pgmap v4171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T12:59:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 12:59:59 smithi183 ceph-mon[37273]: pgmap v4172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 12:59:59 smithi144 ceph-mon[32129]: pgmap v4172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:59:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 12:59:59 smithi160 ceph-mon[36715]: pgmap v4172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:01 smithi183 ceph-mon[37273]: pgmap v4173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:01 smithi144 ceph-mon[32129]: pgmap v4173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:01 smithi160 ceph-mon[36715]: pgmap v4173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:02.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6844.682489 secs 2024-07-26T13:00:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6844.682489 secs 2024-07-26T13:00:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6844.682489 secs 2024-07-26T13:00:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:02.837+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:02.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:02.838+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:02.838+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:02.838+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:02.839+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:03 smithi183 ceph-mon[37273]: pgmap v4174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:03 smithi144 ceph-mon[32129]: pgmap v4174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:03 smithi160 ceph-mon[36715]: pgmap v4174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:05 smithi183 ceph-mon[37273]: pgmap v4175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:05 smithi144 ceph-mon[32129]: pgmap v4175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:05 smithi160 ceph-mon[36715]: pgmap v4175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6849.682569 secs 2024-07-26T13:00:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:07 smithi183 ceph-mon[37273]: pgmap v4176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6849.682569 secs 2024-07-26T13:00:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:07 smithi144 ceph-mon[32129]: pgmap v4176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6849.682569 secs 2024-07-26T13:00:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:07 smithi160 ceph-mon[36715]: pgmap v4176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:09 smithi183 ceph-mon[37273]: pgmap v4177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:09 smithi144 ceph-mon[32129]: pgmap v4177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:09 smithi160 ceph-mon[36715]: pgmap v4177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:11.371 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: pgmap v4178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:00:11.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:11 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: pgmap v4178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:00:11.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:11 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: pgmap v4178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:00:11.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:11 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:00:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6854.682664 secs 2024-07-26T13:00:12.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6854.682664 secs 2024-07-26T13:00:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6854.682664 secs 2024-07-26T13:00:13.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:13 smithi183 ceph-mon[37273]: pgmap v4179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:13 smithi144 ceph-mon[32129]: pgmap v4179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:13 smithi160 ceph-mon[36715]: pgmap v4179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:00:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:15.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:15.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: pgmap v4180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:15.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: pgmap v4181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:15.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:00:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:00:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: pgmap v4180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: pgmap v4181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: pgmap v4180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: pgmap v4181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:15.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:00:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6859.682796 secs 2024-07-26T13:00:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:16 smithi144 ceph-mon[32129]: pgmap v4182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6859.682796 secs 2024-07-26T13:00:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:16 smithi160 ceph-mon[36715]: pgmap v4182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6859.682796 secs 2024-07-26T13:00:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:16 smithi183 ceph-mon[37273]: pgmap v4182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:17.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:17.842+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:17.842+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:17.842+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:17.842+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:18 smithi144 ceph-mon[32129]: pgmap v4183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:18 smithi160 ceph-mon[36715]: pgmap v4183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:18 smithi183 ceph-mon[37273]: pgmap v4183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:20 smithi160 ceph-mon[36715]: pgmap v4184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:20 smithi183 ceph-mon[37273]: pgmap v4184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:20 smithi144 ceph-mon[32129]: pgmap v4184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:22.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6864.682959 secs 2024-07-26T13:00:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6864.682959 secs 2024-07-26T13:00:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6864.682959 secs 2024-07-26T13:00:23.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:22 smithi183 ceph-mon[37273]: pgmap v4185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:22 smithi160 ceph-mon[36715]: pgmap v4185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:23.325 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:22 smithi144 ceph-mon[32129]: pgmap v4185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:00:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:24 smithi144 ceph-mon[32129]: pgmap v4186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:00:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:24 smithi160 ceph-mon[36715]: pgmap v4186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:00:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:24 smithi183 ceph-mon[37273]: pgmap v4186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:00:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6869.683091 secs 2024-07-26T13:00:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:26 smithi160 ceph-mon[36715]: pgmap v4187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6869.683091 secs 2024-07-26T13:00:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:26 smithi183 ceph-mon[37273]: pgmap v4187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6869.683091 secs 2024-07-26T13:00:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:26 smithi144 ceph-mon[32129]: pgmap v4187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:29 smithi160 ceph-mon[36715]: pgmap v4188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:29 smithi183 ceph-mon[37273]: pgmap v4188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:29 smithi144 ceph-mon[32129]: pgmap v4188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:31 smithi160 ceph-mon[36715]: pgmap v4189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:31 smithi183 ceph-mon[37273]: pgmap v4189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:31 smithi144 ceph-mon[32129]: pgmap v4189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6874.683230 secs 2024-07-26T13:00:32.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6874.683230 secs 2024-07-26T13:00:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6874.683230 secs 2024-07-26T13:00:32.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:32.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:32.860+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:32.860+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:32.860+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:32.860+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:33 smithi160 ceph-mon[36715]: pgmap v4190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:33.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:33 smithi183 ceph-mon[37273]: pgmap v4190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:33 smithi144 ceph-mon[32129]: pgmap v4190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:35 smithi160 ceph-mon[36715]: pgmap v4191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:35 smithi183 ceph-mon[37273]: pgmap v4191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:35 smithi144 ceph-mon[32129]: pgmap v4191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6879.683360 secs 2024-07-26T13:00:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:37 smithi160 ceph-mon[36715]: pgmap v4192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6879.683360 secs 2024-07-26T13:00:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:37 smithi183 ceph-mon[37273]: pgmap v4192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6879.683360 secs 2024-07-26T13:00:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:37 smithi144 ceph-mon[32129]: pgmap v4192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:39 smithi160 ceph-mon[36715]: pgmap v4193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:39 smithi183 ceph-mon[37273]: pgmap v4193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:39 smithi144 ceph-mon[32129]: pgmap v4193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:41 smithi160 ceph-mon[36715]: pgmap v4194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:41 smithi183 ceph-mon[37273]: pgmap v4194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:41 smithi144 ceph-mon[32129]: pgmap v4194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6884.683486 secs 2024-07-26T13:00:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6884.683486 secs 2024-07-26T13:00:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6884.683486 secs 2024-07-26T13:00:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:43 smithi183 ceph-mon[37273]: pgmap v4195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:43 smithi144 ceph-mon[32129]: pgmap v4195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:43 smithi160 ceph-mon[36715]: pgmap v4195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:45 smithi183 ceph-mon[37273]: pgmap v4196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:45 smithi144 ceph-mon[32129]: pgmap v4196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:45 smithi160 ceph-mon[36715]: pgmap v4196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6889.683618 secs 2024-07-26T13:00:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:47 smithi183 ceph-mon[37273]: pgmap v4197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6889.683618 secs 2024-07-26T13:00:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:47 smithi144 ceph-mon[32129]: pgmap v4197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6889.683618 secs 2024-07-26T13:00:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:47 smithi160 ceph-mon[36715]: pgmap v4197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:47.874 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:47.873+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:47.874+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:47.874+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:47.874+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:00:47.874+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:49 smithi183 ceph-mon[37273]: pgmap v4198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:49 smithi144 ceph-mon[32129]: pgmap v4198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:49 smithi160 ceph-mon[36715]: pgmap v4198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:51 smithi183 ceph-mon[37273]: pgmap v4199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:51 smithi144 ceph-mon[32129]: pgmap v4199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:51 smithi160 ceph-mon[36715]: pgmap v4199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6894.683731 secs 2024-07-26T13:00:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6894.683731 secs 2024-07-26T13:00:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6894.683731 secs 2024-07-26T13:00:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:53 smithi183 ceph-mon[37273]: pgmap v4200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:53 smithi144 ceph-mon[32129]: pgmap v4200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:53 smithi160 ceph-mon[36715]: pgmap v4200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:00:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:55 smithi183 ceph-mon[37273]: pgmap v4201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:55 smithi144 ceph-mon[32129]: pgmap v4201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:55 smithi160 ceph-mon[36715]: pgmap v4201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6899.683853 secs 2024-07-26T13:00:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:57 smithi183 ceph-mon[37273]: pgmap v4202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6899.683853 secs 2024-07-26T13:00:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:57 smithi144 ceph-mon[32129]: pgmap v4202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6899.683853 secs 2024-07-26T13:00:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:57 smithi160 ceph-mon[36715]: pgmap v4202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:00:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:00:59 smithi183 ceph-mon[37273]: pgmap v4203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:00:59 smithi144 ceph-mon[32129]: pgmap v4203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:00:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:00:59 smithi160 ceph-mon[36715]: pgmap v4203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:01 smithi183 ceph-mon[37273]: pgmap v4204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:01.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:01 smithi160 ceph-mon[36715]: pgmap v4204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:01 smithi144 ceph-mon[32129]: pgmap v4204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6904.683981 secs 2024-07-26T13:01:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6904.683981 secs 2024-07-26T13:01:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6904.683981 secs 2024-07-26T13:01:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:02.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:02.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:02.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:02.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:02.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:03 smithi144 ceph-mon[32129]: pgmap v4205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:03 smithi160 ceph-mon[36715]: pgmap v4205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:03 smithi183 ceph-mon[37273]: pgmap v4205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:05 smithi144 ceph-mon[32129]: pgmap v4206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:05 smithi160 ceph-mon[36715]: pgmap v4206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:05 smithi183 ceph-mon[37273]: pgmap v4206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6909.684102 secs 2024-07-26T13:01:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:07 smithi144 ceph-mon[32129]: pgmap v4207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6909.684102 secs 2024-07-26T13:01:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:07 smithi160 ceph-mon[36715]: pgmap v4207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6909.684102 secs 2024-07-26T13:01:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:07 smithi183 ceph-mon[37273]: pgmap v4207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:09 smithi144 ceph-mon[32129]: pgmap v4208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:09 smithi160 ceph-mon[36715]: pgmap v4208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:09 smithi183 ceph-mon[37273]: pgmap v4208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:11 smithi144 ceph-mon[32129]: pgmap v4209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:11 smithi160 ceph-mon[36715]: pgmap v4209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:11 smithi183 ceph-mon[37273]: pgmap v4209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:12.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6914.684206 secs 2024-07-26T13:01:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6914.684206 secs 2024-07-26T13:01:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6914.684206 secs 2024-07-26T13:01:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:13 smithi144 ceph-mon[32129]: pgmap v4210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:13 smithi160 ceph-mon[36715]: pgmap v4210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:13 smithi183 ceph-mon[37273]: pgmap v4210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:15 smithi144 ceph-mon[32129]: pgmap v4211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:01:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:15 smithi160 ceph-mon[36715]: pgmap v4211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:01:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:15 smithi183 ceph-mon[37273]: pgmap v4211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:01:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:16 smithi144 ceph-mon[32129]: pgmap v4212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:01:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:16 smithi144 ceph-mon[32129]: pgmap v4213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:01:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:16 smithi160 ceph-mon[36715]: pgmap v4212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:01:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:16 smithi160 ceph-mon[36715]: pgmap v4213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:01:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:16 smithi183 ceph-mon[37273]: pgmap v4212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:01:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:16 smithi183 ceph-mon[37273]: pgmap v4213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:01:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.323+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:01:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.323+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:01:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.323+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:01:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.323+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d0e0 con 0x7f74b0109990 2024-07-26T13:01:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6919.684331 secs 2024-07-26T13:01:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6919.684331 secs 2024-07-26T13:01:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6919.684331 secs 2024-07-26T13:01:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.895+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.895+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:17.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:18 smithi144 ceph-mon[32129]: pgmap v4214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:01:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:18 smithi160 ceph-mon[36715]: pgmap v4214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:01:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:18 smithi183 ceph-mon[37273]: pgmap v4214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:01:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:20 smithi144 ceph-mon[32129]: pgmap v4215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:01:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:20 smithi160 ceph-mon[36715]: pgmap v4215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:01:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:20 smithi183 ceph-mon[37273]: pgmap v4215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:01:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6924.684405 secs 2024-07-26T13:01:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:22 smithi160 ceph-mon[36715]: pgmap v4216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6924.684405 secs 2024-07-26T13:01:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:22 smithi183 ceph-mon[37273]: pgmap v4216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6924.684405 secs 2024-07-26T13:01:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:22 smithi144 ceph-mon[32129]: pgmap v4216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:24 smithi160 ceph-mon[36715]: pgmap v4217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:01:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:24 smithi183 ceph-mon[37273]: pgmap v4217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:01:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:24 smithi144 ceph-mon[32129]: pgmap v4217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:01:26.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:26 smithi144 ceph-mon[32129]: pgmap v4218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:01:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:26 smithi160 ceph-mon[36715]: pgmap v4218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:01:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:26 smithi183 ceph-mon[37273]: pgmap v4218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:01:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6929.684555 secs 2024-07-26T13:01:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6929.684555 secs 2024-07-26T13:01:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6929.684555 secs 2024-07-26T13:01:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:28 smithi160 ceph-mon[36715]: pgmap v4219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:01:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:28 smithi183 ceph-mon[37273]: pgmap v4219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:01:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:28 smithi144 ceph-mon[32129]: pgmap v4219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:01:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:30 smithi160 ceph-mon[36715]: pgmap v4220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:01:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:30 smithi183 ceph-mon[37273]: pgmap v4220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:01:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:30 smithi144 ceph-mon[32129]: pgmap v4220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:01:32.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6934.684705 secs 2024-07-26T13:01:32.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:32 smithi144 ceph-mon[32129]: pgmap v4221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:01:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6934.684705 secs 2024-07-26T13:01:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:32 smithi160 ceph-mon[36715]: pgmap v4221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:01:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6934.684705 secs 2024-07-26T13:01:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:32 smithi183 ceph-mon[37273]: pgmap v4221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:01:32.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:32.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:32.913+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:32.913+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:32.913+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:32.913+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:34 smithi160 ceph-mon[36715]: pgmap v4222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:34 smithi183 ceph-mon[37273]: pgmap v4222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:34 smithi144 ceph-mon[32129]: pgmap v4222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:36 smithi160 ceph-mon[36715]: pgmap v4223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:36 smithi183 ceph-mon[37273]: pgmap v4223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:36 smithi144 ceph-mon[32129]: pgmap v4223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6939.684832 secs 2024-07-26T13:01:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6939.684832 secs 2024-07-26T13:01:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6939.684832 secs 2024-07-26T13:01:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:38 smithi160 ceph-mon[36715]: pgmap v4224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:38 smithi183 ceph-mon[37273]: pgmap v4224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:38 smithi144 ceph-mon[32129]: pgmap v4224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:40 smithi160 ceph-mon[36715]: pgmap v4225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:40 smithi183 ceph-mon[37273]: pgmap v4225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:40 smithi144 ceph-mon[32129]: pgmap v4225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6944.684961 secs 2024-07-26T13:01:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:42 smithi144 ceph-mon[32129]: pgmap v4226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6944.684961 secs 2024-07-26T13:01:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:42 smithi160 ceph-mon[36715]: pgmap v4226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6944.684961 secs 2024-07-26T13:01:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:42 smithi183 ceph-mon[37273]: pgmap v4226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:44 smithi144 ceph-mon[32129]: pgmap v4227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:44 smithi160 ceph-mon[36715]: pgmap v4227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:44 smithi183 ceph-mon[37273]: pgmap v4227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:45 smithi144 ceph-mon[32129]: pgmap v4228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:45 smithi160 ceph-mon[36715]: pgmap v4228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:45 smithi183 ceph-mon[37273]: pgmap v4228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6949.685089 secs 2024-07-26T13:01:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6949.685089 secs 2024-07-26T13:01:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6949.685089 secs 2024-07-26T13:01:47.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:47.929+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:47.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:47.930+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:47.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:01:47.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:47 smithi144 ceph-mon[32129]: pgmap v4229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:47 smithi160 ceph-mon[36715]: pgmap v4229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:47 smithi183 ceph-mon[37273]: pgmap v4229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:50 smithi144 ceph-mon[32129]: pgmap v4230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:50 smithi160 ceph-mon[36715]: pgmap v4230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:50 smithi183 ceph-mon[37273]: pgmap v4230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6954.685194 secs 2024-07-26T13:01:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:52 smithi144 ceph-mon[32129]: pgmap v4231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6954.685194 secs 2024-07-26T13:01:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:52 smithi160 ceph-mon[36715]: pgmap v4231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6954.685194 secs 2024-07-26T13:01:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:52 smithi183 ceph-mon[37273]: pgmap v4231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:54 smithi144 ceph-mon[32129]: pgmap v4232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:54 smithi160 ceph-mon[36715]: pgmap v4232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:54 smithi183 ceph-mon[37273]: pgmap v4232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:01:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:56 smithi144 ceph-mon[32129]: pgmap v4233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:56 smithi160 ceph-mon[36715]: pgmap v4233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:56 smithi183 ceph-mon[37273]: pgmap v4233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6959.685334 secs 2024-07-26T13:01:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6959.685334 secs 2024-07-26T13:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6959.685334 secs 2024-07-26T13:01:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:01:58 smithi144 ceph-mon[32129]: pgmap v4234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:01:58 smithi160 ceph-mon[36715]: pgmap v4234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:01:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:01:58 smithi183 ceph-mon[37273]: pgmap v4234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:00 smithi160 ceph-mon[36715]: pgmap v4235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:00 smithi183 ceph-mon[37273]: pgmap v4235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:00 smithi144 ceph-mon[32129]: pgmap v4235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:02.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6964.685439 secs 2024-07-26T13:02:02.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:02 smithi144 ceph-mon[32129]: pgmap v4236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6964.685439 secs 2024-07-26T13:02:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:02 smithi160 ceph-mon[36715]: pgmap v4236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6964.685439 secs 2024-07-26T13:02:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:02 smithi183 ceph-mon[37273]: pgmap v4236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:02.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:02.934+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:02.934+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:02.934+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:02.935+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:04 smithi160 ceph-mon[36715]: pgmap v4237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:04 smithi183 ceph-mon[37273]: pgmap v4237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:04 smithi144 ceph-mon[32129]: pgmap v4237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:06 smithi160 ceph-mon[36715]: pgmap v4238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:02:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:06 smithi183 ceph-mon[37273]: pgmap v4238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:02:06.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:06 smithi144 ceph-mon[32129]: pgmap v4238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:02:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6969.685567 secs 2024-07-26T13:02:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6969.685567 secs 2024-07-26T13:02:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6969.685567 secs 2024-07-26T13:02:08.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:08 smithi160 ceph-mon[36715]: pgmap v4239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:08 smithi183 ceph-mon[37273]: pgmap v4239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:08 smithi144 ceph-mon[32129]: pgmap v4239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:10 smithi160 ceph-mon[36715]: pgmap v4240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:10 smithi183 ceph-mon[37273]: pgmap v4240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:10 smithi144 ceph-mon[32129]: pgmap v4240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6974.685688 secs 2024-07-26T13:02:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:12 smithi160 ceph-mon[36715]: pgmap v4241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6974.685688 secs 2024-07-26T13:02:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:12 smithi183 ceph-mon[37273]: pgmap v4241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6974.685688 secs 2024-07-26T13:02:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:12 smithi144 ceph-mon[32129]: pgmap v4241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:14 smithi160 ceph-mon[36715]: pgmap v4242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:14 smithi183 ceph-mon[37273]: pgmap v4242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:14 smithi144 ceph-mon[32129]: pgmap v4242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:16 smithi160 ceph-mon[36715]: pgmap v4243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:02:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:16 smithi183 ceph-mon[37273]: pgmap v4243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:02:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:16 smithi144 ceph-mon[32129]: pgmap v4243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:02:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6979.685767 secs 2024-07-26T13:02:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6979.685767 secs 2024-07-26T13:02:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6979.685767 secs 2024-07-26T13:02:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:02:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:17.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:17.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:17.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:17.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:17.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:17.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:17.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:18 smithi183 ceph-mon[37273]: pgmap v4244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:18 smithi183 ceph-mon[37273]: pgmap v4245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:18 smithi144 ceph-mon[32129]: pgmap v4244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:18 smithi144 ceph-mon[32129]: pgmap v4245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:18 smithi160 ceph-mon[36715]: pgmap v4244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:18 smithi160 ceph-mon[36715]: pgmap v4245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:20 smithi183 ceph-mon[37273]: pgmap v4246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:02:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:20 smithi144 ceph-mon[32129]: pgmap v4246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:02:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:20 smithi160 ceph-mon[36715]: pgmap v4246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:02:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6984.685888 secs 2024-07-26T13:02:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:22 smithi183 ceph-mon[37273]: pgmap v4247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6984.685888 secs 2024-07-26T13:02:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:22 smithi144 ceph-mon[32129]: pgmap v4247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6984.685888 secs 2024-07-26T13:02:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:22 smithi160 ceph-mon[36715]: pgmap v4247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:24 smithi183 ceph-mon[37273]: pgmap v4248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:24 smithi144 ceph-mon[32129]: pgmap v4248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:24 smithi160 ceph-mon[36715]: pgmap v4248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:02:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:26 smithi183 ceph-mon[37273]: pgmap v4249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:02:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:26 smithi144 ceph-mon[32129]: pgmap v4249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:02:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:26 smithi160 ceph-mon[36715]: pgmap v4249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:02:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6989.686019 secs 2024-07-26T13:02:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6989.686019 secs 2024-07-26T13:02:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6989.686019 secs 2024-07-26T13:02:28.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:28 smithi144 ceph-mon[32129]: pgmap v4250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:02:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:28 smithi183 ceph-mon[37273]: pgmap v4250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:02:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:28 smithi160 ceph-mon[36715]: pgmap v4250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:02:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:30 smithi183 ceph-mon[37273]: pgmap v4251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:30 smithi144 ceph-mon[32129]: pgmap v4251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:30 smithi160 ceph-mon[36715]: pgmap v4251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6994.686144 secs 2024-07-26T13:02:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:32 smithi183 ceph-mon[37273]: pgmap v4252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6994.686144 secs 2024-07-26T13:02:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:32 smithi144 ceph-mon[32129]: pgmap v4252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6994.686144 secs 2024-07-26T13:02:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:32 smithi160 ceph-mon[36715]: pgmap v4252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:32.965+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:32.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:32.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:32.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:32.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:34 smithi183 ceph-mon[37273]: pgmap v4253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:34 smithi144 ceph-mon[32129]: pgmap v4253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:34 smithi160 ceph-mon[36715]: pgmap v4253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:36 smithi183 ceph-mon[37273]: pgmap v4254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:36 smithi144 ceph-mon[32129]: pgmap v4254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:36 smithi160 ceph-mon[36715]: pgmap v4254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 6999.686267 secs 2024-07-26T13:02:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 6999.686267 secs 2024-07-26T13:02:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 6999.686267 secs 2024-07-26T13:02:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:38 smithi144 ceph-mon[32129]: pgmap v4255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:38 smithi160 ceph-mon[36715]: pgmap v4255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:38 smithi183 ceph-mon[37273]: pgmap v4255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:40 smithi144 ceph-mon[32129]: pgmap v4256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:40 smithi160 ceph-mon[36715]: pgmap v4256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:40 smithi183 ceph-mon[37273]: pgmap v4256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7004.686395 secs 2024-07-26T13:02:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:42 smithi144 ceph-mon[32129]: pgmap v4257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7004.686395 secs 2024-07-26T13:02:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:42 smithi160 ceph-mon[36715]: pgmap v4257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7004.686395 secs 2024-07-26T13:02:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:42 smithi183 ceph-mon[37273]: pgmap v4257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:44 smithi144 ceph-mon[32129]: pgmap v4258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:44 smithi160 ceph-mon[36715]: pgmap v4258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:44 smithi183 ceph-mon[37273]: pgmap v4258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:46 smithi144 ceph-mon[32129]: pgmap v4259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:46 smithi160 ceph-mon[36715]: pgmap v4259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:46 smithi183 ceph-mon[37273]: pgmap v4259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:47.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.340+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:02:47.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.340+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:02:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.341+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:02:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.341+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:02:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7009.686529 secs 2024-07-26T13:02:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7009.686529 secs 2024-07-26T13:02:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7009.686529 secs 2024-07-26T13:02:47.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:47.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.970+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:47.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.970+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:47.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:47.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:02:47.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:48 smithi144 ceph-mon[32129]: pgmap v4260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:48 smithi160 ceph-mon[36715]: pgmap v4260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:48 smithi183 ceph-mon[37273]: pgmap v4260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:50 smithi144 ceph-mon[32129]: pgmap v4261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:50 smithi160 ceph-mon[36715]: pgmap v4261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:50.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:50 smithi183 ceph-mon[37273]: pgmap v4261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:52.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7014.686657 secs 2024-07-26T13:02:52.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:52 smithi183 ceph-mon[37273]: pgmap v4262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7014.686657 secs 2024-07-26T13:02:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:52 smithi144 ceph-mon[32129]: pgmap v4262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7014.686657 secs 2024-07-26T13:02:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:52 smithi160 ceph-mon[36715]: pgmap v4262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:54 smithi144 ceph-mon[32129]: pgmap v4263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:54 smithi160 ceph-mon[36715]: pgmap v4263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:54 smithi183 ceph-mon[37273]: pgmap v4263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:02:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:56 smithi144 ceph-mon[32129]: pgmap v4264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:56 smithi160 ceph-mon[36715]: pgmap v4264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:56 smithi183 ceph-mon[37273]: pgmap v4264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:02:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7019.686784 secs 2024-07-26T13:02:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7019.686784 secs 2024-07-26T13:02:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7019.686784 secs 2024-07-26T13:02:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:02:58 smithi144 ceph-mon[32129]: pgmap v4265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:02:58 smithi160 ceph-mon[36715]: pgmap v4265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:02:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:02:58 smithi183 ceph-mon[37273]: pgmap v4265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:00 smithi160 ceph-mon[36715]: pgmap v4266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:00 smithi183 ceph-mon[37273]: pgmap v4266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:00 smithi144 ceph-mon[32129]: pgmap v4266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7024.686841 secs 2024-07-26T13:03:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:02 smithi160 ceph-mon[36715]: pgmap v4267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:02.818 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7024.686841 secs 2024-07-26T13:03:02.818 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:02 smithi144 ceph-mon[32129]: pgmap v4267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7024.686841 secs 2024-07-26T13:03:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:02 smithi183 ceph-mon[37273]: pgmap v4267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:02.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:02.987+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:02.987+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:02.987+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:02.987+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:04 smithi160 ceph-mon[36715]: pgmap v4268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:04 smithi183 ceph-mon[37273]: pgmap v4268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:04 smithi144 ceph-mon[32129]: pgmap v4268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:06 smithi160 ceph-mon[36715]: pgmap v4269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:06 smithi183 ceph-mon[37273]: pgmap v4269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:06 smithi144 ceph-mon[32129]: pgmap v4269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7029.686953 secs 2024-07-26T13:03:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7029.686953 secs 2024-07-26T13:03:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7029.686953 secs 2024-07-26T13:03:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:08 smithi160 ceph-mon[36715]: pgmap v4270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:08 smithi183 ceph-mon[37273]: pgmap v4270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:08 smithi144 ceph-mon[32129]: pgmap v4270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:10 smithi160 ceph-mon[36715]: pgmap v4271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:10 smithi183 ceph-mon[37273]: pgmap v4271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:10 smithi144 ceph-mon[32129]: pgmap v4271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7034.687099 secs 2024-07-26T13:03:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:12 smithi160 ceph-mon[36715]: pgmap v4272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7034.687099 secs 2024-07-26T13:03:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:12 smithi183 ceph-mon[37273]: pgmap v4272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7034.687099 secs 2024-07-26T13:03:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:12 smithi144 ceph-mon[32129]: pgmap v4272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:14 smithi160 ceph-mon[36715]: pgmap v4273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:14.846 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:14 smithi144 ceph-mon[32129]: pgmap v4273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:14 smithi183 ceph-mon[37273]: pgmap v4273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:16 smithi160 ceph-mon[36715]: pgmap v4274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:16 smithi183 ceph-mon[37273]: pgmap v4274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:16 smithi144 ceph-mon[32129]: pgmap v4274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7039.687230 secs 2024-07-26T13:03:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7039.687230 secs 2024-07-26T13:03:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7039.687230 secs 2024-07-26T13:03:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:18.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:18.001+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:18.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:18.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:18.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:18.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:18.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:18.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:18.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:18.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:18 smithi160 ceph-mon[36715]: pgmap v4275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:18 smithi183 ceph-mon[37273]: pgmap v4275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:18 smithi144 ceph-mon[32129]: pgmap v4275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:20 smithi160 ceph-mon[36715]: pgmap v4276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:20 smithi183 ceph-mon[37273]: pgmap v4276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:20 smithi144 ceph-mon[32129]: pgmap v4276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7044.687326 secs 2024-07-26T13:03:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:22 smithi160 ceph-mon[36715]: pgmap v4277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7044.687326 secs 2024-07-26T13:03:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:22 smithi183 ceph-mon[37273]: pgmap v4277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7044.687326 secs 2024-07-26T13:03:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:22 smithi144 ceph-mon[32129]: pgmap v4277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:23 smithi160 ceph-mon[36715]: pgmap v4278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:03:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:23 smithi160 ceph-mon[36715]: pgmap v4279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:03:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:23 smithi183 ceph-mon[37273]: pgmap v4278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:03:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:23 smithi183 ceph-mon[37273]: pgmap v4279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:03:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:23 smithi144 ceph-mon[32129]: pgmap v4278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:03:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:23 smithi144 ceph-mon[32129]: pgmap v4279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:03:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:03:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:24 smithi160 ceph-mon[36715]: pgmap v4280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:03:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:24 smithi183 ceph-mon[37273]: pgmap v4280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:03:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:24 smithi144 ceph-mon[32129]: pgmap v4280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:03:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7049.687425 secs 2024-07-26T13:03:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:27 smithi183 ceph-mon[37273]: pgmap v4281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:03:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7049.687425 secs 2024-07-26T13:03:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:27 smithi144 ceph-mon[32129]: pgmap v4281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:03:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7049.687425 secs 2024-07-26T13:03:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:27 smithi160 ceph-mon[36715]: pgmap v4281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:03:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:29 smithi183 ceph-mon[37273]: pgmap v4282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:29 smithi144 ceph-mon[32129]: pgmap v4282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:29 smithi160 ceph-mon[36715]: pgmap v4282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:31 smithi183 ceph-mon[37273]: pgmap v4283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:31.408 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:31 smithi144 ceph-mon[32129]: pgmap v4283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:31 smithi160 ceph-mon[36715]: pgmap v4283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7054.687567 secs 2024-07-26T13:03:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7054.687567 secs 2024-07-26T13:03:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7054.687567 secs 2024-07-26T13:03:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:33.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:33.019+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:33.019+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:33.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:33.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:33 smithi144 ceph-mon[32129]: pgmap v4284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:03:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:33 smithi160 ceph-mon[36715]: pgmap v4284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:03:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:33 smithi183 ceph-mon[37273]: pgmap v4284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:03:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:35 smithi144 ceph-mon[32129]: pgmap v4285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:35 smithi160 ceph-mon[36715]: pgmap v4285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:35 smithi183 ceph-mon[37273]: pgmap v4285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7059.687676 secs 2024-07-26T13:03:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:37 smithi144 ceph-mon[32129]: pgmap v4286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7059.687676 secs 2024-07-26T13:03:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:37 smithi160 ceph-mon[36715]: pgmap v4286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7059.687676 secs 2024-07-26T13:03:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:37 smithi183 ceph-mon[37273]: pgmap v4286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:39 smithi144 ceph-mon[32129]: pgmap v4287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:39 smithi160 ceph-mon[36715]: pgmap v4287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:39 smithi183 ceph-mon[37273]: pgmap v4287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:41 smithi144 ceph-mon[32129]: pgmap v4288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:41 smithi160 ceph-mon[36715]: pgmap v4288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:41 smithi183 ceph-mon[37273]: pgmap v4288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:42.448 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7064.687798 secs 2024-07-26T13:03:42.462 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7064.687798 secs 2024-07-26T13:03:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7064.687798 secs 2024-07-26T13:03:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:43 smithi144 ceph-mon[32129]: pgmap v4289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:43 smithi160 ceph-mon[36715]: pgmap v4289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:43 smithi183 ceph-mon[37273]: pgmap v4289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:45 smithi144 ceph-mon[32129]: pgmap v4290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:45 smithi160 ceph-mon[36715]: pgmap v4290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:45 smithi183 ceph-mon[37273]: pgmap v4290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7069.687917 secs 2024-07-26T13:03:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:47 smithi144 ceph-mon[32129]: pgmap v4291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7069.687917 secs 2024-07-26T13:03:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:47 smithi160 ceph-mon[36715]: pgmap v4291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7069.687917 secs 2024-07-26T13:03:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:47 smithi183 ceph-mon[37273]: pgmap v4291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:48.035+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:48.036+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:48.036+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:48.036+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:03:48.036+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:49 smithi144 ceph-mon[32129]: pgmap v4292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:49 smithi160 ceph-mon[36715]: pgmap v4292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:49 smithi183 ceph-mon[37273]: pgmap v4292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:51 smithi144 ceph-mon[32129]: pgmap v4293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:51 smithi160 ceph-mon[36715]: pgmap v4293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:51 smithi183 ceph-mon[37273]: pgmap v4293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7074.688071 secs 2024-07-26T13:03:52.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7074.688071 secs 2024-07-26T13:03:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7074.688071 secs 2024-07-26T13:03:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:53 smithi144 ceph-mon[32129]: pgmap v4294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:53 smithi160 ceph-mon[36715]: pgmap v4294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:53 smithi183 ceph-mon[37273]: pgmap v4294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:03:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:55 smithi144 ceph-mon[32129]: pgmap v4295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:55 smithi160 ceph-mon[36715]: pgmap v4295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:55 smithi183 ceph-mon[37273]: pgmap v4295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7079.688192 secs 2024-07-26T13:03:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:57 smithi144 ceph-mon[32129]: pgmap v4296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7079.688192 secs 2024-07-26T13:03:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:57 smithi160 ceph-mon[36715]: pgmap v4296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7079.688192 secs 2024-07-26T13:03:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:57 smithi183 ceph-mon[37273]: pgmap v4296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:03:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:03:59 smithi160 ceph-mon[36715]: pgmap v4297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:03:59 smithi183 ceph-mon[37273]: pgmap v4297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:03:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:03:59 smithi144 ceph-mon[32129]: pgmap v4297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:01 smithi160 ceph-mon[36715]: pgmap v4298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:01 smithi183 ceph-mon[37273]: pgmap v4298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:01 smithi144 ceph-mon[32129]: pgmap v4298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7084.688276 secs 2024-07-26T13:04:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7084.688276 secs 2024-07-26T13:04:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7084.688276 secs 2024-07-26T13:04:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:03.041+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:03.042+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:03.042+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:03.042+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:03.042+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:03 smithi160 ceph-mon[36715]: pgmap v4299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:03 smithi183 ceph-mon[37273]: pgmap v4299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:03 smithi144 ceph-mon[32129]: pgmap v4299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:05 smithi160 ceph-mon[36715]: pgmap v4300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:05 smithi183 ceph-mon[37273]: pgmap v4300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:05 smithi144 ceph-mon[32129]: pgmap v4300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7089.688396 secs 2024-07-26T13:04:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:07 smithi160 ceph-mon[36715]: pgmap v4301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7089.688396 secs 2024-07-26T13:04:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:07 smithi183 ceph-mon[37273]: pgmap v4301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7089.688396 secs 2024-07-26T13:04:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:07 smithi144 ceph-mon[32129]: pgmap v4301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:09 smithi160 ceph-mon[36715]: pgmap v4302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:09 smithi183 ceph-mon[37273]: pgmap v4302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:09 smithi144 ceph-mon[32129]: pgmap v4302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:11 smithi160 ceph-mon[36715]: pgmap v4303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:11 smithi183 ceph-mon[37273]: pgmap v4303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:11 smithi144 ceph-mon[32129]: pgmap v4303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7094.688479 secs 2024-07-26T13:04:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7094.688479 secs 2024-07-26T13:04:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7094.688479 secs 2024-07-26T13:04:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:13 smithi160 ceph-mon[36715]: pgmap v4304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:13 smithi183 ceph-mon[37273]: pgmap v4304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:13 smithi144 ceph-mon[32129]: pgmap v4304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:15 smithi160 ceph-mon[36715]: pgmap v4305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:15 smithi183 ceph-mon[37273]: pgmap v4305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:15 smithi144 ceph-mon[32129]: pgmap v4305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7099.688600 secs 2024-07-26T13:04:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:17 smithi160 ceph-mon[36715]: pgmap v4306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7099.688600 secs 2024-07-26T13:04:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:17 smithi183 ceph-mon[37273]: pgmap v4306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7099.688600 secs 2024-07-26T13:04:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:17 smithi144 ceph-mon[32129]: pgmap v4306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:18.058 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:18.057+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:18.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:18.057+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:18.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:18.057+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:18.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:18.057+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:18.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:18.057+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:19 smithi160 ceph-mon[36715]: pgmap v4307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:19 smithi183 ceph-mon[37273]: pgmap v4307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:19 smithi144 ceph-mon[32129]: pgmap v4307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:21 smithi160 ceph-mon[36715]: pgmap v4308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:21 smithi183 ceph-mon[37273]: pgmap v4308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:21 smithi144 ceph-mon[32129]: pgmap v4308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7104.688727 secs 2024-07-26T13:04:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7104.688727 secs 2024-07-26T13:04:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7104.688727 secs 2024-07-26T13:04:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:23 smithi183 ceph-mon[37273]: pgmap v4309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:04:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:23 smithi144 ceph-mon[32129]: pgmap v4309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:04:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:23 smithi160 ceph-mon[36715]: pgmap v4309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:04:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:24 smithi183 ceph-mon[37273]: pgmap v4310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:04:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:24 smithi183 ceph-mon[37273]: pgmap v4311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:04:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:24 smithi144 ceph-mon[32129]: pgmap v4310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:04:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:24 smithi144 ceph-mon[32129]: pgmap v4311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:04:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:24 smithi160 ceph-mon[36715]: pgmap v4310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:04:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:24 smithi160 ceph-mon[36715]: pgmap v4311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:04:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:26 smithi183 ceph-mon[37273]: pgmap v4312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:26 smithi144 ceph-mon[32129]: pgmap v4312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:26 smithi160 ceph-mon[36715]: pgmap v4312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7109.688869 secs 2024-07-26T13:04:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7109.688869 secs 2024-07-26T13:04:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7109.688869 secs 2024-07-26T13:04:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:28 smithi183 ceph-mon[37273]: pgmap v4313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:28 smithi144 ceph-mon[32129]: pgmap v4313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:28 smithi160 ceph-mon[36715]: pgmap v4313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:30 smithi183 ceph-mon[37273]: pgmap v4314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:30 smithi144 ceph-mon[32129]: pgmap v4314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:30 smithi160 ceph-mon[36715]: pgmap v4314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7114.688989 secs 2024-07-26T13:04:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:32 smithi183 ceph-mon[37273]: pgmap v4315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7114.688989 secs 2024-07-26T13:04:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:32 smithi144 ceph-mon[32129]: pgmap v4315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7114.688989 secs 2024-07-26T13:04:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:32 smithi160 ceph-mon[36715]: pgmap v4315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:33.065+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:33.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:33.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:33.066+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:33.067+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:34 smithi183 ceph-mon[37273]: pgmap v4316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:04:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:34 smithi144 ceph-mon[32129]: pgmap v4316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:04:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:34 smithi160 ceph-mon[36715]: pgmap v4316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:04:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:36 smithi183 ceph-mon[37273]: pgmap v4317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T13:04:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:36 smithi144 ceph-mon[32129]: pgmap v4317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T13:04:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:36 smithi160 ceph-mon[36715]: pgmap v4317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T13:04:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7119.689111 secs 2024-07-26T13:04:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7119.689111 secs 2024-07-26T13:04:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7119.689111 secs 2024-07-26T13:04:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:38 smithi183 ceph-mon[37273]: pgmap v4318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:38 smithi144 ceph-mon[32129]: pgmap v4318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:38 smithi160 ceph-mon[36715]: pgmap v4318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:40 smithi183 ceph-mon[37273]: pgmap v4319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:40 smithi144 ceph-mon[32129]: pgmap v4319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:40 smithi160 ceph-mon[36715]: pgmap v4319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7124.689229 secs 2024-07-26T13:04:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:42 smithi144 ceph-mon[32129]: pgmap v4320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7124.689229 secs 2024-07-26T13:04:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:42 smithi160 ceph-mon[36715]: pgmap v4320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7124.689229 secs 2024-07-26T13:04:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:42 smithi183 ceph-mon[37273]: pgmap v4320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:44 smithi144 ceph-mon[32129]: pgmap v4321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:44 smithi160 ceph-mon[36715]: pgmap v4321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:44 smithi183 ceph-mon[37273]: pgmap v4321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:46 smithi144 ceph-mon[32129]: pgmap v4322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:04:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:46 smithi160 ceph-mon[36715]: pgmap v4322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:04:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:46 smithi183 ceph-mon[37273]: pgmap v4322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:04:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7129.689348 secs 2024-07-26T13:04:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7129.689348 secs 2024-07-26T13:04:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7129.689348 secs 2024-07-26T13:04:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:48.082+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:48.083+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:48.083+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:48.083+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:04:48.083+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:48 smithi144 ceph-mon[32129]: pgmap v4323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:48 smithi160 ceph-mon[36715]: pgmap v4323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:48 smithi183 ceph-mon[37273]: pgmap v4323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:50 smithi144 ceph-mon[32129]: pgmap v4324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:50 smithi160 ceph-mon[36715]: pgmap v4324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:50 smithi183 ceph-mon[37273]: pgmap v4324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:04:52.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7134.689467 secs 2024-07-26T13:04:52.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:52 smithi183 ceph-mon[37273]: pgmap v4325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7134.689467 secs 2024-07-26T13:04:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:52 smithi144 ceph-mon[32129]: pgmap v4325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7134.689467 secs 2024-07-26T13:04:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:52 smithi160 ceph-mon[36715]: pgmap v4325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:04:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:54 smithi144 ceph-mon[32129]: pgmap v4326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:54 smithi160 ceph-mon[36715]: pgmap v4326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:54 smithi183 ceph-mon[37273]: pgmap v4326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:56 smithi144 ceph-mon[32129]: pgmap v4327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:04:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:56 smithi160 ceph-mon[36715]: pgmap v4327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:04:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:56 smithi183 ceph-mon[37273]: pgmap v4327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:04:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7139.689622 secs 2024-07-26T13:04:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7139.689622 secs 2024-07-26T13:04:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7139.689622 secs 2024-07-26T13:04:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:04:58 smithi144 ceph-mon[32129]: pgmap v4328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:04:58 smithi160 ceph-mon[36715]: pgmap v4328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:04:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:04:58 smithi183 ceph-mon[37273]: pgmap v4328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:00 smithi144 ceph-mon[32129]: pgmap v4329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:00 smithi160 ceph-mon[36715]: pgmap v4329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:00 smithi183 ceph-mon[37273]: pgmap v4329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7144.689762 secs 2024-07-26T13:05:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:02 smithi144 ceph-mon[32129]: pgmap v4330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7144.689762 secs 2024-07-26T13:05:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:02 smithi160 ceph-mon[36715]: pgmap v4330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7144.689762 secs 2024-07-26T13:05:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:02 smithi183 ceph-mon[37273]: pgmap v4330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:03.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:03.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:03.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:03.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:03.098+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:04.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:04 smithi144 ceph-mon[32129]: pgmap v4331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:04 smithi160 ceph-mon[36715]: pgmap v4331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:04 smithi183 ceph-mon[37273]: pgmap v4331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:06 smithi160 ceph-mon[36715]: pgmap v4332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:05:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:06 smithi183 ceph-mon[37273]: pgmap v4332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:05:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:06 smithi144 ceph-mon[32129]: pgmap v4332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:05:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7149.689900 secs 2024-07-26T13:05:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7149.689900 secs 2024-07-26T13:05:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7149.689900 secs 2024-07-26T13:05:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:08 smithi160 ceph-mon[36715]: pgmap v4333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:08 smithi183 ceph-mon[37273]: pgmap v4333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:08 smithi144 ceph-mon[32129]: pgmap v4333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:10 smithi160 ceph-mon[36715]: pgmap v4334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:10 smithi183 ceph-mon[37273]: pgmap v4334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:10 smithi144 ceph-mon[32129]: pgmap v4334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7154.690038 secs 2024-07-26T13:05:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:12 smithi160 ceph-mon[36715]: pgmap v4335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7154.690038 secs 2024-07-26T13:05:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:12 smithi183 ceph-mon[37273]: pgmap v4335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7154.690038 secs 2024-07-26T13:05:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:12 smithi144 ceph-mon[32129]: pgmap v4335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:14 smithi160 ceph-mon[36715]: pgmap v4336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:14 smithi183 ceph-mon[37273]: pgmap v4336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:14 smithi144 ceph-mon[32129]: pgmap v4336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:16 smithi160 ceph-mon[36715]: pgmap v4337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:05:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:16 smithi183 ceph-mon[37273]: pgmap v4337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:05:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:16 smithi144 ceph-mon[32129]: pgmap v4337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:05:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7159.690177 secs 2024-07-26T13:05:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7159.690177 secs 2024-07-26T13:05:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7159.690177 secs 2024-07-26T13:05:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:18.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:18.114+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:18.114+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:18.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:18.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:18 smithi160 ceph-mon[36715]: pgmap v4338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:18 smithi183 ceph-mon[37273]: pgmap v4338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:18 smithi144 ceph-mon[32129]: pgmap v4338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:20 smithi160 ceph-mon[36715]: pgmap v4339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:20 smithi183 ceph-mon[37273]: pgmap v4339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:20 smithi144 ceph-mon[32129]: pgmap v4339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7164.690271 secs 2024-07-26T13:05:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:22 smithi160 ceph-mon[36715]: pgmap v4340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7164.690271 secs 2024-07-26T13:05:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:22 smithi183 ceph-mon[37273]: pgmap v4340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7164.690271 secs 2024-07-26T13:05:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:22 smithi144 ceph-mon[32129]: pgmap v4340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:23.763 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:24 smithi160 ceph-mon[36715]: pgmap v4341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:05:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:24 smithi183 ceph-mon[37273]: pgmap v4341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:05:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:24 smithi144 ceph-mon[32129]: pgmap v4341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:05:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:25 smithi160 ceph-mon[36715]: pgmap v4342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:25 smithi160 ceph-mon[36715]: pgmap v4343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:05:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:25 smithi183 ceph-mon[37273]: pgmap v4342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:25 smithi183 ceph-mon[37273]: pgmap v4343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:05:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:25 smithi144 ceph-mon[32129]: pgmap v4342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:25 smithi144 ceph-mon[32129]: pgmap v4343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:05:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7169.690405 secs 2024-07-26T13:05:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:27 smithi183 ceph-mon[37273]: pgmap v4344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7169.690405 secs 2024-07-26T13:05:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:27 smithi144 ceph-mon[32129]: pgmap v4344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7169.690405 secs 2024-07-26T13:05:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:27 smithi160 ceph-mon[36715]: pgmap v4344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:29 smithi183 ceph-mon[37273]: pgmap v4345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:05:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:29 smithi144 ceph-mon[32129]: pgmap v4345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:05:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:29 smithi160 ceph-mon[36715]: pgmap v4345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:05:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:31 smithi183 ceph-mon[37273]: pgmap v4346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:31 smithi144 ceph-mon[32129]: pgmap v4346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:31 smithi160 ceph-mon[36715]: pgmap v4346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:32.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7174.690533 secs 2024-07-26T13:05:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7174.690533 secs 2024-07-26T13:05:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7174.690533 secs 2024-07-26T13:05:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:33.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:33.131+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:33.131+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:33.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:33.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:33.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:33 smithi183 ceph-mon[37273]: pgmap v4347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:33 smithi144 ceph-mon[32129]: pgmap v4347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:33 smithi160 ceph-mon[36715]: pgmap v4347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:05:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:35 smithi183 ceph-mon[37273]: pgmap v4348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:05:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:35 smithi144 ceph-mon[32129]: pgmap v4348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:05:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:35 smithi160 ceph-mon[36715]: pgmap v4348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:05:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7179.690652 secs 2024-07-26T13:05:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:37 smithi183 ceph-mon[37273]: pgmap v4349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7179.690652 secs 2024-07-26T13:05:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:37 smithi144 ceph-mon[32129]: pgmap v4349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7179.690652 secs 2024-07-26T13:05:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:37 smithi160 ceph-mon[36715]: pgmap v4349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:39 smithi183 ceph-mon[37273]: pgmap v4350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:39 smithi144 ceph-mon[32129]: pgmap v4350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:39 smithi160 ceph-mon[36715]: pgmap v4350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:41 smithi183 ceph-mon[37273]: pgmap v4351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:41 smithi144 ceph-mon[32129]: pgmap v4351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:41 smithi160 ceph-mon[36715]: pgmap v4351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7184.690751 secs 2024-07-26T13:05:42.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7184.690751 secs 2024-07-26T13:05:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7184.690751 secs 2024-07-26T13:05:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:43 smithi144 ceph-mon[32129]: pgmap v4352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:43 smithi160 ceph-mon[36715]: pgmap v4352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:43 smithi183 ceph-mon[37273]: pgmap v4352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:45 smithi144 ceph-mon[32129]: pgmap v4353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:45 smithi160 ceph-mon[36715]: pgmap v4353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.115+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.115+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.115+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a8063070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.115+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a8063070 comp rx=0 tx=0).stop 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.115+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2cc0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.116+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 msgr2=0x7f74900c2cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.116+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c23b0 0x7f74900c2cc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:05:46.117 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.116+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:05:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.117+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:46.120 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:46.119+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a804c040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:05:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:45 smithi183 ceph-mon[37273]: pgmap v4353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:47.372+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:05:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:47.372+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:05:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:47.372+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:05:47.374 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:47.373+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:05:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7189.690875 secs 2024-07-26T13:05:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:47 smithi144 ceph-mon[32129]: pgmap v4354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7189.690875 secs 2024-07-26T13:05:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:47 smithi160 ceph-mon[36715]: pgmap v4354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7189.690875 secs 2024-07-26T13:05:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:47 smithi183 ceph-mon[37273]: pgmap v4354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:48.141+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:48.142+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:48.142+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:48.142+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:05:48.142+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:49 smithi144 ceph-mon[32129]: pgmap v4355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:49 smithi160 ceph-mon[36715]: pgmap v4355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:49 smithi183 ceph-mon[37273]: pgmap v4355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:51 smithi144 ceph-mon[32129]: pgmap v4356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:51 smithi160 ceph-mon[36715]: pgmap v4356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:51 smithi183 ceph-mon[37273]: pgmap v4356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7194.691002 secs 2024-07-26T13:05:53.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7194.691002 secs 2024-07-26T13:05:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7194.691002 secs 2024-07-26T13:05:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:53 smithi144 ceph-mon[32129]: pgmap v4357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:53 smithi160 ceph-mon[36715]: pgmap v4357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:53 smithi183 ceph-mon[37273]: pgmap v4357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:05:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:55 smithi144 ceph-mon[32129]: pgmap v4358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:55 smithi160 ceph-mon[36715]: pgmap v4358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:55 smithi183 ceph-mon[37273]: pgmap v4358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:05:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7199.691142 secs 2024-07-26T13:05:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:57 smithi144 ceph-mon[32129]: pgmap v4359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7199.691142 secs 2024-07-26T13:05:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:57 smithi160 ceph-mon[36715]: pgmap v4359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7199.691142 secs 2024-07-26T13:05:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:57 smithi183 ceph-mon[37273]: pgmap v4359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:05:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:05:59 smithi144 ceph-mon[32129]: pgmap v4360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:05:59 smithi160 ceph-mon[36715]: pgmap v4360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:05:59 smithi183 ceph-mon[37273]: pgmap v4360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:01 smithi144 ceph-mon[32129]: pgmap v4361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7204.691280 secs 2024-07-26T13:06:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:01 smithi160 ceph-mon[36715]: pgmap v4361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7204.691280 secs 2024-07-26T13:06:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:01 smithi183 ceph-mon[37273]: pgmap v4361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7204.691280 secs 2024-07-26T13:06:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:03.158+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:03.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:03.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:03.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:03.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:03 smithi144 ceph-mon[32129]: pgmap v4362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:03 smithi160 ceph-mon[36715]: pgmap v4362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:03 smithi183 ceph-mon[37273]: pgmap v4362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:05 smithi144 ceph-mon[32129]: pgmap v4363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:05 smithi160 ceph-mon[36715]: pgmap v4363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:05 smithi183 ceph-mon[37273]: pgmap v4363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7209.691310 secs 2024-07-26T13:06:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7209.691310 secs 2024-07-26T13:06:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7209.691310 secs 2024-07-26T13:06:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:07 smithi160 ceph-mon[36715]: pgmap v4364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:07 smithi183 ceph-mon[37273]: pgmap v4364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:07 smithi144 ceph-mon[32129]: pgmap v4364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:09 smithi160 ceph-mon[36715]: pgmap v4365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:09 smithi183 ceph-mon[37273]: pgmap v4365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:09 smithi144 ceph-mon[32129]: pgmap v4365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:11 smithi160 ceph-mon[36715]: pgmap v4366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7214.691431 secs 2024-07-26T13:06:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:11 smithi183 ceph-mon[37273]: pgmap v4366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7214.691431 secs 2024-07-26T13:06:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:11 smithi144 ceph-mon[32129]: pgmap v4366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7214.691431 secs 2024-07-26T13:06:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:12 smithi160 ceph-mon[36715]: pgmap v4367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:12 smithi183 ceph-mon[37273]: pgmap v4367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:12 smithi144 ceph-mon[32129]: pgmap v4367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:15 smithi160 ceph-mon[36715]: pgmap v4368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:15 smithi183 ceph-mon[37273]: pgmap v4368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:15 smithi144 ceph-mon[32129]: pgmap v4368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7219.691585 secs 2024-07-26T13:06:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:17 smithi160 ceph-mon[36715]: pgmap v4369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7219.691585 secs 2024-07-26T13:06:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:17 smithi183 ceph-mon[37273]: pgmap v4369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7219.691585 secs 2024-07-26T13:06:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:17 smithi144 ceph-mon[32129]: pgmap v4369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:18.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:18.175+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:18.176+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:18.176+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:18.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:18.178 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:18.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:19 smithi160 ceph-mon[36715]: pgmap v4370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:19 smithi183 ceph-mon[37273]: pgmap v4370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:19 smithi144 ceph-mon[32129]: pgmap v4370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:21 smithi160 ceph-mon[36715]: pgmap v4371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:21 smithi183 ceph-mon[37273]: pgmap v4371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:21 smithi144 ceph-mon[32129]: pgmap v4371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7224.691708 secs 2024-07-26T13:06:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7224.691708 secs 2024-07-26T13:06:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7224.691708 secs 2024-07-26T13:06:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:23 smithi160 ceph-mon[36715]: pgmap v4372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:23 smithi183 ceph-mon[37273]: pgmap v4372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:23 smithi144 ceph-mon[32129]: pgmap v4372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:24 smithi183 ceph-mon[37273]: pgmap v4373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:24 smithi144 ceph-mon[32129]: pgmap v4373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:24 smithi160 ceph-mon[36715]: pgmap v4373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:25 smithi183 ceph-mon[37273]: pgmap v4374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:25 smithi183 ceph-mon[37273]: pgmap v4375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:06:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:25 smithi144 ceph-mon[32129]: pgmap v4374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:25 smithi144 ceph-mon[32129]: pgmap v4375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:06:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:25 smithi160 ceph-mon[36715]: pgmap v4374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:25 smithi160 ceph-mon[36715]: pgmap v4375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:06:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7229.691840 secs 2024-07-26T13:06:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7229.691840 secs 2024-07-26T13:06:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7229.691840 secs 2024-07-26T13:06:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:27 smithi183 ceph-mon[37273]: pgmap v4376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:06:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:27 smithi144 ceph-mon[32129]: pgmap v4376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:06:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:27 smithi160 ceph-mon[36715]: pgmap v4376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:06:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:30 smithi144 ceph-mon[32129]: pgmap v4377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:06:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:30 smithi160 ceph-mon[36715]: pgmap v4377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:06:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:30 smithi183 ceph-mon[37273]: pgmap v4377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:06:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7234.691965 secs 2024-07-26T13:06:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:32 smithi144 ceph-mon[32129]: pgmap v4378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7234.691965 secs 2024-07-26T13:06:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:32 smithi160 ceph-mon[36715]: pgmap v4378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7234.691965 secs 2024-07-26T13:06:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:32 smithi183 ceph-mon[37273]: pgmap v4378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:06:33.193 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:33.192+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:33.193+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:33.193+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:33.193+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:33.193+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:34 smithi144 ceph-mon[32129]: pgmap v4379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:06:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:34 smithi160 ceph-mon[36715]: pgmap v4379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:06:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:34 smithi183 ceph-mon[37273]: pgmap v4379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:06:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:36 smithi144 ceph-mon[32129]: pgmap v4380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:06:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7239.692058 secs 2024-07-26T13:06:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:36 smithi160 ceph-mon[36715]: pgmap v4380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:06:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7239.692058 secs 2024-07-26T13:06:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:36 smithi183 ceph-mon[37273]: pgmap v4380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:06:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7239.692058 secs 2024-07-26T13:06:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:38 smithi144 ceph-mon[32129]: pgmap v4381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:38 smithi160 ceph-mon[36715]: pgmap v4381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:38 smithi183 ceph-mon[37273]: pgmap v4381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:40 smithi144 ceph-mon[32129]: pgmap v4382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:40 smithi160 ceph-mon[36715]: pgmap v4382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:40 smithi183 ceph-mon[37273]: pgmap v4382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7244.692189 secs 2024-07-26T13:06:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7244.692189 secs 2024-07-26T13:06:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7244.692189 secs 2024-07-26T13:06:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:42 smithi160 ceph-mon[36715]: pgmap v4383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:42 smithi183 ceph-mon[37273]: pgmap v4383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:42 smithi144 ceph-mon[32129]: pgmap v4383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:44 smithi160 ceph-mon[36715]: pgmap v4384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:44 smithi183 ceph-mon[37273]: pgmap v4384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:44 smithi144 ceph-mon[32129]: pgmap v4384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:46 smithi160 ceph-mon[36715]: pgmap v4385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7249.692324 secs 2024-07-26T13:06:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:46 smithi183 ceph-mon[37273]: pgmap v4385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7249.692324 secs 2024-07-26T13:06:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:46 smithi144 ceph-mon[32129]: pgmap v4385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7249.692324 secs 2024-07-26T13:06:48.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:47 smithi160 ceph-mon[36715]: pgmap v4386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:47 smithi183 ceph-mon[37273]: pgmap v4386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:48.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:48.210+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:48.210+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:48.210+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:06:48.210+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:47 smithi144 ceph-mon[32129]: pgmap v4386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:50 smithi144 ceph-mon[32129]: pgmap v4387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:50 smithi160 ceph-mon[36715]: pgmap v4387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:50 smithi183 ceph-mon[37273]: pgmap v4387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7254.692389 secs 2024-07-26T13:06:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7254.692389 secs 2024-07-26T13:06:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7254.692389 secs 2024-07-26T13:06:53.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:52 smithi183 ceph-mon[37273]: pgmap v4388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:52 smithi160 ceph-mon[36715]: pgmap v4388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:52 smithi144 ceph-mon[32129]: pgmap v4388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:06:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:54 smithi160 ceph-mon[36715]: pgmap v4389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:54 smithi183 ceph-mon[37273]: pgmap v4389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:54 smithi144 ceph-mon[32129]: pgmap v4389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:56 smithi160 ceph-mon[36715]: pgmap v4390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7259.692484 secs 2024-07-26T13:06:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:56 smithi183 ceph-mon[37273]: pgmap v4390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7259.692484 secs 2024-07-26T13:06:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:56 smithi144 ceph-mon[32129]: pgmap v4390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:06:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7259.692484 secs 2024-07-26T13:06:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:06:58 smithi160 ceph-mon[36715]: pgmap v4391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:06:58 smithi183 ceph-mon[37273]: pgmap v4391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:06:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:06:58 smithi144 ceph-mon[32129]: pgmap v4391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:00 smithi160 ceph-mon[36715]: pgmap v4392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:00 smithi183 ceph-mon[37273]: pgmap v4392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:00 smithi144 ceph-mon[32129]: pgmap v4392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7264.692607 secs 2024-07-26T13:07:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7264.692607 secs 2024-07-26T13:07:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7264.692607 secs 2024-07-26T13:07:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:02 smithi160 ceph-mon[36715]: pgmap v4393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:02 smithi183 ceph-mon[37273]: pgmap v4393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:02 smithi144 ceph-mon[32129]: pgmap v4393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:03.226+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:03.227+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:03.227+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:03.227+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:03.227+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:03 smithi160 ceph-mon[36715]: pgmap v4394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:04.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:03 smithi144 ceph-mon[32129]: pgmap v4394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:03 smithi183 ceph-mon[37273]: pgmap v4394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:06 smithi144 ceph-mon[32129]: pgmap v4395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7269.692671 secs 2024-07-26T13:07:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:06 smithi160 ceph-mon[36715]: pgmap v4395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7269.692671 secs 2024-07-26T13:07:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:06 smithi183 ceph-mon[37273]: pgmap v4395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7269.692671 secs 2024-07-26T13:07:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:08 smithi144 ceph-mon[32129]: pgmap v4396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:08 smithi160 ceph-mon[36715]: pgmap v4396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:08 smithi183 ceph-mon[37273]: pgmap v4396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:10 smithi160 ceph-mon[36715]: pgmap v4397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:10 smithi183 ceph-mon[37273]: pgmap v4397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:10 smithi144 ceph-mon[32129]: pgmap v4397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7274.692801 secs 2024-07-26T13:07:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7274.692801 secs 2024-07-26T13:07:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7274.692801 secs 2024-07-26T13:07:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:12 smithi160 ceph-mon[36715]: pgmap v4398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:12 smithi183 ceph-mon[37273]: pgmap v4398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:12 smithi144 ceph-mon[32129]: pgmap v4398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:14 smithi160 ceph-mon[36715]: pgmap v4399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:14 smithi183 ceph-mon[37273]: pgmap v4399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:14 smithi144 ceph-mon[32129]: pgmap v4399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:16 smithi160 ceph-mon[36715]: pgmap v4400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7279.692928 secs 2024-07-26T13:07:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:16 smithi183 ceph-mon[37273]: pgmap v4400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7279.692928 secs 2024-07-26T13:07:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:16 smithi144 ceph-mon[32129]: pgmap v4400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7279.692928 secs 2024-07-26T13:07:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:17 smithi160 ceph-mon[36715]: pgmap v4401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:17 smithi183 ceph-mon[37273]: pgmap v4401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:17 smithi144 ceph-mon[32129]: pgmap v4401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:18.243+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:18.244+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:18.244+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:18.244+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:18.244+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:20 smithi160 ceph-mon[36715]: pgmap v4402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:20 smithi183 ceph-mon[37273]: pgmap v4402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:20 smithi144 ceph-mon[32129]: pgmap v4402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7284.693048 secs 2024-07-26T13:07:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7284.693048 secs 2024-07-26T13:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7284.693048 secs 2024-07-26T13:07:23.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:22 smithi183 ceph-mon[37273]: pgmap v4403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:22 smithi160 ceph-mon[36715]: pgmap v4403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:23.165 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:22 smithi144 ceph-mon[32129]: pgmap v4403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:24 smithi160 ceph-mon[36715]: pgmap v4404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:24 smithi183 ceph-mon[37273]: pgmap v4404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:24 smithi144 ceph-mon[32129]: pgmap v4404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:26 smithi160 ceph-mon[36715]: pgmap v4405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7289.693173 secs 2024-07-26T13:07:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:26 smithi183 ceph-mon[37273]: pgmap v4405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7289.693173 secs 2024-07-26T13:07:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:26 smithi144 ceph-mon[32129]: pgmap v4405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7289.693173 secs 2024-07-26T13:07:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:07:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:07:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:07:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:28 smithi160 ceph-mon[36715]: pgmap v4406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:07:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:28 smithi160 ceph-mon[36715]: pgmap v4407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:07:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:28 smithi183 ceph-mon[37273]: pgmap v4406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:07:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:28 smithi183 ceph-mon[37273]: pgmap v4407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:07:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:28 smithi144 ceph-mon[32129]: pgmap v4406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:07:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:28 smithi144 ceph-mon[32129]: pgmap v4407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:07:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:29 smithi160 ceph-mon[36715]: pgmap v4408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:07:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:29 smithi183 ceph-mon[37273]: pgmap v4408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:07:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:29 smithi144 ceph-mon[32129]: pgmap v4408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:07:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7294.693313 secs 2024-07-26T13:07:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:31 smithi144 ceph-mon[32129]: pgmap v4409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:07:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7294.693313 secs 2024-07-26T13:07:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:31 smithi160 ceph-mon[36715]: pgmap v4409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:07:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7294.693313 secs 2024-07-26T13:07:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:31 smithi183 ceph-mon[37273]: pgmap v4409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:07:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:33.260+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:33.261+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:33.261+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:33.261+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:33.261+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:33 smithi144 ceph-mon[32129]: pgmap v4410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:07:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:33 smithi160 ceph-mon[36715]: pgmap v4410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:07:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:33 smithi183 ceph-mon[37273]: pgmap v4410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:07:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:35 smithi144 ceph-mon[32129]: pgmap v4411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:07:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:35 smithi160 ceph-mon[36715]: pgmap v4411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:07:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:35 smithi183 ceph-mon[37273]: pgmap v4411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:07:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7299.693442 secs 2024-07-26T13:07:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7299.693442 secs 2024-07-26T13:07:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7299.693442 secs 2024-07-26T13:07:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:37 smithi144 ceph-mon[32129]: pgmap v4412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:07:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:37 smithi160 ceph-mon[36715]: pgmap v4412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:07:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:37 smithi183 ceph-mon[37273]: pgmap v4412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:07:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:39 smithi144 ceph-mon[32129]: pgmap v4413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:39 smithi160 ceph-mon[36715]: pgmap v4413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:39 smithi183 ceph-mon[37273]: pgmap v4413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7304.693567 secs 2024-07-26T13:07:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:41 smithi144 ceph-mon[32129]: pgmap v4414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7304.693567 secs 2024-07-26T13:07:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:41 smithi160 ceph-mon[36715]: pgmap v4414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7304.693567 secs 2024-07-26T13:07:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:41 smithi183 ceph-mon[37273]: pgmap v4414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:43 smithi144 ceph-mon[32129]: pgmap v4415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:43 smithi160 ceph-mon[36715]: pgmap v4415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:43 smithi183 ceph-mon[37273]: pgmap v4415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:45 smithi144 ceph-mon[32129]: pgmap v4416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:45 smithi160 ceph-mon[36715]: pgmap v4416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:45 smithi183 ceph-mon[37273]: pgmap v4416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7309.693695 secs 2024-07-26T13:07:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7309.693695 secs 2024-07-26T13:07:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7309.693695 secs 2024-07-26T13:07:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:47 smithi144 ceph-mon[32129]: pgmap v4417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:48.277+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:48.278+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:48.278+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:48.278+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:07:48.278+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:47 smithi160 ceph-mon[36715]: pgmap v4417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:47 smithi183 ceph-mon[37273]: pgmap v4417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:49 smithi144 ceph-mon[32129]: pgmap v4418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:49 smithi160 ceph-mon[36715]: pgmap v4418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:49 smithi183 ceph-mon[37273]: pgmap v4418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7314.693796 secs 2024-07-26T13:07:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:51 smithi144 ceph-mon[32129]: pgmap v4419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7314.693796 secs 2024-07-26T13:07:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:51 smithi160 ceph-mon[36715]: pgmap v4419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7314.693796 secs 2024-07-26T13:07:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:51 smithi183 ceph-mon[37273]: pgmap v4419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:53 smithi144 ceph-mon[32129]: pgmap v4420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:53 smithi160 ceph-mon[36715]: pgmap v4420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:53 smithi183 ceph-mon[37273]: pgmap v4420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:07:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:55 smithi160 ceph-mon[36715]: pgmap v4421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:55 smithi183 ceph-mon[37273]: pgmap v4421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:55 smithi144 ceph-mon[32129]: pgmap v4421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:07:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7319.693918 secs 2024-07-26T13:07:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7319.693918 secs 2024-07-26T13:07:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7319.693918 secs 2024-07-26T13:07:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:57 smithi160 ceph-mon[36715]: pgmap v4422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:57 smithi183 ceph-mon[37273]: pgmap v4422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:07:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:57 smithi144 ceph-mon[32129]: pgmap v4422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:07:59 smithi160 ceph-mon[36715]: pgmap v4423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:07:59 smithi183 ceph-mon[37273]: pgmap v4423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:07:59 smithi144 ceph-mon[32129]: pgmap v4423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7324.694054 secs 2024-07-26T13:08:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:01 smithi160 ceph-mon[36715]: pgmap v4424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7324.694054 secs 2024-07-26T13:08:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:01 smithi183 ceph-mon[37273]: pgmap v4424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7324.694054 secs 2024-07-26T13:08:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:01 smithi144 ceph-mon[32129]: pgmap v4424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:03.293+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:03.294+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:03.294+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:03.294+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:03.295+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:04.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:04 smithi144 ceph-mon[32129]: pgmap v4425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:04 smithi160 ceph-mon[36715]: pgmap v4425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:04 smithi183 ceph-mon[37273]: pgmap v4425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:06 smithi160 ceph-mon[36715]: pgmap v4426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:06 smithi183 ceph-mon[37273]: pgmap v4426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:06 smithi144 ceph-mon[32129]: pgmap v4426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7329.694151 secs 2024-07-26T13:08:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7329.694151 secs 2024-07-26T13:08:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7329.694151 secs 2024-07-26T13:08:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:08 smithi160 ceph-mon[36715]: pgmap v4427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:08 smithi183 ceph-mon[37273]: pgmap v4427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:08 smithi144 ceph-mon[32129]: pgmap v4427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:10 smithi160 ceph-mon[36715]: pgmap v4428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:10 smithi183 ceph-mon[37273]: pgmap v4428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:10 smithi144 ceph-mon[32129]: pgmap v4428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7334.694311 secs 2024-07-26T13:08:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7334.694311 secs 2024-07-26T13:08:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7334.694311 secs 2024-07-26T13:08:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:12 smithi160 ceph-mon[36715]: pgmap v4429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:12 smithi183 ceph-mon[37273]: pgmap v4429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:12 smithi144 ceph-mon[32129]: pgmap v4429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:14 smithi160 ceph-mon[36715]: pgmap v4430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:14 smithi183 ceph-mon[37273]: pgmap v4430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:14 smithi144 ceph-mon[32129]: pgmap v4430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:16 smithi160 ceph-mon[36715]: pgmap v4431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7339.694416 secs 2024-07-26T13:08:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:16 smithi183 ceph-mon[37273]: pgmap v4431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7339.694416 secs 2024-07-26T13:08:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:16 smithi144 ceph-mon[32129]: pgmap v4431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7339.694416 secs 2024-07-26T13:08:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:18 smithi160 ceph-mon[36715]: pgmap v4432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:18.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:18.310+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:18.311+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:18.311+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:18.311+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:18.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:18.311+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:18 smithi183 ceph-mon[37273]: pgmap v4432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:18.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:18 smithi144 ceph-mon[32129]: pgmap v4432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:20 smithi183 ceph-mon[37273]: pgmap v4433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:20 smithi144 ceph-mon[32129]: pgmap v4433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:20 smithi160 ceph-mon[36715]: pgmap v4433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7344.694483 secs 2024-07-26T13:08:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7344.694483 secs 2024-07-26T13:08:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7344.694483 secs 2024-07-26T13:08:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:22 smithi183 ceph-mon[37273]: pgmap v4434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:22.817 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:22 smithi160 ceph-mon[36715]: pgmap v4434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:22.817 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:22 smithi144 ceph-mon[32129]: pgmap v4434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:24 smithi183 ceph-mon[37273]: pgmap v4435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:24 smithi144 ceph-mon[32129]: pgmap v4435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:24.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:24 smithi160 ceph-mon[36715]: pgmap v4435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:26 smithi183 ceph-mon[37273]: pgmap v4436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7349.694601 secs 2024-07-26T13:08:26.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:26 smithi144 ceph-mon[32129]: pgmap v4436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:26.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7349.694601 secs 2024-07-26T13:08:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:26 smithi160 ceph-mon[36715]: pgmap v4436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7349.694601 secs 2024-07-26T13:08:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:28 smithi183 ceph-mon[37273]: pgmap v4437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:08:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:28 smithi144 ceph-mon[32129]: pgmap v4437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:08:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:28 smithi160 ceph-mon[36715]: pgmap v4437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:08:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:29 smithi183 ceph-mon[37273]: pgmap v4438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:08:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:29 smithi183 ceph-mon[37273]: pgmap v4439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:08:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:29 smithi144 ceph-mon[32129]: pgmap v4438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:08:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:29 smithi144 ceph-mon[32129]: pgmap v4439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:08:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:29 smithi160 ceph-mon[36715]: pgmap v4438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:08:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:29 smithi160 ceph-mon[36715]: pgmap v4439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:08:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:31 smithi183 ceph-mon[37273]: pgmap v4440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:08:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7354.694727 secs 2024-07-26T13:08:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:31 smithi144 ceph-mon[32129]: pgmap v4440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:08:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7354.694727 secs 2024-07-26T13:08:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:31 smithi160 ceph-mon[36715]: pgmap v4440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:08:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7354.694727 secs 2024-07-26T13:08:33.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:33.327+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:33.328+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:33.328+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:33.328+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:33.328+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:33 smithi183 ceph-mon[37273]: pgmap v4441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:33 smithi144 ceph-mon[32129]: pgmap v4441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:33 smithi160 ceph-mon[36715]: pgmap v4441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:35 smithi183 ceph-mon[37273]: pgmap v4442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:35 smithi144 ceph-mon[32129]: pgmap v4442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:35 smithi160 ceph-mon[36715]: pgmap v4442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7359.694871 secs 2024-07-26T13:08:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7359.694871 secs 2024-07-26T13:08:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7359.694871 secs 2024-07-26T13:08:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:37 smithi144 ceph-mon[32129]: pgmap v4443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:37 smithi160 ceph-mon[36715]: pgmap v4443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:37 smithi183 ceph-mon[37273]: pgmap v4443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:08:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:39 smithi144 ceph-mon[32129]: pgmap v4444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:08:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:39 smithi160 ceph-mon[36715]: pgmap v4444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:08:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:39 smithi183 ceph-mon[37273]: pgmap v4444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:08:41.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:41 smithi144 ceph-mon[32129]: pgmap v4445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:41.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7364.695010 secs 2024-07-26T13:08:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:41 smithi160 ceph-mon[36715]: pgmap v4445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7364.695010 secs 2024-07-26T13:08:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:41 smithi183 ceph-mon[37273]: pgmap v4445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7364.695010 secs 2024-07-26T13:08:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:43 smithi144 ceph-mon[32129]: pgmap v4446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:43 smithi160 ceph-mon[36715]: pgmap v4446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:43 smithi183 ceph-mon[37273]: pgmap v4446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:45 smithi144 ceph-mon[32129]: pgmap v4447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:45 smithi160 ceph-mon[36715]: pgmap v4447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:45 smithi183 ceph-mon[37273]: pgmap v4447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7369.695133 secs 2024-07-26T13:08:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7369.695133 secs 2024-07-26T13:08:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7369.695133 secs 2024-07-26T13:08:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:47.404+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:08:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:47.404+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:08:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:47.405+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T13:08:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:47.406+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:08:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:47 smithi144 ceph-mon[32129]: pgmap v4448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:47 smithi160 ceph-mon[36715]: pgmap v4448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:47 smithi183 ceph-mon[37273]: pgmap v4448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:48.344+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:48.345+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:48.345+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:48.345+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:08:48.345+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:49 smithi144 ceph-mon[32129]: pgmap v4449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:49 smithi160 ceph-mon[36715]: pgmap v4449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:49 smithi183 ceph-mon[37273]: pgmap v4449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:08:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:51 smithi144 ceph-mon[32129]: pgmap v4450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7374.695253 secs 2024-07-26T13:08:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:51 smithi160 ceph-mon[36715]: pgmap v4450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7374.695253 secs 2024-07-26T13:08:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:51 smithi183 ceph-mon[37273]: pgmap v4450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7374.695253 secs 2024-07-26T13:08:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:53 smithi144 ceph-mon[32129]: pgmap v4451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:53 smithi160 ceph-mon[36715]: pgmap v4451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:53 smithi183 ceph-mon[37273]: pgmap v4451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:55 smithi144 ceph-mon[32129]: pgmap v4452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:55 smithi160 ceph-mon[36715]: pgmap v4452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:55 smithi183 ceph-mon[37273]: pgmap v4452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:08:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7379.695377 secs 2024-07-26T13:08:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7379.695377 secs 2024-07-26T13:08:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7379.695377 secs 2024-07-26T13:08:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:57 smithi144 ceph-mon[32129]: pgmap v4453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:57 smithi160 ceph-mon[36715]: pgmap v4453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:57 smithi183 ceph-mon[37273]: pgmap v4453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:08:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:08:59 smithi160 ceph-mon[36715]: pgmap v4454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:08:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:08:59 smithi183 ceph-mon[37273]: pgmap v4454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:08:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:08:59 smithi144 ceph-mon[32129]: pgmap v4454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:01 smithi160 ceph-mon[36715]: pgmap v4455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:09:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7384.695500 secs 2024-07-26T13:09:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:01 smithi183 ceph-mon[37273]: pgmap v4455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:09:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7384.695500 secs 2024-07-26T13:09:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:01 smithi144 ceph-mon[32129]: pgmap v4455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:09:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7384.695500 secs 2024-07-26T13:09:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:03.360+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:03.361+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:03.361+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:03.361+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:03.361+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:03 smithi160 ceph-mon[36715]: pgmap v4456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:03 smithi183 ceph-mon[37273]: pgmap v4456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:03 smithi144 ceph-mon[32129]: pgmap v4456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:05.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:05 smithi144 ceph-mon[32129]: pgmap v4457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:09:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:05 smithi160 ceph-mon[36715]: pgmap v4457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:09:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:05 smithi183 ceph-mon[37273]: pgmap v4457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:09:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7389.695597 secs 2024-07-26T13:09:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7389.695597 secs 2024-07-26T13:09:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7389.695597 secs 2024-07-26T13:09:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:07 smithi160 ceph-mon[36715]: pgmap v4458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:07 smithi183 ceph-mon[37273]: pgmap v4458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:07 smithi144 ceph-mon[32129]: pgmap v4458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:09 smithi160 ceph-mon[36715]: pgmap v4459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:09 smithi183 ceph-mon[37273]: pgmap v4459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:09 smithi144 ceph-mon[32129]: pgmap v4459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:11 smithi144 ceph-mon[32129]: pgmap v4460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7394.695666 secs 2024-07-26T13:09:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:11 smithi160 ceph-mon[36715]: pgmap v4460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7394.695666 secs 2024-07-26T13:09:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:11 smithi183 ceph-mon[37273]: pgmap v4460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7394.695666 secs 2024-07-26T13:09:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:13 smithi160 ceph-mon[36715]: pgmap v4461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:13 smithi183 ceph-mon[37273]: pgmap v4461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:13 smithi144 ceph-mon[32129]: pgmap v4461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:15 smithi160 ceph-mon[36715]: pgmap v4462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:15 smithi183 ceph-mon[37273]: pgmap v4462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:15 smithi144 ceph-mon[32129]: pgmap v4462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7399.695783 secs 2024-07-26T13:09:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7399.695783 secs 2024-07-26T13:09:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7399.695783 secs 2024-07-26T13:09:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:17 smithi160 ceph-mon[36715]: pgmap v4463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:17 smithi183 ceph-mon[37273]: pgmap v4463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:17 smithi144 ceph-mon[32129]: pgmap v4463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:18.378+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:18.379+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:18.379+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:18.379+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:18.379+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:19 smithi160 ceph-mon[36715]: pgmap v4464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:19 smithi183 ceph-mon[37273]: pgmap v4464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:19 smithi144 ceph-mon[32129]: pgmap v4464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:21 smithi160 ceph-mon[36715]: pgmap v4465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7404.695920 secs 2024-07-26T13:09:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:21 smithi183 ceph-mon[37273]: pgmap v4465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7404.695920 secs 2024-07-26T13:09:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:21 smithi144 ceph-mon[32129]: pgmap v4465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7404.695920 secs 2024-07-26T13:09:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:23 smithi183 ceph-mon[37273]: pgmap v4466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:23 smithi144 ceph-mon[32129]: pgmap v4466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:23 smithi160 ceph-mon[36715]: pgmap v4466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:25 smithi183 ceph-mon[37273]: pgmap v4467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:25 smithi144 ceph-mon[32129]: pgmap v4467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:25 smithi160 ceph-mon[36715]: pgmap v4467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7409.696041 secs 2024-07-26T13:09:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7409.696041 secs 2024-07-26T13:09:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7409.696041 secs 2024-07-26T13:09:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:27 smithi183 ceph-mon[37273]: pgmap v4468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:27 smithi144 ceph-mon[32129]: pgmap v4468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:27 smithi160 ceph-mon[36715]: pgmap v4468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:28.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: pgmap v4469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:09:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: pgmap v4469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:29.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: pgmap v4469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:29.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:09:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:30 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:09:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:30 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:09:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:30 smithi183 ceph-mon[37273]: pgmap v4470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:09:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:30 smithi183 ceph-mon[37273]: pgmap v4471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:30 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:09:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:30 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:09:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:30 smithi144 ceph-mon[32129]: pgmap v4470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:09:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:30 smithi144 ceph-mon[32129]: pgmap v4471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:30 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:09:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:30 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:09:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:30 smithi160 ceph-mon[36715]: pgmap v4470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:09:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:30 smithi160 ceph-mon[36715]: pgmap v4471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7414.696171 secs 2024-07-26T13:09:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7414.696171 secs 2024-07-26T13:09:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7414.696171 secs 2024-07-26T13:09:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:32.412+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:09:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:32.412+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:09:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:32.413+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T13:09:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:32.413+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:09:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:32 smithi183 ceph-mon[37273]: pgmap v4472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:32 smithi144 ceph-mon[32129]: pgmap v4472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:32 smithi160 ceph-mon[36715]: pgmap v4472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:33.395+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:33.396+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:33.396+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:33.396+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:33.396+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:34 smithi144 ceph-mon[32129]: pgmap v4473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:09:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:34 smithi160 ceph-mon[36715]: pgmap v4473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:09:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:34 smithi183 ceph-mon[37273]: pgmap v4473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:09:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:36 smithi144 ceph-mon[32129]: pgmap v4474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7419.696305 secs 2024-07-26T13:09:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:36 smithi160 ceph-mon[36715]: pgmap v4474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7419.696305 secs 2024-07-26T13:09:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:36 smithi183 ceph-mon[37273]: pgmap v4474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:09:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7419.696305 secs 2024-07-26T13:09:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:38 smithi144 ceph-mon[32129]: pgmap v4475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:09:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:38 smithi160 ceph-mon[36715]: pgmap v4475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:09:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:38 smithi183 ceph-mon[37273]: pgmap v4475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:09:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:40 smithi144 ceph-mon[32129]: pgmap v4476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:40 smithi160 ceph-mon[36715]: pgmap v4476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:40 smithi183 ceph-mon[37273]: pgmap v4476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:09:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7424.696407 secs 2024-07-26T13:09:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7424.696407 secs 2024-07-26T13:09:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7424.696407 secs 2024-07-26T13:09:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:42 smithi144 ceph-mon[32129]: pgmap v4477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:42 smithi160 ceph-mon[36715]: pgmap v4477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:42 smithi183 ceph-mon[37273]: pgmap v4477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:44 smithi144 ceph-mon[32129]: pgmap v4478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:44 smithi160 ceph-mon[36715]: pgmap v4478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:44 smithi183 ceph-mon[37273]: pgmap v4478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:46 smithi144 ceph-mon[32129]: pgmap v4479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7429.696522 secs 2024-07-26T13:09:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:46 smithi160 ceph-mon[36715]: pgmap v4479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7429.696522 secs 2024-07-26T13:09:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:46 smithi183 ceph-mon[37273]: pgmap v4479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7429.696522 secs 2024-07-26T13:09:48.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:48.412+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:48.412+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:48.413+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:48.413+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:09:48.413+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:48 smithi144 ceph-mon[32129]: pgmap v4480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:48 smithi160 ceph-mon[36715]: pgmap v4480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:48 smithi183 ceph-mon[37273]: pgmap v4480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:50 smithi144 ceph-mon[32129]: pgmap v4481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:50 smithi160 ceph-mon[36715]: pgmap v4481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:50 smithi183 ceph-mon[37273]: pgmap v4481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7434.696647 secs 2024-07-26T13:09:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7434.696647 secs 2024-07-26T13:09:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7434.696647 secs 2024-07-26T13:09:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:52 smithi144 ceph-mon[32129]: pgmap v4482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:52 smithi160 ceph-mon[36715]: pgmap v4482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:52 smithi183 ceph-mon[37273]: pgmap v4482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:09:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:54 smithi144 ceph-mon[32129]: pgmap v4483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:54 smithi160 ceph-mon[36715]: pgmap v4483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:54 smithi183 ceph-mon[37273]: pgmap v4483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:56 smithi144 ceph-mon[32129]: pgmap v4484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7439.696845 secs 2024-07-26T13:09:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:56 smithi160 ceph-mon[36715]: pgmap v4484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7439.696845 secs 2024-07-26T13:09:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:56 smithi183 ceph-mon[37273]: pgmap v4484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:09:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7439.696845 secs 2024-07-26T13:09:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:58 smithi144 ceph-mon[32129]: pgmap v4485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:58 smithi160 ceph-mon[36715]: pgmap v4485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:09:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:58 smithi183 ceph-mon[37273]: pgmap v4485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:09:59 smithi144 ceph-mon[32129]: pgmap v4486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:09:59 smithi160 ceph-mon[36715]: pgmap v4486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:09:59 smithi183 ceph-mon[37273]: pgmap v4486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7444.696919 secs 2024-07-26T13:10:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7444.696919 secs 2024-07-26T13:10:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7444.696919 secs 2024-07-26T13:10:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:01 smithi144 ceph-mon[32129]: pgmap v4487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:01 smithi160 ceph-mon[36715]: pgmap v4487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:01 smithi183 ceph-mon[37273]: pgmap v4487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:03.427+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:03.428+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:03.428+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:03.428+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:03.428+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:04 smithi183 ceph-mon[37273]: pgmap v4488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:04 smithi144 ceph-mon[32129]: pgmap v4488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:04 smithi160 ceph-mon[36715]: pgmap v4488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:06 smithi183 ceph-mon[37273]: pgmap v4489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7449.697044 secs 2024-07-26T13:10:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:06 smithi144 ceph-mon[32129]: pgmap v4489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7449.697044 secs 2024-07-26T13:10:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:06 smithi160 ceph-mon[36715]: pgmap v4489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7449.697044 secs 2024-07-26T13:10:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:08 smithi183 ceph-mon[37273]: pgmap v4490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:08 smithi144 ceph-mon[32129]: pgmap v4490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:08 smithi160 ceph-mon[36715]: pgmap v4490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:10 smithi183 ceph-mon[37273]: pgmap v4491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:10 smithi144 ceph-mon[32129]: pgmap v4491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:10 smithi160 ceph-mon[36715]: pgmap v4491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7454.697166 secs 2024-07-26T13:10:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7454.697166 secs 2024-07-26T13:10:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7454.697166 secs 2024-07-26T13:10:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:12 smithi144 ceph-mon[32129]: pgmap v4492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:12 smithi160 ceph-mon[36715]: pgmap v4492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:12 smithi183 ceph-mon[37273]: pgmap v4492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:14 smithi144 ceph-mon[32129]: pgmap v4493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:14.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:14 smithi160 ceph-mon[36715]: pgmap v4493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:14 smithi183 ceph-mon[37273]: pgmap v4493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:16 smithi144 ceph-mon[32129]: pgmap v4494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7459.697286 secs 2024-07-26T13:10:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:16 smithi160 ceph-mon[36715]: pgmap v4494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7459.697286 secs 2024-07-26T13:10:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:16 smithi183 ceph-mon[37273]: pgmap v4494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7459.697286 secs 2024-07-26T13:10:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:18.445+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:18.446+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:18.446+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:18.446+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:18.446+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:18 smithi144 ceph-mon[32129]: pgmap v4495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:18 smithi160 ceph-mon[36715]: pgmap v4495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:18 smithi183 ceph-mon[37273]: pgmap v4495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:20 smithi144 ceph-mon[32129]: pgmap v4496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:20 smithi160 ceph-mon[36715]: pgmap v4496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:20 smithi183 ceph-mon[37273]: pgmap v4496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7464.697434 secs 2024-07-26T13:10:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7464.697434 secs 2024-07-26T13:10:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7464.697434 secs 2024-07-26T13:10:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:22 smithi144 ceph-mon[32129]: pgmap v4497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:22 smithi160 ceph-mon[36715]: pgmap v4497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:22 smithi183 ceph-mon[37273]: pgmap v4497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:24 smithi144 ceph-mon[32129]: pgmap v4498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:24 smithi160 ceph-mon[36715]: pgmap v4498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:24 smithi183 ceph-mon[37273]: pgmap v4498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:26 smithi144 ceph-mon[32129]: pgmap v4499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7469.697557 secs 2024-07-26T13:10:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:26 smithi160 ceph-mon[36715]: pgmap v4499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7469.697557 secs 2024-07-26T13:10:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:26 smithi183 ceph-mon[37273]: pgmap v4499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7469.697557 secs 2024-07-26T13:10:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:28 smithi144 ceph-mon[32129]: pgmap v4500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:28 smithi160 ceph-mon[36715]: pgmap v4500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:28 smithi183 ceph-mon[37273]: pgmap v4500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: pgmap v4501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: pgmap v4501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: pgmap v4501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:30.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: pgmap v4502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: pgmap v4503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:10:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7474.697678 secs 2024-07-26T13:10:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:10:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:10:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:10:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:10:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: pgmap v4502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:10:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: pgmap v4503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:10:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:10:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7474.697678 secs 2024-07-26T13:10:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: pgmap v4502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: pgmap v4503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7474.697678 secs 2024-07-26T13:10:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:33.459+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:33.460+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:33.460+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:33.460+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:33.460+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:33 smithi160 ceph-mon[36715]: pgmap v4504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:10:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:33 smithi183 ceph-mon[37273]: pgmap v4504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:10:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:33 smithi144 ceph-mon[32129]: pgmap v4504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:10:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:35 smithi183 ceph-mon[37273]: pgmap v4505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:10:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:35 smithi144 ceph-mon[32129]: pgmap v4505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:10:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:35 smithi160 ceph-mon[36715]: pgmap v4505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:10:36.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7479.697796 secs 2024-07-26T13:10:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7479.697796 secs 2024-07-26T13:10:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7479.697796 secs 2024-07-26T13:10:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:37 smithi183 ceph-mon[37273]: pgmap v4506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:10:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:37 smithi144 ceph-mon[32129]: pgmap v4506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:10:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:37 smithi160 ceph-mon[36715]: pgmap v4506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:10:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:39 smithi183 ceph-mon[37273]: pgmap v4507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:10:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:39 smithi144 ceph-mon[32129]: pgmap v4507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:10:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:39 smithi160 ceph-mon[36715]: pgmap v4507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:10:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:41 smithi183 ceph-mon[37273]: pgmap v4508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:10:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7484.697909 secs 2024-07-26T13:10:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:41 smithi144 ceph-mon[32129]: pgmap v4508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:10:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7484.697909 secs 2024-07-26T13:10:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:41 smithi160 ceph-mon[36715]: pgmap v4508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:10:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7484.697909 secs 2024-07-26T13:10:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:43 smithi183 ceph-mon[37273]: pgmap v4509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:43 smithi144 ceph-mon[32129]: pgmap v4509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:43 smithi160 ceph-mon[36715]: pgmap v4509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:45.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:45 smithi160 ceph-mon[36715]: pgmap v4510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:45 smithi183 ceph-mon[37273]: pgmap v4510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:45 smithi144 ceph-mon[32129]: pgmap v4510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7489.698036 secs 2024-07-26T13:10:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7489.698036 secs 2024-07-26T13:10:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7489.698036 secs 2024-07-26T13:10:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:47 smithi183 ceph-mon[37273]: pgmap v4511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:47 smithi144 ceph-mon[32129]: pgmap v4511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:47 smithi160 ceph-mon[36715]: pgmap v4511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:48.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:48.476+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:48.476+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:48.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:10:48.477+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:49 smithi183 ceph-mon[37273]: pgmap v4512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:49 smithi144 ceph-mon[32129]: pgmap v4512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:49 smithi160 ceph-mon[36715]: pgmap v4512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:51 smithi183 ceph-mon[37273]: pgmap v4513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7494.698114 secs 2024-07-26T13:10:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:51 smithi144 ceph-mon[32129]: pgmap v4513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7494.698114 secs 2024-07-26T13:10:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:51 smithi160 ceph-mon[36715]: pgmap v4513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7494.698114 secs 2024-07-26T13:10:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:53 smithi183 ceph-mon[37273]: pgmap v4514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:53 smithi144 ceph-mon[32129]: pgmap v4514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:53 smithi160 ceph-mon[36715]: pgmap v4514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:10:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:55 smithi183 ceph-mon[37273]: pgmap v4515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:55 smithi144 ceph-mon[32129]: pgmap v4515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:55 smithi160 ceph-mon[36715]: pgmap v4515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:10:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7499.698240 secs 2024-07-26T13:10:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7499.698240 secs 2024-07-26T13:10:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7499.698240 secs 2024-07-26T13:10:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:57 smithi144 ceph-mon[32129]: pgmap v4516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:57 smithi160 ceph-mon[36715]: pgmap v4516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:57 smithi183 ceph-mon[37273]: pgmap v4516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:10:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:10:59 smithi144 ceph-mon[32129]: pgmap v4517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:10:59 smithi160 ceph-mon[36715]: pgmap v4517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:10:59 smithi183 ceph-mon[37273]: pgmap v4517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:01 smithi144 ceph-mon[32129]: pgmap v4518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7504.698366 secs 2024-07-26T13:11:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:01 smithi160 ceph-mon[36715]: pgmap v4518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7504.698366 secs 2024-07-26T13:11:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:01 smithi183 ceph-mon[37273]: pgmap v4518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7504.698366 secs 2024-07-26T13:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:03.491+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:03.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:03.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:03.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:03.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:03 smithi144 ceph-mon[32129]: pgmap v4519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:03 smithi160 ceph-mon[36715]: pgmap v4519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:03 smithi183 ceph-mon[37273]: pgmap v4519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:05 smithi144 ceph-mon[32129]: pgmap v4520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:05 smithi160 ceph-mon[36715]: pgmap v4520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:05 smithi183 ceph-mon[37273]: pgmap v4520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7509.698492 secs 2024-07-26T13:11:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7509.698492 secs 2024-07-26T13:11:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7509.698492 secs 2024-07-26T13:11:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:07 smithi144 ceph-mon[32129]: pgmap v4521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:07 smithi160 ceph-mon[36715]: pgmap v4521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:07 smithi183 ceph-mon[37273]: pgmap v4521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:09 smithi144 ceph-mon[32129]: pgmap v4522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:09 smithi160 ceph-mon[36715]: pgmap v4522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:09 smithi183 ceph-mon[37273]: pgmap v4522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:11 smithi144 ceph-mon[32129]: pgmap v4523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7514.698580 secs 2024-07-26T13:11:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:11 smithi160 ceph-mon[36715]: pgmap v4523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7514.698580 secs 2024-07-26T13:11:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:11 smithi183 ceph-mon[37273]: pgmap v4523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7514.698580 secs 2024-07-26T13:11:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:13 smithi144 ceph-mon[32129]: pgmap v4524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:13 smithi160 ceph-mon[36715]: pgmap v4524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:13 smithi183 ceph-mon[37273]: pgmap v4524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:15 smithi144 ceph-mon[32129]: pgmap v4525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:15 smithi160 ceph-mon[36715]: pgmap v4525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:15 smithi183 ceph-mon[37273]: pgmap v4525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7519.698666 secs 2024-07-26T13:11:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7519.698666 secs 2024-07-26T13:11:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7519.698666 secs 2024-07-26T13:11:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:17 smithi160 ceph-mon[36715]: pgmap v4526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:17 smithi183 ceph-mon[37273]: pgmap v4526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:17 smithi144 ceph-mon[32129]: pgmap v4526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:18.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:18.503+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:18.503+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:18.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:18.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:19 smithi160 ceph-mon[36715]: pgmap v4527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:19 smithi183 ceph-mon[37273]: pgmap v4527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:19 smithi144 ceph-mon[32129]: pgmap v4527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:21 smithi160 ceph-mon[36715]: pgmap v4528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7524.698808 secs 2024-07-26T13:11:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:21 smithi183 ceph-mon[37273]: pgmap v4528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7524.698808 secs 2024-07-26T13:11:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:21 smithi144 ceph-mon[32129]: pgmap v4528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7524.698808 secs 2024-07-26T13:11:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:23 smithi160 ceph-mon[36715]: pgmap v4529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:23 smithi183 ceph-mon[37273]: pgmap v4529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:23 smithi144 ceph-mon[32129]: pgmap v4529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:24 smithi144 ceph-mon[32129]: pgmap v4530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:24 smithi160 ceph-mon[36715]: pgmap v4530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:24 smithi183 ceph-mon[37273]: pgmap v4530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7529.698924 secs 2024-07-26T13:11:26.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7529.698924 secs 2024-07-26T13:11:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7529.698924 secs 2024-07-26T13:11:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:26 smithi144 ceph-mon[32129]: pgmap v4531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:26 smithi160 ceph-mon[36715]: pgmap v4531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:26 smithi183 ceph-mon[37273]: pgmap v4531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:29 smithi183 ceph-mon[37273]: pgmap v4532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:29 smithi144 ceph-mon[32129]: pgmap v4532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:29 smithi160 ceph-mon[36715]: pgmap v4532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:31 smithi183 ceph-mon[37273]: pgmap v4533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7534.699074 secs 2024-07-26T13:11:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:31 smithi144 ceph-mon[32129]: pgmap v4533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7534.699074 secs 2024-07-26T13:11:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:31 smithi160 ceph-mon[36715]: pgmap v4533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7534.699074 secs 2024-07-26T13:11:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:32 smithi183 ceph-mon[37273]: pgmap v4534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:11:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:32 smithi183 ceph-mon[37273]: pgmap v4535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:11:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:11:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:32 smithi144 ceph-mon[32129]: pgmap v4534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:11:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:32 smithi144 ceph-mon[32129]: pgmap v4535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:11:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:11:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:32 smithi160 ceph-mon[36715]: pgmap v4534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:11:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:32 smithi160 ceph-mon[36715]: pgmap v4535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:11:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:11:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:33.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:33.520+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:33.520+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:33.520+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:33.520+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:34 smithi160 ceph-mon[36715]: pgmap v4536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:11:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:34 smithi183 ceph-mon[37273]: pgmap v4536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:11:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:34 smithi144 ceph-mon[32129]: pgmap v4536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:11:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7539.699175 secs 2024-07-26T13:11:36.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7539.699175 secs 2024-07-26T13:11:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7539.699175 secs 2024-07-26T13:11:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:36 smithi160 ceph-mon[36715]: pgmap v4537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:11:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:36 smithi183 ceph-mon[37273]: pgmap v4537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:11:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:36 smithi144 ceph-mon[32129]: pgmap v4537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:11:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:38 smithi160 ceph-mon[36715]: pgmap v4538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:11:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:38 smithi183 ceph-mon[37273]: pgmap v4538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:11:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:38 smithi144 ceph-mon[32129]: pgmap v4538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:40 smithi160 ceph-mon[36715]: pgmap v4539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:11:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7544.699266 secs 2024-07-26T13:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:40 smithi183 ceph-mon[37273]: pgmap v4539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:11:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7544.699266 secs 2024-07-26T13:11:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:40 smithi144 ceph-mon[32129]: pgmap v4539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:11:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7544.699266 secs 2024-07-26T13:11:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:42 smithi160 ceph-mon[36715]: pgmap v4540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:11:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:42 smithi183 ceph-mon[37273]: pgmap v4540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:11:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:42 smithi144 ceph-mon[32129]: pgmap v4540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:11:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:44 smithi160 ceph-mon[36715]: pgmap v4541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:44 smithi183 ceph-mon[37273]: pgmap v4541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:44 smithi144 ceph-mon[32129]: pgmap v4541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7549.699328 secs 2024-07-26T13:11:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7549.699328 secs 2024-07-26T13:11:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7549.699328 secs 2024-07-26T13:11:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:46 smithi160 ceph-mon[36715]: pgmap v4542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:47.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:46 smithi144 ceph-mon[32129]: pgmap v4542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:46 smithi183 ceph-mon[37273]: pgmap v4542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:47.437+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:11:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:47.437+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:11:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:47.437+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T13:11:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:47.437+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:11:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:48.536+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:48.536+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:48.536+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:48.536+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:11:48.537+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:48 smithi160 ceph-mon[36715]: pgmap v4543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:48 smithi183 ceph-mon[37273]: pgmap v4543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:48 smithi144 ceph-mon[32129]: pgmap v4543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:49 smithi183 ceph-mon[37273]: pgmap v4544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:49 smithi144 ceph-mon[32129]: pgmap v4544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:49 smithi160 ceph-mon[36715]: pgmap v4544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7554.699447 secs 2024-07-26T13:11:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7554.699447 secs 2024-07-26T13:11:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7554.699447 secs 2024-07-26T13:11:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:51 smithi183 ceph-mon[37273]: pgmap v4545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:51 smithi144 ceph-mon[32129]: pgmap v4545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:51 smithi160 ceph-mon[36715]: pgmap v4545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:11:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:54 smithi160 ceph-mon[36715]: pgmap v4546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:54 smithi183 ceph-mon[37273]: pgmap v4546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:54 smithi144 ceph-mon[32129]: pgmap v4546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:56.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7559.699564 secs 2024-07-26T13:11:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7559.699564 secs 2024-07-26T13:11:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7559.699564 secs 2024-07-26T13:11:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:56 smithi160 ceph-mon[36715]: pgmap v4547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:56 smithi183 ceph-mon[37273]: pgmap v4547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:56 smithi144 ceph-mon[32129]: pgmap v4547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:11:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:11:58 smithi160 ceph-mon[36715]: pgmap v4548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:11:58 smithi183 ceph-mon[37273]: pgmap v4548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:11:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:11:58 smithi144 ceph-mon[32129]: pgmap v4548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:00 smithi160 ceph-mon[36715]: pgmap v4549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7564.699664 secs 2024-07-26T13:12:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:00 smithi183 ceph-mon[37273]: pgmap v4549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7564.699664 secs 2024-07-26T13:12:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:00 smithi144 ceph-mon[32129]: pgmap v4549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7564.699664 secs 2024-07-26T13:12:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:02 smithi160 ceph-mon[36715]: pgmap v4550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:02 smithi183 ceph-mon[37273]: pgmap v4550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:02 smithi144 ceph-mon[32129]: pgmap v4550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:03.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:03.550+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:03.550+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:03.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:03.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:04 smithi160 ceph-mon[36715]: pgmap v4551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:04 smithi183 ceph-mon[37273]: pgmap v4551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:04 smithi144 ceph-mon[32129]: pgmap v4551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:06.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7569.699809 secs 2024-07-26T13:12:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7569.699809 secs 2024-07-26T13:12:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7569.699809 secs 2024-07-26T13:12:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:06 smithi183 ceph-mon[37273]: pgmap v4552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:06 smithi144 ceph-mon[32129]: pgmap v4552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:06 smithi160 ceph-mon[36715]: pgmap v4552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:07 smithi183 ceph-mon[37273]: pgmap v4553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:07 smithi144 ceph-mon[32129]: pgmap v4553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:07 smithi160 ceph-mon[36715]: pgmap v4553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:10 smithi160 ceph-mon[36715]: pgmap v4554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7574.699954 secs 2024-07-26T13:12:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:10 smithi183 ceph-mon[37273]: pgmap v4554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7574.699954 secs 2024-07-26T13:12:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:10 smithi144 ceph-mon[32129]: pgmap v4554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7574.699954 secs 2024-07-26T13:12:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:12 smithi160 ceph-mon[36715]: pgmap v4555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:12 smithi183 ceph-mon[37273]: pgmap v4555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:12 smithi144 ceph-mon[32129]: pgmap v4555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:14 smithi160 ceph-mon[36715]: pgmap v4556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:14 smithi183 ceph-mon[37273]: pgmap v4556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:14 smithi144 ceph-mon[32129]: pgmap v4556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7579.700099 secs 2024-07-26T13:12:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7579.700099 secs 2024-07-26T13:12:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7579.700099 secs 2024-07-26T13:12:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:16 smithi160 ceph-mon[36715]: pgmap v4557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:16 smithi183 ceph-mon[37273]: pgmap v4557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:16 smithi144 ceph-mon[32129]: pgmap v4557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:18.567+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:18.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:18.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:18.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:18.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:18 smithi160 ceph-mon[36715]: pgmap v4558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:12:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:18 smithi183 ceph-mon[37273]: pgmap v4558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:12:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:18 smithi144 ceph-mon[32129]: pgmap v4558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:12:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:20 smithi183 ceph-mon[37273]: pgmap v4559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7584.700217 secs 2024-07-26T13:12:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:20 smithi144 ceph-mon[32129]: pgmap v4559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7584.700217 secs 2024-07-26T13:12:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:20 smithi160 ceph-mon[36715]: pgmap v4559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7584.700217 secs 2024-07-26T13:12:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:22 smithi183 ceph-mon[37273]: pgmap v4560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:12:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:22 smithi144 ceph-mon[32129]: pgmap v4560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:12:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:22 smithi160 ceph-mon[36715]: pgmap v4560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:12:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:23 smithi183 ceph-mon[37273]: pgmap v4561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:23 smithi144 ceph-mon[32129]: pgmap v4561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:23 smithi160 ceph-mon[36715]: pgmap v4561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7589.700361 secs 2024-07-26T13:12:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7589.700361 secs 2024-07-26T13:12:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7589.700361 secs 2024-07-26T13:12:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:26 smithi160 ceph-mon[36715]: pgmap v4562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:26 smithi183 ceph-mon[37273]: pgmap v4562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:26 smithi144 ceph-mon[32129]: pgmap v4562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:28 smithi183 ceph-mon[37273]: pgmap v4563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:28 smithi144 ceph-mon[32129]: pgmap v4563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:28 smithi160 ceph-mon[36715]: pgmap v4563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:30 smithi183 ceph-mon[37273]: pgmap v4564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7594.700486 secs 2024-07-26T13:12:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:30 smithi144 ceph-mon[32129]: pgmap v4564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7594.700486 secs 2024-07-26T13:12:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:30 smithi160 ceph-mon[36715]: pgmap v4564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7594.700486 secs 2024-07-26T13:12:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:31 smithi183 ceph-mon[37273]: pgmap v4565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:31 smithi144 ceph-mon[32129]: pgmap v4565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:31 smithi160 ceph-mon[36715]: pgmap v4565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:33.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:33.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:33.584+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:33.584+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:33.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:33.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:33 smithi183 ceph-mon[37273]: pgmap v4566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:33 smithi183 ceph-mon[37273]: pgmap v4567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:33 smithi144 ceph-mon[32129]: pgmap v4566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:33 smithi144 ceph-mon[32129]: pgmap v4567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:33 smithi160 ceph-mon[36715]: pgmap v4566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:33 smithi160 ceph-mon[36715]: pgmap v4567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:12:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:36 smithi183 ceph-mon[37273]: pgmap v4568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7599.700616 secs 2024-07-26T13:12:36.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:36 smithi144 ceph-mon[32129]: pgmap v4568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:36.923 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7599.700616 secs 2024-07-26T13:12:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:36 smithi160 ceph-mon[36715]: pgmap v4568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7599.700616 secs 2024-07-26T13:12:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:38 smithi183 ceph-mon[37273]: pgmap v4569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:38 smithi144 ceph-mon[32129]: pgmap v4569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:38 smithi160 ceph-mon[36715]: pgmap v4569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:40 smithi183 ceph-mon[37273]: pgmap v4570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:40 smithi144 ceph-mon[32129]: pgmap v4570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:40 smithi160 ceph-mon[36715]: pgmap v4570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7604.700742 secs 2024-07-26T13:12:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7604.700742 secs 2024-07-26T13:12:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7604.700742 secs 2024-07-26T13:12:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:42 smithi183 ceph-mon[37273]: pgmap v4571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:42 smithi144 ceph-mon[32129]: pgmap v4571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:42 smithi160 ceph-mon[36715]: pgmap v4571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:12:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:44 smithi183 ceph-mon[37273]: pgmap v4572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:12:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:44 smithi144 ceph-mon[32129]: pgmap v4572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:12:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:44 smithi160 ceph-mon[36715]: pgmap v4572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:12:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:46 smithi144 ceph-mon[32129]: pgmap v4573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7609.700860 secs 2024-07-26T13:12:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:46 smithi160 ceph-mon[36715]: pgmap v4573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7609.700860 secs 2024-07-26T13:12:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:46 smithi183 ceph-mon[37273]: pgmap v4573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7609.700860 secs 2024-07-26T13:12:48.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:48.597+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:48.598+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:48.598+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:48.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:12:48.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:48 smithi144 ceph-mon[32129]: pgmap v4574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:48 smithi160 ceph-mon[36715]: pgmap v4574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:48 smithi183 ceph-mon[37273]: pgmap v4574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:50 smithi144 ceph-mon[32129]: pgmap v4575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:50 smithi160 ceph-mon[36715]: pgmap v4575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:50 smithi183 ceph-mon[37273]: pgmap v4575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7614.700991 secs 2024-07-26T13:12:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7614.700991 secs 2024-07-26T13:12:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7614.700991 secs 2024-07-26T13:12:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:52 smithi144 ceph-mon[32129]: pgmap v4576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:52 smithi160 ceph-mon[36715]: pgmap v4576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:52 smithi183 ceph-mon[37273]: pgmap v4576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:12:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:54 smithi144 ceph-mon[32129]: pgmap v4577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:54 smithi160 ceph-mon[36715]: pgmap v4577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:54 smithi183 ceph-mon[37273]: pgmap v4577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:56 smithi144 ceph-mon[32129]: pgmap v4578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7619.701114 secs 2024-07-26T13:12:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:56 smithi160 ceph-mon[36715]: pgmap v4578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7619.701114 secs 2024-07-26T13:12:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:56 smithi183 ceph-mon[37273]: pgmap v4578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:12:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7619.701114 secs 2024-07-26T13:12:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:12:58 smithi144 ceph-mon[32129]: pgmap v4579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:12:58 smithi160 ceph-mon[36715]: pgmap v4579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:12:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:12:58 smithi183 ceph-mon[37273]: pgmap v4579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:00 smithi144 ceph-mon[32129]: pgmap v4580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:00 smithi160 ceph-mon[36715]: pgmap v4580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:00 smithi183 ceph-mon[37273]: pgmap v4580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7624.701243 secs 2024-07-26T13:13:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7624.701243 secs 2024-07-26T13:13:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7624.701243 secs 2024-07-26T13:13:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:02 smithi144 ceph-mon[32129]: pgmap v4581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:02 smithi160 ceph-mon[36715]: pgmap v4581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:02 smithi183 ceph-mon[37273]: pgmap v4581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:03.613+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:03.614+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:03.614+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:03.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:03.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:04 smithi144 ceph-mon[32129]: pgmap v4582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:04 smithi160 ceph-mon[36715]: pgmap v4582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:04 smithi183 ceph-mon[37273]: pgmap v4582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:06 smithi160 ceph-mon[36715]: pgmap v4583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7629.701363 secs 2024-07-26T13:13:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:06 smithi183 ceph-mon[37273]: pgmap v4583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7629.701363 secs 2024-07-26T13:13:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:06 smithi144 ceph-mon[32129]: pgmap v4583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7629.701363 secs 2024-07-26T13:13:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:08 smithi160 ceph-mon[36715]: pgmap v4584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:08 smithi183 ceph-mon[37273]: pgmap v4584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:08 smithi144 ceph-mon[32129]: pgmap v4584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:10 smithi160 ceph-mon[36715]: pgmap v4585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:10 smithi183 ceph-mon[37273]: pgmap v4585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:10 smithi144 ceph-mon[32129]: pgmap v4585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7634.701504 secs 2024-07-26T13:13:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7634.701504 secs 2024-07-26T13:13:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7634.701504 secs 2024-07-26T13:13:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:12 smithi160 ceph-mon[36715]: pgmap v4586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:12 smithi183 ceph-mon[37273]: pgmap v4586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:12 smithi144 ceph-mon[32129]: pgmap v4586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:14 smithi160 ceph-mon[36715]: pgmap v4587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:14 smithi183 ceph-mon[37273]: pgmap v4587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:14 smithi144 ceph-mon[32129]: pgmap v4587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:16 smithi160 ceph-mon[36715]: pgmap v4588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7639.701628 secs 2024-07-26T13:13:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:16 smithi183 ceph-mon[37273]: pgmap v4588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7639.701628 secs 2024-07-26T13:13:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:16 smithi144 ceph-mon[32129]: pgmap v4588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7639.701628 secs 2024-07-26T13:13:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:18.626+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:18.627+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:18.628 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:18.627+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:18.628 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:18.627+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:18.628 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:18.627+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:18 smithi160 ceph-mon[36715]: pgmap v4589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:18 smithi183 ceph-mon[37273]: pgmap v4589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:18 smithi144 ceph-mon[32129]: pgmap v4589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:21.053 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:20 smithi144 ceph-mon[32129]: pgmap v4590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:21.054 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7644.701754 secs 2024-07-26T13:13:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:20 smithi160 ceph-mon[36715]: pgmap v4590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7644.701754 secs 2024-07-26T13:13:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:20 smithi183 ceph-mon[37273]: pgmap v4590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7644.701754 secs 2024-07-26T13:13:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:22 smithi160 ceph-mon[36715]: pgmap v4591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:22 smithi183 ceph-mon[37273]: pgmap v4591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:22 smithi144 ceph-mon[32129]: pgmap v4591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:24 smithi160 ceph-mon[36715]: pgmap v4592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:24 smithi183 ceph-mon[37273]: pgmap v4592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:24 smithi144 ceph-mon[32129]: pgmap v4592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7649.701830 secs 2024-07-26T13:13:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7649.701830 secs 2024-07-26T13:13:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7649.701830 secs 2024-07-26T13:13:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:26 smithi160 ceph-mon[36715]: pgmap v4593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:26 smithi183 ceph-mon[37273]: pgmap v4593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:26 smithi144 ceph-mon[32129]: pgmap v4593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:28 smithi160 ceph-mon[36715]: pgmap v4594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:28 smithi183 ceph-mon[37273]: pgmap v4594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:28 smithi144 ceph-mon[32129]: pgmap v4594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:30 smithi160 ceph-mon[36715]: pgmap v4595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7654.701914 secs 2024-07-26T13:13:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:30 smithi183 ceph-mon[37273]: pgmap v4595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7654.701914 secs 2024-07-26T13:13:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:30 smithi144 ceph-mon[32129]: pgmap v4595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7654.701914 secs 2024-07-26T13:13:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:32 smithi183 ceph-mon[37273]: pgmap v4596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:32 smithi144 ceph-mon[32129]: pgmap v4596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:32 smithi160 ceph-mon[36715]: pgmap v4596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:33.642+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:33.644+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:33.644+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:33.645 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:33.644+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:33.645 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:33.644+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:34.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:34 smithi160 ceph-mon[36715]: pgmap v4597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:34 smithi183 ceph-mon[37273]: pgmap v4597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:35.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:34 smithi144 ceph-mon[32129]: pgmap v4597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7659.701980 secs 2024-07-26T13:13:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7659.701980 secs 2024-07-26T13:13:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7659.701980 secs 2024-07-26T13:13:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:37.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:36 smithi144 ceph-mon[32129]: pgmap v4598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:37.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:37.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:36 smithi183 ceph-mon[37273]: pgmap v4598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:36 smithi160 ceph-mon[36715]: pgmap v4598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:38 smithi144 ceph-mon[32129]: pgmap v4599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:38 smithi160 ceph-mon[36715]: pgmap v4599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:38 smithi183 ceph-mon[37273]: pgmap v4599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:13:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:39 smithi144 ceph-mon[32129]: pgmap v4600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:39 smithi144 ceph-mon[32129]: pgmap v4601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T13:13:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:39 smithi160 ceph-mon[36715]: pgmap v4600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:39 smithi160 ceph-mon[36715]: pgmap v4601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T13:13:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:39 smithi183 ceph-mon[37273]: pgmap v4600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:39 smithi183 ceph-mon[37273]: pgmap v4601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T13:13:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:41 smithi160 ceph-mon[36715]: pgmap v4602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7664.702105 secs 2024-07-26T13:13:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:41 smithi183 ceph-mon[37273]: pgmap v4602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7664.702105 secs 2024-07-26T13:13:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:41 smithi144 ceph-mon[32129]: pgmap v4602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7664.702105 secs 2024-07-26T13:13:43.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:43 smithi160 ceph-mon[36715]: pgmap v4603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:43 smithi183 ceph-mon[37273]: pgmap v4603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:43 smithi144 ceph-mon[32129]: pgmap v4603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:45 smithi160 ceph-mon[36715]: pgmap v4604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:45 smithi183 ceph-mon[37273]: pgmap v4604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:45 smithi144 ceph-mon[32129]: pgmap v4604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7669.702228 secs 2024-07-26T13:13:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7669.702228 secs 2024-07-26T13:13:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7669.702228 secs 2024-07-26T13:13:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:47 smithi160 ceph-mon[36715]: pgmap v4605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:47 smithi183 ceph-mon[37273]: pgmap v4605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:47 smithi144 ceph-mon[32129]: pgmap v4605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T13:13:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:48.659+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:48.660+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:48.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:48.660+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:48.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:48.660+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:48.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:13:48.661+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:49 smithi160 ceph-mon[36715]: pgmap v4606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:13:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:49 smithi183 ceph-mon[37273]: pgmap v4606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:13:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:49 smithi144 ceph-mon[32129]: pgmap v4606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:13:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:51 smithi160 ceph-mon[36715]: pgmap v4607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7674.702335 secs 2024-07-26T13:13:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:51 smithi183 ceph-mon[37273]: pgmap v4607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:51.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7674.702335 secs 2024-07-26T13:13:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:51 smithi144 ceph-mon[32129]: pgmap v4607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7674.702335 secs 2024-07-26T13:13:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:53 smithi160 ceph-mon[36715]: pgmap v4608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:53 smithi183 ceph-mon[37273]: pgmap v4608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:53 smithi144 ceph-mon[32129]: pgmap v4608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:13:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:55 smithi160 ceph-mon[36715]: pgmap v4609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:55 smithi183 ceph-mon[37273]: pgmap v4609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:55 smithi144 ceph-mon[32129]: pgmap v4609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7679.702460 secs 2024-07-26T13:13:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7679.702460 secs 2024-07-26T13:13:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7679.702460 secs 2024-07-26T13:13:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:57 smithi160 ceph-mon[36715]: pgmap v4610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:57 smithi183 ceph-mon[37273]: pgmap v4610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:57 smithi144 ceph-mon[32129]: pgmap v4610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:13:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:13:59 smithi160 ceph-mon[36715]: pgmap v4611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:13:59 smithi183 ceph-mon[37273]: pgmap v4611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:13:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:13:59 smithi144 ceph-mon[32129]: pgmap v4611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:01 smithi160 ceph-mon[36715]: pgmap v4612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7684.702559 secs 2024-07-26T13:14:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:01 smithi183 ceph-mon[37273]: pgmap v4612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7684.702559 secs 2024-07-26T13:14:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:01 smithi144 ceph-mon[32129]: pgmap v4612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7684.702559 secs 2024-07-26T13:14:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:03.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:03.676+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:03.676+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:03.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:03.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:03 smithi160 ceph-mon[36715]: pgmap v4613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:03 smithi183 ceph-mon[37273]: pgmap v4613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:03 smithi144 ceph-mon[32129]: pgmap v4613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:05 smithi183 ceph-mon[37273]: pgmap v4614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:05 smithi144 ceph-mon[32129]: pgmap v4614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:05 smithi160 ceph-mon[36715]: pgmap v4614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7689.702685 secs 2024-07-26T13:14:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7689.702685 secs 2024-07-26T13:14:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7689.702685 secs 2024-07-26T13:14:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:07 smithi183 ceph-mon[37273]: pgmap v4615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:07 smithi144 ceph-mon[32129]: pgmap v4615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:07 smithi160 ceph-mon[36715]: pgmap v4615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:09 smithi183 ceph-mon[37273]: pgmap v4616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:09 smithi144 ceph-mon[32129]: pgmap v4616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:09 smithi160 ceph-mon[36715]: pgmap v4616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:11 smithi183 ceph-mon[37273]: pgmap v4617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7694.702833 secs 2024-07-26T13:14:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:11 smithi144 ceph-mon[32129]: pgmap v4617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7694.702833 secs 2024-07-26T13:14:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:11 smithi160 ceph-mon[36715]: pgmap v4617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7694.702833 secs 2024-07-26T13:14:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:13 smithi183 ceph-mon[37273]: pgmap v4618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:13 smithi144 ceph-mon[32129]: pgmap v4618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:13 smithi160 ceph-mon[36715]: pgmap v4618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:15 smithi183 ceph-mon[37273]: pgmap v4619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:15 smithi144 ceph-mon[32129]: pgmap v4619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:15 smithi160 ceph-mon[36715]: pgmap v4619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7699.702949 secs 2024-07-26T13:14:16.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7699.702949 secs 2024-07-26T13:14:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7699.702949 secs 2024-07-26T13:14:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:17 smithi183 ceph-mon[37273]: pgmap v4620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:17 smithi144 ceph-mon[32129]: pgmap v4620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:17 smithi160 ceph-mon[36715]: pgmap v4620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:18.693+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:18.694+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:18.694+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:18.694+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:18.694+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:19 smithi183 ceph-mon[37273]: pgmap v4621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:19 smithi144 ceph-mon[32129]: pgmap v4621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:19 smithi160 ceph-mon[36715]: pgmap v4621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:21 smithi183 ceph-mon[37273]: pgmap v4622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7704.703083 secs 2024-07-26T13:14:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:21 smithi144 ceph-mon[32129]: pgmap v4622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7704.703083 secs 2024-07-26T13:14:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:21 smithi160 ceph-mon[36715]: pgmap v4622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:22.104 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7704.703083 secs 2024-07-26T13:14:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:23 smithi183 ceph-mon[37273]: pgmap v4623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:23 smithi144 ceph-mon[32129]: pgmap v4623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:23 smithi160 ceph-mon[36715]: pgmap v4623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:25 smithi183 ceph-mon[37273]: pgmap v4624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:25 smithi144 ceph-mon[32129]: pgmap v4624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:25 smithi160 ceph-mon[36715]: pgmap v4624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7709.703210 secs 2024-07-26T13:14:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7709.703210 secs 2024-07-26T13:14:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7709.703210 secs 2024-07-26T13:14:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:27 smithi144 ceph-mon[32129]: pgmap v4625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:27 smithi160 ceph-mon[36715]: pgmap v4625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:27 smithi183 ceph-mon[37273]: pgmap v4625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:29.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:29 smithi183 ceph-mon[37273]: pgmap v4626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:29 smithi144 ceph-mon[32129]: pgmap v4626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:29 smithi160 ceph-mon[36715]: pgmap v4626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:31 smithi144 ceph-mon[32129]: pgmap v4627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7714.703341 secs 2024-07-26T13:14:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:31 smithi160 ceph-mon[36715]: pgmap v4627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7714.703341 secs 2024-07-26T13:14:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:31 smithi183 ceph-mon[37273]: pgmap v4627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7714.703341 secs 2024-07-26T13:14:33.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:33.710+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:33.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:33.711+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:33.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:33.711+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:33.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:33.711+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:33.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:33.711+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:33 smithi144 ceph-mon[32129]: pgmap v4628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:33 smithi160 ceph-mon[36715]: pgmap v4628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:33 smithi183 ceph-mon[37273]: pgmap v4628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:35 smithi144 ceph-mon[32129]: pgmap v4629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:35 smithi160 ceph-mon[36715]: pgmap v4629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:35 smithi183 ceph-mon[37273]: pgmap v4629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7719.703474 secs 2024-07-26T13:14:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7719.703474 secs 2024-07-26T13:14:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7719.703474 secs 2024-07-26T13:14:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:37 smithi144 ceph-mon[32129]: pgmap v4630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:37 smithi160 ceph-mon[36715]: pgmap v4630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:37 smithi183 ceph-mon[37273]: pgmap v4630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:39 smithi144 ceph-mon[32129]: pgmap v4631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:14:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:39 smithi160 ceph-mon[36715]: pgmap v4631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:14:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:39 smithi183 ceph-mon[37273]: pgmap v4631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:14:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:40 smithi144 ceph-mon[32129]: pgmap v4632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:14:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:40 smithi144 ceph-mon[32129]: pgmap v4633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:14:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:40 smithi160 ceph-mon[36715]: pgmap v4632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:14:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:40 smithi160 ceph-mon[36715]: pgmap v4633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:14:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:40 smithi183 ceph-mon[37273]: pgmap v4632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:14:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:40 smithi183 ceph-mon[37273]: pgmap v4633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:14:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7724.703543 secs 2024-07-26T13:14:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7724.703543 secs 2024-07-26T13:14:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7724.703543 secs 2024-07-26T13:14:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:42 smithi160 ceph-mon[36715]: pgmap v4634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:14:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:42 smithi183 ceph-mon[37273]: pgmap v4634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:14:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:42 smithi144 ceph-mon[32129]: pgmap v4634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:14:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:44 smithi160 ceph-mon[36715]: pgmap v4635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:14:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:44 smithi183 ceph-mon[37273]: pgmap v4635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:14:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:44 smithi144 ceph-mon[32129]: pgmap v4635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:14:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:46 smithi160 ceph-mon[36715]: pgmap v4636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:14:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7729.703621 secs 2024-07-26T13:14:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:46 smithi183 ceph-mon[37273]: pgmap v4636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:14:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7729.703621 secs 2024-07-26T13:14:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:46 smithi144 ceph-mon[32129]: pgmap v4636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:14:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7729.703621 secs 2024-07-26T13:14:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:47.468+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:14:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:47.468+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:14:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:47.469+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:14:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:47.469+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:14:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:48.727+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:48.728+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:48.728+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:48.728+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:14:48.728+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:48 smithi160 ceph-mon[36715]: pgmap v4637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:14:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:48 smithi183 ceph-mon[37273]: pgmap v4637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:14:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:48 smithi144 ceph-mon[32129]: pgmap v4637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:14:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:50 smithi160 ceph-mon[36715]: pgmap v4638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:14:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:50 smithi183 ceph-mon[37273]: pgmap v4638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:14:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:50 smithi144 ceph-mon[32129]: pgmap v4638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:14:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7734.703744 secs 2024-07-26T13:14:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7734.703744 secs 2024-07-26T13:14:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7734.703744 secs 2024-07-26T13:14:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:52 smithi160 ceph-mon[36715]: pgmap v4639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:52 smithi183 ceph-mon[37273]: pgmap v4639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:53.213 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:52 smithi144 ceph-mon[32129]: pgmap v4639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:14:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:54 smithi160 ceph-mon[36715]: pgmap v4640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:54 smithi183 ceph-mon[37273]: pgmap v4640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:54 smithi144 ceph-mon[32129]: pgmap v4640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7739.703863 secs 2024-07-26T13:14:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7739.703863 secs 2024-07-26T13:14:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7739.703863 secs 2024-07-26T13:14:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:56 smithi160 ceph-mon[36715]: pgmap v4641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:56 smithi183 ceph-mon[37273]: pgmap v4641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:56 smithi144 ceph-mon[32129]: pgmap v4641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:14:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:14:58 smithi160 ceph-mon[36715]: pgmap v4642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:14:58 smithi183 ceph-mon[37273]: pgmap v4642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:14:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:14:58 smithi144 ceph-mon[32129]: pgmap v4642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:00 smithi160 ceph-mon[36715]: pgmap v4643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:15:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7744.704005 secs 2024-07-26T13:15:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:00 smithi183 ceph-mon[37273]: pgmap v4643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:15:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7744.704005 secs 2024-07-26T13:15:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:00 smithi144 ceph-mon[32129]: pgmap v4643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:15:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7744.704005 secs 2024-07-26T13:15:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:02 smithi183 ceph-mon[37273]: pgmap v4644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:02 smithi144 ceph-mon[32129]: pgmap v4644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:02 smithi160 ceph-mon[36715]: pgmap v4644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:03.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:03.743+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:03.743+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:03.743+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:03.743+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:04 smithi183 ceph-mon[37273]: pgmap v4645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:04 smithi144 ceph-mon[32129]: pgmap v4645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:04 smithi160 ceph-mon[36715]: pgmap v4645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7749.704142 secs 2024-07-26T13:15:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7749.704142 secs 2024-07-26T13:15:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7749.704142 secs 2024-07-26T13:15:07.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:06 smithi144 ceph-mon[32129]: pgmap v4646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:06 smithi183 ceph-mon[37273]: pgmap v4646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:06 smithi160 ceph-mon[36715]: pgmap v4646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:09.093 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:08 smithi144 ceph-mon[32129]: pgmap v4647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:08 smithi183 ceph-mon[37273]: pgmap v4647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:08 smithi160 ceph-mon[36715]: pgmap v4647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:10 smithi183 ceph-mon[37273]: pgmap v4648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:15:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7754.704233 secs 2024-07-26T13:15:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:10 smithi144 ceph-mon[32129]: pgmap v4648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:15:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7754.704233 secs 2024-07-26T13:15:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:10 smithi160 ceph-mon[36715]: pgmap v4648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:15:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7754.704233 secs 2024-07-26T13:15:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:12 smithi183 ceph-mon[37273]: pgmap v4649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:12 smithi144 ceph-mon[32129]: pgmap v4649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:12 smithi160 ceph-mon[36715]: pgmap v4649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:14 smithi183 ceph-mon[37273]: pgmap v4650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:14 smithi144 ceph-mon[32129]: pgmap v4650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:14 smithi160 ceph-mon[36715]: pgmap v4650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:15 smithi183 ceph-mon[37273]: pgmap v4651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7759.704378 secs 2024-07-26T13:15:16.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:15 smithi160 ceph-mon[36715]: pgmap v4651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:16.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7759.704378 secs 2024-07-26T13:15:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:15 smithi144 ceph-mon[32129]: pgmap v4651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7759.704378 secs 2024-07-26T13:15:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:18.760+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:18.760+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:18.761+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:18.761+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:18.761+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:18 smithi144 ceph-mon[32129]: pgmap v4652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:18 smithi160 ceph-mon[36715]: pgmap v4652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:18 smithi183 ceph-mon[37273]: pgmap v4652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:20.713 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:20 smithi160 ceph-mon[36715]: pgmap v4653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:20 smithi144 ceph-mon[32129]: pgmap v4653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:20 smithi183 ceph-mon[37273]: pgmap v4653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7764.704467 secs 2024-07-26T13:15:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7764.704467 secs 2024-07-26T13:15:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7764.704467 secs 2024-07-26T13:15:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:22 smithi160 ceph-mon[36715]: pgmap v4654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:22 smithi183 ceph-mon[37273]: pgmap v4654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:22 smithi144 ceph-mon[32129]: pgmap v4654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:25.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:24 smithi144 ceph-mon[32129]: pgmap v4655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:24 smithi160 ceph-mon[36715]: pgmap v4655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:24 smithi183 ceph-mon[37273]: pgmap v4655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:26 smithi160 ceph-mon[36715]: pgmap v4656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7769.704595 secs 2024-07-26T13:15:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:26 smithi183 ceph-mon[37273]: pgmap v4656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7769.704595 secs 2024-07-26T13:15:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:26 smithi144 ceph-mon[32129]: pgmap v4656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7769.704595 secs 2024-07-26T13:15:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:28 smithi160 ceph-mon[36715]: pgmap v4657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:28 smithi183 ceph-mon[37273]: pgmap v4657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:28 smithi144 ceph-mon[32129]: pgmap v4657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:30 smithi160 ceph-mon[36715]: pgmap v4658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:30 smithi183 ceph-mon[37273]: pgmap v4658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:30 smithi144 ceph-mon[32129]: pgmap v4658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7774.704676 secs 2024-07-26T13:15:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7774.704676 secs 2024-07-26T13:15:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7774.704676 secs 2024-07-26T13:15:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:32 smithi160 ceph-mon[36715]: pgmap v4659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:32 smithi183 ceph-mon[37273]: pgmap v4659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:32 smithi144 ceph-mon[32129]: pgmap v4659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:33.776+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:33.777+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:33.777+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:33.777+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:33.777+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:34 smithi160 ceph-mon[36715]: pgmap v4660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:34 smithi183 ceph-mon[37273]: pgmap v4660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:34 smithi144 ceph-mon[32129]: pgmap v4660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:36 smithi160 ceph-mon[36715]: pgmap v4661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7779.704803 secs 2024-07-26T13:15:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:36 smithi183 ceph-mon[37273]: pgmap v4661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7779.704803 secs 2024-07-26T13:15:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:36 smithi144 ceph-mon[32129]: pgmap v4661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7779.704803 secs 2024-07-26T13:15:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:38 smithi160 ceph-mon[36715]: pgmap v4662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:38 smithi183 ceph-mon[37273]: pgmap v4662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:38 smithi144 ceph-mon[32129]: pgmap v4662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:40 smithi160 ceph-mon[36715]: pgmap v4663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7784.704925 secs 2024-07-26T13:15:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:40 smithi183 ceph-mon[37273]: pgmap v4663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7784.704925 secs 2024-07-26T13:15:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:40 smithi144 ceph-mon[32129]: pgmap v4663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7784.704925 secs 2024-07-26T13:15:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:15:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:15:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:15:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:42 smithi160 ceph-mon[36715]: pgmap v4664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:42 smithi160 ceph-mon[36715]: pgmap v4665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:42 smithi183 ceph-mon[37273]: pgmap v4664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:42 smithi183 ceph-mon[37273]: pgmap v4665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:42 smithi144 ceph-mon[32129]: pgmap v4664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:42 smithi144 ceph-mon[32129]: pgmap v4665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:44 smithi160 ceph-mon[36715]: pgmap v4666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:44 smithi183 ceph-mon[37273]: pgmap v4666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:44 smithi144 ceph-mon[32129]: pgmap v4666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7789.705079 secs 2024-07-26T13:15:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7789.705079 secs 2024-07-26T13:15:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7789.705079 secs 2024-07-26T13:15:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:46 smithi183 ceph-mon[37273]: pgmap v4667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:47.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:46 smithi160 ceph-mon[36715]: pgmap v4667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:46 smithi144 ceph-mon[32129]: pgmap v4667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:15:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:48.786+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:48.787+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:48.787+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:48.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:15:48.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:48 smithi183 ceph-mon[37273]: pgmap v4668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:48 smithi144 ceph-mon[32129]: pgmap v4668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:48 smithi160 ceph-mon[36715]: pgmap v4668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:15:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:50 smithi183 ceph-mon[37273]: pgmap v4669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:15:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7794.705233 secs 2024-07-26T13:15:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:50 smithi144 ceph-mon[32129]: pgmap v4669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:15:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7794.705233 secs 2024-07-26T13:15:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:50 smithi160 ceph-mon[36715]: pgmap v4669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:15:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7794.705233 secs 2024-07-26T13:15:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:52 smithi183 ceph-mon[37273]: pgmap v4670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:52 smithi144 ceph-mon[32129]: pgmap v4670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:52 smithi160 ceph-mon[36715]: pgmap v4670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:15:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:54 smithi183 ceph-mon[37273]: pgmap v4671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:54 smithi144 ceph-mon[32129]: pgmap v4671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:54 smithi160 ceph-mon[36715]: pgmap v4671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:15:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7799.705397 secs 2024-07-26T13:15:56.189 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7799.705397 secs 2024-07-26T13:15:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7799.705397 secs 2024-07-26T13:15:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:56 smithi183 ceph-mon[37273]: pgmap v4672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:56 smithi144 ceph-mon[32129]: pgmap v4672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:56 smithi160 ceph-mon[36715]: pgmap v4672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:15:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:57 smithi183 ceph-mon[37273]: pgmap v4673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:57 smithi144 ceph-mon[32129]: pgmap v4673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:15:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:57 smithi160 ceph-mon[36715]: pgmap v4673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:15:59 smithi183 ceph-mon[37273]: pgmap v4674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:15:59 smithi144 ceph-mon[32129]: pgmap v4674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:15:59 smithi160 ceph-mon[36715]: pgmap v4674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7804.705528 secs 2024-07-26T13:16:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7804.705528 secs 2024-07-26T13:16:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7804.705528 secs 2024-07-26T13:16:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:01 smithi144 ceph-mon[32129]: pgmap v4675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:01 smithi160 ceph-mon[36715]: pgmap v4675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:01 smithi183 ceph-mon[37273]: pgmap v4675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:03.791+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:03.792+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:03.792+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:03.792+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:03.792+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:03 smithi144 ceph-mon[32129]: pgmap v4676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:03 smithi160 ceph-mon[36715]: pgmap v4676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:03 smithi183 ceph-mon[37273]: pgmap v4676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:05 smithi144 ceph-mon[32129]: pgmap v4677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7809.705658 secs 2024-07-26T13:16:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:05 smithi160 ceph-mon[36715]: pgmap v4677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7809.705658 secs 2024-07-26T13:16:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:05 smithi183 ceph-mon[37273]: pgmap v4677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7809.705658 secs 2024-07-26T13:16:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:07 smithi144 ceph-mon[32129]: pgmap v4678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:07 smithi160 ceph-mon[36715]: pgmap v4678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:07 smithi183 ceph-mon[37273]: pgmap v4678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:09 smithi144 ceph-mon[32129]: pgmap v4679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:09 smithi160 ceph-mon[36715]: pgmap v4679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:09 smithi183 ceph-mon[37273]: pgmap v4679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7814.705727 secs 2024-07-26T13:16:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7814.705727 secs 2024-07-26T13:16:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7814.705727 secs 2024-07-26T13:16:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:11 smithi144 ceph-mon[32129]: pgmap v4680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:11 smithi160 ceph-mon[36715]: pgmap v4680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:11 smithi183 ceph-mon[37273]: pgmap v4680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:13 smithi144 ceph-mon[32129]: pgmap v4681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:13 smithi160 ceph-mon[36715]: pgmap v4681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:13 smithi183 ceph-mon[37273]: pgmap v4681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:15 smithi144 ceph-mon[32129]: pgmap v4682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7819.705867 secs 2024-07-26T13:16:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:15 smithi160 ceph-mon[36715]: pgmap v4682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7819.705867 secs 2024-07-26T13:16:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:15 smithi183 ceph-mon[37273]: pgmap v4682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7819.705867 secs 2024-07-26T13:16:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:17 smithi144 ceph-mon[32129]: pgmap v4683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:17 smithi160 ceph-mon[36715]: pgmap v4683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:17 smithi183 ceph-mon[37273]: pgmap v4683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:18.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:18.809+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:18.810+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:18.810+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:18.810+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:19 smithi144 ceph-mon[32129]: pgmap v4684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:19 smithi160 ceph-mon[36715]: pgmap v4684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:19 smithi183 ceph-mon[37273]: pgmap v4684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7824.705994 secs 2024-07-26T13:16:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7824.705994 secs 2024-07-26T13:16:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7824.705994 secs 2024-07-26T13:16:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:21 smithi144 ceph-mon[32129]: pgmap v4685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:21 smithi160 ceph-mon[36715]: pgmap v4685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:21 smithi183 ceph-mon[37273]: pgmap v4685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:23 smithi160 ceph-mon[36715]: pgmap v4686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:23 smithi183 ceph-mon[37273]: pgmap v4686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:23 smithi144 ceph-mon[32129]: pgmap v4686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:25 smithi160 ceph-mon[36715]: pgmap v4687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7829.706117 secs 2024-07-26T13:16:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:25 smithi183 ceph-mon[37273]: pgmap v4687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7829.706117 secs 2024-07-26T13:16:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:25 smithi144 ceph-mon[32129]: pgmap v4687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7829.706117 secs 2024-07-26T13:16:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:27 smithi160 ceph-mon[36715]: pgmap v4688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:27 smithi183 ceph-mon[37273]: pgmap v4688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:27 smithi144 ceph-mon[32129]: pgmap v4688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:30 smithi160 ceph-mon[36715]: pgmap v4689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:30 smithi183 ceph-mon[37273]: pgmap v4689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:30 smithi144 ceph-mon[32129]: pgmap v4689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:31.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7834.706254 secs 2024-07-26T13:16:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7834.706254 secs 2024-07-26T13:16:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7834.706254 secs 2024-07-26T13:16:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:32 smithi160 ceph-mon[36715]: pgmap v4690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:32 smithi183 ceph-mon[37273]: pgmap v4690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:32 smithi144 ceph-mon[32129]: pgmap v4690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:33.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:33.825+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:33.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:33.827+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:33.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:33.827+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:33.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:33.827+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:33.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:33.827+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:34 smithi160 ceph-mon[36715]: pgmap v4691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:34 smithi183 ceph-mon[37273]: pgmap v4691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:34 smithi144 ceph-mon[32129]: pgmap v4691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:36 smithi160 ceph-mon[36715]: pgmap v4692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7839.706375 secs 2024-07-26T13:16:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:36 smithi183 ceph-mon[37273]: pgmap v4692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7839.706375 secs 2024-07-26T13:16:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:36 smithi144 ceph-mon[32129]: pgmap v4692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:36.555 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7839.706375 secs 2024-07-26T13:16:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:38 smithi160 ceph-mon[36715]: pgmap v4693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:38 smithi183 ceph-mon[37273]: pgmap v4693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:38 smithi144 ceph-mon[32129]: pgmap v4693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:16:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:40 smithi160 ceph-mon[36715]: pgmap v4694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:40 smithi183 ceph-mon[37273]: pgmap v4694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:40 smithi144 ceph-mon[32129]: pgmap v4694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7844.706471 secs 2024-07-26T13:16:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7844.706471 secs 2024-07-26T13:16:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7844.706471 secs 2024-07-26T13:16:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:42 smithi160 ceph-mon[36715]: pgmap v4695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:42 smithi183 ceph-mon[37273]: pgmap v4695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:42 smithi144 ceph-mon[32129]: pgmap v4695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:43 smithi183 ceph-mon[37273]: pgmap v4696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:16:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:43 smithi183 ceph-mon[37273]: pgmap v4697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:16:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:16:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:43 smithi144 ceph-mon[32129]: pgmap v4696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:16:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:43 smithi144 ceph-mon[32129]: pgmap v4697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:16:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:16:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:43 smithi160 ceph-mon[36715]: pgmap v4696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:16:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:43 smithi160 ceph-mon[36715]: pgmap v4697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:16:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:16:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:45 smithi183 ceph-mon[37273]: pgmap v4698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:16:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:45 smithi144 ceph-mon[32129]: pgmap v4698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:16:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:45 smithi160 ceph-mon[36715]: pgmap v4698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:16:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7849.706596 secs 2024-07-26T13:16:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7849.706596 secs 2024-07-26T13:16:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7849.706596 secs 2024-07-26T13:16:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:47 smithi144 ceph-mon[32129]: pgmap v4699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:16:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:47 smithi160 ceph-mon[36715]: pgmap v4699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:16:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:47 smithi183 ceph-mon[37273]: pgmap v4699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:16:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:48.842+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:48.843+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:48.843+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:48.843+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:16:48.844+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:49 smithi144 ceph-mon[32129]: pgmap v4700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:16:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:49 smithi160 ceph-mon[36715]: pgmap v4700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:16:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:49 smithi183 ceph-mon[37273]: pgmap v4700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:16:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:51 smithi144 ceph-mon[32129]: pgmap v4701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:16:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7854.706727 secs 2024-07-26T13:16:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:51 smithi160 ceph-mon[36715]: pgmap v4701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:16:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7854.706727 secs 2024-07-26T13:16:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:51 smithi183 ceph-mon[37273]: pgmap v4701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:16:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7854.706727 secs 2024-07-26T13:16:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:53 smithi144 ceph-mon[32129]: pgmap v4702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:16:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:53 smithi160 ceph-mon[36715]: pgmap v4702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:16:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:53 smithi183 ceph-mon[37273]: pgmap v4702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:16:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:55 smithi144 ceph-mon[32129]: pgmap v4703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:55 smithi160 ceph-mon[36715]: pgmap v4703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:55 smithi183 ceph-mon[37273]: pgmap v4703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7859.706875 secs 2024-07-26T13:16:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7859.706875 secs 2024-07-26T13:16:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7859.706875 secs 2024-07-26T13:16:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:57 smithi144 ceph-mon[32129]: pgmap v4704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:57 smithi160 ceph-mon[36715]: pgmap v4704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:57 smithi183 ceph-mon[37273]: pgmap v4704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:16:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:16:59 smithi144 ceph-mon[32129]: pgmap v4705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:16:59 smithi160 ceph-mon[36715]: pgmap v4705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:16:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:16:59 smithi183 ceph-mon[37273]: pgmap v4705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:01.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:01 smithi183 ceph-mon[37273]: pgmap v4706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:01.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7864.706922 secs 2024-07-26T13:17:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:01 smithi144 ceph-mon[32129]: pgmap v4706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7864.706922 secs 2024-07-26T13:17:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:01 smithi160 ceph-mon[36715]: pgmap v4706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7864.706922 secs 2024-07-26T13:17:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:03 smithi144 ceph-mon[32129]: pgmap v4707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:03 smithi160 ceph-mon[36715]: pgmap v4707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:03 smithi183 ceph-mon[37273]: pgmap v4707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:03.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:03.859+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:03.859+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:03.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:03.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:05 smithi144 ceph-mon[32129]: pgmap v4708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:05 smithi160 ceph-mon[36715]: pgmap v4708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:05 smithi183 ceph-mon[37273]: pgmap v4708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7869.707021 secs 2024-07-26T13:17:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7869.707021 secs 2024-07-26T13:17:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7869.707021 secs 2024-07-26T13:17:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:07 smithi144 ceph-mon[32129]: pgmap v4709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:07 smithi160 ceph-mon[36715]: pgmap v4709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:07 smithi183 ceph-mon[37273]: pgmap v4709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:09 smithi160 ceph-mon[36715]: pgmap v4710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:09 smithi183 ceph-mon[37273]: pgmap v4710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:09 smithi144 ceph-mon[32129]: pgmap v4710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:11 smithi160 ceph-mon[36715]: pgmap v4711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7874.707158 secs 2024-07-26T13:17:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:11 smithi183 ceph-mon[37273]: pgmap v4711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7874.707158 secs 2024-07-26T13:17:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:11 smithi144 ceph-mon[32129]: pgmap v4711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7874.707158 secs 2024-07-26T13:17:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:13 smithi160 ceph-mon[36715]: pgmap v4712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:13 smithi183 ceph-mon[37273]: pgmap v4712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:13 smithi144 ceph-mon[32129]: pgmap v4712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:15 smithi160 ceph-mon[36715]: pgmap v4713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:15 smithi183 ceph-mon[37273]: pgmap v4713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:15 smithi144 ceph-mon[32129]: pgmap v4713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7879.707282 secs 2024-07-26T13:17:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7879.707282 secs 2024-07-26T13:17:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7879.707282 secs 2024-07-26T13:17:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:17 smithi160 ceph-mon[36715]: pgmap v4714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:17 smithi183 ceph-mon[37273]: pgmap v4714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:17 smithi144 ceph-mon[32129]: pgmap v4714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:18.876+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:18.877+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:18.877+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:18.877+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:18.877+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:19 smithi160 ceph-mon[36715]: pgmap v4715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:19 smithi183 ceph-mon[37273]: pgmap v4715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:19 smithi144 ceph-mon[32129]: pgmap v4715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:21 smithi160 ceph-mon[36715]: pgmap v4716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7884.707387 secs 2024-07-26T13:17:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:21 smithi183 ceph-mon[37273]: pgmap v4716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7884.707387 secs 2024-07-26T13:17:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:21 smithi144 ceph-mon[32129]: pgmap v4716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7884.707387 secs 2024-07-26T13:17:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:23 smithi160 ceph-mon[36715]: pgmap v4717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:23 smithi183 ceph-mon[37273]: pgmap v4717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:23 smithi144 ceph-mon[32129]: pgmap v4717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:25 smithi160 ceph-mon[36715]: pgmap v4718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:25 smithi183 ceph-mon[37273]: pgmap v4718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:25 smithi144 ceph-mon[32129]: pgmap v4718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7889.707517 secs 2024-07-26T13:17:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7889.707517 secs 2024-07-26T13:17:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7889.707517 secs 2024-07-26T13:17:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:27 smithi160 ceph-mon[36715]: pgmap v4719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:27 smithi183 ceph-mon[37273]: pgmap v4719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:27 smithi144 ceph-mon[32129]: pgmap v4719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:29 smithi160 ceph-mon[36715]: pgmap v4720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:29 smithi183 ceph-mon[37273]: pgmap v4720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:29 smithi144 ceph-mon[32129]: pgmap v4720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:31.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:31 smithi160 ceph-mon[36715]: pgmap v4721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:31.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7894.707640 secs 2024-07-26T13:17:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:31 smithi183 ceph-mon[37273]: pgmap v4721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7894.707640 secs 2024-07-26T13:17:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:31 smithi144 ceph-mon[32129]: pgmap v4721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7894.707640 secs 2024-07-26T13:17:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:33 smithi160 ceph-mon[36715]: pgmap v4722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:33 smithi183 ceph-mon[37273]: pgmap v4722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:33 smithi144 ceph-mon[32129]: pgmap v4722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:33.885+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:33.886+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:33.886+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:33.886+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:33.886+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:35 smithi183 ceph-mon[37273]: pgmap v4723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:35 smithi144 ceph-mon[32129]: pgmap v4723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:35 smithi160 ceph-mon[36715]: pgmap v4723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7899.707774 secs 2024-07-26T13:17:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7899.707774 secs 2024-07-26T13:17:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7899.707774 secs 2024-07-26T13:17:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:37 smithi183 ceph-mon[37273]: pgmap v4724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:37 smithi144 ceph-mon[32129]: pgmap v4724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:37 smithi160 ceph-mon[36715]: pgmap v4724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:39 smithi183 ceph-mon[37273]: pgmap v4725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:39 smithi144 ceph-mon[32129]: pgmap v4725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:39 smithi160 ceph-mon[36715]: pgmap v4725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:41 smithi183 ceph-mon[37273]: pgmap v4726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7904.707904 secs 2024-07-26T13:17:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:41 smithi144 ceph-mon[32129]: pgmap v4726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7904.707904 secs 2024-07-26T13:17:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:41 smithi160 ceph-mon[36715]: pgmap v4726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7904.707904 secs 2024-07-26T13:17:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:43 smithi183 ceph-mon[37273]: pgmap v4727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:17:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:43 smithi144 ceph-mon[32129]: pgmap v4727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:17:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:43 smithi160 ceph-mon[36715]: pgmap v4727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:17:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:17:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:44 smithi183 ceph-mon[37273]: pgmap v4728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:44 smithi183 ceph-mon[37273]: pgmap v4729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:17:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:44 smithi144 ceph-mon[32129]: pgmap v4728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:44 smithi144 ceph-mon[32129]: pgmap v4729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:17:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:44 smithi160 ceph-mon[36715]: pgmap v4728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:44 smithi160 ceph-mon[36715]: pgmap v4729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:17:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:46 smithi183 ceph-mon[37273]: pgmap v4730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7909.708035 secs 2024-07-26T13:17:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:46 smithi144 ceph-mon[32129]: pgmap v4730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7909.708035 secs 2024-07-26T13:17:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:46 smithi160 ceph-mon[36715]: pgmap v4730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7909.708035 secs 2024-07-26T13:17:47.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.500+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:17:47.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.500+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:17:47.501 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.500+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:17:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.500+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:17:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.501+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:17:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.501+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:17:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.501+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:17:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:47.501+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:17:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:48 smithi183 ceph-mon[37273]: pgmap v4731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:48 smithi144 ceph-mon[32129]: pgmap v4731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:48 smithi160 ceph-mon[36715]: pgmap v4731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:48.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:48.893+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:48.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:48.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:48.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:48.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:48.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:48.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:48.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:17:48.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:50 smithi183 ceph-mon[37273]: pgmap v4732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:50 smithi144 ceph-mon[32129]: pgmap v4732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:50 smithi160 ceph-mon[36715]: pgmap v4732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7914.708175 secs 2024-07-26T13:17:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7914.708175 secs 2024-07-26T13:17:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7914.708175 secs 2024-07-26T13:17:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:52 smithi144 ceph-mon[32129]: pgmap v4733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:52 smithi160 ceph-mon[36715]: pgmap v4733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:52 smithi183 ceph-mon[37273]: pgmap v4733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:17:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:54 smithi144 ceph-mon[32129]: pgmap v4734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:17:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:54 smithi160 ceph-mon[36715]: pgmap v4734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:17:54.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:54 smithi183 ceph-mon[37273]: pgmap v4734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:17:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:56 smithi144 ceph-mon[32129]: pgmap v4735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7919.708342 secs 2024-07-26T13:17:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:56 smithi160 ceph-mon[36715]: pgmap v4735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7919.708342 secs 2024-07-26T13:17:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:56 smithi183 ceph-mon[37273]: pgmap v4735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:17:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7919.708342 secs 2024-07-26T13:17:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:17:58 smithi144 ceph-mon[32129]: pgmap v4736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:17:58 smithi160 ceph-mon[36715]: pgmap v4736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:17:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:17:58 smithi183 ceph-mon[37273]: pgmap v4736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:00 smithi144 ceph-mon[32129]: pgmap v4737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:00 smithi160 ceph-mon[36715]: pgmap v4737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:00 smithi183 ceph-mon[37273]: pgmap v4737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7924.708391 secs 2024-07-26T13:18:01.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7924.708391 secs 2024-07-26T13:18:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7924.708391 secs 2024-07-26T13:18:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:02 smithi144 ceph-mon[32129]: pgmap v4738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:02 smithi160 ceph-mon[36715]: pgmap v4738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:02 smithi183 ceph-mon[37273]: pgmap v4738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:03.909+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:03.910+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:03.910+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:03.910+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:03.910+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:04 smithi144 ceph-mon[32129]: pgmap v4739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:04 smithi160 ceph-mon[36715]: pgmap v4739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:04 smithi183 ceph-mon[37273]: pgmap v4739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:06 smithi144 ceph-mon[32129]: pgmap v4740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7929.708460 secs 2024-07-26T13:18:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:06 smithi160 ceph-mon[36715]: pgmap v4740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7929.708460 secs 2024-07-26T13:18:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:06 smithi183 ceph-mon[37273]: pgmap v4740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7929.708460 secs 2024-07-26T13:18:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:08 smithi144 ceph-mon[32129]: pgmap v4741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:08 smithi160 ceph-mon[36715]: pgmap v4741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:08 smithi183 ceph-mon[37273]: pgmap v4741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:10 smithi144 ceph-mon[32129]: pgmap v4742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:10 smithi160 ceph-mon[36715]: pgmap v4742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:10 smithi183 ceph-mon[37273]: pgmap v4742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7934.708585 secs 2024-07-26T13:18:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7934.708585 secs 2024-07-26T13:18:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7934.708585 secs 2024-07-26T13:18:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:12 smithi144 ceph-mon[32129]: pgmap v4743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:12 smithi160 ceph-mon[36715]: pgmap v4743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:12 smithi183 ceph-mon[37273]: pgmap v4743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:14 smithi144 ceph-mon[32129]: pgmap v4744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:14 smithi160 ceph-mon[36715]: pgmap v4744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:14 smithi183 ceph-mon[37273]: pgmap v4744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:16 smithi144 ceph-mon[32129]: pgmap v4745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7939.708653 secs 2024-07-26T13:18:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:16 smithi160 ceph-mon[36715]: pgmap v4745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7939.708653 secs 2024-07-26T13:18:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:16 smithi183 ceph-mon[37273]: pgmap v4745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7939.708653 secs 2024-07-26T13:18:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:18 smithi160 ceph-mon[36715]: pgmap v4746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:18 smithi183 ceph-mon[37273]: pgmap v4746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:18.917+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:18.918+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:18.918+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:18.918+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:18.918+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:18 smithi144 ceph-mon[32129]: pgmap v4746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:20 smithi160 ceph-mon[36715]: pgmap v4747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:20 smithi183 ceph-mon[37273]: pgmap v4747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:20 smithi144 ceph-mon[32129]: pgmap v4747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7944.708783 secs 2024-07-26T13:18:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7944.708783 secs 2024-07-26T13:18:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7944.708783 secs 2024-07-26T13:18:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:22 smithi160 ceph-mon[36715]: pgmap v4748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:22 smithi183 ceph-mon[37273]: pgmap v4748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:22 smithi144 ceph-mon[32129]: pgmap v4748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:24 smithi160 ceph-mon[36715]: pgmap v4749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:24 smithi183 ceph-mon[37273]: pgmap v4749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:24 smithi144 ceph-mon[32129]: pgmap v4749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:26 smithi160 ceph-mon[36715]: pgmap v4750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7949.708879 secs 2024-07-26T13:18:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:26 smithi183 ceph-mon[37273]: pgmap v4750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7949.708879 secs 2024-07-26T13:18:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:26 smithi144 ceph-mon[32129]: pgmap v4750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7949.708879 secs 2024-07-26T13:18:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:28 smithi160 ceph-mon[36715]: pgmap v4751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:28 smithi183 ceph-mon[37273]: pgmap v4751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:28 smithi144 ceph-mon[32129]: pgmap v4751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:30 smithi160 ceph-mon[36715]: pgmap v4752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:30.814 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:30 smithi144 ceph-mon[32129]: pgmap v4752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:30 smithi183 ceph-mon[37273]: pgmap v4752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:31.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7954.709006 secs 2024-07-26T13:18:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7954.709006 secs 2024-07-26T13:18:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7954.709006 secs 2024-07-26T13:18:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:32 smithi160 ceph-mon[36715]: pgmap v4753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:32 smithi183 ceph-mon[37273]: pgmap v4753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:32 smithi144 ceph-mon[32129]: pgmap v4753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:33.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:33.920+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:33.920+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:33.920+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:33.920+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:34 smithi160 ceph-mon[36715]: pgmap v4754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:34 smithi183 ceph-mon[37273]: pgmap v4754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:34 smithi144 ceph-mon[32129]: pgmap v4754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:36 smithi160 ceph-mon[36715]: pgmap v4755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7959.709122 secs 2024-07-26T13:18:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:36 smithi183 ceph-mon[37273]: pgmap v4755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7959.709122 secs 2024-07-26T13:18:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:36 smithi144 ceph-mon[32129]: pgmap v4755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7959.709122 secs 2024-07-26T13:18:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:38 smithi160 ceph-mon[36715]: pgmap v4756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:38 smithi183 ceph-mon[37273]: pgmap v4756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:38 smithi144 ceph-mon[32129]: pgmap v4756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:18:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:40 smithi160 ceph-mon[36715]: pgmap v4757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:40 smithi183 ceph-mon[37273]: pgmap v4757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:40 smithi144 ceph-mon[32129]: pgmap v4757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7964.709253 secs 2024-07-26T13:18:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7964.709253 secs 2024-07-26T13:18:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7964.709253 secs 2024-07-26T13:18:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:42 smithi183 ceph-mon[37273]: pgmap v4758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:42 smithi144 ceph-mon[32129]: pgmap v4758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:42 smithi160 ceph-mon[36715]: pgmap v4758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:44 smithi183 ceph-mon[37273]: pgmap v4759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:18:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:44 smithi144 ceph-mon[32129]: pgmap v4759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:18:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:44 smithi160 ceph-mon[36715]: pgmap v4759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:18:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:18:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:45 smithi183 ceph-mon[37273]: pgmap v4760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:18:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:45 smithi183 ceph-mon[37273]: pgmap v4761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:18:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:45 smithi144 ceph-mon[32129]: pgmap v4760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:18:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:45 smithi144 ceph-mon[32129]: pgmap v4761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:18:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:45 smithi160 ceph-mon[36715]: pgmap v4760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:18:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:45 smithi160 ceph-mon[36715]: pgmap v4761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:18:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7969.709365 secs 2024-07-26T13:18:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7969.709365 secs 2024-07-26T13:18:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7969.709365 secs 2024-07-26T13:18:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:47 smithi183 ceph-mon[37273]: pgmap v4762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:18:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:47 smithi144 ceph-mon[32129]: pgmap v4762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:18:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:47 smithi160 ceph-mon[36715]: pgmap v4762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:18:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:48.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:48.934+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:48.934+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:48.934+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:18:48.934+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:49 smithi183 ceph-mon[37273]: pgmap v4763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:18:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:49 smithi144 ceph-mon[32129]: pgmap v4763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:18:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:49 smithi160 ceph-mon[36715]: pgmap v4763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:18:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:51 smithi183 ceph-mon[37273]: pgmap v4764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:18:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7974.709494 secs 2024-07-26T13:18:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:51 smithi144 ceph-mon[32129]: pgmap v4764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:18:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7974.709494 secs 2024-07-26T13:18:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:51 smithi160 ceph-mon[36715]: pgmap v4764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:18:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7974.709494 secs 2024-07-26T13:18:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:53 smithi183 ceph-mon[37273]: pgmap v4765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:18:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:53 smithi144 ceph-mon[32129]: pgmap v4765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:18:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:53 smithi160 ceph-mon[36715]: pgmap v4765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:18:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:55 smithi183 ceph-mon[37273]: pgmap v4766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:18:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:55 smithi144 ceph-mon[32129]: pgmap v4766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:18:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:55 smithi160 ceph-mon[36715]: pgmap v4766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:18:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7979.709623 secs 2024-07-26T13:18:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7979.709623 secs 2024-07-26T13:18:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7979.709623 secs 2024-07-26T13:18:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:57 smithi144 ceph-mon[32129]: pgmap v4767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:57 smithi160 ceph-mon[36715]: pgmap v4767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:57 smithi183 ceph-mon[37273]: pgmap v4767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:18:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:18:59 smithi144 ceph-mon[32129]: pgmap v4768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:18:59 smithi160 ceph-mon[36715]: pgmap v4768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:18:59 smithi183 ceph-mon[37273]: pgmap v4768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:01.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:01 smithi183 ceph-mon[37273]: pgmap v4769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:01.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7984.709744 secs 2024-07-26T13:19:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:01 smithi144 ceph-mon[32129]: pgmap v4769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7984.709744 secs 2024-07-26T13:19:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:01 smithi160 ceph-mon[36715]: pgmap v4769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7984.709744 secs 2024-07-26T13:19:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:03.948+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:03.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:03.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:03.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:03.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:03 smithi144 ceph-mon[32129]: pgmap v4770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:03 smithi160 ceph-mon[36715]: pgmap v4770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:03 smithi183 ceph-mon[37273]: pgmap v4770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:05 smithi144 ceph-mon[32129]: pgmap v4771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:05 smithi160 ceph-mon[36715]: pgmap v4771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:05 smithi183 ceph-mon[37273]: pgmap v4771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7989.709869 secs 2024-07-26T13:19:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7989.709869 secs 2024-07-26T13:19:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7989.709869 secs 2024-07-26T13:19:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:07 smithi144 ceph-mon[32129]: pgmap v4772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:07 smithi160 ceph-mon[36715]: pgmap v4772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:07 smithi183 ceph-mon[37273]: pgmap v4772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:09 smithi144 ceph-mon[32129]: pgmap v4773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:09 smithi160 ceph-mon[36715]: pgmap v4773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:09 smithi183 ceph-mon[37273]: pgmap v4773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:11 smithi144 ceph-mon[32129]: pgmap v4774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7994.709992 secs 2024-07-26T13:19:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:11 smithi160 ceph-mon[36715]: pgmap v4774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7994.709992 secs 2024-07-26T13:19:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:11 smithi183 ceph-mon[37273]: pgmap v4774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7994.709992 secs 2024-07-26T13:19:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:13 smithi144 ceph-mon[32129]: pgmap v4775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:13 smithi160 ceph-mon[36715]: pgmap v4775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:13 smithi183 ceph-mon[37273]: pgmap v4775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:15 smithi144 ceph-mon[32129]: pgmap v4776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:15 smithi160 ceph-mon[36715]: pgmap v4776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:15 smithi183 ceph-mon[37273]: pgmap v4776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 7999.710132 secs 2024-07-26T13:19:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 7999.710132 secs 2024-07-26T13:19:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 7999.710132 secs 2024-07-26T13:19:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:17 smithi160 ceph-mon[36715]: pgmap v4777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:17 smithi183 ceph-mon[37273]: pgmap v4777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:17 smithi144 ceph-mon[32129]: pgmap v4777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:18.957+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:18.958+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:18.958+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:18.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:18.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:19 smithi160 ceph-mon[36715]: pgmap v4778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:19 smithi183 ceph-mon[37273]: pgmap v4778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:19 smithi144 ceph-mon[32129]: pgmap v4778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:21 smithi160 ceph-mon[36715]: pgmap v4779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8004.710269 secs 2024-07-26T13:19:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:21 smithi183 ceph-mon[37273]: pgmap v4779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8004.710269 secs 2024-07-26T13:19:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:21 smithi144 ceph-mon[32129]: pgmap v4779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8004.710269 secs 2024-07-26T13:19:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:23 smithi160 ceph-mon[36715]: pgmap v4780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:23 smithi183 ceph-mon[37273]: pgmap v4780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:23 smithi144 ceph-mon[32129]: pgmap v4780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:25 smithi144 ceph-mon[32129]: pgmap v4781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:25 smithi160 ceph-mon[36715]: pgmap v4781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:25 smithi183 ceph-mon[37273]: pgmap v4781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8009.710407 secs 2024-07-26T13:19:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8009.710407 secs 2024-07-26T13:19:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8009.710407 secs 2024-07-26T13:19:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:27 smithi144 ceph-mon[32129]: pgmap v4782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:27 smithi160 ceph-mon[36715]: pgmap v4782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:27 smithi183 ceph-mon[37273]: pgmap v4782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:29 smithi183 ceph-mon[37273]: pgmap v4783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:29 smithi144 ceph-mon[32129]: pgmap v4783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:29 smithi160 ceph-mon[36715]: pgmap v4783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:31 smithi144 ceph-mon[32129]: pgmap v4784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8014.710502 secs 2024-07-26T13:19:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:31 smithi160 ceph-mon[36715]: pgmap v4784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8014.710502 secs 2024-07-26T13:19:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:31 smithi183 ceph-mon[37273]: pgmap v4784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8014.710502 secs 2024-07-26T13:19:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:33 smithi144 ceph-mon[32129]: pgmap v4785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:33.972+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:33.973+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:33.973+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:33.973+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:33.973+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:33 smithi160 ceph-mon[36715]: pgmap v4785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:33 smithi183 ceph-mon[37273]: pgmap v4785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:35 smithi144 ceph-mon[32129]: pgmap v4786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:35 smithi160 ceph-mon[36715]: pgmap v4786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:35 smithi183 ceph-mon[37273]: pgmap v4786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8019.710611 secs 2024-07-26T13:19:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8019.710611 secs 2024-07-26T13:19:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8019.710611 secs 2024-07-26T13:19:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:37 smithi144 ceph-mon[32129]: pgmap v4787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:37 smithi160 ceph-mon[36715]: pgmap v4787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:37 smithi183 ceph-mon[37273]: pgmap v4787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:19:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:39 smithi144 ceph-mon[32129]: pgmap v4788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:39 smithi160 ceph-mon[36715]: pgmap v4788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:39 smithi183 ceph-mon[37273]: pgmap v4788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:41 smithi144 ceph-mon[32129]: pgmap v4789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8024.710734 secs 2024-07-26T13:19:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:41 smithi160 ceph-mon[36715]: pgmap v4789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8024.710734 secs 2024-07-26T13:19:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:41 smithi183 ceph-mon[37273]: pgmap v4789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8024.710734 secs 2024-07-26T13:19:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:43 smithi144 ceph-mon[32129]: pgmap v4790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:43 smithi160 ceph-mon[36715]: pgmap v4790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:43 smithi183 ceph-mon[37273]: pgmap v4790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:45 smithi144 ceph-mon[32129]: pgmap v4791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:45 smithi160 ceph-mon[36715]: pgmap v4791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:45 smithi183 ceph-mon[37273]: pgmap v4791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:19:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8029.710832 secs 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:19:46.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:19:46.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:46.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:46.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: pgmap v4792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:46.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: pgmap v4793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:46.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8029.710832 secs 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:19:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:47.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:47.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: pgmap v4792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:47.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: pgmap v4793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:47.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:19:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8029.710832 secs 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: pgmap v4792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:47.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: pgmap v4793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:47.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:19:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:48.978+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:48.979+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:48.979+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:48.979+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:19:48.979+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:48 smithi183 ceph-mon[37273]: pgmap v4794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:19:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:48 smithi144 ceph-mon[32129]: pgmap v4794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:19:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:48 smithi160 ceph-mon[36715]: pgmap v4794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:19:51.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:50 smithi160 ceph-mon[36715]: pgmap v4795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:19:51.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8034.710955 secs 2024-07-26T13:19:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:50 smithi183 ceph-mon[37273]: pgmap v4795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:19:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8034.710955 secs 2024-07-26T13:19:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:50 smithi144 ceph-mon[32129]: pgmap v4795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:19:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8034.710955 secs 2024-07-26T13:19:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:52 smithi183 ceph-mon[37273]: pgmap v4796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:52 smithi144 ceph-mon[32129]: pgmap v4796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:52 smithi160 ceph-mon[36715]: pgmap v4796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:19:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:54 smithi183 ceph-mon[37273]: pgmap v4797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:19:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:54 smithi144 ceph-mon[32129]: pgmap v4797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:19:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:54 smithi160 ceph-mon[36715]: pgmap v4797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:19:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8039.711074 secs 2024-07-26T13:19:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8039.711074 secs 2024-07-26T13:19:56.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8039.711074 secs 2024-07-26T13:19:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:56 smithi183 ceph-mon[37273]: pgmap v4798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:19:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:56 smithi144 ceph-mon[32129]: pgmap v4798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:19:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:56 smithi160 ceph-mon[36715]: pgmap v4798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:19:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:19:58 smithi144 ceph-mon[32129]: pgmap v4799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:19:58 smithi160 ceph-mon[36715]: pgmap v4799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:19:59.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:19:58 smithi183 ceph-mon[37273]: pgmap v4799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:00 smithi144 ceph-mon[32129]: pgmap v4800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:00 smithi160 ceph-mon[36715]: pgmap v4800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:00 smithi183 ceph-mon[37273]: pgmap v4800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8044.711204 secs 2024-07-26T13:20:01.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8044.711204 secs 2024-07-26T13:20:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8044.711204 secs 2024-07-26T13:20:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:02 smithi144 ceph-mon[32129]: pgmap v4801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:02 smithi160 ceph-mon[36715]: pgmap v4801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:02 smithi183 ceph-mon[37273]: pgmap v4801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:03.994+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:03.996 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:03.994+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:03.996 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:03.994+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:03.997 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:03.994+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:03.997 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:03.996+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:04 smithi183 ceph-mon[37273]: pgmap v4802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:04 smithi144 ceph-mon[32129]: pgmap v4802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:04 smithi160 ceph-mon[36715]: pgmap v4802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8049.711325 secs 2024-07-26T13:20:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8049.711325 secs 2024-07-26T13:20:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8049.711325 secs 2024-07-26T13:20:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:06 smithi183 ceph-mon[37273]: pgmap v4803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:06 smithi144 ceph-mon[32129]: pgmap v4803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:06 smithi160 ceph-mon[36715]: pgmap v4803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:08 smithi183 ceph-mon[37273]: pgmap v4804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:08 smithi144 ceph-mon[32129]: pgmap v4804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:08 smithi160 ceph-mon[36715]: pgmap v4804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:10 smithi144 ceph-mon[32129]: pgmap v4805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8054.711450 secs 2024-07-26T13:20:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:10 smithi160 ceph-mon[36715]: pgmap v4805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8054.711450 secs 2024-07-26T13:20:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:10 smithi183 ceph-mon[37273]: pgmap v4805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8054.711450 secs 2024-07-26T13:20:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:12 smithi144 ceph-mon[32129]: pgmap v4806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:12 smithi160 ceph-mon[36715]: pgmap v4806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:12 smithi183 ceph-mon[37273]: pgmap v4806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:14 smithi144 ceph-mon[32129]: pgmap v4807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:14 smithi160 ceph-mon[36715]: pgmap v4807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:14 smithi183 ceph-mon[37273]: pgmap v4807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8059.711573 secs 2024-07-26T13:20:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8059.711573 secs 2024-07-26T13:20:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8059.711573 secs 2024-07-26T13:20:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:16 smithi144 ceph-mon[32129]: pgmap v4808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:16 smithi160 ceph-mon[36715]: pgmap v4808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:16 smithi183 ceph-mon[37273]: pgmap v4808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:17 smithi144 ceph-mon[32129]: pgmap v4809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:17 smithi160 ceph-mon[36715]: pgmap v4809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:17 smithi183 ceph-mon[37273]: pgmap v4809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:19.012+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:19.013+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:19.013+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:19.013+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:19.013+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:20 smithi183 ceph-mon[37273]: pgmap v4810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8064.711690 secs 2024-07-26T13:20:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:20 smithi144 ceph-mon[32129]: pgmap v4810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8064.711690 secs 2024-07-26T13:20:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:20 smithi160 ceph-mon[36715]: pgmap v4810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8064.711690 secs 2024-07-26T13:20:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:22 smithi183 ceph-mon[37273]: pgmap v4811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:22 smithi144 ceph-mon[32129]: pgmap v4811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:22 smithi160 ceph-mon[36715]: pgmap v4811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:24 smithi183 ceph-mon[37273]: pgmap v4812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:24 smithi144 ceph-mon[32129]: pgmap v4812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:24 smithi160 ceph-mon[36715]: pgmap v4812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8069.711836 secs 2024-07-26T13:20:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8069.711836 secs 2024-07-26T13:20:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8069.711836 secs 2024-07-26T13:20:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:26 smithi144 ceph-mon[32129]: pgmap v4813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:26 smithi160 ceph-mon[36715]: pgmap v4813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:26 smithi183 ceph-mon[37273]: pgmap v4813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:28 smithi144 ceph-mon[32129]: pgmap v4814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:28 smithi160 ceph-mon[36715]: pgmap v4814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:28 smithi183 ceph-mon[37273]: pgmap v4814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:31.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:30 smithi183 ceph-mon[37273]: pgmap v4815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:31.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8074.711953 secs 2024-07-26T13:20:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:30 smithi144 ceph-mon[32129]: pgmap v4815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8074.711953 secs 2024-07-26T13:20:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:30 smithi160 ceph-mon[36715]: pgmap v4815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8074.711953 secs 2024-07-26T13:20:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:32 smithi144 ceph-mon[32129]: pgmap v4816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:32 smithi160 ceph-mon[36715]: pgmap v4816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:32 smithi183 ceph-mon[37273]: pgmap v4816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:34.021+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:34.022+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:34.022+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:34.022+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:34.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:34.023+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:34 smithi144 ceph-mon[32129]: pgmap v4817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:34 smithi160 ceph-mon[36715]: pgmap v4817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:34 smithi183 ceph-mon[37273]: pgmap v4817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8079.712090 secs 2024-07-26T13:20:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:35 smithi144 ceph-mon[32129]: pgmap v4818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8079.712090 secs 2024-07-26T13:20:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:35 smithi160 ceph-mon[36715]: pgmap v4818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8079.712090 secs 2024-07-26T13:20:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:35 smithi183 ceph-mon[37273]: pgmap v4818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:38 smithi183 ceph-mon[37273]: pgmap v4819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:38 smithi144 ceph-mon[32129]: pgmap v4819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:38 smithi160 ceph-mon[36715]: pgmap v4819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:20:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:40 smithi144 ceph-mon[32129]: pgmap v4820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8084.712247 secs 2024-07-26T13:20:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:40 smithi160 ceph-mon[36715]: pgmap v4820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8084.712247 secs 2024-07-26T13:20:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:40 smithi183 ceph-mon[37273]: pgmap v4820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8084.712247 secs 2024-07-26T13:20:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:42 smithi144 ceph-mon[32129]: pgmap v4821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:42 smithi160 ceph-mon[36715]: pgmap v4821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:42 smithi183 ceph-mon[37273]: pgmap v4821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:20:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:44 smithi144 ceph-mon[32129]: pgmap v4822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:44 smithi160 ceph-mon[36715]: pgmap v4822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:44 smithi183 ceph-mon[37273]: pgmap v4822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:20:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a804c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a804c040 comp rx=0 tx=0).stop 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 msgr2=0x7f74900b49b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b45a0 0x7f74900b49b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.120+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 0x7f74900b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.121+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 0x7f74900b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:46.122+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 0x7f74900b4ef0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a804c040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:20:46.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8089.712362 secs 2024-07-26T13:20:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8089.712362 secs 2024-07-26T13:20:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8089.712362 secs 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: pgmap v4823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: pgmap v4823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: pgmap v4823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.568 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:47.532+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:20:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:47.532+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:20:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:47.533+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:20:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:47.533+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:20:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:20:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:20:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:20:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:20:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:20:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:20:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:20:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:20:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:20:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:20:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:20:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:49.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:49.031+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:49.031+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:49.031+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:20:49.031+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:48 smithi144 ceph-mon[32129]: pgmap v4824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:20:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:48 smithi144 ceph-mon[32129]: pgmap v4825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:20:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:48 smithi160 ceph-mon[36715]: pgmap v4824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:20:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:48 smithi160 ceph-mon[36715]: pgmap v4825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:20:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:48 smithi183 ceph-mon[37273]: pgmap v4824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:20:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:48 smithi183 ceph-mon[37273]: pgmap v4825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:20:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:49 smithi144 ceph-mon[32129]: pgmap v4826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:20:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:49 smithi160 ceph-mon[36715]: pgmap v4826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:20:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:49 smithi183 ceph-mon[37273]: pgmap v4826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:20:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8094.712444 secs 2024-07-26T13:20:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8094.712444 secs 2024-07-26T13:20:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8094.712444 secs 2024-07-26T13:20:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:51 smithi144 ceph-mon[32129]: pgmap v4827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:20:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:51 smithi160 ceph-mon[36715]: pgmap v4827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:20:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:51 smithi183 ceph-mon[37273]: pgmap v4827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:20:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:54 smithi160 ceph-mon[36715]: pgmap v4828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:20:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:54 smithi183 ceph-mon[37273]: pgmap v4828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:20:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:54 smithi144 ceph-mon[32129]: pgmap v4828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:20:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:56 smithi160 ceph-mon[36715]: pgmap v4829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:20:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8099.712549 secs 2024-07-26T13:20:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:56 smithi183 ceph-mon[37273]: pgmap v4829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:20:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8099.712549 secs 2024-07-26T13:20:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:56 smithi144 ceph-mon[32129]: pgmap v4829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:20:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8099.712549 secs 2024-07-26T13:20:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:20:58 smithi160 ceph-mon[36715]: pgmap v4830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:20:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:20:58 smithi183 ceph-mon[37273]: pgmap v4830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:20:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:20:58 smithi144 ceph-mon[32129]: pgmap v4830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:00 smithi183 ceph-mon[37273]: pgmap v4831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:00 smithi144 ceph-mon[32129]: pgmap v4831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:00 smithi160 ceph-mon[36715]: pgmap v4831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8104.712632 secs 2024-07-26T13:21:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8104.712632 secs 2024-07-26T13:21:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8104.712632 secs 2024-07-26T13:21:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:02 smithi183 ceph-mon[37273]: pgmap v4832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:02 smithi144 ceph-mon[32129]: pgmap v4832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:02 smithi160 ceph-mon[36715]: pgmap v4832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:04.036+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:04.037+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:04.037+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:04.037+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:04.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:04 smithi183 ceph-mon[37273]: pgmap v4833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:04 smithi144 ceph-mon[32129]: pgmap v4833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:04 smithi160 ceph-mon[36715]: pgmap v4833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:06 smithi144 ceph-mon[32129]: pgmap v4834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8109.712737 secs 2024-07-26T13:21:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:06 smithi183 ceph-mon[37273]: pgmap v4834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8109.712737 secs 2024-07-26T13:21:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:06 smithi160 ceph-mon[36715]: pgmap v4834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8109.712737 secs 2024-07-26T13:21:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:08 smithi183 ceph-mon[37273]: pgmap v4835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:08 smithi144 ceph-mon[32129]: pgmap v4835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:08 smithi160 ceph-mon[36715]: pgmap v4835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:10 smithi183 ceph-mon[37273]: pgmap v4836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:10 smithi144 ceph-mon[32129]: pgmap v4836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:10 smithi160 ceph-mon[36715]: pgmap v4836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:11.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8114.712863 secs 2024-07-26T13:21:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8114.712863 secs 2024-07-26T13:21:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8114.712863 secs 2024-07-26T13:21:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:12 smithi183 ceph-mon[37273]: pgmap v4837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:12 smithi144 ceph-mon[32129]: pgmap v4837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:12 smithi160 ceph-mon[36715]: pgmap v4837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:14 smithi183 ceph-mon[37273]: pgmap v4838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:14 smithi144 ceph-mon[32129]: pgmap v4838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:14 smithi160 ceph-mon[36715]: pgmap v4838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:16 smithi183 ceph-mon[37273]: pgmap v4839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8119.712985 secs 2024-07-26T13:21:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:16 smithi144 ceph-mon[32129]: pgmap v4839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8119.712985 secs 2024-07-26T13:21:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:16 smithi160 ceph-mon[36715]: pgmap v4839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8119.712985 secs 2024-07-26T13:21:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:18 smithi183 ceph-mon[37273]: pgmap v4840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:18 smithi144 ceph-mon[32129]: pgmap v4840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:18 smithi160 ceph-mon[36715]: pgmap v4840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:19.043+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:19.044+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:19.044+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:19.044+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:19.044+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:20 smithi183 ceph-mon[37273]: pgmap v4841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:20 smithi144 ceph-mon[32129]: pgmap v4841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:20 smithi160 ceph-mon[36715]: pgmap v4841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:21.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8124.713102 secs 2024-07-26T13:21:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8124.713102 secs 2024-07-26T13:21:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8124.713102 secs 2024-07-26T13:21:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:22 smithi144 ceph-mon[32129]: pgmap v4842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:22 smithi160 ceph-mon[36715]: pgmap v4842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:22 smithi183 ceph-mon[37273]: pgmap v4842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:24 smithi144 ceph-mon[32129]: pgmap v4843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:24 smithi160 ceph-mon[36715]: pgmap v4843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:24 smithi183 ceph-mon[37273]: pgmap v4843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:26 smithi144 ceph-mon[32129]: pgmap v4844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8129.713231 secs 2024-07-26T13:21:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:26 smithi160 ceph-mon[36715]: pgmap v4844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8129.713231 secs 2024-07-26T13:21:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:26 smithi183 ceph-mon[37273]: pgmap v4844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8129.713231 secs 2024-07-26T13:21:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:28 smithi160 ceph-mon[36715]: pgmap v4845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:28 smithi183 ceph-mon[37273]: pgmap v4845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:28 smithi144 ceph-mon[32129]: pgmap v4845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:30 smithi160 ceph-mon[36715]: pgmap v4846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:30 smithi183 ceph-mon[37273]: pgmap v4846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:30 smithi144 ceph-mon[32129]: pgmap v4846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:31.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8134.713362 secs 2024-07-26T13:21:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8134.713362 secs 2024-07-26T13:21:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8134.713362 secs 2024-07-26T13:21:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:32 smithi160 ceph-mon[36715]: pgmap v4847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:32 smithi183 ceph-mon[37273]: pgmap v4847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:32 smithi144 ceph-mon[32129]: pgmap v4847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:34.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:34.053+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:34.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:34.054+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:34.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:34.054+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:34 smithi160 ceph-mon[36715]: pgmap v4848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:21:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:34 smithi183 ceph-mon[37273]: pgmap v4848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:21:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:34 smithi144 ceph-mon[32129]: pgmap v4848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:21:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:36 smithi160 ceph-mon[36715]: pgmap v4849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:21:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8139.713489 secs 2024-07-26T13:21:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:36 smithi183 ceph-mon[37273]: pgmap v4849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:21:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8139.713489 secs 2024-07-26T13:21:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:36 smithi144 ceph-mon[32129]: pgmap v4849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:21:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8139.713489 secs 2024-07-26T13:21:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:38 smithi160 ceph-mon[36715]: pgmap v4850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:38 smithi183 ceph-mon[37273]: pgmap v4850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:38 smithi144 ceph-mon[32129]: pgmap v4850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:40 smithi160 ceph-mon[36715]: pgmap v4851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:40 smithi183 ceph-mon[37273]: pgmap v4851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:40 smithi144 ceph-mon[32129]: pgmap v4851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:21:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8144.713589 secs 2024-07-26T13:21:41.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8144.713589 secs 2024-07-26T13:21:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8144.713589 secs 2024-07-26T13:21:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:42 smithi160 ceph-mon[36715]: pgmap v4852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:42 smithi183 ceph-mon[37273]: pgmap v4852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:42 smithi144 ceph-mon[32129]: pgmap v4852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:44 smithi160 ceph-mon[36715]: pgmap v4853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:44 smithi183 ceph-mon[37273]: pgmap v4853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:44 smithi144 ceph-mon[32129]: pgmap v4853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:21:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:46 smithi160 ceph-mon[36715]: pgmap v4854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8149.713672 secs 2024-07-26T13:21:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:46 smithi183 ceph-mon[37273]: pgmap v4854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8149.713672 secs 2024-07-26T13:21:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:46 smithi144 ceph-mon[32129]: pgmap v4854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:21:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8149.713672 secs 2024-07-26T13:21:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:48 smithi183 ceph-mon[37273]: pgmap v4855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:21:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:48 smithi144 ceph-mon[32129]: pgmap v4855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:21:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:48 smithi160 ceph-mon[36715]: pgmap v4855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:21:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:21:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:49.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:49.071+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:49.071+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:49.071+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:21:49.071+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:49 smithi183 ceph-mon[37273]: pgmap v4856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:49 smithi144 ceph-mon[32129]: pgmap v4856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:49 smithi160 ceph-mon[36715]: pgmap v4856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:51.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:51 smithi160 ceph-mon[36715]: pgmap v4857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:21:51.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8154.713814 secs 2024-07-26T13:21:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:51 smithi183 ceph-mon[37273]: pgmap v4857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:21:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8154.713814 secs 2024-07-26T13:21:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:51 smithi144 ceph-mon[32129]: pgmap v4857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:21:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8154.713814 secs 2024-07-26T13:21:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:53 smithi183 ceph-mon[37273]: pgmap v4858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:21:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:53 smithi144 ceph-mon[32129]: pgmap v4858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:21:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:53 smithi160 ceph-mon[36715]: pgmap v4858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:21:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:55 smithi183 ceph-mon[37273]: pgmap v4859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:55 smithi144 ceph-mon[32129]: pgmap v4859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:55 smithi160 ceph-mon[36715]: pgmap v4859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8159.713956 secs 2024-07-26T13:21:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8159.713956 secs 2024-07-26T13:21:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8159.713956 secs 2024-07-26T13:21:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:57 smithi183 ceph-mon[37273]: pgmap v4860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:57 smithi144 ceph-mon[32129]: pgmap v4860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:57 smithi160 ceph-mon[36715]: pgmap v4860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:21:59 smithi183 ceph-mon[37273]: pgmap v4861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:21:59 smithi144 ceph-mon[32129]: pgmap v4861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:21:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:21:59 smithi160 ceph-mon[36715]: pgmap v4861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:22:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:01 smithi183 ceph-mon[37273]: pgmap v4862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8164.714092 secs 2024-07-26T13:22:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:01 smithi144 ceph-mon[32129]: pgmap v4862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8164.714092 secs 2024-07-26T13:22:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:01 smithi160 ceph-mon[36715]: pgmap v4862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8164.714092 secs 2024-07-26T13:22:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:03 smithi183 ceph-mon[37273]: pgmap v4863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:03 smithi144 ceph-mon[32129]: pgmap v4863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:03 smithi160 ceph-mon[36715]: pgmap v4863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:04.087+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:04.088+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:04.088+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:04.088+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:04.088+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:05 smithi183 ceph-mon[37273]: pgmap v4864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:05 smithi144 ceph-mon[32129]: pgmap v4864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:05 smithi160 ceph-mon[36715]: pgmap v4864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8169.714229 secs 2024-07-26T13:22:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8169.714229 secs 2024-07-26T13:22:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8169.714229 secs 2024-07-26T13:22:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:07 smithi183 ceph-mon[37273]: pgmap v4865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:07 smithi144 ceph-mon[32129]: pgmap v4865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:07 smithi160 ceph-mon[36715]: pgmap v4865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:09 smithi144 ceph-mon[32129]: pgmap v4866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:09 smithi160 ceph-mon[36715]: pgmap v4866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:09 smithi183 ceph-mon[37273]: pgmap v4866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:11 smithi144 ceph-mon[32129]: pgmap v4867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8174.714305 secs 2024-07-26T13:22:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:11 smithi160 ceph-mon[36715]: pgmap v4867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8174.714305 secs 2024-07-26T13:22:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:11 smithi183 ceph-mon[37273]: pgmap v4867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8174.714305 secs 2024-07-26T13:22:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:13 smithi144 ceph-mon[32129]: pgmap v4868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:13 smithi160 ceph-mon[36715]: pgmap v4868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:13 smithi183 ceph-mon[37273]: pgmap v4868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:15 smithi144 ceph-mon[32129]: pgmap v4869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:15 smithi160 ceph-mon[36715]: pgmap v4869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:15 smithi183 ceph-mon[37273]: pgmap v4869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8179.714427 secs 2024-07-26T13:22:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8179.714427 secs 2024-07-26T13:22:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8179.714427 secs 2024-07-26T13:22:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:17 smithi144 ceph-mon[32129]: pgmap v4870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:17 smithi160 ceph-mon[36715]: pgmap v4870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:17 smithi183 ceph-mon[37273]: pgmap v4870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:19.104+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:19.105+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:19.105+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:19.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:19.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:19 smithi144 ceph-mon[32129]: pgmap v4871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:19 smithi160 ceph-mon[36715]: pgmap v4871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:19 smithi183 ceph-mon[37273]: pgmap v4871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:21 smithi144 ceph-mon[32129]: pgmap v4872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8184.714556 secs 2024-07-26T13:22:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:21 smithi160 ceph-mon[36715]: pgmap v4872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8184.714556 secs 2024-07-26T13:22:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:21 smithi183 ceph-mon[37273]: pgmap v4872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8184.714556 secs 2024-07-26T13:22:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:23 smithi144 ceph-mon[32129]: pgmap v4873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:23 smithi160 ceph-mon[36715]: pgmap v4873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:23 smithi183 ceph-mon[37273]: pgmap v4873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:25 smithi144 ceph-mon[32129]: pgmap v4874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:25 smithi160 ceph-mon[36715]: pgmap v4874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:25 smithi183 ceph-mon[37273]: pgmap v4874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8189.714673 secs 2024-07-26T13:22:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8189.714673 secs 2024-07-26T13:22:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8189.714673 secs 2024-07-26T13:22:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:27 smithi144 ceph-mon[32129]: pgmap v4875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:27 smithi160 ceph-mon[36715]: pgmap v4875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:27 smithi183 ceph-mon[37273]: pgmap v4875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:29 smithi144 ceph-mon[32129]: pgmap v4876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:29 smithi160 ceph-mon[36715]: pgmap v4876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:29 smithi183 ceph-mon[37273]: pgmap v4876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:31 smithi160 ceph-mon[36715]: pgmap v4877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8194.714802 secs 2024-07-26T13:22:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:31 smithi183 ceph-mon[37273]: pgmap v4877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8194.714802 secs 2024-07-26T13:22:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:31 smithi144 ceph-mon[32129]: pgmap v4877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8194.714802 secs 2024-07-26T13:22:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:33 smithi160 ceph-mon[36715]: pgmap v4878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:33 smithi183 ceph-mon[37273]: pgmap v4878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:33 smithi144 ceph-mon[32129]: pgmap v4878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:34.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:34.115+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:34.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:34.115+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:34.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:34.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:34.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:34.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:35.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:35 smithi160 ceph-mon[36715]: pgmap v4879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:35 smithi183 ceph-mon[37273]: pgmap v4879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:35 smithi144 ceph-mon[32129]: pgmap v4879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8199.714921 secs 2024-07-26T13:22:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8199.714921 secs 2024-07-26T13:22:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8199.714921 secs 2024-07-26T13:22:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:37 smithi160 ceph-mon[36715]: pgmap v4880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:37 smithi183 ceph-mon[37273]: pgmap v4880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:37 smithi144 ceph-mon[32129]: pgmap v4880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:39 smithi160 ceph-mon[36715]: pgmap v4881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:39 smithi183 ceph-mon[37273]: pgmap v4881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:39 smithi144 ceph-mon[32129]: pgmap v4881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:41 smithi160 ceph-mon[36715]: pgmap v4882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8204.715042 secs 2024-07-26T13:22:41.874 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:41 smithi144 ceph-mon[32129]: pgmap v4882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:41.874 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8204.715042 secs 2024-07-26T13:22:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:41 smithi183 ceph-mon[37273]: pgmap v4882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8204.715042 secs 2024-07-26T13:22:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:43 smithi160 ceph-mon[36715]: pgmap v4883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:43 smithi183 ceph-mon[37273]: pgmap v4883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:43 smithi144 ceph-mon[32129]: pgmap v4883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:45 smithi160 ceph-mon[36715]: pgmap v4884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:45 smithi183 ceph-mon[37273]: pgmap v4884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:45 smithi144 ceph-mon[32129]: pgmap v4884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8209.715146 secs 2024-07-26T13:22:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8209.715146 secs 2024-07-26T13:22:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8209.715146 secs 2024-07-26T13:22:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:47 smithi160 ceph-mon[36715]: pgmap v4885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:47 smithi183 ceph-mon[37273]: pgmap v4885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:47 smithi144 ceph-mon[32129]: pgmap v4885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:22:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:49.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:49.131+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:49.131+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:49.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:22:49.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:49 smithi160 ceph-mon[36715]: pgmap v4886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:49 smithi183 ceph-mon[37273]: pgmap v4886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:49 smithi144 ceph-mon[32129]: pgmap v4886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:22:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:22:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:51 smithi160 ceph-mon[36715]: pgmap v4887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:51 smithi160 ceph-mon[36715]: pgmap v4888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8214.715272 secs 2024-07-26T13:22:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:51 smithi183 ceph-mon[37273]: pgmap v4887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:51 smithi183 ceph-mon[37273]: pgmap v4888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8214.715272 secs 2024-07-26T13:22:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:51 smithi144 ceph-mon[32129]: pgmap v4887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:22:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:51 smithi144 ceph-mon[32129]: pgmap v4888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8214.715272 secs 2024-07-26T13:22:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:53 smithi183 ceph-mon[37273]: pgmap v4889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:53 smithi144 ceph-mon[32129]: pgmap v4889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:53 smithi160 ceph-mon[36715]: pgmap v4889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:55 smithi183 ceph-mon[37273]: pgmap v4890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:22:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:55 smithi144 ceph-mon[32129]: pgmap v4890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:22:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:55 smithi160 ceph-mon[36715]: pgmap v4890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:22:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8219.715375 secs 2024-07-26T13:22:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8219.715375 secs 2024-07-26T13:22:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8219.715375 secs 2024-07-26T13:22:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:57 smithi183 ceph-mon[37273]: pgmap v4891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:57 smithi144 ceph-mon[32129]: pgmap v4891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:57 smithi160 ceph-mon[36715]: pgmap v4891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:22:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:22:59 smithi183 ceph-mon[37273]: pgmap v4892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:22:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:22:59 smithi144 ceph-mon[32129]: pgmap v4892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:23:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:22:59 smithi160 ceph-mon[36715]: pgmap v4892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:23:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:01 smithi183 ceph-mon[37273]: pgmap v4893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8224.715519 secs 2024-07-26T13:23:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:01 smithi144 ceph-mon[32129]: pgmap v4893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8224.715519 secs 2024-07-26T13:23:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:01 smithi160 ceph-mon[36715]: pgmap v4893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8224.715519 secs 2024-07-26T13:23:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:03 smithi183 ceph-mon[37273]: pgmap v4894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:03 smithi144 ceph-mon[32129]: pgmap v4894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:03 smithi160 ceph-mon[36715]: pgmap v4894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:04.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:23:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:04.148+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:23:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:04.148+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:23:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:04.148+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:23:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:04.148+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:05 smithi183 ceph-mon[37273]: pgmap v4895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:05 smithi144 ceph-mon[32129]: pgmap v4895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:05 smithi160 ceph-mon[36715]: pgmap v4895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8229.715641 secs 2024-07-26T13:23:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8229.715641 secs 2024-07-26T13:23:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8229.715641 secs 2024-07-26T13:23:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:07 smithi183 ceph-mon[37273]: pgmap v4896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:07 smithi144 ceph-mon[32129]: pgmap v4896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:07 smithi160 ceph-mon[36715]: pgmap v4896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:09 smithi183 ceph-mon[37273]: pgmap v4897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:09 smithi144 ceph-mon[32129]: pgmap v4897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:09 smithi160 ceph-mon[36715]: pgmap v4897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:11 smithi183 ceph-mon[37273]: pgmap v4898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8234.715708 secs 2024-07-26T13:23:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:11 smithi144 ceph-mon[32129]: pgmap v4898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8234.715708 secs 2024-07-26T13:23:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:11 smithi160 ceph-mon[36715]: pgmap v4898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8234.715708 secs 2024-07-26T13:23:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:13 smithi144 ceph-mon[32129]: pgmap v4899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:13 smithi160 ceph-mon[36715]: pgmap v4899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:13 smithi183 ceph-mon[37273]: pgmap v4899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:15 smithi144 ceph-mon[32129]: pgmap v4900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:15 smithi160 ceph-mon[36715]: pgmap v4900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:15 smithi183 ceph-mon[37273]: pgmap v4900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8239.715832 secs 2024-07-26T13:23:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8239.715832 secs 2024-07-26T13:23:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8239.715832 secs 2024-07-26T13:23:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:17 smithi144 ceph-mon[32129]: pgmap v4901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:17 smithi160 ceph-mon[36715]: pgmap v4901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:17 smithi183 ceph-mon[37273]: pgmap v4901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:19.158+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:23:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:19.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:23:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:19.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:23:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:19.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:23:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:19.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:19 smithi144 ceph-mon[32129]: pgmap v4902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:19 smithi160 ceph-mon[36715]: pgmap v4902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:19 smithi183 ceph-mon[37273]: pgmap v4902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:21 smithi144 ceph-mon[32129]: pgmap v4903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8244.715953 secs 2024-07-26T13:23:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:21 smithi160 ceph-mon[36715]: pgmap v4903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8244.715953 secs 2024-07-26T13:23:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:21 smithi183 ceph-mon[37273]: pgmap v4903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8244.715953 secs 2024-07-26T13:23:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:23 smithi144 ceph-mon[32129]: pgmap v4904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:23 smithi160 ceph-mon[36715]: pgmap v4904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:23 smithi183 ceph-mon[37273]: pgmap v4904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:25 smithi144 ceph-mon[32129]: pgmap v4905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:25 smithi160 ceph-mon[36715]: pgmap v4905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:25 smithi183 ceph-mon[37273]: pgmap v4905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8249.716058 secs 2024-07-26T13:23:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8249.716058 secs 2024-07-26T13:23:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8249.716058 secs 2024-07-26T13:23:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:27 smithi144 ceph-mon[32129]: pgmap v4906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:27 smithi160 ceph-mon[36715]: pgmap v4906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:27 smithi183 ceph-mon[37273]: pgmap v4906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:29 smithi144 ceph-mon[32129]: pgmap v4907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:23:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:29 smithi160 ceph-mon[36715]: pgmap v4907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:23:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:29 smithi183 ceph-mon[37273]: pgmap v4907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:23:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:31 smithi144 ceph-mon[32129]: pgmap v4908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8254.716148 secs 2024-07-26T13:23:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:31 smithi160 ceph-mon[36715]: pgmap v4908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:32.150 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8254.716148 secs 2024-07-26T13:23:32.151 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:31 smithi183 ceph-mon[37273]: pgmap v4908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:23:32.151 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8254.716148 secs 2024-07-26T13:23:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:33 smithi144 ceph-mon[32129]: pgmap v4909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:33 smithi160 ceph-mon[36715]: pgmap v4909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:33 smithi183 ceph-mon[37273]: pgmap v4909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:23:34.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:34.175+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:23:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:34.176+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:23:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:34.176+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:23:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:34.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:23:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:34.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:35 smithi160 ceph-mon[36715]: pgmap v4910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:35 smithi183 ceph-mon[37273]: pgmap v4910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:35 smithi144 ceph-mon[32129]: pgmap v4910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8259.716329 secs 2024-07-26T13:23:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8259.716329 secs 2024-07-26T13:23:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8259.716329 secs 2024-07-26T13:23:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:37 smithi160 ceph-mon[36715]: pgmap v4911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:37 smithi183 ceph-mon[37273]: pgmap v4911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:37 smithi144 ceph-mon[32129]: pgmap v4911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:39 smithi160 ceph-mon[36715]: pgmap v4912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:39 smithi183 ceph-mon[37273]: pgmap v4912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:39 smithi144 ceph-mon[32129]: pgmap v4912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:42.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:41 smithi144 ceph-mon[32129]: pgmap v4913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:42.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8264.716448 secs 2024-07-26T13:23:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:41 smithi160 ceph-mon[36715]: pgmap v4913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:42.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8264.716448 secs 2024-07-26T13:23:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:41 smithi183 ceph-mon[37273]: pgmap v4913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8264.716448 secs 2024-07-26T13:23:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:43 smithi160 ceph-mon[36715]: pgmap v4914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:43 smithi183 ceph-mon[37273]: pgmap v4914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:43 smithi144 ceph-mon[32129]: pgmap v4914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:45 smithi160 ceph-mon[36715]: pgmap v4915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:45 smithi183 ceph-mon[37273]: pgmap v4915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:45 smithi144 ceph-mon[32129]: pgmap v4915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8269.716581 secs 2024-07-26T13:23:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8269.716581 secs 2024-07-26T13:23:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8269.716581 secs 2024-07-26T13:23:47.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:47.564+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:23:47.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:47.564+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:23:47.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:47.564+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T13:23:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:47.565+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:23:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:47 smithi160 ceph-mon[36715]: pgmap v4916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:47 smithi183 ceph-mon[37273]: pgmap v4916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:47 smithi144 ceph-mon[32129]: pgmap v4916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:49.183+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:23:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:49.183+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:23:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:49.184+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:23:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:49.184+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:23:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:23:49.184+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:49 smithi160 ceph-mon[36715]: pgmap v4917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:49 smithi183 ceph-mon[37273]: pgmap v4917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:49 smithi144 ceph-mon[32129]: pgmap v4917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:23:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:51 smithi160 ceph-mon[36715]: pgmap v4918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8274.716651 secs 2024-07-26T13:23:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:51 smithi183 ceph-mon[37273]: pgmap v4918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8274.716651 secs 2024-07-26T13:23:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:51 smithi144 ceph-mon[32129]: pgmap v4918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8274.716651 secs 2024-07-26T13:23:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: pgmap v4919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: pgmap v4919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: pgmap v4919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:23:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:55 smithi160 ceph-mon[36715]: pgmap v4920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:55 smithi183 ceph-mon[37273]: pgmap v4920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:55 smithi144 ceph-mon[32129]: pgmap v4920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:56 smithi160 ceph-mon[36715]: pgmap v4921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:23:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:56 smithi160 ceph-mon[36715]: pgmap v4922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8279.716781 secs 2024-07-26T13:23:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:56 smithi183 ceph-mon[37273]: pgmap v4921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:23:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:56 smithi183 ceph-mon[37273]: pgmap v4922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8279.716781 secs 2024-07-26T13:23:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:56 smithi144 ceph-mon[32129]: pgmap v4921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:23:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:56 smithi144 ceph-mon[32129]: pgmap v4922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:23:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8279.716781 secs 2024-07-26T13:23:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:23:58 smithi183 ceph-mon[37273]: pgmap v4923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:23:58 smithi144 ceph-mon[32129]: pgmap v4923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:23:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:23:58 smithi160 ceph-mon[36715]: pgmap v4923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:00 smithi183 ceph-mon[37273]: pgmap v4924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:00 smithi144 ceph-mon[32129]: pgmap v4924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:00 smithi160 ceph-mon[36715]: pgmap v4924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8284.716907 secs 2024-07-26T13:24:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8284.716907 secs 2024-07-26T13:24:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8284.716907 secs 2024-07-26T13:24:02.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:02 smithi183 ceph-mon[37273]: pgmap v4925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:02 smithi144 ceph-mon[32129]: pgmap v4925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:02 smithi160 ceph-mon[36715]: pgmap v4925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:04.199+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:24:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:04.200+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:24:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:04.200+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:24:04.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:04.200+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:24:04.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:04.200+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:04 smithi183 ceph-mon[37273]: pgmap v4926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:04 smithi144 ceph-mon[32129]: pgmap v4926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:04 smithi160 ceph-mon[36715]: pgmap v4926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:06 smithi183 ceph-mon[37273]: pgmap v4927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:24:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8289.717028 secs 2024-07-26T13:24:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:06 smithi144 ceph-mon[32129]: pgmap v4927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:24:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8289.717028 secs 2024-07-26T13:24:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:06 smithi160 ceph-mon[36715]: pgmap v4927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:24:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8289.717028 secs 2024-07-26T13:24:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:08 smithi183 ceph-mon[37273]: pgmap v4928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:08 smithi144 ceph-mon[32129]: pgmap v4928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:08 smithi160 ceph-mon[36715]: pgmap v4928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:10 smithi183 ceph-mon[37273]: pgmap v4929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:10 smithi144 ceph-mon[32129]: pgmap v4929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:10 smithi160 ceph-mon[36715]: pgmap v4929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8294.717165 secs 2024-07-26T13:24:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8294.717165 secs 2024-07-26T13:24:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8294.717165 secs 2024-07-26T13:24:12.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:12 smithi144 ceph-mon[32129]: pgmap v4930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:12 smithi183 ceph-mon[37273]: pgmap v4930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:12 smithi160 ceph-mon[36715]: pgmap v4930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:14 smithi183 ceph-mon[37273]: pgmap v4931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:14 smithi144 ceph-mon[32129]: pgmap v4931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:14 smithi160 ceph-mon[36715]: pgmap v4931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:16 smithi183 ceph-mon[37273]: pgmap v4932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8299.717234 secs 2024-07-26T13:24:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:16 smithi144 ceph-mon[32129]: pgmap v4932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8299.717234 secs 2024-07-26T13:24:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:16 smithi160 ceph-mon[36715]: pgmap v4932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8299.717234 secs 2024-07-26T13:24:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:18 smithi183 ceph-mon[37273]: pgmap v4933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:18 smithi144 ceph-mon[32129]: pgmap v4933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:18 smithi160 ceph-mon[36715]: pgmap v4933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:19.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:24:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:19.207+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:24:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:19.207+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:24:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:19.207+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:24:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:19.207+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:20 smithi144 ceph-mon[32129]: pgmap v4934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:20 smithi160 ceph-mon[36715]: pgmap v4934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:20 smithi183 ceph-mon[37273]: pgmap v4934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:21.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8304.717359 secs 2024-07-26T13:24:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8304.717359 secs 2024-07-26T13:24:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8304.717359 secs 2024-07-26T13:24:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:22 smithi144 ceph-mon[32129]: pgmap v4935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:22 smithi160 ceph-mon[36715]: pgmap v4935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:22 smithi183 ceph-mon[37273]: pgmap v4935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:24 smithi144 ceph-mon[32129]: pgmap v4936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:24 smithi160 ceph-mon[36715]: pgmap v4936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:24 smithi183 ceph-mon[37273]: pgmap v4936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:26 smithi144 ceph-mon[32129]: pgmap v4937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8309.717499 secs 2024-07-26T13:24:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:26 smithi160 ceph-mon[36715]: pgmap v4937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8309.717499 secs 2024-07-26T13:24:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:26 smithi183 ceph-mon[37273]: pgmap v4937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8309.717499 secs 2024-07-26T13:24:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:28 smithi144 ceph-mon[32129]: pgmap v4938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:28 smithi160 ceph-mon[36715]: pgmap v4938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:28 smithi183 ceph-mon[37273]: pgmap v4938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:30 smithi144 ceph-mon[32129]: pgmap v4939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:30 smithi160 ceph-mon[36715]: pgmap v4939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:30 smithi183 ceph-mon[37273]: pgmap v4939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8314.717630 secs 2024-07-26T13:24:32.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8314.717630 secs 2024-07-26T13:24:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8314.717630 secs 2024-07-26T13:24:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:32 smithi144 ceph-mon[32129]: pgmap v4940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:32 smithi160 ceph-mon[36715]: pgmap v4940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:32 smithi183 ceph-mon[37273]: pgmap v4940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:34.220+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:24:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:34.221+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:24:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:34.221+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:24:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:34.221+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:24:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:34.221+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:34 smithi144 ceph-mon[32129]: pgmap v4941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:34 smithi160 ceph-mon[36715]: pgmap v4941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:34 smithi183 ceph-mon[37273]: pgmap v4941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:36 smithi144 ceph-mon[32129]: pgmap v4942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8319.717767 secs 2024-07-26T13:24:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:36 smithi160 ceph-mon[36715]: pgmap v4942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8319.717767 secs 2024-07-26T13:24:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:36 smithi183 ceph-mon[37273]: pgmap v4942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8319.717767 secs 2024-07-26T13:24:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:38 smithi144 ceph-mon[32129]: pgmap v4943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:38 smithi160 ceph-mon[36715]: pgmap v4943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:38 smithi183 ceph-mon[37273]: pgmap v4943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:40 smithi144 ceph-mon[32129]: pgmap v4944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:40 smithi160 ceph-mon[36715]: pgmap v4944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:40 smithi183 ceph-mon[37273]: pgmap v4944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8324.717902 secs 2024-07-26T13:24:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8324.717902 secs 2024-07-26T13:24:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8324.717902 secs 2024-07-26T13:24:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:42 smithi160 ceph-mon[36715]: pgmap v4945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:43.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:42 smithi144 ceph-mon[32129]: pgmap v4945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:42 smithi183 ceph-mon[37273]: pgmap v4945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:44 smithi160 ceph-mon[36715]: pgmap v4946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:44 smithi183 ceph-mon[37273]: pgmap v4946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:44 smithi144 ceph-mon[32129]: pgmap v4946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:46 smithi160 ceph-mon[36715]: pgmap v4947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8329.718037 secs 2024-07-26T13:24:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:46 smithi183 ceph-mon[37273]: pgmap v4947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8329.718037 secs 2024-07-26T13:24:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:46 smithi144 ceph-mon[32129]: pgmap v4947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8329.718037 secs 2024-07-26T13:24:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:48 smithi160 ceph-mon[36715]: pgmap v4948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:48 smithi183 ceph-mon[37273]: pgmap v4948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:48 smithi144 ceph-mon[32129]: pgmap v4948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:24:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:49.237+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:24:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:49.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:24:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:49.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:24:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:49.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:24:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:24:49.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:50 smithi160 ceph-mon[36715]: pgmap v4949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:50 smithi183 ceph-mon[37273]: pgmap v4949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:50 smithi144 ceph-mon[32129]: pgmap v4949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8334.718085 secs 2024-07-26T13:24:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8334.718085 secs 2024-07-26T13:24:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8334.718085 secs 2024-07-26T13:24:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:52 smithi160 ceph-mon[36715]: pgmap v4950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:52 smithi183 ceph-mon[37273]: pgmap v4950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:52 smithi144 ceph-mon[32129]: pgmap v4950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:24:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:54 smithi160 ceph-mon[36715]: pgmap v4951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:54 smithi183 ceph-mon[37273]: pgmap v4951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:54 smithi144 ceph-mon[32129]: pgmap v4951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:56.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:56 smithi160 ceph-mon[36715]: pgmap v4952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8339.718212 secs 2024-07-26T13:24:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:24:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:56 smithi183 ceph-mon[37273]: pgmap v4952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8339.718212 secs 2024-07-26T13:24:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:24:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:56 smithi144 ceph-mon[32129]: pgmap v4952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:24:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8339.718212 secs 2024-07-26T13:24:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:24:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:57 smithi160 ceph-mon[36715]: pgmap v4953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:24:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:57 smithi160 ceph-mon[36715]: pgmap v4954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:24:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:57 smithi183 ceph-mon[37273]: pgmap v4953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:24:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:57 smithi183 ceph-mon[37273]: pgmap v4954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:24:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:57 smithi144 ceph-mon[32129]: pgmap v4953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:24:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:57 smithi144 ceph-mon[32129]: pgmap v4954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:25:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:24:59 smithi183 ceph-mon[37273]: pgmap v4955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:25:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:24:59 smithi144 ceph-mon[32129]: pgmap v4955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:25:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:24:59 smithi160 ceph-mon[36715]: pgmap v4955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:25:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8344.718332 secs 2024-07-26T13:25:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:01 smithi183 ceph-mon[37273]: pgmap v4956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:25:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8344.718332 secs 2024-07-26T13:25:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:01 smithi144 ceph-mon[32129]: pgmap v4956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:25:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8344.718332 secs 2024-07-26T13:25:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:01 smithi160 ceph-mon[36715]: pgmap v4956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:25:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:03 smithi183 ceph-mon[37273]: pgmap v4957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:03 smithi144 ceph-mon[32129]: pgmap v4957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:04.253+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:25:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:04.255+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:25:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:04.255+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:25:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:04.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:25:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:04.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:03 smithi160 ceph-mon[36715]: pgmap v4957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:05 smithi183 ceph-mon[37273]: pgmap v4958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:05 smithi144 ceph-mon[32129]: pgmap v4958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:05 smithi160 ceph-mon[36715]: pgmap v4958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8349.718461 secs 2024-07-26T13:25:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8349.718461 secs 2024-07-26T13:25:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8349.718461 secs 2024-07-26T13:25:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:07 smithi183 ceph-mon[37273]: pgmap v4959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:25:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:07 smithi144 ceph-mon[32129]: pgmap v4959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:25:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:07 smithi160 ceph-mon[36715]: pgmap v4959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:25:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:09 smithi183 ceph-mon[37273]: pgmap v4960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:09 smithi144 ceph-mon[32129]: pgmap v4960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:09 smithi160 ceph-mon[36715]: pgmap v4960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8354.718568 secs 2024-07-26T13:25:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:11 smithi183 ceph-mon[37273]: pgmap v4961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8354.718568 secs 2024-07-26T13:25:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:11 smithi144 ceph-mon[32129]: pgmap v4961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8354.718568 secs 2024-07-26T13:25:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:11 smithi160 ceph-mon[36715]: pgmap v4961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:13 smithi183 ceph-mon[37273]: pgmap v4962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:13 smithi144 ceph-mon[32129]: pgmap v4962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:13 smithi160 ceph-mon[36715]: pgmap v4962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:15 smithi183 ceph-mon[37273]: pgmap v4963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:15 smithi144 ceph-mon[32129]: pgmap v4963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:16.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:15 smithi160 ceph-mon[36715]: pgmap v4963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8359.718685 secs 2024-07-26T13:25:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8359.718685 secs 2024-07-26T13:25:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8359.718685 secs 2024-07-26T13:25:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:17 smithi183 ceph-mon[37273]: pgmap v4964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:17 smithi144 ceph-mon[32129]: pgmap v4964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:17 smithi160 ceph-mon[36715]: pgmap v4964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:19.268+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:25:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:19.269+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:25:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:19.269+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:25:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:19.269+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:25:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:19.269+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:19 smithi144 ceph-mon[32129]: pgmap v4965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:19 smithi160 ceph-mon[36715]: pgmap v4965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:19 smithi183 ceph-mon[37273]: pgmap v4965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:21.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8364.718813 secs 2024-07-26T13:25:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8364.718813 secs 2024-07-26T13:25:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8364.718813 secs 2024-07-26T13:25:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:21 smithi144 ceph-mon[32129]: pgmap v4966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:22.226 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:21 smithi160 ceph-mon[36715]: pgmap v4966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:21 smithi183 ceph-mon[37273]: pgmap v4966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:23 smithi144 ceph-mon[32129]: pgmap v4967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:23 smithi160 ceph-mon[36715]: pgmap v4967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:23 smithi183 ceph-mon[37273]: pgmap v4967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:25 smithi144 ceph-mon[32129]: pgmap v4968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8369.718926 secs 2024-07-26T13:25:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:25 smithi160 ceph-mon[36715]: pgmap v4968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8369.718926 secs 2024-07-26T13:25:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:25 smithi183 ceph-mon[37273]: pgmap v4968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8369.718926 secs 2024-07-26T13:25:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:27 smithi144 ceph-mon[32129]: pgmap v4969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:27 smithi160 ceph-mon[36715]: pgmap v4969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:27 smithi183 ceph-mon[37273]: pgmap v4969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:29 smithi144 ceph-mon[32129]: pgmap v4970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:29 smithi160 ceph-mon[36715]: pgmap v4970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:29 smithi183 ceph-mon[37273]: pgmap v4970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:30 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8374.719053 secs 2024-07-26T13:25:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:30 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8374.719053 secs 2024-07-26T13:25:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:30 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8374.719053 secs 2024-07-26T13:25:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:31 smithi144 ceph-mon[32129]: pgmap v4971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:31 smithi160 ceph-mon[36715]: pgmap v4971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:31 smithi183 ceph-mon[37273]: pgmap v4971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:33 smithi144 ceph-mon[32129]: pgmap v4972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:34.284 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:34.283+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:25:34.284 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:34.284+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:25:34.284 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:34.284+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:25:34.284 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:34.284+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:25:34.285 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:34.284+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:33 smithi160 ceph-mon[36715]: pgmap v4972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:33 smithi183 ceph-mon[37273]: pgmap v4972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:35 smithi144 ceph-mon[32129]: pgmap v4973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8379.719201 secs 2024-07-26T13:25:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:35 smithi160 ceph-mon[36715]: pgmap v4973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8379.719201 secs 2024-07-26T13:25:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:35 smithi183 ceph-mon[37273]: pgmap v4973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8379.719201 secs 2024-07-26T13:25:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:37 smithi144 ceph-mon[32129]: pgmap v4974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:37 smithi160 ceph-mon[36715]: pgmap v4974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:37 smithi183 ceph-mon[37273]: pgmap v4974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:39 smithi144 ceph-mon[32129]: pgmap v4975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:39 smithi160 ceph-mon[36715]: pgmap v4975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:39 smithi183 ceph-mon[37273]: pgmap v4975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8384.719324 secs 2024-07-26T13:25:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8384.719324 secs 2024-07-26T13:25:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8384.719324 secs 2024-07-26T13:25:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:41 smithi160 ceph-mon[36715]: pgmap v4976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:41 smithi183 ceph-mon[37273]: pgmap v4976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:41 smithi144 ceph-mon[32129]: pgmap v4976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:44.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:43 smithi144 ceph-mon[32129]: pgmap v4977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:43 smithi160 ceph-mon[36715]: pgmap v4977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:43 smithi183 ceph-mon[37273]: pgmap v4977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:44 smithi160 ceph-mon[36715]: pgmap v4978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:44 smithi183 ceph-mon[37273]: pgmap v4978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:44 smithi144 ceph-mon[32129]: pgmap v4978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8389.719415 secs 2024-07-26T13:25:46.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8389.719415 secs 2024-07-26T13:25:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8389.719415 secs 2024-07-26T13:25:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:47 smithi160 ceph-mon[36715]: pgmap v4979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:47 smithi183 ceph-mon[37273]: pgmap v4979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:47 smithi144 ceph-mon[32129]: pgmap v4979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:49.285+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:25:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:49.286+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:25:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:49.286+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:25:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:49.286+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:25:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:25:49.286+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:49 smithi183 ceph-mon[37273]: pgmap v4980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:49 smithi144 ceph-mon[32129]: pgmap v4980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:49 smithi160 ceph-mon[36715]: pgmap v4980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:52.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8394.719532 secs 2024-07-26T13:25:52.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:51 smithi160 ceph-mon[36715]: pgmap v4981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8394.719532 secs 2024-07-26T13:25:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:51 smithi183 ceph-mon[37273]: pgmap v4981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8394.719532 secs 2024-07-26T13:25:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:51 smithi144 ceph-mon[32129]: pgmap v4981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:53 smithi183 ceph-mon[37273]: pgmap v4982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:53 smithi144 ceph-mon[32129]: pgmap v4982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:53 smithi160 ceph-mon[36715]: pgmap v4982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:25:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:55 smithi183 ceph-mon[37273]: pgmap v4983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:55 smithi144 ceph-mon[32129]: pgmap v4983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:55 smithi160 ceph-mon[36715]: pgmap v4983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:25:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8399.719622 secs 2024-07-26T13:25:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8399.719622 secs 2024-07-26T13:25:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8399.719622 secs 2024-07-26T13:25:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:57 smithi183 ceph-mon[37273]: pgmap v4984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:57 smithi144 ceph-mon[32129]: pgmap v4984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:57 smithi160 ceph-mon[36715]: pgmap v4984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:25:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:58 smithi183 ceph-mon[37273]: pgmap v4985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:25:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:58 smithi183 ceph-mon[37273]: pgmap v4986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:25:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:25:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:58 smithi144 ceph-mon[32129]: pgmap v4985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:25:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:58 smithi144 ceph-mon[32129]: pgmap v4986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:25:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:25:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:58 smithi160 ceph-mon[36715]: pgmap v4985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:25:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:58 smithi160 ceph-mon[36715]: pgmap v4986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:25:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:25:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:26:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:00 smithi160 ceph-mon[36715]: pgmap v4987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:26:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:00 smithi183 ceph-mon[37273]: pgmap v4987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:26:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:00 smithi144 ceph-mon[32129]: pgmap v4987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:26:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8404.719755 secs 2024-07-26T13:26:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8404.719755 secs 2024-07-26T13:26:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8404.719755 secs 2024-07-26T13:26:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:02 smithi160 ceph-mon[36715]: pgmap v4988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:26:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:02 smithi183 ceph-mon[37273]: pgmap v4988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:26:02.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:02 smithi144 ceph-mon[32129]: pgmap v4988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:26:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:02.588+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:26:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:02.588+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74780064a0 con 0x563c6c4182f0 2024-07-26T13:26:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:02.589+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74780064a0 con 0x7f74b0109990 2024-07-26T13:26:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:02.589+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:26:04.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:04.294+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:26:04.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:04.295+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:26:04.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:04.295+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:26:04.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:04.295+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:26:04.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:04.295+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:04 smithi160 ceph-mon[36715]: pgmap v4989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:26:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:04 smithi183 ceph-mon[37273]: pgmap v4989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:26:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:04 smithi144 ceph-mon[32129]: pgmap v4989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:26:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8409.719899 secs 2024-07-26T13:26:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8409.719899 secs 2024-07-26T13:26:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8409.719899 secs 2024-07-26T13:26:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:06 smithi144 ceph-mon[32129]: pgmap v4990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:26:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:06 smithi160 ceph-mon[36715]: pgmap v4990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:26:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:06 smithi183 ceph-mon[37273]: pgmap v4990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:26:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:08 smithi160 ceph-mon[36715]: pgmap v4991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:08 smithi183 ceph-mon[37273]: pgmap v4991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:08 smithi144 ceph-mon[32129]: pgmap v4991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:10 smithi160 ceph-mon[36715]: pgmap v4992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:10 smithi183 ceph-mon[37273]: pgmap v4992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:10 smithi144 ceph-mon[32129]: pgmap v4992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8414.720034 secs 2024-07-26T13:26:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8414.720034 secs 2024-07-26T13:26:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8414.720034 secs 2024-07-26T13:26:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:12 smithi160 ceph-mon[36715]: pgmap v4993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:12 smithi183 ceph-mon[37273]: pgmap v4993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:12 smithi144 ceph-mon[32129]: pgmap v4993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:14 smithi160 ceph-mon[36715]: pgmap v4994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:14 smithi183 ceph-mon[37273]: pgmap v4994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:14 smithi144 ceph-mon[32129]: pgmap v4994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8419.720165 secs 2024-07-26T13:26:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8419.720165 secs 2024-07-26T13:26:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8419.720165 secs 2024-07-26T13:26:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:16 smithi144 ceph-mon[32129]: pgmap v4995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:16 smithi160 ceph-mon[36715]: pgmap v4995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:16 smithi183 ceph-mon[37273]: pgmap v4995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:18 smithi160 ceph-mon[36715]: pgmap v4996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:18 smithi183 ceph-mon[37273]: pgmap v4996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:18 smithi144 ceph-mon[32129]: pgmap v4996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:19.309+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:26:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:19.310+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:26:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:19.310+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:26:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:19.310+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:26:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:19.310+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:20 smithi160 ceph-mon[36715]: pgmap v4997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:20 smithi183 ceph-mon[37273]: pgmap v4997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:20 smithi144 ceph-mon[32129]: pgmap v4997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8424.720321 secs 2024-07-26T13:26:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8424.720321 secs 2024-07-26T13:26:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8424.720321 secs 2024-07-26T13:26:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:22 smithi183 ceph-mon[37273]: pgmap v4998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:22 smithi144 ceph-mon[32129]: pgmap v4998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:22 smithi160 ceph-mon[36715]: pgmap v4998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:24 smithi160 ceph-mon[36715]: pgmap v4999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:24 smithi183 ceph-mon[37273]: pgmap v4999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:24 smithi144 ceph-mon[32129]: pgmap v4999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8429.720480 secs 2024-07-26T13:26:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8429.720480 secs 2024-07-26T13:26:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8429.720480 secs 2024-07-26T13:26:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:26 smithi144 ceph-mon[32129]: pgmap v5000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:26 smithi160 ceph-mon[36715]: pgmap v5000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:26 smithi183 ceph-mon[37273]: pgmap v5000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:28 smithi160 ceph-mon[36715]: pgmap v5001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:28 smithi183 ceph-mon[37273]: pgmap v5001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:28 smithi144 ceph-mon[32129]: pgmap v5001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:30 smithi183 ceph-mon[37273]: pgmap v5002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:30 smithi144 ceph-mon[32129]: pgmap v5002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:30 smithi160 ceph-mon[36715]: pgmap v5002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8434.720627 secs 2024-07-26T13:26:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8434.720627 secs 2024-07-26T13:26:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8434.720627 secs 2024-07-26T13:26:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:32 smithi183 ceph-mon[37273]: pgmap v5003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:32 smithi144 ceph-mon[32129]: pgmap v5003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:32 smithi160 ceph-mon[36715]: pgmap v5003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:34 smithi160 ceph-mon[36715]: pgmap v5004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:34.320 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:34.319+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:26:34.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:34.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:26:34.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:34.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:26:34.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:34.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:26:34.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:34.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:34 smithi183 ceph-mon[37273]: pgmap v5004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:34 smithi144 ceph-mon[32129]: pgmap v5004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:35 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8439.720709 secs 2024-07-26T13:26:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:35 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8439.720709 secs 2024-07-26T13:26:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:35 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8439.720709 secs 2024-07-26T13:26:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:36 smithi144 ceph-mon[32129]: pgmap v5005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:36 smithi160 ceph-mon[36715]: pgmap v5005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:36 smithi183 ceph-mon[37273]: pgmap v5005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:38 smithi160 ceph-mon[36715]: pgmap v5006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:38 smithi183 ceph-mon[37273]: pgmap v5006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:38 smithi144 ceph-mon[32129]: pgmap v5006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:40 smithi160 ceph-mon[36715]: pgmap v5007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:40 smithi183 ceph-mon[37273]: pgmap v5007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:40 smithi144 ceph-mon[32129]: pgmap v5007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8444.720834 secs 2024-07-26T13:26:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8444.720834 secs 2024-07-26T13:26:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8444.720834 secs 2024-07-26T13:26:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:42 smithi183 ceph-mon[37273]: pgmap v5008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:42 smithi144 ceph-mon[32129]: pgmap v5008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:42 smithi160 ceph-mon[36715]: pgmap v5008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:26:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:44 smithi160 ceph-mon[36715]: pgmap v5009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:44 smithi183 ceph-mon[37273]: pgmap v5009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:44 smithi144 ceph-mon[32129]: pgmap v5009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8449.720923 secs 2024-07-26T13:26:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8449.720923 secs 2024-07-26T13:26:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8449.720923 secs 2024-07-26T13:26:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:46 smithi144 ceph-mon[32129]: pgmap v5010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:46 smithi160 ceph-mon[36715]: pgmap v5010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:46 smithi183 ceph-mon[37273]: pgmap v5010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:26:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:47.597+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:26:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:47.597+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:26:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:47.597+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:26:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:47.597+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:26:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:48 smithi144 ceph-mon[32129]: pgmap v5011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:48 smithi160 ceph-mon[36715]: pgmap v5011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:48 smithi183 ceph-mon[37273]: pgmap v5011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:26:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:49.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:26:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:49.336+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:26:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:49.336+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:26:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:49.336+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:26:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:26:49.336+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:50 smithi160 ceph-mon[36715]: pgmap v5012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:26:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:50 smithi183 ceph-mon[37273]: pgmap v5012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:26:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:50 smithi144 ceph-mon[32129]: pgmap v5012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:26:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8454.721052 secs 2024-07-26T13:26:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8454.721052 secs 2024-07-26T13:26:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8454.721052 secs 2024-07-26T13:26:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:52 smithi183 ceph-mon[37273]: pgmap v5013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:52 smithi144 ceph-mon[32129]: pgmap v5013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:52 smithi160 ceph-mon[36715]: pgmap v5013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:54 smithi160 ceph-mon[36715]: pgmap v5014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:54 smithi183 ceph-mon[37273]: pgmap v5014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:54 smithi144 ceph-mon[32129]: pgmap v5014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8459.721183 secs 2024-07-26T13:26:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8459.721183 secs 2024-07-26T13:26:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8459.721183 secs 2024-07-26T13:26:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:56 smithi144 ceph-mon[32129]: pgmap v5015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:56 smithi160 ceph-mon[36715]: pgmap v5015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:56 smithi183 ceph-mon[37273]: pgmap v5015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:26:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:58 smithi160 ceph-mon[36715]: pgmap v5016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:26:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:58 smithi183 ceph-mon[37273]: pgmap v5016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:26:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:58 smithi144 ceph-mon[32129]: pgmap v5016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:26:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:26:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:26:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:26:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:00 smithi183 ceph-mon[37273]: pgmap v5017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:00 smithi183 ceph-mon[37273]: pgmap v5018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:27:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:27:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:00 smithi144 ceph-mon[32129]: pgmap v5017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:00 smithi144 ceph-mon[32129]: pgmap v5018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:27:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:27:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:00 smithi160 ceph-mon[36715]: pgmap v5017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:00 smithi160 ceph-mon[36715]: pgmap v5018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:27:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:27:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8464.721332 secs 2024-07-26T13:27:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8464.721332 secs 2024-07-26T13:27:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8464.721332 secs 2024-07-26T13:27:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:02 smithi183 ceph-mon[37273]: pgmap v5019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:02 smithi144 ceph-mon[32129]: pgmap v5019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:02 smithi160 ceph-mon[36715]: pgmap v5019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:04.354 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:04.353+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:27:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:04.354+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:27:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:04.354+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:27:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:04.354+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:27:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:04.354+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:04 smithi144 ceph-mon[32129]: pgmap v5020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:04 smithi160 ceph-mon[36715]: pgmap v5020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:04 smithi183 ceph-mon[37273]: pgmap v5020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:06 smithi144 ceph-mon[32129]: pgmap v5021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8469.721460 secs 2024-07-26T13:27:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:06 smithi160 ceph-mon[36715]: pgmap v5021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8469.721460 secs 2024-07-26T13:27:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:06 smithi183 ceph-mon[37273]: pgmap v5021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8469.721460 secs 2024-07-26T13:27:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:08 smithi144 ceph-mon[32129]: pgmap v5022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:08 smithi160 ceph-mon[36715]: pgmap v5022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:08 smithi183 ceph-mon[37273]: pgmap v5022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:27:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:10 smithi144 ceph-mon[32129]: pgmap v5023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:27:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:10 smithi160 ceph-mon[36715]: pgmap v5023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:27:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:10 smithi183 ceph-mon[37273]: pgmap v5023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:27:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8474.721561 secs 2024-07-26T13:27:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8474.721561 secs 2024-07-26T13:27:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8474.721561 secs 2024-07-26T13:27:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:12 smithi144 ceph-mon[32129]: pgmap v5024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:12 smithi160 ceph-mon[36715]: pgmap v5024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:12 smithi183 ceph-mon[37273]: pgmap v5024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:14 smithi160 ceph-mon[36715]: pgmap v5025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:14 smithi183 ceph-mon[37273]: pgmap v5025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:14 smithi144 ceph-mon[32129]: pgmap v5025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:16 smithi160 ceph-mon[36715]: pgmap v5026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8479.721693 secs 2024-07-26T13:27:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:16 smithi183 ceph-mon[37273]: pgmap v5026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8479.721693 secs 2024-07-26T13:27:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:16 smithi144 ceph-mon[32129]: pgmap v5026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8479.721693 secs 2024-07-26T13:27:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:18 smithi160 ceph-mon[36715]: pgmap v5027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:18 smithi183 ceph-mon[37273]: pgmap v5027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:18 smithi144 ceph-mon[32129]: pgmap v5027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:19.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:27:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:19.366+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:27:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:19.366+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:27:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:19.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:27:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:19.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:20 smithi160 ceph-mon[36715]: pgmap v5028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:20 smithi183 ceph-mon[37273]: pgmap v5028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:20 smithi144 ceph-mon[32129]: pgmap v5028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8484.721812 secs 2024-07-26T13:27:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8484.721812 secs 2024-07-26T13:27:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8484.721812 secs 2024-07-26T13:27:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:22 smithi160 ceph-mon[36715]: pgmap v5029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:22 smithi183 ceph-mon[37273]: pgmap v5029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:22 smithi144 ceph-mon[32129]: pgmap v5029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:24 smithi160 ceph-mon[36715]: pgmap v5030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:24 smithi183 ceph-mon[37273]: pgmap v5030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:24 smithi144 ceph-mon[32129]: pgmap v5030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:26 smithi183 ceph-mon[37273]: pgmap v5031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8489.721938 secs 2024-07-26T13:27:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:26 smithi144 ceph-mon[32129]: pgmap v5031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8489.721938 secs 2024-07-26T13:27:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:26 smithi160 ceph-mon[36715]: pgmap v5031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8489.721938 secs 2024-07-26T13:27:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:28 smithi144 ceph-mon[32129]: pgmap v5032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:28 smithi160 ceph-mon[36715]: pgmap v5032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:28 smithi183 ceph-mon[37273]: pgmap v5032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:30 smithi144 ceph-mon[32129]: pgmap v5033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:30 smithi160 ceph-mon[36715]: pgmap v5033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:30 smithi183 ceph-mon[37273]: pgmap v5033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8494.722055 secs 2024-07-26T13:27:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8494.722055 secs 2024-07-26T13:27:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8494.722055 secs 2024-07-26T13:27:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:32 smithi144 ceph-mon[32129]: pgmap v5034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:32 smithi160 ceph-mon[36715]: pgmap v5034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:32 smithi183 ceph-mon[37273]: pgmap v5034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:34.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:27:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:34.383+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:27:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:34.383+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:27:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:34.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:27:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:34.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:34.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:34 smithi183 ceph-mon[37273]: pgmap v5035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:34 smithi144 ceph-mon[32129]: pgmap v5035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:34 smithi160 ceph-mon[36715]: pgmap v5035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:36 smithi144 ceph-mon[32129]: pgmap v5036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8499.722173 secs 2024-07-26T13:27:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:36 smithi160 ceph-mon[36715]: pgmap v5036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8499.722173 secs 2024-07-26T13:27:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:36 smithi183 ceph-mon[37273]: pgmap v5036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8499.722173 secs 2024-07-26T13:27:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:38 smithi144 ceph-mon[32129]: pgmap v5037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:38 smithi160 ceph-mon[36715]: pgmap v5037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:38 smithi183 ceph-mon[37273]: pgmap v5037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:40 smithi144 ceph-mon[32129]: pgmap v5038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:40 smithi160 ceph-mon[36715]: pgmap v5038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:40 smithi183 ceph-mon[37273]: pgmap v5038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8504.722258 secs 2024-07-26T13:27:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8504.722258 secs 2024-07-26T13:27:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8504.722258 secs 2024-07-26T13:27:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:42 smithi144 ceph-mon[32129]: pgmap v5039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:42 smithi160 ceph-mon[36715]: pgmap v5039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:42 smithi183 ceph-mon[37273]: pgmap v5039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:44 smithi144 ceph-mon[32129]: pgmap v5040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:44 smithi160 ceph-mon[36715]: pgmap v5040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:44 smithi183 ceph-mon[37273]: pgmap v5040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:46 smithi144 ceph-mon[32129]: pgmap v5041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8509.722391 secs 2024-07-26T13:27:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:46 smithi160 ceph-mon[36715]: pgmap v5041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8509.722391 secs 2024-07-26T13:27:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:46 smithi183 ceph-mon[37273]: pgmap v5041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8509.722391 secs 2024-07-26T13:27:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:48 smithi144 ceph-mon[32129]: pgmap v5042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:48 smithi160 ceph-mon[36715]: pgmap v5042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:48 smithi183 ceph-mon[37273]: pgmap v5042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:49.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:27:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:49.399+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:27:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:49.399+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:27:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:49.399+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:27:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:27:49.399+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:50 smithi160 ceph-mon[36715]: pgmap v5043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:50 smithi183 ceph-mon[37273]: pgmap v5043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:50.941 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:50 smithi144 ceph-mon[32129]: pgmap v5043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8514.722481 secs 2024-07-26T13:27:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8514.722481 secs 2024-07-26T13:27:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8514.722481 secs 2024-07-26T13:27:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:52 smithi160 ceph-mon[36715]: pgmap v5044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:52 smithi183 ceph-mon[37273]: pgmap v5044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:52 smithi144 ceph-mon[32129]: pgmap v5044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:54 smithi160 ceph-mon[36715]: pgmap v5045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:54 smithi183 ceph-mon[37273]: pgmap v5045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:54 smithi144 ceph-mon[32129]: pgmap v5045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:56 smithi160 ceph-mon[36715]: pgmap v5046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8519.722589 secs 2024-07-26T13:27:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:56 smithi183 ceph-mon[37273]: pgmap v5046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8519.722589 secs 2024-07-26T13:27:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:56 smithi144 ceph-mon[32129]: pgmap v5046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:27:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8519.722589 secs 2024-07-26T13:27:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:58 smithi160 ceph-mon[36715]: pgmap v5047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:58 smithi183 ceph-mon[37273]: pgmap v5047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:58 smithi144 ceph-mon[32129]: pgmap v5047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:27:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:27:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:27:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:27:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:00 smithi160 ceph-mon[36715]: pgmap v5048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:28:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:00 smithi183 ceph-mon[37273]: pgmap v5048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:28:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:00 smithi144 ceph-mon[32129]: pgmap v5048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:28:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:01 smithi160 ceph-mon[36715]: pgmap v5049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:01 smithi160 ceph-mon[36715]: pgmap v5050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8524.722716 secs 2024-07-26T13:28:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:01 smithi183 ceph-mon[37273]: pgmap v5049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:01 smithi183 ceph-mon[37273]: pgmap v5050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8524.722716 secs 2024-07-26T13:28:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:01 smithi144 ceph-mon[32129]: pgmap v5049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:01 smithi144 ceph-mon[32129]: pgmap v5050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8524.722716 secs 2024-07-26T13:28:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:03 smithi160 ceph-mon[36715]: pgmap v5051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:28:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:03 smithi183 ceph-mon[37273]: pgmap v5051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:28:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:03 smithi144 ceph-mon[32129]: pgmap v5051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:28:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:04.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:28:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:04.417+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:28:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:04.417+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:28:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:04.417+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:28:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:04.417+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:05 smithi160 ceph-mon[36715]: pgmap v5052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:28:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:05 smithi183 ceph-mon[37273]: pgmap v5052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:28:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:05 smithi144 ceph-mon[32129]: pgmap v5052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:28:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8529.722844 secs 2024-07-26T13:28:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8529.722844 secs 2024-07-26T13:28:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8529.722844 secs 2024-07-26T13:28:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:07 smithi160 ceph-mon[36715]: pgmap v5053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:07 smithi183 ceph-mon[37273]: pgmap v5053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:07 smithi144 ceph-mon[32129]: pgmap v5053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:09 smithi160 ceph-mon[36715]: pgmap v5054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:09 smithi183 ceph-mon[37273]: pgmap v5054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:09 smithi144 ceph-mon[32129]: pgmap v5054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:28:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:11 smithi183 ceph-mon[37273]: pgmap v5055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:28:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8534.722913 secs 2024-07-26T13:28:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:11 smithi144 ceph-mon[32129]: pgmap v5055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:28:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8534.722913 secs 2024-07-26T13:28:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:11 smithi160 ceph-mon[36715]: pgmap v5055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:28:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8534.722913 secs 2024-07-26T13:28:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:13 smithi183 ceph-mon[37273]: pgmap v5056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:13 smithi144 ceph-mon[32129]: pgmap v5056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:13 smithi160 ceph-mon[36715]: pgmap v5056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:15 smithi183 ceph-mon[37273]: pgmap v5057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:15 smithi144 ceph-mon[32129]: pgmap v5057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:15 smithi160 ceph-mon[36715]: pgmap v5057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8539.723057 secs 2024-07-26T13:28:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8539.723057 secs 2024-07-26T13:28:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8539.723057 secs 2024-07-26T13:28:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:17 smithi183 ceph-mon[37273]: pgmap v5058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:17 smithi144 ceph-mon[32129]: pgmap v5058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:17 smithi160 ceph-mon[36715]: pgmap v5058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:19.433+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:28:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:19.434+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:28:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:19.434+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:28:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:19.434+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:28:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:19.434+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:19 smithi183 ceph-mon[37273]: pgmap v5059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:19 smithi144 ceph-mon[32129]: pgmap v5059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:19 smithi160 ceph-mon[36715]: pgmap v5059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:21 smithi183 ceph-mon[37273]: pgmap v5060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8544.723193 secs 2024-07-26T13:28:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:21 smithi144 ceph-mon[32129]: pgmap v5060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8544.723193 secs 2024-07-26T13:28:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:21 smithi160 ceph-mon[36715]: pgmap v5060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8544.723193 secs 2024-07-26T13:28:23.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:23 smithi160 ceph-mon[36715]: pgmap v5061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:23 smithi183 ceph-mon[37273]: pgmap v5061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:23 smithi144 ceph-mon[32129]: pgmap v5061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:25 smithi183 ceph-mon[37273]: pgmap v5062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:25 smithi144 ceph-mon[32129]: pgmap v5062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:25 smithi160 ceph-mon[36715]: pgmap v5062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8549.723349 secs 2024-07-26T13:28:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8549.723349 secs 2024-07-26T13:28:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8549.723349 secs 2024-07-26T13:28:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:27 smithi183 ceph-mon[37273]: pgmap v5063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:27 smithi144 ceph-mon[32129]: pgmap v5063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:27 smithi160 ceph-mon[36715]: pgmap v5063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:29 smithi183 ceph-mon[37273]: pgmap v5064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:29 smithi144 ceph-mon[32129]: pgmap v5064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:29 smithi160 ceph-mon[36715]: pgmap v5064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:31 smithi183 ceph-mon[37273]: pgmap v5065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8554.723497 secs 2024-07-26T13:28:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:31 smithi144 ceph-mon[32129]: pgmap v5065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8554.723497 secs 2024-07-26T13:28:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:31 smithi160 ceph-mon[36715]: pgmap v5065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8554.723497 secs 2024-07-26T13:28:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:33 smithi144 ceph-mon[32129]: pgmap v5066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:33 smithi160 ceph-mon[36715]: pgmap v5066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:33 smithi183 ceph-mon[37273]: pgmap v5066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:34.439+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:28:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:34.440+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:28:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:34.440+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:28:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:34.440+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:28:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:34.440+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:35.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:35 smithi183 ceph-mon[37273]: pgmap v5067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:35 smithi144 ceph-mon[32129]: pgmap v5067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:35 smithi160 ceph-mon[36715]: pgmap v5067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8559.723585 secs 2024-07-26T13:28:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8559.723585 secs 2024-07-26T13:28:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8559.723585 secs 2024-07-26T13:28:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:37 smithi144 ceph-mon[32129]: pgmap v5068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:37 smithi160 ceph-mon[36715]: pgmap v5068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:37 smithi183 ceph-mon[37273]: pgmap v5068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:39 smithi144 ceph-mon[32129]: pgmap v5069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:39 smithi160 ceph-mon[36715]: pgmap v5069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:39 smithi183 ceph-mon[37273]: pgmap v5069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:41 smithi144 ceph-mon[32129]: pgmap v5070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8564.723722 secs 2024-07-26T13:28:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:41 smithi160 ceph-mon[36715]: pgmap v5070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8564.723722 secs 2024-07-26T13:28:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:41 smithi183 ceph-mon[37273]: pgmap v5070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8564.723722 secs 2024-07-26T13:28:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:43 smithi144 ceph-mon[32129]: pgmap v5071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:43 smithi160 ceph-mon[36715]: pgmap v5071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:43 smithi183 ceph-mon[37273]: pgmap v5071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:45 smithi144 ceph-mon[32129]: pgmap v5072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:45 smithi160 ceph-mon[36715]: pgmap v5072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:45 smithi183 ceph-mon[37273]: pgmap v5072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8569.723843 secs 2024-07-26T13:28:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8569.723843 secs 2024-07-26T13:28:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8569.723843 secs 2024-07-26T13:28:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:47 smithi144 ceph-mon[32129]: pgmap v5073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:47 smithi160 ceph-mon[36715]: pgmap v5073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:47 smithi183 ceph-mon[37273]: pgmap v5073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:49.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:28:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:49.450+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:28:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:49.450+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:28:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:49.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:28:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:28:49.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:49 smithi144 ceph-mon[32129]: pgmap v5074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:49 smithi160 ceph-mon[36715]: pgmap v5074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:49 smithi183 ceph-mon[37273]: pgmap v5074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:51 smithi144 ceph-mon[32129]: pgmap v5075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8574.724000 secs 2024-07-26T13:28:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:51 smithi160 ceph-mon[36715]: pgmap v5075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8574.724000 secs 2024-07-26T13:28:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:51 smithi183 ceph-mon[37273]: pgmap v5075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8574.724000 secs 2024-07-26T13:28:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:53 smithi144 ceph-mon[32129]: pgmap v5076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:53 smithi160 ceph-mon[36715]: pgmap v5076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:53 smithi183 ceph-mon[37273]: pgmap v5076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:28:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:55 smithi160 ceph-mon[36715]: pgmap v5077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:55 smithi183 ceph-mon[37273]: pgmap v5077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:55 smithi144 ceph-mon[32129]: pgmap v5077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:28:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8579.724151 secs 2024-07-26T13:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8579.724151 secs 2024-07-26T13:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8579.724151 secs 2024-07-26T13:28:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:57 smithi160 ceph-mon[36715]: pgmap v5078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:57 smithi183 ceph-mon[37273]: pgmap v5078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:28:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:57 smithi144 ceph-mon[32129]: pgmap v5078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:28:59 smithi160 ceph-mon[36715]: pgmap v5079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:28:59 smithi183 ceph-mon[37273]: pgmap v5079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:28:59 smithi144 ceph-mon[32129]: pgmap v5079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:01 smithi160 ceph-mon[36715]: pgmap v5080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8584.724219 secs 2024-07-26T13:29:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:01 smithi183 ceph-mon[37273]: pgmap v5080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8584.724219 secs 2024-07-26T13:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:01 smithi144 ceph-mon[32129]: pgmap v5080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8584.724219 secs 2024-07-26T13:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:29:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:02 smithi160 ceph-mon[36715]: pgmap v5081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:29:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:02 smithi160 ceph-mon[36715]: pgmap v5082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:29:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:02 smithi183 ceph-mon[37273]: pgmap v5081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:29:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:02 smithi183 ceph-mon[37273]: pgmap v5082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:29:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:02 smithi144 ceph-mon[32129]: pgmap v5081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:29:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:02 smithi144 ceph-mon[32129]: pgmap v5082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:29:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:04.453+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:29:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:04.454+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:29:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:04.454+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:29:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:04.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:29:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:04.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:04 smithi160 ceph-mon[36715]: pgmap v5083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:29:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:04 smithi183 ceph-mon[37273]: pgmap v5083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:29:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:04 smithi144 ceph-mon[32129]: pgmap v5083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:29:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:06 smithi160 ceph-mon[36715]: pgmap v5084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:29:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8589.724348 secs 2024-07-26T13:29:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:06 smithi183 ceph-mon[37273]: pgmap v5084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:29:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8589.724348 secs 2024-07-26T13:29:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:06 smithi144 ceph-mon[32129]: pgmap v5084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:29:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8589.724348 secs 2024-07-26T13:29:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:08 smithi160 ceph-mon[36715]: pgmap v5085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:29:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:08 smithi183 ceph-mon[37273]: pgmap v5085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:29:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:08 smithi144 ceph-mon[32129]: pgmap v5085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:29:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:10 smithi160 ceph-mon[36715]: pgmap v5086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:29:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:10 smithi183 ceph-mon[37273]: pgmap v5086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:29:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:10 smithi144 ceph-mon[32129]: pgmap v5086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:29:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8594.724470 secs 2024-07-26T13:29:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8594.724470 secs 2024-07-26T13:29:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8594.724470 secs 2024-07-26T13:29:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:12 smithi160 ceph-mon[36715]: pgmap v5087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:29:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:12 smithi183 ceph-mon[37273]: pgmap v5087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:29:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:12 smithi144 ceph-mon[32129]: pgmap v5087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:29:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:14 smithi160 ceph-mon[36715]: pgmap v5088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:14 smithi183 ceph-mon[37273]: pgmap v5088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:14 smithi144 ceph-mon[32129]: pgmap v5088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:16 smithi183 ceph-mon[37273]: pgmap v5089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8599.724615 secs 2024-07-26T13:29:17.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:16 smithi144 ceph-mon[32129]: pgmap v5089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:17.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8599.724615 secs 2024-07-26T13:29:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:16 smithi160 ceph-mon[36715]: pgmap v5089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8599.724615 secs 2024-07-26T13:29:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:18 smithi183 ceph-mon[37273]: pgmap v5090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:18 smithi144 ceph-mon[32129]: pgmap v5090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:18 smithi160 ceph-mon[36715]: pgmap v5090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:19.470+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:29:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:19.471+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:29:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:19.471+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:29:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:19.471+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:29:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:19.471+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:20 smithi183 ceph-mon[37273]: pgmap v5091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:20 smithi144 ceph-mon[32129]: pgmap v5091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:20 smithi160 ceph-mon[36715]: pgmap v5091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8604.724748 secs 2024-07-26T13:29:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8604.724748 secs 2024-07-26T13:29:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8604.724748 secs 2024-07-26T13:29:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:22 smithi183 ceph-mon[37273]: pgmap v5092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:23.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:22 smithi160 ceph-mon[36715]: pgmap v5092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:22 smithi144 ceph-mon[32129]: pgmap v5092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:24 smithi183 ceph-mon[37273]: pgmap v5093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:24 smithi144 ceph-mon[32129]: pgmap v5093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:24 smithi160 ceph-mon[36715]: pgmap v5093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:26 smithi183 ceph-mon[37273]: pgmap v5094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8609.724886 secs 2024-07-26T13:29:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:26 smithi144 ceph-mon[32129]: pgmap v5094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8609.724886 secs 2024-07-26T13:29:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:26 smithi160 ceph-mon[36715]: pgmap v5094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8609.724886 secs 2024-07-26T13:29:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:28 smithi183 ceph-mon[37273]: pgmap v5095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:28 smithi144 ceph-mon[32129]: pgmap v5095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:28 smithi160 ceph-mon[36715]: pgmap v5095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:30 smithi183 ceph-mon[37273]: pgmap v5096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:30 smithi144 ceph-mon[32129]: pgmap v5096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:30 smithi160 ceph-mon[36715]: pgmap v5096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8614.725003 secs 2024-07-26T13:29:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8614.725003 secs 2024-07-26T13:29:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8614.725003 secs 2024-07-26T13:29:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:32 smithi183 ceph-mon[37273]: pgmap v5097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:32 smithi144 ceph-mon[32129]: pgmap v5097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:32 smithi160 ceph-mon[36715]: pgmap v5097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:34.487+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:29:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:34.488+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:29:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:34.488+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:29:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:34.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:29:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:34.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:34 smithi144 ceph-mon[32129]: pgmap v5098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:34 smithi160 ceph-mon[36715]: pgmap v5098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:34 smithi183 ceph-mon[37273]: pgmap v5098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:37.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:36 smithi183 ceph-mon[37273]: pgmap v5099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:37.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8619.725138 secs 2024-07-26T13:29:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:36 smithi144 ceph-mon[32129]: pgmap v5099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8619.725138 secs 2024-07-26T13:29:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:36 smithi160 ceph-mon[36715]: pgmap v5099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8619.725138 secs 2024-07-26T13:29:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:38 smithi144 ceph-mon[32129]: pgmap v5100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:38 smithi160 ceph-mon[36715]: pgmap v5100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:38 smithi183 ceph-mon[37273]: pgmap v5100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:40 smithi144 ceph-mon[32129]: pgmap v5101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:40 smithi160 ceph-mon[36715]: pgmap v5101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:40 smithi183 ceph-mon[37273]: pgmap v5101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8624.725267 secs 2024-07-26T13:29:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8624.725267 secs 2024-07-26T13:29:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8624.725267 secs 2024-07-26T13:29:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:42 smithi144 ceph-mon[32129]: pgmap v5102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:42 smithi160 ceph-mon[36715]: pgmap v5102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:43.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:42 smithi183 ceph-mon[37273]: pgmap v5102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:45.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:44 smithi144 ceph-mon[32129]: pgmap v5103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:44 smithi160 ceph-mon[36715]: pgmap v5103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:44 smithi183 ceph-mon[37273]: pgmap v5103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8629.725378 secs 2024-07-26T13:29:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8629.725378 secs 2024-07-26T13:29:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8629.725378 secs 2024-07-26T13:29:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:46 smithi144 ceph-mon[32129]: pgmap v5104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:46 smithi160 ceph-mon[36715]: pgmap v5104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:46 smithi183 ceph-mon[37273]: pgmap v5104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:47.627+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:29:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:47.627+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:29:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:47.627+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:29:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:47.627+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:29:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:48 smithi144 ceph-mon[32129]: pgmap v5105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:48 smithi160 ceph-mon[36715]: pgmap v5105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:48 smithi183 ceph-mon[37273]: pgmap v5105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:49.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:49.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:29:49.505 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:49.503+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:29:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:49.503+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:29:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:49.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:29:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:29:49.504+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:50 smithi144 ceph-mon[32129]: pgmap v5106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8634.725505 secs 2024-07-26T13:29:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:50 smithi160 ceph-mon[36715]: pgmap v5106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8634.725505 secs 2024-07-26T13:29:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:50 smithi183 ceph-mon[37273]: pgmap v5106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8634.725505 secs 2024-07-26T13:29:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:52 smithi144 ceph-mon[32129]: pgmap v5107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:52 smithi160 ceph-mon[36715]: pgmap v5107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:52 smithi183 ceph-mon[37273]: pgmap v5107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:29:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:54 smithi144 ceph-mon[32129]: pgmap v5108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:54 smithi160 ceph-mon[36715]: pgmap v5108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:54 smithi183 ceph-mon[37273]: pgmap v5108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8639.725650 secs 2024-07-26T13:29:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8639.725650 secs 2024-07-26T13:29:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8639.725650 secs 2024-07-26T13:29:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:56 smithi160 ceph-mon[36715]: pgmap v5109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:56 smithi183 ceph-mon[37273]: pgmap v5109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:56 smithi144 ceph-mon[32129]: pgmap v5109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:29:59.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:29:58 smithi160 ceph-mon[36715]: pgmap v5110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:29:58 smithi183 ceph-mon[37273]: pgmap v5110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:29:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:29:58 smithi144 ceph-mon[32129]: pgmap v5110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:00 smithi160 ceph-mon[36715]: pgmap v5111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8644.725797 secs 2024-07-26T13:30:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:00 smithi183 ceph-mon[37273]: pgmap v5111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8644.725797 secs 2024-07-26T13:30:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:00 smithi144 ceph-mon[32129]: pgmap v5111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8644.725797 secs 2024-07-26T13:30:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:03 smithi160 ceph-mon[36715]: pgmap v5112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:02 smithi183 ceph-mon[37273]: pgmap v5112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:03.412 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:02 smithi144 ceph-mon[32129]: pgmap v5112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:04.520+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:30:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:04.521+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:30:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:04.521+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:30:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:04.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:30:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:04.521+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:05.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:05 smithi144 ceph-mon[32129]: pgmap v5113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:05 smithi160 ceph-mon[36715]: pgmap v5113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:05 smithi183 ceph-mon[37273]: pgmap v5113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:06 smithi183 ceph-mon[37273]: pgmap v5114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8649.725920 secs 2024-07-26T13:30:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:06 smithi144 ceph-mon[32129]: pgmap v5114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8649.725920 secs 2024-07-26T13:30:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:06 smithi160 ceph-mon[36715]: pgmap v5114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8649.725920 secs 2024-07-26T13:30:09.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: pgmap v5115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:09.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:30:09.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:09.140 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:09.140 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: pgmap v5116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:30:09.140 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: pgmap v5117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:09.140 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: pgmap v5115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: pgmap v5116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: pgmap v5117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:09.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: pgmap v5115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: pgmap v5116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: pgmap v5117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:09.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:30:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:10 smithi144 ceph-mon[32129]: pgmap v5118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:30:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8654.726069 secs 2024-07-26T13:30:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:10 smithi160 ceph-mon[36715]: pgmap v5118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:30:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8654.726069 secs 2024-07-26T13:30:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:10 smithi183 ceph-mon[37273]: pgmap v5118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:30:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8654.726069 secs 2024-07-26T13:30:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:12 smithi144 ceph-mon[32129]: pgmap v5119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:12 smithi160 ceph-mon[36715]: pgmap v5119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:12 smithi183 ceph-mon[37273]: pgmap v5119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:14 smithi144 ceph-mon[32129]: pgmap v5120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:14 smithi160 ceph-mon[36715]: pgmap v5120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:14 smithi183 ceph-mon[37273]: pgmap v5120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:30:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8659.726220 secs 2024-07-26T13:30:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8659.726220 secs 2024-07-26T13:30:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8659.726220 secs 2024-07-26T13:30:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:16 smithi144 ceph-mon[32129]: pgmap v5121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T13:30:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:16 smithi160 ceph-mon[36715]: pgmap v5121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T13:30:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:16 smithi183 ceph-mon[37273]: pgmap v5121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T13:30:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:18 smithi160 ceph-mon[36715]: pgmap v5122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:30:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:18 smithi183 ceph-mon[37273]: pgmap v5122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:30:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:18 smithi144 ceph-mon[32129]: pgmap v5122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:30:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:19.522+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:30:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:19.524+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:30:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:19.524+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:30:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:19.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:30:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:19.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:20 smithi160 ceph-mon[36715]: pgmap v5123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8664.726349 secs 2024-07-26T13:30:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:20 smithi183 ceph-mon[37273]: pgmap v5123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8664.726349 secs 2024-07-26T13:30:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:20 smithi144 ceph-mon[32129]: pgmap v5123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8664.726349 secs 2024-07-26T13:30:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:22 smithi160 ceph-mon[36715]: pgmap v5124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:22 smithi183 ceph-mon[37273]: pgmap v5124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:22 smithi144 ceph-mon[32129]: pgmap v5124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:24 smithi160 ceph-mon[36715]: pgmap v5125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:24 smithi183 ceph-mon[37273]: pgmap v5125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:24 smithi144 ceph-mon[32129]: pgmap v5125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8669.726471 secs 2024-07-26T13:30:26.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8669.726471 secs 2024-07-26T13:30:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8669.726471 secs 2024-07-26T13:30:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:27 smithi160 ceph-mon[36715]: pgmap v5126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:27 smithi183 ceph-mon[37273]: pgmap v5126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:27 smithi144 ceph-mon[32129]: pgmap v5126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:29 smithi160 ceph-mon[36715]: pgmap v5127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:29 smithi183 ceph-mon[37273]: pgmap v5127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:29 smithi144 ceph-mon[32129]: pgmap v5127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:31 smithi160 ceph-mon[36715]: pgmap v5128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8674.726547 secs 2024-07-26T13:30:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:31 smithi183 ceph-mon[37273]: pgmap v5128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8674.726547 secs 2024-07-26T13:30:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:31 smithi144 ceph-mon[32129]: pgmap v5128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8674.726547 secs 2024-07-26T13:30:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:33 smithi160 ceph-mon[36715]: pgmap v5129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:33 smithi183 ceph-mon[37273]: pgmap v5129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:33 smithi144 ceph-mon[32129]: pgmap v5129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:34.540+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:30:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:34.541+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:30:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:34.541+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:30:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:34.541+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:30:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:34.541+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:35.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:35 smithi160 ceph-mon[36715]: pgmap v5130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:35 smithi183 ceph-mon[37273]: pgmap v5130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:35 smithi144 ceph-mon[32129]: pgmap v5130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8679.726673 secs 2024-07-26T13:30:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8679.726673 secs 2024-07-26T13:30:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8679.726673 secs 2024-07-26T13:30:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:37 smithi160 ceph-mon[36715]: pgmap v5131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:37 smithi183 ceph-mon[37273]: pgmap v5131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:37 smithi144 ceph-mon[32129]: pgmap v5131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:38 smithi160 ceph-mon[36715]: pgmap v5132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:38 smithi183 ceph-mon[37273]: pgmap v5132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:38 smithi144 ceph-mon[32129]: pgmap v5132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:40 smithi144 ceph-mon[32129]: pgmap v5133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8684.726799 secs 2024-07-26T13:30:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:40 smithi160 ceph-mon[36715]: pgmap v5133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8684.726799 secs 2024-07-26T13:30:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:40 smithi183 ceph-mon[37273]: pgmap v5133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8684.726799 secs 2024-07-26T13:30:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:42 smithi144 ceph-mon[32129]: pgmap v5134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:42 smithi160 ceph-mon[36715]: pgmap v5134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:42 smithi183 ceph-mon[37273]: pgmap v5134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:44 smithi144 ceph-mon[32129]: pgmap v5135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:44 smithi160 ceph-mon[36715]: pgmap v5135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:44 smithi183 ceph-mon[37273]: pgmap v5135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8689.726920 secs 2024-07-26T13:30:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8689.726920 secs 2024-07-26T13:30:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8689.726920 secs 2024-07-26T13:30:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:46 smithi160 ceph-mon[36715]: pgmap v5136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:46 smithi183 ceph-mon[37273]: pgmap v5136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:46 smithi144 ceph-mon[32129]: pgmap v5136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:48 smithi160 ceph-mon[36715]: pgmap v5137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:48 smithi183 ceph-mon[37273]: pgmap v5137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:48 smithi144 ceph-mon[32129]: pgmap v5137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:49.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:30:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:49.551+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:30:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:49.551+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:30:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:49.551+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:30:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:30:49.551+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:50 smithi160 ceph-mon[36715]: pgmap v5138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:50 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8694.727047 secs 2024-07-26T13:30:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:50 smithi183 ceph-mon[37273]: pgmap v5138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:50 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8694.727047 secs 2024-07-26T13:30:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:50 smithi144 ceph-mon[32129]: pgmap v5138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:50 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8694.727047 secs 2024-07-26T13:30:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:52 smithi160 ceph-mon[36715]: pgmap v5139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:52 smithi183 ceph-mon[37273]: pgmap v5139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:52 smithi144 ceph-mon[32129]: pgmap v5139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:30:55.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:55 smithi160 ceph-mon[36715]: pgmap v5140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:55 smithi183 ceph-mon[37273]: pgmap v5140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:55 smithi144 ceph-mon[32129]: pgmap v5140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8699.727180 secs 2024-07-26T13:30:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8699.727180 secs 2024-07-26T13:30:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8699.727180 secs 2024-07-26T13:30:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:57 smithi160 ceph-mon[36715]: pgmap v5141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:57.310 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:57 smithi144 ceph-mon[32129]: pgmap v5141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:57 smithi183 ceph-mon[37273]: pgmap v5141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:30:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:30:59 smithi160 ceph-mon[36715]: pgmap v5142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:30:59 smithi183 ceph-mon[37273]: pgmap v5142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:30:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:30:59 smithi144 ceph-mon[32129]: pgmap v5142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:01 smithi160 ceph-mon[36715]: pgmap v5143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8704.727337 secs 2024-07-26T13:31:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:01 smithi183 ceph-mon[37273]: pgmap v5143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8704.727337 secs 2024-07-26T13:31:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:01 smithi144 ceph-mon[32129]: pgmap v5143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8704.727337 secs 2024-07-26T13:31:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:03 smithi160 ceph-mon[36715]: pgmap v5144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:03 smithi183 ceph-mon[37273]: pgmap v5144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:03 smithi144 ceph-mon[32129]: pgmap v5144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:04.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:31:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:04.569+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:31:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:04.569+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:31:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:04.569+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:31:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:04.569+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:05 smithi160 ceph-mon[36715]: pgmap v5145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:05 smithi183 ceph-mon[37273]: pgmap v5145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:05 smithi144 ceph-mon[32129]: pgmap v5145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8709.727460 secs 2024-07-26T13:31:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8709.727460 secs 2024-07-26T13:31:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8709.727460 secs 2024-07-26T13:31:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:07 smithi160 ceph-mon[36715]: pgmap v5146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:07 smithi183 ceph-mon[37273]: pgmap v5146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:07 smithi144 ceph-mon[32129]: pgmap v5146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:08 smithi183 ceph-mon[37273]: pgmap v5147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:08 smithi144 ceph-mon[32129]: pgmap v5147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:08 smithi160 ceph-mon[36715]: pgmap v5147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.337 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.337 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:31:09.337 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:09 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:31:09.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:09 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:09 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:31:10.304 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:10 smithi183 ceph-mon[37273]: pgmap v5148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:10 smithi144 ceph-mon[32129]: pgmap v5148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:10 smithi160 ceph-mon[36715]: pgmap v5148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:11.280 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8714.727596 secs 2024-07-26T13:31:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8714.727596 secs 2024-07-26T13:31:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8714.727596 secs 2024-07-26T13:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: pgmap v5149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: pgmap v5150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:12.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: pgmap v5149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: pgmap v5150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:13.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: pgmap v5149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: pgmap v5150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:13.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:31:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:14 smithi160 ceph-mon[36715]: pgmap v5151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:14 smithi183 ceph-mon[37273]: pgmap v5151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:14 smithi144 ceph-mon[32129]: pgmap v5151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:16 smithi160 ceph-mon[36715]: pgmap v5152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8719.727679 secs 2024-07-26T13:31:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:16 smithi183 ceph-mon[37273]: pgmap v5152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8719.727679 secs 2024-07-26T13:31:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:16 smithi144 ceph-mon[32129]: pgmap v5152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8719.727679 secs 2024-07-26T13:31:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:18 smithi183 ceph-mon[37273]: pgmap v5153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:18 smithi144 ceph-mon[32129]: pgmap v5153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:18 smithi160 ceph-mon[36715]: pgmap v5153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:19.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:19.580+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:31:19.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:19.581+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:31:19.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:19.581+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:31:19.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:19.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:31:19.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:19.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:20 smithi183 ceph-mon[37273]: pgmap v5154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:20 smithi144 ceph-mon[32129]: pgmap v5154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:20 smithi160 ceph-mon[36715]: pgmap v5154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8724.727807 secs 2024-07-26T13:31:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8724.727807 secs 2024-07-26T13:31:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8724.727807 secs 2024-07-26T13:31:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:22 smithi183 ceph-mon[37273]: pgmap v5155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:31:23.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:22 smithi160 ceph-mon[36715]: pgmap v5155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:31:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:22 smithi144 ceph-mon[32129]: pgmap v5155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:31:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:24 smithi183 ceph-mon[37273]: pgmap v5156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:24 smithi144 ceph-mon[32129]: pgmap v5156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:24 smithi160 ceph-mon[36715]: pgmap v5156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8729.727955 secs 2024-07-26T13:31:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:26 smithi183 ceph-mon[37273]: pgmap v5157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8729.727955 secs 2024-07-26T13:31:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:26 smithi144 ceph-mon[32129]: pgmap v5157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8729.727955 secs 2024-07-26T13:31:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:26 smithi160 ceph-mon[36715]: pgmap v5157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:28 smithi183 ceph-mon[37273]: pgmap v5158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:28 smithi144 ceph-mon[32129]: pgmap v5158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:28 smithi160 ceph-mon[36715]: pgmap v5158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:30 smithi183 ceph-mon[37273]: pgmap v5159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:30 smithi144 ceph-mon[32129]: pgmap v5159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:30 smithi160 ceph-mon[36715]: pgmap v5159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8734.728092 secs 2024-07-26T13:31:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8734.728092 secs 2024-07-26T13:31:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8734.728092 secs 2024-07-26T13:31:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:32 smithi183 ceph-mon[37273]: pgmap v5160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:32 smithi144 ceph-mon[32129]: pgmap v5160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:32 smithi160 ceph-mon[36715]: pgmap v5160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:34.586+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:31:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:34.587+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:31:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:34.587+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:31:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:34.587+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:31:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:34.587+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:34 smithi183 ceph-mon[37273]: pgmap v5161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:34 smithi144 ceph-mon[32129]: pgmap v5161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:34 smithi160 ceph-mon[36715]: pgmap v5161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8739.728192 secs 2024-07-26T13:31:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:36 smithi183 ceph-mon[37273]: pgmap v5162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8739.728192 secs 2024-07-26T13:31:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:36 smithi144 ceph-mon[32129]: pgmap v5162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8739.728192 secs 2024-07-26T13:31:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:36 smithi160 ceph-mon[36715]: pgmap v5162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:38 smithi144 ceph-mon[32129]: pgmap v5163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:38 smithi160 ceph-mon[36715]: pgmap v5163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:38 smithi183 ceph-mon[37273]: pgmap v5163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:41.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:40 smithi183 ceph-mon[37273]: pgmap v5164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:40 smithi144 ceph-mon[32129]: pgmap v5164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:40 smithi160 ceph-mon[36715]: pgmap v5164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8744.728339 secs 2024-07-26T13:31:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8744.728339 secs 2024-07-26T13:31:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8744.728339 secs 2024-07-26T13:31:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:42 smithi144 ceph-mon[32129]: pgmap v5165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:42 smithi160 ceph-mon[36715]: pgmap v5165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:42 smithi183 ceph-mon[37273]: pgmap v5165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:31:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:44 smithi144 ceph-mon[32129]: pgmap v5166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:44 smithi160 ceph-mon[36715]: pgmap v5166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:44 smithi183 ceph-mon[37273]: pgmap v5166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8749.728440 secs 2024-07-26T13:31:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:46 smithi144 ceph-mon[32129]: pgmap v5167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8749.728440 secs 2024-07-26T13:31:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:46 smithi160 ceph-mon[36715]: pgmap v5167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8749.728440 secs 2024-07-26T13:31:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:46 smithi183 ceph-mon[37273]: pgmap v5167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:48 smithi144 ceph-mon[32129]: pgmap v5168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:48 smithi160 ceph-mon[36715]: pgmap v5168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:48 smithi183 ceph-mon[37273]: pgmap v5168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:31:49.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:49.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:31:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:49.601+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:31:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:49.601+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:31:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:49.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:31:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:31:49.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:50 smithi144 ceph-mon[32129]: pgmap v5169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:50 smithi160 ceph-mon[36715]: pgmap v5169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:50 smithi183 ceph-mon[37273]: pgmap v5169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:31:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8754.728569 secs 2024-07-26T13:31:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8754.728569 secs 2024-07-26T13:31:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8754.728569 secs 2024-07-26T13:31:53.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:52 smithi160 ceph-mon[36715]: pgmap v5170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:31:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:52 smithi144 ceph-mon[32129]: pgmap v5170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:31:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:52 smithi183 ceph-mon[37273]: pgmap v5170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:31:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:54 smithi144 ceph-mon[32129]: pgmap v5171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:54 smithi160 ceph-mon[36715]: pgmap v5171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:54 smithi183 ceph-mon[37273]: pgmap v5171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8759.728687 secs 2024-07-26T13:31:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8759.728687 secs 2024-07-26T13:31:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8759.728687 secs 2024-07-26T13:31:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:56 smithi144 ceph-mon[32129]: pgmap v5172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:56 smithi160 ceph-mon[36715]: pgmap v5172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:56 smithi183 ceph-mon[37273]: pgmap v5172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:31:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:31:58 smithi144 ceph-mon[32129]: pgmap v5173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:31:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:31:58 smithi160 ceph-mon[36715]: pgmap v5173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:31:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:31:58 smithi183 ceph-mon[37273]: pgmap v5173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:00 smithi160 ceph-mon[36715]: pgmap v5174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:32:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8764.728808 secs 2024-07-26T13:32:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:00 smithi183 ceph-mon[37273]: pgmap v5174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:32:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8764.728808 secs 2024-07-26T13:32:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:00 smithi144 ceph-mon[32129]: pgmap v5174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:32:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8764.728808 secs 2024-07-26T13:32:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:02 smithi160 ceph-mon[36715]: pgmap v5175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:32:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:02 smithi183 ceph-mon[37273]: pgmap v5175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:32:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:02 smithi144 ceph-mon[32129]: pgmap v5175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:32:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:04.618+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:32:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:04.619+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:32:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:04.619+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:32:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:04.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:32:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:04.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:04 smithi160 ceph-mon[36715]: pgmap v5176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:04 smithi183 ceph-mon[37273]: pgmap v5176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:04 smithi144 ceph-mon[32129]: pgmap v5176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8769.728929 secs 2024-07-26T13:32:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8769.728929 secs 2024-07-26T13:32:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8769.728929 secs 2024-07-26T13:32:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:06 smithi160 ceph-mon[36715]: pgmap v5177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:06 smithi183 ceph-mon[37273]: pgmap v5177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:06 smithi144 ceph-mon[32129]: pgmap v5177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:09 smithi160 ceph-mon[36715]: pgmap v5178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:09 smithi183 ceph-mon[37273]: pgmap v5178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:09 smithi144 ceph-mon[32129]: pgmap v5178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:11 smithi160 ceph-mon[36715]: pgmap v5179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8774.729058 secs 2024-07-26T13:32:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:11 smithi183 ceph-mon[37273]: pgmap v5179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8774.729058 secs 2024-07-26T13:32:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:11 smithi144 ceph-mon[32129]: pgmap v5179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8774.729058 secs 2024-07-26T13:32:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:12.468 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:13 smithi160 ceph-mon[36715]: pgmap v5180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:32:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:13 smithi183 ceph-mon[37273]: pgmap v5180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:32:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:13 smithi144 ceph-mon[32129]: pgmap v5180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:32:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:14 smithi160 ceph-mon[36715]: pgmap v5181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:32:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:14 smithi160 ceph-mon[36715]: pgmap v5182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:14 smithi183 ceph-mon[37273]: pgmap v5181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:32:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:14 smithi183 ceph-mon[37273]: pgmap v5182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:14 smithi144 ceph-mon[32129]: pgmap v5181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:32:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:14 smithi144 ceph-mon[32129]: pgmap v5182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:16 smithi160 ceph-mon[36715]: pgmap v5183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8779.729185 secs 2024-07-26T13:32:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:16 smithi183 ceph-mon[37273]: pgmap v5183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8779.729185 secs 2024-07-26T13:32:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:16 smithi144 ceph-mon[32129]: pgmap v5183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:32:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8779.729185 secs 2024-07-26T13:32:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:18 smithi160 ceph-mon[36715]: pgmap v5184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:32:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:18 smithi183 ceph-mon[37273]: pgmap v5184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:32:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:18 smithi144 ceph-mon[32129]: pgmap v5184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:32:19.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:19.621+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:32:19.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:19.622+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:32:19.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:19.622+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:32:19.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:19.622+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:32:19.623 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:19.622+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:20 smithi160 ceph-mon[36715]: pgmap v5185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:32:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:20 smithi183 ceph-mon[37273]: pgmap v5185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:32:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:20 smithi144 ceph-mon[32129]: pgmap v5185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:32:21.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8784.729329 secs 2024-07-26T13:32:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8784.729329 secs 2024-07-26T13:32:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8784.729329 secs 2024-07-26T13:32:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:22 smithi183 ceph-mon[37273]: pgmap v5186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:32:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:22 smithi144 ceph-mon[32129]: pgmap v5186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:32:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:22 smithi160 ceph-mon[36715]: pgmap v5186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:32:24.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:24 smithi160 ceph-mon[36715]: pgmap v5187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:32:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:24 smithi183 ceph-mon[37273]: pgmap v5187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:32:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:24 smithi144 ceph-mon[32129]: pgmap v5187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:32:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:25 smithi160 ceph-mon[36715]: pgmap v5188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:25 smithi183 ceph-mon[37273]: pgmap v5188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:25 smithi144 ceph-mon[32129]: pgmap v5188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8789.729480 secs 2024-07-26T13:32:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8789.729480 secs 2024-07-26T13:32:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8789.729480 secs 2024-07-26T13:32:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:27 smithi160 ceph-mon[36715]: pgmap v5189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:27 smithi183 ceph-mon[37273]: pgmap v5189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:27 smithi144 ceph-mon[32129]: pgmap v5189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:30 smithi160 ceph-mon[36715]: pgmap v5190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:30 smithi183 ceph-mon[37273]: pgmap v5190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:30 smithi144 ceph-mon[32129]: pgmap v5190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8794.729609 secs 2024-07-26T13:32:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8794.729609 secs 2024-07-26T13:32:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8794.729609 secs 2024-07-26T13:32:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:32 smithi160 ceph-mon[36715]: pgmap v5191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:32.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:32 smithi144 ceph-mon[32129]: pgmap v5191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:32 smithi183 ceph-mon[37273]: pgmap v5191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:34 smithi160 ceph-mon[36715]: pgmap v5192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:34 smithi183 ceph-mon[37273]: pgmap v5192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:34 smithi144 ceph-mon[32129]: pgmap v5192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:34.636+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:32:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:34.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:32:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:34.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:32:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:34.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:32:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:34.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:36 smithi160 ceph-mon[36715]: pgmap v5193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8799.729676 secs 2024-07-26T13:32:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:36 smithi183 ceph-mon[37273]: pgmap v5193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8799.729676 secs 2024-07-26T13:32:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:36 smithi144 ceph-mon[32129]: pgmap v5193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8799.729676 secs 2024-07-26T13:32:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:38 smithi160 ceph-mon[36715]: pgmap v5194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:38 smithi183 ceph-mon[37273]: pgmap v5194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:38 smithi144 ceph-mon[32129]: pgmap v5194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:40 smithi160 ceph-mon[36715]: pgmap v5195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:40 smithi183 ceph-mon[37273]: pgmap v5195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:40 smithi144 ceph-mon[32129]: pgmap v5195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8804.729810 secs 2024-07-26T13:32:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8804.729810 secs 2024-07-26T13:32:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8804.729810 secs 2024-07-26T13:32:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:42 smithi183 ceph-mon[37273]: pgmap v5196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:42 smithi144 ceph-mon[32129]: pgmap v5196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:42 smithi160 ceph-mon[36715]: pgmap v5196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:44 smithi183 ceph-mon[37273]: pgmap v5197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:44 smithi144 ceph-mon[32129]: pgmap v5197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:44 smithi160 ceph-mon[36715]: pgmap v5197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:45 smithi183 ceph-mon[37273]: pgmap v5198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:45 smithi144 ceph-mon[32129]: pgmap v5198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:45 smithi160 ceph-mon[36715]: pgmap v5198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:46.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8809.729941 secs 2024-07-26T13:32:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8809.729941 secs 2024-07-26T13:32:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8809.729941 secs 2024-07-26T13:32:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:47 smithi144 ceph-mon[32129]: pgmap v5199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:47 smithi183 ceph-mon[37273]: pgmap v5199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:47 smithi160 ceph-mon[36715]: pgmap v5199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:47.659+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:32:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:47.659+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:32:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:47.659+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:32:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:47.659+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:32:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:49.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:32:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:49.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:32:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:49.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:32:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:49.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:32:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:32:49.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:50 smithi160 ceph-mon[36715]: pgmap v5200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:50 smithi183 ceph-mon[37273]: pgmap v5200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:50 smithi144 ceph-mon[32129]: pgmap v5200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8814.730088 secs 2024-07-26T13:32:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8814.730088 secs 2024-07-26T13:32:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8814.730088 secs 2024-07-26T13:32:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:52 smithi160 ceph-mon[36715]: pgmap v5201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:52 smithi183 ceph-mon[37273]: pgmap v5201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:52 smithi144 ceph-mon[32129]: pgmap v5201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:54 smithi160 ceph-mon[36715]: pgmap v5202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:54 smithi183 ceph-mon[37273]: pgmap v5202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:54 smithi144 ceph-mon[32129]: pgmap v5202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:32:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:56 smithi160 ceph-mon[36715]: pgmap v5203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8819.730202 secs 2024-07-26T13:32:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:56 smithi183 ceph-mon[37273]: pgmap v5203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8819.730202 secs 2024-07-26T13:32:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:56 smithi144 ceph-mon[32129]: pgmap v5203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:32:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8819.730202 secs 2024-07-26T13:32:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:32:58 smithi183 ceph-mon[37273]: pgmap v5204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:32:58 smithi144 ceph-mon[32129]: pgmap v5204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:32:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:32:58 smithi160 ceph-mon[36715]: pgmap v5204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:00 smithi183 ceph-mon[37273]: pgmap v5205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:00 smithi144 ceph-mon[32129]: pgmap v5205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:00 smithi160 ceph-mon[36715]: pgmap v5205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:01.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8824.730331 secs 2024-07-26T13:33:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8824.730331 secs 2024-07-26T13:33:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8824.730331 secs 2024-07-26T13:33:02.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:02 smithi144 ceph-mon[32129]: pgmap v5206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:02 smithi183 ceph-mon[37273]: pgmap v5206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:02 smithi160 ceph-mon[36715]: pgmap v5206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:03 smithi183 ceph-mon[37273]: pgmap v5207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:03 smithi144 ceph-mon[32129]: pgmap v5207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:03 smithi160 ceph-mon[36715]: pgmap v5207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:04.670+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:33:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:04.671+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:33:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:04.671+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:33:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:04.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:33:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:04.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:06 smithi160 ceph-mon[36715]: pgmap v5208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8829.730427 secs 2024-07-26T13:33:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:06 smithi183 ceph-mon[37273]: pgmap v5208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8829.730427 secs 2024-07-26T13:33:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:06 smithi144 ceph-mon[32129]: pgmap v5208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8829.730427 secs 2024-07-26T13:33:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:08 smithi160 ceph-mon[36715]: pgmap v5209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:08 smithi183 ceph-mon[37273]: pgmap v5209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:08 smithi144 ceph-mon[32129]: pgmap v5209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:10 smithi160 ceph-mon[36715]: pgmap v5210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:10 smithi183 ceph-mon[37273]: pgmap v5210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:10 smithi144 ceph-mon[32129]: pgmap v5210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8834.730514 secs 2024-07-26T13:33:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8834.730514 secs 2024-07-26T13:33:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8834.730514 secs 2024-07-26T13:33:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:12 smithi183 ceph-mon[37273]: pgmap v5211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:12 smithi144 ceph-mon[32129]: pgmap v5211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:12 smithi160 ceph-mon[36715]: pgmap v5211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:14 smithi183 ceph-mon[37273]: pgmap v5212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:14 smithi144 ceph-mon[32129]: pgmap v5212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:14 smithi160 ceph-mon[36715]: pgmap v5212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:15 smithi160 ceph-mon[36715]: pgmap v5213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:15 smithi160 ceph-mon[36715]: pgmap v5214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:15 smithi183 ceph-mon[37273]: pgmap v5213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:15 smithi183 ceph-mon[37273]: pgmap v5214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:15 smithi144 ceph-mon[32129]: pgmap v5213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:15 smithi144 ceph-mon[32129]: pgmap v5214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:33:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8839.730630 secs 2024-07-26T13:33:16.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8839.730630 secs 2024-07-26T13:33:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8839.730630 secs 2024-07-26T13:33:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:17 smithi160 ceph-mon[36715]: pgmap v5215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:17 smithi183 ceph-mon[37273]: pgmap v5215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:17 smithi144 ceph-mon[32129]: pgmap v5215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:19.687+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:33:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:19.688+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:33:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:19.688+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:33:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:19.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:33:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:19.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:19 smithi183 ceph-mon[37273]: pgmap v5216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:19 smithi144 ceph-mon[32129]: pgmap v5216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:19 smithi160 ceph-mon[36715]: pgmap v5216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8844.730772 secs 2024-07-26T13:33:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:21 smithi144 ceph-mon[32129]: pgmap v5217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8844.730772 secs 2024-07-26T13:33:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:21 smithi160 ceph-mon[36715]: pgmap v5217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8844.730772 secs 2024-07-26T13:33:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:21 smithi183 ceph-mon[37273]: pgmap v5217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:23 smithi144 ceph-mon[32129]: pgmap v5218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:23 smithi160 ceph-mon[36715]: pgmap v5218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:23 smithi183 ceph-mon[37273]: pgmap v5218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:25 smithi144 ceph-mon[32129]: pgmap v5219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:25 smithi160 ceph-mon[36715]: pgmap v5219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:25 smithi183 ceph-mon[37273]: pgmap v5219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:33:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8849.730923 secs 2024-07-26T13:33:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8849.730923 secs 2024-07-26T13:33:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8849.730923 secs 2024-07-26T13:33:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:27 smithi144 ceph-mon[32129]: pgmap v5220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:27 smithi160 ceph-mon[36715]: pgmap v5220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:27 smithi183 ceph-mon[37273]: pgmap v5220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:33:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:29 smithi144 ceph-mon[32129]: pgmap v5221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:33:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:29 smithi160 ceph-mon[36715]: pgmap v5221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:33:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:29 smithi183 ceph-mon[37273]: pgmap v5221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:33:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8854.731069 secs 2024-07-26T13:33:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:31 smithi144 ceph-mon[32129]: pgmap v5222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8854.731069 secs 2024-07-26T13:33:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:31 smithi160 ceph-mon[36715]: pgmap v5222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8854.731069 secs 2024-07-26T13:33:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:31 smithi183 ceph-mon[37273]: pgmap v5222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:33 smithi144 ceph-mon[32129]: pgmap v5223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:33 smithi160 ceph-mon[36715]: pgmap v5223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:33 smithi183 ceph-mon[37273]: pgmap v5223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:34.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:33:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:34.705+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:33:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:34.705+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:33:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:34.705+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:33:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:34.705+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:35 smithi144 ceph-mon[32129]: pgmap v5224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:35 smithi160 ceph-mon[36715]: pgmap v5224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:35 smithi183 ceph-mon[37273]: pgmap v5224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8859.731197 secs 2024-07-26T13:33:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8859.731197 secs 2024-07-26T13:33:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8859.731197 secs 2024-07-26T13:33:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:37 smithi144 ceph-mon[32129]: pgmap v5225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:37 smithi160 ceph-mon[36715]: pgmap v5225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:37 smithi183 ceph-mon[37273]: pgmap v5225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:39 smithi144 ceph-mon[32129]: pgmap v5226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:39 smithi160 ceph-mon[36715]: pgmap v5226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:39 smithi183 ceph-mon[37273]: pgmap v5226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:41.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:40 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8864.731345 secs 2024-07-26T13:33:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:40 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8864.731345 secs 2024-07-26T13:33:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:40 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8864.731345 secs 2024-07-26T13:33:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:41 smithi160 ceph-mon[36715]: pgmap v5227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:41 smithi183 ceph-mon[37273]: pgmap v5227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:41 smithi144 ceph-mon[32129]: pgmap v5227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:43 smithi160 ceph-mon[36715]: pgmap v5228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:43 smithi183 ceph-mon[37273]: pgmap v5228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:43 smithi144 ceph-mon[32129]: pgmap v5228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:46.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:45 smithi144 ceph-mon[32129]: pgmap v5229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:46.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8869.731471 secs 2024-07-26T13:33:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:45 smithi160 ceph-mon[36715]: pgmap v5229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:45 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8869.731471 secs 2024-07-26T13:33:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:45 smithi183 ceph-mon[37273]: pgmap v5229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:45 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8869.731471 secs 2024-07-26T13:33:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:47 smithi160 ceph-mon[36715]: pgmap v5230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:47 smithi183 ceph-mon[37273]: pgmap v5230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:47 smithi144 ceph-mon[32129]: pgmap v5230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:49.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:33:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:49.720+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:33:49.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:49.720+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:33:49.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:49.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:33:49.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:33:49.721+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:49 smithi160 ceph-mon[36715]: pgmap v5231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:49 smithi183 ceph-mon[37273]: pgmap v5231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:49 smithi144 ceph-mon[32129]: pgmap v5231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8874.731589 secs 2024-07-26T13:33:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8874.731589 secs 2024-07-26T13:33:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8874.731589 secs 2024-07-26T13:33:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:52 smithi160 ceph-mon[36715]: pgmap v5232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:52 smithi183 ceph-mon[37273]: pgmap v5232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:52 smithi144 ceph-mon[32129]: pgmap v5232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:54 smithi160 ceph-mon[36715]: pgmap v5233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:54 smithi183 ceph-mon[37273]: pgmap v5233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:54 smithi144 ceph-mon[32129]: pgmap v5233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:33:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:56 smithi160 ceph-mon[36715]: pgmap v5234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8879.731721 secs 2024-07-26T13:33:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:56 smithi183 ceph-mon[37273]: pgmap v5234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8879.731721 secs 2024-07-26T13:33:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:56 smithi144 ceph-mon[32129]: pgmap v5234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:33:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8879.731721 secs 2024-07-26T13:33:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:33:58 smithi160 ceph-mon[36715]: pgmap v5235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:33:58 smithi183 ceph-mon[37273]: pgmap v5235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:33:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:33:58 smithi144 ceph-mon[32129]: pgmap v5235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:00 smithi160 ceph-mon[36715]: pgmap v5236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:00 smithi183 ceph-mon[37273]: pgmap v5236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:00 smithi144 ceph-mon[32129]: pgmap v5236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8884.731822 secs 2024-07-26T13:34:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8884.731822 secs 2024-07-26T13:34:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8884.731822 secs 2024-07-26T13:34:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:02 smithi160 ceph-mon[36715]: pgmap v5237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:02 smithi183 ceph-mon[37273]: pgmap v5237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:02 smithi144 ceph-mon[32129]: pgmap v5237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:04 smithi183 ceph-mon[37273]: pgmap v5238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:04 smithi144 ceph-mon[32129]: pgmap v5238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:04 smithi160 ceph-mon[36715]: pgmap v5238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:04.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:34:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:04.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:34:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:04.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:34:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:04.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:34:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:04.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:06 smithi183 ceph-mon[37273]: pgmap v5239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8889.731941 secs 2024-07-26T13:34:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:06 smithi144 ceph-mon[32129]: pgmap v5239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8889.731941 secs 2024-07-26T13:34:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:06 smithi160 ceph-mon[36715]: pgmap v5239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8889.731941 secs 2024-07-26T13:34:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:08 smithi183 ceph-mon[37273]: pgmap v5240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:08 smithi144 ceph-mon[32129]: pgmap v5240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:08 smithi160 ceph-mon[36715]: pgmap v5240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:10 smithi183 ceph-mon[37273]: pgmap v5241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:10 smithi144 ceph-mon[32129]: pgmap v5241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:10 smithi160 ceph-mon[36715]: pgmap v5241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8894.732087 secs 2024-07-26T13:34:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8894.732087 secs 2024-07-26T13:34:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8894.732087 secs 2024-07-26T13:34:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:12 smithi183 ceph-mon[37273]: pgmap v5242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:12 smithi144 ceph-mon[32129]: pgmap v5242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:12 smithi160 ceph-mon[36715]: pgmap v5242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:13 smithi144 ceph-mon[32129]: pgmap v5243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:13 smithi160 ceph-mon[36715]: pgmap v5243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:13 smithi183 ceph-mon[37273]: pgmap v5243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:16.171 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:15 smithi183 ceph-mon[37273]: pgmap v5244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:16.171 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:16.171 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.171 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.171 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.171 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:15 smithi144 ceph-mon[32129]: pgmap v5244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:16.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:16.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:15 smithi160 ceph-mon[36715]: pgmap v5244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:16.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:16.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:16.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8899.732179 secs 2024-07-26T13:34:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8899.732179 secs 2024-07-26T13:34:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8899.732179 secs 2024-07-26T13:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:17.675+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:34:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:17.675+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:34:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:17.676+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:34:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:17.676+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:34:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:17 smithi144 ceph-mon[32129]: pgmap v5245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:17 smithi160 ceph-mon[36715]: pgmap v5245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:17 smithi183 ceph-mon[37273]: pgmap v5245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:19.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:19.754+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:34:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:19.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:34:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:19.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:34:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:19.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:34:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:19.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:20 smithi160 ceph-mon[36715]: pgmap v5246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:20 smithi183 ceph-mon[37273]: pgmap v5246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:20 smithi144 ceph-mon[32129]: pgmap v5246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8904.732336 secs 2024-07-26T13:34:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8904.732336 secs 2024-07-26T13:34:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:34:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8904.732336 secs 2024-07-26T13:34:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:22 smithi183 ceph-mon[37273]: pgmap v5247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:22 smithi144 ceph-mon[32129]: pgmap v5247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:22 smithi160 ceph-mon[36715]: pgmap v5247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:24.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:24 smithi160 ceph-mon[36715]: pgmap v5248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:24 smithi183 ceph-mon[37273]: pgmap v5248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:24 smithi144 ceph-mon[32129]: pgmap v5248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:26 smithi183 ceph-mon[37273]: pgmap v5249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:34:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8909.732457 secs 2024-07-26T13:34:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:26 smithi144 ceph-mon[32129]: pgmap v5249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:34:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8909.732457 secs 2024-07-26T13:34:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:26 smithi160 ceph-mon[36715]: pgmap v5249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:34:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8909.732457 secs 2024-07-26T13:34:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:28 smithi183 ceph-mon[37273]: pgmap v5250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:34:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:28 smithi144 ceph-mon[32129]: pgmap v5250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:34:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:28 smithi160 ceph-mon[36715]: pgmap v5250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:34:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:30 smithi183 ceph-mon[37273]: pgmap v5251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:30 smithi144 ceph-mon[32129]: pgmap v5251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:30 smithi160 ceph-mon[36715]: pgmap v5251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8914.732593 secs 2024-07-26T13:34:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8914.732593 secs 2024-07-26T13:34:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8914.732593 secs 2024-07-26T13:34:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:32 smithi183 ceph-mon[37273]: pgmap v5252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:32 smithi144 ceph-mon[32129]: pgmap v5252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:32 smithi160 ceph-mon[36715]: pgmap v5252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:34:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:33 smithi183 ceph-mon[37273]: pgmap v5253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:33 smithi144 ceph-mon[32129]: pgmap v5253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:33 smithi160 ceph-mon[36715]: pgmap v5253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:34.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:34.757+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:34:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:34.758+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:34:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:34.758+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:34:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:34.758+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:34:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:34.758+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:35 smithi144 ceph-mon[32129]: pgmap v5254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:35 smithi160 ceph-mon[36715]: pgmap v5254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:35 smithi183 ceph-mon[37273]: pgmap v5254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8919.732670 secs 2024-07-26T13:34:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8919.732670 secs 2024-07-26T13:34:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8919.732670 secs 2024-07-26T13:34:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:37 smithi144 ceph-mon[32129]: pgmap v5255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:37 smithi160 ceph-mon[36715]: pgmap v5255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:37 smithi183 ceph-mon[37273]: pgmap v5255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:39 smithi144 ceph-mon[32129]: pgmap v5256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:39 smithi160 ceph-mon[36715]: pgmap v5256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:39 smithi183 ceph-mon[37273]: pgmap v5256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:41 smithi144 ceph-mon[32129]: pgmap v5257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8924.732794 secs 2024-07-26T13:34:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:41 smithi160 ceph-mon[36715]: pgmap v5257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8924.732794 secs 2024-07-26T13:34:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:41 smithi183 ceph-mon[37273]: pgmap v5257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8924.732794 secs 2024-07-26T13:34:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:43 smithi144 ceph-mon[32129]: pgmap v5258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:43 smithi160 ceph-mon[36715]: pgmap v5258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:43 smithi183 ceph-mon[37273]: pgmap v5258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:45 smithi183 ceph-mon[37273]: pgmap v5259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:45 smithi144 ceph-mon[32129]: pgmap v5259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:45 smithi160 ceph-mon[36715]: pgmap v5259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:46.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8929.732912 secs 2024-07-26T13:34:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8929.732912 secs 2024-07-26T13:34:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8929.732912 secs 2024-07-26T13:34:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:47 smithi183 ceph-mon[37273]: pgmap v5260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:47 smithi144 ceph-mon[32129]: pgmap v5260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:47 smithi160 ceph-mon[36715]: pgmap v5260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:49.764+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:34:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:49.764+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:34:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:49.764+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:34:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:49.764+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:34:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:34:49.765+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:49 smithi183 ceph-mon[37273]: pgmap v5261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:49 smithi144 ceph-mon[32129]: pgmap v5261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:49 smithi160 ceph-mon[36715]: pgmap v5261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:34:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:51 smithi183 ceph-mon[37273]: pgmap v5262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8934.733072 secs 2024-07-26T13:34:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:51 smithi144 ceph-mon[32129]: pgmap v5262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8934.733072 secs 2024-07-26T13:34:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:51 smithi160 ceph-mon[36715]: pgmap v5262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8934.733072 secs 2024-07-26T13:34:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:53 smithi183 ceph-mon[37273]: pgmap v5263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:53 smithi144 ceph-mon[32129]: pgmap v5263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:53 smithi160 ceph-mon[36715]: pgmap v5263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:34:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:55 smithi183 ceph-mon[37273]: pgmap v5264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:55 smithi144 ceph-mon[32129]: pgmap v5264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:55 smithi160 ceph-mon[36715]: pgmap v5264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8939.733225 secs 2024-07-26T13:34:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8939.733225 secs 2024-07-26T13:34:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8939.733225 secs 2024-07-26T13:34:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:57 smithi183 ceph-mon[37273]: pgmap v5265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:57 smithi144 ceph-mon[32129]: pgmap v5265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:57 smithi160 ceph-mon[36715]: pgmap v5265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:34:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:34:59 smithi144 ceph-mon[32129]: pgmap v5266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:34:59 smithi160 ceph-mon[36715]: pgmap v5266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:34:59 smithi183 ceph-mon[37273]: pgmap v5266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:01 smithi144 ceph-mon[32129]: pgmap v5267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8944.733346 secs 2024-07-26T13:35:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:01 smithi160 ceph-mon[36715]: pgmap v5267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8944.733346 secs 2024-07-26T13:35:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:01 smithi183 ceph-mon[37273]: pgmap v5267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8944.733346 secs 2024-07-26T13:35:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:03 smithi144 ceph-mon[32129]: pgmap v5268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:03 smithi160 ceph-mon[36715]: pgmap v5268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:03 smithi183 ceph-mon[37273]: pgmap v5268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:04.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:35:04.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:04.775+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:35:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:04.775+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:35:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:04.775+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:35:04.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:04.775+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:05 smithi144 ceph-mon[32129]: pgmap v5269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:05 smithi160 ceph-mon[36715]: pgmap v5269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:05 smithi183 ceph-mon[37273]: pgmap v5269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8949.733508 secs 2024-07-26T13:35:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8949.733508 secs 2024-07-26T13:35:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8949.733508 secs 2024-07-26T13:35:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:07 smithi144 ceph-mon[32129]: pgmap v5270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:07 smithi160 ceph-mon[36715]: pgmap v5270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:07 smithi183 ceph-mon[37273]: pgmap v5270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:09 smithi144 ceph-mon[32129]: pgmap v5271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:09 smithi160 ceph-mon[36715]: pgmap v5271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:09 smithi183 ceph-mon[37273]: pgmap v5271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:11.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:11 smithi183 ceph-mon[37273]: pgmap v5272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:11.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8954.733638 secs 2024-07-26T13:35:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:11 smithi144 ceph-mon[32129]: pgmap v5272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8954.733638 secs 2024-07-26T13:35:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:11 smithi160 ceph-mon[36715]: pgmap v5272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8954.733638 secs 2024-07-26T13:35:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:13 smithi144 ceph-mon[32129]: pgmap v5273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:13 smithi160 ceph-mon[36715]: pgmap v5273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:13 smithi183 ceph-mon[37273]: pgmap v5273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:15 smithi144 ceph-mon[32129]: pgmap v5274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:15 smithi160 ceph-mon[36715]: pgmap v5274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:15 smithi183 ceph-mon[37273]: pgmap v5274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8959.733767 secs 2024-07-26T13:35:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8959.733767 secs 2024-07-26T13:35:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8959.733767 secs 2024-07-26T13:35:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:17 smithi144 ceph-mon[32129]: pgmap v5275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:17 smithi160 ceph-mon[36715]: pgmap v5275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:17 smithi183 ceph-mon[37273]: pgmap v5275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:19.781+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:35:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:19.783+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:35:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:19.783+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:35:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:19.783+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:35:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:19.783+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:19 smithi144 ceph-mon[32129]: pgmap v5276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:19 smithi160 ceph-mon[36715]: pgmap v5276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:19 smithi183 ceph-mon[37273]: pgmap v5276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:21 smithi160 ceph-mon[36715]: pgmap v5277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8964.733918 secs 2024-07-26T13:35:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:35:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:21 smithi183 ceph-mon[37273]: pgmap v5277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8964.733918 secs 2024-07-26T13:35:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:21 smithi144 ceph-mon[32129]: pgmap v5277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8964.733918 secs 2024-07-26T13:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:35:23.037 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:22 smithi144 ceph-mon[32129]: pgmap v5278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:23.037 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:22 smithi144 ceph-mon[32129]: pgmap v5279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:22 smithi160 ceph-mon[36715]: pgmap v5278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:22 smithi160 ceph-mon[36715]: pgmap v5279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:22 smithi183 ceph-mon[37273]: pgmap v5278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:22 smithi183 ceph-mon[37273]: pgmap v5279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:24 smithi160 ceph-mon[36715]: pgmap v5280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:24 smithi183 ceph-mon[37273]: pgmap v5280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:24 smithi144 ceph-mon[32129]: pgmap v5280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:26 smithi160 ceph-mon[36715]: pgmap v5281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8969.733994 secs 2024-07-26T13:35:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:26 smithi183 ceph-mon[37273]: pgmap v5281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8969.733994 secs 2024-07-26T13:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:26 smithi144 ceph-mon[32129]: pgmap v5281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8969.733994 secs 2024-07-26T13:35:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:28 smithi160 ceph-mon[36715]: pgmap v5282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:28 smithi183 ceph-mon[37273]: pgmap v5282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:28 smithi144 ceph-mon[32129]: pgmap v5282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:30 smithi160 ceph-mon[36715]: pgmap v5283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:30 smithi183 ceph-mon[37273]: pgmap v5283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:30 smithi144 ceph-mon[32129]: pgmap v5283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:35:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8974.734091 secs 2024-07-26T13:35:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8974.734091 secs 2024-07-26T13:35:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8974.734091 secs 2024-07-26T13:35:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:32 smithi160 ceph-mon[36715]: pgmap v5284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:35:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:32 smithi183 ceph-mon[37273]: pgmap v5284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:35:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:32 smithi144 ceph-mon[32129]: pgmap v5284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:35:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:34.785+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:35:34.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:34.787+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:35:34.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:34.787+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:35:34.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:34.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:35:34.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:34.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:34 smithi160 ceph-mon[36715]: pgmap v5285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:34 smithi183 ceph-mon[37273]: pgmap v5285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:34 smithi144 ceph-mon[32129]: pgmap v5285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:36 smithi160 ceph-mon[36715]: pgmap v5286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8979.734214 secs 2024-07-26T13:35:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:36 smithi183 ceph-mon[37273]: pgmap v5286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8979.734214 secs 2024-07-26T13:35:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:36 smithi144 ceph-mon[32129]: pgmap v5286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8979.734214 secs 2024-07-26T13:35:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:38 smithi160 ceph-mon[36715]: pgmap v5287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:38 smithi183 ceph-mon[37273]: pgmap v5287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:38 smithi144 ceph-mon[32129]: pgmap v5287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:40 smithi160 ceph-mon[36715]: pgmap v5288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:40 smithi183 ceph-mon[37273]: pgmap v5288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:40 smithi144 ceph-mon[32129]: pgmap v5288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8984.734336 secs 2024-07-26T13:35:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8984.734336 secs 2024-07-26T13:35:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8984.734336 secs 2024-07-26T13:35:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:42 smithi183 ceph-mon[37273]: pgmap v5289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:42 smithi144 ceph-mon[32129]: pgmap v5289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:42 smithi160 ceph-mon[36715]: pgmap v5289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:44 smithi183 ceph-mon[37273]: pgmap v5290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:44 smithi144 ceph-mon[32129]: pgmap v5290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:44 smithi160 ceph-mon[36715]: pgmap v5290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 msgr2=0x7f74900b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 msgr2=0x7f74900b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 0x7f74900b4ef0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a804c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 0x7f74900b4ef0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a804c040 comp rx=0 tx=0).stop 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 msgr2=0x7f74900b4ef0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 msgr2=0x7f74900b4ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900c2a40 0x7f74900b4ef0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:35:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 0x7f74900b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:35:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.122+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 0x7f74900b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:35:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:46.124+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 0x7f74900b5430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a806b000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:35:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:46 smithi183 ceph-mon[37273]: pgmap v5291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8989.734467 secs 2024-07-26T13:35:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:46 smithi144 ceph-mon[32129]: pgmap v5291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8989.734467 secs 2024-07-26T13:35:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:46 smithi160 ceph-mon[36715]: pgmap v5291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8989.734467 secs 2024-07-26T13:35:47.692 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:47.691+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:35:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:47.691+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:35:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:47.691+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:35:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:47.691+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:35:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:48 smithi183 ceph-mon[37273]: pgmap v5292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:48 smithi144 ceph-mon[32129]: pgmap v5292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:48 smithi160 ceph-mon[36715]: pgmap v5292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:49.792+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:35:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:49.793+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:35:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:49.793+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:35:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:49.793+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:35:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:35:49.793+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:50 smithi183 ceph-mon[37273]: pgmap v5293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:50 smithi144 ceph-mon[32129]: pgmap v5293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:50 smithi160 ceph-mon[36715]: pgmap v5293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8994.734551 secs 2024-07-26T13:35:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8994.734551 secs 2024-07-26T13:35:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8994.734551 secs 2024-07-26T13:35:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:52 smithi183 ceph-mon[37273]: pgmap v5294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:52 smithi144 ceph-mon[32129]: pgmap v5294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:52 smithi160 ceph-mon[36715]: pgmap v5294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:35:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:54 smithi183 ceph-mon[37273]: pgmap v5295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:54 smithi144 ceph-mon[32129]: pgmap v5295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:54 smithi160 ceph-mon[36715]: pgmap v5295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:56 smithi183 ceph-mon[37273]: pgmap v5296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 8999.734676 secs 2024-07-26T13:35:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:56 smithi144 ceph-mon[32129]: pgmap v5296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 8999.734676 secs 2024-07-26T13:35:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:56 smithi160 ceph-mon[36715]: pgmap v5296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:35:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 8999.734676 secs 2024-07-26T13:35:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:35:58 smithi183 ceph-mon[37273]: pgmap v5297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:35:58 smithi144 ceph-mon[32129]: pgmap v5297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:35:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:35:58 smithi160 ceph-mon[36715]: pgmap v5297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:00 smithi144 ceph-mon[32129]: pgmap v5298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:00 smithi160 ceph-mon[36715]: pgmap v5298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:00 smithi183 ceph-mon[37273]: pgmap v5298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9004.734790 secs 2024-07-26T13:36:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9004.734790 secs 2024-07-26T13:36:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9004.734790 secs 2024-07-26T13:36:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:02 smithi144 ceph-mon[32129]: pgmap v5299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:02 smithi160 ceph-mon[36715]: pgmap v5299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:02 smithi183 ceph-mon[37273]: pgmap v5299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:04.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:36:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:04.808+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:36:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:04.808+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:36:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:04.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:36:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:04.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:04 smithi144 ceph-mon[32129]: pgmap v5300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:04 smithi160 ceph-mon[36715]: pgmap v5300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:04 smithi183 ceph-mon[37273]: pgmap v5300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:06 smithi144 ceph-mon[32129]: pgmap v5301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9009.734894 secs 2024-07-26T13:36:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:06 smithi160 ceph-mon[36715]: pgmap v5301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9009.734894 secs 2024-07-26T13:36:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:06 smithi183 ceph-mon[37273]: pgmap v5301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9009.734894 secs 2024-07-26T13:36:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:08 smithi144 ceph-mon[32129]: pgmap v5302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:08 smithi160 ceph-mon[36715]: pgmap v5302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:08 smithi183 ceph-mon[37273]: pgmap v5302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:10 smithi144 ceph-mon[32129]: pgmap v5303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:10 smithi160 ceph-mon[36715]: pgmap v5303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:10 smithi183 ceph-mon[37273]: pgmap v5303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9014.735015 secs 2024-07-26T13:36:12.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9014.735015 secs 2024-07-26T13:36:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9014.735015 secs 2024-07-26T13:36:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:12 smithi144 ceph-mon[32129]: pgmap v5304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:12 smithi160 ceph-mon[36715]: pgmap v5304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:12 smithi183 ceph-mon[37273]: pgmap v5304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:14 smithi144 ceph-mon[32129]: pgmap v5305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:14 smithi160 ceph-mon[36715]: pgmap v5305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:14 smithi183 ceph-mon[37273]: pgmap v5305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:16 smithi144 ceph-mon[32129]: pgmap v5306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9019.735139 secs 2024-07-26T13:36:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:16 smithi160 ceph-mon[36715]: pgmap v5306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9019.735139 secs 2024-07-26T13:36:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:16 smithi183 ceph-mon[37273]: pgmap v5306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9019.735139 secs 2024-07-26T13:36:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:18 smithi144 ceph-mon[32129]: pgmap v5307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:18 smithi160 ceph-mon[36715]: pgmap v5307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:18 smithi183 ceph-mon[37273]: pgmap v5307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:19.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:36:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:19.825+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:36:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:19.825+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:36:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:19.825+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:36:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:19.825+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:20 smithi144 ceph-mon[32129]: pgmap v5308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:20 smithi160 ceph-mon[36715]: pgmap v5308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:20 smithi183 ceph-mon[37273]: pgmap v5308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9024.735256 secs 2024-07-26T13:36:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9024.735256 secs 2024-07-26T13:36:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9024.735256 secs 2024-07-26T13:36:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:22 smithi160 ceph-mon[36715]: pgmap v5309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:36:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:22 smithi183 ceph-mon[37273]: pgmap v5309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:36:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:22 smithi144 ceph-mon[32129]: pgmap v5309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:36:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:23 smithi160 ceph-mon[36715]: pgmap v5310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:36:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:23 smithi160 ceph-mon[36715]: pgmap v5311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:36:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:23 smithi183 ceph-mon[37273]: pgmap v5310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:36:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:23 smithi183 ceph-mon[37273]: pgmap v5311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:36:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:23 smithi144 ceph-mon[32129]: pgmap v5310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:36:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:23 smithi144 ceph-mon[32129]: pgmap v5311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:36:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:25 smithi160 ceph-mon[36715]: pgmap v5312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:36:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:25 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9029.735410 secs 2024-07-26T13:36:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:25 smithi183 ceph-mon[37273]: pgmap v5312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:36:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:25 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9029.735410 secs 2024-07-26T13:36:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:25 smithi144 ceph-mon[32129]: pgmap v5312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:36:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:25 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9029.735410 secs 2024-07-26T13:36:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:28 smithi160 ceph-mon[36715]: pgmap v5313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:36:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:28 smithi183 ceph-mon[37273]: pgmap v5313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:36:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:28 smithi144 ceph-mon[32129]: pgmap v5313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:36:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:30 smithi160 ceph-mon[36715]: pgmap v5314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:36:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:30 smithi183 ceph-mon[37273]: pgmap v5314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:36:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:30 smithi144 ceph-mon[32129]: pgmap v5314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:36:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9034.735537 secs 2024-07-26T13:36:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9034.735537 secs 2024-07-26T13:36:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9034.735537 secs 2024-07-26T13:36:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:32 smithi160 ceph-mon[36715]: pgmap v5315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:36:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:32 smithi183 ceph-mon[37273]: pgmap v5315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:36:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:32 smithi144 ceph-mon[32129]: pgmap v5315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:36:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:34 smithi160 ceph-mon[36715]: pgmap v5316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:36:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:34 smithi183 ceph-mon[37273]: pgmap v5316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:36:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:34 smithi144 ceph-mon[32129]: pgmap v5316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:36:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:34.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:36:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:34.842+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:36:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:34.842+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:36:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:34.842+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:36:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:34.842+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:36 smithi160 ceph-mon[36715]: pgmap v5317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9039.735661 secs 2024-07-26T13:36:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:36 smithi183 ceph-mon[37273]: pgmap v5317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9039.735661 secs 2024-07-26T13:36:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:36 smithi144 ceph-mon[32129]: pgmap v5317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9039.735661 secs 2024-07-26T13:36:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:38 smithi160 ceph-mon[36715]: pgmap v5318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:38 smithi183 ceph-mon[37273]: pgmap v5318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:38 smithi144 ceph-mon[32129]: pgmap v5318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:40 smithi144 ceph-mon[32129]: pgmap v5319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:40 smithi160 ceph-mon[36715]: pgmap v5319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:40 smithi183 ceph-mon[37273]: pgmap v5319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9044.735747 secs 2024-07-26T13:36:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9044.735747 secs 2024-07-26T13:36:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9044.735747 secs 2024-07-26T13:36:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:42 smithi183 ceph-mon[37273]: pgmap v5320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:42 smithi144 ceph-mon[32129]: pgmap v5320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:42 smithi160 ceph-mon[36715]: pgmap v5320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:44 smithi183 ceph-mon[37273]: pgmap v5321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:44 smithi144 ceph-mon[32129]: pgmap v5321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:44 smithi160 ceph-mon[36715]: pgmap v5321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:46 smithi183 ceph-mon[37273]: pgmap v5322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9049.735813 secs 2024-07-26T13:36:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:46 smithi144 ceph-mon[32129]: pgmap v5322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9049.735813 secs 2024-07-26T13:36:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:46 smithi160 ceph-mon[36715]: pgmap v5322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9049.735813 secs 2024-07-26T13:36:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:48 smithi183 ceph-mon[37273]: pgmap v5323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:48 smithi144 ceph-mon[32129]: pgmap v5323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:48 smithi160 ceph-mon[36715]: pgmap v5323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:49.857+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:36:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:49.858+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:36:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:49.858+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:36:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:49.858+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:36:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:36:49.858+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:50 smithi183 ceph-mon[37273]: pgmap v5324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:50 smithi144 ceph-mon[32129]: pgmap v5324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:50 smithi160 ceph-mon[36715]: pgmap v5324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9054.735973 secs 2024-07-26T13:36:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9054.735973 secs 2024-07-26T13:36:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9054.735973 secs 2024-07-26T13:36:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:52 smithi183 ceph-mon[37273]: pgmap v5325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:52 smithi144 ceph-mon[32129]: pgmap v5325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:52 smithi160 ceph-mon[36715]: pgmap v5325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:54 smithi183 ceph-mon[37273]: pgmap v5326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:54 smithi144 ceph-mon[32129]: pgmap v5326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:54 smithi160 ceph-mon[36715]: pgmap v5326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:36:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:56 smithi183 ceph-mon[37273]: pgmap v5327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9059.736041 secs 2024-07-26T13:36:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:56 smithi144 ceph-mon[32129]: pgmap v5327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9059.736041 secs 2024-07-26T13:36:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:56 smithi160 ceph-mon[36715]: pgmap v5327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:36:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9059.736041 secs 2024-07-26T13:36:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:36:58 smithi183 ceph-mon[37273]: pgmap v5328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:36:58 smithi144 ceph-mon[32129]: pgmap v5328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:36:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:36:58 smithi160 ceph-mon[36715]: pgmap v5328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:00 smithi183 ceph-mon[37273]: pgmap v5329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:00 smithi144 ceph-mon[32129]: pgmap v5329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:00 smithi160 ceph-mon[36715]: pgmap v5329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9064.736178 secs 2024-07-26T13:37:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9064.736178 secs 2024-07-26T13:37:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9064.736178 secs 2024-07-26T13:37:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:02 smithi144 ceph-mon[32129]: pgmap v5330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:02 smithi160 ceph-mon[36715]: pgmap v5330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:02 smithi183 ceph-mon[37273]: pgmap v5330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:03 smithi144 ceph-mon[32129]: pgmap v5331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:03 smithi160 ceph-mon[36715]: pgmap v5331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:03 smithi183 ceph-mon[37273]: pgmap v5331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:04.874+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:37:04.876 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:04.876+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:37:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:04.876+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:37:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:04.876+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:37:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:04.876+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:05 smithi144 ceph-mon[32129]: pgmap v5332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:05 smithi160 ceph-mon[36715]: pgmap v5332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:05 smithi183 ceph-mon[37273]: pgmap v5332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9069.736325 secs 2024-07-26T13:37:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9069.736325 secs 2024-07-26T13:37:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9069.736325 secs 2024-07-26T13:37:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:07 smithi144 ceph-mon[32129]: pgmap v5333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:07 smithi160 ceph-mon[36715]: pgmap v5333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:07 smithi183 ceph-mon[37273]: pgmap v5333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:09 smithi144 ceph-mon[32129]: pgmap v5334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:09 smithi160 ceph-mon[36715]: pgmap v5334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:09 smithi183 ceph-mon[37273]: pgmap v5334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:11 smithi144 ceph-mon[32129]: pgmap v5335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9074.736425 secs 2024-07-26T13:37:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:11 smithi160 ceph-mon[36715]: pgmap v5335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9074.736425 secs 2024-07-26T13:37:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:11 smithi183 ceph-mon[37273]: pgmap v5335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9074.736425 secs 2024-07-26T13:37:13.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:13 smithi183 ceph-mon[37273]: pgmap v5336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:13 smithi144 ceph-mon[32129]: pgmap v5336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:13 smithi160 ceph-mon[36715]: pgmap v5336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:15 smithi144 ceph-mon[32129]: pgmap v5337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:15 smithi160 ceph-mon[36715]: pgmap v5337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:15 smithi183 ceph-mon[37273]: pgmap v5337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9079.736530 secs 2024-07-26T13:37:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9079.736530 secs 2024-07-26T13:37:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9079.736530 secs 2024-07-26T13:37:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:17 smithi144 ceph-mon[32129]: pgmap v5338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:17 smithi160 ceph-mon[36715]: pgmap v5338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:17 smithi183 ceph-mon[37273]: pgmap v5338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:19.891+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:37:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:19.893+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:37:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:19.893+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:37:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:19.893+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:37:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:19.893+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:19 smithi144 ceph-mon[32129]: pgmap v5339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:19 smithi160 ceph-mon[36715]: pgmap v5339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:19 smithi183 ceph-mon[37273]: pgmap v5339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:21 smithi160 ceph-mon[36715]: pgmap v5340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9084.736659 secs 2024-07-26T13:37:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:21 smithi183 ceph-mon[37273]: pgmap v5340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9084.736659 secs 2024-07-26T13:37:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:21 smithi144 ceph-mon[32129]: pgmap v5340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9084.736659 secs 2024-07-26T13:37:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:23 smithi160 ceph-mon[36715]: pgmap v5341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:23 smithi183 ceph-mon[37273]: pgmap v5341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:23 smithi144 ceph-mon[32129]: pgmap v5341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:24 smithi160 ceph-mon[36715]: pgmap v5342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:24 smithi160 ceph-mon[36715]: pgmap v5343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:37:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:37:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:24 smithi183 ceph-mon[37273]: pgmap v5342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:24 smithi183 ceph-mon[37273]: pgmap v5343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:37:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:37:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:24 smithi144 ceph-mon[32129]: pgmap v5342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:24 smithi144 ceph-mon[32129]: pgmap v5343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:37:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:37:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9089.736777 secs 2024-07-26T13:37:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:26 smithi183 ceph-mon[37273]: pgmap v5344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9089.736777 secs 2024-07-26T13:37:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:26 smithi144 ceph-mon[32129]: pgmap v5344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9089.736777 secs 2024-07-26T13:37:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:26 smithi160 ceph-mon[36715]: pgmap v5344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:29.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:28 smithi160 ceph-mon[36715]: pgmap v5345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:37:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:28 smithi183 ceph-mon[37273]: pgmap v5345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:37:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:28 smithi144 ceph-mon[32129]: pgmap v5345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:37:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:30 smithi183 ceph-mon[37273]: pgmap v5346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:37:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:30 smithi144 ceph-mon[32129]: pgmap v5346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:37:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:30 smithi160 ceph-mon[36715]: pgmap v5346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:37:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9094.736905 secs 2024-07-26T13:37:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9094.736905 secs 2024-07-26T13:37:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9094.736905 secs 2024-07-26T13:37:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:32 smithi183 ceph-mon[37273]: pgmap v5347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:32 smithi144 ceph-mon[32129]: pgmap v5347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:32 smithi160 ceph-mon[36715]: pgmap v5347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:37:34.909 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:34.908+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:37:34.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:34.909+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:37:34.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:34.909+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:37:34.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:34.909+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:37:34.910 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:34.909+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:34 smithi183 ceph-mon[37273]: pgmap v5348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:37:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:34 smithi144 ceph-mon[32129]: pgmap v5348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:37:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:34 smithi160 ceph-mon[36715]: pgmap v5348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:37:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9099.736969 secs 2024-07-26T13:37:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:36 smithi183 ceph-mon[37273]: pgmap v5349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9099.736969 secs 2024-07-26T13:37:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:36 smithi144 ceph-mon[32129]: pgmap v5349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9099.736969 secs 2024-07-26T13:37:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:36 smithi160 ceph-mon[36715]: pgmap v5349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:38 smithi183 ceph-mon[37273]: pgmap v5350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:38 smithi144 ceph-mon[32129]: pgmap v5350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:38 smithi160 ceph-mon[36715]: pgmap v5350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:40 smithi183 ceph-mon[37273]: pgmap v5351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:40 smithi144 ceph-mon[32129]: pgmap v5351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:40 smithi160 ceph-mon[36715]: pgmap v5351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9104.737133 secs 2024-07-26T13:37:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9104.737133 secs 2024-07-26T13:37:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9104.737133 secs 2024-07-26T13:37:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:42 smithi183 ceph-mon[37273]: pgmap v5352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:42 smithi144 ceph-mon[32129]: pgmap v5352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:42 smithi160 ceph-mon[36715]: pgmap v5352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:44 smithi183 ceph-mon[37273]: pgmap v5353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:44 smithi144 ceph-mon[32129]: pgmap v5353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:44 smithi160 ceph-mon[36715]: pgmap v5353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9109.737239 secs 2024-07-26T13:37:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:46 smithi144 ceph-mon[32129]: pgmap v5354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9109.737239 secs 2024-07-26T13:37:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:46 smithi160 ceph-mon[36715]: pgmap v5354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9109.737239 secs 2024-07-26T13:37:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:46 smithi183 ceph-mon[37273]: pgmap v5354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:48 smithi144 ceph-mon[32129]: pgmap v5355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:48 smithi160 ceph-mon[36715]: pgmap v5355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:48 smithi183 ceph-mon[37273]: pgmap v5355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:49.924+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:37:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:49.925+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:37:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:49.925+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:37:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:49.925+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:37:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:37:49.926+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:50 smithi144 ceph-mon[32129]: pgmap v5356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:50 smithi160 ceph-mon[36715]: pgmap v5356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:50 smithi183 ceph-mon[37273]: pgmap v5356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9114.737376 secs 2024-07-26T13:37:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9114.737376 secs 2024-07-26T13:37:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9114.737376 secs 2024-07-26T13:37:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:52 smithi144 ceph-mon[32129]: pgmap v5357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:52 smithi160 ceph-mon[36715]: pgmap v5357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:52 smithi183 ceph-mon[37273]: pgmap v5357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:37:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:54 smithi144 ceph-mon[32129]: pgmap v5358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:54 smithi160 ceph-mon[36715]: pgmap v5358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:54 smithi183 ceph-mon[37273]: pgmap v5358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9119.737498 secs 2024-07-26T13:37:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:56 smithi144 ceph-mon[32129]: pgmap v5359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9119.737498 secs 2024-07-26T13:37:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:56 smithi160 ceph-mon[36715]: pgmap v5359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9119.737498 secs 2024-07-26T13:37:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:56 smithi183 ceph-mon[37273]: pgmap v5359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:37:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:37:58 smithi144 ceph-mon[32129]: pgmap v5360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:37:58 smithi160 ceph-mon[36715]: pgmap v5360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:37:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:37:58 smithi183 ceph-mon[37273]: pgmap v5360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:00 smithi144 ceph-mon[32129]: pgmap v5361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:00 smithi160 ceph-mon[36715]: pgmap v5361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:00 smithi183 ceph-mon[37273]: pgmap v5361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9124.737602 secs 2024-07-26T13:38:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9124.737602 secs 2024-07-26T13:38:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9124.737602 secs 2024-07-26T13:38:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:02 smithi144 ceph-mon[32129]: pgmap v5362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:02 smithi160 ceph-mon[36715]: pgmap v5362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:02 smithi183 ceph-mon[37273]: pgmap v5362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:04.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:38:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:04.943+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:38:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:04.943+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:38:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:04.943+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:38:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:04.943+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:04 smithi144 ceph-mon[32129]: pgmap v5363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:04 smithi160 ceph-mon[36715]: pgmap v5363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:04 smithi183 ceph-mon[37273]: pgmap v5363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9129.737730 secs 2024-07-26T13:38:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:06 smithi144 ceph-mon[32129]: pgmap v5364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9129.737730 secs 2024-07-26T13:38:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:06 smithi160 ceph-mon[36715]: pgmap v5364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9129.737730 secs 2024-07-26T13:38:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:06 smithi183 ceph-mon[37273]: pgmap v5364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:08 smithi160 ceph-mon[36715]: pgmap v5365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:08 smithi183 ceph-mon[37273]: pgmap v5365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:08 smithi144 ceph-mon[32129]: pgmap v5365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:10 smithi160 ceph-mon[36715]: pgmap v5366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:10 smithi183 ceph-mon[37273]: pgmap v5366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:10 smithi144 ceph-mon[32129]: pgmap v5366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9134.737863 secs 2024-07-26T13:38:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9134.737863 secs 2024-07-26T13:38:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9134.737863 secs 2024-07-26T13:38:13.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:12 smithi183 ceph-mon[37273]: pgmap v5367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:12 smithi160 ceph-mon[36715]: pgmap v5367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:12 smithi144 ceph-mon[32129]: pgmap v5367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:14 smithi160 ceph-mon[36715]: pgmap v5368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:14 smithi183 ceph-mon[37273]: pgmap v5368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:14 smithi144 ceph-mon[32129]: pgmap v5368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9139.737990 secs 2024-07-26T13:38:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9139.737990 secs 2024-07-26T13:38:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9139.737990 secs 2024-07-26T13:38:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:17 smithi160 ceph-mon[36715]: pgmap v5369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:17 smithi183 ceph-mon[37273]: pgmap v5369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:17 smithi144 ceph-mon[32129]: pgmap v5369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:19.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:19 smithi144 ceph-mon[32129]: pgmap v5370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:19 smithi160 ceph-mon[36715]: pgmap v5370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:19 smithi183 ceph-mon[37273]: pgmap v5370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:19.951+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:38:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:19.952+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:38:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:19.952+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:38:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:19.952+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:38:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:19.953+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:21 smithi160 ceph-mon[36715]: pgmap v5371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9144.738106 secs 2024-07-26T13:38:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:21 smithi183 ceph-mon[37273]: pgmap v5371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9144.738106 secs 2024-07-26T13:38:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:21 smithi144 ceph-mon[32129]: pgmap v5371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9144.738106 secs 2024-07-26T13:38:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:23 smithi160 ceph-mon[36715]: pgmap v5372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:23 smithi183 ceph-mon[37273]: pgmap v5372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:23 smithi144 ceph-mon[32129]: pgmap v5372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:25 smithi160 ceph-mon[36715]: pgmap v5373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:38:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:25 smithi183 ceph-mon[37273]: pgmap v5373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:38:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:25 smithi144 ceph-mon[32129]: pgmap v5373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:38:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:26 smithi160 ceph-mon[36715]: pgmap v5374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:38:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:26 smithi160 ceph-mon[36715]: pgmap v5375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:38:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9149.738257 secs 2024-07-26T13:38:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:26 smithi183 ceph-mon[37273]: pgmap v5374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:38:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:26 smithi183 ceph-mon[37273]: pgmap v5375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:38:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9149.738257 secs 2024-07-26T13:38:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:26 smithi144 ceph-mon[32129]: pgmap v5374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:38:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:26 smithi144 ceph-mon[32129]: pgmap v5375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:38:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9149.738257 secs 2024-07-26T13:38:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:28 smithi160 ceph-mon[36715]: pgmap v5376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:38:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:28 smithi183 ceph-mon[37273]: pgmap v5376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:38:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:28 smithi144 ceph-mon[32129]: pgmap v5376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:38:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:30 smithi183 ceph-mon[37273]: pgmap v5377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:38:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:30 smithi144 ceph-mon[32129]: pgmap v5377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:38:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:30 smithi160 ceph-mon[36715]: pgmap v5377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:38:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9154.738369 secs 2024-07-26T13:38:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9154.738369 secs 2024-07-26T13:38:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9154.738369 secs 2024-07-26T13:38:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:32 smithi183 ceph-mon[37273]: pgmap v5378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:38:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:32 smithi144 ceph-mon[32129]: pgmap v5378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:38:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:32 smithi160 ceph-mon[36715]: pgmap v5378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:38:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:34 smithi183 ceph-mon[37273]: pgmap v5379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:38:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:34 smithi144 ceph-mon[32129]: pgmap v5379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:38:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:34 smithi160 ceph-mon[36715]: pgmap v5379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:38:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:34.968+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:38:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:34.969+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:38:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:34.969+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:38:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:34.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:38:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:34.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:36 smithi183 ceph-mon[37273]: pgmap v5380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:38:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9159.738495 secs 2024-07-26T13:38:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:36 smithi144 ceph-mon[32129]: pgmap v5380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:38:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9159.738495 secs 2024-07-26T13:38:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:36 smithi160 ceph-mon[36715]: pgmap v5380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:38:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9159.738495 secs 2024-07-26T13:38:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:38 smithi183 ceph-mon[37273]: pgmap v5381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:38 smithi144 ceph-mon[32129]: pgmap v5381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:38 smithi160 ceph-mon[36715]: pgmap v5381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:40 smithi183 ceph-mon[37273]: pgmap v5382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:40 smithi144 ceph-mon[32129]: pgmap v5382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:40 smithi160 ceph-mon[36715]: pgmap v5382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9164.738619 secs 2024-07-26T13:38:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9164.738619 secs 2024-07-26T13:38:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9164.738619 secs 2024-07-26T13:38:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:42 smithi183 ceph-mon[37273]: pgmap v5383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:42 smithi144 ceph-mon[32129]: pgmap v5383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:42 smithi160 ceph-mon[36715]: pgmap v5383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:44 smithi183 ceph-mon[37273]: pgmap v5384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:44 smithi144 ceph-mon[32129]: pgmap v5384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:44 smithi160 ceph-mon[36715]: pgmap v5384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:46 smithi183 ceph-mon[37273]: pgmap v5385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9169.738752 secs 2024-07-26T13:38:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:46 smithi144 ceph-mon[32129]: pgmap v5385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9169.738752 secs 2024-07-26T13:38:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:46 smithi160 ceph-mon[36715]: pgmap v5385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9169.738752 secs 2024-07-26T13:38:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:47.723+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:38:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:47.723+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:38:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:47.723+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:38:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:47.724+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:38:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:48 smithi183 ceph-mon[37273]: pgmap v5386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:48 smithi144 ceph-mon[32129]: pgmap v5386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:48 smithi160 ceph-mon[36715]: pgmap v5386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:49.980+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:38:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:49.981+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:38:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:49.981+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:38:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:49.981+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:38:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:38:49.981+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:50 smithi144 ceph-mon[32129]: pgmap v5387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:50 smithi160 ceph-mon[36715]: pgmap v5387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:50 smithi183 ceph-mon[37273]: pgmap v5387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9174.738899 secs 2024-07-26T13:38:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9174.738899 secs 2024-07-26T13:38:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9174.738899 secs 2024-07-26T13:38:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:52 smithi144 ceph-mon[32129]: pgmap v5388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:52 smithi160 ceph-mon[36715]: pgmap v5388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:52 smithi183 ceph-mon[37273]: pgmap v5388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:53 smithi144 ceph-mon[32129]: pgmap v5389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:53 smithi160 ceph-mon[36715]: pgmap v5389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:53 smithi183 ceph-mon[37273]: pgmap v5389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:38:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:56 smithi160 ceph-mon[36715]: pgmap v5390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9179.739046 secs 2024-07-26T13:38:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:56 smithi183 ceph-mon[37273]: pgmap v5390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:56.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9179.739046 secs 2024-07-26T13:38:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:56 smithi144 ceph-mon[32129]: pgmap v5390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:38:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9179.739046 secs 2024-07-26T13:38:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:38:58 smithi160 ceph-mon[36715]: pgmap v5391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:38:58 smithi183 ceph-mon[37273]: pgmap v5391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:38:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:38:58 smithi144 ceph-mon[32129]: pgmap v5391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:00 smithi183 ceph-mon[37273]: pgmap v5392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:00 smithi144 ceph-mon[32129]: pgmap v5392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:00 smithi160 ceph-mon[36715]: pgmap v5392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9184.739189 secs 2024-07-26T13:39:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9184.739189 secs 2024-07-26T13:39:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9184.739189 secs 2024-07-26T13:39:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:02 smithi183 ceph-mon[37273]: pgmap v5393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:02 smithi144 ceph-mon[32129]: pgmap v5393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:02 smithi160 ceph-mon[36715]: pgmap v5393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:04 smithi183 ceph-mon[37273]: pgmap v5394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:04 smithi144 ceph-mon[32129]: pgmap v5394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:04 smithi160 ceph-mon[36715]: pgmap v5394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:04.997+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:04.998+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:04.998+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:04.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:04.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:06 smithi183 ceph-mon[37273]: pgmap v5395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9189.739338 secs 2024-07-26T13:39:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:06 smithi144 ceph-mon[32129]: pgmap v5395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9189.739338 secs 2024-07-26T13:39:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:06 smithi160 ceph-mon[36715]: pgmap v5395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9189.739338 secs 2024-07-26T13:39:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:08 smithi183 ceph-mon[37273]: pgmap v5396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:08 smithi144 ceph-mon[32129]: pgmap v5396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:08.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:08 smithi160 ceph-mon[36715]: pgmap v5396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:10 smithi183 ceph-mon[37273]: pgmap v5397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:10 smithi144 ceph-mon[32129]: pgmap v5397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:10 smithi160 ceph-mon[36715]: pgmap v5397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9194.739467 secs 2024-07-26T13:39:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9194.739467 secs 2024-07-26T13:39:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9194.739467 secs 2024-07-26T13:39:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:12 smithi183 ceph-mon[37273]: pgmap v5398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:12 smithi144 ceph-mon[32129]: pgmap v5398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:12 smithi160 ceph-mon[36715]: pgmap v5398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:14 smithi183 ceph-mon[37273]: pgmap v5399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:14 smithi144 ceph-mon[32129]: pgmap v5399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:14 smithi160 ceph-mon[36715]: pgmap v5399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:16.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:16 smithi144 ceph-mon[32129]: pgmap v5400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:16.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9199.739591 secs 2024-07-26T13:39:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:16 smithi183 ceph-mon[37273]: pgmap v5400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9199.739591 secs 2024-07-26T13:39:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:16 smithi160 ceph-mon[36715]: pgmap v5400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9199.739591 secs 2024-07-26T13:39:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:18 smithi183 ceph-mon[37273]: pgmap v5401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:18 smithi144 ceph-mon[32129]: pgmap v5401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:18 smithi160 ceph-mon[36715]: pgmap v5401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:20.014+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:39:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:20.015+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:39:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:20.015+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:39:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:20.015+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:39:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:20.015+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:20 smithi144 ceph-mon[32129]: pgmap v5402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:20 smithi160 ceph-mon[36715]: pgmap v5402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:20 smithi183 ceph-mon[37273]: pgmap v5402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9204.739712 secs 2024-07-26T13:39:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9204.739712 secs 2024-07-26T13:39:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9204.739712 secs 2024-07-26T13:39:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:22 smithi144 ceph-mon[32129]: pgmap v5403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:22 smithi160 ceph-mon[36715]: pgmap v5403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:22 smithi183 ceph-mon[37273]: pgmap v5403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:24 smithi144 ceph-mon[32129]: pgmap v5404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:24 smithi160 ceph-mon[36715]: pgmap v5404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:24 smithi183 ceph-mon[37273]: pgmap v5404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:25 smithi144 ceph-mon[32129]: pgmap v5405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:25 smithi160 ceph-mon[36715]: pgmap v5405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:25 smithi183 ceph-mon[37273]: pgmap v5405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9209.739868 secs 2024-07-26T13:39:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:39:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9209.739868 secs 2024-07-26T13:39:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:39:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9209.739868 secs 2024-07-26T13:39:27.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:27.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:27.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:39:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:27 smithi160 ceph-mon[36715]: pgmap v5406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:27 smithi160 ceph-mon[36715]: pgmap v5407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:27 smithi183 ceph-mon[37273]: pgmap v5406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:27 smithi183 ceph-mon[37273]: pgmap v5407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:27 smithi144 ceph-mon[32129]: pgmap v5406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:27 smithi144 ceph-mon[32129]: pgmap v5407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:30.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:29 smithi160 ceph-mon[36715]: pgmap v5408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:29 smithi183 ceph-mon[37273]: pgmap v5408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:29 smithi144 ceph-mon[32129]: pgmap v5408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:31 smithi160 ceph-mon[36715]: pgmap v5409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9214.739974 secs 2024-07-26T13:39:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:31 smithi183 ceph-mon[37273]: pgmap v5409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9214.739974 secs 2024-07-26T13:39:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:31 smithi144 ceph-mon[32129]: pgmap v5409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9214.739974 secs 2024-07-26T13:39:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:33 smithi160 ceph-mon[36715]: pgmap v5410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:33 smithi183 ceph-mon[37273]: pgmap v5410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:33 smithi144 ceph-mon[32129]: pgmap v5410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:35.017+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:39:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:35.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:39:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:35.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:39:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:35.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:39:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:35.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:35 smithi160 ceph-mon[36715]: pgmap v5411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:35 smithi183 ceph-mon[37273]: pgmap v5411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:35 smithi144 ceph-mon[32129]: pgmap v5411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:39:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9219.740125 secs 2024-07-26T13:39:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9219.740125 secs 2024-07-26T13:39:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9219.740125 secs 2024-07-26T13:39:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:37 smithi160 ceph-mon[36715]: pgmap v5412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:37 smithi183 ceph-mon[37273]: pgmap v5412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:37 smithi144 ceph-mon[32129]: pgmap v5412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:39 smithi160 ceph-mon[36715]: pgmap v5413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:39 smithi183 ceph-mon[37273]: pgmap v5413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:39 smithi144 ceph-mon[32129]: pgmap v5413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:41 smithi160 ceph-mon[36715]: pgmap v5414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9224.740247 secs 2024-07-26T13:39:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:41 smithi183 ceph-mon[37273]: pgmap v5414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9224.740247 secs 2024-07-26T13:39:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:41 smithi144 ceph-mon[32129]: pgmap v5414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9224.740247 secs 2024-07-26T13:39:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:43 smithi160 ceph-mon[36715]: pgmap v5415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:43 smithi183 ceph-mon[37273]: pgmap v5415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:43 smithi144 ceph-mon[32129]: pgmap v5415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:39:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:45 smithi160 ceph-mon[36715]: pgmap v5416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:45 smithi183 ceph-mon[37273]: pgmap v5416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:46.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:45 smithi144 ceph-mon[32129]: pgmap v5416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:39:47.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9229.740350 secs 2024-07-26T13:39:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9229.740350 secs 2024-07-26T13:39:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9229.740350 secs 2024-07-26T13:39:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:47 smithi160 ceph-mon[36715]: pgmap v5417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:47 smithi183 ceph-mon[37273]: pgmap v5417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:47 smithi144 ceph-mon[32129]: pgmap v5417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:39:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:50.020+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:39:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:50.021+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:39:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:50.021+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:39:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:50.021+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:39:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:39:50.021+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:49 smithi160 ceph-mon[36715]: pgmap v5418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:49 smithi183 ceph-mon[37273]: pgmap v5418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:49 smithi144 ceph-mon[32129]: pgmap v5418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:51 smithi183 ceph-mon[37273]: pgmap v5419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9234.740419 secs 2024-07-26T13:39:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:51 smithi144 ceph-mon[32129]: pgmap v5419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9234.740419 secs 2024-07-26T13:39:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:51 smithi160 ceph-mon[36715]: pgmap v5419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9234.740419 secs 2024-07-26T13:39:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:53 smithi183 ceph-mon[37273]: pgmap v5420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:39:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:53 smithi144 ceph-mon[32129]: pgmap v5420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:39:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:53 smithi160 ceph-mon[36715]: pgmap v5420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:39:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:55 smithi183 ceph-mon[37273]: pgmap v5421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:39:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:55 smithi144 ceph-mon[32129]: pgmap v5421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:39:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:55 smithi160 ceph-mon[36715]: pgmap v5421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:39:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9239.740547 secs 2024-07-26T13:39:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9239.740547 secs 2024-07-26T13:39:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9239.740547 secs 2024-07-26T13:39:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:57 smithi183 ceph-mon[37273]: pgmap v5422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:57 smithi144 ceph-mon[32129]: pgmap v5422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:39:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:57 smithi160 ceph-mon[36715]: pgmap v5422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:39:59 smithi183 ceph-mon[37273]: pgmap v5423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:39:59 smithi144 ceph-mon[32129]: pgmap v5423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:39:59 smithi160 ceph-mon[36715]: pgmap v5423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:01.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:01 smithi183 ceph-mon[37273]: pgmap v5424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9244.740671 secs 2024-07-26T13:40:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:01 smithi144 ceph-mon[32129]: pgmap v5424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9244.740671 secs 2024-07-26T13:40:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:01 smithi160 ceph-mon[36715]: pgmap v5424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9244.740671 secs 2024-07-26T13:40:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:03 smithi183 ceph-mon[37273]: pgmap v5425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:03 smithi144 ceph-mon[32129]: pgmap v5425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:03 smithi160 ceph-mon[36715]: pgmap v5425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:05.029+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:40:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:05.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:40:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:05.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:40:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:05.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:40:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:05.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:05 smithi183 ceph-mon[37273]: pgmap v5426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:05 smithi144 ceph-mon[32129]: pgmap v5426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:05 smithi160 ceph-mon[36715]: pgmap v5426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9249.740788 secs 2024-07-26T13:40:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9249.740788 secs 2024-07-26T13:40:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9249.740788 secs 2024-07-26T13:40:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:07 smithi144 ceph-mon[32129]: pgmap v5427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:07 smithi160 ceph-mon[36715]: pgmap v5427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:07 smithi183 ceph-mon[37273]: pgmap v5427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:09 smithi144 ceph-mon[32129]: pgmap v5428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:09 smithi160 ceph-mon[36715]: pgmap v5428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:09 smithi183 ceph-mon[37273]: pgmap v5428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:11 smithi144 ceph-mon[32129]: pgmap v5429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9254.740899 secs 2024-07-26T13:40:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:11 smithi160 ceph-mon[36715]: pgmap v5429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9254.740899 secs 2024-07-26T13:40:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:11 smithi183 ceph-mon[37273]: pgmap v5429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9254.740899 secs 2024-07-26T13:40:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:13 smithi144 ceph-mon[32129]: pgmap v5430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:13 smithi160 ceph-mon[36715]: pgmap v5430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:13 smithi183 ceph-mon[37273]: pgmap v5430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:16.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:15 smithi183 ceph-mon[37273]: pgmap v5431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:15 smithi144 ceph-mon[32129]: pgmap v5431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:16.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:15 smithi160 ceph-mon[36715]: pgmap v5431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:17.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9259.741022 secs 2024-07-26T13:40:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9259.741022 secs 2024-07-26T13:40:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9259.741022 secs 2024-07-26T13:40:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:17 smithi144 ceph-mon[32129]: pgmap v5432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:17 smithi160 ceph-mon[36715]: pgmap v5432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:17 smithi183 ceph-mon[37273]: pgmap v5432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:20.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:40:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:20.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:40:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:20.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:40:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:20.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:40:20.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:20.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:19 smithi144 ceph-mon[32129]: pgmap v5433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:19 smithi160 ceph-mon[36715]: pgmap v5433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:19 smithi183 ceph-mon[37273]: pgmap v5433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:21 smithi144 ceph-mon[32129]: pgmap v5434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9264.741163 secs 2024-07-26T13:40:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:21 smithi160 ceph-mon[36715]: pgmap v5434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9264.741163 secs 2024-07-26T13:40:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:21 smithi183 ceph-mon[37273]: pgmap v5434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9264.741163 secs 2024-07-26T13:40:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:23 smithi144 ceph-mon[32129]: pgmap v5435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:23 smithi160 ceph-mon[36715]: pgmap v5435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:23 smithi183 ceph-mon[37273]: pgmap v5435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:25 smithi144 ceph-mon[32129]: pgmap v5436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:25 smithi160 ceph-mon[36715]: pgmap v5436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:25 smithi183 ceph-mon[37273]: pgmap v5436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9269.741335 secs 2024-07-26T13:40:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9269.741335 secs 2024-07-26T13:40:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9269.741335 secs 2024-07-26T13:40:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: pgmap v5437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: pgmap v5437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:28.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:40:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: pgmap v5437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:40:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:28 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:40:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:28 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:40:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:28 smithi144 ceph-mon[32129]: pgmap v5438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:40:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:28 smithi144 ceph-mon[32129]: pgmap v5439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:40:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:28 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:40:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:28 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:40:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:28 smithi160 ceph-mon[36715]: pgmap v5438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:40:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:28 smithi160 ceph-mon[36715]: pgmap v5439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:40:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:28 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:40:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:28 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:40:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:28 smithi183 ceph-mon[37273]: pgmap v5438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:40:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:28 smithi183 ceph-mon[37273]: pgmap v5439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:40:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:30 smithi160 ceph-mon[36715]: pgmap v5440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:40:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:30 smithi183 ceph-mon[37273]: pgmap v5440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:40:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:30 smithi144 ceph-mon[32129]: pgmap v5440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:40:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9274.741467 secs 2024-07-26T13:40:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9274.741467 secs 2024-07-26T13:40:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9274.741467 secs 2024-07-26T13:40:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:32 smithi160 ceph-mon[36715]: pgmap v5441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:40:33.309 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:32 smithi144 ceph-mon[32129]: pgmap v5441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:40:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:32 smithi183 ceph-mon[37273]: pgmap v5441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:40:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:35.063+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:40:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:35.064+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:40:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:35.064+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:40:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:35.064+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:40:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:35.064+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:34 smithi160 ceph-mon[36715]: pgmap v5442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:40:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:34 smithi183 ceph-mon[37273]: pgmap v5442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:40:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:34 smithi144 ceph-mon[32129]: pgmap v5442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:40:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:36 smithi160 ceph-mon[36715]: pgmap v5443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:40:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9279.741600 secs 2024-07-26T13:40:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:36 smithi183 ceph-mon[37273]: pgmap v5443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:40:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9279.741600 secs 2024-07-26T13:40:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:36 smithi144 ceph-mon[32129]: pgmap v5443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:40:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9279.741600 secs 2024-07-26T13:40:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:39 smithi160 ceph-mon[36715]: pgmap v5444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:39 smithi183 ceph-mon[37273]: pgmap v5444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:39 smithi144 ceph-mon[32129]: pgmap v5444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:40:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:41 smithi160 ceph-mon[36715]: pgmap v5445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:41 smithi183 ceph-mon[37273]: pgmap v5445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:41 smithi144 ceph-mon[32129]: pgmap v5445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9284.741683 secs 2024-07-26T13:40:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9284.741683 secs 2024-07-26T13:40:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9284.741683 secs 2024-07-26T13:40:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:43 smithi160 ceph-mon[36715]: pgmap v5446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:43 smithi183 ceph-mon[37273]: pgmap v5446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:43 smithi144 ceph-mon[32129]: pgmap v5446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:45 smithi160 ceph-mon[36715]: pgmap v5447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:45 smithi183 ceph-mon[37273]: pgmap v5447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:45 smithi144 ceph-mon[32129]: pgmap v5447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9289.741760 secs 2024-07-26T13:40:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9289.741760 secs 2024-07-26T13:40:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9289.741760 secs 2024-07-26T13:40:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:47 smithi160 ceph-mon[36715]: pgmap v5448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:47 smithi183 ceph-mon[37273]: pgmap v5448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:47 smithi144 ceph-mon[32129]: pgmap v5448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:49 smithi160 ceph-mon[36715]: pgmap v5449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:49 smithi183 ceph-mon[37273]: pgmap v5449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:49 smithi144 ceph-mon[32129]: pgmap v5449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:50.079+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:40:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:50.081+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:40:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:50.081+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:40:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:50.081+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:40:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:40:50.081+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:51 smithi160 ceph-mon[36715]: pgmap v5450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9294.741883 secs 2024-07-26T13:40:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:51 smithi183 ceph-mon[37273]: pgmap v5450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9294.741883 secs 2024-07-26T13:40:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:51 smithi144 ceph-mon[32129]: pgmap v5450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9294.741883 secs 2024-07-26T13:40:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:53 smithi183 ceph-mon[37273]: pgmap v5451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:53 smithi144 ceph-mon[32129]: pgmap v5451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:53 smithi160 ceph-mon[36715]: pgmap v5451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:40:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:55 smithi183 ceph-mon[37273]: pgmap v5452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:55 smithi144 ceph-mon[32129]: pgmap v5452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:55 smithi160 ceph-mon[36715]: pgmap v5452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9299.742015 secs 2024-07-26T13:40:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9299.742015 secs 2024-07-26T13:40:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9299.742015 secs 2024-07-26T13:40:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:57 smithi183 ceph-mon[37273]: pgmap v5453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:57 smithi144 ceph-mon[32129]: pgmap v5453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:57 smithi160 ceph-mon[36715]: pgmap v5453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:40:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:40:59 smithi183 ceph-mon[37273]: pgmap v5454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:40:59 smithi144 ceph-mon[32129]: pgmap v5454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:40:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:40:59 smithi160 ceph-mon[36715]: pgmap v5454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:01.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:01 smithi160 ceph-mon[36715]: pgmap v5455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:01.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9304.742138 secs 2024-07-26T13:41:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:01 smithi183 ceph-mon[37273]: pgmap v5455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9304.742138 secs 2024-07-26T13:41:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:01 smithi144 ceph-mon[32129]: pgmap v5455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9304.742138 secs 2024-07-26T13:41:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:03 smithi183 ceph-mon[37273]: pgmap v5456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:03 smithi144 ceph-mon[32129]: pgmap v5456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:03 smithi160 ceph-mon[36715]: pgmap v5456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:05.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:41:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:05.098+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:41:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:05.098+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:41:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:05.098+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:41:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:05.098+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:05 smithi183 ceph-mon[37273]: pgmap v5457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:05 smithi144 ceph-mon[32129]: pgmap v5457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:05 smithi160 ceph-mon[36715]: pgmap v5457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9309.742237 secs 2024-07-26T13:41:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9309.742237 secs 2024-07-26T13:41:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9309.742237 secs 2024-07-26T13:41:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:07 smithi183 ceph-mon[37273]: pgmap v5458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:07 smithi144 ceph-mon[32129]: pgmap v5458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:07 smithi160 ceph-mon[36715]: pgmap v5458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:09 smithi183 ceph-mon[37273]: pgmap v5459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:09 smithi144 ceph-mon[32129]: pgmap v5459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:09 smithi160 ceph-mon[36715]: pgmap v5459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:11 smithi144 ceph-mon[32129]: pgmap v5460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9314.742364 secs 2024-07-26T13:41:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:11 smithi160 ceph-mon[36715]: pgmap v5460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9314.742364 secs 2024-07-26T13:41:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:11 smithi183 ceph-mon[37273]: pgmap v5460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9314.742364 secs 2024-07-26T13:41:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:13 smithi144 ceph-mon[32129]: pgmap v5461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:13 smithi160 ceph-mon[36715]: pgmap v5461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:13 smithi183 ceph-mon[37273]: pgmap v5461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:15 smithi144 ceph-mon[32129]: pgmap v5462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:15 smithi160 ceph-mon[36715]: pgmap v5462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:15 smithi183 ceph-mon[37273]: pgmap v5462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9319.742487 secs 2024-07-26T13:41:16.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9319.742487 secs 2024-07-26T13:41:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9319.742487 secs 2024-07-26T13:41:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:17 smithi144 ceph-mon[32129]: pgmap v5463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:17 smithi160 ceph-mon[36715]: pgmap v5463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:17 smithi183 ceph-mon[37273]: pgmap v5463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:19 smithi144 ceph-mon[32129]: pgmap v5464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:19 smithi160 ceph-mon[36715]: pgmap v5464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:19 smithi183 ceph-mon[37273]: pgmap v5464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:20.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:41:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:20.106+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:41:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:20.106+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:41:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:20.106+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:41:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:20.106+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:21 smithi144 ceph-mon[32129]: pgmap v5465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9324.742596 secs 2024-07-26T13:41:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:21 smithi160 ceph-mon[36715]: pgmap v5465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9324.742596 secs 2024-07-26T13:41:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:21 smithi183 ceph-mon[37273]: pgmap v5465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9324.742596 secs 2024-07-26T13:41:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:23 smithi144 ceph-mon[32129]: pgmap v5466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:23 smithi160 ceph-mon[36715]: pgmap v5466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:23 smithi183 ceph-mon[37273]: pgmap v5466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:24 smithi144 ceph-mon[32129]: pgmap v5467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:24 smithi160 ceph-mon[36715]: pgmap v5467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:24 smithi183 ceph-mon[37273]: pgmap v5467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:26 smithi183 ceph-mon[37273]: pgmap v5468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9329.742705 secs 2024-07-26T13:41:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:26 smithi144 ceph-mon[32129]: pgmap v5468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9329.742705 secs 2024-07-26T13:41:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:26 smithi160 ceph-mon[36715]: pgmap v5468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9329.742705 secs 2024-07-26T13:41:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: pgmap v5469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: pgmap v5469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:28.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: pgmap v5469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:41:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: pgmap v5470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: pgmap v5471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:41:29.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:41:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:41:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:41:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:41:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:41:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: pgmap v5470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: pgmap v5471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:41:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: pgmap v5470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: pgmap v5471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:41:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:41:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:31 smithi144 ceph-mon[32129]: pgmap v5472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9334.742831 secs 2024-07-26T13:41:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:31 smithi160 ceph-mon[36715]: pgmap v5472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9334.742831 secs 2024-07-26T13:41:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:31 smithi183 ceph-mon[37273]: pgmap v5472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9334.742831 secs 2024-07-26T13:41:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:33 smithi144 ceph-mon[32129]: pgmap v5473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:41:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:33 smithi160 ceph-mon[36715]: pgmap v5473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:41:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:33 smithi183 ceph-mon[37273]: pgmap v5473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:41:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:35.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:41:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:35.114+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:41:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:35.114+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:41:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:35.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:41:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:35.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:35 smithi144 ceph-mon[32129]: pgmap v5474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:41:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:35 smithi160 ceph-mon[36715]: pgmap v5474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:41:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:35 smithi183 ceph-mon[37273]: pgmap v5474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:41:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9339.742927 secs 2024-07-26T13:41:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9339.742927 secs 2024-07-26T13:41:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9339.742927 secs 2024-07-26T13:41:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:37 smithi160 ceph-mon[36715]: pgmap v5475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:37 smithi183 ceph-mon[37273]: pgmap v5475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:37 smithi144 ceph-mon[32129]: pgmap v5475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:41:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:39 smithi160 ceph-mon[36715]: pgmap v5476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:41:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:39 smithi183 ceph-mon[37273]: pgmap v5476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:41:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:39 smithi144 ceph-mon[32129]: pgmap v5476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:41:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:41 smithi160 ceph-mon[36715]: pgmap v5477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9344.743052 secs 2024-07-26T13:41:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:41 smithi183 ceph-mon[37273]: pgmap v5477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9344.743052 secs 2024-07-26T13:41:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:41 smithi144 ceph-mon[32129]: pgmap v5477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9344.743052 secs 2024-07-26T13:41:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:43 smithi160 ceph-mon[36715]: pgmap v5478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:43 smithi183 ceph-mon[37273]: pgmap v5478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:43 smithi144 ceph-mon[32129]: pgmap v5478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:45 smithi160 ceph-mon[36715]: pgmap v5479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:45 smithi183 ceph-mon[37273]: pgmap v5479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:45 smithi144 ceph-mon[32129]: pgmap v5479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:47.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9349.743141 secs 2024-07-26T13:41:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9349.743141 secs 2024-07-26T13:41:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9349.743141 secs 2024-07-26T13:41:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:47.753+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:41:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:47.753+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:41:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:47.753+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:41:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:47.753+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:41:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:47 smithi160 ceph-mon[36715]: pgmap v5480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:47 smithi183 ceph-mon[37273]: pgmap v5480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:47 smithi144 ceph-mon[32129]: pgmap v5480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:49 smithi160 ceph-mon[36715]: pgmap v5481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:50.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:41:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:50.130+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:41:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:50.130+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:41:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:50.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:41:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:41:50.130+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:49 smithi183 ceph-mon[37273]: pgmap v5481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:49 smithi144 ceph-mon[32129]: pgmap v5481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:51 smithi160 ceph-mon[36715]: pgmap v5482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9354.743254 secs 2024-07-26T13:41:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:51 smithi183 ceph-mon[37273]: pgmap v5482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9354.743254 secs 2024-07-26T13:41:52.157 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:51 smithi144 ceph-mon[32129]: pgmap v5482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:52.157 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9354.743254 secs 2024-07-26T13:41:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:53 smithi160 ceph-mon[36715]: pgmap v5483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:53 smithi183 ceph-mon[37273]: pgmap v5483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:53 smithi144 ceph-mon[32129]: pgmap v5483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:41:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:55 smithi160 ceph-mon[36715]: pgmap v5484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:55 smithi183 ceph-mon[37273]: pgmap v5484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:55 smithi144 ceph-mon[32129]: pgmap v5484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:41:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9359.743367 secs 2024-07-26T13:41:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9359.743367 secs 2024-07-26T13:41:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9359.743367 secs 2024-07-26T13:41:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:57 smithi183 ceph-mon[37273]: pgmap v5485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:57 smithi144 ceph-mon[32129]: pgmap v5485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:41:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:57 smithi160 ceph-mon[36715]: pgmap v5485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:41:59 smithi183 ceph-mon[37273]: pgmap v5486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:41:59 smithi144 ceph-mon[32129]: pgmap v5486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:41:59 smithi160 ceph-mon[36715]: pgmap v5486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:02.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:01 smithi160 ceph-mon[36715]: pgmap v5487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:02.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9364.743487 secs 2024-07-26T13:42:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:01 smithi183 ceph-mon[37273]: pgmap v5487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9364.743487 secs 2024-07-26T13:42:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:01 smithi144 ceph-mon[32129]: pgmap v5487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9364.743487 secs 2024-07-26T13:42:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:03 smithi183 ceph-mon[37273]: pgmap v5488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:03 smithi144 ceph-mon[32129]: pgmap v5488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:03 smithi160 ceph-mon[36715]: pgmap v5488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:05.135+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:42:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:05.136+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:42:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:05.136+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:42:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:05.136+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:42:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:05.136+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:05 smithi183 ceph-mon[37273]: pgmap v5489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:05 smithi144 ceph-mon[32129]: pgmap v5489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:05 smithi160 ceph-mon[36715]: pgmap v5489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9369.743605 secs 2024-07-26T13:42:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9369.743605 secs 2024-07-26T13:42:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9369.743605 secs 2024-07-26T13:42:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:07 smithi183 ceph-mon[37273]: pgmap v5490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:07 smithi144 ceph-mon[32129]: pgmap v5490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:07 smithi160 ceph-mon[36715]: pgmap v5490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:09 smithi183 ceph-mon[37273]: pgmap v5491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:09 smithi144 ceph-mon[32129]: pgmap v5491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:09 smithi160 ceph-mon[36715]: pgmap v5491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:11 smithi183 ceph-mon[37273]: pgmap v5492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9374.743707 secs 2024-07-26T13:42:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:11 smithi144 ceph-mon[32129]: pgmap v5492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9374.743707 secs 2024-07-26T13:42:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:11 smithi160 ceph-mon[36715]: pgmap v5492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9374.743707 secs 2024-07-26T13:42:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:13 smithi183 ceph-mon[37273]: pgmap v5493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:13 smithi144 ceph-mon[32129]: pgmap v5493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:13 smithi160 ceph-mon[36715]: pgmap v5493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:15 smithi183 ceph-mon[37273]: pgmap v5494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:15 smithi144 ceph-mon[32129]: pgmap v5494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:15 smithi160 ceph-mon[36715]: pgmap v5494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9379.743833 secs 2024-07-26T13:42:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9379.743833 secs 2024-07-26T13:42:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9379.743833 secs 2024-07-26T13:42:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:17 smithi144 ceph-mon[32129]: pgmap v5495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:17 smithi160 ceph-mon[36715]: pgmap v5495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:17 smithi183 ceph-mon[37273]: pgmap v5495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:20.152 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:20.151+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:42:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:20.152+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:42:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:20.152+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:42:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:20.152+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:42:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:20.152+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:19 smithi144 ceph-mon[32129]: pgmap v5496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:19 smithi160 ceph-mon[36715]: pgmap v5496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:19 smithi183 ceph-mon[37273]: pgmap v5496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:21 smithi144 ceph-mon[32129]: pgmap v5497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9384.743929 secs 2024-07-26T13:42:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:21 smithi160 ceph-mon[36715]: pgmap v5497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9384.743929 secs 2024-07-26T13:42:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:21 smithi183 ceph-mon[37273]: pgmap v5497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9384.743929 secs 2024-07-26T13:42:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:23 smithi144 ceph-mon[32129]: pgmap v5498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:23 smithi160 ceph-mon[36715]: pgmap v5498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:23 smithi183 ceph-mon[37273]: pgmap v5498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:25 smithi144 ceph-mon[32129]: pgmap v5499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:25 smithi160 ceph-mon[36715]: pgmap v5499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:25 smithi183 ceph-mon[37273]: pgmap v5499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9389.744047 secs 2024-07-26T13:42:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9389.744047 secs 2024-07-26T13:42:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9389.744047 secs 2024-07-26T13:42:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:27 smithi144 ceph-mon[32129]: pgmap v5500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:27 smithi160 ceph-mon[36715]: pgmap v5500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:27 smithi183 ceph-mon[37273]: pgmap v5500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:42:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:29 smithi144 ceph-mon[32129]: pgmap v5501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:42:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:42:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:42:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:29 smithi160 ceph-mon[36715]: pgmap v5501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:42:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:42:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:42:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:29 smithi183 ceph-mon[37273]: pgmap v5501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:42:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:42:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:42:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:30 smithi144 ceph-mon[32129]: pgmap v5502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:42:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:30 smithi144 ceph-mon[32129]: pgmap v5503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:31.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:30 smithi160 ceph-mon[36715]: pgmap v5502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:42:31.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:30 smithi160 ceph-mon[36715]: pgmap v5503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:30 smithi183 ceph-mon[37273]: pgmap v5502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:42:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:30 smithi183 ceph-mon[37273]: pgmap v5503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:32.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9394.744166 secs 2024-07-26T13:42:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9394.744166 secs 2024-07-26T13:42:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9394.744166 secs 2024-07-26T13:42:32.762 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:32.762+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:42:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:32.762+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:42:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:32.762+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:42:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:32.762+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:42:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:32 smithi144 ceph-mon[32129]: pgmap v5504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:42:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:32 smithi160 ceph-mon[36715]: pgmap v5504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:42:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:32 smithi183 ceph-mon[37273]: pgmap v5504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:42:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:35.157+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:42:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:35.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:42:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:35.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:42:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:35.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:42:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:35.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:34 smithi144 ceph-mon[32129]: pgmap v5505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:42:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:34 smithi160 ceph-mon[36715]: pgmap v5505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:42:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:34 smithi183 ceph-mon[37273]: pgmap v5505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:42:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9399.744312 secs 2024-07-26T13:42:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:36 smithi160 ceph-mon[36715]: pgmap v5506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:42:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9399.744312 secs 2024-07-26T13:42:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:36 smithi183 ceph-mon[37273]: pgmap v5506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:42:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9399.744312 secs 2024-07-26T13:42:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:36 smithi144 ceph-mon[32129]: pgmap v5506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:42:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:38 smithi160 ceph-mon[36715]: pgmap v5507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:38 smithi183 ceph-mon[37273]: pgmap v5507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:38 smithi144 ceph-mon[32129]: pgmap v5507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:40 smithi160 ceph-mon[36715]: pgmap v5508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:40 smithi183 ceph-mon[37273]: pgmap v5508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:40 smithi144 ceph-mon[32129]: pgmap v5508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9404.744363 secs 2024-07-26T13:42:42.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9404.744363 secs 2024-07-26T13:42:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9404.744363 secs 2024-07-26T13:42:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:42 smithi160 ceph-mon[36715]: pgmap v5509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:42 smithi183 ceph-mon[37273]: pgmap v5509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:42 smithi144 ceph-mon[32129]: pgmap v5509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:45 smithi160 ceph-mon[36715]: pgmap v5510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:45 smithi183 ceph-mon[37273]: pgmap v5510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:45 smithi144 ceph-mon[32129]: pgmap v5510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9409.744481 secs 2024-07-26T13:42:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:47 smithi160 ceph-mon[36715]: pgmap v5511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9409.744481 secs 2024-07-26T13:42:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:47 smithi183 ceph-mon[37273]: pgmap v5511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9409.744481 secs 2024-07-26T13:42:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:47 smithi144 ceph-mon[32129]: pgmap v5511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:49 smithi160 ceph-mon[36715]: pgmap v5512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:49 smithi183 ceph-mon[37273]: pgmap v5512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:49 smithi144 ceph-mon[32129]: pgmap v5512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:42:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:50.171+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:42:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:50.172+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:42:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:50.172+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:42:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:50.172+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:42:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:42:50.172+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:51 smithi160 ceph-mon[36715]: pgmap v5513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:51 smithi183 ceph-mon[37273]: pgmap v5513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:51 smithi144 ceph-mon[32129]: pgmap v5513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:42:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9414.744604 secs 2024-07-26T13:42:52.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9414.744604 secs 2024-07-26T13:42:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9414.744604 secs 2024-07-26T13:42:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:53 smithi160 ceph-mon[36715]: pgmap v5514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:53 smithi183 ceph-mon[37273]: pgmap v5514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:53 smithi144 ceph-mon[32129]: pgmap v5514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:42:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:55 smithi160 ceph-mon[36715]: pgmap v5515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:55 smithi183 ceph-mon[37273]: pgmap v5515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:55 smithi144 ceph-mon[32129]: pgmap v5515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9419.744669 secs 2024-07-26T13:42:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9419.744669 secs 2024-07-26T13:42:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9419.744669 secs 2024-07-26T13:42:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:57 smithi160 ceph-mon[36715]: pgmap v5516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:57 smithi183 ceph-mon[37273]: pgmap v5516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:57 smithi144 ceph-mon[32129]: pgmap v5516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:42:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:42:59 smithi183 ceph-mon[37273]: pgmap v5517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:42:59 smithi144 ceph-mon[32129]: pgmap v5517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:42:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:42:59 smithi160 ceph-mon[36715]: pgmap v5517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:00 smithi160 ceph-mon[36715]: pgmap v5518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:00 smithi183 ceph-mon[37273]: pgmap v5518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:00 smithi144 ceph-mon[32129]: pgmap v5518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9424.744754 secs 2024-07-26T13:43:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9424.744754 secs 2024-07-26T13:43:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9424.744754 secs 2024-07-26T13:43:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:02 smithi183 ceph-mon[37273]: pgmap v5519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:03.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:02 smithi160 ceph-mon[36715]: pgmap v5519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:02 smithi144 ceph-mon[32129]: pgmap v5519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:05.184+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:43:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:05.185+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:43:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:05.185+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:43:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:05.185+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:43:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:05.185+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:04 smithi144 ceph-mon[32129]: pgmap v5520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:04 smithi160 ceph-mon[36715]: pgmap v5520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:04 smithi183 ceph-mon[37273]: pgmap v5520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9429.744877 secs 2024-07-26T13:43:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:06 smithi144 ceph-mon[32129]: pgmap v5521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9429.744877 secs 2024-07-26T13:43:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:06 smithi160 ceph-mon[36715]: pgmap v5521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9429.744877 secs 2024-07-26T13:43:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:06 smithi183 ceph-mon[37273]: pgmap v5521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:08 smithi144 ceph-mon[32129]: pgmap v5522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:08 smithi160 ceph-mon[36715]: pgmap v5522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:08 smithi183 ceph-mon[37273]: pgmap v5522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:10 smithi144 ceph-mon[32129]: pgmap v5523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:10 smithi160 ceph-mon[36715]: pgmap v5523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:10 smithi183 ceph-mon[37273]: pgmap v5523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9434.744996 secs 2024-07-26T13:43:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9434.744996 secs 2024-07-26T13:43:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9434.744996 secs 2024-07-26T13:43:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:12 smithi144 ceph-mon[32129]: pgmap v5524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:12 smithi160 ceph-mon[36715]: pgmap v5524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:12 smithi183 ceph-mon[37273]: pgmap v5524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:14 smithi144 ceph-mon[32129]: pgmap v5525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:14 smithi160 ceph-mon[36715]: pgmap v5525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:14 smithi183 ceph-mon[37273]: pgmap v5525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9439.745113 secs 2024-07-26T13:43:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:16 smithi160 ceph-mon[36715]: pgmap v5526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9439.745113 secs 2024-07-26T13:43:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:16 smithi183 ceph-mon[37273]: pgmap v5526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9439.745113 secs 2024-07-26T13:43:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:16 smithi144 ceph-mon[32129]: pgmap v5526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:18 smithi160 ceph-mon[36715]: pgmap v5527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:18 smithi183 ceph-mon[37273]: pgmap v5527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:18 smithi144 ceph-mon[32129]: pgmap v5527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:20.189+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:43:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:20.190+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:43:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:20.190+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:43:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:20.190+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:43:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:20.190+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:20 smithi160 ceph-mon[36715]: pgmap v5528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:20 smithi183 ceph-mon[37273]: pgmap v5528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:20 smithi144 ceph-mon[32129]: pgmap v5528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9444.745232 secs 2024-07-26T13:43:22.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9444.745232 secs 2024-07-26T13:43:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9444.745232 secs 2024-07-26T13:43:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:22 smithi160 ceph-mon[36715]: pgmap v5529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:22 smithi183 ceph-mon[37273]: pgmap v5529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:22 smithi144 ceph-mon[32129]: pgmap v5529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:24 smithi144 ceph-mon[32129]: pgmap v5530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:24 smithi160 ceph-mon[36715]: pgmap v5530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:24 smithi183 ceph-mon[37273]: pgmap v5530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9449.745353 secs 2024-07-26T13:43:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:27 smithi160 ceph-mon[36715]: pgmap v5531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9449.745353 secs 2024-07-26T13:43:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:27 smithi183 ceph-mon[37273]: pgmap v5531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9449.745353 secs 2024-07-26T13:43:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:27 smithi144 ceph-mon[32129]: pgmap v5531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:29 smithi160 ceph-mon[36715]: pgmap v5532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:29 smithi183 ceph-mon[37273]: pgmap v5532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:29 smithi144 ceph-mon[32129]: pgmap v5532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:31 smithi160 ceph-mon[36715]: pgmap v5533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:31 smithi183 ceph-mon[37273]: pgmap v5533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:31 smithi144 ceph-mon[32129]: pgmap v5533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9454.745453 secs 2024-07-26T13:43:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9454.745453 secs 2024-07-26T13:43:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9454.745453 secs 2024-07-26T13:43:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:43:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:33 smithi160 ceph-mon[36715]: pgmap v5534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:33 smithi160 ceph-mon[36715]: pgmap v5535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:33 smithi183 ceph-mon[37273]: pgmap v5534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:33 smithi183 ceph-mon[37273]: pgmap v5535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:33 smithi144 ceph-mon[32129]: pgmap v5534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:33 smithi144 ceph-mon[32129]: pgmap v5535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:35.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:35.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:43:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:35.207+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:43:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:35.207+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:43:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:35.207+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:43:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:35.207+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:35 smithi160 ceph-mon[36715]: pgmap v5536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:35 smithi183 ceph-mon[37273]: pgmap v5536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:35 smithi144 ceph-mon[32129]: pgmap v5536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9459.745521 secs 2024-07-26T13:43:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9459.745521 secs 2024-07-26T13:43:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9459.745521 secs 2024-07-26T13:43:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:37 smithi160 ceph-mon[36715]: pgmap v5537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:37 smithi183 ceph-mon[37273]: pgmap v5537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:37 smithi144 ceph-mon[32129]: pgmap v5537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:39 smithi183 ceph-mon[37273]: pgmap v5538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:39 smithi144 ceph-mon[32129]: pgmap v5538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:39 smithi160 ceph-mon[36715]: pgmap v5538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:41 smithi183 ceph-mon[37273]: pgmap v5539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9464.745658 secs 2024-07-26T13:43:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:41 smithi144 ceph-mon[32129]: pgmap v5539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9464.745658 secs 2024-07-26T13:43:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:41 smithi160 ceph-mon[36715]: pgmap v5539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:43:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9464.745658 secs 2024-07-26T13:43:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:43 smithi183 ceph-mon[37273]: pgmap v5540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:43:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:43 smithi144 ceph-mon[32129]: pgmap v5540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:43:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:43 smithi160 ceph-mon[36715]: pgmap v5540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:43:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:45 smithi183 ceph-mon[37273]: pgmap v5541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:45 smithi144 ceph-mon[32129]: pgmap v5541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:45 smithi160 ceph-mon[36715]: pgmap v5541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9469.745769 secs 2024-07-26T13:43:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9469.745769 secs 2024-07-26T13:43:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9469.745769 secs 2024-07-26T13:43:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:47 smithi183 ceph-mon[37273]: pgmap v5542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:47 smithi144 ceph-mon[32129]: pgmap v5542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:47 smithi160 ceph-mon[36715]: pgmap v5542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:49 smithi183 ceph-mon[37273]: pgmap v5543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:49 smithi144 ceph-mon[32129]: pgmap v5543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:49 smithi160 ceph-mon[36715]: pgmap v5543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:50.222+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:43:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:50.223+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:43:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:50.223+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:43:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:50.223+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:43:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:43:50.223+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:51 smithi183 ceph-mon[37273]: pgmap v5544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9474.745904 secs 2024-07-26T13:43:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:51 smithi144 ceph-mon[32129]: pgmap v5544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9474.745904 secs 2024-07-26T13:43:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:51 smithi160 ceph-mon[36715]: pgmap v5544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9474.745904 secs 2024-07-26T13:43:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:53 smithi183 ceph-mon[37273]: pgmap v5545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:53 smithi144 ceph-mon[32129]: pgmap v5545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:53 smithi160 ceph-mon[36715]: pgmap v5545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:43:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:55 smithi183 ceph-mon[37273]: pgmap v5546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:55 smithi144 ceph-mon[32129]: pgmap v5546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:55 smithi160 ceph-mon[36715]: pgmap v5546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:56.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9479.746005 secs 2024-07-26T13:43:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9479.746005 secs 2024-07-26T13:43:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9479.746005 secs 2024-07-26T13:43:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:57 smithi183 ceph-mon[37273]: pgmap v5547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:57 smithi144 ceph-mon[32129]: pgmap v5547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:57 smithi160 ceph-mon[36715]: pgmap v5547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:43:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:43:59 smithi183 ceph-mon[37273]: pgmap v5548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:43:59 smithi144 ceph-mon[32129]: pgmap v5548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:43:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:43:59 smithi160 ceph-mon[36715]: pgmap v5548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:01 smithi144 ceph-mon[32129]: pgmap v5549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9484.746147 secs 2024-07-26T13:44:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:01 smithi160 ceph-mon[36715]: pgmap v5549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9484.746147 secs 2024-07-26T13:44:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:01 smithi183 ceph-mon[37273]: pgmap v5549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9484.746147 secs 2024-07-26T13:44:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:03 smithi144 ceph-mon[32129]: pgmap v5550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:03 smithi160 ceph-mon[36715]: pgmap v5550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:03 smithi183 ceph-mon[37273]: pgmap v5550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:05.240+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:44:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:05.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:44:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:05.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:44:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:05.241+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:44:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:05.241+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:05 smithi144 ceph-mon[32129]: pgmap v5551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:05 smithi160 ceph-mon[36715]: pgmap v5551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:05 smithi183 ceph-mon[37273]: pgmap v5551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9489.746283 secs 2024-07-26T13:44:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9489.746283 secs 2024-07-26T13:44:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9489.746283 secs 2024-07-26T13:44:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:07 smithi144 ceph-mon[32129]: pgmap v5552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:07 smithi160 ceph-mon[36715]: pgmap v5552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:07 smithi183 ceph-mon[37273]: pgmap v5552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:09 smithi144 ceph-mon[32129]: pgmap v5553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:09 smithi160 ceph-mon[36715]: pgmap v5553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:09 smithi183 ceph-mon[37273]: pgmap v5553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:11 smithi144 ceph-mon[32129]: pgmap v5554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9494.746429 secs 2024-07-26T13:44:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:11 smithi160 ceph-mon[36715]: pgmap v5554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9494.746429 secs 2024-07-26T13:44:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:11 smithi183 ceph-mon[37273]: pgmap v5554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9494.746429 secs 2024-07-26T13:44:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:13 smithi144 ceph-mon[32129]: pgmap v5555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:13 smithi160 ceph-mon[36715]: pgmap v5555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:13 smithi183 ceph-mon[37273]: pgmap v5555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:15 smithi144 ceph-mon[32129]: pgmap v5556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:15 smithi160 ceph-mon[36715]: pgmap v5556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:15 smithi183 ceph-mon[37273]: pgmap v5556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9499.746496 secs 2024-07-26T13:44:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9499.746496 secs 2024-07-26T13:44:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9499.746496 secs 2024-07-26T13:44:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:17 smithi144 ceph-mon[32129]: pgmap v5557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:17.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:17 smithi183 ceph-mon[37273]: pgmap v5557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:17 smithi160 ceph-mon[36715]: pgmap v5557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:19 smithi144 ceph-mon[32129]: pgmap v5558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:19 smithi160 ceph-mon[36715]: pgmap v5558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:19 smithi183 ceph-mon[37273]: pgmap v5558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:20.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:44:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:20.256+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:44:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:20.256+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:44:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:20.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:44:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:20.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:21 smithi144 ceph-mon[32129]: pgmap v5559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9504.746617 secs 2024-07-26T13:44:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:21 smithi160 ceph-mon[36715]: pgmap v5559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9504.746617 secs 2024-07-26T13:44:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:21 smithi183 ceph-mon[37273]: pgmap v5559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9504.746617 secs 2024-07-26T13:44:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:23 smithi144 ceph-mon[32129]: pgmap v5560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:23 smithi160 ceph-mon[36715]: pgmap v5560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:23 smithi183 ceph-mon[37273]: pgmap v5560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:25 smithi160 ceph-mon[36715]: pgmap v5561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:25 smithi183 ceph-mon[37273]: pgmap v5561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:25 smithi144 ceph-mon[32129]: pgmap v5561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:26 smithi160 ceph-mon[36715]: pgmap v5562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9509.746765 secs 2024-07-26T13:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:26 smithi183 ceph-mon[37273]: pgmap v5562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9509.746765 secs 2024-07-26T13:44:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:26 smithi144 ceph-mon[32129]: pgmap v5562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9509.746765 secs 2024-07-26T13:44:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:28 smithi160 ceph-mon[36715]: pgmap v5563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:28 smithi183 ceph-mon[37273]: pgmap v5563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:29.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:28 smithi144 ceph-mon[32129]: pgmap v5563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:30 smithi160 ceph-mon[36715]: pgmap v5564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:30 smithi183 ceph-mon[37273]: pgmap v5564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:30 smithi144 ceph-mon[32129]: pgmap v5564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9514.746866 secs 2024-07-26T13:44:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9514.746866 secs 2024-07-26T13:44:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9514.746866 secs 2024-07-26T13:44:33.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:32 smithi160 ceph-mon[36715]: pgmap v5565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:33.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:33.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.095 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:32 smithi183 ceph-mon[37273]: pgmap v5565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:33.095 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:33.095 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.095 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.095 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.095 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.103 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:32 smithi144 ceph-mon[32129]: pgmap v5565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:33.103 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:33.103 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.103 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.103 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:33.103 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:34 smithi144 ceph-mon[32129]: pgmap v5566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.273 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:35.272+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:44:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:35.273+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:44:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:35.273+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:44:35.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:35.273+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:44:35.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:35.273+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:34 smithi160 ceph-mon[36715]: pgmap v5566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:34 smithi183 ceph-mon[37273]: pgmap v5566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:36 smithi144 ceph-mon[32129]: pgmap v5567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9519.746999 secs 2024-07-26T13:44:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:36 smithi160 ceph-mon[36715]: pgmap v5567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9519.746999 secs 2024-07-26T13:44:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:36 smithi183 ceph-mon[37273]: pgmap v5567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9519.746999 secs 2024-07-26T13:44:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:37 smithi144 ceph-mon[32129]: pgmap v5568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:44:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:37 smithi144 ceph-mon[32129]: pgmap v5569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:44:37.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:37 smithi160 ceph-mon[36715]: pgmap v5568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:44:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:37 smithi160 ceph-mon[36715]: pgmap v5569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:44:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:37 smithi183 ceph-mon[37273]: pgmap v5568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:44:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:37 smithi183 ceph-mon[37273]: pgmap v5569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:44:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:44:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:39 smithi160 ceph-mon[36715]: pgmap v5570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:44:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:39 smithi183 ceph-mon[37273]: pgmap v5570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:44:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:39 smithi144 ceph-mon[32129]: pgmap v5570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:44:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:41 smithi160 ceph-mon[36715]: pgmap v5571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:44:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9524.747102 secs 2024-07-26T13:44:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:41 smithi183 ceph-mon[37273]: pgmap v5571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:44:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9524.747102 secs 2024-07-26T13:44:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:41 smithi144 ceph-mon[32129]: pgmap v5571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:44:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9524.747102 secs 2024-07-26T13:44:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:43 smithi183 ceph-mon[37273]: pgmap v5572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:43 smithi144 ceph-mon[32129]: pgmap v5572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:43 smithi160 ceph-mon[36715]: pgmap v5572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:45 smithi183 ceph-mon[37273]: pgmap v5573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:45 smithi144 ceph-mon[32129]: pgmap v5573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:45 smithi160 ceph-mon[36715]: pgmap v5573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9529.747226 secs 2024-07-26T13:44:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9529.747226 secs 2024-07-26T13:44:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9529.747226 secs 2024-07-26T13:44:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:47.784+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:44:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:47.784+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780065a0 con 0x563c6c4182f0 2024-07-26T13:44:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:47.784+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780065a0 con 0x7f74b0109990 2024-07-26T13:44:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:47.785+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:44:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:47 smithi183 ceph-mon[37273]: pgmap v5574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:44:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:47 smithi144 ceph-mon[32129]: pgmap v5574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:44:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:47 smithi160 ceph-mon[36715]: pgmap v5574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:44:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:49 smithi183 ceph-mon[37273]: pgmap v5575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:49 smithi144 ceph-mon[32129]: pgmap v5575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:50.279+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:44:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:50.280+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:44:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:50.280+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:44:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:50.280+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:44:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:44:50.280+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:49 smithi160 ceph-mon[36715]: pgmap v5575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:51 smithi183 ceph-mon[37273]: pgmap v5576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9534.747355 secs 2024-07-26T13:44:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:51 smithi144 ceph-mon[32129]: pgmap v5576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9534.747355 secs 2024-07-26T13:44:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:51 smithi160 ceph-mon[36715]: pgmap v5576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9534.747355 secs 2024-07-26T13:44:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:53 smithi183 ceph-mon[37273]: pgmap v5577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:53 smithi144 ceph-mon[32129]: pgmap v5577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:53 smithi160 ceph-mon[36715]: pgmap v5577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:44:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:55 smithi183 ceph-mon[37273]: pgmap v5578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:55 smithi144 ceph-mon[32129]: pgmap v5578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:55 smithi160 ceph-mon[36715]: pgmap v5578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:44:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9539.747421 secs 2024-07-26T13:44:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9539.747421 secs 2024-07-26T13:44:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9539.747421 secs 2024-07-26T13:44:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:57 smithi183 ceph-mon[37273]: pgmap v5579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:57 smithi144 ceph-mon[32129]: pgmap v5579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:44:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:57 smithi160 ceph-mon[36715]: pgmap v5579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:44:59 smithi183 ceph-mon[37273]: pgmap v5580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:44:59 smithi144 ceph-mon[32129]: pgmap v5580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:44:59 smithi160 ceph-mon[36715]: pgmap v5580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:01 smithi183 ceph-mon[37273]: pgmap v5581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9544.747516 secs 2024-07-26T13:45:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:01 smithi144 ceph-mon[32129]: pgmap v5581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9544.747516 secs 2024-07-26T13:45:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:01 smithi160 ceph-mon[36715]: pgmap v5581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9544.747516 secs 2024-07-26T13:45:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:03 smithi183 ceph-mon[37273]: pgmap v5582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:03 smithi144 ceph-mon[32129]: pgmap v5582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:03 smithi160 ceph-mon[36715]: pgmap v5582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:05.289+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:45:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:05.290+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:45:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:05.290+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:45:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:05.290+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:45:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:05.290+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:05 smithi144 ceph-mon[32129]: pgmap v5583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:05 smithi160 ceph-mon[36715]: pgmap v5583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:05 smithi183 ceph-mon[37273]: pgmap v5583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9549.747635 secs 2024-07-26T13:45:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9549.747635 secs 2024-07-26T13:45:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9549.747635 secs 2024-07-26T13:45:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:07 smithi144 ceph-mon[32129]: pgmap v5584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:07 smithi160 ceph-mon[36715]: pgmap v5584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:07 smithi183 ceph-mon[37273]: pgmap v5584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:09 smithi144 ceph-mon[32129]: pgmap v5585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:09 smithi160 ceph-mon[36715]: pgmap v5585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:09 smithi183 ceph-mon[37273]: pgmap v5585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:11 smithi144 ceph-mon[32129]: pgmap v5586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9554.747715 secs 2024-07-26T13:45:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:11 smithi160 ceph-mon[36715]: pgmap v5586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9554.747715 secs 2024-07-26T13:45:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:11 smithi183 ceph-mon[37273]: pgmap v5586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9554.747715 secs 2024-07-26T13:45:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:13 smithi144 ceph-mon[32129]: pgmap v5587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:13 smithi160 ceph-mon[36715]: pgmap v5587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:13 smithi183 ceph-mon[37273]: pgmap v5587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:15 smithi144 ceph-mon[32129]: pgmap v5588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:15 smithi160 ceph-mon[36715]: pgmap v5588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:15 smithi183 ceph-mon[37273]: pgmap v5588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9559.747858 secs 2024-07-26T13:45:17.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9559.747858 secs 2024-07-26T13:45:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9559.747858 secs 2024-07-26T13:45:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:17 smithi144 ceph-mon[32129]: pgmap v5589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:17 smithi160 ceph-mon[36715]: pgmap v5589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:17 smithi183 ceph-mon[37273]: pgmap v5589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:19 smithi144 ceph-mon[32129]: pgmap v5590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:20.306+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:45:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:20.307+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:45:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:20.307+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:45:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:20.307+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:45:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:20.307+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:19 smithi160 ceph-mon[36715]: pgmap v5590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:19 smithi183 ceph-mon[37273]: pgmap v5590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9564.747987 secs 2024-07-26T13:45:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:21 smithi144 ceph-mon[32129]: pgmap v5591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9564.747987 secs 2024-07-26T13:45:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:21 smithi160 ceph-mon[36715]: pgmap v5591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9564.747987 secs 2024-07-26T13:45:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:21 smithi183 ceph-mon[37273]: pgmap v5591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:23 smithi144 ceph-mon[32129]: pgmap v5592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:23 smithi160 ceph-mon[36715]: pgmap v5592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:23 smithi183 ceph-mon[37273]: pgmap v5592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:25 smithi144 ceph-mon[32129]: pgmap v5593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:25 smithi160 ceph-mon[36715]: pgmap v5593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:25 smithi183 ceph-mon[37273]: pgmap v5593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9569.748130 secs 2024-07-26T13:45:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9569.748130 secs 2024-07-26T13:45:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9569.748130 secs 2024-07-26T13:45:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:27 smithi160 ceph-mon[36715]: pgmap v5594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:27 smithi183 ceph-mon[37273]: pgmap v5594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:27 smithi144 ceph-mon[32129]: pgmap v5594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:29 smithi160 ceph-mon[36715]: pgmap v5595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:29 smithi183 ceph-mon[37273]: pgmap v5595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:29 smithi144 ceph-mon[32129]: pgmap v5595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9574.748273 secs 2024-07-26T13:45:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:31 smithi160 ceph-mon[36715]: pgmap v5596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:32.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9574.748273 secs 2024-07-26T13:45:32.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:31 smithi183 ceph-mon[37273]: pgmap v5596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9574.748273 secs 2024-07-26T13:45:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:31 smithi144 ceph-mon[32129]: pgmap v5596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:33 smithi160 ceph-mon[36715]: pgmap v5597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:33 smithi183 ceph-mon[37273]: pgmap v5597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:33 smithi144 ceph-mon[32129]: pgmap v5597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:35.323+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:45:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:35.324+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:45:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:35.324+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:45:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:35.324+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:45:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:35.324+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:35 smithi160 ceph-mon[36715]: pgmap v5598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:35 smithi183 ceph-mon[37273]: pgmap v5598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:35 smithi144 ceph-mon[32129]: pgmap v5598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9579.748403 secs 2024-07-26T13:45:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9579.748403 secs 2024-07-26T13:45:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9579.748403 secs 2024-07-26T13:45:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:38 smithi160 ceph-mon[36715]: pgmap v5599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:38.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:45:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:38 smithi183 ceph-mon[37273]: pgmap v5599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:45:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:38 smithi144 ceph-mon[32129]: pgmap v5599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:45:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:39 smithi160 ceph-mon[36715]: pgmap v5600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:45:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:39 smithi160 ceph-mon[36715]: pgmap v5601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:45:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:39 smithi183 ceph-mon[37273]: pgmap v5600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:45:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:39 smithi183 ceph-mon[37273]: pgmap v5601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:45:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:39 smithi144 ceph-mon[32129]: pgmap v5600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:45:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:39 smithi144 ceph-mon[32129]: pgmap v5601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:45:41.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:41 smithi160 ceph-mon[36715]: pgmap v5602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:45:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:41 smithi183 ceph-mon[37273]: pgmap v5602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:45:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:41 smithi144 ceph-mon[32129]: pgmap v5602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:45:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9584.748470 secs 2024-07-26T13:45:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9584.748470 secs 2024-07-26T13:45:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9584.748470 secs 2024-07-26T13:45:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:43 smithi160 ceph-mon[36715]: pgmap v5603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:45:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:43 smithi183 ceph-mon[37273]: pgmap v5603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:45:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:43 smithi144 ceph-mon[32129]: pgmap v5603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:45:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:45 smithi160 ceph-mon[36715]: pgmap v5604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:45:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:45 smithi183 ceph-mon[37273]: pgmap v5604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:45:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:45 smithi144 ceph-mon[32129]: pgmap v5604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:45:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9589.748617 secs 2024-07-26T13:45:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:47 smithi160 ceph-mon[36715]: pgmap v5605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:45:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9589.748617 secs 2024-07-26T13:45:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:47 smithi183 ceph-mon[37273]: pgmap v5605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:45:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9589.748617 secs 2024-07-26T13:45:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:47 smithi144 ceph-mon[32129]: pgmap v5605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:45:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:49 smithi160 ceph-mon[36715]: pgmap v5606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:45:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:49 smithi183 ceph-mon[37273]: pgmap v5606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:45:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:49 smithi144 ceph-mon[32129]: pgmap v5606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:45:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:50.331+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:45:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:50.332+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:45:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:50.332+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:45:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:50.332+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:45:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:45:50.332+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:51 smithi160 ceph-mon[36715]: pgmap v5607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:51 smithi183 ceph-mon[37273]: pgmap v5607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:51 smithi144 ceph-mon[32129]: pgmap v5607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9594.748699 secs 2024-07-26T13:45:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9594.748699 secs 2024-07-26T13:45:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9594.748699 secs 2024-07-26T13:45:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:53 smithi183 ceph-mon[37273]: pgmap v5608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:53 smithi144 ceph-mon[32129]: pgmap v5608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:53 smithi160 ceph-mon[36715]: pgmap v5608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:45:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:55 smithi183 ceph-mon[37273]: pgmap v5609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:55 smithi144 ceph-mon[32129]: pgmap v5609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:55 smithi160 ceph-mon[36715]: pgmap v5609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9599.748828 secs 2024-07-26T13:45:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9599.748828 secs 2024-07-26T13:45:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9599.748828 secs 2024-07-26T13:45:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:57 smithi183 ceph-mon[37273]: pgmap v5610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:57 smithi144 ceph-mon[32129]: pgmap v5610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:57 smithi160 ceph-mon[36715]: pgmap v5610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:45:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:45:59 smithi183 ceph-mon[37273]: pgmap v5611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:45:59 smithi144 ceph-mon[32129]: pgmap v5611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:45:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:45:59 smithi160 ceph-mon[36715]: pgmap v5611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:01.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:01 smithi144 ceph-mon[32129]: pgmap v5612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:01.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9604.748949 secs 2024-07-26T13:46:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:01 smithi183 ceph-mon[37273]: pgmap v5612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9604.748949 secs 2024-07-26T13:46:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:01 smithi160 ceph-mon[36715]: pgmap v5612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9604.748949 secs 2024-07-26T13:46:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:03 smithi183 ceph-mon[37273]: pgmap v5613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:03 smithi144 ceph-mon[32129]: pgmap v5613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:03 smithi160 ceph-mon[36715]: pgmap v5613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:05.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:46:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:05.350+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:46:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:05.350+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:46:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:05.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:46:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:05.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:05 smithi183 ceph-mon[37273]: pgmap v5614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:05 smithi144 ceph-mon[32129]: pgmap v5614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:05 smithi160 ceph-mon[36715]: pgmap v5614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9609.749092 secs 2024-07-26T13:46:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9609.749092 secs 2024-07-26T13:46:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9609.749092 secs 2024-07-26T13:46:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:07 smithi183 ceph-mon[37273]: pgmap v5615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:07 smithi144 ceph-mon[32129]: pgmap v5615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:07 smithi160 ceph-mon[36715]: pgmap v5615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:09 smithi144 ceph-mon[32129]: pgmap v5616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:09 smithi160 ceph-mon[36715]: pgmap v5616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:09 smithi183 ceph-mon[37273]: pgmap v5616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:11 smithi144 ceph-mon[32129]: pgmap v5617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9614.749172 secs 2024-07-26T13:46:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:11 smithi160 ceph-mon[36715]: pgmap v5617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9614.749172 secs 2024-07-26T13:46:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:11 smithi183 ceph-mon[37273]: pgmap v5617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9614.749172 secs 2024-07-26T13:46:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:13 smithi144 ceph-mon[32129]: pgmap v5618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:13 smithi160 ceph-mon[36715]: pgmap v5618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:13 smithi183 ceph-mon[37273]: pgmap v5618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:15 smithi144 ceph-mon[32129]: pgmap v5619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:15 smithi160 ceph-mon[36715]: pgmap v5619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:15 smithi183 ceph-mon[37273]: pgmap v5619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9619.749254 secs 2024-07-26T13:46:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9619.749254 secs 2024-07-26T13:46:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9619.749254 secs 2024-07-26T13:46:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:17 smithi144 ceph-mon[32129]: pgmap v5620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:17.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:17 smithi183 ceph-mon[37273]: pgmap v5620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:17 smithi160 ceph-mon[36715]: pgmap v5620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:19 smithi144 ceph-mon[32129]: pgmap v5621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:19 smithi160 ceph-mon[36715]: pgmap v5621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:19 smithi183 ceph-mon[37273]: pgmap v5621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:20.361+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:46:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:20.362+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:46:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:20.362+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:46:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:20.362+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:46:20.365 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:20.362+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:21 smithi144 ceph-mon[32129]: pgmap v5622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9624.749382 secs 2024-07-26T13:46:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:21 smithi160 ceph-mon[36715]: pgmap v5622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9624.749382 secs 2024-07-26T13:46:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:21 smithi183 ceph-mon[37273]: pgmap v5622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9624.749382 secs 2024-07-26T13:46:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:23 smithi144 ceph-mon[32129]: pgmap v5623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:23 smithi160 ceph-mon[36715]: pgmap v5623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:23 smithi183 ceph-mon[37273]: pgmap v5623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:25 smithi144 ceph-mon[32129]: pgmap v5624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:25 smithi160 ceph-mon[36715]: pgmap v5624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:25 smithi183 ceph-mon[37273]: pgmap v5624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9629.749478 secs 2024-07-26T13:46:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9629.749478 secs 2024-07-26T13:46:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9629.749478 secs 2024-07-26T13:46:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:27 smithi144 ceph-mon[32129]: pgmap v5625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:27 smithi160 ceph-mon[36715]: pgmap v5625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:27 smithi183 ceph-mon[37273]: pgmap v5625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:29 smithi144 ceph-mon[32129]: pgmap v5626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:29 smithi160 ceph-mon[36715]: pgmap v5626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:29 smithi183 ceph-mon[37273]: pgmap v5626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:31 smithi144 ceph-mon[32129]: pgmap v5627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:46:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9634.749608 secs 2024-07-26T13:46:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:31 smithi160 ceph-mon[36715]: pgmap v5627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:46:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9634.749608 secs 2024-07-26T13:46:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:31 smithi183 ceph-mon[37273]: pgmap v5627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:46:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9634.749608 secs 2024-07-26T13:46:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:33 smithi160 ceph-mon[36715]: pgmap v5628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:33 smithi183 ceph-mon[37273]: pgmap v5628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:33 smithi144 ceph-mon[32129]: pgmap v5628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:35.375+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:46:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:35.376+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:46:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:35.376+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:46:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:35.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:46:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:35.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:35 smithi160 ceph-mon[36715]: pgmap v5629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:46:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:35 smithi183 ceph-mon[37273]: pgmap v5629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:46:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:35 smithi144 ceph-mon[32129]: pgmap v5629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:46:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9639.749736 secs 2024-07-26T13:46:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9639.749736 secs 2024-07-26T13:46:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9639.749736 secs 2024-07-26T13:46:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:37 smithi160 ceph-mon[36715]: pgmap v5630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:37 smithi183 ceph-mon[37273]: pgmap v5630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:37 smithi144 ceph-mon[32129]: pgmap v5630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:39 smithi160 ceph-mon[36715]: pgmap v5631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:46:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:39 smithi183 ceph-mon[37273]: pgmap v5631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:46:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:39 smithi144 ceph-mon[32129]: pgmap v5631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:46:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:40 smithi160 ceph-mon[36715]: pgmap v5632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:40 smithi160 ceph-mon[36715]: pgmap v5633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:40 smithi183 ceph-mon[37273]: pgmap v5632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:40 smithi183 ceph-mon[37273]: pgmap v5633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:40 smithi144 ceph-mon[32129]: pgmap v5632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:40 smithi144 ceph-mon[32129]: pgmap v5633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9644.749885 secs 2024-07-26T13:46:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:41 smithi160 ceph-mon[36715]: pgmap v5634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9644.749885 secs 2024-07-26T13:46:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:41 smithi183 ceph-mon[37273]: pgmap v5634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9644.749885 secs 2024-07-26T13:46:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:41 smithi144 ceph-mon[32129]: pgmap v5634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:44 smithi144 ceph-mon[32129]: pgmap v5635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:44 smithi160 ceph-mon[36715]: pgmap v5635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:44 smithi183 ceph-mon[37273]: pgmap v5635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:46 smithi144 ceph-mon[32129]: pgmap v5636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9649.750016 secs 2024-07-26T13:46:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:46 smithi160 ceph-mon[36715]: pgmap v5636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9649.750016 secs 2024-07-26T13:46:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:46 smithi183 ceph-mon[37273]: pgmap v5636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:46:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9649.750016 secs 2024-07-26T13:46:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:48 smithi144 ceph-mon[32129]: pgmap v5637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:48 smithi160 ceph-mon[36715]: pgmap v5637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:48 smithi183 ceph-mon[37273]: pgmap v5637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:46:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:50.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:46:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:50.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:46:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:50.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:46:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:50.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:46:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:46:50.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:50 smithi144 ceph-mon[32129]: pgmap v5638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:50 smithi160 ceph-mon[36715]: pgmap v5638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:50 smithi183 ceph-mon[37273]: pgmap v5638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:46:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9654.750079 secs 2024-07-26T13:46:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9654.750079 secs 2024-07-26T13:46:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9654.750079 secs 2024-07-26T13:46:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:52 smithi160 ceph-mon[36715]: pgmap v5639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:52 smithi183 ceph-mon[37273]: pgmap v5639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:52 smithi144 ceph-mon[32129]: pgmap v5639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:54 smithi160 ceph-mon[36715]: pgmap v5640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:54 smithi183 ceph-mon[37273]: pgmap v5640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:54 smithi144 ceph-mon[32129]: pgmap v5640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:46:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:55 smithi144 ceph-mon[32129]: pgmap v5641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:55 smithi160 ceph-mon[36715]: pgmap v5641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:55 smithi183 ceph-mon[37273]: pgmap v5641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:46:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9659.750208 secs 2024-07-26T13:46:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9659.750208 secs 2024-07-26T13:46:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9659.750208 secs 2024-07-26T13:46:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:46:57 smithi144 ceph-mon[32129]: pgmap v5642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:46:57 smithi160 ceph-mon[36715]: pgmap v5642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:46:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:46:57 smithi183 ceph-mon[37273]: pgmap v5642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:00 smithi144 ceph-mon[32129]: pgmap v5643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:00 smithi160 ceph-mon[36715]: pgmap v5643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:00 smithi183 ceph-mon[37273]: pgmap v5643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9664.750342 secs 2024-07-26T13:47:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9664.750342 secs 2024-07-26T13:47:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9664.750342 secs 2024-07-26T13:47:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:02 smithi144 ceph-mon[32129]: pgmap v5644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:02 smithi160 ceph-mon[36715]: pgmap v5644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:02 smithi183 ceph-mon[37273]: pgmap v5644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:04 smithi144 ceph-mon[32129]: pgmap v5645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:04 smithi160 ceph-mon[36715]: pgmap v5645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:04 smithi183 ceph-mon[37273]: pgmap v5645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:05.404+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:47:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:05.405+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:47:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:05.405+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:47:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:05.405+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:47:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:05.405+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:06 smithi160 ceph-mon[36715]: pgmap v5646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9669.750467 secs 2024-07-26T13:47:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:06 smithi183 ceph-mon[37273]: pgmap v5646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9669.750467 secs 2024-07-26T13:47:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:06 smithi144 ceph-mon[32129]: pgmap v5646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9669.750467 secs 2024-07-26T13:47:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:08 smithi160 ceph-mon[36715]: pgmap v5647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:08 smithi183 ceph-mon[37273]: pgmap v5647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:08 smithi144 ceph-mon[32129]: pgmap v5647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:10 smithi160 ceph-mon[36715]: pgmap v5648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:10 smithi183 ceph-mon[37273]: pgmap v5648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:10 smithi144 ceph-mon[32129]: pgmap v5648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9674.750533 secs 2024-07-26T13:47:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9674.750533 secs 2024-07-26T13:47:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9674.750533 secs 2024-07-26T13:47:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:12 smithi160 ceph-mon[36715]: pgmap v5649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:12 smithi183 ceph-mon[37273]: pgmap v5649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:12 smithi144 ceph-mon[32129]: pgmap v5649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:14 smithi160 ceph-mon[36715]: pgmap v5650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:14 smithi183 ceph-mon[37273]: pgmap v5650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:14 smithi144 ceph-mon[32129]: pgmap v5650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:16 smithi160 ceph-mon[36715]: pgmap v5651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9679.750665 secs 2024-07-26T13:47:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:16 smithi183 ceph-mon[37273]: pgmap v5651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9679.750665 secs 2024-07-26T13:47:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:16 smithi144 ceph-mon[32129]: pgmap v5651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9679.750665 secs 2024-07-26T13:47:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:17 smithi160 ceph-mon[36715]: pgmap v5652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:17 smithi183 ceph-mon[37273]: pgmap v5652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:17 smithi144 ceph-mon[32129]: pgmap v5652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:20.421+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:47:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:20.422+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:47:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:20.422+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:47:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:20.422+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:47:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:20.422+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:20 smithi144 ceph-mon[32129]: pgmap v5653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:20 smithi160 ceph-mon[36715]: pgmap v5653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:20 smithi183 ceph-mon[37273]: pgmap v5653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9684.750792 secs 2024-07-26T13:47:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9684.750792 secs 2024-07-26T13:47:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9684.750792 secs 2024-07-26T13:47:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:22 smithi160 ceph-mon[36715]: pgmap v5654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:22 smithi183 ceph-mon[37273]: pgmap v5654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:22 smithi144 ceph-mon[32129]: pgmap v5654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:24 smithi160 ceph-mon[36715]: pgmap v5655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:24 smithi183 ceph-mon[37273]: pgmap v5655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:24 smithi144 ceph-mon[32129]: pgmap v5655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:25 smithi144 ceph-mon[32129]: pgmap v5656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:25 smithi160 ceph-mon[36715]: pgmap v5656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:25 smithi183 ceph-mon[37273]: pgmap v5656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9689.750928 secs 2024-07-26T13:47:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9689.750928 secs 2024-07-26T13:47:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9689.750928 secs 2024-07-26T13:47:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:27 smithi144 ceph-mon[32129]: pgmap v5657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:27 smithi160 ceph-mon[36715]: pgmap v5657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:27 smithi183 ceph-mon[37273]: pgmap v5657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:30 smithi144 ceph-mon[32129]: pgmap v5658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:30 smithi160 ceph-mon[36715]: pgmap v5658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:30 smithi183 ceph-mon[37273]: pgmap v5658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9694.751057 secs 2024-07-26T13:47:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9694.751057 secs 2024-07-26T13:47:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9694.751057 secs 2024-07-26T13:47:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:32 smithi160 ceph-mon[36715]: pgmap v5659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:32 smithi183 ceph-mon[37273]: pgmap v5659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:32 smithi144 ceph-mon[32129]: pgmap v5659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:34 smithi144 ceph-mon[32129]: pgmap v5660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:34 smithi160 ceph-mon[36715]: pgmap v5660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:34 smithi183 ceph-mon[37273]: pgmap v5660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:35.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:35.437+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:47:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:35.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:47:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:35.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:47:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:35.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:47:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:35.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:36 smithi160 ceph-mon[36715]: pgmap v5661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9699.751188 secs 2024-07-26T13:47:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:36 smithi183 ceph-mon[37273]: pgmap v5661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9699.751188 secs 2024-07-26T13:47:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:36 smithi144 ceph-mon[32129]: pgmap v5661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9699.751188 secs 2024-07-26T13:47:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:38 smithi160 ceph-mon[36715]: pgmap v5662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:38 smithi183 ceph-mon[37273]: pgmap v5662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:38 smithi144 ceph-mon[32129]: pgmap v5662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:40 smithi160 ceph-mon[36715]: pgmap v5663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:40 smithi183 ceph-mon[37273]: pgmap v5663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:40 smithi144 ceph-mon[32129]: pgmap v5663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:47:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9704.751337 secs 2024-07-26T13:47:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:47:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9704.751337 secs 2024-07-26T13:47:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:47:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9704.751337 secs 2024-07-26T13:47:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:42 smithi160 ceph-mon[36715]: pgmap v5664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:42 smithi160 ceph-mon[36715]: pgmap v5665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:42 smithi183 ceph-mon[37273]: pgmap v5664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:42 smithi183 ceph-mon[37273]: pgmap v5665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:42 smithi144 ceph-mon[32129]: pgmap v5664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:42 smithi144 ceph-mon[32129]: pgmap v5665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:43 smithi160 ceph-mon[36715]: pgmap v5666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:43 smithi183 ceph-mon[37273]: pgmap v5666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:43 smithi144 ceph-mon[32129]: pgmap v5666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:45 smithi144 ceph-mon[32129]: pgmap v5667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:45 smithi160 ceph-mon[36715]: pgmap v5667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:45 smithi183 ceph-mon[37273]: pgmap v5667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9709.751462 secs 2024-07-26T13:47:46.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9709.751462 secs 2024-07-26T13:47:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9709.751462 secs 2024-07-26T13:47:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:47 smithi144 ceph-mon[32129]: pgmap v5668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:47 smithi160 ceph-mon[36715]: pgmap v5668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:47.816+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:47:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:47.816+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:47:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:47.816+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:47:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:47.816+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:47:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:47 smithi183 ceph-mon[37273]: pgmap v5668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:47:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:49 smithi144 ceph-mon[32129]: pgmap v5669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:47:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:49 smithi160 ceph-mon[36715]: pgmap v5669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:47:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:49 smithi183 ceph-mon[37273]: pgmap v5669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:47:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:50.440+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:47:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:50.441+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:47:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:50.441+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:47:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:50.441+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:47:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:47:50.441+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:51 smithi144 ceph-mon[32129]: pgmap v5670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:47:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9714.751566 secs 2024-07-26T13:47:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:51 smithi160 ceph-mon[36715]: pgmap v5670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:47:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9714.751566 secs 2024-07-26T13:47:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:51 smithi183 ceph-mon[37273]: pgmap v5670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:47:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9714.751566 secs 2024-07-26T13:47:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:53 smithi144 ceph-mon[32129]: pgmap v5671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:53 smithi160 ceph-mon[36715]: pgmap v5671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:53 smithi183 ceph-mon[37273]: pgmap v5671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:47:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:55 smithi144 ceph-mon[32129]: pgmap v5672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:55 smithi160 ceph-mon[36715]: pgmap v5672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:55 smithi183 ceph-mon[37273]: pgmap v5672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9719.751664 secs 2024-07-26T13:47:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9719.751664 secs 2024-07-26T13:47:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9719.751664 secs 2024-07-26T13:47:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:57 smithi160 ceph-mon[36715]: pgmap v5673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:57 smithi183 ceph-mon[37273]: pgmap v5673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:57 smithi144 ceph-mon[32129]: pgmap v5673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:47:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:47:59 smithi160 ceph-mon[36715]: pgmap v5674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:47:59 smithi183 ceph-mon[37273]: pgmap v5674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:47:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:47:59 smithi144 ceph-mon[32129]: pgmap v5674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:01 smithi160 ceph-mon[36715]: pgmap v5675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9724.751761 secs 2024-07-26T13:48:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:01 smithi183 ceph-mon[37273]: pgmap v5675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9724.751761 secs 2024-07-26T13:48:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:01 smithi144 ceph-mon[32129]: pgmap v5675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9724.751761 secs 2024-07-26T13:48:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:03 smithi160 ceph-mon[36715]: pgmap v5676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:03 smithi183 ceph-mon[37273]: pgmap v5676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:03 smithi144 ceph-mon[32129]: pgmap v5676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:05.458 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:05.457+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:48:05.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:05.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:48:05.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:05.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:48:05.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:05.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:48:05.459 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:05.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:05 smithi160 ceph-mon[36715]: pgmap v5677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:05 smithi183 ceph-mon[37273]: pgmap v5677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:05.917 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:05 smithi144 ceph-mon[32129]: pgmap v5677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9729.751885 secs 2024-07-26T13:48:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9729.751885 secs 2024-07-26T13:48:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9729.751885 secs 2024-07-26T13:48:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:07 smithi160 ceph-mon[36715]: pgmap v5678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:07 smithi183 ceph-mon[37273]: pgmap v5678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:07 smithi144 ceph-mon[32129]: pgmap v5678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:09 smithi160 ceph-mon[36715]: pgmap v5679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:09 smithi183 ceph-mon[37273]: pgmap v5679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:09 smithi144 ceph-mon[32129]: pgmap v5679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:11 smithi160 ceph-mon[36715]: pgmap v5680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9734.751993 secs 2024-07-26T13:48:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:11 smithi183 ceph-mon[37273]: pgmap v5680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9734.751993 secs 2024-07-26T13:48:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:11 smithi144 ceph-mon[32129]: pgmap v5680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9734.751993 secs 2024-07-26T13:48:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:13 smithi160 ceph-mon[36715]: pgmap v5681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:13 smithi183 ceph-mon[37273]: pgmap v5681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:13 smithi144 ceph-mon[32129]: pgmap v5681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:15 smithi160 ceph-mon[36715]: pgmap v5682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:15 smithi183 ceph-mon[37273]: pgmap v5682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:15 smithi144 ceph-mon[32129]: pgmap v5682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9739.752160 secs 2024-07-26T13:48:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9739.752160 secs 2024-07-26T13:48:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9739.752160 secs 2024-07-26T13:48:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:17 smithi183 ceph-mon[37273]: pgmap v5683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:17 smithi144 ceph-mon[32129]: pgmap v5683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:17 smithi160 ceph-mon[36715]: pgmap v5683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:19 smithi183 ceph-mon[37273]: pgmap v5684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:19 smithi144 ceph-mon[32129]: pgmap v5684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:19 smithi160 ceph-mon[36715]: pgmap v5684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:20.474+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:48:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:20.475+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:48:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:20.475+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:48:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:20.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:48:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:20.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:21 smithi183 ceph-mon[37273]: pgmap v5685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9744.752278 secs 2024-07-26T13:48:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:21 smithi144 ceph-mon[32129]: pgmap v5685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9744.752278 secs 2024-07-26T13:48:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:21 smithi160 ceph-mon[36715]: pgmap v5685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9744.752278 secs 2024-07-26T13:48:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:23 smithi183 ceph-mon[37273]: pgmap v5686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:23 smithi144 ceph-mon[32129]: pgmap v5686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:23 smithi160 ceph-mon[36715]: pgmap v5686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:25 smithi183 ceph-mon[37273]: pgmap v5687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:25 smithi144 ceph-mon[32129]: pgmap v5687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:25 smithi160 ceph-mon[36715]: pgmap v5687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:26.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9749.752411 secs 2024-07-26T13:48:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9749.752411 secs 2024-07-26T13:48:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9749.752411 secs 2024-07-26T13:48:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:27 smithi183 ceph-mon[37273]: pgmap v5688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:27 smithi144 ceph-mon[32129]: pgmap v5688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:27 smithi160 ceph-mon[36715]: pgmap v5688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:29 smithi183 ceph-mon[37273]: pgmap v5689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:29 smithi144 ceph-mon[32129]: pgmap v5689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:29 smithi160 ceph-mon[36715]: pgmap v5689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:31 smithi183 ceph-mon[37273]: pgmap v5690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9754.752477 secs 2024-07-26T13:48:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:31 smithi144 ceph-mon[32129]: pgmap v5690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9754.752477 secs 2024-07-26T13:48:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:31 smithi160 ceph-mon[36715]: pgmap v5690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9754.752477 secs 2024-07-26T13:48:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:33 smithi183 ceph-mon[37273]: pgmap v5691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:33 smithi144 ceph-mon[32129]: pgmap v5691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:33 smithi160 ceph-mon[36715]: pgmap v5691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:35.486 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:35.485+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:48:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:35.486+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:48:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:35.486+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:48:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:35.486+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:48:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:35.487+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:35 smithi183 ceph-mon[37273]: pgmap v5692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:35 smithi144 ceph-mon[32129]: pgmap v5692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:35 smithi160 ceph-mon[36715]: pgmap v5692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:36.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9759.752573 secs 2024-07-26T13:48:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9759.752573 secs 2024-07-26T13:48:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9759.752573 secs 2024-07-26T13:48:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:37 smithi144 ceph-mon[32129]: pgmap v5693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:37 smithi160 ceph-mon[36715]: pgmap v5693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:37 smithi183 ceph-mon[37273]: pgmap v5693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:48:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:39 smithi144 ceph-mon[32129]: pgmap v5694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:39 smithi160 ceph-mon[36715]: pgmap v5694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:39 smithi183 ceph-mon[37273]: pgmap v5694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:41 smithi144 ceph-mon[32129]: pgmap v5695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9764.752639 secs 2024-07-26T13:48:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:48:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:41 smithi160 ceph-mon[36715]: pgmap v5695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9764.752639 secs 2024-07-26T13:48:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:48:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:41 smithi183 ceph-mon[37273]: pgmap v5695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9764.752639 secs 2024-07-26T13:48:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:48:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:42 smithi144 ceph-mon[32129]: pgmap v5696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:48:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:42 smithi144 ceph-mon[32129]: pgmap v5697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:48:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:42 smithi160 ceph-mon[36715]: pgmap v5696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:48:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:42 smithi160 ceph-mon[36715]: pgmap v5697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:48:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:42 smithi183 ceph-mon[37273]: pgmap v5696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:48:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:42 smithi183 ceph-mon[37273]: pgmap v5697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:48:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:44 smithi144 ceph-mon[32129]: pgmap v5698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:48:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:44 smithi160 ceph-mon[36715]: pgmap v5698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:48:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:44 smithi183 ceph-mon[37273]: pgmap v5698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:48:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:46 smithi144 ceph-mon[32129]: pgmap v5699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:48:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9769.752754 secs 2024-07-26T13:48:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:46 smithi160 ceph-mon[36715]: pgmap v5699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:48:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9769.752754 secs 2024-07-26T13:48:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:46 smithi183 ceph-mon[37273]: pgmap v5699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:48:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9769.752754 secs 2024-07-26T13:48:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:48 smithi144 ceph-mon[32129]: pgmap v5700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:48:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:48 smithi160 ceph-mon[36715]: pgmap v5700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:48:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:48 smithi183 ceph-mon[37273]: pgmap v5700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:48:50.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:50.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:48:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:50.489+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:48:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:50.489+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:48:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:50.489+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:48:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:48:50.489+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:50 smithi144 ceph-mon[32129]: pgmap v5701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:48:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:50 smithi160 ceph-mon[36715]: pgmap v5701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:48:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:50 smithi183 ceph-mon[37273]: pgmap v5701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:48:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9774.752819 secs 2024-07-26T13:48:52.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9774.752819 secs 2024-07-26T13:48:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9774.752819 secs 2024-07-26T13:48:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:52 smithi144 ceph-mon[32129]: pgmap v5702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:48:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:52 smithi160 ceph-mon[36715]: pgmap v5702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:48:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:52 smithi183 ceph-mon[37273]: pgmap v5702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:48:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:54 smithi144 ceph-mon[32129]: pgmap v5703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:54 smithi160 ceph-mon[36715]: pgmap v5703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:54 smithi183 ceph-mon[37273]: pgmap v5703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:56 smithi144 ceph-mon[32129]: pgmap v5704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9779.752947 secs 2024-07-26T13:48:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:56 smithi160 ceph-mon[36715]: pgmap v5704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9779.752947 secs 2024-07-26T13:48:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:56 smithi183 ceph-mon[37273]: pgmap v5704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:48:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9779.752947 secs 2024-07-26T13:48:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:48:58 smithi144 ceph-mon[32129]: pgmap v5705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:48:58 smithi160 ceph-mon[36715]: pgmap v5705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:48:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:48:58 smithi183 ceph-mon[37273]: pgmap v5705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:00 smithi160 ceph-mon[36715]: pgmap v5706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:00 smithi183 ceph-mon[37273]: pgmap v5706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:00 smithi144 ceph-mon[32129]: pgmap v5706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9784.753074 secs 2024-07-26T13:49:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9784.753074 secs 2024-07-26T13:49:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9784.753074 secs 2024-07-26T13:49:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:02 smithi160 ceph-mon[36715]: pgmap v5707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:02 smithi183 ceph-mon[37273]: pgmap v5707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:02 smithi144 ceph-mon[32129]: pgmap v5707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:04 smithi160 ceph-mon[36715]: pgmap v5708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:04 smithi183 ceph-mon[37273]: pgmap v5708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:04 smithi144 ceph-mon[32129]: pgmap v5708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:05.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:05.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:49:05.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:05.497+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:49:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:05.497+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:49:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:05.497+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:49:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:05.497+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:06 smithi160 ceph-mon[36715]: pgmap v5709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9789.753164 secs 2024-07-26T13:49:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:06 smithi183 ceph-mon[37273]: pgmap v5709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9789.753164 secs 2024-07-26T13:49:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:06 smithi144 ceph-mon[32129]: pgmap v5709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9789.753164 secs 2024-07-26T13:49:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:08 smithi160 ceph-mon[36715]: pgmap v5710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:08 smithi183 ceph-mon[37273]: pgmap v5710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:08 smithi144 ceph-mon[32129]: pgmap v5710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:10 smithi160 ceph-mon[36715]: pgmap v5711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:10 smithi183 ceph-mon[37273]: pgmap v5711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:10 smithi144 ceph-mon[32129]: pgmap v5711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9794.753308 secs 2024-07-26T13:49:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9794.753308 secs 2024-07-26T13:49:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9794.753308 secs 2024-07-26T13:49:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:12 smithi160 ceph-mon[36715]: pgmap v5712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:12 smithi183 ceph-mon[37273]: pgmap v5712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:12 smithi144 ceph-mon[32129]: pgmap v5712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:14 smithi160 ceph-mon[36715]: pgmap v5713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:14 smithi183 ceph-mon[37273]: pgmap v5713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:14 smithi144 ceph-mon[32129]: pgmap v5713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:16 smithi160 ceph-mon[36715]: pgmap v5714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9799.753402 secs 2024-07-26T13:49:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:16 smithi183 ceph-mon[37273]: pgmap v5714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9799.753402 secs 2024-07-26T13:49:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:16 smithi144 ceph-mon[32129]: pgmap v5714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9799.753402 secs 2024-07-26T13:49:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:18 smithi160 ceph-mon[36715]: pgmap v5715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:18 smithi183 ceph-mon[37273]: pgmap v5715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:18 smithi144 ceph-mon[32129]: pgmap v5715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:20.513+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:49:20.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:20.514+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:49:20.515 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:20.514+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:49:20.515 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:20.514+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:49:20.515 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:20.514+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:20 smithi160 ceph-mon[36715]: pgmap v5716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:20 smithi183 ceph-mon[37273]: pgmap v5716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:20 smithi144 ceph-mon[32129]: pgmap v5716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9804.753497 secs 2024-07-26T13:49:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9804.753497 secs 2024-07-26T13:49:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9804.753497 secs 2024-07-26T13:49:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:22 smithi183 ceph-mon[37273]: pgmap v5717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:22 smithi144 ceph-mon[32129]: pgmap v5717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:22 smithi160 ceph-mon[36715]: pgmap v5717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:24 smithi183 ceph-mon[37273]: pgmap v5718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:24 smithi144 ceph-mon[32129]: pgmap v5718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:24 smithi160 ceph-mon[36715]: pgmap v5718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:26 smithi183 ceph-mon[37273]: pgmap v5719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9809.753630 secs 2024-07-26T13:49:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:26 smithi144 ceph-mon[32129]: pgmap v5719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9809.753630 secs 2024-07-26T13:49:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:26 smithi160 ceph-mon[36715]: pgmap v5719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9809.753630 secs 2024-07-26T13:49:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:28 smithi183 ceph-mon[37273]: pgmap v5720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:28 smithi144 ceph-mon[32129]: pgmap v5720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:28 smithi160 ceph-mon[36715]: pgmap v5720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:30 smithi183 ceph-mon[37273]: pgmap v5721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:30 smithi144 ceph-mon[32129]: pgmap v5721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:30 smithi160 ceph-mon[36715]: pgmap v5721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9814.753711 secs 2024-07-26T13:49:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9814.753711 secs 2024-07-26T13:49:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9814.753711 secs 2024-07-26T13:49:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:32 smithi183 ceph-mon[37273]: pgmap v5722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:32 smithi144 ceph-mon[32129]: pgmap v5722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:32 smithi160 ceph-mon[36715]: pgmap v5722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:34 smithi183 ceph-mon[37273]: pgmap v5723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:34 smithi144 ceph-mon[32129]: pgmap v5723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:34 smithi160 ceph-mon[36715]: pgmap v5723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:35.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:49:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:35.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:49:35.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:35.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:49:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:35.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:49:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:35.531+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:37.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:36 smithi160 ceph-mon[36715]: pgmap v5724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:37.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9819.753834 secs 2024-07-26T13:49:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:36 smithi183 ceph-mon[37273]: pgmap v5724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9819.753834 secs 2024-07-26T13:49:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:36 smithi144 ceph-mon[32129]: pgmap v5724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9819.753834 secs 2024-07-26T13:49:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:38 smithi183 ceph-mon[37273]: pgmap v5725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:38 smithi144 ceph-mon[32129]: pgmap v5725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:38 smithi160 ceph-mon[36715]: pgmap v5725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:49:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:40 smithi183 ceph-mon[37273]: pgmap v5726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:40 smithi144 ceph-mon[32129]: pgmap v5726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:40 smithi160 ceph-mon[36715]: pgmap v5726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9824.753928 secs 2024-07-26T13:49:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9824.753928 secs 2024-07-26T13:49:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9824.753928 secs 2024-07-26T13:49:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:42 smithi183 ceph-mon[37273]: pgmap v5727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:49:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:42 smithi144 ceph-mon[32129]: pgmap v5727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:49:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:42 smithi160 ceph-mon[36715]: pgmap v5727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:49:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:43 smithi144 ceph-mon[32129]: pgmap v5728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:49:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:43 smithi144 ceph-mon[32129]: pgmap v5729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:49:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:43 smithi160 ceph-mon[36715]: pgmap v5728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:49:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:43 smithi160 ceph-mon[36715]: pgmap v5729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:49:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:43 smithi183 ceph-mon[37273]: pgmap v5728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:49:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:43 smithi183 ceph-mon[37273]: pgmap v5729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:49:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:45 smithi144 ceph-mon[32129]: pgmap v5730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:49:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:45 smithi160 ceph-mon[36715]: pgmap v5730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:49:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:45 smithi183 ceph-mon[37273]: pgmap v5730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:49:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9829.754058 secs 2024-07-26T13:49:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9829.754058 secs 2024-07-26T13:49:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9829.754058 secs 2024-07-26T13:49:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:47 smithi144 ceph-mon[32129]: pgmap v5731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:49:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:47 smithi160 ceph-mon[36715]: pgmap v5731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:49:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:47 smithi183 ceph-mon[37273]: pgmap v5731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:49:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:49 smithi144 ceph-mon[32129]: pgmap v5732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:49:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:49 smithi160 ceph-mon[36715]: pgmap v5732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:49:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:49 smithi183 ceph-mon[37273]: pgmap v5732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:49:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:50.546+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:49:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:50.547+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:49:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:50.547+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:49:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:50.547+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:49:50.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:49:50.548+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:52.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:51 smithi183 ceph-mon[37273]: pgmap v5733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:49:52.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9834.754202 secs 2024-07-26T13:49:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:51 smithi144 ceph-mon[32129]: pgmap v5733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:49:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9834.754202 secs 2024-07-26T13:49:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:51 smithi160 ceph-mon[36715]: pgmap v5733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:49:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9834.754202 secs 2024-07-26T13:49:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:53 smithi144 ceph-mon[32129]: pgmap v5734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:49:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:53 smithi160 ceph-mon[36715]: pgmap v5734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:49:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:53 smithi183 ceph-mon[37273]: pgmap v5734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:49:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:55 smithi144 ceph-mon[32129]: pgmap v5735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:55 smithi160 ceph-mon[36715]: pgmap v5735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:55 smithi183 ceph-mon[37273]: pgmap v5735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:49:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9839.754283 secs 2024-07-26T13:49:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9839.754283 secs 2024-07-26T13:49:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9839.754283 secs 2024-07-26T13:49:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:57 smithi144 ceph-mon[32129]: pgmap v5736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:57 smithi160 ceph-mon[36715]: pgmap v5736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:49:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:57 smithi183 ceph-mon[37273]: pgmap v5736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:49:59 smithi144 ceph-mon[32129]: pgmap v5737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:49:59 smithi160 ceph-mon[36715]: pgmap v5737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:49:59 smithi183 ceph-mon[37273]: pgmap v5737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:01 smithi160 ceph-mon[36715]: pgmap v5738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9844.754347 secs 2024-07-26T13:50:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:01 smithi183 ceph-mon[37273]: pgmap v5738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9844.754347 secs 2024-07-26T13:50:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:01 smithi144 ceph-mon[32129]: pgmap v5738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9844.754347 secs 2024-07-26T13:50:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:03 smithi160 ceph-mon[36715]: pgmap v5739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:03 smithi183 ceph-mon[37273]: pgmap v5739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:03 smithi144 ceph-mon[32129]: pgmap v5739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:05.563+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:50:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:05.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:50:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:05.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:50:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:05.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:50:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:05.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:05 smithi160 ceph-mon[36715]: pgmap v5740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:05 smithi183 ceph-mon[37273]: pgmap v5740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:05 smithi144 ceph-mon[32129]: pgmap v5740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9849.754451 secs 2024-07-26T13:50:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9849.754451 secs 2024-07-26T13:50:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9849.754451 secs 2024-07-26T13:50:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:07 smithi160 ceph-mon[36715]: pgmap v5741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:07 smithi183 ceph-mon[37273]: pgmap v5741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:07 smithi144 ceph-mon[32129]: pgmap v5741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:10 smithi144 ceph-mon[32129]: pgmap v5742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:10 smithi160 ceph-mon[36715]: pgmap v5742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:10 smithi183 ceph-mon[37273]: pgmap v5742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:12 smithi160 ceph-mon[36715]: pgmap v5743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9854.754519 secs 2024-07-26T13:50:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:12 smithi183 ceph-mon[37273]: pgmap v5743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9854.754519 secs 2024-07-26T13:50:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:12 smithi144 ceph-mon[32129]: pgmap v5743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9854.754519 secs 2024-07-26T13:50:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:14 smithi160 ceph-mon[36715]: pgmap v5744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:14 smithi183 ceph-mon[37273]: pgmap v5744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:14 smithi144 ceph-mon[32129]: pgmap v5744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:16 smithi160 ceph-mon[36715]: pgmap v5745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:16 smithi183 ceph-mon[37273]: pgmap v5745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:16 smithi144 ceph-mon[32129]: pgmap v5745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9859.754640 secs 2024-07-26T13:50:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9859.754640 secs 2024-07-26T13:50:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9859.754640 secs 2024-07-26T13:50:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:18 smithi160 ceph-mon[36715]: pgmap v5746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:18 smithi183 ceph-mon[37273]: pgmap v5746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:18 smithi144 ceph-mon[32129]: pgmap v5746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:20 smithi144 ceph-mon[32129]: pgmap v5747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:20 smithi160 ceph-mon[36715]: pgmap v5747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:20 smithi183 ceph-mon[37273]: pgmap v5747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:20.581+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:50:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:20.582+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:50:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:20.582+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:50:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:20.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:50:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:20.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:22 smithi160 ceph-mon[36715]: pgmap v5748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9864.754759 secs 2024-07-26T13:50:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:22 smithi183 ceph-mon[37273]: pgmap v5748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9864.754759 secs 2024-07-26T13:50:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:22 smithi144 ceph-mon[32129]: pgmap v5748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9864.754759 secs 2024-07-26T13:50:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:24 smithi160 ceph-mon[36715]: pgmap v5749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:24 smithi183 ceph-mon[37273]: pgmap v5749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:24 smithi144 ceph-mon[32129]: pgmap v5749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:26 smithi160 ceph-mon[36715]: pgmap v5750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:26 smithi183 ceph-mon[37273]: pgmap v5750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:26 smithi144 ceph-mon[32129]: pgmap v5750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:27.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9869.754882 secs 2024-07-26T13:50:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9869.754882 secs 2024-07-26T13:50:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9869.754882 secs 2024-07-26T13:50:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:28 smithi183 ceph-mon[37273]: pgmap v5751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:28 smithi144 ceph-mon[32129]: pgmap v5751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:28 smithi160 ceph-mon[36715]: pgmap v5751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:30 smithi183 ceph-mon[37273]: pgmap v5752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:30 smithi144 ceph-mon[32129]: pgmap v5752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:30 smithi160 ceph-mon[36715]: pgmap v5752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:32 smithi183 ceph-mon[37273]: pgmap v5753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9874.754994 secs 2024-07-26T13:50:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:32 smithi144 ceph-mon[32129]: pgmap v5753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9874.754994 secs 2024-07-26T13:50:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:32 smithi160 ceph-mon[36715]: pgmap v5753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9874.754994 secs 2024-07-26T13:50:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:34 smithi183 ceph-mon[37273]: pgmap v5754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:34 smithi144 ceph-mon[32129]: pgmap v5754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:34 smithi160 ceph-mon[36715]: pgmap v5754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:35.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:35.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:50:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:35.599+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:50:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:35.599+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:50:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:35.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:50:35.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:35.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:36 smithi183 ceph-mon[37273]: pgmap v5755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:36 smithi144 ceph-mon[32129]: pgmap v5755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:36 smithi160 ceph-mon[36715]: pgmap v5755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9879.755116 secs 2024-07-26T13:50:37.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9879.755116 secs 2024-07-26T13:50:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9879.755116 secs 2024-07-26T13:50:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:38 smithi183 ceph-mon[37273]: pgmap v5756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:38 smithi144 ceph-mon[32129]: pgmap v5756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:38 smithi160 ceph-mon[36715]: pgmap v5756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:50:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:40 smithi183 ceph-mon[37273]: pgmap v5757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:40 smithi144 ceph-mon[32129]: pgmap v5757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:40 smithi160 ceph-mon[36715]: pgmap v5757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:41.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9884.755261 secs 2024-07-26T13:50:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9884.755261 secs 2024-07-26T13:50:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9884.755261 secs 2024-07-26T13:50:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:42 smithi183 ceph-mon[37273]: pgmap v5758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:42 smithi144 ceph-mon[32129]: pgmap v5758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:42 smithi160 ceph-mon[36715]: pgmap v5758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: pgmap v5759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:44.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: pgmap v5759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:50:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: pgmap v5759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:50:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:45 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:50:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:45 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:50:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:45 smithi144 ceph-mon[32129]: pgmap v5760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:45 smithi144 ceph-mon[32129]: pgmap v5761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:50:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:45 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:50:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:45 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:50:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:45 smithi160 ceph-mon[36715]: pgmap v5760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:45 smithi160 ceph-mon[36715]: pgmap v5761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:50:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:45 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T13:50:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:45 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T13:50:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:45 smithi183 ceph-mon[37273]: pgmap v5760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:45 smithi183 ceph-mon[37273]: pgmap v5761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:50:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 msgr2=0x7f74900b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 msgr2=0x7f74900b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 0x7f74900b5430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a806b000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 0x7f74900b5430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a806b000 comp rx=0 tx=0).stop 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 msgr2=0x7f74900b5430 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 msgr2=0x7f74900b5430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b49b0 0x7f74900b5430 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:50:46.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.126+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:50:46.129 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:46.128+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5970 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a804c040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:50:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9889.755387 secs 2024-07-26T13:50:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9889.755387 secs 2024-07-26T13:50:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9889.755387 secs 2024-07-26T13:50:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:47 smithi144 ceph-mon[32129]: pgmap v5762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:47 smithi160 ceph-mon[36715]: pgmap v5762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:47 smithi183 ceph-mon[37273]: pgmap v5762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.848+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T13:50:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.848+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T13:50:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.849+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T13:50:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.849+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T13:50:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.849+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:50:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.849+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:50:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.850+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T13:50:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:47.850+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:50:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:49 smithi144 ceph-mon[32129]: pgmap v5763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:50:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:49 smithi160 ceph-mon[36715]: pgmap v5763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:50:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:49 smithi183 ceph-mon[37273]: pgmap v5763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:50:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:50.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:50:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:50.615+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:50:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:50.615+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:50:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:50.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:50:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:50:50.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:51 smithi144 ceph-mon[32129]: pgmap v5764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:50:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9894.755458 secs 2024-07-26T13:50:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:51 smithi160 ceph-mon[36715]: pgmap v5764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:50:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9894.755458 secs 2024-07-26T13:50:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:51 smithi183 ceph-mon[37273]: pgmap v5764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:50:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9894.755458 secs 2024-07-26T13:50:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:53 smithi144 ceph-mon[32129]: pgmap v5765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:53 smithi160 ceph-mon[36715]: pgmap v5765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:53 smithi183 ceph-mon[37273]: pgmap v5765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:50:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:55 smithi144 ceph-mon[32129]: pgmap v5766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:50:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:55 smithi160 ceph-mon[36715]: pgmap v5766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:50:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:55 smithi183 ceph-mon[37273]: pgmap v5766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:50:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9899.755579 secs 2024-07-26T13:50:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9899.755579 secs 2024-07-26T13:50:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9899.755579 secs 2024-07-26T13:50:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:57 smithi144 ceph-mon[32129]: pgmap v5767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:57 smithi160 ceph-mon[36715]: pgmap v5767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:57 smithi183 ceph-mon[37273]: pgmap v5767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:50:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:50:59 smithi144 ceph-mon[32129]: pgmap v5768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:50:59 smithi160 ceph-mon[36715]: pgmap v5768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:50:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:50:59 smithi183 ceph-mon[37273]: pgmap v5768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:01 smithi160 ceph-mon[36715]: pgmap v5769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9904.755685 secs 2024-07-26T13:51:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:01 smithi183 ceph-mon[37273]: pgmap v5769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9904.755685 secs 2024-07-26T13:51:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:01 smithi144 ceph-mon[32129]: pgmap v5769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9904.755685 secs 2024-07-26T13:51:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:03 smithi160 ceph-mon[36715]: pgmap v5770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:03 smithi183 ceph-mon[37273]: pgmap v5770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:03 smithi144 ceph-mon[32129]: pgmap v5770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:05 smithi160 ceph-mon[36715]: pgmap v5771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:05.628 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:05.627+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:51:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:05.628+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:51:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:05.628+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:51:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:05.628+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:51:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:05.629+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:05 smithi183 ceph-mon[37273]: pgmap v5771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:05 smithi144 ceph-mon[32129]: pgmap v5771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9909.755802 secs 2024-07-26T13:51:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9909.755802 secs 2024-07-26T13:51:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9909.755802 secs 2024-07-26T13:51:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:07 smithi160 ceph-mon[36715]: pgmap v5772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:07 smithi183 ceph-mon[37273]: pgmap v5772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:07 smithi144 ceph-mon[32129]: pgmap v5772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:09 smithi160 ceph-mon[36715]: pgmap v5773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:09.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:09 smithi183 ceph-mon[37273]: pgmap v5773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:09 smithi144 ceph-mon[32129]: pgmap v5773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:11.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:11 smithi144 ceph-mon[32129]: pgmap v5774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:11.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9914.755924 secs 2024-07-26T13:51:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:11 smithi160 ceph-mon[36715]: pgmap v5774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9914.755924 secs 2024-07-26T13:51:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:11 smithi183 ceph-mon[37273]: pgmap v5774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9914.755924 secs 2024-07-26T13:51:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:13 smithi160 ceph-mon[36715]: pgmap v5775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:13 smithi183 ceph-mon[37273]: pgmap v5775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:13 smithi144 ceph-mon[32129]: pgmap v5775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:15 smithi160 ceph-mon[36715]: pgmap v5776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:15 smithi183 ceph-mon[37273]: pgmap v5776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:15 smithi144 ceph-mon[32129]: pgmap v5776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9919.756056 secs 2024-07-26T13:51:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9919.756056 secs 2024-07-26T13:51:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9919.756056 secs 2024-07-26T13:51:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:17 smithi160 ceph-mon[36715]: pgmap v5777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:17 smithi183 ceph-mon[37273]: pgmap v5777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:17 smithi144 ceph-mon[32129]: pgmap v5777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:19 smithi160 ceph-mon[36715]: pgmap v5778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:19 smithi183 ceph-mon[37273]: pgmap v5778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:19 smithi144 ceph-mon[32129]: pgmap v5778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:20.645+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:20.646+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:20.646+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:20.646+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:20.646+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:21 smithi160 ceph-mon[36715]: pgmap v5779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9924.756180 secs 2024-07-26T13:51:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:21 smithi183 ceph-mon[37273]: pgmap v5779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9924.756180 secs 2024-07-26T13:51:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:21 smithi144 ceph-mon[32129]: pgmap v5779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9924.756180 secs 2024-07-26T13:51:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:23 smithi160 ceph-mon[36715]: pgmap v5780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:23 smithi183 ceph-mon[37273]: pgmap v5780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:23 smithi144 ceph-mon[32129]: pgmap v5780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:25 smithi160 ceph-mon[36715]: pgmap v5781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:25 smithi183 ceph-mon[37273]: pgmap v5781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:25 smithi144 ceph-mon[32129]: pgmap v5781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9929.756320 secs 2024-07-26T13:51:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9929.756320 secs 2024-07-26T13:51:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9929.756320 secs 2024-07-26T13:51:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:27 smithi144 ceph-mon[32129]: pgmap v5782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:27 smithi183 ceph-mon[37273]: pgmap v5782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:27 smithi160 ceph-mon[36715]: pgmap v5782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:28 smithi183 ceph-mon[37273]: pgmap v5783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:28 smithi144 ceph-mon[32129]: pgmap v5783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:28 smithi160 ceph-mon[36715]: pgmap v5783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:31 smithi183 ceph-mon[37273]: pgmap v5784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:31 smithi144 ceph-mon[32129]: pgmap v5784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:31 smithi160 ceph-mon[36715]: pgmap v5784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9934.756410 secs 2024-07-26T13:51:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9934.756410 secs 2024-07-26T13:51:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9934.756410 secs 2024-07-26T13:51:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:33 smithi183 ceph-mon[37273]: pgmap v5785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:33 smithi144 ceph-mon[32129]: pgmap v5785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:33 smithi160 ceph-mon[36715]: pgmap v5785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:35 smithi183 ceph-mon[37273]: pgmap v5786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:35 smithi144 ceph-mon[32129]: pgmap v5786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:35 smithi160 ceph-mon[36715]: pgmap v5786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:35.662+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:51:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:35.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:51:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:35.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:51:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:35.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:51:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:35.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:37.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9939.756550 secs 2024-07-26T13:51:37.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:37 smithi160 ceph-mon[36715]: pgmap v5787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9939.756550 secs 2024-07-26T13:51:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:37 smithi183 ceph-mon[37273]: pgmap v5787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9939.756550 secs 2024-07-26T13:51:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:37 smithi144 ceph-mon[32129]: pgmap v5787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:39 smithi183 ceph-mon[37273]: pgmap v5788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:39 smithi144 ceph-mon[32129]: pgmap v5788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:39 smithi160 ceph-mon[36715]: pgmap v5788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:51:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:41 smithi183 ceph-mon[37273]: pgmap v5789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:41 smithi144 ceph-mon[32129]: pgmap v5789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:41 smithi160 ceph-mon[36715]: pgmap v5789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9944.756668 secs 2024-07-26T13:51:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9944.756668 secs 2024-07-26T13:51:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9944.756668 secs 2024-07-26T13:51:43.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:43 smithi144 ceph-mon[32129]: pgmap v5790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:43 smithi183 ceph-mon[37273]: pgmap v5790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:43 smithi160 ceph-mon[36715]: pgmap v5790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: pgmap v5791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: pgmap v5791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: pgmap v5791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:51:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:51:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:51:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:51:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:51:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: pgmap v5792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:51:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: pgmap v5793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:51:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:51:46.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9949.756784 secs 2024-07-26T13:51:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:51:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:51:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: pgmap v5792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: pgmap v5793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:51:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9949.756784 secs 2024-07-26T13:51:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: pgmap v5792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: pgmap v5793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:51:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9949.756784 secs 2024-07-26T13:51:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:48 smithi160 ceph-mon[36715]: pgmap v5794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:48 smithi183 ceph-mon[37273]: pgmap v5794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:48 smithi144 ceph-mon[32129]: pgmap v5794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:50 smithi160 ceph-mon[36715]: pgmap v5795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:50 smithi183 ceph-mon[37273]: pgmap v5795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:50.666+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:51:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:50.667+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:51:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:50.667+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:51:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:50.667+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:51:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:51:50.667+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:50 smithi144 ceph-mon[32129]: pgmap v5795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9954.756909 secs 2024-07-26T13:51:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9954.756909 secs 2024-07-26T13:51:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9954.756909 secs 2024-07-26T13:51:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:52 smithi183 ceph-mon[37273]: pgmap v5796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:52 smithi144 ceph-mon[32129]: pgmap v5796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:52 smithi160 ceph-mon[36715]: pgmap v5796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:51:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:54 smithi183 ceph-mon[37273]: pgmap v5797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:51:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:54 smithi144 ceph-mon[32129]: pgmap v5797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:51:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:54 smithi160 ceph-mon[36715]: pgmap v5797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:51:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:55 smithi183 ceph-mon[37273]: pgmap v5798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:51:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:55 smithi144 ceph-mon[32129]: pgmap v5798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:51:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:55 smithi160 ceph-mon[36715]: pgmap v5798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:51:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9959.757013 secs 2024-07-26T13:51:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9959.757013 secs 2024-07-26T13:51:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9959.757013 secs 2024-07-26T13:51:57.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:51:57 smithi144 ceph-mon[32129]: pgmap v5799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:51:57 smithi183 ceph-mon[37273]: pgmap v5799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:51:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:51:57 smithi160 ceph-mon[36715]: pgmap v5799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:00 smithi160 ceph-mon[36715]: pgmap v5800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:00 smithi183 ceph-mon[37273]: pgmap v5800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:00 smithi144 ceph-mon[32129]: pgmap v5800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9964.757132 secs 2024-07-26T13:52:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9964.757132 secs 2024-07-26T13:52:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9964.757132 secs 2024-07-26T13:52:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:02 smithi160 ceph-mon[36715]: pgmap v5801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:02 smithi183 ceph-mon[37273]: pgmap v5801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:02 smithi144 ceph-mon[32129]: pgmap v5801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:04 smithi160 ceph-mon[36715]: pgmap v5802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:04 smithi183 ceph-mon[37273]: pgmap v5802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:04 smithi144 ceph-mon[32129]: pgmap v5802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:05.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:05.672+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:52:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:05.673+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:52:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:05.673+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:52:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:05.673+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:52:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:05.673+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:06 smithi183 ceph-mon[37273]: pgmap v5803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9969.757260 secs 2024-07-26T13:52:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:06 smithi144 ceph-mon[32129]: pgmap v5803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9969.757260 secs 2024-07-26T13:52:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:06 smithi160 ceph-mon[36715]: pgmap v5803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9969.757260 secs 2024-07-26T13:52:08.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:08 smithi160 ceph-mon[36715]: pgmap v5804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:08 smithi183 ceph-mon[37273]: pgmap v5804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:08 smithi144 ceph-mon[32129]: pgmap v5804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:09 smithi183 ceph-mon[37273]: pgmap v5805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:09 smithi144 ceph-mon[32129]: pgmap v5805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:09 smithi160 ceph-mon[36715]: pgmap v5805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9974.757374 secs 2024-07-26T13:52:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9974.757374 secs 2024-07-26T13:52:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9974.757374 secs 2024-07-26T13:52:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:12 smithi144 ceph-mon[32129]: pgmap v5806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:12 smithi160 ceph-mon[36715]: pgmap v5806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:12 smithi183 ceph-mon[37273]: pgmap v5806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:14 smithi160 ceph-mon[36715]: pgmap v5807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:14 smithi183 ceph-mon[37273]: pgmap v5807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:14 smithi144 ceph-mon[32129]: pgmap v5807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:16 smithi160 ceph-mon[36715]: pgmap v5808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9979.757484 secs 2024-07-26T13:52:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:16 smithi183 ceph-mon[37273]: pgmap v5808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9979.757484 secs 2024-07-26T13:52:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:16 smithi144 ceph-mon[32129]: pgmap v5808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9979.757484 secs 2024-07-26T13:52:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:18 smithi160 ceph-mon[36715]: pgmap v5809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:18 smithi183 ceph-mon[37273]: pgmap v5809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:18 smithi144 ceph-mon[32129]: pgmap v5809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:20 smithi183 ceph-mon[37273]: pgmap v5810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:20.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:52:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:20.681+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:52:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:20.681+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:52:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:20.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:52:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:20.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:20 smithi144 ceph-mon[32129]: pgmap v5810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:20 smithi160 ceph-mon[36715]: pgmap v5810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9984.757606 secs 2024-07-26T13:52:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9984.757606 secs 2024-07-26T13:52:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9984.757606 secs 2024-07-26T13:52:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:22 smithi183 ceph-mon[37273]: pgmap v5811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:22 smithi144 ceph-mon[32129]: pgmap v5811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:22 smithi160 ceph-mon[36715]: pgmap v5811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:23 smithi183 ceph-mon[37273]: pgmap v5812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:23 smithi144 ceph-mon[32129]: pgmap v5812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:23 smithi160 ceph-mon[36715]: pgmap v5812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:26 smithi160 ceph-mon[36715]: pgmap v5813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9989.757724 secs 2024-07-26T13:52:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:26 smithi183 ceph-mon[37273]: pgmap v5813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9989.757724 secs 2024-07-26T13:52:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:26 smithi144 ceph-mon[32129]: pgmap v5813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9989.757724 secs 2024-07-26T13:52:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:28 smithi160 ceph-mon[36715]: pgmap v5814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:28 smithi183 ceph-mon[37273]: pgmap v5814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:28 smithi144 ceph-mon[32129]: pgmap v5814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:30 smithi183 ceph-mon[37273]: pgmap v5815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:30 smithi144 ceph-mon[32129]: pgmap v5815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:30 smithi160 ceph-mon[36715]: pgmap v5815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9994.757844 secs 2024-07-26T13:52:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9994.757844 secs 2024-07-26T13:52:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9994.757844 secs 2024-07-26T13:52:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:32 smithi183 ceph-mon[37273]: pgmap v5816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:32.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:32 smithi144 ceph-mon[32129]: pgmap v5816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:32 smithi160 ceph-mon[36715]: pgmap v5816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:33 smithi183 ceph-mon[37273]: pgmap v5817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:33 smithi144 ceph-mon[32129]: pgmap v5817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:33 smithi160 ceph-mon[36715]: pgmap v5817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:35.686+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:35.687+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:35.687+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:35.687+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:35.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:36 smithi183 ceph-mon[37273]: pgmap v5818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 9999.757963 secs 2024-07-26T13:52:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:36 smithi144 ceph-mon[32129]: pgmap v5818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 9999.757963 secs 2024-07-26T13:52:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:36 smithi160 ceph-mon[36715]: pgmap v5818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 9999.757963 secs 2024-07-26T13:52:38.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:38 smithi160 ceph-mon[36715]: pgmap v5819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:38 smithi183 ceph-mon[37273]: pgmap v5819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:38 smithi144 ceph-mon[32129]: pgmap v5819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:40 smithi183 ceph-mon[37273]: pgmap v5820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:40 smithi144 ceph-mon[32129]: pgmap v5820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:40 smithi160 ceph-mon[36715]: pgmap v5820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10004.758084 secs 2024-07-26T13:52:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10004.758084 secs 2024-07-26T13:52:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10004.758084 secs 2024-07-26T13:52:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:42 smithi183 ceph-mon[37273]: pgmap v5821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:42 smithi144 ceph-mon[32129]: pgmap v5821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:42 smithi160 ceph-mon[36715]: pgmap v5821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:52:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:43 smithi183 ceph-mon[37273]: pgmap v5822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:43 smithi144 ceph-mon[32129]: pgmap v5822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:43 smithi160 ceph-mon[36715]: pgmap v5822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:52:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:46 smithi183 ceph-mon[37273]: pgmap v5823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10009.758174 secs 2024-07-26T13:52:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:46 smithi144 ceph-mon[32129]: pgmap v5823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10009.758174 secs 2024-07-26T13:52:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:46 smithi160 ceph-mon[36715]: pgmap v5823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10009.758174 secs 2024-07-26T13:52:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:52:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:52:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:48 smithi183 ceph-mon[37273]: pgmap v5824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:48 smithi183 ceph-mon[37273]: pgmap v5825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:48 smithi144 ceph-mon[32129]: pgmap v5824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:48 smithi144 ceph-mon[32129]: pgmap v5825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:48 smithi160 ceph-mon[36715]: pgmap v5824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:48 smithi160 ceph-mon[36715]: pgmap v5825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:49 smithi183 ceph-mon[37273]: pgmap v5826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:52:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:49 smithi144 ceph-mon[32129]: pgmap v5826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:52:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:49 smithi160 ceph-mon[36715]: pgmap v5826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:52:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:50.697+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:52:50.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:50.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:52:50.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:50.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:52:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:50.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:52:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:52:50.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:51 smithi160 ceph-mon[36715]: pgmap v5827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10014.758311 secs 2024-07-26T13:52:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:51 smithi183 ceph-mon[37273]: pgmap v5827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10014.758311 secs 2024-07-26T13:52:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:51 smithi144 ceph-mon[32129]: pgmap v5827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10014.758311 secs 2024-07-26T13:52:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:53 smithi160 ceph-mon[36715]: pgmap v5828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:53 smithi183 ceph-mon[37273]: pgmap v5828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:53 smithi144 ceph-mon[32129]: pgmap v5828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:52:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:55 smithi160 ceph-mon[36715]: pgmap v5829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:55 smithi183 ceph-mon[37273]: pgmap v5829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:55 smithi144 ceph-mon[32129]: pgmap v5829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:52:56.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10019.758454 secs 2024-07-26T13:52:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10019.758454 secs 2024-07-26T13:52:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10019.758454 secs 2024-07-26T13:52:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:57 smithi160 ceph-mon[36715]: pgmap v5830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:52:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:57 smithi183 ceph-mon[37273]: pgmap v5830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:52:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:57 smithi144 ceph-mon[32129]: pgmap v5830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:52:59.771 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:52:59 smithi144 ceph-mon[32129]: pgmap v5831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:52:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:52:59 smithi160 ceph-mon[36715]: pgmap v5831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:52:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:52:59 smithi183 ceph-mon[37273]: pgmap v5831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:01 smithi160 ceph-mon[36715]: pgmap v5832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10024.758573 secs 2024-07-26T13:53:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:01 smithi183 ceph-mon[37273]: pgmap v5832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10024.758573 secs 2024-07-26T13:53:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:01 smithi144 ceph-mon[32129]: pgmap v5832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10024.758573 secs 2024-07-26T13:53:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:03 smithi160 ceph-mon[36715]: pgmap v5833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:53:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:03 smithi183 ceph-mon[37273]: pgmap v5833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:53:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:03 smithi144 ceph-mon[32129]: pgmap v5833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:53:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:05.714+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:53:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:05.715+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:53:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:05.715+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:53:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:05.715+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:53:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:05.716+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:05 smithi160 ceph-mon[36715]: pgmap v5834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:53:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:05 smithi183 ceph-mon[37273]: pgmap v5834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:53:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:05 smithi144 ceph-mon[32129]: pgmap v5834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:53:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10029.758695 secs 2024-07-26T13:53:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10029.758695 secs 2024-07-26T13:53:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10029.758695 secs 2024-07-26T13:53:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:07 smithi160 ceph-mon[36715]: pgmap v5835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:07 smithi183 ceph-mon[37273]: pgmap v5835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:07 smithi144 ceph-mon[32129]: pgmap v5835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:09 smithi160 ceph-mon[36715]: pgmap v5836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:09 smithi183 ceph-mon[37273]: pgmap v5836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:09 smithi144 ceph-mon[32129]: pgmap v5836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:11 smithi183 ceph-mon[37273]: pgmap v5837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10034.758823 secs 2024-07-26T13:53:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:11 smithi144 ceph-mon[32129]: pgmap v5837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10034.758823 secs 2024-07-26T13:53:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:11 smithi160 ceph-mon[36715]: pgmap v5837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10034.758823 secs 2024-07-26T13:53:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:13 smithi183 ceph-mon[37273]: pgmap v5838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:13 smithi144 ceph-mon[32129]: pgmap v5838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:13 smithi160 ceph-mon[36715]: pgmap v5838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:15 smithi183 ceph-mon[37273]: pgmap v5839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:15 smithi144 ceph-mon[32129]: pgmap v5839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:15 smithi160 ceph-mon[36715]: pgmap v5839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:16.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10039.758944 secs 2024-07-26T13:53:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10039.758944 secs 2024-07-26T13:53:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10039.758944 secs 2024-07-26T13:53:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:17 smithi183 ceph-mon[37273]: pgmap v5840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:17 smithi144 ceph-mon[32129]: pgmap v5840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:17 smithi160 ceph-mon[36715]: pgmap v5840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:19 smithi183 ceph-mon[37273]: pgmap v5841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:19 smithi144 ceph-mon[32129]: pgmap v5841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:19 smithi160 ceph-mon[36715]: pgmap v5841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:20.731+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:53:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:20.732+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:53:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:20.732+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:53:20.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:20.732+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:53:20.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:20.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:21 smithi183 ceph-mon[37273]: pgmap v5842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10044.759065 secs 2024-07-26T13:53:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:21 smithi144 ceph-mon[32129]: pgmap v5842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10044.759065 secs 2024-07-26T13:53:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:21 smithi160 ceph-mon[36715]: pgmap v5842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10044.759065 secs 2024-07-26T13:53:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:23 smithi183 ceph-mon[37273]: pgmap v5843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:23 smithi144 ceph-mon[32129]: pgmap v5843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:23 smithi160 ceph-mon[36715]: pgmap v5843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:25 smithi183 ceph-mon[37273]: pgmap v5844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:25 smithi144 ceph-mon[32129]: pgmap v5844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:25 smithi160 ceph-mon[36715]: pgmap v5844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10049.759183 secs 2024-07-26T13:53:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10049.759183 secs 2024-07-26T13:53:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10049.759183 secs 2024-07-26T13:53:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:27 smithi183 ceph-mon[37273]: pgmap v5845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:27 smithi144 ceph-mon[32129]: pgmap v5845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:27 smithi160 ceph-mon[36715]: pgmap v5845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:29 smithi183 ceph-mon[37273]: pgmap v5846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:29 smithi144 ceph-mon[32129]: pgmap v5846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:29 smithi160 ceph-mon[36715]: pgmap v5846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:31 smithi144 ceph-mon[32129]: pgmap v5847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10054.759327 secs 2024-07-26T13:53:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:31 smithi160 ceph-mon[36715]: pgmap v5847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10054.759327 secs 2024-07-26T13:53:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:31 smithi183 ceph-mon[37273]: pgmap v5847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10054.759327 secs 2024-07-26T13:53:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:33 smithi144 ceph-mon[32129]: pgmap v5848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:33 smithi160 ceph-mon[36715]: pgmap v5848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:33 smithi183 ceph-mon[37273]: pgmap v5848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:35.749+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:53:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:35.749+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:53:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:35.749+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:53:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:35.749+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:53:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:35.749+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:36 smithi183 ceph-mon[37273]: pgmap v5849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:36 smithi144 ceph-mon[32129]: pgmap v5849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:36 smithi160 ceph-mon[36715]: pgmap v5849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10059.759366 secs 2024-07-26T13:53:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10059.759366 secs 2024-07-26T13:53:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10059.759366 secs 2024-07-26T13:53:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:38 smithi183 ceph-mon[37273]: pgmap v5850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:38 smithi144 ceph-mon[32129]: pgmap v5850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:38 smithi160 ceph-mon[36715]: pgmap v5850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:53:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:40 smithi183 ceph-mon[37273]: pgmap v5851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:40 smithi144 ceph-mon[32129]: pgmap v5851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:40 smithi160 ceph-mon[36715]: pgmap v5851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:42 smithi183 ceph-mon[37273]: pgmap v5852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10064.759493 secs 2024-07-26T13:53:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:42 smithi144 ceph-mon[32129]: pgmap v5852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10064.759493 secs 2024-07-26T13:53:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:42 smithi160 ceph-mon[36715]: pgmap v5852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10064.759493 secs 2024-07-26T13:53:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:44 smithi183 ceph-mon[37273]: pgmap v5853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:44 smithi144 ceph-mon[32129]: pgmap v5853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:44 smithi160 ceph-mon[36715]: pgmap v5853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:53:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:46 smithi144 ceph-mon[32129]: pgmap v5854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10069.759618 secs 2024-07-26T13:53:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:46 smithi183 ceph-mon[37273]: pgmap v5854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10069.759618 secs 2024-07-26T13:53:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:46 smithi160 ceph-mon[36715]: pgmap v5854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:53:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10069.759618 secs 2024-07-26T13:53:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:47.879+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:53:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:47.880+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:53:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:47.884+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:53:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:47.885+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:53:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:48 smithi160 ceph-mon[36715]: pgmap v5855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:48 smithi183 ceph-mon[37273]: pgmap v5855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:48 smithi144 ceph-mon[32129]: pgmap v5855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:53:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:49 smithi160 ceph-mon[36715]: pgmap v5856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:53:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:49 smithi160 ceph-mon[36715]: pgmap v5857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-07-26T13:53:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:49 smithi183 ceph-mon[37273]: pgmap v5856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:53:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:49 smithi183 ceph-mon[37273]: pgmap v5857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-07-26T13:53:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:49 smithi144 ceph-mon[32129]: pgmap v5856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:53:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:49 smithi144 ceph-mon[32129]: pgmap v5857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-07-26T13:53:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:53:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:50 smithi144 ceph-mon[32129]: pgmap v5858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s 2024-07-26T13:53:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:50.765+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:53:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:50.766+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:53:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:50.766+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:53:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:50.766+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:53:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:53:50.766+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:50 smithi160 ceph-mon[36715]: pgmap v5858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s 2024-07-26T13:53:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:50 smithi183 ceph-mon[37273]: pgmap v5858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s 2024-07-26T13:53:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10074.759730 secs 2024-07-26T13:53:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10074.759730 secs 2024-07-26T13:53:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10074.759730 secs 2024-07-26T13:53:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:52 smithi144 ceph-mon[32129]: pgmap v5859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:53:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:52 smithi160 ceph-mon[36715]: pgmap v5859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:53:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:52 smithi183 ceph-mon[37273]: pgmap v5859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:53:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:54 smithi144 ceph-mon[32129]: pgmap v5860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:53:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:54 smithi160 ceph-mon[36715]: pgmap v5860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:53:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:54 smithi183 ceph-mon[37273]: pgmap v5860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:53:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10079.759855 secs 2024-07-26T13:53:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10079.759855 secs 2024-07-26T13:53:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10079.759855 secs 2024-07-26T13:53:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:57 smithi144 ceph-mon[32129]: pgmap v5861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:53:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:57 smithi160 ceph-mon[36715]: pgmap v5861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:53:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:57 smithi183 ceph-mon[37273]: pgmap v5861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T13:53:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:53:58 smithi144 ceph-mon[32129]: pgmap v5862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:53:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:53:58 smithi160 ceph-mon[36715]: pgmap v5862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:53:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:53:58 smithi183 ceph-mon[37273]: pgmap v5862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T13:54:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:00 smithi144 ceph-mon[32129]: pgmap v5863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:00 smithi160 ceph-mon[36715]: pgmap v5863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:00 smithi183 ceph-mon[37273]: pgmap v5863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10084.759979 secs 2024-07-26T13:54:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10084.759979 secs 2024-07-26T13:54:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10084.759979 secs 2024-07-26T13:54:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:02 smithi144 ceph-mon[32129]: pgmap v5864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:02 smithi160 ceph-mon[36715]: pgmap v5864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:02 smithi183 ceph-mon[37273]: pgmap v5864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:04 smithi144 ceph-mon[32129]: pgmap v5865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:04 smithi160 ceph-mon[36715]: pgmap v5865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:04 smithi183 ceph-mon[37273]: pgmap v5865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:05.781+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:54:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:05.782+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:54:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:05.782+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:54:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:05.782+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:54:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:05.782+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10089.760101 secs 2024-07-26T13:54:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10089.760101 secs 2024-07-26T13:54:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10089.760101 secs 2024-07-26T13:54:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:07 smithi144 ceph-mon[32129]: pgmap v5866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:07 smithi160 ceph-mon[36715]: pgmap v5866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:07 smithi183 ceph-mon[37273]: pgmap v5866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:08 smithi144 ceph-mon[32129]: pgmap v5867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:08 smithi160 ceph-mon[36715]: pgmap v5867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:08 smithi183 ceph-mon[37273]: pgmap v5867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:10 smithi144 ceph-mon[32129]: pgmap v5868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:10 smithi160 ceph-mon[36715]: pgmap v5868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:10 smithi183 ceph-mon[37273]: pgmap v5868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10094.760222 secs 2024-07-26T13:54:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10094.760222 secs 2024-07-26T13:54:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10094.760222 secs 2024-07-26T13:54:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:12 smithi144 ceph-mon[32129]: pgmap v5869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:12 smithi160 ceph-mon[36715]: pgmap v5869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:12 smithi183 ceph-mon[37273]: pgmap v5869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:14 smithi144 ceph-mon[32129]: pgmap v5870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:14 smithi160 ceph-mon[36715]: pgmap v5870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:14 smithi183 ceph-mon[37273]: pgmap v5870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10099.760362 secs 2024-07-26T13:54:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10099.760362 secs 2024-07-26T13:54:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10099.760362 secs 2024-07-26T13:54:17.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:17 smithi144 ceph-mon[32129]: pgmap v5871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:17 smithi160 ceph-mon[36715]: pgmap v5871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:17 smithi183 ceph-mon[37273]: pgmap v5871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:18 smithi144 ceph-mon[32129]: pgmap v5872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:18 smithi160 ceph-mon[36715]: pgmap v5872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:18 smithi183 ceph-mon[37273]: pgmap v5872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:20 smithi144 ceph-mon[32129]: pgmap v5873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:20.797 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:20.796+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:54:20.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:20.797+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:54:20.799 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:20.797+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:54:20.799 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:20.797+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:54:20.799 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:20.798+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:20 smithi160 ceph-mon[36715]: pgmap v5873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:20 smithi183 ceph-mon[37273]: pgmap v5873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10104.760455 secs 2024-07-26T13:54:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10104.760455 secs 2024-07-26T13:54:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10104.760455 secs 2024-07-26T13:54:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:22 smithi144 ceph-mon[32129]: pgmap v5874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:22 smithi160 ceph-mon[36715]: pgmap v5874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:22 smithi183 ceph-mon[37273]: pgmap v5874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:24 smithi144 ceph-mon[32129]: pgmap v5875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:24 smithi160 ceph-mon[36715]: pgmap v5875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:24 smithi183 ceph-mon[37273]: pgmap v5875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10109.760563 secs 2024-07-26T13:54:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10109.760563 secs 2024-07-26T13:54:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10109.760563 secs 2024-07-26T13:54:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:27 smithi144 ceph-mon[32129]: pgmap v5876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:27 smithi160 ceph-mon[36715]: pgmap v5876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:27 smithi183 ceph-mon[37273]: pgmap v5876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:28 smithi144 ceph-mon[32129]: pgmap v5877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:28 smithi160 ceph-mon[36715]: pgmap v5877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:28 smithi183 ceph-mon[37273]: pgmap v5877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:30 smithi144 ceph-mon[32129]: pgmap v5878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:30 smithi160 ceph-mon[36715]: pgmap v5878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:30 smithi183 ceph-mon[37273]: pgmap v5878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10114.760689 secs 2024-07-26T13:54:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10114.760689 secs 2024-07-26T13:54:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10114.760689 secs 2024-07-26T13:54:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:32 smithi144 ceph-mon[32129]: pgmap v5879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:32 smithi160 ceph-mon[36715]: pgmap v5879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:32 smithi183 ceph-mon[37273]: pgmap v5879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:34 smithi144 ceph-mon[32129]: pgmap v5880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:34 smithi160 ceph-mon[36715]: pgmap v5880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:34 smithi183 ceph-mon[37273]: pgmap v5880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:35.812+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:35.813+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:35.813+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:35.813+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:35.813+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10119.760829 secs 2024-07-26T13:54:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10119.760829 secs 2024-07-26T13:54:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10119.760829 secs 2024-07-26T13:54:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:37 smithi144 ceph-mon[32129]: pgmap v5881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:37 smithi160 ceph-mon[36715]: pgmap v5881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:37 smithi183 ceph-mon[37273]: pgmap v5881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:38 smithi144 ceph-mon[32129]: pgmap v5882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:38 smithi160 ceph-mon[36715]: pgmap v5882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:38 smithi183 ceph-mon[37273]: pgmap v5882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:40 smithi144 ceph-mon[32129]: pgmap v5883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:40 smithi160 ceph-mon[36715]: pgmap v5883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:40 smithi183 ceph-mon[37273]: pgmap v5883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:54:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10124.760928 secs 2024-07-26T13:54:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10124.760928 secs 2024-07-26T13:54:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10124.760928 secs 2024-07-26T13:54:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:42 smithi144 ceph-mon[32129]: pgmap v5884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:42 smithi160 ceph-mon[36715]: pgmap v5884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:42 smithi183 ceph-mon[37273]: pgmap v5884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:44 smithi144 ceph-mon[32129]: pgmap v5885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:44 smithi160 ceph-mon[36715]: pgmap v5885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:44 smithi183 ceph-mon[37273]: pgmap v5885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10129.761041 secs 2024-07-26T13:54:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10129.761041 secs 2024-07-26T13:54:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10129.761041 secs 2024-07-26T13:54:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:47 smithi144 ceph-mon[32129]: pgmap v5886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:47 smithi160 ceph-mon[36715]: pgmap v5886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:47 smithi183 ceph-mon[37273]: pgmap v5886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:48 smithi144 ceph-mon[32129]: pgmap v5887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:48 smithi160 ceph-mon[36715]: pgmap v5887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:48 smithi183 ceph-mon[37273]: pgmap v5887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:49.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:49.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:49.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:49.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:50 smithi144 ceph-mon[32129]: pgmap v5888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:50 smithi160 ceph-mon[36715]: pgmap v5888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:50.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:54:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:50.831+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:54:50.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:50.831+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:54:50.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:50.831+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:54:50.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:54:50.831+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:50 smithi183 ceph-mon[37273]: pgmap v5888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10134.761126 secs 2024-07-26T13:54:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10134.761126 secs 2024-07-26T13:54:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10134.761126 secs 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:52 smithi160 ceph-mon[36715]: pgmap v5889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:52 smithi183 ceph-mon[37273]: pgmap v5889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:52 smithi144 ceph-mon[32129]: pgmap v5889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: pgmap v5890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: pgmap v5891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: pgmap v5890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:54:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: pgmap v5891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: pgmap v5890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:54:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: pgmap v5891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:54:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:54:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10139.761256 secs 2024-07-26T13:54:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10139.761256 secs 2024-07-26T13:54:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10139.761256 secs 2024-07-26T13:54:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:57 smithi144 ceph-mon[32129]: pgmap v5892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:57 smithi160 ceph-mon[36715]: pgmap v5892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:57 smithi183 ceph-mon[37273]: pgmap v5892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:54:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:54:59 smithi144 ceph-mon[32129]: pgmap v5893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:54:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:54:59 smithi160 ceph-mon[36715]: pgmap v5893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:54:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:54:59 smithi183 ceph-mon[37273]: pgmap v5893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:01 smithi144 ceph-mon[32129]: pgmap v5894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10144.761394 secs 2024-07-26T13:55:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:01 smithi160 ceph-mon[36715]: pgmap v5894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10144.761394 secs 2024-07-26T13:55:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:01 smithi183 ceph-mon[37273]: pgmap v5894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10144.761394 secs 2024-07-26T13:55:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:03 smithi144 ceph-mon[32129]: pgmap v5895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:03 smithi160 ceph-mon[36715]: pgmap v5895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:03 smithi183 ceph-mon[37273]: pgmap v5895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:05 smithi144 ceph-mon[32129]: pgmap v5896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:55:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:05 smithi160 ceph-mon[36715]: pgmap v5896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:55:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:05 smithi183 ceph-mon[37273]: pgmap v5896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:55:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:05.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:55:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:05.847+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:55:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:05.847+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:55:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:05.847+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:55:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:05.848+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10149.761513 secs 2024-07-26T13:55:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10149.761513 secs 2024-07-26T13:55:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10149.761513 secs 2024-07-26T13:55:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:07 smithi160 ceph-mon[36715]: pgmap v5897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:07 smithi183 ceph-mon[37273]: pgmap v5897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:07.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:07 smithi144 ceph-mon[32129]: pgmap v5897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:09 smithi160 ceph-mon[36715]: pgmap v5898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:09 smithi183 ceph-mon[37273]: pgmap v5898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:09 smithi144 ceph-mon[32129]: pgmap v5898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:11 smithi160 ceph-mon[36715]: pgmap v5899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10154.761607 secs 2024-07-26T13:55:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:11 smithi183 ceph-mon[37273]: pgmap v5899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10154.761607 secs 2024-07-26T13:55:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:11 smithi144 ceph-mon[32129]: pgmap v5899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10154.761607 secs 2024-07-26T13:55:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:13 smithi160 ceph-mon[36715]: pgmap v5900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:13 smithi183 ceph-mon[37273]: pgmap v5900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:13 smithi144 ceph-mon[32129]: pgmap v5900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:15 smithi160 ceph-mon[36715]: pgmap v5901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:15 smithi183 ceph-mon[37273]: pgmap v5901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:15 smithi144 ceph-mon[32129]: pgmap v5901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10159.761731 secs 2024-07-26T13:55:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10159.761731 secs 2024-07-26T13:55:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10159.761731 secs 2024-07-26T13:55:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:17 smithi160 ceph-mon[36715]: pgmap v5902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:17 smithi183 ceph-mon[37273]: pgmap v5902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:17 smithi144 ceph-mon[32129]: pgmap v5902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:19 smithi144 ceph-mon[32129]: pgmap v5903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:19 smithi160 ceph-mon[36715]: pgmap v5903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:19 smithi183 ceph-mon[37273]: pgmap v5903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:20.853+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:55:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:20.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:55:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:20.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:55:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:20.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:55:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:20.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:21 smithi160 ceph-mon[36715]: pgmap v5904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10164.761845 secs 2024-07-26T13:55:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:21 smithi183 ceph-mon[37273]: pgmap v5904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:21.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10164.761845 secs 2024-07-26T13:55:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:21 smithi144 ceph-mon[32129]: pgmap v5904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10164.761845 secs 2024-07-26T13:55:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:23 smithi160 ceph-mon[36715]: pgmap v5905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:23 smithi183 ceph-mon[37273]: pgmap v5905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:23 smithi144 ceph-mon[32129]: pgmap v5905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:25 smithi160 ceph-mon[36715]: pgmap v5906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:25 smithi183 ceph-mon[37273]: pgmap v5906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:25 smithi144 ceph-mon[32129]: pgmap v5906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10169.761939 secs 2024-07-26T13:55:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10169.761939 secs 2024-07-26T13:55:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10169.761939 secs 2024-07-26T13:55:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:27 smithi183 ceph-mon[37273]: pgmap v5907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:27 smithi144 ceph-mon[32129]: pgmap v5907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:27 smithi160 ceph-mon[36715]: pgmap v5907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:29 smithi183 ceph-mon[37273]: pgmap v5908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:29 smithi144 ceph-mon[32129]: pgmap v5908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:29 smithi160 ceph-mon[36715]: pgmap v5908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:31 smithi183 ceph-mon[37273]: pgmap v5909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10174.762061 secs 2024-07-26T13:55:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:31 smithi144 ceph-mon[32129]: pgmap v5909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10174.762061 secs 2024-07-26T13:55:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:31 smithi160 ceph-mon[36715]: pgmap v5909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10174.762061 secs 2024-07-26T13:55:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:33 smithi183 ceph-mon[37273]: pgmap v5910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:33 smithi144 ceph-mon[32129]: pgmap v5910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:33 smithi160 ceph-mon[36715]: pgmap v5910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:35 smithi183 ceph-mon[37273]: pgmap v5911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:35 smithi144 ceph-mon[32129]: pgmap v5911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:35 smithi160 ceph-mon[36715]: pgmap v5911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:35.864+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:55:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:35.865+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:55:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:35.865+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:55:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:35.865+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:55:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:35.865+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10179.762177 secs 2024-07-26T13:55:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10179.762177 secs 2024-07-26T13:55:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10179.762177 secs 2024-07-26T13:55:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:37 smithi183 ceph-mon[37273]: pgmap v5912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:37 smithi144 ceph-mon[32129]: pgmap v5912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:37 smithi160 ceph-mon[36715]: pgmap v5912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:39 smithi183 ceph-mon[37273]: pgmap v5913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:39 smithi144 ceph-mon[32129]: pgmap v5913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:39 smithi160 ceph-mon[36715]: pgmap v5913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:41.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:41 smithi160 ceph-mon[36715]: pgmap v5914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:41.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10184.762320 secs 2024-07-26T13:55:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:41 smithi183 ceph-mon[37273]: pgmap v5914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10184.762320 secs 2024-07-26T13:55:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:41 smithi144 ceph-mon[32129]: pgmap v5914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10184.762320 secs 2024-07-26T13:55:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:43 smithi183 ceph-mon[37273]: pgmap v5915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:43 smithi144 ceph-mon[32129]: pgmap v5915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:43 smithi160 ceph-mon[36715]: pgmap v5915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:44 smithi183 ceph-mon[37273]: pgmap v5916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:44 smithi144 ceph-mon[32129]: pgmap v5916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:44 smithi160 ceph-mon[36715]: pgmap v5916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10189.762411 secs 2024-07-26T13:55:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:46 smithi144 ceph-mon[32129]: pgmap v5917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10189.762411 secs 2024-07-26T13:55:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:46 smithi160 ceph-mon[36715]: pgmap v5917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10189.762411 secs 2024-07-26T13:55:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:46 smithi183 ceph-mon[37273]: pgmap v5917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:48 smithi144 ceph-mon[32129]: pgmap v5918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:48 smithi160 ceph-mon[36715]: pgmap v5918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:48 smithi183 ceph-mon[37273]: pgmap v5918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:55:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:50.882+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:55:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:50.883+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:55:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:50.883+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:55:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:50.883+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:55:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:55:50.883+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:50 smithi144 ceph-mon[32129]: pgmap v5919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:50 smithi160 ceph-mon[36715]: pgmap v5919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:50 smithi183 ceph-mon[37273]: pgmap v5919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10194.762496 secs 2024-07-26T13:55:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10194.762496 secs 2024-07-26T13:55:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10194.762496 secs 2024-07-26T13:55:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:52 smithi144 ceph-mon[32129]: pgmap v5920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:52 smithi160 ceph-mon[36715]: pgmap v5920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:52 smithi183 ceph-mon[37273]: pgmap v5920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:55:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:54 smithi144 ceph-mon[32129]: pgmap v5921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:54 smithi160 ceph-mon[36715]: pgmap v5921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:54 smithi183 ceph-mon[37273]: pgmap v5921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:55:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:55 smithi144 ceph-mon[32129]: pgmap v5922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:55:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:55 smithi144 ceph-mon[32129]: pgmap v5923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:55:56.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:56.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:56.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:55 smithi183 ceph-mon[37273]: pgmap v5922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:55:56.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:55 smithi183 ceph-mon[37273]: pgmap v5923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:56.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:55:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:55 smithi160 ceph-mon[36715]: pgmap v5922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:55:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:55 smithi160 ceph-mon[36715]: pgmap v5923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:55:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:55:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10199.762592 secs 2024-07-26T13:55:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10199.762592 secs 2024-07-26T13:55:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10199.762592 secs 2024-07-26T13:55:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:55:57 smithi160 ceph-mon[36715]: pgmap v5924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:55:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:55:57 smithi183 ceph-mon[37273]: pgmap v5924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:55:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:55:57 smithi144 ceph-mon[32129]: pgmap v5924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:00 smithi183 ceph-mon[37273]: pgmap v5925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:56:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:00 smithi144 ceph-mon[32129]: pgmap v5925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:56:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:00 smithi160 ceph-mon[36715]: pgmap v5925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:56:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10204.762714 secs 2024-07-26T13:56:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:02 smithi183 ceph-mon[37273]: pgmap v5926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10204.762714 secs 2024-07-26T13:56:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:02 smithi144 ceph-mon[32129]: pgmap v5926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10204.762714 secs 2024-07-26T13:56:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:02 smithi160 ceph-mon[36715]: pgmap v5926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:04 smithi183 ceph-mon[37273]: pgmap v5927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:04 smithi144 ceph-mon[32129]: pgmap v5927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:04 smithi160 ceph-mon[36715]: pgmap v5927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:05.899+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:56:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:05.900+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:56:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:05.900+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:56:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:05.900+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:56:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:05.900+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:06 smithi183 ceph-mon[37273]: pgmap v5928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:56:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:06 smithi144 ceph-mon[32129]: pgmap v5928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:56:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:06 smithi160 ceph-mon[36715]: pgmap v5928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:56:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10209.762834 secs 2024-07-26T13:56:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10209.762834 secs 2024-07-26T13:56:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10209.762834 secs 2024-07-26T13:56:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:08 smithi183 ceph-mon[37273]: pgmap v5929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:08 smithi144 ceph-mon[32129]: pgmap v5929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:08 smithi160 ceph-mon[36715]: pgmap v5929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:10 smithi144 ceph-mon[32129]: pgmap v5930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:10 smithi160 ceph-mon[36715]: pgmap v5930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:10 smithi183 ceph-mon[37273]: pgmap v5930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10214.762972 secs 2024-07-26T13:56:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:12 smithi144 ceph-mon[32129]: pgmap v5931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10214.762972 secs 2024-07-26T13:56:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:12 smithi160 ceph-mon[36715]: pgmap v5931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10214.762972 secs 2024-07-26T13:56:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:12 smithi183 ceph-mon[37273]: pgmap v5931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:14 smithi144 ceph-mon[32129]: pgmap v5932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:14 smithi160 ceph-mon[36715]: pgmap v5932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:14 smithi183 ceph-mon[37273]: pgmap v5932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:16 smithi144 ceph-mon[32129]: pgmap v5933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:16 smithi160 ceph-mon[36715]: pgmap v5933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:16 smithi183 ceph-mon[37273]: pgmap v5933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10219.763097 secs 2024-07-26T13:56:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10219.763097 secs 2024-07-26T13:56:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10219.763097 secs 2024-07-26T13:56:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:18 smithi144 ceph-mon[32129]: pgmap v5934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:18 smithi160 ceph-mon[36715]: pgmap v5934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:18 smithi183 ceph-mon[37273]: pgmap v5934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:20 smithi144 ceph-mon[32129]: pgmap v5935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:20 smithi160 ceph-mon[36715]: pgmap v5935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:20 smithi183 ceph-mon[37273]: pgmap v5935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:20.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:20.904+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:56:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:20.905+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:56:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:20.905+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:56:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:20.905+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:56:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:20.905+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10224.763197 secs 2024-07-26T13:56:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10224.763197 secs 2024-07-26T13:56:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10224.763197 secs 2024-07-26T13:56:22.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:22 smithi144 ceph-mon[32129]: pgmap v5936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:22 smithi160 ceph-mon[36715]: pgmap v5936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:22 smithi183 ceph-mon[37273]: pgmap v5936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:24 smithi144 ceph-mon[32129]: pgmap v5937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:24 smithi160 ceph-mon[36715]: pgmap v5937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:24 smithi183 ceph-mon[37273]: pgmap v5937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:26 smithi144 ceph-mon[32129]: pgmap v5938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10229.763329 secs 2024-07-26T13:56:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:26 smithi160 ceph-mon[36715]: pgmap v5938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10229.763329 secs 2024-07-26T13:56:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:26 smithi183 ceph-mon[37273]: pgmap v5938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10229.763329 secs 2024-07-26T13:56:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:28 smithi144 ceph-mon[32129]: pgmap v5939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:28 smithi160 ceph-mon[36715]: pgmap v5939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:28 smithi183 ceph-mon[37273]: pgmap v5939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:30 smithi144 ceph-mon[32129]: pgmap v5940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:30 smithi160 ceph-mon[36715]: pgmap v5940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:30 smithi183 ceph-mon[37273]: pgmap v5940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10234.763449 secs 2024-07-26T13:56:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10234.763449 secs 2024-07-26T13:56:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10234.763449 secs 2024-07-26T13:56:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:32 smithi160 ceph-mon[36715]: pgmap v5941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:32.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:32 smithi144 ceph-mon[32129]: pgmap v5941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:32 smithi183 ceph-mon[37273]: pgmap v5941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:34 smithi160 ceph-mon[36715]: pgmap v5942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:34 smithi183 ceph-mon[37273]: pgmap v5942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:34 smithi144 ceph-mon[32129]: pgmap v5942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:35.920+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:56:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:35.921+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:56:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:35.921+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:56:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:35.921+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:56:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:35.921+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:36 smithi160 ceph-mon[36715]: pgmap v5943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10239.763570 secs 2024-07-26T13:56:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:36 smithi183 ceph-mon[37273]: pgmap v5943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10239.763570 secs 2024-07-26T13:56:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:36 smithi144 ceph-mon[32129]: pgmap v5943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10239.763570 secs 2024-07-26T13:56:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:38 smithi160 ceph-mon[36715]: pgmap v5944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:38 smithi183 ceph-mon[37273]: pgmap v5944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:38 smithi144 ceph-mon[32129]: pgmap v5944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:40 smithi160 ceph-mon[36715]: pgmap v5945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:40 smithi183 ceph-mon[37273]: pgmap v5945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:40 smithi144 ceph-mon[32129]: pgmap v5945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10244.763689 secs 2024-07-26T13:56:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10244.763689 secs 2024-07-26T13:56:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10244.763689 secs 2024-07-26T13:56:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:42 smithi160 ceph-mon[36715]: pgmap v5946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:42 smithi183 ceph-mon[37273]: pgmap v5946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:42 smithi144 ceph-mon[32129]: pgmap v5946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:44 smithi160 ceph-mon[36715]: pgmap v5947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:44 smithi183 ceph-mon[37273]: pgmap v5947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:44 smithi144 ceph-mon[32129]: pgmap v5947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:46 smithi160 ceph-mon[36715]: pgmap v5948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10249.763811 secs 2024-07-26T13:56:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:46 smithi183 ceph-mon[37273]: pgmap v5948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10249.763811 secs 2024-07-26T13:56:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:46 smithi144 ceph-mon[32129]: pgmap v5948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10249.763811 secs 2024-07-26T13:56:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:47.910+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:56:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:47.911+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:56:47.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:47.913+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:56:47.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:47.913+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:56:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:48 smithi160 ceph-mon[36715]: pgmap v5949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:48 smithi183 ceph-mon[37273]: pgmap v5949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:48 smithi144 ceph-mon[32129]: pgmap v5949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:56:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:50 smithi160 ceph-mon[36715]: pgmap v5950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:50 smithi183 ceph-mon[37273]: pgmap v5950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:50 smithi144 ceph-mon[32129]: pgmap v5950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:50.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:50.938+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:56:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:50.939+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:56:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:50.939+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:56:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:50.939+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:56:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:56:50.939+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10254.763931 secs 2024-07-26T13:56:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10254.763931 secs 2024-07-26T13:56:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10254.763931 secs 2024-07-26T13:56:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:52 smithi160 ceph-mon[36715]: pgmap v5951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:52.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:52 smithi144 ceph-mon[32129]: pgmap v5951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:52 smithi183 ceph-mon[37273]: pgmap v5951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:54 smithi160 ceph-mon[36715]: pgmap v5952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:54 smithi183 ceph-mon[37273]: pgmap v5952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:54 smithi144 ceph-mon[32129]: pgmap v5952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:55.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:56 smithi183 ceph-mon[37273]: pgmap v5953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10259.764065 secs 2024-07-26T13:56:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:56:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:56 smithi144 ceph-mon[32129]: pgmap v5953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10259.764065 secs 2024-07-26T13:56:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:56:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:56 smithi160 ceph-mon[36715]: pgmap v5953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:56:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10259.764065 secs 2024-07-26T13:56:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:56:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:57 smithi183 ceph-mon[37273]: pgmap v5954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:57 smithi183 ceph-mon[37273]: pgmap v5955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:57 smithi144 ceph-mon[32129]: pgmap v5954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:57 smithi144 ceph-mon[32129]: pgmap v5955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:57 smithi160 ceph-mon[36715]: pgmap v5954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:57 smithi160 ceph-mon[36715]: pgmap v5955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:56:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:56:59 smithi183 ceph-mon[37273]: pgmap v5956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:56:59 smithi144 ceph-mon[32129]: pgmap v5956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:56:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:56:59 smithi160 ceph-mon[36715]: pgmap v5956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:01 smithi183 ceph-mon[37273]: pgmap v5957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10264.764167 secs 2024-07-26T13:57:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:01 smithi144 ceph-mon[32129]: pgmap v5957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10264.764167 secs 2024-07-26T13:57:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:01 smithi160 ceph-mon[36715]: pgmap v5957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10264.764167 secs 2024-07-26T13:57:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:03 smithi183 ceph-mon[37273]: pgmap v5958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:57:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:03 smithi144 ceph-mon[32129]: pgmap v5958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:57:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:03 smithi160 ceph-mon[36715]: pgmap v5958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T13:57:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:05 smithi183 ceph-mon[37273]: pgmap v5959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:05 smithi144 ceph-mon[32129]: pgmap v5959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:05 smithi160 ceph-mon[36715]: pgmap v5959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:05.951+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:57:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:05.952+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:57:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:05.952+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:57:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:05.952+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:57:05.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:05.952+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10269.764329 secs 2024-07-26T13:57:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10269.764329 secs 2024-07-26T13:57:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10269.764329 secs 2024-07-26T13:57:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:07 smithi183 ceph-mon[37273]: pgmap v5960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:57:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:07 smithi144 ceph-mon[32129]: pgmap v5960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:57:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:07 smithi160 ceph-mon[36715]: pgmap v5960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T13:57:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:09 smithi183 ceph-mon[37273]: pgmap v5961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:57:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:09 smithi144 ceph-mon[32129]: pgmap v5961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:57:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:09 smithi160 ceph-mon[36715]: pgmap v5961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:57:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:11 smithi183 ceph-mon[37273]: pgmap v5962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10274.764447 secs 2024-07-26T13:57:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:11 smithi144 ceph-mon[32129]: pgmap v5962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10274.764447 secs 2024-07-26T13:57:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:11 smithi160 ceph-mon[36715]: pgmap v5962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10274.764447 secs 2024-07-26T13:57:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:13 smithi183 ceph-mon[37273]: pgmap v5963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:13 smithi144 ceph-mon[32129]: pgmap v5963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:13 smithi160 ceph-mon[36715]: pgmap v5963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:14 smithi144 ceph-mon[32129]: pgmap v5964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:14 smithi160 ceph-mon[36715]: pgmap v5964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:14 smithi183 ceph-mon[37273]: pgmap v5964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10279.764566 secs 2024-07-26T13:57:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10279.764566 secs 2024-07-26T13:57:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10279.764566 secs 2024-07-26T13:57:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:17 smithi160 ceph-mon[36715]: pgmap v5965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:17 smithi183 ceph-mon[37273]: pgmap v5965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:17 smithi144 ceph-mon[32129]: pgmap v5965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:19 smithi183 ceph-mon[37273]: pgmap v5966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:19 smithi144 ceph-mon[32129]: pgmap v5966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:19 smithi160 ceph-mon[36715]: pgmap v5966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:20.957+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:57:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:20.958+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:57:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:20.958+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:57:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:20.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:57:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:20.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:21 smithi183 ceph-mon[37273]: pgmap v5967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10284.764663 secs 2024-07-26T13:57:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:21 smithi144 ceph-mon[32129]: pgmap v5967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10284.764663 secs 2024-07-26T13:57:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:21 smithi160 ceph-mon[36715]: pgmap v5967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10284.764663 secs 2024-07-26T13:57:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:23 smithi183 ceph-mon[37273]: pgmap v5968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:23 smithi144 ceph-mon[32129]: pgmap v5968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:23 smithi160 ceph-mon[36715]: pgmap v5968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:25 smithi183 ceph-mon[37273]: pgmap v5969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:25 smithi144 ceph-mon[32129]: pgmap v5969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:25 smithi160 ceph-mon[36715]: pgmap v5969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10289.764787 secs 2024-07-26T13:57:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10289.764787 secs 2024-07-26T13:57:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10289.764787 secs 2024-07-26T13:57:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:27 smithi183 ceph-mon[37273]: pgmap v5970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:27 smithi144 ceph-mon[32129]: pgmap v5970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:27 smithi160 ceph-mon[36715]: pgmap v5970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:29 smithi183 ceph-mon[37273]: pgmap v5971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:29 smithi144 ceph-mon[32129]: pgmap v5971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:29 smithi160 ceph-mon[36715]: pgmap v5971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:31 smithi183 ceph-mon[37273]: pgmap v5972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10294.764915 secs 2024-07-26T13:57:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:31 smithi144 ceph-mon[32129]: pgmap v5972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10294.764915 secs 2024-07-26T13:57:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:31 smithi160 ceph-mon[36715]: pgmap v5972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10294.764915 secs 2024-07-26T13:57:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:33 smithi183 ceph-mon[37273]: pgmap v5973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:33 smithi144 ceph-mon[32129]: pgmap v5973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:33 smithi160 ceph-mon[36715]: pgmap v5973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:35 smithi183 ceph-mon[37273]: pgmap v5974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:35 smithi144 ceph-mon[32129]: pgmap v5974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:35 smithi160 ceph-mon[36715]: pgmap v5974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:35.973+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:57:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:35.974+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:57:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:35.974+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:57:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:35.974+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:57:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:35.974+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10299.765031 secs 2024-07-26T13:57:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10299.765031 secs 2024-07-26T13:57:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10299.765031 secs 2024-07-26T13:57:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:37 smithi183 ceph-mon[37273]: pgmap v5975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:37 smithi144 ceph-mon[32129]: pgmap v5975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:37 smithi160 ceph-mon[36715]: pgmap v5975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:39 smithi144 ceph-mon[32129]: pgmap v5976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:39 smithi160 ceph-mon[36715]: pgmap v5976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:39 smithi183 ceph-mon[37273]: pgmap v5976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:40 smithi144 ceph-mon[32129]: pgmap v5977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:40 smithi160 ceph-mon[36715]: pgmap v5977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:40 smithi183 ceph-mon[37273]: pgmap v5977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10304.765151 secs 2024-07-26T13:57:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10304.765151 secs 2024-07-26T13:57:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10304.765151 secs 2024-07-26T13:57:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:42 smithi144 ceph-mon[32129]: pgmap v5978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:42 smithi160 ceph-mon[36715]: pgmap v5978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:42 smithi183 ceph-mon[37273]: pgmap v5978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:45 smithi183 ceph-mon[37273]: pgmap v5979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:45 smithi144 ceph-mon[32129]: pgmap v5979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:45 smithi160 ceph-mon[36715]: pgmap v5979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10309.765272 secs 2024-07-26T13:57:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10309.765272 secs 2024-07-26T13:57:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10309.765272 secs 2024-07-26T13:57:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:47 smithi183 ceph-mon[37273]: pgmap v5980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:47 smithi144 ceph-mon[32129]: pgmap v5980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:47 smithi160 ceph-mon[36715]: pgmap v5980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:49 smithi183 ceph-mon[37273]: pgmap v5981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:49 smithi144 ceph-mon[32129]: pgmap v5981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:49 smithi160 ceph-mon[36715]: pgmap v5981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:50.984+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:57:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:50.985+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:57:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:50.985+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:57:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:50.985+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:57:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:57:50.985+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:51 smithi183 ceph-mon[37273]: pgmap v5982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10314.765398 secs 2024-07-26T13:57:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:51 smithi144 ceph-mon[32129]: pgmap v5982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10314.765398 secs 2024-07-26T13:57:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:51 smithi160 ceph-mon[36715]: pgmap v5982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10314.765398 secs 2024-07-26T13:57:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:53 smithi183 ceph-mon[37273]: pgmap v5983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:53 smithi144 ceph-mon[32129]: pgmap v5983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:53 smithi160 ceph-mon[36715]: pgmap v5983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:57:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:55 smithi183 ceph-mon[37273]: pgmap v5984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:55 smithi144 ceph-mon[32129]: pgmap v5984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:55 smithi160 ceph-mon[36715]: pgmap v5984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:57:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10319.765521 secs 2024-07-26T13:57:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10319.765521 secs 2024-07-26T13:57:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10319.765521 secs 2024-07-26T13:57:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:57 smithi183 ceph-mon[37273]: pgmap v5985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:57 smithi144 ceph-mon[32129]: pgmap v5985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:57 smithi160 ceph-mon[36715]: pgmap v5985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:57:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:57:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:57:58.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:58.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:58.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:57:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:57:59 smithi144 ceph-mon[32129]: pgmap v5986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:57:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:57:59 smithi160 ceph-mon[36715]: pgmap v5986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:57:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:57:59 smithi183 ceph-mon[37273]: pgmap v5986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:00 smithi144 ceph-mon[32129]: pgmap v5987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:58:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:00 smithi160 ceph-mon[36715]: pgmap v5987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:58:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:00 smithi183 ceph-mon[37273]: pgmap v5987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T13:58:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10324.765589 secs 2024-07-26T13:58:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10324.765589 secs 2024-07-26T13:58:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10324.765589 secs 2024-07-26T13:58:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:02 smithi144 ceph-mon[32129]: pgmap v5988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:02 smithi160 ceph-mon[36715]: pgmap v5988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:02 smithi183 ceph-mon[37273]: pgmap v5988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:04 smithi144 ceph-mon[32129]: pgmap v5989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:04 smithi160 ceph-mon[36715]: pgmap v5989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:04 smithi183 ceph-mon[37273]: pgmap v5989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:06.002 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:06.001+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:58:06.002 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:06.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:58:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:06.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:58:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:06.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:58:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:06.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:06 smithi144 ceph-mon[32129]: pgmap v5990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10329.765707 secs 2024-07-26T13:58:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:06 smithi160 ceph-mon[36715]: pgmap v5990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10329.765707 secs 2024-07-26T13:58:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:06 smithi183 ceph-mon[37273]: pgmap v5990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10329.765707 secs 2024-07-26T13:58:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:08 smithi144 ceph-mon[32129]: pgmap v5991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:08 smithi160 ceph-mon[36715]: pgmap v5991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:08 smithi183 ceph-mon[37273]: pgmap v5991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T13:58:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:10 smithi144 ceph-mon[32129]: pgmap v5992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:10 smithi160 ceph-mon[36715]: pgmap v5992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:10 smithi183 ceph-mon[37273]: pgmap v5992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10334.765832 secs 2024-07-26T13:58:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10334.765832 secs 2024-07-26T13:58:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10334.765832 secs 2024-07-26T13:58:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:12 smithi144 ceph-mon[32129]: pgmap v5993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:12 smithi160 ceph-mon[36715]: pgmap v5993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:12 smithi183 ceph-mon[37273]: pgmap v5993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:14 smithi144 ceph-mon[32129]: pgmap v5994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:14 smithi160 ceph-mon[36715]: pgmap v5994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:14 smithi183 ceph-mon[37273]: pgmap v5994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:16 smithi160 ceph-mon[36715]: pgmap v5995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10339.765934 secs 2024-07-26T13:58:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:16 smithi183 ceph-mon[37273]: pgmap v5995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10339.765934 secs 2024-07-26T13:58:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:16 smithi144 ceph-mon[32129]: pgmap v5995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10339.765934 secs 2024-07-26T13:58:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:18 smithi160 ceph-mon[36715]: pgmap v5996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:18 smithi183 ceph-mon[37273]: pgmap v5996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:18 smithi144 ceph-mon[32129]: pgmap v5996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:20 smithi160 ceph-mon[36715]: pgmap v5997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:20 smithi183 ceph-mon[37273]: pgmap v5997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:20 smithi144 ceph-mon[32129]: pgmap v5997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:21.017+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:58:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:21.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:58:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:21.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:58:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:21.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:58:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:21.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10344.766031 secs 2024-07-26T13:58:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10344.766031 secs 2024-07-26T13:58:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10344.766031 secs 2024-07-26T13:58:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:22 smithi160 ceph-mon[36715]: pgmap v5998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:22 smithi183 ceph-mon[37273]: pgmap v5998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:22 smithi144 ceph-mon[32129]: pgmap v5998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:24 smithi160 ceph-mon[36715]: pgmap v5999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:24 smithi183 ceph-mon[37273]: pgmap v5999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:24 smithi144 ceph-mon[32129]: pgmap v5999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:26 smithi160 ceph-mon[36715]: pgmap v6000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10349.766131 secs 2024-07-26T13:58:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:26 smithi183 ceph-mon[37273]: pgmap v6000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10349.766131 secs 2024-07-26T13:58:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:26 smithi144 ceph-mon[32129]: pgmap v6000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10349.766131 secs 2024-07-26T13:58:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:28 smithi160 ceph-mon[36715]: pgmap v6001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:28 smithi183 ceph-mon[37273]: pgmap v6001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:28 smithi144 ceph-mon[32129]: pgmap v6001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:30 smithi160 ceph-mon[36715]: pgmap v6002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:30 smithi183 ceph-mon[37273]: pgmap v6002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:30 smithi144 ceph-mon[32129]: pgmap v6002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10354.766254 secs 2024-07-26T13:58:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10354.766254 secs 2024-07-26T13:58:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10354.766254 secs 2024-07-26T13:58:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:32 smithi160 ceph-mon[36715]: pgmap v6003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:32 smithi183 ceph-mon[37273]: pgmap v6003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:32 smithi144 ceph-mon[32129]: pgmap v6003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:34 smithi160 ceph-mon[36715]: pgmap v6004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:34 smithi183 ceph-mon[37273]: pgmap v6004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:34 smithi144 ceph-mon[32129]: pgmap v6004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:36.032+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:58:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:36.033+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:58:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:36.033+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:58:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:36.033+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:58:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:36.033+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:36 smithi160 ceph-mon[36715]: pgmap v6005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10359.766377 secs 2024-07-26T13:58:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:36 smithi183 ceph-mon[37273]: pgmap v6005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10359.766377 secs 2024-07-26T13:58:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:36 smithi144 ceph-mon[32129]: pgmap v6005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10359.766377 secs 2024-07-26T13:58:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:38 smithi160 ceph-mon[36715]: pgmap v6006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:38 smithi183 ceph-mon[37273]: pgmap v6006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:38 smithi144 ceph-mon[32129]: pgmap v6006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:40 smithi160 ceph-mon[36715]: pgmap v6007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:40 smithi183 ceph-mon[37273]: pgmap v6007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:40 smithi144 ceph-mon[32129]: pgmap v6007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10364.766519 secs 2024-07-26T13:58:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10364.766519 secs 2024-07-26T13:58:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10364.766519 secs 2024-07-26T13:58:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:42 smithi183 ceph-mon[37273]: pgmap v6008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:42 smithi144 ceph-mon[32129]: pgmap v6008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:42 smithi160 ceph-mon[36715]: pgmap v6008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:44 smithi183 ceph-mon[37273]: pgmap v6009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:44 smithi144 ceph-mon[32129]: pgmap v6009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:44 smithi160 ceph-mon[36715]: pgmap v6009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:46.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:46 smithi160 ceph-mon[36715]: pgmap v6010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:46.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10369.766668 secs 2024-07-26T13:58:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:46 smithi183 ceph-mon[37273]: pgmap v6010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10369.766668 secs 2024-07-26T13:58:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:46 smithi144 ceph-mon[32129]: pgmap v6010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10369.766668 secs 2024-07-26T13:58:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:48 smithi183 ceph-mon[37273]: pgmap v6011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:48 smithi144 ceph-mon[32129]: pgmap v6011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:48 smithi160 ceph-mon[36715]: pgmap v6011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:50 smithi183 ceph-mon[37273]: pgmap v6012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:50 smithi144 ceph-mon[32129]: pgmap v6012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:51.051 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:51.050+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:58:51.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:51.051+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:58:51.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:51.051+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:58:51.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:51.051+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:58:51.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:58:51.051+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:50 smithi160 ceph-mon[36715]: pgmap v6012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10374.766836 secs 2024-07-26T13:58:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10374.766836 secs 2024-07-26T13:58:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10374.766836 secs 2024-07-26T13:58:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:52 smithi183 ceph-mon[37273]: pgmap v6013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:52 smithi144 ceph-mon[32129]: pgmap v6013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:52 smithi160 ceph-mon[36715]: pgmap v6013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:58:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:54 smithi183 ceph-mon[37273]: pgmap v6014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:54 smithi144 ceph-mon[32129]: pgmap v6014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:54 smithi160 ceph-mon[36715]: pgmap v6014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:56 smithi183 ceph-mon[37273]: pgmap v6015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10379.766907 secs 2024-07-26T13:58:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:56 smithi144 ceph-mon[32129]: pgmap v6015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10379.766907 secs 2024-07-26T13:58:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:56 smithi160 ceph-mon[36715]: pgmap v6015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:58:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10379.766907 secs 2024-07-26T13:58:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:58 smithi144 ceph-mon[32129]: pgmap v6016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:58:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:58 smithi160 ceph-mon[36715]: pgmap v6016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:58:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:58 smithi183 ceph-mon[37273]: pgmap v6016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:58:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T13:58:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:59 smithi144 ceph-mon[32129]: pgmap v6017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:58:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:58:59 smithi144 ceph-mon[32129]: pgmap v6018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:59:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:59 smithi160 ceph-mon[36715]: pgmap v6017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:58:59 smithi160 ceph-mon[36715]: pgmap v6018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:59:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:59 smithi183 ceph-mon[37273]: pgmap v6017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:58:59 smithi183 ceph-mon[37273]: pgmap v6018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T13:59:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:01 smithi144 ceph-mon[32129]: pgmap v6019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10384.767042 secs 2024-07-26T13:59:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:01 smithi160 ceph-mon[36715]: pgmap v6019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10384.767042 secs 2024-07-26T13:59:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:01 smithi183 ceph-mon[37273]: pgmap v6019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10384.767042 secs 2024-07-26T13:59:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:02.934+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:59:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:02.934+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T13:59:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:02.935+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:59:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:02.935+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T13:59:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:03 smithi144 ceph-mon[32129]: pgmap v6020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:59:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:03 smithi160 ceph-mon[36715]: pgmap v6020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:59:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:03 smithi183 ceph-mon[37273]: pgmap v6020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:59:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:05 smithi144 ceph-mon[32129]: pgmap v6021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:59:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:05 smithi160 ceph-mon[36715]: pgmap v6021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:59:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:06.067+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:59:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:06.068+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:59:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:06.068+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:59:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:06.068+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:59:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:06.068+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:05 smithi183 ceph-mon[37273]: pgmap v6021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T13:59:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10389.767178 secs 2024-07-26T13:59:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10389.767178 secs 2024-07-26T13:59:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10389.767178 secs 2024-07-26T13:59:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:07 smithi144 ceph-mon[32129]: pgmap v6022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:07 smithi160 ceph-mon[36715]: pgmap v6022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:07 smithi183 ceph-mon[37273]: pgmap v6022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T13:59:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:09 smithi144 ceph-mon[32129]: pgmap v6023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:59:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:09 smithi160 ceph-mon[36715]: pgmap v6023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:59:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:09 smithi183 ceph-mon[37273]: pgmap v6023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T13:59:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:11 smithi144 ceph-mon[32129]: pgmap v6024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10394.767280 secs 2024-07-26T13:59:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:11 smithi160 ceph-mon[36715]: pgmap v6024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10394.767280 secs 2024-07-26T13:59:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:11 smithi183 ceph-mon[37273]: pgmap v6024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10394.767280 secs 2024-07-26T13:59:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:13 smithi144 ceph-mon[32129]: pgmap v6025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:13 smithi160 ceph-mon[36715]: pgmap v6025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:13 smithi183 ceph-mon[37273]: pgmap v6025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:15 smithi144 ceph-mon[32129]: pgmap v6026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:15 smithi160 ceph-mon[36715]: pgmap v6026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:15 smithi183 ceph-mon[37273]: pgmap v6026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10399.767361 secs 2024-07-26T13:59:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10399.767361 secs 2024-07-26T13:59:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10399.767361 secs 2024-07-26T13:59:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:17 smithi160 ceph-mon[36715]: pgmap v6027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:17 smithi183 ceph-mon[37273]: pgmap v6027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:17 smithi144 ceph-mon[32129]: pgmap v6027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:19 smithi160 ceph-mon[36715]: pgmap v6028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:19 smithi183 ceph-mon[37273]: pgmap v6028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:19 smithi144 ceph-mon[32129]: pgmap v6028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:21.077+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:59:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:21.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:59:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:21.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:59:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:21.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:59:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:21.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:21 smithi160 ceph-mon[36715]: pgmap v6029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10404.767518 secs 2024-07-26T13:59:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:21 smithi183 ceph-mon[37273]: pgmap v6029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10404.767518 secs 2024-07-26T13:59:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:21 smithi144 ceph-mon[32129]: pgmap v6029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10404.767518 secs 2024-07-26T13:59:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:23 smithi160 ceph-mon[36715]: pgmap v6030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:23 smithi183 ceph-mon[37273]: pgmap v6030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:23 smithi144 ceph-mon[32129]: pgmap v6030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:26 smithi160 ceph-mon[36715]: pgmap v6031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10409.767583 secs 2024-07-26T13:59:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:26 smithi183 ceph-mon[37273]: pgmap v6031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10409.767583 secs 2024-07-26T13:59:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:26 smithi144 ceph-mon[32129]: pgmap v6031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10409.767583 secs 2024-07-26T13:59:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:28 smithi160 ceph-mon[36715]: pgmap v6032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:28 smithi183 ceph-mon[37273]: pgmap v6032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:28 smithi144 ceph-mon[32129]: pgmap v6032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:30 smithi160 ceph-mon[36715]: pgmap v6033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:30 smithi183 ceph-mon[37273]: pgmap v6033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:30 smithi144 ceph-mon[32129]: pgmap v6033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10414.767652 secs 2024-07-26T13:59:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10414.767652 secs 2024-07-26T13:59:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10414.767652 secs 2024-07-26T13:59:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:32 smithi160 ceph-mon[36715]: pgmap v6034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:32 smithi183 ceph-mon[37273]: pgmap v6034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:32 smithi144 ceph-mon[32129]: pgmap v6034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:34 smithi160 ceph-mon[36715]: pgmap v6035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:34 smithi183 ceph-mon[37273]: pgmap v6035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:34 smithi144 ceph-mon[32129]: pgmap v6035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:36.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:36.092+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:36.093+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:36.093+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:36.093+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:36.093+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:36 smithi160 ceph-mon[36715]: pgmap v6036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10419.767771 secs 2024-07-26T13:59:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:36 smithi183 ceph-mon[37273]: pgmap v6036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10419.767771 secs 2024-07-26T13:59:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:36 smithi144 ceph-mon[32129]: pgmap v6036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10419.767771 secs 2024-07-26T13:59:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:38 smithi160 ceph-mon[36715]: pgmap v6037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:38 smithi183 ceph-mon[37273]: pgmap v6037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:38 smithi144 ceph-mon[32129]: pgmap v6037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:40 smithi183 ceph-mon[37273]: pgmap v6038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:40 smithi144 ceph-mon[32129]: pgmap v6038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:40 smithi160 ceph-mon[36715]: pgmap v6038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10424.767871 secs 2024-07-26T13:59:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10424.767871 secs 2024-07-26T13:59:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10424.767871 secs 2024-07-26T13:59:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:42 smithi183 ceph-mon[37273]: pgmap v6039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:42 smithi144 ceph-mon[32129]: pgmap v6039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:42.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:42 smithi160 ceph-mon[36715]: pgmap v6039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:44 smithi183 ceph-mon[37273]: pgmap v6040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:44 smithi144 ceph-mon[32129]: pgmap v6040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:44 smithi160 ceph-mon[36715]: pgmap v6040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:46.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:46 smithi160 ceph-mon[36715]: pgmap v6041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:46.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10429.767937 secs 2024-07-26T13:59:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:46 smithi183 ceph-mon[37273]: pgmap v6041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10429.767937 secs 2024-07-26T13:59:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:46 smithi144 ceph-mon[32129]: pgmap v6041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10429.767937 secs 2024-07-26T13:59:47.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:47.941+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:59:47.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:47.942+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T13:59:47.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:47.942+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T13:59:47.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:47.942+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T13:59:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:48 smithi183 ceph-mon[37273]: pgmap v6042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:48 smithi144 ceph-mon[32129]: pgmap v6042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:48 smithi160 ceph-mon[36715]: pgmap v6042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:50 smithi183 ceph-mon[37273]: pgmap v6043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:50 smithi144 ceph-mon[32129]: pgmap v6043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:50 smithi160 ceph-mon[36715]: pgmap v6043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:51.109+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:59:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:51.110+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:59:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:51.110+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:59:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:51.110+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:59:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T13:59:51.110+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10434.768080 secs 2024-07-26T13:59:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10434.768080 secs 2024-07-26T13:59:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10434.768080 secs 2024-07-26T13:59:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:52 smithi183 ceph-mon[37273]: pgmap v6044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:52 smithi144 ceph-mon[32129]: pgmap v6044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:52 smithi160 ceph-mon[36715]: pgmap v6044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:54 smithi183 ceph-mon[37273]: pgmap v6045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:54 smithi144 ceph-mon[32129]: pgmap v6045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:54 smithi160 ceph-mon[36715]: pgmap v6045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T13:59:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:56 smithi183 ceph-mon[37273]: pgmap v6046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10439.768201 secs 2024-07-26T13:59:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:56 smithi144 ceph-mon[32129]: pgmap v6046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10439.768201 secs 2024-07-26T13:59:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:56 smithi160 ceph-mon[36715]: pgmap v6046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T13:59:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10439.768201 secs 2024-07-26T13:59:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:58 smithi183 ceph-mon[37273]: pgmap v6047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:58 smithi144 ceph-mon[32129]: pgmap v6047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:58 smithi160 ceph-mon[36715]: pgmap v6047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T13:59:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 13:59:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 13:59:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 13:59:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:00.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:00 smithi144 ceph-mon[32129]: pgmap v6048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:00.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:00.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:00.409 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:00:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:00 smithi160 ceph-mon[36715]: pgmap v6048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:00:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:00 smithi183 ceph-mon[37273]: pgmap v6048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:00:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:01 smithi144 ceph-mon[32129]: pgmap v6049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:00:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:01 smithi144 ceph-mon[32129]: pgmap v6050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:00:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10444.768322 secs 2024-07-26T14:00:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:01 smithi160 ceph-mon[36715]: pgmap v6049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:00:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:01 smithi160 ceph-mon[36715]: pgmap v6050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:00:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10444.768322 secs 2024-07-26T14:00:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:01 smithi183 ceph-mon[37273]: pgmap v6049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:00:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:01 smithi183 ceph-mon[37273]: pgmap v6050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:00:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10444.768322 secs 2024-07-26T14:00:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:03 smithi144 ceph-mon[32129]: pgmap v6051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:00:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:03 smithi160 ceph-mon[36715]: pgmap v6051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:00:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:03 smithi183 ceph-mon[37273]: pgmap v6051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:00:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:04 smithi160 ceph-mon[36715]: pgmap v6052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:00:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:04 smithi183 ceph-mon[37273]: pgmap v6052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:00:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:04 smithi144 ceph-mon[32129]: pgmap v6052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:00:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:06.117+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:00:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:06.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:00:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:06.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:00:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:06.118+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:00:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:06.118+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10449.768439 secs 2024-07-26T14:00:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:06 smithi183 ceph-mon[37273]: pgmap v6053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:00:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10449.768439 secs 2024-07-26T14:00:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:06 smithi144 ceph-mon[32129]: pgmap v6053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:00:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10449.768439 secs 2024-07-26T14:00:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:06 smithi160 ceph-mon[36715]: pgmap v6053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:00:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:08 smithi183 ceph-mon[37273]: pgmap v6054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:00:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:08 smithi144 ceph-mon[32129]: pgmap v6054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:00:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:08 smithi160 ceph-mon[36715]: pgmap v6054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:11 smithi160 ceph-mon[36715]: pgmap v6055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10454.768558 secs 2024-07-26T14:00:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:11 smithi183 ceph-mon[37273]: pgmap v6055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:00:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10454.768558 secs 2024-07-26T14:00:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:11 smithi144 ceph-mon[32129]: pgmap v6055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:00:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10454.768558 secs 2024-07-26T14:00:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:13 smithi160 ceph-mon[36715]: pgmap v6056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:13 smithi183 ceph-mon[37273]: pgmap v6056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:13 smithi144 ceph-mon[32129]: pgmap v6056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:15 smithi160 ceph-mon[36715]: pgmap v6057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:15 smithi183 ceph-mon[37273]: pgmap v6057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:15 smithi144 ceph-mon[32129]: pgmap v6057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10459.768666 secs 2024-07-26T14:00:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10459.768666 secs 2024-07-26T14:00:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10459.768666 secs 2024-07-26T14:00:17.330 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:17 smithi160 ceph-mon[36715]: pgmap v6058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:17 smithi183 ceph-mon[37273]: pgmap v6058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:17 smithi144 ceph-mon[32129]: pgmap v6058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:19 smithi183 ceph-mon[37273]: pgmap v6059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:19 smithi144 ceph-mon[32129]: pgmap v6059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:19 smithi160 ceph-mon[36715]: pgmap v6059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:21.119+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:00:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:21.120+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:00:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:21.120+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:00:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:21.120+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:00:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:21.120+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:21 smithi183 ceph-mon[37273]: pgmap v6060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10464.768808 secs 2024-07-26T14:00:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:21 smithi144 ceph-mon[32129]: pgmap v6060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10464.768808 secs 2024-07-26T14:00:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:21 smithi160 ceph-mon[36715]: pgmap v6060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10464.768808 secs 2024-07-26T14:00:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:23 smithi183 ceph-mon[37273]: pgmap v6061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:23 smithi144 ceph-mon[32129]: pgmap v6061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:23 smithi160 ceph-mon[36715]: pgmap v6061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:25 smithi183 ceph-mon[37273]: pgmap v6062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:25 smithi144 ceph-mon[32129]: pgmap v6062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:25 smithi160 ceph-mon[36715]: pgmap v6062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10469.768938 secs 2024-07-26T14:00:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10469.768938 secs 2024-07-26T14:00:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10469.768938 secs 2024-07-26T14:00:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:27 smithi183 ceph-mon[37273]: pgmap v6063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:27 smithi144 ceph-mon[32129]: pgmap v6063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:27.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:27 smithi160 ceph-mon[36715]: pgmap v6063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:29 smithi183 ceph-mon[37273]: pgmap v6064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:29 smithi144 ceph-mon[32129]: pgmap v6064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:29 smithi160 ceph-mon[36715]: pgmap v6064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:31 smithi183 ceph-mon[37273]: pgmap v6065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10474.769058 secs 2024-07-26T14:00:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:31 smithi144 ceph-mon[32129]: pgmap v6065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10474.769058 secs 2024-07-26T14:00:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:31 smithi160 ceph-mon[36715]: pgmap v6065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10474.769058 secs 2024-07-26T14:00:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:33 smithi183 ceph-mon[37273]: pgmap v6066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:33 smithi144 ceph-mon[32129]: pgmap v6066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:33 smithi160 ceph-mon[36715]: pgmap v6066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:35 smithi183 ceph-mon[37273]: pgmap v6067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:35 smithi144 ceph-mon[32129]: pgmap v6067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:35 smithi160 ceph-mon[36715]: pgmap v6067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:36.135+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:00:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:36.136+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:00:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:36.136+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:00:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:36.136+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:00:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:36.136+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10479.769165 secs 2024-07-26T14:00:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10479.769165 secs 2024-07-26T14:00:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10479.769165 secs 2024-07-26T14:00:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:37 smithi183 ceph-mon[37273]: pgmap v6068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:37 smithi144 ceph-mon[32129]: pgmap v6068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:37 smithi160 ceph-mon[36715]: pgmap v6068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:39 smithi144 ceph-mon[32129]: pgmap v6069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:39 smithi160 ceph-mon[36715]: pgmap v6069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:39 smithi183 ceph-mon[37273]: pgmap v6069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:41 smithi144 ceph-mon[32129]: pgmap v6070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10484.769284 secs 2024-07-26T14:00:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:41 smithi160 ceph-mon[36715]: pgmap v6070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10484.769284 secs 2024-07-26T14:00:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:41 smithi183 ceph-mon[37273]: pgmap v6070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10484.769284 secs 2024-07-26T14:00:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:43 smithi144 ceph-mon[32129]: pgmap v6071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:43 smithi160 ceph-mon[36715]: pgmap v6071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:43 smithi183 ceph-mon[37273]: pgmap v6071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:45 smithi144 ceph-mon[32129]: pgmap v6072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:45 smithi160 ceph-mon[36715]: pgmap v6072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:45 smithi183 ceph-mon[37273]: pgmap v6072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:46.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10489.769407 secs 2024-07-26T14:00:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10489.769407 secs 2024-07-26T14:00:46.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10489.769407 secs 2024-07-26T14:00:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:47 smithi144 ceph-mon[32129]: pgmap v6073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:47 smithi160 ceph-mon[36715]: pgmap v6073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:47 smithi183 ceph-mon[37273]: pgmap v6073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:49 smithi144 ceph-mon[32129]: pgmap v6074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:49 smithi160 ceph-mon[36715]: pgmap v6074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:49 smithi183 ceph-mon[37273]: pgmap v6074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:51.146+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:00:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:51.147+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:00:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:51.147+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:00:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:51.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:00:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:00:51.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:51 smithi144 ceph-mon[32129]: pgmap v6075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10494.769528 secs 2024-07-26T14:00:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:51 smithi160 ceph-mon[36715]: pgmap v6075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10494.769528 secs 2024-07-26T14:00:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:51 smithi183 ceph-mon[37273]: pgmap v6075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10494.769528 secs 2024-07-26T14:00:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:52 smithi144 ceph-mon[32129]: pgmap v6076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:52 smithi160 ceph-mon[36715]: pgmap v6076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:52 smithi183 ceph-mon[37273]: pgmap v6076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:00:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:54 smithi144 ceph-mon[32129]: pgmap v6077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:54 smithi160 ceph-mon[36715]: pgmap v6077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:54 smithi183 ceph-mon[37273]: pgmap v6077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:57.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10499.769644 secs 2024-07-26T14:00:57.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:56 smithi183 ceph-mon[37273]: pgmap v6078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10499.769644 secs 2024-07-26T14:00:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:56 smithi144 ceph-mon[32129]: pgmap v6078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10499.769644 secs 2024-07-26T14:00:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:56 smithi160 ceph-mon[36715]: pgmap v6078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:00:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:00:58 smithi144 ceph-mon[32129]: pgmap v6079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:00:58 smithi160 ceph-mon[36715]: pgmap v6079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:00:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:00:58 smithi183 ceph-mon[37273]: pgmap v6079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:00 smithi144 ceph-mon[32129]: pgmap v6080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:01.247 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:00 smithi160 ceph-mon[36715]: pgmap v6080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:01.247 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:00 smithi183 ceph-mon[37273]: pgmap v6080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10504.769715 secs 2024-07-26T14:01:02.275 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10504.769715 secs 2024-07-26T14:01:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10504.769715 secs 2024-07-26T14:01:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:02 smithi144 ceph-mon[32129]: pgmap v6081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:03.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:02 smithi160 ceph-mon[36715]: pgmap v6081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:02 smithi183 ceph-mon[37273]: pgmap v6081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:04 smithi144 ceph-mon[32129]: pgmap v6082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:04 smithi160 ceph-mon[36715]: pgmap v6082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:04 smithi183 ceph-mon[37273]: pgmap v6082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:06.158+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:01:06.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:06.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:01:06.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:06.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:01:06.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:06.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:01:06.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:06.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10509.769843 secs 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: pgmap v6083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:01:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:01:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:07.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10509.769843 secs 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: pgmap v6083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10509.769843 secs 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: pgmap v6083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:07.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:01:08.603 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:08 smithi144 ceph-mon[32129]: pgmap v6084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:01:08.603 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:08 smithi144 ceph-mon[32129]: pgmap v6085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:08 smithi160 ceph-mon[36715]: pgmap v6084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:01:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:08 smithi160 ceph-mon[36715]: pgmap v6085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:08 smithi183 ceph-mon[37273]: pgmap v6084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:01:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:08 smithi183 ceph-mon[37273]: pgmap v6085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:10 smithi144 ceph-mon[32129]: pgmap v6086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:10 smithi160 ceph-mon[36715]: pgmap v6086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:10 smithi183 ceph-mon[37273]: pgmap v6086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10514.769993 secs 2024-07-26T14:01:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10514.769993 secs 2024-07-26T14:01:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10514.769993 secs 2024-07-26T14:01:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:12 smithi144 ceph-mon[32129]: pgmap v6087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:12 smithi160 ceph-mon[36715]: pgmap v6087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:12 smithi183 ceph-mon[37273]: pgmap v6087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:01:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:14 smithi160 ceph-mon[36715]: pgmap v6088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:14 smithi144 ceph-mon[32129]: pgmap v6088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:14 smithi183 ceph-mon[37273]: pgmap v6088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:16 smithi144 ceph-mon[32129]: pgmap v6089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T14:01:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10519.770134 secs 2024-07-26T14:01:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:16 smithi160 ceph-mon[36715]: pgmap v6089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T14:01:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10519.770134 secs 2024-07-26T14:01:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:16 smithi183 ceph-mon[37273]: pgmap v6089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T14:01:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10519.770134 secs 2024-07-26T14:01:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:17 smithi144 ceph-mon[32129]: pgmap v6090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:01:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:17 smithi160 ceph-mon[36715]: pgmap v6090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:01:17.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:17 smithi183 ceph-mon[37273]: pgmap v6090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:01:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:19 smithi144 ceph-mon[32129]: pgmap v6091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:19 smithi160 ceph-mon[36715]: pgmap v6091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:19 smithi183 ceph-mon[37273]: pgmap v6091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:21.175+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:01:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:21.176+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:01:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:21.176+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:01:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:21.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:01:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:21.176+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:21 smithi144 ceph-mon[32129]: pgmap v6092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10524.770256 secs 2024-07-26T14:01:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:21 smithi160 ceph-mon[36715]: pgmap v6092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10524.770256 secs 2024-07-26T14:01:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:21 smithi183 ceph-mon[37273]: pgmap v6092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10524.770256 secs 2024-07-26T14:01:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:23 smithi144 ceph-mon[32129]: pgmap v6093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:23 smithi160 ceph-mon[36715]: pgmap v6093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:23 smithi183 ceph-mon[37273]: pgmap v6093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:25 smithi144 ceph-mon[32129]: pgmap v6094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:25 smithi160 ceph-mon[36715]: pgmap v6094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:25 smithi183 ceph-mon[37273]: pgmap v6094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10529.770393 secs 2024-07-26T14:01:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10529.770393 secs 2024-07-26T14:01:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10529.770393 secs 2024-07-26T14:01:27.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:27 smithi183 ceph-mon[37273]: pgmap v6095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:27 smithi160 ceph-mon[36715]: pgmap v6095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:27 smithi144 ceph-mon[32129]: pgmap v6095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:29 smithi160 ceph-mon[36715]: pgmap v6096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:29 smithi183 ceph-mon[37273]: pgmap v6096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:29 smithi144 ceph-mon[32129]: pgmap v6096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:31 smithi160 ceph-mon[36715]: pgmap v6097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10534.770512 secs 2024-07-26T14:01:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:31 smithi183 ceph-mon[37273]: pgmap v6097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10534.770512 secs 2024-07-26T14:01:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:31 smithi144 ceph-mon[32129]: pgmap v6097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10534.770512 secs 2024-07-26T14:01:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:33 smithi160 ceph-mon[36715]: pgmap v6098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:33 smithi183 ceph-mon[37273]: pgmap v6098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:33 smithi144 ceph-mon[32129]: pgmap v6098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:35 smithi160 ceph-mon[36715]: pgmap v6099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:35 smithi183 ceph-mon[37273]: pgmap v6099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:35 smithi144 ceph-mon[32129]: pgmap v6099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:36.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:36.178+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:01:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:36.179+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:01:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:36.179+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:01:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:36.179+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:01:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:36.179+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10539.770650 secs 2024-07-26T14:01:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10539.770650 secs 2024-07-26T14:01:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10539.770650 secs 2024-07-26T14:01:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:37 smithi160 ceph-mon[36715]: pgmap v6100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:37.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:37 smithi144 ceph-mon[32129]: pgmap v6100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:37 smithi183 ceph-mon[37273]: pgmap v6100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:39 smithi160 ceph-mon[36715]: pgmap v6101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:39 smithi183 ceph-mon[37273]: pgmap v6101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:39 smithi144 ceph-mon[32129]: pgmap v6101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:41 smithi160 ceph-mon[36715]: pgmap v6102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10544.770781 secs 2024-07-26T14:01:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:41 smithi183 ceph-mon[37273]: pgmap v6102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10544.770781 secs 2024-07-26T14:01:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:41 smithi144 ceph-mon[32129]: pgmap v6102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10544.770781 secs 2024-07-26T14:01:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:43 smithi160 ceph-mon[36715]: pgmap v6103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:43 smithi183 ceph-mon[37273]: pgmap v6103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:43 smithi144 ceph-mon[32129]: pgmap v6103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:45 smithi160 ceph-mon[36715]: pgmap v6104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:45 smithi183 ceph-mon[37273]: pgmap v6104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:45 smithi144 ceph-mon[32129]: pgmap v6104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10549.770920 secs 2024-07-26T14:01:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10549.770920 secs 2024-07-26T14:01:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10549.770920 secs 2024-07-26T14:01:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:47 smithi160 ceph-mon[36715]: pgmap v6105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:47 smithi183 ceph-mon[37273]: pgmap v6105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:47 smithi144 ceph-mon[32129]: pgmap v6105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:49 smithi183 ceph-mon[37273]: pgmap v6106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:49 smithi144 ceph-mon[32129]: pgmap v6106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:49 smithi160 ceph-mon[36715]: pgmap v6106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:51.196+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:01:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:51.197+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:01:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:51.197+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:01:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:51.197+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:01:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:01:51.197+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:51 smithi183 ceph-mon[37273]: pgmap v6107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10554.770998 secs 2024-07-26T14:01:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:51 smithi144 ceph-mon[32129]: pgmap v6107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10554.770998 secs 2024-07-26T14:01:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:51 smithi160 ceph-mon[36715]: pgmap v6107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10554.770998 secs 2024-07-26T14:01:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:53 smithi183 ceph-mon[37273]: pgmap v6108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:53 smithi144 ceph-mon[32129]: pgmap v6108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:53 smithi160 ceph-mon[36715]: pgmap v6108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:01:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:55 smithi183 ceph-mon[37273]: pgmap v6109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:55 smithi144 ceph-mon[32129]: pgmap v6109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:55 smithi160 ceph-mon[36715]: pgmap v6109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10559.771119 secs 2024-07-26T14:01:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10559.771119 secs 2024-07-26T14:01:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10559.771119 secs 2024-07-26T14:01:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:57 smithi183 ceph-mon[37273]: pgmap v6110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:57 smithi144 ceph-mon[32129]: pgmap v6110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:57 smithi160 ceph-mon[36715]: pgmap v6110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:01:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:01:59 smithi183 ceph-mon[37273]: pgmap v6111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:01:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:01:59 smithi144 ceph-mon[32129]: pgmap v6111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:01:59 smithi160 ceph-mon[36715]: pgmap v6111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:01 smithi183 ceph-mon[37273]: pgmap v6112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10564.771254 secs 2024-07-26T14:02:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:01 smithi144 ceph-mon[32129]: pgmap v6112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10564.771254 secs 2024-07-26T14:02:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:01 smithi160 ceph-mon[36715]: pgmap v6112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10564.771254 secs 2024-07-26T14:02:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:03 smithi183 ceph-mon[37273]: pgmap v6113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:03 smithi144 ceph-mon[32129]: pgmap v6113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:03 smithi160 ceph-mon[36715]: pgmap v6113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:05 smithi183 ceph-mon[37273]: pgmap v6114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:05 smithi144 ceph-mon[32129]: pgmap v6114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:05 smithi160 ceph-mon[36715]: pgmap v6114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:06.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:06.213+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:06.214+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:06.214+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:06.214+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:06.214+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10569.771373 secs 2024-07-26T14:02:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10569.771373 secs 2024-07-26T14:02:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10569.771373 secs 2024-07-26T14:02:07.808 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: pgmap v6115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:02:07.809 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:07 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:02:07.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: pgmap v6115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:07.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:07.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:07.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:02:07.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:07 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:02:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: pgmap v6115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:02:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:07 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:02:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:09 smithi144 ceph-mon[32129]: pgmap v6116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:09.748 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:09 smithi183 ceph-mon[37273]: pgmap v6116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:09 smithi160 ceph-mon[36715]: pgmap v6116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: pgmap v6117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: pgmap v6118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: pgmap v6119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10574.771504 secs 2024-07-26T14:02:11.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: pgmap v6117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:02:11.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: pgmap v6118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: pgmap v6119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10574.771504 secs 2024-07-26T14:02:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: pgmap v6117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: pgmap v6118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: pgmap v6119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:02:11.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10574.771504 secs 2024-07-26T14:02:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:13 smithi183 ceph-mon[37273]: pgmap v6120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T14:02:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:13 smithi144 ceph-mon[32129]: pgmap v6120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T14:02:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:13 smithi160 ceph-mon[36715]: pgmap v6120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T14:02:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:15 smithi183 ceph-mon[37273]: pgmap v6121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T14:02:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:15 smithi144 ceph-mon[32129]: pgmap v6121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T14:02:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:15 smithi160 ceph-mon[36715]: pgmap v6121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T14:02:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10579.771647 secs 2024-07-26T14:02:16.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10579.771647 secs 2024-07-26T14:02:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10579.771647 secs 2024-07-26T14:02:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:17 smithi183 ceph-mon[37273]: pgmap v6122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:17 smithi144 ceph-mon[32129]: pgmap v6122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:17 smithi160 ceph-mon[36715]: pgmap v6122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:19.911 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:19 smithi183 ceph-mon[37273]: pgmap v6123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:02:19.933 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:19 smithi144 ceph-mon[32129]: pgmap v6123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:02:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:19 smithi160 ceph-mon[36715]: pgmap v6123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:02:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:21.229+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:02:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:21.230+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:02:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:21.230+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:02:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:21.230+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:02:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:21.230+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:21 smithi144 ceph-mon[32129]: pgmap v6124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:02:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10584.771744 secs 2024-07-26T14:02:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:21 smithi160 ceph-mon[36715]: pgmap v6124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:02:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10584.771744 secs 2024-07-26T14:02:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:21 smithi183 ceph-mon[37273]: pgmap v6124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:02:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10584.771744 secs 2024-07-26T14:02:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:23 smithi144 ceph-mon[32129]: pgmap v6125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:23 smithi160 ceph-mon[36715]: pgmap v6125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:23 smithi183 ceph-mon[37273]: pgmap v6125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:25 smithi144 ceph-mon[32129]: pgmap v6126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:25 smithi160 ceph-mon[36715]: pgmap v6126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:25 smithi183 ceph-mon[37273]: pgmap v6126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10589.771828 secs 2024-07-26T14:02:27.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10589.771828 secs 2024-07-26T14:02:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10589.771828 secs 2024-07-26T14:02:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:27 smithi144 ceph-mon[32129]: pgmap v6127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:28.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:27 smithi183 ceph-mon[37273]: pgmap v6127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:27 smithi160 ceph-mon[36715]: pgmap v6127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:29 smithi144 ceph-mon[32129]: pgmap v6128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:29 smithi160 ceph-mon[36715]: pgmap v6128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:29 smithi183 ceph-mon[37273]: pgmap v6128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:31 smithi144 ceph-mon[32129]: pgmap v6129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10594.771953 secs 2024-07-26T14:02:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:31 smithi160 ceph-mon[36715]: pgmap v6129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10594.771953 secs 2024-07-26T14:02:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:31 smithi183 ceph-mon[37273]: pgmap v6129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:32.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10594.771953 secs 2024-07-26T14:02:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:33 smithi144 ceph-mon[32129]: pgmap v6130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:33 smithi160 ceph-mon[36715]: pgmap v6130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:33 smithi183 ceph-mon[37273]: pgmap v6130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:35 smithi144 ceph-mon[32129]: pgmap v6131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:35 smithi160 ceph-mon[36715]: pgmap v6131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:35 smithi183 ceph-mon[37273]: pgmap v6131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:36.245+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:02:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:36.246+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:02:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:36.246+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:02:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:36.246+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:02:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:36.246+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10599.772096 secs 2024-07-26T14:02:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10599.772096 secs 2024-07-26T14:02:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10599.772096 secs 2024-07-26T14:02:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:37 smithi144 ceph-mon[32129]: pgmap v6132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:37 smithi160 ceph-mon[36715]: pgmap v6132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:37 smithi183 ceph-mon[37273]: pgmap v6132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:39 smithi144 ceph-mon[32129]: pgmap v6133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:39 smithi160 ceph-mon[36715]: pgmap v6133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:39 smithi183 ceph-mon[37273]: pgmap v6133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:41 smithi144 ceph-mon[32129]: pgmap v6134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10604.772217 secs 2024-07-26T14:02:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:41 smithi160 ceph-mon[36715]: pgmap v6134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10604.772217 secs 2024-07-26T14:02:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:41 smithi183 ceph-mon[37273]: pgmap v6134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10604.772217 secs 2024-07-26T14:02:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:43 smithi160 ceph-mon[36715]: pgmap v6135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:43 smithi183 ceph-mon[37273]: pgmap v6135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:43 smithi144 ceph-mon[32129]: pgmap v6135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:45 smithi160 ceph-mon[36715]: pgmap v6136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:45 smithi183 ceph-mon[37273]: pgmap v6136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:45 smithi144 ceph-mon[32129]: pgmap v6136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10609.772368 secs 2024-07-26T14:02:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10609.772368 secs 2024-07-26T14:02:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10609.772368 secs 2024-07-26T14:02:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:47.974+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:02:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:47.974+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:02:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:47.975+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:02:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:47.975+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T14:02:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:47 smithi160 ceph-mon[36715]: pgmap v6137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:47 smithi183 ceph-mon[37273]: pgmap v6137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:47 smithi144 ceph-mon[32129]: pgmap v6137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:49 smithi160 ceph-mon[36715]: pgmap v6138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:49 smithi183 ceph-mon[37273]: pgmap v6138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:49 smithi144 ceph-mon[32129]: pgmap v6138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:51.253+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:02:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:51.254+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:02:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:51.254+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:02:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:51.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:02:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:02:51.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:51 smithi160 ceph-mon[36715]: pgmap v6139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10614.772465 secs 2024-07-26T14:02:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:51 smithi183 ceph-mon[37273]: pgmap v6139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10614.772465 secs 2024-07-26T14:02:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:51 smithi144 ceph-mon[32129]: pgmap v6139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10614.772465 secs 2024-07-26T14:02:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:53 smithi160 ceph-mon[36715]: pgmap v6140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:53 smithi183 ceph-mon[37273]: pgmap v6140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:53 smithi144 ceph-mon[32129]: pgmap v6140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:02:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:55 smithi160 ceph-mon[36715]: pgmap v6141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:55 smithi183 ceph-mon[37273]: pgmap v6141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:55 smithi144 ceph-mon[32129]: pgmap v6141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:02:57.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10619.772627 secs 2024-07-26T14:02:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10619.772627 secs 2024-07-26T14:02:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10619.772627 secs 2024-07-26T14:02:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:57 smithi160 ceph-mon[36715]: pgmap v6142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:57 smithi183 ceph-mon[37273]: pgmap v6142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:02:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:57 smithi144 ceph-mon[32129]: pgmap v6142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:02:59 smithi160 ceph-mon[36715]: pgmap v6143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:02:59 smithi183 ceph-mon[37273]: pgmap v6143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:02:59 smithi144 ceph-mon[32129]: pgmap v6143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:01 smithi160 ceph-mon[36715]: pgmap v6144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10624.772722 secs 2024-07-26T14:03:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:01 smithi183 ceph-mon[37273]: pgmap v6144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10624.772722 secs 2024-07-26T14:03:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:01 smithi144 ceph-mon[32129]: pgmap v6144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10624.772722 secs 2024-07-26T14:03:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:03 smithi160 ceph-mon[36715]: pgmap v6145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:03 smithi183 ceph-mon[37273]: pgmap v6145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:03 smithi144 ceph-mon[32129]: pgmap v6145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:05 smithi160 ceph-mon[36715]: pgmap v6146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:05 smithi183 ceph-mon[37273]: pgmap v6146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:05 smithi144 ceph-mon[32129]: pgmap v6146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:06.258+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:03:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:06.259+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:03:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:06.259+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:03:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:06.259+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:03:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:06.259+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:07.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10629.772879 secs 2024-07-26T14:03:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10629.772879 secs 2024-07-26T14:03:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10629.772879 secs 2024-07-26T14:03:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:07 smithi183 ceph-mon[37273]: pgmap v6147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:07 smithi144 ceph-mon[32129]: pgmap v6147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:07 smithi160 ceph-mon[36715]: pgmap v6147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:09 smithi183 ceph-mon[37273]: pgmap v6148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:09 smithi144 ceph-mon[32129]: pgmap v6148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:09 smithi160 ceph-mon[36715]: pgmap v6148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:11.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:11 smithi183 ceph-mon[37273]: pgmap v6149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10634.772992 secs 2024-07-26T14:03:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:03:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:11 smithi144 ceph-mon[32129]: pgmap v6149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10634.772992 secs 2024-07-26T14:03:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:03:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:11 smithi160 ceph-mon[36715]: pgmap v6149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10634.772992 secs 2024-07-26T14:03:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:03:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:12 smithi183 ceph-mon[37273]: pgmap v6150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:03:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:12 smithi183 ceph-mon[37273]: pgmap v6151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:03:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:12 smithi144 ceph-mon[32129]: pgmap v6150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:03:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:12 smithi144 ceph-mon[32129]: pgmap v6151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:03:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:12 smithi160 ceph-mon[36715]: pgmap v6150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:03:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:12 smithi160 ceph-mon[36715]: pgmap v6151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:03:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:14 smithi183 ceph-mon[37273]: pgmap v6152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:03:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:14 smithi144 ceph-mon[32129]: pgmap v6152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:03:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:14 smithi160 ceph-mon[36715]: pgmap v6152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:03:17.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:16 smithi160 ceph-mon[36715]: pgmap v6153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:03:17.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10639.773065 secs 2024-07-26T14:03:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:16 smithi183 ceph-mon[37273]: pgmap v6153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:03:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10639.773065 secs 2024-07-26T14:03:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:16 smithi144 ceph-mon[32129]: pgmap v6153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:03:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10639.773065 secs 2024-07-26T14:03:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:18 smithi183 ceph-mon[37273]: pgmap v6154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:18 smithi144 ceph-mon[32129]: pgmap v6154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:18 smithi160 ceph-mon[36715]: pgmap v6154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:20 smithi183 ceph-mon[37273]: pgmap v6155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:20 smithi144 ceph-mon[32129]: pgmap v6155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:21.275+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:03:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:21.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:03:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:21.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:03:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:21.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:03:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:21.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:20 smithi160 ceph-mon[36715]: pgmap v6155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10644.773128 secs 2024-07-26T14:03:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10644.773128 secs 2024-07-26T14:03:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10644.773128 secs 2024-07-26T14:03:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:22 smithi144 ceph-mon[32129]: pgmap v6156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:22 smithi160 ceph-mon[36715]: pgmap v6156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:22 smithi183 ceph-mon[37273]: pgmap v6156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:24 smithi144 ceph-mon[32129]: pgmap v6157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:24 smithi160 ceph-mon[36715]: pgmap v6157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:24 smithi183 ceph-mon[37273]: pgmap v6157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10649.773272 secs 2024-07-26T14:03:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10649.773272 secs 2024-07-26T14:03:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10649.773272 secs 2024-07-26T14:03:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:27 smithi183 ceph-mon[37273]: pgmap v6158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:27 smithi144 ceph-mon[32129]: pgmap v6158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:27 smithi160 ceph-mon[36715]: pgmap v6158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:29 smithi144 ceph-mon[32129]: pgmap v6159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:29 smithi160 ceph-mon[36715]: pgmap v6159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:29 smithi183 ceph-mon[37273]: pgmap v6159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:31 smithi144 ceph-mon[32129]: pgmap v6160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:03:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10654.773382 secs 2024-07-26T14:03:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:31 smithi160 ceph-mon[36715]: pgmap v6160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:03:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10654.773382 secs 2024-07-26T14:03:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:31 smithi183 ceph-mon[37273]: pgmap v6160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:03:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10654.773382 secs 2024-07-26T14:03:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:33 smithi144 ceph-mon[32129]: pgmap v6161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:33 smithi160 ceph-mon[36715]: pgmap v6161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:33 smithi183 ceph-mon[37273]: pgmap v6161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:35 smithi144 ceph-mon[32129]: pgmap v6162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:35 smithi160 ceph-mon[36715]: pgmap v6162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:35 smithi183 ceph-mon[37273]: pgmap v6162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:36.291+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:03:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:36.292+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:03:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:36.292+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:03:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:36.292+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:03:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:36.292+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10659.773506 secs 2024-07-26T14:03:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10659.773506 secs 2024-07-26T14:03:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10659.773506 secs 2024-07-26T14:03:37.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:37 smithi144 ceph-mon[32129]: pgmap v6163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:37 smithi160 ceph-mon[36715]: pgmap v6163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:37 smithi183 ceph-mon[37273]: pgmap v6163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:03:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:39 smithi144 ceph-mon[32129]: pgmap v6164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:39 smithi160 ceph-mon[36715]: pgmap v6164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:39 smithi183 ceph-mon[37273]: pgmap v6164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:03:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:41 smithi144 ceph-mon[32129]: pgmap v6165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:03:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10664.773614 secs 2024-07-26T14:03:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:41 smithi160 ceph-mon[36715]: pgmap v6165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:03:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10664.773614 secs 2024-07-26T14:03:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:41 smithi183 ceph-mon[37273]: pgmap v6165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:03:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10664.773614 secs 2024-07-26T14:03:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:43 smithi144 ceph-mon[32129]: pgmap v6166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:43 smithi160 ceph-mon[36715]: pgmap v6166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:43 smithi183 ceph-mon[37273]: pgmap v6166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:45 smithi144 ceph-mon[32129]: pgmap v6167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:45 smithi160 ceph-mon[36715]: pgmap v6167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:45 smithi183 ceph-mon[37273]: pgmap v6167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10669.773771 secs 2024-07-26T14:03:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10669.773771 secs 2024-07-26T14:03:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10669.773771 secs 2024-07-26T14:03:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:47 smithi144 ceph-mon[32129]: pgmap v6168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:47 smithi160 ceph-mon[36715]: pgmap v6168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:47 smithi183 ceph-mon[37273]: pgmap v6168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:49 smithi144 ceph-mon[32129]: pgmap v6169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:49 smithi160 ceph-mon[36715]: pgmap v6169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:49 smithi183 ceph-mon[37273]: pgmap v6169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:50 smithi160 ceph-mon[36715]: pgmap v6170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:50 smithi183 ceph-mon[37273]: pgmap v6170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:50 smithi144 ceph-mon[32129]: pgmap v6170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:51.303+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:03:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:51.304+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:03:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:51.304+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:03:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:51.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:03:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:03:51.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10674.773913 secs 2024-07-26T14:03:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10674.773913 secs 2024-07-26T14:03:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10674.773913 secs 2024-07-26T14:03:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:52 smithi160 ceph-mon[36715]: pgmap v6171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:52 smithi183 ceph-mon[37273]: pgmap v6171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:52 smithi144 ceph-mon[32129]: pgmap v6171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:03:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:54 smithi183 ceph-mon[37273]: pgmap v6172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:54 smithi144 ceph-mon[32129]: pgmap v6172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:54 smithi160 ceph-mon[36715]: pgmap v6172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:56 smithi183 ceph-mon[37273]: pgmap v6173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10679.774034 secs 2024-07-26T14:03:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:56 smithi144 ceph-mon[32129]: pgmap v6173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10679.774034 secs 2024-07-26T14:03:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:56 smithi160 ceph-mon[36715]: pgmap v6173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:03:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10679.774034 secs 2024-07-26T14:03:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:03:58 smithi183 ceph-mon[37273]: pgmap v6174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:03:58 smithi144 ceph-mon[32129]: pgmap v6174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:03:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:03:58 smithi160 ceph-mon[36715]: pgmap v6174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:00 smithi183 ceph-mon[37273]: pgmap v6175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:00 smithi144 ceph-mon[32129]: pgmap v6175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:00 smithi160 ceph-mon[36715]: pgmap v6175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10684.774158 secs 2024-07-26T14:04:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10684.774158 secs 2024-07-26T14:04:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10684.774158 secs 2024-07-26T14:04:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:02 smithi183 ceph-mon[37273]: pgmap v6176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:02 smithi144 ceph-mon[32129]: pgmap v6176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:02 smithi160 ceph-mon[36715]: pgmap v6176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:04 smithi183 ceph-mon[37273]: pgmap v6177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:04 smithi144 ceph-mon[32129]: pgmap v6177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:04 smithi160 ceph-mon[36715]: pgmap v6177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:06.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:04:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:06.321+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:04:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:06.321+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:04:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:06.321+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:04:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:06.321+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:06 smithi183 ceph-mon[37273]: pgmap v6178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10689.774280 secs 2024-07-26T14:04:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:06 smithi144 ceph-mon[32129]: pgmap v6178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10689.774280 secs 2024-07-26T14:04:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:06 smithi160 ceph-mon[36715]: pgmap v6178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10689.774280 secs 2024-07-26T14:04:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:08 smithi183 ceph-mon[37273]: pgmap v6179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:08 smithi144 ceph-mon[32129]: pgmap v6179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:08 smithi160 ceph-mon[36715]: pgmap v6179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:10 smithi183 ceph-mon[37273]: pgmap v6180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:10 smithi144 ceph-mon[32129]: pgmap v6180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:10 smithi160 ceph-mon[36715]: pgmap v6180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10694.774403 secs 2024-07-26T14:04:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10694.774403 secs 2024-07-26T14:04:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10694.774403 secs 2024-07-26T14:04:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:12 smithi144 ceph-mon[32129]: pgmap v6181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:12 smithi160 ceph-mon[36715]: pgmap v6181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:12 smithi183 ceph-mon[37273]: pgmap v6181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:04:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:14 smithi144 ceph-mon[32129]: pgmap v6182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:14 smithi144 ceph-mon[32129]: pgmap v6183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:14 smithi160 ceph-mon[36715]: pgmap v6182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:14 smithi160 ceph-mon[36715]: pgmap v6183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:14 smithi183 ceph-mon[37273]: pgmap v6182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:14 smithi183 ceph-mon[37273]: pgmap v6183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:16 smithi144 ceph-mon[32129]: pgmap v6184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10699.774482 secs 2024-07-26T14:04:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:16 smithi160 ceph-mon[36715]: pgmap v6184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10699.774482 secs 2024-07-26T14:04:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:16 smithi183 ceph-mon[37273]: pgmap v6184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10699.774482 secs 2024-07-26T14:04:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:18 smithi144 ceph-mon[32129]: pgmap v6185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:18 smithi160 ceph-mon[36715]: pgmap v6185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:18 smithi183 ceph-mon[37273]: pgmap v6185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:20 smithi144 ceph-mon[32129]: pgmap v6186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:20 smithi160 ceph-mon[36715]: pgmap v6186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:21.332+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:04:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:21.333+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:04:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:21.333+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:04:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:21.333+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:04:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:21.334+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:20 smithi183 ceph-mon[37273]: pgmap v6186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10704.774603 secs 2024-07-26T14:04:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10704.774603 secs 2024-07-26T14:04:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10704.774603 secs 2024-07-26T14:04:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:22 smithi144 ceph-mon[32129]: pgmap v6187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:22 smithi160 ceph-mon[36715]: pgmap v6187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:22 smithi183 ceph-mon[37273]: pgmap v6187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:04:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:24 smithi144 ceph-mon[32129]: pgmap v6188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:04:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:24 smithi160 ceph-mon[36715]: pgmap v6188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:04:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:24 smithi183 ceph-mon[37273]: pgmap v6188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:04:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:26 smithi144 ceph-mon[32129]: pgmap v6189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10709.774735 secs 2024-07-26T14:04:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:26 smithi160 ceph-mon[36715]: pgmap v6189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10709.774735 secs 2024-07-26T14:04:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:26 smithi183 ceph-mon[37273]: pgmap v6189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10709.774735 secs 2024-07-26T14:04:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:28 smithi144 ceph-mon[32129]: pgmap v6190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:28 smithi160 ceph-mon[36715]: pgmap v6190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:28 smithi183 ceph-mon[37273]: pgmap v6190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:30 smithi144 ceph-mon[32129]: pgmap v6191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:30 smithi160 ceph-mon[36715]: pgmap v6191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:30 smithi183 ceph-mon[37273]: pgmap v6191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10714.774836 secs 2024-07-26T14:04:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10714.774836 secs 2024-07-26T14:04:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10714.774836 secs 2024-07-26T14:04:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:32 smithi160 ceph-mon[36715]: pgmap v6192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:32 smithi183 ceph-mon[37273]: pgmap v6192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:32 smithi144 ceph-mon[32129]: pgmap v6192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:34 smithi160 ceph-mon[36715]: pgmap v6193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:34 smithi183 ceph-mon[37273]: pgmap v6193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:34 smithi144 ceph-mon[32129]: pgmap v6193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:36.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:04:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:36.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:04:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:36.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:04:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:36.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:04:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:36.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:37.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:36 smithi144 ceph-mon[32129]: pgmap v6194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:37.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10719.774975 secs 2024-07-26T14:04:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:36 smithi160 ceph-mon[36715]: pgmap v6194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10719.774975 secs 2024-07-26T14:04:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:36 smithi183 ceph-mon[37273]: pgmap v6194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10719.774975 secs 2024-07-26T14:04:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:38 smithi160 ceph-mon[36715]: pgmap v6195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:38 smithi183 ceph-mon[37273]: pgmap v6195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:38 smithi144 ceph-mon[32129]: pgmap v6195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:40 smithi160 ceph-mon[36715]: pgmap v6196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:40 smithi183 ceph-mon[37273]: pgmap v6196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:40 smithi144 ceph-mon[32129]: pgmap v6196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10724.775098 secs 2024-07-26T14:04:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10724.775098 secs 2024-07-26T14:04:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10724.775098 secs 2024-07-26T14:04:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:43 smithi160 ceph-mon[36715]: pgmap v6197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:43 smithi183 ceph-mon[37273]: pgmap v6197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:43 smithi144 ceph-mon[32129]: pgmap v6197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:45 smithi160 ceph-mon[36715]: pgmap v6198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:45 smithi183 ceph-mon[37273]: pgmap v6198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:45 smithi144 ceph-mon[32129]: pgmap v6198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:47 smithi160 ceph-mon[36715]: pgmap v6199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10729.775221 secs 2024-07-26T14:04:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:47 smithi183 ceph-mon[37273]: pgmap v6199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10729.775221 secs 2024-07-26T14:04:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:47 smithi144 ceph-mon[32129]: pgmap v6199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10729.775221 secs 2024-07-26T14:04:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:49 smithi160 ceph-mon[36715]: pgmap v6200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:49 smithi183 ceph-mon[37273]: pgmap v6200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:49 smithi144 ceph-mon[32129]: pgmap v6200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:51 smithi160 ceph-mon[36715]: pgmap v6201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:51.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:04:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:51.367+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:04:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:51.367+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:04:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:51.367+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:04:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:04:51.367+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:51 smithi183 ceph-mon[37273]: pgmap v6201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:51 smithi144 ceph-mon[32129]: pgmap v6201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10734.775324 secs 2024-07-26T14:04:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10734.775324 secs 2024-07-26T14:04:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10734.775324 secs 2024-07-26T14:04:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:53 smithi160 ceph-mon[36715]: pgmap v6202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:53 smithi183 ceph-mon[37273]: pgmap v6202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:53 smithi144 ceph-mon[32129]: pgmap v6202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:04:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:55 smithi160 ceph-mon[36715]: pgmap v6203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:55 smithi183 ceph-mon[37273]: pgmap v6203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:55 smithi144 ceph-mon[32129]: pgmap v6203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:57 smithi160 ceph-mon[36715]: pgmap v6204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10739.775425 secs 2024-07-26T14:04:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:57 smithi183 ceph-mon[37273]: pgmap v6204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10739.775425 secs 2024-07-26T14:04:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:57 smithi144 ceph-mon[32129]: pgmap v6204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:04:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10739.775425 secs 2024-07-26T14:04:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:04:59 smithi183 ceph-mon[37273]: pgmap v6205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:04:59 smithi144 ceph-mon[32129]: pgmap v6205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:04:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:04:59 smithi160 ceph-mon[36715]: pgmap v6205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:01 smithi183 ceph-mon[37273]: pgmap v6206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:01 smithi144 ceph-mon[32129]: pgmap v6206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:01 smithi160 ceph-mon[36715]: pgmap v6206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10744.775510 secs 2024-07-26T14:05:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10744.775510 secs 2024-07-26T14:05:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10744.775510 secs 2024-07-26T14:05:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:03 smithi183 ceph-mon[37273]: pgmap v6207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:03 smithi144 ceph-mon[32129]: pgmap v6207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:03 smithi160 ceph-mon[36715]: pgmap v6207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:05 smithi183 ceph-mon[37273]: pgmap v6208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:05 smithi144 ceph-mon[32129]: pgmap v6208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:05 smithi160 ceph-mon[36715]: pgmap v6208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:06.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:05:06.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:06.383+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:05:06.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:06.383+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:05:06.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:06.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:05:06.385 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:06.384+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:07 smithi183 ceph-mon[37273]: pgmap v6209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10749.775579 secs 2024-07-26T14:05:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:07 smithi144 ceph-mon[32129]: pgmap v6209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10749.775579 secs 2024-07-26T14:05:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:07 smithi160 ceph-mon[36715]: pgmap v6209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10749.775579 secs 2024-07-26T14:05:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:09 smithi183 ceph-mon[37273]: pgmap v6210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:09 smithi144 ceph-mon[32129]: pgmap v6210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:09 smithi160 ceph-mon[36715]: pgmap v6210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:11 smithi144 ceph-mon[32129]: pgmap v6211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:11 smithi183 ceph-mon[37273]: pgmap v6211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:11 smithi160 ceph-mon[36715]: pgmap v6211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10754.775707 secs 2024-07-26T14:05:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10754.775707 secs 2024-07-26T14:05:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10754.775707 secs 2024-07-26T14:05:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:13 smithi183 ceph-mon[37273]: pgmap v6212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:13 smithi144 ceph-mon[32129]: pgmap v6212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:13 smithi160 ceph-mon[36715]: pgmap v6212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:14.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:14.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:15 smithi183 ceph-mon[37273]: pgmap v6213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:15 smithi144 ceph-mon[32129]: pgmap v6213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:15 smithi160 ceph-mon[36715]: pgmap v6213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:16 smithi160 ceph-mon[36715]: pgmap v6214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10759.775772 secs 2024-07-26T14:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:16 smithi183 ceph-mon[37273]: pgmap v6214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10759.775772 secs 2024-07-26T14:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:16 smithi144 ceph-mon[32129]: pgmap v6214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10759.775772 secs 2024-07-26T14:05:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:19 smithi144 ceph-mon[32129]: pgmap v6215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:19 smithi160 ceph-mon[36715]: pgmap v6215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:19 smithi183 ceph-mon[37273]: pgmap v6215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:05:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:20 smithi144 ceph-mon[32129]: pgmap v6216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:05:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:20 smithi144 ceph-mon[32129]: pgmap v6217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:05:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:20 smithi160 ceph-mon[36715]: pgmap v6216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:05:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:20 smithi160 ceph-mon[36715]: pgmap v6217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:05:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:20 smithi183 ceph-mon[37273]: pgmap v6216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:05:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:20 smithi183 ceph-mon[37273]: pgmap v6217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:05:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:21.385+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:05:21.387 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:21.386+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:05:21.387 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:21.386+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:05:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:21.386+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:05:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:21.386+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10764.775860 secs 2024-07-26T14:05:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10764.775860 secs 2024-07-26T14:05:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10764.775860 secs 2024-07-26T14:05:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:22 smithi144 ceph-mon[32129]: pgmap v6218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:05:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:22 smithi160 ceph-mon[36715]: pgmap v6218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:05:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:22 smithi183 ceph-mon[37273]: pgmap v6218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:05:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:24 smithi144 ceph-mon[32129]: pgmap v6219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:24 smithi160 ceph-mon[36715]: pgmap v6219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:24 smithi183 ceph-mon[37273]: pgmap v6219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:26 smithi144 ceph-mon[32129]: pgmap v6220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10769.776014 secs 2024-07-26T14:05:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:26 smithi160 ceph-mon[36715]: pgmap v6220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10769.776014 secs 2024-07-26T14:05:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:26 smithi183 ceph-mon[37273]: pgmap v6220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10769.776014 secs 2024-07-26T14:05:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:28 smithi160 ceph-mon[36715]: pgmap v6221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:05:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:28 smithi183 ceph-mon[37273]: pgmap v6221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:05:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:28 smithi144 ceph-mon[32129]: pgmap v6221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:05:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:30 smithi160 ceph-mon[36715]: pgmap v6222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:05:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:30 smithi183 ceph-mon[37273]: pgmap v6222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:05:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:30 smithi144 ceph-mon[32129]: pgmap v6222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:05:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10774.776141 secs 2024-07-26T14:05:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10774.776141 secs 2024-07-26T14:05:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10774.776141 secs 2024-07-26T14:05:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:32 smithi160 ceph-mon[36715]: pgmap v6223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:32 smithi183 ceph-mon[37273]: pgmap v6223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:32 smithi144 ceph-mon[32129]: pgmap v6223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:33 smithi160 ceph-mon[36715]: pgmap v6224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:33 smithi183 ceph-mon[37273]: pgmap v6224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:33 smithi144 ceph-mon[32129]: pgmap v6224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:35 smithi160 ceph-mon[36715]: pgmap v6225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:35 smithi183 ceph-mon[37273]: pgmap v6225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:35 smithi144 ceph-mon[32129]: pgmap v6225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:36.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:05:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:36.394+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:05:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:36.394+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:05:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:36.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:05:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:36.395+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10779.776271 secs 2024-07-26T14:05:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10779.776271 secs 2024-07-26T14:05:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10779.776271 secs 2024-07-26T14:05:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:37 smithi160 ceph-mon[36715]: pgmap v6226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:37.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:37 smithi144 ceph-mon[32129]: pgmap v6226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:37 smithi183 ceph-mon[37273]: pgmap v6226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:39 smithi160 ceph-mon[36715]: pgmap v6227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:39 smithi183 ceph-mon[37273]: pgmap v6227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:39 smithi144 ceph-mon[32129]: pgmap v6227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:41 smithi160 ceph-mon[36715]: pgmap v6228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10784.776390 secs 2024-07-26T14:05:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:41 smithi183 ceph-mon[37273]: pgmap v6228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10784.776390 secs 2024-07-26T14:05:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:41 smithi144 ceph-mon[32129]: pgmap v6228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10784.776390 secs 2024-07-26T14:05:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:43 smithi160 ceph-mon[36715]: pgmap v6229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:43 smithi183 ceph-mon[37273]: pgmap v6229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:43 smithi144 ceph-mon[32129]: pgmap v6229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:45 smithi160 ceph-mon[36715]: pgmap v6230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:45 smithi183 ceph-mon[37273]: pgmap v6230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:45 smithi144 ceph-mon[32129]: pgmap v6230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5970 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a804c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5970 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a804c040 comp rx=0 tx=0).stop 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5970 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5970 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:05:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.129+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:05:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.130+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:05:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:46.132+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a8044000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:05:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10789.776519 secs 2024-07-26T14:05:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10789.776519 secs 2024-07-26T14:05:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10789.776519 secs 2024-07-26T14:05:47.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:47 smithi183 ceph-mon[37273]: pgmap v6231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:47.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:47 smithi160 ceph-mon[36715]: pgmap v6231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:47 smithi144 ceph-mon[32129]: pgmap v6231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:48.006+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:05:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:48.006+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:05:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:48.006+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:05:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:48.007+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:05:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:49 smithi183 ceph-mon[37273]: pgmap v6232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:49 smithi144 ceph-mon[32129]: pgmap v6232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:49 smithi160 ceph-mon[36715]: pgmap v6232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:51.400+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:05:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:51.401+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:05:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:51.401+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:05:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:51.401+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:05:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:05:51.401+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:51 smithi183 ceph-mon[37273]: pgmap v6233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10794.776679 secs 2024-07-26T14:05:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:51 smithi144 ceph-mon[32129]: pgmap v6233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10794.776679 secs 2024-07-26T14:05:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:51 smithi160 ceph-mon[36715]: pgmap v6233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10794.776679 secs 2024-07-26T14:05:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:53 smithi183 ceph-mon[37273]: pgmap v6234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:53 smithi144 ceph-mon[32129]: pgmap v6234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:53 smithi160 ceph-mon[36715]: pgmap v6234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:05:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:55 smithi183 ceph-mon[37273]: pgmap v6235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:55 smithi144 ceph-mon[32129]: pgmap v6235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:55 smithi160 ceph-mon[36715]: pgmap v6235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10799.776821 secs 2024-07-26T14:05:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10799.776821 secs 2024-07-26T14:05:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10799.776821 secs 2024-07-26T14:05:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:57 smithi183 ceph-mon[37273]: pgmap v6236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:57 smithi144 ceph-mon[32129]: pgmap v6236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:57 smithi160 ceph-mon[36715]: pgmap v6236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:05:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:05:59 smithi183 ceph-mon[37273]: pgmap v6237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:05:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:05:59 smithi144 ceph-mon[32129]: pgmap v6237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:05:59 smithi160 ceph-mon[36715]: pgmap v6237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:01 smithi183 ceph-mon[37273]: pgmap v6238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10804.776963 secs 2024-07-26T14:06:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:01 smithi144 ceph-mon[32129]: pgmap v6238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10804.776963 secs 2024-07-26T14:06:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:01 smithi160 ceph-mon[36715]: pgmap v6238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10804.776963 secs 2024-07-26T14:06:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:03 smithi183 ceph-mon[37273]: pgmap v6239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:03 smithi144 ceph-mon[32129]: pgmap v6239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:03 smithi160 ceph-mon[36715]: pgmap v6239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:05 smithi183 ceph-mon[37273]: pgmap v6240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:05 smithi144 ceph-mon[32129]: pgmap v6240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:05 smithi160 ceph-mon[36715]: pgmap v6240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:06.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:06.417+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:06:06.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:06.418+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:06:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:06.418+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:06:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:06.418+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:06:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:06.418+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10809.777111 secs 2024-07-26T14:06:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10809.777111 secs 2024-07-26T14:06:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10809.777111 secs 2024-07-26T14:06:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:07 smithi144 ceph-mon[32129]: pgmap v6241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:07 smithi160 ceph-mon[36715]: pgmap v6241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:07 smithi183 ceph-mon[37273]: pgmap v6241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:09 smithi144 ceph-mon[32129]: pgmap v6242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:09 smithi160 ceph-mon[36715]: pgmap v6242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:09 smithi183 ceph-mon[37273]: pgmap v6242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:11 smithi144 ceph-mon[32129]: pgmap v6243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10814.777233 secs 2024-07-26T14:06:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:11 smithi160 ceph-mon[36715]: pgmap v6243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10814.777233 secs 2024-07-26T14:06:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:11 smithi183 ceph-mon[37273]: pgmap v6243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10814.777233 secs 2024-07-26T14:06:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:13 smithi144 ceph-mon[32129]: pgmap v6244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:13 smithi160 ceph-mon[36715]: pgmap v6244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:13 smithi183 ceph-mon[37273]: pgmap v6244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:15 smithi144 ceph-mon[32129]: pgmap v6245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:15 smithi160 ceph-mon[36715]: pgmap v6245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:15 smithi183 ceph-mon[37273]: pgmap v6245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10819.777322 secs 2024-07-26T14:06:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10819.777322 secs 2024-07-26T14:06:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10819.777322 secs 2024-07-26T14:06:17.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:17 smithi160 ceph-mon[36715]: pgmap v6246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:17 smithi144 ceph-mon[32129]: pgmap v6246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:17 smithi183 ceph-mon[37273]: pgmap v6246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:19 smithi144 ceph-mon[32129]: pgmap v6247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:19 smithi160 ceph-mon[36715]: pgmap v6247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:19 smithi183 ceph-mon[37273]: pgmap v6247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:20 smithi144 ceph-mon[32129]: pgmap v6248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:06:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:20 smithi144 ceph-mon[32129]: pgmap v6249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:06:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:06:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:20 smithi160 ceph-mon[36715]: pgmap v6248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:06:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:20 smithi160 ceph-mon[36715]: pgmap v6249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:06:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:06:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:20 smithi183 ceph-mon[37273]: pgmap v6248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:06:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:20 smithi183 ceph-mon[37273]: pgmap v6249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:06:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:06:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:21.434+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:21.435+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:21.435+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:21.435+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:21.435+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10824.777445 secs 2024-07-26T14:06:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10824.777445 secs 2024-07-26T14:06:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10824.777445 secs 2024-07-26T14:06:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:22 smithi144 ceph-mon[32129]: pgmap v6250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:06:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:22 smithi160 ceph-mon[36715]: pgmap v6250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:06:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:22 smithi183 ceph-mon[37273]: pgmap v6250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:06:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:24 smithi183 ceph-mon[37273]: pgmap v6251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:06:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:24 smithi144 ceph-mon[32129]: pgmap v6251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:06:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:24 smithi160 ceph-mon[36715]: pgmap v6251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:06:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:26 smithi183 ceph-mon[37273]: pgmap v6252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:06:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10829.777567 secs 2024-07-26T14:06:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:26 smithi144 ceph-mon[32129]: pgmap v6252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:06:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10829.777567 secs 2024-07-26T14:06:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:26 smithi160 ceph-mon[36715]: pgmap v6252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:06:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10829.777567 secs 2024-07-26T14:06:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:28 smithi183 ceph-mon[37273]: pgmap v6253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:06:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:28 smithi144 ceph-mon[32129]: pgmap v6253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:06:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:28 smithi160 ceph-mon[36715]: pgmap v6253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:06:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:30 smithi183 ceph-mon[37273]: pgmap v6254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:06:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:30 smithi144 ceph-mon[32129]: pgmap v6254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:06:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:30 smithi160 ceph-mon[36715]: pgmap v6254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:06:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10834.777689 secs 2024-07-26T14:06:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10834.777689 secs 2024-07-26T14:06:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10834.777689 secs 2024-07-26T14:06:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:32 smithi183 ceph-mon[37273]: pgmap v6255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:32 smithi144 ceph-mon[32129]: pgmap v6255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:32 smithi160 ceph-mon[36715]: pgmap v6255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:34 smithi183 ceph-mon[37273]: pgmap v6256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:34 smithi144 ceph-mon[32129]: pgmap v6256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:34 smithi160 ceph-mon[36715]: pgmap v6256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:36.436+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:06:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:36.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:06:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:36.438+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:06:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:36.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:06:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:36.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:36 smithi183 ceph-mon[37273]: pgmap v6257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10839.777806 secs 2024-07-26T14:06:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:36 smithi144 ceph-mon[32129]: pgmap v6257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10839.777806 secs 2024-07-26T14:06:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:36 smithi160 ceph-mon[36715]: pgmap v6257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10839.777806 secs 2024-07-26T14:06:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:38 smithi183 ceph-mon[37273]: pgmap v6258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:38 smithi144 ceph-mon[32129]: pgmap v6258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:38 smithi160 ceph-mon[36715]: pgmap v6258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:40 smithi144 ceph-mon[32129]: pgmap v6259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:40 smithi160 ceph-mon[36715]: pgmap v6259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:40 smithi183 ceph-mon[37273]: pgmap v6259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10844.777930 secs 2024-07-26T14:06:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10844.777930 secs 2024-07-26T14:06:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10844.777930 secs 2024-07-26T14:06:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:42 smithi144 ceph-mon[32129]: pgmap v6260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:42 smithi160 ceph-mon[36715]: pgmap v6260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:42 smithi183 ceph-mon[37273]: pgmap v6260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:44 smithi144 ceph-mon[32129]: pgmap v6261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:44 smithi160 ceph-mon[36715]: pgmap v6261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:44 smithi183 ceph-mon[37273]: pgmap v6261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:46 smithi144 ceph-mon[32129]: pgmap v6262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10849.778019 secs 2024-07-26T14:06:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:46 smithi160 ceph-mon[36715]: pgmap v6262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10849.778019 secs 2024-07-26T14:06:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:46 smithi183 ceph-mon[37273]: pgmap v6262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10849.778019 secs 2024-07-26T14:06:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:48 smithi144 ceph-mon[32129]: pgmap v6263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:48 smithi160 ceph-mon[36715]: pgmap v6263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:48 smithi183 ceph-mon[37273]: pgmap v6263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:50 smithi144 ceph-mon[32129]: pgmap v6264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:50 smithi160 ceph-mon[36715]: pgmap v6264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:50 smithi183 ceph-mon[37273]: pgmap v6264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:51.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:51.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:06:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:51.451+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:06:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:51.451+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:06:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:51.451+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:06:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:06:51.451+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10854.778134 secs 2024-07-26T14:06:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10854.778134 secs 2024-07-26T14:06:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10854.778134 secs 2024-07-26T14:06:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:52 smithi144 ceph-mon[32129]: pgmap v6265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:52 smithi160 ceph-mon[36715]: pgmap v6265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:52 smithi183 ceph-mon[37273]: pgmap v6265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:06:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:54 smithi144 ceph-mon[32129]: pgmap v6266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:54 smithi160 ceph-mon[36715]: pgmap v6266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:54 smithi183 ceph-mon[37273]: pgmap v6266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10859.778259 secs 2024-07-26T14:06:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10859.778259 secs 2024-07-26T14:06:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10859.778259 secs 2024-07-26T14:06:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:57 smithi160 ceph-mon[36715]: pgmap v6267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:57 smithi183 ceph-mon[37273]: pgmap v6267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:57 smithi144 ceph-mon[32129]: pgmap v6267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:06:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:06:59 smithi160 ceph-mon[36715]: pgmap v6268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:06:59 smithi183 ceph-mon[37273]: pgmap v6268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:06:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:06:59 smithi144 ceph-mon[32129]: pgmap v6268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:01 smithi160 ceph-mon[36715]: pgmap v6269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10864.778377 secs 2024-07-26T14:07:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:01 smithi183 ceph-mon[37273]: pgmap v6269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10864.778377 secs 2024-07-26T14:07:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:01 smithi144 ceph-mon[32129]: pgmap v6269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10864.778377 secs 2024-07-26T14:07:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:03 smithi160 ceph-mon[36715]: pgmap v6270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:03 smithi183 ceph-mon[37273]: pgmap v6270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:03 smithi144 ceph-mon[32129]: pgmap v6270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:05 smithi160 ceph-mon[36715]: pgmap v6271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:05 smithi183 ceph-mon[37273]: pgmap v6271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:05 smithi144 ceph-mon[32129]: pgmap v6271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:06.453+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:07:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:06.454+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:07:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:06.454+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:07:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:06.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:07:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:06.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10869.778496 secs 2024-07-26T14:07:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10869.778496 secs 2024-07-26T14:07:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10869.778496 secs 2024-07-26T14:07:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:07 smithi160 ceph-mon[36715]: pgmap v6272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:07.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:07 smithi144 ceph-mon[32129]: pgmap v6272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:07 smithi183 ceph-mon[37273]: pgmap v6272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:08 smithi160 ceph-mon[36715]: pgmap v6273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:08 smithi183 ceph-mon[37273]: pgmap v6273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:08 smithi144 ceph-mon[32129]: pgmap v6273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:10 smithi183 ceph-mon[37273]: pgmap v6274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:10 smithi144 ceph-mon[32129]: pgmap v6274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:10 smithi160 ceph-mon[36715]: pgmap v6274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10874.778592 secs 2024-07-26T14:07:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10874.778592 secs 2024-07-26T14:07:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10874.778592 secs 2024-07-26T14:07:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:12 smithi183 ceph-mon[37273]: pgmap v6275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:12 smithi144 ceph-mon[32129]: pgmap v6275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:12 smithi160 ceph-mon[36715]: pgmap v6275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:14 smithi183 ceph-mon[37273]: pgmap v6276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:14 smithi144 ceph-mon[32129]: pgmap v6276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:14 smithi160 ceph-mon[36715]: pgmap v6276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10879.778712 secs 2024-07-26T14:07:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:16 smithi183 ceph-mon[37273]: pgmap v6277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10879.778712 secs 2024-07-26T14:07:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:16 smithi144 ceph-mon[32129]: pgmap v6277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10879.778712 secs 2024-07-26T14:07:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:16 smithi160 ceph-mon[36715]: pgmap v6277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:18.022+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:07:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:18.022+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478004740 con 0x563c6c4182f0 2024-07-26T14:07:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:18.023+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:07:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:18.023+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:07:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:18 smithi144 ceph-mon[32129]: pgmap v6278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:18 smithi160 ceph-mon[36715]: pgmap v6278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:18 smithi183 ceph-mon[37273]: pgmap v6278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:20 smithi144 ceph-mon[32129]: pgmap v6279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:20 smithi160 ceph-mon[36715]: pgmap v6279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:20 smithi183 ceph-mon[37273]: pgmap v6279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:21.469+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:07:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:21.470+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:07:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:21.470+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:07:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:21.470+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:07:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:21.470+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10884.778854 secs 2024-07-26T14:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:21 smithi144 ceph-mon[32129]: pgmap v6280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:21 smithi144 ceph-mon[32129]: pgmap v6281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:07:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10884.778854 secs 2024-07-26T14:07:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:21 smithi160 ceph-mon[36715]: pgmap v6280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:21 smithi160 ceph-mon[36715]: pgmap v6281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:07:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10884.778854 secs 2024-07-26T14:07:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:21 smithi183 ceph-mon[37273]: pgmap v6280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:21 smithi183 ceph-mon[37273]: pgmap v6281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:07:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:24 smithi160 ceph-mon[36715]: pgmap v6282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:07:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:24 smithi183 ceph-mon[37273]: pgmap v6282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:07:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:24 smithi144 ceph-mon[32129]: pgmap v6282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:07:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:26 smithi160 ceph-mon[36715]: pgmap v6283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:07:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10889.778975 secs 2024-07-26T14:07:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:26 smithi183 ceph-mon[37273]: pgmap v6283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:07:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10889.778975 secs 2024-07-26T14:07:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:26 smithi144 ceph-mon[32129]: pgmap v6283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:07:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10889.778975 secs 2024-07-26T14:07:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:28 smithi160 ceph-mon[36715]: pgmap v6284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:28 smithi183 ceph-mon[37273]: pgmap v6284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:28 smithi144 ceph-mon[32129]: pgmap v6284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:07:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:30 smithi160 ceph-mon[36715]: pgmap v6285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:07:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:30 smithi183 ceph-mon[37273]: pgmap v6285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:07:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:30 smithi144 ceph-mon[32129]: pgmap v6285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:07:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10894.779086 secs 2024-07-26T14:07:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10894.779086 secs 2024-07-26T14:07:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10894.779086 secs 2024-07-26T14:07:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:32 smithi160 ceph-mon[36715]: pgmap v6286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:07:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:32 smithi183 ceph-mon[37273]: pgmap v6286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:07:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:32 smithi144 ceph-mon[32129]: pgmap v6286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:07:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:33 smithi183 ceph-mon[37273]: pgmap v6287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:33 smithi144 ceph-mon[32129]: pgmap v6287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:33 smithi160 ceph-mon[36715]: pgmap v6287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:36 smithi160 ceph-mon[36715]: pgmap v6288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:36 smithi183 ceph-mon[37273]: pgmap v6288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:36 smithi144 ceph-mon[32129]: pgmap v6288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:36.484+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:07:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:36.485+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:07:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:36.485+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:07:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:36.485+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:07:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:36.485+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:37.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10899.779193 secs 2024-07-26T14:07:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10899.779193 secs 2024-07-26T14:07:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10899.779193 secs 2024-07-26T14:07:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:38 smithi183 ceph-mon[37273]: pgmap v6289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:38 smithi144 ceph-mon[32129]: pgmap v6289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:38 smithi160 ceph-mon[36715]: pgmap v6289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:40 smithi183 ceph-mon[37273]: pgmap v6290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:40 smithi144 ceph-mon[32129]: pgmap v6290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:40 smithi160 ceph-mon[36715]: pgmap v6290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10904.779351 secs 2024-07-26T14:07:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:42 smithi183 ceph-mon[37273]: pgmap v6291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10904.779351 secs 2024-07-26T14:07:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:42 smithi144 ceph-mon[32129]: pgmap v6291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10904.779351 secs 2024-07-26T14:07:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:42 smithi160 ceph-mon[36715]: pgmap v6291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:44 smithi183 ceph-mon[37273]: pgmap v6292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:44 smithi144 ceph-mon[32129]: pgmap v6292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:44 smithi160 ceph-mon[36715]: pgmap v6292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:46 smithi160 ceph-mon[36715]: pgmap v6293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10909.779480 secs 2024-07-26T14:07:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:46 smithi183 ceph-mon[37273]: pgmap v6293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10909.779480 secs 2024-07-26T14:07:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:46 smithi144 ceph-mon[32129]: pgmap v6293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10909.779480 secs 2024-07-26T14:07:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:48 smithi160 ceph-mon[36715]: pgmap v6294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:48 smithi183 ceph-mon[37273]: pgmap v6294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:48 smithi144 ceph-mon[32129]: pgmap v6294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:50 smithi160 ceph-mon[36715]: pgmap v6295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:50 smithi183 ceph-mon[37273]: pgmap v6295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:50 smithi144 ceph-mon[32129]: pgmap v6295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:51.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:07:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:51.503+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:07:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:51.503+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:07:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:51.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:07:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:07:51.503+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10914.779600 secs 2024-07-26T14:07:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10914.779600 secs 2024-07-26T14:07:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10914.779600 secs 2024-07-26T14:07:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:52 smithi160 ceph-mon[36715]: pgmap v6296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:52 smithi183 ceph-mon[37273]: pgmap v6296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:52 smithi144 ceph-mon[32129]: pgmap v6296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:53 smithi160 ceph-mon[36715]: pgmap v6297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:53 smithi183 ceph-mon[37273]: pgmap v6297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:53 smithi144 ceph-mon[32129]: pgmap v6297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:07:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:56 smithi183 ceph-mon[37273]: pgmap v6298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:56 smithi144 ceph-mon[32129]: pgmap v6298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:56 smithi160 ceph-mon[36715]: pgmap v6298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:07:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10919.779720 secs 2024-07-26T14:07:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10919.779720 secs 2024-07-26T14:07:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10919.779720 secs 2024-07-26T14:07:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:07:58 smithi183 ceph-mon[37273]: pgmap v6299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:07:58 smithi144 ceph-mon[32129]: pgmap v6299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:07:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:07:58 smithi160 ceph-mon[36715]: pgmap v6299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:00 smithi183 ceph-mon[37273]: pgmap v6300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:00 smithi144 ceph-mon[32129]: pgmap v6300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:00 smithi160 ceph-mon[36715]: pgmap v6300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10924.779868 secs 2024-07-26T14:08:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:02 smithi144 ceph-mon[32129]: pgmap v6301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10924.779868 secs 2024-07-26T14:08:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:02 smithi183 ceph-mon[37273]: pgmap v6301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10924.779868 secs 2024-07-26T14:08:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:02 smithi160 ceph-mon[36715]: pgmap v6301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:04 smithi183 ceph-mon[37273]: pgmap v6302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:04 smithi144 ceph-mon[32129]: pgmap v6302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:04 smithi160 ceph-mon[36715]: pgmap v6302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:06 smithi183 ceph-mon[37273]: pgmap v6303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:06 smithi144 ceph-mon[32129]: pgmap v6303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:06.512+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:08:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:06.513+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:08:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:06.513+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:08:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:06.513+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:08:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:06.513+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:06 smithi160 ceph-mon[36715]: pgmap v6303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:07.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10929.779998 secs 2024-07-26T14:08:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10929.779998 secs 2024-07-26T14:08:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10929.779998 secs 2024-07-26T14:08:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:08 smithi183 ceph-mon[37273]: pgmap v6304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:08 smithi144 ceph-mon[32129]: pgmap v6304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:08 smithi160 ceph-mon[36715]: pgmap v6304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:10 smithi183 ceph-mon[37273]: pgmap v6305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:10 smithi144 ceph-mon[32129]: pgmap v6305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:10 smithi160 ceph-mon[36715]: pgmap v6305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10934.780137 secs 2024-07-26T14:08:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:12 smithi183 ceph-mon[37273]: pgmap v6306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10934.780137 secs 2024-07-26T14:08:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:12 smithi144 ceph-mon[32129]: pgmap v6306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10934.780137 secs 2024-07-26T14:08:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:12 smithi160 ceph-mon[36715]: pgmap v6306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:14 smithi183 ceph-mon[37273]: pgmap v6307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:14 smithi144 ceph-mon[32129]: pgmap v6307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:14 smithi160 ceph-mon[36715]: pgmap v6307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:16 smithi144 ceph-mon[32129]: pgmap v6308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:16 smithi160 ceph-mon[36715]: pgmap v6308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:16 smithi183 ceph-mon[37273]: pgmap v6308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10939.780256 secs 2024-07-26T14:08:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10939.780256 secs 2024-07-26T14:08:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10939.780256 secs 2024-07-26T14:08:18.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:18 smithi160 ceph-mon[36715]: pgmap v6309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:18 smithi144 ceph-mon[32129]: pgmap v6309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:18 smithi183 ceph-mon[37273]: pgmap v6309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:20 smithi144 ceph-mon[32129]: pgmap v6310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:20 smithi160 ceph-mon[36715]: pgmap v6310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:20 smithi183 ceph-mon[37273]: pgmap v6310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:21.529+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:08:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:21.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:08:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:21.530+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:08:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:21.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:08:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:21.530+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10944.780343 secs 2024-07-26T14:08:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:22 smithi144 ceph-mon[32129]: pgmap v6311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10944.780343 secs 2024-07-26T14:08:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:22 smithi160 ceph-mon[36715]: pgmap v6311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10944.780343 secs 2024-07-26T14:08:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:22 smithi183 ceph-mon[37273]: pgmap v6311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:23 smithi144 ceph-mon[32129]: pgmap v6312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:08:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:23 smithi144 ceph-mon[32129]: pgmap v6313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:08:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:23 smithi160 ceph-mon[36715]: pgmap v6312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:08:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:23 smithi160 ceph-mon[36715]: pgmap v6313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:08:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:23 smithi183 ceph-mon[37273]: pgmap v6312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:08:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:23 smithi183 ceph-mon[37273]: pgmap v6313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:08:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:25 smithi160 ceph-mon[36715]: pgmap v6314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:25 smithi183 ceph-mon[37273]: pgmap v6314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:25 smithi144 ceph-mon[32129]: pgmap v6314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10949.780481 secs 2024-07-26T14:08:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10949.780481 secs 2024-07-26T14:08:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10949.780481 secs 2024-07-26T14:08:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:27 smithi160 ceph-mon[36715]: pgmap v6315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:27 smithi183 ceph-mon[37273]: pgmap v6315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:27 smithi144 ceph-mon[32129]: pgmap v6315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:29 smithi160 ceph-mon[36715]: pgmap v6316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:29 smithi183 ceph-mon[37273]: pgmap v6316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:29 smithi144 ceph-mon[32129]: pgmap v6316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:31 smithi160 ceph-mon[36715]: pgmap v6317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10954.780602 secs 2024-07-26T14:08:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:31 smithi183 ceph-mon[37273]: pgmap v6317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10954.780602 secs 2024-07-26T14:08:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:31 smithi144 ceph-mon[32129]: pgmap v6317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:08:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10954.780602 secs 2024-07-26T14:08:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:33 smithi144 ceph-mon[32129]: pgmap v6318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:08:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:33 smithi160 ceph-mon[36715]: pgmap v6318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:08:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:33 smithi183 ceph-mon[37273]: pgmap v6318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:08:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:35 smithi160 ceph-mon[36715]: pgmap v6319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:35 smithi183 ceph-mon[37273]: pgmap v6319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:35.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:35 smithi144 ceph-mon[32129]: pgmap v6319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:36.545+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:08:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:36.546+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:08:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:36.546+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:08:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:36.546+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:08:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:36.546+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10959.780728 secs 2024-07-26T14:08:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10959.780728 secs 2024-07-26T14:08:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10959.780728 secs 2024-07-26T14:08:37.620 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:37 smithi144 ceph-mon[32129]: pgmap v6320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:37 smithi183 ceph-mon[37273]: pgmap v6320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:37 smithi160 ceph-mon[36715]: pgmap v6320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:39 smithi183 ceph-mon[37273]: pgmap v6321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:39 smithi144 ceph-mon[32129]: pgmap v6321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:39.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:39 smithi160 ceph-mon[36715]: pgmap v6321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:41 smithi183 ceph-mon[37273]: pgmap v6322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10964.780832 secs 2024-07-26T14:08:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:41 smithi144 ceph-mon[32129]: pgmap v6322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10964.780832 secs 2024-07-26T14:08:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:41 smithi160 ceph-mon[36715]: pgmap v6322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10964.780832 secs 2024-07-26T14:08:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:43 smithi183 ceph-mon[37273]: pgmap v6323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:43 smithi144 ceph-mon[32129]: pgmap v6323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:43 smithi160 ceph-mon[36715]: pgmap v6323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:45 smithi183 ceph-mon[37273]: pgmap v6324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:45 smithi144 ceph-mon[32129]: pgmap v6324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:45 smithi160 ceph-mon[36715]: pgmap v6324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10969.780959 secs 2024-07-26T14:08:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10969.780959 secs 2024-07-26T14:08:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10969.780959 secs 2024-07-26T14:08:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:47 smithi183 ceph-mon[37273]: pgmap v6325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:47 smithi144 ceph-mon[32129]: pgmap v6325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:47 smithi160 ceph-mon[36715]: pgmap v6325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:48.038+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:08:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:48.038+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:08:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:48.038+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:08:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:48.039+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:08:49.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:49 smithi160 ceph-mon[36715]: pgmap v6326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:49 smithi183 ceph-mon[37273]: pgmap v6326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:49 smithi144 ceph-mon[32129]: pgmap v6326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:51.561+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:08:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:51.562+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:08:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:51.562+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:08:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:51.562+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:08:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:08:51.562+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:51 smithi183 ceph-mon[37273]: pgmap v6327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10974.781054 secs 2024-07-26T14:08:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:51 smithi144 ceph-mon[32129]: pgmap v6327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10974.781054 secs 2024-07-26T14:08:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:51 smithi160 ceph-mon[36715]: pgmap v6327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10974.781054 secs 2024-07-26T14:08:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:53 smithi183 ceph-mon[37273]: pgmap v6328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:53 smithi144 ceph-mon[32129]: pgmap v6328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:53 smithi160 ceph-mon[36715]: pgmap v6328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:08:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:55 smithi183 ceph-mon[37273]: pgmap v6329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:55 smithi144 ceph-mon[32129]: pgmap v6329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:55 smithi160 ceph-mon[36715]: pgmap v6329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10979.781179 secs 2024-07-26T14:08:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10979.781179 secs 2024-07-26T14:08:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10979.781179 secs 2024-07-26T14:08:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:57 smithi144 ceph-mon[32129]: pgmap v6330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:57 smithi160 ceph-mon[36715]: pgmap v6330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:57 smithi183 ceph-mon[37273]: pgmap v6330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:08:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:08:59 smithi144 ceph-mon[32129]: pgmap v6331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:08:59 smithi160 ceph-mon[36715]: pgmap v6331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:08:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:08:59 smithi183 ceph-mon[37273]: pgmap v6331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:01 smithi144 ceph-mon[32129]: pgmap v6332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10984.781331 secs 2024-07-26T14:09:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:01 smithi160 ceph-mon[36715]: pgmap v6332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10984.781331 secs 2024-07-26T14:09:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:01 smithi183 ceph-mon[37273]: pgmap v6332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10984.781331 secs 2024-07-26T14:09:03.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:03 smithi144 ceph-mon[32129]: pgmap v6333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:03 smithi160 ceph-mon[36715]: pgmap v6333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:03 smithi183 ceph-mon[37273]: pgmap v6333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:05.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:05 smithi183 ceph-mon[37273]: pgmap v6334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:05 smithi144 ceph-mon[32129]: pgmap v6334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:05 smithi160 ceph-mon[36715]: pgmap v6334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:06.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:09:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:06.567+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:09:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:06.568+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:09:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:06.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:09:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:06.568+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10989.781439 secs 2024-07-26T14:09:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10989.781439 secs 2024-07-26T14:09:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10989.781439 secs 2024-07-26T14:09:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:07 smithi144 ceph-mon[32129]: pgmap v6335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:07 smithi160 ceph-mon[36715]: pgmap v6335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:07 smithi183 ceph-mon[37273]: pgmap v6335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:09 smithi144 ceph-mon[32129]: pgmap v6336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:09 smithi160 ceph-mon[36715]: pgmap v6336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:09 smithi183 ceph-mon[37273]: pgmap v6336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:11 smithi144 ceph-mon[32129]: pgmap v6337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10994.781528 secs 2024-07-26T14:09:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:11 smithi160 ceph-mon[36715]: pgmap v6337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10994.781528 secs 2024-07-26T14:09:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:11 smithi183 ceph-mon[37273]: pgmap v6337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10994.781528 secs 2024-07-26T14:09:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:13 smithi144 ceph-mon[32129]: pgmap v6338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:13 smithi160 ceph-mon[36715]: pgmap v6338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:13 smithi183 ceph-mon[37273]: pgmap v6338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:15 smithi144 ceph-mon[32129]: pgmap v6339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:15 smithi160 ceph-mon[36715]: pgmap v6339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:15 smithi183 ceph-mon[37273]: pgmap v6339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 10999.781617 secs 2024-07-26T14:09:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 10999.781617 secs 2024-07-26T14:09:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 10999.781617 secs 2024-07-26T14:09:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:17 smithi144 ceph-mon[32129]: pgmap v6340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:17 smithi160 ceph-mon[36715]: pgmap v6340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:17 smithi183 ceph-mon[37273]: pgmap v6340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:19.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:19 smithi144 ceph-mon[32129]: pgmap v6341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:19 smithi160 ceph-mon[36715]: pgmap v6341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:19 smithi183 ceph-mon[37273]: pgmap v6341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:21.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:21.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:09:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:21.584+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:09:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:21.584+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:09:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:21.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:09:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:21.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:21 smithi160 ceph-mon[36715]: pgmap v6342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11004.781722 secs 2024-07-26T14:09:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:21 smithi183 ceph-mon[37273]: pgmap v6342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11004.781722 secs 2024-07-26T14:09:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:21 smithi144 ceph-mon[32129]: pgmap v6342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11004.781722 secs 2024-07-26T14:09:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:23 smithi160 ceph-mon[36715]: pgmap v6343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:09:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:23 smithi183 ceph-mon[37273]: pgmap v6343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:09:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:23 smithi144 ceph-mon[32129]: pgmap v6343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:09:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:24 smithi160 ceph-mon[36715]: pgmap v6344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:24 smithi160 ceph-mon[36715]: pgmap v6345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:09:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:24 smithi183 ceph-mon[37273]: pgmap v6344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:24 smithi183 ceph-mon[37273]: pgmap v6345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:09:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:24 smithi144 ceph-mon[32129]: pgmap v6344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:24 smithi144 ceph-mon[32129]: pgmap v6345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:09:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:26 smithi160 ceph-mon[36715]: pgmap v6346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11009.781846 secs 2024-07-26T14:09:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:26 smithi183 ceph-mon[37273]: pgmap v6346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11009.781846 secs 2024-07-26T14:09:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:26 smithi144 ceph-mon[32129]: pgmap v6346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11009.781846 secs 2024-07-26T14:09:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:28 smithi160 ceph-mon[36715]: pgmap v6347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:09:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:28 smithi183 ceph-mon[37273]: pgmap v6347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:09:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:28 smithi144 ceph-mon[32129]: pgmap v6347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:09:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:30 smithi160 ceph-mon[36715]: pgmap v6348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:09:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:30 smithi183 ceph-mon[37273]: pgmap v6348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:09:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:30 smithi144 ceph-mon[32129]: pgmap v6348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:09:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11014.781986 secs 2024-07-26T14:09:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11014.781986 secs 2024-07-26T14:09:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11014.781986 secs 2024-07-26T14:09:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:32 smithi160 ceph-mon[36715]: pgmap v6349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:32 smithi183 ceph-mon[37273]: pgmap v6349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:32 smithi144 ceph-mon[32129]: pgmap v6349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:09:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:34 smithi160 ceph-mon[36715]: pgmap v6350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:09:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:34 smithi183 ceph-mon[37273]: pgmap v6350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:09:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:34 smithi144 ceph-mon[32129]: pgmap v6350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:09:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:35 smithi160 ceph-mon[36715]: pgmap v6351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:35 smithi183 ceph-mon[37273]: pgmap v6351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:35 smithi144 ceph-mon[32129]: pgmap v6351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:36.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:09:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:36.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:09:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:36.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:09:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:36.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:09:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:36.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11019.782139 secs 2024-07-26T14:09:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11019.782139 secs 2024-07-26T14:09:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11019.782139 secs 2024-07-26T14:09:37.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:37 smithi144 ceph-mon[32129]: pgmap v6352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:37 smithi183 ceph-mon[37273]: pgmap v6352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:37 smithi160 ceph-mon[36715]: pgmap v6352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:40 smithi160 ceph-mon[36715]: pgmap v6353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:40 smithi183 ceph-mon[37273]: pgmap v6353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:40 smithi144 ceph-mon[32129]: pgmap v6353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11024.782261 secs 2024-07-26T14:09:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11024.782261 secs 2024-07-26T14:09:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11024.782261 secs 2024-07-26T14:09:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:42 smithi160 ceph-mon[36715]: pgmap v6354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:42 smithi183 ceph-mon[37273]: pgmap v6354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:42 smithi144 ceph-mon[32129]: pgmap v6354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:44 smithi160 ceph-mon[36715]: pgmap v6355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:44 smithi183 ceph-mon[37273]: pgmap v6355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:44 smithi144 ceph-mon[32129]: pgmap v6355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:46 smithi160 ceph-mon[36715]: pgmap v6356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11029.782382 secs 2024-07-26T14:09:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:46 smithi183 ceph-mon[37273]: pgmap v6356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11029.782382 secs 2024-07-26T14:09:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:46 smithi144 ceph-mon[32129]: pgmap v6356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11029.782382 secs 2024-07-26T14:09:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:48 smithi160 ceph-mon[36715]: pgmap v6357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:48 smithi183 ceph-mon[37273]: pgmap v6357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:48 smithi144 ceph-mon[32129]: pgmap v6357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:50 smithi160 ceph-mon[36715]: pgmap v6358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:50 smithi183 ceph-mon[37273]: pgmap v6358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:50 smithi144 ceph-mon[32129]: pgmap v6358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:51.606+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:09:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:51.607+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:09:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:51.607+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:09:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:51.607+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:09:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:09:51.607+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11034.782461 secs 2024-07-26T14:09:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11034.782461 secs 2024-07-26T14:09:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11034.782461 secs 2024-07-26T14:09:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:52 smithi160 ceph-mon[36715]: pgmap v6359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:52 smithi183 ceph-mon[37273]: pgmap v6359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:52 smithi144 ceph-mon[32129]: pgmap v6359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:09:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:53 smithi183 ceph-mon[37273]: pgmap v6360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:53 smithi144 ceph-mon[32129]: pgmap v6360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:53 smithi160 ceph-mon[36715]: pgmap v6360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:56 smithi160 ceph-mon[36715]: pgmap v6361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11039.782545 secs 2024-07-26T14:09:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:56 smithi183 ceph-mon[37273]: pgmap v6361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11039.782545 secs 2024-07-26T14:09:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:56 smithi144 ceph-mon[32129]: pgmap v6361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:09:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11039.782545 secs 2024-07-26T14:09:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:09:58 smithi160 ceph-mon[36715]: pgmap v6362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:09:58 smithi183 ceph-mon[37273]: pgmap v6362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:09:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:09:58 smithi144 ceph-mon[32129]: pgmap v6362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:00 smithi160 ceph-mon[36715]: pgmap v6363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:10:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:00 smithi183 ceph-mon[37273]: pgmap v6363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:10:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:00 smithi144 ceph-mon[32129]: pgmap v6363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:10:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11044.782667 secs 2024-07-26T14:10:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11044.782667 secs 2024-07-26T14:10:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11044.782667 secs 2024-07-26T14:10:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:02 smithi160 ceph-mon[36715]: pgmap v6364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:02 smithi183 ceph-mon[37273]: pgmap v6364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:02 smithi144 ceph-mon[32129]: pgmap v6364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:04 smithi160 ceph-mon[36715]: pgmap v6365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:04 smithi183 ceph-mon[37273]: pgmap v6365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:04 smithi144 ceph-mon[32129]: pgmap v6365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:05.853 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:05 smithi144 ceph-mon[32129]: pgmap v6366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:05 smithi183 ceph-mon[37273]: pgmap v6366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:05 smithi160 ceph-mon[36715]: pgmap v6366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:06.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:10:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:06.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:10:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:06.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:10:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:06.617+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:10:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:06.617+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11049.782787 secs 2024-07-26T14:10:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11049.782787 secs 2024-07-26T14:10:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11049.782787 secs 2024-07-26T14:10:07.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:07 smithi144 ceph-mon[32129]: pgmap v6367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:07 smithi183 ceph-mon[37273]: pgmap v6367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:07 smithi160 ceph-mon[36715]: pgmap v6367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:10 smithi160 ceph-mon[36715]: pgmap v6368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:10:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:10 smithi183 ceph-mon[37273]: pgmap v6368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:10:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:10 smithi144 ceph-mon[32129]: pgmap v6368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:10:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11054.782912 secs 2024-07-26T14:10:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11054.782912 secs 2024-07-26T14:10:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11054.782912 secs 2024-07-26T14:10:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:12 smithi160 ceph-mon[36715]: pgmap v6369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:12 smithi183 ceph-mon[37273]: pgmap v6369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:12 smithi144 ceph-mon[32129]: pgmap v6369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:14 smithi160 ceph-mon[36715]: pgmap v6370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:14 smithi183 ceph-mon[37273]: pgmap v6370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:14 smithi144 ceph-mon[32129]: pgmap v6370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:16 smithi160 ceph-mon[36715]: pgmap v6371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11059.783039 secs 2024-07-26T14:10:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:16 smithi183 ceph-mon[37273]: pgmap v6371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11059.783039 secs 2024-07-26T14:10:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:16 smithi144 ceph-mon[32129]: pgmap v6371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11059.783039 secs 2024-07-26T14:10:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:17 smithi183 ceph-mon[37273]: pgmap v6372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:17 smithi144 ceph-mon[32129]: pgmap v6372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:17 smithi160 ceph-mon[36715]: pgmap v6372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:20 smithi160 ceph-mon[36715]: pgmap v6373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:20 smithi183 ceph-mon[37273]: pgmap v6373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:20 smithi144 ceph-mon[32129]: pgmap v6373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:21.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:21.632+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:10:21.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:21.633+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:10:21.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:21.633+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:10:21.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:21.633+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:10:21.635 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:21.634+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11064.783165 secs 2024-07-26T14:10:21.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11064.783165 secs 2024-07-26T14:10:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11064.783165 secs 2024-07-26T14:10:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:22 smithi160 ceph-mon[36715]: pgmap v6374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:22.849 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:22 smithi144 ceph-mon[32129]: pgmap v6374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:22 smithi183 ceph-mon[37273]: pgmap v6374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:24 smithi160 ceph-mon[36715]: pgmap v6375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:24 smithi183 ceph-mon[37273]: pgmap v6375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:24 smithi144 ceph-mon[32129]: pgmap v6375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:25 smithi183 ceph-mon[37273]: pgmap v6376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:10:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:25 smithi183 ceph-mon[37273]: pgmap v6377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:10:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:25 smithi144 ceph-mon[32129]: pgmap v6376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:10:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:25 smithi144 ceph-mon[32129]: pgmap v6377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:10:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:25 smithi160 ceph-mon[36715]: pgmap v6376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:10:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:25 smithi160 ceph-mon[36715]: pgmap v6377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:10:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11069.783284 secs 2024-07-26T14:10:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11069.783284 secs 2024-07-26T14:10:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11069.783284 secs 2024-07-26T14:10:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:27 smithi183 ceph-mon[37273]: pgmap v6378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:10:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:27 smithi144 ceph-mon[32129]: pgmap v6378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:10:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:27 smithi160 ceph-mon[36715]: pgmap v6378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:10:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:29 smithi144 ceph-mon[32129]: pgmap v6379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:10:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:29 smithi160 ceph-mon[36715]: pgmap v6379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:10:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:29 smithi183 ceph-mon[37273]: pgmap v6379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:10:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:31 smithi144 ceph-mon[32129]: pgmap v6380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:10:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11074.783410 secs 2024-07-26T14:10:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:31 smithi160 ceph-mon[36715]: pgmap v6380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:10:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11074.783410 secs 2024-07-26T14:10:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:31 smithi183 ceph-mon[37273]: pgmap v6380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:10:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11074.783410 secs 2024-07-26T14:10:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:33 smithi144 ceph-mon[32129]: pgmap v6381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:33 smithi160 ceph-mon[36715]: pgmap v6381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:33 smithi183 ceph-mon[37273]: pgmap v6381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:10:35.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:35 smithi183 ceph-mon[37273]: pgmap v6382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:35 smithi144 ceph-mon[32129]: pgmap v6382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:35 smithi160 ceph-mon[36715]: pgmap v6382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:10:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:36.634+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:10:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:36.634+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:10:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:36.634+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:10:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:36.634+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:10:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:36.635+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11079.783533 secs 2024-07-26T14:10:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11079.783533 secs 2024-07-26T14:10:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11079.783533 secs 2024-07-26T14:10:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:37 smithi144 ceph-mon[32129]: pgmap v6383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:37 smithi160 ceph-mon[36715]: pgmap v6383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:37 smithi183 ceph-mon[37273]: pgmap v6383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:39 smithi144 ceph-mon[32129]: pgmap v6384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:39 smithi160 ceph-mon[36715]: pgmap v6384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:39 smithi183 ceph-mon[37273]: pgmap v6384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:41 smithi144 ceph-mon[32129]: pgmap v6385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11084.783666 secs 2024-07-26T14:10:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:41 smithi160 ceph-mon[36715]: pgmap v6385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11084.783666 secs 2024-07-26T14:10:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:41 smithi183 ceph-mon[37273]: pgmap v6385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11084.783666 secs 2024-07-26T14:10:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:43 smithi144 ceph-mon[32129]: pgmap v6386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:43 smithi160 ceph-mon[36715]: pgmap v6386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:43 smithi183 ceph-mon[37273]: pgmap v6386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:45 smithi144 ceph-mon[32129]: pgmap v6387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:45 smithi160 ceph-mon[36715]: pgmap v6387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:45 smithi183 ceph-mon[37273]: pgmap v6387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11089.783796 secs 2024-07-26T14:10:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11089.783796 secs 2024-07-26T14:10:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11089.783796 secs 2024-07-26T14:10:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:47 smithi160 ceph-mon[36715]: pgmap v6388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:47 smithi183 ceph-mon[37273]: pgmap v6388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:47 smithi144 ceph-mon[32129]: pgmap v6388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:49 smithi160 ceph-mon[36715]: pgmap v6389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:49 smithi183 ceph-mon[37273]: pgmap v6389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:49 smithi144 ceph-mon[32129]: pgmap v6389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:51.652 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:51.651+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:10:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:51.652+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:10:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:51.652+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:10:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:51.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:10:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:10:51.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:51 smithi160 ceph-mon[36715]: pgmap v6390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11094.783949 secs 2024-07-26T14:10:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:51 smithi183 ceph-mon[37273]: pgmap v6390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11094.783949 secs 2024-07-26T14:10:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:51 smithi144 ceph-mon[32129]: pgmap v6390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11094.783949 secs 2024-07-26T14:10:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:53 smithi160 ceph-mon[36715]: pgmap v6391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:53 smithi183 ceph-mon[37273]: pgmap v6391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:53 smithi144 ceph-mon[32129]: pgmap v6391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:10:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:55 smithi160 ceph-mon[36715]: pgmap v6392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:55 smithi183 ceph-mon[37273]: pgmap v6392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:55 smithi144 ceph-mon[32129]: pgmap v6392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:10:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11099.784078 secs 2024-07-26T14:10:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11099.784078 secs 2024-07-26T14:10:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11099.784078 secs 2024-07-26T14:10:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:57 smithi160 ceph-mon[36715]: pgmap v6393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:57 smithi183 ceph-mon[37273]: pgmap v6393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:10:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:57 smithi144 ceph-mon[32129]: pgmap v6393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:10:59 smithi160 ceph-mon[36715]: pgmap v6394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:10:59 smithi183 ceph-mon[37273]: pgmap v6394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:10:59 smithi144 ceph-mon[32129]: pgmap v6394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:01 smithi160 ceph-mon[36715]: pgmap v6395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11104.784216 secs 2024-07-26T14:11:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:01 smithi183 ceph-mon[37273]: pgmap v6395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11104.784216 secs 2024-07-26T14:11:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:01 smithi144 ceph-mon[32129]: pgmap v6395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11104.784216 secs 2024-07-26T14:11:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:03 smithi160 ceph-mon[36715]: pgmap v6396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:03 smithi183 ceph-mon[37273]: pgmap v6396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:03 smithi144 ceph-mon[32129]: pgmap v6396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:05 smithi160 ceph-mon[36715]: pgmap v6397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:05 smithi183 ceph-mon[37273]: pgmap v6397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:05 smithi144 ceph-mon[32129]: pgmap v6397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:06.662+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:11:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:06.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:11:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:06.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:11:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:06.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:11:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:06.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11109.784343 secs 2024-07-26T14:11:07.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11109.784343 secs 2024-07-26T14:11:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11109.784343 secs 2024-07-26T14:11:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:07 smithi160 ceph-mon[36715]: pgmap v6398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:07 smithi183 ceph-mon[37273]: pgmap v6398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:07 smithi144 ceph-mon[32129]: pgmap v6398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:09 smithi160 ceph-mon[36715]: pgmap v6399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:09 smithi183 ceph-mon[37273]: pgmap v6399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:09 smithi144 ceph-mon[32129]: pgmap v6399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:11 smithi160 ceph-mon[36715]: pgmap v6400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11114.784455 secs 2024-07-26T14:11:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:11 smithi183 ceph-mon[37273]: pgmap v6400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11114.784455 secs 2024-07-26T14:11:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:11 smithi144 ceph-mon[32129]: pgmap v6400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11114.784455 secs 2024-07-26T14:11:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:13 smithi183 ceph-mon[37273]: pgmap v6401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:13 smithi144 ceph-mon[32129]: pgmap v6401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:13 smithi160 ceph-mon[36715]: pgmap v6401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:15 smithi183 ceph-mon[37273]: pgmap v6402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:15 smithi144 ceph-mon[32129]: pgmap v6402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:15 smithi160 ceph-mon[36715]: pgmap v6402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11119.784587 secs 2024-07-26T14:11:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11119.784587 secs 2024-07-26T14:11:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11119.784587 secs 2024-07-26T14:11:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:17 smithi183 ceph-mon[37273]: pgmap v6403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:17 smithi144 ceph-mon[32129]: pgmap v6403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:17 smithi160 ceph-mon[36715]: pgmap v6403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:19 smithi183 ceph-mon[37273]: pgmap v6404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:19 smithi144 ceph-mon[32129]: pgmap v6404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:19 smithi160 ceph-mon[36715]: pgmap v6404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:21.664+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:11:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:21.665+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:11:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:21.665+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:11:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:21.665+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:11:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:21.665+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:22.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:21 smithi160 ceph-mon[36715]: pgmap v6405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:22.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11124.784672 secs 2024-07-26T14:11:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:21 smithi183 ceph-mon[37273]: pgmap v6405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11124.784672 secs 2024-07-26T14:11:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:21 smithi144 ceph-mon[32129]: pgmap v6405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11124.784672 secs 2024-07-26T14:11:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:23 smithi183 ceph-mon[37273]: pgmap v6406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:24.189 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:23 smithi144 ceph-mon[32129]: pgmap v6406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:23 smithi160 ceph-mon[36715]: pgmap v6406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: pgmap v6407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:11:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: pgmap v6407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: pgmap v6407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:26.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:11:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11129.784738 secs 2024-07-26T14:11:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:26 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:11:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:26 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:11:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:26 smithi183 ceph-mon[37273]: pgmap v6408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:11:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:26 smithi183 ceph-mon[37273]: pgmap v6409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:11:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11129.784738 secs 2024-07-26T14:11:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:26 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:11:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:26 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:11:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:26 smithi144 ceph-mon[32129]: pgmap v6408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:11:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:26 smithi144 ceph-mon[32129]: pgmap v6409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:11:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11129.784738 secs 2024-07-26T14:11:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:26 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:11:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:26 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:11:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:26 smithi160 ceph-mon[36715]: pgmap v6408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:11:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:26 smithi160 ceph-mon[36715]: pgmap v6409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:11:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:28 smithi183 ceph-mon[37273]: pgmap v6410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:11:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:28 smithi144 ceph-mon[32129]: pgmap v6410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:11:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:28 smithi160 ceph-mon[36715]: pgmap v6410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:11:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:30 smithi183 ceph-mon[37273]: pgmap v6411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:11:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:30 smithi144 ceph-mon[32129]: pgmap v6411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:11:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:30 smithi160 ceph-mon[36715]: pgmap v6411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:11:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11134.784824 secs 2024-07-26T14:11:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11134.784824 secs 2024-07-26T14:11:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11134.784824 secs 2024-07-26T14:11:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:32 smithi144 ceph-mon[32129]: pgmap v6412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:11:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:32 smithi160 ceph-mon[36715]: pgmap v6412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:11:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:32 smithi183 ceph-mon[37273]: pgmap v6412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:11:35.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:34 smithi183 ceph-mon[37273]: pgmap v6413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:11:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:34 smithi144 ceph-mon[32129]: pgmap v6413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:11:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:34 smithi160 ceph-mon[36715]: pgmap v6413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:11:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:36.680+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:11:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:36.681+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:11:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:36.681+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:11:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:36.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:11:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:36.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11139.784968 secs 2024-07-26T14:11:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:36 smithi144 ceph-mon[32129]: pgmap v6414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:11:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11139.784968 secs 2024-07-26T14:11:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:36 smithi160 ceph-mon[36715]: pgmap v6414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:11:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11139.784968 secs 2024-07-26T14:11:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:36 smithi183 ceph-mon[37273]: pgmap v6414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:11:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:38 smithi144 ceph-mon[32129]: pgmap v6415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:38 smithi160 ceph-mon[36715]: pgmap v6415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:38 smithi183 ceph-mon[37273]: pgmap v6415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:40 smithi144 ceph-mon[32129]: pgmap v6416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:41.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:40 smithi160 ceph-mon[36715]: pgmap v6416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:40 smithi183 ceph-mon[37273]: pgmap v6416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11144.785107 secs 2024-07-26T14:11:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11144.785107 secs 2024-07-26T14:11:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11144.785107 secs 2024-07-26T14:11:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:42 smithi144 ceph-mon[32129]: pgmap v6417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:42 smithi160 ceph-mon[36715]: pgmap v6417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:42 smithi183 ceph-mon[37273]: pgmap v6417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:44 smithi144 ceph-mon[32129]: pgmap v6418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:44 smithi160 ceph-mon[36715]: pgmap v6418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:44 smithi183 ceph-mon[37273]: pgmap v6418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11149.785228 secs 2024-07-26T14:11:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:46 smithi144 ceph-mon[32129]: pgmap v6419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11149.785228 secs 2024-07-26T14:11:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:46 smithi160 ceph-mon[36715]: pgmap v6419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11149.785228 secs 2024-07-26T14:11:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:46 smithi183 ceph-mon[37273]: pgmap v6419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:48.071+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:11:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:48.071+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:11:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:48.072+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T14:11:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:48.072+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:11:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:48 smithi144 ceph-mon[32129]: pgmap v6420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:48 smithi160 ceph-mon[36715]: pgmap v6420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:48 smithi183 ceph-mon[37273]: pgmap v6420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:50 smithi144 ceph-mon[32129]: pgmap v6421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:50 smithi160 ceph-mon[36715]: pgmap v6421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:50 smithi183 ceph-mon[37273]: pgmap v6421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:51.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:11:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:51.699+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:11:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:51.699+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:11:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:51.699+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:11:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:11:51.699+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11154.785357 secs 2024-07-26T14:11:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11154.785357 secs 2024-07-26T14:11:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11154.785357 secs 2024-07-26T14:11:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:52 smithi160 ceph-mon[36715]: pgmap v6422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:52 smithi183 ceph-mon[37273]: pgmap v6422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:52 smithi144 ceph-mon[32129]: pgmap v6422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:11:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:54 smithi144 ceph-mon[32129]: pgmap v6423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:54 smithi160 ceph-mon[36715]: pgmap v6423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:54 smithi183 ceph-mon[37273]: pgmap v6423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11159.785441 secs 2024-07-26T14:11:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:56 smithi160 ceph-mon[36715]: pgmap v6424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11159.785441 secs 2024-07-26T14:11:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:56 smithi183 ceph-mon[37273]: pgmap v6424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11159.785441 secs 2024-07-26T14:11:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:56 smithi144 ceph-mon[32129]: pgmap v6424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:11:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:11:58 smithi160 ceph-mon[36715]: pgmap v6425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:11:58 smithi183 ceph-mon[37273]: pgmap v6425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:11:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:11:58 smithi144 ceph-mon[32129]: pgmap v6425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:00 smithi160 ceph-mon[36715]: pgmap v6426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:00 smithi183 ceph-mon[37273]: pgmap v6426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:00 smithi144 ceph-mon[32129]: pgmap v6426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11164.785587 secs 2024-07-26T14:12:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11164.785587 secs 2024-07-26T14:12:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11164.785587 secs 2024-07-26T14:12:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:03 smithi160 ceph-mon[36715]: pgmap v6427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:03 smithi183 ceph-mon[37273]: pgmap v6427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:03 smithi144 ceph-mon[32129]: pgmap v6427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:05 smithi160 ceph-mon[36715]: pgmap v6428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:05 smithi183 ceph-mon[37273]: pgmap v6428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:05 smithi144 ceph-mon[32129]: pgmap v6428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:06.715+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:12:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:06.716+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:12:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:06.716+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:12:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:06.716+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:12:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:06.716+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:07.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11169.785704 secs 2024-07-26T14:12:07.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:07 smithi144 ceph-mon[32129]: pgmap v6429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11169.785704 secs 2024-07-26T14:12:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:07 smithi160 ceph-mon[36715]: pgmap v6429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11169.785704 secs 2024-07-26T14:12:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:07 smithi183 ceph-mon[37273]: pgmap v6429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:09 smithi160 ceph-mon[36715]: pgmap v6430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:09 smithi183 ceph-mon[37273]: pgmap v6430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:09 smithi144 ceph-mon[32129]: pgmap v6430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:11 smithi160 ceph-mon[36715]: pgmap v6431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:11 smithi183 ceph-mon[37273]: pgmap v6431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:11 smithi144 ceph-mon[32129]: pgmap v6431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:12:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11174.785831 secs 2024-07-26T14:12:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11174.785831 secs 2024-07-26T14:12:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11174.785831 secs 2024-07-26T14:12:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:13 smithi160 ceph-mon[36715]: pgmap v6432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:13 smithi183 ceph-mon[37273]: pgmap v6432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:13 smithi144 ceph-mon[32129]: pgmap v6432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:15 smithi160 ceph-mon[36715]: pgmap v6433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:15 smithi183 ceph-mon[37273]: pgmap v6433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:15 smithi144 ceph-mon[32129]: pgmap v6433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11179.785955 secs 2024-07-26T14:12:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:17 smithi183 ceph-mon[37273]: pgmap v6434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11179.785955 secs 2024-07-26T14:12:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:17 smithi144 ceph-mon[32129]: pgmap v6434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11179.785955 secs 2024-07-26T14:12:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:17 smithi160 ceph-mon[36715]: pgmap v6434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:19.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:19 smithi183 ceph-mon[37273]: pgmap v6435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:19 smithi144 ceph-mon[32129]: pgmap v6435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:19 smithi160 ceph-mon[36715]: pgmap v6435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:21 smithi183 ceph-mon[37273]: pgmap v6436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:21 smithi144 ceph-mon[32129]: pgmap v6436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:21 smithi160 ceph-mon[36715]: pgmap v6436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:21.732+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:12:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:21.733+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:12:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:21.733+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:12:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:21.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:12:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:21.733+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11184.786062 secs 2024-07-26T14:12:22.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11184.786062 secs 2024-07-26T14:12:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11184.786062 secs 2024-07-26T14:12:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:23 smithi183 ceph-mon[37273]: pgmap v6437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:23 smithi144 ceph-mon[32129]: pgmap v6437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:23 smithi160 ceph-mon[36715]: pgmap v6437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:25 smithi183 ceph-mon[37273]: pgmap v6438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:25 smithi144 ceph-mon[32129]: pgmap v6438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:25 smithi160 ceph-mon[36715]: pgmap v6438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:26.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11189.786179 secs 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: pgmap v6439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:27.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11189.786179 secs 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: pgmap v6439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:27.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:12:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11189.786179 secs 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: pgmap v6439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:12:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:28 smithi183 ceph-mon[37273]: pgmap v6440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:12:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:28 smithi183 ceph-mon[37273]: pgmap v6441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:12:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:28 smithi144 ceph-mon[32129]: pgmap v6440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:12:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:28 smithi144 ceph-mon[32129]: pgmap v6441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:12:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:28 smithi160 ceph-mon[36715]: pgmap v6440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:12:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:28 smithi160 ceph-mon[36715]: pgmap v6441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:12:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:30 smithi183 ceph-mon[37273]: pgmap v6442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:12:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:30 smithi144 ceph-mon[32129]: pgmap v6442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:12:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:30 smithi160 ceph-mon[36715]: pgmap v6442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:12:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11194.786321 secs 2024-07-26T14:12:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:32 smithi183 ceph-mon[37273]: pgmap v6443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:12:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11194.786321 secs 2024-07-26T14:12:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:32 smithi144 ceph-mon[32129]: pgmap v6443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:12:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11194.786321 secs 2024-07-26T14:12:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:32 smithi160 ceph-mon[36715]: pgmap v6443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:12:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:34 smithi183 ceph-mon[37273]: pgmap v6444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:34 smithi144 ceph-mon[32129]: pgmap v6444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:34 smithi160 ceph-mon[36715]: pgmap v6444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:36 smithi144 ceph-mon[32129]: pgmap v6445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:36 smithi160 ceph-mon[36715]: pgmap v6445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:36 smithi183 ceph-mon[37273]: pgmap v6445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:12:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:36.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:12:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:36.739+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:12:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:36.739+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:12:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:36.739+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:12:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:36.739+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11199.786443 secs 2024-07-26T14:12:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11199.786443 secs 2024-07-26T14:12:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11199.786443 secs 2024-07-26T14:12:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:38 smithi144 ceph-mon[32129]: pgmap v6446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:38 smithi160 ceph-mon[36715]: pgmap v6446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:38 smithi183 ceph-mon[37273]: pgmap v6446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:12:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:40 smithi144 ceph-mon[32129]: pgmap v6447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:40 smithi160 ceph-mon[36715]: pgmap v6447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:40 smithi183 ceph-mon[37273]: pgmap v6447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11204.786561 secs 2024-07-26T14:12:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:42 smithi144 ceph-mon[32129]: pgmap v6448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11204.786561 secs 2024-07-26T14:12:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:42 smithi160 ceph-mon[36715]: pgmap v6448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11204.786561 secs 2024-07-26T14:12:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:42 smithi183 ceph-mon[37273]: pgmap v6448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:44 smithi144 ceph-mon[32129]: pgmap v6449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:44 smithi160 ceph-mon[36715]: pgmap v6449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:44 smithi183 ceph-mon[37273]: pgmap v6449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:46 smithi144 ceph-mon[32129]: pgmap v6450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:46 smithi160 ceph-mon[36715]: pgmap v6450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:46 smithi183 ceph-mon[37273]: pgmap v6450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11209.786686 secs 2024-07-26T14:12:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11209.786686 secs 2024-07-26T14:12:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11209.786686 secs 2024-07-26T14:12:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:48 smithi144 ceph-mon[32129]: pgmap v6451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:48 smithi160 ceph-mon[36715]: pgmap v6451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:48 smithi183 ceph-mon[37273]: pgmap v6451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:50 smithi144 ceph-mon[32129]: pgmap v6452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:50 smithi160 ceph-mon[36715]: pgmap v6452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:50 smithi183 ceph-mon[37273]: pgmap v6452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:51.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:51.756+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:12:51.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:51.756+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:12:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:51.756+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:12:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:51.756+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:12:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:12:51.757+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11214.786815 secs 2024-07-26T14:12:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:52 smithi144 ceph-mon[32129]: pgmap v6453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11214.786815 secs 2024-07-26T14:12:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:52 smithi160 ceph-mon[36715]: pgmap v6453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11214.786815 secs 2024-07-26T14:12:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:52 smithi183 ceph-mon[37273]: pgmap v6453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:54 smithi144 ceph-mon[32129]: pgmap v6454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:54 smithi160 ceph-mon[36715]: pgmap v6454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:54 smithi183 ceph-mon[37273]: pgmap v6454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:12:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:56 smithi144 ceph-mon[32129]: pgmap v6455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:56 smithi160 ceph-mon[36715]: pgmap v6455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:56 smithi183 ceph-mon[37273]: pgmap v6455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:12:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11219.786940 secs 2024-07-26T14:12:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11219.786940 secs 2024-07-26T14:12:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11219.786940 secs 2024-07-26T14:12:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:12:58 smithi160 ceph-mon[36715]: pgmap v6456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:12:58 smithi183 ceph-mon[37273]: pgmap v6456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:12:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:12:58 smithi144 ceph-mon[32129]: pgmap v6456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:00 smithi160 ceph-mon[36715]: pgmap v6457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:00 smithi183 ceph-mon[37273]: pgmap v6457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:00 smithi144 ceph-mon[32129]: pgmap v6457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11224.787075 secs 2024-07-26T14:13:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:02 smithi160 ceph-mon[36715]: pgmap v6458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11224.787075 secs 2024-07-26T14:13:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:02 smithi183 ceph-mon[37273]: pgmap v6458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11224.787075 secs 2024-07-26T14:13:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:02 smithi144 ceph-mon[32129]: pgmap v6458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:04 smithi160 ceph-mon[36715]: pgmap v6459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:04 smithi183 ceph-mon[37273]: pgmap v6459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:04 smithi144 ceph-mon[32129]: pgmap v6459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:06 smithi160 ceph-mon[36715]: pgmap v6460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:06 smithi183 ceph-mon[37273]: pgmap v6460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:06 smithi144 ceph-mon[32129]: pgmap v6460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:06.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:13:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:06.773+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:13:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:06.773+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:13:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:06.773+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:13:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:06.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11229.787240 secs 2024-07-26T14:13:07.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11229.787240 secs 2024-07-26T14:13:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11229.787240 secs 2024-07-26T14:13:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:08 smithi160 ceph-mon[36715]: pgmap v6461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:08 smithi183 ceph-mon[37273]: pgmap v6461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:08 smithi144 ceph-mon[32129]: pgmap v6461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:10 smithi160 ceph-mon[36715]: pgmap v6462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:10 smithi183 ceph-mon[37273]: pgmap v6462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:10 smithi144 ceph-mon[32129]: pgmap v6462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11234.787371 secs 2024-07-26T14:13:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:12 smithi160 ceph-mon[36715]: pgmap v6463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11234.787371 secs 2024-07-26T14:13:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:12 smithi183 ceph-mon[37273]: pgmap v6463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11234.787371 secs 2024-07-26T14:13:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:12 smithi144 ceph-mon[32129]: pgmap v6463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:14 smithi160 ceph-mon[36715]: pgmap v6464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:14 smithi183 ceph-mon[37273]: pgmap v6464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:14 smithi144 ceph-mon[32129]: pgmap v6464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:16 smithi160 ceph-mon[36715]: pgmap v6465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:16 smithi183 ceph-mon[37273]: pgmap v6465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:16 smithi144 ceph-mon[32129]: pgmap v6465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11239.787456 secs 2024-07-26T14:13:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11239.787456 secs 2024-07-26T14:13:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11239.787456 secs 2024-07-26T14:13:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:18 smithi183 ceph-mon[37273]: pgmap v6466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:18 smithi144 ceph-mon[32129]: pgmap v6466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:18 smithi160 ceph-mon[36715]: pgmap v6466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:20 smithi183 ceph-mon[37273]: pgmap v6467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:20 smithi144 ceph-mon[32129]: pgmap v6467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:20 smithi160 ceph-mon[36715]: pgmap v6467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11244.787579 secs 2024-07-26T14:13:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11244.787579 secs 2024-07-26T14:13:21.787 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:21.786+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:13:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:21.787+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:13:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:21.787+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:13:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:21.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:13:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:21.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11244.787579 secs 2024-07-26T14:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:22 smithi183 ceph-mon[37273]: pgmap v6468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:22.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:22 smithi160 ceph-mon[36715]: pgmap v6468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:22 smithi144 ceph-mon[32129]: pgmap v6468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:24 smithi183 ceph-mon[37273]: pgmap v6469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:24 smithi144 ceph-mon[32129]: pgmap v6469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:24 smithi160 ceph-mon[36715]: pgmap v6469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:26 smithi183 ceph-mon[37273]: pgmap v6470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11249.787709 secs 2024-07-26T14:13:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:26 smithi144 ceph-mon[32129]: pgmap v6470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11249.787709 secs 2024-07-26T14:13:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:26 smithi160 ceph-mon[36715]: pgmap v6470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11249.787709 secs 2024-07-26T14:13:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:27.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:28 smithi183 ceph-mon[37273]: pgmap v6471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:13:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:13:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:13:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:28 smithi144 ceph-mon[32129]: pgmap v6471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:13:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:13:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:13:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:28 smithi160 ceph-mon[36715]: pgmap v6471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:13:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:13:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:13:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:29 smithi183 ceph-mon[37273]: pgmap v6472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:13:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:29 smithi183 ceph-mon[37273]: pgmap v6473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:13:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:29 smithi144 ceph-mon[32129]: pgmap v6472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:13:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:29 smithi144 ceph-mon[32129]: pgmap v6473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:13:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:29 smithi160 ceph-mon[36715]: pgmap v6472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:13:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:29 smithi160 ceph-mon[36715]: pgmap v6473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:13:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:31 smithi183 ceph-mon[37273]: pgmap v6474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11254.787859 secs 2024-07-26T14:13:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:31 smithi144 ceph-mon[32129]: pgmap v6474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11254.787859 secs 2024-07-26T14:13:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:31 smithi160 ceph-mon[36715]: pgmap v6474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11254.787859 secs 2024-07-26T14:13:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:33 smithi183 ceph-mon[37273]: pgmap v6475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:33 smithi144 ceph-mon[32129]: pgmap v6475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:33 smithi160 ceph-mon[36715]: pgmap v6475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:35 smithi183 ceph-mon[37273]: pgmap v6476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:35 smithi144 ceph-mon[32129]: pgmap v6476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:35 smithi160 ceph-mon[36715]: pgmap v6476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11259.787931 secs 2024-07-26T14:13:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:36.795+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:13:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:36.796+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:13:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:36.796+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:13:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:36.796+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:13:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:36.796+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:36.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11259.787931 secs 2024-07-26T14:13:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11259.787931 secs 2024-07-26T14:13:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:37 smithi144 ceph-mon[32129]: pgmap v6477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:37 smithi160 ceph-mon[36715]: pgmap v6477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:37 smithi183 ceph-mon[37273]: pgmap v6477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:13:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:39 smithi144 ceph-mon[32129]: pgmap v6478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:13:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:39 smithi160 ceph-mon[36715]: pgmap v6478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:13:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:39 smithi183 ceph-mon[37273]: pgmap v6478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:13:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:41 smithi144 ceph-mon[32129]: pgmap v6479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:13:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11264.788034 secs 2024-07-26T14:13:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:41 smithi160 ceph-mon[36715]: pgmap v6479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:13:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11264.788034 secs 2024-07-26T14:13:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:41 smithi183 ceph-mon[37273]: pgmap v6479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:13:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11264.788034 secs 2024-07-26T14:13:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:43 smithi144 ceph-mon[32129]: pgmap v6480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:43 smithi160 ceph-mon[36715]: pgmap v6480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:43 smithi183 ceph-mon[37273]: pgmap v6480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:45 smithi144 ceph-mon[32129]: pgmap v6481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:45 smithi160 ceph-mon[36715]: pgmap v6481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:45 smithi183 ceph-mon[37273]: pgmap v6481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11269.788182 secs 2024-07-26T14:13:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11269.788182 secs 2024-07-26T14:13:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11269.788182 secs 2024-07-26T14:13:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:47 smithi144 ceph-mon[32129]: pgmap v6482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:47 smithi160 ceph-mon[36715]: pgmap v6482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:47 smithi183 ceph-mon[37273]: pgmap v6482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:49 smithi144 ceph-mon[32129]: pgmap v6483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:49 smithi160 ceph-mon[36715]: pgmap v6483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:49 smithi183 ceph-mon[37273]: pgmap v6483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:51 smithi144 ceph-mon[32129]: pgmap v6484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:13:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11274.788330 secs 2024-07-26T14:13:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:51 smithi160 ceph-mon[36715]: pgmap v6484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:13:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11274.788330 secs 2024-07-26T14:13:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:51.810+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:13:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:51.811+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:13:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:51.811+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:13:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:51.811+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:13:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:13:51.811+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:51 smithi183 ceph-mon[37273]: pgmap v6484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:13:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11274.788330 secs 2024-07-26T14:13:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:53 smithi144 ceph-mon[32129]: pgmap v6485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:53 smithi160 ceph-mon[36715]: pgmap v6485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:53 smithi183 ceph-mon[37273]: pgmap v6485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:55 smithi144 ceph-mon[32129]: pgmap v6486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:55 smithi160 ceph-mon[36715]: pgmap v6486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:55 smithi183 ceph-mon[37273]: pgmap v6486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:13:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11279.788396 secs 2024-07-26T14:13:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11279.788396 secs 2024-07-26T14:13:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11279.788396 secs 2024-07-26T14:13:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:57 smithi144 ceph-mon[32129]: pgmap v6487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:57 smithi160 ceph-mon[36715]: pgmap v6487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:57 smithi183 ceph-mon[37273]: pgmap v6487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:13:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:13:59 smithi144 ceph-mon[32129]: pgmap v6488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:13:59 smithi160 ceph-mon[36715]: pgmap v6488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:13:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:13:59 smithi183 ceph-mon[37273]: pgmap v6488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:01 smithi160 ceph-mon[36715]: pgmap v6489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11284.788467 secs 2024-07-26T14:14:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:01 smithi183 ceph-mon[37273]: pgmap v6489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11284.788467 secs 2024-07-26T14:14:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:01 smithi144 ceph-mon[32129]: pgmap v6489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11284.788467 secs 2024-07-26T14:14:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:03 smithi160 ceph-mon[36715]: pgmap v6490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:03 smithi183 ceph-mon[37273]: pgmap v6490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:03 smithi144 ceph-mon[32129]: pgmap v6490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:05 smithi160 ceph-mon[36715]: pgmap v6491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:05 smithi183 ceph-mon[37273]: pgmap v6491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:05 smithi144 ceph-mon[32129]: pgmap v6491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:06.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11289.788598 secs 2024-07-26T14:14:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11289.788598 secs 2024-07-26T14:14:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:06.826+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:14:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:06.827+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:14:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:06.827+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:14:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:06.827+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:14:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:06.827+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11289.788598 secs 2024-07-26T14:14:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:07 smithi160 ceph-mon[36715]: pgmap v6492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:07.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:07 smithi144 ceph-mon[32129]: pgmap v6492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:07 smithi183 ceph-mon[37273]: pgmap v6492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:09 smithi160 ceph-mon[36715]: pgmap v6493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:09 smithi183 ceph-mon[37273]: pgmap v6493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:09 smithi144 ceph-mon[32129]: pgmap v6493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:11 smithi160 ceph-mon[36715]: pgmap v6494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11294.788686 secs 2024-07-26T14:14:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:11 smithi183 ceph-mon[37273]: pgmap v6494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11294.788686 secs 2024-07-26T14:14:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:11 smithi144 ceph-mon[32129]: pgmap v6494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11294.788686 secs 2024-07-26T14:14:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:13 smithi160 ceph-mon[36715]: pgmap v6495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:13 smithi183 ceph-mon[37273]: pgmap v6495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:13 smithi144 ceph-mon[32129]: pgmap v6495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:15 smithi160 ceph-mon[36715]: pgmap v6496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:15 smithi183 ceph-mon[37273]: pgmap v6496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:15 smithi144 ceph-mon[32129]: pgmap v6496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11299.788846 secs 2024-07-26T14:14:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11299.788846 secs 2024-07-26T14:14:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11299.788846 secs 2024-07-26T14:14:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:17 smithi160 ceph-mon[36715]: pgmap v6497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:17 smithi183 ceph-mon[37273]: pgmap v6497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:17 smithi144 ceph-mon[32129]: pgmap v6497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:19 smithi160 ceph-mon[36715]: pgmap v6498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:19 smithi183 ceph-mon[37273]: pgmap v6498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:19 smithi144 ceph-mon[32129]: pgmap v6498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:21 smithi160 ceph-mon[36715]: pgmap v6499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11304.789049 secs 2024-07-26T14:14:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:21.843+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:14:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:21.844+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:14:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:21.844+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:14:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:21.844+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:14:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:21.844+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:21 smithi183 ceph-mon[37273]: pgmap v6499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11304.789049 secs 2024-07-26T14:14:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:21 smithi144 ceph-mon[32129]: pgmap v6499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:14:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11304.789049 secs 2024-07-26T14:14:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:23 smithi160 ceph-mon[36715]: pgmap v6500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:23 smithi183 ceph-mon[37273]: pgmap v6500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:23 smithi144 ceph-mon[32129]: pgmap v6500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:25 smithi183 ceph-mon[37273]: pgmap v6501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:25 smithi144 ceph-mon[32129]: pgmap v6501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:25 smithi160 ceph-mon[36715]: pgmap v6501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11309.789143 secs 2024-07-26T14:14:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11309.789143 secs 2024-07-26T14:14:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11309.789143 secs 2024-07-26T14:14:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:27 smithi183 ceph-mon[37273]: pgmap v6502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:27 smithi144 ceph-mon[32129]: pgmap v6502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:27 smithi160 ceph-mon[36715]: pgmap v6502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:14:28.968 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:14:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:14:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:29 smithi183 ceph-mon[37273]: pgmap v6503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:29 smithi144 ceph-mon[32129]: pgmap v6503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:29 smithi160 ceph-mon[36715]: pgmap v6503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:14:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:31 smithi183 ceph-mon[37273]: pgmap v6504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:31 smithi183 ceph-mon[37273]: pgmap v6505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:14:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11314.789271 secs 2024-07-26T14:14:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:31 smithi144 ceph-mon[32129]: pgmap v6504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:31 smithi144 ceph-mon[32129]: pgmap v6505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:14:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11314.789271 secs 2024-07-26T14:14:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:31 smithi160 ceph-mon[36715]: pgmap v6504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:31 smithi160 ceph-mon[36715]: pgmap v6505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:14:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11314.789271 secs 2024-07-26T14:14:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:32 smithi183 ceph-mon[37273]: pgmap v6506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:32 smithi144 ceph-mon[32129]: pgmap v6506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:32 smithi160 ceph-mon[36715]: pgmap v6506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:35 smithi160 ceph-mon[36715]: pgmap v6507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:35 smithi183 ceph-mon[37273]: pgmap v6507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:35 smithi144 ceph-mon[32129]: pgmap v6507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:36.859+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:14:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:36.860+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:14:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:36.860+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:14:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:36.860+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:14:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:36.860+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11319.789318 secs 2024-07-26T14:14:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:37 smithi183 ceph-mon[37273]: pgmap v6508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11319.789318 secs 2024-07-26T14:14:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:37 smithi144 ceph-mon[32129]: pgmap v6508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11319.789318 secs 2024-07-26T14:14:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:37 smithi160 ceph-mon[36715]: pgmap v6508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:39 smithi183 ceph-mon[37273]: pgmap v6509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:39 smithi144 ceph-mon[32129]: pgmap v6509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:39 smithi160 ceph-mon[36715]: pgmap v6509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:41 smithi183 ceph-mon[37273]: pgmap v6510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:14:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:41 smithi144 ceph-mon[32129]: pgmap v6510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:14:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:41 smithi160 ceph-mon[36715]: pgmap v6510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:14:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11324.789446 secs 2024-07-26T14:14:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11324.789446 secs 2024-07-26T14:14:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11324.789446 secs 2024-07-26T14:14:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:43 smithi183 ceph-mon[37273]: pgmap v6511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:43 smithi144 ceph-mon[32129]: pgmap v6511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:43 smithi160 ceph-mon[36715]: pgmap v6511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:45 smithi144 ceph-mon[32129]: pgmap v6512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:45.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:45 smithi183 ceph-mon[37273]: pgmap v6512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:45 smithi160 ceph-mon[36715]: pgmap v6512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11329.789523 secs 2024-07-26T14:14:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:47 smithi183 ceph-mon[37273]: pgmap v6513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11329.789523 secs 2024-07-26T14:14:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:47 smithi144 ceph-mon[32129]: pgmap v6513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11329.789523 secs 2024-07-26T14:14:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:47 smithi160 ceph-mon[36715]: pgmap v6513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:48.104 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:48.103+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:14:48.104 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:48.103+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:14:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:48.107+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T14:14:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:48.108+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:14:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:49 smithi183 ceph-mon[37273]: pgmap v6514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:49 smithi144 ceph-mon[32129]: pgmap v6514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:49 smithi160 ceph-mon[36715]: pgmap v6514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:51 smithi183 ceph-mon[37273]: pgmap v6515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:51 smithi144 ceph-mon[32129]: pgmap v6515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:51 smithi160 ceph-mon[36715]: pgmap v6515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:51.877+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:14:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:51.878+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:14:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:51.878+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:14:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:51.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:14:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:14:51.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11334.789600 secs 2024-07-26T14:14:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11334.789600 secs 2024-07-26T14:14:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11334.789600 secs 2024-07-26T14:14:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:53 smithi183 ceph-mon[37273]: pgmap v6516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:53.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:53 smithi160 ceph-mon[36715]: pgmap v6516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:53 smithi144 ceph-mon[32129]: pgmap v6516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:14:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:55 smithi183 ceph-mon[37273]: pgmap v6517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:55 smithi144 ceph-mon[32129]: pgmap v6517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:55 smithi160 ceph-mon[36715]: pgmap v6517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11339.789720 secs 2024-07-26T14:14:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:57 smithi144 ceph-mon[32129]: pgmap v6518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11339.789720 secs 2024-07-26T14:14:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:57 smithi160 ceph-mon[36715]: pgmap v6518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11339.789720 secs 2024-07-26T14:14:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:57 smithi183 ceph-mon[37273]: pgmap v6518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:14:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:14:59 smithi144 ceph-mon[32129]: pgmap v6519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:14:59 smithi160 ceph-mon[36715]: pgmap v6519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:14:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:14:59 smithi183 ceph-mon[37273]: pgmap v6519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:01 smithi144 ceph-mon[32129]: pgmap v6520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:01 smithi160 ceph-mon[36715]: pgmap v6520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:01 smithi183 ceph-mon[37273]: pgmap v6520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11344.789858 secs 2024-07-26T14:15:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11344.789858 secs 2024-07-26T14:15:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11344.789858 secs 2024-07-26T14:15:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:03 smithi144 ceph-mon[32129]: pgmap v6521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:03 smithi160 ceph-mon[36715]: pgmap v6521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:03 smithi183 ceph-mon[37273]: pgmap v6521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:05 smithi144 ceph-mon[32129]: pgmap v6522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:05 smithi160 ceph-mon[36715]: pgmap v6522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:05 smithi183 ceph-mon[37273]: pgmap v6522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:06.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:15:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:06.895+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:15:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:06.895+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:15:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:06.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:15:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:06.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:07.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11349.789958 secs 2024-07-26T14:15:07.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:07 smithi183 ceph-mon[37273]: pgmap v6523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11349.789958 secs 2024-07-26T14:15:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:07 smithi144 ceph-mon[32129]: pgmap v6523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11349.789958 secs 2024-07-26T14:15:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:07 smithi160 ceph-mon[36715]: pgmap v6523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:09 smithi144 ceph-mon[32129]: pgmap v6524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:09 smithi160 ceph-mon[36715]: pgmap v6524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:09 smithi183 ceph-mon[37273]: pgmap v6524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:11 smithi144 ceph-mon[32129]: pgmap v6525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:11 smithi160 ceph-mon[36715]: pgmap v6525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:11 smithi183 ceph-mon[37273]: pgmap v6525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11354.790103 secs 2024-07-26T14:15:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11354.790103 secs 2024-07-26T14:15:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11354.790103 secs 2024-07-26T14:15:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:13 smithi144 ceph-mon[32129]: pgmap v6526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:13 smithi160 ceph-mon[36715]: pgmap v6526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:13 smithi183 ceph-mon[37273]: pgmap v6526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:15 smithi144 ceph-mon[32129]: pgmap v6527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:15 smithi160 ceph-mon[36715]: pgmap v6527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:15 smithi183 ceph-mon[37273]: pgmap v6527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11359.790243 secs 2024-07-26T14:15:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:17 smithi144 ceph-mon[32129]: pgmap v6528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11359.790243 secs 2024-07-26T14:15:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:17 smithi160 ceph-mon[36715]: pgmap v6528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11359.790243 secs 2024-07-26T14:15:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:17 smithi183 ceph-mon[37273]: pgmap v6528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:19 smithi144 ceph-mon[32129]: pgmap v6529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:19 smithi160 ceph-mon[36715]: pgmap v6529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:19 smithi183 ceph-mon[37273]: pgmap v6529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:21 smithi144 ceph-mon[32129]: pgmap v6530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:21 smithi160 ceph-mon[36715]: pgmap v6530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:21 smithi183 ceph-mon[37273]: pgmap v6530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:21.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:15:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:21.911+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:15:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:21.911+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:15:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:21.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:15:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:21.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11364.790408 secs 2024-07-26T14:15:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11364.790408 secs 2024-07-26T14:15:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11364.790408 secs 2024-07-26T14:15:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:23 smithi160 ceph-mon[36715]: pgmap v6531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:23 smithi183 ceph-mon[37273]: pgmap v6531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:23 smithi144 ceph-mon[32129]: pgmap v6531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:25 smithi160 ceph-mon[36715]: pgmap v6532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:25 smithi183 ceph-mon[37273]: pgmap v6532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:25 smithi144 ceph-mon[32129]: pgmap v6532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11369.790457 secs 2024-07-26T14:15:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:26 smithi144 ceph-mon[32129]: pgmap v6533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11369.790457 secs 2024-07-26T14:15:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:26 smithi160 ceph-mon[36715]: pgmap v6533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11369.790457 secs 2024-07-26T14:15:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:26 smithi183 ceph-mon[37273]: pgmap v6533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:29 smithi183 ceph-mon[37273]: pgmap v6534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:29 smithi144 ceph-mon[32129]: pgmap v6534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:29 smithi160 ceph-mon[36715]: pgmap v6534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:31.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:31 smithi183 ceph-mon[37273]: pgmap v6535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:31.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:31.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.424 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:31 smithi160 ceph-mon[36715]: pgmap v6535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:31.424 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:31.424 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.424 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.424 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.424 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:31 smithi144 ceph-mon[32129]: pgmap v6535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:31.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:31.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:31.461 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11374.790522 secs 2024-07-26T14:15:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11374.790522 secs 2024-07-26T14:15:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11374.790522 secs 2024-07-26T14:15:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:33.110+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:15:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:33.110+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:15:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:33.111+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:15:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:33.111+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T14:15:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:33 smithi144 ceph-mon[32129]: pgmap v6536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:33 smithi160 ceph-mon[36715]: pgmap v6536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:33 smithi183 ceph-mon[37273]: pgmap v6536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:34 smithi183 ceph-mon[37273]: pgmap v6537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:34 smithi144 ceph-mon[32129]: pgmap v6537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:34 smithi160 ceph-mon[36715]: pgmap v6537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:35 smithi183 ceph-mon[37273]: pgmap v6538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:15:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:35 smithi183 ceph-mon[37273]: pgmap v6539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:35 smithi144 ceph-mon[32129]: pgmap v6538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:15:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:35 smithi144 ceph-mon[32129]: pgmap v6539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:35 smithi160 ceph-mon[36715]: pgmap v6538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:15:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:35 smithi160 ceph-mon[36715]: pgmap v6539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:15:36.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:36.918+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:15:36.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:36.919+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:15:36.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:36.919+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:15:36.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:36.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:15:36.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:36.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:37.120 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11379.790655 secs 2024-07-26T14:15:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11379.790655 secs 2024-07-26T14:15:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11379.790655 secs 2024-07-26T14:15:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:37 smithi183 ceph-mon[37273]: pgmap v6540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:15:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:37 smithi144 ceph-mon[32129]: pgmap v6540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:15:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:37 smithi160 ceph-mon[36715]: pgmap v6540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:15:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:39 smithi144 ceph-mon[32129]: pgmap v6541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T14:15:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:39 smithi160 ceph-mon[36715]: pgmap v6541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T14:15:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:39 smithi183 ceph-mon[37273]: pgmap v6541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T14:15:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11384.790787 secs 2024-07-26T14:15:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:41 smithi144 ceph-mon[32129]: pgmap v6542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:15:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11384.790787 secs 2024-07-26T14:15:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:41 smithi160 ceph-mon[36715]: pgmap v6542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:15:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11384.790787 secs 2024-07-26T14:15:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:41 smithi183 ceph-mon[37273]: pgmap v6542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:15:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:43 smithi144 ceph-mon[32129]: pgmap v6543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:43 smithi160 ceph-mon[36715]: pgmap v6543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:43 smithi183 ceph-mon[37273]: pgmap v6543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:45 smithi144 ceph-mon[32129]: pgmap v6544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:15:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:45 smithi160 ceph-mon[36715]: pgmap v6544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:15:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:45 smithi183 ceph-mon[37273]: pgmap v6544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:15:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11389.790917 secs 2024-07-26T14:15:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11389.790917 secs 2024-07-26T14:15:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11389.790917 secs 2024-07-26T14:15:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:47 smithi144 ceph-mon[32129]: pgmap v6545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:47 smithi160 ceph-mon[36715]: pgmap v6545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:47 smithi183 ceph-mon[37273]: pgmap v6545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:49 smithi144 ceph-mon[32129]: pgmap v6546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:49 smithi160 ceph-mon[36715]: pgmap v6546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:49 smithi183 ceph-mon[37273]: pgmap v6546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:51.935+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:15:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:51.936+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:15:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:51.936+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:15:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:51.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:15:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:15:51.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11394.791064 secs 2024-07-26T14:15:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:51 smithi144 ceph-mon[32129]: pgmap v6547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11394.791064 secs 2024-07-26T14:15:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:51 smithi160 ceph-mon[36715]: pgmap v6547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11394.791064 secs 2024-07-26T14:15:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:51 smithi183 ceph-mon[37273]: pgmap v6547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:53 smithi160 ceph-mon[36715]: pgmap v6548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:53 smithi183 ceph-mon[37273]: pgmap v6548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:53 smithi144 ceph-mon[32129]: pgmap v6548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:15:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:55 smithi160 ceph-mon[36715]: pgmap v6549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:55 smithi183 ceph-mon[37273]: pgmap v6549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:55 smithi144 ceph-mon[32129]: pgmap v6549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:15:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11399.791162 secs 2024-07-26T14:15:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11399.791162 secs 2024-07-26T14:15:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11399.791162 secs 2024-07-26T14:15:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:57 smithi160 ceph-mon[36715]: pgmap v6550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:57 smithi183 ceph-mon[37273]: pgmap v6550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:15:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:57 smithi144 ceph-mon[32129]: pgmap v6550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:15:59 smithi160 ceph-mon[36715]: pgmap v6551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:15:59 smithi183 ceph-mon[37273]: pgmap v6551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:15:59 smithi144 ceph-mon[32129]: pgmap v6551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11404.791280 secs 2024-07-26T14:16:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:01 smithi160 ceph-mon[36715]: pgmap v6552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11404.791280 secs 2024-07-26T14:16:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:01 smithi183 ceph-mon[37273]: pgmap v6552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:02.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11404.791280 secs 2024-07-26T14:16:02.205 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:01 smithi144 ceph-mon[32129]: pgmap v6552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:03 smithi144 ceph-mon[32129]: pgmap v6553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:03 smithi160 ceph-mon[36715]: pgmap v6553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:03 smithi183 ceph-mon[37273]: pgmap v6553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:05 smithi144 ceph-mon[32129]: pgmap v6554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:05 smithi160 ceph-mon[36715]: pgmap v6554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:05 smithi183 ceph-mon[37273]: pgmap v6554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:06.951+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:16:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:06.952+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:16:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:06.952+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:16:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:06.952+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:16:06.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:06.952+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11409.791379 secs 2024-07-26T14:16:07.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11409.791379 secs 2024-07-26T14:16:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11409.791379 secs 2024-07-26T14:16:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:07 smithi144 ceph-mon[32129]: pgmap v6555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:07 smithi160 ceph-mon[36715]: pgmap v6555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:07 smithi183 ceph-mon[37273]: pgmap v6555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:09 smithi144 ceph-mon[32129]: pgmap v6556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:09 smithi160 ceph-mon[36715]: pgmap v6556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:09 smithi183 ceph-mon[37273]: pgmap v6556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11414.791494 secs 2024-07-26T14:16:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:11 smithi144 ceph-mon[32129]: pgmap v6557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11414.791494 secs 2024-07-26T14:16:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:11 smithi160 ceph-mon[36715]: pgmap v6557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11414.791494 secs 2024-07-26T14:16:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:11 smithi183 ceph-mon[37273]: pgmap v6557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:13 smithi160 ceph-mon[36715]: pgmap v6558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:13 smithi183 ceph-mon[37273]: pgmap v6558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:13 smithi144 ceph-mon[32129]: pgmap v6558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:15 smithi160 ceph-mon[36715]: pgmap v6559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:15 smithi183 ceph-mon[37273]: pgmap v6559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:15 smithi144 ceph-mon[32129]: pgmap v6559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11419.791645 secs 2024-07-26T14:16:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11419.791645 secs 2024-07-26T14:16:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11419.791645 secs 2024-07-26T14:16:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:17 smithi160 ceph-mon[36715]: pgmap v6560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:17 smithi183 ceph-mon[37273]: pgmap v6560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:17 smithi144 ceph-mon[32129]: pgmap v6560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:19 smithi160 ceph-mon[36715]: pgmap v6561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:19 smithi183 ceph-mon[37273]: pgmap v6561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:19 smithi144 ceph-mon[32129]: pgmap v6561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:21.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:21.968+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:16:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:21.969+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:16:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:21.969+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:16:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:21.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:16:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:21.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11424.791781 secs 2024-07-26T14:16:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:22 smithi160 ceph-mon[36715]: pgmap v6562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11424.791781 secs 2024-07-26T14:16:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:22 smithi183 ceph-mon[37273]: pgmap v6562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11424.791781 secs 2024-07-26T14:16:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:22 smithi144 ceph-mon[32129]: pgmap v6562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:24 smithi160 ceph-mon[36715]: pgmap v6563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:24 smithi183 ceph-mon[37273]: pgmap v6563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:24 smithi144 ceph-mon[32129]: pgmap v6563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:26 smithi160 ceph-mon[36715]: pgmap v6564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:26 smithi183 ceph-mon[37273]: pgmap v6564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:26 smithi144 ceph-mon[32129]: pgmap v6564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11429.791914 secs 2024-07-26T14:16:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11429.791914 secs 2024-07-26T14:16:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11429.791914 secs 2024-07-26T14:16:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:28 smithi160 ceph-mon[36715]: pgmap v6565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:28 smithi183 ceph-mon[37273]: pgmap v6565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:28 smithi144 ceph-mon[32129]: pgmap v6565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:30 smithi160 ceph-mon[36715]: pgmap v6566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:30 smithi183 ceph-mon[37273]: pgmap v6566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:30 smithi144 ceph-mon[32129]: pgmap v6566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11434.792012 secs 2024-07-26T14:16:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:32 smithi160 ceph-mon[36715]: pgmap v6567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11434.792012 secs 2024-07-26T14:16:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:32 smithi183 ceph-mon[37273]: pgmap v6567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11434.792012 secs 2024-07-26T14:16:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:32 smithi144 ceph-mon[32129]: pgmap v6567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:34 smithi144 ceph-mon[32129]: pgmap v6568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:34 smithi183 ceph-mon[37273]: pgmap v6568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:34 smithi160 ceph-mon[36715]: pgmap v6568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:36 smithi183 ceph-mon[37273]: pgmap v6569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:16:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:36 smithi144 ceph-mon[32129]: pgmap v6569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:16:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:36 smithi160 ceph-mon[36715]: pgmap v6569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:16:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:36.985+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:16:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:36.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:16:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:36.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:16:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:36.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:16:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:36.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:37.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11439.792167 secs 2024-07-26T14:16:37.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:37.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:37 smithi144 ceph-mon[32129]: pgmap v6570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:37.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:37 smithi144 ceph-mon[32129]: pgmap v6571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:37.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:16:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11439.792167 secs 2024-07-26T14:16:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:37 smithi183 ceph-mon[37273]: pgmap v6570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:37 smithi183 ceph-mon[37273]: pgmap v6571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:16:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11439.792167 secs 2024-07-26T14:16:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:37 smithi160 ceph-mon[36715]: pgmap v6570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:37 smithi160 ceph-mon[36715]: pgmap v6571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:16:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:39 smithi183 ceph-mon[37273]: pgmap v6572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:16:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:39 smithi144 ceph-mon[32129]: pgmap v6572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:16:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:39 smithi160 ceph-mon[36715]: pgmap v6572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:16:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:41 smithi183 ceph-mon[37273]: pgmap v6573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:16:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:41 smithi144 ceph-mon[32129]: pgmap v6573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:16:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:41 smithi160 ceph-mon[36715]: pgmap v6573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:16:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11444.792273 secs 2024-07-26T14:16:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11444.792273 secs 2024-07-26T14:16:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11444.792273 secs 2024-07-26T14:16:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:43 smithi183 ceph-mon[37273]: pgmap v6574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:43 smithi144 ceph-mon[32129]: pgmap v6574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:43 smithi160 ceph-mon[36715]: pgmap v6574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:45 smithi183 ceph-mon[37273]: pgmap v6575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:16:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:45 smithi144 ceph-mon[32129]: pgmap v6575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:16:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:45 smithi160 ceph-mon[36715]: pgmap v6575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:16:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11449.792415 secs 2024-07-26T14:16:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:47 smithi183 ceph-mon[37273]: pgmap v6576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:16:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11449.792415 secs 2024-07-26T14:16:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:47 smithi144 ceph-mon[32129]: pgmap v6576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:16:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11449.792415 secs 2024-07-26T14:16:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:47 smithi160 ceph-mon[36715]: pgmap v6576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:16:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:49 smithi183 ceph-mon[37273]: pgmap v6577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:49 smithi144 ceph-mon[32129]: pgmap v6577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:49 smithi160 ceph-mon[36715]: pgmap v6577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:51 smithi183 ceph-mon[37273]: pgmap v6578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:51 smithi144 ceph-mon[32129]: pgmap v6578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:51 smithi160 ceph-mon[36715]: pgmap v6578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:51.989+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:16:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:51.990+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:16:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:51.990+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:16:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:51.990+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:16:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:16:51.990+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11454.792502 secs 2024-07-26T14:16:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11454.792502 secs 2024-07-26T14:16:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11454.792502 secs 2024-07-26T14:16:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:53 smithi144 ceph-mon[32129]: pgmap v6579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:53 smithi160 ceph-mon[36715]: pgmap v6579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:53 smithi183 ceph-mon[37273]: pgmap v6579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:16:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:55 smithi144 ceph-mon[32129]: pgmap v6580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:55 smithi160 ceph-mon[36715]: pgmap v6580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:55 smithi183 ceph-mon[37273]: pgmap v6580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11459.792619 secs 2024-07-26T14:16:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:57 smithi144 ceph-mon[32129]: pgmap v6581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11459.792619 secs 2024-07-26T14:16:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:57 smithi160 ceph-mon[36715]: pgmap v6581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11459.792619 secs 2024-07-26T14:16:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:57 smithi183 ceph-mon[37273]: pgmap v6581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:16:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:16:59 smithi144 ceph-mon[32129]: pgmap v6582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:16:59 smithi160 ceph-mon[36715]: pgmap v6582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:16:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:16:59 smithi183 ceph-mon[37273]: pgmap v6582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:01 smithi144 ceph-mon[32129]: pgmap v6583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:01 smithi160 ceph-mon[36715]: pgmap v6583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:01 smithi183 ceph-mon[37273]: pgmap v6583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11464.792738 secs 2024-07-26T14:17:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11464.792738 secs 2024-07-26T14:17:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11464.792738 secs 2024-07-26T14:17:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:03 smithi144 ceph-mon[32129]: pgmap v6584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:03 smithi160 ceph-mon[36715]: pgmap v6584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:03 smithi183 ceph-mon[37273]: pgmap v6584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:05 smithi144 ceph-mon[32129]: pgmap v6585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:05 smithi160 ceph-mon[36715]: pgmap v6585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:05 smithi183 ceph-mon[37273]: pgmap v6585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:07.006+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:17:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:07.007+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:17:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:07.007+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:17:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:07.007+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:17:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:07.007+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11469.792873 secs 2024-07-26T14:17:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:07 smithi144 ceph-mon[32129]: pgmap v6586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11469.792873 secs 2024-07-26T14:17:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:07 smithi160 ceph-mon[36715]: pgmap v6586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11469.792873 secs 2024-07-26T14:17:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:07 smithi183 ceph-mon[37273]: pgmap v6586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:09 smithi144 ceph-mon[32129]: pgmap v6587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:09 smithi160 ceph-mon[36715]: pgmap v6587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:09 smithi183 ceph-mon[37273]: pgmap v6587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:11 smithi144 ceph-mon[32129]: pgmap v6588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:11 smithi160 ceph-mon[36715]: pgmap v6588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:11 smithi183 ceph-mon[37273]: pgmap v6588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11474.792995 secs 2024-07-26T14:17:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11474.792995 secs 2024-07-26T14:17:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11474.792995 secs 2024-07-26T14:17:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:13 smithi144 ceph-mon[32129]: pgmap v6589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:13 smithi160 ceph-mon[36715]: pgmap v6589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:13 smithi183 ceph-mon[37273]: pgmap v6589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:15 smithi160 ceph-mon[36715]: pgmap v6590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:15 smithi183 ceph-mon[37273]: pgmap v6590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:15 smithi144 ceph-mon[32129]: pgmap v6590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11479.793116 secs 2024-07-26T14:17:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:17 smithi160 ceph-mon[36715]: pgmap v6591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11479.793116 secs 2024-07-26T14:17:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:17 smithi183 ceph-mon[37273]: pgmap v6591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11479.793116 secs 2024-07-26T14:17:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:17 smithi144 ceph-mon[32129]: pgmap v6591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:19 smithi160 ceph-mon[36715]: pgmap v6592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:19 smithi183 ceph-mon[37273]: pgmap v6592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:19 smithi144 ceph-mon[32129]: pgmap v6592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:21 smithi144 ceph-mon[32129]: pgmap v6593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:21 smithi160 ceph-mon[36715]: pgmap v6593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:21 smithi183 ceph-mon[37273]: pgmap v6593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:22.018 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:22.017+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:17:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:22.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:17:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:22.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:17:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:22.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:17:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:22.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11484.793235 secs 2024-07-26T14:17:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11484.793235 secs 2024-07-26T14:17:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11484.793235 secs 2024-07-26T14:17:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:23 smithi160 ceph-mon[36715]: pgmap v6594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:23 smithi183 ceph-mon[37273]: pgmap v6594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:23 smithi144 ceph-mon[32129]: pgmap v6594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:25 smithi160 ceph-mon[36715]: pgmap v6595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:25 smithi183 ceph-mon[37273]: pgmap v6595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:25 smithi144 ceph-mon[32129]: pgmap v6595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11489.793335 secs 2024-07-26T14:17:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:27 smithi160 ceph-mon[36715]: pgmap v6596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11489.793335 secs 2024-07-26T14:17:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:27 smithi183 ceph-mon[37273]: pgmap v6596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11489.793335 secs 2024-07-26T14:17:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:27 smithi144 ceph-mon[32129]: pgmap v6596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:29 smithi160 ceph-mon[36715]: pgmap v6597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:29 smithi183 ceph-mon[37273]: pgmap v6597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:29 smithi144 ceph-mon[32129]: pgmap v6597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:31 smithi160 ceph-mon[36715]: pgmap v6598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:31 smithi183 ceph-mon[37273]: pgmap v6598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:31 smithi144 ceph-mon[32129]: pgmap v6598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11494.793481 secs 2024-07-26T14:17:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11494.793481 secs 2024-07-26T14:17:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11494.793481 secs 2024-07-26T14:17:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:33 smithi160 ceph-mon[36715]: pgmap v6599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:33 smithi183 ceph-mon[37273]: pgmap v6599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:33 smithi144 ceph-mon[32129]: pgmap v6599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:35 smithi160 ceph-mon[36715]: pgmap v6600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:35 smithi183 ceph-mon[37273]: pgmap v6600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:35 smithi144 ceph-mon[32129]: pgmap v6600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:36.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:37.033+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:17:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:37.034+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:17:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:37.034+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:17:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:37.034+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:17:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:37.034+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:37.623 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11499.793596 secs 2024-07-26T14:17:37.623 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:37 smithi144 ceph-mon[32129]: pgmap v6601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:37.623 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:37.623 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:37.624 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:17:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11499.793596 secs 2024-07-26T14:17:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:37 smithi183 ceph-mon[37273]: pgmap v6601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:17:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11499.793596 secs 2024-07-26T14:17:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:37 smithi160 ceph-mon[36715]: pgmap v6601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:17:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:38 smithi183 ceph-mon[37273]: pgmap v6602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:17:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:38 smithi183 ceph-mon[37273]: pgmap v6603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:38 smithi144 ceph-mon[32129]: pgmap v6602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:17:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:38 smithi144 ceph-mon[32129]: pgmap v6603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:38 smithi160 ceph-mon[36715]: pgmap v6602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:17:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:38 smithi160 ceph-mon[36715]: pgmap v6603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:40 smithi183 ceph-mon[37273]: pgmap v6604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:40 smithi144 ceph-mon[32129]: pgmap v6604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:40 smithi160 ceph-mon[36715]: pgmap v6604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11504.793681 secs 2024-07-26T14:17:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:42 smithi183 ceph-mon[37273]: pgmap v6605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11504.793681 secs 2024-07-26T14:17:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:42 smithi144 ceph-mon[32129]: pgmap v6605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11504.793681 secs 2024-07-26T14:17:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:42 smithi160 ceph-mon[36715]: pgmap v6605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:44 smithi183 ceph-mon[37273]: pgmap v6606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T14:17:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:44 smithi144 ceph-mon[32129]: pgmap v6606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T14:17:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:44 smithi160 ceph-mon[36715]: pgmap v6606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T14:17:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:46 smithi183 ceph-mon[37273]: pgmap v6607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:46 smithi144 ceph-mon[32129]: pgmap v6607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:46 smithi160 ceph-mon[36715]: pgmap v6607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:17:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11509.793801 secs 2024-07-26T14:17:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11509.793801 secs 2024-07-26T14:17:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11509.793801 secs 2024-07-26T14:17:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:48.135+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:17:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:48.135+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:17:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:48.136+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T14:17:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:48.136+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:17:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:48 smithi183 ceph-mon[37273]: pgmap v6608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:17:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:48 smithi144 ceph-mon[32129]: pgmap v6608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:17:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:48 smithi160 ceph-mon[36715]: pgmap v6608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:17:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:50 smithi183 ceph-mon[37273]: pgmap v6609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:50 smithi144 ceph-mon[32129]: pgmap v6609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:50 smithi160 ceph-mon[36715]: pgmap v6609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:17:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11514.793882 secs 2024-07-26T14:17:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11514.793882 secs 2024-07-26T14:17:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11514.793882 secs 2024-07-26T14:17:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:52.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:17:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:52.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:17:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:52.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:17:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:52.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:17:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:17:52.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:52 smithi183 ceph-mon[37273]: pgmap v6610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:52 smithi144 ceph-mon[32129]: pgmap v6610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:52 smithi160 ceph-mon[36715]: pgmap v6610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:17:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:54 smithi183 ceph-mon[37273]: pgmap v6611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:54 smithi144 ceph-mon[32129]: pgmap v6611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:54 smithi160 ceph-mon[36715]: pgmap v6611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:56 smithi183 ceph-mon[37273]: pgmap v6612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:17:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11519.794020 secs 2024-07-26T14:17:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:56 smithi144 ceph-mon[32129]: pgmap v6612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:17:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11519.794020 secs 2024-07-26T14:17:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:56 smithi160 ceph-mon[36715]: pgmap v6612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:17:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11519.794020 secs 2024-07-26T14:17:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:17:58 smithi144 ceph-mon[32129]: pgmap v6613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:17:58 smithi160 ceph-mon[36715]: pgmap v6613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:17:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:17:58 smithi183 ceph-mon[37273]: pgmap v6613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:00 smithi144 ceph-mon[32129]: pgmap v6614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:00 smithi160 ceph-mon[36715]: pgmap v6614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:00.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:00 smithi183 ceph-mon[37273]: pgmap v6614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:01 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 11524.794156 secs 2024-07-26T14:18:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:01 smithi144 ceph-mon[32129]: slow request 11524.794155 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T14:18:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:01 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 11524.794156 secs 2024-07-26T14:18:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:01 smithi160 ceph-mon[36715]: slow request 11524.794155 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T14:18:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:01 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 11524.794156 secs 2024-07-26T14:18:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:01 smithi183 ceph-mon[37273]: slow request 11524.794155 seconds old, received at 2024-07-26T11:05:56.017112+0000: client_request(client.15201:17774 rmdir #0x10000002bcc/p7 2024-07-26T11:05:56.015615+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T14:18:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:02 smithi144 ceph-mon[32129]: pgmap v6615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:02 smithi160 ceph-mon[36715]: pgmap v6615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:02 smithi183 ceph-mon[37273]: pgmap v6615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:04 smithi144 ceph-mon[32129]: pgmap v6616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:04 smithi160 ceph-mon[36715]: pgmap v6616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:04 smithi183 ceph-mon[37273]: pgmap v6616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:06 smithi144 ceph-mon[32129]: pgmap v6617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:06 smithi144 ceph-mon[32129]: 2 slow requests, 1 included below; oldest blocked for > 11529.794277 secs 2024-07-26T14:18:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:06 smithi144 ceph-mon[32129]: slow request 11524.933466 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T14:18:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:06 smithi160 ceph-mon[36715]: pgmap v6617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:06 smithi160 ceph-mon[36715]: 2 slow requests, 1 included below; oldest blocked for > 11529.794277 secs 2024-07-26T14:18:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:06 smithi160 ceph-mon[36715]: slow request 11524.933466 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T14:18:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:06 smithi183 ceph-mon[37273]: pgmap v6617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:06 smithi183 ceph-mon[37273]: 2 slow requests, 1 included below; oldest blocked for > 11529.794277 secs 2024-07-26T14:18:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:06 smithi183 ceph-mon[37273]: slow request 11524.933466 seconds old, received at 2024-07-26T11:06:00.877923+0000: client_request(client.25502:24 mksnap #0x1//scheduled-2024-07-26-11_06_00_UTC 2024-07-26T11:06:00.877772+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T14:18:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:07.059+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:18:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:07.060+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:18:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:07.060+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:18:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:07.060+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:18:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:07.060+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:08.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:08 smithi183 ceph-mon[37273]: pgmap v6618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:08 smithi144 ceph-mon[32129]: pgmap v6618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:08 smithi160 ceph-mon[36715]: pgmap v6618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:10 smithi144 ceph-mon[32129]: pgmap v6619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:10 smithi160 ceph-mon[36715]: pgmap v6619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:10 smithi183 ceph-mon[37273]: pgmap v6619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11534.794362 secs 2024-07-26T14:18:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11534.794362 secs 2024-07-26T14:18:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11534.794362 secs 2024-07-26T14:18:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:12 smithi144 ceph-mon[32129]: pgmap v6620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:12 smithi160 ceph-mon[36715]: pgmap v6620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:12 smithi183 ceph-mon[37273]: pgmap v6620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:14 smithi144 ceph-mon[32129]: pgmap v6621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:14 smithi160 ceph-mon[36715]: pgmap v6621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:14 smithi183 ceph-mon[37273]: pgmap v6621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:16 smithi144 ceph-mon[32129]: pgmap v6622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11539.794455 secs 2024-07-26T14:18:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:16 smithi160 ceph-mon[36715]: pgmap v6622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11539.794455 secs 2024-07-26T14:18:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:16 smithi183 ceph-mon[37273]: pgmap v6622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11539.794455 secs 2024-07-26T14:18:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:18 smithi144 ceph-mon[32129]: pgmap v6623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:18 smithi160 ceph-mon[36715]: pgmap v6623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:18 smithi183 ceph-mon[37273]: pgmap v6623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:20 smithi144 ceph-mon[32129]: pgmap v6624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:20 smithi160 ceph-mon[36715]: pgmap v6624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:20 smithi183 ceph-mon[37273]: pgmap v6624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11544.794525 secs 2024-07-26T14:18:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11544.794525 secs 2024-07-26T14:18:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11544.794525 secs 2024-07-26T14:18:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:22.076+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:18:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:22.076+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:18:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:22.076+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:18:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:22.076+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:18:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:22.077+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:22 smithi160 ceph-mon[36715]: pgmap v6625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:22 smithi183 ceph-mon[37273]: pgmap v6625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:22 smithi144 ceph-mon[32129]: pgmap v6625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:24 smithi160 ceph-mon[36715]: pgmap v6626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:24 smithi183 ceph-mon[37273]: pgmap v6626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:24 smithi144 ceph-mon[32129]: pgmap v6626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:26 smithi160 ceph-mon[36715]: pgmap v6627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11549.794646 secs 2024-07-26T14:18:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:26 smithi183 ceph-mon[37273]: pgmap v6627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11549.794646 secs 2024-07-26T14:18:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:26 smithi144 ceph-mon[32129]: pgmap v6627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11549.794646 secs 2024-07-26T14:18:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:28 smithi160 ceph-mon[36715]: pgmap v6628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:28 smithi183 ceph-mon[37273]: pgmap v6628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:28 smithi144 ceph-mon[32129]: pgmap v6628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:30 smithi160 ceph-mon[36715]: pgmap v6629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:30 smithi183 ceph-mon[37273]: pgmap v6629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:30 smithi144 ceph-mon[32129]: pgmap v6629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11554.794715 secs 2024-07-26T14:18:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11554.794715 secs 2024-07-26T14:18:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11554.794715 secs 2024-07-26T14:18:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:32 smithi160 ceph-mon[36715]: pgmap v6630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:32 smithi183 ceph-mon[37273]: pgmap v6630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:32 smithi144 ceph-mon[32129]: pgmap v6630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:34 smithi160 ceph-mon[36715]: pgmap v6631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:34 smithi183 ceph-mon[37273]: pgmap v6631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:34 smithi144 ceph-mon[32129]: pgmap v6631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:36 smithi160 ceph-mon[36715]: pgmap v6632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11559.794835 secs 2024-07-26T14:18:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:36 smithi183 ceph-mon[37273]: pgmap v6632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11559.794835 secs 2024-07-26T14:18:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:36 smithi144 ceph-mon[32129]: pgmap v6632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:18:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11559.794835 secs 2024-07-26T14:18:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:37.092+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:18:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:37.093+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:18:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:37.093+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:18:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:37.093+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:18:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:37.093+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:37.856 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:38 smithi160 ceph-mon[36715]: pgmap v6633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:18:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:38 smithi183 ceph-mon[37273]: pgmap v6633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:18:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:38 smithi144 ceph-mon[32129]: pgmap v6633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:18:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:39 smithi160 ceph-mon[36715]: pgmap v6634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:39 smithi160 ceph-mon[36715]: pgmap v6635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:18:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:39 smithi183 ceph-mon[37273]: pgmap v6634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:39 smithi183 ceph-mon[37273]: pgmap v6635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:18:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:39 smithi144 ceph-mon[32129]: pgmap v6634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:39 smithi144 ceph-mon[32129]: pgmap v6635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:18:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:41 smithi160 ceph-mon[36715]: pgmap v6636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11564.794953 secs 2024-07-26T14:18:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:41 smithi183 ceph-mon[37273]: pgmap v6636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11564.794953 secs 2024-07-26T14:18:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:41 smithi144 ceph-mon[32129]: pgmap v6636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11564.794953 secs 2024-07-26T14:18:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:43 smithi183 ceph-mon[37273]: pgmap v6637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:43 smithi144 ceph-mon[32129]: pgmap v6637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:43 smithi160 ceph-mon[36715]: pgmap v6637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:45 smithi183 ceph-mon[37273]: pgmap v6638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:18:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:45 smithi144 ceph-mon[32129]: pgmap v6638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:18:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:45 smithi160 ceph-mon[36715]: pgmap v6638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:18:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11569.795078 secs 2024-07-26T14:18:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11569.795078 secs 2024-07-26T14:18:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11569.795078 secs 2024-07-26T14:18:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:47 smithi183 ceph-mon[37273]: pgmap v6639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:47 smithi144 ceph-mon[32129]: pgmap v6639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:47 smithi160 ceph-mon[36715]: pgmap v6639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:18:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:49 smithi183 ceph-mon[37273]: pgmap v6640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:18:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:49 smithi144 ceph-mon[32129]: pgmap v6640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:18:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:49 smithi160 ceph-mon[36715]: pgmap v6640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:18:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:51 smithi183 ceph-mon[37273]: pgmap v6641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11574.795175 secs 2024-07-26T14:18:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:51 smithi144 ceph-mon[32129]: pgmap v6641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11574.795175 secs 2024-07-26T14:18:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:51 smithi160 ceph-mon[36715]: pgmap v6641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11574.795175 secs 2024-07-26T14:18:52.110 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:52.109+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:18:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:52.110+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:18:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:52.110+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:18:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:52.110+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:18:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:18:52.110+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:53 smithi183 ceph-mon[37273]: pgmap v6642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:53 smithi144 ceph-mon[32129]: pgmap v6642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:53 smithi160 ceph-mon[36715]: pgmap v6642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:18:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:55 smithi183 ceph-mon[37273]: pgmap v6643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:55 smithi144 ceph-mon[32129]: pgmap v6643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:55 smithi160 ceph-mon[36715]: pgmap v6643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11579.795312 secs 2024-07-26T14:18:56.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11579.795312 secs 2024-07-26T14:18:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11579.795312 secs 2024-07-26T14:18:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:57 smithi183 ceph-mon[37273]: pgmap v6644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:57 smithi144 ceph-mon[32129]: pgmap v6644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:57 smithi160 ceph-mon[36715]: pgmap v6644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:18:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:18:59 smithi183 ceph-mon[37273]: pgmap v6645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:18:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:18:59 smithi144 ceph-mon[32129]: pgmap v6645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:18:59 smithi160 ceph-mon[36715]: pgmap v6645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:01 smithi183 ceph-mon[37273]: pgmap v6646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11584.795442 secs 2024-07-26T14:19:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:01 smithi144 ceph-mon[32129]: pgmap v6646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11584.795442 secs 2024-07-26T14:19:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:01 smithi160 ceph-mon[36715]: pgmap v6646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11584.795442 secs 2024-07-26T14:19:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:03 smithi144 ceph-mon[32129]: pgmap v6647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:03 smithi160 ceph-mon[36715]: pgmap v6647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:03 smithi183 ceph-mon[37273]: pgmap v6647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:05 smithi144 ceph-mon[32129]: pgmap v6648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:05 smithi160 ceph-mon[36715]: pgmap v6648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:05 smithi183 ceph-mon[37273]: pgmap v6648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11589.795525 secs 2024-07-26T14:19:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11589.795525 secs 2024-07-26T14:19:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:07.125+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:19:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:07.126+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:19:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:07.126+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:19:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:07.126+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:19:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:07.126+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11589.795525 secs 2024-07-26T14:19:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:07 smithi144 ceph-mon[32129]: pgmap v6649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:08.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:07 smithi183 ceph-mon[37273]: pgmap v6649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:07 smithi160 ceph-mon[36715]: pgmap v6649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:09 smithi144 ceph-mon[32129]: pgmap v6650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:09 smithi160 ceph-mon[36715]: pgmap v6650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:09 smithi183 ceph-mon[37273]: pgmap v6650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:10 smithi144 ceph-mon[32129]: pgmap v6651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:10 smithi160 ceph-mon[36715]: pgmap v6651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:10 smithi183 ceph-mon[37273]: pgmap v6651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11594.795645 secs 2024-07-26T14:19:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11594.795645 secs 2024-07-26T14:19:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11594.795645 secs 2024-07-26T14:19:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:12 smithi144 ceph-mon[32129]: pgmap v6652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:12 smithi160 ceph-mon[36715]: pgmap v6652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:12 smithi183 ceph-mon[37273]: pgmap v6652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:15 smithi160 ceph-mon[36715]: pgmap v6653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:15 smithi183 ceph-mon[37273]: pgmap v6653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:15 smithi144 ceph-mon[32129]: pgmap v6653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11599.795713 secs 2024-07-26T14:19:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11599.795713 secs 2024-07-26T14:19:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11599.795713 secs 2024-07-26T14:19:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:17 smithi160 ceph-mon[36715]: pgmap v6654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:17 smithi183 ceph-mon[37273]: pgmap v6654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:17 smithi144 ceph-mon[32129]: pgmap v6654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:19 smithi183 ceph-mon[37273]: pgmap v6655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:19 smithi144 ceph-mon[32129]: pgmap v6655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:19 smithi160 ceph-mon[36715]: pgmap v6655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:21 smithi183 ceph-mon[37273]: pgmap v6656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11604.795797 secs 2024-07-26T14:19:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:21 smithi144 ceph-mon[32129]: pgmap v6656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11604.795797 secs 2024-07-26T14:19:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:21 smithi160 ceph-mon[36715]: pgmap v6656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11604.795797 secs 2024-07-26T14:19:22.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:22.139+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:19:22.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:22.140+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:19:22.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:22.140+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:19:22.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:22.140+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:19:22.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:22.140+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:23 smithi183 ceph-mon[37273]: pgmap v6657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:23 smithi144 ceph-mon[32129]: pgmap v6657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:23 smithi160 ceph-mon[36715]: pgmap v6657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:25 smithi183 ceph-mon[37273]: pgmap v6658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:25 smithi144 ceph-mon[32129]: pgmap v6658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:25 smithi160 ceph-mon[36715]: pgmap v6658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11609.795876 secs 2024-07-26T14:19:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:26 smithi183 ceph-mon[37273]: pgmap v6659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11609.795876 secs 2024-07-26T14:19:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:26 smithi144 ceph-mon[32129]: pgmap v6659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11609.795876 secs 2024-07-26T14:19:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:26 smithi160 ceph-mon[36715]: pgmap v6659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:29 smithi160 ceph-mon[36715]: pgmap v6660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:29 smithi183 ceph-mon[37273]: pgmap v6660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:29 smithi144 ceph-mon[32129]: pgmap v6660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:31 smithi160 ceph-mon[36715]: pgmap v6661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11614.795997 secs 2024-07-26T14:19:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:31 smithi183 ceph-mon[37273]: pgmap v6661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11614.795997 secs 2024-07-26T14:19:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:31 smithi144 ceph-mon[32129]: pgmap v6661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11614.795997 secs 2024-07-26T14:19:33.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:33 smithi183 ceph-mon[37273]: pgmap v6662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:33 smithi144 ceph-mon[32129]: pgmap v6662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:33 smithi160 ceph-mon[36715]: pgmap v6662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:19:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:35 smithi183 ceph-mon[37273]: pgmap v6663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:35 smithi144 ceph-mon[32129]: pgmap v6663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:35 smithi160 ceph-mon[36715]: pgmap v6663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11619.796105 secs 2024-07-26T14:19:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11619.796105 secs 2024-07-26T14:19:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11619.796105 secs 2024-07-26T14:19:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:37.149+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:19:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:37.149+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:19:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:37.149+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:19:37.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:37.149+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:19:37.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:37.150+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:37.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:37 smithi144 ceph-mon[32129]: pgmap v6664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:37 smithi183 ceph-mon[37273]: pgmap v6664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:37 smithi160 ceph-mon[36715]: pgmap v6664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:39 smithi183 ceph-mon[37273]: pgmap v6665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:39 smithi144 ceph-mon[32129]: pgmap v6665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:39 smithi160 ceph-mon[36715]: pgmap v6665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:19:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:19:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:19:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:41 smithi183 ceph-mon[37273]: pgmap v6666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:19:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:41 smithi183 ceph-mon[37273]: pgmap v6667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:19:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11624.796171 secs 2024-07-26T14:19:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:41 smithi144 ceph-mon[32129]: pgmap v6666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:19:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:41 smithi144 ceph-mon[32129]: pgmap v6667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:19:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11624.796171 secs 2024-07-26T14:19:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:41 smithi160 ceph-mon[36715]: pgmap v6666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:19:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:41 smithi160 ceph-mon[36715]: pgmap v6667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:19:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11624.796171 secs 2024-07-26T14:19:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:43 smithi183 ceph-mon[37273]: pgmap v6668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:19:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:43 smithi144 ceph-mon[32129]: pgmap v6668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:19:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:43 smithi160 ceph-mon[36715]: pgmap v6668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:19:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:45 smithi183 ceph-mon[37273]: pgmap v6669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:19:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:45 smithi144 ceph-mon[32129]: pgmap v6669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:19:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:45 smithi160 ceph-mon[36715]: pgmap v6669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:19:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11629.796323 secs 2024-07-26T14:19:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11629.796323 secs 2024-07-26T14:19:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11629.796323 secs 2024-07-26T14:19:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:47 smithi183 ceph-mon[37273]: pgmap v6670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:19:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:47 smithi144 ceph-mon[32129]: pgmap v6670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:19:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:47 smithi160 ceph-mon[36715]: pgmap v6670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:19:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:49 smithi183 ceph-mon[37273]: pgmap v6671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:19:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:49 smithi144 ceph-mon[32129]: pgmap v6671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:19:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:49 smithi160 ceph-mon[36715]: pgmap v6671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:19:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:51 smithi144 ceph-mon[32129]: pgmap v6672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:19:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11634.796407 secs 2024-07-26T14:19:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:51 smithi160 ceph-mon[36715]: pgmap v6672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:19:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11634.796407 secs 2024-07-26T14:19:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:51 smithi183 ceph-mon[37273]: pgmap v6672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:19:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11634.796407 secs 2024-07-26T14:19:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:52.165+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:19:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:52.166+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:19:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:52.166+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:19:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:52.166+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:19:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:19:52.166+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:53 smithi144 ceph-mon[32129]: pgmap v6673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:53 smithi160 ceph-mon[36715]: pgmap v6673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:53 smithi183 ceph-mon[37273]: pgmap v6673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:19:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:55 smithi144 ceph-mon[32129]: pgmap v6674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:55 smithi160 ceph-mon[36715]: pgmap v6674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:55 smithi183 ceph-mon[37273]: pgmap v6674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:56.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11639.796512 secs 2024-07-26T14:19:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11639.796512 secs 2024-07-26T14:19:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11639.796512 secs 2024-07-26T14:19:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:57 smithi144 ceph-mon[32129]: pgmap v6675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:57 smithi160 ceph-mon[36715]: pgmap v6675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:57 smithi183 ceph-mon[37273]: pgmap v6675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:19:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:19:59 smithi144 ceph-mon[32129]: pgmap v6676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:19:59 smithi160 ceph-mon[36715]: pgmap v6676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:19:59 smithi183 ceph-mon[37273]: pgmap v6676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:00 smithi144 ceph-mon[32129]: pgmap v6677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:20:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:00 smithi160 ceph-mon[36715]: pgmap v6677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:20:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:00 smithi183 ceph-mon[37273]: pgmap v6677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:20:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11644.796629 secs 2024-07-26T14:20:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11644.796629 secs 2024-07-26T14:20:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11644.796629 secs 2024-07-26T14:20:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:02 smithi160 ceph-mon[36715]: pgmap v6678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:02 smithi183 ceph-mon[37273]: pgmap v6678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:02 smithi144 ceph-mon[32129]: pgmap v6678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:04 smithi160 ceph-mon[36715]: pgmap v6679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:04 smithi183 ceph-mon[37273]: pgmap v6679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:04 smithi144 ceph-mon[32129]: pgmap v6679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:07.021 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:06 smithi144 ceph-mon[32129]: pgmap v6680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:07.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11649.796758 secs 2024-07-26T14:20:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:06 smithi160 ceph-mon[36715]: pgmap v6680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11649.796758 secs 2024-07-26T14:20:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:06 smithi183 ceph-mon[37273]: pgmap v6680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11649.796758 secs 2024-07-26T14:20:07.179 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:07.178+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:20:07.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:07.179+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:20:07.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:07.179+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:20:07.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:07.179+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:20:07.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:07.180+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:08 smithi160 ceph-mon[36715]: pgmap v6681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:08 smithi183 ceph-mon[37273]: pgmap v6681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:08 smithi144 ceph-mon[32129]: pgmap v6681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:10 smithi160 ceph-mon[36715]: pgmap v6682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:10 smithi183 ceph-mon[37273]: pgmap v6682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:10 smithi144 ceph-mon[32129]: pgmap v6682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11654.796890 secs 2024-07-26T14:20:12.061 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11654.796890 secs 2024-07-26T14:20:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11654.796890 secs 2024-07-26T14:20:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:12 smithi160 ceph-mon[36715]: pgmap v6683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:12 smithi183 ceph-mon[37273]: pgmap v6683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:12 smithi144 ceph-mon[32129]: pgmap v6683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:14 smithi160 ceph-mon[36715]: pgmap v6684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:14 smithi183 ceph-mon[37273]: pgmap v6684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:14 smithi144 ceph-mon[32129]: pgmap v6684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:16 smithi160 ceph-mon[36715]: pgmap v6685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11659.797020 secs 2024-07-26T14:20:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:16 smithi183 ceph-mon[37273]: pgmap v6685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11659.797020 secs 2024-07-26T14:20:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:16 smithi144 ceph-mon[32129]: pgmap v6685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11659.797020 secs 2024-07-26T14:20:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:18 smithi183 ceph-mon[37273]: pgmap v6686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:18 smithi144 ceph-mon[32129]: pgmap v6686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:18 smithi160 ceph-mon[36715]: pgmap v6686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:20 smithi183 ceph-mon[37273]: pgmap v6687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:20 smithi144 ceph-mon[32129]: pgmap v6687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:20 smithi160 ceph-mon[36715]: pgmap v6687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11664.797158 secs 2024-07-26T14:20:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:22.195+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:20:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:22.196+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:20:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:22.196+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:20:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:22.196+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:20:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:22.197+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:22.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11664.797158 secs 2024-07-26T14:20:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11664.797158 secs 2024-07-26T14:20:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:22 smithi183 ceph-mon[37273]: pgmap v6688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:22 smithi144 ceph-mon[32129]: pgmap v6688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:22 smithi160 ceph-mon[36715]: pgmap v6688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:24 smithi183 ceph-mon[37273]: pgmap v6689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:24 smithi144 ceph-mon[32129]: pgmap v6689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:24 smithi160 ceph-mon[36715]: pgmap v6689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:27.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:26 smithi160 ceph-mon[36715]: pgmap v6690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:27.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11669.797258 secs 2024-07-26T14:20:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:26 smithi183 ceph-mon[37273]: pgmap v6690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11669.797258 secs 2024-07-26T14:20:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:26 smithi144 ceph-mon[32129]: pgmap v6690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11669.797258 secs 2024-07-26T14:20:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:28 smithi183 ceph-mon[37273]: pgmap v6691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:28 smithi144 ceph-mon[32129]: pgmap v6691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:28 smithi160 ceph-mon[36715]: pgmap v6691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:30 smithi183 ceph-mon[37273]: pgmap v6692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:30 smithi144 ceph-mon[32129]: pgmap v6692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:30 smithi160 ceph-mon[36715]: pgmap v6692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11674.797425 secs 2024-07-26T14:20:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11674.797425 secs 2024-07-26T14:20:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11674.797425 secs 2024-07-26T14:20:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:32 smithi183 ceph-mon[37273]: pgmap v6693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:32 smithi144 ceph-mon[32129]: pgmap v6693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:32 smithi160 ceph-mon[36715]: pgmap v6693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:34 smithi144 ceph-mon[32129]: pgmap v6694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:34 smithi160 ceph-mon[36715]: pgmap v6694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:34 smithi183 ceph-mon[37273]: pgmap v6694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:37.212+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:20:37.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:37.213+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:20:37.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:37.213+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:20:37.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:37.213+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:20:37.214 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:37.213+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:36 smithi144 ceph-mon[32129]: pgmap v6695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11679.797516 secs 2024-07-26T14:20:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:36 smithi160 ceph-mon[36715]: pgmap v6695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11679.797516 secs 2024-07-26T14:20:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:36 smithi183 ceph-mon[37273]: pgmap v6695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11679.797516 secs 2024-07-26T14:20:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:38 smithi144 ceph-mon[32129]: pgmap v6696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:38 smithi160 ceph-mon[36715]: pgmap v6696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:38 smithi183 ceph-mon[37273]: pgmap v6696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:40.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:40 smithi144 ceph-mon[32129]: pgmap v6697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:20:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:40 smithi160 ceph-mon[36715]: pgmap v6697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:20:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:40 smithi183 ceph-mon[37273]: pgmap v6697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:20:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11684.797614 secs 2024-07-26T14:20:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:41 smithi144 ceph-mon[32129]: pgmap v6698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:41 smithi144 ceph-mon[32129]: pgmap v6699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11684.797614 secs 2024-07-26T14:20:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:41 smithi160 ceph-mon[36715]: pgmap v6698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:41 smithi160 ceph-mon[36715]: pgmap v6699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11684.797614 secs 2024-07-26T14:20:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:41 smithi183 ceph-mon[37273]: pgmap v6698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:41 smithi183 ceph-mon[37273]: pgmap v6699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:43 smithi144 ceph-mon[32129]: pgmap v6700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:20:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:43 smithi160 ceph-mon[36715]: pgmap v6700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:20:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:43 smithi183 ceph-mon[37273]: pgmap v6700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.133+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.133+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.133+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a8044000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.133+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f74a80315a0 tx=0x7f74a8044000 comp rx=0 tx=0).stop 2024-07-26T14:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.133+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T14:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.133+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.134+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.134+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.134+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:46.136+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a8072000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:20:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:45 smithi144 ceph-mon[32129]: pgmap v6701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:20:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:45 smithi160 ceph-mon[36715]: pgmap v6701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:20:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:45 smithi183 ceph-mon[37273]: pgmap v6701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:20:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11689.797679 secs 2024-07-26T14:20:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11689.797679 secs 2024-07-26T14:20:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11689.797679 secs 2024-07-26T14:20:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:48.167+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:20:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:48.167+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:20:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:48.168+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T14:20:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:48.168+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T14:20:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:47 smithi144 ceph-mon[32129]: pgmap v6702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:47 smithi160 ceph-mon[36715]: pgmap v6702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:47 smithi183 ceph-mon[37273]: pgmap v6702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:20:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:49 smithi144 ceph-mon[32129]: pgmap v6703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:20:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:49 smithi160 ceph-mon[36715]: pgmap v6703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:20:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:49 smithi183 ceph-mon[37273]: pgmap v6703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:20:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11694.797746 secs 2024-07-26T14:20:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:51 smithi144 ceph-mon[32129]: pgmap v6704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:20:52.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:52.229+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:20:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:52.230+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:20:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:52.230+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:20:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:52.230+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:20:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:20:52.230+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11694.797746 secs 2024-07-26T14:20:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:51 smithi160 ceph-mon[36715]: pgmap v6704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:20:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11694.797746 secs 2024-07-26T14:20:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:51 smithi183 ceph-mon[37273]: pgmap v6704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:20:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:53 smithi144 ceph-mon[32129]: pgmap v6705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:53 smithi160 ceph-mon[36715]: pgmap v6705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:53 smithi183 ceph-mon[37273]: pgmap v6705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:20:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:55 smithi160 ceph-mon[36715]: pgmap v6706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:55 smithi183 ceph-mon[37273]: pgmap v6706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:55 smithi144 ceph-mon[32129]: pgmap v6706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:20:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11699.797873 secs 2024-07-26T14:20:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11699.797873 secs 2024-07-26T14:20:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11699.797873 secs 2024-07-26T14:20:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:57 smithi160 ceph-mon[36715]: pgmap v6707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:57 smithi183 ceph-mon[37273]: pgmap v6707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:20:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:57 smithi144 ceph-mon[32129]: pgmap v6707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:20:59 smithi160 ceph-mon[36715]: pgmap v6708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:20:59 smithi183 ceph-mon[37273]: pgmap v6708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:20:59 smithi144 ceph-mon[32129]: pgmap v6708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11704.798003 secs 2024-07-26T14:21:02.325 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:01 smithi160 ceph-mon[36715]: pgmap v6709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11704.798003 secs 2024-07-26T14:21:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:01 smithi183 ceph-mon[37273]: pgmap v6709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11704.798003 secs 2024-07-26T14:21:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:01 smithi144 ceph-mon[32129]: pgmap v6709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:04 smithi160 ceph-mon[36715]: pgmap v6710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:04 smithi183 ceph-mon[37273]: pgmap v6710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:04 smithi144 ceph-mon[32129]: pgmap v6710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:06 smithi160 ceph-mon[36715]: pgmap v6711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:06 smithi183 ceph-mon[37273]: pgmap v6711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:06 smithi144 ceph-mon[32129]: pgmap v6711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:07.237+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:21:07.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:07.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:21:07.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:07.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:21:07.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:07.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:21:07.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:07.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11709.798118 secs 2024-07-26T14:21:07.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11709.798118 secs 2024-07-26T14:21:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11709.798118 secs 2024-07-26T14:21:08.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:08 smithi160 ceph-mon[36715]: pgmap v6712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:08 smithi183 ceph-mon[37273]: pgmap v6712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:08 smithi144 ceph-mon[32129]: pgmap v6712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:10 smithi160 ceph-mon[36715]: pgmap v6713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:10 smithi183 ceph-mon[37273]: pgmap v6713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:10 smithi144 ceph-mon[32129]: pgmap v6713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11714.798239 secs 2024-07-26T14:21:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:12 smithi160 ceph-mon[36715]: pgmap v6714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11714.798239 secs 2024-07-26T14:21:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:12 smithi183 ceph-mon[37273]: pgmap v6714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11714.798239 secs 2024-07-26T14:21:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:12 smithi144 ceph-mon[32129]: pgmap v6714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:14.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:14 smithi144 ceph-mon[32129]: pgmap v6715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:14 smithi160 ceph-mon[36715]: pgmap v6715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:14 smithi183 ceph-mon[37273]: pgmap v6715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:16 smithi160 ceph-mon[36715]: pgmap v6716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:16 smithi183 ceph-mon[37273]: pgmap v6716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:16 smithi144 ceph-mon[32129]: pgmap v6716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11719.798345 secs 2024-07-26T14:21:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11719.798345 secs 2024-07-26T14:21:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11719.798345 secs 2024-07-26T14:21:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:18 smithi183 ceph-mon[37273]: pgmap v6717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:18 smithi144 ceph-mon[32129]: pgmap v6717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:18 smithi160 ceph-mon[36715]: pgmap v6717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:20 smithi183 ceph-mon[37273]: pgmap v6718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:20 smithi144 ceph-mon[32129]: pgmap v6718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:20 smithi160 ceph-mon[36715]: pgmap v6718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:22.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:21:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:22.255+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:21:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:22.255+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:21:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:22.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:21:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:22.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11724.798464 secs 2024-07-26T14:21:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:22 smithi183 ceph-mon[37273]: pgmap v6719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11724.798464 secs 2024-07-26T14:21:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:22 smithi144 ceph-mon[32129]: pgmap v6719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11724.798464 secs 2024-07-26T14:21:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:22 smithi160 ceph-mon[36715]: pgmap v6719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:24 smithi183 ceph-mon[37273]: pgmap v6720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:24 smithi144 ceph-mon[32129]: pgmap v6720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:24 smithi160 ceph-mon[36715]: pgmap v6720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:26 smithi183 ceph-mon[37273]: pgmap v6721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:26 smithi144 ceph-mon[32129]: pgmap v6721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:26 smithi160 ceph-mon[36715]: pgmap v6721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:27.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11729.798590 secs 2024-07-26T14:21:27.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11729.798590 secs 2024-07-26T14:21:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11729.798590 secs 2024-07-26T14:21:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:28 smithi183 ceph-mon[37273]: pgmap v6722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:28 smithi144 ceph-mon[32129]: pgmap v6722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:28 smithi160 ceph-mon[36715]: pgmap v6722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:30 smithi183 ceph-mon[37273]: pgmap v6723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:30 smithi144 ceph-mon[32129]: pgmap v6723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:30 smithi160 ceph-mon[36715]: pgmap v6723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11734.798713 secs 2024-07-26T14:21:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:32 smithi183 ceph-mon[37273]: pgmap v6724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11734.798713 secs 2024-07-26T14:21:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:32 smithi144 ceph-mon[32129]: pgmap v6724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11734.798713 secs 2024-07-26T14:21:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:32 smithi160 ceph-mon[36715]: pgmap v6724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:34 smithi183 ceph-mon[37273]: pgmap v6725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:34 smithi144 ceph-mon[32129]: pgmap v6725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:34 smithi160 ceph-mon[36715]: pgmap v6725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:36 smithi183 ceph-mon[37273]: pgmap v6726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:36 smithi144 ceph-mon[32129]: pgmap v6726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:36 smithi160 ceph-mon[36715]: pgmap v6726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:37.263+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:21:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:37.264+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:21:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:37.264+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:21:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:37.264+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:21:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:37.264+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11739.798801 secs 2024-07-26T14:21:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11739.798801 secs 2024-07-26T14:21:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11739.798801 secs 2024-07-26T14:21:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:38 smithi144 ceph-mon[32129]: pgmap v6727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:38 smithi160 ceph-mon[36715]: pgmap v6727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:38 smithi183 ceph-mon[37273]: pgmap v6727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:21:40.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:40 smithi183 ceph-mon[37273]: pgmap v6728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:40 smithi144 ceph-mon[32129]: pgmap v6728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:40 smithi160 ceph-mon[36715]: pgmap v6728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11744.798901 secs 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: pgmap v6729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:21:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:21:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11744.798901 secs 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: pgmap v6729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:21:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:21:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11744.798901 secs 2024-07-26T14:21:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: pgmap v6729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:21:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:43 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:21:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:43 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:21:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:43 smithi144 ceph-mon[32129]: pgmap v6730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:21:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:43 smithi144 ceph-mon[32129]: pgmap v6731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:21:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:43 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:21:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:43 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:21:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:43 smithi160 ceph-mon[36715]: pgmap v6730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:21:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:43 smithi160 ceph-mon[36715]: pgmap v6731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:21:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:43 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:21:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:43 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:21:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:43 smithi183 ceph-mon[37273]: pgmap v6730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:21:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:43 smithi183 ceph-mon[37273]: pgmap v6731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:21:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:45 smithi144 ceph-mon[32129]: pgmap v6732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:21:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:45 smithi160 ceph-mon[36715]: pgmap v6732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:21:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:45 smithi183 ceph-mon[37273]: pgmap v6732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:21:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11749.799046 secs 2024-07-26T14:21:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:47 smithi144 ceph-mon[32129]: pgmap v6733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:21:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11749.799046 secs 2024-07-26T14:21:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:47 smithi160 ceph-mon[36715]: pgmap v6733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:21:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11749.799046 secs 2024-07-26T14:21:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:47 smithi183 ceph-mon[37273]: pgmap v6733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:21:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:49 smithi144 ceph-mon[32129]: pgmap v6734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:21:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:49 smithi160 ceph-mon[36715]: pgmap v6734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:21:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:49 smithi183 ceph-mon[37273]: pgmap v6734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:21:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:51 smithi144 ceph-mon[32129]: pgmap v6735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:21:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:51 smithi160 ceph-mon[36715]: pgmap v6735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:21:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:51 smithi183 ceph-mon[37273]: pgmap v6735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:21:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:52.280+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:21:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:52.281+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:21:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:52.281+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:21:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:52.281+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:21:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:21:52.281+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11754.799179 secs 2024-07-26T14:21:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11754.799179 secs 2024-07-26T14:21:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11754.799179 secs 2024-07-26T14:21:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:53 smithi144 ceph-mon[32129]: pgmap v6736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:21:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:53 smithi160 ceph-mon[36715]: pgmap v6736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:21:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:53 smithi183 ceph-mon[37273]: pgmap v6736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:21:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:55 smithi144 ceph-mon[32129]: pgmap v6737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:55 smithi160 ceph-mon[36715]: pgmap v6737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:55 smithi183 ceph-mon[37273]: pgmap v6737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11759.799329 secs 2024-07-26T14:21:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:57 smithi144 ceph-mon[32129]: pgmap v6738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11759.799329 secs 2024-07-26T14:21:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:57 smithi160 ceph-mon[36715]: pgmap v6738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11759.799329 secs 2024-07-26T14:21:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:57 smithi183 ceph-mon[37273]: pgmap v6738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:21:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:21:59 smithi144 ceph-mon[32129]: pgmap v6739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:21:59 smithi160 ceph-mon[36715]: pgmap v6739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:21:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:21:59 smithi183 ceph-mon[37273]: pgmap v6739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:01 smithi160 ceph-mon[36715]: pgmap v6740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:01 smithi183 ceph-mon[37273]: pgmap v6740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:01 smithi144 ceph-mon[32129]: pgmap v6740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11764.799454 secs 2024-07-26T14:22:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11764.799454 secs 2024-07-26T14:22:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11764.799454 secs 2024-07-26T14:22:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:03 smithi160 ceph-mon[36715]: pgmap v6741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:03 smithi183 ceph-mon[37273]: pgmap v6741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:03 smithi144 ceph-mon[32129]: pgmap v6741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:05 smithi160 ceph-mon[36715]: pgmap v6742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:05 smithi183 ceph-mon[37273]: pgmap v6742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:05 smithi144 ceph-mon[32129]: pgmap v6742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:07.297+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:22:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:07.298+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:22:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:07.298+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:22:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:07.298+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:22:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:07.298+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:07.521 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11769.799543 secs 2024-07-26T14:22:07.521 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:07 smithi144 ceph-mon[32129]: pgmap v6743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11769.799543 secs 2024-07-26T14:22:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:07 smithi160 ceph-mon[36715]: pgmap v6743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11769.799543 secs 2024-07-26T14:22:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:07 smithi183 ceph-mon[37273]: pgmap v6743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:09 smithi160 ceph-mon[36715]: pgmap v6744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:09 smithi183 ceph-mon[37273]: pgmap v6744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:09 smithi144 ceph-mon[32129]: pgmap v6744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:11 smithi160 ceph-mon[36715]: pgmap v6745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:11 smithi183 ceph-mon[37273]: pgmap v6745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:11 smithi144 ceph-mon[32129]: pgmap v6745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11774.799676 secs 2024-07-26T14:22:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11774.799676 secs 2024-07-26T14:22:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11774.799676 secs 2024-07-26T14:22:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:13 smithi160 ceph-mon[36715]: pgmap v6746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:13 smithi183 ceph-mon[37273]: pgmap v6746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:13 smithi144 ceph-mon[32129]: pgmap v6746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:15 smithi160 ceph-mon[36715]: pgmap v6747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:15 smithi183 ceph-mon[37273]: pgmap v6747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:15 smithi144 ceph-mon[32129]: pgmap v6747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11779.799806 secs 2024-07-26T14:22:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:17 smithi160 ceph-mon[36715]: pgmap v6748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11779.799806 secs 2024-07-26T14:22:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:17 smithi183 ceph-mon[37273]: pgmap v6748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11779.799806 secs 2024-07-26T14:22:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:17 smithi144 ceph-mon[32129]: pgmap v6748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:19 smithi160 ceph-mon[36715]: pgmap v6749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:19 smithi183 ceph-mon[37273]: pgmap v6749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:19 smithi144 ceph-mon[32129]: pgmap v6749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:21 smithi160 ceph-mon[36715]: pgmap v6750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:21 smithi183 ceph-mon[37273]: pgmap v6750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:21 smithi144 ceph-mon[32129]: pgmap v6750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:22.314 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:22.313+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:22:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:22.314+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:22:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:22.314+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:22:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:22.314+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:22:22.315 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:22.314+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11784.799938 secs 2024-07-26T14:22:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11784.799938 secs 2024-07-26T14:22:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11784.799938 secs 2024-07-26T14:22:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:23 smithi183 ceph-mon[37273]: pgmap v6751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:23 smithi144 ceph-mon[32129]: pgmap v6751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:23 smithi160 ceph-mon[36715]: pgmap v6751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:25 smithi183 ceph-mon[37273]: pgmap v6752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:25 smithi144 ceph-mon[32129]: pgmap v6752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:25 smithi160 ceph-mon[36715]: pgmap v6752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:27.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11789.800082 secs 2024-07-26T14:22:27.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:27 smithi160 ceph-mon[36715]: pgmap v6753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11789.800082 secs 2024-07-26T14:22:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:27 smithi183 ceph-mon[37273]: pgmap v6753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11789.800082 secs 2024-07-26T14:22:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:27 smithi144 ceph-mon[32129]: pgmap v6753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:29 smithi183 ceph-mon[37273]: pgmap v6754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:29 smithi144 ceph-mon[32129]: pgmap v6754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:29 smithi160 ceph-mon[36715]: pgmap v6754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:31 smithi183 ceph-mon[37273]: pgmap v6755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:31 smithi144 ceph-mon[32129]: pgmap v6755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:31 smithi160 ceph-mon[36715]: pgmap v6755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11794.800206 secs 2024-07-26T14:22:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11794.800206 secs 2024-07-26T14:22:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11794.800206 secs 2024-07-26T14:22:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:33 smithi183 ceph-mon[37273]: pgmap v6756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:33 smithi144 ceph-mon[32129]: pgmap v6756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:33 smithi160 ceph-mon[36715]: pgmap v6756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:35 smithi183 ceph-mon[37273]: pgmap v6757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:35 smithi144 ceph-mon[32129]: pgmap v6757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:35 smithi160 ceph-mon[36715]: pgmap v6757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:37.330+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:22:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:37.331+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:22:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:37.331+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:22:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:37.331+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:22:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:37.331+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11799.800407 secs 2024-07-26T14:22:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:37 smithi183 ceph-mon[37273]: pgmap v6758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11799.800407 secs 2024-07-26T14:22:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:37 smithi144 ceph-mon[32129]: pgmap v6758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11799.800407 secs 2024-07-26T14:22:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:37 smithi160 ceph-mon[36715]: pgmap v6758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:39 smithi183 ceph-mon[37273]: pgmap v6759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:39 smithi144 ceph-mon[32129]: pgmap v6759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:39 smithi160 ceph-mon[36715]: pgmap v6759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:41 smithi183 ceph-mon[37273]: pgmap v6760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:41 smithi144 ceph-mon[32129]: pgmap v6760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:41 smithi160 ceph-mon[36715]: pgmap v6760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:22:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11804.800528 secs 2024-07-26T14:22:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11804.800528 secs 2024-07-26T14:22:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11804.800528 secs 2024-07-26T14:22:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: pgmap v6761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:43.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:22:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: pgmap v6761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:43.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:22:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: pgmap v6761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:22:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:43.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:22:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:44 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:22:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:44 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:22:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:44 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:22:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:44 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:22:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:44 smithi144 ceph-mon[32129]: pgmap v6762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:22:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:44 smithi144 ceph-mon[32129]: pgmap v6763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T14:22:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:44 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:22:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:44 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:22:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:44 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:22:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:44 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:22:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:44 smithi160 ceph-mon[36715]: pgmap v6762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:22:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:44 smithi160 ceph-mon[36715]: pgmap v6763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T14:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:44 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:44 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:44 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:44 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:44 smithi183 ceph-mon[37273]: pgmap v6762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:44 smithi183 ceph-mon[37273]: pgmap v6763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2 op/s 2024-07-26T14:22:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:46 smithi144 ceph-mon[32129]: pgmap v6764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11809.800652 secs 2024-07-26T14:22:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:46 smithi160 ceph-mon[36715]: pgmap v6764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11809.800652 secs 2024-07-26T14:22:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:46 smithi183 ceph-mon[37273]: pgmap v6764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11809.800652 secs 2024-07-26T14:22:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:48 smithi144 ceph-mon[32129]: pgmap v6765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:48 smithi160 ceph-mon[36715]: pgmap v6765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:48 smithi183 ceph-mon[37273]: pgmap v6765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:50 smithi144 ceph-mon[32129]: pgmap v6766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:50 smithi160 ceph-mon[36715]: pgmap v6766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:50 smithi183 ceph-mon[37273]: pgmap v6766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11814.800735 secs 2024-07-26T14:22:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11814.800735 secs 2024-07-26T14:22:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11814.800735 secs 2024-07-26T14:22:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:52.343+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:22:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:52.343+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:22:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:52.343+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:22:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:52.343+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:22:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:22:52.344+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:52 smithi144 ceph-mon[32129]: pgmap v6767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:52 smithi160 ceph-mon[36715]: pgmap v6767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:52 smithi183 ceph-mon[37273]: pgmap v6767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:22:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:54 smithi144 ceph-mon[32129]: pgmap v6768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:22:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:54 smithi160 ceph-mon[36715]: pgmap v6768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:22:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:54 smithi183 ceph-mon[37273]: pgmap v6768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:22:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:56 smithi144 ceph-mon[32129]: pgmap v6769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:22:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11819.800801 secs 2024-07-26T14:22:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:56 smithi160 ceph-mon[36715]: pgmap v6769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:22:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11819.800801 secs 2024-07-26T14:22:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:56 smithi183 ceph-mon[37273]: pgmap v6769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:22:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11819.800801 secs 2024-07-26T14:22:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:22:58 smithi144 ceph-mon[32129]: pgmap v6770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:22:58 smithi160 ceph-mon[36715]: pgmap v6770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:22:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:22:58 smithi183 ceph-mon[37273]: pgmap v6770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:00 smithi144 ceph-mon[32129]: pgmap v6771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:00 smithi160 ceph-mon[36715]: pgmap v6771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:00 smithi183 ceph-mon[37273]: pgmap v6771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11824.800952 secs 2024-07-26T14:23:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11824.800952 secs 2024-07-26T14:23:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11824.800952 secs 2024-07-26T14:23:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:02 smithi144 ceph-mon[32129]: pgmap v6772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:02 smithi160 ceph-mon[36715]: pgmap v6772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:02 smithi183 ceph-mon[37273]: pgmap v6772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:04 smithi144 ceph-mon[32129]: pgmap v6773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:04 smithi160 ceph-mon[36715]: pgmap v6773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:04 smithi183 ceph-mon[37273]: pgmap v6773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:06 smithi160 ceph-mon[36715]: pgmap v6774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:23:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11829.801048 secs 2024-07-26T14:23:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:06 smithi183 ceph-mon[37273]: pgmap v6774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:23:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11829.801048 secs 2024-07-26T14:23:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:06 smithi144 ceph-mon[32129]: pgmap v6774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:23:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11829.801048 secs 2024-07-26T14:23:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:07.359+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:23:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:07.360+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:23:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:07.360+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:23:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:07.360+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:23:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:07.360+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:08.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:08 smithi144 ceph-mon[32129]: pgmap v6775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:08 smithi160 ceph-mon[36715]: pgmap v6775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:08 smithi183 ceph-mon[37273]: pgmap v6775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:10 smithi160 ceph-mon[36715]: pgmap v6776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T14:23:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:10 smithi183 ceph-mon[37273]: pgmap v6776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T14:23:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:10 smithi144 ceph-mon[32129]: pgmap v6776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T14:23:11.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11834.801209 secs 2024-07-26T14:23:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11834.801209 secs 2024-07-26T14:23:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11834.801209 secs 2024-07-26T14:23:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:12 smithi160 ceph-mon[36715]: pgmap v6777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:23:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:12 smithi183 ceph-mon[37273]: pgmap v6777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:23:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:12 smithi144 ceph-mon[32129]: pgmap v6777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:23:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:14 smithi160 ceph-mon[36715]: pgmap v6778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:23:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:14 smithi183 ceph-mon[37273]: pgmap v6778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:23:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:14 smithi144 ceph-mon[32129]: pgmap v6778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:23:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:16 smithi160 ceph-mon[36715]: pgmap v6779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:23:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11839.801339 secs 2024-07-26T14:23:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:16 smithi183 ceph-mon[37273]: pgmap v6779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:23:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11839.801339 secs 2024-07-26T14:23:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:16 smithi144 ceph-mon[32129]: pgmap v6779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:23:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11839.801339 secs 2024-07-26T14:23:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:18 smithi160 ceph-mon[36715]: pgmap v6780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:18 smithi183 ceph-mon[37273]: pgmap v6780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:18.942 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:18 smithi144 ceph-mon[32129]: pgmap v6780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:20 smithi160 ceph-mon[36715]: pgmap v6781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:23:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:20 smithi183 ceph-mon[37273]: pgmap v6781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:23:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:20 smithi144 ceph-mon[32129]: pgmap v6781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:23:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11844.801433 secs 2024-07-26T14:23:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11844.801433 secs 2024-07-26T14:23:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11844.801433 secs 2024-07-26T14:23:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:22.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:23:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:22.377+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:23:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:22.377+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:23:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:22.377+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:23:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:22.377+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:22 smithi160 ceph-mon[36715]: pgmap v6782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:22 smithi183 ceph-mon[37273]: pgmap v6782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:22 smithi144 ceph-mon[32129]: pgmap v6782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:24 smithi160 ceph-mon[36715]: pgmap v6783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:24 smithi183 ceph-mon[37273]: pgmap v6783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:24 smithi144 ceph-mon[32129]: pgmap v6783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:26 smithi160 ceph-mon[36715]: pgmap v6784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:23:26.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11849.801550 secs 2024-07-26T14:23:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:26 smithi183 ceph-mon[37273]: pgmap v6784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:23:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11849.801550 secs 2024-07-26T14:23:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:26 smithi144 ceph-mon[32129]: pgmap v6784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:23:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11849.801550 secs 2024-07-26T14:23:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:28 smithi160 ceph-mon[36715]: pgmap v6785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:28 smithi183 ceph-mon[37273]: pgmap v6785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:28 smithi144 ceph-mon[32129]: pgmap v6785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:30 smithi160 ceph-mon[36715]: pgmap v6786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:30 smithi183 ceph-mon[37273]: pgmap v6786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:30 smithi144 ceph-mon[32129]: pgmap v6786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11854.801708 secs 2024-07-26T14:23:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11854.801708 secs 2024-07-26T14:23:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11854.801708 secs 2024-07-26T14:23:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:32 smithi183 ceph-mon[37273]: pgmap v6787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:32 smithi144 ceph-mon[32129]: pgmap v6787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:32 smithi160 ceph-mon[36715]: pgmap v6787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:34 smithi183 ceph-mon[37273]: pgmap v6788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:34 smithi144 ceph-mon[32129]: pgmap v6788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:34 smithi160 ceph-mon[36715]: pgmap v6788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:36 smithi183 ceph-mon[37273]: pgmap v6789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:23:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11859.801849 secs 2024-07-26T14:23:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:36 smithi144 ceph-mon[32129]: pgmap v6789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:23:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11859.801849 secs 2024-07-26T14:23:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:36 smithi160 ceph-mon[36715]: pgmap v6789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T14:23:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11859.801849 secs 2024-07-26T14:23:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:37.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:23:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:37.394+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:23:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:37.394+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:23:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:37.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:23:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:37.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:38 smithi183 ceph-mon[37273]: pgmap v6790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:38 smithi144 ceph-mon[32129]: pgmap v6790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:38 smithi160 ceph-mon[36715]: pgmap v6790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:40 smithi183 ceph-mon[37273]: pgmap v6791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:40 smithi144 ceph-mon[32129]: pgmap v6791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:40 smithi160 ceph-mon[36715]: pgmap v6791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:23:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11864.801980 secs 2024-07-26T14:23:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11864.801980 secs 2024-07-26T14:23:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11864.801980 secs 2024-07-26T14:23:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:42 smithi183 ceph-mon[37273]: pgmap v6792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:42 smithi144 ceph-mon[32129]: pgmap v6792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:42 smithi160 ceph-mon[36715]: pgmap v6792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:43.968 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:44 smithi183 ceph-mon[37273]: pgmap v6793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:23:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:44 smithi144 ceph-mon[32129]: pgmap v6793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:23:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:44 smithi160 ceph-mon[36715]: pgmap v6793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:23:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:23:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:45 smithi144 ceph-mon[32129]: pgmap v6794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:45 smithi144 ceph-mon[32129]: pgmap v6795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:45 smithi160 ceph-mon[36715]: pgmap v6794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:45 smithi160 ceph-mon[36715]: pgmap v6795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:45 smithi183 ceph-mon[37273]: pgmap v6794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:45 smithi183 ceph-mon[37273]: pgmap v6795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:23:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11869.802112 secs 2024-07-26T14:23:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11869.802112 secs 2024-07-26T14:23:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11869.802112 secs 2024-07-26T14:23:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:47 smithi144 ceph-mon[32129]: pgmap v6796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:47 smithi160 ceph-mon[36715]: pgmap v6796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:47 smithi183 ceph-mon[37273]: pgmap v6796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.198+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T14:23:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.198+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:23:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.199+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f74a009d0e0 con 0x7f74b0109990 2024-07-26T14:23:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.199+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:23:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.199+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7480004740 con 0x563c6c4182f0 2024-07-26T14:23:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.199+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:23:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.200+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:23:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:48.200+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:23:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:49 smithi144 ceph-mon[32129]: pgmap v6797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:23:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:49 smithi160 ceph-mon[36715]: pgmap v6797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:23:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:49 smithi183 ceph-mon[37273]: pgmap v6797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:23:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:51 smithi144 ceph-mon[32129]: pgmap v6798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11874.802270 secs 2024-07-26T14:23:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:51 smithi160 ceph-mon[36715]: pgmap v6798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11874.802270 secs 2024-07-26T14:23:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:51 smithi183 ceph-mon[37273]: pgmap v6798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11874.802270 secs 2024-07-26T14:23:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:52.402+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:23:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:52.403+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:23:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:52.403+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:23:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:52.403+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:23:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:23:52.403+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:53 smithi144 ceph-mon[32129]: pgmap v6799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:53 smithi160 ceph-mon[36715]: pgmap v6799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:53 smithi183 ceph-mon[37273]: pgmap v6799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:23:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:55 smithi144 ceph-mon[32129]: pgmap v6800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:23:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:55 smithi160 ceph-mon[36715]: pgmap v6800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:23:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:55 smithi183 ceph-mon[37273]: pgmap v6800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:23:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11879.802408 secs 2024-07-26T14:23:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11879.802408 secs 2024-07-26T14:23:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11879.802408 secs 2024-07-26T14:23:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:57 smithi144 ceph-mon[32129]: pgmap v6801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:57 smithi160 ceph-mon[36715]: pgmap v6801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:57 smithi183 ceph-mon[37273]: pgmap v6801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:23:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:23:59 smithi144 ceph-mon[32129]: pgmap v6802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:23:59 smithi160 ceph-mon[36715]: pgmap v6802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:23:59 smithi183 ceph-mon[37273]: pgmap v6802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:01 smithi144 ceph-mon[32129]: pgmap v6803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11884.802558 secs 2024-07-26T14:24:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:01 smithi160 ceph-mon[36715]: pgmap v6803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11884.802558 secs 2024-07-26T14:24:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:01 smithi183 ceph-mon[37273]: pgmap v6803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11884.802558 secs 2024-07-26T14:24:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:03 smithi144 ceph-mon[32129]: pgmap v6804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:03 smithi160 ceph-mon[36715]: pgmap v6804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:03 smithi183 ceph-mon[37273]: pgmap v6804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:05 smithi144 ceph-mon[32129]: pgmap v6805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:05 smithi160 ceph-mon[36715]: pgmap v6805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:05 smithi183 ceph-mon[37273]: pgmap v6805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11889.802691 secs 2024-07-26T14:24:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11889.802691 secs 2024-07-26T14:24:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11889.802691 secs 2024-07-26T14:24:07.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:07.419+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:24:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:07.420+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:24:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:07.420+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:24:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:07.420+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:24:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:07.420+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:07 smithi160 ceph-mon[36715]: pgmap v6806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:07 smithi183 ceph-mon[37273]: pgmap v6806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:07 smithi144 ceph-mon[32129]: pgmap v6806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:08 smithi160 ceph-mon[36715]: pgmap v6807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:09.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:08 smithi144 ceph-mon[32129]: pgmap v6807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:08 smithi183 ceph-mon[37273]: pgmap v6807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:11 smithi160 ceph-mon[36715]: pgmap v6808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11894.802812 secs 2024-07-26T14:24:11.837 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:11 smithi183 ceph-mon[37273]: pgmap v6808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:11.837 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11894.802812 secs 2024-07-26T14:24:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:11 smithi144 ceph-mon[32129]: pgmap v6808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11894.802812 secs 2024-07-26T14:24:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:13 smithi160 ceph-mon[36715]: pgmap v6809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:13 smithi183 ceph-mon[37273]: pgmap v6809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:13 smithi144 ceph-mon[32129]: pgmap v6809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:15 smithi183 ceph-mon[37273]: pgmap v6810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:15 smithi144 ceph-mon[32129]: pgmap v6810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:15 smithi160 ceph-mon[36715]: pgmap v6810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11899.802966 secs 2024-07-26T14:24:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:16 smithi144 ceph-mon[32129]: pgmap v6811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11899.802966 secs 2024-07-26T14:24:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:16 smithi160 ceph-mon[36715]: pgmap v6811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11899.802966 secs 2024-07-26T14:24:17.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:16 smithi183 ceph-mon[37273]: pgmap v6811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:19 smithi160 ceph-mon[36715]: pgmap v6812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:19 smithi183 ceph-mon[37273]: pgmap v6812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:19 smithi144 ceph-mon[32129]: pgmap v6812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:21 smithi183 ceph-mon[37273]: pgmap v6813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11904.803091 secs 2024-07-26T14:24:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:21 smithi144 ceph-mon[32129]: pgmap v6813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11904.803091 secs 2024-07-26T14:24:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:21 smithi160 ceph-mon[36715]: pgmap v6813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11904.803091 secs 2024-07-26T14:24:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:22.435+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:24:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:22.436+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:24:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:22.436+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:24:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:22.436+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:24:22.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:22.436+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:23 smithi183 ceph-mon[37273]: pgmap v6814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:23 smithi144 ceph-mon[32129]: pgmap v6814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:23 smithi160 ceph-mon[36715]: pgmap v6814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:25 smithi183 ceph-mon[37273]: pgmap v6815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:25 smithi144 ceph-mon[32129]: pgmap v6815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:25 smithi160 ceph-mon[36715]: pgmap v6815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11909.803231 secs 2024-07-26T14:24:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11909.803231 secs 2024-07-26T14:24:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11909.803231 secs 2024-07-26T14:24:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:27 smithi183 ceph-mon[37273]: pgmap v6816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:27 smithi144 ceph-mon[32129]: pgmap v6816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:27 smithi160 ceph-mon[36715]: pgmap v6816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:29.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:29 smithi160 ceph-mon[36715]: pgmap v6817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:29 smithi183 ceph-mon[37273]: pgmap v6817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:29 smithi144 ceph-mon[32129]: pgmap v6817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:31 smithi183 ceph-mon[37273]: pgmap v6818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11914.803354 secs 2024-07-26T14:24:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:31 smithi144 ceph-mon[32129]: pgmap v6818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11914.803354 secs 2024-07-26T14:24:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:31 smithi160 ceph-mon[36715]: pgmap v6818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11914.803354 secs 2024-07-26T14:24:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:33 smithi183 ceph-mon[37273]: pgmap v6819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:33 smithi144 ceph-mon[32129]: pgmap v6819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:33 smithi160 ceph-mon[36715]: pgmap v6819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:35 smithi183 ceph-mon[37273]: pgmap v6820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:35 smithi144 ceph-mon[32129]: pgmap v6820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:35 smithi160 ceph-mon[36715]: pgmap v6820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11919.803486 secs 2024-07-26T14:24:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11919.803486 secs 2024-07-26T14:24:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11919.803486 secs 2024-07-26T14:24:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:37.445+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:24:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:37.446+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:24:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:37.446+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:24:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:37.446+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:24:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:37.446+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:37 smithi183 ceph-mon[37273]: pgmap v6821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:37 smithi144 ceph-mon[32129]: pgmap v6821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:37 smithi160 ceph-mon[36715]: pgmap v6821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:39 smithi183 ceph-mon[37273]: pgmap v6822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:39 smithi144 ceph-mon[32129]: pgmap v6822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:39 smithi160 ceph-mon[36715]: pgmap v6822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:41 smithi183 ceph-mon[37273]: pgmap v6823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11924.803621 secs 2024-07-26T14:24:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:41 smithi144 ceph-mon[32129]: pgmap v6823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11924.803621 secs 2024-07-26T14:24:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:41 smithi160 ceph-mon[36715]: pgmap v6823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11924.803621 secs 2024-07-26T14:24:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:43 smithi144 ceph-mon[32129]: pgmap v6824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:43 smithi160 ceph-mon[36715]: pgmap v6824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:43 smithi183 ceph-mon[37273]: pgmap v6824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:24:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:45 smithi144 ceph-mon[32129]: pgmap v6825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:45.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:45.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:45 smithi160 ceph-mon[36715]: pgmap v6825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:45 smithi183 ceph-mon[37273]: pgmap v6825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:24:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11929.803747 secs 2024-07-26T14:24:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:46 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11929.803747 secs 2024-07-26T14:24:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:46 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11929.803747 secs 2024-07-26T14:24:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:46 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:24:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:47 smithi144 ceph-mon[32129]: pgmap v6826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:47 smithi144 ceph-mon[32129]: pgmap v6827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:47 smithi160 ceph-mon[36715]: pgmap v6826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:47 smithi160 ceph-mon[36715]: pgmap v6827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:47 smithi183 ceph-mon[37273]: pgmap v6826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:24:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:47 smithi183 ceph-mon[37273]: pgmap v6827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:49 smithi144 ceph-mon[32129]: pgmap v6828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:49 smithi160 ceph-mon[36715]: pgmap v6828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:49 smithi183 ceph-mon[37273]: pgmap v6828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:51 smithi144 ceph-mon[32129]: pgmap v6829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11934.803836 secs 2024-07-26T14:24:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:51 smithi160 ceph-mon[36715]: pgmap v6829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11934.803836 secs 2024-07-26T14:24:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:51 smithi183 ceph-mon[37273]: pgmap v6829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11934.803836 secs 2024-07-26T14:24:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:52.456+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:24:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:52.457+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:24:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:52.457+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:24:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:52.457+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:24:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:24:52.457+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:53 smithi144 ceph-mon[32129]: pgmap v6830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:53 smithi160 ceph-mon[36715]: pgmap v6830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:53 smithi183 ceph-mon[37273]: pgmap v6830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:55 smithi144 ceph-mon[32129]: pgmap v6831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:55 smithi160 ceph-mon[36715]: pgmap v6831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:55 smithi183 ceph-mon[37273]: pgmap v6831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:24:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11939.803965 secs 2024-07-26T14:24:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11939.803965 secs 2024-07-26T14:24:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11939.803965 secs 2024-07-26T14:24:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:57 smithi144 ceph-mon[32129]: pgmap v6832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:24:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:57 smithi160 ceph-mon[36715]: pgmap v6832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:24:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:57 smithi183 ceph-mon[37273]: pgmap v6832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:24:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:24:59 smithi144 ceph-mon[32129]: pgmap v6833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:24:59 smithi160 ceph-mon[36715]: pgmap v6833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:24:59 smithi183 ceph-mon[37273]: pgmap v6833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:01 smithi144 ceph-mon[32129]: pgmap v6834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11944.804110 secs 2024-07-26T14:25:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:01 smithi160 ceph-mon[36715]: pgmap v6834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11944.804110 secs 2024-07-26T14:25:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:01 smithi183 ceph-mon[37273]: pgmap v6834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11944.804110 secs 2024-07-26T14:25:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:03 smithi144 ceph-mon[32129]: pgmap v6835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:03 smithi160 ceph-mon[36715]: pgmap v6835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:03 smithi183 ceph-mon[37273]: pgmap v6835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:05 smithi160 ceph-mon[36715]: pgmap v6836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:05 smithi183 ceph-mon[37273]: pgmap v6836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:05 smithi144 ceph-mon[32129]: pgmap v6836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11949.804243 secs 2024-07-26T14:25:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11949.804243 secs 2024-07-26T14:25:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11949.804243 secs 2024-07-26T14:25:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:07.464+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:25:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:07.465+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:25:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:07.465+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:25:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:07.465+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:25:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:07.465+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:07.997 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:07 smithi144 ceph-mon[32129]: pgmap v6837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:07 smithi160 ceph-mon[36715]: pgmap v6837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:07 smithi183 ceph-mon[37273]: pgmap v6837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:08 smithi160 ceph-mon[36715]: pgmap v6838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:08 smithi183 ceph-mon[37273]: pgmap v6838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:08 smithi144 ceph-mon[32129]: pgmap v6838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:11 smithi183 ceph-mon[37273]: pgmap v6839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:11 smithi144 ceph-mon[32129]: pgmap v6839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:11 smithi160 ceph-mon[36715]: pgmap v6839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11954.804349 secs 2024-07-26T14:25:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11954.804349 secs 2024-07-26T14:25:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11954.804349 secs 2024-07-26T14:25:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:13 smithi144 ceph-mon[32129]: pgmap v6840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:13 smithi160 ceph-mon[36715]: pgmap v6840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:13 smithi183 ceph-mon[37273]: pgmap v6840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:15 smithi144 ceph-mon[32129]: pgmap v6841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:15 smithi160 ceph-mon[36715]: pgmap v6841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:15 smithi183 ceph-mon[37273]: pgmap v6841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11959.804441 secs 2024-07-26T14:25:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:17 smithi144 ceph-mon[32129]: pgmap v6842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11959.804441 secs 2024-07-26T14:25:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:17 smithi160 ceph-mon[36715]: pgmap v6842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11959.804441 secs 2024-07-26T14:25:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:17 smithi183 ceph-mon[37273]: pgmap v6842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:19 smithi144 ceph-mon[32129]: pgmap v6843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:19 smithi160 ceph-mon[36715]: pgmap v6843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:19 smithi183 ceph-mon[37273]: pgmap v6843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:21 smithi144 ceph-mon[32129]: pgmap v6844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:21 smithi160 ceph-mon[36715]: pgmap v6844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:21 smithi183 ceph-mon[37273]: pgmap v6844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:22.473+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:25:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:22.475+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:25:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:22.475+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:25:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:22.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:25:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:22.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11964.804576 secs 2024-07-26T14:25:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11964.804576 secs 2024-07-26T14:25:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11964.804576 secs 2024-07-26T14:25:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:23 smithi144 ceph-mon[32129]: pgmap v6845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:23 smithi160 ceph-mon[36715]: pgmap v6845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:23 smithi183 ceph-mon[37273]: pgmap v6845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:25 smithi144 ceph-mon[32129]: pgmap v6846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:25 smithi160 ceph-mon[36715]: pgmap v6846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:25 smithi183 ceph-mon[37273]: pgmap v6846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11969.804643 secs 2024-07-26T14:25:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11969.804643 secs 2024-07-26T14:25:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11969.804643 secs 2024-07-26T14:25:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:27 smithi144 ceph-mon[32129]: pgmap v6847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:27 smithi160 ceph-mon[36715]: pgmap v6847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:27 smithi183 ceph-mon[37273]: pgmap v6847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:29 smithi144 ceph-mon[32129]: pgmap v6848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:29 smithi160 ceph-mon[36715]: pgmap v6848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:29 smithi183 ceph-mon[37273]: pgmap v6848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:31 smithi144 ceph-mon[32129]: pgmap v6849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11974.804799 secs 2024-07-26T14:25:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:31 smithi160 ceph-mon[36715]: pgmap v6849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11974.804799 secs 2024-07-26T14:25:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:31 smithi183 ceph-mon[37273]: pgmap v6849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11974.804799 secs 2024-07-26T14:25:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:33 smithi144 ceph-mon[32129]: pgmap v6850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:33 smithi160 ceph-mon[36715]: pgmap v6850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:33 smithi183 ceph-mon[37273]: pgmap v6850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:35 smithi160 ceph-mon[36715]: pgmap v6851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:35 smithi183 ceph-mon[37273]: pgmap v6851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:35 smithi144 ceph-mon[32129]: pgmap v6851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11979.804947 secs 2024-07-26T14:25:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11979.804947 secs 2024-07-26T14:25:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11979.804947 secs 2024-07-26T14:25:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:37.491+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:25:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:37.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:25:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:37.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:25:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:37.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:25:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:37.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:37 smithi160 ceph-mon[36715]: pgmap v6852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:37 smithi183 ceph-mon[37273]: pgmap v6852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:37 smithi144 ceph-mon[32129]: pgmap v6852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:39 smithi160 ceph-mon[36715]: pgmap v6853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:39 smithi183 ceph-mon[37273]: pgmap v6853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:39 smithi144 ceph-mon[32129]: pgmap v6853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:41 smithi160 ceph-mon[36715]: pgmap v6854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11984.805089 secs 2024-07-26T14:25:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:41 smithi183 ceph-mon[37273]: pgmap v6854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11984.805089 secs 2024-07-26T14:25:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:41 smithi144 ceph-mon[32129]: pgmap v6854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11984.805089 secs 2024-07-26T14:25:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:43 smithi160 ceph-mon[36715]: pgmap v6855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:43 smithi183 ceph-mon[37273]: pgmap v6855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:43 smithi144 ceph-mon[32129]: pgmap v6855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:45 smithi160 ceph-mon[36715]: pgmap v6856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:45 smithi183 ceph-mon[37273]: pgmap v6856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:45 smithi144 ceph-mon[32129]: pgmap v6856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11989.805234 secs 2024-07-26T14:25:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11989.805234 secs 2024-07-26T14:25:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11989.805234 secs 2024-07-26T14:25:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:47 smithi160 ceph-mon[36715]: pgmap v6857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:47 smithi144 ceph-mon[32129]: pgmap v6857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:47.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:47.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.864 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:47 smithi183 ceph-mon[37273]: pgmap v6857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:25:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: pgmap v6858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: pgmap v6858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: pgmap v6858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:50 smithi160 ceph-mon[36715]: pgmap v6859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:50 smithi183 ceph-mon[37273]: pgmap v6859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:50 smithi144 ceph-mon[32129]: pgmap v6859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:25:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11994.805335 secs 2024-07-26T14:25:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11994.805335 secs 2024-07-26T14:25:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11994.805335 secs 2024-07-26T14:25:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:52.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:25:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:52.509+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:25:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:52.509+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:25:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:52.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:25:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:25:52.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: pgmap v6860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: pgmap v6861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: pgmap v6860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: pgmap v6861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:53.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: pgmap v6860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:25:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: pgmap v6861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:25:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:55 smithi160 ceph-mon[36715]: pgmap v6862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:55 smithi183 ceph-mon[37273]: pgmap v6862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:55 smithi144 ceph-mon[32129]: pgmap v6862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 11999.805478 secs 2024-07-26T14:25:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:57 smithi160 ceph-mon[36715]: pgmap v6863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 11999.805478 secs 2024-07-26T14:25:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:57 smithi183 ceph-mon[37273]: pgmap v6863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 11999.805478 secs 2024-07-26T14:25:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:57 smithi144 ceph-mon[32129]: pgmap v6863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:25:59 smithi160 ceph-mon[36715]: pgmap v6864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:25:59 smithi183 ceph-mon[37273]: pgmap v6864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:25:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:25:59 smithi144 ceph-mon[32129]: pgmap v6864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:26:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:01 smithi160 ceph-mon[36715]: pgmap v6865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:26:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:01 smithi183 ceph-mon[37273]: pgmap v6865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:26:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:01 smithi144 ceph-mon[32129]: pgmap v6865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:26:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12004.805613 secs 2024-07-26T14:26:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12004.805613 secs 2024-07-26T14:26:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12004.805613 secs 2024-07-26T14:26:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:03 smithi160 ceph-mon[36715]: pgmap v6866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:26:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:03 smithi183 ceph-mon[37273]: pgmap v6866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:26:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:03 smithi144 ceph-mon[32129]: pgmap v6866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:26:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:05 smithi160 ceph-mon[36715]: pgmap v6867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:05 smithi183 ceph-mon[37273]: pgmap v6867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:05 smithi144 ceph-mon[32129]: pgmap v6867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:07.525+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:26:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:07.526+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:26:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:07.526+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:26:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:07.526+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:26:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:07.526+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12009.805740 secs 2024-07-26T14:26:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:07 smithi160 ceph-mon[36715]: pgmap v6868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12009.805740 secs 2024-07-26T14:26:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:07 smithi183 ceph-mon[37273]: pgmap v6868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12009.805740 secs 2024-07-26T14:26:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:07 smithi144 ceph-mon[32129]: pgmap v6868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:09 smithi160 ceph-mon[36715]: pgmap v6869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:09 smithi183 ceph-mon[37273]: pgmap v6869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:09 smithi144 ceph-mon[32129]: pgmap v6869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:11 smithi160 ceph-mon[36715]: pgmap v6870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:11 smithi183 ceph-mon[37273]: pgmap v6870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:11 smithi144 ceph-mon[32129]: pgmap v6870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12014.805871 secs 2024-07-26T14:26:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12014.805871 secs 2024-07-26T14:26:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12014.805871 secs 2024-07-26T14:26:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:13 smithi183 ceph-mon[37273]: pgmap v6871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:13 smithi144 ceph-mon[32129]: pgmap v6871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:13 smithi160 ceph-mon[36715]: pgmap v6871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:15 smithi183 ceph-mon[37273]: pgmap v6872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:15 smithi144 ceph-mon[32129]: pgmap v6872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:15 smithi160 ceph-mon[36715]: pgmap v6872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12019.806015 secs 2024-07-26T14:26:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:17 smithi183 ceph-mon[37273]: pgmap v6873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12019.806015 secs 2024-07-26T14:26:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:17 smithi144 ceph-mon[32129]: pgmap v6873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12019.806015 secs 2024-07-26T14:26:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:17 smithi160 ceph-mon[36715]: pgmap v6873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:19 smithi183 ceph-mon[37273]: pgmap v6874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:19 smithi144 ceph-mon[32129]: pgmap v6874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:19 smithi160 ceph-mon[36715]: pgmap v6874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:21 smithi183 ceph-mon[37273]: pgmap v6875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:21 smithi144 ceph-mon[32129]: pgmap v6875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:21 smithi160 ceph-mon[36715]: pgmap v6875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:22.531+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:26:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:22.531+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:26:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:22.531+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:26:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:22.531+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:26:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:22.532+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12024.806140 secs 2024-07-26T14:26:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12024.806140 secs 2024-07-26T14:26:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12024.806140 secs 2024-07-26T14:26:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:23 smithi183 ceph-mon[37273]: pgmap v6876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:23 smithi144 ceph-mon[32129]: pgmap v6876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:23 smithi160 ceph-mon[36715]: pgmap v6876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:25 smithi183 ceph-mon[37273]: pgmap v6877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:25 smithi144 ceph-mon[32129]: pgmap v6877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:25 smithi160 ceph-mon[36715]: pgmap v6877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12029.806250 secs 2024-07-26T14:26:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:27 smithi183 ceph-mon[37273]: pgmap v6878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12029.806250 secs 2024-07-26T14:26:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:27 smithi144 ceph-mon[32129]: pgmap v6878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12029.806250 secs 2024-07-26T14:26:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:27 smithi160 ceph-mon[36715]: pgmap v6878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:29 smithi183 ceph-mon[37273]: pgmap v6879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:29 smithi144 ceph-mon[32129]: pgmap v6879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:29 smithi160 ceph-mon[36715]: pgmap v6879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:31 smithi183 ceph-mon[37273]: pgmap v6880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:31 smithi144 ceph-mon[32129]: pgmap v6880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:31 smithi160 ceph-mon[36715]: pgmap v6880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12034.806409 secs 2024-07-26T14:26:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12034.806409 secs 2024-07-26T14:26:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12034.806409 secs 2024-07-26T14:26:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:33 smithi144 ceph-mon[32129]: pgmap v6881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:33 smithi160 ceph-mon[36715]: pgmap v6881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:33 smithi183 ceph-mon[37273]: pgmap v6881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:35 smithi144 ceph-mon[32129]: pgmap v6882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:35 smithi160 ceph-mon[36715]: pgmap v6882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:35 smithi183 ceph-mon[37273]: pgmap v6882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:37.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:37.548+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:26:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:37.549+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:26:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:37.549+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:26:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:37.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:26:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:37.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12039.806478 secs 2024-07-26T14:26:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:37 smithi144 ceph-mon[32129]: pgmap v6883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12039.806478 secs 2024-07-26T14:26:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:37 smithi160 ceph-mon[36715]: pgmap v6883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12039.806478 secs 2024-07-26T14:26:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:37 smithi183 ceph-mon[37273]: pgmap v6883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:39 smithi144 ceph-mon[32129]: pgmap v6884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:39 smithi160 ceph-mon[36715]: pgmap v6884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:39 smithi183 ceph-mon[37273]: pgmap v6884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:41 smithi144 ceph-mon[32129]: pgmap v6885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:41.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:41 smithi183 ceph-mon[37273]: pgmap v6885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:41 smithi160 ceph-mon[36715]: pgmap v6885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12044.806613 secs 2024-07-26T14:26:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12044.806613 secs 2024-07-26T14:26:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12044.806613 secs 2024-07-26T14:26:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:43 smithi144 ceph-mon[32129]: pgmap v6886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:43 smithi160 ceph-mon[36715]: pgmap v6886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:43 smithi183 ceph-mon[37273]: pgmap v6886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:45 smithi144 ceph-mon[32129]: pgmap v6887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:45 smithi160 ceph-mon[36715]: pgmap v6887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:45 smithi183 ceph-mon[37273]: pgmap v6887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12049.806770 secs 2024-07-26T14:26:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12049.806770 secs 2024-07-26T14:26:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12049.806770 secs 2024-07-26T14:26:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:47 smithi144 ceph-mon[32129]: pgmap v6888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:47 smithi160 ceph-mon[36715]: pgmap v6888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:47 smithi183 ceph-mon[37273]: pgmap v6888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:48.229+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:26:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:48.229+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:26:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:48.230+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:26:48.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:48.230+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:26:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:49 smithi144 ceph-mon[32129]: pgmap v6889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:49 smithi160 ceph-mon[36715]: pgmap v6889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:49 smithi183 ceph-mon[37273]: pgmap v6889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:26:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:51 smithi144 ceph-mon[32129]: pgmap v6890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12054.806869 secs 2024-07-26T14:26:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:51 smithi160 ceph-mon[36715]: pgmap v6890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12054.806869 secs 2024-07-26T14:26:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:51 smithi183 ceph-mon[37273]: pgmap v6890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:26:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12054.806869 secs 2024-07-26T14:26:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:52.564+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:26:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:52.564+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:26:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:52.564+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:26:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:52.564+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:26:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:26:52.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:53 smithi160 ceph-mon[36715]: pgmap v6891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:26:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:53 smithi183 ceph-mon[37273]: pgmap v6891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:26:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:53 smithi144 ceph-mon[32129]: pgmap v6891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:26:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:26:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:54 smithi160 ceph-mon[36715]: pgmap v6892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:26:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:54 smithi160 ceph-mon[36715]: pgmap v6893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:26:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:54 smithi183 ceph-mon[37273]: pgmap v6892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:26:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:54 smithi183 ceph-mon[37273]: pgmap v6893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:26:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:54 smithi144 ceph-mon[32129]: pgmap v6892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:26:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:54 smithi144 ceph-mon[32129]: pgmap v6893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:26:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:56 smithi160 ceph-mon[36715]: pgmap v6894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:26:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12059.806992 secs 2024-07-26T14:26:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:56 smithi183 ceph-mon[37273]: pgmap v6894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:26:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12059.806992 secs 2024-07-26T14:26:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:56 smithi144 ceph-mon[32129]: pgmap v6894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:26:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12059.806992 secs 2024-07-26T14:26:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:26:58 smithi160 ceph-mon[36715]: pgmap v6895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:26:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:26:58 smithi183 ceph-mon[37273]: pgmap v6895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:26:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:26:58 smithi144 ceph-mon[32129]: pgmap v6895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:00 smithi160 ceph-mon[36715]: pgmap v6896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:00 smithi183 ceph-mon[37273]: pgmap v6896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:00 smithi144 ceph-mon[32129]: pgmap v6896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12064.807129 secs 2024-07-26T14:27:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12064.807129 secs 2024-07-26T14:27:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12064.807129 secs 2024-07-26T14:27:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:02 smithi160 ceph-mon[36715]: pgmap v6897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:02 smithi183 ceph-mon[37273]: pgmap v6897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:02 smithi144 ceph-mon[32129]: pgmap v6897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:04 smithi160 ceph-mon[36715]: pgmap v6898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:04 smithi183 ceph-mon[37273]: pgmap v6898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:04 smithi144 ceph-mon[32129]: pgmap v6898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:06 smithi160 ceph-mon[36715]: pgmap v6899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12069.807257 secs 2024-07-26T14:27:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:06 smithi183 ceph-mon[37273]: pgmap v6899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12069.807257 secs 2024-07-26T14:27:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:06 smithi144 ceph-mon[32129]: pgmap v6899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12069.807257 secs 2024-07-26T14:27:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:07.579+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:27:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:07.580+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:27:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:07.580+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:27:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:07.580+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:27:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:07.580+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:08 smithi160 ceph-mon[36715]: pgmap v6900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:08 smithi183 ceph-mon[37273]: pgmap v6900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:08 smithi144 ceph-mon[32129]: pgmap v6900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:10 smithi160 ceph-mon[36715]: pgmap v6901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:10 smithi183 ceph-mon[37273]: pgmap v6901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:10 smithi144 ceph-mon[32129]: pgmap v6901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12074.807370 secs 2024-07-26T14:27:11.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12074.807370 secs 2024-07-26T14:27:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12074.807370 secs 2024-07-26T14:27:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:12 smithi183 ceph-mon[37273]: pgmap v6902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:12 smithi144 ceph-mon[32129]: pgmap v6902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:12 smithi160 ceph-mon[36715]: pgmap v6902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:14 smithi183 ceph-mon[37273]: pgmap v6903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:14 smithi144 ceph-mon[32129]: pgmap v6903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:14 smithi160 ceph-mon[36715]: pgmap v6903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:16 smithi183 ceph-mon[37273]: pgmap v6904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12079.807436 secs 2024-07-26T14:27:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:16 smithi144 ceph-mon[32129]: pgmap v6904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12079.807436 secs 2024-07-26T14:27:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:16 smithi160 ceph-mon[36715]: pgmap v6904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12079.807436 secs 2024-07-26T14:27:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:18 smithi183 ceph-mon[37273]: pgmap v6905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:18 smithi144 ceph-mon[32129]: pgmap v6905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:18 smithi160 ceph-mon[36715]: pgmap v6905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:20 smithi183 ceph-mon[37273]: pgmap v6906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:27:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:20 smithi144 ceph-mon[32129]: pgmap v6906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:27:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:20 smithi160 ceph-mon[36715]: pgmap v6906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:27:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12084.807583 secs 2024-07-26T14:27:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12084.807583 secs 2024-07-26T14:27:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12084.807583 secs 2024-07-26T14:27:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:22.595+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:27:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:22.596+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:27:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:22.596+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:27:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:22.596+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:27:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:22.596+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:22 smithi183 ceph-mon[37273]: pgmap v6907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:22 smithi144 ceph-mon[32129]: pgmap v6907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:22 smithi160 ceph-mon[36715]: pgmap v6907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:24 smithi183 ceph-mon[37273]: pgmap v6908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:24 smithi144 ceph-mon[32129]: pgmap v6908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:24 smithi160 ceph-mon[36715]: pgmap v6908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:27:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:26 smithi183 ceph-mon[37273]: pgmap v6909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12089.807659 secs 2024-07-26T14:27:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:26 smithi144 ceph-mon[32129]: pgmap v6909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12089.807659 secs 2024-07-26T14:27:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:26 smithi160 ceph-mon[36715]: pgmap v6909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12089.807659 secs 2024-07-26T14:27:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:28 smithi183 ceph-mon[37273]: pgmap v6910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:28 smithi144 ceph-mon[32129]: pgmap v6910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:28 smithi160 ceph-mon[36715]: pgmap v6910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:30 smithi183 ceph-mon[37273]: pgmap v6911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:30 smithi144 ceph-mon[32129]: pgmap v6911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:30 smithi160 ceph-mon[36715]: pgmap v6911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12094.807782 secs 2024-07-26T14:27:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12094.807782 secs 2024-07-26T14:27:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12094.807782 secs 2024-07-26T14:27:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:32 smithi144 ceph-mon[32129]: pgmap v6912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:32 smithi160 ceph-mon[36715]: pgmap v6912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:32 smithi183 ceph-mon[37273]: pgmap v6912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:34 smithi144 ceph-mon[32129]: pgmap v6913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:34 smithi160 ceph-mon[36715]: pgmap v6913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:34 smithi183 ceph-mon[37273]: pgmap v6913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:36 smithi144 ceph-mon[32129]: pgmap v6914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12099.807928 secs 2024-07-26T14:27:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:36 smithi160 ceph-mon[36715]: pgmap v6914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12099.807928 secs 2024-07-26T14:27:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:36 smithi183 ceph-mon[37273]: pgmap v6914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12099.807928 secs 2024-07-26T14:27:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:37.613+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:27:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:37.614+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:27:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:37.614+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:27:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:37.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:27:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:37.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:38 smithi144 ceph-mon[32129]: pgmap v6915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:38 smithi160 ceph-mon[36715]: pgmap v6915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:38 smithi183 ceph-mon[37273]: pgmap v6915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:40 smithi144 ceph-mon[32129]: pgmap v6916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:40 smithi160 ceph-mon[36715]: pgmap v6916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:40 smithi183 ceph-mon[37273]: pgmap v6916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12104.808061 secs 2024-07-26T14:27:42.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12104.808061 secs 2024-07-26T14:27:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12104.808061 secs 2024-07-26T14:27:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:42 smithi144 ceph-mon[32129]: pgmap v6917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:42 smithi160 ceph-mon[36715]: pgmap v6917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:42 smithi183 ceph-mon[37273]: pgmap v6917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:44 smithi144 ceph-mon[32129]: pgmap v6918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:44 smithi160 ceph-mon[36715]: pgmap v6918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:44 smithi183 ceph-mon[37273]: pgmap v6918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:46 smithi144 ceph-mon[32129]: pgmap v6919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12109.808207 secs 2024-07-26T14:27:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:46 smithi160 ceph-mon[36715]: pgmap v6919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12109.808207 secs 2024-07-26T14:27:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:46 smithi183 ceph-mon[37273]: pgmap v6919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12109.808207 secs 2024-07-26T14:27:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:48 smithi144 ceph-mon[32129]: pgmap v6920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:48 smithi160 ceph-mon[36715]: pgmap v6920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:48 smithi183 ceph-mon[37273]: pgmap v6920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:27:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:50 smithi144 ceph-mon[32129]: pgmap v6921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:50 smithi160 ceph-mon[36715]: pgmap v6921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:50 smithi183 ceph-mon[37273]: pgmap v6921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12114.808330 secs 2024-07-26T14:27:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12114.808330 secs 2024-07-26T14:27:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12114.808330 secs 2024-07-26T14:27:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:52.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:27:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:52.631+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:27:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:52.631+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:27:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:52.631+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:27:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:27:52.631+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:52 smithi144 ceph-mon[32129]: pgmap v6922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:52 smithi160 ceph-mon[36715]: pgmap v6922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:52 smithi183 ceph-mon[37273]: pgmap v6922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:27:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:54 smithi160 ceph-mon[36715]: pgmap v6923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:27:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:54 smithi183 ceph-mon[37273]: pgmap v6923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:27:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:54 smithi144 ceph-mon[32129]: pgmap v6923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:27:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:27:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:55 smithi160 ceph-mon[36715]: pgmap v6924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:27:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:55 smithi160 ceph-mon[36715]: pgmap v6925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:55 smithi183 ceph-mon[37273]: pgmap v6924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:27:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:55 smithi183 ceph-mon[37273]: pgmap v6925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:55 smithi144 ceph-mon[32129]: pgmap v6924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:27:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:55 smithi144 ceph-mon[32129]: pgmap v6925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:27:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12119.808454 secs 2024-07-26T14:27:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12119.808454 secs 2024-07-26T14:27:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12119.808454 secs 2024-07-26T14:27:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:57 smithi160 ceph-mon[36715]: pgmap v6926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:57 smithi183 ceph-mon[37273]: pgmap v6926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:27:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:57 smithi144 ceph-mon[32129]: pgmap v6926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:27:59 smithi160 ceph-mon[36715]: pgmap v6927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:28:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:27:59 smithi183 ceph-mon[37273]: pgmap v6927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:28:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:27:59 smithi144 ceph-mon[32129]: pgmap v6927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:28:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:01 smithi160 ceph-mon[36715]: pgmap v6928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12124.808585 secs 2024-07-26T14:28:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:01 smithi183 ceph-mon[37273]: pgmap v6928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12124.808585 secs 2024-07-26T14:28:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:01 smithi144 ceph-mon[32129]: pgmap v6928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12124.808585 secs 2024-07-26T14:28:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:03 smithi160 ceph-mon[36715]: pgmap v6929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:28:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:03 smithi183 ceph-mon[37273]: pgmap v6929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:28:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:03 smithi144 ceph-mon[32129]: pgmap v6929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:28:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:04 smithi160 ceph-mon[36715]: pgmap v6930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:28:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:04 smithi183 ceph-mon[37273]: pgmap v6930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:28:05.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:04 smithi144 ceph-mon[32129]: pgmap v6930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:28:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12129.808664 secs 2024-07-26T14:28:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12129.808664 secs 2024-07-26T14:28:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12129.808664 secs 2024-07-26T14:28:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:07.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:28:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:07.639+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:28:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:07.639+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:28:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:07.639+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:28:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:07.639+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:07 smithi160 ceph-mon[36715]: pgmap v6931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:07 smithi183 ceph-mon[37273]: pgmap v6931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:07 smithi144 ceph-mon[32129]: pgmap v6931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:09 smithi160 ceph-mon[36715]: pgmap v6932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:09 smithi183 ceph-mon[37273]: pgmap v6932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:10.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:09 smithi144 ceph-mon[32129]: pgmap v6932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:12.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:11 smithi144 ceph-mon[32129]: pgmap v6933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:12.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12134.808797 secs 2024-07-26T14:28:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:11 smithi160 ceph-mon[36715]: pgmap v6933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12134.808797 secs 2024-07-26T14:28:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:11 smithi183 ceph-mon[37273]: pgmap v6933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12134.808797 secs 2024-07-26T14:28:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:13 smithi160 ceph-mon[36715]: pgmap v6934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:13 smithi183 ceph-mon[37273]: pgmap v6934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:13 smithi144 ceph-mon[32129]: pgmap v6934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:15 smithi160 ceph-mon[36715]: pgmap v6935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:15 smithi183 ceph-mon[37273]: pgmap v6935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:15 smithi144 ceph-mon[32129]: pgmap v6935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12139.808925 secs 2024-07-26T14:28:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12139.808925 secs 2024-07-26T14:28:17.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12139.808925 secs 2024-07-26T14:28:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:17 smithi160 ceph-mon[36715]: pgmap v6936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:17 smithi183 ceph-mon[37273]: pgmap v6936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:17 smithi144 ceph-mon[32129]: pgmap v6936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:19 smithi160 ceph-mon[36715]: pgmap v6937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:19 smithi183 ceph-mon[37273]: pgmap v6937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:19 smithi144 ceph-mon[32129]: pgmap v6937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:21 smithi160 ceph-mon[36715]: pgmap v6938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12144.809068 secs 2024-07-26T14:28:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:21 smithi183 ceph-mon[37273]: pgmap v6938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12144.809068 secs 2024-07-26T14:28:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:21 smithi144 ceph-mon[32129]: pgmap v6938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12144.809068 secs 2024-07-26T14:28:22.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:22.654+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:28:22.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:22.655+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:28:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:22.655+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:28:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:22.655+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:28:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:22.655+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:23 smithi160 ceph-mon[36715]: pgmap v6939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:23 smithi183 ceph-mon[37273]: pgmap v6939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:23 smithi144 ceph-mon[32129]: pgmap v6939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:25 smithi160 ceph-mon[36715]: pgmap v6940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:25 smithi183 ceph-mon[37273]: pgmap v6940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:25 smithi144 ceph-mon[32129]: pgmap v6940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12149.809193 secs 2024-07-26T14:28:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12149.809193 secs 2024-07-26T14:28:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12149.809193 secs 2024-07-26T14:28:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:27 smithi160 ceph-mon[36715]: pgmap v6941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:27 smithi183 ceph-mon[37273]: pgmap v6941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:27 smithi144 ceph-mon[32129]: pgmap v6941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:28 smithi183 ceph-mon[37273]: pgmap v6942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:29.213 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:28 smithi144 ceph-mon[32129]: pgmap v6942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:28 smithi160 ceph-mon[36715]: pgmap v6942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:32.012 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:31 smithi160 ceph-mon[36715]: pgmap v6943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:32.012 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12154.809324 secs 2024-07-26T14:28:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:31 smithi183 ceph-mon[37273]: pgmap v6943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12154.809324 secs 2024-07-26T14:28:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:31 smithi144 ceph-mon[32129]: pgmap v6943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12154.809324 secs 2024-07-26T14:28:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:33 smithi160 ceph-mon[36715]: pgmap v6944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:33 smithi183 ceph-mon[37273]: pgmap v6944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:33 smithi144 ceph-mon[32129]: pgmap v6944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:35 smithi160 ceph-mon[36715]: pgmap v6945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:35 smithi183 ceph-mon[37273]: pgmap v6945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:35 smithi144 ceph-mon[32129]: pgmap v6945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12159.809404 secs 2024-07-26T14:28:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12159.809404 secs 2024-07-26T14:28:37.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12159.809404 secs 2024-07-26T14:28:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:37.672+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:28:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:37.673+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:28:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:37.673+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:28:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:37.673+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:28:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:37.673+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:37 smithi160 ceph-mon[36715]: pgmap v6946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:37 smithi183 ceph-mon[37273]: pgmap v6946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:37 smithi144 ceph-mon[32129]: pgmap v6946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:39 smithi160 ceph-mon[36715]: pgmap v6947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:39 smithi183 ceph-mon[37273]: pgmap v6947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:39 smithi144 ceph-mon[32129]: pgmap v6947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:42.023 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:41 smithi144 ceph-mon[32129]: pgmap v6948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:42.023 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12164.809503 secs 2024-07-26T14:28:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:41 smithi160 ceph-mon[36715]: pgmap v6948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12164.809503 secs 2024-07-26T14:28:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:41 smithi183 ceph-mon[37273]: pgmap v6948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12164.809503 secs 2024-07-26T14:28:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:43 smithi160 ceph-mon[36715]: pgmap v6949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:43 smithi183 ceph-mon[37273]: pgmap v6949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:43 smithi144 ceph-mon[32129]: pgmap v6949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:45 smithi160 ceph-mon[36715]: pgmap v6950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:45 smithi183 ceph-mon[37273]: pgmap v6950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:45 smithi144 ceph-mon[32129]: pgmap v6950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12169.809609 secs 2024-07-26T14:28:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12169.809609 secs 2024-07-26T14:28:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12169.809609 secs 2024-07-26T14:28:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:47 smithi160 ceph-mon[36715]: pgmap v6951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:47 smithi183 ceph-mon[37273]: pgmap v6951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:47 smithi144 ceph-mon[32129]: pgmap v6951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:28:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:48 smithi183 ceph-mon[37273]: pgmap v6952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:48 smithi144 ceph-mon[32129]: pgmap v6952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:48 smithi160 ceph-mon[36715]: pgmap v6952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:51 smithi160 ceph-mon[36715]: pgmap v6953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12174.809738 secs 2024-07-26T14:28:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:51 smithi183 ceph-mon[37273]: pgmap v6953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12174.809738 secs 2024-07-26T14:28:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:51 smithi144 ceph-mon[32129]: pgmap v6953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12174.809738 secs 2024-07-26T14:28:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:52.675+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:28:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:52.676+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:28:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:52.676+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:28:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:52.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:28:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:28:52.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:53 smithi160 ceph-mon[36715]: pgmap v6954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:53 smithi183 ceph-mon[37273]: pgmap v6954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:53 smithi144 ceph-mon[32129]: pgmap v6954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:28:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:55 smithi160 ceph-mon[36715]: pgmap v6955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:55 smithi183 ceph-mon[37273]: pgmap v6955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:55 smithi144 ceph-mon[32129]: pgmap v6955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:28:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12179.809813 secs 2024-07-26T14:28:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:56 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12179.809813 secs 2024-07-26T14:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:56 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12179.809813 secs 2024-07-26T14:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:56 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:28:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:57 smithi160 ceph-mon[36715]: pgmap v6956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:57 smithi160 ceph-mon[36715]: pgmap v6957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:57 smithi183 ceph-mon[37273]: pgmap v6956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:57 smithi183 ceph-mon[37273]: pgmap v6957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:57 smithi144 ceph-mon[32129]: pgmap v6956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:28:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:57 smithi144 ceph-mon[32129]: pgmap v6957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:29:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:28:59 smithi160 ceph-mon[36715]: pgmap v6958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:29:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:28:59 smithi183 ceph-mon[37273]: pgmap v6958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:29:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:28:59 smithi144 ceph-mon[32129]: pgmap v6958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:29:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:01 smithi160 ceph-mon[36715]: pgmap v6959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:29:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12184.809937 secs 2024-07-26T14:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:01 smithi183 ceph-mon[37273]: pgmap v6959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12184.809937 secs 2024-07-26T14:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:01 smithi144 ceph-mon[32129]: pgmap v6959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12184.809937 secs 2024-07-26T14:29:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:03 smithi160 ceph-mon[36715]: pgmap v6960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:29:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:03 smithi183 ceph-mon[37273]: pgmap v6960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:29:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:03 smithi144 ceph-mon[32129]: pgmap v6960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:29:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:04 smithi160 ceph-mon[36715]: pgmap v6961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:29:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:04 smithi183 ceph-mon[37273]: pgmap v6961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:29:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:04 smithi144 ceph-mon[32129]: pgmap v6961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:29:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12189.810057 secs 2024-07-26T14:29:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:06 smithi144 ceph-mon[32129]: pgmap v6962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:29:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12189.810057 secs 2024-07-26T14:29:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:06 smithi160 ceph-mon[36715]: pgmap v6962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:29:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12189.810057 secs 2024-07-26T14:29:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:06 smithi183 ceph-mon[37273]: pgmap v6962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:29:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:07.683+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:29:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:07.684+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:29:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:07.684+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:29:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:07.684+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:29:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:07.685+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:08 smithi144 ceph-mon[32129]: pgmap v6963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:08 smithi160 ceph-mon[36715]: pgmap v6963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:08 smithi183 ceph-mon[37273]: pgmap v6963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:10 smithi144 ceph-mon[32129]: pgmap v6964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:10 smithi160 ceph-mon[36715]: pgmap v6964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:10 smithi183 ceph-mon[37273]: pgmap v6964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12194.810175 secs 2024-07-26T14:29:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12194.810175 secs 2024-07-26T14:29:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12194.810175 secs 2024-07-26T14:29:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:12 smithi144 ceph-mon[32129]: pgmap v6965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:13.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:12 smithi183 ceph-mon[37273]: pgmap v6965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:12 smithi160 ceph-mon[36715]: pgmap v6965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:14 smithi144 ceph-mon[32129]: pgmap v6966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:14 smithi160 ceph-mon[36715]: pgmap v6966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:14 smithi183 ceph-mon[37273]: pgmap v6966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12199.810327 secs 2024-07-26T14:29:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:16 smithi144 ceph-mon[32129]: pgmap v6967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12199.810327 secs 2024-07-26T14:29:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:16 smithi160 ceph-mon[36715]: pgmap v6967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12199.810327 secs 2024-07-26T14:29:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:16 smithi183 ceph-mon[37273]: pgmap v6967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:18 smithi144 ceph-mon[32129]: pgmap v6968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:18 smithi160 ceph-mon[36715]: pgmap v6968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:18 smithi183 ceph-mon[37273]: pgmap v6968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:20 smithi144 ceph-mon[32129]: pgmap v6969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:20 smithi160 ceph-mon[36715]: pgmap v6969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:20 smithi183 ceph-mon[37273]: pgmap v6969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12204.810396 secs 2024-07-26T14:29:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12204.810396 secs 2024-07-26T14:29:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12204.810396 secs 2024-07-26T14:29:22.691 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:22.689+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:29:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:22.690+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:29:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:22.690+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:29:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:22.690+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:29:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:22.690+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:22 smithi144 ceph-mon[32129]: pgmap v6970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:22 smithi160 ceph-mon[36715]: pgmap v6970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:22 smithi183 ceph-mon[37273]: pgmap v6970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:24 smithi160 ceph-mon[36715]: pgmap v6971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:24 smithi183 ceph-mon[37273]: pgmap v6971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:24 smithi144 ceph-mon[32129]: pgmap v6971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12209.810555 secs 2024-07-26T14:29:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:26 smithi160 ceph-mon[36715]: pgmap v6972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12209.810555 secs 2024-07-26T14:29:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:26 smithi183 ceph-mon[37273]: pgmap v6972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12209.810555 secs 2024-07-26T14:29:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:26 smithi144 ceph-mon[32129]: pgmap v6972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:28 smithi160 ceph-mon[36715]: pgmap v6973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:28 smithi183 ceph-mon[37273]: pgmap v6973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:28 smithi144 ceph-mon[32129]: pgmap v6973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:30 smithi160 ceph-mon[36715]: pgmap v6974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:30 smithi183 ceph-mon[37273]: pgmap v6974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:30 smithi144 ceph-mon[32129]: pgmap v6974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12214.810687 secs 2024-07-26T14:29:32.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12214.810687 secs 2024-07-26T14:29:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12214.810687 secs 2024-07-26T14:29:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:33 smithi160 ceph-mon[36715]: pgmap v6975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:33 smithi183 ceph-mon[37273]: pgmap v6975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:33 smithi144 ceph-mon[32129]: pgmap v6975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:35 smithi160 ceph-mon[36715]: pgmap v6976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:35 smithi183 ceph-mon[37273]: pgmap v6976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:35 smithi144 ceph-mon[32129]: pgmap v6976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12219.810812 secs 2024-07-26T14:29:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:37 smithi160 ceph-mon[36715]: pgmap v6977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12219.810812 secs 2024-07-26T14:29:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:37 smithi183 ceph-mon[37273]: pgmap v6977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12219.810812 secs 2024-07-26T14:29:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:37 smithi144 ceph-mon[32129]: pgmap v6977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:37.693+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:29:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:37.694+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:29:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:37.694+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:29:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:37.694+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:29:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:37.694+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:39 smithi160 ceph-mon[36715]: pgmap v6978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:39 smithi183 ceph-mon[37273]: pgmap v6978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:39 smithi144 ceph-mon[32129]: pgmap v6978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:41 smithi160 ceph-mon[36715]: pgmap v6979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:41 smithi183 ceph-mon[37273]: pgmap v6979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:41 smithi144 ceph-mon[32129]: pgmap v6979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12224.810946 secs 2024-07-26T14:29:42.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12224.810946 secs 2024-07-26T14:29:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12224.810946 secs 2024-07-26T14:29:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:43 smithi160 ceph-mon[36715]: pgmap v6980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:43 smithi183 ceph-mon[37273]: pgmap v6980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:43 smithi144 ceph-mon[32129]: pgmap v6980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:45 smithi160 ceph-mon[36715]: pgmap v6981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:45 smithi183 ceph-mon[37273]: pgmap v6981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:45 smithi144 ceph-mon[32129]: pgmap v6981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12229.811067 secs 2024-07-26T14:29:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:47 smithi160 ceph-mon[36715]: pgmap v6982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12229.811067 secs 2024-07-26T14:29:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:47 smithi183 ceph-mon[37273]: pgmap v6982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12229.811067 secs 2024-07-26T14:29:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:47 smithi144 ceph-mon[32129]: pgmap v6982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:48.262+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:29:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:48.262+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:29:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:48.263+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:29:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:48.263+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:29:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:49 smithi160 ceph-mon[36715]: pgmap v6983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:49 smithi183 ceph-mon[37273]: pgmap v6983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:49 smithi144 ceph-mon[32129]: pgmap v6983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:29:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:51 smithi183 ceph-mon[37273]: pgmap v6984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:51 smithi144 ceph-mon[32129]: pgmap v6984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:51 smithi160 ceph-mon[36715]: pgmap v6984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12234.811211 secs 2024-07-26T14:29:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12234.811211 secs 2024-07-26T14:29:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12234.811211 secs 2024-07-26T14:29:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:52.710+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:29:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:52.711+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:29:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:52.711+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:29:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:52.711+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:29:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:29:52.711+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:53 smithi183 ceph-mon[37273]: pgmap v6985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:53 smithi144 ceph-mon[32129]: pgmap v6985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:53 smithi160 ceph-mon[36715]: pgmap v6985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:29:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:55 smithi183 ceph-mon[37273]: pgmap v6986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:55 smithi144 ceph-mon[32129]: pgmap v6986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:55 smithi160 ceph-mon[36715]: pgmap v6986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:56.468 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12239.811357 secs 2024-07-26T14:29:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:57 smithi183 ceph-mon[37273]: pgmap v6987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:29:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12239.811357 secs 2024-07-26T14:29:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:57 smithi144 ceph-mon[32129]: pgmap v6987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:29:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12239.811357 secs 2024-07-26T14:29:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:57 smithi160 ceph-mon[36715]: pgmap v6987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:29:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:58 smithi183 ceph-mon[37273]: pgmap v6988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:29:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:29:58 smithi183 ceph-mon[37273]: pgmap v6989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:29:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:58 smithi144 ceph-mon[32129]: pgmap v6988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:29:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:29:58 smithi144 ceph-mon[32129]: pgmap v6989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:29:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:58 smithi160 ceph-mon[36715]: pgmap v6988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:29:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:29:58 smithi160 ceph-mon[36715]: pgmap v6989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:00 smithi183 ceph-mon[37273]: pgmap v6990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:30:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:30:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:00 smithi144 ceph-mon[32129]: pgmap v6990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:30:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:30:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:00 smithi160 ceph-mon[36715]: pgmap v6990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:30:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:30:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12244.811472 secs 2024-07-26T14:30:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:02 smithi183 ceph-mon[37273]: pgmap v6991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12244.811472 secs 2024-07-26T14:30:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:02 smithi144 ceph-mon[32129]: pgmap v6991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12244.811472 secs 2024-07-26T14:30:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:02 smithi160 ceph-mon[36715]: pgmap v6991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:04 smithi183 ceph-mon[37273]: pgmap v6992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:30:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:04 smithi144 ceph-mon[32129]: pgmap v6992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:30:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:04 smithi160 ceph-mon[36715]: pgmap v6992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:30:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:06 smithi183 ceph-mon[37273]: pgmap v6993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:30:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:06 smithi144 ceph-mon[32129]: pgmap v6993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:30:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:06 smithi160 ceph-mon[36715]: pgmap v6993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:30:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12249.811577 secs 2024-07-26T14:30:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12249.811577 secs 2024-07-26T14:30:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12249.811577 secs 2024-07-26T14:30:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:07.727+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:07.728+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:07.728+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:07.728+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:07.728+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:08 smithi144 ceph-mon[32129]: pgmap v6994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:30:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:08 smithi160 ceph-mon[36715]: pgmap v6994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:30:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:08 smithi183 ceph-mon[37273]: pgmap v6994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:30:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:10 smithi144 ceph-mon[32129]: pgmap v6995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:10 smithi160 ceph-mon[36715]: pgmap v6995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:10 smithi183 ceph-mon[37273]: pgmap v6995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12254.811705 secs 2024-07-26T14:30:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:12 smithi144 ceph-mon[32129]: pgmap v6996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12254.811705 secs 2024-07-26T14:30:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:12 smithi160 ceph-mon[36715]: pgmap v6996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12254.811705 secs 2024-07-26T14:30:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:12 smithi183 ceph-mon[37273]: pgmap v6996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:14.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:14 smithi183 ceph-mon[37273]: pgmap v6997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:14 smithi144 ceph-mon[32129]: pgmap v6997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:14 smithi160 ceph-mon[36715]: pgmap v6997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:16 smithi144 ceph-mon[32129]: pgmap v6998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:16 smithi160 ceph-mon[36715]: pgmap v6998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:16 smithi183 ceph-mon[37273]: pgmap v6998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12259.811808 secs 2024-07-26T14:30:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12259.811808 secs 2024-07-26T14:30:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12259.811808 secs 2024-07-26T14:30:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:18 smithi144 ceph-mon[32129]: pgmap v6999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:18 smithi160 ceph-mon[36715]: pgmap v6999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:18 smithi183 ceph-mon[37273]: pgmap v6999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:20 smithi144 ceph-mon[32129]: pgmap v7000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:20 smithi160 ceph-mon[36715]: pgmap v7000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:20 smithi183 ceph-mon[37273]: pgmap v7000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12264.811938 secs 2024-07-26T14:30:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:22 smithi144 ceph-mon[32129]: pgmap v7001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12264.811938 secs 2024-07-26T14:30:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:22 smithi160 ceph-mon[36715]: pgmap v7001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12264.811938 secs 2024-07-26T14:30:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:22 smithi183 ceph-mon[37273]: pgmap v7001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:22.743+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:30:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:22.744+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:30:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:22.744+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:30:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:22.744+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:30:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:22.744+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:24 smithi144 ceph-mon[32129]: pgmap v7002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:24 smithi160 ceph-mon[36715]: pgmap v7002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:24 smithi183 ceph-mon[37273]: pgmap v7002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:26 smithi144 ceph-mon[32129]: pgmap v7003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:26 smithi160 ceph-mon[36715]: pgmap v7003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:26 smithi183 ceph-mon[37273]: pgmap v7003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12269.812061 secs 2024-07-26T14:30:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12269.812061 secs 2024-07-26T14:30:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12269.812061 secs 2024-07-26T14:30:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:28 smithi160 ceph-mon[36715]: pgmap v7004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:28 smithi183 ceph-mon[37273]: pgmap v7004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:28 smithi144 ceph-mon[32129]: pgmap v7004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:30 smithi160 ceph-mon[36715]: pgmap v7005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:30 smithi183 ceph-mon[37273]: pgmap v7005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:30 smithi144 ceph-mon[32129]: pgmap v7005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12274.812143 secs 2024-07-26T14:30:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:32 smithi160 ceph-mon[36715]: pgmap v7006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12274.812143 secs 2024-07-26T14:30:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:32 smithi183 ceph-mon[37273]: pgmap v7006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12274.812143 secs 2024-07-26T14:30:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:32 smithi144 ceph-mon[32129]: pgmap v7006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:34 smithi144 ceph-mon[32129]: pgmap v7007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:34 smithi160 ceph-mon[36715]: pgmap v7007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:34 smithi183 ceph-mon[37273]: pgmap v7007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:36 smithi144 ceph-mon[32129]: pgmap v7008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:36 smithi160 ceph-mon[36715]: pgmap v7008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:36 smithi183 ceph-mon[37273]: pgmap v7008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12279.812233 secs 2024-07-26T14:30:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12279.812233 secs 2024-07-26T14:30:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12279.812233 secs 2024-07-26T14:30:37.754 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:37.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:30:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:37.754+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:30:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:37.754+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:30:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:37.754+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:30:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:37.754+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:38 smithi160 ceph-mon[36715]: pgmap v7009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:38 smithi183 ceph-mon[37273]: pgmap v7009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:38 smithi144 ceph-mon[32129]: pgmap v7009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:40 smithi160 ceph-mon[36715]: pgmap v7010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:40 smithi183 ceph-mon[37273]: pgmap v7010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:40 smithi144 ceph-mon[32129]: pgmap v7010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:42.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12284.812334 secs 2024-07-26T14:30:42.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:42 smithi144 ceph-mon[32129]: pgmap v7011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12284.812334 secs 2024-07-26T14:30:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:42 smithi160 ceph-mon[36715]: pgmap v7011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12284.812334 secs 2024-07-26T14:30:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:42 smithi183 ceph-mon[37273]: pgmap v7011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:44 smithi160 ceph-mon[36715]: pgmap v7012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:44 smithi183 ceph-mon[37273]: pgmap v7012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:44 smithi144 ceph-mon[32129]: pgmap v7012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:46 smithi144 ceph-mon[32129]: pgmap v7013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:46 smithi160 ceph-mon[36715]: pgmap v7013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:46 smithi183 ceph-mon[37273]: pgmap v7013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12289.812471 secs 2024-07-26T14:30:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12289.812471 secs 2024-07-26T14:30:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12289.812471 secs 2024-07-26T14:30:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:48 smithi183 ceph-mon[37273]: pgmap v7014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:48 smithi144 ceph-mon[32129]: pgmap v7014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:48 smithi160 ceph-mon[36715]: pgmap v7014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:50 smithi183 ceph-mon[37273]: pgmap v7015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:50 smithi144 ceph-mon[32129]: pgmap v7015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:50 smithi160 ceph-mon[36715]: pgmap v7015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12294.812599 secs 2024-07-26T14:30:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:52 smithi183 ceph-mon[37273]: pgmap v7016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12294.812599 secs 2024-07-26T14:30:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:52 smithi144 ceph-mon[32129]: pgmap v7016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:52.761+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:30:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:52.762+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:30:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:52.762+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:30:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:52.762+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:30:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:30:52.762+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12294.812599 secs 2024-07-26T14:30:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:52 smithi160 ceph-mon[36715]: pgmap v7016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:54 smithi183 ceph-mon[37273]: pgmap v7017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:54 smithi144 ceph-mon[32129]: pgmap v7017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:54 smithi160 ceph-mon[36715]: pgmap v7017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:30:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:56 smithi183 ceph-mon[37273]: pgmap v7018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:56 smithi144 ceph-mon[32129]: pgmap v7018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:56 smithi160 ceph-mon[36715]: pgmap v7018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:30:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12299.812757 secs 2024-07-26T14:30:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12299.812757 secs 2024-07-26T14:30:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12299.812757 secs 2024-07-26T14:30:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:58 smithi183 ceph-mon[37273]: pgmap v7019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:58 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:30:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:58 smithi144 ceph-mon[32129]: pgmap v7019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:58 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:30:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:58 smithi160 ceph-mon[36715]: pgmap v7019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:30:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:58 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:30:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:59 smithi183 ceph-mon[37273]: pgmap v7020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:30:59 smithi183 ceph-mon[37273]: pgmap v7021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:59 smithi144 ceph-mon[32129]: pgmap v7020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:30:59 smithi144 ceph-mon[32129]: pgmap v7021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:59 smithi160 ceph-mon[36715]: pgmap v7020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:30:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:30:59 smithi160 ceph-mon[36715]: pgmap v7021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:01 smithi183 ceph-mon[37273]: pgmap v7022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:31:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:01 smithi144 ceph-mon[32129]: pgmap v7022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:31:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:01 smithi160 ceph-mon[36715]: pgmap v7022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:31:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12304.812904 secs 2024-07-26T14:31:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12304.812904 secs 2024-07-26T14:31:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12304.812904 secs 2024-07-26T14:31:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:03 smithi183 ceph-mon[37273]: pgmap v7023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:03 smithi144 ceph-mon[32129]: pgmap v7023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:03 smithi160 ceph-mon[36715]: pgmap v7023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:05 smithi183 ceph-mon[37273]: pgmap v7024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:05 smithi144 ceph-mon[32129]: pgmap v7024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:05 smithi160 ceph-mon[36715]: pgmap v7024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12309.813052 secs 2024-07-26T14:31:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:07 smithi183 ceph-mon[37273]: pgmap v7025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12309.813052 secs 2024-07-26T14:31:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:07 smithi144 ceph-mon[32129]: pgmap v7025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:07.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:31:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:07.772+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:31:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:07.772+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:31:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:07.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:31:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:07.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12309.813052 secs 2024-07-26T14:31:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:07 smithi160 ceph-mon[36715]: pgmap v7025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:31:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:09 smithi144 ceph-mon[32129]: pgmap v7026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:31:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:09 smithi160 ceph-mon[36715]: pgmap v7026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:31:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:09 smithi183 ceph-mon[37273]: pgmap v7026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:31:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:11 smithi144 ceph-mon[32129]: pgmap v7027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:11 smithi160 ceph-mon[36715]: pgmap v7027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:11 smithi183 ceph-mon[37273]: pgmap v7027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12314.813179 secs 2024-07-26T14:31:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12314.813179 secs 2024-07-26T14:31:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12314.813179 secs 2024-07-26T14:31:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:13 smithi144 ceph-mon[32129]: pgmap v7028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:13 smithi160 ceph-mon[36715]: pgmap v7028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:13 smithi183 ceph-mon[37273]: pgmap v7028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:15.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:15 smithi183 ceph-mon[37273]: pgmap v7029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:15 smithi144 ceph-mon[32129]: pgmap v7029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:15 smithi160 ceph-mon[36715]: pgmap v7029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12319.813329 secs 2024-07-26T14:31:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:17 smithi144 ceph-mon[32129]: pgmap v7030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12319.813329 secs 2024-07-26T14:31:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:17 smithi160 ceph-mon[36715]: pgmap v7030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12319.813329 secs 2024-07-26T14:31:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:17 smithi183 ceph-mon[37273]: pgmap v7030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:19 smithi144 ceph-mon[32129]: pgmap v7031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:19 smithi160 ceph-mon[36715]: pgmap v7031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:19 smithi183 ceph-mon[37273]: pgmap v7031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:21 smithi144 ceph-mon[32129]: pgmap v7032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:21 smithi160 ceph-mon[36715]: pgmap v7032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:21 smithi183 ceph-mon[37273]: pgmap v7032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12324.813457 secs 2024-07-26T14:31:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:22.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:31:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:22.788+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:31:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:22.788+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:31:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:22.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:31:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:22.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12324.813457 secs 2024-07-26T14:31:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12324.813457 secs 2024-07-26T14:31:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:23 smithi144 ceph-mon[32129]: pgmap v7033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:23 smithi160 ceph-mon[36715]: pgmap v7033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:23 smithi183 ceph-mon[37273]: pgmap v7033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:25 smithi144 ceph-mon[32129]: pgmap v7034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:25 smithi160 ceph-mon[36715]: pgmap v7034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:25 smithi183 ceph-mon[37273]: pgmap v7034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12329.813543 secs 2024-07-26T14:31:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:27 smithi144 ceph-mon[32129]: pgmap v7035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12329.813543 secs 2024-07-26T14:31:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:27 smithi160 ceph-mon[36715]: pgmap v7035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12329.813543 secs 2024-07-26T14:31:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:27 smithi183 ceph-mon[37273]: pgmap v7035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:29 smithi160 ceph-mon[36715]: pgmap v7036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:29 smithi183 ceph-mon[37273]: pgmap v7036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:29 smithi144 ceph-mon[32129]: pgmap v7036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:31 smithi183 ceph-mon[37273]: pgmap v7037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:31 smithi144 ceph-mon[32129]: pgmap v7037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:31 smithi160 ceph-mon[36715]: pgmap v7037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12334.813667 secs 2024-07-26T14:31:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12334.813667 secs 2024-07-26T14:31:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12334.813667 secs 2024-07-26T14:31:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:33 smithi183 ceph-mon[37273]: pgmap v7038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:33 smithi144 ceph-mon[32129]: pgmap v7038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:33 smithi160 ceph-mon[36715]: pgmap v7038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:35.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:35 smithi160 ceph-mon[36715]: pgmap v7039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:35 smithi183 ceph-mon[37273]: pgmap v7039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:35 smithi144 ceph-mon[32129]: pgmap v7039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12339.813776 secs 2024-07-26T14:31:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:37 smithi183 ceph-mon[37273]: pgmap v7040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12339.813776 secs 2024-07-26T14:31:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:37 smithi144 ceph-mon[32129]: pgmap v7040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:37.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:37.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:31:37.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:37.806+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:31:37.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:37.806+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:31:37.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:37.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:31:37.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:37.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12339.813776 secs 2024-07-26T14:31:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:37 smithi160 ceph-mon[36715]: pgmap v7040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:39 smithi183 ceph-mon[37273]: pgmap v7041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:39 smithi144 ceph-mon[32129]: pgmap v7041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:39 smithi160 ceph-mon[36715]: pgmap v7041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:41 smithi183 ceph-mon[37273]: pgmap v7042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:41 smithi144 ceph-mon[32129]: pgmap v7042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:41 smithi160 ceph-mon[36715]: pgmap v7042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:42.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12344.813897 secs 2024-07-26T14:31:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12344.813897 secs 2024-07-26T14:31:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12344.813897 secs 2024-07-26T14:31:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:43 smithi183 ceph-mon[37273]: pgmap v7043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:43 smithi144 ceph-mon[32129]: pgmap v7043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:43 smithi160 ceph-mon[36715]: pgmap v7043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:45 smithi183 ceph-mon[37273]: pgmap v7044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:45 smithi144 ceph-mon[32129]: pgmap v7044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:45 smithi160 ceph-mon[36715]: pgmap v7044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12349.814018 secs 2024-07-26T14:31:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:47 smithi183 ceph-mon[37273]: pgmap v7045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12349.814018 secs 2024-07-26T14:31:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:47 smithi144 ceph-mon[32129]: pgmap v7045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12349.814018 secs 2024-07-26T14:31:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:47 smithi160 ceph-mon[36715]: pgmap v7045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:49 smithi183 ceph-mon[37273]: pgmap v7046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:49 smithi144 ceph-mon[32129]: pgmap v7046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:49 smithi160 ceph-mon[36715]: pgmap v7046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:51 smithi144 ceph-mon[32129]: pgmap v7047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:51 smithi160 ceph-mon[36715]: pgmap v7047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:51 smithi183 ceph-mon[37273]: pgmap v7047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:31:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12354.814158 secs 2024-07-26T14:31:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12354.814158 secs 2024-07-26T14:31:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:52.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:31:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:52.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:31:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:52.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:31:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:52.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:31:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:31:52.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12354.814158 secs 2024-07-26T14:31:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:53 smithi144 ceph-mon[32129]: pgmap v7048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:53 smithi160 ceph-mon[36715]: pgmap v7048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:53 smithi183 ceph-mon[37273]: pgmap v7048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:55 smithi144 ceph-mon[32129]: pgmap v7049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:55 smithi160 ceph-mon[36715]: pgmap v7049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:55 smithi183 ceph-mon[37273]: pgmap v7049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12359.814328 secs 2024-07-26T14:31:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:57 smithi144 ceph-mon[32129]: pgmap v7050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12359.814328 secs 2024-07-26T14:31:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:57 smithi160 ceph-mon[36715]: pgmap v7050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12359.814328 secs 2024-07-26T14:31:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:57 smithi183 ceph-mon[37273]: pgmap v7050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:31:59.584 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:59 smithi160 ceph-mon[36715]: pgmap v7051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:59.584 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:31:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:59 smithi144 ceph-mon[32129]: pgmap v7051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:31:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:59 smithi183 ceph-mon[37273]: pgmap v7051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:31:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:31:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:01 smithi144 ceph-mon[32129]: pgmap v7052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:01.735 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:01 smithi160 ceph-mon[36715]: pgmap v7052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:01 smithi183 ceph-mon[37273]: pgmap v7052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12364.814395 secs 2024-07-26T14:32:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12364.814395 secs 2024-07-26T14:32:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12364.814395 secs 2024-07-26T14:32:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:03.285+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:32:03.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:03.285+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:32:03.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:03.287+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:32:03.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:03.287+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:32:03.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:03 smithi144 ceph-mon[32129]: pgmap v7053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:03 smithi160 ceph-mon[36715]: pgmap v7053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:03 smithi183 ceph-mon[37273]: pgmap v7053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: pgmap v7054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: pgmap v7055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: pgmap v7056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:32:05.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: pgmap v7054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: pgmap v7055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: pgmap v7056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:32:06.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: pgmap v7054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: pgmap v7055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: pgmap v7056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:32:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12369.814549 secs 2024-07-26T14:32:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12369.814549 secs 2024-07-26T14:32:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12369.814549 secs 2024-07-26T14:32:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:07.829+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:32:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:07.830+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:32:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:07.830+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:32:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:07.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:32:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:07.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:07 smithi144 ceph-mon[32129]: pgmap v7057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:32:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:07 smithi160 ceph-mon[36715]: pgmap v7057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:32:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:07 smithi183 ceph-mon[37273]: pgmap v7057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:32:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:09 smithi160 ceph-mon[36715]: pgmap v7058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:32:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:09 smithi183 ceph-mon[37273]: pgmap v7058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:32:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:09 smithi144 ceph-mon[32129]: pgmap v7058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:32:12.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:11 smithi144 ceph-mon[32129]: pgmap v7059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:32:12.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12374.814685 secs 2024-07-26T14:32:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:11 smithi160 ceph-mon[36715]: pgmap v7059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:32:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12374.814685 secs 2024-07-26T14:32:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:11 smithi183 ceph-mon[37273]: pgmap v7059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T14:32:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12374.814685 secs 2024-07-26T14:32:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:13 smithi160 ceph-mon[36715]: pgmap v7060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T14:32:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:13 smithi183 ceph-mon[37273]: pgmap v7060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T14:32:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:13 smithi144 ceph-mon[32129]: pgmap v7060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T14:32:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:15 smithi160 ceph-mon[36715]: pgmap v7061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:32:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:15 smithi183 ceph-mon[37273]: pgmap v7061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:32:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:15 smithi144 ceph-mon[32129]: pgmap v7061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:32:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12379.814770 secs 2024-07-26T14:32:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12379.814770 secs 2024-07-26T14:32:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12379.814770 secs 2024-07-26T14:32:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:17 smithi160 ceph-mon[36715]: pgmap v7062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:17 smithi183 ceph-mon[37273]: pgmap v7062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:17 smithi144 ceph-mon[32129]: pgmap v7062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:19 smithi160 ceph-mon[36715]: pgmap v7063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:19 smithi183 ceph-mon[37273]: pgmap v7063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:19 smithi144 ceph-mon[32129]: pgmap v7063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:21 smithi160 ceph-mon[36715]: pgmap v7064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12384.814922 secs 2024-07-26T14:32:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:21 smithi183 ceph-mon[37273]: pgmap v7064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12384.814922 secs 2024-07-26T14:32:22.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:21 smithi144 ceph-mon[32129]: pgmap v7064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:22.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12384.814922 secs 2024-07-26T14:32:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:22.845+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:32:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:22.846+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:32:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:22.846+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:32:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:22.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:32:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:22.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:23 smithi160 ceph-mon[36715]: pgmap v7065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:23 smithi183 ceph-mon[37273]: pgmap v7065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:23 smithi144 ceph-mon[32129]: pgmap v7065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:25 smithi183 ceph-mon[37273]: pgmap v7066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:25 smithi144 ceph-mon[32129]: pgmap v7066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:25 smithi160 ceph-mon[36715]: pgmap v7066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12389.815066 secs 2024-07-26T14:32:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12389.815066 secs 2024-07-26T14:32:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12389.815066 secs 2024-07-26T14:32:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:27 smithi183 ceph-mon[37273]: pgmap v7067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:27 smithi144 ceph-mon[32129]: pgmap v7067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:27 smithi160 ceph-mon[36715]: pgmap v7067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:29 smithi183 ceph-mon[37273]: pgmap v7068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:29 smithi144 ceph-mon[32129]: pgmap v7068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:29 smithi160 ceph-mon[36715]: pgmap v7068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:30 smithi144 ceph-mon[32129]: pgmap v7069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:30 smithi160 ceph-mon[36715]: pgmap v7069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:30 smithi183 ceph-mon[37273]: pgmap v7069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12394.815160 secs 2024-07-26T14:32:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12394.815160 secs 2024-07-26T14:32:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12394.815160 secs 2024-07-26T14:32:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:33 smithi144 ceph-mon[32129]: pgmap v7070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:33 smithi160 ceph-mon[36715]: pgmap v7070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:33 smithi183 ceph-mon[37273]: pgmap v7070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:35 smithi160 ceph-mon[36715]: pgmap v7071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:35 smithi183 ceph-mon[37273]: pgmap v7071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:35 smithi144 ceph-mon[32129]: pgmap v7071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12399.815301 secs 2024-07-26T14:32:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12399.815301 secs 2024-07-26T14:32:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12399.815301 secs 2024-07-26T14:32:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:37.863+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:32:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:37.864+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:32:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:37.864+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:32:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:37.864+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:32:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:37.864+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:38.045 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:37 smithi144 ceph-mon[32129]: pgmap v7072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:37 smithi160 ceph-mon[36715]: pgmap v7072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:37 smithi183 ceph-mon[37273]: pgmap v7072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:39 smithi160 ceph-mon[36715]: pgmap v7073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:39 smithi183 ceph-mon[37273]: pgmap v7073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:39 smithi144 ceph-mon[32129]: pgmap v7073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:41 smithi160 ceph-mon[36715]: pgmap v7074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12404.815338 secs 2024-07-26T14:32:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:41 smithi183 ceph-mon[37273]: pgmap v7074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12404.815338 secs 2024-07-26T14:32:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:41 smithi144 ceph-mon[32129]: pgmap v7074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12404.815338 secs 2024-07-26T14:32:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:43 smithi160 ceph-mon[36715]: pgmap v7075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:43 smithi183 ceph-mon[37273]: pgmap v7075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:43 smithi144 ceph-mon[32129]: pgmap v7075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:45 smithi183 ceph-mon[37273]: pgmap v7076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:45 smithi144 ceph-mon[32129]: pgmap v7076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:45 smithi160 ceph-mon[36715]: pgmap v7076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12409.815467 secs 2024-07-26T14:32:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12409.815467 secs 2024-07-26T14:32:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12409.815467 secs 2024-07-26T14:32:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:47 smithi183 ceph-mon[37273]: pgmap v7077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:47 smithi144 ceph-mon[32129]: pgmap v7077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:48.294+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:32:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:48.294+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:32:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:48.295+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:32:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:48.295+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:32:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:47 smithi160 ceph-mon[36715]: pgmap v7077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:49 smithi183 ceph-mon[37273]: pgmap v7078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:49 smithi144 ceph-mon[32129]: pgmap v7078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:49 smithi160 ceph-mon[36715]: pgmap v7078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:51 smithi183 ceph-mon[37273]: pgmap v7079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12414.815591 secs 2024-07-26T14:32:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:51 smithi144 ceph-mon[32129]: pgmap v7079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12414.815591 secs 2024-07-26T14:32:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:51 smithi160 ceph-mon[36715]: pgmap v7079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12414.815591 secs 2024-07-26T14:32:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:52.866+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:32:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:52.867+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:32:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:52.867+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:32:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:52.867+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:32:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:32:52.867+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:52 smithi144 ceph-mon[32129]: pgmap v7080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:52 smithi160 ceph-mon[36715]: pgmap v7080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:52 smithi183 ceph-mon[37273]: pgmap v7080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:32:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:55 smithi160 ceph-mon[36715]: pgmap v7081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:55 smithi183 ceph-mon[37273]: pgmap v7081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:55 smithi144 ceph-mon[32129]: pgmap v7081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:32:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12419.815674 secs 2024-07-26T14:32:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12419.815674 secs 2024-07-26T14:32:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12419.815674 secs 2024-07-26T14:32:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:57 smithi160 ceph-mon[36715]: pgmap v7082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:57 smithi183 ceph-mon[37273]: pgmap v7082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:32:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:57 smithi144 ceph-mon[32129]: pgmap v7082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:32:59 smithi160 ceph-mon[36715]: pgmap v7083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:32:59 smithi183 ceph-mon[37273]: pgmap v7083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:32:59 smithi144 ceph-mon[32129]: pgmap v7083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:01 smithi183 ceph-mon[37273]: pgmap v7084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12424.815793 secs 2024-07-26T14:33:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:01 smithi144 ceph-mon[32129]: pgmap v7084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12424.815793 secs 2024-07-26T14:33:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:01 smithi160 ceph-mon[36715]: pgmap v7084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12424.815793 secs 2024-07-26T14:33:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:03 smithi183 ceph-mon[37273]: pgmap v7085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:03 smithi144 ceph-mon[32129]: pgmap v7085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:03 smithi160 ceph-mon[36715]: pgmap v7085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: pgmap v7086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.086 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.086 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.086 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: pgmap v7086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.112 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.112 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.112 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: pgmap v7086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:07.122 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:06 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:33:07.123 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:06 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:33:07.123 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12429.815854 secs 2024-07-26T14:33:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:06 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:33:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:06 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:33:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12429.815854 secs 2024-07-26T14:33:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:06 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:33:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:06 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:33:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12429.815854 secs 2024-07-26T14:33:07.878 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:07.877+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:33:07.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:07.878+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:33:07.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:07.878+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:33:07.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:07.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:33:07.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:07.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:07 smithi183 ceph-mon[37273]: pgmap v7087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:07 smithi144 ceph-mon[32129]: pgmap v7087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:07 smithi160 ceph-mon[36715]: pgmap v7087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: pgmap v7088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: pgmap v7089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:09.961 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:09.961 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:09.962 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.962 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.962 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.962 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.962 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.962 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: pgmap v7088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: pgmap v7089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:09.963 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:10.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: pgmap v7088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:10.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: pgmap v7089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:10.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:33:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:11 smithi144 ceph-mon[32129]: pgmap v7090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12434.815982 secs 2024-07-26T14:33:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:11 smithi160 ceph-mon[36715]: pgmap v7090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12434.815982 secs 2024-07-26T14:33:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:11 smithi183 ceph-mon[37273]: pgmap v7090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12434.815982 secs 2024-07-26T14:33:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:13 smithi144 ceph-mon[32129]: pgmap v7091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:13 smithi160 ceph-mon[36715]: pgmap v7091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:13 smithi183 ceph-mon[37273]: pgmap v7091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:15 smithi144 ceph-mon[32129]: pgmap v7092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:15 smithi160 ceph-mon[36715]: pgmap v7092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:15 smithi183 ceph-mon[37273]: pgmap v7092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12439.816125 secs 2024-07-26T14:33:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12439.816125 secs 2024-07-26T14:33:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12439.816125 secs 2024-07-26T14:33:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:17 smithi160 ceph-mon[36715]: pgmap v7093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:17 smithi183 ceph-mon[37273]: pgmap v7093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:17 smithi144 ceph-mon[32129]: pgmap v7093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:19 smithi160 ceph-mon[36715]: pgmap v7094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:19 smithi183 ceph-mon[37273]: pgmap v7094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:19 smithi144 ceph-mon[32129]: pgmap v7094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:21 smithi160 ceph-mon[36715]: pgmap v7095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12444.816247 secs 2024-07-26T14:33:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:21 smithi183 ceph-mon[37273]: pgmap v7095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12444.816247 secs 2024-07-26T14:33:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:21 smithi144 ceph-mon[32129]: pgmap v7095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12444.816247 secs 2024-07-26T14:33:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:22.893+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:33:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:22.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:33:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:22.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:33:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:22.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:33:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:22.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:23 smithi160 ceph-mon[36715]: pgmap v7096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:23 smithi183 ceph-mon[37273]: pgmap v7096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:23 smithi144 ceph-mon[32129]: pgmap v7096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:25 smithi160 ceph-mon[36715]: pgmap v7097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:25 smithi183 ceph-mon[37273]: pgmap v7097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:25 smithi144 ceph-mon[32129]: pgmap v7097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12449.816365 secs 2024-07-26T14:33:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12449.816365 secs 2024-07-26T14:33:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12449.816365 secs 2024-07-26T14:33:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:27 smithi160 ceph-mon[36715]: pgmap v7098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:27 smithi183 ceph-mon[37273]: pgmap v7098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:27 smithi144 ceph-mon[32129]: pgmap v7098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:29 smithi160 ceph-mon[36715]: pgmap v7099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:29 smithi183 ceph-mon[37273]: pgmap v7099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:29 smithi144 ceph-mon[32129]: pgmap v7099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:31 smithi160 ceph-mon[36715]: pgmap v7100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12454.816490 secs 2024-07-26T14:33:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:31 smithi183 ceph-mon[37273]: pgmap v7100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12454.816490 secs 2024-07-26T14:33:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:31 smithi144 ceph-mon[32129]: pgmap v7100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12454.816490 secs 2024-07-26T14:33:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:33 smithi160 ceph-mon[36715]: pgmap v7101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:33 smithi183 ceph-mon[37273]: pgmap v7101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:33 smithi144 ceph-mon[32129]: pgmap v7101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:35 smithi160 ceph-mon[36715]: pgmap v7102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:35 smithi183 ceph-mon[37273]: pgmap v7102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:35 smithi144 ceph-mon[32129]: pgmap v7102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:33:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12459.816589 secs 2024-07-26T14:33:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12459.816589 secs 2024-07-26T14:33:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12459.816589 secs 2024-07-26T14:33:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:37.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:33:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:37.912+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:33:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:37.912+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:33:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:37.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:33:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:37.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:37 smithi160 ceph-mon[36715]: pgmap v7103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:37 smithi183 ceph-mon[37273]: pgmap v7103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:37 smithi144 ceph-mon[32129]: pgmap v7103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:39 smithi160 ceph-mon[36715]: pgmap v7104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:39 smithi183 ceph-mon[37273]: pgmap v7104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:39 smithi144 ceph-mon[32129]: pgmap v7104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:41 smithi160 ceph-mon[36715]: pgmap v7105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12464.816712 secs 2024-07-26T14:33:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:41 smithi183 ceph-mon[37273]: pgmap v7105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12464.816712 secs 2024-07-26T14:33:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:41 smithi144 ceph-mon[32129]: pgmap v7105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12464.816712 secs 2024-07-26T14:33:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:43 smithi160 ceph-mon[36715]: pgmap v7106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:43 smithi183 ceph-mon[37273]: pgmap v7106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:43 smithi144 ceph-mon[32129]: pgmap v7106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:33:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:45 smithi183 ceph-mon[37273]: pgmap v7107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:45 smithi144 ceph-mon[32129]: pgmap v7107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:45 smithi160 ceph-mon[36715]: pgmap v7107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12469.816837 secs 2024-07-26T14:33:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12469.816837 secs 2024-07-26T14:33:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12469.816837 secs 2024-07-26T14:33:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:47 smithi183 ceph-mon[37273]: pgmap v7108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:47 smithi144 ceph-mon[32129]: pgmap v7108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:47 smithi160 ceph-mon[36715]: pgmap v7108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:49 smithi183 ceph-mon[37273]: pgmap v7109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:49 smithi144 ceph-mon[32129]: pgmap v7109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:49 smithi160 ceph-mon[36715]: pgmap v7109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:33:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:51 smithi183 ceph-mon[37273]: pgmap v7110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12474.816956 secs 2024-07-26T14:33:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:51 smithi144 ceph-mon[32129]: pgmap v7110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12474.816956 secs 2024-07-26T14:33:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:51 smithi160 ceph-mon[36715]: pgmap v7110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12474.816956 secs 2024-07-26T14:33:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:52.928+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:33:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:52.929+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:33:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:52.929+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:33:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:52.929+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:33:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:33:52.929+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:53 smithi183 ceph-mon[37273]: pgmap v7111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:33:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:53 smithi144 ceph-mon[32129]: pgmap v7111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:33:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:53 smithi160 ceph-mon[36715]: pgmap v7111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:33:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:54 smithi144 ceph-mon[32129]: pgmap v7112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:54 smithi160 ceph-mon[36715]: pgmap v7112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:54 smithi183 ceph-mon[37273]: pgmap v7112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:33:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12479.817056 secs 2024-07-26T14:33:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12479.817056 secs 2024-07-26T14:33:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12479.817056 secs 2024-07-26T14:33:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:57 smithi160 ceph-mon[36715]: pgmap v7113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:57 smithi183 ceph-mon[37273]: pgmap v7113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:33:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:57 smithi144 ceph-mon[32129]: pgmap v7113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:33:59 smithi160 ceph-mon[36715]: pgmap v7114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:33:59 smithi183 ceph-mon[37273]: pgmap v7114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:00.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:33:59 smithi144 ceph-mon[32129]: pgmap v7114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:01 smithi160 ceph-mon[36715]: pgmap v7115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12484.817182 secs 2024-07-26T14:34:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:01 smithi183 ceph-mon[37273]: pgmap v7115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12484.817182 secs 2024-07-26T14:34:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:01 smithi144 ceph-mon[32129]: pgmap v7115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12484.817182 secs 2024-07-26T14:34:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:03 smithi160 ceph-mon[36715]: pgmap v7116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:03 smithi183 ceph-mon[37273]: pgmap v7116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:03 smithi144 ceph-mon[32129]: pgmap v7116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:05 smithi160 ceph-mon[36715]: pgmap v7117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:05 smithi183 ceph-mon[37273]: pgmap v7117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:05 smithi144 ceph-mon[32129]: pgmap v7117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12489.817271 secs 2024-07-26T14:34:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:07 smithi160 ceph-mon[36715]: pgmap v7118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:34:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12489.817271 secs 2024-07-26T14:34:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:07 smithi183 ceph-mon[37273]: pgmap v7118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:34:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12489.817271 secs 2024-07-26T14:34:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:07 smithi144 ceph-mon[32129]: pgmap v7118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:34:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:07.930+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:34:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:07.931+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:34:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:07.931+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:34:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:07.931+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:34:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:07.931+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:09 smithi160 ceph-mon[36715]: pgmap v7119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:09 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:09 smithi183 ceph-mon[37273]: pgmap v7119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:09 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:09 smithi144 ceph-mon[32129]: pgmap v7119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:34:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:09 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:34:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:34:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:34:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:11 smithi160 ceph-mon[36715]: pgmap v7120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:11 smithi160 ceph-mon[36715]: pgmap v7121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12494.817361 secs 2024-07-26T14:34:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:11 smithi183 ceph-mon[37273]: pgmap v7120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:11 smithi183 ceph-mon[37273]: pgmap v7121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12494.817361 secs 2024-07-26T14:34:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:11 smithi144 ceph-mon[32129]: pgmap v7120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:11 smithi144 ceph-mon[32129]: pgmap v7121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12494.817361 secs 2024-07-26T14:34:14.026 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:13 smithi144 ceph-mon[32129]: pgmap v7122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:34:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:13 smithi160 ceph-mon[36715]: pgmap v7122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:34:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:13 smithi183 ceph-mon[37273]: pgmap v7122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:34:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:15 smithi160 ceph-mon[36715]: pgmap v7123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:34:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:15 smithi183 ceph-mon[37273]: pgmap v7123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:34:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:15 smithi144 ceph-mon[32129]: pgmap v7123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:34:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12499.817475 secs 2024-07-26T14:34:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12499.817475 secs 2024-07-26T14:34:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12499.817475 secs 2024-07-26T14:34:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:17 smithi160 ceph-mon[36715]: pgmap v7124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:34:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:17 smithi183 ceph-mon[37273]: pgmap v7124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:34:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:17 smithi144 ceph-mon[32129]: pgmap v7124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:34:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:19 smithi160 ceph-mon[36715]: pgmap v7125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:19 smithi183 ceph-mon[37273]: pgmap v7125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:19 smithi144 ceph-mon[32129]: pgmap v7125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:34:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:21 smithi160 ceph-mon[36715]: pgmap v7126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:34:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12504.817596 secs 2024-07-26T14:34:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:21 smithi183 ceph-mon[37273]: pgmap v7126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:34:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12504.817596 secs 2024-07-26T14:34:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:21 smithi144 ceph-mon[32129]: pgmap v7126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:34:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12504.817596 secs 2024-07-26T14:34:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:22.931+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:34:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:22.932+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:34:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:22.932+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:34:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:22.932+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:34:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:22.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:23 smithi160 ceph-mon[36715]: pgmap v7127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:23 smithi183 ceph-mon[37273]: pgmap v7127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:23 smithi144 ceph-mon[32129]: pgmap v7127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:25 smithi160 ceph-mon[36715]: pgmap v7128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:25 smithi183 ceph-mon[37273]: pgmap v7128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:25 smithi144 ceph-mon[32129]: pgmap v7128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:27.101 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12509.817720 secs 2024-07-26T14:34:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12509.817720 secs 2024-07-26T14:34:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12509.817720 secs 2024-07-26T14:34:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:27 smithi183 ceph-mon[37273]: pgmap v7129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:27 smithi144 ceph-mon[32129]: pgmap v7129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:27 smithi160 ceph-mon[36715]: pgmap v7129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:29 smithi183 ceph-mon[37273]: pgmap v7130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:29 smithi144 ceph-mon[32129]: pgmap v7130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:29 smithi160 ceph-mon[36715]: pgmap v7130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:32.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:31 smithi183 ceph-mon[37273]: pgmap v7131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:32.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12514.817784 secs 2024-07-26T14:34:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:31 smithi144 ceph-mon[32129]: pgmap v7131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12514.817784 secs 2024-07-26T14:34:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:31 smithi160 ceph-mon[36715]: pgmap v7131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12514.817784 secs 2024-07-26T14:34:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:33 smithi183 ceph-mon[37273]: pgmap v7132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:33 smithi144 ceph-mon[32129]: pgmap v7132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:33 smithi160 ceph-mon[36715]: pgmap v7132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:35 smithi183 ceph-mon[37273]: pgmap v7133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:35 smithi144 ceph-mon[32129]: pgmap v7133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:35 smithi160 ceph-mon[36715]: pgmap v7133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12519.817924 secs 2024-07-26T14:34:37.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12519.817924 secs 2024-07-26T14:34:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12519.817924 secs 2024-07-26T14:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:37.947+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:37.949+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:37.949+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:37.949+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:34:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:37.949+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:37 smithi183 ceph-mon[37273]: pgmap v7134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:37 smithi144 ceph-mon[32129]: pgmap v7134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:37 smithi160 ceph-mon[36715]: pgmap v7134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:38 smithi183 ceph-mon[37273]: pgmap v7135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:38 smithi144 ceph-mon[32129]: pgmap v7135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:38 smithi160 ceph-mon[36715]: pgmap v7135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:40 smithi144 ceph-mon[32129]: pgmap v7136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:40 smithi160 ceph-mon[36715]: pgmap v7136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:40 smithi183 ceph-mon[37273]: pgmap v7136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12524.818064 secs 2024-07-26T14:34:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12524.818064 secs 2024-07-26T14:34:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12524.818064 secs 2024-07-26T14:34:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:42 smithi144 ceph-mon[32129]: pgmap v7137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:42 smithi160 ceph-mon[36715]: pgmap v7137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:42 smithi183 ceph-mon[37273]: pgmap v7137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:44 smithi144 ceph-mon[32129]: pgmap v7138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:44 smithi160 ceph-mon[36715]: pgmap v7138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:44 smithi183 ceph-mon[37273]: pgmap v7138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12529.818202 secs 2024-07-26T14:34:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:46 smithi144 ceph-mon[32129]: pgmap v7139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12529.818202 secs 2024-07-26T14:34:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:46 smithi160 ceph-mon[36715]: pgmap v7139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12529.818202 secs 2024-07-26T14:34:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:46 smithi183 ceph-mon[37273]: pgmap v7139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:48 smithi160 ceph-mon[36715]: pgmap v7140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:48 smithi183 ceph-mon[37273]: pgmap v7140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:48 smithi144 ceph-mon[32129]: pgmap v7140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:50 smithi160 ceph-mon[36715]: pgmap v7141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:50 smithi183 ceph-mon[37273]: pgmap v7141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:50 smithi144 ceph-mon[32129]: pgmap v7141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12534.818366 secs 2024-07-26T14:34:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12534.818366 secs 2024-07-26T14:34:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12534.818366 secs 2024-07-26T14:34:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:52.965+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:34:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:52.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:34:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:52.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:34:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:52.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:34:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:34:52.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:52 smithi160 ceph-mon[36715]: pgmap v7142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:52 smithi183 ceph-mon[37273]: pgmap v7142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:52 smithi144 ceph-mon[32129]: pgmap v7142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:34:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:54 smithi160 ceph-mon[36715]: pgmap v7143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:54 smithi183 ceph-mon[37273]: pgmap v7143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:54 smithi144 ceph-mon[32129]: pgmap v7143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12539.818430 secs 2024-07-26T14:34:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:57 smithi160 ceph-mon[36715]: pgmap v7144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12539.818430 secs 2024-07-26T14:34:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:57 smithi183 ceph-mon[37273]: pgmap v7144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12539.818430 secs 2024-07-26T14:34:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:57 smithi144 ceph-mon[32129]: pgmap v7144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:34:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:34:59 smithi160 ceph-mon[36715]: pgmap v7145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:34:59 smithi183 ceph-mon[37273]: pgmap v7145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:34:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:34:59 smithi144 ceph-mon[32129]: pgmap v7145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:01 smithi160 ceph-mon[36715]: pgmap v7146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:01 smithi183 ceph-mon[37273]: pgmap v7146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:01 smithi144 ceph-mon[32129]: pgmap v7146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12544.818559 secs 2024-07-26T14:35:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12544.818559 secs 2024-07-26T14:35:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12544.818559 secs 2024-07-26T14:35:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:03 smithi160 ceph-mon[36715]: pgmap v7147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:03 smithi183 ceph-mon[37273]: pgmap v7147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:03 smithi144 ceph-mon[32129]: pgmap v7147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:05 smithi160 ceph-mon[36715]: pgmap v7148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:05 smithi183 ceph-mon[37273]: pgmap v7148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:05 smithi144 ceph-mon[32129]: pgmap v7148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12549.818684 secs 2024-07-26T14:35:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:07 smithi160 ceph-mon[36715]: pgmap v7149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12549.818684 secs 2024-07-26T14:35:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:07 smithi183 ceph-mon[37273]: pgmap v7149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12549.818684 secs 2024-07-26T14:35:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:07 smithi144 ceph-mon[32129]: pgmap v7149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:07.982+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:35:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:07.983+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:35:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:07.983+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:35:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:07.983+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:35:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:07.983+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:09 smithi160 ceph-mon[36715]: pgmap v7150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:09 smithi183 ceph-mon[37273]: pgmap v7150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:09 smithi144 ceph-mon[32129]: pgmap v7150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:11 smithi160 ceph-mon[36715]: pgmap v7151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:11 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:11 smithi183 ceph-mon[37273]: pgmap v7151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:11 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:11 smithi144 ceph-mon[32129]: pgmap v7151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:11 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12554.818808 secs 2024-07-26T14:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:12 smithi183 ceph-mon[37273]: pgmap v7152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:12 smithi183 ceph-mon[37273]: pgmap v7153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12554.818808 secs 2024-07-26T14:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:12 smithi144 ceph-mon[32129]: pgmap v7152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:12 smithi144 ceph-mon[32129]: pgmap v7153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12554.818808 secs 2024-07-26T14:35:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:12 smithi160 ceph-mon[36715]: pgmap v7152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:12 smithi160 ceph-mon[36715]: pgmap v7153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:35:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:14 smithi160 ceph-mon[36715]: pgmap v7154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:15.087 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:14 smithi183 ceph-mon[37273]: pgmap v7154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:14 smithi144 ceph-mon[32129]: pgmap v7154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:16 smithi160 ceph-mon[36715]: pgmap v7155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12559.818936 secs 2024-07-26T14:35:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:16 smithi183 ceph-mon[37273]: pgmap v7155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12559.818936 secs 2024-07-26T14:35:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:16 smithi144 ceph-mon[32129]: pgmap v7155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12559.818936 secs 2024-07-26T14:35:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:18 smithi160 ceph-mon[36715]: pgmap v7156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:18 smithi183 ceph-mon[37273]: pgmap v7156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:18 smithi144 ceph-mon[32129]: pgmap v7156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:20 smithi160 ceph-mon[36715]: pgmap v7157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:20 smithi183 ceph-mon[37273]: pgmap v7157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:20 smithi144 ceph-mon[32129]: pgmap v7157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:35:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12564.819085 secs 2024-07-26T14:35:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12564.819085 secs 2024-07-26T14:35:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12564.819085 secs 2024-07-26T14:35:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:22.997+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:35:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:22.998+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:35:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:22.998+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:35:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:22.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:35:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:22.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:22 smithi160 ceph-mon[36715]: pgmap v7158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:35:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:22 smithi183 ceph-mon[37273]: pgmap v7158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:35:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:22 smithi144 ceph-mon[32129]: pgmap v7158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:35:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:24 smithi160 ceph-mon[36715]: pgmap v7159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:24 smithi183 ceph-mon[37273]: pgmap v7159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:24 smithi144 ceph-mon[32129]: pgmap v7159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:26 smithi160 ceph-mon[36715]: pgmap v7160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12569.819196 secs 2024-07-26T14:35:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:26 smithi183 ceph-mon[37273]: pgmap v7160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12569.819196 secs 2024-07-26T14:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:26 smithi144 ceph-mon[32129]: pgmap v7160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12569.819196 secs 2024-07-26T14:35:29.053 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:28 smithi144 ceph-mon[32129]: pgmap v7161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:28 smithi160 ceph-mon[36715]: pgmap v7161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:28 smithi183 ceph-mon[37273]: pgmap v7161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:30 smithi160 ceph-mon[36715]: pgmap v7162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:30 smithi183 ceph-mon[37273]: pgmap v7162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:30 smithi144 ceph-mon[32129]: pgmap v7162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12574.819335 secs 2024-07-26T14:35:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12574.819335 secs 2024-07-26T14:35:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12574.819335 secs 2024-07-26T14:35:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:32 smithi160 ceph-mon[36715]: pgmap v7163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:32 smithi183 ceph-mon[37273]: pgmap v7163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:32 smithi144 ceph-mon[32129]: pgmap v7163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:34 smithi183 ceph-mon[37273]: pgmap v7164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:34 smithi144 ceph-mon[32129]: pgmap v7164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:34 smithi160 ceph-mon[36715]: pgmap v7164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:37.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:36 smithi160 ceph-mon[36715]: pgmap v7165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:37.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12579.819463 secs 2024-07-26T14:35:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:36 smithi183 ceph-mon[37273]: pgmap v7165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12579.819463 secs 2024-07-26T14:35:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:36 smithi144 ceph-mon[32129]: pgmap v7165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12579.819463 secs 2024-07-26T14:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:38.015+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:35:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:38.016+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:35:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:38.016+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:35:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:38.016+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:35:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:38.016+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:38 smithi183 ceph-mon[37273]: pgmap v7166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:38 smithi144 ceph-mon[32129]: pgmap v7166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:38 smithi160 ceph-mon[36715]: pgmap v7166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:40 smithi183 ceph-mon[37273]: pgmap v7167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:40 smithi144 ceph-mon[32129]: pgmap v7167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:40 smithi160 ceph-mon[36715]: pgmap v7167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12584.819547 secs 2024-07-26T14:35:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12584.819547 secs 2024-07-26T14:35:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12584.819547 secs 2024-07-26T14:35:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:42 smithi183 ceph-mon[37273]: pgmap v7168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:42 smithi144 ceph-mon[32129]: pgmap v7168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:42 smithi160 ceph-mon[36715]: pgmap v7168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:44 smithi183 ceph-mon[37273]: pgmap v7169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:44 smithi144 ceph-mon[32129]: pgmap v7169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:44 smithi160 ceph-mon[36715]: pgmap v7169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:35:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:35:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a8072000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:35:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a8072000 comp rx=0 tx=0).stop 2024-07-26T14:35:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T14:35:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:35:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:35:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.137+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:35:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.138+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:35:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:46.140+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f74a8002230 tx=0x7f74a804c070 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:35:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:46 smithi183 ceph-mon[37273]: pgmap v7170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12589.819680 secs 2024-07-26T14:35:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:46 smithi144 ceph-mon[32129]: pgmap v7170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12589.819680 secs 2024-07-26T14:35:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:46 smithi160 ceph-mon[36715]: pgmap v7170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12589.819680 secs 2024-07-26T14:35:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:48.326+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:35:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:48.326+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:35:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:48.327+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:35:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:48.327+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:35:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:48 smithi183 ceph-mon[37273]: pgmap v7171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:48 smithi144 ceph-mon[32129]: pgmap v7171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:48 smithi160 ceph-mon[36715]: pgmap v7171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:50 smithi183 ceph-mon[37273]: pgmap v7172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:50 smithi144 ceph-mon[32129]: pgmap v7172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:50 smithi160 ceph-mon[36715]: pgmap v7172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12594.819804 secs 2024-07-26T14:35:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12594.819804 secs 2024-07-26T14:35:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12594.819804 secs 2024-07-26T14:35:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:53.032+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:35:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:53.033+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:35:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:53.033+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:35:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:53.033+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:35:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:35:53.033+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:52 smithi183 ceph-mon[37273]: pgmap v7173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:52 smithi144 ceph-mon[32129]: pgmap v7173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:52 smithi160 ceph-mon[36715]: pgmap v7173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:35:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:53 smithi144 ceph-mon[32129]: pgmap v7174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:53 smithi160 ceph-mon[36715]: pgmap v7174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:53 smithi183 ceph-mon[37273]: pgmap v7174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:57.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:56 smithi160 ceph-mon[36715]: pgmap v7175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:57.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12599.819938 secs 2024-07-26T14:35:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:56 smithi183 ceph-mon[37273]: pgmap v7175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12599.819938 secs 2024-07-26T14:35:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:56 smithi144 ceph-mon[32129]: pgmap v7175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:35:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12599.819938 secs 2024-07-26T14:35:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:35:58 smithi160 ceph-mon[36715]: pgmap v7176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:35:58 smithi183 ceph-mon[37273]: pgmap v7176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:35:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:35:58 smithi144 ceph-mon[32129]: pgmap v7176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:00 smithi160 ceph-mon[36715]: pgmap v7177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:00 smithi183 ceph-mon[37273]: pgmap v7177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:00 smithi144 ceph-mon[32129]: pgmap v7177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12604.820018 secs 2024-07-26T14:36:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12604.820018 secs 2024-07-26T14:36:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12604.820018 secs 2024-07-26T14:36:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:02 smithi160 ceph-mon[36715]: pgmap v7178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:02 smithi183 ceph-mon[37273]: pgmap v7178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:02 smithi144 ceph-mon[32129]: pgmap v7178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:04 smithi160 ceph-mon[36715]: pgmap v7179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:04 smithi183 ceph-mon[37273]: pgmap v7179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:04 smithi144 ceph-mon[32129]: pgmap v7179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:06 smithi160 ceph-mon[36715]: pgmap v7180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12609.820142 secs 2024-07-26T14:36:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:06 smithi183 ceph-mon[37273]: pgmap v7180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12609.820142 secs 2024-07-26T14:36:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:06 smithi144 ceph-mon[32129]: pgmap v7180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12609.820142 secs 2024-07-26T14:36:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:08.049+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:36:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:08.050+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:36:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:08.050+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:36:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:08.050+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:36:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:08.050+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:08 smithi160 ceph-mon[36715]: pgmap v7181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:08 smithi183 ceph-mon[37273]: pgmap v7181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:08 smithi144 ceph-mon[32129]: pgmap v7181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:10 smithi183 ceph-mon[37273]: pgmap v7182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:10 smithi144 ceph-mon[32129]: pgmap v7182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:10 smithi160 ceph-mon[36715]: pgmap v7182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12614.820284 secs 2024-07-26T14:36:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12614.820284 secs 2024-07-26T14:36:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12614.820284 secs 2024-07-26T14:36:13.115 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:12 smithi160 ceph-mon[36715]: pgmap v7183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:13.116 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:13.116 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.116 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.116 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.116 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.119 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:12 smithi183 ceph-mon[37273]: pgmap v7183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:13.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:13.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.129 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:12 smithi144 ceph-mon[32129]: pgmap v7183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:13.129 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:13.130 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.130 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.130 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:13.130 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:14 smithi183 ceph-mon[37273]: pgmap v7184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:14 smithi144 ceph-mon[32129]: pgmap v7184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:14 smithi160 ceph-mon[36715]: pgmap v7184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: pgmap v7185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12619.820394 secs 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:16.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: pgmap v7185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12619.820394 secs 2024-07-26T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: pgmap v7185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12619.820394 secs 2024-07-26T14:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:36:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:17 smithi144 ceph-mon[32129]: pgmap v7186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:17 smithi160 ceph-mon[36715]: pgmap v7186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:17 smithi183 ceph-mon[37273]: pgmap v7186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:19 smithi144 ceph-mon[32129]: pgmap v7187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:36:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:19 smithi160 ceph-mon[36715]: pgmap v7187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:36:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:19 smithi183 ceph-mon[37273]: pgmap v7187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:36:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:21 smithi144 ceph-mon[32129]: pgmap v7188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:36:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12624.820530 secs 2024-07-26T14:36:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:21 smithi160 ceph-mon[36715]: pgmap v7188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:36:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12624.820530 secs 2024-07-26T14:36:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:21 smithi183 ceph-mon[37273]: pgmap v7188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T14:36:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12624.820530 secs 2024-07-26T14:36:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:23.065+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:36:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:23.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:36:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:23.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:36:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:23.066+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:36:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:23.067+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:23 smithi144 ceph-mon[32129]: pgmap v7189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:23 smithi160 ceph-mon[36715]: pgmap v7189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:23 smithi183 ceph-mon[37273]: pgmap v7189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:25 smithi144 ceph-mon[32129]: pgmap v7190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:36:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:25 smithi160 ceph-mon[36715]: pgmap v7190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:36:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:25 smithi183 ceph-mon[37273]: pgmap v7190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:36:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12629.820657 secs 2024-07-26T14:36:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12629.820657 secs 2024-07-26T14:36:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12629.820657 secs 2024-07-26T14:36:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:27 smithi144 ceph-mon[32129]: pgmap v7191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:27 smithi160 ceph-mon[36715]: pgmap v7191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:27 smithi183 ceph-mon[37273]: pgmap v7191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:36:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:29 smithi144 ceph-mon[32129]: pgmap v7192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:29 smithi160 ceph-mon[36715]: pgmap v7192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:29 smithi183 ceph-mon[37273]: pgmap v7192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:31 smithi144 ceph-mon[32129]: pgmap v7193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12634.820784 secs 2024-07-26T14:36:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:31 smithi160 ceph-mon[36715]: pgmap v7193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12634.820784 secs 2024-07-26T14:36:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:31 smithi183 ceph-mon[37273]: pgmap v7193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12634.820784 secs 2024-07-26T14:36:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:33 smithi144 ceph-mon[32129]: pgmap v7194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:33 smithi160 ceph-mon[36715]: pgmap v7194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:33 smithi183 ceph-mon[37273]: pgmap v7194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:35 smithi144 ceph-mon[32129]: pgmap v7195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:35 smithi160 ceph-mon[36715]: pgmap v7195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:35 smithi183 ceph-mon[37273]: pgmap v7195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12639.820913 secs 2024-07-26T14:36:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12639.820913 secs 2024-07-26T14:36:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12639.820913 secs 2024-07-26T14:36:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:37 smithi160 ceph-mon[36715]: pgmap v7196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:38.083+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:36:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:38.084+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:36:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:38.084+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:36:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:38.084+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:36:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:38.084+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:37 smithi183 ceph-mon[37273]: pgmap v7196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:37 smithi144 ceph-mon[32129]: pgmap v7196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:39 smithi160 ceph-mon[36715]: pgmap v7197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:39 smithi183 ceph-mon[37273]: pgmap v7197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:39 smithi144 ceph-mon[32129]: pgmap v7197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:41 smithi160 ceph-mon[36715]: pgmap v7198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12644.821039 secs 2024-07-26T14:36:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:41 smithi183 ceph-mon[37273]: pgmap v7198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12644.821039 secs 2024-07-26T14:36:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:41 smithi144 ceph-mon[32129]: pgmap v7198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12644.821039 secs 2024-07-26T14:36:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:43 smithi160 ceph-mon[36715]: pgmap v7199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:43 smithi183 ceph-mon[37273]: pgmap v7199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:43 smithi144 ceph-mon[32129]: pgmap v7199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:46.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:45 smithi144 ceph-mon[32129]: pgmap v7200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:45 smithi160 ceph-mon[36715]: pgmap v7200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:45 smithi183 ceph-mon[37273]: pgmap v7200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:47.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12649.821161 secs 2024-07-26T14:36:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12649.821161 secs 2024-07-26T14:36:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12649.821161 secs 2024-07-26T14:36:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:47 smithi160 ceph-mon[36715]: pgmap v7201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:47 smithi183 ceph-mon[37273]: pgmap v7201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:47 smithi144 ceph-mon[32129]: pgmap v7201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:49 smithi160 ceph-mon[36715]: pgmap v7202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:49 smithi183 ceph-mon[37273]: pgmap v7202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:49 smithi144 ceph-mon[32129]: pgmap v7202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:51 smithi160 ceph-mon[36715]: pgmap v7203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12654.821270 secs 2024-07-26T14:36:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:51 smithi183 ceph-mon[37273]: pgmap v7203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12654.821270 secs 2024-07-26T14:36:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:51 smithi144 ceph-mon[32129]: pgmap v7203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12654.821270 secs 2024-07-26T14:36:53.101 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:53.100+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:36:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:53.101+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:36:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:53.101+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:36:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:53.101+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:36:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:36:53.101+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:53 smithi160 ceph-mon[36715]: pgmap v7204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:53 smithi183 ceph-mon[37273]: pgmap v7204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:53 smithi144 ceph-mon[32129]: pgmap v7204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:36:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:55 smithi160 ceph-mon[36715]: pgmap v7205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:55 smithi183 ceph-mon[37273]: pgmap v7205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:55 smithi144 ceph-mon[32129]: pgmap v7205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:36:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12659.821397 secs 2024-07-26T14:36:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12659.821397 secs 2024-07-26T14:36:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12659.821397 secs 2024-07-26T14:36:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:57 smithi183 ceph-mon[37273]: pgmap v7206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:57 smithi144 ceph-mon[32129]: pgmap v7206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:36:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:57 smithi160 ceph-mon[36715]: pgmap v7206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:36:59 smithi183 ceph-mon[37273]: pgmap v7207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:36:59 smithi144 ceph-mon[32129]: pgmap v7207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:36:59 smithi160 ceph-mon[36715]: pgmap v7207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:01 smithi183 ceph-mon[37273]: pgmap v7208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12664.821483 secs 2024-07-26T14:37:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:01 smithi144 ceph-mon[32129]: pgmap v7208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12664.821483 secs 2024-07-26T14:37:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:01 smithi160 ceph-mon[36715]: pgmap v7208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12664.821483 secs 2024-07-26T14:37:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:03 smithi183 ceph-mon[37273]: pgmap v7209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:03 smithi144 ceph-mon[32129]: pgmap v7209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:03 smithi160 ceph-mon[36715]: pgmap v7209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:05 smithi183 ceph-mon[37273]: pgmap v7210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:05 smithi144 ceph-mon[32129]: pgmap v7210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:05 smithi160 ceph-mon[36715]: pgmap v7210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12669.821556 secs 2024-07-26T14:37:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12669.821556 secs 2024-07-26T14:37:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12669.821556 secs 2024-07-26T14:37:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:08.117+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:37:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:08.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:37:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:08.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:37:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:08.118+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:37:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:08.118+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:07 smithi183 ceph-mon[37273]: pgmap v7211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:07 smithi144 ceph-mon[32129]: pgmap v7211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:07 smithi160 ceph-mon[36715]: pgmap v7211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:10.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:09 smithi160 ceph-mon[36715]: pgmap v7212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:09 smithi183 ceph-mon[37273]: pgmap v7212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:09 smithi144 ceph-mon[32129]: pgmap v7212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:11 smithi183 ceph-mon[37273]: pgmap v7213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12674.821691 secs 2024-07-26T14:37:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:11 smithi144 ceph-mon[32129]: pgmap v7213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12674.821691 secs 2024-07-26T14:37:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:11 smithi160 ceph-mon[36715]: pgmap v7213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12674.821691 secs 2024-07-26T14:37:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:13 smithi183 ceph-mon[37273]: pgmap v7214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:13 smithi144 ceph-mon[32129]: pgmap v7214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:13 smithi160 ceph-mon[36715]: pgmap v7214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:15 smithi183 ceph-mon[37273]: pgmap v7215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:15 smithi144 ceph-mon[32129]: pgmap v7215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:16.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:15 smithi160 ceph-mon[36715]: pgmap v7215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12679.821796 secs 2024-07-26T14:37:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12679.821796 secs 2024-07-26T14:37:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12679.821796 secs 2024-07-26T14:37:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:17 smithi144 ceph-mon[32129]: pgmap v7216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:37:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:17 smithi160 ceph-mon[36715]: pgmap v7216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:37:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:17 smithi183 ceph-mon[37273]: pgmap v7216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:37:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:18 smithi144 ceph-mon[32129]: pgmap v7217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:37:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:18 smithi144 ceph-mon[32129]: pgmap v7218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:37:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:18 smithi160 ceph-mon[36715]: pgmap v7217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:37:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:18 smithi160 ceph-mon[36715]: pgmap v7218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:37:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:18 smithi183 ceph-mon[37273]: pgmap v7217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:37:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:18 smithi183 ceph-mon[37273]: pgmap v7218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:37:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:19 smithi144 ceph-mon[32129]: pgmap v7219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:37:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:19 smithi160 ceph-mon[36715]: pgmap v7219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:37:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:19 smithi183 ceph-mon[37273]: pgmap v7219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:37:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12684.821895 secs 2024-07-26T14:37:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12684.821895 secs 2024-07-26T14:37:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12684.821895 secs 2024-07-26T14:37:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:22 smithi183 ceph-mon[37273]: pgmap v7220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:37:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:22 smithi144 ceph-mon[32129]: pgmap v7220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:37:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:22 smithi160 ceph-mon[36715]: pgmap v7220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:37:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:23.133+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:37:23.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:23.134+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:37:23.136 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:23.134+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:37:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:23.134+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:37:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:23.135+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:24 smithi183 ceph-mon[37273]: pgmap v7221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:37:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:24 smithi144 ceph-mon[32129]: pgmap v7221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:37:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:24 smithi160 ceph-mon[36715]: pgmap v7221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:37:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:26 smithi183 ceph-mon[37273]: pgmap v7222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:37:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12689.822049 secs 2024-07-26T14:37:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:26 smithi144 ceph-mon[32129]: pgmap v7222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:37:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12689.822049 secs 2024-07-26T14:37:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:26 smithi160 ceph-mon[36715]: pgmap v7222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:37:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12689.822049 secs 2024-07-26T14:37:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:28 smithi183 ceph-mon[37273]: pgmap v7223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:37:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:28 smithi144 ceph-mon[32129]: pgmap v7223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:37:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:28 smithi160 ceph-mon[36715]: pgmap v7223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:37:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:30 smithi183 ceph-mon[37273]: pgmap v7224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:30 smithi144 ceph-mon[32129]: pgmap v7224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:30 smithi160 ceph-mon[36715]: pgmap v7224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12694.822135 secs 2024-07-26T14:37:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12694.822135 secs 2024-07-26T14:37:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12694.822135 secs 2024-07-26T14:37:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:32 smithi183 ceph-mon[37273]: pgmap v7225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:32 smithi144 ceph-mon[32129]: pgmap v7225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:32 smithi160 ceph-mon[36715]: pgmap v7225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:34 smithi144 ceph-mon[32129]: pgmap v7226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:34 smithi160 ceph-mon[36715]: pgmap v7226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:34 smithi183 ceph-mon[37273]: pgmap v7226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:36 smithi144 ceph-mon[32129]: pgmap v7227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12699.822279 secs 2024-07-26T14:37:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:36 smithi160 ceph-mon[36715]: pgmap v7227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12699.822279 secs 2024-07-26T14:37:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:36 smithi183 ceph-mon[37273]: pgmap v7227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12699.822279 secs 2024-07-26T14:37:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:38.151+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:37:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:38.152+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:37:38.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:38.152+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:37:38.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:38.152+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:37:38.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:38.152+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:38 smithi144 ceph-mon[32129]: pgmap v7228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:38 smithi160 ceph-mon[36715]: pgmap v7228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:38 smithi183 ceph-mon[37273]: pgmap v7228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:40.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:39 smithi160 ceph-mon[36715]: pgmap v7229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:39 smithi144 ceph-mon[32129]: pgmap v7229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:39 smithi183 ceph-mon[37273]: pgmap v7229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12704.822410 secs 2024-07-26T14:37:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12704.822410 secs 2024-07-26T14:37:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12704.822410 secs 2024-07-26T14:37:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:42 smithi183 ceph-mon[37273]: pgmap v7230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:42 smithi144 ceph-mon[32129]: pgmap v7230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:42 smithi160 ceph-mon[36715]: pgmap v7230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:44 smithi183 ceph-mon[37273]: pgmap v7231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:44 smithi144 ceph-mon[32129]: pgmap v7231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:44 smithi160 ceph-mon[36715]: pgmap v7231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:46 smithi183 ceph-mon[37273]: pgmap v7232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12709.822544 secs 2024-07-26T14:37:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:46 smithi144 ceph-mon[32129]: pgmap v7232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12709.822544 secs 2024-07-26T14:37:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:46 smithi160 ceph-mon[36715]: pgmap v7232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12709.822544 secs 2024-07-26T14:37:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:48 smithi183 ceph-mon[37273]: pgmap v7233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:48 smithi144 ceph-mon[32129]: pgmap v7233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:48 smithi160 ceph-mon[36715]: pgmap v7233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:50 smithi144 ceph-mon[32129]: pgmap v7234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:50 smithi160 ceph-mon[36715]: pgmap v7234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:50 smithi183 ceph-mon[37273]: pgmap v7234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12714.822642 secs 2024-07-26T14:37:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12714.822642 secs 2024-07-26T14:37:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12714.822642 secs 2024-07-26T14:37:53.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:53.168+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:37:53.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:53.169+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:37:53.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:53.169+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:37:53.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:53.169+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:37:53.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:37:53.169+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:52 smithi144 ceph-mon[32129]: pgmap v7235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:52 smithi160 ceph-mon[36715]: pgmap v7235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:52 smithi183 ceph-mon[37273]: pgmap v7235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:37:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:53 smithi144 ceph-mon[32129]: pgmap v7236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:53 smithi160 ceph-mon[36715]: pgmap v7236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:53 smithi183 ceph-mon[37273]: pgmap v7236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:56 smithi183 ceph-mon[37273]: pgmap v7237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12719.822766 secs 2024-07-26T14:37:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:56 smithi144 ceph-mon[32129]: pgmap v7237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12719.822766 secs 2024-07-26T14:37:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:56 smithi160 ceph-mon[36715]: pgmap v7237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:37:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12719.822766 secs 2024-07-26T14:37:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:37:58 smithi183 ceph-mon[37273]: pgmap v7238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:37:58 smithi144 ceph-mon[32129]: pgmap v7238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:37:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:37:58 smithi160 ceph-mon[36715]: pgmap v7238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:00 smithi183 ceph-mon[37273]: pgmap v7239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:00 smithi144 ceph-mon[32129]: pgmap v7239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:00 smithi160 ceph-mon[36715]: pgmap v7239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12724.822890 secs 2024-07-26T14:38:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12724.822890 secs 2024-07-26T14:38:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12724.822890 secs 2024-07-26T14:38:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:02 smithi144 ceph-mon[32129]: pgmap v7240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:02 smithi160 ceph-mon[36715]: pgmap v7240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:02 smithi183 ceph-mon[37273]: pgmap v7240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:04 smithi144 ceph-mon[32129]: pgmap v7241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:04 smithi160 ceph-mon[36715]: pgmap v7241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:04 smithi183 ceph-mon[37273]: pgmap v7241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:06 smithi144 ceph-mon[32129]: pgmap v7242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12729.823017 secs 2024-07-26T14:38:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:06 smithi160 ceph-mon[36715]: pgmap v7242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12729.823017 secs 2024-07-26T14:38:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:06 smithi183 ceph-mon[37273]: pgmap v7242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12729.823017 secs 2024-07-26T14:38:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:08.185+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:38:08.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:08.186+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:38:08.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:08.186+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:38:08.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:08.186+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:38:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:08.187+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:07 smithi144 ceph-mon[32129]: pgmap v7243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:07 smithi160 ceph-mon[36715]: pgmap v7243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:08.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:07 smithi183 ceph-mon[37273]: pgmap v7243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:10 smithi183 ceph-mon[37273]: pgmap v7244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:10 smithi144 ceph-mon[32129]: pgmap v7244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:10 smithi160 ceph-mon[36715]: pgmap v7244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12734.823151 secs 2024-07-26T14:38:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12734.823151 secs 2024-07-26T14:38:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12734.823151 secs 2024-07-26T14:38:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:12 smithi144 ceph-mon[32129]: pgmap v7245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:12 smithi160 ceph-mon[36715]: pgmap v7245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:12 smithi183 ceph-mon[37273]: pgmap v7245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:14 smithi144 ceph-mon[32129]: pgmap v7246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:14 smithi160 ceph-mon[36715]: pgmap v7246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:14 smithi183 ceph-mon[37273]: pgmap v7246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:16.920 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:16 smithi144 ceph-mon[32129]: pgmap v7247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12739.823304 secs 2024-07-26T14:38:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:16 smithi160 ceph-mon[36715]: pgmap v7247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12739.823304 secs 2024-07-26T14:38:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:16 smithi183 ceph-mon[37273]: pgmap v7247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12739.823304 secs 2024-07-26T14:38:19.185 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:18 smithi183 ceph-mon[37273]: pgmap v7248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:19.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:18 smithi144 ceph-mon[32129]: pgmap v7248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:18 smithi160 ceph-mon[36715]: pgmap v7248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:38:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:38:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:38:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:20 smithi144 ceph-mon[32129]: pgmap v7249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:20 smithi144 ceph-mon[32129]: pgmap v7250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:38:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:20 smithi160 ceph-mon[36715]: pgmap v7249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:20 smithi160 ceph-mon[36715]: pgmap v7250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:38:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:20 smithi183 ceph-mon[37273]: pgmap v7249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:20 smithi183 ceph-mon[37273]: pgmap v7250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:38:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12744.823425 secs 2024-07-26T14:38:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:21 smithi144 ceph-mon[32129]: pgmap v7251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12744.823425 secs 2024-07-26T14:38:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:21 smithi160 ceph-mon[36715]: pgmap v7251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12744.823425 secs 2024-07-26T14:38:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:21 smithi183 ceph-mon[37273]: pgmap v7251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:23.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:23.202+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:38:23.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:23.202+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:38:23.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:23.202+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:38:23.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:23.202+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:38:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:23.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:24 smithi160 ceph-mon[36715]: pgmap v7252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:38:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:24 smithi183 ceph-mon[37273]: pgmap v7252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:38:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:24 smithi144 ceph-mon[32129]: pgmap v7252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:38:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:26 smithi160 ceph-mon[36715]: pgmap v7253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:38:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:26 smithi183 ceph-mon[37273]: pgmap v7253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:38:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:26 smithi144 ceph-mon[32129]: pgmap v7253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:38:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12749.823549 secs 2024-07-26T14:38:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12749.823549 secs 2024-07-26T14:38:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12749.823549 secs 2024-07-26T14:38:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:28 smithi183 ceph-mon[37273]: pgmap v7254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:28 smithi144 ceph-mon[32129]: pgmap v7254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:28 smithi160 ceph-mon[36715]: pgmap v7254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:38:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:30 smithi183 ceph-mon[37273]: pgmap v7255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:38:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:30 smithi144 ceph-mon[32129]: pgmap v7255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:38:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:30 smithi160 ceph-mon[36715]: pgmap v7255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:38:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12754.823676 secs 2024-07-26T14:38:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:32 smithi183 ceph-mon[37273]: pgmap v7256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12754.823676 secs 2024-07-26T14:38:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:32 smithi144 ceph-mon[32129]: pgmap v7256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12754.823676 secs 2024-07-26T14:38:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:32 smithi160 ceph-mon[36715]: pgmap v7256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:34 smithi183 ceph-mon[37273]: pgmap v7257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:34 smithi144 ceph-mon[32129]: pgmap v7257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:34 smithi160 ceph-mon[36715]: pgmap v7257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:36 smithi183 ceph-mon[37273]: pgmap v7258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:36 smithi144 ceph-mon[32129]: pgmap v7258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:36 smithi160 ceph-mon[36715]: pgmap v7258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12759.823797 secs 2024-07-26T14:38:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12759.823797 secs 2024-07-26T14:38:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12759.823797 secs 2024-07-26T14:38:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:38.208+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:38:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:38.209+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:38:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:38.209+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:38:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:38.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:38:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:38.209+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:38 smithi144 ceph-mon[32129]: pgmap v7259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:38.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:38 smithi160 ceph-mon[36715]: pgmap v7259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:38 smithi183 ceph-mon[37273]: pgmap v7259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:40 smithi144 ceph-mon[32129]: pgmap v7260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:40 smithi160 ceph-mon[36715]: pgmap v7260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:40 smithi183 ceph-mon[37273]: pgmap v7260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12764.823873 secs 2024-07-26T14:38:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:42 smithi144 ceph-mon[32129]: pgmap v7261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12764.823873 secs 2024-07-26T14:38:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:42 smithi160 ceph-mon[36715]: pgmap v7261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12764.823873 secs 2024-07-26T14:38:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:42 smithi183 ceph-mon[37273]: pgmap v7261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:44 smithi144 ceph-mon[32129]: pgmap v7262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:44 smithi160 ceph-mon[36715]: pgmap v7262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:44 smithi183 ceph-mon[37273]: pgmap v7262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:46 smithi144 ceph-mon[32129]: pgmap v7263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:46 smithi160 ceph-mon[36715]: pgmap v7263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:46 smithi183 ceph-mon[37273]: pgmap v7263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12769.823989 secs 2024-07-26T14:38:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12769.823989 secs 2024-07-26T14:38:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12769.823989 secs 2024-07-26T14:38:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:48.358+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:38:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:48.358+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:38:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:48.359+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:38:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:48.359+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:38:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:48 smithi144 ceph-mon[32129]: pgmap v7264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:48 smithi160 ceph-mon[36715]: pgmap v7264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:48 smithi183 ceph-mon[37273]: pgmap v7264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:50.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:50 smithi183 ceph-mon[37273]: pgmap v7265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:50 smithi144 ceph-mon[32129]: pgmap v7265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:50 smithi160 ceph-mon[36715]: pgmap v7265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12774.824088 secs 2024-07-26T14:38:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:52 smithi144 ceph-mon[32129]: pgmap v7266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12774.824088 secs 2024-07-26T14:38:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:52 smithi160 ceph-mon[36715]: pgmap v7266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12774.824088 secs 2024-07-26T14:38:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:52 smithi183 ceph-mon[37273]: pgmap v7266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:53.224+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:38:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:53.225+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:38:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:53.225+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:38:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:53.225+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:38:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:38:53.225+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:54 smithi144 ceph-mon[32129]: pgmap v7267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:54 smithi160 ceph-mon[36715]: pgmap v7267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:54 smithi183 ceph-mon[37273]: pgmap v7267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:38:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:56 smithi144 ceph-mon[32129]: pgmap v7268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:56 smithi160 ceph-mon[36715]: pgmap v7268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:56 smithi183 ceph-mon[37273]: pgmap v7268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:38:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12779.824217 secs 2024-07-26T14:38:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12779.824217 secs 2024-07-26T14:38:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12779.824217 secs 2024-07-26T14:38:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:38:58 smithi144 ceph-mon[32129]: pgmap v7269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:38:58 smithi160 ceph-mon[36715]: pgmap v7269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:38:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:38:58 smithi183 ceph-mon[37273]: pgmap v7269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:00 smithi160 ceph-mon[36715]: pgmap v7270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:00 smithi183 ceph-mon[37273]: pgmap v7270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:00 smithi144 ceph-mon[32129]: pgmap v7270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12784.824333 secs 2024-07-26T14:39:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:02 smithi160 ceph-mon[36715]: pgmap v7271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12784.824333 secs 2024-07-26T14:39:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:02 smithi183 ceph-mon[37273]: pgmap v7271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12784.824333 secs 2024-07-26T14:39:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:02 smithi144 ceph-mon[32129]: pgmap v7271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:04 smithi160 ceph-mon[36715]: pgmap v7272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:04 smithi183 ceph-mon[37273]: pgmap v7272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:04 smithi144 ceph-mon[32129]: pgmap v7272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:06 smithi160 ceph-mon[36715]: pgmap v7273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:39:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:06 smithi183 ceph-mon[37273]: pgmap v7273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:39:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:06 smithi144 ceph-mon[32129]: pgmap v7273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:39:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12789.824400 secs 2024-07-26T14:39:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12789.824400 secs 2024-07-26T14:39:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12789.824400 secs 2024-07-26T14:39:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:08.241+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:39:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:08.242+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:39:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:08.242+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:39:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:08.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:39:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:08.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:08 smithi160 ceph-mon[36715]: pgmap v7274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:08 smithi183 ceph-mon[37273]: pgmap v7274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:08 smithi144 ceph-mon[32129]: pgmap v7274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:10 smithi160 ceph-mon[36715]: pgmap v7275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:10 smithi183 ceph-mon[37273]: pgmap v7275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:10 smithi144 ceph-mon[32129]: pgmap v7275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12794.824467 secs 2024-07-26T14:39:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:12 smithi160 ceph-mon[36715]: pgmap v7276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12794.824467 secs 2024-07-26T14:39:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:12 smithi183 ceph-mon[37273]: pgmap v7276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12794.824467 secs 2024-07-26T14:39:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:12 smithi144 ceph-mon[32129]: pgmap v7276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:14 smithi160 ceph-mon[36715]: pgmap v7277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:39:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:14 smithi183 ceph-mon[37273]: pgmap v7277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:39:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:14 smithi144 ceph-mon[32129]: pgmap v7277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:39:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:16 smithi160 ceph-mon[36715]: pgmap v7278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:39:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:16 smithi183 ceph-mon[37273]: pgmap v7278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:39:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:16 smithi144 ceph-mon[32129]: pgmap v7278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:39:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12799.824536 secs 2024-07-26T14:39:17.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12799.824536 secs 2024-07-26T14:39:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12799.824536 secs 2024-07-26T14:39:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:18 smithi160 ceph-mon[36715]: pgmap v7279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:18 smithi183 ceph-mon[37273]: pgmap v7279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:18 smithi144 ceph-mon[32129]: pgmap v7279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:19.968 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:20 smithi183 ceph-mon[37273]: pgmap v7280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:39:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:20 smithi144 ceph-mon[32129]: pgmap v7280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:39:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:20 smithi160 ceph-mon[36715]: pgmap v7280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:39:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:21 smithi183 ceph-mon[37273]: pgmap v7281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:39:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:21 smithi183 ceph-mon[37273]: pgmap v7282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:39:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12804.824626 secs 2024-07-26T14:39:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:21 smithi144 ceph-mon[32129]: pgmap v7281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:39:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:21 smithi144 ceph-mon[32129]: pgmap v7282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:39:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12804.824626 secs 2024-07-26T14:39:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:21 smithi160 ceph-mon[36715]: pgmap v7281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:39:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:21 smithi160 ceph-mon[36715]: pgmap v7282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:39:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12804.824626 secs 2024-07-26T14:39:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:23.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:39:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:23.257+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:39:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:23.257+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:39:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:23.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:39:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:23.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:23 smithi183 ceph-mon[37273]: pgmap v7283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:23 smithi144 ceph-mon[32129]: pgmap v7283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:23 smithi160 ceph-mon[36715]: pgmap v7283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:25 smithi183 ceph-mon[37273]: pgmap v7284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:25 smithi144 ceph-mon[32129]: pgmap v7284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:25 smithi160 ceph-mon[36715]: pgmap v7284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12809.824748 secs 2024-07-26T14:39:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:27 smithi183 ceph-mon[37273]: pgmap v7285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12809.824748 secs 2024-07-26T14:39:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:27 smithi144 ceph-mon[32129]: pgmap v7285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12809.824748 secs 2024-07-26T14:39:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:27 smithi160 ceph-mon[36715]: pgmap v7285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:29 smithi183 ceph-mon[37273]: pgmap v7286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:29 smithi144 ceph-mon[32129]: pgmap v7286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:29 smithi160 ceph-mon[36715]: pgmap v7286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:39:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:31 smithi183 ceph-mon[37273]: pgmap v7287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:31 smithi144 ceph-mon[32129]: pgmap v7287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:31 smithi160 ceph-mon[36715]: pgmap v7287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:39:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12814.824876 secs 2024-07-26T14:39:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12814.824876 secs 2024-07-26T14:39:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12814.824876 secs 2024-07-26T14:39:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:33 smithi144 ceph-mon[32129]: pgmap v7288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:33 smithi160 ceph-mon[36715]: pgmap v7288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:33 smithi183 ceph-mon[37273]: pgmap v7288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:35 smithi144 ceph-mon[32129]: pgmap v7289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:35 smithi160 ceph-mon[36715]: pgmap v7289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:35 smithi183 ceph-mon[37273]: pgmap v7289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12819.824996 secs 2024-07-26T14:39:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:37 smithi144 ceph-mon[32129]: pgmap v7290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12819.824996 secs 2024-07-26T14:39:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:37 smithi160 ceph-mon[36715]: pgmap v7290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12819.824996 secs 2024-07-26T14:39:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:37 smithi183 ceph-mon[37273]: pgmap v7290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:38.270+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:39:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:38.271+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:39:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:38.271+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:39:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:38.271+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:39:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:38.271+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:39 smithi144 ceph-mon[32129]: pgmap v7291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:39 smithi160 ceph-mon[36715]: pgmap v7291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:39 smithi183 ceph-mon[37273]: pgmap v7291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:41 smithi144 ceph-mon[32129]: pgmap v7292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:41 smithi160 ceph-mon[36715]: pgmap v7292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:41 smithi183 ceph-mon[37273]: pgmap v7292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:42.709 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12824.825132 secs 2024-07-26T14:39:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12824.825132 secs 2024-07-26T14:39:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12824.825132 secs 2024-07-26T14:39:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:43 smithi144 ceph-mon[32129]: pgmap v7293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:43 smithi160 ceph-mon[36715]: pgmap v7293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:43 smithi183 ceph-mon[37273]: pgmap v7293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:45 smithi144 ceph-mon[32129]: pgmap v7294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:45 smithi160 ceph-mon[36715]: pgmap v7294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:45 smithi183 ceph-mon[37273]: pgmap v7294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12829.825256 secs 2024-07-26T14:39:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:47 smithi144 ceph-mon[32129]: pgmap v7295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12829.825256 secs 2024-07-26T14:39:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:47 smithi160 ceph-mon[36715]: pgmap v7295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12829.825256 secs 2024-07-26T14:39:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:47 smithi183 ceph-mon[37273]: pgmap v7295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:49 smithi144 ceph-mon[32129]: pgmap v7296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:49 smithi160 ceph-mon[36715]: pgmap v7296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:49 smithi183 ceph-mon[37273]: pgmap v7296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:51 smithi144 ceph-mon[32129]: pgmap v7297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:51 smithi160 ceph-mon[36715]: pgmap v7297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:51 smithi183 ceph-mon[37273]: pgmap v7297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:39:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12834.825375 secs 2024-07-26T14:39:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12834.825375 secs 2024-07-26T14:39:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12834.825375 secs 2024-07-26T14:39:53.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:53.287+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:39:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:53.288+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:39:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:53.288+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:39:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:53.288+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:39:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:39:53.288+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:53.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:53 smithi144 ceph-mon[32129]: pgmap v7298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:53 smithi160 ceph-mon[36715]: pgmap v7298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:53 smithi183 ceph-mon[37273]: pgmap v7298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:55 smithi160 ceph-mon[36715]: pgmap v7299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:55 smithi183 ceph-mon[37273]: pgmap v7299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:55 smithi144 ceph-mon[32129]: pgmap v7299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12839.825492 secs 2024-07-26T14:39:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:57 smithi160 ceph-mon[36715]: pgmap v7300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12839.825492 secs 2024-07-26T14:39:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:57 smithi183 ceph-mon[37273]: pgmap v7300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12839.825492 secs 2024-07-26T14:39:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:57 smithi144 ceph-mon[32129]: pgmap v7300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:39:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:39:59 smithi160 ceph-mon[36715]: pgmap v7301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:39:59 smithi183 ceph-mon[37273]: pgmap v7301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:39:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:39:59 smithi144 ceph-mon[32129]: pgmap v7301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:40:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:40:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:40:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:01 smithi160 ceph-mon[36715]: pgmap v7302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:01 smithi183 ceph-mon[37273]: pgmap v7302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:01 smithi144 ceph-mon[32129]: pgmap v7302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12844.825610 secs 2024-07-26T14:40:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12844.825610 secs 2024-07-26T14:40:02.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12844.825610 secs 2024-07-26T14:40:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:03 smithi160 ceph-mon[36715]: pgmap v7303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:03 smithi183 ceph-mon[37273]: pgmap v7303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:03 smithi144 ceph-mon[32129]: pgmap v7303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:05 smithi160 ceph-mon[36715]: pgmap v7304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:05 smithi183 ceph-mon[37273]: pgmap v7304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:05 smithi144 ceph-mon[32129]: pgmap v7304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:07.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12849.825728 secs 2024-07-26T14:40:07.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:07 smithi160 ceph-mon[36715]: pgmap v7305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12849.825728 secs 2024-07-26T14:40:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:07 smithi183 ceph-mon[37273]: pgmap v7305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12849.825728 secs 2024-07-26T14:40:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:07 smithi144 ceph-mon[32129]: pgmap v7305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:08.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:40:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:08.305+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:40:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:08.305+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:40:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:08.305+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:40:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:08.305+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:09 smithi160 ceph-mon[36715]: pgmap v7306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:09 smithi183 ceph-mon[37273]: pgmap v7306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:09 smithi144 ceph-mon[32129]: pgmap v7306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:11 smithi160 ceph-mon[36715]: pgmap v7307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:11 smithi183 ceph-mon[37273]: pgmap v7307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:11 smithi144 ceph-mon[32129]: pgmap v7307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12854.825849 secs 2024-07-26T14:40:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12854.825849 secs 2024-07-26T14:40:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12854.825849 secs 2024-07-26T14:40:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:13 smithi160 ceph-mon[36715]: pgmap v7308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:13 smithi183 ceph-mon[37273]: pgmap v7308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:13 smithi144 ceph-mon[32129]: pgmap v7308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:15 smithi183 ceph-mon[37273]: pgmap v7309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:15 smithi144 ceph-mon[32129]: pgmap v7309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:15 smithi160 ceph-mon[36715]: pgmap v7309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12859.826004 secs 2024-07-26T14:40:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12859.826004 secs 2024-07-26T14:40:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12859.826004 secs 2024-07-26T14:40:17.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:17 smithi144 ceph-mon[32129]: pgmap v7310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:17.883 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:17 smithi183 ceph-mon[37273]: pgmap v7310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:17 smithi160 ceph-mon[36715]: pgmap v7310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:18.374 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:18.373+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:40:18.374 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:18.373+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:40:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:18.373+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:40:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:18.373+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:40:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:19 smithi183 ceph-mon[37273]: pgmap v7311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:19.886 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:19 smithi144 ceph-mon[32129]: pgmap v7311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:19 smithi160 ceph-mon[36715]: pgmap v7311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:21 smithi183 ceph-mon[37273]: pgmap v7312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12864.826099 secs 2024-07-26T14:40:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:40:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:21 smithi144 ceph-mon[32129]: pgmap v7312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12864.826099 secs 2024-07-26T14:40:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:40:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:21 smithi160 ceph-mon[36715]: pgmap v7312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12864.826099 secs 2024-07-26T14:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:40:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:22 smithi183 ceph-mon[37273]: pgmap v7313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:22 smithi183 ceph-mon[37273]: pgmap v7314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:22 smithi144 ceph-mon[32129]: pgmap v7313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:22 smithi144 ceph-mon[32129]: pgmap v7314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:22 smithi160 ceph-mon[36715]: pgmap v7313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:22 smithi160 ceph-mon[36715]: pgmap v7314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:23.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:40:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:23.321+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:40:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:23.321+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:40:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:23.322+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:40:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:23.322+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:24 smithi183 ceph-mon[37273]: pgmap v7315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:40:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:24 smithi144 ceph-mon[32129]: pgmap v7315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:40:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:24 smithi160 ceph-mon[36715]: pgmap v7315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:40:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:26 smithi183 ceph-mon[37273]: pgmap v7316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12869.826221 secs 2024-07-26T14:40:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:26 smithi144 ceph-mon[32129]: pgmap v7316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12869.826221 secs 2024-07-26T14:40:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:26 smithi160 ceph-mon[36715]: pgmap v7316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12869.826221 secs 2024-07-26T14:40:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:28 smithi183 ceph-mon[37273]: pgmap v7317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:40:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:28 smithi144 ceph-mon[32129]: pgmap v7317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:40:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:28 smithi160 ceph-mon[36715]: pgmap v7317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:40:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:30 smithi183 ceph-mon[37273]: pgmap v7318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:30 smithi144 ceph-mon[32129]: pgmap v7318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:30 smithi160 ceph-mon[36715]: pgmap v7318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:40:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12874.826355 secs 2024-07-26T14:40:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12874.826355 secs 2024-07-26T14:40:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12874.826355 secs 2024-07-26T14:40:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:32 smithi144 ceph-mon[32129]: pgmap v7319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:40:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:32 smithi160 ceph-mon[36715]: pgmap v7319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:40:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:32 smithi183 ceph-mon[37273]: pgmap v7319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:40:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:34 smithi144 ceph-mon[32129]: pgmap v7320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:34 smithi160 ceph-mon[36715]: pgmap v7320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:34 smithi183 ceph-mon[37273]: pgmap v7320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:36 smithi144 ceph-mon[32129]: pgmap v7321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12879.826428 secs 2024-07-26T14:40:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:36 smithi160 ceph-mon[36715]: pgmap v7321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12879.826428 secs 2024-07-26T14:40:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:36 smithi183 ceph-mon[37273]: pgmap v7321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12879.826428 secs 2024-07-26T14:40:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:38.329+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:40:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:38.330+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:40:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:38.330+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:40:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:38.330+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:40:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:38.330+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:38 smithi144 ceph-mon[32129]: pgmap v7322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:38 smithi160 ceph-mon[36715]: pgmap v7322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:38 smithi183 ceph-mon[37273]: pgmap v7322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:40 smithi144 ceph-mon[32129]: pgmap v7323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:40 smithi160 ceph-mon[36715]: pgmap v7323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:40 smithi183 ceph-mon[37273]: pgmap v7323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12884.826519 secs 2024-07-26T14:40:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12884.826519 secs 2024-07-26T14:40:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12884.826519 secs 2024-07-26T14:40:42.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:42 smithi160 ceph-mon[36715]: pgmap v7324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:42 smithi144 ceph-mon[32129]: pgmap v7324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:42 smithi183 ceph-mon[37273]: pgmap v7324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:44 smithi144 ceph-mon[32129]: pgmap v7325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:44 smithi160 ceph-mon[36715]: pgmap v7325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:44 smithi183 ceph-mon[37273]: pgmap v7325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:46 smithi144 ceph-mon[32129]: pgmap v7326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12889.826645 secs 2024-07-26T14:40:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:46 smithi160 ceph-mon[36715]: pgmap v7326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12889.826645 secs 2024-07-26T14:40:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:46 smithi183 ceph-mon[37273]: pgmap v7326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12889.826645 secs 2024-07-26T14:40:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:48 smithi144 ceph-mon[32129]: pgmap v7327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:48 smithi160 ceph-mon[36715]: pgmap v7327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:48 smithi183 ceph-mon[37273]: pgmap v7327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:50 smithi144 ceph-mon[32129]: pgmap v7328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:50 smithi160 ceph-mon[36715]: pgmap v7328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:50 smithi183 ceph-mon[37273]: pgmap v7328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12894.826779 secs 2024-07-26T14:40:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:52 smithi144 ceph-mon[32129]: pgmap v7329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12894.826779 secs 2024-07-26T14:40:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:52 smithi160 ceph-mon[36715]: pgmap v7329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12894.826779 secs 2024-07-26T14:40:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:52 smithi183 ceph-mon[37273]: pgmap v7329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:40:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:53.346+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:40:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:53.347+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:40:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:53.347+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:40:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:53.347+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:40:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:40:53.347+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:54 smithi144 ceph-mon[32129]: pgmap v7330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:54 smithi160 ceph-mon[36715]: pgmap v7330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:54 smithi183 ceph-mon[37273]: pgmap v7330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:56 smithi144 ceph-mon[32129]: pgmap v7331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:56 smithi160 ceph-mon[36715]: pgmap v7331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:56 smithi183 ceph-mon[37273]: pgmap v7331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:40:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12899.826880 secs 2024-07-26T14:40:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12899.826880 secs 2024-07-26T14:40:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12899.826880 secs 2024-07-26T14:40:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:40:58 smithi144 ceph-mon[32129]: pgmap v7332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:40:58 smithi160 ceph-mon[36715]: pgmap v7332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:40:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:40:58 smithi183 ceph-mon[37273]: pgmap v7332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:00 smithi144 ceph-mon[32129]: pgmap v7333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:00 smithi160 ceph-mon[36715]: pgmap v7333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:00 smithi183 ceph-mon[37273]: pgmap v7333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12904.827024 secs 2024-07-26T14:41:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:02 smithi144 ceph-mon[32129]: pgmap v7334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12904.827024 secs 2024-07-26T14:41:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:02 smithi160 ceph-mon[36715]: pgmap v7334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12904.827024 secs 2024-07-26T14:41:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:02 smithi183 ceph-mon[37273]: pgmap v7334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:04 smithi144 ceph-mon[32129]: pgmap v7335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:04 smithi160 ceph-mon[36715]: pgmap v7335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:04 smithi183 ceph-mon[37273]: pgmap v7335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:06 smithi144 ceph-mon[32129]: pgmap v7336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:06 smithi160 ceph-mon[36715]: pgmap v7336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:06 smithi183 ceph-mon[37273]: pgmap v7336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12909.827146 secs 2024-07-26T14:41:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12909.827146 secs 2024-07-26T14:41:07.968 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12909.827146 secs 2024-07-26T14:41:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:08.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:41:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:08.350+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:41:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:08.350+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:41:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:08.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:41:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:08.351+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:08 smithi160 ceph-mon[36715]: pgmap v7337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:08 smithi183 ceph-mon[37273]: pgmap v7337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:08 smithi144 ceph-mon[32129]: pgmap v7337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:10 smithi160 ceph-mon[36715]: pgmap v7338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:10 smithi183 ceph-mon[37273]: pgmap v7338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:10 smithi144 ceph-mon[32129]: pgmap v7338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12914.827261 secs 2024-07-26T14:41:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:12 smithi160 ceph-mon[36715]: pgmap v7339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12914.827261 secs 2024-07-26T14:41:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:12 smithi183 ceph-mon[37273]: pgmap v7339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12914.827261 secs 2024-07-26T14:41:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:12 smithi144 ceph-mon[32129]: pgmap v7339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:14 smithi160 ceph-mon[36715]: pgmap v7340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:14 smithi183 ceph-mon[37273]: pgmap v7340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:14 smithi144 ceph-mon[32129]: pgmap v7340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:16 smithi160 ceph-mon[36715]: pgmap v7341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:16 smithi183 ceph-mon[37273]: pgmap v7341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:16 smithi144 ceph-mon[32129]: pgmap v7341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12919.827382 secs 2024-07-26T14:41:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12919.827382 secs 2024-07-26T14:41:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12919.827382 secs 2024-07-26T14:41:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:18 smithi160 ceph-mon[36715]: pgmap v7342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:18.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:18 smithi144 ceph-mon[32129]: pgmap v7342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:18 smithi183 ceph-mon[37273]: pgmap v7342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:20 smithi160 ceph-mon[36715]: pgmap v7343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:20 smithi183 ceph-mon[37273]: pgmap v7343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:20 smithi144 ceph-mon[32129]: pgmap v7343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12924.827487 secs 2024-07-26T14:41:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:22 smithi160 ceph-mon[36715]: pgmap v7344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12924.827487 secs 2024-07-26T14:41:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:22 smithi183 ceph-mon[37273]: pgmap v7344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12924.827487 secs 2024-07-26T14:41:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:22 smithi144 ceph-mon[32129]: pgmap v7344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:23.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:41:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:23.367+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:41:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:23.367+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:41:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:23.367+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:41:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:23.368+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:23 smithi160 ceph-mon[36715]: pgmap v7345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:41:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:23 smithi160 ceph-mon[36715]: pgmap v7346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:41:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:41:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:23 smithi183 ceph-mon[37273]: pgmap v7345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:41:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:23 smithi183 ceph-mon[37273]: pgmap v7346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:41:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:41:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:23 smithi144 ceph-mon[32129]: pgmap v7345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:41:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:23 smithi144 ceph-mon[32129]: pgmap v7346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:41:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:41:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:25 smithi144 ceph-mon[32129]: pgmap v7347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:41:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:25 smithi160 ceph-mon[36715]: pgmap v7347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:41:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:25 smithi183 ceph-mon[37273]: pgmap v7347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:41:26.941 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12929.827582 secs 2024-07-26T14:41:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12929.827582 secs 2024-07-26T14:41:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12929.827582 secs 2024-07-26T14:41:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:27 smithi144 ceph-mon[32129]: pgmap v7348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:41:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:27 smithi160 ceph-mon[36715]: pgmap v7348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:41:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:27 smithi183 ceph-mon[37273]: pgmap v7348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:41:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:29 smithi144 ceph-mon[32129]: pgmap v7349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:41:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:29 smithi160 ceph-mon[36715]: pgmap v7349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:41:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:29 smithi183 ceph-mon[37273]: pgmap v7349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:41:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:31 smithi144 ceph-mon[32129]: pgmap v7350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:41:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12934.827706 secs 2024-07-26T14:41:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:31 smithi160 ceph-mon[36715]: pgmap v7350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:41:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12934.827706 secs 2024-07-26T14:41:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:31 smithi183 ceph-mon[37273]: pgmap v7350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:41:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12934.827706 secs 2024-07-26T14:41:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:33 smithi144 ceph-mon[32129]: pgmap v7351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:41:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:33 smithi160 ceph-mon[36715]: pgmap v7351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:41:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:33 smithi183 ceph-mon[37273]: pgmap v7351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:41:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:35 smithi144 ceph-mon[32129]: pgmap v7352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:35 smithi160 ceph-mon[36715]: pgmap v7352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:35 smithi183 ceph-mon[37273]: pgmap v7352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12939.827830 secs 2024-07-26T14:41:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12939.827830 secs 2024-07-26T14:41:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12939.827830 secs 2024-07-26T14:41:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:37 smithi144 ceph-mon[32129]: pgmap v7353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:37 smithi160 ceph-mon[36715]: pgmap v7353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:37 smithi183 ceph-mon[37273]: pgmap v7353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:38.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:41:38.385 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:38.384+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:41:38.385 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:38.384+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:41:38.385 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:38.384+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:41:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:38.385+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:39 smithi144 ceph-mon[32129]: pgmap v7354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:39 smithi160 ceph-mon[36715]: pgmap v7354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:39 smithi183 ceph-mon[37273]: pgmap v7354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:41 smithi144 ceph-mon[32129]: pgmap v7355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12944.827951 secs 2024-07-26T14:41:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:41 smithi160 ceph-mon[36715]: pgmap v7355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12944.827951 secs 2024-07-26T14:41:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:41 smithi183 ceph-mon[37273]: pgmap v7355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12944.827951 secs 2024-07-26T14:41:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:43 smithi144 ceph-mon[32129]: pgmap v7356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:43 smithi160 ceph-mon[36715]: pgmap v7356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:43 smithi183 ceph-mon[37273]: pgmap v7356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:45 smithi160 ceph-mon[36715]: pgmap v7357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:45 smithi183 ceph-mon[37273]: pgmap v7357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:45 smithi144 ceph-mon[32129]: pgmap v7357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12949.828090 secs 2024-07-26T14:41:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12949.828090 secs 2024-07-26T14:41:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12949.828090 secs 2024-07-26T14:41:48.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:47 smithi160 ceph-mon[36715]: pgmap v7358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:48.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:47 smithi144 ceph-mon[32129]: pgmap v7358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:47 smithi183 ceph-mon[37273]: pgmap v7358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:48.389+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:41:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:48.389+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:41:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:48.389+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:41:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:48.389+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:41:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:49 smithi160 ceph-mon[36715]: pgmap v7359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:49 smithi183 ceph-mon[37273]: pgmap v7359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:49 smithi144 ceph-mon[32129]: pgmap v7359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:51 smithi160 ceph-mon[36715]: pgmap v7360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12954.828207 secs 2024-07-26T14:41:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:51 smithi183 ceph-mon[37273]: pgmap v7360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12954.828207 secs 2024-07-26T14:41:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:51 smithi144 ceph-mon[32129]: pgmap v7360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12954.828207 secs 2024-07-26T14:41:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:53.390+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:41:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:53.391+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:41:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:53.391+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:41:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:53.391+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:41:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:41:53.391+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:53 smithi160 ceph-mon[36715]: pgmap v7361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:53 smithi183 ceph-mon[37273]: pgmap v7361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:53 smithi144 ceph-mon[32129]: pgmap v7361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:41:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:55 smithi160 ceph-mon[36715]: pgmap v7362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:55 smithi183 ceph-mon[37273]: pgmap v7362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:55 smithi144 ceph-mon[32129]: pgmap v7362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:41:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12959.828375 secs 2024-07-26T14:41:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12959.828375 secs 2024-07-26T14:41:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12959.828375 secs 2024-07-26T14:41:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:57 smithi160 ceph-mon[36715]: pgmap v7363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:57 smithi183 ceph-mon[37273]: pgmap v7363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:41:58.173 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:57 smithi144 ceph-mon[32129]: pgmap v7363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:41:59 smithi160 ceph-mon[36715]: pgmap v7364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:41:59 smithi183 ceph-mon[37273]: pgmap v7364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:41:59 smithi144 ceph-mon[32129]: pgmap v7364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:01 smithi160 ceph-mon[36715]: pgmap v7365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12964.828498 secs 2024-07-26T14:42:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:01 smithi183 ceph-mon[37273]: pgmap v7365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12964.828498 secs 2024-07-26T14:42:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:01 smithi144 ceph-mon[32129]: pgmap v7365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12964.828498 secs 2024-07-26T14:42:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:03 smithi160 ceph-mon[36715]: pgmap v7366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:03 smithi183 ceph-mon[37273]: pgmap v7366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:03 smithi144 ceph-mon[32129]: pgmap v7366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:05 smithi160 ceph-mon[36715]: pgmap v7367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:05 smithi183 ceph-mon[37273]: pgmap v7367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:05 smithi144 ceph-mon[32129]: pgmap v7367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12969.828612 secs 2024-07-26T14:42:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12969.828612 secs 2024-07-26T14:42:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12969.828612 secs 2024-07-26T14:42:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:07 smithi183 ceph-mon[37273]: pgmap v7368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:07 smithi144 ceph-mon[32129]: pgmap v7368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:07 smithi160 ceph-mon[36715]: pgmap v7368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:08.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:42:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:08.394+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:42:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:08.394+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:42:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:08.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:42:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:08.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:09 smithi183 ceph-mon[37273]: pgmap v7369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:09 smithi144 ceph-mon[32129]: pgmap v7369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:09 smithi160 ceph-mon[36715]: pgmap v7369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:11 smithi183 ceph-mon[37273]: pgmap v7370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12974.828742 secs 2024-07-26T14:42:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:11 smithi144 ceph-mon[32129]: pgmap v7370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12974.828742 secs 2024-07-26T14:42:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:11 smithi160 ceph-mon[36715]: pgmap v7370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12974.828742 secs 2024-07-26T14:42:14.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:13 smithi160 ceph-mon[36715]: pgmap v7371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:13 smithi183 ceph-mon[37273]: pgmap v7371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:13 smithi144 ceph-mon[32129]: pgmap v7371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:15 smithi183 ceph-mon[37273]: pgmap v7372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:15 smithi144 ceph-mon[32129]: pgmap v7372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:15 smithi160 ceph-mon[36715]: pgmap v7372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12979.828865 secs 2024-07-26T14:42:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12979.828865 secs 2024-07-26T14:42:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12979.828865 secs 2024-07-26T14:42:18.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:17 smithi144 ceph-mon[32129]: pgmap v7373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:17 smithi183 ceph-mon[37273]: pgmap v7373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:17 smithi160 ceph-mon[36715]: pgmap v7373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:19 smithi183 ceph-mon[37273]: pgmap v7374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:19 smithi144 ceph-mon[32129]: pgmap v7374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:19 smithi160 ceph-mon[36715]: pgmap v7374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:21 smithi183 ceph-mon[37273]: pgmap v7375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12984.829009 secs 2024-07-26T14:42:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:21 smithi144 ceph-mon[32129]: pgmap v7375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12984.829009 secs 2024-07-26T14:42:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:21 smithi160 ceph-mon[36715]: pgmap v7375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12984.829009 secs 2024-07-26T14:42:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:23.409+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:42:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:23.410+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:42:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:23.410+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:42:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:23.410+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:42:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:23.410+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: pgmap v7376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:42:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: pgmap v7376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:42:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: pgmap v7376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:42:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:42:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:24 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:42:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:24 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:42:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:24 smithi144 ceph-mon[32129]: pgmap v7377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:42:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:24 smithi144 ceph-mon[32129]: pgmap v7378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:42:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:24 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:42:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:24 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:42:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:24 smithi160 ceph-mon[36715]: pgmap v7377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:42:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:24 smithi160 ceph-mon[36715]: pgmap v7378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:42:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:24 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:42:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:24 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:42:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:24 smithi183 ceph-mon[37273]: pgmap v7377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:42:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:24 smithi183 ceph-mon[37273]: pgmap v7378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:42:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:26 smithi144 ceph-mon[32129]: pgmap v7379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12989.829140 secs 2024-07-26T14:42:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:26 smithi160 ceph-mon[36715]: pgmap v7379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12989.829140 secs 2024-07-26T14:42:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:26 smithi183 ceph-mon[37273]: pgmap v7379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12989.829140 secs 2024-07-26T14:42:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:28 smithi144 ceph-mon[32129]: pgmap v7380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:42:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:28 smithi160 ceph-mon[36715]: pgmap v7380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:42:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:28 smithi183 ceph-mon[37273]: pgmap v7380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:42:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:30 smithi144 ceph-mon[32129]: pgmap v7381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:42:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:30 smithi160 ceph-mon[36715]: pgmap v7381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:42:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:30 smithi183 ceph-mon[37273]: pgmap v7381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:42:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12994.829271 secs 2024-07-26T14:42:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12994.829271 secs 2024-07-26T14:42:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12994.829271 secs 2024-07-26T14:42:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:32 smithi144 ceph-mon[32129]: pgmap v7382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:32 smithi160 ceph-mon[36715]: pgmap v7382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:32 smithi183 ceph-mon[37273]: pgmap v7382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:34 smithi144 ceph-mon[32129]: pgmap v7383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:42:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:34 smithi160 ceph-mon[36715]: pgmap v7383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:42:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:34 smithi183 ceph-mon[37273]: pgmap v7383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:42:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:36 smithi144 ceph-mon[32129]: pgmap v7384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:42:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 12999.829389 secs 2024-07-26T14:42:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:36 smithi160 ceph-mon[36715]: pgmap v7384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:42:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 12999.829389 secs 2024-07-26T14:42:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:36 smithi183 ceph-mon[37273]: pgmap v7384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:42:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 12999.829389 secs 2024-07-26T14:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:38.412+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:38.412+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:38.412+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:38.412+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:38.413+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:38 smithi144 ceph-mon[32129]: pgmap v7385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:38 smithi160 ceph-mon[36715]: pgmap v7385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:38 smithi183 ceph-mon[37273]: pgmap v7385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:40 smithi144 ceph-mon[32129]: pgmap v7386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:40 smithi160 ceph-mon[36715]: pgmap v7386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:40 smithi183 ceph-mon[37273]: pgmap v7386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13004.829528 secs 2024-07-26T14:42:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13004.829528 secs 2024-07-26T14:42:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13004.829528 secs 2024-07-26T14:42:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:42 smithi144 ceph-mon[32129]: pgmap v7387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:42 smithi160 ceph-mon[36715]: pgmap v7387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:42 smithi183 ceph-mon[37273]: pgmap v7387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:44 smithi144 ceph-mon[32129]: pgmap v7388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:44 smithi160 ceph-mon[36715]: pgmap v7388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:44 smithi183 ceph-mon[37273]: pgmap v7388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:46 smithi160 ceph-mon[36715]: pgmap v7389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:46 smithi183 ceph-mon[37273]: pgmap v7389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:46 smithi144 ceph-mon[32129]: pgmap v7389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13009.829603 secs 2024-07-26T14:42:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13009.829603 secs 2024-07-26T14:42:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13009.829603 secs 2024-07-26T14:42:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:48 smithi160 ceph-mon[36715]: pgmap v7390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:48.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:48 smithi144 ceph-mon[32129]: pgmap v7390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:48 smithi183 ceph-mon[37273]: pgmap v7390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:50 smithi183 ceph-mon[37273]: pgmap v7391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:50 smithi144 ceph-mon[32129]: pgmap v7391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:50 smithi160 ceph-mon[36715]: pgmap v7391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13014.829725 secs 2024-07-26T14:42:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13014.829725 secs 2024-07-26T14:42:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13014.829725 secs 2024-07-26T14:42:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:52 smithi144 ceph-mon[32129]: pgmap v7392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:53.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:52 smithi183 ceph-mon[37273]: pgmap v7392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:52 smithi160 ceph-mon[36715]: pgmap v7392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:42:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:53.428+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:42:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:53.429+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:42:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:53.429+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:42:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:53.429+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:42:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:42:53.429+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:54 smithi144 ceph-mon[32129]: pgmap v7393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:54 smithi160 ceph-mon[36715]: pgmap v7393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:54 smithi183 ceph-mon[37273]: pgmap v7393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:56 smithi144 ceph-mon[32129]: pgmap v7394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13019.829823 secs 2024-07-26T14:42:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:56 smithi160 ceph-mon[36715]: pgmap v7394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13019.829823 secs 2024-07-26T14:42:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:56 smithi183 ceph-mon[37273]: pgmap v7394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:42:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13019.829823 secs 2024-07-26T14:42:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:42:58 smithi144 ceph-mon[32129]: pgmap v7395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:42:58 smithi160 ceph-mon[36715]: pgmap v7395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:42:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:42:58 smithi183 ceph-mon[37273]: pgmap v7395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:00 smithi144 ceph-mon[32129]: pgmap v7396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:00 smithi160 ceph-mon[36715]: pgmap v7396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:00 smithi183 ceph-mon[37273]: pgmap v7396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13024.829924 secs 2024-07-26T14:43:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13024.829924 secs 2024-07-26T14:43:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13024.829924 secs 2024-07-26T14:43:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:02 smithi144 ceph-mon[32129]: pgmap v7397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:02 smithi160 ceph-mon[36715]: pgmap v7397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:02 smithi183 ceph-mon[37273]: pgmap v7397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:04 smithi144 ceph-mon[32129]: pgmap v7398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:04 smithi160 ceph-mon[36715]: pgmap v7398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:04 smithi183 ceph-mon[37273]: pgmap v7398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:06 smithi144 ceph-mon[32129]: pgmap v7399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13029.829990 secs 2024-07-26T14:43:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:06 smithi160 ceph-mon[36715]: pgmap v7399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13029.829990 secs 2024-07-26T14:43:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:06 smithi183 ceph-mon[37273]: pgmap v7399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13029.829990 secs 2024-07-26T14:43:08.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:08.445+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:43:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:08.447+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:43:08.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:08.447+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:43:08.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:08.447+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:43:08.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:08.447+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:08 smithi144 ceph-mon[32129]: pgmap v7400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:08 smithi160 ceph-mon[36715]: pgmap v7400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:08 smithi183 ceph-mon[37273]: pgmap v7400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:10 smithi144 ceph-mon[32129]: pgmap v7401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:10 smithi160 ceph-mon[36715]: pgmap v7401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:10 smithi183 ceph-mon[37273]: pgmap v7401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13034.830133 secs 2024-07-26T14:43:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13034.830133 secs 2024-07-26T14:43:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13034.830133 secs 2024-07-26T14:43:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:12 smithi144 ceph-mon[32129]: pgmap v7402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:12 smithi160 ceph-mon[36715]: pgmap v7402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:12 smithi183 ceph-mon[37273]: pgmap v7402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:14 smithi160 ceph-mon[36715]: pgmap v7403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:14 smithi183 ceph-mon[37273]: pgmap v7403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:14 smithi144 ceph-mon[32129]: pgmap v7403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:16 smithi144 ceph-mon[32129]: pgmap v7404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13039.830242 secs 2024-07-26T14:43:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:16 smithi160 ceph-mon[36715]: pgmap v7404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13039.830242 secs 2024-07-26T14:43:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:16 smithi183 ceph-mon[37273]: pgmap v7404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13039.830242 secs 2024-07-26T14:43:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:18 smithi144 ceph-mon[32129]: pgmap v7405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:18 smithi160 ceph-mon[36715]: pgmap v7405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:18 smithi183 ceph-mon[37273]: pgmap v7405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:20 smithi144 ceph-mon[32129]: pgmap v7406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:20 smithi160 ceph-mon[36715]: pgmap v7406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:20 smithi183 ceph-mon[37273]: pgmap v7406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13044.830360 secs 2024-07-26T14:43:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13044.830360 secs 2024-07-26T14:43:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13044.830360 secs 2024-07-26T14:43:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:22 smithi144 ceph-mon[32129]: pgmap v7407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:23.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:22 smithi183 ceph-mon[37273]: pgmap v7407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:22 smithi160 ceph-mon[36715]: pgmap v7407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:23.463+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:43:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:23.464+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:43:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:23.464+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:43:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:23.464+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:43:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:23.464+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: pgmap v7408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: pgmap v7408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: pgmap v7408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:25.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:43:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:43:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:43:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:43:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:43:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:43:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:43:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:43:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:43:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:43:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:43:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:43:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:43:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:43:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:43:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:43:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:26 smithi144 ceph-mon[32129]: pgmap v7409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:43:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:26 smithi144 ceph-mon[32129]: pgmap v7410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:43:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13049.830441 secs 2024-07-26T14:43:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:26 smithi160 ceph-mon[36715]: pgmap v7409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:43:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:26 smithi160 ceph-mon[36715]: pgmap v7410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:43:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13049.830441 secs 2024-07-26T14:43:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:26 smithi183 ceph-mon[37273]: pgmap v7409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:43:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:26 smithi183 ceph-mon[37273]: pgmap v7410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:43:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13049.830441 secs 2024-07-26T14:43:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:28 smithi144 ceph-mon[32129]: pgmap v7411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:43:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:28 smithi160 ceph-mon[36715]: pgmap v7411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:43:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:28 smithi183 ceph-mon[37273]: pgmap v7411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:43:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:30 smithi144 ceph-mon[32129]: pgmap v7412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:43:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:30 smithi160 ceph-mon[36715]: pgmap v7412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:43:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:30 smithi183 ceph-mon[37273]: pgmap v7412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:43:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13054.830563 secs 2024-07-26T14:43:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13054.830563 secs 2024-07-26T14:43:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13054.830563 secs 2024-07-26T14:43:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:32 smithi144 ceph-mon[32129]: pgmap v7413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:43:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:32 smithi160 ceph-mon[36715]: pgmap v7413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:43:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:32 smithi183 ceph-mon[37273]: pgmap v7413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:43:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:34 smithi160 ceph-mon[36715]: pgmap v7414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:43:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:34 smithi183 ceph-mon[37273]: pgmap v7414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:43:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:34 smithi144 ceph-mon[32129]: pgmap v7414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:43:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:36 smithi160 ceph-mon[36715]: pgmap v7415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:43:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:36 smithi183 ceph-mon[37273]: pgmap v7415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:43:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:36 smithi144 ceph-mon[32129]: pgmap v7415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:43:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13059.830627 secs 2024-07-26T14:43:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13059.830627 secs 2024-07-26T14:43:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13059.830627 secs 2024-07-26T14:43:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:38 smithi183 ceph-mon[37273]: pgmap v7416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:38 smithi144 ceph-mon[32129]: pgmap v7416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:38.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:38.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:43:38.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:38.481+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:43:38.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:38.481+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:43:38.482 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:38.481+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:43:38.482 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:38.481+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:38 smithi160 ceph-mon[36715]: pgmap v7416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:40 smithi183 ceph-mon[37273]: pgmap v7417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:40 smithi144 ceph-mon[32129]: pgmap v7417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:40 smithi160 ceph-mon[36715]: pgmap v7417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13064.830767 secs 2024-07-26T14:43:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:42 smithi183 ceph-mon[37273]: pgmap v7418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13064.830767 secs 2024-07-26T14:43:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:42 smithi144 ceph-mon[32129]: pgmap v7418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13064.830767 secs 2024-07-26T14:43:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:42 smithi160 ceph-mon[36715]: pgmap v7418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:44 smithi183 ceph-mon[37273]: pgmap v7419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:44 smithi144 ceph-mon[32129]: pgmap v7419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:44 smithi160 ceph-mon[36715]: pgmap v7419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:46 smithi183 ceph-mon[37273]: pgmap v7420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:46 smithi144 ceph-mon[32129]: pgmap v7420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:46 smithi160 ceph-mon[36715]: pgmap v7420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13069.830883 secs 2024-07-26T14:43:47.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13069.830883 secs 2024-07-26T14:43:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13069.830883 secs 2024-07-26T14:43:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:48 smithi183 ceph-mon[37273]: pgmap v7421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:48 smithi144 ceph-mon[32129]: pgmap v7421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:48 smithi160 ceph-mon[36715]: pgmap v7421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:50 smithi183 ceph-mon[37273]: pgmap v7422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:50 smithi144 ceph-mon[32129]: pgmap v7422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:50 smithi160 ceph-mon[36715]: pgmap v7422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13074.831021 secs 2024-07-26T14:43:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:52 smithi183 ceph-mon[37273]: pgmap v7423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13074.831021 secs 2024-07-26T14:43:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:52 smithi144 ceph-mon[32129]: pgmap v7423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13074.831021 secs 2024-07-26T14:43:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:52 smithi160 ceph-mon[36715]: pgmap v7423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:53.490+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:43:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:53.491+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:43:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:53.492+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:43:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:53.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:43:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:43:53.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:54 smithi144 ceph-mon[32129]: pgmap v7424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:54 smithi160 ceph-mon[36715]: pgmap v7424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:54 smithi183 ceph-mon[37273]: pgmap v7424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:43:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:56 smithi144 ceph-mon[32129]: pgmap v7425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:56 smithi160 ceph-mon[36715]: pgmap v7425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:56 smithi183 ceph-mon[37273]: pgmap v7425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:43:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13079.831158 secs 2024-07-26T14:43:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13079.831158 secs 2024-07-26T14:43:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13079.831158 secs 2024-07-26T14:43:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:43:58 smithi144 ceph-mon[32129]: pgmap v7426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:43:58 smithi160 ceph-mon[36715]: pgmap v7426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:43:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:43:58 smithi183 ceph-mon[37273]: pgmap v7426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:00 smithi144 ceph-mon[32129]: pgmap v7427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:00 smithi160 ceph-mon[36715]: pgmap v7427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:00 smithi183 ceph-mon[37273]: pgmap v7427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13084.831310 secs 2024-07-26T14:44:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:02 smithi144 ceph-mon[32129]: pgmap v7428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13084.831310 secs 2024-07-26T14:44:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:02 smithi160 ceph-mon[36715]: pgmap v7428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13084.831310 secs 2024-07-26T14:44:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:02 smithi183 ceph-mon[37273]: pgmap v7428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:04 smithi144 ceph-mon[32129]: pgmap v7429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:04 smithi160 ceph-mon[36715]: pgmap v7429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:04 smithi183 ceph-mon[37273]: pgmap v7429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:06 smithi144 ceph-mon[32129]: pgmap v7430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:06 smithi160 ceph-mon[36715]: pgmap v7430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:06 smithi183 ceph-mon[37273]: pgmap v7430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13089.831375 secs 2024-07-26T14:44:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13089.831375 secs 2024-07-26T14:44:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13089.831375 secs 2024-07-26T14:44:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:08 smithi144 ceph-mon[32129]: pgmap v7431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:08.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:44:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:08.508+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:44:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:08.508+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:44:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:08.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:44:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:08.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:08 smithi160 ceph-mon[36715]: pgmap v7431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:08 smithi183 ceph-mon[37273]: pgmap v7431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:10 smithi144 ceph-mon[32129]: pgmap v7432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:10 smithi160 ceph-mon[36715]: pgmap v7432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:10 smithi183 ceph-mon[37273]: pgmap v7432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13094.831505 secs 2024-07-26T14:44:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:12 smithi144 ceph-mon[32129]: pgmap v7433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13094.831505 secs 2024-07-26T14:44:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:12 smithi160 ceph-mon[36715]: pgmap v7433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13094.831505 secs 2024-07-26T14:44:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:12 smithi183 ceph-mon[37273]: pgmap v7433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:14 smithi144 ceph-mon[32129]: pgmap v7434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:14 smithi160 ceph-mon[36715]: pgmap v7434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:14 smithi183 ceph-mon[37273]: pgmap v7434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:16 smithi144 ceph-mon[32129]: pgmap v7435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:16 smithi160 ceph-mon[36715]: pgmap v7435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:16 smithi183 ceph-mon[37273]: pgmap v7435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13099.831624 secs 2024-07-26T14:44:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13099.831624 secs 2024-07-26T14:44:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13099.831624 secs 2024-07-26T14:44:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:18 smithi160 ceph-mon[36715]: pgmap v7436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:18 smithi183 ceph-mon[37273]: pgmap v7436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:18 smithi144 ceph-mon[32129]: pgmap v7436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:20.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:20 smithi144 ceph-mon[32129]: pgmap v7437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:20 smithi160 ceph-mon[36715]: pgmap v7437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:20 smithi183 ceph-mon[37273]: pgmap v7437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13104.831742 secs 2024-07-26T14:44:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:22 smithi160 ceph-mon[36715]: pgmap v7438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13104.831742 secs 2024-07-26T14:44:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:22 smithi183 ceph-mon[37273]: pgmap v7438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13104.831742 secs 2024-07-26T14:44:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:22 smithi144 ceph-mon[32129]: pgmap v7438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:23.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:44:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:23.525+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:44:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:23.525+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:44:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:23.525+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:44:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:23.526+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:24.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:24 smithi160 ceph-mon[36715]: pgmap v7439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:24 smithi183 ceph-mon[37273]: pgmap v7439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:24 smithi144 ceph-mon[32129]: pgmap v7439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:25.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:26 smithi160 ceph-mon[36715]: pgmap v7440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:26 smithi183 ceph-mon[37273]: pgmap v7440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:44:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:26 smithi144 ceph-mon[32129]: pgmap v7440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:44:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13109.831858 secs 2024-07-26T14:44:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:27 smithi160 ceph-mon[36715]: pgmap v7441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:27 smithi160 ceph-mon[36715]: pgmap v7442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13109.831858 secs 2024-07-26T14:44:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:27 smithi183 ceph-mon[37273]: pgmap v7441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:27 smithi183 ceph-mon[37273]: pgmap v7442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13109.831858 secs 2024-07-26T14:44:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:27 smithi144 ceph-mon[32129]: pgmap v7441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:27 smithi144 ceph-mon[32129]: pgmap v7442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:29 smithi160 ceph-mon[36715]: pgmap v7443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:29 smithi183 ceph-mon[37273]: pgmap v7443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:29 smithi144 ceph-mon[32129]: pgmap v7443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:31 smithi160 ceph-mon[36715]: pgmap v7444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:31 smithi183 ceph-mon[37273]: pgmap v7444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:31 smithi144 ceph-mon[32129]: pgmap v7444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:32.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13114.831996 secs 2024-07-26T14:44:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13114.831996 secs 2024-07-26T14:44:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13114.831996 secs 2024-07-26T14:44:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:33 smithi160 ceph-mon[36715]: pgmap v7445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:33.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:33 smithi144 ceph-mon[32129]: pgmap v7445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:33 smithi183 ceph-mon[37273]: pgmap v7445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:35 smithi160 ceph-mon[36715]: pgmap v7446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:35 smithi183 ceph-mon[37273]: pgmap v7446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:35 smithi144 ceph-mon[32129]: pgmap v7446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:44:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13119.832135 secs 2024-07-26T14:44:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:37 smithi160 ceph-mon[36715]: pgmap v7447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:44:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13119.832135 secs 2024-07-26T14:44:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:37 smithi183 ceph-mon[37273]: pgmap v7447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:44:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13119.832135 secs 2024-07-26T14:44:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:37 smithi144 ceph-mon[32129]: pgmap v7447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:44:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:38.541+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:44:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:38.542+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:44:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:38.542+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:44:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:38.542+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:44:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:38.543+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:39 smithi183 ceph-mon[37273]: pgmap v7448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:39 smithi144 ceph-mon[32129]: pgmap v7448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:39 smithi160 ceph-mon[36715]: pgmap v7448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:41 smithi183 ceph-mon[37273]: pgmap v7449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:41 smithi144 ceph-mon[32129]: pgmap v7449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:41 smithi160 ceph-mon[36715]: pgmap v7449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13124.832273 secs 2024-07-26T14:44:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13124.832273 secs 2024-07-26T14:44:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13124.832273 secs 2024-07-26T14:44:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:43 smithi183 ceph-mon[37273]: pgmap v7450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:43 smithi144 ceph-mon[32129]: pgmap v7450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:43 smithi160 ceph-mon[36715]: pgmap v7450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:45 smithi183 ceph-mon[37273]: pgmap v7451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:45 smithi144 ceph-mon[32129]: pgmap v7451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:45 smithi160 ceph-mon[36715]: pgmap v7451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:47.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13129.832399 secs 2024-07-26T14:44:47.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:47 smithi160 ceph-mon[36715]: pgmap v7452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13129.832399 secs 2024-07-26T14:44:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:47 smithi183 ceph-mon[37273]: pgmap v7452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13129.832399 secs 2024-07-26T14:44:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:47 smithi144 ceph-mon[32129]: pgmap v7452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:48.421+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:44:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:48.421+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:44:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:48.422+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:44:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:48.422+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:44:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:49 smithi144 ceph-mon[32129]: pgmap v7453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:49 smithi183 ceph-mon[37273]: pgmap v7453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:49 smithi160 ceph-mon[36715]: pgmap v7453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:51 smithi183 ceph-mon[37273]: pgmap v7454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:51 smithi144 ceph-mon[32129]: pgmap v7454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:51 smithi160 ceph-mon[36715]: pgmap v7454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13134.832525 secs 2024-07-26T14:44:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13134.832525 secs 2024-07-26T14:44:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13134.832525 secs 2024-07-26T14:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:53.558+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:53.559+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:53.560+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:53.560+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:44:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:44:53.560+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:53 smithi183 ceph-mon[37273]: pgmap v7455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:53 smithi144 ceph-mon[32129]: pgmap v7455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:53 smithi160 ceph-mon[36715]: pgmap v7455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:44:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:55 smithi183 ceph-mon[37273]: pgmap v7456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:55 smithi144 ceph-mon[32129]: pgmap v7456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:55 smithi160 ceph-mon[36715]: pgmap v7456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13139.832647 secs 2024-07-26T14:44:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:57 smithi183 ceph-mon[37273]: pgmap v7457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13139.832647 secs 2024-07-26T14:44:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:57 smithi144 ceph-mon[32129]: pgmap v7457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13139.832647 secs 2024-07-26T14:44:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:57 smithi160 ceph-mon[36715]: pgmap v7457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:44:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:44:59 smithi144 ceph-mon[32129]: pgmap v7458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:44:59 smithi160 ceph-mon[36715]: pgmap v7458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:44:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:44:59 smithi183 ceph-mon[37273]: pgmap v7458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:01 smithi144 ceph-mon[32129]: pgmap v7459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:01.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:01 smithi160 ceph-mon[36715]: pgmap v7459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:01 smithi183 ceph-mon[37273]: pgmap v7459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13144.832781 secs 2024-07-26T14:45:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13144.832781 secs 2024-07-26T14:45:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13144.832781 secs 2024-07-26T14:45:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:03 smithi144 ceph-mon[32129]: pgmap v7460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:03 smithi160 ceph-mon[36715]: pgmap v7460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:03 smithi183 ceph-mon[37273]: pgmap v7460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:05 smithi144 ceph-mon[32129]: pgmap v7461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:05 smithi160 ceph-mon[36715]: pgmap v7461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:05 smithi183 ceph-mon[37273]: pgmap v7461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13149.832902 secs 2024-07-26T14:45:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:07 smithi144 ceph-mon[32129]: pgmap v7462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13149.832902 secs 2024-07-26T14:45:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:07 smithi160 ceph-mon[36715]: pgmap v7462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13149.832902 secs 2024-07-26T14:45:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:07 smithi183 ceph-mon[37273]: pgmap v7462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:08.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:08.575+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:45:08.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:08.575+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:45:08.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:08.576+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:45:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:08.576+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:45:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:08.576+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:09 smithi144 ceph-mon[32129]: pgmap v7463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:09 smithi160 ceph-mon[36715]: pgmap v7463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:09 smithi183 ceph-mon[37273]: pgmap v7463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:11 smithi144 ceph-mon[32129]: pgmap v7464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:11 smithi160 ceph-mon[36715]: pgmap v7464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:11 smithi183 ceph-mon[37273]: pgmap v7464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13154.833001 secs 2024-07-26T14:45:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13154.833001 secs 2024-07-26T14:45:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13154.833001 secs 2024-07-26T14:45:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:13 smithi144 ceph-mon[32129]: pgmap v7465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:13 smithi160 ceph-mon[36715]: pgmap v7465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:13 smithi183 ceph-mon[37273]: pgmap v7465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:15 smithi144 ceph-mon[32129]: pgmap v7466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:15 smithi160 ceph-mon[36715]: pgmap v7466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:15 smithi183 ceph-mon[37273]: pgmap v7466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13159.833150 secs 2024-07-26T14:45:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:17 smithi144 ceph-mon[32129]: pgmap v7467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13159.833150 secs 2024-07-26T14:45:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:17 smithi160 ceph-mon[36715]: pgmap v7467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13159.833150 secs 2024-07-26T14:45:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:17 smithi183 ceph-mon[37273]: pgmap v7467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:19 smithi160 ceph-mon[36715]: pgmap v7468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:19 smithi183 ceph-mon[37273]: pgmap v7468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:19.901 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:19 smithi144 ceph-mon[32129]: pgmap v7468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:21 smithi160 ceph-mon[36715]: pgmap v7469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:21 smithi183 ceph-mon[37273]: pgmap v7469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:21 smithi144 ceph-mon[32129]: pgmap v7469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13164.833298 secs 2024-07-26T14:45:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13164.833298 secs 2024-07-26T14:45:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13164.833298 secs 2024-07-26T14:45:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:23.592+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:45:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:23.593+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:45:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:23.593+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:45:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:23.593+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:45:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:23.593+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:23 smithi160 ceph-mon[36715]: pgmap v7470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:23 smithi183 ceph-mon[37273]: pgmap v7470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:23 smithi144 ceph-mon[32129]: pgmap v7470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:25 smithi160 ceph-mon[36715]: pgmap v7471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:25 smithi183 ceph-mon[37273]: pgmap v7471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:25 smithi144 ceph-mon[32129]: pgmap v7471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13169.833429 secs 2024-07-26T14:45:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:27 smithi160 ceph-mon[36715]: pgmap v7472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13169.833429 secs 2024-07-26T14:45:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:27 smithi183 ceph-mon[37273]: pgmap v7472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13169.833429 secs 2024-07-26T14:45:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:27 smithi144 ceph-mon[32129]: pgmap v7472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:28.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:45:28.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:45:28.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:45:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:45:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:45:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:45:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:45:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:29 smithi160 ceph-mon[36715]: pgmap v7473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:29 smithi160 ceph-mon[36715]: pgmap v7474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:29 smithi183 ceph-mon[37273]: pgmap v7473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:29 smithi183 ceph-mon[37273]: pgmap v7474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:29 smithi144 ceph-mon[32129]: pgmap v7473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:29 smithi144 ceph-mon[32129]: pgmap v7474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:31 smithi160 ceph-mon[36715]: pgmap v7475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:45:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:31 smithi183 ceph-mon[37273]: pgmap v7475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:45:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:31 smithi144 ceph-mon[32129]: pgmap v7475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T14:45:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13174.833546 secs 2024-07-26T14:45:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13174.833546 secs 2024-07-26T14:45:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13174.833546 secs 2024-07-26T14:45:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:33 smithi160 ceph-mon[36715]: pgmap v7476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:33 smithi183 ceph-mon[37273]: pgmap v7476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:33 smithi144 ceph-mon[32129]: pgmap v7476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:35 smithi160 ceph-mon[36715]: pgmap v7477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:35 smithi183 ceph-mon[37273]: pgmap v7477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:35 smithi144 ceph-mon[32129]: pgmap v7477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13179.833666 secs 2024-07-26T14:45:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:37 smithi160 ceph-mon[36715]: pgmap v7478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13179.833666 secs 2024-07-26T14:45:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:37 smithi183 ceph-mon[37273]: pgmap v7478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13179.833666 secs 2024-07-26T14:45:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:37 smithi144 ceph-mon[32129]: pgmap v7478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:38.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:45:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:38.601+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:45:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:38.601+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:45:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:38.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:45:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:38.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:39 smithi183 ceph-mon[37273]: pgmap v7479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:39 smithi144 ceph-mon[32129]: pgmap v7479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:39 smithi160 ceph-mon[36715]: pgmap v7479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T14:45:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:41 smithi183 ceph-mon[37273]: pgmap v7480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:45:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:41 smithi144 ceph-mon[32129]: pgmap v7480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:45:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:41 smithi160 ceph-mon[36715]: pgmap v7480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:45:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13184.833740 secs 2024-07-26T14:45:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13184.833740 secs 2024-07-26T14:45:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13184.833740 secs 2024-07-26T14:45:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:43 smithi183 ceph-mon[37273]: pgmap v7481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:45:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:43 smithi144 ceph-mon[32129]: pgmap v7481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:45:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:43 smithi160 ceph-mon[36715]: pgmap v7481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:45 smithi183 ceph-mon[37273]: pgmap v7482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:45 smithi144 ceph-mon[32129]: pgmap v7482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:45 smithi160 ceph-mon[36715]: pgmap v7482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:45:47.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13189.833894 secs 2024-07-26T14:45:47.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:47 smithi160 ceph-mon[36715]: pgmap v7483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13189.833894 secs 2024-07-26T14:45:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:47 smithi183 ceph-mon[37273]: pgmap v7483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13189.833894 secs 2024-07-26T14:45:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:47 smithi144 ceph-mon[32129]: pgmap v7483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:49 smithi183 ceph-mon[37273]: pgmap v7484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:49 smithi144 ceph-mon[32129]: pgmap v7484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:49 smithi160 ceph-mon[36715]: pgmap v7484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:51 smithi183 ceph-mon[37273]: pgmap v7485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:51 smithi144 ceph-mon[32129]: pgmap v7485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:51 smithi160 ceph-mon[36715]: pgmap v7485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:45:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13194.834014 secs 2024-07-26T14:45:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13194.834014 secs 2024-07-26T14:45:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13194.834014 secs 2024-07-26T14:45:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:53.613+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:45:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:53.615+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:45:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:53.615+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:45:53.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:53.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:45:53.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:45:53.615+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:53 smithi183 ceph-mon[37273]: pgmap v7486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:53 smithi144 ceph-mon[32129]: pgmap v7486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:53 smithi160 ceph-mon[36715]: pgmap v7486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:55 smithi183 ceph-mon[37273]: pgmap v7487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:55 smithi144 ceph-mon[32129]: pgmap v7487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:55 smithi160 ceph-mon[36715]: pgmap v7487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:45:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13199.834134 secs 2024-07-26T14:45:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13199.834134 secs 2024-07-26T14:45:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13199.834134 secs 2024-07-26T14:45:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:57 smithi144 ceph-mon[32129]: pgmap v7488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:57 smithi160 ceph-mon[36715]: pgmap v7488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:57 smithi183 ceph-mon[37273]: pgmap v7488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:45:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:45:59 smithi144 ceph-mon[32129]: pgmap v7489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:45:59 smithi160 ceph-mon[36715]: pgmap v7489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:45:59 smithi183 ceph-mon[37273]: pgmap v7489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:01 smithi144 ceph-mon[32129]: pgmap v7490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13204.834273 secs 2024-07-26T14:46:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:01 smithi160 ceph-mon[36715]: pgmap v7490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:02.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13204.834273 secs 2024-07-26T14:46:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:01 smithi183 ceph-mon[37273]: pgmap v7490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13204.834273 secs 2024-07-26T14:46:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:03 smithi144 ceph-mon[32129]: pgmap v7491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:03 smithi160 ceph-mon[36715]: pgmap v7491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:03 smithi183 ceph-mon[37273]: pgmap v7491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:05 smithi144 ceph-mon[32129]: pgmap v7492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:05 smithi160 ceph-mon[36715]: pgmap v7492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:05 smithi183 ceph-mon[37273]: pgmap v7492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13209.834389 secs 2024-07-26T14:46:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13209.834389 secs 2024-07-26T14:46:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13209.834389 secs 2024-07-26T14:46:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:07 smithi144 ceph-mon[32129]: pgmap v7493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:07 smithi160 ceph-mon[36715]: pgmap v7493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:07 smithi183 ceph-mon[37273]: pgmap v7493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:08.618+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:46:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:08.619+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:46:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:08.619+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:46:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:08.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:46:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:08.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:09 smithi144 ceph-mon[32129]: pgmap v7494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:09 smithi160 ceph-mon[36715]: pgmap v7494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:09 smithi183 ceph-mon[37273]: pgmap v7494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:11 smithi144 ceph-mon[32129]: pgmap v7495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13214.834512 secs 2024-07-26T14:46:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:11 smithi160 ceph-mon[36715]: pgmap v7495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13214.834512 secs 2024-07-26T14:46:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:11 smithi183 ceph-mon[37273]: pgmap v7495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13214.834512 secs 2024-07-26T14:46:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:13 smithi144 ceph-mon[32129]: pgmap v7496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:13 smithi160 ceph-mon[36715]: pgmap v7496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:13 smithi183 ceph-mon[37273]: pgmap v7496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:15 smithi144 ceph-mon[32129]: pgmap v7497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:15 smithi160 ceph-mon[36715]: pgmap v7497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:15 smithi183 ceph-mon[37273]: pgmap v7497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13219.834627 secs 2024-07-26T14:46:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13219.834627 secs 2024-07-26T14:46:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13219.834627 secs 2024-07-26T14:46:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:17 smithi144 ceph-mon[32129]: pgmap v7498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:17 smithi160 ceph-mon[36715]: pgmap v7498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:17 smithi183 ceph-mon[37273]: pgmap v7498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:19 smithi144 ceph-mon[32129]: pgmap v7499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:19 smithi160 ceph-mon[36715]: pgmap v7499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:19 smithi183 ceph-mon[37273]: pgmap v7499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:21 smithi160 ceph-mon[36715]: pgmap v7500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13224.834748 secs 2024-07-26T14:46:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:21 smithi183 ceph-mon[37273]: pgmap v7500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13224.834748 secs 2024-07-26T14:46:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:21 smithi144 ceph-mon[32129]: pgmap v7500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13224.834748 secs 2024-07-26T14:46:23.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:23.635+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:46:23.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:23.636+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:46:23.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:23.636+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:46:23.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:23.636+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:46:23.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:23.636+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:23 smithi160 ceph-mon[36715]: pgmap v7501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:23 smithi183 ceph-mon[37273]: pgmap v7501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:23 smithi144 ceph-mon[32129]: pgmap v7501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:25 smithi160 ceph-mon[36715]: pgmap v7502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:25 smithi183 ceph-mon[37273]: pgmap v7502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:25 smithi144 ceph-mon[32129]: pgmap v7502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13229.834866 secs 2024-07-26T14:46:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13229.834866 secs 2024-07-26T14:46:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13229.834866 secs 2024-07-26T14:46:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:27 smithi160 ceph-mon[36715]: pgmap v7503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:27 smithi183 ceph-mon[37273]: pgmap v7503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:27 smithi144 ceph-mon[32129]: pgmap v7503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:28 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:46:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:28 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:46:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:28 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:46:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:29 smithi160 ceph-mon[36715]: pgmap v7504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:29 smithi183 ceph-mon[37273]: pgmap v7504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:30.218 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:29 smithi144 ceph-mon[32129]: pgmap v7504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:31 smithi144 ceph-mon[32129]: pgmap v7505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:31 smithi160 ceph-mon[36715]: pgmap v7505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:31 smithi183 ceph-mon[37273]: pgmap v7505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13234.835011 secs 2024-07-26T14:46:32.728 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:32.728 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13234.835011 secs 2024-07-26T14:46:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13234.835011 secs 2024-07-26T14:46:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:33 smithi144 ceph-mon[32129]: pgmap v7506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:33 smithi160 ceph-mon[36715]: pgmap v7506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:33 smithi183 ceph-mon[37273]: pgmap v7506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:46:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:34 smithi144 ceph-mon[32129]: pgmap v7507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:46:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:34 smithi144 ceph-mon[32129]: pgmap v7508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T14:46:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:34 smithi160 ceph-mon[36715]: pgmap v7507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:46:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:34 smithi160 ceph-mon[36715]: pgmap v7508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T14:46:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:34 smithi183 ceph-mon[37273]: pgmap v7507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:46:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:34 smithi183 ceph-mon[37273]: pgmap v7508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-07-26T14:46:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:36 smithi144 ceph-mon[32129]: pgmap v7509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:36 smithi160 ceph-mon[36715]: pgmap v7509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:36 smithi183 ceph-mon[37273]: pgmap v7509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13239.835111 secs 2024-07-26T14:46:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13239.835111 secs 2024-07-26T14:46:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13239.835111 secs 2024-07-26T14:46:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:38.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:38.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:38.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:38.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:38.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:38 smithi144 ceph-mon[32129]: pgmap v7510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:38 smithi160 ceph-mon[36715]: pgmap v7510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:38 smithi183 ceph-mon[37273]: pgmap v7510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:40 smithi144 ceph-mon[32129]: pgmap v7511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:40 smithi160 ceph-mon[36715]: pgmap v7511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:40 smithi183 ceph-mon[37273]: pgmap v7511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13244.835237 secs 2024-07-26T14:46:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:42 smithi144 ceph-mon[32129]: pgmap v7512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:46:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13244.835237 secs 2024-07-26T14:46:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:42 smithi160 ceph-mon[36715]: pgmap v7512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:46:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13244.835237 secs 2024-07-26T14:46:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:42 smithi183 ceph-mon[37273]: pgmap v7512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:46:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:44 smithi144 ceph-mon[32129]: pgmap v7513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:46:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:44 smithi160 ceph-mon[36715]: pgmap v7513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:46:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:44 smithi183 ceph-mon[37273]: pgmap v7513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:46:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:46 smithi144 ceph-mon[32129]: pgmap v7514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:46 smithi160 ceph-mon[36715]: pgmap v7514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:46 smithi183 ceph-mon[37273]: pgmap v7514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:47.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13249.835355 secs 2024-07-26T14:46:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13249.835355 secs 2024-07-26T14:46:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13249.835355 secs 2024-07-26T14:46:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:48 smithi144 ceph-mon[32129]: pgmap v7515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:48 smithi160 ceph-mon[36715]: pgmap v7515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:48 smithi183 ceph-mon[37273]: pgmap v7515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:50 smithi144 ceph-mon[32129]: pgmap v7516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:50 smithi160 ceph-mon[36715]: pgmap v7516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:50 smithi183 ceph-mon[37273]: pgmap v7516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13254.835477 secs 2024-07-26T14:46:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:52 smithi160 ceph-mon[36715]: pgmap v7517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13254.835477 secs 2024-07-26T14:46:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:52 smithi183 ceph-mon[37273]: pgmap v7517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13254.835477 secs 2024-07-26T14:46:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:52 smithi144 ceph-mon[32129]: pgmap v7517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:53.669+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:46:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:53.670+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:46:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:53.670+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:46:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:53.670+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:46:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:46:53.670+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:54 smithi160 ceph-mon[36715]: pgmap v7518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:54 smithi183 ceph-mon[37273]: pgmap v7518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:54 smithi144 ceph-mon[32129]: pgmap v7518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:46:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:56 smithi160 ceph-mon[36715]: pgmap v7519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:56 smithi183 ceph-mon[37273]: pgmap v7519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:56 smithi144 ceph-mon[32129]: pgmap v7519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:46:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13259.835597 secs 2024-07-26T14:46:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13259.835597 secs 2024-07-26T14:46:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13259.835597 secs 2024-07-26T14:46:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:46:58 smithi160 ceph-mon[36715]: pgmap v7520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:46:58 smithi183 ceph-mon[37273]: pgmap v7520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:46:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:46:58 smithi144 ceph-mon[32129]: pgmap v7520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:00 smithi160 ceph-mon[36715]: pgmap v7521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:00 smithi183 ceph-mon[37273]: pgmap v7521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:00 smithi144 ceph-mon[32129]: pgmap v7521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13264.835717 secs 2024-07-26T14:47:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:02 smithi160 ceph-mon[36715]: pgmap v7522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13264.835717 secs 2024-07-26T14:47:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:02 smithi183 ceph-mon[37273]: pgmap v7522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13264.835717 secs 2024-07-26T14:47:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:02 smithi144 ceph-mon[32129]: pgmap v7522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:04 smithi160 ceph-mon[36715]: pgmap v7523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:04 smithi183 ceph-mon[37273]: pgmap v7523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:04 smithi144 ceph-mon[32129]: pgmap v7523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:06 smithi160 ceph-mon[36715]: pgmap v7524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:06 smithi183 ceph-mon[37273]: pgmap v7524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:06 smithi144 ceph-mon[32129]: pgmap v7524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13269.835828 secs 2024-07-26T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13269.835828 secs 2024-07-26T14:47:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13269.835828 secs 2024-07-26T14:47:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:08.685+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:47:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:08.686+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:47:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:08.686+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:47:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:08.686+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:47:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:08.686+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:08 smithi160 ceph-mon[36715]: pgmap v7525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:08 smithi183 ceph-mon[37273]: pgmap v7525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:08.957 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:08 smithi144 ceph-mon[32129]: pgmap v7525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:10 smithi160 ceph-mon[36715]: pgmap v7526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:10 smithi183 ceph-mon[37273]: pgmap v7526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:10 smithi144 ceph-mon[32129]: pgmap v7526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13274.835948 secs 2024-07-26T14:47:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:12 smithi160 ceph-mon[36715]: pgmap v7527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13274.835948 secs 2024-07-26T14:47:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:12 smithi183 ceph-mon[37273]: pgmap v7527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13274.835948 secs 2024-07-26T14:47:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:12 smithi144 ceph-mon[32129]: pgmap v7527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:14 smithi160 ceph-mon[36715]: pgmap v7528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:14 smithi183 ceph-mon[37273]: pgmap v7528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:14 smithi144 ceph-mon[32129]: pgmap v7528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:16 smithi160 ceph-mon[36715]: pgmap v7529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:16 smithi183 ceph-mon[37273]: pgmap v7529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:16 smithi144 ceph-mon[32129]: pgmap v7529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13279.836067 secs 2024-07-26T14:47:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13279.836067 secs 2024-07-26T14:47:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13279.836067 secs 2024-07-26T14:47:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:18 smithi183 ceph-mon[37273]: pgmap v7530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:18 smithi144 ceph-mon[32129]: pgmap v7530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:18 smithi160 ceph-mon[36715]: pgmap v7530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:20 smithi183 ceph-mon[37273]: pgmap v7531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:20 smithi144 ceph-mon[32129]: pgmap v7531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:20 smithi160 ceph-mon[36715]: pgmap v7531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13284.836164 secs 2024-07-26T14:47:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:22 smithi183 ceph-mon[37273]: pgmap v7532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13284.836164 secs 2024-07-26T14:47:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:22 smithi144 ceph-mon[32129]: pgmap v7532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13284.836164 secs 2024-07-26T14:47:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:22 smithi160 ceph-mon[36715]: pgmap v7532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:23.703+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:47:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:23.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:47:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:23.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:47:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:23.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:47:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:23.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:24 smithi183 ceph-mon[37273]: pgmap v7533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:24 smithi144 ceph-mon[32129]: pgmap v7533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:24 smithi160 ceph-mon[36715]: pgmap v7533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:26 smithi183 ceph-mon[37273]: pgmap v7534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:26 smithi144 ceph-mon[32129]: pgmap v7534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:26 smithi160 ceph-mon[36715]: pgmap v7534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13289.836283 secs 2024-07-26T14:47:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13289.836283 secs 2024-07-26T14:47:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13289.836283 secs 2024-07-26T14:47:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:28 smithi183 ceph-mon[37273]: pgmap v7535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:28 smithi144 ceph-mon[32129]: pgmap v7535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:28 smithi160 ceph-mon[36715]: pgmap v7535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:30 smithi183 ceph-mon[37273]: pgmap v7536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:30 smithi144 ceph-mon[32129]: pgmap v7536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:30 smithi160 ceph-mon[36715]: pgmap v7536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13294.836406 secs 2024-07-26T14:47:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:32 smithi183 ceph-mon[37273]: pgmap v7537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13294.836406 secs 2024-07-26T14:47:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:32 smithi144 ceph-mon[32129]: pgmap v7537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13294.836406 secs 2024-07-26T14:47:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:32 smithi160 ceph-mon[36715]: pgmap v7537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:34 smithi144 ceph-mon[32129]: pgmap v7538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:47:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:34 smithi160 ceph-mon[36715]: pgmap v7538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:47:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:34 smithi183 ceph-mon[37273]: pgmap v7538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:47:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:35 smithi144 ceph-mon[32129]: pgmap v7539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:47:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:35 smithi144 ceph-mon[32129]: pgmap v7540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:35 smithi160 ceph-mon[36715]: pgmap v7539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:47:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:35 smithi160 ceph-mon[36715]: pgmap v7540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:35 smithi183 ceph-mon[37273]: pgmap v7539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T14:47:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:35 smithi183 ceph-mon[37273]: pgmap v7540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13299.836528 secs 2024-07-26T14:47:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:37 smithi183 ceph-mon[37273]: pgmap v7541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13299.836528 secs 2024-07-26T14:47:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:37 smithi144 ceph-mon[32129]: pgmap v7541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13299.836528 secs 2024-07-26T14:47:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:37 smithi160 ceph-mon[36715]: pgmap v7541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:38.717+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:47:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:38.718+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:47:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:38.718+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:47:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:38.718+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:47:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:38.718+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:39 smithi183 ceph-mon[37273]: pgmap v7542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:47:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:39 smithi144 ceph-mon[32129]: pgmap v7542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:47:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:39 smithi160 ceph-mon[36715]: pgmap v7542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:47:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:41 smithi183 ceph-mon[37273]: pgmap v7543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:41 smithi144 ceph-mon[32129]: pgmap v7543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:41 smithi160 ceph-mon[36715]: pgmap v7543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:47:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13304.836685 secs 2024-07-26T14:47:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13304.836685 secs 2024-07-26T14:47:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13304.836685 secs 2024-07-26T14:47:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:43 smithi183 ceph-mon[37273]: pgmap v7544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:47:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:43 smithi144 ceph-mon[32129]: pgmap v7544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:47:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:43 smithi160 ceph-mon[36715]: pgmap v7544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:47:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:45 smithi183 ceph-mon[37273]: pgmap v7545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:47:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:45 smithi144 ceph-mon[32129]: pgmap v7545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:47:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:45 smithi160 ceph-mon[36715]: pgmap v7545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:47:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13309.836811 secs 2024-07-26T14:47:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13309.836811 secs 2024-07-26T14:47:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13309.836811 secs 2024-07-26T14:47:47.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:47 smithi160 ceph-mon[36715]: pgmap v7546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:47 smithi144 ceph-mon[32129]: pgmap v7546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:47 smithi183 ceph-mon[37273]: pgmap v7546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:48.452+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:47:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:48.452+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:47:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:48.453+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:47:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:48.453+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:47:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:49 smithi144 ceph-mon[32129]: pgmap v7547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:49 smithi160 ceph-mon[36715]: pgmap v7547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:49 smithi183 ceph-mon[37273]: pgmap v7547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:51 smithi144 ceph-mon[32129]: pgmap v7548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13314.836933 secs 2024-07-26T14:47:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:51 smithi160 ceph-mon[36715]: pgmap v7548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13314.836933 secs 2024-07-26T14:47:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:51 smithi183 ceph-mon[37273]: pgmap v7548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13314.836933 secs 2024-07-26T14:47:53.726 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:53.725+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:47:53.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:53.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:47:53.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:53.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:47:53.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:53.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:47:53.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:47:53.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:53 smithi144 ceph-mon[32129]: pgmap v7549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:53 smithi160 ceph-mon[36715]: pgmap v7549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:53 smithi183 ceph-mon[37273]: pgmap v7549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:47:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:55 smithi144 ceph-mon[32129]: pgmap v7550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:55 smithi160 ceph-mon[36715]: pgmap v7550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:55 smithi183 ceph-mon[37273]: pgmap v7550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:47:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13319.837050 secs 2024-07-26T14:47:57.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13319.837050 secs 2024-07-26T14:47:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13319.837050 secs 2024-07-26T14:47:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:57 smithi144 ceph-mon[32129]: pgmap v7551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:57 smithi160 ceph-mon[36715]: pgmap v7551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:57 smithi183 ceph-mon[37273]: pgmap v7551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:47:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:47:59 smithi144 ceph-mon[32129]: pgmap v7552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:47:59 smithi160 ceph-mon[36715]: pgmap v7552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:47:59 smithi183 ceph-mon[37273]: pgmap v7552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:01 smithi160 ceph-mon[36715]: pgmap v7553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13324.837146 secs 2024-07-26T14:48:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:01 smithi183 ceph-mon[37273]: pgmap v7553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13324.837146 secs 2024-07-26T14:48:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:01 smithi144 ceph-mon[32129]: pgmap v7553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13324.837146 secs 2024-07-26T14:48:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:03 smithi160 ceph-mon[36715]: pgmap v7554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:03 smithi183 ceph-mon[37273]: pgmap v7554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:03 smithi144 ceph-mon[32129]: pgmap v7554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:05 smithi160 ceph-mon[36715]: pgmap v7555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:05 smithi183 ceph-mon[37273]: pgmap v7555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:05 smithi144 ceph-mon[32129]: pgmap v7555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13329.837285 secs 2024-07-26T14:48:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13329.837285 secs 2024-07-26T14:48:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13329.837285 secs 2024-07-26T14:48:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:07 smithi160 ceph-mon[36715]: pgmap v7556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:07 smithi183 ceph-mon[37273]: pgmap v7556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:07 smithi144 ceph-mon[32129]: pgmap v7556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:08.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:48:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:08.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:48:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:08.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:48:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:08.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:48:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:08.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:09 smithi160 ceph-mon[36715]: pgmap v7557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:09 smithi183 ceph-mon[37273]: pgmap v7557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:09 smithi144 ceph-mon[32129]: pgmap v7557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:12.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:11 smithi144 ceph-mon[32129]: pgmap v7558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:12.023 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13334.837365 secs 2024-07-26T14:48:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:11 smithi160 ceph-mon[36715]: pgmap v7558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13334.837365 secs 2024-07-26T14:48:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:11 smithi183 ceph-mon[37273]: pgmap v7558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13334.837365 secs 2024-07-26T14:48:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:13 smithi160 ceph-mon[36715]: pgmap v7559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:13 smithi183 ceph-mon[37273]: pgmap v7559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:13 smithi144 ceph-mon[32129]: pgmap v7559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:15 smithi160 ceph-mon[36715]: pgmap v7560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:15 smithi183 ceph-mon[37273]: pgmap v7560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:15 smithi144 ceph-mon[32129]: pgmap v7560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13339.837486 secs 2024-07-26T14:48:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13339.837486 secs 2024-07-26T14:48:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13339.837486 secs 2024-07-26T14:48:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:17 smithi160 ceph-mon[36715]: pgmap v7561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:17 smithi183 ceph-mon[37273]: pgmap v7561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:17 smithi144 ceph-mon[32129]: pgmap v7561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:19 smithi160 ceph-mon[36715]: pgmap v7562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:19 smithi183 ceph-mon[37273]: pgmap v7562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:19 smithi144 ceph-mon[32129]: pgmap v7562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:21 smithi160 ceph-mon[36715]: pgmap v7563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13344.837615 secs 2024-07-26T14:48:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:21 smithi183 ceph-mon[37273]: pgmap v7563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13344.837615 secs 2024-07-26T14:48:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:21 smithi144 ceph-mon[32129]: pgmap v7563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13344.837615 secs 2024-07-26T14:48:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:23.754+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:48:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:23.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:48:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:23.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:48:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:23.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:48:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:23.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:23 smithi160 ceph-mon[36715]: pgmap v7564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:23 smithi183 ceph-mon[37273]: pgmap v7564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:23 smithi144 ceph-mon[32129]: pgmap v7564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:25 smithi183 ceph-mon[37273]: pgmap v7565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:25 smithi144 ceph-mon[32129]: pgmap v7565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:25 smithi160 ceph-mon[36715]: pgmap v7565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13349.837760 secs 2024-07-26T14:48:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13349.837760 secs 2024-07-26T14:48:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13349.837760 secs 2024-07-26T14:48:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:27 smithi183 ceph-mon[37273]: pgmap v7566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:27 smithi144 ceph-mon[32129]: pgmap v7566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:27 smithi160 ceph-mon[36715]: pgmap v7566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:29 smithi183 ceph-mon[37273]: pgmap v7567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:29 smithi144 ceph-mon[32129]: pgmap v7567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:29 smithi160 ceph-mon[36715]: pgmap v7567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:31 smithi183 ceph-mon[37273]: pgmap v7568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13354.837878 secs 2024-07-26T14:48:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:31 smithi144 ceph-mon[32129]: pgmap v7568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13354.837878 secs 2024-07-26T14:48:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:31 smithi160 ceph-mon[36715]: pgmap v7568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13354.837878 secs 2024-07-26T14:48:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:33.460+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:48:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:33.460+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:48:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:33.461+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:48:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:33.461+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:48:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:33 smithi183 ceph-mon[37273]: pgmap v7569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:33 smithi144 ceph-mon[32129]: pgmap v7569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:33 smithi160 ceph-mon[36715]: pgmap v7569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:35 smithi183 ceph-mon[37273]: pgmap v7570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:48:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:35 smithi144 ceph-mon[32129]: pgmap v7570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:48:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:35 smithi160 ceph-mon[36715]: pgmap v7570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:48:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:36 smithi183 ceph-mon[37273]: pgmap v7571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:36 smithi183 ceph-mon[37273]: pgmap v7572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13359.837997 secs 2024-07-26T14:48:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:36 smithi144 ceph-mon[32129]: pgmap v7571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:36 smithi144 ceph-mon[32129]: pgmap v7572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13359.837997 secs 2024-07-26T14:48:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:36 smithi160 ceph-mon[36715]: pgmap v7571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:36 smithi160 ceph-mon[36715]: pgmap v7572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13359.837997 secs 2024-07-26T14:48:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:38.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:48:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:38.772+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:48:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:38.772+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:48:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:38.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:48:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:38.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:38 smithi183 ceph-mon[37273]: pgmap v7573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:48:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:38 smithi144 ceph-mon[32129]: pgmap v7573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:48:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:38 smithi160 ceph-mon[36715]: pgmap v7573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:48:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:40 smithi144 ceph-mon[32129]: pgmap v7574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:48:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:40 smithi160 ceph-mon[36715]: pgmap v7574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:48:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:40 smithi183 ceph-mon[37273]: pgmap v7574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:48:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13364.838116 secs 2024-07-26T14:48:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13364.838116 secs 2024-07-26T14:48:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13364.838116 secs 2024-07-26T14:48:43.165 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:42 smithi144 ceph-mon[32129]: pgmap v7575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:42 smithi160 ceph-mon[36715]: pgmap v7575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:42 smithi183 ceph-mon[37273]: pgmap v7575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:48:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:44 smithi144 ceph-mon[32129]: pgmap v7576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:48:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:44 smithi160 ceph-mon[36715]: pgmap v7576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:48:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:44 smithi183 ceph-mon[37273]: pgmap v7576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:48:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:46 smithi144 ceph-mon[32129]: pgmap v7577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:48:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13369.838241 secs 2024-07-26T14:48:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:46 smithi160 ceph-mon[36715]: pgmap v7577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:48:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13369.838241 secs 2024-07-26T14:48:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:46 smithi183 ceph-mon[37273]: pgmap v7577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:48:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13369.838241 secs 2024-07-26T14:48:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:48 smithi144 ceph-mon[32129]: pgmap v7578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:48 smithi160 ceph-mon[36715]: pgmap v7578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:48 smithi183 ceph-mon[37273]: pgmap v7578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:50 smithi144 ceph-mon[32129]: pgmap v7579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:50 smithi160 ceph-mon[36715]: pgmap v7579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:50 smithi183 ceph-mon[37273]: pgmap v7579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13374.838386 secs 2024-07-26T14:48:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13374.838386 secs 2024-07-26T14:48:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13374.838386 secs 2024-07-26T14:48:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:52 smithi144 ceph-mon[32129]: pgmap v7580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:52 smithi160 ceph-mon[36715]: pgmap v7580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:52 smithi183 ceph-mon[37273]: pgmap v7580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:48:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:53.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:48:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:53.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:48:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:53.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:48:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:53.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:48:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:48:53.790+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:54 smithi144 ceph-mon[32129]: pgmap v7581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:54 smithi160 ceph-mon[36715]: pgmap v7581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:54 smithi183 ceph-mon[37273]: pgmap v7581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:56 smithi144 ceph-mon[32129]: pgmap v7582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13379.838521 secs 2024-07-26T14:48:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:56 smithi160 ceph-mon[36715]: pgmap v7582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13379.838521 secs 2024-07-26T14:48:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:56 smithi183 ceph-mon[37273]: pgmap v7582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:48:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13379.838521 secs 2024-07-26T14:48:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:48:58 smithi144 ceph-mon[32129]: pgmap v7583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:48:58 smithi160 ceph-mon[36715]: pgmap v7583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:48:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:48:58 smithi183 ceph-mon[37273]: pgmap v7583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:00 smithi144 ceph-mon[32129]: pgmap v7584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:00 smithi160 ceph-mon[36715]: pgmap v7584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:00 smithi183 ceph-mon[37273]: pgmap v7584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13384.838643 secs 2024-07-26T14:49:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13384.838643 secs 2024-07-26T14:49:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13384.838643 secs 2024-07-26T14:49:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:02 smithi160 ceph-mon[36715]: pgmap v7585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:02 smithi183 ceph-mon[37273]: pgmap v7585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:02 smithi144 ceph-mon[32129]: pgmap v7585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:04 smithi160 ceph-mon[36715]: pgmap v7586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:04 smithi183 ceph-mon[37273]: pgmap v7586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:04 smithi144 ceph-mon[32129]: pgmap v7586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:06 smithi160 ceph-mon[36715]: pgmap v7587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13389.838709 secs 2024-07-26T14:49:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:06 smithi183 ceph-mon[37273]: pgmap v7587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13389.838709 secs 2024-07-26T14:49:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:06 smithi144 ceph-mon[32129]: pgmap v7587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13389.838709 secs 2024-07-26T14:49:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:08.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:49:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:08.805+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:49:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:08.805+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:49:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:08.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:49:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:08.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:08 smithi160 ceph-mon[36715]: pgmap v7588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:08 smithi183 ceph-mon[37273]: pgmap v7588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:08 smithi144 ceph-mon[32129]: pgmap v7588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:11 smithi160 ceph-mon[36715]: pgmap v7589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:11 smithi183 ceph-mon[37273]: pgmap v7589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:11 smithi144 ceph-mon[32129]: pgmap v7589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13394.838832 secs 2024-07-26T14:49:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13394.838832 secs 2024-07-26T14:49:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13394.838832 secs 2024-07-26T14:49:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:13 smithi160 ceph-mon[36715]: pgmap v7590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:13 smithi183 ceph-mon[37273]: pgmap v7590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:13 smithi144 ceph-mon[32129]: pgmap v7590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:15 smithi160 ceph-mon[36715]: pgmap v7591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:15 smithi183 ceph-mon[37273]: pgmap v7591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:15 smithi144 ceph-mon[32129]: pgmap v7591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:17 smithi160 ceph-mon[36715]: pgmap v7592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13399.838956 secs 2024-07-26T14:49:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:17 smithi183 ceph-mon[37273]: pgmap v7592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13399.838956 secs 2024-07-26T14:49:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:17 smithi144 ceph-mon[32129]: pgmap v7592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13399.838956 secs 2024-07-26T14:49:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:19 smithi160 ceph-mon[36715]: pgmap v7593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:19 smithi183 ceph-mon[37273]: pgmap v7593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:19 smithi144 ceph-mon[32129]: pgmap v7593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:21 smithi160 ceph-mon[36715]: pgmap v7594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:21 smithi183 ceph-mon[37273]: pgmap v7594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:21 smithi144 ceph-mon[32129]: pgmap v7594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13404.839083 secs 2024-07-26T14:49:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13404.839083 secs 2024-07-26T14:49:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13404.839083 secs 2024-07-26T14:49:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:23 smithi160 ceph-mon[36715]: pgmap v7595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:23 smithi183 ceph-mon[37273]: pgmap v7595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:23 smithi144 ceph-mon[32129]: pgmap v7595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:23.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:23.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:23.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:23.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:23.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:24 smithi183 ceph-mon[37273]: pgmap v7596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:24 smithi144 ceph-mon[32129]: pgmap v7596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:24 smithi160 ceph-mon[36715]: pgmap v7596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:26 smithi183 ceph-mon[37273]: pgmap v7597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:26 smithi144 ceph-mon[32129]: pgmap v7597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:26 smithi160 ceph-mon[36715]: pgmap v7597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13409.839200 secs 2024-07-26T14:49:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13409.839200 secs 2024-07-26T14:49:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13409.839200 secs 2024-07-26T14:49:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:28 smithi144 ceph-mon[32129]: pgmap v7598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:28 smithi160 ceph-mon[36715]: pgmap v7598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:28 smithi183 ceph-mon[37273]: pgmap v7598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:30 smithi144 ceph-mon[32129]: pgmap v7599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:30 smithi160 ceph-mon[36715]: pgmap v7599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:30 smithi183 ceph-mon[37273]: pgmap v7599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13414.839335 secs 2024-07-26T14:49:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13414.839335 secs 2024-07-26T14:49:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13414.839335 secs 2024-07-26T14:49:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:32 smithi144 ceph-mon[32129]: pgmap v7600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:32 smithi160 ceph-mon[36715]: pgmap v7600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:32 smithi183 ceph-mon[37273]: pgmap v7600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:34 smithi144 ceph-mon[32129]: pgmap v7601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:34 smithi160 ceph-mon[36715]: pgmap v7601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:34 smithi183 ceph-mon[37273]: pgmap v7601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:36 smithi144 ceph-mon[32129]: pgmap v7602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13419.839411 secs 2024-07-26T14:49:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:36 smithi160 ceph-mon[36715]: pgmap v7602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13419.839411 secs 2024-07-26T14:49:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:36 smithi183 ceph-mon[37273]: pgmap v7602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13419.839411 secs 2024-07-26T14:49:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:37 smithi144 ceph-mon[32129]: pgmap v7603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:49:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:37 smithi144 ceph-mon[32129]: pgmap v7604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:49:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:49:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:37 smithi160 ceph-mon[36715]: pgmap v7603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:49:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:37 smithi160 ceph-mon[36715]: pgmap v7604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:49:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:49:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:37 smithi183 ceph-mon[37273]: pgmap v7603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:49:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:37 smithi183 ceph-mon[37273]: pgmap v7604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:49:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:49:38.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:38.839+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:49:38.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:38.840+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:49:38.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:38.840+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:49:38.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:38.840+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:49:38.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:38.840+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:39 smithi160 ceph-mon[36715]: pgmap v7605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:49:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:39 smithi183 ceph-mon[37273]: pgmap v7605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:49:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:39 smithi144 ceph-mon[32129]: pgmap v7605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:49:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:41 smithi160 ceph-mon[36715]: pgmap v7606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:49:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13424.839523 secs 2024-07-26T14:49:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:41 smithi183 ceph-mon[37273]: pgmap v7606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:49:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13424.839523 secs 2024-07-26T14:49:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:41 smithi144 ceph-mon[32129]: pgmap v7606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:49:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13424.839523 secs 2024-07-26T14:49:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:43 smithi160 ceph-mon[36715]: pgmap v7607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:49:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:43 smithi183 ceph-mon[37273]: pgmap v7607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:49:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:43 smithi144 ceph-mon[32129]: pgmap v7607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:49:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:45 smithi160 ceph-mon[36715]: pgmap v7608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:49:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:45 smithi183 ceph-mon[37273]: pgmap v7608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:49:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:45 smithi144 ceph-mon[32129]: pgmap v7608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:49:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13429.839643 secs 2024-07-26T14:49:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13429.839643 secs 2024-07-26T14:49:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13429.839643 secs 2024-07-26T14:49:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:47 smithi160 ceph-mon[36715]: pgmap v7609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:49:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:47 smithi183 ceph-mon[37273]: pgmap v7609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:49:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:47 smithi144 ceph-mon[32129]: pgmap v7609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:49:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:49 smithi160 ceph-mon[36715]: pgmap v7610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:49 smithi183 ceph-mon[37273]: pgmap v7610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:49 smithi144 ceph-mon[32129]: pgmap v7610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:51 smithi160 ceph-mon[36715]: pgmap v7611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13434.839758 secs 2024-07-26T14:49:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:51 smithi183 ceph-mon[37273]: pgmap v7611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13434.839758 secs 2024-07-26T14:49:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:51 smithi144 ceph-mon[32129]: pgmap v7611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13434.839758 secs 2024-07-26T14:49:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:53.848+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:49:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:53.849+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:49:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:53.849+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:49:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:53.849+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:49:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:49:53.849+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:53 smithi160 ceph-mon[36715]: pgmap v7612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:53 smithi183 ceph-mon[37273]: pgmap v7612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:53 smithi144 ceph-mon[32129]: pgmap v7612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:49:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:55 smithi160 ceph-mon[36715]: pgmap v7613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:55 smithi183 ceph-mon[37273]: pgmap v7613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:55 smithi144 ceph-mon[32129]: pgmap v7613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:49:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13439.839823 secs 2024-07-26T14:49:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13439.839823 secs 2024-07-26T14:49:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13439.839823 secs 2024-07-26T14:49:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:57 smithi183 ceph-mon[37273]: pgmap v7614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:57 smithi144 ceph-mon[32129]: pgmap v7614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:49:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:57 smithi160 ceph-mon[36715]: pgmap v7614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:49:59 smithi183 ceph-mon[37273]: pgmap v7615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:49:59 smithi144 ceph-mon[32129]: pgmap v7615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:49:59 smithi160 ceph-mon[36715]: pgmap v7615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:50:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:50:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:50:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:01 smithi183 ceph-mon[37273]: pgmap v7616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13444.839941 secs 2024-07-26T14:50:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:01 smithi144 ceph-mon[32129]: pgmap v7616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13444.839941 secs 2024-07-26T14:50:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:01 smithi160 ceph-mon[36715]: pgmap v7616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13444.839941 secs 2024-07-26T14:50:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:03 smithi183 ceph-mon[37273]: pgmap v7617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:03 smithi144 ceph-mon[32129]: pgmap v7617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:03 smithi160 ceph-mon[36715]: pgmap v7617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:05 smithi183 ceph-mon[37273]: pgmap v7618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:05 smithi144 ceph-mon[32129]: pgmap v7618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:05 smithi160 ceph-mon[36715]: pgmap v7618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13449.840065 secs 2024-07-26T14:50:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13449.840065 secs 2024-07-26T14:50:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13449.840065 secs 2024-07-26T14:50:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:07 smithi183 ceph-mon[37273]: pgmap v7619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:07 smithi144 ceph-mon[32129]: pgmap v7619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:07 smithi160 ceph-mon[36715]: pgmap v7619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:08.863+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:50:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:08.864+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:50:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:08.864+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:50:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:08.864+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:50:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:08.864+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:09 smithi183 ceph-mon[37273]: pgmap v7620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:09 smithi144 ceph-mon[32129]: pgmap v7620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:09 smithi160 ceph-mon[36715]: pgmap v7620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:11 smithi183 ceph-mon[37273]: pgmap v7621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13454.840186 secs 2024-07-26T14:50:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:11 smithi144 ceph-mon[32129]: pgmap v7621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13454.840186 secs 2024-07-26T14:50:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:11 smithi160 ceph-mon[36715]: pgmap v7621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13454.840186 secs 2024-07-26T14:50:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:13 smithi183 ceph-mon[37273]: pgmap v7622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:13 smithi144 ceph-mon[32129]: pgmap v7622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:13 smithi160 ceph-mon[36715]: pgmap v7622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:15 smithi144 ceph-mon[32129]: pgmap v7623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:15 smithi160 ceph-mon[36715]: pgmap v7623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:15 smithi183 ceph-mon[37273]: pgmap v7623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13459.840328 secs 2024-07-26T14:50:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13459.840328 secs 2024-07-26T14:50:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13459.840328 secs 2024-07-26T14:50:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:17 smithi144 ceph-mon[32129]: pgmap v7624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:17 smithi160 ceph-mon[36715]: pgmap v7624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:17 smithi183 ceph-mon[37273]: pgmap v7624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:19 smithi144 ceph-mon[32129]: pgmap v7625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:19 smithi160 ceph-mon[36715]: pgmap v7625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:19 smithi183 ceph-mon[37273]: pgmap v7625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:21 smithi144 ceph-mon[32129]: pgmap v7626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13464.840444 secs 2024-07-26T14:50:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:21 smithi160 ceph-mon[36715]: pgmap v7626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13464.840444 secs 2024-07-26T14:50:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:21 smithi183 ceph-mon[37273]: pgmap v7626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13464.840444 secs 2024-07-26T14:50:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:23.881+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:50:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:23.882+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:50:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:23.882+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:50:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:23.882+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:50:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:23.882+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:23 smithi144 ceph-mon[32129]: pgmap v7627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:23 smithi160 ceph-mon[36715]: pgmap v7627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:23 smithi183 ceph-mon[37273]: pgmap v7627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:25 smithi144 ceph-mon[32129]: pgmap v7628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:25 smithi160 ceph-mon[36715]: pgmap v7628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:25 smithi183 ceph-mon[37273]: pgmap v7628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13469.840562 secs 2024-07-26T14:50:27.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13469.840562 secs 2024-07-26T14:50:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13469.840562 secs 2024-07-26T14:50:28.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:27 smithi144 ceph-mon[32129]: pgmap v7629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:27 smithi160 ceph-mon[36715]: pgmap v7629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:27 smithi183 ceph-mon[37273]: pgmap v7629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:29 smithi144 ceph-mon[32129]: pgmap v7630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:29 smithi160 ceph-mon[36715]: pgmap v7630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:29 smithi183 ceph-mon[37273]: pgmap v7630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:31 smithi144 ceph-mon[32129]: pgmap v7631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13474.840680 secs 2024-07-26T14:50:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:31 smithi160 ceph-mon[36715]: pgmap v7631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13474.840680 secs 2024-07-26T14:50:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:31 smithi183 ceph-mon[37273]: pgmap v7631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13474.840680 secs 2024-07-26T14:50:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:33 smithi144 ceph-mon[32129]: pgmap v7632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:33 smithi160 ceph-mon[36715]: pgmap v7632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:33 smithi183 ceph-mon[37273]: pgmap v7632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:35 smithi144 ceph-mon[32129]: pgmap v7633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:35 smithi160 ceph-mon[36715]: pgmap v7633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:35 smithi183 ceph-mon[37273]: pgmap v7633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13479.840799 secs 2024-07-26T14:50:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13479.840799 secs 2024-07-26T14:50:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13479.840799 secs 2024-07-26T14:50:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:37 smithi144 ceph-mon[32129]: pgmap v7634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:50:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:37 smithi160 ceph-mon[36715]: pgmap v7634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:50:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:37 smithi183 ceph-mon[37273]: pgmap v7634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:50:38.894 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:38.893+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:50:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:38.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:50:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:38.894+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:50:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:38.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:50:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:38.894+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:38 smithi160 ceph-mon[36715]: pgmap v7635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:50:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:38 smithi160 ceph-mon[36715]: pgmap v7636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:50:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:38 smithi183 ceph-mon[37273]: pgmap v7635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:50:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:38 smithi183 ceph-mon[37273]: pgmap v7636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:50:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:38 smithi144 ceph-mon[32129]: pgmap v7635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:50:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:38 smithi144 ceph-mon[32129]: pgmap v7636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:50:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:40 smithi160 ceph-mon[36715]: pgmap v7637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:50:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:40 smithi183 ceph-mon[37273]: pgmap v7637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:50:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:40 smithi144 ceph-mon[32129]: pgmap v7637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:50:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13484.840940 secs 2024-07-26T14:50:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13484.840940 secs 2024-07-26T14:50:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13484.840940 secs 2024-07-26T14:50:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:43 smithi160 ceph-mon[36715]: pgmap v7638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:50:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:43 smithi183 ceph-mon[37273]: pgmap v7638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:50:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:43 smithi144 ceph-mon[32129]: pgmap v7638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:50:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:45 smithi160 ceph-mon[36715]: pgmap v7639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:50:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:45 smithi183 ceph-mon[37273]: pgmap v7639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:50:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:45 smithi144 ceph-mon[32129]: pgmap v7639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:50:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:50:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:50:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f74a8002230 tx=0x7f74a804c070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:50:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f74a8002230 tx=0x7f74a804c070 comp rx=0 tx=0).stop 2024-07-26T14:50:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:50:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:50:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.140+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:50:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.141+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:50:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:46.143+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a806c070 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:50:47.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:47 smithi144 ceph-mon[32129]: pgmap v7640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:50:47.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13489.841056 secs 2024-07-26T14:50:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:47 smithi160 ceph-mon[36715]: pgmap v7640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:50:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13489.841056 secs 2024-07-26T14:50:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:47 smithi183 ceph-mon[37273]: pgmap v7640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:50:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13489.841056 secs 2024-07-26T14:50:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:48.484+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:50:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:48.484+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74780065a0 con 0x563c6c4182f0 2024-07-26T14:50:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:48.485+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74780065a0 con 0x7f74b0109990 2024-07-26T14:50:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:48.485+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:50:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:49 smithi160 ceph-mon[36715]: pgmap v7641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:50:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:49 smithi183 ceph-mon[37273]: pgmap v7641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:50:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:49 smithi144 ceph-mon[32129]: pgmap v7641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:50:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:51 smithi160 ceph-mon[36715]: pgmap v7642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:51 smithi183 ceph-mon[37273]: pgmap v7642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:51 smithi144 ceph-mon[32129]: pgmap v7642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13494.841178 secs 2024-07-26T14:50:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13494.841178 secs 2024-07-26T14:50:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13494.841178 secs 2024-07-26T14:50:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:53 smithi160 ceph-mon[36715]: pgmap v7643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:53 smithi183 ceph-mon[37273]: pgmap v7643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:53 smithi144 ceph-mon[32129]: pgmap v7643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:50:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:53.910+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:50:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:53.911+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:50:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:53.911+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:50:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:53.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:50:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:50:53.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:55 smithi160 ceph-mon[36715]: pgmap v7644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:55 smithi183 ceph-mon[37273]: pgmap v7644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:55 smithi144 ceph-mon[32129]: pgmap v7644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13499.841273 secs 2024-07-26T14:50:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:57 smithi160 ceph-mon[36715]: pgmap v7645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13499.841273 secs 2024-07-26T14:50:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:57 smithi183 ceph-mon[37273]: pgmap v7645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13499.841273 secs 2024-07-26T14:50:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:57 smithi144 ceph-mon[32129]: pgmap v7645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:50:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:50:59 smithi183 ceph-mon[37273]: pgmap v7646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:50:59 smithi144 ceph-mon[32129]: pgmap v7646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:50:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:50:59 smithi160 ceph-mon[36715]: pgmap v7646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:01 smithi183 ceph-mon[37273]: pgmap v7647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:01 smithi144 ceph-mon[32129]: pgmap v7647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:01 smithi160 ceph-mon[36715]: pgmap v7647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13504.841404 secs 2024-07-26T14:51:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13504.841404 secs 2024-07-26T14:51:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13504.841404 secs 2024-07-26T14:51:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:03 smithi144 ceph-mon[32129]: pgmap v7648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:03 smithi183 ceph-mon[37273]: pgmap v7648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:03 smithi160 ceph-mon[36715]: pgmap v7648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:04 smithi183 ceph-mon[37273]: pgmap v7649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:04 smithi144 ceph-mon[32129]: pgmap v7649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:04 smithi160 ceph-mon[36715]: pgmap v7649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13509.841527 secs 2024-07-26T14:51:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:06 smithi144 ceph-mon[32129]: pgmap v7650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13509.841527 secs 2024-07-26T14:51:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:06 smithi160 ceph-mon[36715]: pgmap v7650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13509.841527 secs 2024-07-26T14:51:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:06 smithi183 ceph-mon[37273]: pgmap v7650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:08.923+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:51:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:08.923+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:51:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:08.923+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:51:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:08.924+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:51:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:08.924+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:08 smithi144 ceph-mon[32129]: pgmap v7651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:08 smithi160 ceph-mon[36715]: pgmap v7651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:08 smithi183 ceph-mon[37273]: pgmap v7651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:10 smithi144 ceph-mon[32129]: pgmap v7652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:10 smithi160 ceph-mon[36715]: pgmap v7652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:10 smithi183 ceph-mon[37273]: pgmap v7652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13514.841613 secs 2024-07-26T14:51:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13514.841613 secs 2024-07-26T14:51:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13514.841613 secs 2024-07-26T14:51:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:12 smithi144 ceph-mon[32129]: pgmap v7653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:12 smithi160 ceph-mon[36715]: pgmap v7653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:12 smithi183 ceph-mon[37273]: pgmap v7653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:14 smithi144 ceph-mon[32129]: pgmap v7654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:14 smithi160 ceph-mon[36715]: pgmap v7654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:14 smithi183 ceph-mon[37273]: pgmap v7654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:16.966 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13519.841729 secs 2024-07-26T14:51:16.966 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:16 smithi183 ceph-mon[37273]: pgmap v7655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13519.841729 secs 2024-07-26T14:51:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:16 smithi144 ceph-mon[32129]: pgmap v7655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13519.841729 secs 2024-07-26T14:51:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:16 smithi160 ceph-mon[36715]: pgmap v7655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:18 smithi160 ceph-mon[36715]: pgmap v7656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:19.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:18 smithi183 ceph-mon[37273]: pgmap v7656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:18 smithi144 ceph-mon[32129]: pgmap v7656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:20 smithi160 ceph-mon[36715]: pgmap v7657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:20 smithi183 ceph-mon[37273]: pgmap v7657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:20 smithi144 ceph-mon[32129]: pgmap v7657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13524.841850 secs 2024-07-26T14:51:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13524.841850 secs 2024-07-26T14:51:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13524.841850 secs 2024-07-26T14:51:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:22 smithi160 ceph-mon[36715]: pgmap v7658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:22 smithi183 ceph-mon[37273]: pgmap v7658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:22 smithi144 ceph-mon[32129]: pgmap v7658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:23.940+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:51:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:23.941+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:51:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:23.942+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:51:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:23.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:51:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:23.942+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:24 smithi160 ceph-mon[36715]: pgmap v7659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:24 smithi183 ceph-mon[37273]: pgmap v7659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:24 smithi144 ceph-mon[32129]: pgmap v7659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:27.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13529.841976 secs 2024-07-26T14:51:27.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:27 smithi144 ceph-mon[32129]: pgmap v7660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13529.841976 secs 2024-07-26T14:51:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:27 smithi160 ceph-mon[36715]: pgmap v7660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13529.841976 secs 2024-07-26T14:51:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:27 smithi183 ceph-mon[37273]: pgmap v7660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:29 smithi160 ceph-mon[36715]: pgmap v7661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:29 smithi183 ceph-mon[37273]: pgmap v7661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:29 smithi144 ceph-mon[32129]: pgmap v7661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:31 smithi160 ceph-mon[36715]: pgmap v7662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:31 smithi183 ceph-mon[37273]: pgmap v7662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:31 smithi144 ceph-mon[32129]: pgmap v7662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13534.842096 secs 2024-07-26T14:51:32.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13534.842096 secs 2024-07-26T14:51:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13534.842096 secs 2024-07-26T14:51:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:33 smithi160 ceph-mon[36715]: pgmap v7663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:33 smithi183 ceph-mon[37273]: pgmap v7663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:33 smithi144 ceph-mon[32129]: pgmap v7663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:35 smithi160 ceph-mon[36715]: pgmap v7664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:35 smithi183 ceph-mon[37273]: pgmap v7664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:35 smithi144 ceph-mon[32129]: pgmap v7664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13539.842218 secs 2024-07-26T14:51:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:37 smithi160 ceph-mon[36715]: pgmap v7665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13539.842218 secs 2024-07-26T14:51:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:37 smithi183 ceph-mon[37273]: pgmap v7665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13539.842218 secs 2024-07-26T14:51:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:37 smithi144 ceph-mon[32129]: pgmap v7665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:38.958+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:51:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:38.959+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:38.959+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:38.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:38.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:39 smithi160 ceph-mon[36715]: pgmap v7666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:39 smithi183 ceph-mon[37273]: pgmap v7666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:39 smithi144 ceph-mon[32129]: pgmap v7666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:40 smithi183 ceph-mon[37273]: pgmap v7667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:40 smithi183 ceph-mon[37273]: pgmap v7668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:51:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:51:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:40 smithi144 ceph-mon[32129]: pgmap v7667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:40 smithi144 ceph-mon[32129]: pgmap v7668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:51:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:51:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:40 smithi160 ceph-mon[36715]: pgmap v7667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:40 smithi160 ceph-mon[36715]: pgmap v7668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:51:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:51:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13544.842345 secs 2024-07-26T14:51:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:42 smithi183 ceph-mon[37273]: pgmap v7669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13544.842345 secs 2024-07-26T14:51:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:42 smithi144 ceph-mon[32129]: pgmap v7669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13544.842345 secs 2024-07-26T14:51:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:42 smithi160 ceph-mon[36715]: pgmap v7669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:44 smithi183 ceph-mon[37273]: pgmap v7670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:51:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:44 smithi144 ceph-mon[32129]: pgmap v7670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:51:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:44 smithi160 ceph-mon[36715]: pgmap v7670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:51:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:46 smithi183 ceph-mon[37273]: pgmap v7671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:51:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:46 smithi144 ceph-mon[32129]: pgmap v7671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:51:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:46 smithi160 ceph-mon[36715]: pgmap v7671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:51:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13549.842440 secs 2024-07-26T14:51:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:47 smithi183 ceph-mon[37273]: pgmap v7672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:47.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13549.842440 secs 2024-07-26T14:51:47.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:47 smithi160 ceph-mon[36715]: pgmap v7672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13549.842440 secs 2024-07-26T14:51:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:47 smithi144 ceph-mon[32129]: pgmap v7672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:51:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:49 smithi183 ceph-mon[37273]: pgmap v7673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:51:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:49 smithi144 ceph-mon[32129]: pgmap v7673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:51:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:49 smithi160 ceph-mon[36715]: pgmap v7673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:51:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:51 smithi183 ceph-mon[37273]: pgmap v7674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:51 smithi144 ceph-mon[32129]: pgmap v7674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:51 smithi160 ceph-mon[36715]: pgmap v7674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13554.842562 secs 2024-07-26T14:51:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13554.842562 secs 2024-07-26T14:51:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13554.842562 secs 2024-07-26T14:51:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:53 smithi144 ceph-mon[32129]: pgmap v7675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:53 smithi160 ceph-mon[36715]: pgmap v7675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:53 smithi183 ceph-mon[37273]: pgmap v7675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:51:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:53.974+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:51:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:53.976+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:51:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:53.976+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:51:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:53.976+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:51:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:51:53.976+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:55 smithi183 ceph-mon[37273]: pgmap v7676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:55 smithi144 ceph-mon[32129]: pgmap v7676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:55 smithi160 ceph-mon[36715]: pgmap v7676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13559.842664 secs 2024-07-26T14:51:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13559.842664 secs 2024-07-26T14:51:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13559.842664 secs 2024-07-26T14:51:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:57 smithi160 ceph-mon[36715]: pgmap v7677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:57 smithi183 ceph-mon[37273]: pgmap v7677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:57 smithi144 ceph-mon[32129]: pgmap v7677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:51:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:51:59 smithi183 ceph-mon[37273]: pgmap v7678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:51:59 smithi144 ceph-mon[32129]: pgmap v7678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:51:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:51:59 smithi160 ceph-mon[36715]: pgmap v7678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:01 smithi183 ceph-mon[37273]: pgmap v7679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:01 smithi144 ceph-mon[32129]: pgmap v7679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:01 smithi160 ceph-mon[36715]: pgmap v7679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13564.842794 secs 2024-07-26T14:52:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13564.842794 secs 2024-07-26T14:52:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13564.842794 secs 2024-07-26T14:52:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:03 smithi144 ceph-mon[32129]: pgmap v7680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:03 smithi160 ceph-mon[36715]: pgmap v7680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:03 smithi183 ceph-mon[37273]: pgmap v7680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:05 smithi144 ceph-mon[32129]: pgmap v7681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:52:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:05 smithi160 ceph-mon[36715]: pgmap v7681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:52:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:05 smithi183 ceph-mon[37273]: pgmap v7681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:52:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13569.842888 secs 2024-07-26T14:52:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13569.842888 secs 2024-07-26T14:52:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13569.842888 secs 2024-07-26T14:52:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:07 smithi160 ceph-mon[36715]: pgmap v7682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:07 smithi183 ceph-mon[37273]: pgmap v7682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:07 smithi144 ceph-mon[32129]: pgmap v7682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:08.991+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:52:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:08.992+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:52:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:08.992+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:52:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:08.992+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:52:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:08.992+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:09 smithi144 ceph-mon[32129]: pgmap v7683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:09 smithi160 ceph-mon[36715]: pgmap v7683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:09 smithi183 ceph-mon[37273]: pgmap v7683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:11 smithi144 ceph-mon[32129]: pgmap v7684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:11 smithi160 ceph-mon[36715]: pgmap v7684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:11 smithi183 ceph-mon[37273]: pgmap v7684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13574.843015 secs 2024-07-26T14:52:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13574.843015 secs 2024-07-26T14:52:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13574.843015 secs 2024-07-26T14:52:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:13 smithi144 ceph-mon[32129]: pgmap v7685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:52:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:13 smithi160 ceph-mon[36715]: pgmap v7685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:52:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:13 smithi183 ceph-mon[37273]: pgmap v7685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:52:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:15 smithi144 ceph-mon[32129]: pgmap v7686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:52:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:15 smithi160 ceph-mon[36715]: pgmap v7686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:52:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:15 smithi183 ceph-mon[37273]: pgmap v7686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:52:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13579.843110 secs 2024-07-26T14:52:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13579.843110 secs 2024-07-26T14:52:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13579.843110 secs 2024-07-26T14:52:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:17 smithi144 ceph-mon[32129]: pgmap v7687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:17 smithi160 ceph-mon[36715]: pgmap v7687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:17 smithi183 ceph-mon[37273]: pgmap v7687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:19 smithi144 ceph-mon[32129]: pgmap v7688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:19 smithi160 ceph-mon[36715]: pgmap v7688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:19 smithi183 ceph-mon[37273]: pgmap v7688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:21 smithi144 ceph-mon[32129]: pgmap v7689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:21 smithi160 ceph-mon[36715]: pgmap v7689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:21 smithi183 ceph-mon[37273]: pgmap v7689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13584.843227 secs 2024-07-26T14:52:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13584.843227 secs 2024-07-26T14:52:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13584.843227 secs 2024-07-26T14:52:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:23 smithi144 ceph-mon[32129]: pgmap v7690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:23 smithi160 ceph-mon[36715]: pgmap v7690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:23 smithi183 ceph-mon[37273]: pgmap v7690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:24.010 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:24.009+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:52:24.010 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:24.010+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:52:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:24.010+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:52:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:24.010+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:52:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:24.010+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:25 smithi144 ceph-mon[32129]: pgmap v7691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:25 smithi160 ceph-mon[36715]: pgmap v7691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:25 smithi183 ceph-mon[37273]: pgmap v7691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13589.843363 secs 2024-07-26T14:52:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13589.843363 secs 2024-07-26T14:52:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13589.843363 secs 2024-07-26T14:52:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:27 smithi160 ceph-mon[36715]: pgmap v7692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:27 smithi183 ceph-mon[37273]: pgmap v7692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:27 smithi144 ceph-mon[32129]: pgmap v7692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:29 smithi144 ceph-mon[32129]: pgmap v7693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:29 smithi160 ceph-mon[36715]: pgmap v7693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:29 smithi183 ceph-mon[37273]: pgmap v7693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:31 smithi144 ceph-mon[32129]: pgmap v7694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:31 smithi160 ceph-mon[36715]: pgmap v7694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:31 smithi183 ceph-mon[37273]: pgmap v7694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13594.843504 secs 2024-07-26T14:52:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13594.843504 secs 2024-07-26T14:52:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13594.843504 secs 2024-07-26T14:52:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:33 smithi144 ceph-mon[32129]: pgmap v7695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:33 smithi160 ceph-mon[36715]: pgmap v7695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:33 smithi183 ceph-mon[37273]: pgmap v7695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:35 smithi144 ceph-mon[32129]: pgmap v7696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:35 smithi160 ceph-mon[36715]: pgmap v7696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:35 smithi183 ceph-mon[37273]: pgmap v7696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13599.843650 secs 2024-07-26T14:52:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13599.843650 secs 2024-07-26T14:52:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13599.843650 secs 2024-07-26T14:52:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:37 smithi160 ceph-mon[36715]: pgmap v7697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:37 smithi183 ceph-mon[37273]: pgmap v7697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:37 smithi144 ceph-mon[32129]: pgmap v7697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:39.026 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:39.025+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:52:39.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:39.026+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:52:39.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:39.026+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:52:39.027 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:39.026+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:52:39.028 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:39.027+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:39 smithi144 ceph-mon[32129]: pgmap v7698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:39 smithi160 ceph-mon[36715]: pgmap v7698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:39 smithi183 ceph-mon[37273]: pgmap v7698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:52:41.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:52:41.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:41.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:41.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: pgmap v7699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:41.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: pgmap v7700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:41.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:52:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: pgmap v7699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: pgmap v7700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:52:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: pgmap v7699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: pgmap v7700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:52:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13604.843802 secs 2024-07-26T14:52:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13604.843802 secs 2024-07-26T14:52:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13604.843802 secs 2024-07-26T14:52:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:43 smithi144 ceph-mon[32129]: pgmap v7701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:43 smithi160 ceph-mon[36715]: pgmap v7701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:43 smithi183 ceph-mon[37273]: pgmap v7701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:45 smithi183 ceph-mon[37273]: pgmap v7702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:45 smithi144 ceph-mon[32129]: pgmap v7702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:45 smithi160 ceph-mon[36715]: pgmap v7702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:47.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13609.843882 secs 2024-07-26T14:52:47.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:47 smithi160 ceph-mon[36715]: pgmap v7703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13609.843882 secs 2024-07-26T14:52:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:47 smithi183 ceph-mon[37273]: pgmap v7703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13609.843882 secs 2024-07-26T14:52:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:47 smithi144 ceph-mon[32129]: pgmap v7703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:49 smithi183 ceph-mon[37273]: pgmap v7704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:49 smithi144 ceph-mon[32129]: pgmap v7704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:49 smithi160 ceph-mon[36715]: pgmap v7704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:52:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:51 smithi183 ceph-mon[37273]: pgmap v7705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:51 smithi144 ceph-mon[32129]: pgmap v7705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:51 smithi160 ceph-mon[36715]: pgmap v7705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:52:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13614.843978 secs 2024-07-26T14:52:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13614.843978 secs 2024-07-26T14:52:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13614.843978 secs 2024-07-26T14:52:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:53 smithi183 ceph-mon[37273]: pgmap v7706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:53 smithi144 ceph-mon[32129]: pgmap v7706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:53 smithi160 ceph-mon[36715]: pgmap v7706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:52:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:54.042+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:52:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:54.043+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:52:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:54.043+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:52:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:54.043+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:52:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:52:54.043+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:55 smithi183 ceph-mon[37273]: pgmap v7707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:55 smithi144 ceph-mon[32129]: pgmap v7707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:55 smithi160 ceph-mon[36715]: pgmap v7707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13619.844093 secs 2024-07-26T14:52:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:57 smithi183 ceph-mon[37273]: pgmap v7708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13619.844093 secs 2024-07-26T14:52:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:57 smithi144 ceph-mon[32129]: pgmap v7708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13619.844093 secs 2024-07-26T14:52:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:57 smithi160 ceph-mon[36715]: pgmap v7708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:52:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:52:59 smithi183 ceph-mon[37273]: pgmap v7709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:52:59 smithi144 ceph-mon[32129]: pgmap v7709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:52:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:52:59 smithi160 ceph-mon[36715]: pgmap v7709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:01 smithi183 ceph-mon[37273]: pgmap v7710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:01 smithi144 ceph-mon[32129]: pgmap v7710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:01 smithi160 ceph-mon[36715]: pgmap v7710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13624.844159 secs 2024-07-26T14:53:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13624.844159 secs 2024-07-26T14:53:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13624.844159 secs 2024-07-26T14:53:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:03 smithi144 ceph-mon[32129]: pgmap v7711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:03 smithi160 ceph-mon[36715]: pgmap v7711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:03 smithi183 ceph-mon[37273]: pgmap v7711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:05 smithi144 ceph-mon[32129]: pgmap v7712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:05 smithi160 ceph-mon[36715]: pgmap v7712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:05 smithi183 ceph-mon[37273]: pgmap v7712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13629.844316 secs 2024-07-26T14:53:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:07 smithi144 ceph-mon[32129]: pgmap v7713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13629.844316 secs 2024-07-26T14:53:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:07 smithi160 ceph-mon[36715]: pgmap v7713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13629.844316 secs 2024-07-26T14:53:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:07 smithi183 ceph-mon[37273]: pgmap v7713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:09.052+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:53:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:09.053+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:53:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:09.053+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:53:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:09.053+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:53:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:09.053+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:09 smithi144 ceph-mon[32129]: pgmap v7714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:09 smithi160 ceph-mon[36715]: pgmap v7714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:09 smithi183 ceph-mon[37273]: pgmap v7714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:11 smithi144 ceph-mon[32129]: pgmap v7715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:11 smithi160 ceph-mon[36715]: pgmap v7715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:11 smithi183 ceph-mon[37273]: pgmap v7715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13634.844349 secs 2024-07-26T14:53:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13634.844349 secs 2024-07-26T14:53:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13634.844349 secs 2024-07-26T14:53:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:13 smithi144 ceph-mon[32129]: pgmap v7716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:13 smithi160 ceph-mon[36715]: pgmap v7716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:13 smithi183 ceph-mon[37273]: pgmap v7716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:15 smithi144 ceph-mon[32129]: pgmap v7717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:15 smithi160 ceph-mon[36715]: pgmap v7717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:15 smithi183 ceph-mon[37273]: pgmap v7717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13639.844456 secs 2024-07-26T14:53:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:17 smithi144 ceph-mon[32129]: pgmap v7718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13639.844456 secs 2024-07-26T14:53:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:17 smithi160 ceph-mon[36715]: pgmap v7718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13639.844456 secs 2024-07-26T14:53:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:17 smithi183 ceph-mon[37273]: pgmap v7718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:19 smithi144 ceph-mon[32129]: pgmap v7719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:19 smithi160 ceph-mon[36715]: pgmap v7719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:19 smithi183 ceph-mon[37273]: pgmap v7719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:21 smithi144 ceph-mon[32129]: pgmap v7720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:21 smithi160 ceph-mon[36715]: pgmap v7720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:21 smithi183 ceph-mon[37273]: pgmap v7720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13644.844525 secs 2024-07-26T14:53:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13644.844525 secs 2024-07-26T14:53:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13644.844525 secs 2024-07-26T14:53:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:23 smithi144 ceph-mon[32129]: pgmap v7721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:23 smithi160 ceph-mon[36715]: pgmap v7721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:23 smithi183 ceph-mon[37273]: pgmap v7721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:24.070 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:24.069+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:53:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:24.070+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:53:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:24.070+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:53:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:24.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:53:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:24.070+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:25 smithi160 ceph-mon[36715]: pgmap v7722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:25 smithi183 ceph-mon[37273]: pgmap v7722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:25 smithi144 ceph-mon[32129]: pgmap v7722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:27.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13649.844646 secs 2024-07-26T14:53:27.772 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:27 smithi144 ceph-mon[32129]: pgmap v7723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13649.844646 secs 2024-07-26T14:53:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:27 smithi160 ceph-mon[36715]: pgmap v7723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13649.844646 secs 2024-07-26T14:53:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:27 smithi183 ceph-mon[37273]: pgmap v7723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:53:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:29 smithi160 ceph-mon[36715]: pgmap v7724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:29 smithi183 ceph-mon[37273]: pgmap v7724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:29 smithi144 ceph-mon[32129]: pgmap v7724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:31 smithi160 ceph-mon[36715]: pgmap v7725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:31 smithi183 ceph-mon[37273]: pgmap v7725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:31 smithi144 ceph-mon[32129]: pgmap v7725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13654.844762 secs 2024-07-26T14:53:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13654.844762 secs 2024-07-26T14:53:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13654.844762 secs 2024-07-26T14:53:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:33 smithi160 ceph-mon[36715]: pgmap v7726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:33 smithi183 ceph-mon[37273]: pgmap v7726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:33 smithi144 ceph-mon[32129]: pgmap v7726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:35 smithi160 ceph-mon[36715]: pgmap v7727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:35 smithi183 ceph-mon[37273]: pgmap v7727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:35 smithi144 ceph-mon[32129]: pgmap v7727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13659.844883 secs 2024-07-26T14:53:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:37 smithi160 ceph-mon[36715]: pgmap v7728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:53:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13659.844883 secs 2024-07-26T14:53:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:37 smithi183 ceph-mon[37273]: pgmap v7728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:53:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13659.844883 secs 2024-07-26T14:53:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:37 smithi144 ceph-mon[32129]: pgmap v7728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:53:39.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:39.077+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:53:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:39.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:53:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:39.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:53:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:39.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:53:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:39.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:39 smithi160 ceph-mon[36715]: pgmap v7729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:39 smithi183 ceph-mon[37273]: pgmap v7729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:39 smithi144 ceph-mon[32129]: pgmap v7729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: pgmap v7730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:53:41.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:53:41.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:53:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: pgmap v7730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:53:41.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: pgmap v7730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T14:53:41.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T14:53:41.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:53:41.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T14:53:41.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:53:41.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:53:41.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:53:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13664.845036 secs 2024-07-26T14:53:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:42 smithi160 ceph-mon[36715]: pgmap v7731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:53:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:42 smithi160 ceph-mon[36715]: pgmap v7732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:53:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13664.845036 secs 2024-07-26T14:53:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:42 smithi183 ceph-mon[37273]: pgmap v7731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:53:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:42 smithi183 ceph-mon[37273]: pgmap v7732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:53:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13664.845036 secs 2024-07-26T14:53:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:42 smithi144 ceph-mon[32129]: pgmap v7731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:53:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:42 smithi144 ceph-mon[32129]: pgmap v7732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:53:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:44 smithi160 ceph-mon[36715]: pgmap v7733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:53:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:44 smithi183 ceph-mon[37273]: pgmap v7733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:53:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:44 smithi144 ceph-mon[32129]: pgmap v7733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:53:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:46 smithi160 ceph-mon[36715]: pgmap v7734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:53:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:46 smithi183 ceph-mon[37273]: pgmap v7734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:53:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:46 smithi144 ceph-mon[32129]: pgmap v7734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:53:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13669.845187 secs 2024-07-26T14:53:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13669.845187 secs 2024-07-26T14:53:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13669.845187 secs 2024-07-26T14:53:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:48.516+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:53:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:48.516+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:53:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:48.517+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:53:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:48.517+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:53:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:48 smithi183 ceph-mon[37273]: pgmap v7735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:48 smithi144 ceph-mon[32129]: pgmap v7735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:48 smithi160 ceph-mon[36715]: pgmap v7735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:50 smithi183 ceph-mon[37273]: pgmap v7736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:50 smithi144 ceph-mon[32129]: pgmap v7736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:50 smithi160 ceph-mon[36715]: pgmap v7736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:53:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13674.845337 secs 2024-07-26T14:53:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:52 smithi183 ceph-mon[37273]: pgmap v7737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:53:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13674.845337 secs 2024-07-26T14:53:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:52 smithi144 ceph-mon[32129]: pgmap v7737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:53:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13674.845337 secs 2024-07-26T14:53:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:52 smithi160 ceph-mon[36715]: pgmap v7737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:53:54.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:54.094+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:53:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:54.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:53:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:54.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:53:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:54.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:53:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:53:54.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:54 smithi183 ceph-mon[37273]: pgmap v7738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:54 smithi144 ceph-mon[32129]: pgmap v7738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:54 smithi160 ceph-mon[36715]: pgmap v7738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:56 smithi183 ceph-mon[37273]: pgmap v7739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:56 smithi144 ceph-mon[32129]: pgmap v7739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:56 smithi160 ceph-mon[36715]: pgmap v7739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:53:57.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13679.845451 secs 2024-07-26T14:53:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13679.845451 secs 2024-07-26T14:53:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13679.845451 secs 2024-07-26T14:53:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:53:58 smithi183 ceph-mon[37273]: pgmap v7740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:53:58 smithi144 ceph-mon[32129]: pgmap v7740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:53:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:53:58 smithi160 ceph-mon[36715]: pgmap v7740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:00 smithi183 ceph-mon[37273]: pgmap v7741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:00 smithi144 ceph-mon[32129]: pgmap v7741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:00 smithi160 ceph-mon[36715]: pgmap v7741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13684.845563 secs 2024-07-26T14:54:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:02 smithi183 ceph-mon[37273]: pgmap v7742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13684.845563 secs 2024-07-26T14:54:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:02 smithi144 ceph-mon[32129]: pgmap v7742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13684.845563 secs 2024-07-26T14:54:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:02 smithi160 ceph-mon[36715]: pgmap v7742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:04 smithi183 ceph-mon[37273]: pgmap v7743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:04 smithi144 ceph-mon[32129]: pgmap v7743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:04 smithi160 ceph-mon[36715]: pgmap v7743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:06 smithi183 ceph-mon[37273]: pgmap v7744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:06 smithi144 ceph-mon[32129]: pgmap v7744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:06 smithi160 ceph-mon[36715]: pgmap v7744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13689.845693 secs 2024-07-26T14:54:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13689.845693 secs 2024-07-26T14:54:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13689.845693 secs 2024-07-26T14:54:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:08 smithi144 ceph-mon[32129]: pgmap v7745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:08 smithi160 ceph-mon[36715]: pgmap v7745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:09.104+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:54:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:09.105+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:54:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:09.105+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:54:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:09.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:54:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:09.105+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:08 smithi183 ceph-mon[37273]: pgmap v7745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:10 smithi144 ceph-mon[32129]: pgmap v7746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:10 smithi160 ceph-mon[36715]: pgmap v7746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:10 smithi183 ceph-mon[37273]: pgmap v7746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13694.845807 secs 2024-07-26T14:54:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:12 smithi144 ceph-mon[32129]: pgmap v7747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13694.845807 secs 2024-07-26T14:54:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:12 smithi160 ceph-mon[36715]: pgmap v7747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13694.845807 secs 2024-07-26T14:54:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:12 smithi183 ceph-mon[37273]: pgmap v7747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:14 smithi144 ceph-mon[32129]: pgmap v7748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:14 smithi160 ceph-mon[36715]: pgmap v7748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:14 smithi183 ceph-mon[37273]: pgmap v7748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:16 smithi144 ceph-mon[32129]: pgmap v7749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:16 smithi160 ceph-mon[36715]: pgmap v7749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:16 smithi183 ceph-mon[37273]: pgmap v7749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:17.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13699.845931 secs 2024-07-26T14:54:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13699.845931 secs 2024-07-26T14:54:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13699.845931 secs 2024-07-26T14:54:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:18 smithi160 ceph-mon[36715]: pgmap v7750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:18 smithi183 ceph-mon[37273]: pgmap v7750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:18 smithi144 ceph-mon[32129]: pgmap v7750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:20 smithi160 ceph-mon[36715]: pgmap v7751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:20 smithi183 ceph-mon[37273]: pgmap v7751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:20 smithi144 ceph-mon[32129]: pgmap v7751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13704.846079 secs 2024-07-26T14:54:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13704.846079 secs 2024-07-26T14:54:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13704.846079 secs 2024-07-26T14:54:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:22 smithi160 ceph-mon[36715]: pgmap v7752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:22 smithi183 ceph-mon[37273]: pgmap v7752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:22 smithi144 ceph-mon[32129]: pgmap v7752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:24.119+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:54:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:24.121+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:54:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:24.121+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:54:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:24.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:54:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:24.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:24 smithi183 ceph-mon[37273]: pgmap v7753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:25.181 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:24 smithi144 ceph-mon[32129]: pgmap v7753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:24 smithi160 ceph-mon[36715]: pgmap v7753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:26 smithi183 ceph-mon[37273]: pgmap v7754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13709.846207 secs 2024-07-26T14:54:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:26 smithi144 ceph-mon[32129]: pgmap v7754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13709.846207 secs 2024-07-26T14:54:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:26 smithi160 ceph-mon[36715]: pgmap v7754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13709.846207 secs 2024-07-26T14:54:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:28 smithi183 ceph-mon[37273]: pgmap v7755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:28 smithi144 ceph-mon[32129]: pgmap v7755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:28 smithi160 ceph-mon[36715]: pgmap v7755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:30 smithi183 ceph-mon[37273]: pgmap v7756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:30 smithi144 ceph-mon[32129]: pgmap v7756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:30 smithi160 ceph-mon[36715]: pgmap v7756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13714.846312 secs 2024-07-26T14:54:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13714.846312 secs 2024-07-26T14:54:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13714.846312 secs 2024-07-26T14:54:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:32 smithi183 ceph-mon[37273]: pgmap v7757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:32 smithi144 ceph-mon[32129]: pgmap v7757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:32 smithi160 ceph-mon[36715]: pgmap v7757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:34 smithi183 ceph-mon[37273]: pgmap v7758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:34 smithi144 ceph-mon[32129]: pgmap v7758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:34 smithi160 ceph-mon[36715]: pgmap v7758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:36 smithi183 ceph-mon[37273]: pgmap v7759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13719.846382 secs 2024-07-26T14:54:37.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:36 smithi144 ceph-mon[32129]: pgmap v7759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:37.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13719.846382 secs 2024-07-26T14:54:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:36 smithi160 ceph-mon[36715]: pgmap v7759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13719.846382 secs 2024-07-26T14:54:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:38 smithi183 ceph-mon[37273]: pgmap v7760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:39.137+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:54:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:39.138+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:54:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:39.138+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:54:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:39.138+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:54:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:39.138+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:38 smithi144 ceph-mon[32129]: pgmap v7760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:38 smithi160 ceph-mon[36715]: pgmap v7760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:54:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:40 smithi183 ceph-mon[37273]: pgmap v7761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:40 smithi144 ceph-mon[32129]: pgmap v7761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:40 smithi160 ceph-mon[36715]: pgmap v7761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13724.846451 secs 2024-07-26T14:54:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:54:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13724.846451 secs 2024-07-26T14:54:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:54:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13724.846451 secs 2024-07-26T14:54:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:54:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:42 smithi144 ceph-mon[32129]: pgmap v7762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:54:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:42 smithi160 ceph-mon[36715]: pgmap v7762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:54:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:42 smithi183 ceph-mon[37273]: pgmap v7762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:54:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:43 smithi144 ceph-mon[32129]: pgmap v7763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:54:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:43 smithi144 ceph-mon[32129]: pgmap v7764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:54:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:43 smithi160 ceph-mon[36715]: pgmap v7763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:54:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:43 smithi160 ceph-mon[36715]: pgmap v7764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:54:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:43 smithi183 ceph-mon[37273]: pgmap v7763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:54:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:43 smithi183 ceph-mon[37273]: pgmap v7764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:54:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:45 smithi144 ceph-mon[32129]: pgmap v7765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:54:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:45 smithi160 ceph-mon[36715]: pgmap v7765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:54:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:45 smithi183 ceph-mon[37273]: pgmap v7765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:54:47.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13729.846595 secs 2024-07-26T14:54:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13729.846595 secs 2024-07-26T14:54:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13729.846595 secs 2024-07-26T14:54:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:47 smithi144 ceph-mon[32129]: pgmap v7766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:54:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:47 smithi160 ceph-mon[36715]: pgmap v7766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:54:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:47 smithi183 ceph-mon[37273]: pgmap v7766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:54:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:49 smithi144 ceph-mon[32129]: pgmap v7767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:54:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:49 smithi160 ceph-mon[36715]: pgmap v7767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:54:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:49 smithi183 ceph-mon[37273]: pgmap v7767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:54:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:51 smithi144 ceph-mon[32129]: pgmap v7768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:54:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13734.846719 secs 2024-07-26T14:54:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:51 smithi160 ceph-mon[36715]: pgmap v7768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:54:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13734.846719 secs 2024-07-26T14:54:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:51 smithi183 ceph-mon[37273]: pgmap v7768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:54:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13734.846719 secs 2024-07-26T14:54:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:54.153+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:54:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:54.155+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:54:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:54.155+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:54:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:54.155+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:54:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:54:54.155+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:53 smithi144 ceph-mon[32129]: pgmap v7769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:54:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:53 smithi160 ceph-mon[36715]: pgmap v7769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:54:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:53 smithi183 ceph-mon[37273]: pgmap v7769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:54:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:55 smithi144 ceph-mon[32129]: pgmap v7770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:55 smithi160 ceph-mon[36715]: pgmap v7770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:55 smithi183 ceph-mon[37273]: pgmap v7770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:54:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13739.846865 secs 2024-07-26T14:54:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13739.846865 secs 2024-07-26T14:54:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13739.846865 secs 2024-07-26T14:54:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:57 smithi144 ceph-mon[32129]: pgmap v7771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:58.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:57 smithi183 ceph-mon[37273]: pgmap v7771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:54:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:57 smithi160 ceph-mon[36715]: pgmap v7771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:54:59 smithi144 ceph-mon[32129]: pgmap v7772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:54:59 smithi160 ceph-mon[36715]: pgmap v7772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:54:59 smithi183 ceph-mon[37273]: pgmap v7772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:01 smithi144 ceph-mon[32129]: pgmap v7773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13744.846986 secs 2024-07-26T14:55:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:01 smithi160 ceph-mon[36715]: pgmap v7773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13744.846986 secs 2024-07-26T14:55:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:01 smithi183 ceph-mon[37273]: pgmap v7773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13744.846986 secs 2024-07-26T14:55:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:03 smithi144 ceph-mon[32129]: pgmap v7774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:03 smithi160 ceph-mon[36715]: pgmap v7774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:03 smithi183 ceph-mon[37273]: pgmap v7774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:05 smithi144 ceph-mon[32129]: pgmap v7775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:05 smithi160 ceph-mon[36715]: pgmap v7775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:05 smithi183 ceph-mon[37273]: pgmap v7775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13749.847083 secs 2024-07-26T14:55:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13749.847083 secs 2024-07-26T14:55:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13749.847083 secs 2024-07-26T14:55:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:07 smithi160 ceph-mon[36715]: pgmap v7776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:07 smithi183 ceph-mon[37273]: pgmap v7776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:07 smithi144 ceph-mon[32129]: pgmap v7776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:09.158+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:55:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:09.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:55:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:09.159+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:55:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:09.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:55:09.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:09.159+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:09 smithi160 ceph-mon[36715]: pgmap v7777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:09 smithi183 ceph-mon[37273]: pgmap v7777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:09 smithi144 ceph-mon[32129]: pgmap v7777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:11 smithi160 ceph-mon[36715]: pgmap v7778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13754.847205 secs 2024-07-26T14:55:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:11 smithi183 ceph-mon[37273]: pgmap v7778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13754.847205 secs 2024-07-26T14:55:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:11 smithi144 ceph-mon[32129]: pgmap v7778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13754.847205 secs 2024-07-26T14:55:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:13 smithi160 ceph-mon[36715]: pgmap v7779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:13 smithi183 ceph-mon[37273]: pgmap v7779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:13 smithi144 ceph-mon[32129]: pgmap v7779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:16 smithi160 ceph-mon[36715]: pgmap v7780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:16 smithi183 ceph-mon[37273]: pgmap v7780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:16 smithi144 ceph-mon[32129]: pgmap v7780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13759.847306 secs 2024-07-26T14:55:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13759.847306 secs 2024-07-26T14:55:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13759.847306 secs 2024-07-26T14:55:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:18 smithi160 ceph-mon[36715]: pgmap v7781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:18 smithi183 ceph-mon[37273]: pgmap v7781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:18 smithi144 ceph-mon[32129]: pgmap v7781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:20 smithi160 ceph-mon[36715]: pgmap v7782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:20 smithi183 ceph-mon[37273]: pgmap v7782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:20 smithi144 ceph-mon[32129]: pgmap v7782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:22 smithi160 ceph-mon[36715]: pgmap v7783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13764.847429 secs 2024-07-26T14:55:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:22 smithi183 ceph-mon[37273]: pgmap v7783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13764.847429 secs 2024-07-26T14:55:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:22 smithi144 ceph-mon[32129]: pgmap v7783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13764.847429 secs 2024-07-26T14:55:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:24.164+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:55:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:24.165+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:55:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:24.165+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:55:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:24.165+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:55:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:24.165+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:24 smithi160 ceph-mon[36715]: pgmap v7784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:24 smithi183 ceph-mon[37273]: pgmap v7784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:24 smithi144 ceph-mon[32129]: pgmap v7784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:26 smithi160 ceph-mon[36715]: pgmap v7785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:26 smithi183 ceph-mon[37273]: pgmap v7785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:26 smithi144 ceph-mon[32129]: pgmap v7785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13769.847552 secs 2024-07-26T14:55:27.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13769.847552 secs 2024-07-26T14:55:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13769.847552 secs 2024-07-26T14:55:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:28 smithi160 ceph-mon[36715]: pgmap v7786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:28 smithi183 ceph-mon[37273]: pgmap v7786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:28 smithi144 ceph-mon[32129]: pgmap v7786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:30 smithi183 ceph-mon[37273]: pgmap v7787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:30 smithi144 ceph-mon[32129]: pgmap v7787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:30 smithi160 ceph-mon[36715]: pgmap v7787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:32 smithi183 ceph-mon[37273]: pgmap v7788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13774.847681 secs 2024-07-26T14:55:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:32 smithi144 ceph-mon[32129]: pgmap v7788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13774.847681 secs 2024-07-26T14:55:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:32 smithi160 ceph-mon[36715]: pgmap v7788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13774.847681 secs 2024-07-26T14:55:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:34 smithi183 ceph-mon[37273]: pgmap v7789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:34 smithi144 ceph-mon[32129]: pgmap v7789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:34 smithi160 ceph-mon[36715]: pgmap v7789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:36 smithi183 ceph-mon[37273]: pgmap v7790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:36 smithi144 ceph-mon[32129]: pgmap v7790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:36 smithi160 ceph-mon[36715]: pgmap v7790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13779.847801 secs 2024-07-26T14:55:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13779.847801 secs 2024-07-26T14:55:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13779.847801 secs 2024-07-26T14:55:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:38 smithi183 ceph-mon[37273]: pgmap v7791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:38 smithi144 ceph-mon[32129]: pgmap v7791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:38 smithi160 ceph-mon[36715]: pgmap v7791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:39.181+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:55:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:39.182+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:55:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:39.182+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:55:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:39.182+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:55:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:39.182+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:40 smithi183 ceph-mon[37273]: pgmap v7792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:40 smithi144 ceph-mon[32129]: pgmap v7792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:40 smithi160 ceph-mon[36715]: pgmap v7792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:42 smithi183 ceph-mon[37273]: pgmap v7793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13784.847930 secs 2024-07-26T14:55:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:42 smithi144 ceph-mon[32129]: pgmap v7793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13784.847930 secs 2024-07-26T14:55:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:42 smithi160 ceph-mon[36715]: pgmap v7793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13784.847930 secs 2024-07-26T14:55:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:44 smithi183 ceph-mon[37273]: pgmap v7794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:44 smithi144 ceph-mon[32129]: pgmap v7794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:44 smithi160 ceph-mon[36715]: pgmap v7794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:55:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:45 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:45 smithi144 ceph-mon[32129]: pgmap v7795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:45 smithi144 ceph-mon[32129]: pgmap v7796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:45 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:45 smithi160 ceph-mon[36715]: pgmap v7795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:45 smithi160 ceph-mon[36715]: pgmap v7796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:45 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:45 smithi183 ceph-mon[37273]: pgmap v7795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:45 smithi183 ceph-mon[37273]: pgmap v7796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:55:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13789.848047 secs 2024-07-26T14:55:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:47 smithi160 ceph-mon[36715]: pgmap v7797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13789.848047 secs 2024-07-26T14:55:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:47 smithi183 ceph-mon[37273]: pgmap v7797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13789.848047 secs 2024-07-26T14:55:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:47 smithi144 ceph-mon[32129]: pgmap v7797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:49 smithi160 ceph-mon[36715]: pgmap v7798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:49 smithi183 ceph-mon[37273]: pgmap v7798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:49 smithi144 ceph-mon[32129]: pgmap v7798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:51 smithi160 ceph-mon[36715]: pgmap v7799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:51 smithi183 ceph-mon[37273]: pgmap v7799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:51 smithi144 ceph-mon[32129]: pgmap v7799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13794.848165 secs 2024-07-26T14:55:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13794.848165 secs 2024-07-26T14:55:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13794.848165 secs 2024-07-26T14:55:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:53 smithi183 ceph-mon[37273]: pgmap v7800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:53 smithi144 ceph-mon[32129]: pgmap v7800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:53 smithi160 ceph-mon[36715]: pgmap v7800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:55:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:54.197+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:55:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:54.198+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:55:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:54.198+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:55:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:54.198+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:55:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:55:54.198+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:55 smithi183 ceph-mon[37273]: pgmap v7801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:55:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:55 smithi144 ceph-mon[32129]: pgmap v7801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:55:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:55 smithi160 ceph-mon[36715]: pgmap v7801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:55:57.853 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13799.848283 secs 2024-07-26T14:55:57.854 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:57 smithi144 ceph-mon[32129]: pgmap v7802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13799.848283 secs 2024-07-26T14:55:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:57 smithi183 ceph-mon[37273]: pgmap v7802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13799.848283 secs 2024-07-26T14:55:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:57 smithi160 ceph-mon[36715]: pgmap v7802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:55:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:55:59 smithi183 ceph-mon[37273]: pgmap v7803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:55:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:55:59 smithi144 ceph-mon[32129]: pgmap v7803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:55:59 smithi160 ceph-mon[36715]: pgmap v7803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:01 smithi183 ceph-mon[37273]: pgmap v7804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:01 smithi144 ceph-mon[32129]: pgmap v7804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:01 smithi160 ceph-mon[36715]: pgmap v7804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13804.848334 secs 2024-07-26T14:56:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13804.848334 secs 2024-07-26T14:56:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13804.848334 secs 2024-07-26T14:56:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:03 smithi183 ceph-mon[37273]: pgmap v7805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:03 smithi144 ceph-mon[32129]: pgmap v7805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:03 smithi160 ceph-mon[36715]: pgmap v7805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:05 smithi183 ceph-mon[37273]: pgmap v7806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:05 smithi144 ceph-mon[32129]: pgmap v7806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:05 smithi160 ceph-mon[36715]: pgmap v7806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13809.848425 secs 2024-07-26T14:56:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:07 smithi183 ceph-mon[37273]: pgmap v7807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13809.848425 secs 2024-07-26T14:56:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:07 smithi144 ceph-mon[32129]: pgmap v7807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13809.848425 secs 2024-07-26T14:56:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:07 smithi160 ceph-mon[36715]: pgmap v7807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:09.213+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:56:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:09.214+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:56:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:09.214+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:56:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:09.214+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:56:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:09.214+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:09 smithi183 ceph-mon[37273]: pgmap v7808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:09 smithi144 ceph-mon[32129]: pgmap v7808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:09 smithi160 ceph-mon[36715]: pgmap v7808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:11 smithi183 ceph-mon[37273]: pgmap v7809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:11 smithi144 ceph-mon[32129]: pgmap v7809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:11 smithi160 ceph-mon[36715]: pgmap v7809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13814.848513 secs 2024-07-26T14:56:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13814.848513 secs 2024-07-26T14:56:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13814.848513 secs 2024-07-26T14:56:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:13 smithi144 ceph-mon[32129]: pgmap v7810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:13 smithi160 ceph-mon[36715]: pgmap v7810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:13 smithi183 ceph-mon[37273]: pgmap v7810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:15 smithi144 ceph-mon[32129]: pgmap v7811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:15 smithi160 ceph-mon[36715]: pgmap v7811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:15 smithi183 ceph-mon[37273]: pgmap v7811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13819.848591 secs 2024-07-26T14:56:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:17 smithi144 ceph-mon[32129]: pgmap v7812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13819.848591 secs 2024-07-26T14:56:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:17 smithi160 ceph-mon[36715]: pgmap v7812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13819.848591 secs 2024-07-26T14:56:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:17 smithi183 ceph-mon[37273]: pgmap v7812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:19 smithi144 ceph-mon[32129]: pgmap v7813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:19 smithi160 ceph-mon[36715]: pgmap v7813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:19 smithi183 ceph-mon[37273]: pgmap v7813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:21 smithi144 ceph-mon[32129]: pgmap v7814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:21 smithi160 ceph-mon[36715]: pgmap v7814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:21 smithi183 ceph-mon[37273]: pgmap v7814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13824.848712 secs 2024-07-26T14:56:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13824.848712 secs 2024-07-26T14:56:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13824.848712 secs 2024-07-26T14:56:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:23 smithi144 ceph-mon[32129]: pgmap v7815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:23 smithi160 ceph-mon[36715]: pgmap v7815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:23 smithi183 ceph-mon[37273]: pgmap v7815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:24.231+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:56:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:24.232+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:56:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:24.232+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:56:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:24.232+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:56:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:24.232+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:25 smithi144 ceph-mon[32129]: pgmap v7816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:25 smithi160 ceph-mon[36715]: pgmap v7816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:25 smithi183 ceph-mon[37273]: pgmap v7816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13829.848826 secs 2024-07-26T14:56:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:27 smithi144 ceph-mon[32129]: pgmap v7817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13829.848826 secs 2024-07-26T14:56:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:27 smithi160 ceph-mon[36715]: pgmap v7817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13829.848826 secs 2024-07-26T14:56:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:27 smithi183 ceph-mon[37273]: pgmap v7817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:29 smithi144 ceph-mon[32129]: pgmap v7818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:29 smithi160 ceph-mon[36715]: pgmap v7818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:29 smithi183 ceph-mon[37273]: pgmap v7818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:31 smithi144 ceph-mon[32129]: pgmap v7819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:31 smithi160 ceph-mon[36715]: pgmap v7819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:31 smithi183 ceph-mon[37273]: pgmap v7819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13834.848905 secs 2024-07-26T14:56:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13834.848905 secs 2024-07-26T14:56:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13834.848905 secs 2024-07-26T14:56:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:33 smithi144 ceph-mon[32129]: pgmap v7820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:33 smithi160 ceph-mon[36715]: pgmap v7820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:33 smithi183 ceph-mon[37273]: pgmap v7820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:35 smithi144 ceph-mon[32129]: pgmap v7821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:35 smithi160 ceph-mon[36715]: pgmap v7821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:35 smithi183 ceph-mon[37273]: pgmap v7821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13839.848967 secs 2024-07-26T14:56:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13839.848967 secs 2024-07-26T14:56:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13839.848967 secs 2024-07-26T14:56:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:37 smithi160 ceph-mon[36715]: pgmap v7822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:37 smithi183 ceph-mon[37273]: pgmap v7822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:37 smithi144 ceph-mon[32129]: pgmap v7822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:39.237+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:56:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:39.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:56:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:39.238+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:56:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:39.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:56:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:39.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:39 smithi160 ceph-mon[36715]: pgmap v7823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:39 smithi183 ceph-mon[37273]: pgmap v7823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:39 smithi144 ceph-mon[32129]: pgmap v7823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:41 smithi160 ceph-mon[36715]: pgmap v7824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13844.849084 secs 2024-07-26T14:56:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:41 smithi183 ceph-mon[37273]: pgmap v7824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13844.849084 secs 2024-07-26T14:56:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:41 smithi144 ceph-mon[32129]: pgmap v7824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13844.849084 secs 2024-07-26T14:56:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:43 smithi160 ceph-mon[36715]: pgmap v7825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:43 smithi183 ceph-mon[37273]: pgmap v7825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:43 smithi144 ceph-mon[32129]: pgmap v7825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:44 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:44 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:44 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:46 smithi160 ceph-mon[36715]: pgmap v7826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:46 smithi183 ceph-mon[37273]: pgmap v7826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:46 smithi144 ceph-mon[32129]: pgmap v7826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13849.849177 secs 2024-07-26T14:56:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13849.849177 secs 2024-07-26T14:56:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13849.849177 secs 2024-07-26T14:56:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: pgmap v7827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: pgmap v7827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.548+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T14:56:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.548+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T14:56:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.548+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T14:56:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.548+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74a009d0e0 con 0x7f74b0109990 2024-07-26T14:56:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.549+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d0e0 con 0x7f74b0109990 2024-07-26T14:56:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.550+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T14:56:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.551+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x563c6c4182f0 2024-07-26T14:56:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:48.551+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480004740 con 0x563c6c4182f0 2024-07-26T14:56:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: pgmap v7827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:56:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:49 smithi160 ceph-mon[36715]: pgmap v7828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:49 smithi183 ceph-mon[37273]: pgmap v7828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:49 smithi144 ceph-mon[32129]: pgmap v7828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: pgmap v7829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: pgmap v7830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:56:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: pgmap v7829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: pgmap v7830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:56:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: pgmap v7829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:56:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: pgmap v7830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:56:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:56:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13854.849323 secs 2024-07-26T14:56:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13854.849323 secs 2024-07-26T14:56:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13854.849323 secs 2024-07-26T14:56:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:53 smithi144 ceph-mon[32129]: pgmap v7831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:53 smithi160 ceph-mon[36715]: pgmap v7831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:53 smithi183 ceph-mon[37273]: pgmap v7831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:54.254+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:56:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:54.255+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:56:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:54.255+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:56:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:54.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:56:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:56:54.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:55 smithi160 ceph-mon[36715]: pgmap v7832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:55 smithi183 ceph-mon[37273]: pgmap v7832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:55 smithi144 ceph-mon[32129]: pgmap v7832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:57.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13859.849454 secs 2024-07-26T14:56:57.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:57 smithi144 ceph-mon[32129]: pgmap v7833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13859.849454 secs 2024-07-26T14:56:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:57 smithi160 ceph-mon[36715]: pgmap v7833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13859.849454 secs 2024-07-26T14:56:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:57 smithi183 ceph-mon[37273]: pgmap v7833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:56:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:56:59 smithi160 ceph-mon[36715]: pgmap v7834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:56:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:56:59 smithi183 ceph-mon[37273]: pgmap v7834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:56:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:56:59 smithi144 ceph-mon[32129]: pgmap v7834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:57:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:01 smithi160 ceph-mon[36715]: pgmap v7835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:57:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:01 smithi183 ceph-mon[37273]: pgmap v7835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:57:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:01 smithi144 ceph-mon[32129]: pgmap v7835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:57:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13864.849516 secs 2024-07-26T14:57:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13864.849516 secs 2024-07-26T14:57:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13864.849516 secs 2024-07-26T14:57:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:03 smithi160 ceph-mon[36715]: pgmap v7836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:03 smithi183 ceph-mon[37273]: pgmap v7836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:03 smithi144 ceph-mon[32129]: pgmap v7836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:05 smithi160 ceph-mon[36715]: pgmap v7837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:05 smithi183 ceph-mon[37273]: pgmap v7837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:05 smithi144 ceph-mon[32129]: pgmap v7837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13869.849620 secs 2024-07-26T14:57:07.582 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:07 smithi160 ceph-mon[36715]: pgmap v7838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13869.849620 secs 2024-07-26T14:57:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:07 smithi183 ceph-mon[37273]: pgmap v7838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13869.849620 secs 2024-07-26T14:57:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:07 smithi144 ceph-mon[32129]: pgmap v7838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:09.267+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:57:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:09.268+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:57:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:09.268+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:57:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:09.268+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:57:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:09.268+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:09 smithi160 ceph-mon[36715]: pgmap v7839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:09 smithi183 ceph-mon[37273]: pgmap v7839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:09 smithi144 ceph-mon[32129]: pgmap v7839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:11 smithi160 ceph-mon[36715]: pgmap v7840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:11 smithi183 ceph-mon[37273]: pgmap v7840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:11 smithi144 ceph-mon[32129]: pgmap v7840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13874.849750 secs 2024-07-26T14:57:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13874.849750 secs 2024-07-26T14:57:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13874.849750 secs 2024-07-26T14:57:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:13 smithi160 ceph-mon[36715]: pgmap v7841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:13 smithi183 ceph-mon[37273]: pgmap v7841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:13 smithi144 ceph-mon[32129]: pgmap v7841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:15 smithi183 ceph-mon[37273]: pgmap v7842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:15 smithi144 ceph-mon[32129]: pgmap v7842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:15 smithi160 ceph-mon[36715]: pgmap v7842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:17.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13879.849824 secs 2024-07-26T14:57:17.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:17 smithi160 ceph-mon[36715]: pgmap v7843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13879.849824 secs 2024-07-26T14:57:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:17 smithi183 ceph-mon[37273]: pgmap v7843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13879.849824 secs 2024-07-26T14:57:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:17 smithi144 ceph-mon[32129]: pgmap v7843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:19 smithi183 ceph-mon[37273]: pgmap v7844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:19 smithi144 ceph-mon[32129]: pgmap v7844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:19 smithi160 ceph-mon[36715]: pgmap v7844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:21 smithi183 ceph-mon[37273]: pgmap v7845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:21 smithi144 ceph-mon[32129]: pgmap v7845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:21 smithi160 ceph-mon[36715]: pgmap v7845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13884.849907 secs 2024-07-26T14:57:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13884.849907 secs 2024-07-26T14:57:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13884.849907 secs 2024-07-26T14:57:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:23 smithi183 ceph-mon[37273]: pgmap v7846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:23 smithi144 ceph-mon[32129]: pgmap v7846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:24.285+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:57:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:24.286+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:57:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:24.286+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:57:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:24.286+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:57:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:24.286+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:23 smithi160 ceph-mon[36715]: pgmap v7846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:25 smithi183 ceph-mon[37273]: pgmap v7847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:25 smithi144 ceph-mon[32129]: pgmap v7847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:25 smithi160 ceph-mon[36715]: pgmap v7847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13889.849965 secs 2024-07-26T14:57:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13889.849965 secs 2024-07-26T14:57:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13889.849965 secs 2024-07-26T14:57:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:27 smithi144 ceph-mon[32129]: pgmap v7848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:27 smithi160 ceph-mon[36715]: pgmap v7848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:28.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:27 smithi183 ceph-mon[37273]: pgmap v7848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:30.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:29 smithi183 ceph-mon[37273]: pgmap v7849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:29 smithi144 ceph-mon[32129]: pgmap v7849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:29 smithi160 ceph-mon[36715]: pgmap v7849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:31 smithi144 ceph-mon[32129]: pgmap v7850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13894.850044 secs 2024-07-26T14:57:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:31 smithi160 ceph-mon[36715]: pgmap v7850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13894.850044 secs 2024-07-26T14:57:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:31 smithi183 ceph-mon[37273]: pgmap v7850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13894.850044 secs 2024-07-26T14:57:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:33 smithi144 ceph-mon[32129]: pgmap v7851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:33 smithi160 ceph-mon[36715]: pgmap v7851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:33 smithi183 ceph-mon[37273]: pgmap v7851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:35 smithi144 ceph-mon[32129]: pgmap v7852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:35 smithi160 ceph-mon[36715]: pgmap v7852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:35 smithi183 ceph-mon[37273]: pgmap v7852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13899.850141 secs 2024-07-26T14:57:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13899.850141 secs 2024-07-26T14:57:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13899.850141 secs 2024-07-26T14:57:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:37 smithi144 ceph-mon[32129]: pgmap v7853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:37 smithi160 ceph-mon[36715]: pgmap v7853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:37 smithi183 ceph-mon[37273]: pgmap v7853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:39.298+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:57:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:39.298+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:57:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:39.299+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:57:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:39.299+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:57:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:39.299+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:39 smithi144 ceph-mon[32129]: pgmap v7854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:39 smithi160 ceph-mon[36715]: pgmap v7854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:39 smithi183 ceph-mon[37273]: pgmap v7854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:41 smithi144 ceph-mon[32129]: pgmap v7855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13904.850313 secs 2024-07-26T14:57:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:41 smithi160 ceph-mon[36715]: pgmap v7855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13904.850313 secs 2024-07-26T14:57:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:41 smithi183 ceph-mon[37273]: pgmap v7855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13904.850313 secs 2024-07-26T14:57:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:43 smithi144 ceph-mon[32129]: pgmap v7856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:43 smithi160 ceph-mon[36715]: pgmap v7856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:43 smithi183 ceph-mon[37273]: pgmap v7856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:45 smithi144 ceph-mon[32129]: pgmap v7857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:45 smithi160 ceph-mon[36715]: pgmap v7857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:45 smithi183 ceph-mon[37273]: pgmap v7857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:47.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13909.850415 secs 2024-07-26T14:57:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13909.850415 secs 2024-07-26T14:57:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13909.850415 secs 2024-07-26T14:57:48.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:47 smithi144 ceph-mon[32129]: pgmap v7858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:47 smithi160 ceph-mon[36715]: pgmap v7858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:47 smithi183 ceph-mon[37273]: pgmap v7858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:57:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:49 smithi160 ceph-mon[36715]: pgmap v7859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:49 smithi183 ceph-mon[37273]: pgmap v7859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:49 smithi144 ceph-mon[32129]: pgmap v7859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:57:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:51.468 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:51 smithi160 ceph-mon[36715]: pgmap v7860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13914.850536 secs 2024-07-26T14:57:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:57:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:51 smithi183 ceph-mon[37273]: pgmap v7860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13914.850536 secs 2024-07-26T14:57:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:57:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:51 smithi144 ceph-mon[32129]: pgmap v7860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:57:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13914.850536 secs 2024-07-26T14:57:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:57:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:53 smithi160 ceph-mon[36715]: pgmap v7861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:53 smithi160 ceph-mon[36715]: pgmap v7862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:53 smithi183 ceph-mon[37273]: pgmap v7861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:53 smithi183 ceph-mon[37273]: pgmap v7862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:53 smithi144 ceph-mon[32129]: pgmap v7861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:53 smithi144 ceph-mon[32129]: pgmap v7862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:54.315+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:57:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:54.315+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:57:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:54.316+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:57:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:54.316+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:57:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:57:54.316+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:55 smithi160 ceph-mon[36715]: pgmap v7863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:57:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:55 smithi183 ceph-mon[37273]: pgmap v7863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:57:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:55 smithi144 ceph-mon[32129]: pgmap v7863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:57:57.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:57 smithi144 ceph-mon[32129]: pgmap v7864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:57.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13919.850662 secs 2024-07-26T14:57:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:57 smithi160 ceph-mon[36715]: pgmap v7864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13919.850662 secs 2024-07-26T14:57:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:57 smithi183 ceph-mon[37273]: pgmap v7864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13919.850662 secs 2024-07-26T14:57:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:57:59 smithi160 ceph-mon[36715]: pgmap v7865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:57:59 smithi183 ceph-mon[37273]: pgmap v7865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:57:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:57:59 smithi144 ceph-mon[32129]: pgmap v7865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:01 smithi160 ceph-mon[36715]: pgmap v7866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:01 smithi183 ceph-mon[37273]: pgmap v7866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:01 smithi144 ceph-mon[32129]: pgmap v7866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13924.850789 secs 2024-07-26T14:58:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13924.850789 secs 2024-07-26T14:58:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13924.850789 secs 2024-07-26T14:58:03.293 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:03 smithi144 ceph-mon[32129]: pgmap v7867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:03 smithi160 ceph-mon[36715]: pgmap v7867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:03 smithi183 ceph-mon[37273]: pgmap v7867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:05 smithi160 ceph-mon[36715]: pgmap v7868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:05 smithi183 ceph-mon[37273]: pgmap v7868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:05 smithi144 ceph-mon[32129]: pgmap v7868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:07 smithi160 ceph-mon[36715]: pgmap v7869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13929.850935 secs 2024-07-26T14:58:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:07 smithi183 ceph-mon[37273]: pgmap v7869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13929.850935 secs 2024-07-26T14:58:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:07 smithi144 ceph-mon[32129]: pgmap v7869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13929.850935 secs 2024-07-26T14:58:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:09 smithi160 ceph-mon[36715]: pgmap v7870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:09.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:58:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:09.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:58:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:09.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:58:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:09.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:58:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:09.321+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:09 smithi183 ceph-mon[37273]: pgmap v7870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:09 smithi144 ceph-mon[32129]: pgmap v7870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:11 smithi183 ceph-mon[37273]: pgmap v7871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:11 smithi144 ceph-mon[32129]: pgmap v7871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:11 smithi160 ceph-mon[36715]: pgmap v7871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13934.851055 secs 2024-07-26T14:58:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13934.851055 secs 2024-07-26T14:58:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13934.851055 secs 2024-07-26T14:58:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:13 smithi183 ceph-mon[37273]: pgmap v7872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:13 smithi144 ceph-mon[32129]: pgmap v7872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:13 smithi160 ceph-mon[36715]: pgmap v7872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:15.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:15 smithi183 ceph-mon[37273]: pgmap v7873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:15 smithi144 ceph-mon[32129]: pgmap v7873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:15 smithi160 ceph-mon[36715]: pgmap v7873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:17.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:17 smithi160 ceph-mon[36715]: pgmap v7874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:17.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13939.851199 secs 2024-07-26T14:58:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:17 smithi183 ceph-mon[37273]: pgmap v7874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13939.851199 secs 2024-07-26T14:58:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:17 smithi144 ceph-mon[32129]: pgmap v7874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13939.851199 secs 2024-07-26T14:58:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:19 smithi183 ceph-mon[37273]: pgmap v7875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:19 smithi144 ceph-mon[32129]: pgmap v7875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:19 smithi160 ceph-mon[36715]: pgmap v7875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:21 smithi183 ceph-mon[37273]: pgmap v7876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:21 smithi144 ceph-mon[32129]: pgmap v7876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:21 smithi160 ceph-mon[36715]: pgmap v7876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13944.851321 secs 2024-07-26T14:58:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13944.851321 secs 2024-07-26T14:58:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13944.851321 secs 2024-07-26T14:58:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:23 smithi183 ceph-mon[37273]: pgmap v7877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:23 smithi144 ceph-mon[32129]: pgmap v7877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:23 smithi160 ceph-mon[36715]: pgmap v7877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:24.338 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:24.337+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:58:24.339 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:24.338+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:58:24.339 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:24.338+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:58:24.339 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:24.338+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:58:24.339 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:24.338+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:25 smithi183 ceph-mon[37273]: pgmap v7878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:25 smithi144 ceph-mon[32129]: pgmap v7878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:25 smithi160 ceph-mon[36715]: pgmap v7878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:27 smithi183 ceph-mon[37273]: pgmap v7879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13949.851448 secs 2024-07-26T14:58:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:27 smithi144 ceph-mon[32129]: pgmap v7879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13949.851448 secs 2024-07-26T14:58:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:27 smithi160 ceph-mon[36715]: pgmap v7879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13949.851448 secs 2024-07-26T14:58:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:29 smithi183 ceph-mon[37273]: pgmap v7880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:29 smithi144 ceph-mon[32129]: pgmap v7880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:29 smithi160 ceph-mon[36715]: pgmap v7880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:31 smithi144 ceph-mon[32129]: pgmap v7881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:58:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:31 smithi160 ceph-mon[36715]: pgmap v7881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:58:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:31 smithi183 ceph-mon[37273]: pgmap v7881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:58:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13954.851597 secs 2024-07-26T14:58:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13954.851597 secs 2024-07-26T14:58:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13954.851597 secs 2024-07-26T14:58:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:33 smithi144 ceph-mon[32129]: pgmap v7882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:33 smithi160 ceph-mon[36715]: pgmap v7882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:33 smithi183 ceph-mon[37273]: pgmap v7882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:35 smithi144 ceph-mon[32129]: pgmap v7883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:35 smithi160 ceph-mon[36715]: pgmap v7883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:35 smithi183 ceph-mon[37273]: pgmap v7883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:37 smithi144 ceph-mon[32129]: pgmap v7884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13959.851725 secs 2024-07-26T14:58:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:37 smithi160 ceph-mon[36715]: pgmap v7884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13959.851725 secs 2024-07-26T14:58:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:37 smithi183 ceph-mon[37273]: pgmap v7884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:58:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13959.851725 secs 2024-07-26T14:58:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:39.340+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:58:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:39.341+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:58:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:39.341+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:58:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:39.341+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:58:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:39.341+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:39 smithi144 ceph-mon[32129]: pgmap v7885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:39 smithi160 ceph-mon[36715]: pgmap v7885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:39 smithi183 ceph-mon[37273]: pgmap v7885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:41 smithi144 ceph-mon[32129]: pgmap v7886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:58:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:41 smithi160 ceph-mon[36715]: pgmap v7886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:58:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:41 smithi183 ceph-mon[37273]: pgmap v7886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T14:58:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13964.851824 secs 2024-07-26T14:58:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:42 smithi144 ceph-mon[32129]: pgmap v7887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13964.851824 secs 2024-07-26T14:58:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:42 smithi160 ceph-mon[36715]: pgmap v7887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13964.851824 secs 2024-07-26T14:58:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:42 smithi183 ceph-mon[37273]: pgmap v7887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:44 smithi160 ceph-mon[36715]: pgmap v7888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:44 smithi183 ceph-mon[37273]: pgmap v7888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:44 smithi144 ceph-mon[32129]: pgmap v7888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:46 smithi160 ceph-mon[36715]: pgmap v7889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:46 smithi183 ceph-mon[37273]: pgmap v7889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:46 smithi144 ceph-mon[32129]: pgmap v7889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13969.851954 secs 2024-07-26T14:58:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13969.851954 secs 2024-07-26T14:58:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13969.851954 secs 2024-07-26T14:58:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:48 smithi160 ceph-mon[36715]: pgmap v7890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:48 smithi183 ceph-mon[37273]: pgmap v7890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:48 smithi144 ceph-mon[32129]: pgmap v7890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:58:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:50 smithi160 ceph-mon[36715]: pgmap v7891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:50 smithi183 ceph-mon[37273]: pgmap v7891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:50 smithi144 ceph-mon[32129]: pgmap v7891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:58:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13974.852072 secs 2024-07-26T14:58:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:52 smithi160 ceph-mon[36715]: pgmap v7892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13974.852072 secs 2024-07-26T14:58:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:52 smithi183 ceph-mon[37273]: pgmap v7892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13974.852072 secs 2024-07-26T14:58:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:52 smithi144 ceph-mon[32129]: pgmap v7892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:58:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:53 smithi160 ceph-mon[36715]: pgmap v7893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:58:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:53 smithi160 ceph-mon[36715]: pgmap v7894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:58:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:53 smithi183 ceph-mon[37273]: pgmap v7893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:58:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:53 smithi183 ceph-mon[37273]: pgmap v7894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:58:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:53 smithi144 ceph-mon[32129]: pgmap v7893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T14:58:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:53 smithi144 ceph-mon[32129]: pgmap v7894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:58:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:54.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:58:54.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:54.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:58:54.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:54.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:58:54.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:54.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:58:54.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:58:54.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:55 smithi183 ceph-mon[37273]: pgmap v7895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:55 smithi144 ceph-mon[32129]: pgmap v7895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:55 smithi160 ceph-mon[36715]: pgmap v7895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:58:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13979.852204 secs 2024-07-26T14:58:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:57 smithi144 ceph-mon[32129]: pgmap v7896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13979.852204 secs 2024-07-26T14:58:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:57 smithi160 ceph-mon[36715]: pgmap v7896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13979.852204 secs 2024-07-26T14:58:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:57 smithi183 ceph-mon[37273]: pgmap v7896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:58:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:58:59 smithi144 ceph-mon[32129]: pgmap v7897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:59:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:58:59 smithi160 ceph-mon[36715]: pgmap v7897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:59:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:58:59 smithi183 ceph-mon[37273]: pgmap v7897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:59:01.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:01 smithi183 ceph-mon[37273]: pgmap v7898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:01 smithi144 ceph-mon[32129]: pgmap v7898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:01 smithi160 ceph-mon[36715]: pgmap v7898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13984.852331 secs 2024-07-26T14:59:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13984.852331 secs 2024-07-26T14:59:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13984.852331 secs 2024-07-26T14:59:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:03 smithi144 ceph-mon[32129]: pgmap v7899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:59:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:03 smithi160 ceph-mon[36715]: pgmap v7899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:59:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:03 smithi183 ceph-mon[37273]: pgmap v7899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T14:59:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:05 smithi144 ceph-mon[32129]: pgmap v7900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:05 smithi160 ceph-mon[36715]: pgmap v7900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:05 smithi183 ceph-mon[37273]: pgmap v7900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13989.852413 secs 2024-07-26T14:59:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:07 smithi144 ceph-mon[32129]: pgmap v7901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13989.852413 secs 2024-07-26T14:59:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:07 smithi160 ceph-mon[36715]: pgmap v7901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13989.852413 secs 2024-07-26T14:59:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:07 smithi183 ceph-mon[37273]: pgmap v7901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:09.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:09.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:59:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:09.366+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:59:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:09.366+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:59:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:09.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:59:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:09.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:09 smithi144 ceph-mon[32129]: pgmap v7902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:09 smithi160 ceph-mon[36715]: pgmap v7902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:09 smithi183 ceph-mon[37273]: pgmap v7902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:11 smithi144 ceph-mon[32129]: pgmap v7903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:11 smithi160 ceph-mon[36715]: pgmap v7903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:11 smithi183 ceph-mon[37273]: pgmap v7903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13994.852539 secs 2024-07-26T14:59:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13994.852539 secs 2024-07-26T14:59:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13994.852539 secs 2024-07-26T14:59:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:13 smithi144 ceph-mon[32129]: pgmap v7904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:13 smithi160 ceph-mon[36715]: pgmap v7904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:13 smithi183 ceph-mon[37273]: pgmap v7904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:15 smithi160 ceph-mon[36715]: pgmap v7905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:15 smithi183 ceph-mon[37273]: pgmap v7905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:15 smithi144 ceph-mon[32129]: pgmap v7905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 13999.852674 secs 2024-07-26T14:59:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:17 smithi160 ceph-mon[36715]: pgmap v7906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 13999.852674 secs 2024-07-26T14:59:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:17 smithi183 ceph-mon[37273]: pgmap v7906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 13999.852674 secs 2024-07-26T14:59:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:17 smithi144 ceph-mon[32129]: pgmap v7906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:19 smithi160 ceph-mon[36715]: pgmap v7907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:19 smithi183 ceph-mon[37273]: pgmap v7907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:19 smithi144 ceph-mon[32129]: pgmap v7907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:21 smithi160 ceph-mon[36715]: pgmap v7908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14004.852807 secs 2024-07-26T14:59:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:21 smithi183 ceph-mon[37273]: pgmap v7908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14004.852807 secs 2024-07-26T14:59:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:21 smithi144 ceph-mon[32129]: pgmap v7908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14004.852807 secs 2024-07-26T14:59:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:23 smithi160 ceph-mon[36715]: pgmap v7909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:23 smithi183 ceph-mon[37273]: pgmap v7909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:23 smithi144 ceph-mon[32129]: pgmap v7909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:24.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:59:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:24.383+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:59:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:24.383+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:59:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:24.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:59:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:24.383+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:25 smithi160 ceph-mon[36715]: pgmap v7910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:25 smithi183 ceph-mon[37273]: pgmap v7910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:25 smithi144 ceph-mon[32129]: pgmap v7910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14009.852931 secs 2024-07-26T14:59:27.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14009.852931 secs 2024-07-26T14:59:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14009.852931 secs 2024-07-26T14:59:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:27 smithi144 ceph-mon[32129]: pgmap v7911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:27 smithi160 ceph-mon[36715]: pgmap v7911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:27 smithi183 ceph-mon[37273]: pgmap v7911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:29 smithi144 ceph-mon[32129]: pgmap v7912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:29 smithi160 ceph-mon[36715]: pgmap v7912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:29 smithi183 ceph-mon[37273]: pgmap v7912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:32.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:31 smithi183 ceph-mon[37273]: pgmap v7913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:32.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14014.853073 secs 2024-07-26T14:59:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:31 smithi144 ceph-mon[32129]: pgmap v7913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14014.853073 secs 2024-07-26T14:59:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:31 smithi160 ceph-mon[36715]: pgmap v7913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14014.853073 secs 2024-07-26T14:59:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:33 smithi144 ceph-mon[32129]: pgmap v7914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:33 smithi160 ceph-mon[36715]: pgmap v7914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:33 smithi183 ceph-mon[37273]: pgmap v7914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:35 smithi144 ceph-mon[32129]: pgmap v7915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:35 smithi160 ceph-mon[36715]: pgmap v7915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:35 smithi183 ceph-mon[37273]: pgmap v7915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14019.853200 secs 2024-07-26T14:59:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14019.853200 secs 2024-07-26T14:59:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14019.853200 secs 2024-07-26T14:59:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:37 smithi144 ceph-mon[32129]: pgmap v7916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:37 smithi160 ceph-mon[36715]: pgmap v7916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:37 smithi183 ceph-mon[37273]: pgmap v7916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:39.389+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:59:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:39.390+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:59:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:39.390+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:59:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:39.390+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:59:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:39.390+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:39 smithi144 ceph-mon[32129]: pgmap v7917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:39 smithi160 ceph-mon[36715]: pgmap v7917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:39 smithi183 ceph-mon[37273]: pgmap v7917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:41 smithi160 ceph-mon[36715]: pgmap v7918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14024.853341 secs 2024-07-26T14:59:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:41 smithi183 ceph-mon[37273]: pgmap v7918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14024.853341 secs 2024-07-26T14:59:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:41 smithi144 ceph-mon[32129]: pgmap v7918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14024.853341 secs 2024-07-26T14:59:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:43 smithi160 ceph-mon[36715]: pgmap v7919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:43 smithi183 ceph-mon[37273]: pgmap v7919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:43 smithi144 ceph-mon[32129]: pgmap v7919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:45 smithi160 ceph-mon[36715]: pgmap v7920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:45 smithi183 ceph-mon[37273]: pgmap v7920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:45 smithi144 ceph-mon[32129]: pgmap v7920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14029.853410 secs 2024-07-26T14:59:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14029.853410 secs 2024-07-26T14:59:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14029.853410 secs 2024-07-26T14:59:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:48 smithi160 ceph-mon[36715]: pgmap v7921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:48 smithi183 ceph-mon[37273]: pgmap v7921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:48 smithi144 ceph-mon[32129]: pgmap v7921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T14:59:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:48.580+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:59:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:48.580+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T14:59:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:48.581+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T14:59:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:48.581+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d020 con 0x563c6c4182f0 2024-07-26T14:59:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:50 smithi160 ceph-mon[36715]: pgmap v7922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:50 smithi183 ceph-mon[37273]: pgmap v7922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:50 smithi144 ceph-mon[32129]: pgmap v7922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:52 smithi144 ceph-mon[32129]: pgmap v7923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14034.853530 secs 2024-07-26T14:59:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:52 smithi160 ceph-mon[36715]: pgmap v7923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14034.853530 secs 2024-07-26T14:59:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:52 smithi183 ceph-mon[37273]: pgmap v7923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T14:59:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14034.853530 secs 2024-07-26T14:59:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:54 smithi160 ceph-mon[36715]: pgmap v7924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:59:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:54 smithi183 ceph-mon[37273]: pgmap v7924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:59:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:54.406+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:59:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:54.407+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T14:59:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:54.407+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T14:59:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:54.407+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:59:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T14:59:54.407+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:54 smithi144 ceph-mon[32129]: pgmap v7924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T14:59:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T14:59:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:55 smithi160 ceph-mon[36715]: pgmap v7925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:55 smithi160 ceph-mon[36715]: pgmap v7926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:59:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:55 smithi183 ceph-mon[37273]: pgmap v7925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:55 smithi183 ceph-mon[37273]: pgmap v7926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:59:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:55 smithi144 ceph-mon[32129]: pgmap v7925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:55 smithi144 ceph-mon[32129]: pgmap v7926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T14:59:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:57 smithi160 ceph-mon[36715]: pgmap v7927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14039.853614 secs 2024-07-26T14:59:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:57 smithi183 ceph-mon[37273]: pgmap v7927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14039.853614 secs 2024-07-26T14:59:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:57 smithi144 ceph-mon[32129]: pgmap v7927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T14:59:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14039.853614 secs 2024-07-26T14:59:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 14:59:59 smithi160 ceph-mon[36715]: pgmap v7928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:59:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 14:59:59 smithi183 ceph-mon[37273]: pgmap v7928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T14:59:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 14:59:59 smithi144 ceph-mon[32129]: pgmap v7928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:00:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:00:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:00:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:01 smithi183 ceph-mon[37273]: pgmap v7929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:01 smithi144 ceph-mon[32129]: pgmap v7929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:01 smithi160 ceph-mon[36715]: pgmap v7929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14044.853743 secs 2024-07-26T15:00:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14044.853743 secs 2024-07-26T15:00:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14044.853743 secs 2024-07-26T15:00:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:03 smithi183 ceph-mon[37273]: pgmap v7930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:00:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:03 smithi144 ceph-mon[32129]: pgmap v7930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:00:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:03 smithi160 ceph-mon[36715]: pgmap v7930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:00:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:05 smithi183 ceph-mon[37273]: pgmap v7931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:00:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:05 smithi144 ceph-mon[32129]: pgmap v7931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:00:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:05 smithi160 ceph-mon[36715]: pgmap v7931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:00:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:07 smithi183 ceph-mon[37273]: pgmap v7932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14049.853862 secs 2024-07-26T15:00:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:07 smithi144 ceph-mon[32129]: pgmap v7932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14049.853862 secs 2024-07-26T15:00:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:07 smithi160 ceph-mon[36715]: pgmap v7932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14049.853862 secs 2024-07-26T15:00:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:09 smithi183 ceph-mon[37273]: pgmap v7933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:09.423+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:00:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:09.424+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:00:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:09.424+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:00:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:09.424+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:00:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:09.424+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:09 smithi144 ceph-mon[32129]: pgmap v7933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:09 smithi160 ceph-mon[36715]: pgmap v7933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:11 smithi183 ceph-mon[37273]: pgmap v7934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:11 smithi144 ceph-mon[32129]: pgmap v7934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:11 smithi160 ceph-mon[36715]: pgmap v7934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14054.853945 secs 2024-07-26T15:00:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14054.853945 secs 2024-07-26T15:00:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14054.853945 secs 2024-07-26T15:00:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:13 smithi183 ceph-mon[37273]: pgmap v7935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:13 smithi144 ceph-mon[32129]: pgmap v7935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:13 smithi160 ceph-mon[36715]: pgmap v7935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:15 smithi183 ceph-mon[37273]: pgmap v7936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:15 smithi144 ceph-mon[32129]: pgmap v7936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:15 smithi160 ceph-mon[36715]: pgmap v7936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:17 smithi144 ceph-mon[32129]: pgmap v7937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14059.854086 secs 2024-07-26T15:00:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:17 smithi160 ceph-mon[36715]: pgmap v7937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14059.854086 secs 2024-07-26T15:00:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:17 smithi183 ceph-mon[37273]: pgmap v7937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14059.854086 secs 2024-07-26T15:00:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:19 smithi144 ceph-mon[32129]: pgmap v7938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:19 smithi160 ceph-mon[36715]: pgmap v7938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:19 smithi183 ceph-mon[37273]: pgmap v7938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:21 smithi144 ceph-mon[32129]: pgmap v7939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:21 smithi160 ceph-mon[36715]: pgmap v7939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:21 smithi183 ceph-mon[37273]: pgmap v7939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14064.854209 secs 2024-07-26T15:00:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14064.854209 secs 2024-07-26T15:00:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14064.854209 secs 2024-07-26T15:00:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:23 smithi144 ceph-mon[32129]: pgmap v7940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:23 smithi160 ceph-mon[36715]: pgmap v7940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:23 smithi183 ceph-mon[37273]: pgmap v7940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:24.429+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:00:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:24.430+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:00:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:24.430+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:00:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:24.430+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:00:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:24.430+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:25 smithi144 ceph-mon[32129]: pgmap v7941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:25 smithi160 ceph-mon[36715]: pgmap v7941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:25 smithi183 ceph-mon[37273]: pgmap v7941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:27 smithi144 ceph-mon[32129]: pgmap v7942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14069.854333 secs 2024-07-26T15:00:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:27 smithi160 ceph-mon[36715]: pgmap v7942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14069.854333 secs 2024-07-26T15:00:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:27 smithi183 ceph-mon[37273]: pgmap v7942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14069.854333 secs 2024-07-26T15:00:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:29 smithi144 ceph-mon[32129]: pgmap v7943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:29 smithi160 ceph-mon[36715]: pgmap v7943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:29 smithi183 ceph-mon[37273]: pgmap v7943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:31 smithi144 ceph-mon[32129]: pgmap v7944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:31 smithi160 ceph-mon[36715]: pgmap v7944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:31 smithi183 ceph-mon[37273]: pgmap v7944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14074.854427 secs 2024-07-26T15:00:32.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14074.854427 secs 2024-07-26T15:00:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14074.854427 secs 2024-07-26T15:00:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:33 smithi144 ceph-mon[32129]: pgmap v7945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:33 smithi160 ceph-mon[36715]: pgmap v7945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:33 smithi183 ceph-mon[37273]: pgmap v7945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:34 smithi144 ceph-mon[32129]: pgmap v7946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:34 smithi160 ceph-mon[36715]: pgmap v7946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:34 smithi183 ceph-mon[37273]: pgmap v7946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:36 smithi183 ceph-mon[37273]: pgmap v7947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14079.854558 secs 2024-07-26T15:00:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:36 smithi144 ceph-mon[32129]: pgmap v7947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14079.854558 secs 2024-07-26T15:00:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:36 smithi160 ceph-mon[36715]: pgmap v7947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14079.854558 secs 2024-07-26T15:00:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:38 smithi183 ceph-mon[37273]: pgmap v7948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:38 smithi144 ceph-mon[32129]: pgmap v7948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:38 smithi160 ceph-mon[36715]: pgmap v7948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:39.446+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:00:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:39.447+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:00:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:39.447+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:00:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:39.447+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:00:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:39.447+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:40 smithi183 ceph-mon[37273]: pgmap v7949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:40 smithi144 ceph-mon[32129]: pgmap v7949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:40 smithi160 ceph-mon[36715]: pgmap v7949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14084.854683 secs 2024-07-26T15:00:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14084.854683 secs 2024-07-26T15:00:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14084.854683 secs 2024-07-26T15:00:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:42 smithi183 ceph-mon[37273]: pgmap v7950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:42 smithi144 ceph-mon[32129]: pgmap v7950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:42 smithi160 ceph-mon[36715]: pgmap v7950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:44 smithi183 ceph-mon[37273]: pgmap v7951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:44 smithi144 ceph-mon[32129]: pgmap v7951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:44 smithi160 ceph-mon[36715]: pgmap v7951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:46 smithi183 ceph-mon[37273]: pgmap v7952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14089.854747 secs 2024-07-26T15:00:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:46 smithi144 ceph-mon[32129]: pgmap v7952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14089.854747 secs 2024-07-26T15:00:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:46 smithi160 ceph-mon[36715]: pgmap v7952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14089.854747 secs 2024-07-26T15:00:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:48 smithi183 ceph-mon[37273]: pgmap v7953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:48 smithi144 ceph-mon[32129]: pgmap v7953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:48 smithi160 ceph-mon[36715]: pgmap v7953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:00:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:50 smithi144 ceph-mon[32129]: pgmap v7954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:50 smithi160 ceph-mon[36715]: pgmap v7954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:50 smithi183 ceph-mon[37273]: pgmap v7954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14094.854874 secs 2024-07-26T15:00:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14094.854874 secs 2024-07-26T15:00:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14094.854874 secs 2024-07-26T15:00:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:52 smithi144 ceph-mon[32129]: pgmap v7955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:52 smithi160 ceph-mon[36715]: pgmap v7955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:52 smithi183 ceph-mon[37273]: pgmap v7955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:00:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:54.462+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:00:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:54.463+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:00:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:54.463+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:00:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:54.463+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:00:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:00:54.463+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:54 smithi144 ceph-mon[32129]: pgmap v7956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:54 smithi160 ceph-mon[36715]: pgmap v7956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:54 smithi183 ceph-mon[37273]: pgmap v7956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:00:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:55 smithi144 ceph-mon[32129]: pgmap v7957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:00:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:55 smithi144 ceph-mon[32129]: pgmap v7958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:00:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:55 smithi160 ceph-mon[36715]: pgmap v7957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:00:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:55 smithi160 ceph-mon[36715]: pgmap v7958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:00:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:55 smithi183 ceph-mon[37273]: pgmap v7957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:00:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:55 smithi183 ceph-mon[37273]: pgmap v7958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:00:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:00:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14099.854970 secs 2024-07-26T15:00:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14099.854970 secs 2024-07-26T15:00:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14099.854970 secs 2024-07-26T15:00:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:00:57 smithi144 ceph-mon[32129]: pgmap v7959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:00:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:00:57 smithi160 ceph-mon[36715]: pgmap v7959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:00:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:00:57 smithi183 ceph-mon[37273]: pgmap v7959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:00 smithi183 ceph-mon[37273]: pgmap v7960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:01:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:00 smithi144 ceph-mon[32129]: pgmap v7960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:01:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:00 smithi160 ceph-mon[36715]: pgmap v7960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:01:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14104.855091 secs 2024-07-26T15:01:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:02 smithi183 ceph-mon[37273]: pgmap v7961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14104.855091 secs 2024-07-26T15:01:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:02 smithi144 ceph-mon[32129]: pgmap v7961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14104.855091 secs 2024-07-26T15:01:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:02 smithi160 ceph-mon[36715]: pgmap v7961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:04 smithi183 ceph-mon[37273]: pgmap v7962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:01:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:04 smithi144 ceph-mon[32129]: pgmap v7962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:01:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:04 smithi160 ceph-mon[36715]: pgmap v7962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:01:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:06 smithi183 ceph-mon[37273]: pgmap v7963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:01:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:06 smithi144 ceph-mon[32129]: pgmap v7963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:01:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:06 smithi160 ceph-mon[36715]: pgmap v7963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:01:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14109.855210 secs 2024-07-26T15:01:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14109.855210 secs 2024-07-26T15:01:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14109.855210 secs 2024-07-26T15:01:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:08 smithi144 ceph-mon[32129]: pgmap v7964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:08 smithi160 ceph-mon[36715]: pgmap v7964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:08 smithi183 ceph-mon[37273]: pgmap v7964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:09.478+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:01:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:09.479+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:01:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:09.479+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:01:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:09.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:01:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:09.480+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:10 smithi144 ceph-mon[32129]: pgmap v7965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:10 smithi160 ceph-mon[36715]: pgmap v7965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:10 smithi183 ceph-mon[37273]: pgmap v7965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14114.855306 secs 2024-07-26T15:01:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:12 smithi144 ceph-mon[32129]: pgmap v7966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14114.855306 secs 2024-07-26T15:01:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:12 smithi160 ceph-mon[36715]: pgmap v7966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14114.855306 secs 2024-07-26T15:01:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:12 smithi183 ceph-mon[37273]: pgmap v7966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:14 smithi144 ceph-mon[32129]: pgmap v7967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:14 smithi160 ceph-mon[36715]: pgmap v7967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:14 smithi183 ceph-mon[37273]: pgmap v7967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:16 smithi144 ceph-mon[32129]: pgmap v7968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:16 smithi160 ceph-mon[36715]: pgmap v7968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:16 smithi183 ceph-mon[37273]: pgmap v7968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:17.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14119.855424 secs 2024-07-26T15:01:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14119.855424 secs 2024-07-26T15:01:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14119.855424 secs 2024-07-26T15:01:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:18 smithi144 ceph-mon[32129]: pgmap v7969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:18 smithi160 ceph-mon[36715]: pgmap v7969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:18 smithi183 ceph-mon[37273]: pgmap v7969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:20 smithi144 ceph-mon[32129]: pgmap v7970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:20 smithi160 ceph-mon[36715]: pgmap v7970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:20 smithi183 ceph-mon[37273]: pgmap v7970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14124.855565 secs 2024-07-26T15:01:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:22 smithi144 ceph-mon[32129]: pgmap v7971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14124.855565 secs 2024-07-26T15:01:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:22 smithi160 ceph-mon[36715]: pgmap v7971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14124.855565 secs 2024-07-26T15:01:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:22 smithi183 ceph-mon[37273]: pgmap v7971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:24 smithi144 ceph-mon[32129]: pgmap v7972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:24.496 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:24.495+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:01:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:24.496+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:01:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:24.496+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:01:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:24.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:01:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:24.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:24 smithi160 ceph-mon[36715]: pgmap v7972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:24 smithi183 ceph-mon[37273]: pgmap v7972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:26 smithi144 ceph-mon[32129]: pgmap v7973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:26 smithi160 ceph-mon[36715]: pgmap v7973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:26 smithi183 ceph-mon[37273]: pgmap v7973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14129.855662 secs 2024-07-26T15:01:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14129.855662 secs 2024-07-26T15:01:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14129.855662 secs 2024-07-26T15:01:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:28 smithi144 ceph-mon[32129]: pgmap v7974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:28 smithi160 ceph-mon[36715]: pgmap v7974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:28 smithi183 ceph-mon[37273]: pgmap v7974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:29 smithi160 ceph-mon[36715]: pgmap v7975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:29 smithi183 ceph-mon[37273]: pgmap v7975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:29 smithi144 ceph-mon[32129]: pgmap v7975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14134.855802 secs 2024-07-26T15:01:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:32 smithi183 ceph-mon[37273]: pgmap v7976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14134.855802 secs 2024-07-26T15:01:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:32 smithi144 ceph-mon[32129]: pgmap v7976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14134.855802 secs 2024-07-26T15:01:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:32 smithi160 ceph-mon[36715]: pgmap v7976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:34 smithi183 ceph-mon[37273]: pgmap v7977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:34 smithi144 ceph-mon[32129]: pgmap v7977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:34 smithi160 ceph-mon[36715]: pgmap v7977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:36 smithi144 ceph-mon[32129]: pgmap v7978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:36 smithi160 ceph-mon[36715]: pgmap v7978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:36 smithi183 ceph-mon[37273]: pgmap v7978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14139.855924 secs 2024-07-26T15:01:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14139.855924 secs 2024-07-26T15:01:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14139.855924 secs 2024-07-26T15:01:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:38 smithi144 ceph-mon[32129]: pgmap v7979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:38 smithi160 ceph-mon[36715]: pgmap v7979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:38 smithi183 ceph-mon[37273]: pgmap v7979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:39.501+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:01:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:39.502+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:01:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:39.502+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:01:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:39.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:01:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:39.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:40 smithi144 ceph-mon[32129]: pgmap v7980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:40 smithi160 ceph-mon[36715]: pgmap v7980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:40 smithi183 ceph-mon[37273]: pgmap v7980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14144.856045 secs 2024-07-26T15:01:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:42 smithi144 ceph-mon[32129]: pgmap v7981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14144.856045 secs 2024-07-26T15:01:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:42 smithi160 ceph-mon[36715]: pgmap v7981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14144.856045 secs 2024-07-26T15:01:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:42 smithi183 ceph-mon[37273]: pgmap v7981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:44 smithi144 ceph-mon[32129]: pgmap v7982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:44 smithi160 ceph-mon[36715]: pgmap v7982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:44 smithi183 ceph-mon[37273]: pgmap v7982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:46 smithi144 ceph-mon[32129]: pgmap v7983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:46 smithi160 ceph-mon[36715]: pgmap v7983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:46 smithi183 ceph-mon[37273]: pgmap v7983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:47.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14149.856183 secs 2024-07-26T15:01:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14149.856183 secs 2024-07-26T15:01:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14149.856183 secs 2024-07-26T15:01:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:48 smithi144 ceph-mon[32129]: pgmap v7984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:48 smithi160 ceph-mon[36715]: pgmap v7984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:48 smithi183 ceph-mon[37273]: pgmap v7984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:01:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:50 smithi144 ceph-mon[32129]: pgmap v7985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:50 smithi160 ceph-mon[36715]: pgmap v7985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:50 smithi183 ceph-mon[37273]: pgmap v7985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14154.856324 secs 2024-07-26T15:01:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:52 smithi144 ceph-mon[32129]: pgmap v7986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14154.856324 secs 2024-07-26T15:01:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:52 smithi160 ceph-mon[36715]: pgmap v7986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14154.856324 secs 2024-07-26T15:01:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:52 smithi183 ceph-mon[37273]: pgmap v7986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:54 smithi144 ceph-mon[32129]: pgmap v7987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:54.517+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:01:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:54.518+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:01:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:54.518+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:01:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:54.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:01:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:01:54.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:54 smithi160 ceph-mon[36715]: pgmap v7987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:54 smithi183 ceph-mon[37273]: pgmap v7987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:01:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:56 smithi144 ceph-mon[32129]: pgmap v7988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:56 smithi160 ceph-mon[36715]: pgmap v7988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:56 smithi183 ceph-mon[37273]: pgmap v7988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:01:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14159.856413 secs 2024-07-26T15:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:57 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:57 smithi160 ceph-mon[36715]: pgmap v7989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:57 smithi160 ceph-mon[36715]: pgmap v7990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:57 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:01:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14159.856413 secs 2024-07-26T15:01:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:57 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:57 smithi183 ceph-mon[37273]: pgmap v7989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:57 smithi183 ceph-mon[37273]: pgmap v7990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:57 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14159.856413 secs 2024-07-26T15:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:57 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:57 smithi144 ceph-mon[32129]: pgmap v7989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:57 smithi144 ceph-mon[32129]: pgmap v7990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:01:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:57 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:01:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:01:59 smithi160 ceph-mon[36715]: pgmap v7991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:01:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:01:59 smithi183 ceph-mon[37273]: pgmap v7991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:01:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:01:59 smithi144 ceph-mon[32129]: pgmap v7991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:02:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:01 smithi160 ceph-mon[36715]: pgmap v7992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:02:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:01 smithi183 ceph-mon[37273]: pgmap v7992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:02:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:01 smithi144 ceph-mon[32129]: pgmap v7992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:02:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14164.856512 secs 2024-07-26T15:02:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14164.856512 secs 2024-07-26T15:02:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14164.856512 secs 2024-07-26T15:02:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:03 smithi160 ceph-mon[36715]: pgmap v7993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:02:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:03 smithi183 ceph-mon[37273]: pgmap v7993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:02:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:03 smithi144 ceph-mon[32129]: pgmap v7993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:02:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:05 smithi183 ceph-mon[37273]: pgmap v7994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:02:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:05 smithi144 ceph-mon[32129]: pgmap v7994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:02:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:05 smithi160 ceph-mon[36715]: pgmap v7994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:02:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14169.856634 secs 2024-07-26T15:02:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:07 smithi183 ceph-mon[37273]: pgmap v7995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:02:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14169.856634 secs 2024-07-26T15:02:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:07 smithi144 ceph-mon[32129]: pgmap v7995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:02:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14169.856634 secs 2024-07-26T15:02:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:07 smithi160 ceph-mon[36715]: pgmap v7995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:02:09.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:09.522+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:02:09.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:09.523+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:02:09.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:09.523+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:02:09.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:09.523+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:02:09.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:09.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:09 smithi183 ceph-mon[37273]: pgmap v7996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:09 smithi144 ceph-mon[32129]: pgmap v7996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:09 smithi160 ceph-mon[36715]: pgmap v7996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:11 smithi183 ceph-mon[37273]: pgmap v7997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:11 smithi144 ceph-mon[32129]: pgmap v7997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:11 smithi160 ceph-mon[36715]: pgmap v7997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14174.856761 secs 2024-07-26T15:02:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14174.856761 secs 2024-07-26T15:02:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14174.856761 secs 2024-07-26T15:02:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:13 smithi183 ceph-mon[37273]: pgmap v7998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:13 smithi144 ceph-mon[32129]: pgmap v7998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:13 smithi160 ceph-mon[36715]: pgmap v7998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:15 smithi183 ceph-mon[37273]: pgmap v7999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:15 smithi144 ceph-mon[32129]: pgmap v7999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:15 smithi160 ceph-mon[36715]: pgmap v7999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:17.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14179.856874 secs 2024-07-26T15:02:17.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:17 smithi160 ceph-mon[36715]: pgmap v8000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14179.856874 secs 2024-07-26T15:02:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:17 smithi183 ceph-mon[37273]: pgmap v8000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14179.856874 secs 2024-07-26T15:02:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:17 smithi144 ceph-mon[32129]: pgmap v8000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:19 smithi183 ceph-mon[37273]: pgmap v8001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:19 smithi144 ceph-mon[32129]: pgmap v8001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:19 smithi160 ceph-mon[36715]: pgmap v8001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:21 smithi183 ceph-mon[37273]: pgmap v8002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:21 smithi144 ceph-mon[32129]: pgmap v8002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:21 smithi160 ceph-mon[36715]: pgmap v8002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14184.856997 secs 2024-07-26T15:02:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14184.856997 secs 2024-07-26T15:02:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14184.856997 secs 2024-07-26T15:02:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:23 smithi183 ceph-mon[37273]: pgmap v8003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:23 smithi144 ceph-mon[32129]: pgmap v8003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:23 smithi160 ceph-mon[36715]: pgmap v8003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:24.539+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:02:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:24.540+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:02:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:24.540+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:02:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:24.540+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:02:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:24.540+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:25 smithi183 ceph-mon[37273]: pgmap v8004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:25 smithi144 ceph-mon[32129]: pgmap v8004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:25 smithi160 ceph-mon[36715]: pgmap v8004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14189.857147 secs 2024-07-26T15:02:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:27 smithi144 ceph-mon[32129]: pgmap v8005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14189.857147 secs 2024-07-26T15:02:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:27 smithi160 ceph-mon[36715]: pgmap v8005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14189.857147 secs 2024-07-26T15:02:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:27 smithi183 ceph-mon[37273]: pgmap v8005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:29 smithi144 ceph-mon[32129]: pgmap v8006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:29 smithi160 ceph-mon[36715]: pgmap v8006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:29 smithi183 ceph-mon[37273]: pgmap v8006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:31 smithi144 ceph-mon[32129]: pgmap v8007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:31 smithi160 ceph-mon[36715]: pgmap v8007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:31 smithi183 ceph-mon[37273]: pgmap v8007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14194.857263 secs 2024-07-26T15:02:32.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14194.857263 secs 2024-07-26T15:02:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14194.857263 secs 2024-07-26T15:02:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:33 smithi144 ceph-mon[32129]: pgmap v8008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:33 smithi160 ceph-mon[36715]: pgmap v8008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:33 smithi183 ceph-mon[37273]: pgmap v8008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:35 smithi144 ceph-mon[32129]: pgmap v8009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:35 smithi160 ceph-mon[36715]: pgmap v8009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:35 smithi183 ceph-mon[37273]: pgmap v8009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14199.857380 secs 2024-07-26T15:02:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:37 smithi144 ceph-mon[32129]: pgmap v8010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14199.857380 secs 2024-07-26T15:02:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:37 smithi160 ceph-mon[36715]: pgmap v8010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14199.857380 secs 2024-07-26T15:02:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:37 smithi183 ceph-mon[37273]: pgmap v8010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:39.555+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:02:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:39.556+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:02:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:39.556+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:02:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:39.556+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:02:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:39.556+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:39 smithi144 ceph-mon[32129]: pgmap v8011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:39 smithi160 ceph-mon[36715]: pgmap v8011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:39 smithi183 ceph-mon[37273]: pgmap v8011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:41 smithi144 ceph-mon[32129]: pgmap v8012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:41 smithi160 ceph-mon[36715]: pgmap v8012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:41 smithi183 ceph-mon[37273]: pgmap v8012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14204.857496 secs 2024-07-26T15:02:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14204.857496 secs 2024-07-26T15:02:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14204.857496 secs 2024-07-26T15:02:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:43 smithi144 ceph-mon[32129]: pgmap v8013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:43 smithi160 ceph-mon[36715]: pgmap v8013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:43 smithi183 ceph-mon[37273]: pgmap v8013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:45 smithi160 ceph-mon[36715]: pgmap v8014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:45 smithi183 ceph-mon[37273]: pgmap v8014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:45 smithi144 ceph-mon[32129]: pgmap v8014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14209.857590 secs 2024-07-26T15:02:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:47 smithi160 ceph-mon[36715]: pgmap v8015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14209.857590 secs 2024-07-26T15:02:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:47 smithi183 ceph-mon[37273]: pgmap v8015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14209.857590 secs 2024-07-26T15:02:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:47 smithi144 ceph-mon[32129]: pgmap v8015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:48.613+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:02:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:48.613+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478004740 con 0x563c6c4182f0 2024-07-26T15:02:48.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:48.614+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:02:48.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:48.614+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:02:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:49 smithi160 ceph-mon[36715]: pgmap v8016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:49 smithi183 ceph-mon[37273]: pgmap v8016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:49 smithi144 ceph-mon[32129]: pgmap v8016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:51 smithi160 ceph-mon[36715]: pgmap v8017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:51 smithi183 ceph-mon[37273]: pgmap v8017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:51 smithi144 ceph-mon[32129]: pgmap v8017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14214.857659 secs 2024-07-26T15:02:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14214.857659 secs 2024-07-26T15:02:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14214.857659 secs 2024-07-26T15:02:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:53 smithi160 ceph-mon[36715]: pgmap v8018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:53 smithi183 ceph-mon[37273]: pgmap v8018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:53 smithi144 ceph-mon[32129]: pgmap v8018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:54.572+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:02:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:54.573+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:02:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:54.573+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:02:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:54.573+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:02:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:02:54.573+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:55 smithi160 ceph-mon[36715]: pgmap v8019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:55.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:55 smithi183 ceph-mon[37273]: pgmap v8019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:55 smithi144 ceph-mon[32129]: pgmap v8019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:02:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:56 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:56 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:56 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:57.693 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14219.857780 secs 2024-07-26T15:02:57.694 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:57 smithi160 ceph-mon[36715]: pgmap v8020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:57.870 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14219.857780 secs 2024-07-26T15:02:57.870 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:57 smithi144 ceph-mon[32129]: pgmap v8020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14219.857780 secs 2024-07-26T15:02:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:57 smithi183 ceph-mon[37273]: pgmap v8020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:02:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:02:59 smithi144 ceph-mon[32129]: pgmap v8021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:02:59 smithi160 ceph-mon[36715]: pgmap v8021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:02:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:02:59 smithi183 ceph-mon[37273]: pgmap v8021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:02 smithi144 ceph-mon[32129]: pgmap v8022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14224.857875 secs 2024-07-26T15:03:02.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:02 smithi183 ceph-mon[37273]: pgmap v8022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:02.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14224.857875 secs 2024-07-26T15:03:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:02 smithi160 ceph-mon[36715]: pgmap v8022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14224.857875 secs 2024-07-26T15:03:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: pgmap v8023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:03:03.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:03:03.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:03.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:03.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: pgmap v8024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:03:03.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: pgmap v8025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T15:03:03.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:03 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: pgmap v8023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: pgmap v8024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: pgmap v8025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T15:03:03.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:03 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: pgmap v8023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: pgmap v8024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: pgmap v8025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T15:03:04.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:03 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:03:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:05 smithi160 ceph-mon[36715]: pgmap v8026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:03:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:05 smithi183 ceph-mon[37273]: pgmap v8026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:03:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:05 smithi144 ceph-mon[32129]: pgmap v8026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:03:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14229.857992 secs 2024-07-26T15:03:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:07 smithi160 ceph-mon[36715]: pgmap v8027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14229.857992 secs 2024-07-26T15:03:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:07 smithi183 ceph-mon[37273]: pgmap v8027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14229.857992 secs 2024-07-26T15:03:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:07 smithi144 ceph-mon[32129]: pgmap v8027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:09.589+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:03:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:09.590+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:03:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:09.590+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:03:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:09.590+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:03:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:09.590+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:09 smithi160 ceph-mon[36715]: pgmap v8028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:09 smithi183 ceph-mon[37273]: pgmap v8028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:09 smithi144 ceph-mon[32129]: pgmap v8028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:11 smithi160 ceph-mon[36715]: pgmap v8029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:11 smithi183 ceph-mon[37273]: pgmap v8029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:11 smithi144 ceph-mon[32129]: pgmap v8029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:03:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14234.858140 secs 2024-07-26T15:03:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14234.858140 secs 2024-07-26T15:03:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14234.858140 secs 2024-07-26T15:03:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:13 smithi160 ceph-mon[36715]: pgmap v8030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:03:14.077 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:13 smithi144 ceph-mon[32129]: pgmap v8030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:03:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:13 smithi183 ceph-mon[37273]: pgmap v8030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:03:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:15 smithi160 ceph-mon[36715]: pgmap v8031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:15 smithi183 ceph-mon[37273]: pgmap v8031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:15 smithi144 ceph-mon[32129]: pgmap v8031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14239.858262 secs 2024-07-26T15:03:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14239.858262 secs 2024-07-26T15:03:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14239.858262 secs 2024-07-26T15:03:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:17 smithi160 ceph-mon[36715]: pgmap v8032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:17 smithi183 ceph-mon[37273]: pgmap v8032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:17 smithi144 ceph-mon[32129]: pgmap v8032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:19 smithi160 ceph-mon[36715]: pgmap v8033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:19 smithi183 ceph-mon[37273]: pgmap v8033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:19 smithi144 ceph-mon[32129]: pgmap v8033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:21 smithi160 ceph-mon[36715]: pgmap v8034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14244.858378 secs 2024-07-26T15:03:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:21 smithi183 ceph-mon[37273]: pgmap v8034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14244.858378 secs 2024-07-26T15:03:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:21 smithi144 ceph-mon[32129]: pgmap v8034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14244.858378 secs 2024-07-26T15:03:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:23 smithi183 ceph-mon[37273]: pgmap v8035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:23 smithi144 ceph-mon[32129]: pgmap v8035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:23 smithi160 ceph-mon[36715]: pgmap v8035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:24.591+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:03:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:24.592+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:03:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:24.592+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:03:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:24.592+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:03:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:24.592+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:25 smithi183 ceph-mon[37273]: pgmap v8036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:25 smithi144 ceph-mon[32129]: pgmap v8036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:25 smithi160 ceph-mon[36715]: pgmap v8036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:27.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14249.858495 secs 2024-07-26T15:03:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14249.858495 secs 2024-07-26T15:03:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14249.858495 secs 2024-07-26T15:03:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:28 smithi160 ceph-mon[36715]: pgmap v8037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:28 smithi183 ceph-mon[37273]: pgmap v8037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:28 smithi144 ceph-mon[32129]: pgmap v8037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:30 smithi144 ceph-mon[32129]: pgmap v8038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:30 smithi160 ceph-mon[36715]: pgmap v8038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:30 smithi183 ceph-mon[37273]: pgmap v8038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:32 smithi160 ceph-mon[36715]: pgmap v8039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14254.858641 secs 2024-07-26T15:03:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:32 smithi183 ceph-mon[37273]: pgmap v8039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14254.858641 secs 2024-07-26T15:03:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:32 smithi144 ceph-mon[32129]: pgmap v8039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14254.858641 secs 2024-07-26T15:03:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:34 smithi160 ceph-mon[36715]: pgmap v8040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:34 smithi183 ceph-mon[37273]: pgmap v8040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:34 smithi144 ceph-mon[32129]: pgmap v8040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:36 smithi183 ceph-mon[37273]: pgmap v8041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:36 smithi144 ceph-mon[32129]: pgmap v8041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:36 smithi160 ceph-mon[36715]: pgmap v8041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14259.858789 secs 2024-07-26T15:03:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14259.858789 secs 2024-07-26T15:03:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14259.858789 secs 2024-07-26T15:03:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:38 smithi183 ceph-mon[37273]: pgmap v8042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:38 smithi144 ceph-mon[32129]: pgmap v8042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:38 smithi160 ceph-mon[36715]: pgmap v8042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:39.609 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:39.608+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:03:39.609 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:39.609+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:03:39.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:39.609+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:03:39.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:39.609+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:03:39.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:39.609+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:40 smithi183 ceph-mon[37273]: pgmap v8043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:40 smithi144 ceph-mon[32129]: pgmap v8043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:40 smithi160 ceph-mon[36715]: pgmap v8043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:42 smithi183 ceph-mon[37273]: pgmap v8044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14264.858908 secs 2024-07-26T15:03:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:42 smithi144 ceph-mon[32129]: pgmap v8044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14264.858908 secs 2024-07-26T15:03:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:42 smithi160 ceph-mon[36715]: pgmap v8044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14264.858908 secs 2024-07-26T15:03:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:44 smithi183 ceph-mon[37273]: pgmap v8045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:44 smithi144 ceph-mon[32129]: pgmap v8045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:44 smithi160 ceph-mon[36715]: pgmap v8045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:46 smithi183 ceph-mon[37273]: pgmap v8046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:46 smithi144 ceph-mon[32129]: pgmap v8046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:46 smithi160 ceph-mon[36715]: pgmap v8046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14269.859029 secs 2024-07-26T15:03:47.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14269.859029 secs 2024-07-26T15:03:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14269.859029 secs 2024-07-26T15:03:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:48 smithi183 ceph-mon[37273]: pgmap v8047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:48 smithi144 ceph-mon[32129]: pgmap v8047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:48 smithi160 ceph-mon[36715]: pgmap v8047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:50 smithi183 ceph-mon[37273]: pgmap v8048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:50 smithi144 ceph-mon[32129]: pgmap v8048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:50 smithi160 ceph-mon[36715]: pgmap v8048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:52 smithi183 ceph-mon[37273]: pgmap v8049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14274.859154 secs 2024-07-26T15:03:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:52 smithi144 ceph-mon[32129]: pgmap v8049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14274.859154 secs 2024-07-26T15:03:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:52 smithi160 ceph-mon[36715]: pgmap v8049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14274.859154 secs 2024-07-26T15:03:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:54 smithi183 ceph-mon[37273]: pgmap v8050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:54 smithi144 ceph-mon[32129]: pgmap v8050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:54 smithi160 ceph-mon[36715]: pgmap v8050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:03:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:54.624+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:03:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:54.625+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:03:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:54.625+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:03:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:54.625+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:03:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:03:54.625+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:56 smithi144 ceph-mon[32129]: pgmap v8051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:56 smithi160 ceph-mon[36715]: pgmap v8051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:56 smithi183 ceph-mon[37273]: pgmap v8051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:03:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14279.859273 secs 2024-07-26T15:03:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14279.859273 secs 2024-07-26T15:03:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14279.859273 secs 2024-07-26T15:03:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:03:58 smithi144 ceph-mon[32129]: pgmap v8052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:03:58 smithi160 ceph-mon[36715]: pgmap v8052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:03:58.647 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:03:58 smithi183 ceph-mon[37273]: pgmap v8052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:00 smithi144 ceph-mon[32129]: pgmap v8053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:00 smithi160 ceph-mon[36715]: pgmap v8053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:00 smithi183 ceph-mon[37273]: pgmap v8053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:02.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:02 smithi183 ceph-mon[37273]: pgmap v8054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:02.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14284.859391 secs 2024-07-26T15:04:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:02 smithi144 ceph-mon[32129]: pgmap v8054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14284.859391 secs 2024-07-26T15:04:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:02 smithi160 ceph-mon[36715]: pgmap v8054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14284.859391 secs 2024-07-26T15:04:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: pgmap v8055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: pgmap v8055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: pgmap v8055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:05.362 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:05 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:04:05.362 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:05 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:04:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:05 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:04:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:05 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:04:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:05 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:04:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:05 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:04:06.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:06 smithi183 ceph-mon[37273]: pgmap v8056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:06 smithi144 ceph-mon[32129]: pgmap v8056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:06 smithi160 ceph-mon[36715]: pgmap v8056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14289.859540 secs 2024-07-26T15:04:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: pgmap v8057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: pgmap v8058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: pgmap v8059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:07.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14289.859540 secs 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: pgmap v8057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: pgmap v8058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: pgmap v8059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:08.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14289.859540 secs 2024-07-26T15:04:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: pgmap v8057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: pgmap v8058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: pgmap v8059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:08.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:04:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:09.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:04:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:09.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:04:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:09.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:04:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:09.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:04:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:09.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:09 smithi160 ceph-mon[36715]: pgmap v8060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:09 smithi183 ceph-mon[37273]: pgmap v8060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:09 smithi144 ceph-mon[32129]: pgmap v8060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:11 smithi160 ceph-mon[36715]: pgmap v8061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T15:04:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14294.859657 secs 2024-07-26T15:04:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:11 smithi183 ceph-mon[37273]: pgmap v8061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T15:04:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14294.859657 secs 2024-07-26T15:04:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:11 smithi144 ceph-mon[32129]: pgmap v8061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T15:04:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14294.859657 secs 2024-07-26T15:04:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:13 smithi160 ceph-mon[36715]: pgmap v8062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:13 smithi183 ceph-mon[37273]: pgmap v8062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:13 smithi144 ceph-mon[32129]: pgmap v8062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:04:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:15 smithi160 ceph-mon[36715]: pgmap v8063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T15:04:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:15 smithi183 ceph-mon[37273]: pgmap v8063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T15:04:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:15 smithi144 ceph-mon[32129]: pgmap v8063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T15:04:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14299.859784 secs 2024-07-26T15:04:17.061 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14299.859784 secs 2024-07-26T15:04:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14299.859784 secs 2024-07-26T15:04:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:17 smithi183 ceph-mon[37273]: pgmap v8064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:17 smithi144 ceph-mon[32129]: pgmap v8064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:18.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:17 smithi160 ceph-mon[36715]: pgmap v8064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:19 smithi183 ceph-mon[37273]: pgmap v8065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:19 smithi144 ceph-mon[32129]: pgmap v8065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:19 smithi160 ceph-mon[36715]: pgmap v8065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:21 smithi183 ceph-mon[37273]: pgmap v8066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14304.859850 secs 2024-07-26T15:04:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:21 smithi144 ceph-mon[32129]: pgmap v8066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14304.859850 secs 2024-07-26T15:04:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:21 smithi160 ceph-mon[36715]: pgmap v8066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14304.859850 secs 2024-07-26T15:04:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:23 smithi183 ceph-mon[37273]: pgmap v8067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:23 smithi144 ceph-mon[32129]: pgmap v8067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:23 smithi160 ceph-mon[36715]: pgmap v8067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:24.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:04:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:24.654+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:04:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:24.654+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:04:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:24.654+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:04:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:24.654+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:25 smithi183 ceph-mon[37273]: pgmap v8068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:25 smithi144 ceph-mon[32129]: pgmap v8068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:25 smithi160 ceph-mon[36715]: pgmap v8068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:27.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14309.860011 secs 2024-07-26T15:04:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14309.860011 secs 2024-07-26T15:04:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14309.860011 secs 2024-07-26T15:04:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:27 smithi183 ceph-mon[37273]: pgmap v8069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:27 smithi144 ceph-mon[32129]: pgmap v8069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:27 smithi160 ceph-mon[36715]: pgmap v8069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:29 smithi183 ceph-mon[37273]: pgmap v8070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:29 smithi144 ceph-mon[32129]: pgmap v8070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:29 smithi160 ceph-mon[36715]: pgmap v8070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:31 smithi183 ceph-mon[37273]: pgmap v8071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14314.860152 secs 2024-07-26T15:04:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:31 smithi144 ceph-mon[32129]: pgmap v8071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14314.860152 secs 2024-07-26T15:04:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:31 smithi160 ceph-mon[36715]: pgmap v8071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14314.860152 secs 2024-07-26T15:04:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:33 smithi183 ceph-mon[37273]: pgmap v8072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:04:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:33 smithi144 ceph-mon[32129]: pgmap v8072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:04:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:33 smithi160 ceph-mon[36715]: pgmap v8072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:04:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:35 smithi183 ceph-mon[37273]: pgmap v8073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:35 smithi144 ceph-mon[32129]: pgmap v8073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:35 smithi160 ceph-mon[36715]: pgmap v8073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14319.860260 secs 2024-07-26T15:04:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14319.860260 secs 2024-07-26T15:04:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14319.860260 secs 2024-07-26T15:04:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:37 smithi144 ceph-mon[32129]: pgmap v8074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:37 smithi160 ceph-mon[36715]: pgmap v8074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:37 smithi183 ceph-mon[37273]: pgmap v8074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:04:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:39.670+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:04:39.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:39.671+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:04:39.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:39.671+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:04:39.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:39.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:04:39.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:39.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:39 smithi144 ceph-mon[32129]: pgmap v8075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:39 smithi160 ceph-mon[36715]: pgmap v8075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:39 smithi183 ceph-mon[37273]: pgmap v8075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:41 smithi144 ceph-mon[32129]: pgmap v8076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14324.860385 secs 2024-07-26T15:04:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:41 smithi160 ceph-mon[36715]: pgmap v8076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14324.860385 secs 2024-07-26T15:04:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:41 smithi183 ceph-mon[37273]: pgmap v8076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:04:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14324.860385 secs 2024-07-26T15:04:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:43 smithi144 ceph-mon[32129]: pgmap v8077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:44.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:43 smithi160 ceph-mon[36715]: pgmap v8077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:43 smithi183 ceph-mon[37273]: pgmap v8077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:46.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:45 smithi183 ceph-mon[37273]: pgmap v8078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:45 smithi144 ceph-mon[32129]: pgmap v8078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:45 smithi160 ceph-mon[36715]: pgmap v8078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:47.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14329.860511 secs 2024-07-26T15:04:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14329.860511 secs 2024-07-26T15:04:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14329.860511 secs 2024-07-26T15:04:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:47 smithi144 ceph-mon[32129]: pgmap v8079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:47 smithi160 ceph-mon[36715]: pgmap v8079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:47 smithi183 ceph-mon[37273]: pgmap v8079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:49 smithi144 ceph-mon[32129]: pgmap v8080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:49 smithi160 ceph-mon[36715]: pgmap v8080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:49 smithi183 ceph-mon[37273]: pgmap v8080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:51 smithi144 ceph-mon[32129]: pgmap v8081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14334.860595 secs 2024-07-26T15:04:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:51 smithi160 ceph-mon[36715]: pgmap v8081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14334.860595 secs 2024-07-26T15:04:52.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:51 smithi183 ceph-mon[37273]: pgmap v8081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:52.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14334.860595 secs 2024-07-26T15:04:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:53 smithi144 ceph-mon[32129]: pgmap v8082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:53 smithi160 ceph-mon[36715]: pgmap v8082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:53 smithi183 ceph-mon[37273]: pgmap v8082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:04:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:54.686+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:04:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:54.687+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:04:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:54.687+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:04:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:54.687+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:04:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:04:54.687+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:55 smithi144 ceph-mon[32129]: pgmap v8083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:55 smithi160 ceph-mon[36715]: pgmap v8083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:55 smithi183 ceph-mon[37273]: pgmap v8083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:04:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14339.860723 secs 2024-07-26T15:04:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14339.860723 secs 2024-07-26T15:04:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14339.860723 secs 2024-07-26T15:04:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:57 smithi160 ceph-mon[36715]: pgmap v8084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:57 smithi183 ceph-mon[37273]: pgmap v8084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:04:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:57 smithi144 ceph-mon[32129]: pgmap v8084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:04:59 smithi160 ceph-mon[36715]: pgmap v8085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:04:59 smithi183 ceph-mon[37273]: pgmap v8085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:04:59 smithi144 ceph-mon[32129]: pgmap v8085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:01 smithi160 ceph-mon[36715]: pgmap v8086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14344.860845 secs 2024-07-26T15:05:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:01 smithi183 ceph-mon[37273]: pgmap v8086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14344.860845 secs 2024-07-26T15:05:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:01 smithi144 ceph-mon[32129]: pgmap v8086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14344.860845 secs 2024-07-26T15:05:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:03.637+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:05:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:03.637+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:05:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:03.637+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:05:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:03.637+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:05:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:03 smithi160 ceph-mon[36715]: pgmap v8087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:03 smithi183 ceph-mon[37273]: pgmap v8087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:03 smithi144 ceph-mon[32129]: pgmap v8087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:05 smithi160 ceph-mon[36715]: pgmap v8088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:05:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:05 smithi183 ceph-mon[37273]: pgmap v8088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:05:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:05 smithi144 ceph-mon[32129]: pgmap v8088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:05:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14349.860959 secs 2024-07-26T15:05:07.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14349.860959 secs 2024-07-26T15:05:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14349.860959 secs 2024-07-26T15:05:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:08 smithi160 ceph-mon[36715]: pgmap v8089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:05:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:08 smithi183 ceph-mon[37273]: pgmap v8089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:05:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:08 smithi144 ceph-mon[32129]: pgmap v8089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:05:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:09 smithi160 ceph-mon[36715]: pgmap v8090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:09 smithi160 ceph-mon[36715]: pgmap v8091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:09 smithi183 ceph-mon[37273]: pgmap v8090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:09 smithi183 ceph-mon[37273]: pgmap v8091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:09 smithi144 ceph-mon[32129]: pgmap v8090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:09 smithi144 ceph-mon[32129]: pgmap v8091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:09.703+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:05:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:09.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:05:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:09.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:05:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:09.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:05:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:09.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:11 smithi160 ceph-mon[36715]: pgmap v8092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:11 smithi183 ceph-mon[37273]: pgmap v8092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:11 smithi144 ceph-mon[32129]: pgmap v8092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:05:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14354.861083 secs 2024-07-26T15:05:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14354.861083 secs 2024-07-26T15:05:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14354.861083 secs 2024-07-26T15:05:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:13 smithi160 ceph-mon[36715]: pgmap v8093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:05:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:13 smithi183 ceph-mon[37273]: pgmap v8093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:05:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:13 smithi144 ceph-mon[32129]: pgmap v8093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:05:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:15 smithi160 ceph-mon[36715]: pgmap v8094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:05:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:15 smithi183 ceph-mon[37273]: pgmap v8094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:05:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:15 smithi144 ceph-mon[32129]: pgmap v8094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:05:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14359.861198 secs 2024-07-26T15:05:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:17 smithi160 ceph-mon[36715]: pgmap v8095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:05:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14359.861198 secs 2024-07-26T15:05:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:17 smithi183 ceph-mon[37273]: pgmap v8095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:05:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14359.861198 secs 2024-07-26T15:05:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:17 smithi144 ceph-mon[32129]: pgmap v8095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:05:19.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:19 smithi144 ceph-mon[32129]: pgmap v8096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:19 smithi160 ceph-mon[36715]: pgmap v8096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:19 smithi183 ceph-mon[37273]: pgmap v8096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:05:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:21 smithi160 ceph-mon[36715]: pgmap v8097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:21 smithi183 ceph-mon[37273]: pgmap v8097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:21 smithi144 ceph-mon[32129]: pgmap v8097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14364.861337 secs 2024-07-26T15:05:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14364.861337 secs 2024-07-26T15:05:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14364.861337 secs 2024-07-26T15:05:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:23 smithi183 ceph-mon[37273]: pgmap v8098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:23 smithi144 ceph-mon[32129]: pgmap v8098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:23 smithi160 ceph-mon[36715]: pgmap v8098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:24.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:05:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:24.720+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:05:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:24.720+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:05:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:24.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:05:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:24.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:25 smithi183 ceph-mon[37273]: pgmap v8099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:25 smithi144 ceph-mon[32129]: pgmap v8099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:25 smithi160 ceph-mon[36715]: pgmap v8099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14369.861474 secs 2024-07-26T15:05:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:27 smithi183 ceph-mon[37273]: pgmap v8100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14369.861474 secs 2024-07-26T15:05:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:27 smithi144 ceph-mon[32129]: pgmap v8100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14369.861474 secs 2024-07-26T15:05:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:27 smithi160 ceph-mon[36715]: pgmap v8100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:29 smithi183 ceph-mon[37273]: pgmap v8101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:29 smithi144 ceph-mon[32129]: pgmap v8101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:29 smithi160 ceph-mon[36715]: pgmap v8101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:31 smithi183 ceph-mon[37273]: pgmap v8102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:31 smithi144 ceph-mon[32129]: pgmap v8102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:31 smithi160 ceph-mon[36715]: pgmap v8102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14374.861575 secs 2024-07-26T15:05:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14374.861575 secs 2024-07-26T15:05:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14374.861575 secs 2024-07-26T15:05:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:33 smithi183 ceph-mon[37273]: pgmap v8103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:33 smithi144 ceph-mon[32129]: pgmap v8103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:33 smithi160 ceph-mon[36715]: pgmap v8103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:35 smithi183 ceph-mon[37273]: pgmap v8104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:35 smithi144 ceph-mon[32129]: pgmap v8104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:35 smithi160 ceph-mon[36715]: pgmap v8104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14379.861720 secs 2024-07-26T15:05:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:37 smithi183 ceph-mon[37273]: pgmap v8105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14379.861720 secs 2024-07-26T15:05:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:37 smithi144 ceph-mon[32129]: pgmap v8105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14379.861720 secs 2024-07-26T15:05:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:37 smithi160 ceph-mon[36715]: pgmap v8105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:39 smithi183 ceph-mon[37273]: pgmap v8106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:39 smithi144 ceph-mon[32129]: pgmap v8106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:39 smithi160 ceph-mon[36715]: pgmap v8106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:39.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:05:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:39.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:05:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:39.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:05:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:39.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:05:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:39.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:41 smithi183 ceph-mon[37273]: pgmap v8107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:41 smithi144 ceph-mon[32129]: pgmap v8107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:41 smithi160 ceph-mon[36715]: pgmap v8107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14384.861842 secs 2024-07-26T15:05:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14384.861842 secs 2024-07-26T15:05:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14384.861842 secs 2024-07-26T15:05:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:43 smithi144 ceph-mon[32129]: pgmap v8108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:43 smithi160 ceph-mon[36715]: pgmap v8108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:43 smithi183 ceph-mon[37273]: pgmap v8108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:45 smithi144 ceph-mon[32129]: pgmap v8109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:45 smithi160 ceph-mon[36715]: pgmap v8109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:45 smithi183 ceph-mon[37273]: pgmap v8109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.144+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000811) for more than 900000000 us, fault. 2024-07-26T15:05:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.144+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a806c070 comp rx=0 tx=0).stop 2024-07-26T15:05:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.144+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:05:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.144+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:05:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.144+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:05:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.144+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:05:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:46.146+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f74a8002230 tx=0x7f74a803e040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:05:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14389.861987 secs 2024-07-26T15:05:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:47 smithi144 ceph-mon[32129]: pgmap v8110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14389.861987 secs 2024-07-26T15:05:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:47 smithi160 ceph-mon[36715]: pgmap v8110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14389.861987 secs 2024-07-26T15:05:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:47 smithi183 ceph-mon[37273]: pgmap v8110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:48.645+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:05:48.646 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:48.645+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:05:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:48.646+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:05:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:48.646+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d020 con 0x563c6c4182f0 2024-07-26T15:05:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:49 smithi144 ceph-mon[32129]: pgmap v8111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:49 smithi160 ceph-mon[36715]: pgmap v8111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:49 smithi183 ceph-mon[37273]: pgmap v8111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:51 smithi144 ceph-mon[32129]: pgmap v8112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:51 smithi160 ceph-mon[36715]: pgmap v8112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:51 smithi183 ceph-mon[37273]: pgmap v8112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14394.862130 secs 2024-07-26T15:05:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14394.862130 secs 2024-07-26T15:05:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14394.862130 secs 2024-07-26T15:05:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:53 smithi144 ceph-mon[32129]: pgmap v8113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:53 smithi160 ceph-mon[36715]: pgmap v8113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:53 smithi183 ceph-mon[37273]: pgmap v8113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:05:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:54.752+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:54.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:54.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:54.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:05:54.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:55 smithi144 ceph-mon[32129]: pgmap v8114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:55 smithi160 ceph-mon[36715]: pgmap v8114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:55 smithi183 ceph-mon[37273]: pgmap v8114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14399.862242 secs 2024-07-26T15:05:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:57 smithi144 ceph-mon[32129]: pgmap v8115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14399.862242 secs 2024-07-26T15:05:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:57 smithi160 ceph-mon[36715]: pgmap v8115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14399.862242 secs 2024-07-26T15:05:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:57 smithi183 ceph-mon[37273]: pgmap v8115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:05:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:05:59 smithi144 ceph-mon[32129]: pgmap v8116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:05:59 smithi160 ceph-mon[36715]: pgmap v8116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:05:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:05:59 smithi183 ceph-mon[37273]: pgmap v8116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:01 smithi144 ceph-mon[32129]: pgmap v8117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:01 smithi160 ceph-mon[36715]: pgmap v8117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:01 smithi183 ceph-mon[37273]: pgmap v8117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14404.862360 secs 2024-07-26T15:06:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14404.862360 secs 2024-07-26T15:06:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14404.862360 secs 2024-07-26T15:06:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:03 smithi160 ceph-mon[36715]: pgmap v8118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:03 smithi183 ceph-mon[37273]: pgmap v8118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:03 smithi144 ceph-mon[32129]: pgmap v8118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:05 smithi144 ceph-mon[32129]: pgmap v8119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:05 smithi160 ceph-mon[36715]: pgmap v8119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:05 smithi183 ceph-mon[37273]: pgmap v8119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14409.862425 secs 2024-07-26T15:06:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:07 smithi160 ceph-mon[36715]: pgmap v8120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14409.862425 secs 2024-07-26T15:06:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:07 smithi183 ceph-mon[37273]: pgmap v8120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14409.862425 secs 2024-07-26T15:06:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:07 smithi144 ceph-mon[32129]: pgmap v8120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:08.718 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:09 smithi160 ceph-mon[36715]: pgmap v8121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:09 smithi183 ceph-mon[37273]: pgmap v8121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:09 smithi144 ceph-mon[32129]: pgmap v8121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:09.764+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:06:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:09.765+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:06:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:09.765+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:06:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:09.765+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:06:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:09.765+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:10 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:10 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:10 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:06:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:11 smithi160 ceph-mon[36715]: pgmap v8122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:11 smithi160 ceph-mon[36715]: pgmap v8123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:11 smithi183 ceph-mon[37273]: pgmap v8122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:11 smithi183 ceph-mon[37273]: pgmap v8123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:11 smithi144 ceph-mon[32129]: pgmap v8122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:11 smithi144 ceph-mon[32129]: pgmap v8123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14414.862547 secs 2024-07-26T15:06:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14414.862547 secs 2024-07-26T15:06:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14414.862547 secs 2024-07-26T15:06:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:13 smithi160 ceph-mon[36715]: pgmap v8124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:13 smithi183 ceph-mon[37273]: pgmap v8124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:13 smithi144 ceph-mon[32129]: pgmap v8124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:14 smithi160 ceph-mon[36715]: pgmap v8125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:14 smithi183 ceph-mon[37273]: pgmap v8125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:14 smithi144 ceph-mon[32129]: pgmap v8125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:16 smithi183 ceph-mon[37273]: pgmap v8126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14419.862671 secs 2024-07-26T15:06:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:16 smithi144 ceph-mon[32129]: pgmap v8126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14419.862671 secs 2024-07-26T15:06:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:16 smithi160 ceph-mon[36715]: pgmap v8126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14419.862671 secs 2024-07-26T15:06:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:18 smithi183 ceph-mon[37273]: pgmap v8127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:18 smithi144 ceph-mon[32129]: pgmap v8127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:18 smithi160 ceph-mon[36715]: pgmap v8127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:06:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:20 smithi183 ceph-mon[37273]: pgmap v8128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:06:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:20 smithi144 ceph-mon[32129]: pgmap v8128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:06:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:20 smithi160 ceph-mon[36715]: pgmap v8128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:06:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14424.862755 secs 2024-07-26T15:06:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14424.862755 secs 2024-07-26T15:06:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14424.862755 secs 2024-07-26T15:06:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:22 smithi144 ceph-mon[32129]: pgmap v8129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:22 smithi183 ceph-mon[37273]: pgmap v8129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:22 smithi160 ceph-mon[36715]: pgmap v8129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:24.768+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:06:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:24.769+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:06:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:24.769+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:06:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:24.769+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:06:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:24.769+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:24 smithi183 ceph-mon[37273]: pgmap v8130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:24 smithi144 ceph-mon[32129]: pgmap v8130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:24 smithi160 ceph-mon[36715]: pgmap v8130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:26 smithi183 ceph-mon[37273]: pgmap v8131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14429.862897 secs 2024-07-26T15:06:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:26 smithi144 ceph-mon[32129]: pgmap v8131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14429.862897 secs 2024-07-26T15:06:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:26 smithi160 ceph-mon[36715]: pgmap v8131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14429.862897 secs 2024-07-26T15:06:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:28 smithi183 ceph-mon[37273]: pgmap v8132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:28 smithi144 ceph-mon[32129]: pgmap v8132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:28 smithi160 ceph-mon[36715]: pgmap v8132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:30 smithi183 ceph-mon[37273]: pgmap v8133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:30 smithi144 ceph-mon[32129]: pgmap v8133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:30 smithi160 ceph-mon[36715]: pgmap v8133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14434.863020 secs 2024-07-26T15:06:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14434.863020 secs 2024-07-26T15:06:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14434.863020 secs 2024-07-26T15:06:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:32 smithi183 ceph-mon[37273]: pgmap v8134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:32 smithi144 ceph-mon[32129]: pgmap v8134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:32 smithi160 ceph-mon[36715]: pgmap v8134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:34 smithi144 ceph-mon[32129]: pgmap v8135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:34 smithi160 ceph-mon[36715]: pgmap v8135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:34 smithi183 ceph-mon[37273]: pgmap v8135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:37.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:36 smithi183 ceph-mon[37273]: pgmap v8136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:37.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14439.863156 secs 2024-07-26T15:06:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:36 smithi144 ceph-mon[32129]: pgmap v8136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14439.863156 secs 2024-07-26T15:06:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:36 smithi160 ceph-mon[36715]: pgmap v8136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14439.863156 secs 2024-07-26T15:06:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:38 smithi144 ceph-mon[32129]: pgmap v8137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:38 smithi160 ceph-mon[36715]: pgmap v8137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:38 smithi183 ceph-mon[37273]: pgmap v8137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:39.782+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:06:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:39.783+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:06:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:39.783+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:06:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:39.783+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:06:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:39.783+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:40 smithi144 ceph-mon[32129]: pgmap v8138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:40 smithi160 ceph-mon[36715]: pgmap v8138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:40 smithi183 ceph-mon[37273]: pgmap v8138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14444.863317 secs 2024-07-26T15:06:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14444.863317 secs 2024-07-26T15:06:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14444.863317 secs 2024-07-26T15:06:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:42 smithi144 ceph-mon[32129]: pgmap v8139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:43.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:42 smithi160 ceph-mon[36715]: pgmap v8139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:42 smithi183 ceph-mon[37273]: pgmap v8139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:44 smithi144 ceph-mon[32129]: pgmap v8140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:45.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:44 smithi160 ceph-mon[36715]: pgmap v8140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:44 smithi183 ceph-mon[37273]: pgmap v8140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:46 smithi144 ceph-mon[32129]: pgmap v8141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14449.863451 secs 2024-07-26T15:06:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:46 smithi160 ceph-mon[36715]: pgmap v8141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14449.863451 secs 2024-07-26T15:06:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:46 smithi183 ceph-mon[37273]: pgmap v8141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14449.863451 secs 2024-07-26T15:06:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:48 smithi144 ceph-mon[32129]: pgmap v8142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:48 smithi160 ceph-mon[36715]: pgmap v8142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:48 smithi183 ceph-mon[37273]: pgmap v8142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:50 smithi144 ceph-mon[32129]: pgmap v8143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:50 smithi160 ceph-mon[36715]: pgmap v8143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:50 smithi183 ceph-mon[37273]: pgmap v8143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14454.863575 secs 2024-07-26T15:06:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14454.863575 secs 2024-07-26T15:06:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14454.863575 secs 2024-07-26T15:06:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:52 smithi144 ceph-mon[32129]: pgmap v8144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:53.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:52 smithi160 ceph-mon[36715]: pgmap v8144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:52 smithi183 ceph-mon[37273]: pgmap v8144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:06:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:54.798+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:06:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:54.799+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:06:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:54.799+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:06:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:54.799+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:06:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:06:54.799+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:54 smithi144 ceph-mon[32129]: pgmap v8145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:54 smithi160 ceph-mon[36715]: pgmap v8145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:54 smithi183 ceph-mon[37273]: pgmap v8145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:56 smithi144 ceph-mon[32129]: pgmap v8146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14459.863650 secs 2024-07-26T15:06:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:56 smithi160 ceph-mon[36715]: pgmap v8146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14459.863650 secs 2024-07-26T15:06:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:56 smithi183 ceph-mon[37273]: pgmap v8146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:06:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14459.863650 secs 2024-07-26T15:06:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:06:58 smithi160 ceph-mon[36715]: pgmap v8147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:06:58 smithi183 ceph-mon[37273]: pgmap v8147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:06:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:06:58 smithi144 ceph-mon[32129]: pgmap v8147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:00 smithi160 ceph-mon[36715]: pgmap v8148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:00 smithi183 ceph-mon[37273]: pgmap v8148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:00 smithi144 ceph-mon[32129]: pgmap v8148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14464.863786 secs 2024-07-26T15:07:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14464.863786 secs 2024-07-26T15:07:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14464.863786 secs 2024-07-26T15:07:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:02 smithi160 ceph-mon[36715]: pgmap v8149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:02 smithi183 ceph-mon[37273]: pgmap v8149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:02 smithi144 ceph-mon[32129]: pgmap v8149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:04 smithi160 ceph-mon[36715]: pgmap v8150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:04 smithi183 ceph-mon[37273]: pgmap v8150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:04 smithi144 ceph-mon[32129]: pgmap v8150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:07 smithi160 ceph-mon[36715]: pgmap v8151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14469.863925 secs 2024-07-26T15:07:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:07 smithi183 ceph-mon[37273]: pgmap v8151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14469.863925 secs 2024-07-26T15:07:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:07 smithi144 ceph-mon[32129]: pgmap v8151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14469.863925 secs 2024-07-26T15:07:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:09 smithi160 ceph-mon[36715]: pgmap v8152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:09 smithi183 ceph-mon[37273]: pgmap v8152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:09 smithi144 ceph-mon[32129]: pgmap v8152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:09.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:09.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:07:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:09.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:07:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:09.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:07:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:09.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:07:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:09.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:10 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:10 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:10 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:11 smithi160 ceph-mon[36715]: pgmap v8153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:11 smithi183 ceph-mon[37273]: pgmap v8153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:11 smithi144 ceph-mon[32129]: pgmap v8153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14474.864065 secs 2024-07-26T15:07:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14474.864065 secs 2024-07-26T15:07:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14474.864065 secs 2024-07-26T15:07:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:13 smithi160 ceph-mon[36715]: pgmap v8154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:13 smithi183 ceph-mon[37273]: pgmap v8154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:13 smithi144 ceph-mon[32129]: pgmap v8154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:14 smithi160 ceph-mon[36715]: pgmap v8155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:14 smithi183 ceph-mon[37273]: pgmap v8155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:14 smithi144 ceph-mon[32129]: pgmap v8155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:15 smithi160 ceph-mon[36715]: pgmap v8156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:07:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:15 smithi160 ceph-mon[36715]: pgmap v8157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:15 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:15 smithi183 ceph-mon[37273]: pgmap v8156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:07:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:15 smithi183 ceph-mon[37273]: pgmap v8157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:15 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:15 smithi144 ceph-mon[32129]: pgmap v8156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:07:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:15 smithi144 ceph-mon[32129]: pgmap v8157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:15 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:07:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14479.864191 secs 2024-07-26T15:07:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14479.864191 secs 2024-07-26T15:07:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14479.864191 secs 2024-07-26T15:07:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:17 smithi183 ceph-mon[37273]: pgmap v8158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:07:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:17 smithi144 ceph-mon[32129]: pgmap v8158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:07:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:17 smithi160 ceph-mon[36715]: pgmap v8158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:07:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:19 smithi144 ceph-mon[32129]: pgmap v8159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:07:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:19 smithi160 ceph-mon[36715]: pgmap v8159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:07:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:19 smithi183 ceph-mon[37273]: pgmap v8159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:21 smithi144 ceph-mon[32129]: pgmap v8160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:07:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14484.864328 secs 2024-07-26T15:07:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:21 smithi160 ceph-mon[36715]: pgmap v8160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:07:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14484.864328 secs 2024-07-26T15:07:22.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:21 smithi183 ceph-mon[37273]: pgmap v8160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:07:22.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14484.864328 secs 2024-07-26T15:07:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:23 smithi144 ceph-mon[32129]: pgmap v8161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:23 smithi160 ceph-mon[36715]: pgmap v8161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:23 smithi183 ceph-mon[37273]: pgmap v8161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:24.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:07:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:24.824+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:07:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:24.824+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:07:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:24.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:07:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:24.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:25 smithi144 ceph-mon[32129]: pgmap v8162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:07:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:25 smithi160 ceph-mon[36715]: pgmap v8162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:07:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:25 smithi183 ceph-mon[37273]: pgmap v8162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:07:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14489.864431 secs 2024-07-26T15:07:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14489.864431 secs 2024-07-26T15:07:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14489.864431 secs 2024-07-26T15:07:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:28 smithi144 ceph-mon[32129]: pgmap v8163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:28 smithi160 ceph-mon[36715]: pgmap v8163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:28 smithi183 ceph-mon[37273]: pgmap v8163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:30 smithi144 ceph-mon[32129]: pgmap v8164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:30 smithi160 ceph-mon[36715]: pgmap v8164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:30 smithi183 ceph-mon[37273]: pgmap v8164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:32 smithi144 ceph-mon[32129]: pgmap v8165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14494.864536 secs 2024-07-26T15:07:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:32 smithi160 ceph-mon[36715]: pgmap v8165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14494.864536 secs 2024-07-26T15:07:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:32 smithi183 ceph-mon[37273]: pgmap v8165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14494.864536 secs 2024-07-26T15:07:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:34 smithi144 ceph-mon[32129]: pgmap v8166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:34 smithi160 ceph-mon[36715]: pgmap v8166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:34 smithi183 ceph-mon[37273]: pgmap v8166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:36 smithi144 ceph-mon[32129]: pgmap v8167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:36 smithi160 ceph-mon[36715]: pgmap v8167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:36 smithi183 ceph-mon[37273]: pgmap v8167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14499.864663 secs 2024-07-26T15:07:37.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14499.864663 secs 2024-07-26T15:07:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14499.864663 secs 2024-07-26T15:07:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:38 smithi144 ceph-mon[32129]: pgmap v8168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:38 smithi160 ceph-mon[36715]: pgmap v8168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:38 smithi183 ceph-mon[37273]: pgmap v8168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:39.840+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:07:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:39.841+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:07:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:39.841+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:07:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:39.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:07:39.843 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:39.841+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:40 smithi144 ceph-mon[32129]: pgmap v8169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:40 smithi160 ceph-mon[36715]: pgmap v8169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:40 smithi183 ceph-mon[37273]: pgmap v8169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:42 smithi144 ceph-mon[32129]: pgmap v8170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14504.864788 secs 2024-07-26T15:07:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:42 smithi160 ceph-mon[36715]: pgmap v8170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14504.864788 secs 2024-07-26T15:07:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:42 smithi183 ceph-mon[37273]: pgmap v8170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14504.864788 secs 2024-07-26T15:07:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:44 smithi144 ceph-mon[32129]: pgmap v8171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:44 smithi160 ceph-mon[36715]: pgmap v8171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:44 smithi183 ceph-mon[37273]: pgmap v8171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:46 smithi144 ceph-mon[32129]: pgmap v8172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:46 smithi160 ceph-mon[36715]: pgmap v8172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:46 smithi183 ceph-mon[37273]: pgmap v8172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14509.864863 secs 2024-07-26T15:07:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14509.864863 secs 2024-07-26T15:07:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14509.864863 secs 2024-07-26T15:07:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:48 smithi160 ceph-mon[36715]: pgmap v8173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:48 smithi183 ceph-mon[37273]: pgmap v8173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:48 smithi144 ceph-mon[32129]: pgmap v8173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:50 smithi160 ceph-mon[36715]: pgmap v8174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:50 smithi183 ceph-mon[37273]: pgmap v8174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:50 smithi144 ceph-mon[32129]: pgmap v8174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:52 smithi160 ceph-mon[36715]: pgmap v8175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14514.865003 secs 2024-07-26T15:07:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:52 smithi183 ceph-mon[37273]: pgmap v8175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14514.865003 secs 2024-07-26T15:07:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:52 smithi144 ceph-mon[32129]: pgmap v8175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14514.865003 secs 2024-07-26T15:07:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:54 smithi160 ceph-mon[36715]: pgmap v8176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:54 smithi183 ceph-mon[37273]: pgmap v8176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:54 smithi144 ceph-mon[32129]: pgmap v8176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:07:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:54.856+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:07:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:54.856+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:07:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:54.856+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:07:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:54.856+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:07:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:07:54.856+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:56 smithi160 ceph-mon[36715]: pgmap v8177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:56 smithi183 ceph-mon[37273]: pgmap v8177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:56 smithi144 ceph-mon[32129]: pgmap v8177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:07:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14519.865140 secs 2024-07-26T15:07:57.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14519.865140 secs 2024-07-26T15:07:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14519.865140 secs 2024-07-26T15:07:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:07:58 smithi160 ceph-mon[36715]: pgmap v8178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:07:58 smithi183 ceph-mon[37273]: pgmap v8178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:07:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:07:58 smithi144 ceph-mon[32129]: pgmap v8178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:00 smithi160 ceph-mon[36715]: pgmap v8179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:00 smithi183 ceph-mon[37273]: pgmap v8179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:00 smithi144 ceph-mon[32129]: pgmap v8179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:02 smithi160 ceph-mon[36715]: pgmap v8180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14524.865281 secs 2024-07-26T15:08:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:02 smithi183 ceph-mon[37273]: pgmap v8180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14524.865281 secs 2024-07-26T15:08:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:02 smithi144 ceph-mon[32129]: pgmap v8180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14524.865281 secs 2024-07-26T15:08:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:04 smithi160 ceph-mon[36715]: pgmap v8181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:04 smithi183 ceph-mon[37273]: pgmap v8181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:04 smithi144 ceph-mon[32129]: pgmap v8181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:06 smithi160 ceph-mon[36715]: pgmap v8182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:06 smithi183 ceph-mon[37273]: pgmap v8182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:06 smithi144 ceph-mon[32129]: pgmap v8182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14529.865374 secs 2024-07-26T15:08:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14529.865374 secs 2024-07-26T15:08:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14529.865374 secs 2024-07-26T15:08:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:08 smithi160 ceph-mon[36715]: pgmap v8183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:08 smithi183 ceph-mon[37273]: pgmap v8183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:08 smithi144 ceph-mon[32129]: pgmap v8183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:09 smithi183 ceph-mon[37273]: pgmap v8184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:09 smithi144 ceph-mon[32129]: pgmap v8184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:09 smithi160 ceph-mon[36715]: pgmap v8184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:09.861+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:08:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:09.862+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:08:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:09.862+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:08:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:09.862+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:08:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:09.862+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:11 smithi144 ceph-mon[32129]: pgmap v8185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14534.865495 secs 2024-07-26T15:08:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:11 smithi160 ceph-mon[36715]: pgmap v8185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14534.865495 secs 2024-07-26T15:08:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:11 smithi183 ceph-mon[37273]: pgmap v8185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14534.865495 secs 2024-07-26T15:08:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:13 smithi144 ceph-mon[32129]: pgmap v8186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:13 smithi160 ceph-mon[36715]: pgmap v8186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:13 smithi183 ceph-mon[37273]: pgmap v8186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:15 smithi144 ceph-mon[32129]: pgmap v8187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:15 smithi160 ceph-mon[36715]: pgmap v8187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:15 smithi183 ceph-mon[37273]: pgmap v8187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14539.865575 secs 2024-07-26T15:08:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:16 smithi160 ceph-mon[36715]: pgmap v8188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:16 smithi160 ceph-mon[36715]: pgmap v8189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:17.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:08:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14539.865575 secs 2024-07-26T15:08:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:16 smithi144 ceph-mon[32129]: pgmap v8188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:16 smithi144 ceph-mon[32129]: pgmap v8189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:08:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14539.865575 secs 2024-07-26T15:08:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:16 smithi183 ceph-mon[37273]: pgmap v8188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:16 smithi183 ceph-mon[37273]: pgmap v8189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:08:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:19 smithi183 ceph-mon[37273]: pgmap v8190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:08:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:19 smithi144 ceph-mon[32129]: pgmap v8190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:08:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:19 smithi160 ceph-mon[36715]: pgmap v8190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:08:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:21 smithi183 ceph-mon[37273]: pgmap v8191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:08:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:21 smithi144 ceph-mon[32129]: pgmap v8191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:08:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:21 smithi160 ceph-mon[36715]: pgmap v8191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:08:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14544.865698 secs 2024-07-26T15:08:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14544.865698 secs 2024-07-26T15:08:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14544.865698 secs 2024-07-26T15:08:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:23 smithi183 ceph-mon[37273]: pgmap v8192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:23 smithi144 ceph-mon[32129]: pgmap v8192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:23 smithi160 ceph-mon[36715]: pgmap v8192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:08:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:24.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:08:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:24.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:08:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:24.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:08:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:24.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:08:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:24.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:25 smithi144 ceph-mon[32129]: pgmap v8193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:08:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:25 smithi160 ceph-mon[36715]: pgmap v8193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:08:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:25 smithi183 ceph-mon[37273]: pgmap v8193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:08:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14549.865836 secs 2024-07-26T15:08:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:27 smithi144 ceph-mon[32129]: pgmap v8194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:08:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14549.865836 secs 2024-07-26T15:08:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:27 smithi160 ceph-mon[36715]: pgmap v8194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:08:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14549.865836 secs 2024-07-26T15:08:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:27 smithi183 ceph-mon[37273]: pgmap v8194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:08:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:29 smithi144 ceph-mon[32129]: pgmap v8195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:29 smithi160 ceph-mon[36715]: pgmap v8195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:29 smithi183 ceph-mon[37273]: pgmap v8195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:31 smithi144 ceph-mon[32129]: pgmap v8196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:31 smithi160 ceph-mon[36715]: pgmap v8196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:31 smithi183 ceph-mon[37273]: pgmap v8196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14554.865918 secs 2024-07-26T15:08:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14554.865918 secs 2024-07-26T15:08:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14554.865918 secs 2024-07-26T15:08:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:33 smithi144 ceph-mon[32129]: pgmap v8197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:33 smithi160 ceph-mon[36715]: pgmap v8197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:33 smithi183 ceph-mon[37273]: pgmap v8197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:35 smithi144 ceph-mon[32129]: pgmap v8198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:35 smithi160 ceph-mon[36715]: pgmap v8198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:35 smithi183 ceph-mon[37273]: pgmap v8198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:37.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14559.866016 secs 2024-07-26T15:08:37.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:37 smithi183 ceph-mon[37273]: pgmap v8199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14559.866016 secs 2024-07-26T15:08:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:37 smithi144 ceph-mon[32129]: pgmap v8199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14559.866016 secs 2024-07-26T15:08:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:37 smithi160 ceph-mon[36715]: pgmap v8199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:39 smithi144 ceph-mon[32129]: pgmap v8200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:39 smithi160 ceph-mon[36715]: pgmap v8200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:39 smithi183 ceph-mon[37273]: pgmap v8200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:39.882+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:08:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:39.883+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:08:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:39.883+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:08:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:39.883+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:08:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:39.883+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:41 smithi144 ceph-mon[32129]: pgmap v8201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:41 smithi160 ceph-mon[36715]: pgmap v8201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:41 smithi183 ceph-mon[37273]: pgmap v8201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14564.866137 secs 2024-07-26T15:08:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14564.866137 secs 2024-07-26T15:08:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14564.866137 secs 2024-07-26T15:08:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:43 smithi144 ceph-mon[32129]: pgmap v8202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:43 smithi160 ceph-mon[36715]: pgmap v8202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:43 smithi183 ceph-mon[37273]: pgmap v8202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:45 smithi144 ceph-mon[32129]: pgmap v8203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:45 smithi160 ceph-mon[36715]: pgmap v8203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:45 smithi183 ceph-mon[37273]: pgmap v8203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14569.866255 secs 2024-07-26T15:08:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:47 smithi144 ceph-mon[32129]: pgmap v8204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14569.866255 secs 2024-07-26T15:08:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:47 smithi160 ceph-mon[36715]: pgmap v8204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14569.866255 secs 2024-07-26T15:08:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:47 smithi183 ceph-mon[37273]: pgmap v8204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:48.677+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:08:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:48.677+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:08:48.679 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:48.678+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T15:08:48.679 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:48.678+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:08:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:49 smithi160 ceph-mon[36715]: pgmap v8205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:49 smithi183 ceph-mon[37273]: pgmap v8205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:49 smithi144 ceph-mon[32129]: pgmap v8205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:08:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:51 smithi160 ceph-mon[36715]: pgmap v8206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:51 smithi183 ceph-mon[37273]: pgmap v8206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:51 smithi144 ceph-mon[32129]: pgmap v8206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14574.866386 secs 2024-07-26T15:08:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14574.866386 secs 2024-07-26T15:08:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14574.866386 secs 2024-07-26T15:08:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:53 smithi160 ceph-mon[36715]: pgmap v8207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:53 smithi183 ceph-mon[37273]: pgmap v8207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:53 smithi144 ceph-mon[32129]: pgmap v8207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:08:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:54.898+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:54.899+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:54.899+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:54.899+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:08:54.899+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:55 smithi160 ceph-mon[36715]: pgmap v8208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:55 smithi183 ceph-mon[37273]: pgmap v8208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:55 smithi144 ceph-mon[32129]: pgmap v8208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14579.866470 secs 2024-07-26T15:08:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:57 smithi144 ceph-mon[32129]: pgmap v8209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14579.866470 secs 2024-07-26T15:08:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:57 smithi160 ceph-mon[36715]: pgmap v8209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14579.866470 secs 2024-07-26T15:08:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:57 smithi183 ceph-mon[37273]: pgmap v8209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:08:59 smithi160 ceph-mon[36715]: pgmap v8210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:08:59 smithi183 ceph-mon[37273]: pgmap v8210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:08:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:08:59 smithi144 ceph-mon[32129]: pgmap v8210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:01 smithi160 ceph-mon[36715]: pgmap v8211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:01 smithi183 ceph-mon[37273]: pgmap v8211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:01 smithi144 ceph-mon[32129]: pgmap v8211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14584.866588 secs 2024-07-26T15:09:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14584.866588 secs 2024-07-26T15:09:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14584.866588 secs 2024-07-26T15:09:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:03 smithi160 ceph-mon[36715]: pgmap v8212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:03 smithi183 ceph-mon[37273]: pgmap v8212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:03 smithi144 ceph-mon[32129]: pgmap v8212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:05 smithi160 ceph-mon[36715]: pgmap v8213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:05 smithi183 ceph-mon[37273]: pgmap v8213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:05 smithi144 ceph-mon[32129]: pgmap v8213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14589.866650 secs 2024-07-26T15:09:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:07 smithi160 ceph-mon[36715]: pgmap v8214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14589.866650 secs 2024-07-26T15:09:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:07 smithi183 ceph-mon[37273]: pgmap v8214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14589.866650 secs 2024-07-26T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:07 smithi144 ceph-mon[32129]: pgmap v8214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:09 smithi160 ceph-mon[36715]: pgmap v8215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:09 smithi183 ceph-mon[37273]: pgmap v8215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:09 smithi144 ceph-mon[32129]: pgmap v8215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:09.917 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:09.916+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:09:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:09.917+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:09:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:09.917+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:09:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:09.917+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:09:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:09.917+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:11 smithi160 ceph-mon[36715]: pgmap v8216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:11 smithi183 ceph-mon[37273]: pgmap v8216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:11 smithi144 ceph-mon[32129]: pgmap v8216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14594.866779 secs 2024-07-26T15:09:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:12 smithi183 ceph-mon[37273]: pgmap v8217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14594.866779 secs 2024-07-26T15:09:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:12 smithi144 ceph-mon[32129]: pgmap v8217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14594.866779 secs 2024-07-26T15:09:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:12 smithi160 ceph-mon[36715]: pgmap v8217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:15 smithi144 ceph-mon[32129]: pgmap v8218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:15 smithi160 ceph-mon[36715]: pgmap v8218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:15 smithi183 ceph-mon[37273]: pgmap v8218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14599.866903 secs 2024-07-26T15:09:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:17 smithi144 ceph-mon[32129]: pgmap v8219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:17.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14599.866903 secs 2024-07-26T15:09:17.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:17 smithi160 ceph-mon[36715]: pgmap v8219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:17.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14599.866903 secs 2024-07-26T15:09:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:17 smithi183 ceph-mon[37273]: pgmap v8219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:18 smithi144 ceph-mon[32129]: pgmap v8220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:18 smithi144 ceph-mon[32129]: pgmap v8221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:09:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:18 smithi160 ceph-mon[36715]: pgmap v8220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:18 smithi160 ceph-mon[36715]: pgmap v8221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:09:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:18 smithi183 ceph-mon[37273]: pgmap v8220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:18 smithi183 ceph-mon[37273]: pgmap v8221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:09:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:19 smithi183 ceph-mon[37273]: pgmap v8222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:19 smithi144 ceph-mon[32129]: pgmap v8222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:19.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:19 smithi160 ceph-mon[36715]: pgmap v8222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:21 smithi183 ceph-mon[37273]: pgmap v8223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:21 smithi144 ceph-mon[32129]: pgmap v8223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:21 smithi160 ceph-mon[36715]: pgmap v8223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14604.867005 secs 2024-07-26T15:09:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14604.867005 secs 2024-07-26T15:09:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14604.867005 secs 2024-07-26T15:09:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:23 smithi183 ceph-mon[37273]: pgmap v8224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:23 smithi144 ceph-mon[32129]: pgmap v8224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:23 smithi160 ceph-mon[36715]: pgmap v8224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:09:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:24.933+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:09:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:24.934+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:09:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:24.934+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:09:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:24.934+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:09:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:24.934+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:25 smithi183 ceph-mon[37273]: pgmap v8225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:25 smithi144 ceph-mon[32129]: pgmap v8225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:25 smithi160 ceph-mon[36715]: pgmap v8225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:09:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14609.867096 secs 2024-07-26T15:09:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14609.867096 secs 2024-07-26T15:09:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14609.867096 secs 2024-07-26T15:09:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:27 smithi144 ceph-mon[32129]: pgmap v8226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:27 smithi160 ceph-mon[36715]: pgmap v8226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:28.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:27 smithi183 ceph-mon[37273]: pgmap v8226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:09:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:29 smithi183 ceph-mon[37273]: pgmap v8227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:29 smithi144 ceph-mon[32129]: pgmap v8227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:29 smithi160 ceph-mon[36715]: pgmap v8227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:31 smithi183 ceph-mon[37273]: pgmap v8228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:31 smithi144 ceph-mon[32129]: pgmap v8228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:31 smithi160 ceph-mon[36715]: pgmap v8228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14614.867224 secs 2024-07-26T15:09:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14614.867224 secs 2024-07-26T15:09:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14614.867224 secs 2024-07-26T15:09:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:33 smithi183 ceph-mon[37273]: pgmap v8229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:33 smithi144 ceph-mon[32129]: pgmap v8229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:33 smithi160 ceph-mon[36715]: pgmap v8229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:35 smithi183 ceph-mon[37273]: pgmap v8230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:35 smithi144 ceph-mon[32129]: pgmap v8230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:35 smithi160 ceph-mon[36715]: pgmap v8230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14619.867377 secs 2024-07-26T15:09:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14619.867377 secs 2024-07-26T15:09:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14619.867377 secs 2024-07-26T15:09:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:37 smithi144 ceph-mon[32129]: pgmap v8231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:37 smithi160 ceph-mon[36715]: pgmap v8231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:37 smithi183 ceph-mon[37273]: pgmap v8231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:39.637 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:39 smithi183 ceph-mon[37273]: pgmap v8232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:39 smithi144 ceph-mon[32129]: pgmap v8232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:39 smithi160 ceph-mon[36715]: pgmap v8232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:39.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:09:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:39.951+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:09:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:39.951+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:09:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:39.951+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:09:39.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:39.952+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:41 smithi183 ceph-mon[37273]: pgmap v8233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:41 smithi144 ceph-mon[32129]: pgmap v8233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:41 smithi160 ceph-mon[36715]: pgmap v8233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14624.867506 secs 2024-07-26T15:09:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14624.867506 secs 2024-07-26T15:09:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14624.867506 secs 2024-07-26T15:09:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:43 smithi144 ceph-mon[32129]: pgmap v8234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:43 smithi183 ceph-mon[37273]: pgmap v8234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:43 smithi160 ceph-mon[36715]: pgmap v8234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:45 smithi183 ceph-mon[37273]: pgmap v8235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:45 smithi144 ceph-mon[32129]: pgmap v8235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:45 smithi160 ceph-mon[36715]: pgmap v8235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14629.867623 secs 2024-07-26T15:09:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14629.867623 secs 2024-07-26T15:09:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14629.867623 secs 2024-07-26T15:09:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:47 smithi144 ceph-mon[32129]: pgmap v8236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:47 smithi160 ceph-mon[36715]: pgmap v8236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:47 smithi183 ceph-mon[37273]: pgmap v8236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:49 smithi183 ceph-mon[37273]: pgmap v8237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:49 smithi144 ceph-mon[32129]: pgmap v8237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:49 smithi160 ceph-mon[36715]: pgmap v8237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:51 smithi183 ceph-mon[37273]: pgmap v8238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:51 smithi144 ceph-mon[32129]: pgmap v8238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:51 smithi160 ceph-mon[36715]: pgmap v8238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14634.867736 secs 2024-07-26T15:09:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14634.867736 secs 2024-07-26T15:09:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14634.867736 secs 2024-07-26T15:09:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:53 smithi183 ceph-mon[37273]: pgmap v8239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:53 smithi144 ceph-mon[32129]: pgmap v8239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:53 smithi160 ceph-mon[36715]: pgmap v8239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:54.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:54.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:09:54.969 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:54.967+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:09:54.969 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:54.967+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:09:54.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:54.967+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:09:54.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:09:54.968+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:55 smithi183 ceph-mon[37273]: pgmap v8240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:55 smithi144 ceph-mon[32129]: pgmap v8240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:55 smithi160 ceph-mon[36715]: pgmap v8240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:09:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14639.867868 secs 2024-07-26T15:09:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14639.867868 secs 2024-07-26T15:09:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14639.867868 secs 2024-07-26T15:09:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:57 smithi144 ceph-mon[32129]: pgmap v8241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:57 smithi160 ceph-mon[36715]: pgmap v8241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:57 smithi183 ceph-mon[37273]: pgmap v8241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:09:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:09:59 smithi183 ceph-mon[37273]: pgmap v8242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:09:59 smithi144 ceph-mon[32129]: pgmap v8242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:09:59.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:09:59 smithi160 ceph-mon[36715]: pgmap v8242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:10:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:10:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:10:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:01 smithi144 ceph-mon[32129]: pgmap v8243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:01 smithi160 ceph-mon[36715]: pgmap v8243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:01 smithi183 ceph-mon[37273]: pgmap v8243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14644.867986 secs 2024-07-26T15:10:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14644.867986 secs 2024-07-26T15:10:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14644.867986 secs 2024-07-26T15:10:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:03 smithi144 ceph-mon[32129]: pgmap v8244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:03 smithi160 ceph-mon[36715]: pgmap v8244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:03 smithi183 ceph-mon[37273]: pgmap v8244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:05 smithi183 ceph-mon[37273]: pgmap v8245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:05 smithi144 ceph-mon[32129]: pgmap v8245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:05 smithi160 ceph-mon[36715]: pgmap v8245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14649.868135 secs 2024-07-26T15:10:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14649.868135 secs 2024-07-26T15:10:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14649.868135 secs 2024-07-26T15:10:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:08 smithi183 ceph-mon[37273]: pgmap v8246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:08 smithi144 ceph-mon[32129]: pgmap v8246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:08 smithi160 ceph-mon[36715]: pgmap v8246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:09.984+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:10:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:09.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:10:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:09.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:10:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:09.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:10:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:09.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:10 smithi144 ceph-mon[32129]: pgmap v8247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:10 smithi160 ceph-mon[36715]: pgmap v8247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:10 smithi183 ceph-mon[37273]: pgmap v8247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14654.868280 secs 2024-07-26T15:10:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14654.868280 secs 2024-07-26T15:10:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14654.868280 secs 2024-07-26T15:10:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:12 smithi144 ceph-mon[32129]: pgmap v8248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:12 smithi160 ceph-mon[36715]: pgmap v8248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:12 smithi183 ceph-mon[37273]: pgmap v8248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:14 smithi144 ceph-mon[32129]: pgmap v8249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:14 smithi160 ceph-mon[36715]: pgmap v8249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:14 smithi183 ceph-mon[37273]: pgmap v8249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:16 smithi144 ceph-mon[32129]: pgmap v8250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14659.868333 secs 2024-07-26T15:10:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:16 smithi160 ceph-mon[36715]: pgmap v8250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14659.868333 secs 2024-07-26T15:10:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:16 smithi183 ceph-mon[37273]: pgmap v8250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14659.868333 secs 2024-07-26T15:10:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:18 smithi144 ceph-mon[32129]: pgmap v8251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:10:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:18 smithi160 ceph-mon[36715]: pgmap v8251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:10:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:18 smithi183 ceph-mon[37273]: pgmap v8251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:10:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:19 smithi144 ceph-mon[32129]: pgmap v8252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:19 smithi144 ceph-mon[32129]: pgmap v8253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:10:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:19 smithi160 ceph-mon[36715]: pgmap v8252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:19 smithi160 ceph-mon[36715]: pgmap v8253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:10:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:19 smithi183 ceph-mon[37273]: pgmap v8252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:19 smithi183 ceph-mon[37273]: pgmap v8253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:10:22.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:21 smithi144 ceph-mon[32129]: pgmap v8254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:22.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14664.868424 secs 2024-07-26T15:10:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:21 smithi160 ceph-mon[36715]: pgmap v8254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14664.868424 secs 2024-07-26T15:10:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:21 smithi183 ceph-mon[37273]: pgmap v8254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14664.868424 secs 2024-07-26T15:10:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:23 smithi144 ceph-mon[32129]: pgmap v8255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:10:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:23 smithi160 ceph-mon[36715]: pgmap v8255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:10:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:23 smithi183 ceph-mon[37273]: pgmap v8255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:10:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:25.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:10:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:25.003+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:10:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:25.003+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:10:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:25.003+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:10:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:25.003+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:25 smithi144 ceph-mon[32129]: pgmap v8256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:25 smithi160 ceph-mon[36715]: pgmap v8256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:25 smithi183 ceph-mon[37273]: pgmap v8256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14669.868513 secs 2024-07-26T15:10:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14669.868513 secs 2024-07-26T15:10:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14669.868513 secs 2024-07-26T15:10:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:27 smithi160 ceph-mon[36715]: pgmap v8257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:27 smithi183 ceph-mon[37273]: pgmap v8257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:27 smithi144 ceph-mon[32129]: pgmap v8257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:10:30.301 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:29 smithi144 ceph-mon[32129]: pgmap v8258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:10:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:29 smithi160 ceph-mon[36715]: pgmap v8258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:10:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:29 smithi183 ceph-mon[37273]: pgmap v8258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:10:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:31 smithi160 ceph-mon[36715]: pgmap v8259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14674.868589 secs 2024-07-26T15:10:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:31 smithi183 ceph-mon[37273]: pgmap v8259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14674.868589 secs 2024-07-26T15:10:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:31 smithi144 ceph-mon[32129]: pgmap v8259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14674.868589 secs 2024-07-26T15:10:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:33 smithi160 ceph-mon[36715]: pgmap v8260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:33 smithi183 ceph-mon[37273]: pgmap v8260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:33 smithi144 ceph-mon[32129]: pgmap v8260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:35 smithi160 ceph-mon[36715]: pgmap v8261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:35 smithi183 ceph-mon[37273]: pgmap v8261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:35 smithi144 ceph-mon[32129]: pgmap v8261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:37.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14679.868707 secs 2024-07-26T15:10:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14679.868707 secs 2024-07-26T15:10:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14679.868707 secs 2024-07-26T15:10:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:38 smithi160 ceph-mon[36715]: pgmap v8262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:38 smithi183 ceph-mon[37273]: pgmap v8262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:38 smithi144 ceph-mon[32129]: pgmap v8262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:40.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:10:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:40.019+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:10:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:40.019+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:10:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:40.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:10:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:40.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:40 smithi144 ceph-mon[32129]: pgmap v8263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:40 smithi160 ceph-mon[36715]: pgmap v8263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:40 smithi183 ceph-mon[37273]: pgmap v8263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:42 smithi160 ceph-mon[36715]: pgmap v8264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14684.868826 secs 2024-07-26T15:10:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:42 smithi183 ceph-mon[37273]: pgmap v8264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14684.868826 secs 2024-07-26T15:10:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:42 smithi144 ceph-mon[32129]: pgmap v8264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14684.868826 secs 2024-07-26T15:10:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:44 smithi160 ceph-mon[36715]: pgmap v8265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:44 smithi183 ceph-mon[37273]: pgmap v8265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:44 smithi144 ceph-mon[32129]: pgmap v8265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:46 smithi160 ceph-mon[36715]: pgmap v8266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:46 smithi183 ceph-mon[37273]: pgmap v8266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:46 smithi144 ceph-mon[32129]: pgmap v8266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14689.868945 secs 2024-07-26T15:10:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14689.868945 secs 2024-07-26T15:10:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14689.868945 secs 2024-07-26T15:10:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:48 smithi160 ceph-mon[36715]: pgmap v8267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:48 smithi183 ceph-mon[37273]: pgmap v8267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:48 smithi144 ceph-mon[32129]: pgmap v8267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:50 smithi160 ceph-mon[36715]: pgmap v8268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:50 smithi183 ceph-mon[37273]: pgmap v8268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:50 smithi144 ceph-mon[32129]: pgmap v8268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:52 smithi183 ceph-mon[37273]: pgmap v8269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14694.869096 secs 2024-07-26T15:10:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:52 smithi144 ceph-mon[32129]: pgmap v8269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14694.869096 secs 2024-07-26T15:10:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:52 smithi160 ceph-mon[36715]: pgmap v8269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14694.869096 secs 2024-07-26T15:10:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:54 smithi183 ceph-mon[37273]: pgmap v8270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:54 smithi144 ceph-mon[32129]: pgmap v8270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:54 smithi160 ceph-mon[36715]: pgmap v8270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:10:55.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:55.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:10:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:55.031+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:10:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:55.031+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:10:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:55.031+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:10:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:10:55.031+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:56 smithi183 ceph-mon[37273]: pgmap v8271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:56 smithi144 ceph-mon[32129]: pgmap v8271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:56 smithi160 ceph-mon[36715]: pgmap v8271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:10:57.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14699.869194 secs 2024-07-26T15:10:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14699.869194 secs 2024-07-26T15:10:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14699.869194 secs 2024-07-26T15:10:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:10:58 smithi183 ceph-mon[37273]: pgmap v8272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:10:58 smithi144 ceph-mon[32129]: pgmap v8272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:10:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:10:58 smithi160 ceph-mon[36715]: pgmap v8272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:00 smithi183 ceph-mon[37273]: pgmap v8273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:00 smithi144 ceph-mon[32129]: pgmap v8273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:00 smithi160 ceph-mon[36715]: pgmap v8273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:02 smithi183 ceph-mon[37273]: pgmap v8274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14704.869304 secs 2024-07-26T15:11:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:02 smithi144 ceph-mon[32129]: pgmap v8274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14704.869304 secs 2024-07-26T15:11:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:02 smithi160 ceph-mon[36715]: pgmap v8274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14704.869304 secs 2024-07-26T15:11:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:04 smithi183 ceph-mon[37273]: pgmap v8275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:04 smithi144 ceph-mon[32129]: pgmap v8275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:04 smithi160 ceph-mon[36715]: pgmap v8275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:06 smithi183 ceph-mon[37273]: pgmap v8276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:06 smithi144 ceph-mon[32129]: pgmap v8276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:06 smithi160 ceph-mon[36715]: pgmap v8276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14709.869425 secs 2024-07-26T15:11:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14709.869425 secs 2024-07-26T15:11:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14709.869425 secs 2024-07-26T15:11:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:08 smithi183 ceph-mon[37273]: pgmap v8277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:08 smithi144 ceph-mon[32129]: pgmap v8277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:08 smithi160 ceph-mon[36715]: pgmap v8277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:10.048+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:11:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:10.049+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:11:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:10.049+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:11:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:10.049+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:11:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:10.049+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:10 smithi183 ceph-mon[37273]: pgmap v8278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:10 smithi144 ceph-mon[32129]: pgmap v8278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:10 smithi160 ceph-mon[36715]: pgmap v8278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:12 smithi183 ceph-mon[37273]: pgmap v8279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14714.869581 secs 2024-07-26T15:11:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:12 smithi144 ceph-mon[32129]: pgmap v8279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14714.869581 secs 2024-07-26T15:11:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:12 smithi160 ceph-mon[36715]: pgmap v8279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14714.869581 secs 2024-07-26T15:11:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:14 smithi144 ceph-mon[32129]: pgmap v8280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:14 smithi160 ceph-mon[36715]: pgmap v8280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:14 smithi183 ceph-mon[37273]: pgmap v8280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:16 smithi144 ceph-mon[32129]: pgmap v8281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:16 smithi160 ceph-mon[36715]: pgmap v8281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:16 smithi183 ceph-mon[37273]: pgmap v8281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:17.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14719.869698 secs 2024-07-26T15:11:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14719.869698 secs 2024-07-26T15:11:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14719.869698 secs 2024-07-26T15:11:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:18 smithi144 ceph-mon[32129]: pgmap v8282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:18 smithi160 ceph-mon[36715]: pgmap v8282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:18 smithi183 ceph-mon[37273]: pgmap v8282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:19.468 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:20 smithi144 ceph-mon[32129]: pgmap v8283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:11:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:20 smithi160 ceph-mon[36715]: pgmap v8283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:11:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:20 smithi183 ceph-mon[37273]: pgmap v8283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:11:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:21 smithi144 ceph-mon[32129]: pgmap v8284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:11:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:21 smithi144 ceph-mon[32129]: pgmap v8285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:11:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:21 smithi160 ceph-mon[36715]: pgmap v8284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:11:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:21 smithi160 ceph-mon[36715]: pgmap v8285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:11:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:21 smithi183 ceph-mon[37273]: pgmap v8284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:11:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:21 smithi183 ceph-mon[37273]: pgmap v8285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:11:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14724.869820 secs 2024-07-26T15:11:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14724.869820 secs 2024-07-26T15:11:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14724.869820 secs 2024-07-26T15:11:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:23 smithi144 ceph-mon[32129]: pgmap v8286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:11:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:23 smithi160 ceph-mon[36715]: pgmap v8286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:11:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:23 smithi183 ceph-mon[37273]: pgmap v8286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:11:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:25.065+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:11:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:25.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:11:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:25.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:11:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:25.066+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:11:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:25.066+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:25 smithi144 ceph-mon[32129]: pgmap v8287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:11:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:25 smithi160 ceph-mon[36715]: pgmap v8287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:11:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:25 smithi183 ceph-mon[37273]: pgmap v8287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:11:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:27 smithi144 ceph-mon[32129]: pgmap v8288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:11:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14729.869957 secs 2024-07-26T15:11:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:27 smithi160 ceph-mon[36715]: pgmap v8288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:11:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14729.869957 secs 2024-07-26T15:11:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:27 smithi183 ceph-mon[37273]: pgmap v8288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:11:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14729.869957 secs 2024-07-26T15:11:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:29 smithi160 ceph-mon[36715]: pgmap v8289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:11:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:29 smithi183 ceph-mon[37273]: pgmap v8289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:11:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:29 smithi144 ceph-mon[32129]: pgmap v8289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:11:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:31 smithi160 ceph-mon[36715]: pgmap v8290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:11:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:31 smithi183 ceph-mon[37273]: pgmap v8290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:11:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:31 smithi144 ceph-mon[32129]: pgmap v8290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:11:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14734.870060 secs 2024-07-26T15:11:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14734.870060 secs 2024-07-26T15:11:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14734.870060 secs 2024-07-26T15:11:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:33 smithi160 ceph-mon[36715]: pgmap v8291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:33.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:33 smithi144 ceph-mon[32129]: pgmap v8291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:33 smithi183 ceph-mon[37273]: pgmap v8291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:35 smithi160 ceph-mon[36715]: pgmap v8292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:35 smithi183 ceph-mon[37273]: pgmap v8292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:35 smithi144 ceph-mon[32129]: pgmap v8292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:37 smithi160 ceph-mon[36715]: pgmap v8293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14739.870180 secs 2024-07-26T15:11:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:37 smithi183 ceph-mon[37273]: pgmap v8293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14739.870180 secs 2024-07-26T15:11:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:37 smithi144 ceph-mon[32129]: pgmap v8293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14739.870180 secs 2024-07-26T15:11:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:39 smithi160 ceph-mon[36715]: pgmap v8294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:39 smithi183 ceph-mon[37273]: pgmap v8294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:39 smithi144 ceph-mon[32129]: pgmap v8294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:40.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:40.081+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:11:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:40.082+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:11:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:40.082+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:11:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:40.082+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:11:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:40.082+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:41 smithi160 ceph-mon[36715]: pgmap v8295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:41 smithi183 ceph-mon[37273]: pgmap v8295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:41 smithi144 ceph-mon[32129]: pgmap v8295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14744.870321 secs 2024-07-26T15:11:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14744.870321 secs 2024-07-26T15:11:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14744.870321 secs 2024-07-26T15:11:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:43 smithi183 ceph-mon[37273]: pgmap v8296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:43 smithi144 ceph-mon[32129]: pgmap v8296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:43 smithi160 ceph-mon[36715]: pgmap v8296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:45 smithi183 ceph-mon[37273]: pgmap v8297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:45 smithi144 ceph-mon[32129]: pgmap v8297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:45.975 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:45 smithi160 ceph-mon[36715]: pgmap v8297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:47.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:47 smithi160 ceph-mon[36715]: pgmap v8298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:47.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14749.870448 secs 2024-07-26T15:11:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:47 smithi183 ceph-mon[37273]: pgmap v8298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14749.870448 secs 2024-07-26T15:11:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:47 smithi144 ceph-mon[32129]: pgmap v8298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14749.870448 secs 2024-07-26T15:11:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:48.709+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:11:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:48.709+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:11:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:48.710+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T15:11:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:48.710+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:11:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:49 smithi144 ceph-mon[32129]: pgmap v8299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:49 smithi183 ceph-mon[37273]: pgmap v8299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:49 smithi160 ceph-mon[36715]: pgmap v8299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:51 smithi183 ceph-mon[37273]: pgmap v8300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:51 smithi144 ceph-mon[32129]: pgmap v8300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:51 smithi160 ceph-mon[36715]: pgmap v8300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14754.870587 secs 2024-07-26T15:11:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:52 smithi183 ceph-mon[37273]: pgmap v8301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14754.870587 secs 2024-07-26T15:11:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:52 smithi144 ceph-mon[32129]: pgmap v8301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14754.870587 secs 2024-07-26T15:11:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:52 smithi160 ceph-mon[36715]: pgmap v8301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:11:55.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:55.096+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:11:55.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:55.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:11:55.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:55.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:11:55.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:55.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:11:55.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:11:55.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:54 smithi183 ceph-mon[37273]: pgmap v8302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:54 smithi144 ceph-mon[32129]: pgmap v8302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:55.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:54 smithi160 ceph-mon[36715]: pgmap v8302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:56 smithi183 ceph-mon[37273]: pgmap v8303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14759.870727 secs 2024-07-26T15:11:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:56 smithi144 ceph-mon[32129]: pgmap v8303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14759.870727 secs 2024-07-26T15:11:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:56 smithi160 ceph-mon[36715]: pgmap v8303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:11:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14759.870727 secs 2024-07-26T15:11:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:11:58 smithi183 ceph-mon[37273]: pgmap v8304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:11:58 smithi144 ceph-mon[32129]: pgmap v8304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:11:59.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:11:58 smithi160 ceph-mon[36715]: pgmap v8304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:00 smithi183 ceph-mon[37273]: pgmap v8305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:00 smithi144 ceph-mon[32129]: pgmap v8305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:00 smithi160 ceph-mon[36715]: pgmap v8305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14764.870850 secs 2024-07-26T15:12:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14764.870850 secs 2024-07-26T15:12:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14764.870850 secs 2024-07-26T15:12:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:02 smithi144 ceph-mon[32129]: pgmap v8306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:02 smithi160 ceph-mon[36715]: pgmap v8306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:02 smithi183 ceph-mon[37273]: pgmap v8306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:04 smithi144 ceph-mon[32129]: pgmap v8307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:05.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:04 smithi160 ceph-mon[36715]: pgmap v8307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:04 smithi183 ceph-mon[37273]: pgmap v8307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:07.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:06 smithi183 ceph-mon[37273]: pgmap v8308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:07.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14769.870992 secs 2024-07-26T15:12:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:06 smithi144 ceph-mon[32129]: pgmap v8308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14769.870992 secs 2024-07-26T15:12:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:06 smithi160 ceph-mon[36715]: pgmap v8308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14769.870992 secs 2024-07-26T15:12:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:08 smithi144 ceph-mon[32129]: pgmap v8309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:08 smithi160 ceph-mon[36715]: pgmap v8309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:08 smithi183 ceph-mon[37273]: pgmap v8309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:10.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:10.114+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:12:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:10.115+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:12:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:10.115+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:12:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:10.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:12:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:10.115+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:10 smithi144 ceph-mon[32129]: pgmap v8310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:10 smithi160 ceph-mon[36715]: pgmap v8310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:10 smithi183 ceph-mon[37273]: pgmap v8310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14774.871070 secs 2024-07-26T15:12:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14774.871070 secs 2024-07-26T15:12:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14774.871070 secs 2024-07-26T15:12:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:12 smithi144 ceph-mon[32129]: pgmap v8311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:12 smithi160 ceph-mon[36715]: pgmap v8311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:12 smithi183 ceph-mon[37273]: pgmap v8311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:14 smithi144 ceph-mon[32129]: pgmap v8312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:14 smithi160 ceph-mon[36715]: pgmap v8312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:14 smithi183 ceph-mon[37273]: pgmap v8312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:16 smithi144 ceph-mon[32129]: pgmap v8313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14779.871187 secs 2024-07-26T15:12:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:16 smithi160 ceph-mon[36715]: pgmap v8313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14779.871187 secs 2024-07-26T15:12:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:16 smithi183 ceph-mon[37273]: pgmap v8313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14779.871187 secs 2024-07-26T15:12:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:18 smithi144 ceph-mon[32129]: pgmap v8314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:18 smithi160 ceph-mon[36715]: pgmap v8314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:18 smithi183 ceph-mon[37273]: pgmap v8314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:20 smithi144 ceph-mon[32129]: pgmap v8315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:20 smithi160 ceph-mon[36715]: pgmap v8315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:20 smithi183 ceph-mon[37273]: pgmap v8315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14784.871303 secs 2024-07-26T15:12:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:21 smithi144 ceph-mon[32129]: pgmap v8316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:21 smithi144 ceph-mon[32129]: pgmap v8317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:12:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14784.871303 secs 2024-07-26T15:12:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:21 smithi160 ceph-mon[36715]: pgmap v8316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:21 smithi160 ceph-mon[36715]: pgmap v8317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14784.871303 secs 2024-07-26T15:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:21 smithi183 ceph-mon[37273]: pgmap v8316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:21 smithi183 ceph-mon[37273]: pgmap v8317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:12:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:24 smithi183 ceph-mon[37273]: pgmap v8318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:12:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:24 smithi144 ceph-mon[32129]: pgmap v8318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:12:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:24 smithi160 ceph-mon[36715]: pgmap v8318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:12:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:25.131+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:12:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:25.132+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:12:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:25.132+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:12:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:25.132+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:12:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:25.132+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:26 smithi183 ceph-mon[37273]: pgmap v8319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:12:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:26 smithi144 ceph-mon[32129]: pgmap v8319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:12:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:26 smithi160 ceph-mon[36715]: pgmap v8319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:12:27.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14789.871425 secs 2024-07-26T15:12:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14789.871425 secs 2024-07-26T15:12:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14789.871425 secs 2024-07-26T15:12:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:28 smithi183 ceph-mon[37273]: pgmap v8320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:28 smithi144 ceph-mon[32129]: pgmap v8320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:28 smithi160 ceph-mon[36715]: pgmap v8320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:12:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:30 smithi183 ceph-mon[37273]: pgmap v8321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:12:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:30 smithi144 ceph-mon[32129]: pgmap v8321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:12:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:30 smithi160 ceph-mon[36715]: pgmap v8321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:12:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14794.871545 secs 2024-07-26T15:12:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:32 smithi183 ceph-mon[37273]: pgmap v8322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:12:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14794.871545 secs 2024-07-26T15:12:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:32 smithi144 ceph-mon[32129]: pgmap v8322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:12:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14794.871545 secs 2024-07-26T15:12:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:32 smithi160 ceph-mon[36715]: pgmap v8322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:12:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:34 smithi183 ceph-mon[37273]: pgmap v8323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:34 smithi144 ceph-mon[32129]: pgmap v8323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:34 smithi160 ceph-mon[36715]: pgmap v8323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:36 smithi183 ceph-mon[37273]: pgmap v8324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:36 smithi144 ceph-mon[32129]: pgmap v8324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:36 smithi160 ceph-mon[36715]: pgmap v8324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14799.871663 secs 2024-07-26T15:12:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14799.871663 secs 2024-07-26T15:12:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14799.871663 secs 2024-07-26T15:12:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:38 smithi183 ceph-mon[37273]: pgmap v8325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:38 smithi144 ceph-mon[32129]: pgmap v8325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:38 smithi160 ceph-mon[36715]: pgmap v8325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:40.147+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:12:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:40.148+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:12:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:40.148+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:12:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:40.148+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:12:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:40.148+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:40 smithi183 ceph-mon[37273]: pgmap v8326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:40 smithi144 ceph-mon[32129]: pgmap v8326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:40 smithi160 ceph-mon[36715]: pgmap v8326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14804.871803 secs 2024-07-26T15:12:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:42 smithi183 ceph-mon[37273]: pgmap v8327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14804.871803 secs 2024-07-26T15:12:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:42 smithi144 ceph-mon[32129]: pgmap v8327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14804.871803 secs 2024-07-26T15:12:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:42 smithi160 ceph-mon[36715]: pgmap v8327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:44 smithi144 ceph-mon[32129]: pgmap v8328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:44 smithi160 ceph-mon[36715]: pgmap v8328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:44 smithi183 ceph-mon[37273]: pgmap v8328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:46 smithi144 ceph-mon[32129]: pgmap v8329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:46 smithi160 ceph-mon[36715]: pgmap v8329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:46 smithi183 ceph-mon[37273]: pgmap v8329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:47.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14809.871938 secs 2024-07-26T15:12:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14809.871938 secs 2024-07-26T15:12:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14809.871938 secs 2024-07-26T15:12:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:48 smithi144 ceph-mon[32129]: pgmap v8330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:48 smithi160 ceph-mon[36715]: pgmap v8330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:48 smithi183 ceph-mon[37273]: pgmap v8330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:50 smithi144 ceph-mon[32129]: pgmap v8331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:50 smithi160 ceph-mon[36715]: pgmap v8331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:50 smithi183 ceph-mon[37273]: pgmap v8331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14814.872064 secs 2024-07-26T15:12:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:52 smithi144 ceph-mon[32129]: pgmap v8332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14814.872064 secs 2024-07-26T15:12:52.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:52 smithi160 ceph-mon[36715]: pgmap v8332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14814.872064 secs 2024-07-26T15:12:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:52 smithi183 ceph-mon[37273]: pgmap v8332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:54 smithi144 ceph-mon[32129]: pgmap v8333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:54.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:54 smithi160 ceph-mon[36715]: pgmap v8333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:54 smithi183 ceph-mon[37273]: pgmap v8333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:12:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:55.163+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:12:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:55.164+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:12:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:55.164+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:12:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:55.164+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:12:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:12:55.164+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:56 smithi144 ceph-mon[32129]: pgmap v8334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:56 smithi160 ceph-mon[36715]: pgmap v8334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:56 smithi183 ceph-mon[37273]: pgmap v8334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:12:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14819.872182 secs 2024-07-26T15:12:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14819.872182 secs 2024-07-26T15:12:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14819.872182 secs 2024-07-26T15:12:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:12:58 smithi144 ceph-mon[32129]: pgmap v8335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:58.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:12:58 smithi160 ceph-mon[36715]: pgmap v8335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:12:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:12:58 smithi183 ceph-mon[37273]: pgmap v8335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:00 smithi144 ceph-mon[32129]: pgmap v8336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:00 smithi160 ceph-mon[36715]: pgmap v8336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:00 smithi183 ceph-mon[37273]: pgmap v8336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14824.872334 secs 2024-07-26T15:13:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:02 smithi144 ceph-mon[32129]: pgmap v8337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14824.872334 secs 2024-07-26T15:13:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:02 smithi160 ceph-mon[36715]: pgmap v8337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14824.872334 secs 2024-07-26T15:13:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:02 smithi183 ceph-mon[37273]: pgmap v8337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:04 smithi144 ceph-mon[32129]: pgmap v8338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:13:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:04 smithi160 ceph-mon[36715]: pgmap v8338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:13:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:04 smithi183 ceph-mon[37273]: pgmap v8338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:13:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:06 smithi144 ceph-mon[32129]: pgmap v8339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:06 smithi160 ceph-mon[36715]: pgmap v8339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:06 smithi183 ceph-mon[37273]: pgmap v8339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:07.535 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14829.872455 secs 2024-07-26T15:13:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14829.872455 secs 2024-07-26T15:13:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14829.872455 secs 2024-07-26T15:13:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:08 smithi160 ceph-mon[36715]: pgmap v8340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:13:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:08 smithi183 ceph-mon[37273]: pgmap v8340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:13:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:08 smithi144 ceph-mon[32129]: pgmap v8340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:13:10.167 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:10.166+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:13:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:10.167+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:13:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:10.167+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:13:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:10.167+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:13:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:10.167+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:10 smithi160 ceph-mon[36715]: pgmap v8341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:10 smithi183 ceph-mon[37273]: pgmap v8341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:10 smithi144 ceph-mon[32129]: pgmap v8341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14834.872573 secs 2024-07-26T15:13:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:12 smithi160 ceph-mon[36715]: pgmap v8342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14834.872573 secs 2024-07-26T15:13:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:12 smithi183 ceph-mon[37273]: pgmap v8342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14834.872573 secs 2024-07-26T15:13:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:12 smithi144 ceph-mon[32129]: pgmap v8342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:13:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:14 smithi160 ceph-mon[36715]: pgmap v8343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:14 smithi183 ceph-mon[37273]: pgmap v8343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:14 smithi144 ceph-mon[32129]: pgmap v8343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:16 smithi160 ceph-mon[36715]: pgmap v8344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:16 smithi183 ceph-mon[37273]: pgmap v8344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:16 smithi144 ceph-mon[32129]: pgmap v8344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14839.872656 secs 2024-07-26T15:13:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14839.872656 secs 2024-07-26T15:13:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14839.872656 secs 2024-07-26T15:13:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:18 smithi160 ceph-mon[36715]: pgmap v8345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:18 smithi183 ceph-mon[37273]: pgmap v8345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:18 smithi144 ceph-mon[32129]: pgmap v8345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:18.725 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:18.725+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:13:18.726 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:18.725+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T15:13:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:18.725+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:13:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:18.725+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T15:13:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:20 smithi160 ceph-mon[36715]: pgmap v8346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:20 smithi183 ceph-mon[37273]: pgmap v8346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:20 smithi144 ceph-mon[32129]: pgmap v8346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14844.872776 secs 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: pgmap v8347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:22.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14844.872776 secs 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: pgmap v8347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:13:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14844.872776 secs 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: pgmap v8347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:13:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:23 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:23 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:23 smithi160 ceph-mon[36715]: pgmap v8348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:13:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:23 smithi160 ceph-mon[36715]: pgmap v8349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:23 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:13:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:23 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:13:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:23 smithi183 ceph-mon[37273]: pgmap v8348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:13:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:23 smithi183 ceph-mon[37273]: pgmap v8349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:23 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:13:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:23 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:13:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:23 smithi144 ceph-mon[32129]: pgmap v8348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:13:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:23 smithi144 ceph-mon[32129]: pgmap v8349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:25.172+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:13:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:25.173+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:13:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:25.173+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:13:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:25.173+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:13:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:25.174+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:25 smithi160 ceph-mon[36715]: pgmap v8350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:25 smithi183 ceph-mon[37273]: pgmap v8350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:25 smithi144 ceph-mon[32129]: pgmap v8350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14849.872906 secs 2024-07-26T15:13:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:27 smithi183 ceph-mon[37273]: pgmap v8351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14849.872906 secs 2024-07-26T15:13:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:27 smithi144 ceph-mon[32129]: pgmap v8351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14849.872906 secs 2024-07-26T15:13:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:27 smithi160 ceph-mon[36715]: pgmap v8351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:29 smithi183 ceph-mon[37273]: pgmap v8352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:29 smithi144 ceph-mon[32129]: pgmap v8352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:29 smithi160 ceph-mon[36715]: pgmap v8352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:31 smithi183 ceph-mon[37273]: pgmap v8353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:31 smithi144 ceph-mon[32129]: pgmap v8353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:31 smithi160 ceph-mon[36715]: pgmap v8353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:13:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14854.873047 secs 2024-07-26T15:13:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14854.873047 secs 2024-07-26T15:13:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14854.873047 secs 2024-07-26T15:13:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:33 smithi183 ceph-mon[37273]: pgmap v8354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:13:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:33 smithi144 ceph-mon[32129]: pgmap v8354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:13:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:33 smithi160 ceph-mon[36715]: pgmap v8354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:13:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:35 smithi183 ceph-mon[37273]: pgmap v8355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:35 smithi144 ceph-mon[32129]: pgmap v8355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:35 smithi160 ceph-mon[36715]: pgmap v8355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14859.873175 secs 2024-07-26T15:13:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:37 smithi144 ceph-mon[32129]: pgmap v8356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14859.873175 secs 2024-07-26T15:13:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:37 smithi183 ceph-mon[37273]: pgmap v8356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14859.873175 secs 2024-07-26T15:13:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:37 smithi160 ceph-mon[36715]: pgmap v8356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:39 smithi183 ceph-mon[37273]: pgmap v8357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:39 smithi144 ceph-mon[32129]: pgmap v8357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:39 smithi160 ceph-mon[36715]: pgmap v8357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:40.189+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:13:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:40.190+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:13:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:40.190+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:13:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:40.190+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:13:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:40.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:41 smithi183 ceph-mon[37273]: pgmap v8358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:41 smithi144 ceph-mon[32129]: pgmap v8358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:41 smithi160 ceph-mon[36715]: pgmap v8358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14864.873311 secs 2024-07-26T15:13:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14864.873311 secs 2024-07-26T15:13:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14864.873311 secs 2024-07-26T15:13:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:43 smithi183 ceph-mon[37273]: pgmap v8359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:43 smithi144 ceph-mon[32129]: pgmap v8359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:43 smithi160 ceph-mon[36715]: pgmap v8359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:44 smithi183 ceph-mon[37273]: pgmap v8360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:44 smithi144 ceph-mon[32129]: pgmap v8360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:44 smithi160 ceph-mon[36715]: pgmap v8360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14869.873433 secs 2024-07-26T15:13:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:47 smithi160 ceph-mon[36715]: pgmap v8361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14869.873433 secs 2024-07-26T15:13:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:47 smithi183 ceph-mon[37273]: pgmap v8361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14869.873433 secs 2024-07-26T15:13:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:47 smithi144 ceph-mon[32129]: pgmap v8361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:49 smithi160 ceph-mon[36715]: pgmap v8362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:49 smithi183 ceph-mon[37273]: pgmap v8362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:49 smithi144 ceph-mon[32129]: pgmap v8362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:51 smithi160 ceph-mon[36715]: pgmap v8363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:51 smithi183 ceph-mon[37273]: pgmap v8363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:51 smithi144 ceph-mon[32129]: pgmap v8363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14874.873517 secs 2024-07-26T15:13:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14874.873517 secs 2024-07-26T15:13:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14874.873517 secs 2024-07-26T15:13:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:53 smithi183 ceph-mon[37273]: pgmap v8364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:53 smithi144 ceph-mon[32129]: pgmap v8364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:53 smithi160 ceph-mon[36715]: pgmap v8364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:13:55.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:55.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:13:55.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:55.206+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:13:55.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:55.206+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:13:55.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:55.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:13:55.209 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:13:55.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:55 smithi183 ceph-mon[37273]: pgmap v8365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:55 smithi144 ceph-mon[32129]: pgmap v8365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:55 smithi160 ceph-mon[36715]: pgmap v8365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14879.873631 secs 2024-07-26T15:13:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:57 smithi183 ceph-mon[37273]: pgmap v8366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14879.873631 secs 2024-07-26T15:13:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:57 smithi144 ceph-mon[32129]: pgmap v8366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14879.873631 secs 2024-07-26T15:13:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:57 smithi160 ceph-mon[36715]: pgmap v8366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:13:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:13:59 smithi183 ceph-mon[37273]: pgmap v8367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:13:59 smithi144 ceph-mon[32129]: pgmap v8367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:13:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:13:59 smithi160 ceph-mon[36715]: pgmap v8367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:01 smithi183 ceph-mon[37273]: pgmap v8368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:01 smithi144 ceph-mon[32129]: pgmap v8368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:01 smithi160 ceph-mon[36715]: pgmap v8368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14884.873753 secs 2024-07-26T15:14:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14884.873753 secs 2024-07-26T15:14:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14884.873753 secs 2024-07-26T15:14:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:03 smithi183 ceph-mon[37273]: pgmap v8369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:03 smithi144 ceph-mon[32129]: pgmap v8369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:03 smithi160 ceph-mon[36715]: pgmap v8369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:04 smithi183 ceph-mon[37273]: pgmap v8370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:04 smithi144 ceph-mon[32129]: pgmap v8370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:04 smithi160 ceph-mon[36715]: pgmap v8370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14889.873874 secs 2024-07-26T15:14:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:07 smithi160 ceph-mon[36715]: pgmap v8371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14889.873874 secs 2024-07-26T15:14:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:07 smithi183 ceph-mon[37273]: pgmap v8371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14889.873874 secs 2024-07-26T15:14:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:07 smithi144 ceph-mon[32129]: pgmap v8371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:09 smithi183 ceph-mon[37273]: pgmap v8372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:09 smithi144 ceph-mon[32129]: pgmap v8372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:09 smithi160 ceph-mon[36715]: pgmap v8372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:10.223+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:14:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:10.224+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:14:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:10.224+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:14:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:10.224+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:14:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:10.224+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:11 smithi183 ceph-mon[37273]: pgmap v8373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:11 smithi144 ceph-mon[32129]: pgmap v8373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:11 smithi160 ceph-mon[36715]: pgmap v8373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14894.874015 secs 2024-07-26T15:14:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14894.874015 secs 2024-07-26T15:14:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14894.874015 secs 2024-07-26T15:14:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:13 smithi183 ceph-mon[37273]: pgmap v8374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:13 smithi144 ceph-mon[32129]: pgmap v8374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:13 smithi160 ceph-mon[36715]: pgmap v8374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:15 smithi183 ceph-mon[37273]: pgmap v8375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:15 smithi144 ceph-mon[32129]: pgmap v8375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:15 smithi160 ceph-mon[36715]: pgmap v8375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:17.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14899.874155 secs 2024-07-26T15:14:17.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:17 smithi160 ceph-mon[36715]: pgmap v8376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14899.874155 secs 2024-07-26T15:14:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:17 smithi183 ceph-mon[37273]: pgmap v8376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14899.874155 secs 2024-07-26T15:14:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:17 smithi144 ceph-mon[32129]: pgmap v8376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:19 smithi183 ceph-mon[37273]: pgmap v8377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:19 smithi144 ceph-mon[32129]: pgmap v8377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:19 smithi160 ceph-mon[36715]: pgmap v8377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:21 smithi183 ceph-mon[37273]: pgmap v8378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:21 smithi144 ceph-mon[32129]: pgmap v8378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:21 smithi160 ceph-mon[36715]: pgmap v8378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14904.874307 secs 2024-07-26T15:14:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14904.874307 secs 2024-07-26T15:14:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14904.874307 secs 2024-07-26T15:14:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: pgmap v8379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: pgmap v8379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: pgmap v8379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:23.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:14:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:14:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:14:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:14:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:14:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:14:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:14:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:14:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:14:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:14:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:14:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:14:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:14:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:14:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:14:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:14:25.236 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:25.235+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:14:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:25.236+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:14:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:25.236+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:14:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:25.236+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:14:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:25.236+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:25 smithi183 ceph-mon[37273]: pgmap v8380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:25 smithi183 ceph-mon[37273]: pgmap v8381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:14:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:25 smithi144 ceph-mon[32129]: pgmap v8380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:25 smithi144 ceph-mon[32129]: pgmap v8381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:14:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:25 smithi160 ceph-mon[36715]: pgmap v8380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:25 smithi160 ceph-mon[36715]: pgmap v8381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:14:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:26 smithi144 ceph-mon[32129]: pgmap v8382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:26 smithi160 ceph-mon[36715]: pgmap v8382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:26 smithi183 ceph-mon[37273]: pgmap v8382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14909.874389 secs 2024-07-26T15:14:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14909.874389 secs 2024-07-26T15:14:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14909.874389 secs 2024-07-26T15:14:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:29 smithi183 ceph-mon[37273]: pgmap v8383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:14:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:29 smithi144 ceph-mon[32129]: pgmap v8383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:14:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:29 smithi160 ceph-mon[36715]: pgmap v8383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:14:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:31 smithi183 ceph-mon[37273]: pgmap v8384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:14:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:31 smithi144 ceph-mon[32129]: pgmap v8384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:14:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:31 smithi160 ceph-mon[36715]: pgmap v8384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:14:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14914.874509 secs 2024-07-26T15:14:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14914.874509 secs 2024-07-26T15:14:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14914.874509 secs 2024-07-26T15:14:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:33 smithi183 ceph-mon[37273]: pgmap v8385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:33 smithi144 ceph-mon[32129]: pgmap v8385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:33 smithi160 ceph-mon[36715]: pgmap v8385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:14:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:34 smithi183 ceph-mon[37273]: pgmap v8386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:14:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:34 smithi144 ceph-mon[32129]: pgmap v8386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:14:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:34 smithi160 ceph-mon[36715]: pgmap v8386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:14:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:36 smithi144 ceph-mon[32129]: pgmap v8387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:36 smithi160 ceph-mon[36715]: pgmap v8387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:36 smithi183 ceph-mon[37273]: pgmap v8387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14919.874630 secs 2024-07-26T15:14:37.785 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14919.874630 secs 2024-07-26T15:14:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14919.874630 secs 2024-07-26T15:14:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:38 smithi144 ceph-mon[32129]: pgmap v8388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:38 smithi160 ceph-mon[36715]: pgmap v8388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:38 smithi183 ceph-mon[37273]: pgmap v8388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:40.240+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:14:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:40.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:14:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:40.241+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:14:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:40.241+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:14:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:40.241+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:40 smithi144 ceph-mon[32129]: pgmap v8389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:40 smithi160 ceph-mon[36715]: pgmap v8389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:40 smithi183 ceph-mon[37273]: pgmap v8389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14924.874755 secs 2024-07-26T15:14:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:42 smithi160 ceph-mon[36715]: pgmap v8390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14924.874755 secs 2024-07-26T15:14:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:42 smithi183 ceph-mon[37273]: pgmap v8390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14924.874755 secs 2024-07-26T15:14:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:42 smithi144 ceph-mon[32129]: pgmap v8390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:44 smithi160 ceph-mon[36715]: pgmap v8391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:44 smithi183 ceph-mon[37273]: pgmap v8391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:44 smithi144 ceph-mon[32129]: pgmap v8391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:46 smithi160 ceph-mon[36715]: pgmap v8392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:46 smithi183 ceph-mon[37273]: pgmap v8392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:46 smithi144 ceph-mon[32129]: pgmap v8392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14929.874859 secs 2024-07-26T15:14:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14929.874859 secs 2024-07-26T15:14:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14929.874859 secs 2024-07-26T15:14:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:48.739+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:14:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:48.739+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:14:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:48.740+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T15:14:48.742 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:48.740+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:14:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:48 smithi160 ceph-mon[36715]: pgmap v8393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:48 smithi183 ceph-mon[37273]: pgmap v8393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:48 smithi144 ceph-mon[32129]: pgmap v8393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:50 smithi160 ceph-mon[36715]: pgmap v8394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:50 smithi183 ceph-mon[37273]: pgmap v8394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:50 smithi144 ceph-mon[32129]: pgmap v8394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14934.874981 secs 2024-07-26T15:14:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:52 smithi160 ceph-mon[36715]: pgmap v8395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14934.874981 secs 2024-07-26T15:14:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:52 smithi183 ceph-mon[37273]: pgmap v8395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14934.874981 secs 2024-07-26T15:14:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:52 smithi144 ceph-mon[32129]: pgmap v8395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:14:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:54 smithi160 ceph-mon[36715]: pgmap v8396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:54 smithi183 ceph-mon[37273]: pgmap v8396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:54 smithi144 ceph-mon[32129]: pgmap v8396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:55.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:14:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:55.257+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:14:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:55.257+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:14:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:55.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:14:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:14:55.257+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:56 smithi160 ceph-mon[36715]: pgmap v8397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:56 smithi183 ceph-mon[37273]: pgmap v8397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:56 smithi144 ceph-mon[32129]: pgmap v8397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:14:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14939.875080 secs 2024-07-26T15:14:57.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14939.875080 secs 2024-07-26T15:14:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14939.875080 secs 2024-07-26T15:14:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:14:58 smithi160 ceph-mon[36715]: pgmap v8398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:14:58 smithi183 ceph-mon[37273]: pgmap v8398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:14:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:14:58 smithi144 ceph-mon[32129]: pgmap v8398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:00 smithi160 ceph-mon[36715]: pgmap v8399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:00 smithi183 ceph-mon[37273]: pgmap v8399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:00 smithi144 ceph-mon[32129]: pgmap v8399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14944.875198 secs 2024-07-26T15:15:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:02 smithi160 ceph-mon[36715]: pgmap v8400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14944.875198 secs 2024-07-26T15:15:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:02 smithi183 ceph-mon[37273]: pgmap v8400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14944.875198 secs 2024-07-26T15:15:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:02 smithi144 ceph-mon[32129]: pgmap v8400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:04 smithi160 ceph-mon[36715]: pgmap v8401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:04 smithi183 ceph-mon[37273]: pgmap v8401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:04 smithi144 ceph-mon[32129]: pgmap v8401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:06 smithi183 ceph-mon[37273]: pgmap v8402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:06 smithi144 ceph-mon[32129]: pgmap v8402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:06 smithi160 ceph-mon[36715]: pgmap v8402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14949.875325 secs 2024-07-26T15:15:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14949.875325 secs 2024-07-26T15:15:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14949.875325 secs 2024-07-26T15:15:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:08 smithi183 ceph-mon[37273]: pgmap v8403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:08 smithi144 ceph-mon[32129]: pgmap v8403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:08 smithi160 ceph-mon[36715]: pgmap v8403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:10.274+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:15:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:10.274+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:15:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:10.275+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:15:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:10.275+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:15:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:10.275+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:10.877 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:10 smithi144 ceph-mon[32129]: pgmap v8404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:10 smithi183 ceph-mon[37273]: pgmap v8404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:10 smithi160 ceph-mon[36715]: pgmap v8404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14954.875378 secs 2024-07-26T15:15:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:12 smithi183 ceph-mon[37273]: pgmap v8405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14954.875378 secs 2024-07-26T15:15:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:12 smithi144 ceph-mon[32129]: pgmap v8405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14954.875378 secs 2024-07-26T15:15:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:12 smithi160 ceph-mon[36715]: pgmap v8405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:14 smithi183 ceph-mon[37273]: pgmap v8406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:14 smithi144 ceph-mon[32129]: pgmap v8406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:14 smithi160 ceph-mon[36715]: pgmap v8406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:16 smithi183 ceph-mon[37273]: pgmap v8407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:16 smithi144 ceph-mon[32129]: pgmap v8407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:16 smithi160 ceph-mon[36715]: pgmap v8407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14959.875494 secs 2024-07-26T15:15:17.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14959.875494 secs 2024-07-26T15:15:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14959.875494 secs 2024-07-26T15:15:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:18 smithi183 ceph-mon[37273]: pgmap v8408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:18 smithi144 ceph-mon[32129]: pgmap v8408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:18 smithi160 ceph-mon[36715]: pgmap v8408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:20 smithi183 ceph-mon[37273]: pgmap v8409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:20 smithi144 ceph-mon[32129]: pgmap v8409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:20 smithi160 ceph-mon[36715]: pgmap v8409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:22.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14964.875622 secs 2024-07-26T15:15:22.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:22 smithi183 ceph-mon[37273]: pgmap v8410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14964.875622 secs 2024-07-26T15:15:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:22 smithi144 ceph-mon[32129]: pgmap v8410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14964.875622 secs 2024-07-26T15:15:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:22 smithi160 ceph-mon[36715]: pgmap v8410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:24 smithi144 ceph-mon[32129]: pgmap v8411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:15:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:24 smithi160 ceph-mon[36715]: pgmap v8411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:15:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:24 smithi183 ceph-mon[37273]: pgmap v8411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:15:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:25.286+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:15:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:25.287+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:15:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:25.287+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:15:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:25.287+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:15:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:25.287+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:25 smithi144 ceph-mon[32129]: pgmap v8412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:15:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:25 smithi144 ceph-mon[32129]: pgmap v8413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:25 smithi160 ceph-mon[36715]: pgmap v8412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:15:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:25 smithi160 ceph-mon[36715]: pgmap v8413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:25 smithi183 ceph-mon[37273]: pgmap v8412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:15:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:25 smithi183 ceph-mon[37273]: pgmap v8413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14969.875746 secs 2024-07-26T15:15:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:27 smithi144 ceph-mon[32129]: pgmap v8414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:15:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14969.875746 secs 2024-07-26T15:15:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:27 smithi160 ceph-mon[36715]: pgmap v8414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:15:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14969.875746 secs 2024-07-26T15:15:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:27 smithi183 ceph-mon[37273]: pgmap v8414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:15:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:29 smithi144 ceph-mon[32129]: pgmap v8415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:15:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:29 smithi160 ceph-mon[36715]: pgmap v8415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:15:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:29 smithi183 ceph-mon[37273]: pgmap v8415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:15:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:31 smithi144 ceph-mon[32129]: pgmap v8416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:15:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:31 smithi160 ceph-mon[36715]: pgmap v8416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:15:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:31 smithi183 ceph-mon[37273]: pgmap v8416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:15:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14974.875892 secs 2024-07-26T15:15:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14974.875892 secs 2024-07-26T15:15:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14974.875892 secs 2024-07-26T15:15:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:33 smithi144 ceph-mon[32129]: pgmap v8417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:34.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:33 smithi160 ceph-mon[36715]: pgmap v8417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:33 smithi183 ceph-mon[37273]: pgmap v8417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:35 smithi144 ceph-mon[32129]: pgmap v8418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:15:36.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:35 smithi160 ceph-mon[36715]: pgmap v8418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:15:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:35 smithi183 ceph-mon[37273]: pgmap v8418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:15:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14979.876017 secs 2024-07-26T15:15:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:37 smithi144 ceph-mon[32129]: pgmap v8419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14979.876017 secs 2024-07-26T15:15:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:37 smithi160 ceph-mon[36715]: pgmap v8419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14979.876017 secs 2024-07-26T15:15:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:37 smithi183 ceph-mon[37273]: pgmap v8419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:39 smithi144 ceph-mon[32129]: pgmap v8420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:40.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:39 smithi160 ceph-mon[36715]: pgmap v8420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:39 smithi183 ceph-mon[37273]: pgmap v8420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:40.301+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:15:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:40.302+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:15:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:40.302+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:15:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:40.302+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:15:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:40.302+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:41 smithi144 ceph-mon[32129]: pgmap v8421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:41 smithi160 ceph-mon[36715]: pgmap v8421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:41 smithi183 ceph-mon[37273]: pgmap v8421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:15:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14984.876156 secs 2024-07-26T15:15:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14984.876156 secs 2024-07-26T15:15:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14984.876156 secs 2024-07-26T15:15:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:43 smithi144 ceph-mon[32129]: pgmap v8422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:44.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:43 smithi160 ceph-mon[36715]: pgmap v8422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:43 smithi183 ceph-mon[37273]: pgmap v8422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:15:46.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:45 smithi160 ceph-mon[36715]: pgmap v8423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:15:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:45 smithi183 ceph-mon[37273]: pgmap v8423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:15:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:45 smithi144 ceph-mon[32129]: pgmap v8423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:15:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14989.876275 secs 2024-07-26T15:15:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:47 smithi160 ceph-mon[36715]: pgmap v8424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14989.876275 secs 2024-07-26T15:15:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:47 smithi183 ceph-mon[37273]: pgmap v8424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14989.876275 secs 2024-07-26T15:15:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:47 smithi144 ceph-mon[32129]: pgmap v8424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:49 smithi160 ceph-mon[36715]: pgmap v8425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:49 smithi183 ceph-mon[37273]: pgmap v8425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:49 smithi144 ceph-mon[32129]: pgmap v8425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:51 smithi160 ceph-mon[36715]: pgmap v8426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:51 smithi183 ceph-mon[37273]: pgmap v8426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:51 smithi144 ceph-mon[32129]: pgmap v8426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14994.876370 secs 2024-07-26T15:15:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14994.876370 secs 2024-07-26T15:15:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14994.876370 secs 2024-07-26T15:15:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:53 smithi160 ceph-mon[36715]: pgmap v8427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:53 smithi183 ceph-mon[37273]: pgmap v8427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:53 smithi144 ceph-mon[32129]: pgmap v8427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:15:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:55.317+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:15:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:55.317+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:15:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:55.317+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:15:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:55.317+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:15:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:15:55.317+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:55 smithi160 ceph-mon[36715]: pgmap v8428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:55 smithi183 ceph-mon[37273]: pgmap v8428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:55 smithi144 ceph-mon[32129]: pgmap v8428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:15:58.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 14999.876492 secs 2024-07-26T15:15:58.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:57 smithi144 ceph-mon[32129]: pgmap v8429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 14999.876492 secs 2024-07-26T15:15:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:57 smithi160 ceph-mon[36715]: pgmap v8429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:15:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 14999.876492 secs 2024-07-26T15:15:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:57 smithi183 ceph-mon[37273]: pgmap v8429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:15:59 smithi160 ceph-mon[36715]: pgmap v8430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:15:59 smithi183 ceph-mon[37273]: pgmap v8430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:15:59 smithi144 ceph-mon[32129]: pgmap v8430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:01 smithi160 ceph-mon[36715]: pgmap v8431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:01 smithi183 ceph-mon[37273]: pgmap v8431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:01 smithi144 ceph-mon[32129]: pgmap v8431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15004.876624 secs 2024-07-26T15:16:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15004.876624 secs 2024-07-26T15:16:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15004.876624 secs 2024-07-26T15:16:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:03 smithi160 ceph-mon[36715]: pgmap v8432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:03 smithi183 ceph-mon[37273]: pgmap v8432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:03 smithi144 ceph-mon[32129]: pgmap v8432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:06.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:05 smithi160 ceph-mon[36715]: pgmap v8433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:05 smithi183 ceph-mon[37273]: pgmap v8433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:05 smithi144 ceph-mon[32129]: pgmap v8433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15009.876706 secs 2024-07-26T15:16:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:07 smithi160 ceph-mon[36715]: pgmap v8434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15009.876706 secs 2024-07-26T15:16:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:07 smithi183 ceph-mon[37273]: pgmap v8434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15009.876706 secs 2024-07-26T15:16:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:07 smithi144 ceph-mon[32129]: pgmap v8434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:09 smithi160 ceph-mon[36715]: pgmap v8435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:09 smithi183 ceph-mon[37273]: pgmap v8435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:09 smithi144 ceph-mon[32129]: pgmap v8435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:10.334+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:16:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:10.335+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:16:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:10.335+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:16:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:10.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:16:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:10.335+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:11 smithi183 ceph-mon[37273]: pgmap v8436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:11 smithi144 ceph-mon[32129]: pgmap v8436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:11 smithi160 ceph-mon[36715]: pgmap v8436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:13.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15014.876840 secs 2024-07-26T15:16:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15014.876840 secs 2024-07-26T15:16:13.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15014.876840 secs 2024-07-26T15:16:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:13 smithi183 ceph-mon[37273]: pgmap v8437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:13 smithi144 ceph-mon[32129]: pgmap v8437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:13 smithi160 ceph-mon[36715]: pgmap v8437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:15 smithi183 ceph-mon[37273]: pgmap v8438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:15 smithi144 ceph-mon[32129]: pgmap v8438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:15 smithi160 ceph-mon[36715]: pgmap v8438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:18.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15019.876906 secs 2024-07-26T15:16:18.110 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:17 smithi160 ceph-mon[36715]: pgmap v8439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15019.876906 secs 2024-07-26T15:16:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:17 smithi183 ceph-mon[37273]: pgmap v8439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15019.876906 secs 2024-07-26T15:16:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:17 smithi144 ceph-mon[32129]: pgmap v8439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:19 smithi183 ceph-mon[37273]: pgmap v8440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:19 smithi144 ceph-mon[32129]: pgmap v8440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:19 smithi160 ceph-mon[36715]: pgmap v8440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:21 smithi183 ceph-mon[37273]: pgmap v8441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:21 smithi144 ceph-mon[32129]: pgmap v8441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:21 smithi160 ceph-mon[36715]: pgmap v8441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15024.877033 secs 2024-07-26T15:16:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15024.877033 secs 2024-07-26T15:16:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15024.877033 secs 2024-07-26T15:16:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:23 smithi183 ceph-mon[37273]: pgmap v8442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:23 smithi144 ceph-mon[32129]: pgmap v8442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:23 smithi160 ceph-mon[36715]: pgmap v8442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:25.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:25.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:16:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:25.348+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:16:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:25.348+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:16:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:25.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:16:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:25.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:26 smithi144 ceph-mon[32129]: pgmap v8443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:26 smithi160 ceph-mon[36715]: pgmap v8443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:26 smithi183 ceph-mon[37273]: pgmap v8443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15029.877162 secs 2024-07-26T15:16:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:27 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15029.877162 secs 2024-07-26T15:16:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:27.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:27 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15029.877162 secs 2024-07-26T15:16:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:27 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:16:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:28 smithi144 ceph-mon[32129]: pgmap v8444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:28 smithi144 ceph-mon[32129]: pgmap v8445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:28 smithi160 ceph-mon[36715]: pgmap v8444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:28 smithi160 ceph-mon[36715]: pgmap v8445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:28 smithi183 ceph-mon[37273]: pgmap v8444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:28 smithi183 ceph-mon[37273]: pgmap v8445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:29 smithi144 ceph-mon[32129]: pgmap v8446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:29.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:29 smithi160 ceph-mon[36715]: pgmap v8446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:29 smithi183 ceph-mon[37273]: pgmap v8446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:31.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:31 smithi160 ceph-mon[36715]: pgmap v8447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:31 smithi183 ceph-mon[37273]: pgmap v8447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:31 smithi144 ceph-mon[32129]: pgmap v8447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:32.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15034.877285 secs 2024-07-26T15:16:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15034.877285 secs 2024-07-26T15:16:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15034.877285 secs 2024-07-26T15:16:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:33 smithi160 ceph-mon[36715]: pgmap v8448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:33 smithi183 ceph-mon[37273]: pgmap v8448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:33 smithi144 ceph-mon[32129]: pgmap v8448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:35 smithi160 ceph-mon[36715]: pgmap v8449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:35 smithi183 ceph-mon[37273]: pgmap v8449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:35 smithi144 ceph-mon[32129]: pgmap v8449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:16:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15039.877381 secs 2024-07-26T15:16:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:37 smithi160 ceph-mon[36715]: pgmap v8450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:16:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15039.877381 secs 2024-07-26T15:16:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:37 smithi183 ceph-mon[37273]: pgmap v8450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:16:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15039.877381 secs 2024-07-26T15:16:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:37 smithi144 ceph-mon[32129]: pgmap v8450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:16:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:39 smithi160 ceph-mon[36715]: pgmap v8451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:39 smithi183 ceph-mon[37273]: pgmap v8451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:39 smithi144 ceph-mon[32129]: pgmap v8451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:40.356+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:16:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:40.357+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:16:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:40.357+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:16:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:40.357+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:16:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:40.357+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:41 smithi160 ceph-mon[36715]: pgmap v8452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:41 smithi183 ceph-mon[37273]: pgmap v8452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:41 smithi144 ceph-mon[32129]: pgmap v8452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15044.877477 secs 2024-07-26T15:16:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15044.877477 secs 2024-07-26T15:16:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15044.877477 secs 2024-07-26T15:16:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:43 smithi160 ceph-mon[36715]: pgmap v8453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:43 smithi183 ceph-mon[37273]: pgmap v8453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:43 smithi144 ceph-mon[32129]: pgmap v8453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:45 smithi160 ceph-mon[36715]: pgmap v8454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:45 smithi183 ceph-mon[37273]: pgmap v8454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:45 smithi144 ceph-mon[32129]: pgmap v8454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15049.877571 secs 2024-07-26T15:16:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:47 smithi160 ceph-mon[36715]: pgmap v8455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15049.877571 secs 2024-07-26T15:16:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:47 smithi183 ceph-mon[37273]: pgmap v8455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15049.877571 secs 2024-07-26T15:16:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:47 smithi144 ceph-mon[32129]: pgmap v8455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:49 smithi160 ceph-mon[36715]: pgmap v8456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:49 smithi183 ceph-mon[37273]: pgmap v8456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:49 smithi144 ceph-mon[32129]: pgmap v8456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:51 smithi183 ceph-mon[37273]: pgmap v8457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:51 smithi144 ceph-mon[32129]: pgmap v8457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:51 smithi160 ceph-mon[36715]: pgmap v8457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15054.877653 secs 2024-07-26T15:16:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15054.877653 secs 2024-07-26T15:16:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15054.877653 secs 2024-07-26T15:16:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:53 smithi183 ceph-mon[37273]: pgmap v8458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:53 smithi144 ceph-mon[32129]: pgmap v8458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:54.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:53 smithi160 ceph-mon[36715]: pgmap v8458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:16:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:55.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:16:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:55.365+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:16:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:55.365+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:16:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:55.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:16:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:16:55.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:55 smithi183 ceph-mon[37273]: pgmap v8459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:55 smithi144 ceph-mon[32129]: pgmap v8459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:56.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:55 smithi160 ceph-mon[36715]: pgmap v8459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15059.877735 secs 2024-07-26T15:16:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:57 smithi183 ceph-mon[37273]: pgmap v8460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15059.877735 secs 2024-07-26T15:16:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:57 smithi144 ceph-mon[32129]: pgmap v8460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15059.877735 secs 2024-07-26T15:16:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:57 smithi160 ceph-mon[36715]: pgmap v8460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:16:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:16:59 smithi183 ceph-mon[37273]: pgmap v8461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:16:59.933 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:16:59 smithi144 ceph-mon[32129]: pgmap v8461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:16:59 smithi160 ceph-mon[36715]: pgmap v8461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:01 smithi183 ceph-mon[37273]: pgmap v8462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:01 smithi144 ceph-mon[32129]: pgmap v8462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:01 smithi160 ceph-mon[36715]: pgmap v8462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15064.877858 secs 2024-07-26T15:17:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15064.877858 secs 2024-07-26T15:17:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15064.877858 secs 2024-07-26T15:17:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:03 smithi183 ceph-mon[37273]: pgmap v8463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:03 smithi144 ceph-mon[32129]: pgmap v8463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:04.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:03 smithi160 ceph-mon[36715]: pgmap v8463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:05 smithi183 ceph-mon[37273]: pgmap v8464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:05 smithi144 ceph-mon[32129]: pgmap v8464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:06.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:05 smithi160 ceph-mon[36715]: pgmap v8464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15069.877982 secs 2024-07-26T15:17:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:07 smithi144 ceph-mon[32129]: pgmap v8465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:08.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15069.877982 secs 2024-07-26T15:17:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:07 smithi160 ceph-mon[36715]: pgmap v8465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15069.877982 secs 2024-07-26T15:17:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:07 smithi183 ceph-mon[37273]: pgmap v8465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:09 smithi144 ceph-mon[32129]: pgmap v8466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:10.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:09 smithi160 ceph-mon[36715]: pgmap v8466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:09 smithi183 ceph-mon[37273]: pgmap v8466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:10.374 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:10.373+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:17:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:10.374+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:17:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:10.374+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:17:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:10.374+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:17:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:10.374+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:11 smithi144 ceph-mon[32129]: pgmap v8467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:11 smithi160 ceph-mon[36715]: pgmap v8467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:11 smithi183 ceph-mon[37273]: pgmap v8467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15074.878139 secs 2024-07-26T15:17:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15074.878139 secs 2024-07-26T15:17:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15074.878139 secs 2024-07-26T15:17:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:13 smithi144 ceph-mon[32129]: pgmap v8468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:14.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:13 smithi160 ceph-mon[36715]: pgmap v8468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:13 smithi183 ceph-mon[37273]: pgmap v8468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:15 smithi144 ceph-mon[32129]: pgmap v8469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:16.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:15 smithi160 ceph-mon[36715]: pgmap v8469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:15 smithi183 ceph-mon[37273]: pgmap v8469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15079.878262 secs 2024-07-26T15:17:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:17 smithi144 ceph-mon[32129]: pgmap v8470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:18.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15079.878262 secs 2024-07-26T15:17:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:17 smithi160 ceph-mon[36715]: pgmap v8470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15079.878262 secs 2024-07-26T15:17:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:17 smithi183 ceph-mon[37273]: pgmap v8470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:19 smithi144 ceph-mon[32129]: pgmap v8471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:20.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:19 smithi160 ceph-mon[36715]: pgmap v8471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:19 smithi183 ceph-mon[37273]: pgmap v8471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:21 smithi144 ceph-mon[32129]: pgmap v8472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:21 smithi160 ceph-mon[36715]: pgmap v8472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:21 smithi183 ceph-mon[37273]: pgmap v8472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15084.878382 secs 2024-07-26T15:17:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15084.878382 secs 2024-07-26T15:17:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15084.878382 secs 2024-07-26T15:17:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:23 smithi144 ceph-mon[32129]: pgmap v8473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:24.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:23 smithi160 ceph-mon[36715]: pgmap v8473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:23 smithi183 ceph-mon[37273]: pgmap v8473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:25.389+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:17:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:25.390+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:17:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:25.390+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:17:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:25.390+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:17:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:25.390+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:25 smithi144 ceph-mon[32129]: pgmap v8474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:25 smithi160 ceph-mon[36715]: pgmap v8474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:25 smithi183 ceph-mon[37273]: pgmap v8474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:26 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:26 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:26 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15089.878459 secs 2024-07-26T15:17:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:27 smithi144 ceph-mon[32129]: pgmap v8475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:27 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15089.878459 secs 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:27 smithi183 ceph-mon[37273]: pgmap v8475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:27.970 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:27 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:28.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15089.878459 secs 2024-07-26T15:17:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:27 smithi160 ceph-mon[36715]: pgmap v8475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:27 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:29 smithi144 ceph-mon[32129]: pgmap v8476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:29 smithi160 ceph-mon[36715]: pgmap v8476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:29 smithi183 ceph-mon[37273]: pgmap v8476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:31 smithi160 ceph-mon[36715]: pgmap v8477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:31 smithi183 ceph-mon[37273]: pgmap v8477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:31 smithi144 ceph-mon[32129]: pgmap v8477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:32.487 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15094.878558 secs 2024-07-26T15:17:32.487 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:17:32.487 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:17:32.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:32 smithi160 ceph-mon[36715]: pgmap v8478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:17:32.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:32 smithi160 ceph-mon[36715]: pgmap v8479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:17:32.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15094.878558 secs 2024-07-26T15:17:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:17:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:17:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:32 smithi183 ceph-mon[37273]: pgmap v8478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:17:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:32 smithi183 ceph-mon[37273]: pgmap v8479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:17:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15094.878558 secs 2024-07-26T15:17:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:17:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:17:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:32 smithi144 ceph-mon[32129]: pgmap v8478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:17:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:32 smithi144 ceph-mon[32129]: pgmap v8479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:17:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:17:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:34 smithi183 ceph-mon[37273]: pgmap v8480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:17:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:34 smithi144 ceph-mon[32129]: pgmap v8480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:17:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:34 smithi160 ceph-mon[36715]: pgmap v8480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:17:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:36 smithi183 ceph-mon[37273]: pgmap v8481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:17:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:36 smithi144 ceph-mon[32129]: pgmap v8481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:17:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:36 smithi160 ceph-mon[36715]: pgmap v8481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:17:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15099.878626 secs 2024-07-26T15:17:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15099.878626 secs 2024-07-26T15:17:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15099.878626 secs 2024-07-26T15:17:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:38 smithi183 ceph-mon[37273]: pgmap v8482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:38 smithi144 ceph-mon[32129]: pgmap v8482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:38 smithi160 ceph-mon[36715]: pgmap v8482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:40.406+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:17:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:40.407+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:17:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:40.407+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:17:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:40.407+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:17:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:40.407+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:40 smithi183 ceph-mon[37273]: pgmap v8483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:40 smithi144 ceph-mon[32129]: pgmap v8483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:40 smithi160 ceph-mon[36715]: pgmap v8483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15104.878725 secs 2024-07-26T15:17:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:42 smithi183 ceph-mon[37273]: pgmap v8484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:17:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15104.878725 secs 2024-07-26T15:17:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:42 smithi144 ceph-mon[32129]: pgmap v8484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:17:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15104.878725 secs 2024-07-26T15:17:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:42 smithi160 ceph-mon[36715]: pgmap v8484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:17:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:44 smithi183 ceph-mon[37273]: pgmap v8485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:44 smithi144 ceph-mon[32129]: pgmap v8485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:44 smithi160 ceph-mon[36715]: pgmap v8485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:46 smithi144 ceph-mon[32129]: pgmap v8486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:46 smithi160 ceph-mon[36715]: pgmap v8486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:46 smithi183 ceph-mon[37273]: pgmap v8486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15109.878851 secs 2024-07-26T15:17:48.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15109.878851 secs 2024-07-26T15:17:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15109.878851 secs 2024-07-26T15:17:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:48.771+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:17:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:48.771+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T15:17:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:48.772+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:17:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:48.772+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x7f74b0109990 2024-07-26T15:17:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:48 smithi144 ceph-mon[32129]: pgmap v8487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:49.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:48 smithi160 ceph-mon[36715]: pgmap v8487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:48 smithi183 ceph-mon[37273]: pgmap v8487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:50 smithi144 ceph-mon[32129]: pgmap v8488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:50 smithi160 ceph-mon[36715]: pgmap v8488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:50 smithi183 ceph-mon[37273]: pgmap v8488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15114.878969 secs 2024-07-26T15:17:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:52 smithi144 ceph-mon[32129]: pgmap v8489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15114.878969 secs 2024-07-26T15:17:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:52 smithi160 ceph-mon[36715]: pgmap v8489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15114.878969 secs 2024-07-26T15:17:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:52 smithi183 ceph-mon[37273]: pgmap v8489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:17:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:54 smithi144 ceph-mon[32129]: pgmap v8490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:54 smithi160 ceph-mon[36715]: pgmap v8490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:54 smithi183 ceph-mon[37273]: pgmap v8490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:55.421+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:17:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:55.423+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:17:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:55.423+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:17:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:55.423+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:17:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:17:55.423+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:56 smithi144 ceph-mon[32129]: pgmap v8491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:57.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:56 smithi160 ceph-mon[36715]: pgmap v8491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:56 smithi183 ceph-mon[37273]: pgmap v8491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:17:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15119.879031 secs 2024-07-26T15:17:58.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15119.879031 secs 2024-07-26T15:17:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15119.879031 secs 2024-07-26T15:17:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:17:58 smithi144 ceph-mon[32129]: pgmap v8492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:17:58 smithi160 ceph-mon[36715]: pgmap v8492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:17:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:17:58 smithi183 ceph-mon[37273]: pgmap v8492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:00 smithi144 ceph-mon[32129]: pgmap v8493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:00 smithi160 ceph-mon[36715]: pgmap v8493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:00 smithi183 ceph-mon[37273]: pgmap v8493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15124.879152 secs 2024-07-26T15:18:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15124.879152 secs 2024-07-26T15:18:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15124.879152 secs 2024-07-26T15:18:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:02 smithi144 ceph-mon[32129]: pgmap v8494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:02 smithi160 ceph-mon[36715]: pgmap v8494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:02 smithi183 ceph-mon[37273]: pgmap v8494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:04 smithi144 ceph-mon[32129]: pgmap v8495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:04 smithi160 ceph-mon[36715]: pgmap v8495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:04 smithi183 ceph-mon[37273]: pgmap v8495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:06 smithi160 ceph-mon[36715]: pgmap v8496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15129.879312 secs 2024-07-26T15:18:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:06 smithi183 ceph-mon[37273]: pgmap v8496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15129.879312 secs 2024-07-26T15:18:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:06 smithi144 ceph-mon[32129]: pgmap v8496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15129.879312 secs 2024-07-26T15:18:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:08 smithi160 ceph-mon[36715]: pgmap v8497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:08 smithi183 ceph-mon[37273]: pgmap v8497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:08 smithi144 ceph-mon[32129]: pgmap v8497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:10.438+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:18:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:10.440+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:18:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:10.440+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:18:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:10.440+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:18:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:10.440+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:10 smithi160 ceph-mon[36715]: pgmap v8498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:10 smithi183 ceph-mon[37273]: pgmap v8498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:10 smithi144 ceph-mon[32129]: pgmap v8498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15134.879410 secs 2024-07-26T15:18:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15134.879410 secs 2024-07-26T15:18:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15134.879410 secs 2024-07-26T15:18:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:12 smithi160 ceph-mon[36715]: pgmap v8499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:12 smithi183 ceph-mon[37273]: pgmap v8499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:12 smithi144 ceph-mon[32129]: pgmap v8499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:14 smithi160 ceph-mon[36715]: pgmap v8500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:14 smithi183 ceph-mon[37273]: pgmap v8500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:14 smithi144 ceph-mon[32129]: pgmap v8500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:17.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:16 smithi160 ceph-mon[36715]: pgmap v8501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15139.879531 secs 2024-07-26T15:18:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:16 smithi183 ceph-mon[37273]: pgmap v8501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15139.879531 secs 2024-07-26T15:18:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:16 smithi144 ceph-mon[32129]: pgmap v8501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15139.879531 secs 2024-07-26T15:18:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:18 smithi160 ceph-mon[36715]: pgmap v8502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:18 smithi183 ceph-mon[37273]: pgmap v8502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:18 smithi144 ceph-mon[32129]: pgmap v8502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:20 smithi160 ceph-mon[36715]: pgmap v8503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:20 smithi183 ceph-mon[37273]: pgmap v8503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:20 smithi144 ceph-mon[32129]: pgmap v8503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15144.879664 secs 2024-07-26T15:18:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15144.879664 secs 2024-07-26T15:18:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15144.879664 secs 2024-07-26T15:18:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:22 smithi160 ceph-mon[36715]: pgmap v8504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:22 smithi183 ceph-mon[37273]: pgmap v8504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:22 smithi144 ceph-mon[32129]: pgmap v8504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:24 smithi160 ceph-mon[36715]: pgmap v8505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:24 smithi183 ceph-mon[37273]: pgmap v8505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:24 smithi144 ceph-mon[32129]: pgmap v8505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:25.455+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:18:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:25.456+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:18:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:25.456+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:18:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:25.456+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:18:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:25.456+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:26 smithi160 ceph-mon[36715]: pgmap v8506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15149.879789 secs 2024-07-26T15:18:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:26 smithi183 ceph-mon[37273]: pgmap v8506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15149.879789 secs 2024-07-26T15:18:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:26 smithi144 ceph-mon[32129]: pgmap v8506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15149.879789 secs 2024-07-26T15:18:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:28 smithi160 ceph-mon[36715]: pgmap v8507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:28 smithi183 ceph-mon[37273]: pgmap v8507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:28 smithi144 ceph-mon[32129]: pgmap v8507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:30 smithi160 ceph-mon[36715]: pgmap v8508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:30 smithi183 ceph-mon[37273]: pgmap v8508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:30 smithi144 ceph-mon[32129]: pgmap v8508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15154.879934 secs 2024-07-26T15:18:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15154.879934 secs 2024-07-26T15:18:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15154.879934 secs 2024-07-26T15:18:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:33.117 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:32 smithi144 ceph-mon[32129]: pgmap v8509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:33.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:33.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:33.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:18:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:32 smithi183 ceph-mon[37273]: pgmap v8509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:18:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:32 smithi160 ceph-mon[36715]: pgmap v8509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:18:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:33 smithi183 ceph-mon[37273]: pgmap v8510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:18:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:33 smithi183 ceph-mon[37273]: pgmap v8511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:18:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:33 smithi144 ceph-mon[32129]: pgmap v8510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:18:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:33 smithi144 ceph-mon[32129]: pgmap v8511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:18:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:33 smithi160 ceph-mon[36715]: pgmap v8510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:18:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:33 smithi160 ceph-mon[36715]: pgmap v8511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:18:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:35 smithi183 ceph-mon[37273]: pgmap v8512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:18:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:35 smithi144 ceph-mon[32129]: pgmap v8512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:18:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:35 smithi160 ceph-mon[36715]: pgmap v8512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:18:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15159.880056 secs 2024-07-26T15:18:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15159.880056 secs 2024-07-26T15:18:37.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15159.880056 secs 2024-07-26T15:18:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:37 smithi183 ceph-mon[37273]: pgmap v8513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:18:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:37 smithi144 ceph-mon[32129]: pgmap v8513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:18:38.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:37 smithi160 ceph-mon[36715]: pgmap v8513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:18:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:39 smithi183 ceph-mon[37273]: pgmap v8514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:18:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:39 smithi144 ceph-mon[32129]: pgmap v8514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:18:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:39 smithi160 ceph-mon[36715]: pgmap v8514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:18:40.473 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:40.472+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:18:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:40.473+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:18:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:40.473+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:18:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:40.473+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:18:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:40.473+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:41 smithi183 ceph-mon[37273]: pgmap v8515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:18:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15164.880198 secs 2024-07-26T15:18:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:41 smithi144 ceph-mon[32129]: pgmap v8515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:18:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15164.880198 secs 2024-07-26T15:18:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:41 smithi160 ceph-mon[36715]: pgmap v8515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:18:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15164.880198 secs 2024-07-26T15:18:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:43 smithi183 ceph-mon[37273]: pgmap v8516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:18:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:43 smithi144 ceph-mon[32129]: pgmap v8516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:18:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:43 smithi160 ceph-mon[36715]: pgmap v8516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:18:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:45 smithi183 ceph-mon[37273]: pgmap v8517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:45 smithi144 ceph-mon[32129]: pgmap v8517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:45 smithi160 ceph-mon[36715]: pgmap v8517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15169.880317 secs 2024-07-26T15:18:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15169.880317 secs 2024-07-26T15:18:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15169.880317 secs 2024-07-26T15:18:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:47 smithi144 ceph-mon[32129]: pgmap v8518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:47 smithi160 ceph-mon[36715]: pgmap v8518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:47 smithi183 ceph-mon[37273]: pgmap v8518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:49 smithi144 ceph-mon[32129]: pgmap v8519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:49 smithi160 ceph-mon[36715]: pgmap v8519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:49 smithi183 ceph-mon[37273]: pgmap v8519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:51 smithi144 ceph-mon[32129]: pgmap v8520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15174.880316 secs 2024-07-26T15:18:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:51 smithi160 ceph-mon[36715]: pgmap v8520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15174.880316 secs 2024-07-26T15:18:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:51 smithi183 ceph-mon[37273]: pgmap v8520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15174.880316 secs 2024-07-26T15:18:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:53 smithi144 ceph-mon[32129]: pgmap v8521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:53 smithi160 ceph-mon[36715]: pgmap v8521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:53 smithi183 ceph-mon[37273]: pgmap v8521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:18:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:55.489+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:18:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:55.490+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:18:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:55.490+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:18:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:55.490+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:18:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:18:55.490+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:55 smithi144 ceph-mon[32129]: pgmap v8522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:55 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15179.880439 secs 2024-07-26T15:18:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:55 smithi160 ceph-mon[36715]: pgmap v8522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:55 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15179.880439 secs 2024-07-26T15:18:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:55 smithi183 ceph-mon[37273]: pgmap v8522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:18:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:55 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15179.880439 secs 2024-07-26T15:18:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:57 smithi144 ceph-mon[32129]: pgmap v8523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:57 smithi160 ceph-mon[36715]: pgmap v8523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:18:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:57 smithi183 ceph-mon[37273]: pgmap v8523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:18:59 smithi144 ceph-mon[32129]: pgmap v8524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:18:59 smithi160 ceph-mon[36715]: pgmap v8524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:18:59 smithi183 ceph-mon[37273]: pgmap v8524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:00 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15184.880568 secs 2024-07-26T15:19:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:00 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15184.880568 secs 2024-07-26T15:19:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:00 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15184.880568 secs 2024-07-26T15:19:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:01 smithi144 ceph-mon[32129]: pgmap v8525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:01 smithi160 ceph-mon[36715]: pgmap v8525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:01 smithi183 ceph-mon[37273]: pgmap v8525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:03 smithi144 ceph-mon[32129]: pgmap v8526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:03 smithi160 ceph-mon[36715]: pgmap v8526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:03 smithi183 ceph-mon[37273]: pgmap v8526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:05 smithi144 ceph-mon[32129]: pgmap v8527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:05 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15189.880699 secs 2024-07-26T15:19:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:05 smithi160 ceph-mon[36715]: pgmap v8527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:05 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15189.880699 secs 2024-07-26T15:19:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:05 smithi183 ceph-mon[37273]: pgmap v8527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:05 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15189.880699 secs 2024-07-26T15:19:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:07 smithi144 ceph-mon[32129]: pgmap v8528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:07 smithi160 ceph-mon[36715]: pgmap v8528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:07 smithi183 ceph-mon[37273]: pgmap v8528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:09 smithi144 ceph-mon[32129]: pgmap v8529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:09 smithi160 ceph-mon[36715]: pgmap v8529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:09 smithi183 ceph-mon[37273]: pgmap v8529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:10.506+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:19:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:10.507+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:19:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:10.507+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:19:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:10.507+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:19:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:10.507+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15194.880824 secs 2024-07-26T15:19:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15194.880824 secs 2024-07-26T15:19:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15194.880824 secs 2024-07-26T15:19:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:11 smithi160 ceph-mon[36715]: pgmap v8530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:11 smithi183 ceph-mon[37273]: pgmap v8530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:11 smithi144 ceph-mon[32129]: pgmap v8530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:13 smithi160 ceph-mon[36715]: pgmap v8531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:13 smithi183 ceph-mon[37273]: pgmap v8531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:13 smithi144 ceph-mon[32129]: pgmap v8531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:15 smithi160 ceph-mon[36715]: pgmap v8532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:15 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15199.880932 secs 2024-07-26T15:19:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:15 smithi183 ceph-mon[37273]: pgmap v8532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:15 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15199.880932 secs 2024-07-26T15:19:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:15 smithi144 ceph-mon[32129]: pgmap v8532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:15 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15199.880932 secs 2024-07-26T15:19:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:17 smithi160 ceph-mon[36715]: pgmap v8533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:17 smithi183 ceph-mon[37273]: pgmap v8533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:17 smithi144 ceph-mon[32129]: pgmap v8533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:19 smithi160 ceph-mon[36715]: pgmap v8534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:19 smithi183 ceph-mon[37273]: pgmap v8534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:19 smithi144 ceph-mon[32129]: pgmap v8534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15204.881056 secs 2024-07-26T15:19:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15204.881056 secs 2024-07-26T15:19:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15204.881056 secs 2024-07-26T15:19:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:22 smithi160 ceph-mon[36715]: pgmap v8535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:22 smithi183 ceph-mon[37273]: pgmap v8535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:22 smithi144 ceph-mon[32129]: pgmap v8535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:24 smithi160 ceph-mon[36715]: pgmap v8536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:24 smithi183 ceph-mon[37273]: pgmap v8536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:24 smithi144 ceph-mon[32129]: pgmap v8536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:25.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:19:25.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:25.519+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:19:25.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:25.519+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:19:25.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:25.519+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:19:25.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:25.519+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:26 smithi160 ceph-mon[36715]: pgmap v8537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15209.881168 secs 2024-07-26T15:19:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:26 smithi183 ceph-mon[37273]: pgmap v8537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15209.881168 secs 2024-07-26T15:19:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:26 smithi144 ceph-mon[32129]: pgmap v8537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15209.881168 secs 2024-07-26T15:19:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:28 smithi160 ceph-mon[36715]: pgmap v8538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:28 smithi183 ceph-mon[37273]: pgmap v8538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:28 smithi144 ceph-mon[32129]: pgmap v8538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:30 smithi160 ceph-mon[36715]: pgmap v8539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:30 smithi183 ceph-mon[37273]: pgmap v8539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:30 smithi144 ceph-mon[32129]: pgmap v8539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15214.881320 secs 2024-07-26T15:19:31.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15214.881320 secs 2024-07-26T15:19:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15214.881320 secs 2024-07-26T15:19:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:32 smithi160 ceph-mon[36715]: pgmap v8540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:32 smithi183 ceph-mon[37273]: pgmap v8540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:32 smithi144 ceph-mon[32129]: pgmap v8540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:34 smithi183 ceph-mon[37273]: pgmap v8541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:19:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:34 smithi144 ceph-mon[32129]: pgmap v8541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:19:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:34 smithi160 ceph-mon[36715]: pgmap v8541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:19:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:35 smithi183 ceph-mon[37273]: pgmap v8542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:35 smithi183 ceph-mon[37273]: pgmap v8543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:19:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:35 smithi144 ceph-mon[32129]: pgmap v8542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:35 smithi144 ceph-mon[32129]: pgmap v8543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:19:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:35 smithi160 ceph-mon[36715]: pgmap v8542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:35 smithi160 ceph-mon[36715]: pgmap v8543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:19:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15219.881443 secs 2024-07-26T15:19:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15219.881443 secs 2024-07-26T15:19:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15219.881443 secs 2024-07-26T15:19:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:37 smithi183 ceph-mon[37273]: pgmap v8544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:37 smithi144 ceph-mon[32129]: pgmap v8544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:37 smithi160 ceph-mon[36715]: pgmap v8544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:39 smithi183 ceph-mon[37273]: pgmap v8545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:19:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:39 smithi144 ceph-mon[32129]: pgmap v8545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:19:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:39 smithi160 ceph-mon[36715]: pgmap v8545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:19:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:40.535+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:19:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:40.536+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:19:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:40.537+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:19:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:40.537+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:19:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:40.537+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:41 smithi183 ceph-mon[37273]: pgmap v8546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:19:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15224.881588 secs 2024-07-26T15:19:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:41 smithi144 ceph-mon[32129]: pgmap v8546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:19:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15224.881588 secs 2024-07-26T15:19:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:41 smithi160 ceph-mon[36715]: pgmap v8546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:19:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15224.881588 secs 2024-07-26T15:19:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:43 smithi183 ceph-mon[37273]: pgmap v8547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:43 smithi144 ceph-mon[32129]: pgmap v8547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:43 smithi160 ceph-mon[36715]: pgmap v8547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:19:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:45 smithi183 ceph-mon[37273]: pgmap v8548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:19:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:45 smithi144 ceph-mon[32129]: pgmap v8548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:19:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:45 smithi160 ceph-mon[36715]: pgmap v8548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:19:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15229.881705 secs 2024-07-26T15:19:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15229.881705 secs 2024-07-26T15:19:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15229.881705 secs 2024-07-26T15:19:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:47 smithi183 ceph-mon[37273]: pgmap v8549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:47 smithi144 ceph-mon[32129]: pgmap v8549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:47 smithi160 ceph-mon[36715]: pgmap v8549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:49 smithi183 ceph-mon[37273]: pgmap v8550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:49 smithi144 ceph-mon[32129]: pgmap v8550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:49 smithi160 ceph-mon[36715]: pgmap v8550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:51 smithi144 ceph-mon[32129]: pgmap v8551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15234.881823 secs 2024-07-26T15:19:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:51 smithi160 ceph-mon[36715]: pgmap v8551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15234.881823 secs 2024-07-26T15:19:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:51 smithi183 ceph-mon[37273]: pgmap v8551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15234.881823 secs 2024-07-26T15:19:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:53 smithi144 ceph-mon[32129]: pgmap v8552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:53.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:53 smithi160 ceph-mon[36715]: pgmap v8552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:53 smithi183 ceph-mon[37273]: pgmap v8552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:19:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:55 smithi144 ceph-mon[32129]: pgmap v8553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:55.552+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:19:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:55.553+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:19:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:55.553+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:19:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:55.553+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:19:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:19:55.553+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:55 smithi160 ceph-mon[36715]: pgmap v8553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:55 smithi183 ceph-mon[37273]: pgmap v8553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15239.881958 secs 2024-07-26T15:19:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15239.881958 secs 2024-07-26T15:19:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15239.881958 secs 2024-07-26T15:19:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:57 smithi144 ceph-mon[32129]: pgmap v8554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:57 smithi160 ceph-mon[36715]: pgmap v8554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:57 smithi183 ceph-mon[37273]: pgmap v8554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:19:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:19:59 smithi144 ceph-mon[32129]: pgmap v8555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:19:59 smithi160 ceph-mon[36715]: pgmap v8555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:19:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:19:59 smithi183 ceph-mon[37273]: pgmap v8555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:20:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:20:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:20:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:01 smithi144 ceph-mon[32129]: pgmap v8556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15244.882082 secs 2024-07-26T15:20:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:01 smithi160 ceph-mon[36715]: pgmap v8556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15244.882082 secs 2024-07-26T15:20:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:01 smithi183 ceph-mon[37273]: pgmap v8556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15244.882082 secs 2024-07-26T15:20:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:03 smithi144 ceph-mon[32129]: pgmap v8557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:03 smithi160 ceph-mon[36715]: pgmap v8557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:03 smithi183 ceph-mon[37273]: pgmap v8557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:05 smithi144 ceph-mon[32129]: pgmap v8558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:05 smithi160 ceph-mon[36715]: pgmap v8558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:05 smithi183 ceph-mon[37273]: pgmap v8558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15249.882204 secs 2024-07-26T15:20:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15249.882204 secs 2024-07-26T15:20:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15249.882204 secs 2024-07-26T15:20:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:07 smithi144 ceph-mon[32129]: pgmap v8559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:07 smithi160 ceph-mon[36715]: pgmap v8559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:07 smithi183 ceph-mon[37273]: pgmap v8559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:09 smithi144 ceph-mon[32129]: pgmap v8560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:09 smithi160 ceph-mon[36715]: pgmap v8560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:09 smithi183 ceph-mon[37273]: pgmap v8560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:10.569+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:20:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:10.570+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:20:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:10.570+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:20:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:10.570+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:20:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:10.570+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:11 smithi160 ceph-mon[36715]: pgmap v8561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15254.882331 secs 2024-07-26T15:20:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:11 smithi183 ceph-mon[37273]: pgmap v8561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15254.882331 secs 2024-07-26T15:20:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:11 smithi144 ceph-mon[32129]: pgmap v8561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15254.882331 secs 2024-07-26T15:20:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:13 smithi160 ceph-mon[36715]: pgmap v8562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:13 smithi183 ceph-mon[37273]: pgmap v8562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:13 smithi144 ceph-mon[32129]: pgmap v8562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:15 smithi160 ceph-mon[36715]: pgmap v8563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:15 smithi183 ceph-mon[37273]: pgmap v8563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:15 smithi144 ceph-mon[32129]: pgmap v8563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15259.882468 secs 2024-07-26T15:20:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15259.882468 secs 2024-07-26T15:20:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15259.882468 secs 2024-07-26T15:20:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:17 smithi160 ceph-mon[36715]: pgmap v8564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:17 smithi183 ceph-mon[37273]: pgmap v8564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:17 smithi144 ceph-mon[32129]: pgmap v8564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:19 smithi160 ceph-mon[36715]: pgmap v8565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:19 smithi183 ceph-mon[37273]: pgmap v8565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:19 smithi144 ceph-mon[32129]: pgmap v8565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:21 smithi160 ceph-mon[36715]: pgmap v8566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15264.882585 secs 2024-07-26T15:20:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:21 smithi183 ceph-mon[37273]: pgmap v8566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15264.882585 secs 2024-07-26T15:20:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:21 smithi144 ceph-mon[32129]: pgmap v8566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15264.882585 secs 2024-07-26T15:20:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:23 smithi144 ceph-mon[32129]: pgmap v8567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:23 smithi160 ceph-mon[36715]: pgmap v8567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:23 smithi183 ceph-mon[37273]: pgmap v8567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:25 smithi160 ceph-mon[36715]: pgmap v8568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:25.587+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:20:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:25.588+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:20:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:25.588+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:20:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:25.588+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:20:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:25.588+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:25 smithi183 ceph-mon[37273]: pgmap v8568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:25 smithi144 ceph-mon[32129]: pgmap v8568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15269.882700 secs 2024-07-26T15:20:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15269.882700 secs 2024-07-26T15:20:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15269.882700 secs 2024-07-26T15:20:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:27 smithi160 ceph-mon[36715]: pgmap v8569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:27 smithi183 ceph-mon[37273]: pgmap v8569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:27 smithi144 ceph-mon[32129]: pgmap v8569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:29 smithi160 ceph-mon[36715]: pgmap v8570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:29 smithi183 ceph-mon[37273]: pgmap v8570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:29 smithi144 ceph-mon[32129]: pgmap v8570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:31 smithi160 ceph-mon[36715]: pgmap v8571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15274.882820 secs 2024-07-26T15:20:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:31 smithi183 ceph-mon[37273]: pgmap v8571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15274.882820 secs 2024-07-26T15:20:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:31 smithi144 ceph-mon[32129]: pgmap v8571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15274.882820 secs 2024-07-26T15:20:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:33 smithi183 ceph-mon[37273]: pgmap v8572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:33 smithi144 ceph-mon[32129]: pgmap v8572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:33 smithi160 ceph-mon[36715]: pgmap v8572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:34.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:35 smithi183 ceph-mon[37273]: pgmap v8573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:20:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:35 smithi144 ceph-mon[32129]: pgmap v8573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:20:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:35 smithi160 ceph-mon[36715]: pgmap v8573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:20:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:36 smithi183 ceph-mon[37273]: pgmap v8574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:20:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:36 smithi183 ceph-mon[37273]: pgmap v8575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:20:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15279.882897 secs 2024-07-26T15:20:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:36 smithi144 ceph-mon[32129]: pgmap v8574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:20:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:36 smithi144 ceph-mon[32129]: pgmap v8575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:20:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15279.882897 secs 2024-07-26T15:20:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:36 smithi160 ceph-mon[36715]: pgmap v8574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:20:36.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:36 smithi160 ceph-mon[36715]: pgmap v8575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:20:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15279.882897 secs 2024-07-26T15:20:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:38 smithi183 ceph-mon[37273]: pgmap v8576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:20:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:38 smithi144 ceph-mon[32129]: pgmap v8576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:20:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:38 smithi160 ceph-mon[36715]: pgmap v8576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:20:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:40.604+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:20:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:40.605+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:20:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:40.605+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:20:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:40.605+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:20:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:40.605+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:40 smithi183 ceph-mon[37273]: pgmap v8577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:20:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:40 smithi144 ceph-mon[32129]: pgmap v8577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:20:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:40 smithi160 ceph-mon[36715]: pgmap v8577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:20:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15284.883015 secs 2024-07-26T15:20:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15284.883015 secs 2024-07-26T15:20:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15284.883015 secs 2024-07-26T15:20:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:42 smithi183 ceph-mon[37273]: pgmap v8578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:20:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:42 smithi144 ceph-mon[32129]: pgmap v8578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:20:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:42 smithi160 ceph-mon[36715]: pgmap v8578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:20:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:44 smithi183 ceph-mon[37273]: pgmap v8579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:20:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:44 smithi144 ceph-mon[32129]: pgmap v8579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:20:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:44 smithi160 ceph-mon[36715]: pgmap v8579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:20:46.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:20:46.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:20:46.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f74a8002230 tx=0x7f74a803e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:20:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f74a8002230 tx=0x7f74a803e040 comp rx=0 tx=0).stop 2024-07-26T15:20:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T15:20:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 msgr2=0x7f74900b51d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:20:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b60c0 0x7f74900b51d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:20:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.147+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:20:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.148+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:20:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:46.149+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a803e040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:20:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:46 smithi183 ceph-mon[37273]: pgmap v8580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:20:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15289.883131 secs 2024-07-26T15:20:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:46 smithi144 ceph-mon[32129]: pgmap v8580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:20:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15289.883131 secs 2024-07-26T15:20:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:46 smithi160 ceph-mon[36715]: pgmap v8580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:20:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15289.883131 secs 2024-07-26T15:20:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:48 smithi183 ceph-mon[37273]: pgmap v8581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:48 smithi144 ceph-mon[32129]: pgmap v8581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:48.804+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:20:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:48.804+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:20:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:48.804+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:20:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:48.804+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:20:48.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:48 smithi160 ceph-mon[36715]: pgmap v8581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:50 smithi183 ceph-mon[37273]: pgmap v8582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:50 smithi144 ceph-mon[32129]: pgmap v8582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:50.787 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:50 smithi160 ceph-mon[36715]: pgmap v8582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15294.883249 secs 2024-07-26T15:20:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15294.883249 secs 2024-07-26T15:20:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15294.883249 secs 2024-07-26T15:20:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:52 smithi144 ceph-mon[32129]: pgmap v8583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:52 smithi160 ceph-mon[36715]: pgmap v8583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:52 smithi183 ceph-mon[37273]: pgmap v8583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:54 smithi144 ceph-mon[32129]: pgmap v8584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:54.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:54 smithi160 ceph-mon[36715]: pgmap v8584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:54 smithi183 ceph-mon[37273]: pgmap v8584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:20:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:55.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:20:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:55.620+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:20:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:55.620+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:20:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:55.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:20:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:20:55.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:56 smithi144 ceph-mon[32129]: pgmap v8585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15299.883316 secs 2024-07-26T15:20:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:56 smithi160 ceph-mon[36715]: pgmap v8585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15299.883316 secs 2024-07-26T15:20:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:56 smithi183 ceph-mon[37273]: pgmap v8585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:20:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15299.883316 secs 2024-07-26T15:20:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:20:58 smithi144 ceph-mon[32129]: pgmap v8586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:20:58 smithi160 ceph-mon[36715]: pgmap v8586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:20:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:20:58 smithi183 ceph-mon[37273]: pgmap v8586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:00 smithi144 ceph-mon[32129]: pgmap v8587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:00 smithi160 ceph-mon[36715]: pgmap v8587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:00 smithi183 ceph-mon[37273]: pgmap v8587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15304.883449 secs 2024-07-26T15:21:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15304.883449 secs 2024-07-26T15:21:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15304.883449 secs 2024-07-26T15:21:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:02 smithi144 ceph-mon[32129]: pgmap v8588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:02.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:02 smithi160 ceph-mon[36715]: pgmap v8588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:02 smithi183 ceph-mon[37273]: pgmap v8588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:04 smithi144 ceph-mon[32129]: pgmap v8589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:04.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:04 smithi160 ceph-mon[36715]: pgmap v8589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:04 smithi183 ceph-mon[37273]: pgmap v8589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:06 smithi144 ceph-mon[32129]: pgmap v8590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15309.883572 secs 2024-07-26T15:21:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:06 smithi160 ceph-mon[36715]: pgmap v8590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15309.883572 secs 2024-07-26T15:21:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:06 smithi183 ceph-mon[37273]: pgmap v8590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15309.883572 secs 2024-07-26T15:21:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:08 smithi144 ceph-mon[32129]: pgmap v8591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:08 smithi160 ceph-mon[36715]: pgmap v8591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:08 smithi183 ceph-mon[37273]: pgmap v8591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:09 smithi144 ceph-mon[32129]: pgmap v8592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:09.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:09 smithi160 ceph-mon[36715]: pgmap v8592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:09 smithi183 ceph-mon[37273]: pgmap v8592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:10.627+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:21:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:10.628+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:21:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:10.628+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:21:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:10.628+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:21:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:10.628+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:10 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15314.883694 secs 2024-07-26T15:21:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:10 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15314.883694 secs 2024-07-26T15:21:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:10 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15314.883694 secs 2024-07-26T15:21:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:11 smithi160 ceph-mon[36715]: pgmap v8593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:11 smithi183 ceph-mon[37273]: pgmap v8593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:11 smithi144 ceph-mon[32129]: pgmap v8593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:14 smithi144 ceph-mon[32129]: pgmap v8594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:14 smithi160 ceph-mon[36715]: pgmap v8594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:14 smithi183 ceph-mon[37273]: pgmap v8594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:16 smithi144 ceph-mon[32129]: pgmap v8595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15319.883812 secs 2024-07-26T15:21:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:16 smithi160 ceph-mon[36715]: pgmap v8595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15319.883812 secs 2024-07-26T15:21:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:16 smithi183 ceph-mon[37273]: pgmap v8595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15319.883812 secs 2024-07-26T15:21:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:18 smithi144 ceph-mon[32129]: pgmap v8596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:18 smithi160 ceph-mon[36715]: pgmap v8596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:18 smithi183 ceph-mon[37273]: pgmap v8596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:20 smithi144 ceph-mon[32129]: pgmap v8597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:20 smithi160 ceph-mon[36715]: pgmap v8597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:20 smithi183 ceph-mon[37273]: pgmap v8597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15324.883934 secs 2024-07-26T15:21:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15324.883934 secs 2024-07-26T15:21:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15324.883934 secs 2024-07-26T15:21:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:22 smithi144 ceph-mon[32129]: pgmap v8598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:22 smithi160 ceph-mon[36715]: pgmap v8598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:22 smithi183 ceph-mon[37273]: pgmap v8598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:24 smithi144 ceph-mon[32129]: pgmap v8599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:24 smithi160 ceph-mon[36715]: pgmap v8599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:24 smithi183 ceph-mon[37273]: pgmap v8599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:25.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:25.636+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:21:25.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:25.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:21:25.638 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:25.637+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:21:25.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:25.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:21:25.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:25.637+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:26 smithi160 ceph-mon[36715]: pgmap v8600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15329.884055 secs 2024-07-26T15:21:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:26 smithi183 ceph-mon[37273]: pgmap v8600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15329.884055 secs 2024-07-26T15:21:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:26 smithi144 ceph-mon[32129]: pgmap v8600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15329.884055 secs 2024-07-26T15:21:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:28 smithi160 ceph-mon[36715]: pgmap v8601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:28 smithi183 ceph-mon[37273]: pgmap v8601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:28 smithi144 ceph-mon[32129]: pgmap v8601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:30 smithi160 ceph-mon[36715]: pgmap v8602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:30 smithi183 ceph-mon[37273]: pgmap v8602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:30 smithi144 ceph-mon[32129]: pgmap v8602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15334.884143 secs 2024-07-26T15:21:31.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15334.884143 secs 2024-07-26T15:21:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15334.884143 secs 2024-07-26T15:21:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:32 smithi160 ceph-mon[36715]: pgmap v8603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:32 smithi183 ceph-mon[37273]: pgmap v8603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:32 smithi144 ceph-mon[32129]: pgmap v8603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:33.812+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478004740 con 0x563c6c4182f0 2024-07-26T15:21:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:33.812+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:21:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:33.813+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478004740 con 0x7f74b0109990 2024-07-26T15:21:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:33.813+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d0e0 con 0x563c6c4182f0 2024-07-26T15:21:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:34 smithi160 ceph-mon[36715]: pgmap v8604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:34 smithi183 ceph-mon[37273]: pgmap v8604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:34 smithi144 ceph-mon[32129]: pgmap v8604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:36 smithi160 ceph-mon[36715]: pgmap v8605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15339.884237 secs 2024-07-26T15:21:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:36 smithi183 ceph-mon[37273]: pgmap v8605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15339.884237 secs 2024-07-26T15:21:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:36 smithi144 ceph-mon[32129]: pgmap v8605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15339.884237 secs 2024-07-26T15:21:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:37 smithi160 ceph-mon[36715]: pgmap v8606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:37 smithi160 ceph-mon[36715]: pgmap v8607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:21:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:37 smithi183 ceph-mon[37273]: pgmap v8606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:37 smithi183 ceph-mon[37273]: pgmap v8607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:21:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:37 smithi144 ceph-mon[32129]: pgmap v8606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:37 smithi144 ceph-mon[32129]: pgmap v8607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:21:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:39 smithi144 ceph-mon[32129]: pgmap v8608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:39 smithi160 ceph-mon[36715]: pgmap v8608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:39 smithi183 ceph-mon[37273]: pgmap v8608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:40.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:40.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:21:40.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:40.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:21:40.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:40.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:21:40.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:40.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:21:40.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:40.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:41.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:41 smithi183 ceph-mon[37273]: pgmap v8609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:41.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15344.884388 secs 2024-07-26T15:21:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:41 smithi144 ceph-mon[32129]: pgmap v8609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15344.884388 secs 2024-07-26T15:21:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:41 smithi160 ceph-mon[36715]: pgmap v8609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15344.884388 secs 2024-07-26T15:21:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:43 smithi144 ceph-mon[32129]: pgmap v8610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:43.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:43 smithi160 ceph-mon[36715]: pgmap v8610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:43 smithi183 ceph-mon[37273]: pgmap v8610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:45 smithi144 ceph-mon[32129]: pgmap v8611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:45.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:45 smithi160 ceph-mon[36715]: pgmap v8611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:45 smithi183 ceph-mon[37273]: pgmap v8611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:21:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15349.884503 secs 2024-07-26T15:21:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15349.884503 secs 2024-07-26T15:21:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15349.884503 secs 2024-07-26T15:21:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:47 smithi144 ceph-mon[32129]: pgmap v8612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:21:47.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:47 smithi160 ceph-mon[36715]: pgmap v8612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:21:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:47 smithi183 ceph-mon[37273]: pgmap v8612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:21:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:49 smithi144 ceph-mon[32129]: pgmap v8613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:49.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:49 smithi160 ceph-mon[36715]: pgmap v8613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:49 smithi183 ceph-mon[37273]: pgmap v8613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:51 smithi144 ceph-mon[32129]: pgmap v8614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15354.884570 secs 2024-07-26T15:21:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:51 smithi160 ceph-mon[36715]: pgmap v8614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15354.884570 secs 2024-07-26T15:21:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:51 smithi183 ceph-mon[37273]: pgmap v8614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15354.884570 secs 2024-07-26T15:21:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:53 smithi144 ceph-mon[32129]: pgmap v8615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:53.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:53 smithi160 ceph-mon[36715]: pgmap v8615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:53.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:53 smithi183 ceph-mon[37273]: pgmap v8615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:21:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:55.657+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:21:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:55.658+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:21:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:55.658+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:21:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:55.658+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:21:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:21:55.658+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:55 smithi144 ceph-mon[32129]: pgmap v8616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:55 smithi160 ceph-mon[36715]: pgmap v8616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:55 smithi183 ceph-mon[37273]: pgmap v8616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15359.884714 secs 2024-07-26T15:21:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15359.884714 secs 2024-07-26T15:21:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15359.884714 secs 2024-07-26T15:21:57.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:57 smithi160 ceph-mon[36715]: pgmap v8617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:57 smithi183 ceph-mon[37273]: pgmap v8617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:57 smithi144 ceph-mon[32129]: pgmap v8617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:21:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:21:58 smithi160 ceph-mon[36715]: pgmap v8618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:21:58 smithi183 ceph-mon[37273]: pgmap v8618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:21:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:21:58 smithi144 ceph-mon[32129]: pgmap v8618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:01.673 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:01 smithi144 ceph-mon[32129]: pgmap v8619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:01.673 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15364.884839 secs 2024-07-26T15:22:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:01 smithi160 ceph-mon[36715]: pgmap v8619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15364.884839 secs 2024-07-26T15:22:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:01 smithi183 ceph-mon[37273]: pgmap v8619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15364.884839 secs 2024-07-26T15:22:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:03 smithi144 ceph-mon[32129]: pgmap v8620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:03 smithi160 ceph-mon[36715]: pgmap v8620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:03 smithi183 ceph-mon[37273]: pgmap v8620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:05 smithi144 ceph-mon[32129]: pgmap v8621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:22:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:05 smithi160 ceph-mon[36715]: pgmap v8621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:22:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:05 smithi183 ceph-mon[37273]: pgmap v8621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:22:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15369.884973 secs 2024-07-26T15:22:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15369.884973 secs 2024-07-26T15:22:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15369.884973 secs 2024-07-26T15:22:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:07 smithi144 ceph-mon[32129]: pgmap v8622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:07.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:07 smithi160 ceph-mon[36715]: pgmap v8622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:07 smithi183 ceph-mon[37273]: pgmap v8622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:08 smithi183 ceph-mon[37273]: pgmap v8623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:08 smithi144 ceph-mon[32129]: pgmap v8623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:08 smithi160 ceph-mon[36715]: pgmap v8623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:10.674+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:22:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:10.675+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:22:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:10.675+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:22:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:10.675+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:22:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:10.675+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:11 smithi144 ceph-mon[32129]: pgmap v8624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15374.885045 secs 2024-07-26T15:22:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:11 smithi160 ceph-mon[36715]: pgmap v8624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15374.885045 secs 2024-07-26T15:22:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:11 smithi183 ceph-mon[37273]: pgmap v8624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15374.885045 secs 2024-07-26T15:22:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:13 smithi144 ceph-mon[32129]: pgmap v8625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:13.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:13 smithi160 ceph-mon[36715]: pgmap v8625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:13 smithi183 ceph-mon[37273]: pgmap v8625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:15 smithi144 ceph-mon[32129]: pgmap v8626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:22:15.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:15 smithi160 ceph-mon[36715]: pgmap v8626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:22:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:15 smithi183 ceph-mon[37273]: pgmap v8626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:22:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15379.885170 secs 2024-07-26T15:22:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15379.885170 secs 2024-07-26T15:22:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15379.885170 secs 2024-07-26T15:22:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:17 smithi144 ceph-mon[32129]: pgmap v8627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:17.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:17 smithi160 ceph-mon[36715]: pgmap v8627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:17 smithi183 ceph-mon[37273]: pgmap v8627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:19 smithi144 ceph-mon[32129]: pgmap v8628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:19.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:19 smithi160 ceph-mon[36715]: pgmap v8628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:19 smithi183 ceph-mon[37273]: pgmap v8628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:21 smithi144 ceph-mon[32129]: pgmap v8629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15384.885298 secs 2024-07-26T15:22:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:21 smithi160 ceph-mon[36715]: pgmap v8629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15384.885298 secs 2024-07-26T15:22:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:21 smithi183 ceph-mon[37273]: pgmap v8629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15384.885298 secs 2024-07-26T15:22:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:23 smithi144 ceph-mon[32129]: pgmap v8630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:23.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:23 smithi160 ceph-mon[36715]: pgmap v8630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:23 smithi183 ceph-mon[37273]: pgmap v8630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:25.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:22:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:25.682+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:22:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:25.682+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:22:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:25.682+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:22:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:25.682+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:25.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:25 smithi160 ceph-mon[36715]: pgmap v8631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:25 smithi183 ceph-mon[37273]: pgmap v8631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:25 smithi144 ceph-mon[32129]: pgmap v8631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15389.885446 secs 2024-07-26T15:22:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15389.885446 secs 2024-07-26T15:22:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15389.885446 secs 2024-07-26T15:22:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:27 smithi160 ceph-mon[36715]: pgmap v8632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:27 smithi183 ceph-mon[37273]: pgmap v8632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:27 smithi144 ceph-mon[32129]: pgmap v8632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:28 smithi183 ceph-mon[37273]: pgmap v8633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:28 smithi144 ceph-mon[32129]: pgmap v8633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:28 smithi160 ceph-mon[36715]: pgmap v8633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:31 smithi144 ceph-mon[32129]: pgmap v8634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15394.885567 secs 2024-07-26T15:22:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:31 smithi160 ceph-mon[36715]: pgmap v8634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15394.885567 secs 2024-07-26T15:22:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:31 smithi183 ceph-mon[37273]: pgmap v8634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15394.885567 secs 2024-07-26T15:22:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:33 smithi144 ceph-mon[32129]: pgmap v8635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:33.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:33 smithi160 ceph-mon[36715]: pgmap v8635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:33 smithi183 ceph-mon[37273]: pgmap v8635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:35 smithi144 ceph-mon[32129]: pgmap v8636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:35.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:35 smithi160 ceph-mon[36715]: pgmap v8636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:35 smithi183 ceph-mon[37273]: pgmap v8636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15399.885689 secs 2024-07-26T15:22:36.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15399.885689 secs 2024-07-26T15:22:36.980 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15399.885689 secs 2024-07-26T15:22:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:37 smithi144 ceph-mon[32129]: pgmap v8637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:37 smithi160 ceph-mon[36715]: pgmap v8637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:37.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:37 smithi183 ceph-mon[37273]: pgmap v8637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:38.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:22:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:22:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:22:39.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:39 smithi160 ceph-mon[36715]: pgmap v8638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:22:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:39 smithi160 ceph-mon[36715]: pgmap v8639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:39 smithi183 ceph-mon[37273]: pgmap v8638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:22:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:39 smithi183 ceph-mon[37273]: pgmap v8639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:39 smithi144 ceph-mon[32129]: pgmap v8638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:22:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:39 smithi144 ceph-mon[32129]: pgmap v8639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:40.697+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:22:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:40.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:22:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:40.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:22:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:40.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:22:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:40.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:40 smithi160 ceph-mon[36715]: pgmap v8640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:40 smithi183 ceph-mon[37273]: pgmap v8640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:40 smithi144 ceph-mon[32129]: pgmap v8640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:22:41.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15404.885811 secs 2024-07-26T15:22:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15404.885811 secs 2024-07-26T15:22:41.950 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15404.885811 secs 2024-07-26T15:22:42.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:42 smithi160 ceph-mon[36715]: pgmap v8641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:22:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:42 smithi183 ceph-mon[37273]: pgmap v8641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:22:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:42 smithi144 ceph-mon[32129]: pgmap v8641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:22:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:44 smithi183 ceph-mon[37273]: pgmap v8642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:22:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:44 smithi144 ceph-mon[32129]: pgmap v8642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:22:45.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:44 smithi160 ceph-mon[36715]: pgmap v8642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:22:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:46 smithi183 ceph-mon[37273]: pgmap v8643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:22:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15409.885950 secs 2024-07-26T15:22:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:46 smithi144 ceph-mon[32129]: pgmap v8643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:22:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15409.885950 secs 2024-07-26T15:22:47.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:46 smithi160 ceph-mon[36715]: pgmap v8643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:22:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15409.885950 secs 2024-07-26T15:22:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:48 smithi183 ceph-mon[37273]: pgmap v8644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:48 smithi144 ceph-mon[32129]: pgmap v8644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:48 smithi160 ceph-mon[36715]: pgmap v8644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:22:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:50 smithi183 ceph-mon[37273]: pgmap v8645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:50 smithi144 ceph-mon[32129]: pgmap v8645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:50 smithi160 ceph-mon[36715]: pgmap v8645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15414.886039 secs 2024-07-26T15:22:51.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15414.886039 secs 2024-07-26T15:22:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15414.886039 secs 2024-07-26T15:22:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:52 smithi183 ceph-mon[37273]: pgmap v8646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:52 smithi144 ceph-mon[32129]: pgmap v8646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:53.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:52 smithi160 ceph-mon[36715]: pgmap v8646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:22:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:54 smithi183 ceph-mon[37273]: pgmap v8647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:54 smithi144 ceph-mon[32129]: pgmap v8647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:54 smithi160 ceph-mon[36715]: pgmap v8647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:55.712+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:22:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:55.713+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:22:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:55.713+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:22:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:55.713+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:22:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:22:55.713+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:56 smithi183 ceph-mon[37273]: pgmap v8648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15419.886163 secs 2024-07-26T15:22:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:56 smithi144 ceph-mon[32129]: pgmap v8648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15419.886163 secs 2024-07-26T15:22:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:56 smithi160 ceph-mon[36715]: pgmap v8648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:22:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15419.886163 secs 2024-07-26T15:22:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:22:58 smithi144 ceph-mon[32129]: pgmap v8649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:22:58 smithi160 ceph-mon[36715]: pgmap v8649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:22:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:22:58 smithi183 ceph-mon[37273]: pgmap v8649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:00 smithi144 ceph-mon[32129]: pgmap v8650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:00 smithi160 ceph-mon[36715]: pgmap v8650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:00 smithi183 ceph-mon[37273]: pgmap v8650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15424.886282 secs 2024-07-26T15:23:02.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15424.886282 secs 2024-07-26T15:23:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15424.886282 secs 2024-07-26T15:23:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:02 smithi144 ceph-mon[32129]: pgmap v8651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:02 smithi160 ceph-mon[36715]: pgmap v8651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:02 smithi183 ceph-mon[37273]: pgmap v8651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:04 smithi144 ceph-mon[32129]: pgmap v8652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:04 smithi160 ceph-mon[36715]: pgmap v8652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:04 smithi183 ceph-mon[37273]: pgmap v8652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:06 smithi144 ceph-mon[32129]: pgmap v8653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15429.886368 secs 2024-07-26T15:23:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:06 smithi160 ceph-mon[36715]: pgmap v8653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:07.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15429.886368 secs 2024-07-26T15:23:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:06 smithi183 ceph-mon[37273]: pgmap v8653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15429.886368 secs 2024-07-26T15:23:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:08 smithi144 ceph-mon[32129]: pgmap v8654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:08 smithi160 ceph-mon[36715]: pgmap v8654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:08 smithi183 ceph-mon[37273]: pgmap v8654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:10.719+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:23:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:10.720+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:23:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:10.720+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:23:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:10.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:23:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:10.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:10 smithi144 ceph-mon[32129]: pgmap v8655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:10 smithi160 ceph-mon[36715]: pgmap v8655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:10 smithi183 ceph-mon[37273]: pgmap v8655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15434.886489 secs 2024-07-26T15:23:12.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15434.886489 secs 2024-07-26T15:23:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15434.886489 secs 2024-07-26T15:23:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:12 smithi144 ceph-mon[32129]: pgmap v8656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:12 smithi160 ceph-mon[36715]: pgmap v8656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:12 smithi183 ceph-mon[37273]: pgmap v8656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:14 smithi144 ceph-mon[32129]: pgmap v8657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:14 smithi160 ceph-mon[36715]: pgmap v8657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:14 smithi183 ceph-mon[37273]: pgmap v8657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:16 smithi144 ceph-mon[32129]: pgmap v8658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15439.886587 secs 2024-07-26T15:23:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:16 smithi160 ceph-mon[36715]: pgmap v8658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15439.886587 secs 2024-07-26T15:23:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:16 smithi183 ceph-mon[37273]: pgmap v8658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15439.886587 secs 2024-07-26T15:23:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:18 smithi144 ceph-mon[32129]: pgmap v8659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:18 smithi160 ceph-mon[36715]: pgmap v8659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:18 smithi183 ceph-mon[37273]: pgmap v8659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:20 smithi144 ceph-mon[32129]: pgmap v8660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:21.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:20 smithi160 ceph-mon[36715]: pgmap v8660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:20 smithi183 ceph-mon[37273]: pgmap v8660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15444.886713 secs 2024-07-26T15:23:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15444.886713 secs 2024-07-26T15:23:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15444.886713 secs 2024-07-26T15:23:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:22 smithi160 ceph-mon[36715]: pgmap v8661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:22 smithi183 ceph-mon[37273]: pgmap v8661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:22 smithi144 ceph-mon[32129]: pgmap v8661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:25.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:24 smithi160 ceph-mon[36715]: pgmap v8662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:24 smithi183 ceph-mon[37273]: pgmap v8662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:24 smithi144 ceph-mon[32129]: pgmap v8662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:25.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:23:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:25.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:23:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:25.737+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:23:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:25.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:23:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:25.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:26 smithi160 ceph-mon[36715]: pgmap v8663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15449.886874 secs 2024-07-26T15:23:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:26 smithi183 ceph-mon[37273]: pgmap v8663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15449.886874 secs 2024-07-26T15:23:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:26 smithi144 ceph-mon[32129]: pgmap v8663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15449.886874 secs 2024-07-26T15:23:29.055 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:28 smithi144 ceph-mon[32129]: pgmap v8664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:28 smithi160 ceph-mon[36715]: pgmap v8664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:28 smithi183 ceph-mon[37273]: pgmap v8664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:31.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:30 smithi160 ceph-mon[36715]: pgmap v8665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:30 smithi183 ceph-mon[37273]: pgmap v8665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:30 smithi144 ceph-mon[32129]: pgmap v8665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:32.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15454.886994 secs 2024-07-26T15:23:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15454.886994 secs 2024-07-26T15:23:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15454.886994 secs 2024-07-26T15:23:33.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:32 smithi160 ceph-mon[36715]: pgmap v8666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:32 smithi183 ceph-mon[37273]: pgmap v8666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:32 smithi144 ceph-mon[32129]: pgmap v8666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:35.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:34 smithi160 ceph-mon[36715]: pgmap v8667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:34 smithi183 ceph-mon[37273]: pgmap v8667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:34 smithi144 ceph-mon[32129]: pgmap v8667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:37.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:36 smithi160 ceph-mon[36715]: pgmap v8668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15459.887092 secs 2024-07-26T15:23:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:36 smithi183 ceph-mon[37273]: pgmap v8668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:37.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15459.887092 secs 2024-07-26T15:23:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:36 smithi144 ceph-mon[32129]: pgmap v8668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15459.887092 secs 2024-07-26T15:23:38.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: pgmap v8669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:23:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: pgmap v8669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: pgmap v8669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:23:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:39.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:23:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:39 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:23:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:39 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:23:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:39 smithi183 ceph-mon[37273]: pgmap v8670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:39 smithi183 ceph-mon[37273]: pgmap v8671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:23:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:39 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:23:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:39 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:23:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:39 smithi144 ceph-mon[32129]: pgmap v8670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:39 smithi144 ceph-mon[32129]: pgmap v8671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:23:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:39 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:23:40.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:39 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:23:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:39 smithi160 ceph-mon[36715]: pgmap v8670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:39 smithi160 ceph-mon[36715]: pgmap v8671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:40.752+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:23:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:40.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:23:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:40.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:23:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:40.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:23:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:40.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:41 smithi183 ceph-mon[37273]: pgmap v8672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15464.887235 secs 2024-07-26T15:23:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:41 smithi144 ceph-mon[32129]: pgmap v8672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15464.887235 secs 2024-07-26T15:23:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:41 smithi160 ceph-mon[36715]: pgmap v8672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15464.887235 secs 2024-07-26T15:23:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:43 smithi183 ceph-mon[37273]: pgmap v8673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:23:44.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:43 smithi144 ceph-mon[32129]: pgmap v8673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:23:44.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:43 smithi160 ceph-mon[36715]: pgmap v8673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:23:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:45 smithi183 ceph-mon[37273]: pgmap v8674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:23:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:45 smithi144 ceph-mon[32129]: pgmap v8674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:23:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:45 smithi160 ceph-mon[36715]: pgmap v8674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:23:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15469.887340 secs 2024-07-26T15:23:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15469.887340 secs 2024-07-26T15:23:47.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15469.887340 secs 2024-07-26T15:23:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:47 smithi183 ceph-mon[37273]: pgmap v8675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:47 smithi144 ceph-mon[32129]: pgmap v8675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:48.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:47 smithi160 ceph-mon[36715]: pgmap v8675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:23:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:48.834+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T15:23:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:48.834+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478004740 con 0x563c6c4182f0 2024-07-26T15:23:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:48.835+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480004740 con 0x7f74b0109990 2024-07-26T15:23:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:48.835+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:23:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:49 smithi183 ceph-mon[37273]: pgmap v8676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:23:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:49 smithi144 ceph-mon[32129]: pgmap v8676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:23:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:49 smithi160 ceph-mon[36715]: pgmap v8676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:23:52.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:51 smithi160 ceph-mon[36715]: pgmap v8677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:52.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15474.887459 secs 2024-07-26T15:23:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:51 smithi183 ceph-mon[37273]: pgmap v8677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15474.887459 secs 2024-07-26T15:23:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:51 smithi144 ceph-mon[32129]: pgmap v8677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15474.887459 secs 2024-07-26T15:23:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:53 smithi183 ceph-mon[37273]: pgmap v8678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:53 smithi144 ceph-mon[32129]: pgmap v8678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:53 smithi160 ceph-mon[36715]: pgmap v8678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:23:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:55.757+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:23:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:55.758+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:23:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:55.758+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:23:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:55.758+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:23:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:23:55.759+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:55 smithi183 ceph-mon[37273]: pgmap v8679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:55 smithi144 ceph-mon[32129]: pgmap v8679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:55 smithi160 ceph-mon[36715]: pgmap v8679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:23:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15479.887541 secs 2024-07-26T15:23:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15479.887541 secs 2024-07-26T15:23:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15479.887541 secs 2024-07-26T15:23:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:57 smithi144 ceph-mon[32129]: pgmap v8680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:58.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:57 smithi160 ceph-mon[36715]: pgmap v8680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:23:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:57 smithi183 ceph-mon[37273]: pgmap v8680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:23:59 smithi144 ceph-mon[32129]: pgmap v8681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:23:59 smithi160 ceph-mon[36715]: pgmap v8681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:23:59 smithi183 ceph-mon[37273]: pgmap v8681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:01 smithi144 ceph-mon[32129]: pgmap v8682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15484.887629 secs 2024-07-26T15:24:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:01 smithi160 ceph-mon[36715]: pgmap v8682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15484.887629 secs 2024-07-26T15:24:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:01 smithi183 ceph-mon[37273]: pgmap v8682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15484.887629 secs 2024-07-26T15:24:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:03 smithi144 ceph-mon[32129]: pgmap v8683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:03 smithi160 ceph-mon[36715]: pgmap v8683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:03 smithi183 ceph-mon[37273]: pgmap v8683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:05 smithi144 ceph-mon[32129]: pgmap v8684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:06.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:05 smithi160 ceph-mon[36715]: pgmap v8684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:05 smithi183 ceph-mon[37273]: pgmap v8684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15489.887727 secs 2024-07-26T15:24:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15489.887727 secs 2024-07-26T15:24:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15489.887727 secs 2024-07-26T15:24:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:07 smithi144 ceph-mon[32129]: pgmap v8685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:24:08.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:07 smithi160 ceph-mon[36715]: pgmap v8685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:24:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:07 smithi183 ceph-mon[37273]: pgmap v8685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:24:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:09 smithi144 ceph-mon[32129]: pgmap v8686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:10.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:09 smithi160 ceph-mon[36715]: pgmap v8686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:09 smithi183 ceph-mon[37273]: pgmap v8686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:10.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:24:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:10.775+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:24:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:10.775+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:24:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:10.776+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:24:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:10.776+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:11 smithi144 ceph-mon[32129]: pgmap v8687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15494.887812 secs 2024-07-26T15:24:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:11 smithi160 ceph-mon[36715]: pgmap v8687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:12.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15494.887812 secs 2024-07-26T15:24:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:11 smithi183 ceph-mon[37273]: pgmap v8687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:24:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15494.887812 secs 2024-07-26T15:24:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:13 smithi144 ceph-mon[32129]: pgmap v8688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:14.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:13 smithi160 ceph-mon[36715]: pgmap v8688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:13 smithi183 ceph-mon[37273]: pgmap v8688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:15 smithi144 ceph-mon[32129]: pgmap v8689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:16.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:15 smithi160 ceph-mon[36715]: pgmap v8689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:15 smithi183 ceph-mon[37273]: pgmap v8689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15499.887929 secs 2024-07-26T15:24:17.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15499.887929 secs 2024-07-26T15:24:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15499.887929 secs 2024-07-26T15:24:18.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:17 smithi160 ceph-mon[36715]: pgmap v8690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:17 smithi183 ceph-mon[37273]: pgmap v8690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:17 smithi144 ceph-mon[32129]: pgmap v8690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:20.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:19 smithi160 ceph-mon[36715]: pgmap v8691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:19 smithi183 ceph-mon[37273]: pgmap v8691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:19 smithi144 ceph-mon[32129]: pgmap v8691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:21.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:20 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15504.888051 secs 2024-07-26T15:24:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:20 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15504.888051 secs 2024-07-26T15:24:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:20 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15504.888051 secs 2024-07-26T15:24:22.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:21 smithi160 ceph-mon[36715]: pgmap v8692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:21 smithi183 ceph-mon[37273]: pgmap v8692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:21 smithi144 ceph-mon[32129]: pgmap v8692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:24.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:23 smithi160 ceph-mon[36715]: pgmap v8693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:23 smithi183 ceph-mon[37273]: pgmap v8693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:23 smithi144 ceph-mon[32129]: pgmap v8693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:25.790+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:24:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:25.792+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:24:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:25.792+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:24:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:25.792+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:24:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:25.792+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:26 smithi160 ceph-mon[36715]: pgmap v8694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15509.888167 secs 2024-07-26T15:24:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:26 smithi183 ceph-mon[37273]: pgmap v8694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15509.888167 secs 2024-07-26T15:24:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:25 smithi144 ceph-mon[32129]: pgmap v8694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15509.888167 secs 2024-07-26T15:24:28.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:28 smithi160 ceph-mon[36715]: pgmap v8695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:28 smithi183 ceph-mon[37273]: pgmap v8695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:28 smithi144 ceph-mon[32129]: pgmap v8695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:30.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:30 smithi160 ceph-mon[36715]: pgmap v8696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:30 smithi183 ceph-mon[37273]: pgmap v8696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:30 smithi144 ceph-mon[32129]: pgmap v8696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15514.888265 secs 2024-07-26T15:24:31.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15514.888265 secs 2024-07-26T15:24:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15514.888265 secs 2024-07-26T15:24:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:32 smithi160 ceph-mon[36715]: pgmap v8697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:32 smithi183 ceph-mon[37273]: pgmap v8697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:32 smithi144 ceph-mon[32129]: pgmap v8697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:34.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:34 smithi160 ceph-mon[36715]: pgmap v8698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:34 smithi183 ceph-mon[37273]: pgmap v8698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:34 smithi144 ceph-mon[32129]: pgmap v8698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:36.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:36 smithi160 ceph-mon[36715]: pgmap v8699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15519.888386 secs 2024-07-26T15:24:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:36 smithi183 ceph-mon[37273]: pgmap v8699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15519.888386 secs 2024-07-26T15:24:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:36 smithi144 ceph-mon[32129]: pgmap v8699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15519.888386 secs 2024-07-26T15:24:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:38 smithi160 ceph-mon[36715]: pgmap v8700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:38 smithi183 ceph-mon[37273]: pgmap v8700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:38 smithi144 ceph-mon[32129]: pgmap v8700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:39.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: pgmap v8701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:24:40.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:40.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:40.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:24:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: pgmap v8701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:40.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: pgmap v8701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:40.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:24:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:40.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:24:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:40.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:24:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:40.808+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:24:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:40.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:24:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:40.808+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:41 smithi183 ceph-mon[37273]: pgmap v8702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:24:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:41 smithi183 ceph-mon[37273]: pgmap v8703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15524.888517 secs 2024-07-26T15:24:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:41 smithi144 ceph-mon[32129]: pgmap v8702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:24:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:41 smithi144 ceph-mon[32129]: pgmap v8703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15524.888517 secs 2024-07-26T15:24:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:41 smithi160 ceph-mon[36715]: pgmap v8702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:24:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:41 smithi160 ceph-mon[36715]: pgmap v8703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15524.888517 secs 2024-07-26T15:24:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:43 smithi183 ceph-mon[37273]: pgmap v8704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:24:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:43 smithi144 ceph-mon[32129]: pgmap v8704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:24:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:43 smithi160 ceph-mon[36715]: pgmap v8704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:24:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:45 smithi183 ceph-mon[37273]: pgmap v8705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:24:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:45 smithi144 ceph-mon[32129]: pgmap v8705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:24:45.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:45 smithi160 ceph-mon[36715]: pgmap v8705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:24:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15529.888576 secs 2024-07-26T15:24:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15529.888576 secs 2024-07-26T15:24:46.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15529.888576 secs 2024-07-26T15:24:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:47 smithi144 ceph-mon[32129]: pgmap v8706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:24:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:47 smithi183 ceph-mon[37273]: pgmap v8706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:24:47.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:47 smithi160 ceph-mon[36715]: pgmap v8706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:24:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:49 smithi183 ceph-mon[37273]: pgmap v8707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:49 smithi144 ceph-mon[32129]: pgmap v8707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:49.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:49 smithi160 ceph-mon[36715]: pgmap v8707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:24:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:51 smithi183 ceph-mon[37273]: pgmap v8708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:24:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15534.888703 secs 2024-07-26T15:24:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:51 smithi144 ceph-mon[32129]: pgmap v8708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:24:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15534.888703 secs 2024-07-26T15:24:51.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:51 smithi160 ceph-mon[36715]: pgmap v8708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:24:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15534.888703 secs 2024-07-26T15:24:53.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:53 smithi160 ceph-mon[36715]: pgmap v8709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:53 smithi183 ceph-mon[37273]: pgmap v8709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:53 smithi144 ceph-mon[32129]: pgmap v8709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:24:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:55 smithi183 ceph-mon[37273]: pgmap v8710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:55 smithi144 ceph-mon[32129]: pgmap v8710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:55 smithi160 ceph-mon[36715]: pgmap v8710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:55.825+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:24:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:55.826+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:24:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:55.826+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:24:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:55.826+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:24:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:24:55.826+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15539.888827 secs 2024-07-26T15:24:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15539.888827 secs 2024-07-26T15:24:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15539.888827 secs 2024-07-26T15:24:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:57 smithi183 ceph-mon[37273]: pgmap v8711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:57 smithi144 ceph-mon[32129]: pgmap v8711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:57.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:57 smithi160 ceph-mon[36715]: pgmap v8711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:24:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:24:59 smithi144 ceph-mon[32129]: pgmap v8712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:59.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:24:59 smithi160 ceph-mon[36715]: pgmap v8712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:24:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:24:59 smithi183 ceph-mon[37273]: pgmap v8712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:01 smithi144 ceph-mon[32129]: pgmap v8713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15544.888955 secs 2024-07-26T15:25:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:01 smithi160 ceph-mon[36715]: pgmap v8713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15544.888955 secs 2024-07-26T15:25:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:01 smithi183 ceph-mon[37273]: pgmap v8713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15544.888955 secs 2024-07-26T15:25:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:03 smithi144 ceph-mon[32129]: pgmap v8714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:03.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:03 smithi160 ceph-mon[36715]: pgmap v8714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:03 smithi183 ceph-mon[37273]: pgmap v8714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:05 smithi144 ceph-mon[32129]: pgmap v8715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:05 smithi160 ceph-mon[36715]: pgmap v8715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:05 smithi183 ceph-mon[37273]: pgmap v8715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15549.889083 secs 2024-07-26T15:25:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15549.889083 secs 2024-07-26T15:25:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15549.889083 secs 2024-07-26T15:25:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:07 smithi144 ceph-mon[32129]: pgmap v8716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:07.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:07 smithi160 ceph-mon[36715]: pgmap v8716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:07 smithi183 ceph-mon[37273]: pgmap v8716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:09 smithi144 ceph-mon[32129]: pgmap v8717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:09.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:09 smithi160 ceph-mon[36715]: pgmap v8717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:09 smithi183 ceph-mon[37273]: pgmap v8717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:10.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:25:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:10.831+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:25:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:10.831+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:25:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:10.831+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:25:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:10.831+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:11 smithi144 ceph-mon[32129]: pgmap v8718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15554.889200 secs 2024-07-26T15:25:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:11 smithi160 ceph-mon[36715]: pgmap v8718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15554.889200 secs 2024-07-26T15:25:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:11 smithi183 ceph-mon[37273]: pgmap v8718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15554.889200 secs 2024-07-26T15:25:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:13 smithi144 ceph-mon[32129]: pgmap v8719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:13 smithi160 ceph-mon[36715]: pgmap v8719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:13 smithi183 ceph-mon[37273]: pgmap v8719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:15 smithi144 ceph-mon[32129]: pgmap v8720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:15.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:15 smithi160 ceph-mon[36715]: pgmap v8720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:15 smithi183 ceph-mon[37273]: pgmap v8720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15559.889340 secs 2024-07-26T15:25:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15559.889340 secs 2024-07-26T15:25:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15559.889340 secs 2024-07-26T15:25:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:17 smithi144 ceph-mon[32129]: pgmap v8721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:17 smithi160 ceph-mon[36715]: pgmap v8721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:17 smithi183 ceph-mon[37273]: pgmap v8721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:19 smithi144 ceph-mon[32129]: pgmap v8722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:19.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:19 smithi160 ceph-mon[36715]: pgmap v8722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:19 smithi183 ceph-mon[37273]: pgmap v8722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:21 smithi144 ceph-mon[32129]: pgmap v8723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15564.889463 secs 2024-07-26T15:25:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:21 smithi160 ceph-mon[36715]: pgmap v8723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15564.889463 secs 2024-07-26T15:25:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:21 smithi183 ceph-mon[37273]: pgmap v8723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15564.889463 secs 2024-07-26T15:25:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:23 smithi160 ceph-mon[36715]: pgmap v8724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:23 smithi183 ceph-mon[37273]: pgmap v8724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:23 smithi144 ceph-mon[32129]: pgmap v8724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:25.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:25 smithi160 ceph-mon[36715]: pgmap v8725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:25 smithi183 ceph-mon[37273]: pgmap v8725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:25 smithi144 ceph-mon[32129]: pgmap v8725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:25.847+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:25:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:25.848+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:25:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:25.848+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:25:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:25.848+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:25:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:25.848+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15569.889578 secs 2024-07-26T15:25:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15569.889578 secs 2024-07-26T15:25:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15569.889578 secs 2024-07-26T15:25:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:27 smithi160 ceph-mon[36715]: pgmap v8726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:27 smithi183 ceph-mon[37273]: pgmap v8726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:27 smithi144 ceph-mon[32129]: pgmap v8726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:29.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:29 smithi160 ceph-mon[36715]: pgmap v8727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:29 smithi183 ceph-mon[37273]: pgmap v8727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:29 smithi144 ceph-mon[32129]: pgmap v8727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:31.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:31 smithi144 ceph-mon[32129]: pgmap v8728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:31.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15574.889660 secs 2024-07-26T15:25:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:31 smithi160 ceph-mon[36715]: pgmap v8728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15574.889660 secs 2024-07-26T15:25:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:31 smithi183 ceph-mon[37273]: pgmap v8728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15574.889660 secs 2024-07-26T15:25:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:33 smithi160 ceph-mon[36715]: pgmap v8729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:33 smithi183 ceph-mon[37273]: pgmap v8729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:33 smithi144 ceph-mon[32129]: pgmap v8729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:35.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:35 smithi160 ceph-mon[36715]: pgmap v8730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:35 smithi183 ceph-mon[37273]: pgmap v8730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:35 smithi144 ceph-mon[32129]: pgmap v8730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15579.889807 secs 2024-07-26T15:25:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15579.889807 secs 2024-07-26T15:25:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15579.889807 secs 2024-07-26T15:25:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:37 smithi160 ceph-mon[36715]: pgmap v8731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:37 smithi183 ceph-mon[37273]: pgmap v8731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:37 smithi144 ceph-mon[32129]: pgmap v8731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:39.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:39 smithi160 ceph-mon[36715]: pgmap v8732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:39 smithi183 ceph-mon[37273]: pgmap v8732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:39 smithi144 ceph-mon[32129]: pgmap v8732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:40.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:40.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:25:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:40.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:25:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:40.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:25:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:40.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:25:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:40.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:41 smithi160 ceph-mon[36715]: pgmap v8733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15584.889897 secs 2024-07-26T15:25:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:25:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:25:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:25:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:41 smithi183 ceph-mon[37273]: pgmap v8733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15584.889897 secs 2024-07-26T15:25:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:25:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:25:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:25:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:41 smithi144 ceph-mon[32129]: pgmap v8733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15584.889897 secs 2024-07-26T15:25:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:25:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:25:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:25:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:42 smithi160 ceph-mon[36715]: pgmap v8734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:42 smithi160 ceph-mon[36715]: pgmap v8735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:42 smithi183 ceph-mon[37273]: pgmap v8734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:42 smithi183 ceph-mon[37273]: pgmap v8735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:42 smithi144 ceph-mon[32129]: pgmap v8734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:42 smithi144 ceph-mon[32129]: pgmap v8735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:44 smithi183 ceph-mon[37273]: pgmap v8736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:25:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:44 smithi144 ceph-mon[32129]: pgmap v8736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:25:44.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:44 smithi160 ceph-mon[36715]: pgmap v8736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:25:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:46 smithi183 ceph-mon[37273]: pgmap v8737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:25:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15589.890018 secs 2024-07-26T15:25:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:46 smithi144 ceph-mon[32129]: pgmap v8737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:25:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15589.890018 secs 2024-07-26T15:25:46.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:46 smithi160 ceph-mon[36715]: pgmap v8737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:25:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15589.890018 secs 2024-07-26T15:25:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:48 smithi183 ceph-mon[37273]: pgmap v8738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:48 smithi144 ceph-mon[32129]: pgmap v8738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:48 smithi160 ceph-mon[36715]: pgmap v8738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:50 smithi183 ceph-mon[37273]: pgmap v8739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:50 smithi144 ceph-mon[32129]: pgmap v8739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:50.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:50 smithi160 ceph-mon[36715]: pgmap v8739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:25:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15594.890148 secs 2024-07-26T15:25:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15594.890148 secs 2024-07-26T15:25:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15594.890148 secs 2024-07-26T15:25:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:52 smithi183 ceph-mon[37273]: pgmap v8740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:25:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:52 smithi144 ceph-mon[32129]: pgmap v8740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:25:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:52 smithi160 ceph-mon[36715]: pgmap v8740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:25:54.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:54 smithi160 ceph-mon[36715]: pgmap v8741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:54 smithi183 ceph-mon[37273]: pgmap v8741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:54 smithi144 ceph-mon[32129]: pgmap v8741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:25:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:55.861+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:25:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:55.862+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:25:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:55.862+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:25:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:55.862+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:25:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:25:55.862+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:56 smithi183 ceph-mon[37273]: pgmap v8742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15599.890272 secs 2024-07-26T15:25:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:56 smithi144 ceph-mon[32129]: pgmap v8742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15599.890272 secs 2024-07-26T15:25:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:56 smithi160 ceph-mon[36715]: pgmap v8742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:25:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15599.890272 secs 2024-07-26T15:25:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:25:58 smithi183 ceph-mon[37273]: pgmap v8743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:25:58 smithi144 ceph-mon[32129]: pgmap v8743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:25:58.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:25:58 smithi160 ceph-mon[36715]: pgmap v8743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:00 smithi183 ceph-mon[37273]: pgmap v8744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:00 smithi144 ceph-mon[32129]: pgmap v8744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:00.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:00 smithi160 ceph-mon[36715]: pgmap v8744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15604.890347 secs 2024-07-26T15:26:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15604.890347 secs 2024-07-26T15:26:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15604.890347 secs 2024-07-26T15:26:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:02 smithi183 ceph-mon[37273]: pgmap v8745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:02 smithi144 ceph-mon[32129]: pgmap v8745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:02.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:02 smithi160 ceph-mon[36715]: pgmap v8745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:04 smithi144 ceph-mon[32129]: pgmap v8746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:04 smithi183 ceph-mon[37273]: pgmap v8746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:04 smithi160 ceph-mon[36715]: pgmap v8746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:06 smithi144 ceph-mon[32129]: pgmap v8747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15609.890468 secs 2024-07-26T15:26:06.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:06 smithi160 ceph-mon[36715]: pgmap v8747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15609.890468 secs 2024-07-26T15:26:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:06 smithi183 ceph-mon[37273]: pgmap v8747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15609.890468 secs 2024-07-26T15:26:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:08 smithi144 ceph-mon[32129]: pgmap v8748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:08 smithi160 ceph-mon[36715]: pgmap v8748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:08 smithi183 ceph-mon[37273]: pgmap v8748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:10 smithi144 ceph-mon[32129]: pgmap v8749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:10.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:10 smithi160 ceph-mon[36715]: pgmap v8749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:10.878+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:26:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:10.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:26:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:10.879+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:26:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:10.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:26:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:10.879+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:10.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:10 smithi183 ceph-mon[37273]: pgmap v8749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15614.890588 secs 2024-07-26T15:26:11.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15614.890588 secs 2024-07-26T15:26:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15614.890588 secs 2024-07-26T15:26:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:12 smithi144 ceph-mon[32129]: pgmap v8750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:12.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:12 smithi160 ceph-mon[36715]: pgmap v8750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:12 smithi183 ceph-mon[37273]: pgmap v8750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:14 smithi144 ceph-mon[32129]: pgmap v8751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:14 smithi160 ceph-mon[36715]: pgmap v8751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:14 smithi183 ceph-mon[37273]: pgmap v8751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:16.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:16 smithi183 ceph-mon[37273]: pgmap v8752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:16.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15619.890744 secs 2024-07-26T15:26:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:16 smithi144 ceph-mon[32129]: pgmap v8752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15619.890744 secs 2024-07-26T15:26:16.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:16 smithi160 ceph-mon[36715]: pgmap v8752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15619.890744 secs 2024-07-26T15:26:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:18 smithi144 ceph-mon[32129]: pgmap v8753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:18.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:18 smithi160 ceph-mon[36715]: pgmap v8753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:18 smithi183 ceph-mon[37273]: pgmap v8753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:20 smithi144 ceph-mon[32129]: pgmap v8754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:20.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:20 smithi160 ceph-mon[36715]: pgmap v8754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:20 smithi183 ceph-mon[37273]: pgmap v8754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15624.890864 secs 2024-07-26T15:26:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15624.890864 secs 2024-07-26T15:26:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15624.890864 secs 2024-07-26T15:26:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:22 smithi144 ceph-mon[32129]: pgmap v8755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:22.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:22 smithi160 ceph-mon[36715]: pgmap v8755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:22 smithi183 ceph-mon[37273]: pgmap v8755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:24 smithi144 ceph-mon[32129]: pgmap v8756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:24.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:24 smithi160 ceph-mon[36715]: pgmap v8756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:24 smithi183 ceph-mon[37273]: pgmap v8756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:25.895+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:26:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:25.896+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:26:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:25.896+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:26:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:25.896+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:26:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:25.896+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:26 smithi144 ceph-mon[32129]: pgmap v8757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15629.890937 secs 2024-07-26T15:26:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:26 smithi160 ceph-mon[36715]: pgmap v8757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15629.890937 secs 2024-07-26T15:26:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:26 smithi183 ceph-mon[37273]: pgmap v8757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15629.890937 secs 2024-07-26T15:26:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:28 smithi144 ceph-mon[32129]: pgmap v8758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:28.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:28 smithi160 ceph-mon[36715]: pgmap v8758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:28 smithi183 ceph-mon[37273]: pgmap v8758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:30.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:30 smithi160 ceph-mon[36715]: pgmap v8759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:30 smithi183 ceph-mon[37273]: pgmap v8759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:30 smithi144 ceph-mon[32129]: pgmap v8759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15634.891079 secs 2024-07-26T15:26:31.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15634.891079 secs 2024-07-26T15:26:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15634.891079 secs 2024-07-26T15:26:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:32 smithi160 ceph-mon[36715]: pgmap v8760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:32 smithi183 ceph-mon[37273]: pgmap v8760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:32 smithi144 ceph-mon[32129]: pgmap v8760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:34 smithi160 ceph-mon[36715]: pgmap v8761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:34 smithi183 ceph-mon[37273]: pgmap v8761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:34 smithi144 ceph-mon[32129]: pgmap v8761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:36 smithi160 ceph-mon[36715]: pgmap v8762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15639.891154 secs 2024-07-26T15:26:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:36 smithi183 ceph-mon[37273]: pgmap v8762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15639.891154 secs 2024-07-26T15:26:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:36 smithi144 ceph-mon[32129]: pgmap v8762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15639.891154 secs 2024-07-26T15:26:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:38 smithi160 ceph-mon[36715]: pgmap v8763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:38 smithi183 ceph-mon[37273]: pgmap v8763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:38 smithi144 ceph-mon[32129]: pgmap v8763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:40 smithi160 ceph-mon[36715]: pgmap v8764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:40 smithi183 ceph-mon[37273]: pgmap v8764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:40.911+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:26:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:40.912+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:26:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:40.912+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:26:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:40.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:26:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:40.912+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:40 smithi144 ceph-mon[32129]: pgmap v8764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15644.891284 secs 2024-07-26T15:26:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:26:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15644.891284 secs 2024-07-26T15:26:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:26:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15644.891284 secs 2024-07-26T15:26:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:26:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:42 smithi160 ceph-mon[36715]: pgmap v8765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:42 smithi183 ceph-mon[37273]: pgmap v8765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:42 smithi144 ceph-mon[32129]: pgmap v8765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:43 smithi160 ceph-mon[36715]: pgmap v8766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:43 smithi160 ceph-mon[36715]: pgmap v8767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:43 smithi183 ceph-mon[37273]: pgmap v8766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:43 smithi183 ceph-mon[37273]: pgmap v8767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:43 smithi144 ceph-mon[32129]: pgmap v8766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:26:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:43 smithi144 ceph-mon[32129]: pgmap v8767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:26:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:46 smithi160 ceph-mon[36715]: pgmap v8768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:46.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15649.891401 secs 2024-07-26T15:26:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:46 smithi183 ceph-mon[37273]: pgmap v8768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15649.891401 secs 2024-07-26T15:26:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:45 smithi144 ceph-mon[32129]: pgmap v8768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:45 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15649.891401 secs 2024-07-26T15:26:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:48 smithi160 ceph-mon[36715]: pgmap v8769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:48 smithi183 ceph-mon[37273]: pgmap v8769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:48 smithi144 ceph-mon[32129]: pgmap v8769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:48.864+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:26:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:48.864+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T15:26:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:48.865+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478003690 con 0x7f74b0109990 2024-07-26T15:26:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:48.865+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x563c6c4182f0 2024-07-26T15:26:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:50 smithi160 ceph-mon[36715]: pgmap v8770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:26:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:50 smithi183 ceph-mon[37273]: pgmap v8770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:26:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:50 smithi144 ceph-mon[32129]: pgmap v8770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:26:51.294 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15654.891470 secs 2024-07-26T15:26:51.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15654.891470 secs 2024-07-26T15:26:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15654.891470 secs 2024-07-26T15:26:52.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:52 smithi160 ceph-mon[36715]: pgmap v8771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:52 smithi183 ceph-mon[37273]: pgmap v8771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:52 smithi144 ceph-mon[32129]: pgmap v8771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:26:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:54 smithi160 ceph-mon[36715]: pgmap v8772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:26:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:54 smithi183 ceph-mon[37273]: pgmap v8772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:26:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:54 smithi144 ceph-mon[32129]: pgmap v8772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:26:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:55.921+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:26:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:55.922+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:26:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:55.922+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:26:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:55.922+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:26:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:26:55.922+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:56.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:56 smithi160 ceph-mon[36715]: pgmap v8773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15659.891546 secs 2024-07-26T15:26:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:56 smithi183 ceph-mon[37273]: pgmap v8773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15659.891546 secs 2024-07-26T15:26:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:56 smithi144 ceph-mon[32129]: pgmap v8773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:26:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15659.891546 secs 2024-07-26T15:26:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:26:58 smithi160 ceph-mon[36715]: pgmap v8774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:26:58 smithi183 ceph-mon[37273]: pgmap v8774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:26:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:26:58 smithi144 ceph-mon[32129]: pgmap v8774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:00.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:00 smithi160 ceph-mon[36715]: pgmap v8775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:00 smithi183 ceph-mon[37273]: pgmap v8775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:00 smithi144 ceph-mon[32129]: pgmap v8775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:01.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15664.891674 secs 2024-07-26T15:27:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15664.891674 secs 2024-07-26T15:27:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15664.891674 secs 2024-07-26T15:27:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:02 smithi183 ceph-mon[37273]: pgmap v8776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:02 smithi144 ceph-mon[32129]: pgmap v8776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:02.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:02 smithi160 ceph-mon[36715]: pgmap v8776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:04 smithi183 ceph-mon[37273]: pgmap v8777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:04 smithi144 ceph-mon[32129]: pgmap v8777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:04.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:04 smithi160 ceph-mon[36715]: pgmap v8777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:06 smithi183 ceph-mon[37273]: pgmap v8778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15669.891794 secs 2024-07-26T15:27:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:06 smithi144 ceph-mon[32129]: pgmap v8778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15669.891794 secs 2024-07-26T15:27:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:06 smithi160 ceph-mon[36715]: pgmap v8778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15669.891794 secs 2024-07-26T15:27:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:08 smithi183 ceph-mon[37273]: pgmap v8779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:08 smithi144 ceph-mon[32129]: pgmap v8779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:08.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:08 smithi160 ceph-mon[36715]: pgmap v8779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:10 smithi183 ceph-mon[37273]: pgmap v8780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:10 smithi144 ceph-mon[32129]: pgmap v8780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:10.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:10 smithi160 ceph-mon[36715]: pgmap v8780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:10.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:10.938+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:27:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:10.939+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:27:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:10.939+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:27:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:10.939+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:27:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:10.939+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15674.891920 secs 2024-07-26T15:27:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15674.891920 secs 2024-07-26T15:27:11.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15674.891920 secs 2024-07-26T15:27:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:12 smithi183 ceph-mon[37273]: pgmap v8781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:12 smithi144 ceph-mon[32129]: pgmap v8781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:12.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:12 smithi160 ceph-mon[36715]: pgmap v8781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:14 smithi183 ceph-mon[37273]: pgmap v8782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:14 smithi144 ceph-mon[32129]: pgmap v8782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:14 smithi160 ceph-mon[36715]: pgmap v8782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:16 smithi183 ceph-mon[37273]: pgmap v8783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15679.891991 secs 2024-07-26T15:27:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:16 smithi144 ceph-mon[32129]: pgmap v8783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15679.891991 secs 2024-07-26T15:27:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:16 smithi160 ceph-mon[36715]: pgmap v8783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15679.891991 secs 2024-07-26T15:27:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:18 smithi183 ceph-mon[37273]: pgmap v8784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:18 smithi144 ceph-mon[32129]: pgmap v8784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:18 smithi160 ceph-mon[36715]: pgmap v8784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:20 smithi183 ceph-mon[37273]: pgmap v8785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:20 smithi144 ceph-mon[32129]: pgmap v8785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:20.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:20 smithi160 ceph-mon[36715]: pgmap v8785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15684.892128 secs 2024-07-26T15:27:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15684.892128 secs 2024-07-26T15:27:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15684.892128 secs 2024-07-26T15:27:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:22 smithi144 ceph-mon[32129]: pgmap v8786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:22.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:22 smithi160 ceph-mon[36715]: pgmap v8786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:22 smithi183 ceph-mon[37273]: pgmap v8786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:24 smithi144 ceph-mon[32129]: pgmap v8787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:24 smithi160 ceph-mon[36715]: pgmap v8787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:24 smithi183 ceph-mon[37273]: pgmap v8787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:25.949+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:27:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:25.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:27:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:25.950+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:27:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:25.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:27:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:25.950+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:26 smithi144 ceph-mon[32129]: pgmap v8788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15689.892248 secs 2024-07-26T15:27:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:26 smithi160 ceph-mon[36715]: pgmap v8788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15689.892248 secs 2024-07-26T15:27:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:26 smithi183 ceph-mon[37273]: pgmap v8788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15689.892248 secs 2024-07-26T15:27:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:28 smithi144 ceph-mon[32129]: pgmap v8789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:28.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:28 smithi160 ceph-mon[36715]: pgmap v8789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:28 smithi183 ceph-mon[37273]: pgmap v8789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:30 smithi144 ceph-mon[32129]: pgmap v8790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:30.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:30 smithi160 ceph-mon[36715]: pgmap v8790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:30 smithi183 ceph-mon[37273]: pgmap v8790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15694.892370 secs 2024-07-26T15:27:31.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15694.892370 secs 2024-07-26T15:27:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15694.892370 secs 2024-07-26T15:27:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:32 smithi144 ceph-mon[32129]: pgmap v8791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:32.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:32 smithi160 ceph-mon[36715]: pgmap v8791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:32 smithi183 ceph-mon[37273]: pgmap v8791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:34 smithi144 ceph-mon[32129]: pgmap v8792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:34 smithi160 ceph-mon[36715]: pgmap v8792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:34 smithi183 ceph-mon[37273]: pgmap v8792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:36 smithi144 ceph-mon[32129]: pgmap v8793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15699.892449 secs 2024-07-26T15:27:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:36 smithi160 ceph-mon[36715]: pgmap v8793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15699.892449 secs 2024-07-26T15:27:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:36 smithi183 ceph-mon[37273]: pgmap v8793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15699.892449 secs 2024-07-26T15:27:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:38 smithi144 ceph-mon[32129]: pgmap v8794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:38.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:38 smithi160 ceph-mon[36715]: pgmap v8794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:38 smithi183 ceph-mon[37273]: pgmap v8794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:40 smithi144 ceph-mon[32129]: pgmap v8795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:40 smithi160 ceph-mon[36715]: pgmap v8795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:40 smithi183 ceph-mon[37273]: pgmap v8795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:40.965+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:27:40.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:40.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:27:40.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:40.966+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:27:40.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:40.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:27:40.968 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:40.966+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:41.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15704.892532 secs 2024-07-26T15:27:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15704.892532 secs 2024-07-26T15:27:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15704.892532 secs 2024-07-26T15:27:42.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:42 smithi160 ceph-mon[36715]: pgmap v8796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:42 smithi183 ceph-mon[37273]: pgmap v8796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:42 smithi144 ceph-mon[32129]: pgmap v8796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:43 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:43 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:43 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:44 smithi160 ceph-mon[36715]: pgmap v8797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:44 smithi183 ceph-mon[37273]: pgmap v8797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:44 smithi144 ceph-mon[32129]: pgmap v8797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:27:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: pgmap v8798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15709.892654 secs 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: pgmap v8798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15709.892654 secs 2024-07-26T15:27:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: pgmap v8798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15709.892654 secs 2024-07-26T15:27:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:48 smithi160 ceph-mon[36715]: pgmap v8799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:48 smithi183 ceph-mon[37273]: pgmap v8799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:48 smithi144 ceph-mon[32129]: pgmap v8799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:27:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:27:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:49 smithi160 ceph-mon[36715]: pgmap v8800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:27:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:49 smithi160 ceph-mon[36715]: pgmap v8801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:49 smithi183 ceph-mon[37273]: pgmap v8800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:27:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:49 smithi183 ceph-mon[37273]: pgmap v8801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:49 smithi144 ceph-mon[32129]: pgmap v8800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:27:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:49 smithi144 ceph-mon[32129]: pgmap v8801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:51.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:51 smithi160 ceph-mon[36715]: pgmap v8802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15714.892719 secs 2024-07-26T15:27:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:51 smithi183 ceph-mon[37273]: pgmap v8802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15714.892719 secs 2024-07-26T15:27:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:51 smithi144 ceph-mon[32129]: pgmap v8802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15714.892719 secs 2024-07-26T15:27:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:52 smithi160 ceph-mon[36715]: pgmap v8803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:52 smithi183 ceph-mon[37273]: pgmap v8803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:52 smithi144 ceph-mon[32129]: pgmap v8803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:54 smithi183 ceph-mon[37273]: pgmap v8804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:54 smithi144 ceph-mon[32129]: pgmap v8804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:54 smithi160 ceph-mon[36715]: pgmap v8804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:55.981+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:27:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:55.982+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:27:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:55.982+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:27:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:55.982+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:27:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:27:55.982+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15719.892839 secs 2024-07-26T15:27:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15719.892839 secs 2024-07-26T15:27:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15719.892839 secs 2024-07-26T15:27:57.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:57 smithi160 ceph-mon[36715]: pgmap v8805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:57 smithi183 ceph-mon[37273]: pgmap v8805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:57 smithi144 ceph-mon[32129]: pgmap v8805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:27:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:27:58 smithi183 ceph-mon[37273]: pgmap v8806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:27:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:27:58 smithi144 ceph-mon[32129]: pgmap v8806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:27:59.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:27:58 smithi160 ceph-mon[36715]: pgmap v8806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:00 smithi183 ceph-mon[37273]: pgmap v8807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:00 smithi144 ceph-mon[32129]: pgmap v8807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:00 smithi160 ceph-mon[36715]: pgmap v8807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:01.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15724.892963 secs 2024-07-26T15:28:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15724.892963 secs 2024-07-26T15:28:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15724.892963 secs 2024-07-26T15:28:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:02 smithi183 ceph-mon[37273]: pgmap v8808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:02 smithi144 ceph-mon[32129]: pgmap v8808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:03.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:02 smithi160 ceph-mon[36715]: pgmap v8808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:04 smithi183 ceph-mon[37273]: pgmap v8809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:04 smithi144 ceph-mon[32129]: pgmap v8809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:04 smithi160 ceph-mon[36715]: pgmap v8809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15729.893079 secs 2024-07-26T15:28:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15729.893079 secs 2024-07-26T15:28:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15729.893079 secs 2024-07-26T15:28:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:07 smithi160 ceph-mon[36715]: pgmap v8810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:07 smithi183 ceph-mon[37273]: pgmap v8810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:07 smithi144 ceph-mon[32129]: pgmap v8810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:08 smithi183 ceph-mon[37273]: pgmap v8811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:08 smithi144 ceph-mon[32129]: pgmap v8811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:08 smithi160 ceph-mon[36715]: pgmap v8811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:10 smithi183 ceph-mon[37273]: pgmap v8812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:10 smithi144 ceph-mon[32129]: pgmap v8812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:10.998+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:28:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:10.999+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:28:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:10.999+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:28:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:10.999+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:28:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:10.999+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:10 smithi160 ceph-mon[36715]: pgmap v8812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15734.893196 secs 2024-07-26T15:28:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15734.893196 secs 2024-07-26T15:28:12.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15734.893196 secs 2024-07-26T15:28:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:12 smithi183 ceph-mon[37273]: pgmap v8813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:12 smithi144 ceph-mon[32129]: pgmap v8813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:12 smithi160 ceph-mon[36715]: pgmap v8813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:14 smithi183 ceph-mon[37273]: pgmap v8814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:14 smithi144 ceph-mon[32129]: pgmap v8814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:14 smithi160 ceph-mon[36715]: pgmap v8814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15739.893330 secs 2024-07-26T15:28:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15739.893330 secs 2024-07-26T15:28:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15739.893330 secs 2024-07-26T15:28:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:17 smithi183 ceph-mon[37273]: pgmap v8815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:17 smithi144 ceph-mon[32129]: pgmap v8815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:17.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:17 smithi160 ceph-mon[36715]: pgmap v8815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:18 smithi183 ceph-mon[37273]: pgmap v8816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:18 smithi144 ceph-mon[32129]: pgmap v8816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:19.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:18 smithi160 ceph-mon[36715]: pgmap v8816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:20 smithi183 ceph-mon[37273]: pgmap v8817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:20 smithi144 ceph-mon[32129]: pgmap v8817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:20 smithi160 ceph-mon[36715]: pgmap v8817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15744.893428 secs 2024-07-26T15:28:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15744.893428 secs 2024-07-26T15:28:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15744.893428 secs 2024-07-26T15:28:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:22 smithi183 ceph-mon[37273]: pgmap v8818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:22 smithi144 ceph-mon[32129]: pgmap v8818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:22 smithi160 ceph-mon[36715]: pgmap v8818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:24.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:24 smithi160 ceph-mon[36715]: pgmap v8819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:24 smithi183 ceph-mon[37273]: pgmap v8819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:24 smithi144 ceph-mon[32129]: pgmap v8819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:26.005+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:28:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:26.006+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:28:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:26.006+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:28:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:26.006+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:28:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:26.007+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15749.893494 secs 2024-07-26T15:28:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15749.893494 secs 2024-07-26T15:28:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15749.893494 secs 2024-07-26T15:28:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:27 smithi183 ceph-mon[37273]: pgmap v8820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:27 smithi144 ceph-mon[32129]: pgmap v8820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:27 smithi160 ceph-mon[36715]: pgmap v8820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:28 smithi183 ceph-mon[37273]: pgmap v8821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:28 smithi144 ceph-mon[32129]: pgmap v8821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:29.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:28 smithi160 ceph-mon[36715]: pgmap v8821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:30 smithi183 ceph-mon[37273]: pgmap v8822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:30 smithi144 ceph-mon[32129]: pgmap v8822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:30 smithi160 ceph-mon[36715]: pgmap v8822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:31.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15754.893604 secs 2024-07-26T15:28:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15754.893604 secs 2024-07-26T15:28:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15754.893604 secs 2024-07-26T15:28:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:32 smithi183 ceph-mon[37273]: pgmap v8823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:32 smithi144 ceph-mon[32129]: pgmap v8823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:32 smithi160 ceph-mon[36715]: pgmap v8823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:34 smithi183 ceph-mon[37273]: pgmap v8824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:28:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:34 smithi144 ceph-mon[32129]: pgmap v8824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:28:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:34 smithi160 ceph-mon[36715]: pgmap v8824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:28:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15759.893743 secs 2024-07-26T15:28:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15759.893743 secs 2024-07-26T15:28:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15759.893743 secs 2024-07-26T15:28:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:37 smithi183 ceph-mon[37273]: pgmap v8825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:37 smithi144 ceph-mon[32129]: pgmap v8825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:37 smithi160 ceph-mon[36715]: pgmap v8825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:38 smithi183 ceph-mon[37273]: pgmap v8826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:38 smithi144 ceph-mon[32129]: pgmap v8826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:39.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:38 smithi160 ceph-mon[36715]: pgmap v8826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:28:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:40 smithi183 ceph-mon[37273]: pgmap v8827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:40 smithi144 ceph-mon[32129]: pgmap v8827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:41.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:41.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:28:41.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:41.020+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:28:41.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:41.020+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:28:41.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:41.020+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:28:41.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:41.020+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:40 smithi160 ceph-mon[36715]: pgmap v8827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15764.893862 secs 2024-07-26T15:28:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15764.893862 secs 2024-07-26T15:28:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15764.893862 secs 2024-07-26T15:28:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:42 smithi183 ceph-mon[37273]: pgmap v8828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:42 smithi144 ceph-mon[32129]: pgmap v8828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:43.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:42 smithi160 ceph-mon[36715]: pgmap v8828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:28:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:44 smithi183 ceph-mon[37273]: pgmap v8829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:44 smithi144 ceph-mon[32129]: pgmap v8829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:44 smithi160 ceph-mon[36715]: pgmap v8829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:46.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15769.893983 secs 2024-07-26T15:28:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15769.893983 secs 2024-07-26T15:28:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15769.893983 secs 2024-07-26T15:28:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:47 smithi183 ceph-mon[37273]: pgmap v8830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:47 smithi144 ceph-mon[32129]: pgmap v8830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:47 smithi160 ceph-mon[36715]: pgmap v8830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:28:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:48 smithi183 ceph-mon[37273]: pgmap v8831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:48 smithi144 ceph-mon[32129]: pgmap v8831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:48 smithi160 ceph-mon[36715]: pgmap v8831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:28:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:50.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:50 smithi183 ceph-mon[37273]: pgmap v8832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:28:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:50 smithi183 ceph-mon[37273]: pgmap v8833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:28:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:50 smithi144 ceph-mon[32129]: pgmap v8832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:28:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:50 smithi144 ceph-mon[32129]: pgmap v8833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:50.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:28:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:50 smithi160 ceph-mon[36715]: pgmap v8832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:28:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:50 smithi160 ceph-mon[36715]: pgmap v8833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:28:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15774.894139 secs 2024-07-26T15:28:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15774.894139 secs 2024-07-26T15:28:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15774.894139 secs 2024-07-26T15:28:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:52 smithi183 ceph-mon[37273]: pgmap v8834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:28:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:52 smithi144 ceph-mon[32129]: pgmap v8834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:28:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:52 smithi160 ceph-mon[36715]: pgmap v8834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:28:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:54 smithi160 ceph-mon[36715]: pgmap v8835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:28:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:54 smithi183 ceph-mon[37273]: pgmap v8835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:28:55.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:54 smithi144 ceph-mon[32129]: pgmap v8835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:28:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:56.029+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:28:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:56.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:28:56.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:56.030+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:28:56.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:56.030+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:28:56.032 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:28:56.031+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:56 smithi160 ceph-mon[36715]: pgmap v8836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:28:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15779.894217 secs 2024-07-26T15:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:56 smithi183 ceph-mon[37273]: pgmap v8836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:28:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15779.894217 secs 2024-07-26T15:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:56 smithi144 ceph-mon[32129]: pgmap v8836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:28:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15779.894217 secs 2024-07-26T15:28:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:28:58 smithi160 ceph-mon[36715]: pgmap v8837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:28:58 smithi183 ceph-mon[37273]: pgmap v8837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:28:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:28:58 smithi144 ceph-mon[32129]: pgmap v8837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:00 smithi183 ceph-mon[37273]: pgmap v8838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:29:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:00 smithi144 ceph-mon[32129]: pgmap v8838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:29:01.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:00 smithi160 ceph-mon[36715]: pgmap v8838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:29:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15784.894283 secs 2024-07-26T15:29:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15784.894283 secs 2024-07-26T15:29:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15784.894283 secs 2024-07-26T15:29:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:02 smithi183 ceph-mon[37273]: pgmap v8839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:02 smithi144 ceph-mon[32129]: pgmap v8839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:03.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:02 smithi160 ceph-mon[36715]: pgmap v8839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:04 smithi183 ceph-mon[37273]: pgmap v8840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:04 smithi144 ceph-mon[32129]: pgmap v8840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:04 smithi160 ceph-mon[36715]: pgmap v8840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:06 smithi183 ceph-mon[37273]: pgmap v8841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15789.894428 secs 2024-07-26T15:29:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:06 smithi144 ceph-mon[32129]: pgmap v8841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15789.894428 secs 2024-07-26T15:29:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:06 smithi160 ceph-mon[36715]: pgmap v8841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15789.894428 secs 2024-07-26T15:29:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:08 smithi183 ceph-mon[37273]: pgmap v8842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:08 smithi144 ceph-mon[32129]: pgmap v8842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:09.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:08 smithi160 ceph-mon[36715]: pgmap v8842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:11.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:29:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:11.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:29:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:11.047+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:29:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:11.047+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:29:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:11.048+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:10 smithi183 ceph-mon[37273]: pgmap v8843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:10 smithi144 ceph-mon[32129]: pgmap v8843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:11.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:10 smithi160 ceph-mon[36715]: pgmap v8843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15794.894528 secs 2024-07-26T15:29:12.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15794.894528 secs 2024-07-26T15:29:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15794.894528 secs 2024-07-26T15:29:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:12 smithi183 ceph-mon[37273]: pgmap v8844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:12 smithi144 ceph-mon[32129]: pgmap v8844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:12 smithi160 ceph-mon[36715]: pgmap v8844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:14 smithi183 ceph-mon[37273]: pgmap v8845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:14 smithi144 ceph-mon[32129]: pgmap v8845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:15.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:14 smithi160 ceph-mon[36715]: pgmap v8845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:16 smithi183 ceph-mon[37273]: pgmap v8846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15799.894653 secs 2024-07-26T15:29:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:16 smithi144 ceph-mon[32129]: pgmap v8846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15799.894653 secs 2024-07-26T15:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:16 smithi160 ceph-mon[36715]: pgmap v8846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15799.894653 secs 2024-07-26T15:29:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:18 smithi183 ceph-mon[37273]: pgmap v8847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:18 smithi144 ceph-mon[32129]: pgmap v8847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:19.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:18 smithi160 ceph-mon[36715]: pgmap v8847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:20 smithi144 ceph-mon[32129]: pgmap v8848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:20 smithi160 ceph-mon[36715]: pgmap v8848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:20 smithi183 ceph-mon[37273]: pgmap v8848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15804.894782 secs 2024-07-26T15:29:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15804.894782 secs 2024-07-26T15:29:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15804.894782 secs 2024-07-26T15:29:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:22 smithi144 ceph-mon[32129]: pgmap v8849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:23.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:22 smithi160 ceph-mon[36715]: pgmap v8849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:22 smithi183 ceph-mon[37273]: pgmap v8849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:24 smithi144 ceph-mon[32129]: pgmap v8850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:24 smithi160 ceph-mon[36715]: pgmap v8850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:25.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:24 smithi183 ceph-mon[37273]: pgmap v8850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:26.063+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:29:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:26.064+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:26.064+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:26.064+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:26.064+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:26 smithi144 ceph-mon[32129]: pgmap v8851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15809.894923 secs 2024-07-26T15:29:27.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:26 smithi160 ceph-mon[36715]: pgmap v8851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15809.894923 secs 2024-07-26T15:29:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:26 smithi183 ceph-mon[37273]: pgmap v8851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15809.894923 secs 2024-07-26T15:29:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:28 smithi144 ceph-mon[32129]: pgmap v8852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:28 smithi160 ceph-mon[36715]: pgmap v8852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:28 smithi183 ceph-mon[37273]: pgmap v8852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:30 smithi144 ceph-mon[32129]: pgmap v8853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:31.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:30 smithi160 ceph-mon[36715]: pgmap v8853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:30 smithi183 ceph-mon[37273]: pgmap v8853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15814.895049 secs 2024-07-26T15:29:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15814.895049 secs 2024-07-26T15:29:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15814.895049 secs 2024-07-26T15:29:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:32 smithi144 ceph-mon[32129]: pgmap v8854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:33.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:32 smithi160 ceph-mon[36715]: pgmap v8854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:32 smithi183 ceph-mon[37273]: pgmap v8854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:34 smithi144 ceph-mon[32129]: pgmap v8855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:35.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:34 smithi160 ceph-mon[36715]: pgmap v8855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:35.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:34 smithi183 ceph-mon[37273]: pgmap v8855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:36 smithi144 ceph-mon[32129]: pgmap v8856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15819.895111 secs 2024-07-26T15:29:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:36 smithi160 ceph-mon[36715]: pgmap v8856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15819.895111 secs 2024-07-26T15:29:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:36 smithi183 ceph-mon[37273]: pgmap v8856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15819.895111 secs 2024-07-26T15:29:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:38 smithi144 ceph-mon[32129]: pgmap v8857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:38 smithi160 ceph-mon[36715]: pgmap v8857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:38 smithi183 ceph-mon[37273]: pgmap v8857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:41.065+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:29:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:41.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:29:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:41.066+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:29:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:41.066+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:29:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:41.066+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:40 smithi144 ceph-mon[32129]: pgmap v8858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:41.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:40 smithi160 ceph-mon[36715]: pgmap v8858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:40 smithi183 ceph-mon[37273]: pgmap v8858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:42.302 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15824.895236 secs 2024-07-26T15:29:42.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15824.895236 secs 2024-07-26T15:29:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15824.895236 secs 2024-07-26T15:29:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:42 smithi160 ceph-mon[36715]: pgmap v8859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:42 smithi183 ceph-mon[37273]: pgmap v8859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:42 smithi144 ceph-mon[32129]: pgmap v8859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:29:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:44 smithi160 ceph-mon[36715]: pgmap v8860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:44 smithi183 ceph-mon[37273]: pgmap v8860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:44 smithi144 ceph-mon[32129]: pgmap v8860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:46 smithi160 ceph-mon[36715]: pgmap v8861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15829.895374 secs 2024-07-26T15:29:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:46 smithi183 ceph-mon[37273]: pgmap v8861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15829.895374 secs 2024-07-26T15:29:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:46 smithi144 ceph-mon[32129]: pgmap v8861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15829.895374 secs 2024-07-26T15:29:48.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.894+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478003690 con 0x563c6c4182f0 2024-07-26T15:29:48.895 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.894+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:29:48.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.895+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T15:29:48.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.895+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T15:29:48.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.895+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:29:48.896 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.895+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:29:48.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.896+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:29:48.897 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:48.896+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:29:49.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:48 smithi160 ceph-mon[36715]: pgmap v8862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:48 smithi183 ceph-mon[37273]: pgmap v8862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:48 smithi144 ceph-mon[32129]: pgmap v8862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:29:50.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:51 smithi160 ceph-mon[36715]: pgmap v8863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:29:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:51 smithi183 ceph-mon[37273]: pgmap v8863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:29:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:51 smithi144 ceph-mon[32129]: pgmap v8863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:29:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:29:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15834.895494 secs 2024-07-26T15:29:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:52 smithi160 ceph-mon[36715]: pgmap v8864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:52 smithi160 ceph-mon[36715]: pgmap v8865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15834.895494 secs 2024-07-26T15:29:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:52 smithi183 ceph-mon[37273]: pgmap v8864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:52 smithi183 ceph-mon[37273]: pgmap v8865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15834.895494 secs 2024-07-26T15:29:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:52 smithi144 ceph-mon[32129]: pgmap v8864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:52 smithi144 ceph-mon[32129]: pgmap v8865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:54.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:54 smithi160 ceph-mon[36715]: pgmap v8866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:54.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:54 smithi183 ceph-mon[37273]: pgmap v8866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:54 smithi144 ceph-mon[32129]: pgmap v8866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:56.079+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:29:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:56.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:29:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:56.080+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:29:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:56.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:29:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:29:56.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:56 smithi160 ceph-mon[36715]: pgmap v8867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15839.895626 secs 2024-07-26T15:29:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:56 smithi183 ceph-mon[37273]: pgmap v8867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:56.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15839.895626 secs 2024-07-26T15:29:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:56 smithi144 ceph-mon[32129]: pgmap v8867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:29:56.606 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15839.895626 secs 2024-07-26T15:29:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:29:58 smithi160 ceph-mon[36715]: pgmap v8868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:29:58 smithi183 ceph-mon[37273]: pgmap v8868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:29:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:29:58 smithi144 ceph-mon[32129]: pgmap v8868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:00 smithi160 ceph-mon[36715]: pgmap v8869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:30:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:30:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:00 smithi183 ceph-mon[37273]: pgmap v8869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:30:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:30:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:00 smithi144 ceph-mon[32129]: pgmap v8869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:30:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:30:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15844.895752 secs 2024-07-26T15:30:01.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15844.895752 secs 2024-07-26T15:30:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15844.895752 secs 2024-07-26T15:30:02.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:02 smithi160 ceph-mon[36715]: pgmap v8870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:30:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:02 smithi183 ceph-mon[37273]: pgmap v8870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:30:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:02 smithi144 ceph-mon[32129]: pgmap v8870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:30:04.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:04 smithi160 ceph-mon[36715]: pgmap v8871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:04 smithi183 ceph-mon[37273]: pgmap v8871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:04 smithi144 ceph-mon[32129]: pgmap v8871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:06 smithi160 ceph-mon[36715]: pgmap v8872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15849.895884 secs 2024-07-26T15:30:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:06 smithi183 ceph-mon[37273]: pgmap v8872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15849.895884 secs 2024-07-26T15:30:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:06 smithi144 ceph-mon[32129]: pgmap v8872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15849.895884 secs 2024-07-26T15:30:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:08 smithi183 ceph-mon[37273]: pgmap v8873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:08 smithi144 ceph-mon[32129]: pgmap v8873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:08 smithi160 ceph-mon[36715]: pgmap v8873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:10 smithi183 ceph-mon[37273]: pgmap v8874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:10 smithi144 ceph-mon[32129]: pgmap v8874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:10 smithi160 ceph-mon[36715]: pgmap v8874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:11.096+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:30:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:11.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:30:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:11.097+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:30:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:11.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:30:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:11.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15854.896015 secs 2024-07-26T15:30:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15854.896015 secs 2024-07-26T15:30:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15854.896015 secs 2024-07-26T15:30:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:12 smithi183 ceph-mon[37273]: pgmap v8875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:12 smithi144 ceph-mon[32129]: pgmap v8875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:12 smithi160 ceph-mon[36715]: pgmap v8875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:14 smithi183 ceph-mon[37273]: pgmap v8876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:14 smithi144 ceph-mon[32129]: pgmap v8876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:14.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:14 smithi160 ceph-mon[36715]: pgmap v8876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:16 smithi183 ceph-mon[37273]: pgmap v8877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15859.896081 secs 2024-07-26T15:30:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:16 smithi144 ceph-mon[32129]: pgmap v8877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15859.896081 secs 2024-07-26T15:30:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:16 smithi160 ceph-mon[36715]: pgmap v8877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15859.896081 secs 2024-07-26T15:30:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:18 smithi183 ceph-mon[37273]: pgmap v8878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:18 smithi144 ceph-mon[32129]: pgmap v8878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:18.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:18 smithi160 ceph-mon[36715]: pgmap v8878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:20 smithi144 ceph-mon[32129]: pgmap v8879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:20 smithi183 ceph-mon[37273]: pgmap v8879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:20 smithi160 ceph-mon[36715]: pgmap v8879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15864.896214 secs 2024-07-26T15:30:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15864.896214 secs 2024-07-26T15:30:21.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15864.896214 secs 2024-07-26T15:30:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:22 smithi183 ceph-mon[37273]: pgmap v8880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:22 smithi144 ceph-mon[32129]: pgmap v8880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:22 smithi160 ceph-mon[36715]: pgmap v8880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:23 smithi160 ceph-mon[36715]: pgmap v8881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:23 smithi183 ceph-mon[37273]: pgmap v8881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:23 smithi144 ceph-mon[32129]: pgmap v8881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:26.101+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:30:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:26.102+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:30:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:26.102+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:30:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:26.102+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:30:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:26.102+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:26.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:26 smithi160 ceph-mon[36715]: pgmap v8882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15869.896343 secs 2024-07-26T15:30:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:26 smithi183 ceph-mon[37273]: pgmap v8882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15869.896343 secs 2024-07-26T15:30:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:26 smithi144 ceph-mon[32129]: pgmap v8882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15869.896343 secs 2024-07-26T15:30:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:28 smithi160 ceph-mon[36715]: pgmap v8883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:28 smithi183 ceph-mon[37273]: pgmap v8883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:28 smithi144 ceph-mon[32129]: pgmap v8883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:30 smithi144 ceph-mon[32129]: pgmap v8884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:30 smithi160 ceph-mon[36715]: pgmap v8884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:30 smithi183 ceph-mon[37273]: pgmap v8884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15874.896466 secs 2024-07-26T15:30:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15874.896466 secs 2024-07-26T15:30:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15874.896466 secs 2024-07-26T15:30:32.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:32 smithi160 ceph-mon[36715]: pgmap v8885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:32 smithi183 ceph-mon[37273]: pgmap v8885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:32 smithi144 ceph-mon[32129]: pgmap v8885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:34 smithi183 ceph-mon[37273]: pgmap v8886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:34 smithi144 ceph-mon[32129]: pgmap v8886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:34.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:34 smithi160 ceph-mon[36715]: pgmap v8886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:36 smithi183 ceph-mon[37273]: pgmap v8887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15879.896588 secs 2024-07-26T15:30:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:36 smithi144 ceph-mon[32129]: pgmap v8887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15879.896588 secs 2024-07-26T15:30:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:36 smithi160 ceph-mon[36715]: pgmap v8887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15879.896588 secs 2024-07-26T15:30:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:38 smithi160 ceph-mon[36715]: pgmap v8888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:38 smithi183 ceph-mon[37273]: pgmap v8888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:38 smithi144 ceph-mon[32129]: pgmap v8888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:40 smithi183 ceph-mon[37273]: pgmap v8889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:40 smithi144 ceph-mon[32129]: pgmap v8889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:40 smithi160 ceph-mon[36715]: pgmap v8889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:41.118+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:30:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:41.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:30:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:41.118+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:30:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:41.118+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:30:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:41.119+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15884.896674 secs 2024-07-26T15:30:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15884.896674 secs 2024-07-26T15:30:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15884.896674 secs 2024-07-26T15:30:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:42 smithi183 ceph-mon[37273]: pgmap v8890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:42 smithi144 ceph-mon[32129]: pgmap v8890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:42 smithi160 ceph-mon[36715]: pgmap v8890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:44.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:44 smithi183 ceph-mon[37273]: pgmap v8891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:44 smithi144 ceph-mon[32129]: pgmap v8891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:44.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:44 smithi160 ceph-mon[36715]: pgmap v8891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:30:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:46 smithi183 ceph-mon[37273]: pgmap v8892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15889.896748 secs 2024-07-26T15:30:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:46 smithi144 ceph-mon[32129]: pgmap v8892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15889.896748 secs 2024-07-26T15:30:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:46 smithi160 ceph-mon[36715]: pgmap v8892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15889.896748 secs 2024-07-26T15:30:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:48 smithi183 ceph-mon[37273]: pgmap v8893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:48 smithi144 ceph-mon[32129]: pgmap v8893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:48.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:48 smithi160 ceph-mon[36715]: pgmap v8893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:30:50.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:50 smithi183 ceph-mon[37273]: pgmap v8894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:50 smithi144 ceph-mon[32129]: pgmap v8894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:50.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:50 smithi160 ceph-mon[36715]: pgmap v8894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15894.896883 secs 2024-07-26T15:30:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15894.896883 secs 2024-07-26T15:30:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15894.896883 secs 2024-07-26T15:30:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:52 smithi144 ceph-mon[32129]: pgmap v8895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:52 smithi160 ceph-mon[36715]: pgmap v8895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:52 smithi183 ceph-mon[37273]: pgmap v8895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:30:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:53 smithi144 ceph-mon[32129]: pgmap v8896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:30:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:53 smithi144 ceph-mon[32129]: pgmap v8897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:30:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:53 smithi160 ceph-mon[36715]: pgmap v8896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:30:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:53 smithi160 ceph-mon[36715]: pgmap v8897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:30:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:53 smithi183 ceph-mon[37273]: pgmap v8896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:30:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:53 smithi183 ceph-mon[37273]: pgmap v8897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:30:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:55 smithi144 ceph-mon[32129]: pgmap v8898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:55.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:55 smithi160 ceph-mon[36715]: pgmap v8898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:55 smithi183 ceph-mon[37273]: pgmap v8898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:56.125+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:30:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:56.126+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:30:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:56.126+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:30:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:56.126+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:30:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:30:56.126+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15899.896949 secs 2024-07-26T15:30:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15899.896949 secs 2024-07-26T15:30:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15899.896949 secs 2024-07-26T15:30:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:57 smithi160 ceph-mon[36715]: pgmap v8899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:57 smithi183 ceph-mon[37273]: pgmap v8899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:57 smithi144 ceph-mon[32129]: pgmap v8899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:30:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:30:59 smithi160 ceph-mon[36715]: pgmap v8900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:30:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:30:59 smithi183 ceph-mon[37273]: pgmap v8900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:30:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:30:59 smithi144 ceph-mon[32129]: pgmap v8900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:31:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:01 smithi144 ceph-mon[32129]: pgmap v8901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:31:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15904.897082 secs 2024-07-26T15:31:01.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:01 smithi160 ceph-mon[36715]: pgmap v8901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:31:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15904.897082 secs 2024-07-26T15:31:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:01 smithi183 ceph-mon[37273]: pgmap v8901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:31:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15904.897082 secs 2024-07-26T15:31:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:03 smithi160 ceph-mon[36715]: pgmap v8902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:31:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:03 smithi183 ceph-mon[37273]: pgmap v8902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:31:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:03 smithi144 ceph-mon[32129]: pgmap v8902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:31:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:05 smithi160 ceph-mon[36715]: pgmap v8903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:05 smithi183 ceph-mon[37273]: pgmap v8903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:05 smithi144 ceph-mon[32129]: pgmap v8903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:06.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15909.897232 secs 2024-07-26T15:31:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15909.897232 secs 2024-07-26T15:31:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15909.897232 secs 2024-07-26T15:31:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:07 smithi160 ceph-mon[36715]: pgmap v8904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:07 smithi183 ceph-mon[37273]: pgmap v8904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:07 smithi144 ceph-mon[32129]: pgmap v8904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:09 smithi160 ceph-mon[36715]: pgmap v8905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:09 smithi183 ceph-mon[37273]: pgmap v8905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:09 smithi144 ceph-mon[32129]: pgmap v8905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:11.140+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:31:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:11.141+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:31:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:11.141+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:31:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:11.141+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:31:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:11.141+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:11 smithi160 ceph-mon[36715]: pgmap v8906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15914.897338 secs 2024-07-26T15:31:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:11 smithi183 ceph-mon[37273]: pgmap v8906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15914.897338 secs 2024-07-26T15:31:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:11 smithi144 ceph-mon[32129]: pgmap v8906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15914.897338 secs 2024-07-26T15:31:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:13 smithi160 ceph-mon[36715]: pgmap v8907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:13 smithi183 ceph-mon[37273]: pgmap v8907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:13 smithi144 ceph-mon[32129]: pgmap v8907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:15 smithi160 ceph-mon[36715]: pgmap v8908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:15 smithi183 ceph-mon[37273]: pgmap v8908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:15 smithi144 ceph-mon[32129]: pgmap v8908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:16.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15919.897463 secs 2024-07-26T15:31:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15919.897463 secs 2024-07-26T15:31:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15919.897463 secs 2024-07-26T15:31:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:17 smithi160 ceph-mon[36715]: pgmap v8909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:17 smithi183 ceph-mon[37273]: pgmap v8909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:17 smithi144 ceph-mon[32129]: pgmap v8909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:19 smithi160 ceph-mon[36715]: pgmap v8910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:19 smithi183 ceph-mon[37273]: pgmap v8910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:19 smithi144 ceph-mon[32129]: pgmap v8910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:21 smithi183 ceph-mon[37273]: pgmap v8911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15924.897562 secs 2024-07-26T15:31:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:21 smithi144 ceph-mon[32129]: pgmap v8911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15924.897562 secs 2024-07-26T15:31:21.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:21 smithi160 ceph-mon[36715]: pgmap v8911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15924.897562 secs 2024-07-26T15:31:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:23 smithi183 ceph-mon[37273]: pgmap v8912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:23 smithi144 ceph-mon[32129]: pgmap v8912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:23 smithi160 ceph-mon[36715]: pgmap v8912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:25 smithi183 ceph-mon[37273]: pgmap v8913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:25 smithi144 ceph-mon[32129]: pgmap v8913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:25 smithi160 ceph-mon[36715]: pgmap v8913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:26.156+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:31:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:26.157+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:31:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:26.157+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:31:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:26.157+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:31:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:26.157+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15929.897684 secs 2024-07-26T15:31:26.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15929.897684 secs 2024-07-26T15:31:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15929.897684 secs 2024-07-26T15:31:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:27 smithi183 ceph-mon[37273]: pgmap v8914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:27 smithi144 ceph-mon[32129]: pgmap v8914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:27 smithi160 ceph-mon[36715]: pgmap v8914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:28 smithi183 ceph-mon[37273]: pgmap v8915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:28 smithi144 ceph-mon[32129]: pgmap v8915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:29.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:28 smithi160 ceph-mon[36715]: pgmap v8915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:31 smithi160 ceph-mon[36715]: pgmap v8916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15934.897796 secs 2024-07-26T15:31:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:31 smithi183 ceph-mon[37273]: pgmap v8916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15934.897796 secs 2024-07-26T15:31:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:31 smithi144 ceph-mon[32129]: pgmap v8916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15934.897796 secs 2024-07-26T15:31:33.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:33 smithi160 ceph-mon[36715]: pgmap v8917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:33 smithi183 ceph-mon[37273]: pgmap v8917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:33 smithi144 ceph-mon[32129]: pgmap v8917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:35 smithi160 ceph-mon[36715]: pgmap v8918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:35 smithi183 ceph-mon[37273]: pgmap v8918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:35 smithi144 ceph-mon[32129]: pgmap v8918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15939.897920 secs 2024-07-26T15:31:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15939.897920 secs 2024-07-26T15:31:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15939.897920 secs 2024-07-26T15:31:37.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:37 smithi160 ceph-mon[36715]: pgmap v8919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:37 smithi183 ceph-mon[37273]: pgmap v8919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:37 smithi144 ceph-mon[32129]: pgmap v8919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:39 smithi160 ceph-mon[36715]: pgmap v8920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:39 smithi183 ceph-mon[37273]: pgmap v8920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:39 smithi144 ceph-mon[32129]: pgmap v8920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:41.175 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:41.173+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:31:41.175 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:41.174+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:31:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:41.174+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:31:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:41.174+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:31:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:41.174+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:41 smithi160 ceph-mon[36715]: pgmap v8921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15944.898055 secs 2024-07-26T15:31:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:41 smithi183 ceph-mon[37273]: pgmap v8921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15944.898055 secs 2024-07-26T15:31:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:41 smithi144 ceph-mon[32129]: pgmap v8921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15944.898055 secs 2024-07-26T15:31:43.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:43 smithi160 ceph-mon[36715]: pgmap v8922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:43 smithi183 ceph-mon[37273]: pgmap v8922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:43 smithi144 ceph-mon[32129]: pgmap v8922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:45 smithi160 ceph-mon[36715]: pgmap v8923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:45 smithi183 ceph-mon[37273]: pgmap v8923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:45 smithi144 ceph-mon[32129]: pgmap v8923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15949.898154 secs 2024-07-26T15:31:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15949.898154 secs 2024-07-26T15:31:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15949.898154 secs 2024-07-26T15:31:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:47 smithi160 ceph-mon[36715]: pgmap v8924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:47.623 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:47 smithi144 ceph-mon[32129]: pgmap v8924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:47 smithi183 ceph-mon[37273]: pgmap v8924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:49 smithi160 ceph-mon[36715]: pgmap v8925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:49 smithi183 ceph-mon[37273]: pgmap v8925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:49 smithi144 ceph-mon[32129]: pgmap v8925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:31:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:51 smithi183 ceph-mon[37273]: pgmap v8926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15954.898330 secs 2024-07-26T15:31:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:51 smithi144 ceph-mon[32129]: pgmap v8926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15954.898330 secs 2024-07-26T15:31:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:51 smithi160 ceph-mon[36715]: pgmap v8926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:31:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15954.898330 secs 2024-07-26T15:31:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:53 smithi183 ceph-mon[37273]: pgmap v8927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:53 smithi144 ceph-mon[32129]: pgmap v8927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:53 smithi160 ceph-mon[36715]: pgmap v8927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:31:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:54 smithi183 ceph-mon[37273]: pgmap v8928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:54 smithi183 ceph-mon[37273]: pgmap v8929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:31:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:54 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:31:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:54 smithi144 ceph-mon[32129]: pgmap v8928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:54 smithi144 ceph-mon[32129]: pgmap v8929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:31:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:54 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:31:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:54 smithi160 ceph-mon[36715]: pgmap v8928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:54 smithi160 ceph-mon[36715]: pgmap v8929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:31:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:54 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:31:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:56.190+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:31:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:56.191+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:31:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:56.191+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:31:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:56.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:31:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:31:56.191+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:56 smithi144 ceph-mon[32129]: pgmap v8930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15959.898457 secs 2024-07-26T15:31:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:56 smithi160 ceph-mon[36715]: pgmap v8930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15959.898457 secs 2024-07-26T15:31:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:56 smithi183 ceph-mon[37273]: pgmap v8930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15959.898457 secs 2024-07-26T15:31:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:31:58 smithi144 ceph-mon[32129]: pgmap v8931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:31:58 smithi160 ceph-mon[36715]: pgmap v8931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:31:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:31:58 smithi183 ceph-mon[37273]: pgmap v8931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:00 smithi144 ceph-mon[32129]: pgmap v8932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:00 smithi160 ceph-mon[36715]: pgmap v8932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:00 smithi183 ceph-mon[37273]: pgmap v8932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15964.898524 secs 2024-07-26T15:32:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15964.898524 secs 2024-07-26T15:32:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15964.898524 secs 2024-07-26T15:32:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:02 smithi144 ceph-mon[32129]: pgmap v8933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:02 smithi160 ceph-mon[36715]: pgmap v8933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:02 smithi183 ceph-mon[37273]: pgmap v8933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:04 smithi144 ceph-mon[32129]: pgmap v8934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:32:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:04 smithi160 ceph-mon[36715]: pgmap v8934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:32:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:04 smithi183 ceph-mon[37273]: pgmap v8934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:32:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:06 smithi144 ceph-mon[32129]: pgmap v8935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15969.898645 secs 2024-07-26T15:32:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:06 smithi160 ceph-mon[36715]: pgmap v8935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15969.898645 secs 2024-07-26T15:32:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:06 smithi183 ceph-mon[37273]: pgmap v8935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15969.898645 secs 2024-07-26T15:32:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:08 smithi144 ceph-mon[32129]: pgmap v8936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:08 smithi160 ceph-mon[36715]: pgmap v8936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:08 smithi183 ceph-mon[37273]: pgmap v8936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:10 smithi144 ceph-mon[32129]: pgmap v8937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:10 smithi160 ceph-mon[36715]: pgmap v8937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:10 smithi183 ceph-mon[37273]: pgmap v8937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:11.205+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:11.206+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:11.206+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:11.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:11.206+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15974.898729 secs 2024-07-26T15:32:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15974.898729 secs 2024-07-26T15:32:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15974.898729 secs 2024-07-26T15:32:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:12 smithi160 ceph-mon[36715]: pgmap v8938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:12 smithi183 ceph-mon[37273]: pgmap v8938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:12 smithi144 ceph-mon[32129]: pgmap v8938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:14 smithi160 ceph-mon[36715]: pgmap v8939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:14 smithi183 ceph-mon[37273]: pgmap v8939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:14 smithi144 ceph-mon[32129]: pgmap v8939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:16 smithi160 ceph-mon[36715]: pgmap v8940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15979.898805 secs 2024-07-26T15:32:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:16 smithi183 ceph-mon[37273]: pgmap v8940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15979.898805 secs 2024-07-26T15:32:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:16 smithi144 ceph-mon[32129]: pgmap v8940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15979.898805 secs 2024-07-26T15:32:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:18 smithi160 ceph-mon[36715]: pgmap v8941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:18 smithi183 ceph-mon[37273]: pgmap v8941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:18 smithi144 ceph-mon[32129]: pgmap v8941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:20 smithi160 ceph-mon[36715]: pgmap v8942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:20 smithi183 ceph-mon[37273]: pgmap v8942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:20 smithi144 ceph-mon[32129]: pgmap v8942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15984.898928 secs 2024-07-26T15:32:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15984.898928 secs 2024-07-26T15:32:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15984.898928 secs 2024-07-26T15:32:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:22 smithi160 ceph-mon[36715]: pgmap v8943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:22 smithi183 ceph-mon[37273]: pgmap v8943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:22 smithi144 ceph-mon[32129]: pgmap v8943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:24 smithi160 ceph-mon[36715]: pgmap v8944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:24 smithi183 ceph-mon[37273]: pgmap v8944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:24 smithi144 ceph-mon[32129]: pgmap v8944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:26.222+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:32:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:26.223+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:32:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:26.223+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:32:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:26.223+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:32:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:26.223+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:26 smithi160 ceph-mon[36715]: pgmap v8945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15989.899049 secs 2024-07-26T15:32:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:26 smithi183 ceph-mon[37273]: pgmap v8945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15989.899049 secs 2024-07-26T15:32:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:26 smithi144 ceph-mon[32129]: pgmap v8945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15989.899049 secs 2024-07-26T15:32:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:28 smithi160 ceph-mon[36715]: pgmap v8946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:28 smithi183 ceph-mon[37273]: pgmap v8946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:28 smithi144 ceph-mon[32129]: pgmap v8946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:30 smithi160 ceph-mon[36715]: pgmap v8947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:30 smithi183 ceph-mon[37273]: pgmap v8947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:30 smithi144 ceph-mon[32129]: pgmap v8947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15994.899192 secs 2024-07-26T15:32:31.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15994.899192 secs 2024-07-26T15:32:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15994.899192 secs 2024-07-26T15:32:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:32 smithi160 ceph-mon[36715]: pgmap v8948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:32 smithi183 ceph-mon[37273]: pgmap v8948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:32 smithi144 ceph-mon[32129]: pgmap v8948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:34 smithi160 ceph-mon[36715]: pgmap v8949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:34 smithi183 ceph-mon[37273]: pgmap v8949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:34 smithi144 ceph-mon[32129]: pgmap v8949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:36 smithi160 ceph-mon[36715]: pgmap v8950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 15999.899338 secs 2024-07-26T15:32:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:36 smithi183 ceph-mon[37273]: pgmap v8950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 15999.899338 secs 2024-07-26T15:32:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:36 smithi144 ceph-mon[32129]: pgmap v8950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 15999.899338 secs 2024-07-26T15:32:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:38 smithi183 ceph-mon[37273]: pgmap v8951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:38 smithi144 ceph-mon[32129]: pgmap v8951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:38 smithi160 ceph-mon[36715]: pgmap v8951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:40 smithi183 ceph-mon[37273]: pgmap v8952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:40 smithi144 ceph-mon[32129]: pgmap v8952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:41.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:40 smithi160 ceph-mon[36715]: pgmap v8952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:41.240 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:41.239+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:32:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:41.240+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:32:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:41.240+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:32:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:41.240+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:32:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:41.240+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16004.899482 secs 2024-07-26T15:32:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16004.899482 secs 2024-07-26T15:32:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16004.899482 secs 2024-07-26T15:32:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:42 smithi183 ceph-mon[37273]: pgmap v8953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:42 smithi144 ceph-mon[32129]: pgmap v8953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:42 smithi160 ceph-mon[36715]: pgmap v8953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:44 smithi183 ceph-mon[37273]: pgmap v8954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:44 smithi144 ceph-mon[32129]: pgmap v8954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:44 smithi160 ceph-mon[36715]: pgmap v8954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:46 smithi183 ceph-mon[37273]: pgmap v8955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16009.899565 secs 2024-07-26T15:32:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:46 smithi144 ceph-mon[32129]: pgmap v8955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16009.899565 secs 2024-07-26T15:32:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:46 smithi160 ceph-mon[36715]: pgmap v8955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16009.899565 secs 2024-07-26T15:32:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:48 smithi183 ceph-mon[37273]: pgmap v8956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:48.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:48.925+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:32:48.927 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:48.925+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:32:48.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:48.926+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:32:48.928 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:48.926+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:32:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:48 smithi144 ceph-mon[32129]: pgmap v8956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:48 smithi160 ceph-mon[36715]: pgmap v8956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:32:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:49 smithi144 ceph-mon[32129]: pgmap v8957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:49 smithi160 ceph-mon[36715]: pgmap v8957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:49 smithi183 ceph-mon[37273]: pgmap v8957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16014.899680 secs 2024-07-26T15:32:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16014.899680 secs 2024-07-26T15:32:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16014.899680 secs 2024-07-26T15:32:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:52 smithi183 ceph-mon[37273]: pgmap v8958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:52 smithi144 ceph-mon[32129]: pgmap v8958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:52 smithi160 ceph-mon[36715]: pgmap v8958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:32:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:54 smithi144 ceph-mon[32129]: pgmap v8959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:54.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:54 smithi160 ceph-mon[36715]: pgmap v8959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:54 smithi183 ceph-mon[37273]: pgmap v8959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:32:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:55 smithi144 ceph-mon[32129]: pgmap v8960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:32:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:55 smithi144 ceph-mon[32129]: pgmap v8961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:32:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:55 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:32:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:55 smithi160 ceph-mon[36715]: pgmap v8960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:32:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:55 smithi160 ceph-mon[36715]: pgmap v8961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:32:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:55 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:32:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:55 smithi183 ceph-mon[37273]: pgmap v8960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:32:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:55 smithi183 ceph-mon[37273]: pgmap v8961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:32:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:55 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:32:56.257 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:56.255+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:32:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:56.256+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:32:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:56.256+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:32:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:56.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:32:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:32:56.256+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16019.899794 secs 2024-07-26T15:32:56.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16019.899794 secs 2024-07-26T15:32:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16019.899794 secs 2024-07-26T15:32:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:57 smithi160 ceph-mon[36715]: pgmap v8962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:57 smithi183 ceph-mon[37273]: pgmap v8962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:57 smithi144 ceph-mon[32129]: pgmap v8962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:32:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:32:58 smithi144 ceph-mon[32129]: pgmap v8963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:32:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:32:58 smithi160 ceph-mon[36715]: pgmap v8963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:32:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:32:58 smithi183 ceph-mon[37273]: pgmap v8963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:33:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:00 smithi144 ceph-mon[32129]: pgmap v8964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:33:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:00 smithi160 ceph-mon[36715]: pgmap v8964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:33:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:00 smithi183 ceph-mon[37273]: pgmap v8964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:33:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16024.899936 secs 2024-07-26T15:33:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16024.899936 secs 2024-07-26T15:33:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16024.899936 secs 2024-07-26T15:33:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:02 smithi144 ceph-mon[32129]: pgmap v8965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:33:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:02 smithi160 ceph-mon[36715]: pgmap v8965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:33:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:02 smithi183 ceph-mon[37273]: pgmap v8965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:33:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:04 smithi144 ceph-mon[32129]: pgmap v8966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:33:05.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:04 smithi160 ceph-mon[36715]: pgmap v8966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:33:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:04 smithi183 ceph-mon[37273]: pgmap v8966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:33:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16029.900050 secs 2024-07-26T15:33:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16029.900050 secs 2024-07-26T15:33:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16029.900050 secs 2024-07-26T15:33:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:07 smithi183 ceph-mon[37273]: pgmap v8967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:07 smithi144 ceph-mon[32129]: pgmap v8967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:07 smithi160 ceph-mon[36715]: pgmap v8967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:08 smithi144 ceph-mon[32129]: pgmap v8968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:09.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:08 smithi160 ceph-mon[36715]: pgmap v8968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:08 smithi183 ceph-mon[37273]: pgmap v8968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:10 smithi144 ceph-mon[32129]: pgmap v8969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:11.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:10 smithi160 ceph-mon[36715]: pgmap v8969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:10 smithi183 ceph-mon[37273]: pgmap v8969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:11.261+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:33:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:11.262+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:33:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:11.262+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:33:11.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:11.262+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:33:11.264 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:11.262+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16034.900165 secs 2024-07-26T15:33:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16034.900165 secs 2024-07-26T15:33:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16034.900165 secs 2024-07-26T15:33:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:12 smithi144 ceph-mon[32129]: pgmap v8970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:12 smithi160 ceph-mon[36715]: pgmap v8970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:12 smithi183 ceph-mon[37273]: pgmap v8970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:14 smithi144 ceph-mon[32129]: pgmap v8971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:15.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:14 smithi160 ceph-mon[36715]: pgmap v8971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:14 smithi183 ceph-mon[37273]: pgmap v8971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16039.900286 secs 2024-07-26T15:33:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16039.900286 secs 2024-07-26T15:33:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16039.900286 secs 2024-07-26T15:33:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:17 smithi183 ceph-mon[37273]: pgmap v8972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:17 smithi144 ceph-mon[32129]: pgmap v8972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:17 smithi160 ceph-mon[36715]: pgmap v8972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:18 smithi144 ceph-mon[32129]: pgmap v8973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:18 smithi160 ceph-mon[36715]: pgmap v8973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:18 smithi183 ceph-mon[37273]: pgmap v8973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:20 smithi144 ceph-mon[32129]: pgmap v8974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:20 smithi160 ceph-mon[36715]: pgmap v8974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:21.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:20 smithi183 ceph-mon[37273]: pgmap v8974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16044.900411 secs 2024-07-26T15:33:22.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16044.900411 secs 2024-07-26T15:33:22.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16044.900411 secs 2024-07-26T15:33:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:22 smithi144 ceph-mon[32129]: pgmap v8975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:23.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:22 smithi160 ceph-mon[36715]: pgmap v8975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:22 smithi183 ceph-mon[37273]: pgmap v8975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:24 smithi144 ceph-mon[32129]: pgmap v8976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:24 smithi160 ceph-mon[36715]: pgmap v8976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:25.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:24 smithi183 ceph-mon[37273]: pgmap v8976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:26.278+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:33:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:26.279+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:33:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:26.279+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:33:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:26.279+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:33:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:26.279+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16049.900567 secs 2024-07-26T15:33:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16049.900567 secs 2024-07-26T15:33:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16049.900567 secs 2024-07-26T15:33:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:27 smithi183 ceph-mon[37273]: pgmap v8977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:27 smithi144 ceph-mon[32129]: pgmap v8977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:27.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:27 smithi160 ceph-mon[36715]: pgmap v8977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:28 smithi183 ceph-mon[37273]: pgmap v8978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:28 smithi144 ceph-mon[32129]: pgmap v8978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:28 smithi160 ceph-mon[36715]: pgmap v8978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:30 smithi144 ceph-mon[32129]: pgmap v8979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:30 smithi160 ceph-mon[36715]: pgmap v8979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:30 smithi183 ceph-mon[37273]: pgmap v8979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16054.900692 secs 2024-07-26T15:33:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16054.900692 secs 2024-07-26T15:33:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16054.900692 secs 2024-07-26T15:33:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:32 smithi144 ceph-mon[32129]: pgmap v8980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:32 smithi160 ceph-mon[36715]: pgmap v8980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:32 smithi183 ceph-mon[37273]: pgmap v8980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:34 smithi144 ceph-mon[32129]: pgmap v8981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:34 smithi160 ceph-mon[36715]: pgmap v8981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:34 smithi183 ceph-mon[37273]: pgmap v8981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16059.900813 secs 2024-07-26T15:33:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16059.900813 secs 2024-07-26T15:33:36.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16059.900813 secs 2024-07-26T15:33:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:37 smithi183 ceph-mon[37273]: pgmap v8982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:37 smithi144 ceph-mon[32129]: pgmap v8982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:37 smithi160 ceph-mon[36715]: pgmap v8982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:38 smithi144 ceph-mon[32129]: pgmap v8983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:38 smithi160 ceph-mon[36715]: pgmap v8983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:38 smithi183 ceph-mon[37273]: pgmap v8983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:40 smithi144 ceph-mon[32129]: pgmap v8984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:40 smithi160 ceph-mon[36715]: pgmap v8984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:40 smithi183 ceph-mon[37273]: pgmap v8984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:41.295+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:33:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:41.296+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:33:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:41.296+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:33:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:41.296+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:33:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:41.296+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16064.900932 secs 2024-07-26T15:33:42.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16064.900932 secs 2024-07-26T15:33:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16064.900932 secs 2024-07-26T15:33:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:42 smithi144 ceph-mon[32129]: pgmap v8985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:42 smithi160 ceph-mon[36715]: pgmap v8985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:42 smithi183 ceph-mon[37273]: pgmap v8985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:44 smithi144 ceph-mon[32129]: pgmap v8986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:44 smithi160 ceph-mon[36715]: pgmap v8986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:44 smithi183 ceph-mon[37273]: pgmap v8986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16069.901025 secs 2024-07-26T15:33:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16069.901025 secs 2024-07-26T15:33:46.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16069.901025 secs 2024-07-26T15:33:47.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:47 smithi183 ceph-mon[37273]: pgmap v8987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:47 smithi144 ceph-mon[32129]: pgmap v8987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:47 smithi160 ceph-mon[36715]: pgmap v8987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:48 smithi144 ceph-mon[32129]: pgmap v8988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:48 smithi160 ceph-mon[36715]: pgmap v8988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:48 smithi183 ceph-mon[37273]: pgmap v8988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:50 smithi144 ceph-mon[32129]: pgmap v8989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:51.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:50 smithi160 ceph-mon[36715]: pgmap v8989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:50 smithi183 ceph-mon[37273]: pgmap v8989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16074.901167 secs 2024-07-26T15:33:52.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16074.901167 secs 2024-07-26T15:33:52.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16074.901167 secs 2024-07-26T15:33:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:52 smithi144 ceph-mon[32129]: pgmap v8990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:52 smithi160 ceph-mon[36715]: pgmap v8990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:52 smithi183 ceph-mon[37273]: pgmap v8990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:33:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:54 smithi144 ceph-mon[32129]: pgmap v8991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:54 smithi160 ceph-mon[36715]: pgmap v8991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:54 smithi183 ceph-mon[37273]: pgmap v8991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:55.929 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:55 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:55 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:55 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:56.311+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:33:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:56.311+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:33:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:56.311+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:33:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:56.311+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:33:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:33:56.311+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:56.851 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16079.901310 secs 2024-07-26T15:33:56.851 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:56 smithi144 ceph-mon[32129]: pgmap v8992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:56.967 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16079.901310 secs 2024-07-26T15:33:56.968 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:56 smithi160 ceph-mon[36715]: pgmap v8992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16079.901310 secs 2024-07-26T15:33:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:56 smithi183 ceph-mon[37273]: pgmap v8992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:33:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:33:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:59 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:33:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:33:59 smithi160 ceph-mon[36715]: pgmap v8993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:33:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:59 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:33:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:33:59 smithi183 ceph-mon[37273]: pgmap v8993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:33:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:33:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:59 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:33:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:33:59 smithi144 ceph-mon[32129]: pgmap v8993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:00.920 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:00 smithi144 ceph-mon[32129]: pgmap v8994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:01.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:00 smithi160 ceph-mon[36715]: pgmap v8994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:00 smithi183 ceph-mon[37273]: pgmap v8994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:02.258 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16084.901449 secs 2024-07-26T15:34:02.258 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.258 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.258 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.258 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: pgmap v8995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: pgmap v8996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:34:02.259 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:01 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16084.901449 secs 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: pgmap v8995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: pgmap v8996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:34:02.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:01 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16084.901449 secs 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: pgmap v8995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: pgmap v8996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T15:34:02.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:01 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:34:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:04 smithi183 ceph-mon[37273]: pgmap v8997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T15:34:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:04 smithi144 ceph-mon[32129]: pgmap v8997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T15:34:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:04 smithi160 ceph-mon[36715]: pgmap v8997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2 op/s 2024-07-26T15:34:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:06 smithi183 ceph-mon[37273]: pgmap v8998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:34:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16089.901511 secs 2024-07-26T15:34:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:06 smithi144 ceph-mon[32129]: pgmap v8998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:34:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16089.901511 secs 2024-07-26T15:34:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:06 smithi160 ceph-mon[36715]: pgmap v8998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:34:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16089.901511 secs 2024-07-26T15:34:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:08 smithi160 ceph-mon[36715]: pgmap v8999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:34:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:08 smithi183 ceph-mon[37273]: pgmap v8999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:34:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:08 smithi144 ceph-mon[32129]: pgmap v8999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:34:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:10 smithi183 ceph-mon[37273]: pgmap v9000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:34:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:10 smithi144 ceph-mon[32129]: pgmap v9000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:34:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:10 smithi160 ceph-mon[36715]: pgmap v9000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:34:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:11.327+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:34:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:11.329+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:34:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:11.329+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:34:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:11.329+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:34:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:11.329+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16094.901627 secs 2024-07-26T15:34:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16094.901627 secs 2024-07-26T15:34:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16094.901627 secs 2024-07-26T15:34:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:12 smithi144 ceph-mon[32129]: pgmap v9001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:34:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:12 smithi160 ceph-mon[36715]: pgmap v9001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:34:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:12 smithi183 ceph-mon[37273]: pgmap v9001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:34:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:14 smithi144 ceph-mon[32129]: pgmap v9002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:14 smithi160 ceph-mon[36715]: pgmap v9002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:14 smithi183 ceph-mon[37273]: pgmap v9002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:16 smithi144 ceph-mon[32129]: pgmap v9003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16099.901748 secs 2024-07-26T15:34:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:16 smithi160 ceph-mon[36715]: pgmap v9003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16099.901748 secs 2024-07-26T15:34:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:16 smithi183 ceph-mon[37273]: pgmap v9003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16099.901748 secs 2024-07-26T15:34:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:18 smithi144 ceph-mon[32129]: pgmap v9004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:18 smithi160 ceph-mon[36715]: pgmap v9004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:18 smithi183 ceph-mon[37273]: pgmap v9004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:20 smithi144 ceph-mon[32129]: pgmap v9005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:20 smithi160 ceph-mon[36715]: pgmap v9005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:20 smithi183 ceph-mon[37273]: pgmap v9005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16104.901874 secs 2024-07-26T15:34:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16104.901874 secs 2024-07-26T15:34:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16104.901874 secs 2024-07-26T15:34:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:22 smithi144 ceph-mon[32129]: pgmap v9006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:22.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:22 smithi183 ceph-mon[37273]: pgmap v9006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:22 smithi160 ceph-mon[36715]: pgmap v9006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:24 smithi144 ceph-mon[32129]: pgmap v9007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:24 smithi160 ceph-mon[36715]: pgmap v9007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:24 smithi183 ceph-mon[37273]: pgmap v9007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:26.345+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:34:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:26.346+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:34:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:26.346+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:34:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:26.346+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:34:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:26.346+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:26 smithi144 ceph-mon[32129]: pgmap v9008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16109.902013 secs 2024-07-26T15:34:26.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:26 smithi160 ceph-mon[36715]: pgmap v9008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16109.902013 secs 2024-07-26T15:34:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:26 smithi183 ceph-mon[37273]: pgmap v9008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16109.902013 secs 2024-07-26T15:34:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:28 smithi144 ceph-mon[32129]: pgmap v9009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:28 smithi160 ceph-mon[36715]: pgmap v9009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:28 smithi183 ceph-mon[37273]: pgmap v9009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:30 smithi144 ceph-mon[32129]: pgmap v9010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:30 smithi160 ceph-mon[36715]: pgmap v9010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:30 smithi183 ceph-mon[37273]: pgmap v9010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16114.902111 secs 2024-07-26T15:34:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16114.902111 secs 2024-07-26T15:34:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16114.902111 secs 2024-07-26T15:34:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:32 smithi144 ceph-mon[32129]: pgmap v9011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:32 smithi160 ceph-mon[36715]: pgmap v9011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:32 smithi183 ceph-mon[37273]: pgmap v9011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:34 smithi160 ceph-mon[36715]: pgmap v9012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:34 smithi183 ceph-mon[37273]: pgmap v9012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:34 smithi144 ceph-mon[32129]: pgmap v9012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:36 smithi160 ceph-mon[36715]: pgmap v9013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16119.902233 secs 2024-07-26T15:34:36.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:36 smithi183 ceph-mon[37273]: pgmap v9013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:36.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16119.902233 secs 2024-07-26T15:34:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:36 smithi144 ceph-mon[32129]: pgmap v9013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16119.902233 secs 2024-07-26T15:34:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:38 smithi160 ceph-mon[36715]: pgmap v9014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:38 smithi183 ceph-mon[37273]: pgmap v9014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:38 smithi144 ceph-mon[32129]: pgmap v9014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:40 smithi160 ceph-mon[36715]: pgmap v9015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:40 smithi183 ceph-mon[37273]: pgmap v9015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:40 smithi144 ceph-mon[32129]: pgmap v9015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:41.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:34:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:41.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:34:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:41.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:34:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:41.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:34:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:41.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16124.902355 secs 2024-07-26T15:34:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16124.902355 secs 2024-07-26T15:34:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16124.902355 secs 2024-07-26T15:34:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:42 smithi160 ceph-mon[36715]: pgmap v9016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:42 smithi183 ceph-mon[37273]: pgmap v9016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:42 smithi144 ceph-mon[32129]: pgmap v9016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:44 smithi160 ceph-mon[36715]: pgmap v9017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:44 smithi183 ceph-mon[37273]: pgmap v9017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:44 smithi144 ceph-mon[32129]: pgmap v9017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:46 smithi160 ceph-mon[36715]: pgmap v9018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16129.902419 secs 2024-07-26T15:34:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:46 smithi183 ceph-mon[37273]: pgmap v9018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16129.902419 secs 2024-07-26T15:34:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:46 smithi144 ceph-mon[32129]: pgmap v9018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16129.902419 secs 2024-07-26T15:34:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:48 smithi160 ceph-mon[36715]: pgmap v9019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:48 smithi183 ceph-mon[37273]: pgmap v9019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:48 smithi144 ceph-mon[32129]: pgmap v9019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:50 smithi160 ceph-mon[36715]: pgmap v9020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:50 smithi183 ceph-mon[37273]: pgmap v9020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:50 smithi144 ceph-mon[32129]: pgmap v9020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16134.902544 secs 2024-07-26T15:34:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16134.902544 secs 2024-07-26T15:34:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16134.902544 secs 2024-07-26T15:34:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:52 smithi160 ceph-mon[36715]: pgmap v9021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:52 smithi183 ceph-mon[37273]: pgmap v9021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:52 smithi144 ceph-mon[32129]: pgmap v9021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:54 smithi160 ceph-mon[36715]: pgmap v9022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:54 smithi183 ceph-mon[37273]: pgmap v9022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:54 smithi144 ceph-mon[32129]: pgmap v9022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:34:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:56.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:34:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:56.366+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:34:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:56.366+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:34:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:56.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:34:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:34:56.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:56 smithi160 ceph-mon[36715]: pgmap v9023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16139.902661 secs 2024-07-26T15:34:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:56 smithi183 ceph-mon[37273]: pgmap v9023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16139.902661 secs 2024-07-26T15:34:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:56 smithi144 ceph-mon[32129]: pgmap v9023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:34:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16139.902661 secs 2024-07-26T15:34:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:34:58 smithi160 ceph-mon[36715]: pgmap v9024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:34:58 smithi183 ceph-mon[37273]: pgmap v9024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:34:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:34:58 smithi144 ceph-mon[32129]: pgmap v9024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:00 smithi183 ceph-mon[37273]: pgmap v9025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:00 smithi144 ceph-mon[32129]: pgmap v9025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:00 smithi160 ceph-mon[36715]: pgmap v9025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:01.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16144.902780 secs 2024-07-26T15:35:01.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:01 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16144.902780 secs 2024-07-26T15:35:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:01 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:01.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16144.902780 secs 2024-07-26T15:35:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:01 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:02.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: pgmap v9026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.489 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:02 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: pgmap v9026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:02 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: pgmap v9026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:02 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:03.523 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:03 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:35:03.524 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:03 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:35:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:03 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:35:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:03 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:35:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:03 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:35:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:03 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:35:04.486 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:04 smithi183 ceph-mon[37273]: pgmap v9027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:35:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:04 smithi144 ceph-mon[32129]: pgmap v9027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:35:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:04 smithi160 ceph-mon[36715]: pgmap v9027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:35:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:35:06.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:06.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:06.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: pgmap v9028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:35:06.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: pgmap v9029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:06.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:06.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: pgmap v9028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:35:06.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: pgmap v9029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:06.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: pgmap v9028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: pgmap v9029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:35:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16149.902938 secs 2024-07-26T15:35:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16149.902938 secs 2024-07-26T15:35:07.308 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16149.902938 secs 2024-07-26T15:35:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:07 smithi144 ceph-mon[32129]: pgmap v9030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:35:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:07 smithi160 ceph-mon[36715]: pgmap v9030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:35:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:07 smithi183 ceph-mon[37273]: pgmap v9030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:35:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:09 smithi160 ceph-mon[36715]: pgmap v9031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:09 smithi183 ceph-mon[37273]: pgmap v9031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:09 smithi144 ceph-mon[32129]: pgmap v9031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:11.379+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:35:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:11.380+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:35:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:11.380+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:35:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:11.380+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:35:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:11.381+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:11 smithi160 ceph-mon[36715]: pgmap v9032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16154.903042 secs 2024-07-26T15:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:11 smithi183 ceph-mon[37273]: pgmap v9032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16154.903042 secs 2024-07-26T15:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:11 smithi144 ceph-mon[32129]: pgmap v9032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16154.903042 secs 2024-07-26T15:35:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:13 smithi160 ceph-mon[36715]: pgmap v9033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:13 smithi183 ceph-mon[37273]: pgmap v9033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:13 smithi144 ceph-mon[32129]: pgmap v9033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:15 smithi160 ceph-mon[36715]: pgmap v9034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:15 smithi183 ceph-mon[37273]: pgmap v9034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:15 smithi144 ceph-mon[32129]: pgmap v9034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:35:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16159.903151 secs 2024-07-26T15:35:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16159.903151 secs 2024-07-26T15:35:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16159.903151 secs 2024-07-26T15:35:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:18 smithi160 ceph-mon[36715]: pgmap v9035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:18 smithi183 ceph-mon[37273]: pgmap v9035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:18 smithi144 ceph-mon[32129]: pgmap v9035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:35:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:20 smithi160 ceph-mon[36715]: pgmap v9036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:20 smithi183 ceph-mon[37273]: pgmap v9036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:20 smithi144 ceph-mon[32129]: pgmap v9036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16164.903300 secs 2024-07-26T15:35:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:22 smithi160 ceph-mon[36715]: pgmap v9037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16164.903300 secs 2024-07-26T15:35:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:22 smithi183 ceph-mon[37273]: pgmap v9037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16164.903300 secs 2024-07-26T15:35:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:22 smithi144 ceph-mon[32129]: pgmap v9037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:24 smithi144 ceph-mon[32129]: pgmap v9038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:24 smithi160 ceph-mon[36715]: pgmap v9038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:24 smithi183 ceph-mon[37273]: pgmap v9038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:26 smithi144 ceph-mon[32129]: pgmap v9039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:26 smithi160 ceph-mon[36715]: pgmap v9039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:26 smithi183 ceph-mon[37273]: pgmap v9039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:26.396+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:35:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:26.397+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:35:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:26.397+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:35:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:26.397+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:35:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:26.397+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16169.903404 secs 2024-07-26T15:35:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16169.903404 secs 2024-07-26T15:35:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16169.903404 secs 2024-07-26T15:35:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:28 smithi160 ceph-mon[36715]: pgmap v9040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:28 smithi183 ceph-mon[37273]: pgmap v9040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:28 smithi144 ceph-mon[32129]: pgmap v9040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:30 smithi160 ceph-mon[36715]: pgmap v9041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:30 smithi183 ceph-mon[37273]: pgmap v9041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:30 smithi144 ceph-mon[32129]: pgmap v9041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16174.903473 secs 2024-07-26T15:35:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:32 smithi160 ceph-mon[36715]: pgmap v9042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16174.903473 secs 2024-07-26T15:35:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:32 smithi183 ceph-mon[37273]: pgmap v9042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16174.903473 secs 2024-07-26T15:35:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:32 smithi144 ceph-mon[32129]: pgmap v9042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:34 smithi183 ceph-mon[37273]: pgmap v9043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:34 smithi144 ceph-mon[32129]: pgmap v9043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:34 smithi160 ceph-mon[36715]: pgmap v9043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:36.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:36 smithi183 ceph-mon[37273]: pgmap v9044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16179.903594 secs 2024-07-26T15:35:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:36 smithi144 ceph-mon[32129]: pgmap v9044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16179.903594 secs 2024-07-26T15:35:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:36 smithi160 ceph-mon[36715]: pgmap v9044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16179.903594 secs 2024-07-26T15:35:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:38 smithi183 ceph-mon[37273]: pgmap v9045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:38 smithi144 ceph-mon[32129]: pgmap v9045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:38 smithi160 ceph-mon[36715]: pgmap v9045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:40 smithi183 ceph-mon[37273]: pgmap v9046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:40 smithi144 ceph-mon[32129]: pgmap v9046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:40 smithi160 ceph-mon[36715]: pgmap v9046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16184.903711 secs 2024-07-26T15:35:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:41.410+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:35:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:41.411+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:35:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:41.411+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:35:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:41.411+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:35:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:41.411+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16184.903711 secs 2024-07-26T15:35:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16184.903711 secs 2024-07-26T15:35:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:42 smithi183 ceph-mon[37273]: pgmap v9047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:42 smithi144 ceph-mon[32129]: pgmap v9047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:42 smithi160 ceph-mon[36715]: pgmap v9047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:44 smithi183 ceph-mon[37273]: pgmap v9048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:44 smithi144 ceph-mon[32129]: pgmap v9048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:44 smithi160 ceph-mon[36715]: pgmap v9048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:35:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:35:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a803e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:35:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a803e040 comp rx=0 tx=0).stop 2024-07-26T15:35:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T15:35:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b5710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:35:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.150+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b5710 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:35:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.151+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:35:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.151+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:35:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:46.153+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5e60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74a804c040 tx=0x7f74a8059040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:35:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:46 smithi183 ceph-mon[37273]: pgmap v9049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16189.903828 secs 2024-07-26T15:35:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:46 smithi144 ceph-mon[32129]: pgmap v9049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16189.903828 secs 2024-07-26T15:35:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:46 smithi160 ceph-mon[36715]: pgmap v9049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16189.903828 secs 2024-07-26T15:35:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:48 smithi183 ceph-mon[37273]: pgmap v9050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:48 smithi144 ceph-mon[32129]: pgmap v9050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:48 smithi160 ceph-mon[36715]: pgmap v9050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:48.958 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:48.957+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:35:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:48.957+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:35:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:48.958+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:35:48.959 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:48.958+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:35:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:50 smithi183 ceph-mon[37273]: pgmap v9051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:50 smithi144 ceph-mon[32129]: pgmap v9051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:50 smithi160 ceph-mon[36715]: pgmap v9051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16194.903955 secs 2024-07-26T15:35:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16194.903955 secs 2024-07-26T15:35:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16194.903955 secs 2024-07-26T15:35:52.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:52 smithi183 ceph-mon[37273]: pgmap v9052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:52 smithi144 ceph-mon[32129]: pgmap v9052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:52 smithi160 ceph-mon[36715]: pgmap v9052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:54 smithi144 ceph-mon[32129]: pgmap v9053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:54 smithi160 ceph-mon[36715]: pgmap v9053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:54 smithi183 ceph-mon[37273]: pgmap v9053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:35:56.426 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:56.425+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:35:56.427 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:56.426+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:35:56.427 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:56.426+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:35:56.427 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:56.426+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:35:56.427 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:35:56.426+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:56 smithi144 ceph-mon[32129]: pgmap v9054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16199.904074 secs 2024-07-26T15:35:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:56 smithi160 ceph-mon[36715]: pgmap v9054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16199.904074 secs 2024-07-26T15:35:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:56 smithi183 ceph-mon[37273]: pgmap v9054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:35:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16199.904074 secs 2024-07-26T15:35:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:35:58 smithi144 ceph-mon[32129]: pgmap v9055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:35:58 smithi160 ceph-mon[36715]: pgmap v9055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:35:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:35:58 smithi183 ceph-mon[37273]: pgmap v9055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:00 smithi144 ceph-mon[32129]: pgmap v9056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:00 smithi160 ceph-mon[36715]: pgmap v9056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:00 smithi183 ceph-mon[37273]: pgmap v9056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16204.904199 secs 2024-07-26T15:36:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16204.904199 secs 2024-07-26T15:36:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16204.904199 secs 2024-07-26T15:36:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:02 smithi144 ceph-mon[32129]: pgmap v9057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:02 smithi160 ceph-mon[36715]: pgmap v9057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:02 smithi183 ceph-mon[37273]: pgmap v9057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:04 smithi144 ceph-mon[32129]: pgmap v9058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:04 smithi160 ceph-mon[36715]: pgmap v9058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:04 smithi183 ceph-mon[37273]: pgmap v9058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:05.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:06 smithi144 ceph-mon[32129]: pgmap v9059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16209.904351 secs 2024-07-26T15:36:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:06 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:36:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:06 smithi160 ceph-mon[36715]: pgmap v9059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16209.904351 secs 2024-07-26T15:36:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:06 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:36:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:06 smithi183 ceph-mon[37273]: pgmap v9059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16209.904351 secs 2024-07-26T15:36:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:06 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:36:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:07 smithi144 ceph-mon[32129]: pgmap v9060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:07 smithi144 ceph-mon[32129]: pgmap v9061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:07 smithi160 ceph-mon[36715]: pgmap v9060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:07 smithi160 ceph-mon[36715]: pgmap v9061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:07 smithi183 ceph-mon[37273]: pgmap v9060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:07 smithi183 ceph-mon[37273]: pgmap v9061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:09 smithi144 ceph-mon[32129]: pgmap v9062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:36:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:09 smithi160 ceph-mon[36715]: pgmap v9062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:36:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:09 smithi183 ceph-mon[37273]: pgmap v9062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:36:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:11.442+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:36:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:11.443+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:36:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:11.443+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:36:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:11.443+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:36:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:11.443+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:11 smithi144 ceph-mon[32129]: pgmap v9063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:36:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16214.904414 secs 2024-07-26T15:36:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:11 smithi160 ceph-mon[36715]: pgmap v9063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:36:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16214.904414 secs 2024-07-26T15:36:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:11 smithi183 ceph-mon[37273]: pgmap v9063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:36:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16214.904414 secs 2024-07-26T15:36:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:13 smithi144 ceph-mon[32129]: pgmap v9064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:13.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:13 smithi160 ceph-mon[36715]: pgmap v9064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:13 smithi183 ceph-mon[37273]: pgmap v9064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:15 smithi160 ceph-mon[36715]: pgmap v9065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:15 smithi183 ceph-mon[37273]: pgmap v9065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:15 smithi144 ceph-mon[32129]: pgmap v9065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:36:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16219.904554 secs 2024-07-26T15:36:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16219.904554 secs 2024-07-26T15:36:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16219.904554 secs 2024-07-26T15:36:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:17 smithi160 ceph-mon[36715]: pgmap v9066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:36:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:17 smithi183 ceph-mon[37273]: pgmap v9066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:36:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:17 smithi144 ceph-mon[32129]: pgmap v9066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:36:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:19 smithi160 ceph-mon[36715]: pgmap v9067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:19 smithi183 ceph-mon[37273]: pgmap v9067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:19 smithi144 ceph-mon[32129]: pgmap v9067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:21 smithi160 ceph-mon[36715]: pgmap v9068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16224.904671 secs 2024-07-26T15:36:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:21 smithi183 ceph-mon[37273]: pgmap v9068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16224.904671 secs 2024-07-26T15:36:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:21 smithi144 ceph-mon[32129]: pgmap v9068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16224.904671 secs 2024-07-26T15:36:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:23 smithi160 ceph-mon[36715]: pgmap v9069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:23 smithi183 ceph-mon[37273]: pgmap v9069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:23 smithi144 ceph-mon[32129]: pgmap v9069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:25.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:25 smithi144 ceph-mon[32129]: pgmap v9070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:25 smithi160 ceph-mon[36715]: pgmap v9070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:25 smithi183 ceph-mon[37273]: pgmap v9070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:26.460 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:26.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:36:26.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:26.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:36:26.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:26.458+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:36:26.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:26.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:36:26.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:26.459+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16229.904766 secs 2024-07-26T15:36:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16229.904766 secs 2024-07-26T15:36:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16229.904766 secs 2024-07-26T15:36:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:27 smithi160 ceph-mon[36715]: pgmap v9071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:27 smithi183 ceph-mon[37273]: pgmap v9071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:27 smithi144 ceph-mon[32129]: pgmap v9071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:29 smithi160 ceph-mon[36715]: pgmap v9072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:29 smithi183 ceph-mon[37273]: pgmap v9072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:29 smithi144 ceph-mon[32129]: pgmap v9072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:31 smithi160 ceph-mon[36715]: pgmap v9073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16234.904888 secs 2024-07-26T15:36:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:31 smithi183 ceph-mon[37273]: pgmap v9073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16234.904888 secs 2024-07-26T15:36:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:31 smithi144 ceph-mon[32129]: pgmap v9073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16234.904888 secs 2024-07-26T15:36:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:33 smithi160 ceph-mon[36715]: pgmap v9074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:33 smithi183 ceph-mon[37273]: pgmap v9074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:33 smithi144 ceph-mon[32129]: pgmap v9074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:35 smithi160 ceph-mon[36715]: pgmap v9075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:35 smithi183 ceph-mon[37273]: pgmap v9075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:35 smithi144 ceph-mon[32129]: pgmap v9075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16239.904993 secs 2024-07-26T15:36:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16239.904993 secs 2024-07-26T15:36:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16239.904993 secs 2024-07-26T15:36:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:37 smithi183 ceph-mon[37273]: pgmap v9076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:37 smithi144 ceph-mon[32129]: pgmap v9076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:37 smithi160 ceph-mon[36715]: pgmap v9076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:39 smithi183 ceph-mon[37273]: pgmap v9077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:39 smithi144 ceph-mon[32129]: pgmap v9077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:39 smithi160 ceph-mon[36715]: pgmap v9077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:41 smithi144 ceph-mon[32129]: pgmap v9078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16244.905131 secs 2024-07-26T15:36:41.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:41.475+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:36:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:41.476+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:36:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:41.476+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:36:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:41.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:36:41.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:41.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:41 smithi183 ceph-mon[37273]: pgmap v9078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16244.905131 secs 2024-07-26T15:36:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:41 smithi160 ceph-mon[36715]: pgmap v9078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16244.905131 secs 2024-07-26T15:36:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:43 smithi183 ceph-mon[37273]: pgmap v9079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:43 smithi144 ceph-mon[32129]: pgmap v9079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:43 smithi160 ceph-mon[36715]: pgmap v9079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:45 smithi183 ceph-mon[37273]: pgmap v9080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:45 smithi144 ceph-mon[32129]: pgmap v9080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:45 smithi160 ceph-mon[36715]: pgmap v9080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16249.905251 secs 2024-07-26T15:36:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16249.905251 secs 2024-07-26T15:36:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16249.905251 secs 2024-07-26T15:36:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:47 smithi183 ceph-mon[37273]: pgmap v9081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:47 smithi144 ceph-mon[32129]: pgmap v9081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:47 smithi160 ceph-mon[36715]: pgmap v9081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:49 smithi183 ceph-mon[37273]: pgmap v9082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:49 smithi144 ceph-mon[32129]: pgmap v9082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:49 smithi160 ceph-mon[36715]: pgmap v9082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:51 smithi183 ceph-mon[37273]: pgmap v9083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16254.905372 secs 2024-07-26T15:36:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:51 smithi144 ceph-mon[32129]: pgmap v9083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16254.905372 secs 2024-07-26T15:36:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:51 smithi160 ceph-mon[36715]: pgmap v9083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16254.905372 secs 2024-07-26T15:36:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:53 smithi183 ceph-mon[37273]: pgmap v9084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:53 smithi144 ceph-mon[32129]: pgmap v9084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:53 smithi160 ceph-mon[36715]: pgmap v9084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:55 smithi183 ceph-mon[37273]: pgmap v9085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:55 smithi144 ceph-mon[32129]: pgmap v9085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:55 smithi160 ceph-mon[36715]: pgmap v9085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:36:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:56.484+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:36:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:56.485+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:36:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:56.485+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:36:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:56.485+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:36:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:36:56.485+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16259.905454 secs 2024-07-26T15:36:56.710 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16259.905454 secs 2024-07-26T15:36:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16259.905454 secs 2024-07-26T15:36:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:57 smithi144 ceph-mon[32129]: pgmap v9086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:57 smithi160 ceph-mon[36715]: pgmap v9086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:57 smithi183 ceph-mon[37273]: pgmap v9086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:36:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:36:59 smithi144 ceph-mon[32129]: pgmap v9087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:36:59 smithi160 ceph-mon[36715]: pgmap v9087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:36:59.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:36:59 smithi183 ceph-mon[37273]: pgmap v9087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:01 smithi144 ceph-mon[32129]: pgmap v9088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16264.905517 secs 2024-07-26T15:37:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:01 smithi160 ceph-mon[36715]: pgmap v9088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16264.905517 secs 2024-07-26T15:37:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:01 smithi183 ceph-mon[37273]: pgmap v9088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16264.905517 secs 2024-07-26T15:37:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:03 smithi144 ceph-mon[32129]: pgmap v9089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:03.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:03 smithi160 ceph-mon[36715]: pgmap v9089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:03 smithi183 ceph-mon[37273]: pgmap v9089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:05 smithi144 ceph-mon[32129]: pgmap v9090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:05.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:05 smithi160 ceph-mon[36715]: pgmap v9090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:05 smithi183 ceph-mon[37273]: pgmap v9090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16269.905604 secs 2024-07-26T15:37:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:06 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16269.905604 secs 2024-07-26T15:37:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:06 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16269.905604 secs 2024-07-26T15:37:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:06 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:07 smithi144 ceph-mon[32129]: pgmap v9091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:07 smithi160 ceph-mon[36715]: pgmap v9091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:07 smithi183 ceph-mon[37273]: pgmap v9091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:08 smithi160 ceph-mon[36715]: pgmap v9092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:08 smithi160 ceph-mon[36715]: pgmap v9093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:08 smithi183 ceph-mon[37273]: pgmap v9092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:08 smithi183 ceph-mon[37273]: pgmap v9093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:08 smithi144 ceph-mon[32129]: pgmap v9092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:08 smithi144 ceph-mon[32129]: pgmap v9093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:37:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:11 smithi183 ceph-mon[37273]: pgmap v9094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16274.905721 secs 2024-07-26T15:37:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:11 smithi144 ceph-mon[32129]: pgmap v9094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16274.905721 secs 2024-07-26T15:37:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:11.501+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:37:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:11.502+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:37:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:11.502+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:37:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:11.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:37:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:11.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:11 smithi160 ceph-mon[36715]: pgmap v9094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16274.905721 secs 2024-07-26T15:37:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:13 smithi183 ceph-mon[37273]: pgmap v9095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:13 smithi144 ceph-mon[32129]: pgmap v9095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:13 smithi160 ceph-mon[36715]: pgmap v9095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:15 smithi144 ceph-mon[32129]: pgmap v9096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:37:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:15 smithi160 ceph-mon[36715]: pgmap v9096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:37:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:15 smithi183 ceph-mon[37273]: pgmap v9096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:37:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16279.905839 secs 2024-07-26T15:37:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16279.905839 secs 2024-07-26T15:37:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16279.905839 secs 2024-07-26T15:37:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:17 smithi144 ceph-mon[32129]: pgmap v9097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:17 smithi160 ceph-mon[36715]: pgmap v9097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:17 smithi183 ceph-mon[37273]: pgmap v9097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:37:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:19 smithi144 ceph-mon[32129]: pgmap v9098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:37:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:19 smithi160 ceph-mon[36715]: pgmap v9098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:37:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:19 smithi183 ceph-mon[37273]: pgmap v9098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:37:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:21 smithi144 ceph-mon[32129]: pgmap v9099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16284.905959 secs 2024-07-26T15:37:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:21 smithi160 ceph-mon[36715]: pgmap v9099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16284.905959 secs 2024-07-26T15:37:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:21 smithi183 ceph-mon[37273]: pgmap v9099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16284.905959 secs 2024-07-26T15:37:23.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:23 smithi183 ceph-mon[37273]: pgmap v9100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:23 smithi144 ceph-mon[32129]: pgmap v9100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:23 smithi160 ceph-mon[36715]: pgmap v9100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:25.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:25 smithi144 ceph-mon[32129]: pgmap v9101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:25 smithi160 ceph-mon[36715]: pgmap v9101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:25 smithi183 ceph-mon[37273]: pgmap v9101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:26.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16289.906052 secs 2024-07-26T15:37:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16289.906052 secs 2024-07-26T15:37:26.519 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:26.517+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:37:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:26.517+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:37:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:26.518+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:37:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:26.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:37:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:26.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:26.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16289.906052 secs 2024-07-26T15:37:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:27 smithi144 ceph-mon[32129]: pgmap v9102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:27 smithi160 ceph-mon[36715]: pgmap v9102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:27 smithi183 ceph-mon[37273]: pgmap v9102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:29 smithi144 ceph-mon[32129]: pgmap v9103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:29 smithi160 ceph-mon[36715]: pgmap v9103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:29 smithi183 ceph-mon[37273]: pgmap v9103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:31 smithi144 ceph-mon[32129]: pgmap v9104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16294.906174 secs 2024-07-26T15:37:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:31 smithi160 ceph-mon[36715]: pgmap v9104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16294.906174 secs 2024-07-26T15:37:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:31 smithi183 ceph-mon[37273]: pgmap v9104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16294.906174 secs 2024-07-26T15:37:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:33 smithi144 ceph-mon[32129]: pgmap v9105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:33 smithi160 ceph-mon[36715]: pgmap v9105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:33 smithi183 ceph-mon[37273]: pgmap v9105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:35 smithi144 ceph-mon[32129]: pgmap v9106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:35 smithi160 ceph-mon[36715]: pgmap v9106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:35.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:35 smithi183 ceph-mon[37273]: pgmap v9106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16299.906287 secs 2024-07-26T15:37:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16299.906287 secs 2024-07-26T15:37:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16299.906287 secs 2024-07-26T15:37:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:37 smithi160 ceph-mon[36715]: pgmap v9107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:37.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:37 smithi183 ceph-mon[37273]: pgmap v9107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:37 smithi144 ceph-mon[32129]: pgmap v9107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:39 smithi160 ceph-mon[36715]: pgmap v9108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:39 smithi183 ceph-mon[37273]: pgmap v9108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:39 smithi144 ceph-mon[32129]: pgmap v9108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:41.526+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:37:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:41.527+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:37:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:41.527+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:37:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:41.527+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:37:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:41.527+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:41 smithi160 ceph-mon[36715]: pgmap v9109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16304.906397 secs 2024-07-26T15:37:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:41 smithi183 ceph-mon[37273]: pgmap v9109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16304.906397 secs 2024-07-26T15:37:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:41 smithi144 ceph-mon[32129]: pgmap v9109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16304.906397 secs 2024-07-26T15:37:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:43 smithi144 ceph-mon[32129]: pgmap v9110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:43 smithi160 ceph-mon[36715]: pgmap v9110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:43.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:43 smithi183 ceph-mon[37273]: pgmap v9110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:45 smithi160 ceph-mon[36715]: pgmap v9111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:45 smithi183 ceph-mon[37273]: pgmap v9111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:45 smithi144 ceph-mon[32129]: pgmap v9111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16309.906508 secs 2024-07-26T15:37:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16309.906508 secs 2024-07-26T15:37:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16309.906508 secs 2024-07-26T15:37:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:47 smithi160 ceph-mon[36715]: pgmap v9112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:47 smithi183 ceph-mon[37273]: pgmap v9112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:47.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:47 smithi144 ceph-mon[32129]: pgmap v9112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:49 smithi160 ceph-mon[36715]: pgmap v9113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:49 smithi183 ceph-mon[37273]: pgmap v9113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:49.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:49 smithi144 ceph-mon[32129]: pgmap v9113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:51 smithi160 ceph-mon[36715]: pgmap v9114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16314.906570 secs 2024-07-26T15:37:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:51 smithi183 ceph-mon[37273]: pgmap v9114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16314.906570 secs 2024-07-26T15:37:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:51 smithi144 ceph-mon[32129]: pgmap v9114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16314.906570 secs 2024-07-26T15:37:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:53 smithi160 ceph-mon[36715]: pgmap v9115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:53 smithi183 ceph-mon[37273]: pgmap v9115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:53 smithi144 ceph-mon[32129]: pgmap v9115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:37:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:55 smithi160 ceph-mon[36715]: pgmap v9116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:55.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:55 smithi183 ceph-mon[37273]: pgmap v9116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:55 smithi144 ceph-mon[32129]: pgmap v9116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:56.543+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:37:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:56.544+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:37:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:56.544+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:37:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:56.544+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:37:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:37:56.544+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:56.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16319.906691 secs 2024-07-26T15:37:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16319.906691 secs 2024-07-26T15:37:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16319.906691 secs 2024-07-26T15:37:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:57 smithi160 ceph-mon[36715]: pgmap v9117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:57 smithi183 ceph-mon[37273]: pgmap v9117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:57 smithi144 ceph-mon[32129]: pgmap v9117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:37:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:37:59 smithi183 ceph-mon[37273]: pgmap v9118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:37:59 smithi144 ceph-mon[32129]: pgmap v9118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:37:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:37:59 smithi160 ceph-mon[36715]: pgmap v9118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:01 smithi183 ceph-mon[37273]: pgmap v9119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16324.906812 secs 2024-07-26T15:38:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:01 smithi144 ceph-mon[32129]: pgmap v9119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16324.906812 secs 2024-07-26T15:38:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:01 smithi160 ceph-mon[36715]: pgmap v9119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16324.906812 secs 2024-07-26T15:38:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:03 smithi183 ceph-mon[37273]: pgmap v9120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:03 smithi144 ceph-mon[32129]: pgmap v9120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:03 smithi160 ceph-mon[36715]: pgmap v9120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:03.982 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:03.980+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:38:03.982 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:03.980+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:38:03.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:03.981+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:38:03.983 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:03.981+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:38:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:05 smithi183 ceph-mon[37273]: pgmap v9121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:05 smithi144 ceph-mon[32129]: pgmap v9121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:05 smithi160 ceph-mon[36715]: pgmap v9121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16329.906949 secs 2024-07-26T15:38:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16329.906949 secs 2024-07-26T15:38:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16329.906949 secs 2024-07-26T15:38:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:07 smithi183 ceph-mon[37273]: pgmap v9122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:07 smithi144 ceph-mon[32129]: pgmap v9122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:07 smithi160 ceph-mon[36715]: pgmap v9122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:08.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:08 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:08 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:08.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:08 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:09 smithi144 ceph-mon[32129]: pgmap v9123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:09 smithi160 ceph-mon[36715]: pgmap v9123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:09 smithi183 ceph-mon[37273]: pgmap v9123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:11.559+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:38:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:11.560+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:38:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:11.560+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:38:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:11.560+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:38:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:11.560+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:11 smithi183 ceph-mon[37273]: pgmap v9124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16334.907040 secs 2024-07-26T15:38:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:11 smithi144 ceph-mon[32129]: pgmap v9124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16334.907040 secs 2024-07-26T15:38:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:11 smithi160 ceph-mon[36715]: pgmap v9124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16334.907040 secs 2024-07-26T15:38:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:13 smithi160 ceph-mon[36715]: pgmap v9125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:13 smithi183 ceph-mon[37273]: pgmap v9125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:13 smithi144 ceph-mon[32129]: pgmap v9125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:14 smithi160 ceph-mon[36715]: pgmap v9126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:38:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:14 smithi160 ceph-mon[36715]: pgmap v9127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:38:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:14 smithi183 ceph-mon[37273]: pgmap v9126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:38:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:14 smithi183 ceph-mon[37273]: pgmap v9127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:38:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:14 smithi144 ceph-mon[32129]: pgmap v9126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:38:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:14 smithi144 ceph-mon[32129]: pgmap v9127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:38:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:38:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:16 smithi183 ceph-mon[37273]: pgmap v9128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:38:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16339.907157 secs 2024-07-26T15:38:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:16 smithi144 ceph-mon[32129]: pgmap v9128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:38:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16339.907157 secs 2024-07-26T15:38:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:16 smithi160 ceph-mon[36715]: pgmap v9128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:38:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16339.907157 secs 2024-07-26T15:38:18.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:18 smithi183 ceph-mon[37273]: pgmap v9129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:18 smithi144 ceph-mon[32129]: pgmap v9129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:18 smithi160 ceph-mon[36715]: pgmap v9129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:20 smithi183 ceph-mon[37273]: pgmap v9130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:20 smithi144 ceph-mon[32129]: pgmap v9130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:20 smithi160 ceph-mon[36715]: pgmap v9130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16344.907318 secs 2024-07-26T15:38:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16344.907318 secs 2024-07-26T15:38:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16344.907318 secs 2024-07-26T15:38:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:22 smithi183 ceph-mon[37273]: pgmap v9131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:22 smithi144 ceph-mon[32129]: pgmap v9131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:22 smithi160 ceph-mon[36715]: pgmap v9131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:38:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:24 smithi183 ceph-mon[37273]: pgmap v9132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:38:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:24 smithi144 ceph-mon[32129]: pgmap v9132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:38:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:24 smithi160 ceph-mon[36715]: pgmap v9132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:38:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:26.570+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:38:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:26.571+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:38:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:26.571+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:38:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:26.571+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:38:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:26.571+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:26 smithi183 ceph-mon[37273]: pgmap v9133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16349.907377 secs 2024-07-26T15:38:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:26 smithi144 ceph-mon[32129]: pgmap v9133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16349.907377 secs 2024-07-26T15:38:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:26 smithi160 ceph-mon[36715]: pgmap v9133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16349.907377 secs 2024-07-26T15:38:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:28 smithi183 ceph-mon[37273]: pgmap v9134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:28 smithi144 ceph-mon[32129]: pgmap v9134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:28 smithi160 ceph-mon[36715]: pgmap v9134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:30 smithi144 ceph-mon[32129]: pgmap v9135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:30 smithi160 ceph-mon[36715]: pgmap v9135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:30 smithi183 ceph-mon[37273]: pgmap v9135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16354.907494 secs 2024-07-26T15:38:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16354.907494 secs 2024-07-26T15:38:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16354.907494 secs 2024-07-26T15:38:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:32 smithi144 ceph-mon[32129]: pgmap v9136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:32 smithi160 ceph-mon[36715]: pgmap v9136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:32 smithi183 ceph-mon[37273]: pgmap v9136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:34 smithi144 ceph-mon[32129]: pgmap v9137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:34 smithi160 ceph-mon[36715]: pgmap v9137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:34 smithi183 ceph-mon[37273]: pgmap v9137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:36 smithi144 ceph-mon[32129]: pgmap v9138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16359.907644 secs 2024-07-26T15:38:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:36 smithi160 ceph-mon[36715]: pgmap v9138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16359.907644 secs 2024-07-26T15:38:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:36 smithi183 ceph-mon[37273]: pgmap v9138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16359.907644 secs 2024-07-26T15:38:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:38 smithi144 ceph-mon[32129]: pgmap v9139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:38 smithi160 ceph-mon[36715]: pgmap v9139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:38 smithi183 ceph-mon[37273]: pgmap v9139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:40 smithi144 ceph-mon[32129]: pgmap v9140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:40.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:40 smithi160 ceph-mon[36715]: pgmap v9140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:40 smithi183 ceph-mon[37273]: pgmap v9140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:41.588+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:38:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:41.589+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:38:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:41.589+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:38:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:41.589+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:38:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:41.589+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16364.907766 secs 2024-07-26T15:38:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16364.907766 secs 2024-07-26T15:38:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16364.907766 secs 2024-07-26T15:38:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:42 smithi144 ceph-mon[32129]: pgmap v9141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:42 smithi160 ceph-mon[36715]: pgmap v9141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:42 smithi183 ceph-mon[37273]: pgmap v9141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:44 smithi144 ceph-mon[32129]: pgmap v9142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:44 smithi160 ceph-mon[36715]: pgmap v9142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:44 smithi183 ceph-mon[37273]: pgmap v9142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:46 smithi144 ceph-mon[32129]: pgmap v9143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16369.907882 secs 2024-07-26T15:38:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:46 smithi160 ceph-mon[36715]: pgmap v9143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16369.907882 secs 2024-07-26T15:38:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:46 smithi183 ceph-mon[37273]: pgmap v9143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16369.907882 secs 2024-07-26T15:38:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:48 smithi144 ceph-mon[32129]: pgmap v9144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:48 smithi160 ceph-mon[36715]: pgmap v9144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:48 smithi183 ceph-mon[37273]: pgmap v9144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:48.987+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:38:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:48.988+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:38:48.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:48.988+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:38:48.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:48.988+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:38:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:50 smithi144 ceph-mon[32129]: pgmap v9145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:50 smithi160 ceph-mon[36715]: pgmap v9145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:50 smithi183 ceph-mon[37273]: pgmap v9145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16374.908027 secs 2024-07-26T15:38:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16374.908027 secs 2024-07-26T15:38:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16374.908027 secs 2024-07-26T15:38:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:52 smithi160 ceph-mon[36715]: pgmap v9146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:52 smithi183 ceph-mon[37273]: pgmap v9146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:52 smithi144 ceph-mon[32129]: pgmap v9146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:54 smithi160 ceph-mon[36715]: pgmap v9147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:54 smithi183 ceph-mon[37273]: pgmap v9147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:54 smithi144 ceph-mon[32129]: pgmap v9147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:38:56.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:56.597+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:38:56.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:56.598+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:38:56.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:56.598+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:38:56.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:56.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:38:56.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:38:56.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:56 smithi160 ceph-mon[36715]: pgmap v9148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16379.908126 secs 2024-07-26T15:38:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:56 smithi183 ceph-mon[37273]: pgmap v9148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16379.908126 secs 2024-07-26T15:38:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:56 smithi144 ceph-mon[32129]: pgmap v9148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:38:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16379.908126 secs 2024-07-26T15:38:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:38:58 smithi160 ceph-mon[36715]: pgmap v9149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:38:58 smithi183 ceph-mon[37273]: pgmap v9149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:38:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:38:58 smithi144 ceph-mon[32129]: pgmap v9149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:00 smithi160 ceph-mon[36715]: pgmap v9150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:00 smithi183 ceph-mon[37273]: pgmap v9150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:00.894 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:00 smithi144 ceph-mon[32129]: pgmap v9150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16384.908249 secs 2024-07-26T15:39:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16384.908249 secs 2024-07-26T15:39:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16384.908249 secs 2024-07-26T15:39:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:02 smithi160 ceph-mon[36715]: pgmap v9151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:02 smithi183 ceph-mon[37273]: pgmap v9151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:02 smithi144 ceph-mon[32129]: pgmap v9151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:04.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:04 smithi144 ceph-mon[32129]: pgmap v9152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:04 smithi160 ceph-mon[36715]: pgmap v9152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:04 smithi183 ceph-mon[37273]: pgmap v9152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:06 smithi160 ceph-mon[36715]: pgmap v9153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16389.908380 secs 2024-07-26T15:39:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:06 smithi183 ceph-mon[37273]: pgmap v9153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16389.908380 secs 2024-07-26T15:39:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:06 smithi144 ceph-mon[32129]: pgmap v9153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16389.908380 secs 2024-07-26T15:39:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:08 smithi160 ceph-mon[36715]: pgmap v9154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:08 smithi183 ceph-mon[37273]: pgmap v9154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:08 smithi144 ceph-mon[32129]: pgmap v9154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:10 smithi160 ceph-mon[36715]: pgmap v9155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:39:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:10 smithi183 ceph-mon[37273]: pgmap v9155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:39:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:10 smithi144 ceph-mon[32129]: pgmap v9155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:39:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:11.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:39:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:11.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:39:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:11.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:39:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:11.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:39:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:11.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16394.908535 secs 2024-07-26T15:39:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16394.908535 secs 2024-07-26T15:39:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16394.908535 secs 2024-07-26T15:39:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:12 smithi160 ceph-mon[36715]: pgmap v9156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:12 smithi183 ceph-mon[37273]: pgmap v9156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:12 smithi144 ceph-mon[32129]: pgmap v9156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:13.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:14 smithi183 ceph-mon[37273]: pgmap v9157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:39:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:14 smithi144 ceph-mon[32129]: pgmap v9157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:39:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:14 smithi160 ceph-mon[36715]: pgmap v9157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:39:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:15 smithi183 ceph-mon[37273]: pgmap v9158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:15 smithi183 ceph-mon[37273]: pgmap v9159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:15 smithi144 ceph-mon[32129]: pgmap v9158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:15 smithi144 ceph-mon[32129]: pgmap v9159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:15 smithi160 ceph-mon[36715]: pgmap v9158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:15 smithi160 ceph-mon[36715]: pgmap v9159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16399.908611 secs 2024-07-26T15:39:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16399.908611 secs 2024-07-26T15:39:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16399.908611 secs 2024-07-26T15:39:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:17 smithi183 ceph-mon[37273]: pgmap v9160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:17 smithi144 ceph-mon[32129]: pgmap v9160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:17 smithi160 ceph-mon[36715]: pgmap v9160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:19.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:19 smithi183 ceph-mon[37273]: pgmap v9161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:39:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:19 smithi144 ceph-mon[32129]: pgmap v9161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:39:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:19 smithi160 ceph-mon[36715]: pgmap v9161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:39:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:21 smithi183 ceph-mon[37273]: pgmap v9162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:39:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16404.908732 secs 2024-07-26T15:39:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:21 smithi144 ceph-mon[32129]: pgmap v9162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:39:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16404.908732 secs 2024-07-26T15:39:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:21 smithi160 ceph-mon[36715]: pgmap v9162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:39:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16404.908732 secs 2024-07-26T15:39:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:23 smithi183 ceph-mon[37273]: pgmap v9163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:39:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:23 smithi144 ceph-mon[32129]: pgmap v9163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:39:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:23 smithi160 ceph-mon[36715]: pgmap v9163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:39:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:25 smithi183 ceph-mon[37273]: pgmap v9164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:25 smithi144 ceph-mon[32129]: pgmap v9164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:25 smithi160 ceph-mon[36715]: pgmap v9164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:39:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:26.607+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:39:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:26.608+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:39:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:26.608+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:39:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:26.608+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:39:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:26.608+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16409.908858 secs 2024-07-26T15:39:26.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16409.908858 secs 2024-07-26T15:39:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16409.908858 secs 2024-07-26T15:39:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:27 smithi183 ceph-mon[37273]: pgmap v9165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:27 smithi144 ceph-mon[32129]: pgmap v9165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:27 smithi160 ceph-mon[36715]: pgmap v9165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:29.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:29 smithi183 ceph-mon[37273]: pgmap v9166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:29 smithi144 ceph-mon[32129]: pgmap v9166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:29 smithi160 ceph-mon[36715]: pgmap v9166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:31 smithi183 ceph-mon[37273]: pgmap v9167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16414.908958 secs 2024-07-26T15:39:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:31 smithi144 ceph-mon[32129]: pgmap v9167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16414.908958 secs 2024-07-26T15:39:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:31 smithi160 ceph-mon[36715]: pgmap v9167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16414.908958 secs 2024-07-26T15:39:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:33 smithi183 ceph-mon[37273]: pgmap v9168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:33 smithi144 ceph-mon[32129]: pgmap v9168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:33 smithi160 ceph-mon[36715]: pgmap v9168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:35 smithi183 ceph-mon[37273]: pgmap v9169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:35 smithi144 ceph-mon[32129]: pgmap v9169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:35 smithi160 ceph-mon[36715]: pgmap v9169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16419.909078 secs 2024-07-26T15:39:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16419.909078 secs 2024-07-26T15:39:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16419.909078 secs 2024-07-26T15:39:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:37 smithi183 ceph-mon[37273]: pgmap v9170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:37 smithi144 ceph-mon[32129]: pgmap v9170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:37 smithi160 ceph-mon[36715]: pgmap v9170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:39 smithi183 ceph-mon[37273]: pgmap v9171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:39 smithi144 ceph-mon[32129]: pgmap v9171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:39 smithi160 ceph-mon[36715]: pgmap v9171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:41.624+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:39:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:41.625+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:39:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:41.625+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:39:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:41.625+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:39:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:41.625+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:41 smithi183 ceph-mon[37273]: pgmap v9172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16424.909200 secs 2024-07-26T15:39:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:41 smithi144 ceph-mon[32129]: pgmap v9172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16424.909200 secs 2024-07-26T15:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:41 smithi160 ceph-mon[36715]: pgmap v9172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16424.909200 secs 2024-07-26T15:39:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:43 smithi183 ceph-mon[37273]: pgmap v9173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:43 smithi144 ceph-mon[32129]: pgmap v9173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:43 smithi160 ceph-mon[36715]: pgmap v9173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:45 smithi183 ceph-mon[37273]: pgmap v9174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:45 smithi144 ceph-mon[32129]: pgmap v9174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:45 smithi160 ceph-mon[36715]: pgmap v9174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16429.909352 secs 2024-07-26T15:39:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16429.909352 secs 2024-07-26T15:39:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16429.909352 secs 2024-07-26T15:39:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:47 smithi144 ceph-mon[32129]: pgmap v9175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:47 smithi160 ceph-mon[36715]: pgmap v9175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:47 smithi183 ceph-mon[37273]: pgmap v9175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:49 smithi144 ceph-mon[32129]: pgmap v9176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:49 smithi160 ceph-mon[36715]: pgmap v9176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:49 smithi183 ceph-mon[37273]: pgmap v9176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:51 smithi144 ceph-mon[32129]: pgmap v9177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16434.909476 secs 2024-07-26T15:39:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:51 smithi160 ceph-mon[36715]: pgmap v9177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16434.909476 secs 2024-07-26T15:39:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:51 smithi183 ceph-mon[37273]: pgmap v9177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16434.909476 secs 2024-07-26T15:39:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:53 smithi144 ceph-mon[32129]: pgmap v9178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:53 smithi160 ceph-mon[36715]: pgmap v9178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:53 smithi183 ceph-mon[37273]: pgmap v9178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:39:55.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:55 smithi183 ceph-mon[37273]: pgmap v9179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:55 smithi144 ceph-mon[32129]: pgmap v9179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:55 smithi160 ceph-mon[36715]: pgmap v9179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:39:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:56.629+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:39:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:56.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:39:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:56.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:39:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:56.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:39:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:39:56.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:56.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16439.909612 secs 2024-07-26T15:39:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16439.909612 secs 2024-07-26T15:39:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16439.909612 secs 2024-07-26T15:39:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:57 smithi144 ceph-mon[32129]: pgmap v9180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:57 smithi160 ceph-mon[36715]: pgmap v9180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:57 smithi183 ceph-mon[37273]: pgmap v9180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:39:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:39:59 smithi144 ceph-mon[32129]: pgmap v9181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:39:59 smithi160 ceph-mon[36715]: pgmap v9181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:39:59 smithi183 ceph-mon[37273]: pgmap v9181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:40:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:40:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:40:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:01 smithi144 ceph-mon[32129]: pgmap v9182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16444.909716 secs 2024-07-26T15:40:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:01 smithi160 ceph-mon[36715]: pgmap v9182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16444.909716 secs 2024-07-26T15:40:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:01 smithi183 ceph-mon[37273]: pgmap v9182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16444.909716 secs 2024-07-26T15:40:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:02 smithi144 ceph-mon[32129]: pgmap v9183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:02 smithi160 ceph-mon[36715]: pgmap v9183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:02 smithi183 ceph-mon[37273]: pgmap v9183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:05.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:05 smithi183 ceph-mon[37273]: pgmap v9184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:05 smithi144 ceph-mon[32129]: pgmap v9184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:05 smithi160 ceph-mon[36715]: pgmap v9184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16449.909829 secs 2024-07-26T15:40:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16449.909829 secs 2024-07-26T15:40:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16449.909829 secs 2024-07-26T15:40:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:07 smithi183 ceph-mon[37273]: pgmap v9185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:07 smithi144 ceph-mon[32129]: pgmap v9185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:07 smithi160 ceph-mon[36715]: pgmap v9185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:09.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:09 smithi183 ceph-mon[37273]: pgmap v9186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:09 smithi144 ceph-mon[32129]: pgmap v9186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:09 smithi160 ceph-mon[36715]: pgmap v9186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:11.646+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:40:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:11.647+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:40:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:11.647+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:40:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:11.647+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:40:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:11.647+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:11 smithi183 ceph-mon[37273]: pgmap v9187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16454.909955 secs 2024-07-26T15:40:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:11 smithi144 ceph-mon[32129]: pgmap v9187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16454.909955 secs 2024-07-26T15:40:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:11 smithi160 ceph-mon[36715]: pgmap v9187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16454.909955 secs 2024-07-26T15:40:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:13 smithi144 ceph-mon[32129]: pgmap v9188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:13 smithi160 ceph-mon[36715]: pgmap v9188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:13 smithi183 ceph-mon[37273]: pgmap v9188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:15.894 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:15 smithi183 ceph-mon[37273]: pgmap v9189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:15.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:15 smithi144 ceph-mon[32129]: pgmap v9189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:15 smithi160 ceph-mon[36715]: pgmap v9189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:40:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16459.910108 secs 2024-07-26T15:40:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:40:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16459.910108 secs 2024-07-26T15:40:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:40:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16459.910108 secs 2024-07-26T15:40:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:17 smithi144 ceph-mon[32129]: pgmap v9190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:17 smithi144 ceph-mon[32129]: pgmap v9191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:17 smithi160 ceph-mon[36715]: pgmap v9190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:17 smithi160 ceph-mon[36715]: pgmap v9191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:17 smithi183 ceph-mon[37273]: pgmap v9190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:17 smithi183 ceph-mon[37273]: pgmap v9191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:19 smithi144 ceph-mon[32129]: pgmap v9192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:40:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:19 smithi160 ceph-mon[36715]: pgmap v9192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:40:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:19 smithi183 ceph-mon[37273]: pgmap v9192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:40:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:21 smithi144 ceph-mon[32129]: pgmap v9193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:40:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16464.910230 secs 2024-07-26T15:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:21 smithi160 ceph-mon[36715]: pgmap v9193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:40:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16464.910230 secs 2024-07-26T15:40:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:21 smithi183 ceph-mon[37273]: pgmap v9193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:40:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16464.910230 secs 2024-07-26T15:40:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:23 smithi144 ceph-mon[32129]: pgmap v9194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:23 smithi160 ceph-mon[36715]: pgmap v9194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:23 smithi183 ceph-mon[37273]: pgmap v9194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:40:25.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:25 smithi183 ceph-mon[37273]: pgmap v9195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:40:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:25 smithi144 ceph-mon[32129]: pgmap v9195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:40:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:25 smithi160 ceph-mon[36715]: pgmap v9195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:40:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:26.662+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:40:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:26.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:40:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:26.663+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:40:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:26.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:40:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:26.663+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:26.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16469.910354 secs 2024-07-26T15:40:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16469.910354 secs 2024-07-26T15:40:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16469.910354 secs 2024-07-26T15:40:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:27 smithi144 ceph-mon[32129]: pgmap v9196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:40:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:27 smithi160 ceph-mon[36715]: pgmap v9196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:40:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:27 smithi183 ceph-mon[37273]: pgmap v9196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:40:29.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:29 smithi144 ceph-mon[32129]: pgmap v9197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:29 smithi160 ceph-mon[36715]: pgmap v9197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:29 smithi183 ceph-mon[37273]: pgmap v9197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:30 smithi144 ceph-mon[32129]: pgmap v9198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:30 smithi160 ceph-mon[36715]: pgmap v9198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:30 smithi183 ceph-mon[37273]: pgmap v9198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16474.910454 secs 2024-07-26T15:40:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16474.910454 secs 2024-07-26T15:40:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16474.910454 secs 2024-07-26T15:40:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:32 smithi160 ceph-mon[36715]: pgmap v9199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:32 smithi183 ceph-mon[37273]: pgmap v9199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:32 smithi144 ceph-mon[32129]: pgmap v9199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:34 smithi183 ceph-mon[37273]: pgmap v9200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:34 smithi144 ceph-mon[32129]: pgmap v9200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:34 smithi160 ceph-mon[36715]: pgmap v9200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:36 smithi183 ceph-mon[37273]: pgmap v9201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16479.910571 secs 2024-07-26T15:40:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:36 smithi144 ceph-mon[32129]: pgmap v9201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16479.910571 secs 2024-07-26T15:40:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:36 smithi160 ceph-mon[36715]: pgmap v9201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16479.910571 secs 2024-07-26T15:40:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:38 smithi183 ceph-mon[37273]: pgmap v9202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:38 smithi144 ceph-mon[32129]: pgmap v9202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:38 smithi160 ceph-mon[36715]: pgmap v9202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:40 smithi183 ceph-mon[37273]: pgmap v9203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:40 smithi144 ceph-mon[32129]: pgmap v9203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:40 smithi160 ceph-mon[36715]: pgmap v9203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:41.676 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:41.674+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:40:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:41.675+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:40:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:41.675+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:40:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:41.675+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:40:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:41.675+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16484.910693 secs 2024-07-26T15:40:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16484.910693 secs 2024-07-26T15:40:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16484.910693 secs 2024-07-26T15:40:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:42 smithi183 ceph-mon[37273]: pgmap v9204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:42 smithi144 ceph-mon[32129]: pgmap v9204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:42 smithi160 ceph-mon[36715]: pgmap v9204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:44 smithi183 ceph-mon[37273]: pgmap v9205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:44 smithi144 ceph-mon[32129]: pgmap v9205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:44 smithi160 ceph-mon[36715]: pgmap v9205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:46 smithi183 ceph-mon[37273]: pgmap v9206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16489.910812 secs 2024-07-26T15:40:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:46 smithi144 ceph-mon[32129]: pgmap v9206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16489.910812 secs 2024-07-26T15:40:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:46 smithi160 ceph-mon[36715]: pgmap v9206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16489.910812 secs 2024-07-26T15:40:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:48 smithi183 ceph-mon[37273]: pgmap v9207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:48 smithi144 ceph-mon[32129]: pgmap v9207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:48 smithi160 ceph-mon[36715]: pgmap v9207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:50 smithi183 ceph-mon[37273]: pgmap v9208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:50 smithi144 ceph-mon[32129]: pgmap v9208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:50 smithi160 ceph-mon[36715]: pgmap v9208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16494.910933 secs 2024-07-26T15:40:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16494.910933 secs 2024-07-26T15:40:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16494.910933 secs 2024-07-26T15:40:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:52 smithi144 ceph-mon[32129]: pgmap v9209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:52 smithi160 ceph-mon[36715]: pgmap v9209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:52 smithi183 ceph-mon[37273]: pgmap v9209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:40:55.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:54 smithi183 ceph-mon[37273]: pgmap v9210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:54 smithi144 ceph-mon[32129]: pgmap v9210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:54 smithi160 ceph-mon[36715]: pgmap v9210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:56.679+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:40:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:56.681+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:40:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:56.681+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:40:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:56.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:40:56.682 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:40:56.681+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:56 smithi144 ceph-mon[32129]: pgmap v9211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16499.911048 secs 2024-07-26T15:40:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:56 smithi160 ceph-mon[36715]: pgmap v9211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16499.911048 secs 2024-07-26T15:40:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:56 smithi183 ceph-mon[37273]: pgmap v9211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:40:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16499.911048 secs 2024-07-26T15:40:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:40:58 smithi144 ceph-mon[32129]: pgmap v9212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:40:58 smithi160 ceph-mon[36715]: pgmap v9212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:40:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:40:58 smithi183 ceph-mon[37273]: pgmap v9212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:00 smithi144 ceph-mon[32129]: pgmap v9213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:00 smithi160 ceph-mon[36715]: pgmap v9213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:00 smithi183 ceph-mon[37273]: pgmap v9213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16504.911204 secs 2024-07-26T15:41:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16504.911204 secs 2024-07-26T15:41:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16504.911204 secs 2024-07-26T15:41:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:02 smithi144 ceph-mon[32129]: pgmap v9214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:02 smithi160 ceph-mon[36715]: pgmap v9214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:02 smithi183 ceph-mon[37273]: pgmap v9214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:04 smithi144 ceph-mon[32129]: pgmap v9215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:04 smithi160 ceph-mon[36715]: pgmap v9215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:04 smithi183 ceph-mon[37273]: pgmap v9215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:06 smithi144 ceph-mon[32129]: pgmap v9216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16509.911367 secs 2024-07-26T15:41:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:06 smithi160 ceph-mon[36715]: pgmap v9216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16509.911367 secs 2024-07-26T15:41:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:06 smithi183 ceph-mon[37273]: pgmap v9216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16509.911367 secs 2024-07-26T15:41:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:08 smithi144 ceph-mon[32129]: pgmap v9217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:08 smithi160 ceph-mon[36715]: pgmap v9217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:08 smithi183 ceph-mon[37273]: pgmap v9217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:10 smithi144 ceph-mon[32129]: pgmap v9218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:10 smithi160 ceph-mon[36715]: pgmap v9218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:10 smithi183 ceph-mon[37273]: pgmap v9218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:11.697+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:41:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:11.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:41:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:11.698+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:41:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:11.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:41:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:11.698+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16514.911483 secs 2024-07-26T15:41:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16514.911483 secs 2024-07-26T15:41:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16514.911483 secs 2024-07-26T15:41:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:12 smithi144 ceph-mon[32129]: pgmap v9219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:12 smithi160 ceph-mon[36715]: pgmap v9219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:12 smithi183 ceph-mon[37273]: pgmap v9219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:15 smithi183 ceph-mon[37273]: pgmap v9220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:15 smithi144 ceph-mon[32129]: pgmap v9220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:15 smithi160 ceph-mon[36715]: pgmap v9220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:17 smithi183 ceph-mon[37273]: pgmap v9221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16519.911608 secs 2024-07-26T15:41:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:41:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:17 smithi144 ceph-mon[32129]: pgmap v9221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16519.911608 secs 2024-07-26T15:41:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:41:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:17 smithi160 ceph-mon[36715]: pgmap v9221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16519.911608 secs 2024-07-26T15:41:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:41:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:18 smithi183 ceph-mon[37273]: pgmap v9222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:41:18.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:18 smithi183 ceph-mon[37273]: pgmap v9223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:41:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:18 smithi144 ceph-mon[32129]: pgmap v9222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:41:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:18 smithi144 ceph-mon[32129]: pgmap v9223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:41:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:18 smithi160 ceph-mon[36715]: pgmap v9222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:41:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:18 smithi160 ceph-mon[36715]: pgmap v9223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:41:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:20 smithi183 ceph-mon[37273]: pgmap v9224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:41:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:20 smithi144 ceph-mon[32129]: pgmap v9224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:41:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:20 smithi160 ceph-mon[36715]: pgmap v9224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:41:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16524.911688 secs 2024-07-26T15:41:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16524.911688 secs 2024-07-26T15:41:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16524.911688 secs 2024-07-26T15:41:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:22 smithi144 ceph-mon[32129]: pgmap v9225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:41:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:22 smithi160 ceph-mon[36715]: pgmap v9225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:41:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:22 smithi183 ceph-mon[37273]: pgmap v9225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:41:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:24 smithi144 ceph-mon[32129]: pgmap v9226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:41:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:24 smithi160 ceph-mon[36715]: pgmap v9226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:41:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:24 smithi183 ceph-mon[37273]: pgmap v9226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:41:26.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:26 smithi183 ceph-mon[37273]: pgmap v9227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:41:26.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16529.911783 secs 2024-07-26T15:41:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:26 smithi144 ceph-mon[32129]: pgmap v9227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:41:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16529.911783 secs 2024-07-26T15:41:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:26 smithi160 ceph-mon[36715]: pgmap v9227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:41:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16529.911783 secs 2024-07-26T15:41:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:26.706+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:41:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:26.707+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:41:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:26.707+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:41:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:26.707+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:41:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:26.707+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:28 smithi144 ceph-mon[32129]: pgmap v9228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:41:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:28 smithi160 ceph-mon[36715]: pgmap v9228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:41:28.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:28 smithi183 ceph-mon[37273]: pgmap v9228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:41:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:30 smithi144 ceph-mon[32129]: pgmap v9229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:30 smithi160 ceph-mon[36715]: pgmap v9229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:30 smithi183 ceph-mon[37273]: pgmap v9229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16534.911911 secs 2024-07-26T15:41:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16534.911911 secs 2024-07-26T15:41:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16534.911911 secs 2024-07-26T15:41:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:32 smithi144 ceph-mon[32129]: pgmap v9230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:32 smithi160 ceph-mon[36715]: pgmap v9230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:32.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:32 smithi183 ceph-mon[37273]: pgmap v9230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:34 smithi144 ceph-mon[32129]: pgmap v9231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:34 smithi160 ceph-mon[36715]: pgmap v9231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:34 smithi183 ceph-mon[37273]: pgmap v9231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:36 smithi144 ceph-mon[32129]: pgmap v9232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16539.911993 secs 2024-07-26T15:41:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:36 smithi160 ceph-mon[36715]: pgmap v9232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16539.911993 secs 2024-07-26T15:41:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:36 smithi183 ceph-mon[37273]: pgmap v9232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16539.911993 secs 2024-07-26T15:41:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:38 smithi144 ceph-mon[32129]: pgmap v9233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:38 smithi160 ceph-mon[36715]: pgmap v9233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:38.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:38 smithi183 ceph-mon[37273]: pgmap v9233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:40 smithi144 ceph-mon[32129]: pgmap v9234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:40 smithi160 ceph-mon[36715]: pgmap v9234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:40.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:40 smithi183 ceph-mon[37273]: pgmap v9234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16544.912116 secs 2024-07-26T15:41:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16544.912116 secs 2024-07-26T15:41:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16544.912116 secs 2024-07-26T15:41:41.726 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:41.725+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:41:41.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:41.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:41:41.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:41.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:41:41.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:41.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:41:41.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:41.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:42 smithi160 ceph-mon[36715]: pgmap v9235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:42 smithi183 ceph-mon[37273]: pgmap v9235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:42 smithi144 ceph-mon[32129]: pgmap v9235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:44 smithi160 ceph-mon[36715]: pgmap v9236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:44 smithi183 ceph-mon[37273]: pgmap v9236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:44 smithi144 ceph-mon[32129]: pgmap v9236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:46 smithi160 ceph-mon[36715]: pgmap v9237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16549.912215 secs 2024-07-26T15:41:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:46 smithi183 ceph-mon[37273]: pgmap v9237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16549.912215 secs 2024-07-26T15:41:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:46 smithi144 ceph-mon[32129]: pgmap v9237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16549.912215 secs 2024-07-26T15:41:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:48 smithi160 ceph-mon[36715]: pgmap v9238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:48.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:48 smithi183 ceph-mon[37273]: pgmap v9238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:48 smithi144 ceph-mon[32129]: pgmap v9238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:49.019+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:41:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:49.019+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:41:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:49.019+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:41:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:49.020+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:41:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:50 smithi160 ceph-mon[36715]: pgmap v9239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:50 smithi183 ceph-mon[37273]: pgmap v9239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:50 smithi144 ceph-mon[32129]: pgmap v9239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16554.912344 secs 2024-07-26T15:41:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16554.912344 secs 2024-07-26T15:41:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16554.912344 secs 2024-07-26T15:41:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:52 smithi160 ceph-mon[36715]: pgmap v9240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:52 smithi183 ceph-mon[37273]: pgmap v9240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:52 smithi144 ceph-mon[32129]: pgmap v9240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:54 smithi160 ceph-mon[36715]: pgmap v9241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:54 smithi183 ceph-mon[37273]: pgmap v9241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:54 smithi144 ceph-mon[32129]: pgmap v9241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:41:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:56 smithi160 ceph-mon[36715]: pgmap v9242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16559.912464 secs 2024-07-26T15:41:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:56 smithi183 ceph-mon[37273]: pgmap v9242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16559.912464 secs 2024-07-26T15:41:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:56 smithi144 ceph-mon[32129]: pgmap v9242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:41:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16559.912464 secs 2024-07-26T15:41:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:56.741+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:41:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:56.742+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:41:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:56.742+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:41:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:56.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:41:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:41:56.742+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:41:58 smithi160 ceph-mon[36715]: pgmap v9243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:41:58 smithi183 ceph-mon[37273]: pgmap v9243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:41:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:41:58 smithi144 ceph-mon[32129]: pgmap v9243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:00 smithi160 ceph-mon[36715]: pgmap v9244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:00.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:00 smithi183 ceph-mon[37273]: pgmap v9244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:00 smithi144 ceph-mon[32129]: pgmap v9244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16564.912581 secs 2024-07-26T15:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16564.912581 secs 2024-07-26T15:42:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16564.912581 secs 2024-07-26T15:42:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:02 smithi183 ceph-mon[37273]: pgmap v9245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:02 smithi144 ceph-mon[32129]: pgmap v9245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:02 smithi160 ceph-mon[36715]: pgmap v9245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:04.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:04 smithi183 ceph-mon[37273]: pgmap v9246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:04 smithi144 ceph-mon[32129]: pgmap v9246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:04 smithi160 ceph-mon[36715]: pgmap v9246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:06 smithi183 ceph-mon[37273]: pgmap v9247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16569.912709 secs 2024-07-26T15:42:06.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:06 smithi144 ceph-mon[32129]: pgmap v9247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:06.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16569.912709 secs 2024-07-26T15:42:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:06 smithi160 ceph-mon[36715]: pgmap v9247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16569.912709 secs 2024-07-26T15:42:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:08 smithi183 ceph-mon[37273]: pgmap v9248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:08 smithi144 ceph-mon[32129]: pgmap v9248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:08 smithi160 ceph-mon[36715]: pgmap v9248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:10 smithi183 ceph-mon[37273]: pgmap v9249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:10 smithi144 ceph-mon[32129]: pgmap v9249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:10 smithi160 ceph-mon[36715]: pgmap v9249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16574.912830 secs 2024-07-26T15:42:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16574.912830 secs 2024-07-26T15:42:11.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:11.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:42:11.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:11.756+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:42:11.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:11.756+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:42:11.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:11.756+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:42:11.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:11.756+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16574.912830 secs 2024-07-26T15:42:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:12 smithi183 ceph-mon[37273]: pgmap v9250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:12 smithi144 ceph-mon[32129]: pgmap v9250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:12 smithi160 ceph-mon[36715]: pgmap v9250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:14.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:14 smithi183 ceph-mon[37273]: pgmap v9251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:14 smithi144 ceph-mon[32129]: pgmap v9251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:14 smithi160 ceph-mon[36715]: pgmap v9251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:16 smithi183 ceph-mon[37273]: pgmap v9252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16579.912969 secs 2024-07-26T15:42:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:16 smithi144 ceph-mon[32129]: pgmap v9252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16579.912969 secs 2024-07-26T15:42:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:16 smithi160 ceph-mon[36715]: pgmap v9252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16579.912969 secs 2024-07-26T15:42:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:18 smithi183 ceph-mon[37273]: pgmap v9253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:18 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:42:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:18 smithi144 ceph-mon[32129]: pgmap v9253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:18 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:42:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:18 smithi160 ceph-mon[36715]: pgmap v9253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:42:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:18 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:42:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:19 smithi183 ceph-mon[37273]: pgmap v9254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:19 smithi183 ceph-mon[37273]: pgmap v9255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:19 smithi144 ceph-mon[32129]: pgmap v9254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:19 smithi144 ceph-mon[32129]: pgmap v9255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:19 smithi160 ceph-mon[36715]: pgmap v9254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:19 smithi160 ceph-mon[36715]: pgmap v9255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:21 smithi144 ceph-mon[32129]: pgmap v9256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16584.913081 secs 2024-07-26T15:42:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:21 smithi160 ceph-mon[36715]: pgmap v9256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16584.913081 secs 2024-07-26T15:42:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:21 smithi183 ceph-mon[37273]: pgmap v9256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16584.913081 secs 2024-07-26T15:42:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:23 smithi144 ceph-mon[32129]: pgmap v9257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:23 smithi160 ceph-mon[36715]: pgmap v9257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:23.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:23 smithi183 ceph-mon[37273]: pgmap v9257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:25 smithi144 ceph-mon[32129]: pgmap v9258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:42:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:25 smithi160 ceph-mon[36715]: pgmap v9258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:42:25.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:25 smithi183 ceph-mon[37273]: pgmap v9258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:42:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16589.913213 secs 2024-07-26T15:42:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:26.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:42:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:26.771+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:42:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:26.771+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:42:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:26.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:42:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:26.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:26.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16589.913213 secs 2024-07-26T15:42:26.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16589.913213 secs 2024-07-26T15:42:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:27 smithi144 ceph-mon[32129]: pgmap v9259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:27 smithi160 ceph-mon[36715]: pgmap v9259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:27 smithi183 ceph-mon[37273]: pgmap v9259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:42:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:29 smithi144 ceph-mon[32129]: pgmap v9260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:42:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:29 smithi160 ceph-mon[36715]: pgmap v9260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:42:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:29 smithi183 ceph-mon[37273]: pgmap v9260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:42:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:31 smithi144 ceph-mon[32129]: pgmap v9261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16594.913363 secs 2024-07-26T15:42:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:31 smithi160 ceph-mon[36715]: pgmap v9261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16594.913363 secs 2024-07-26T15:42:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:31 smithi183 ceph-mon[37273]: pgmap v9261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16594.913363 secs 2024-07-26T15:42:33.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:33 smithi144 ceph-mon[32129]: pgmap v9262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:33 smithi160 ceph-mon[36715]: pgmap v9262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:33 smithi183 ceph-mon[37273]: pgmap v9262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:35 smithi144 ceph-mon[32129]: pgmap v9263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:35 smithi160 ceph-mon[36715]: pgmap v9263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:35 smithi183 ceph-mon[37273]: pgmap v9263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16599.913428 secs 2024-07-26T15:42:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16599.913428 secs 2024-07-26T15:42:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16599.913428 secs 2024-07-26T15:42:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:37 smithi144 ceph-mon[32129]: pgmap v9264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:37 smithi160 ceph-mon[36715]: pgmap v9264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:37 smithi183 ceph-mon[37273]: pgmap v9264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:39.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:39 smithi144 ceph-mon[32129]: pgmap v9265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:39 smithi160 ceph-mon[36715]: pgmap v9265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:39 smithi183 ceph-mon[37273]: pgmap v9265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:41 smithi144 ceph-mon[32129]: pgmap v9266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16604.913528 secs 2024-07-26T15:42:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:41.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:42:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:41.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:42:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:41.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:42:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:41.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:42:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:41.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:41.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:41 smithi160 ceph-mon[36715]: pgmap v9266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16604.913528 secs 2024-07-26T15:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:41 smithi183 ceph-mon[37273]: pgmap v9266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16604.913528 secs 2024-07-26T15:42:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:43 smithi160 ceph-mon[36715]: pgmap v9267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:43.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:43 smithi183 ceph-mon[37273]: pgmap v9267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:43 smithi144 ceph-mon[32129]: pgmap v9267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:45 smithi160 ceph-mon[36715]: pgmap v9268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:45 smithi183 ceph-mon[37273]: pgmap v9268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:45 smithi144 ceph-mon[32129]: pgmap v9268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16609.913628 secs 2024-07-26T15:42:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16609.913628 secs 2024-07-26T15:42:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16609.913628 secs 2024-07-26T15:42:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:47 smithi160 ceph-mon[36715]: pgmap v9269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:47.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:47 smithi183 ceph-mon[37273]: pgmap v9269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:47 smithi144 ceph-mon[32129]: pgmap v9269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:49 smithi160 ceph-mon[36715]: pgmap v9270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:49 smithi183 ceph-mon[37273]: pgmap v9270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:49 smithi144 ceph-mon[32129]: pgmap v9270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:51 smithi160 ceph-mon[36715]: pgmap v9271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16614.913755 secs 2024-07-26T15:42:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:51 smithi183 ceph-mon[37273]: pgmap v9271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16614.913755 secs 2024-07-26T15:42:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:51 smithi144 ceph-mon[32129]: pgmap v9271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16614.913755 secs 2024-07-26T15:42:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:53 smithi160 ceph-mon[36715]: pgmap v9272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:53 smithi183 ceph-mon[37273]: pgmap v9272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:53 smithi144 ceph-mon[32129]: pgmap v9272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:42:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:55 smithi160 ceph-mon[36715]: pgmap v9273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:55 smithi183 ceph-mon[37273]: pgmap v9273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:55 smithi144 ceph-mon[32129]: pgmap v9273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:56.786 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16619.913853 secs 2024-07-26T15:42:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:56.805+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:42:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:56.806+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:42:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:56.806+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:42:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:56.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:42:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:42:56.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16619.913853 secs 2024-07-26T15:42:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16619.913853 secs 2024-07-26T15:42:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:57 smithi160 ceph-mon[36715]: pgmap v9274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:57.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:57 smithi183 ceph-mon[37273]: pgmap v9274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:57 smithi144 ceph-mon[32129]: pgmap v9274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:42:59 smithi160 ceph-mon[36715]: pgmap v9275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:59.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:42:59 smithi183 ceph-mon[37273]: pgmap v9275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:42:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:42:59 smithi144 ceph-mon[32129]: pgmap v9275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:01 smithi160 ceph-mon[36715]: pgmap v9276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16624.913974 secs 2024-07-26T15:43:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:01 smithi183 ceph-mon[37273]: pgmap v9276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16624.913974 secs 2024-07-26T15:43:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:01 smithi144 ceph-mon[32129]: pgmap v9276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16624.913974 secs 2024-07-26T15:43:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:03 smithi160 ceph-mon[36715]: pgmap v9277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:03.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:03 smithi183 ceph-mon[37273]: pgmap v9277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:03 smithi144 ceph-mon[32129]: pgmap v9277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:05 smithi160 ceph-mon[36715]: pgmap v9278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:05 smithi183 ceph-mon[37273]: pgmap v9278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:05 smithi144 ceph-mon[32129]: pgmap v9278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:06.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16629.914040 secs 2024-07-26T15:43:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16629.914040 secs 2024-07-26T15:43:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16629.914040 secs 2024-07-26T15:43:07.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:07 smithi183 ceph-mon[37273]: pgmap v9279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:07 smithi144 ceph-mon[32129]: pgmap v9279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:07 smithi160 ceph-mon[36715]: pgmap v9279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:09 smithi183 ceph-mon[37273]: pgmap v9280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:09 smithi144 ceph-mon[32129]: pgmap v9280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:09 smithi160 ceph-mon[36715]: pgmap v9280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:11.822+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:43:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:11.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:43:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:11.823+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:43:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:11.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:43:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:11.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:11 smithi183 ceph-mon[37273]: pgmap v9281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16634.914161 secs 2024-07-26T15:43:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:11 smithi144 ceph-mon[32129]: pgmap v9281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16634.914161 secs 2024-07-26T15:43:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:11 smithi160 ceph-mon[36715]: pgmap v9281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16634.914161 secs 2024-07-26T15:43:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:13 smithi183 ceph-mon[37273]: pgmap v9282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:13 smithi144 ceph-mon[32129]: pgmap v9282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:13 smithi160 ceph-mon[36715]: pgmap v9282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:15.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:15 smithi183 ceph-mon[37273]: pgmap v9283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:15 smithi144 ceph-mon[32129]: pgmap v9283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:15 smithi160 ceph-mon[36715]: pgmap v9283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16639.914260 secs 2024-07-26T15:43:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16639.914260 secs 2024-07-26T15:43:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16639.914260 secs 2024-07-26T15:43:17.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:17 smithi183 ceph-mon[37273]: pgmap v9284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:17 smithi144 ceph-mon[32129]: pgmap v9284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:17 smithi160 ceph-mon[36715]: pgmap v9284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:19 smithi183 ceph-mon[37273]: pgmap v9285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:43:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:19 smithi144 ceph-mon[32129]: pgmap v9285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:19.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:43:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:19 smithi160 ceph-mon[36715]: pgmap v9285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:43:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:20 smithi183 ceph-mon[37273]: pgmap v9286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:43:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:20 smithi183 ceph-mon[37273]: pgmap v9287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:43:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:20 smithi144 ceph-mon[32129]: pgmap v9286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:43:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:20 smithi144 ceph-mon[32129]: pgmap v9287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:43:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:20 smithi160 ceph-mon[36715]: pgmap v9286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:43:21.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:20 smithi160 ceph-mon[36715]: pgmap v9287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:43:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16644.914406 secs 2024-07-26T15:43:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16644.914406 secs 2024-07-26T15:43:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16644.914406 secs 2024-07-26T15:43:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:22 smithi144 ceph-mon[32129]: pgmap v9288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:43:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:22 smithi160 ceph-mon[36715]: pgmap v9288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:43:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:22 smithi183 ceph-mon[37273]: pgmap v9288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:43:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:24 smithi144 ceph-mon[32129]: pgmap v9289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:43:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:24 smithi160 ceph-mon[36715]: pgmap v9289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:43:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:24 smithi183 ceph-mon[37273]: pgmap v9289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:26.836+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:26.837+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:43:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:26.837+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:43:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:26.837+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:43:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:26.837+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:26.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:26 smithi183 ceph-mon[37273]: pgmap v9290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:26.936 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16649.914551 secs 2024-07-26T15:43:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:26 smithi144 ceph-mon[32129]: pgmap v9290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16649.914551 secs 2024-07-26T15:43:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:26 smithi160 ceph-mon[36715]: pgmap v9290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16649.914551 secs 2024-07-26T15:43:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:28 smithi144 ceph-mon[32129]: pgmap v9291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T15:43:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:28 smithi160 ceph-mon[36715]: pgmap v9291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T15:43:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:28 smithi183 ceph-mon[37273]: pgmap v9291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T15:43:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:30 smithi144 ceph-mon[32129]: pgmap v9292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:43:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:30 smithi160 ceph-mon[36715]: pgmap v9292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:43:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:30 smithi183 ceph-mon[37273]: pgmap v9292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T15:43:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16654.914631 secs 2024-07-26T15:43:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16654.914631 secs 2024-07-26T15:43:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16654.914631 secs 2024-07-26T15:43:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:32 smithi144 ceph-mon[32129]: pgmap v9293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:43:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:32 smithi160 ceph-mon[36715]: pgmap v9293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:43:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:32 smithi183 ceph-mon[37273]: pgmap v9293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:43:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:34 smithi144 ceph-mon[32129]: pgmap v9294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:43:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:34 smithi160 ceph-mon[36715]: pgmap v9294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:43:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:34 smithi183 ceph-mon[37273]: pgmap v9294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:43:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:36 smithi144 ceph-mon[32129]: pgmap v9295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:43:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16659.914751 secs 2024-07-26T15:43:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:36 smithi160 ceph-mon[36715]: pgmap v9295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:43:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16659.914751 secs 2024-07-26T15:43:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:36 smithi183 ceph-mon[37273]: pgmap v9295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:43:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16659.914751 secs 2024-07-26T15:43:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:38 smithi144 ceph-mon[32129]: pgmap v9296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:38 smithi160 ceph-mon[36715]: pgmap v9296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:38 smithi183 ceph-mon[37273]: pgmap v9296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:40 smithi144 ceph-mon[32129]: pgmap v9297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:40 smithi160 ceph-mon[36715]: pgmap v9297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:40 smithi183 ceph-mon[37273]: pgmap v9297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:41.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:41.853+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:43:41.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:41.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:43:41.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:41.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:43:41.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:41.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:43:41.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:41.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16664.914873 secs 2024-07-26T15:43:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16664.914873 secs 2024-07-26T15:43:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16664.914873 secs 2024-07-26T15:43:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:42 smithi144 ceph-mon[32129]: pgmap v9298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:42 smithi160 ceph-mon[36715]: pgmap v9298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:42 smithi183 ceph-mon[37273]: pgmap v9298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:44 smithi160 ceph-mon[36715]: pgmap v9299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:44 smithi183 ceph-mon[37273]: pgmap v9299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:44 smithi144 ceph-mon[32129]: pgmap v9299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:46 smithi160 ceph-mon[36715]: pgmap v9300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16669.914990 secs 2024-07-26T15:43:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:46 smithi183 ceph-mon[37273]: pgmap v9300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16669.914990 secs 2024-07-26T15:43:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:46 smithi144 ceph-mon[32129]: pgmap v9300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:47.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16669.914990 secs 2024-07-26T15:43:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:48 smithi160 ceph-mon[36715]: pgmap v9301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:48 smithi183 ceph-mon[37273]: pgmap v9301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:48 smithi144 ceph-mon[32129]: pgmap v9301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:50 smithi160 ceph-mon[36715]: pgmap v9302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:50 smithi183 ceph-mon[37273]: pgmap v9302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:50 smithi144 ceph-mon[32129]: pgmap v9302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16674.915069 secs 2024-07-26T15:43:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16674.915069 secs 2024-07-26T15:43:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16674.915069 secs 2024-07-26T15:43:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:52 smithi160 ceph-mon[36715]: pgmap v9303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:52 smithi183 ceph-mon[37273]: pgmap v9303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:52 smithi144 ceph-mon[32129]: pgmap v9303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:43:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:53 smithi160 ceph-mon[36715]: pgmap v9304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:53 smithi183 ceph-mon[37273]: pgmap v9304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:53 smithi144 ceph-mon[32129]: pgmap v9304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:56 smithi144 ceph-mon[32129]: pgmap v9305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16679.915186 secs 2024-07-26T15:43:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:56 smithi160 ceph-mon[36715]: pgmap v9305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16679.915186 secs 2024-07-26T15:43:56.838 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:56 smithi183 ceph-mon[37273]: pgmap v9305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:43:56.838 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16679.915186 secs 2024-07-26T15:43:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:56.870+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:43:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:56.871+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:43:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:56.871+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:43:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:56.871+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:43:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:43:56.871+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:43:58 smithi144 ceph-mon[32129]: pgmap v9306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:43:58 smithi160 ceph-mon[36715]: pgmap v9306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:43:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:43:58 smithi183 ceph-mon[37273]: pgmap v9306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:00 smithi144 ceph-mon[32129]: pgmap v9307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:00 smithi160 ceph-mon[36715]: pgmap v9307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:00 smithi183 ceph-mon[37273]: pgmap v9307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16684.915327 secs 2024-07-26T15:44:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16684.915327 secs 2024-07-26T15:44:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16684.915327 secs 2024-07-26T15:44:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:02 smithi144 ceph-mon[32129]: pgmap v9308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:02 smithi160 ceph-mon[36715]: pgmap v9308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:02 smithi183 ceph-mon[37273]: pgmap v9308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:04 smithi160 ceph-mon[36715]: pgmap v9309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:04 smithi183 ceph-mon[37273]: pgmap v9309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:04 smithi144 ceph-mon[32129]: pgmap v9309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:06.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:06 smithi144 ceph-mon[32129]: pgmap v9310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:06.773 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16689.915447 secs 2024-07-26T15:44:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:06 smithi160 ceph-mon[36715]: pgmap v9310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16689.915447 secs 2024-07-26T15:44:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:06 smithi183 ceph-mon[37273]: pgmap v9310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16689.915447 secs 2024-07-26T15:44:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:08 smithi160 ceph-mon[36715]: pgmap v9311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:08 smithi183 ceph-mon[37273]: pgmap v9311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:08 smithi144 ceph-mon[32129]: pgmap v9311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:10 smithi160 ceph-mon[36715]: pgmap v9312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:10 smithi183 ceph-mon[37273]: pgmap v9312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:10 smithi144 ceph-mon[32129]: pgmap v9312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16694.915572 secs 2024-07-26T15:44:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16694.915572 secs 2024-07-26T15:44:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:11.887+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:44:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:11.888+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:44:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:11.888+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:44:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:11.888+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:44:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:11.888+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16694.915572 secs 2024-07-26T15:44:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:12 smithi160 ceph-mon[36715]: pgmap v9313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:12 smithi183 ceph-mon[37273]: pgmap v9313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:12 smithi144 ceph-mon[32129]: pgmap v9313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:14 smithi160 ceph-mon[36715]: pgmap v9314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:14 smithi183 ceph-mon[37273]: pgmap v9314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:14 smithi144 ceph-mon[32129]: pgmap v9314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:16 smithi160 ceph-mon[36715]: pgmap v9315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16699.915684 secs 2024-07-26T15:44:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:16 smithi183 ceph-mon[37273]: pgmap v9315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16699.915684 secs 2024-07-26T15:44:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:16 smithi144 ceph-mon[32129]: pgmap v9315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16699.915684 secs 2024-07-26T15:44:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:18 smithi160 ceph-mon[36715]: pgmap v9316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:18 smithi183 ceph-mon[37273]: pgmap v9316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:18 smithi144 ceph-mon[32129]: pgmap v9316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:20 smithi160 ceph-mon[36715]: pgmap v9317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:20 smithi183 ceph-mon[37273]: pgmap v9317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:20 smithi144 ceph-mon[32129]: pgmap v9317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: pgmap v9318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: pgmap v9319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:44:21.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16704.915779 secs 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:44:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: pgmap v9318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: pgmap v9319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:44:21.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16704.915779 secs 2024-07-26T15:44:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: pgmap v9318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: pgmap v9319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:44:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16704.915779 secs 2024-07-26T15:44:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:23 smithi183 ceph-mon[37273]: pgmap v9320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:44:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:23 smithi144 ceph-mon[32129]: pgmap v9320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:44:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:23 smithi160 ceph-mon[36715]: pgmap v9320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:44:25.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:25 smithi183 ceph-mon[37273]: pgmap v9321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:44:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:25 smithi144 ceph-mon[32129]: pgmap v9321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:44:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:25 smithi160 ceph-mon[36715]: pgmap v9321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:44:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16709.915899 secs 2024-07-26T15:44:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:26.903+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:44:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:26.904+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:44:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:26.904+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:44:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:26.904+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:44:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:26.904+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16709.915899 secs 2024-07-26T15:44:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16709.915899 secs 2024-07-26T15:44:27.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:27 smithi183 ceph-mon[37273]: pgmap v9322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:27.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:27 smithi144 ceph-mon[32129]: pgmap v9322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:27 smithi160 ceph-mon[36715]: pgmap v9322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:44:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:29 smithi144 ceph-mon[32129]: pgmap v9323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:44:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:29 smithi160 ceph-mon[36715]: pgmap v9323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:44:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:29 smithi183 ceph-mon[37273]: pgmap v9323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:44:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:31 smithi183 ceph-mon[37273]: pgmap v9324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:44:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16714.916042 secs 2024-07-26T15:44:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:31 smithi144 ceph-mon[32129]: pgmap v9324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:44:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16714.916042 secs 2024-07-26T15:44:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:31 smithi160 ceph-mon[36715]: pgmap v9324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:44:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16714.916042 secs 2024-07-26T15:44:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:33 smithi183 ceph-mon[37273]: pgmap v9325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:33 smithi144 ceph-mon[32129]: pgmap v9325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:33 smithi160 ceph-mon[36715]: pgmap v9325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:35 smithi183 ceph-mon[37273]: pgmap v9326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:35 smithi144 ceph-mon[32129]: pgmap v9326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:35 smithi160 ceph-mon[36715]: pgmap v9326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16719.916181 secs 2024-07-26T15:44:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16719.916181 secs 2024-07-26T15:44:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16719.916181 secs 2024-07-26T15:44:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:37 smithi144 ceph-mon[32129]: pgmap v9327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:37 smithi160 ceph-mon[36715]: pgmap v9327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:37 smithi183 ceph-mon[37273]: pgmap v9327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:39 smithi144 ceph-mon[32129]: pgmap v9328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:39 smithi160 ceph-mon[36715]: pgmap v9328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:39 smithi183 ceph-mon[37273]: pgmap v9328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:41.921+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:44:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:41.922+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:44:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:41.922+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:44:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:41.922+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:44:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:41.922+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:41 smithi144 ceph-mon[32129]: pgmap v9329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16724.916316 secs 2024-07-26T15:44:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:41 smithi160 ceph-mon[36715]: pgmap v9329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16724.916316 secs 2024-07-26T15:44:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:41 smithi183 ceph-mon[37273]: pgmap v9329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16724.916316 secs 2024-07-26T15:44:43.934 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:43 smithi144 ceph-mon[32129]: pgmap v9330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:43 smithi160 ceph-mon[36715]: pgmap v9330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:43 smithi183 ceph-mon[37273]: pgmap v9330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:45 smithi144 ceph-mon[32129]: pgmap v9331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:45 smithi160 ceph-mon[36715]: pgmap v9331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:45 smithi183 ceph-mon[37273]: pgmap v9331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16729.916436 secs 2024-07-26T15:44:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16729.916436 secs 2024-07-26T15:44:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16729.916436 secs 2024-07-26T15:44:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:47 smithi144 ceph-mon[32129]: pgmap v9332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:47 smithi160 ceph-mon[36715]: pgmap v9332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:48.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:47 smithi183 ceph-mon[37273]: pgmap v9332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:49.050+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:44:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:49.050+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:44:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:49.051+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:44:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:49.051+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:44:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:49 smithi144 ceph-mon[32129]: pgmap v9333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:49 smithi160 ceph-mon[36715]: pgmap v9333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:49 smithi183 ceph-mon[37273]: pgmap v9333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:51 smithi144 ceph-mon[32129]: pgmap v9334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16734.916567 secs 2024-07-26T15:44:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:51 smithi160 ceph-mon[36715]: pgmap v9334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16734.916567 secs 2024-07-26T15:44:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:51 smithi183 ceph-mon[37273]: pgmap v9334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16734.916567 secs 2024-07-26T15:44:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:53 smithi144 ceph-mon[32129]: pgmap v9335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:53 smithi160 ceph-mon[36715]: pgmap v9335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:53 smithi183 ceph-mon[37273]: pgmap v9335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:44:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:55 smithi144 ceph-mon[32129]: pgmap v9336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:55 smithi160 ceph-mon[36715]: pgmap v9336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:55 smithi183 ceph-mon[37273]: pgmap v9336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:44:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:56.938+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:44:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:56.939+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:44:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:56.939+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:44:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:56.939+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:44:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:44:56.939+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:57.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16739.916692 secs 2024-07-26T15:44:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16739.916692 secs 2024-07-26T15:44:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16739.916692 secs 2024-07-26T15:44:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:57 smithi160 ceph-mon[36715]: pgmap v9337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:57 smithi183 ceph-mon[37273]: pgmap v9337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:44:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:57 smithi144 ceph-mon[32129]: pgmap v9337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:44:59 smithi160 ceph-mon[36715]: pgmap v9338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:44:59 smithi183 ceph-mon[37273]: pgmap v9338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:44:59 smithi144 ceph-mon[32129]: pgmap v9338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:01 smithi160 ceph-mon[36715]: pgmap v9339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16744.916767 secs 2024-07-26T15:45:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:01 smithi183 ceph-mon[37273]: pgmap v9339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16744.916767 secs 2024-07-26T15:45:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:01 smithi144 ceph-mon[32129]: pgmap v9339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16744.916767 secs 2024-07-26T15:45:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:03 smithi160 ceph-mon[36715]: pgmap v9340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:03 smithi183 ceph-mon[37273]: pgmap v9340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:03 smithi144 ceph-mon[32129]: pgmap v9340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:04 smithi160 ceph-mon[36715]: pgmap v9341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:04 smithi183 ceph-mon[37273]: pgmap v9341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:04 smithi144 ceph-mon[32129]: pgmap v9341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16749.916913 secs 2024-07-26T15:45:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16749.916913 secs 2024-07-26T15:45:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16749.916913 secs 2024-07-26T15:45:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:07 smithi144 ceph-mon[32129]: pgmap v9342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:07 smithi160 ceph-mon[36715]: pgmap v9342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:07 smithi183 ceph-mon[37273]: pgmap v9342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:09 smithi144 ceph-mon[32129]: pgmap v9343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:09 smithi160 ceph-mon[36715]: pgmap v9343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:09 smithi183 ceph-mon[37273]: pgmap v9343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:11.954+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:45:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:11.955+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:45:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:11.955+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:45:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:11.955+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:45:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:11.955+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:11 smithi144 ceph-mon[32129]: pgmap v9344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16754.917046 secs 2024-07-26T15:45:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:11 smithi160 ceph-mon[36715]: pgmap v9344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16754.917046 secs 2024-07-26T15:45:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:11 smithi183 ceph-mon[37273]: pgmap v9344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16754.917046 secs 2024-07-26T15:45:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:13 smithi144 ceph-mon[32129]: pgmap v9345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:13 smithi160 ceph-mon[36715]: pgmap v9345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:13 smithi183 ceph-mon[37273]: pgmap v9345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:15 smithi144 ceph-mon[32129]: pgmap v9346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:15 smithi160 ceph-mon[36715]: pgmap v9346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:15 smithi183 ceph-mon[37273]: pgmap v9346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16759.917153 secs 2024-07-26T15:45:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16759.917153 secs 2024-07-26T15:45:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16759.917153 secs 2024-07-26T15:45:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:17 smithi144 ceph-mon[32129]: pgmap v9347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:17 smithi160 ceph-mon[36715]: pgmap v9347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:17 smithi183 ceph-mon[37273]: pgmap v9347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:19 smithi144 ceph-mon[32129]: pgmap v9348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:19 smithi160 ceph-mon[36715]: pgmap v9348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:19 smithi183 ceph-mon[37273]: pgmap v9348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: pgmap v9349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16764.917249 secs 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:21.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: pgmap v9349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16764.917249 secs 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: pgmap v9349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16764.917249 secs 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:22 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:22 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:45:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:45:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:45:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:45:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:45:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:22 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:45:23.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:23 smithi144 ceph-mon[32129]: pgmap v9350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:45:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:23 smithi144 ceph-mon[32129]: pgmap v9351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:45:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:23 smithi160 ceph-mon[36715]: pgmap v9350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:45:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:23 smithi160 ceph-mon[36715]: pgmap v9351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:45:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:23 smithi183 ceph-mon[37273]: pgmap v9350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:45:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:23 smithi183 ceph-mon[37273]: pgmap v9351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:45:25.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:25 smithi144 ceph-mon[32129]: pgmap v9352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:45:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:25 smithi160 ceph-mon[36715]: pgmap v9352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:45:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:25 smithi183 ceph-mon[37273]: pgmap v9352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:45:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:26.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:45:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:26.971+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:45:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:26.971+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:45:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:26.971+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:45:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:26.971+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:27.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16769.917367 secs 2024-07-26T15:45:27.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16769.917367 secs 2024-07-26T15:45:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16769.917367 secs 2024-07-26T15:45:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:27 smithi160 ceph-mon[36715]: pgmap v9353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:45:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:27 smithi183 ceph-mon[37273]: pgmap v9353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:45:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:27 smithi144 ceph-mon[32129]: pgmap v9353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:45:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:29 smithi160 ceph-mon[36715]: pgmap v9354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:29 smithi183 ceph-mon[37273]: pgmap v9354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:29 smithi144 ceph-mon[32129]: pgmap v9354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:31 smithi160 ceph-mon[36715]: pgmap v9355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16774.917451 secs 2024-07-26T15:45:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:31 smithi183 ceph-mon[37273]: pgmap v9355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16774.917451 secs 2024-07-26T15:45:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:31 smithi144 ceph-mon[32129]: pgmap v9355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16774.917451 secs 2024-07-26T15:45:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:33 smithi160 ceph-mon[36715]: pgmap v9356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:34.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:33 smithi183 ceph-mon[37273]: pgmap v9356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:33 smithi144 ceph-mon[32129]: pgmap v9356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:36.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:35 smithi144 ceph-mon[32129]: pgmap v9357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:35 smithi160 ceph-mon[36715]: pgmap v9357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:35 smithi183 ceph-mon[37273]: pgmap v9357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16779.917575 secs 2024-07-26T15:45:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16779.917575 secs 2024-07-26T15:45:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16779.917575 secs 2024-07-26T15:45:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:37 smithi160 ceph-mon[36715]: pgmap v9358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:37 smithi183 ceph-mon[37273]: pgmap v9358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:37 smithi144 ceph-mon[32129]: pgmap v9358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:39 smithi160 ceph-mon[36715]: pgmap v9359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:40.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:39 smithi183 ceph-mon[37273]: pgmap v9359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:39 smithi144 ceph-mon[32129]: pgmap v9359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:40 smithi160 ceph-mon[36715]: pgmap v9360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:45:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:40 smithi183 ceph-mon[37273]: pgmap v9360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:45:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:40 smithi144 ceph-mon[32129]: pgmap v9360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:45:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:41.988+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:45:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:41.989+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:45:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:41.989+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:45:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:41.989+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:45:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:41.989+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16784.917709 secs 2024-07-26T15:45:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16784.917709 secs 2024-07-26T15:45:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16784.917709 secs 2024-07-26T15:45:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:42 smithi160 ceph-mon[36715]: pgmap v9361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:42 smithi183 ceph-mon[37273]: pgmap v9361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:42 smithi144 ceph-mon[32129]: pgmap v9361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:44 smithi183 ceph-mon[37273]: pgmap v9362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:44 smithi144 ceph-mon[32129]: pgmap v9362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:44 smithi160 ceph-mon[36715]: pgmap v9362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:46 smithi183 ceph-mon[37273]: pgmap v9363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16789.917849 secs 2024-07-26T15:45:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:46 smithi144 ceph-mon[32129]: pgmap v9363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16789.917849 secs 2024-07-26T15:45:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:46 smithi160 ceph-mon[36715]: pgmap v9363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:45:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16789.917849 secs 2024-07-26T15:45:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:48 smithi183 ceph-mon[37273]: pgmap v9364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:48 smithi144 ceph-mon[32129]: pgmap v9364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:48 smithi160 ceph-mon[36715]: pgmap v9364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:45:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:50 smithi183 ceph-mon[37273]: pgmap v9365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:45:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:50 smithi144 ceph-mon[32129]: pgmap v9365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:45:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:50 smithi160 ceph-mon[36715]: pgmap v9365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:45:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16794.917983 secs 2024-07-26T15:45:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16794.917983 secs 2024-07-26T15:45:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16794.917983 secs 2024-07-26T15:45:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:52 smithi144 ceph-mon[32129]: pgmap v9366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:52 smithi160 ceph-mon[36715]: pgmap v9366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:53.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:52 smithi183 ceph-mon[37273]: pgmap v9366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:45:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:54 smithi144 ceph-mon[32129]: pgmap v9367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:54 smithi160 ceph-mon[36715]: pgmap v9367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:54 smithi183 ceph-mon[37273]: pgmap v9367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:57.005+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:45:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:57.006+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:45:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:57.006+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:45:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:57.006+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:45:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:45:57.006+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:57.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:56 smithi183 ceph-mon[37273]: pgmap v9368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:57.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16799.918135 secs 2024-07-26T15:45:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:56 smithi144 ceph-mon[32129]: pgmap v9368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16799.918135 secs 2024-07-26T15:45:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:56 smithi160 ceph-mon[36715]: pgmap v9368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:45:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16799.918135 secs 2024-07-26T15:45:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:45:58 smithi144 ceph-mon[32129]: pgmap v9369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:45:58 smithi160 ceph-mon[36715]: pgmap v9369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:45:59.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:45:58 smithi183 ceph-mon[37273]: pgmap v9369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:00 smithi144 ceph-mon[32129]: pgmap v9370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:00 smithi160 ceph-mon[36715]: pgmap v9370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:01.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:00 smithi183 ceph-mon[37273]: pgmap v9370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16804.918312 secs 2024-07-26T15:46:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16804.918312 secs 2024-07-26T15:46:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16804.918312 secs 2024-07-26T15:46:03.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:02 smithi160 ceph-mon[36715]: pgmap v9371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:02 smithi144 ceph-mon[32129]: pgmap v9371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:03.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:02 smithi183 ceph-mon[37273]: pgmap v9371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:04 smithi144 ceph-mon[32129]: pgmap v9372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:04 smithi160 ceph-mon[36715]: pgmap v9372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:05.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:04 smithi183 ceph-mon[37273]: pgmap v9372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:06 smithi144 ceph-mon[32129]: pgmap v9373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16809.918406 secs 2024-07-26T15:46:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:06 smithi160 ceph-mon[36715]: pgmap v9373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16809.918406 secs 2024-07-26T15:46:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:06 smithi183 ceph-mon[37273]: pgmap v9373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16809.918406 secs 2024-07-26T15:46:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:08 smithi144 ceph-mon[32129]: pgmap v9374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:08 smithi160 ceph-mon[36715]: pgmap v9374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:08 smithi183 ceph-mon[37273]: pgmap v9374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:10 smithi144 ceph-mon[32129]: pgmap v9375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:10 smithi160 ceph-mon[36715]: pgmap v9375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:11.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:10 smithi183 ceph-mon[37273]: pgmap v9375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:12.021+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:46:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:12.022+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:46:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:12.022+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:46:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:12.022+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:46:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:12.022+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16814.918489 secs 2024-07-26T15:46:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16814.918489 secs 2024-07-26T15:46:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16814.918489 secs 2024-07-26T15:46:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:12 smithi144 ceph-mon[32129]: pgmap v9376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:12 smithi160 ceph-mon[36715]: pgmap v9376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:12 smithi183 ceph-mon[37273]: pgmap v9376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:14 smithi160 ceph-mon[36715]: pgmap v9377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:15.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:14 smithi183 ceph-mon[37273]: pgmap v9377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:14 smithi144 ceph-mon[32129]: pgmap v9377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:17.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:16 smithi144 ceph-mon[32129]: pgmap v9378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:17.274 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16819.918548 secs 2024-07-26T15:46:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:16 smithi160 ceph-mon[36715]: pgmap v9378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16819.918548 secs 2024-07-26T15:46:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:16 smithi183 ceph-mon[37273]: pgmap v9378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16819.918548 secs 2024-07-26T15:46:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:19.065+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:46:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:19.065+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:46:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:19.065+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:46:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:19.065+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:46:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:18 smithi160 ceph-mon[36715]: pgmap v9379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:19.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:18 smithi183 ceph-mon[37273]: pgmap v9379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:18 smithi144 ceph-mon[32129]: pgmap v9379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:20 smithi160 ceph-mon[36715]: pgmap v9380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:21.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:20 smithi183 ceph-mon[37273]: pgmap v9380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:20 smithi144 ceph-mon[32129]: pgmap v9380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16824.918688 secs 2024-07-26T15:46:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16824.918688 secs 2024-07-26T15:46:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16824.918688 secs 2024-07-26T15:46:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:22 smithi160 ceph-mon[36715]: pgmap v9381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:22 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:23.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:22 smithi183 ceph-mon[37273]: pgmap v9381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:22 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:22 smithi144 ceph-mon[32129]: pgmap v9381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:22 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:24 smithi160 ceph-mon[36715]: pgmap v9382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:46:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:24 smithi160 ceph-mon[36715]: pgmap v9383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:46:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:46:24.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:24 smithi183 ceph-mon[37273]: pgmap v9382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:46:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:24 smithi183 ceph-mon[37273]: pgmap v9383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:46:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:46:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:24 smithi144 ceph-mon[32129]: pgmap v9382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:46:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:24 smithi144 ceph-mon[32129]: pgmap v9383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:46:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:46:26.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:26 smithi183 ceph-mon[37273]: pgmap v9384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:46:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:26 smithi144 ceph-mon[32129]: pgmap v9384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:46:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:26 smithi160 ceph-mon[36715]: pgmap v9384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:46:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:27.037+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:46:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:27.038+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:46:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:27.038+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:46:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:27.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:46:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:27.038+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16829.918813 secs 2024-07-26T15:46:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16829.918813 secs 2024-07-26T15:46:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16829.918813 secs 2024-07-26T15:46:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:28 smithi183 ceph-mon[37273]: pgmap v9385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:46:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:28 smithi144 ceph-mon[32129]: pgmap v9385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:46:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:28 smithi160 ceph-mon[36715]: pgmap v9385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:46:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:30 smithi183 ceph-mon[37273]: pgmap v9386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:46:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:30 smithi144 ceph-mon[32129]: pgmap v9386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:46:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:30 smithi160 ceph-mon[36715]: pgmap v9386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:46:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16834.918941 secs 2024-07-26T15:46:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:32 smithi183 ceph-mon[37273]: pgmap v9387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:46:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16834.918941 secs 2024-07-26T15:46:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:32 smithi144 ceph-mon[32129]: pgmap v9387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:46:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16834.918941 secs 2024-07-26T15:46:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:32 smithi160 ceph-mon[36715]: pgmap v9387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:46:34.361 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:34 smithi160 ceph-mon[36715]: pgmap v9388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:46:34.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:34 smithi183 ceph-mon[37273]: pgmap v9388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:46:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:34 smithi144 ceph-mon[32129]: pgmap v9388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:46:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:36 smithi183 ceph-mon[37273]: pgmap v9389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:36.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:36 smithi144 ceph-mon[32129]: pgmap v9389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:36 smithi160 ceph-mon[36715]: pgmap v9389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:37.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16839.919063 secs 2024-07-26T15:46:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16839.919063 secs 2024-07-26T15:46:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16839.919063 secs 2024-07-26T15:46:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:38 smithi183 ceph-mon[37273]: pgmap v9390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:38 smithi144 ceph-mon[32129]: pgmap v9390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:38 smithi160 ceph-mon[36715]: pgmap v9390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:40.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:40 smithi183 ceph-mon[37273]: pgmap v9391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:40 smithi144 ceph-mon[32129]: pgmap v9391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:40 smithi160 ceph-mon[36715]: pgmap v9391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:42.055+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:46:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:42.056+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:46:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:42.056+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:46:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:42.056+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:46:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:42.056+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16844.919183 secs 2024-07-26T15:46:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:42 smithi183 ceph-mon[37273]: pgmap v9392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16844.919183 secs 2024-07-26T15:46:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:42 smithi144 ceph-mon[32129]: pgmap v9392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16844.919183 secs 2024-07-26T15:46:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:42 smithi160 ceph-mon[36715]: pgmap v9392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:44 smithi144 ceph-mon[32129]: pgmap v9393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:44 smithi160 ceph-mon[36715]: pgmap v9393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:44 smithi183 ceph-mon[37273]: pgmap v9393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:46 smithi144 ceph-mon[32129]: pgmap v9394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:46 smithi160 ceph-mon[36715]: pgmap v9394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:46.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:46 smithi183 ceph-mon[37273]: pgmap v9394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16849.919315 secs 2024-07-26T15:46:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16849.919315 secs 2024-07-26T15:46:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16849.919315 secs 2024-07-26T15:46:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:48 smithi144 ceph-mon[32129]: pgmap v9395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:48 smithi160 ceph-mon[36715]: pgmap v9395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:48 smithi183 ceph-mon[37273]: pgmap v9395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:50 smithi144 ceph-mon[32129]: pgmap v9396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:50 smithi160 ceph-mon[36715]: pgmap v9396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:50 smithi183 ceph-mon[37273]: pgmap v9396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16854.919335 secs 2024-07-26T15:46:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16854.919335 secs 2024-07-26T15:46:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16854.919335 secs 2024-07-26T15:46:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:52 smithi160 ceph-mon[36715]: pgmap v9397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:52 smithi183 ceph-mon[37273]: pgmap v9397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:52 smithi144 ceph-mon[32129]: pgmap v9397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:54 smithi160 ceph-mon[36715]: pgmap v9398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:54 smithi183 ceph-mon[37273]: pgmap v9398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:54 smithi144 ceph-mon[32129]: pgmap v9398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:46:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:56 smithi160 ceph-mon[36715]: pgmap v9399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16859.919440 secs 2024-07-26T15:46:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:56 smithi183 ceph-mon[37273]: pgmap v9399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16859.919440 secs 2024-07-26T15:46:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:56 smithi144 ceph-mon[32129]: pgmap v9399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:46:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16859.919440 secs 2024-07-26T15:46:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:57.072+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:46:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:57.073+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:46:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:57.073+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:46:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:57.073+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:46:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:46:57.073+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:46:58 smithi160 ceph-mon[36715]: pgmap v9400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:46:58 smithi183 ceph-mon[37273]: pgmap v9400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:46:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:46:58 smithi144 ceph-mon[32129]: pgmap v9400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:00 smithi160 ceph-mon[36715]: pgmap v9401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:00 smithi183 ceph-mon[37273]: pgmap v9401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:00 smithi144 ceph-mon[32129]: pgmap v9401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16864.919505 secs 2024-07-26T15:47:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16864.919505 secs 2024-07-26T15:47:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16864.919505 secs 2024-07-26T15:47:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:02 smithi160 ceph-mon[36715]: pgmap v9402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:02 smithi183 ceph-mon[37273]: pgmap v9402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:02 smithi144 ceph-mon[32129]: pgmap v9402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:04 smithi160 ceph-mon[36715]: pgmap v9403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:04 smithi183 ceph-mon[37273]: pgmap v9403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:04 smithi144 ceph-mon[32129]: pgmap v9403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:06 smithi160 ceph-mon[36715]: pgmap v9404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16869.919622 secs 2024-07-26T15:47:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:06 smithi183 ceph-mon[37273]: pgmap v9404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16869.919622 secs 2024-07-26T15:47:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:06 smithi144 ceph-mon[32129]: pgmap v9404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16869.919622 secs 2024-07-26T15:47:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:08 smithi160 ceph-mon[36715]: pgmap v9405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:08 smithi183 ceph-mon[37273]: pgmap v9405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:08 smithi144 ceph-mon[32129]: pgmap v9405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:10.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:10 smithi183 ceph-mon[37273]: pgmap v9406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:10 smithi144 ceph-mon[32129]: pgmap v9406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:10 smithi160 ceph-mon[36715]: pgmap v9406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16874.919765 secs 2024-07-26T15:47:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16874.919765 secs 2024-07-26T15:47:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16874.919765 secs 2024-07-26T15:47:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:12.075+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:47:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:12.076+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:47:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:12.076+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:47:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:12.076+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:47:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:12.076+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:12.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:12 smithi183 ceph-mon[37273]: pgmap v9407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:12 smithi144 ceph-mon[32129]: pgmap v9407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:12 smithi160 ceph-mon[36715]: pgmap v9407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:14 smithi183 ceph-mon[37273]: pgmap v9408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:14 smithi144 ceph-mon[32129]: pgmap v9408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:14 smithi160 ceph-mon[36715]: pgmap v9408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:16 smithi183 ceph-mon[37273]: pgmap v9409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16879.919893 secs 2024-07-26T15:47:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:16 smithi144 ceph-mon[32129]: pgmap v9409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16879.919893 secs 2024-07-26T15:47:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:16 smithi160 ceph-mon[36715]: pgmap v9409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16879.919893 secs 2024-07-26T15:47:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:18 smithi183 ceph-mon[37273]: pgmap v9410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:18 smithi144 ceph-mon[32129]: pgmap v9410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:18 smithi160 ceph-mon[36715]: pgmap v9410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:20.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:20 smithi183 ceph-mon[37273]: pgmap v9411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:20 smithi144 ceph-mon[32129]: pgmap v9411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:20 smithi160 ceph-mon[36715]: pgmap v9411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16884.919990 secs 2024-07-26T15:47:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16884.919990 secs 2024-07-26T15:47:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16884.919990 secs 2024-07-26T15:47:22.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:22 smithi183 ceph-mon[37273]: pgmap v9412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:22 smithi144 ceph-mon[32129]: pgmap v9412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:22 smithi160 ceph-mon[36715]: pgmap v9412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:24.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:24 smithi183 ceph-mon[37273]: pgmap v9413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:24 smithi144 ceph-mon[32129]: pgmap v9413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:24 smithi160 ceph-mon[36715]: pgmap v9413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:25.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:47:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:26 smithi144 ceph-mon[32129]: pgmap v9414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:26 smithi144 ceph-mon[32129]: pgmap v9415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16889.920114 secs 2024-07-26T15:47:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:26 smithi160 ceph-mon[36715]: pgmap v9414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:26 smithi160 ceph-mon[36715]: pgmap v9415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16889.920114 secs 2024-07-26T15:47:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:26 smithi183 ceph-mon[37273]: pgmap v9414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:26 smithi183 ceph-mon[37273]: pgmap v9415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16889.920114 secs 2024-07-26T15:47:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:27.091+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:47:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:27.092+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:47:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:27.092+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:47:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:27.092+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:47:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:27.092+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:28.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:28 smithi183 ceph-mon[37273]: pgmap v9416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:28 smithi144 ceph-mon[32129]: pgmap v9416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:28 smithi160 ceph-mon[36715]: pgmap v9416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:30 smithi144 ceph-mon[32129]: pgmap v9417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:30 smithi160 ceph-mon[36715]: pgmap v9417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:30 smithi183 ceph-mon[37273]: pgmap v9417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16894.920184 secs 2024-07-26T15:47:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16894.920184 secs 2024-07-26T15:47:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16894.920184 secs 2024-07-26T15:47:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:32 smithi144 ceph-mon[32129]: pgmap v9418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:32 smithi160 ceph-mon[36715]: pgmap v9418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:32 smithi183 ceph-mon[37273]: pgmap v9418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:34 smithi144 ceph-mon[32129]: pgmap v9419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:34 smithi160 ceph-mon[36715]: pgmap v9419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:34 smithi183 ceph-mon[37273]: pgmap v9419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:47:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:36 smithi144 ceph-mon[32129]: pgmap v9420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:47:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16899.920337 secs 2024-07-26T15:47:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:36 smithi160 ceph-mon[36715]: pgmap v9420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:47:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16899.920337 secs 2024-07-26T15:47:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:36 smithi183 ceph-mon[37273]: pgmap v9420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:47:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16899.920337 secs 2024-07-26T15:47:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:38 smithi144 ceph-mon[32129]: pgmap v9421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:38 smithi160 ceph-mon[36715]: pgmap v9421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:38 smithi183 ceph-mon[37273]: pgmap v9421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:40 smithi144 ceph-mon[32129]: pgmap v9422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:40 smithi160 ceph-mon[36715]: pgmap v9422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:40.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:40 smithi183 ceph-mon[37273]: pgmap v9422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16904.920460 secs 2024-07-26T15:47:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16904.920460 secs 2024-07-26T15:47:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16904.920460 secs 2024-07-26T15:47:42.104 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:42.103+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:47:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:42.104+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:47:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:42.104+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:47:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:42.104+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:47:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:42.104+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:42 smithi144 ceph-mon[32129]: pgmap v9423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:42 smithi160 ceph-mon[36715]: pgmap v9423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:42 smithi183 ceph-mon[37273]: pgmap v9423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:44 smithi144 ceph-mon[32129]: pgmap v9424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:44 smithi160 ceph-mon[36715]: pgmap v9424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:44 smithi183 ceph-mon[37273]: pgmap v9424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:46 smithi144 ceph-mon[32129]: pgmap v9425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16909.920589 secs 2024-07-26T15:47:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:46 smithi160 ceph-mon[36715]: pgmap v9425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16909.920589 secs 2024-07-26T15:47:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:46 smithi183 ceph-mon[37273]: pgmap v9425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16909.920589 secs 2024-07-26T15:47:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:48 smithi144 ceph-mon[32129]: pgmap v9426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:48 smithi160 ceph-mon[36715]: pgmap v9426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:48.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:48 smithi183 ceph-mon[37273]: pgmap v9426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:49.082+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:47:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:49.082+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:47:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:49.082+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:47:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:49.082+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:47:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:50 smithi144 ceph-mon[32129]: pgmap v9427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:50 smithi160 ceph-mon[36715]: pgmap v9427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:50 smithi183 ceph-mon[37273]: pgmap v9427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16914.920676 secs 2024-07-26T15:47:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16914.920676 secs 2024-07-26T15:47:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16914.920676 secs 2024-07-26T15:47:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:52 smithi160 ceph-mon[36715]: pgmap v9428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:52 smithi183 ceph-mon[37273]: pgmap v9428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:52 smithi144 ceph-mon[32129]: pgmap v9428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:54 smithi160 ceph-mon[36715]: pgmap v9429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:54 smithi183 ceph-mon[37273]: pgmap v9429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:54 smithi144 ceph-mon[32129]: pgmap v9429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:47:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:56 smithi160 ceph-mon[36715]: pgmap v9430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16919.920803 secs 2024-07-26T15:47:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:56 smithi183 ceph-mon[37273]: pgmap v9430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16919.920803 secs 2024-07-26T15:47:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:56 smithi144 ceph-mon[32129]: pgmap v9430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:47:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16919.920803 secs 2024-07-26T15:47:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:57.120+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:47:57.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:57.121+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:47:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:57.121+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:47:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:57.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:47:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:47:57.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:47:58 smithi160 ceph-mon[36715]: pgmap v9431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:47:58 smithi183 ceph-mon[37273]: pgmap v9431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:47:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:47:58 smithi144 ceph-mon[32129]: pgmap v9431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:00 smithi160 ceph-mon[36715]: pgmap v9432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:00.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:00 smithi183 ceph-mon[37273]: pgmap v9432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:00 smithi144 ceph-mon[32129]: pgmap v9432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16924.920935 secs 2024-07-26T15:48:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16924.920935 secs 2024-07-26T15:48:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16924.920935 secs 2024-07-26T15:48:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:02 smithi160 ceph-mon[36715]: pgmap v9433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:02 smithi183 ceph-mon[37273]: pgmap v9433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:02 smithi144 ceph-mon[32129]: pgmap v9433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:04 smithi160 ceph-mon[36715]: pgmap v9434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:04 smithi183 ceph-mon[37273]: pgmap v9434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:04 smithi144 ceph-mon[32129]: pgmap v9434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:06 smithi160 ceph-mon[36715]: pgmap v9435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16929.921063 secs 2024-07-26T15:48:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:06 smithi183 ceph-mon[37273]: pgmap v9435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16929.921063 secs 2024-07-26T15:48:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:06 smithi144 ceph-mon[32129]: pgmap v9435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16929.921063 secs 2024-07-26T15:48:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:08 smithi160 ceph-mon[36715]: pgmap v9436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:08 smithi183 ceph-mon[37273]: pgmap v9436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:08 smithi144 ceph-mon[32129]: pgmap v9436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:10 smithi160 ceph-mon[36715]: pgmap v9437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:10.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:10 smithi183 ceph-mon[37273]: pgmap v9437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:10 smithi144 ceph-mon[32129]: pgmap v9437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16934.921128 secs 2024-07-26T15:48:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16934.921128 secs 2024-07-26T15:48:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16934.921128 secs 2024-07-26T15:48:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:12.137+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:48:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:12.138+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:48:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:12.138+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:48:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:12.138+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:48:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:12.138+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:12 smithi183 ceph-mon[37273]: pgmap v9438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:12 smithi144 ceph-mon[32129]: pgmap v9438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:12 smithi160 ceph-mon[36715]: pgmap v9438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:14 smithi183 ceph-mon[37273]: pgmap v9439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:14 smithi144 ceph-mon[32129]: pgmap v9439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:14 smithi160 ceph-mon[36715]: pgmap v9439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:16 smithi183 ceph-mon[37273]: pgmap v9440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16939.921251 secs 2024-07-26T15:48:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:16 smithi144 ceph-mon[32129]: pgmap v9440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16939.921251 secs 2024-07-26T15:48:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:16 smithi160 ceph-mon[36715]: pgmap v9440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16939.921251 secs 2024-07-26T15:48:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:18 smithi183 ceph-mon[37273]: pgmap v9441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:18 smithi144 ceph-mon[32129]: pgmap v9441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:18 smithi160 ceph-mon[36715]: pgmap v9441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:20 smithi183 ceph-mon[37273]: pgmap v9442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:20 smithi144 ceph-mon[32129]: pgmap v9442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:20 smithi160 ceph-mon[36715]: pgmap v9442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16944.921398 secs 2024-07-26T15:48:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16944.921398 secs 2024-07-26T15:48:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16944.921398 secs 2024-07-26T15:48:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:22 smithi183 ceph-mon[37273]: pgmap v9443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:22 smithi144 ceph-mon[32129]: pgmap v9443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:23.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:22 smithi160 ceph-mon[36715]: pgmap v9443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:24 smithi183 ceph-mon[37273]: pgmap v9444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:24 smithi144 ceph-mon[32129]: pgmap v9444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:24 smithi160 ceph-mon[36715]: pgmap v9444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:25 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:25 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:25 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:25 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:26.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:25 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:26.057 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:26.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:25 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:26.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:26 smithi183 ceph-mon[37273]: pgmap v9445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16949.921525 secs 2024-07-26T15:48:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:26 smithi144 ceph-mon[32129]: pgmap v9445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16949.921525 secs 2024-07-26T15:48:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:26 smithi160 ceph-mon[36715]: pgmap v9445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16949.921525 secs 2024-07-26T15:48:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:27.153+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:48:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:27.153+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:48:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:27.153+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:48:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:27.153+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:48:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:27.153+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:28 smithi183 ceph-mon[37273]: pgmap v9446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:28 smithi144 ceph-mon[32129]: pgmap v9446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:28 smithi160 ceph-mon[36715]: pgmap v9446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:29 smithi144 ceph-mon[32129]: pgmap v9447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:29 smithi160 ceph-mon[36715]: pgmap v9447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:29 smithi183 ceph-mon[37273]: pgmap v9447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:30.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:30 smithi144 ceph-mon[32129]: pgmap v9448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:48:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:30 smithi144 ceph-mon[32129]: pgmap v9449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:48:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:30 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:30 smithi160 ceph-mon[36715]: pgmap v9448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:48:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:30 smithi160 ceph-mon[36715]: pgmap v9449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:48:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:30 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:31.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:30 smithi183 ceph-mon[37273]: pgmap v9448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:48:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:30 smithi183 ceph-mon[37273]: pgmap v9449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:48:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:30 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:48:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16954.921649 secs 2024-07-26T15:48:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16954.921649 secs 2024-07-26T15:48:32.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16954.921649 secs 2024-07-26T15:48:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:32 smithi160 ceph-mon[36715]: pgmap v9450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:48:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:32 smithi183 ceph-mon[37273]: pgmap v9450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:48:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:32 smithi144 ceph-mon[32129]: pgmap v9450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:48:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:35 smithi183 ceph-mon[37273]: pgmap v9451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:48:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:35 smithi144 ceph-mon[32129]: pgmap v9451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:48:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:35 smithi160 ceph-mon[36715]: pgmap v9451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:48:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16959.921767 secs 2024-07-26T15:48:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:37 smithi144 ceph-mon[32129]: pgmap v9452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:48:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16959.921767 secs 2024-07-26T15:48:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:37 smithi183 ceph-mon[37273]: pgmap v9452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:48:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16959.921767 secs 2024-07-26T15:48:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:37 smithi160 ceph-mon[36715]: pgmap v9452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:48:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:39 smithi183 ceph-mon[37273]: pgmap v9453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:48:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:39 smithi144 ceph-mon[32129]: pgmap v9453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:48:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:39 smithi160 ceph-mon[36715]: pgmap v9453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:48:41.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:41 smithi183 ceph-mon[37273]: pgmap v9454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:48:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:41 smithi144 ceph-mon[32129]: pgmap v9454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:48:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:41 smithi160 ceph-mon[36715]: pgmap v9454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:48:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:42.169+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:48:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:42.170+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:48:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:42.170+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:48:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:42.170+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:48:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:42.171+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:42.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16964.921865 secs 2024-07-26T15:48:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16964.921865 secs 2024-07-26T15:48:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16964.921865 secs 2024-07-26T15:48:43.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:43 smithi183 ceph-mon[37273]: pgmap v9455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:43 smithi144 ceph-mon[32129]: pgmap v9455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:43 smithi160 ceph-mon[36715]: pgmap v9455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:45.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:45 smithi183 ceph-mon[37273]: pgmap v9456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:45 smithi144 ceph-mon[32129]: pgmap v9456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:45 smithi160 ceph-mon[36715]: pgmap v9456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16969.921988 secs 2024-07-26T15:48:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:47 smithi144 ceph-mon[32129]: pgmap v9457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16969.921988 secs 2024-07-26T15:48:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:47 smithi160 ceph-mon[36715]: pgmap v9457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16969.921988 secs 2024-07-26T15:48:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:47 smithi183 ceph-mon[37273]: pgmap v9457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:49 smithi144 ceph-mon[32129]: pgmap v9458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:49 smithi160 ceph-mon[36715]: pgmap v9458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:49 smithi183 ceph-mon[37273]: pgmap v9458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:51.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:51 smithi144 ceph-mon[32129]: pgmap v9459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:51 smithi160 ceph-mon[36715]: pgmap v9459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:51 smithi183 ceph-mon[37273]: pgmap v9459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16974.922056 secs 2024-07-26T15:48:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16974.922056 secs 2024-07-26T15:48:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16974.922056 secs 2024-07-26T15:48:53.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:53 smithi144 ceph-mon[32129]: pgmap v9460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:53 smithi160 ceph-mon[36715]: pgmap v9460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:53 smithi183 ceph-mon[37273]: pgmap v9460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:48:55.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:55 smithi144 ceph-mon[32129]: pgmap v9461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:55 smithi160 ceph-mon[36715]: pgmap v9461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:55 smithi183 ceph-mon[37273]: pgmap v9461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:57.186+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:48:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:57.187+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:48:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:57.187+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:48:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:57.187+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:48:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:48:57.188+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16979.922186 secs 2024-07-26T15:48:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:57 smithi144 ceph-mon[32129]: pgmap v9462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16979.922186 secs 2024-07-26T15:48:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:57 smithi160 ceph-mon[36715]: pgmap v9462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16979.922186 secs 2024-07-26T15:48:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:57 smithi183 ceph-mon[37273]: pgmap v9462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:48:59.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:48:59 smithi183 ceph-mon[37273]: pgmap v9463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:48:59 smithi144 ceph-mon[32129]: pgmap v9463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:48:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:48:59 smithi160 ceph-mon[36715]: pgmap v9463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:01 smithi144 ceph-mon[32129]: pgmap v9464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16984.922329 secs 2024-07-26T15:49:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:01 smithi160 ceph-mon[36715]: pgmap v9464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16984.922329 secs 2024-07-26T15:49:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:01 smithi183 ceph-mon[37273]: pgmap v9464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16984.922329 secs 2024-07-26T15:49:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:03 smithi144 ceph-mon[32129]: pgmap v9465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:03 smithi160 ceph-mon[36715]: pgmap v9465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:03.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:03 smithi183 ceph-mon[37273]: pgmap v9465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:05 smithi144 ceph-mon[32129]: pgmap v9466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:05 smithi160 ceph-mon[36715]: pgmap v9466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:05 smithi183 ceph-mon[37273]: pgmap v9466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16989.922461 secs 2024-07-26T15:49:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16989.922461 secs 2024-07-26T15:49:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16989.922461 secs 2024-07-26T15:49:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:07 smithi144 ceph-mon[32129]: pgmap v9467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:07 smithi160 ceph-mon[36715]: pgmap v9467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:07 smithi183 ceph-mon[37273]: pgmap v9467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:09 smithi144 ceph-mon[32129]: pgmap v9468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:09 smithi160 ceph-mon[36715]: pgmap v9468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:09 smithi183 ceph-mon[37273]: pgmap v9468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:11 smithi160 ceph-mon[36715]: pgmap v9469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16994.922584 secs 2024-07-26T15:49:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:11 smithi183 ceph-mon[37273]: pgmap v9469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16994.922584 secs 2024-07-26T15:49:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:11 smithi144 ceph-mon[32129]: pgmap v9469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16994.922584 secs 2024-07-26T15:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:12.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:49:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:12.205+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:49:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:12.205+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:49:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:12.205+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:49:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:12.205+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:13 smithi160 ceph-mon[36715]: pgmap v9470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:13 smithi183 ceph-mon[37273]: pgmap v9470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:13 smithi144 ceph-mon[32129]: pgmap v9470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:15 smithi160 ceph-mon[36715]: pgmap v9471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:15 smithi183 ceph-mon[37273]: pgmap v9471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:15 smithi144 ceph-mon[32129]: pgmap v9471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 16999.922672 secs 2024-07-26T15:49:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 16999.922672 secs 2024-07-26T15:49:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 16999.922672 secs 2024-07-26T15:49:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:17 smithi160 ceph-mon[36715]: pgmap v9472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:17 smithi183 ceph-mon[37273]: pgmap v9472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:17 smithi144 ceph-mon[32129]: pgmap v9472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:19 smithi160 ceph-mon[36715]: pgmap v9473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:19 smithi183 ceph-mon[37273]: pgmap v9473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:19 smithi144 ceph-mon[32129]: pgmap v9473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:21 smithi160 ceph-mon[36715]: pgmap v9474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17004.922822 secs 2024-07-26T15:49:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:21 smithi183 ceph-mon[37273]: pgmap v9474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17004.922822 secs 2024-07-26T15:49:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:21 smithi144 ceph-mon[32129]: pgmap v9474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17004.922822 secs 2024-07-26T15:49:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:23 smithi160 ceph-mon[36715]: pgmap v9475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:23 smithi183 ceph-mon[37273]: pgmap v9475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:23 smithi144 ceph-mon[32129]: pgmap v9475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:25 smithi160 ceph-mon[36715]: pgmap v9476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:25 smithi183 ceph-mon[37273]: pgmap v9476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:25 smithi144 ceph-mon[32129]: pgmap v9476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17009.922962 secs 2024-07-26T15:49:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17009.922962 secs 2024-07-26T15:49:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17009.922962 secs 2024-07-26T15:49:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:27.219+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:49:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:27.220+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:49:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:27.220+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:49:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:27.220+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:49:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:27.220+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:27 smithi160 ceph-mon[36715]: pgmap v9477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:27 smithi183 ceph-mon[37273]: pgmap v9477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:27 smithi144 ceph-mon[32129]: pgmap v9477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:29 smithi160 ceph-mon[36715]: pgmap v9478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:29 smithi183 ceph-mon[37273]: pgmap v9478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:29 smithi144 ceph-mon[32129]: pgmap v9478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:30.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:30.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:31.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:31 smithi183 ceph-mon[37273]: pgmap v9479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17014.923026 secs 2024-07-26T15:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:49:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:31 smithi144 ceph-mon[32129]: pgmap v9479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17014.923026 secs 2024-07-26T15:49:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:49:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:31 smithi160 ceph-mon[36715]: pgmap v9479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17014.923026 secs 2024-07-26T15:49:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:49:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:32 smithi183 ceph-mon[37273]: pgmap v9480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:32 smithi183 ceph-mon[37273]: pgmap v9481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:32 smithi144 ceph-mon[32129]: pgmap v9480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:32 smithi144 ceph-mon[32129]: pgmap v9481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:32 smithi160 ceph-mon[36715]: pgmap v9480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:32 smithi160 ceph-mon[36715]: pgmap v9481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:34.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:34 smithi183 ceph-mon[37273]: pgmap v9482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:34.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:34 smithi144 ceph-mon[32129]: pgmap v9482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:34 smithi160 ceph-mon[36715]: pgmap v9482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:36.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:36 smithi160 ceph-mon[36715]: pgmap v9483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:36.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17019.923167 secs 2024-07-26T15:49:36.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:36 smithi183 ceph-mon[37273]: pgmap v9483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17019.923167 secs 2024-07-26T15:49:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:36 smithi144 ceph-mon[32129]: pgmap v9483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17019.923167 secs 2024-07-26T15:49:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:38 smithi183 ceph-mon[37273]: pgmap v9484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:38.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:38 smithi144 ceph-mon[32129]: pgmap v9484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:38 smithi160 ceph-mon[36715]: pgmap v9484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:40 smithi183 ceph-mon[37273]: pgmap v9485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:40 smithi144 ceph-mon[32129]: pgmap v9485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:40 smithi160 ceph-mon[36715]: pgmap v9485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:49:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17024.923333 secs 2024-07-26T15:49:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17024.923333 secs 2024-07-26T15:49:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17024.923333 secs 2024-07-26T15:49:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:42.227+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:49:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:42.228+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:49:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:42.228+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:49:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:42.228+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:49:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:42.228+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:42.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:42 smithi183 ceph-mon[37273]: pgmap v9486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:49:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:42 smithi144 ceph-mon[32129]: pgmap v9486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:49:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:42 smithi160 ceph-mon[36715]: pgmap v9486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:49:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:44 smithi183 ceph-mon[37273]: pgmap v9487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:44 smithi144 ceph-mon[32129]: pgmap v9487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:44 smithi160 ceph-mon[36715]: pgmap v9487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:46 smithi183 ceph-mon[37273]: pgmap v9488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17029.923463 secs 2024-07-26T15:49:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:46 smithi144 ceph-mon[32129]: pgmap v9488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17029.923463 secs 2024-07-26T15:49:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:46 smithi160 ceph-mon[36715]: pgmap v9488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17029.923463 secs 2024-07-26T15:49:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:48 smithi183 ceph-mon[37273]: pgmap v9489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:48 smithi144 ceph-mon[32129]: pgmap v9489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:48 smithi160 ceph-mon[36715]: pgmap v9489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:50.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:50 smithi183 ceph-mon[37273]: pgmap v9490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:50 smithi144 ceph-mon[32129]: pgmap v9490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:50 smithi160 ceph-mon[36715]: pgmap v9490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17034.923564 secs 2024-07-26T15:49:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17034.923564 secs 2024-07-26T15:49:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17034.923564 secs 2024-07-26T15:49:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:52 smithi144 ceph-mon[32129]: pgmap v9491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:52 smithi160 ceph-mon[36715]: pgmap v9491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:52.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:52 smithi183 ceph-mon[37273]: pgmap v9491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:49:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:54 smithi144 ceph-mon[32129]: pgmap v9492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:54 smithi160 ceph-mon[36715]: pgmap v9492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:54 smithi183 ceph-mon[37273]: pgmap v9492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:56 smithi144 ceph-mon[32129]: pgmap v9493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17039.923725 secs 2024-07-26T15:49:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:56 smithi160 ceph-mon[36715]: pgmap v9493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17039.923725 secs 2024-07-26T15:49:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:56 smithi183 ceph-mon[37273]: pgmap v9493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:49:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17039.923725 secs 2024-07-26T15:49:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:57.236+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:49:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:57.237+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:49:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:57.237+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:49:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:57.237+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:49:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:49:57.238+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:49:58 smithi144 ceph-mon[32129]: pgmap v9494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:49:58 smithi160 ceph-mon[36715]: pgmap v9494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:49:58.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:49:58 smithi183 ceph-mon[37273]: pgmap v9494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:00.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:00 smithi183 ceph-mon[37273]: pgmap v9495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:00.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:50:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:00 smithi144 ceph-mon[32129]: pgmap v9495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:50:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:00 smithi160 ceph-mon[36715]: pgmap v9495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:50:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17044.923851 secs 2024-07-26T15:50:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17044.923851 secs 2024-07-26T15:50:01.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17044.923851 secs 2024-07-26T15:50:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:02 smithi144 ceph-mon[32129]: pgmap v9496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:02 smithi160 ceph-mon[36715]: pgmap v9496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:02.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:02 smithi183 ceph-mon[37273]: pgmap v9496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:04.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:04 smithi144 ceph-mon[32129]: pgmap v9497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:04 smithi160 ceph-mon[36715]: pgmap v9497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:04 smithi183 ceph-mon[37273]: pgmap v9497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:06 smithi144 ceph-mon[32129]: pgmap v9498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17049.923929 secs 2024-07-26T15:50:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:06 smithi160 ceph-mon[36715]: pgmap v9498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17049.923929 secs 2024-07-26T15:50:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:06 smithi183 ceph-mon[37273]: pgmap v9498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:06.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17049.923929 secs 2024-07-26T15:50:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:08 smithi144 ceph-mon[32129]: pgmap v9499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:08 smithi160 ceph-mon[36715]: pgmap v9499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:08 smithi183 ceph-mon[37273]: pgmap v9499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:10 smithi144 ceph-mon[32129]: pgmap v9500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:10 smithi160 ceph-mon[36715]: pgmap v9500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:10 smithi183 ceph-mon[37273]: pgmap v9500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17054.924027 secs 2024-07-26T15:50:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17054.924027 secs 2024-07-26T15:50:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17054.924027 secs 2024-07-26T15:50:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:12.242+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:12.243+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:12.243+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:12.243+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:12.243+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:12 smithi144 ceph-mon[32129]: pgmap v9501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:12 smithi160 ceph-mon[36715]: pgmap v9501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:12.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:12 smithi183 ceph-mon[37273]: pgmap v9501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:14 smithi144 ceph-mon[32129]: pgmap v9502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:14.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:14 smithi160 ceph-mon[36715]: pgmap v9502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:14.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:14 smithi183 ceph-mon[37273]: pgmap v9502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:16 smithi160 ceph-mon[36715]: pgmap v9503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17059.924153 secs 2024-07-26T15:50:16.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:16 smithi183 ceph-mon[37273]: pgmap v9503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17059.924153 secs 2024-07-26T15:50:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:16 smithi144 ceph-mon[32129]: pgmap v9503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17059.924153 secs 2024-07-26T15:50:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:18 smithi160 ceph-mon[36715]: pgmap v9504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:18 smithi183 ceph-mon[37273]: pgmap v9504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:18 smithi144 ceph-mon[32129]: pgmap v9504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:20 smithi160 ceph-mon[36715]: pgmap v9505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:20.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:20 smithi183 ceph-mon[37273]: pgmap v9505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:20 smithi144 ceph-mon[32129]: pgmap v9505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17064.924318 secs 2024-07-26T15:50:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17064.924318 secs 2024-07-26T15:50:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17064.924318 secs 2024-07-26T15:50:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:22 smithi160 ceph-mon[36715]: pgmap v9506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:22 smithi183 ceph-mon[37273]: pgmap v9506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:22 smithi144 ceph-mon[32129]: pgmap v9506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:24 smithi160 ceph-mon[36715]: pgmap v9507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:24.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:24 smithi183 ceph-mon[37273]: pgmap v9507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:24 smithi144 ceph-mon[32129]: pgmap v9507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:26 smithi160 ceph-mon[36715]: pgmap v9508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17069.924381 secs 2024-07-26T15:50:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:26 smithi183 ceph-mon[37273]: pgmap v9508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17069.924381 secs 2024-07-26T15:50:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:26 smithi144 ceph-mon[32129]: pgmap v9508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17069.924381 secs 2024-07-26T15:50:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:27.258+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:50:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:27.259+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:50:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:27.259+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:50:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:27.259+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:50:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:27.260+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:28 smithi160 ceph-mon[36715]: pgmap v9509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:28 smithi183 ceph-mon[37273]: pgmap v9509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:28 smithi144 ceph-mon[32129]: pgmap v9509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:30 smithi160 ceph-mon[36715]: pgmap v9510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:30 smithi183 ceph-mon[37273]: pgmap v9510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:30 smithi144 ceph-mon[32129]: pgmap v9510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:31.759 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17074.924483 secs 2024-07-26T15:50:31.759 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17074.924483 secs 2024-07-26T15:50:31.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17074.924483 secs 2024-07-26T15:50:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:32 smithi160 ceph-mon[36715]: pgmap v9511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:50:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:50:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:50:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:32 smithi183 ceph-mon[37273]: pgmap v9511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:50:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:50:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:50:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:32 smithi144 ceph-mon[32129]: pgmap v9511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:50:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:50:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:50:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:33 smithi160 ceph-mon[36715]: pgmap v9512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:50:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:33 smithi160 ceph-mon[36715]: pgmap v9513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:50:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:33 smithi183 ceph-mon[37273]: pgmap v9512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:50:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:33 smithi183 ceph-mon[37273]: pgmap v9513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:50:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:33 smithi144 ceph-mon[32129]: pgmap v9512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:50:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:33 smithi144 ceph-mon[32129]: pgmap v9513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:50:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:35 smithi160 ceph-mon[36715]: pgmap v9514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:50:35.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:35 smithi183 ceph-mon[37273]: pgmap v9514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:50:35.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:35 smithi144 ceph-mon[32129]: pgmap v9514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:50:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17079.924629 secs 2024-07-26T15:50:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17079.924629 secs 2024-07-26T15:50:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17079.924629 secs 2024-07-26T15:50:37.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:37 smithi183 ceph-mon[37273]: pgmap v9515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:50:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:37 smithi144 ceph-mon[32129]: pgmap v9515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:50:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:37 smithi160 ceph-mon[36715]: pgmap v9515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:50:39.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:39 smithi183 ceph-mon[37273]: pgmap v9516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:50:39.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:39 smithi144 ceph-mon[32129]: pgmap v9516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:50:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:39 smithi160 ceph-mon[36715]: pgmap v9516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:50:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:41 smithi183 ceph-mon[37273]: pgmap v9517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:50:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17084.924772 secs 2024-07-26T15:50:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:41 smithi144 ceph-mon[32129]: pgmap v9517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:50:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17084.924772 secs 2024-07-26T15:50:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:41 smithi160 ceph-mon[36715]: pgmap v9517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:50:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17084.924772 secs 2024-07-26T15:50:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:42.275+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:50:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:42.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:50:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:42.276+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:50:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:42.276+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:50:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:42.277+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:43 smithi183 ceph-mon[37273]: pgmap v9518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:50:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:43 smithi144 ceph-mon[32129]: pgmap v9518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:50:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:43 smithi160 ceph-mon[36715]: pgmap v9518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:50:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:45 smithi183 ceph-mon[37273]: pgmap v9519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:45 smithi144 ceph-mon[32129]: pgmap v9519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:45 smithi160 ceph-mon[36715]: pgmap v9519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:50:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:50:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5e60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74a804c040 tx=0x7f74a8059040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:50:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5e60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74a804c040 tx=0x7f74a8059040 comp rx=0 tx=0).stop 2024-07-26T15:50:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5e60 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T15:50:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:50:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5e60 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:50:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.154+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:50:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.155+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:50:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:46.157+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a803e040 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:50:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17089.924915 secs 2024-07-26T15:50:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17089.924915 secs 2024-07-26T15:50:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17089.924915 secs 2024-07-26T15:50:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:47 smithi183 ceph-mon[37273]: pgmap v9520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:47 smithi144 ceph-mon[32129]: pgmap v9520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:47 smithi160 ceph-mon[36715]: pgmap v9520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:49.114+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:50:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:49.114+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:50:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:49.115+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:50:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:49.115+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:50:49.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:49 smithi183 ceph-mon[37273]: pgmap v9521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:49 smithi144 ceph-mon[32129]: pgmap v9521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:49 smithi160 ceph-mon[36715]: pgmap v9521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:51 smithi183 ceph-mon[37273]: pgmap v9522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17094.925013 secs 2024-07-26T15:50:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:51 smithi144 ceph-mon[32129]: pgmap v9522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17094.925013 secs 2024-07-26T15:50:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:51 smithi160 ceph-mon[36715]: pgmap v9522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17094.925013 secs 2024-07-26T15:50:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:53 smithi183 ceph-mon[37273]: pgmap v9523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:53 smithi144 ceph-mon[32129]: pgmap v9523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:53 smithi160 ceph-mon[36715]: pgmap v9523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:50:55.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:55 smithi183 ceph-mon[37273]: pgmap v9524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:55.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:55 smithi144 ceph-mon[32129]: pgmap v9524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:55 smithi160 ceph-mon[36715]: pgmap v9524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:50:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17099.925136 secs 2024-07-26T15:50:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17099.925136 secs 2024-07-26T15:50:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17099.925136 secs 2024-07-26T15:50:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:57.288+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:50:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:57.289+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:50:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:57.289+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:50:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:57.289+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:50:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:50:57.289+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:57 smithi144 ceph-mon[32129]: pgmap v9525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:57 smithi160 ceph-mon[36715]: pgmap v9525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:57 smithi183 ceph-mon[37273]: pgmap v9525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:50:59.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:50:59 smithi144 ceph-mon[32129]: pgmap v9526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:50:59 smithi160 ceph-mon[36715]: pgmap v9526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:50:59 smithi183 ceph-mon[37273]: pgmap v9526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:01.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:01 smithi183 ceph-mon[37273]: pgmap v9527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:01.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17104.925274 secs 2024-07-26T15:51:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:01 smithi144 ceph-mon[32129]: pgmap v9527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17104.925274 secs 2024-07-26T15:51:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:01 smithi160 ceph-mon[36715]: pgmap v9527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17104.925274 secs 2024-07-26T15:51:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:03 smithi144 ceph-mon[32129]: pgmap v9528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:03 smithi160 ceph-mon[36715]: pgmap v9528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:04.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:03 smithi183 ceph-mon[37273]: pgmap v9528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:05 smithi144 ceph-mon[32129]: pgmap v9529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:05 smithi160 ceph-mon[36715]: pgmap v9529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:05 smithi183 ceph-mon[37273]: pgmap v9529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:06.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17109.925405 secs 2024-07-26T15:51:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17109.925405 secs 2024-07-26T15:51:07.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17109.925405 secs 2024-07-26T15:51:07.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:07 smithi144 ceph-mon[32129]: pgmap v9530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:07 smithi160 ceph-mon[36715]: pgmap v9530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:07 smithi183 ceph-mon[37273]: pgmap v9530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:09 smithi144 ceph-mon[32129]: pgmap v9531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:09 smithi160 ceph-mon[36715]: pgmap v9531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:09 smithi183 ceph-mon[37273]: pgmap v9531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:11 smithi144 ceph-mon[32129]: pgmap v9532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17114.925534 secs 2024-07-26T15:51:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:11 smithi160 ceph-mon[36715]: pgmap v9532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17114.925534 secs 2024-07-26T15:51:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:11 smithi183 ceph-mon[37273]: pgmap v9532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17114.925534 secs 2024-07-26T15:51:12.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:12.304+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:51:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:12.306+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:51:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:12.306+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:51:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:12.306+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:51:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:12.306+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:13.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:13 smithi144 ceph-mon[32129]: pgmap v9533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:13 smithi160 ceph-mon[36715]: pgmap v9533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:14.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:13 smithi183 ceph-mon[37273]: pgmap v9533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:15.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:15 smithi144 ceph-mon[32129]: pgmap v9534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:15 smithi160 ceph-mon[36715]: pgmap v9534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:16.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:15 smithi183 ceph-mon[37273]: pgmap v9534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17119.925653 secs 2024-07-26T15:51:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17119.925653 secs 2024-07-26T15:51:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17119.925653 secs 2024-07-26T15:51:17.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:17 smithi144 ceph-mon[32129]: pgmap v9535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:17 smithi160 ceph-mon[36715]: pgmap v9535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:17 smithi183 ceph-mon[37273]: pgmap v9535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:19 smithi144 ceph-mon[32129]: pgmap v9536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:19 smithi160 ceph-mon[36715]: pgmap v9536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:19 smithi183 ceph-mon[37273]: pgmap v9536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:21 smithi160 ceph-mon[36715]: pgmap v9537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17124.925750 secs 2024-07-26T15:51:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:21 smithi183 ceph-mon[37273]: pgmap v9537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:22.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17124.925750 secs 2024-07-26T15:51:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:21 smithi144 ceph-mon[32129]: pgmap v9537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17124.925750 secs 2024-07-26T15:51:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:23 smithi160 ceph-mon[36715]: pgmap v9538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:23 smithi183 ceph-mon[37273]: pgmap v9538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:23 smithi144 ceph-mon[32129]: pgmap v9538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:25 smithi160 ceph-mon[36715]: pgmap v9539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:26.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:25 smithi183 ceph-mon[37273]: pgmap v9539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:25 smithi144 ceph-mon[32129]: pgmap v9539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17129.925816 secs 2024-07-26T15:51:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17129.925816 secs 2024-07-26T15:51:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17129.925816 secs 2024-07-26T15:51:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:27.322+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:51:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:27.323+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:51:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:27.323+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:51:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:27.323+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:51:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:27.323+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:27 smithi160 ceph-mon[36715]: pgmap v9540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:28.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:27 smithi144 ceph-mon[32129]: pgmap v9540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:27 smithi183 ceph-mon[37273]: pgmap v9540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:29 smithi144 ceph-mon[32129]: pgmap v9541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:29 smithi160 ceph-mon[36715]: pgmap v9541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:29.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:29 smithi183 ceph-mon[37273]: pgmap v9541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:31 smithi160 ceph-mon[36715]: pgmap v9542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17134.925911 secs 2024-07-26T15:51:31.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:31 smithi183 ceph-mon[37273]: pgmap v9542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17134.925911 secs 2024-07-26T15:51:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:31 smithi144 ceph-mon[32129]: pgmap v9542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17134.925911 secs 2024-07-26T15:51:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:33 smithi160 ceph-mon[36715]: pgmap v9543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:33.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:33 smithi183 ceph-mon[37273]: pgmap v9543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:33 smithi144 ceph-mon[32129]: pgmap v9543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:33.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:34 smithi183 ceph-mon[37273]: pgmap v9544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:34 smithi183 ceph-mon[37273]: pgmap v9545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:51:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:34 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:51:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:34 smithi144 ceph-mon[32129]: pgmap v9544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:34 smithi144 ceph-mon[32129]: pgmap v9545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:51:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:34 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:51:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:34 smithi160 ceph-mon[36715]: pgmap v9544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:34 smithi160 ceph-mon[36715]: pgmap v9545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:51:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:34 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:51:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:36 smithi160 ceph-mon[36715]: pgmap v9546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17139.926001 secs 2024-07-26T15:51:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:36 smithi183 ceph-mon[37273]: pgmap v9546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17139.926001 secs 2024-07-26T15:51:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:36 smithi144 ceph-mon[32129]: pgmap v9546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17139.926001 secs 2024-07-26T15:51:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:38 smithi160 ceph-mon[36715]: pgmap v9547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:51:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:38 smithi183 ceph-mon[37273]: pgmap v9547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:51:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:38 smithi144 ceph-mon[32129]: pgmap v9547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:51:41.022 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:40 smithi144 ceph-mon[32129]: pgmap v9548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:51:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:40 smithi160 ceph-mon[36715]: pgmap v9548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:51:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:40 smithi183 ceph-mon[37273]: pgmap v9548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:51:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17144.926121 secs 2024-07-26T15:51:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17144.926121 secs 2024-07-26T15:51:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17144.926121 secs 2024-07-26T15:51:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:42.331+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:51:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:42.332+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:51:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:42.332+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:51:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:42.332+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:51:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:42.332+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:42 smithi160 ceph-mon[36715]: pgmap v9549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:42 smithi183 ceph-mon[37273]: pgmap v9549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:42 smithi144 ceph-mon[32129]: pgmap v9549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:51:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:44 smithi160 ceph-mon[36715]: pgmap v9550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:51:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:44 smithi183 ceph-mon[37273]: pgmap v9550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:51:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:44 smithi144 ceph-mon[32129]: pgmap v9550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:51:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:46 smithi160 ceph-mon[36715]: pgmap v9551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17149.926261 secs 2024-07-26T15:51:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:46 smithi183 ceph-mon[37273]: pgmap v9551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17149.926261 secs 2024-07-26T15:51:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:46 smithi144 ceph-mon[32129]: pgmap v9551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17149.926261 secs 2024-07-26T15:51:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:48 smithi160 ceph-mon[36715]: pgmap v9552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:48 smithi183 ceph-mon[37273]: pgmap v9552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:48 smithi144 ceph-mon[32129]: pgmap v9552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:50 smithi160 ceph-mon[36715]: pgmap v9553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:50 smithi183 ceph-mon[37273]: pgmap v9553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:50 smithi144 ceph-mon[32129]: pgmap v9553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17154.926366 secs 2024-07-26T15:51:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17154.926366 secs 2024-07-26T15:51:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17154.926366 secs 2024-07-26T15:51:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:52 smithi160 ceph-mon[36715]: pgmap v9554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:52 smithi183 ceph-mon[37273]: pgmap v9554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:52 smithi144 ceph-mon[32129]: pgmap v9554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:51:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:54 smithi183 ceph-mon[37273]: pgmap v9555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:54 smithi144 ceph-mon[32129]: pgmap v9555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:54 smithi160 ceph-mon[36715]: pgmap v9555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:56 smithi183 ceph-mon[37273]: pgmap v9556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17159.926432 secs 2024-07-26T15:51:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:56 smithi144 ceph-mon[32129]: pgmap v9556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17159.926432 secs 2024-07-26T15:51:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:56 smithi160 ceph-mon[36715]: pgmap v9556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:51:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17159.926432 secs 2024-07-26T15:51:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:57.348+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:51:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:57.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:51:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:57.349+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:51:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:57.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:51:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:51:57.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:59.134 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:51:58 smithi144 ceph-mon[32129]: pgmap v9557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:51:58 smithi183 ceph-mon[37273]: pgmap v9557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:51:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:51:58 smithi160 ceph-mon[36715]: pgmap v9557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:00 smithi183 ceph-mon[37273]: pgmap v9558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:00 smithi144 ceph-mon[32129]: pgmap v9558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:00 smithi160 ceph-mon[36715]: pgmap v9558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17164.926554 secs 2024-07-26T15:52:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17164.926554 secs 2024-07-26T15:52:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17164.926554 secs 2024-07-26T15:52:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:02 smithi183 ceph-mon[37273]: pgmap v9559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:02 smithi144 ceph-mon[32129]: pgmap v9559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:02 smithi160 ceph-mon[36715]: pgmap v9559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:03 smithi144 ceph-mon[32129]: pgmap v9560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:03 smithi160 ceph-mon[36715]: pgmap v9560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:03 smithi183 ceph-mon[37273]: pgmap v9560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:06 smithi160 ceph-mon[36715]: pgmap v9561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17169.926669 secs 2024-07-26T15:52:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:06 smithi183 ceph-mon[37273]: pgmap v9561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17169.926669 secs 2024-07-26T15:52:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:06 smithi144 ceph-mon[32129]: pgmap v9561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17169.926669 secs 2024-07-26T15:52:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:08 smithi160 ceph-mon[36715]: pgmap v9562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:08 smithi183 ceph-mon[37273]: pgmap v9562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:08 smithi144 ceph-mon[32129]: pgmap v9562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:10 smithi160 ceph-mon[36715]: pgmap v9563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:52:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:10 smithi183 ceph-mon[37273]: pgmap v9563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:52:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:10 smithi144 ceph-mon[32129]: pgmap v9563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:52:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17174.926792 secs 2024-07-26T15:52:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17174.926792 secs 2024-07-26T15:52:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17174.926792 secs 2024-07-26T15:52:12.366 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:12.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:12.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:12.365+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:52:12.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:12.365+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:52:12.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:12.365+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:52:12.367 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:12.366+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:12 smithi160 ceph-mon[36715]: pgmap v9564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:13.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:12 smithi183 ceph-mon[37273]: pgmap v9564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:12 smithi144 ceph-mon[32129]: pgmap v9564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:14 smithi160 ceph-mon[36715]: pgmap v9565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:14 smithi183 ceph-mon[37273]: pgmap v9565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:14 smithi144 ceph-mon[32129]: pgmap v9565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:16 smithi160 ceph-mon[36715]: pgmap v9566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17179.926909 secs 2024-07-26T15:52:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:16 smithi183 ceph-mon[37273]: pgmap v9566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17179.926909 secs 2024-07-26T15:52:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:16 smithi144 ceph-mon[32129]: pgmap v9566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17179.926909 secs 2024-07-26T15:52:19.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:18 smithi183 ceph-mon[37273]: pgmap v9567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:18 smithi144 ceph-mon[32129]: pgmap v9567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:18 smithi160 ceph-mon[36715]: pgmap v9567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:20 smithi183 ceph-mon[37273]: pgmap v9568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:52:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:20 smithi144 ceph-mon[32129]: pgmap v9568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:52:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:20 smithi160 ceph-mon[36715]: pgmap v9568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:52:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17184.926984 secs 2024-07-26T15:52:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17184.926984 secs 2024-07-26T15:52:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17184.926984 secs 2024-07-26T15:52:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:22 smithi183 ceph-mon[37273]: pgmap v9569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:22 smithi144 ceph-mon[32129]: pgmap v9569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:22 smithi160 ceph-mon[36715]: pgmap v9569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:24 smithi183 ceph-mon[37273]: pgmap v9570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:24 smithi144 ceph-mon[32129]: pgmap v9570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:24 smithi160 ceph-mon[36715]: pgmap v9570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:27.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:26 smithi183 ceph-mon[37273]: pgmap v9571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17189.927114 secs 2024-07-26T15:52:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:26 smithi144 ceph-mon[32129]: pgmap v9571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17189.927114 secs 2024-07-26T15:52:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:26 smithi160 ceph-mon[36715]: pgmap v9571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17189.927114 secs 2024-07-26T15:52:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:27.381+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:27.382+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:52:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:27.382+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:52:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:27.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:52:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:27.382+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:29.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:28 smithi183 ceph-mon[37273]: pgmap v9572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:28 smithi144 ceph-mon[32129]: pgmap v9572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:28 smithi160 ceph-mon[36715]: pgmap v9572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:31.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:30 smithi183 ceph-mon[37273]: pgmap v9573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:30 smithi144 ceph-mon[32129]: pgmap v9573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:30 smithi160 ceph-mon[36715]: pgmap v9573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17194.927256 secs 2024-07-26T15:52:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:31 smithi183 ceph-mon[37273]: pgmap v9574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17194.927256 secs 2024-07-26T15:52:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:31 smithi144 ceph-mon[32129]: pgmap v9574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17194.927256 secs 2024-07-26T15:52:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:31 smithi160 ceph-mon[36715]: pgmap v9574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:34 smithi160 ceph-mon[36715]: pgmap v9575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:35.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:34 smithi183 ceph-mon[37273]: pgmap v9575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:34 smithi144 ceph-mon[32129]: pgmap v9575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:52:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:36.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:52:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:52:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:36 smithi160 ceph-mon[36715]: pgmap v9576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:52:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:36 smithi160 ceph-mon[36715]: pgmap v9577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17199.927377 secs 2024-07-26T15:52:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:36 smithi183 ceph-mon[37273]: pgmap v9576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:52:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:36 smithi183 ceph-mon[37273]: pgmap v9577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17199.927377 secs 2024-07-26T15:52:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:36 smithi144 ceph-mon[32129]: pgmap v9576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:52:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:36 smithi144 ceph-mon[32129]: pgmap v9577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17199.927377 secs 2024-07-26T15:52:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:38 smithi160 ceph-mon[36715]: pgmap v9578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:52:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:38 smithi183 ceph-mon[37273]: pgmap v9578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:52:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:38 smithi144 ceph-mon[32129]: pgmap v9578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:52:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:40 smithi160 ceph-mon[36715]: pgmap v9579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:52:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:40 smithi183 ceph-mon[37273]: pgmap v9579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:52:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:40 smithi144 ceph-mon[32129]: pgmap v9579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:52:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17204.927492 secs 2024-07-26T15:52:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17204.927492 secs 2024-07-26T15:52:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17204.927492 secs 2024-07-26T15:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:42.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:42.399+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:42.399+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:42.399+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:52:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:42.399+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:42 smithi183 ceph-mon[37273]: pgmap v9580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:52:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:42 smithi144 ceph-mon[32129]: pgmap v9580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:52:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:42 smithi160 ceph-mon[36715]: pgmap v9580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:52:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:44 smithi183 ceph-mon[37273]: pgmap v9581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:45.214 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:44 smithi144 ceph-mon[32129]: pgmap v9581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:44 smithi160 ceph-mon[36715]: pgmap v9581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:52:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:46 smithi183 ceph-mon[37273]: pgmap v9582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17209.927632 secs 2024-07-26T15:52:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:46 smithi144 ceph-mon[32129]: pgmap v9582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17209.927632 secs 2024-07-26T15:52:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:46 smithi160 ceph-mon[36715]: pgmap v9582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:52:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17209.927632 secs 2024-07-26T15:52:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:48 smithi183 ceph-mon[37273]: pgmap v9583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:48 smithi144 ceph-mon[32129]: pgmap v9583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:48 smithi160 ceph-mon[36715]: pgmap v9583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:50 smithi183 ceph-mon[37273]: pgmap v9584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:50 smithi144 ceph-mon[32129]: pgmap v9584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:51.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:50 smithi160 ceph-mon[36715]: pgmap v9584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17214.927766 secs 2024-07-26T15:52:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17214.927766 secs 2024-07-26T15:52:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17214.927766 secs 2024-07-26T15:52:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:52 smithi183 ceph-mon[37273]: pgmap v9585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:52 smithi144 ceph-mon[32129]: pgmap v9585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:52 smithi160 ceph-mon[36715]: pgmap v9585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:54 smithi183 ceph-mon[37273]: pgmap v9586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:54 smithi144 ceph-mon[32129]: pgmap v9586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:54 smithi160 ceph-mon[36715]: pgmap v9586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:52:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:55 smithi183 ceph-mon[37273]: pgmap v9587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:55 smithi144 ceph-mon[32129]: pgmap v9587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:55 smithi160 ceph-mon[36715]: pgmap v9587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:52:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17219.927882 secs 2024-07-26T15:52:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17219.927882 secs 2024-07-26T15:52:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17219.927882 secs 2024-07-26T15:52:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:57.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:57.416+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:52:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:57.416+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:52:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:57.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:52:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:52:57.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:57 smithi144 ceph-mon[32129]: pgmap v9588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:57 smithi160 ceph-mon[36715]: pgmap v9588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:52:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:57 smithi183 ceph-mon[37273]: pgmap v9588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:52:59 smithi144 ceph-mon[32129]: pgmap v9589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:52:59 smithi160 ceph-mon[36715]: pgmap v9589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:52:59 smithi183 ceph-mon[37273]: pgmap v9589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:01 smithi144 ceph-mon[32129]: pgmap v9590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17224.927976 secs 2024-07-26T15:53:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:01 smithi160 ceph-mon[36715]: pgmap v9590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17224.927976 secs 2024-07-26T15:53:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:01 smithi183 ceph-mon[37273]: pgmap v9590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17224.927976 secs 2024-07-26T15:53:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:03 smithi160 ceph-mon[36715]: pgmap v9591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:04.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:03 smithi183 ceph-mon[37273]: pgmap v9591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:03 smithi144 ceph-mon[32129]: pgmap v9591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:05 smithi160 ceph-mon[36715]: pgmap v9592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:06.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:05 smithi183 ceph-mon[37273]: pgmap v9592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:05 smithi144 ceph-mon[32129]: pgmap v9592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17229.928097 secs 2024-07-26T15:53:07.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17229.928097 secs 2024-07-26T15:53:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17229.928097 secs 2024-07-26T15:53:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:07 smithi160 ceph-mon[36715]: pgmap v9593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:08.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:07 smithi183 ceph-mon[37273]: pgmap v9593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:07 smithi144 ceph-mon[32129]: pgmap v9593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:09 smithi160 ceph-mon[36715]: pgmap v9594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:10.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:09 smithi183 ceph-mon[37273]: pgmap v9594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:09 smithi144 ceph-mon[32129]: pgmap v9594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:12.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:12 smithi144 ceph-mon[32129]: pgmap v9595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:12.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17234.928221 secs 2024-07-26T15:53:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:12 smithi160 ceph-mon[36715]: pgmap v9595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17234.928221 secs 2024-07-26T15:53:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:12 smithi183 ceph-mon[37273]: pgmap v9595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17234.928221 secs 2024-07-26T15:53:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:12.431+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:53:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:12.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:53:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:12.432+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:53:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:12.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:53:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:12.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:14 smithi160 ceph-mon[36715]: pgmap v9596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:14.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:14 smithi183 ceph-mon[37273]: pgmap v9596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:14 smithi144 ceph-mon[32129]: pgmap v9596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:16 smithi160 ceph-mon[36715]: pgmap v9597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:16.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:16 smithi183 ceph-mon[37273]: pgmap v9597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:16 smithi144 ceph-mon[32129]: pgmap v9597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17239.928442 secs 2024-07-26T15:53:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17239.928442 secs 2024-07-26T15:53:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17239.928442 secs 2024-07-26T15:53:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:18 smithi160 ceph-mon[36715]: pgmap v9598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:18 smithi183 ceph-mon[37273]: pgmap v9598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:18 smithi144 ceph-mon[32129]: pgmap v9598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:20 smithi160 ceph-mon[36715]: pgmap v9599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:20 smithi183 ceph-mon[37273]: pgmap v9599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:20 smithi144 ceph-mon[32129]: pgmap v9599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:22 smithi160 ceph-mon[36715]: pgmap v9600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17244.928527 secs 2024-07-26T15:53:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:22 smithi183 ceph-mon[37273]: pgmap v9600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:22.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17244.928527 secs 2024-07-26T15:53:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:22 smithi144 ceph-mon[32129]: pgmap v9600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17244.928527 secs 2024-07-26T15:53:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:24 smithi160 ceph-mon[36715]: pgmap v9601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:24.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:24 smithi183 ceph-mon[37273]: pgmap v9601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:24 smithi144 ceph-mon[32129]: pgmap v9601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:26 smithi160 ceph-mon[36715]: pgmap v9602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:26 smithi183 ceph-mon[37273]: pgmap v9602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:26 smithi144 ceph-mon[32129]: pgmap v9602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17249.928658 secs 2024-07-26T15:53:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:27.436+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:53:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:27.437+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:53:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:27.437+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:53:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:27.437+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:53:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:27.437+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17249.928658 secs 2024-07-26T15:53:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17249.928658 secs 2024-07-26T15:53:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:28 smithi183 ceph-mon[37273]: pgmap v9603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:28 smithi144 ceph-mon[32129]: pgmap v9603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:28 smithi160 ceph-mon[36715]: pgmap v9603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:30 smithi183 ceph-mon[37273]: pgmap v9604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:30 smithi144 ceph-mon[32129]: pgmap v9604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:30 smithi160 ceph-mon[36715]: pgmap v9604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:32 smithi183 ceph-mon[37273]: pgmap v9605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17254.928806 secs 2024-07-26T15:53:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:32 smithi144 ceph-mon[32129]: pgmap v9605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17254.928806 secs 2024-07-26T15:53:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:32 smithi160 ceph-mon[36715]: pgmap v9605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17254.928806 secs 2024-07-26T15:53:34.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:34 smithi183 ceph-mon[37273]: pgmap v9606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:34 smithi144 ceph-mon[32129]: pgmap v9606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:34 smithi160 ceph-mon[36715]: pgmap v9606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:36 smithi183 ceph-mon[37273]: pgmap v9607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:36 smithi144 ceph-mon[32129]: pgmap v9607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:36 smithi160 ceph-mon[36715]: pgmap v9607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17259.928893 secs 2024-07-26T15:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:37 smithi183 ceph-mon[37273]: pgmap v9608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:37 smithi183 ceph-mon[37273]: pgmap v9609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:53:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17259.928893 secs 2024-07-26T15:53:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:37 smithi144 ceph-mon[32129]: pgmap v9608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:37 smithi144 ceph-mon[32129]: pgmap v9609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:53:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17259.928893 secs 2024-07-26T15:53:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:37 smithi160 ceph-mon[36715]: pgmap v9608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:37 smithi160 ceph-mon[36715]: pgmap v9609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:53:39.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:39 smithi144 ceph-mon[32129]: pgmap v9610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:53:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:39 smithi160 ceph-mon[36715]: pgmap v9610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:53:39.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:39 smithi183 ceph-mon[37273]: pgmap v9610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:53:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:41 smithi144 ceph-mon[32129]: pgmap v9611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:53:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:41 smithi160 ceph-mon[36715]: pgmap v9611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:53:41.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:41 smithi183 ceph-mon[37273]: pgmap v9611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:53:42.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:42.453+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:53:42.455 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:42.454+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:53:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:42.454+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:53:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:42.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:53:42.456 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:42.454+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17264.929016 secs 2024-07-26T15:53:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17264.929016 secs 2024-07-26T15:53:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17264.929016 secs 2024-07-26T15:53:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:43 smithi144 ceph-mon[32129]: pgmap v9612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:43 smithi160 ceph-mon[36715]: pgmap v9612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:43 smithi183 ceph-mon[37273]: pgmap v9612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:45 smithi144 ceph-mon[32129]: pgmap v9613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:45 smithi160 ceph-mon[36715]: pgmap v9613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:45.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:45 smithi183 ceph-mon[37273]: pgmap v9613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:53:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17269.929136 secs 2024-07-26T15:53:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:47 smithi144 ceph-mon[32129]: pgmap v9614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:53:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17269.929136 secs 2024-07-26T15:53:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:47 smithi160 ceph-mon[36715]: pgmap v9614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:53:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17269.929136 secs 2024-07-26T15:53:47.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:47 smithi183 ceph-mon[37273]: pgmap v9614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:53:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:49.147+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:53:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:49.147+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:53:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:49.147+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:53:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:49.147+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:53:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:49 smithi144 ceph-mon[32129]: pgmap v9615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:49 smithi160 ceph-mon[36715]: pgmap v9615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:49.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:49 smithi183 ceph-mon[37273]: pgmap v9615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:51 smithi160 ceph-mon[36715]: pgmap v9616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:51.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:51 smithi183 ceph-mon[37273]: pgmap v9616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:51.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:51 smithi144 ceph-mon[32129]: pgmap v9616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17274.929277 secs 2024-07-26T15:53:52.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17274.929277 secs 2024-07-26T15:53:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17274.929277 secs 2024-07-26T15:53:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:53 smithi160 ceph-mon[36715]: pgmap v9617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:53.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:53 smithi183 ceph-mon[37273]: pgmap v9617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:53.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:53 smithi144 ceph-mon[32129]: pgmap v9617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:55 smithi160 ceph-mon[36715]: pgmap v9618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:55 smithi183 ceph-mon[37273]: pgmap v9618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:55.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:55 smithi144 ceph-mon[32129]: pgmap v9618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:53:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17279.929391 secs 2024-07-26T15:53:56.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17279.929391 secs 2024-07-26T15:53:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17279.929391 secs 2024-07-26T15:53:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:57.464+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:53:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:57.465+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:53:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:57.465+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:53:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:57.465+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:53:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:53:57.465+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:57 smithi160 ceph-mon[36715]: pgmap v9619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:57.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:57 smithi183 ceph-mon[37273]: pgmap v9619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:57 smithi144 ceph-mon[32129]: pgmap v9619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:53:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:53:59 smithi160 ceph-mon[36715]: pgmap v9620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:59.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:53:59 smithi183 ceph-mon[37273]: pgmap v9620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:53:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:53:59 smithi144 ceph-mon[32129]: pgmap v9620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:01 smithi160 ceph-mon[36715]: pgmap v9621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17284.929498 secs 2024-07-26T15:54:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:01 smithi183 ceph-mon[37273]: pgmap v9621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17284.929498 secs 2024-07-26T15:54:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:01 smithi144 ceph-mon[32129]: pgmap v9621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17284.929498 secs 2024-07-26T15:54:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:03 smithi144 ceph-mon[32129]: pgmap v9622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:03 smithi160 ceph-mon[36715]: pgmap v9622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:03 smithi183 ceph-mon[37273]: pgmap v9622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:05 smithi160 ceph-mon[36715]: pgmap v9623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:05.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:05 smithi183 ceph-mon[37273]: pgmap v9623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:05 smithi144 ceph-mon[32129]: pgmap v9623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17289.929607 secs 2024-07-26T15:54:06.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17289.929607 secs 2024-07-26T15:54:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17289.929607 secs 2024-07-26T15:54:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:07 smithi160 ceph-mon[36715]: pgmap v9624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:07.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:07 smithi183 ceph-mon[37273]: pgmap v9624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:07 smithi144 ceph-mon[32129]: pgmap v9624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:09 smithi160 ceph-mon[36715]: pgmap v9625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:09.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:09 smithi183 ceph-mon[37273]: pgmap v9625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:09.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:09 smithi144 ceph-mon[32129]: pgmap v9625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:11 smithi160 ceph-mon[36715]: pgmap v9626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17294.929742 secs 2024-07-26T15:54:11.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:11 smithi183 ceph-mon[37273]: pgmap v9626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17294.929742 secs 2024-07-26T15:54:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:11 smithi144 ceph-mon[32129]: pgmap v9626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17294.929742 secs 2024-07-26T15:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:12.479+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:12.480+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:12.480+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:12.480+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:12.480+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:13.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:13 smithi183 ceph-mon[37273]: pgmap v9627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:13.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:13 smithi144 ceph-mon[32129]: pgmap v9627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:13.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:13 smithi160 ceph-mon[36715]: pgmap v9627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:15.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:15 smithi183 ceph-mon[37273]: pgmap v9628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:15.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:15 smithi144 ceph-mon[32129]: pgmap v9628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:15.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:15 smithi160 ceph-mon[36715]: pgmap v9628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:16.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17299.929884 secs 2024-07-26T15:54:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17299.929884 secs 2024-07-26T15:54:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17299.929884 secs 2024-07-26T15:54:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:17 smithi183 ceph-mon[37273]: pgmap v9629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:17.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:17 smithi144 ceph-mon[32129]: pgmap v9629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:17 smithi160 ceph-mon[36715]: pgmap v9629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:19.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:19 smithi183 ceph-mon[37273]: pgmap v9630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:19.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:19 smithi144 ceph-mon[32129]: pgmap v9630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:19.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:19 smithi160 ceph-mon[36715]: pgmap v9630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:21 smithi183 ceph-mon[37273]: pgmap v9631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17304.930001 secs 2024-07-26T15:54:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:21 smithi144 ceph-mon[32129]: pgmap v9631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17304.930001 secs 2024-07-26T15:54:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:21 smithi160 ceph-mon[36715]: pgmap v9631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17304.930001 secs 2024-07-26T15:54:23.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:23 smithi183 ceph-mon[37273]: pgmap v9632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:23 smithi144 ceph-mon[32129]: pgmap v9632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:23.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:23 smithi160 ceph-mon[36715]: pgmap v9632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:25.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:25 smithi183 ceph-mon[37273]: pgmap v9633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:54:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:25 smithi144 ceph-mon[32129]: pgmap v9633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:54:25.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:25 smithi160 ceph-mon[36715]: pgmap v9633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:54:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17309.930149 secs 2024-07-26T15:54:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17309.930149 secs 2024-07-26T15:54:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17309.930149 secs 2024-07-26T15:54:27.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:27.487+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:54:27.489 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:27.488+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:54:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:27.488+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:54:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:27.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:54:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:27.488+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:27.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:27 smithi183 ceph-mon[37273]: pgmap v9634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:27 smithi144 ceph-mon[32129]: pgmap v9634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:27 smithi160 ceph-mon[36715]: pgmap v9634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:29 smithi183 ceph-mon[37273]: pgmap v9635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:54:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:29 smithi144 ceph-mon[32129]: pgmap v9635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:54:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:29 smithi160 ceph-mon[36715]: pgmap v9635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:54:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:31 smithi183 ceph-mon[37273]: pgmap v9636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:54:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17314.930270 secs 2024-07-26T15:54:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:31 smithi144 ceph-mon[32129]: pgmap v9636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:54:31.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17314.930270 secs 2024-07-26T15:54:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:31 smithi160 ceph-mon[36715]: pgmap v9636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:54:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17314.930270 secs 2024-07-26T15:54:33.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:33 smithi144 ceph-mon[32129]: pgmap v9637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:33 smithi160 ceph-mon[36715]: pgmap v9637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:33 smithi183 ceph-mon[37273]: pgmap v9637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:34.154+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:54:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:34.154+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:54:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:34.155+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:54:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:34.155+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:54:35.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:35 smithi144 ceph-mon[32129]: pgmap v9638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:35.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:35 smithi160 ceph-mon[36715]: pgmap v9638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:35 smithi183 ceph-mon[37273]: pgmap v9638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17319.930393 secs 2024-07-26T15:54:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17319.930393 secs 2024-07-26T15:54:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17319.930393 secs 2024-07-26T15:54:36.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: pgmap v9639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:37.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: pgmap v9639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:37.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: pgmap v9639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:54:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:38 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:54:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:38 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:54:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:38 smithi144 ceph-mon[32129]: pgmap v9640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:54:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:38 smithi144 ceph-mon[32129]: pgmap v9641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:38 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:54:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:38 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:54:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:38 smithi160 ceph-mon[36715]: pgmap v9640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:54:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:38 smithi160 ceph-mon[36715]: pgmap v9641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:38.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:38 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T15:54:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:38 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T15:54:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:38 smithi183 ceph-mon[37273]: pgmap v9640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T15:54:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:38 smithi183 ceph-mon[37273]: pgmap v9641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:40.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:40 smithi144 ceph-mon[32129]: pgmap v9642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:40 smithi160 ceph-mon[36715]: pgmap v9642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:40 smithi183 ceph-mon[37273]: pgmap v9642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17324.930460 secs 2024-07-26T15:54:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17324.930460 secs 2024-07-26T15:54:41.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17324.930460 secs 2024-07-26T15:54:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:42.492+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:54:42.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:42.493+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:54:42.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:42.493+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:54:42.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:42.493+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:54:42.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:42.493+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:42 smithi144 ceph-mon[32129]: pgmap v9643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:42 smithi160 ceph-mon[36715]: pgmap v9643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:42.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:42 smithi183 ceph-mon[37273]: pgmap v9643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:44 smithi144 ceph-mon[32129]: pgmap v9644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:54:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:44 smithi160 ceph-mon[36715]: pgmap v9644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:54:44.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:44 smithi183 ceph-mon[37273]: pgmap v9644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:54:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:46 smithi144 ceph-mon[32129]: pgmap v9645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17329.930583 secs 2024-07-26T15:54:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:46 smithi160 ceph-mon[36715]: pgmap v9645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17329.930583 secs 2024-07-26T15:54:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:46 smithi183 ceph-mon[37273]: pgmap v9645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:54:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17329.930583 secs 2024-07-26T15:54:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:48 smithi144 ceph-mon[32129]: pgmap v9646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:48 smithi160 ceph-mon[36715]: pgmap v9646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:48 smithi183 ceph-mon[37273]: pgmap v9646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:54:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:50 smithi160 ceph-mon[36715]: pgmap v9647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:50.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:50 smithi183 ceph-mon[37273]: pgmap v9647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:50.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:50 smithi144 ceph-mon[32129]: pgmap v9647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17334.930735 secs 2024-07-26T15:54:51.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17334.930735 secs 2024-07-26T15:54:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17334.930735 secs 2024-07-26T15:54:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:52 smithi160 ceph-mon[36715]: pgmap v9648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:52 smithi183 ceph-mon[37273]: pgmap v9648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:52.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:52 smithi144 ceph-mon[32129]: pgmap v9648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:54:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:54 smithi160 ceph-mon[36715]: pgmap v9649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:54.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:54 smithi183 ceph-mon[37273]: pgmap v9649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:54 smithi144 ceph-mon[32129]: pgmap v9649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:56 smithi160 ceph-mon[36715]: pgmap v9650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17339.930853 secs 2024-07-26T15:54:56.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:56 smithi183 ceph-mon[37273]: pgmap v9650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17339.930853 secs 2024-07-26T15:54:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:56 smithi144 ceph-mon[32129]: pgmap v9650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:54:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17339.930853 secs 2024-07-26T15:54:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:57.495+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:54:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:57.496+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:54:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:57.496+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:54:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:57.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:54:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:54:57.497+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:54:58 smithi160 ceph-mon[36715]: pgmap v9651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:54:58 smithi183 ceph-mon[37273]: pgmap v9651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:54:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:54:58 smithi144 ceph-mon[32129]: pgmap v9651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:00 smithi160 ceph-mon[36715]: pgmap v9652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:00 smithi183 ceph-mon[37273]: pgmap v9652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:00 smithi144 ceph-mon[32129]: pgmap v9652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17344.930958 secs 2024-07-26T15:55:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17344.930958 secs 2024-07-26T15:55:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17344.930958 secs 2024-07-26T15:55:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:02 smithi160 ceph-mon[36715]: pgmap v9653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:02 smithi183 ceph-mon[37273]: pgmap v9653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:02 smithi144 ceph-mon[32129]: pgmap v9653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:04 smithi160 ceph-mon[36715]: pgmap v9654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:04.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:04 smithi183 ceph-mon[37273]: pgmap v9654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:04 smithi144 ceph-mon[32129]: pgmap v9654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:06 smithi160 ceph-mon[36715]: pgmap v9655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17349.931083 secs 2024-07-26T15:55:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:06 smithi183 ceph-mon[37273]: pgmap v9655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17349.931083 secs 2024-07-26T15:55:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:06 smithi144 ceph-mon[32129]: pgmap v9655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17349.931083 secs 2024-07-26T15:55:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:08 smithi160 ceph-mon[36715]: pgmap v9656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:08.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:08 smithi183 ceph-mon[37273]: pgmap v9656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:08 smithi144 ceph-mon[32129]: pgmap v9656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:10.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:10 smithi160 ceph-mon[36715]: pgmap v9657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:10 smithi183 ceph-mon[37273]: pgmap v9657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:10 smithi144 ceph-mon[32129]: pgmap v9657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:11.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17354.931168 secs 2024-07-26T15:55:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17354.931168 secs 2024-07-26T15:55:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17354.931168 secs 2024-07-26T15:55:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:12.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:55:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:12.508+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:55:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:12.508+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:55:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:12.508+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:55:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:12.509+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:12 smithi183 ceph-mon[37273]: pgmap v9658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:12 smithi144 ceph-mon[32129]: pgmap v9658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:12 smithi160 ceph-mon[36715]: pgmap v9658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:14 smithi183 ceph-mon[37273]: pgmap v9659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:14 smithi144 ceph-mon[32129]: pgmap v9659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:14 smithi160 ceph-mon[36715]: pgmap v9659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:16 smithi183 ceph-mon[37273]: pgmap v9660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17359.931284 secs 2024-07-26T15:55:16.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:16 smithi144 ceph-mon[32129]: pgmap v9660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17359.931284 secs 2024-07-26T15:55:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:16 smithi160 ceph-mon[36715]: pgmap v9660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17359.931284 secs 2024-07-26T15:55:18.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:18 smithi183 ceph-mon[37273]: pgmap v9661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:18 smithi144 ceph-mon[32129]: pgmap v9661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:18 smithi160 ceph-mon[36715]: pgmap v9661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:20 smithi183 ceph-mon[37273]: pgmap v9662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:20 smithi144 ceph-mon[32129]: pgmap v9662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:20 smithi160 ceph-mon[36715]: pgmap v9662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:21.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17364.931426 secs 2024-07-26T15:55:21.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17364.931426 secs 2024-07-26T15:55:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17364.931426 secs 2024-07-26T15:55:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:22 smithi183 ceph-mon[37273]: pgmap v9663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:22 smithi144 ceph-mon[32129]: pgmap v9663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:22 smithi160 ceph-mon[36715]: pgmap v9663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:24 smithi183 ceph-mon[37273]: pgmap v9664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:24 smithi144 ceph-mon[32129]: pgmap v9664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:24 smithi160 ceph-mon[36715]: pgmap v9664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:26 smithi183 ceph-mon[37273]: pgmap v9665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17369.931486 secs 2024-07-26T15:55:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:26 smithi144 ceph-mon[32129]: pgmap v9665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17369.931486 secs 2024-07-26T15:55:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:26 smithi160 ceph-mon[36715]: pgmap v9665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17369.931486 secs 2024-07-26T15:55:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:27.514+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:55:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:27.515+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:55:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:27.515+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:55:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:27.515+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:55:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:27.515+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:28.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:28 smithi183 ceph-mon[37273]: pgmap v9666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:28 smithi144 ceph-mon[32129]: pgmap v9666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:28 smithi160 ceph-mon[36715]: pgmap v9666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:30.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:30 smithi183 ceph-mon[37273]: pgmap v9667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:30 smithi144 ceph-mon[32129]: pgmap v9667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:30 smithi160 ceph-mon[36715]: pgmap v9667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17374.931554 secs 2024-07-26T15:55:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17374.931554 secs 2024-07-26T15:55:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17374.931554 secs 2024-07-26T15:55:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:32 smithi144 ceph-mon[32129]: pgmap v9668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:32 smithi160 ceph-mon[36715]: pgmap v9668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:32 smithi183 ceph-mon[37273]: pgmap v9668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:34.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:34 smithi183 ceph-mon[37273]: pgmap v9669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:34 smithi144 ceph-mon[32129]: pgmap v9669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:34 smithi160 ceph-mon[36715]: pgmap v9669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:36 smithi144 ceph-mon[32129]: pgmap v9670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17379.931674 secs 2024-07-26T15:55:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:36 smithi160 ceph-mon[36715]: pgmap v9670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17379.931674 secs 2024-07-26T15:55:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:36 smithi183 ceph-mon[37273]: pgmap v9670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17379.931674 secs 2024-07-26T15:55:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: pgmap v9671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: pgmap v9671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:39.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: pgmap v9671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:39.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:55:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:39 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:55:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:39 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:55:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:39 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:55:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:39 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:55:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:39 smithi144 ceph-mon[32129]: pgmap v9672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:39 smithi144 ceph-mon[32129]: pgmap v9673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:55:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:39 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:55:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:39 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:55:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:39 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:55:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:39 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:55:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:39 smithi160 ceph-mon[36715]: pgmap v9672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:39 smithi160 ceph-mon[36715]: pgmap v9673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:55:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:39 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T15:55:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:39 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T15:55:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:39 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T15:55:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:39 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T15:55:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:39 smithi183 ceph-mon[37273]: pgmap v9672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:39 smithi183 ceph-mon[37273]: pgmap v9673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:55:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:41 smithi144 ceph-mon[32129]: pgmap v9674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17384.931795 secs 2024-07-26T15:55:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:41 smithi160 ceph-mon[36715]: pgmap v9674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17384.931795 secs 2024-07-26T15:55:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:41 smithi183 ceph-mon[37273]: pgmap v9674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17384.931795 secs 2024-07-26T15:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:42.531+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:42.532+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:42.532+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:42.532+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:42.532+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:43 smithi144 ceph-mon[32129]: pgmap v9675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:55:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:43 smithi160 ceph-mon[36715]: pgmap v9675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:55:44.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:43 smithi183 ceph-mon[37273]: pgmap v9675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:55:45.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:45 smithi144 ceph-mon[32129]: pgmap v9676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:55:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:45 smithi160 ceph-mon[36715]: pgmap v9676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:55:46.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:45 smithi183 ceph-mon[37273]: pgmap v9676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:55:46.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17389.931893 secs 2024-07-26T15:55:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17389.931893 secs 2024-07-26T15:55:47.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17389.931893 secs 2024-07-26T15:55:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:47 smithi144 ceph-mon[32129]: pgmap v9677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:47 smithi160 ceph-mon[36715]: pgmap v9677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:47 smithi183 ceph-mon[37273]: pgmap v9677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:55:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:49 smithi160 ceph-mon[36715]: pgmap v9678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:55:50.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:49 smithi183 ceph-mon[37273]: pgmap v9678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:55:50.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:49 smithi144 ceph-mon[32129]: pgmap v9678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:55:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:51 smithi160 ceph-mon[36715]: pgmap v9679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17394.932009 secs 2024-07-26T15:55:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:51 smithi183 ceph-mon[37273]: pgmap v9679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17394.932009 secs 2024-07-26T15:55:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:51 smithi144 ceph-mon[32129]: pgmap v9679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17394.932009 secs 2024-07-26T15:55:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:53 smithi160 ceph-mon[36715]: pgmap v9680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:54.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:53 smithi183 ceph-mon[37273]: pgmap v9680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:53 smithi144 ceph-mon[32129]: pgmap v9680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:55:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:55 smithi160 ceph-mon[36715]: pgmap v9681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:55 smithi183 ceph-mon[37273]: pgmap v9681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:55 smithi144 ceph-mon[32129]: pgmap v9681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:55:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17399.932109 secs 2024-07-26T15:55:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17399.932109 secs 2024-07-26T15:55:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17399.932109 secs 2024-07-26T15:55:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:57.548+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:55:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:57.549+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:55:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:57.549+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:55:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:57.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:55:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:55:57.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:57 smithi160 ceph-mon[36715]: pgmap v9682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:58.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:57 smithi183 ceph-mon[37273]: pgmap v9682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:55:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:57 smithi144 ceph-mon[32129]: pgmap v9682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:55:59 smithi160 ceph-mon[36715]: pgmap v9683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:00.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:55:59 smithi183 ceph-mon[37273]: pgmap v9683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:55:59 smithi144 ceph-mon[32129]: pgmap v9683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:01 smithi160 ceph-mon[36715]: pgmap v9684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17404.932237 secs 2024-07-26T15:56:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:01 smithi183 ceph-mon[37273]: pgmap v9684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17404.932237 secs 2024-07-26T15:56:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:01 smithi144 ceph-mon[32129]: pgmap v9684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17404.932237 secs 2024-07-26T15:56:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:03 smithi160 ceph-mon[36715]: pgmap v9685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:03 smithi183 ceph-mon[37273]: pgmap v9685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:03 smithi144 ceph-mon[32129]: pgmap v9685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:05 smithi160 ceph-mon[36715]: pgmap v9686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:05 smithi183 ceph-mon[37273]: pgmap v9686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:05 smithi144 ceph-mon[32129]: pgmap v9686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17409.932349 secs 2024-07-26T15:56:07.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17409.932349 secs 2024-07-26T15:56:07.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17409.932349 secs 2024-07-26T15:56:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:07 smithi160 ceph-mon[36715]: pgmap v9687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:07 smithi183 ceph-mon[37273]: pgmap v9687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:07 smithi144 ceph-mon[32129]: pgmap v9687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:09 smithi160 ceph-mon[36715]: pgmap v9688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:09 smithi183 ceph-mon[37273]: pgmap v9688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:09 smithi144 ceph-mon[32129]: pgmap v9688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:11 smithi160 ceph-mon[36715]: pgmap v9689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17414.932464 secs 2024-07-26T15:56:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:11 smithi183 ceph-mon[37273]: pgmap v9689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17414.932464 secs 2024-07-26T15:56:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:11 smithi144 ceph-mon[32129]: pgmap v9689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17414.932464 secs 2024-07-26T15:56:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:12.564+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:56:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:12.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:56:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:12.565+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:56:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:12.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:56:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:12.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:13 smithi160 ceph-mon[36715]: pgmap v9690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:13 smithi183 ceph-mon[37273]: pgmap v9690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:13 smithi144 ceph-mon[32129]: pgmap v9690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:15 smithi183 ceph-mon[37273]: pgmap v9691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:15 smithi144 ceph-mon[32129]: pgmap v9691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:15 smithi160 ceph-mon[36715]: pgmap v9691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17419.932584 secs 2024-07-26T15:56:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17419.932584 secs 2024-07-26T15:56:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17419.932584 secs 2024-07-26T15:56:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:17 smithi183 ceph-mon[37273]: pgmap v9692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:17 smithi144 ceph-mon[32129]: pgmap v9692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:17 smithi160 ceph-mon[36715]: pgmap v9692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:20.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:19 smithi183 ceph-mon[37273]: pgmap v9693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:19 smithi144 ceph-mon[32129]: pgmap v9693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:19 smithi160 ceph-mon[36715]: pgmap v9693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:21 smithi183 ceph-mon[37273]: pgmap v9694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17424.932650 secs 2024-07-26T15:56:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:21 smithi144 ceph-mon[32129]: pgmap v9694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17424.932650 secs 2024-07-26T15:56:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:21 smithi160 ceph-mon[36715]: pgmap v9694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17424.932650 secs 2024-07-26T15:56:24.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:23 smithi183 ceph-mon[37273]: pgmap v9695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:23 smithi144 ceph-mon[32129]: pgmap v9695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:23 smithi160 ceph-mon[36715]: pgmap v9695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:25 smithi183 ceph-mon[37273]: pgmap v9696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:25 smithi144 ceph-mon[32129]: pgmap v9696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:25 smithi160 ceph-mon[36715]: pgmap v9696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17429.932776 secs 2024-07-26T15:56:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17429.932776 secs 2024-07-26T15:56:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17429.932776 secs 2024-07-26T15:56:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:27.574+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:56:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:27.575+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:56:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:27.575+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:56:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:27.575+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:56:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:27.576+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:27 smithi183 ceph-mon[37273]: pgmap v9697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:27 smithi144 ceph-mon[32129]: pgmap v9697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:27 smithi160 ceph-mon[36715]: pgmap v9697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:30.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:29 smithi183 ceph-mon[37273]: pgmap v9698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:29 smithi144 ceph-mon[32129]: pgmap v9698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:29 smithi160 ceph-mon[36715]: pgmap v9698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:31 smithi183 ceph-mon[37273]: pgmap v9699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17434.932898 secs 2024-07-26T15:56:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:31 smithi144 ceph-mon[32129]: pgmap v9699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17434.932898 secs 2024-07-26T15:56:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:31 smithi160 ceph-mon[36715]: pgmap v9699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17434.932898 secs 2024-07-26T15:56:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:33 smithi144 ceph-mon[32129]: pgmap v9700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:33 smithi160 ceph-mon[36715]: pgmap v9700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:33 smithi183 ceph-mon[37273]: pgmap v9700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:36.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:35 smithi183 ceph-mon[37273]: pgmap v9701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:35 smithi144 ceph-mon[32129]: pgmap v9701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:35 smithi160 ceph-mon[36715]: pgmap v9701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17439.933000 secs 2024-07-26T15:56:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17439.933000 secs 2024-07-26T15:56:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17439.933000 secs 2024-07-26T15:56:38.174 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:37 smithi144 ceph-mon[32129]: pgmap v9702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:37 smithi160 ceph-mon[36715]: pgmap v9702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:38.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:37 smithi183 ceph-mon[37273]: pgmap v9702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:38 smithi144 ceph-mon[32129]: pgmap v9703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:38 smithi160 ceph-mon[36715]: pgmap v9703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:38 smithi183 ceph-mon[37273]: pgmap v9703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:39.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:39 smithi144 ceph-mon[32129]: pgmap v9704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:56:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:39 smithi144 ceph-mon[32129]: pgmap v9705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:56:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:56:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:39 smithi160 ceph-mon[36715]: pgmap v9704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:56:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:39 smithi160 ceph-mon[36715]: pgmap v9705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:56:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:56:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:39 smithi183 ceph-mon[37273]: pgmap v9704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T15:56:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:39 smithi183 ceph-mon[37273]: pgmap v9705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:56:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:56:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17444.933116 secs 2024-07-26T15:56:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17444.933116 secs 2024-07-26T15:56:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17444.933116 secs 2024-07-26T15:56:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:42.591+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:56:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:42.593+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:56:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:42.593+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:56:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:42.593+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:56:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:42.593+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:42 smithi183 ceph-mon[37273]: pgmap v9706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:56:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:42 smithi144 ceph-mon[32129]: pgmap v9706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:56:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:42 smithi160 ceph-mon[36715]: pgmap v9706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:56:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:44 smithi160 ceph-mon[36715]: pgmap v9707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:56:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:44 smithi183 ceph-mon[37273]: pgmap v9707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:56:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:44 smithi144 ceph-mon[32129]: pgmap v9707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:56:47.023 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:46 smithi144 ceph-mon[32129]: pgmap v9708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:56:47.023 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17449.933224 secs 2024-07-26T15:56:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:46 smithi160 ceph-mon[36715]: pgmap v9708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:56:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17449.933224 secs 2024-07-26T15:56:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:46 smithi183 ceph-mon[37273]: pgmap v9708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:56:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17449.933224 secs 2024-07-26T15:56:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:48 smithi160 ceph-mon[36715]: pgmap v9709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:56:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:48 smithi183 ceph-mon[37273]: pgmap v9709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:56:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:49.178+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:56:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:49.178+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:56:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:49.179+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:56:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:49.179+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:56:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:48 smithi144 ceph-mon[32129]: pgmap v9709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:56:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:50 smithi160 ceph-mon[36715]: pgmap v9710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:56:51.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:50 smithi183 ceph-mon[37273]: pgmap v9710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:56:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:50 smithi144 ceph-mon[32129]: pgmap v9710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:56:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17454.933348 secs 2024-07-26T15:56:52.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17454.933348 secs 2024-07-26T15:56:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17454.933348 secs 2024-07-26T15:56:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:52 smithi160 ceph-mon[36715]: pgmap v9711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:52 smithi183 ceph-mon[37273]: pgmap v9711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:52 smithi144 ceph-mon[32129]: pgmap v9711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:56:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:54 smithi160 ceph-mon[36715]: pgmap v9712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:54 smithi183 ceph-mon[37273]: pgmap v9712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:54 smithi144 ceph-mon[32129]: pgmap v9712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:56 smithi160 ceph-mon[36715]: pgmap v9713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17459.933471 secs 2024-07-26T15:56:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:56 smithi183 ceph-mon[37273]: pgmap v9713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17459.933471 secs 2024-07-26T15:56:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:56 smithi144 ceph-mon[32129]: pgmap v9713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:56:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17459.933471 secs 2024-07-26T15:56:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:57.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:56:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:57.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:56:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:57.600+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:56:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:57.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:56:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:56:57.600+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:56:58 smithi160 ceph-mon[36715]: pgmap v9714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:56:58 smithi183 ceph-mon[37273]: pgmap v9714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:56:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:56:58 smithi144 ceph-mon[32129]: pgmap v9714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:00 smithi160 ceph-mon[36715]: pgmap v9715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:00 smithi183 ceph-mon[37273]: pgmap v9715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:00 smithi144 ceph-mon[32129]: pgmap v9715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17464.933597 secs 2024-07-26T15:57:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17464.933597 secs 2024-07-26T15:57:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17464.933597 secs 2024-07-26T15:57:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:02 smithi160 ceph-mon[36715]: pgmap v9716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:03.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:02 smithi183 ceph-mon[37273]: pgmap v9716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:02 smithi144 ceph-mon[32129]: pgmap v9716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:04 smithi160 ceph-mon[36715]: pgmap v9717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:04 smithi183 ceph-mon[37273]: pgmap v9717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:04 smithi144 ceph-mon[32129]: pgmap v9717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:06 smithi183 ceph-mon[37273]: pgmap v9718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17469.933719 secs 2024-07-26T15:57:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:06 smithi144 ceph-mon[32129]: pgmap v9718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17469.933719 secs 2024-07-26T15:57:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:06 smithi160 ceph-mon[36715]: pgmap v9718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17469.933719 secs 2024-07-26T15:57:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:08 smithi183 ceph-mon[37273]: pgmap v9719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:08 smithi144 ceph-mon[32129]: pgmap v9719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:08 smithi160 ceph-mon[36715]: pgmap v9719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:11.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:10 smithi160 ceph-mon[36715]: pgmap v9720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:11.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:10 smithi183 ceph-mon[37273]: pgmap v9720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:10 smithi144 ceph-mon[32129]: pgmap v9720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:12.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17474.933870 secs 2024-07-26T15:57:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17474.933870 secs 2024-07-26T15:57:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17474.933870 secs 2024-07-26T15:57:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:12.601+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:57:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:12.602+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:57:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:12.602+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:57:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:12.602+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:57:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:12.602+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:12 smithi183 ceph-mon[37273]: pgmap v9721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:12 smithi144 ceph-mon[32129]: pgmap v9721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:12 smithi160 ceph-mon[36715]: pgmap v9721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:14 smithi183 ceph-mon[37273]: pgmap v9722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:14 smithi144 ceph-mon[32129]: pgmap v9722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:14 smithi160 ceph-mon[36715]: pgmap v9722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:17.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:16 smithi183 ceph-mon[37273]: pgmap v9723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17479.933989 secs 2024-07-26T15:57:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:16 smithi144 ceph-mon[32129]: pgmap v9723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17479.933989 secs 2024-07-26T15:57:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:16 smithi160 ceph-mon[36715]: pgmap v9723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17479.933989 secs 2024-07-26T15:57:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:18 smithi183 ceph-mon[37273]: pgmap v9724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:18 smithi144 ceph-mon[32129]: pgmap v9724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:18 smithi160 ceph-mon[36715]: pgmap v9724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:20 smithi183 ceph-mon[37273]: pgmap v9725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:20 smithi144 ceph-mon[32129]: pgmap v9725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:20 smithi160 ceph-mon[36715]: pgmap v9725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17484.934110 secs 2024-07-26T15:57:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17484.934110 secs 2024-07-26T15:57:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17484.934110 secs 2024-07-26T15:57:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:22 smithi183 ceph-mon[37273]: pgmap v9726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:22 smithi144 ceph-mon[32129]: pgmap v9726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:22 smithi160 ceph-mon[36715]: pgmap v9726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:24 smithi183 ceph-mon[37273]: pgmap v9727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:24 smithi144 ceph-mon[32129]: pgmap v9727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:24 smithi160 ceph-mon[36715]: pgmap v9727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:26 smithi144 ceph-mon[32129]: pgmap v9728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17489.934229 secs 2024-07-26T15:57:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:26 smithi160 ceph-mon[36715]: pgmap v9728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17489.934229 secs 2024-07-26T15:57:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:26 smithi183 ceph-mon[37273]: pgmap v9728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17489.934229 secs 2024-07-26T15:57:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:27.619+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:57:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:27.619+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:57:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:27.619+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:57:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:27.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:57:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:27.620+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:28 smithi144 ceph-mon[32129]: pgmap v9729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:28 smithi160 ceph-mon[36715]: pgmap v9729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:28 smithi183 ceph-mon[37273]: pgmap v9729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:30 smithi144 ceph-mon[32129]: pgmap v9730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:30 smithi160 ceph-mon[36715]: pgmap v9730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:30 smithi183 ceph-mon[37273]: pgmap v9730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17494.934357 secs 2024-07-26T15:57:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17494.934357 secs 2024-07-26T15:57:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17494.934357 secs 2024-07-26T15:57:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:32 smithi144 ceph-mon[32129]: pgmap v9731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:32 smithi160 ceph-mon[36715]: pgmap v9731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:33.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:32 smithi183 ceph-mon[37273]: pgmap v9731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:33 smithi144 ceph-mon[32129]: pgmap v9732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:33 smithi160 ceph-mon[36715]: pgmap v9732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:33 smithi183 ceph-mon[37273]: pgmap v9732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:36 smithi160 ceph-mon[36715]: pgmap v9733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17499.934481 secs 2024-07-26T15:57:37.088 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:36 smithi183 ceph-mon[37273]: pgmap v9733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:37.088 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17499.934481 secs 2024-07-26T15:57:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:36 smithi144 ceph-mon[32129]: pgmap v9733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17499.934481 secs 2024-07-26T15:57:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:38 smithi160 ceph-mon[36715]: pgmap v9734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:39.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:38 smithi183 ceph-mon[37273]: pgmap v9734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:38 smithi144 ceph-mon[32129]: pgmap v9734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:40 smithi160 ceph-mon[36715]: pgmap v9735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:40 smithi183 ceph-mon[37273]: pgmap v9735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:40 smithi144 ceph-mon[32129]: pgmap v9735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17504.934598 secs 2024-07-26T15:57:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17504.934598 secs 2024-07-26T15:57:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17504.934598 secs 2024-07-26T15:57:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:57:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:42.624+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:57:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:42.625+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:57:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:42.625+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:57:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:42.625+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:57:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:42.625+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:42 smithi160 ceph-mon[36715]: pgmap v9736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:42 smithi160 ceph-mon[36715]: pgmap v9737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:43.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:42 smithi183 ceph-mon[37273]: pgmap v9736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:42 smithi183 ceph-mon[37273]: pgmap v9737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:42 smithi144 ceph-mon[32129]: pgmap v9736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:57:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:42 smithi144 ceph-mon[32129]: pgmap v9737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:44 smithi160 ceph-mon[36715]: pgmap v9738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:44 smithi183 ceph-mon[37273]: pgmap v9738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:44 smithi144 ceph-mon[32129]: pgmap v9738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:46 smithi160 ceph-mon[36715]: pgmap v9739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17509.934664 secs 2024-07-26T15:57:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:46 smithi183 ceph-mon[37273]: pgmap v9739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17509.934664 secs 2024-07-26T15:57:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:46 smithi144 ceph-mon[32129]: pgmap v9739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17509.934664 secs 2024-07-26T15:57:49.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:48 smithi183 ceph-mon[37273]: pgmap v9740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:48 smithi144 ceph-mon[32129]: pgmap v9740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:48 smithi160 ceph-mon[36715]: pgmap v9740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:50 smithi183 ceph-mon[37273]: pgmap v9741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:50 smithi144 ceph-mon[32129]: pgmap v9741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:50 smithi160 ceph-mon[36715]: pgmap v9741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:57:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17514.934747 secs 2024-07-26T15:57:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17514.934747 secs 2024-07-26T15:57:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17514.934747 secs 2024-07-26T15:57:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:52 smithi183 ceph-mon[37273]: pgmap v9742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:57:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:52 smithi144 ceph-mon[32129]: pgmap v9742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:57:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:52 smithi160 ceph-mon[36715]: pgmap v9742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:57:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:54 smithi183 ceph-mon[37273]: pgmap v9743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:54 smithi144 ceph-mon[32129]: pgmap v9743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:54 smithi160 ceph-mon[36715]: pgmap v9743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:56 smithi183 ceph-mon[37273]: pgmap v9744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17519.934828 secs 2024-07-26T15:57:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:56 smithi144 ceph-mon[32129]: pgmap v9744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17519.934828 secs 2024-07-26T15:57:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:56 smithi160 ceph-mon[36715]: pgmap v9744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:57:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17519.934828 secs 2024-07-26T15:57:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:57.629+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:57:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:57.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:57:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:57.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:57:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:57.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:57:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:57:57.631+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:57:58 smithi183 ceph-mon[37273]: pgmap v9745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:57:58 smithi144 ceph-mon[32129]: pgmap v9745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:57:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:57:58 smithi160 ceph-mon[36715]: pgmap v9745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:00 smithi183 ceph-mon[37273]: pgmap v9746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:00 smithi144 ceph-mon[32129]: pgmap v9746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:00 smithi160 ceph-mon[36715]: pgmap v9746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17524.934982 secs 2024-07-26T15:58:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17524.934982 secs 2024-07-26T15:58:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17524.934982 secs 2024-07-26T15:58:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:02 smithi183 ceph-mon[37273]: pgmap v9747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:02 smithi144 ceph-mon[32129]: pgmap v9747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:02 smithi160 ceph-mon[36715]: pgmap v9747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:04 smithi183 ceph-mon[37273]: pgmap v9748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:04 smithi144 ceph-mon[32129]: pgmap v9748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:04 smithi160 ceph-mon[36715]: pgmap v9748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:06 smithi183 ceph-mon[37273]: pgmap v9749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17529.935086 secs 2024-07-26T15:58:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:06 smithi144 ceph-mon[32129]: pgmap v9749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17529.935086 secs 2024-07-26T15:58:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:06 smithi160 ceph-mon[36715]: pgmap v9749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17529.935086 secs 2024-07-26T15:58:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:08 smithi144 ceph-mon[32129]: pgmap v9750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:08 smithi160 ceph-mon[36715]: pgmap v9750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:08 smithi183 ceph-mon[37273]: pgmap v9750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:10 smithi144 ceph-mon[32129]: pgmap v9751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:10 smithi160 ceph-mon[36715]: pgmap v9751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:10 smithi183 ceph-mon[37273]: pgmap v9751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17534.935250 secs 2024-07-26T15:58:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17534.935250 secs 2024-07-26T15:58:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17534.935250 secs 2024-07-26T15:58:12.639 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:12.636+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:58:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:12.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:58:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:12.638+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:58:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:12.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:58:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:12.638+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:12 smithi144 ceph-mon[32129]: pgmap v9752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:12 smithi160 ceph-mon[36715]: pgmap v9752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:13.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:12 smithi183 ceph-mon[37273]: pgmap v9752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:15.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:14 smithi144 ceph-mon[32129]: pgmap v9753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:14 smithi160 ceph-mon[36715]: pgmap v9753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:14 smithi183 ceph-mon[37273]: pgmap v9753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:16 smithi144 ceph-mon[32129]: pgmap v9754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17539.935380 secs 2024-07-26T15:58:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:16 smithi160 ceph-mon[36715]: pgmap v9754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17539.935380 secs 2024-07-26T15:58:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:16 smithi183 ceph-mon[37273]: pgmap v9754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17539.935380 secs 2024-07-26T15:58:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:18 smithi144 ceph-mon[32129]: pgmap v9755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:18 smithi160 ceph-mon[36715]: pgmap v9755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:18 smithi183 ceph-mon[37273]: pgmap v9755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:19 smithi144 ceph-mon[32129]: pgmap v9756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:19 smithi160 ceph-mon[36715]: pgmap v9756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:20.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:19 smithi183 ceph-mon[37273]: pgmap v9756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:21.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17544.935478 secs 2024-07-26T15:58:21.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17544.935478 secs 2024-07-26T15:58:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17544.935478 secs 2024-07-26T15:58:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:22 smithi183 ceph-mon[37273]: pgmap v9757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:22 smithi144 ceph-mon[32129]: pgmap v9757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:22 smithi160 ceph-mon[36715]: pgmap v9757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:24 smithi183 ceph-mon[37273]: pgmap v9758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:24 smithi144 ceph-mon[32129]: pgmap v9758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:24 smithi160 ceph-mon[36715]: pgmap v9758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:26 smithi183 ceph-mon[37273]: pgmap v9759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17549.935602 secs 2024-07-26T15:58:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:26 smithi144 ceph-mon[32129]: pgmap v9759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17549.935602 secs 2024-07-26T15:58:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:26 smithi160 ceph-mon[36715]: pgmap v9759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17549.935602 secs 2024-07-26T15:58:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:27.642+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:58:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:27.644+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:58:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:27.644+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:58:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:27.644+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:58:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:27.644+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:28 smithi183 ceph-mon[37273]: pgmap v9760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:28.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:28 smithi144 ceph-mon[32129]: pgmap v9760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:28 smithi160 ceph-mon[36715]: pgmap v9760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:30 smithi183 ceph-mon[37273]: pgmap v9761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:30 smithi144 ceph-mon[32129]: pgmap v9761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:30 smithi160 ceph-mon[36715]: pgmap v9761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17554.935692 secs 2024-07-26T15:58:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17554.935692 secs 2024-07-26T15:58:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17554.935692 secs 2024-07-26T15:58:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:32 smithi183 ceph-mon[37273]: pgmap v9762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:32 smithi144 ceph-mon[32129]: pgmap v9762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:32 smithi160 ceph-mon[36715]: pgmap v9762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:58:34.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:34 smithi183 ceph-mon[37273]: pgmap v9763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:34 smithi144 ceph-mon[32129]: pgmap v9763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:34 smithi160 ceph-mon[36715]: pgmap v9763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:36 smithi183 ceph-mon[37273]: pgmap v9764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17559.935794 secs 2024-07-26T15:58:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:36 smithi144 ceph-mon[32129]: pgmap v9764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17559.935794 secs 2024-07-26T15:58:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:36 smithi160 ceph-mon[36715]: pgmap v9764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17559.935794 secs 2024-07-26T15:58:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:38 smithi183 ceph-mon[37273]: pgmap v9765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:38 smithi144 ceph-mon[32129]: pgmap v9765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:38 smithi160 ceph-mon[36715]: pgmap v9765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:58:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:40 smithi183 ceph-mon[37273]: pgmap v9766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:58:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:40 smithi144 ceph-mon[32129]: pgmap v9766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:58:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:40 smithi160 ceph-mon[36715]: pgmap v9766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T15:58:41.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17564.935941 secs 2024-07-26T15:58:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17564.935941 secs 2024-07-26T15:58:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17564.935941 secs 2024-07-26T15:58:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:42.659+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:58:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:42.660+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:58:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:42.660+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:58:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:42.660+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:58:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:42.660+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:42.760 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:42 smithi144 ceph-mon[32129]: pgmap v9767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:42.760 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:42 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:42.760 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.760 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.760 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.760 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:42 smithi183 ceph-mon[37273]: pgmap v9767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:42.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:42 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:42.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.946 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:42 smithi160 ceph-mon[36715]: pgmap v9767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:42.946 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:42 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:42.946 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.946 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.946 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:42.946 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: pgmap v9768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: pgmap v9768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: pgmap v9768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:45 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:45 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:45 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:46 smithi144 ceph-mon[32129]: pgmap v9769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17569.936062 secs 2024-07-26T15:58:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:46 smithi160 ceph-mon[36715]: pgmap v9769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17569.936062 secs 2024-07-26T15:58:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:46 smithi183 ceph-mon[37273]: pgmap v9769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17569.936062 secs 2024-07-26T15:58:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:58:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:58:48.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: pgmap v9770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:48.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: pgmap v9771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:48.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:58:48.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:58:48.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: pgmap v9770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:48.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: pgmap v9771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:48.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: pgmap v9770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: pgmap v9771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:58:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:50 smithi160 ceph-mon[36715]: pgmap v9772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:50 smithi183 ceph-mon[37273]: pgmap v9772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:50.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:50 smithi144 ceph-mon[32129]: pgmap v9772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17574.936179 secs 2024-07-26T15:58:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:52 smithi160 ceph-mon[36715]: pgmap v9773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17574.936179 secs 2024-07-26T15:58:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:52 smithi183 ceph-mon[37273]: pgmap v9773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17574.936179 secs 2024-07-26T15:58:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:52 smithi144 ceph-mon[32129]: pgmap v9773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:58:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:54 smithi160 ceph-mon[36715]: pgmap v9774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:54.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:54 smithi183 ceph-mon[37273]: pgmap v9774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:54 smithi144 ceph-mon[32129]: pgmap v9774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:56 smithi160 ceph-mon[36715]: pgmap v9775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:56 smithi183 ceph-mon[37273]: pgmap v9775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:56.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:56 smithi144 ceph-mon[32129]: pgmap v9775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T15:58:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17579.936339 secs 2024-07-26T15:58:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17579.936339 secs 2024-07-26T15:58:57.677 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:57.675+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:58:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:57.676+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:58:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:57.676+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:58:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:57.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:58:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:58:57.676+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17579.936339 secs 2024-07-26T15:58:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:58:58 smithi160 ceph-mon[36715]: pgmap v9776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:58.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:58:58 smithi183 ceph-mon[37273]: pgmap v9776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:58:58.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:58:58 smithi144 ceph-mon[32129]: pgmap v9776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T15:59:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:00 smithi160 ceph-mon[36715]: pgmap v9777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:00 smithi183 ceph-mon[37273]: pgmap v9777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:00.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:00 smithi144 ceph-mon[32129]: pgmap v9777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17584.936443 secs 2024-07-26T15:59:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17584.936443 secs 2024-07-26T15:59:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17584.936443 secs 2024-07-26T15:59:02.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:02 smithi183 ceph-mon[37273]: pgmap v9778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:02.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:02 smithi144 ceph-mon[32129]: pgmap v9778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:02 smithi160 ceph-mon[36715]: pgmap v9778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:04 smithi183 ceph-mon[37273]: pgmap v9779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:04 smithi144 ceph-mon[32129]: pgmap v9779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:04 smithi160 ceph-mon[36715]: pgmap v9779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:06 smithi183 ceph-mon[37273]: pgmap v9780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17589.936567 secs 2024-07-26T15:59:06.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:06 smithi144 ceph-mon[32129]: pgmap v9780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17589.936567 secs 2024-07-26T15:59:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:06 smithi160 ceph-mon[36715]: pgmap v9780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17589.936567 secs 2024-07-26T15:59:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:08 smithi183 ceph-mon[37273]: pgmap v9781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:08 smithi144 ceph-mon[32129]: pgmap v9781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:08 smithi160 ceph-mon[36715]: pgmap v9781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:10 smithi183 ceph-mon[37273]: pgmap v9782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:10.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:10 smithi144 ceph-mon[32129]: pgmap v9782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:10 smithi160 ceph-mon[36715]: pgmap v9782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17594.936686 secs 2024-07-26T15:59:11.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17594.936686 secs 2024-07-26T15:59:11.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17594.936686 secs 2024-07-26T15:59:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:12 smithi183 ceph-mon[37273]: pgmap v9783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:12.691+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:59:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:12.692+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:59:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:12.692+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:59:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:12.692+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:59:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:12.692+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:12 smithi144 ceph-mon[32129]: pgmap v9783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:12 smithi160 ceph-mon[36715]: pgmap v9783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:14 smithi183 ceph-mon[37273]: pgmap v9784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:14.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:14 smithi144 ceph-mon[32129]: pgmap v9784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:14 smithi160 ceph-mon[36715]: pgmap v9784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:16 smithi183 ceph-mon[37273]: pgmap v9785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17599.936827 secs 2024-07-26T15:59:16.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:16 smithi144 ceph-mon[32129]: pgmap v9785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17599.936827 secs 2024-07-26T15:59:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:16 smithi160 ceph-mon[36715]: pgmap v9785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17599.936827 secs 2024-07-26T15:59:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:18 smithi183 ceph-mon[37273]: pgmap v9786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:18 smithi144 ceph-mon[32129]: pgmap v9786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:18 smithi160 ceph-mon[36715]: pgmap v9786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:20 smithi144 ceph-mon[32129]: pgmap v9787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:20 smithi160 ceph-mon[36715]: pgmap v9787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:20 smithi183 ceph-mon[37273]: pgmap v9787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17604.936943 secs 2024-07-26T15:59:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17604.936943 secs 2024-07-26T15:59:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17604.936943 secs 2024-07-26T15:59:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:22 smithi144 ceph-mon[32129]: pgmap v9788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:22 smithi160 ceph-mon[36715]: pgmap v9788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:22.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:22 smithi183 ceph-mon[37273]: pgmap v9788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:24.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:24 smithi144 ceph-mon[32129]: pgmap v9789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:24 smithi160 ceph-mon[36715]: pgmap v9789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:24 smithi183 ceph-mon[37273]: pgmap v9789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:26.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:26 smithi144 ceph-mon[32129]: pgmap v9790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17609.937062 secs 2024-07-26T15:59:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:26 smithi160 ceph-mon[36715]: pgmap v9790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17609.937062 secs 2024-07-26T15:59:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:26 smithi183 ceph-mon[37273]: pgmap v9790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17609.937062 secs 2024-07-26T15:59:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:27.708+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:59:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:27.709+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:59:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:27.709+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:59:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:27.709+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:59:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:27.709+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:28 smithi144 ceph-mon[32129]: pgmap v9791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:28 smithi160 ceph-mon[36715]: pgmap v9791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:28 smithi183 ceph-mon[37273]: pgmap v9791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:30 smithi183 ceph-mon[37273]: pgmap v9792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:30.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:30 smithi144 ceph-mon[32129]: pgmap v9792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:30 smithi160 ceph-mon[36715]: pgmap v9792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17614.937204 secs 2024-07-26T15:59:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17614.937204 secs 2024-07-26T15:59:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17614.937204 secs 2024-07-26T15:59:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:32 smithi183 ceph-mon[37273]: pgmap v9793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:32 smithi144 ceph-mon[32129]: pgmap v9793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:32 smithi160 ceph-mon[36715]: pgmap v9793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:34 smithi183 ceph-mon[37273]: pgmap v9794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:34.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:34 smithi144 ceph-mon[32129]: pgmap v9794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:34 smithi160 ceph-mon[36715]: pgmap v9794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:36 smithi144 ceph-mon[32129]: pgmap v9795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17619.937341 secs 2024-07-26T15:59:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:36 smithi160 ceph-mon[36715]: pgmap v9795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17619.937341 secs 2024-07-26T15:59:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:36 smithi183 ceph-mon[37273]: pgmap v9795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17619.937341 secs 2024-07-26T15:59:38.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:38 smithi183 ceph-mon[37273]: pgmap v9796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:38.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:38 smithi144 ceph-mon[32129]: pgmap v9796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:38 smithi160 ceph-mon[36715]: pgmap v9796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:40 smithi144 ceph-mon[32129]: pgmap v9797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:40 smithi160 ceph-mon[36715]: pgmap v9797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:40 smithi183 ceph-mon[37273]: pgmap v9797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:41.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17624.937441 secs 2024-07-26T15:59:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17624.937441 secs 2024-07-26T15:59:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17624.937441 secs 2024-07-26T15:59:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:42.725+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:59:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:42.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:59:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:42.726+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:59:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:42.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:59:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:42.726+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:42.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:42 smithi144 ceph-mon[32129]: pgmap v9798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:42 smithi160 ceph-mon[36715]: pgmap v9798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:42 smithi183 ceph-mon[37273]: pgmap v9798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:44 smithi144 ceph-mon[32129]: pgmap v9799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:44 smithi160 ceph-mon[36715]: pgmap v9799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:44 smithi183 ceph-mon[37273]: pgmap v9799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T15:59:46.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:46 smithi144 ceph-mon[32129]: pgmap v9800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:46.973 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17629.937566 secs 2024-07-26T15:59:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:46 smithi160 ceph-mon[36715]: pgmap v9800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17629.937566 secs 2024-07-26T15:59:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:46 smithi183 ceph-mon[37273]: pgmap v9800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T15:59:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17629.937566 secs 2024-07-26T15:59:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:48 smithi144 ceph-mon[32129]: pgmap v9801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:59:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:48 smithi160 ceph-mon[36715]: pgmap v9801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:59:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:48 smithi183 ceph-mon[37273]: pgmap v9801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T15:59:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T15:59:49.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:49.210+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T15:59:49.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:49.210+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T15:59:49.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:49.211+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006250 con 0x7f74b0109990 2024-07-26T15:59:49.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:49.211+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x563c6c4182f0 2024-07-26T15:59:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:49 smithi144 ceph-mon[32129]: pgmap v9802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:49.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:49 smithi144 ceph-mon[32129]: pgmap v9803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:59:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:49 smithi160 ceph-mon[36715]: pgmap v9802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:49 smithi160 ceph-mon[36715]: pgmap v9803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:59:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:49 smithi183 ceph-mon[37273]: pgmap v9802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:49 smithi183 ceph-mon[37273]: pgmap v9803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T15:59:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:51 smithi144 ceph-mon[32129]: pgmap v9804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17634.937642 secs 2024-07-26T15:59:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:51 smithi160 ceph-mon[36715]: pgmap v9804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17634.937642 secs 2024-07-26T15:59:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:51 smithi183 ceph-mon[37273]: pgmap v9804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17634.937642 secs 2024-07-26T15:59:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:53 smithi144 ceph-mon[32129]: pgmap v9805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:53 smithi160 ceph-mon[36715]: pgmap v9805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:53 smithi183 ceph-mon[37273]: pgmap v9805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:55 smithi160 ceph-mon[36715]: pgmap v9806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:56.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:55 smithi183 ceph-mon[37273]: pgmap v9806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:55 smithi144 ceph-mon[32129]: pgmap v9806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17639.937776 secs 2024-07-26T15:59:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17639.937776 secs 2024-07-26T15:59:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17639.937776 secs 2024-07-26T15:59:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:57.735+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:59:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:57.736+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T15:59:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:57.736+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T15:59:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:57.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:59:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T15:59:57.736+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:57 smithi160 ceph-mon[36715]: pgmap v9807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:57 smithi183 ceph-mon[37273]: pgmap v9807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T15:59:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:57 smithi144 ceph-mon[32129]: pgmap v9807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 15:59:59 smithi160 ceph-mon[36715]: pgmap v9808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:00:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 15:59:59 smithi183 ceph-mon[37273]: pgmap v9808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:00:00.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 15:59:59 smithi144 ceph-mon[32129]: pgmap v9808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:00:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:00:01.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:00:01.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:00:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:01 smithi160 ceph-mon[36715]: pgmap v9809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17644.937887 secs 2024-07-26T16:00:02.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:01 smithi183 ceph-mon[37273]: pgmap v9809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17644.937887 secs 2024-07-26T16:00:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:01 smithi144 ceph-mon[32129]: pgmap v9809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17644.937887 secs 2024-07-26T16:00:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:03 smithi160 ceph-mon[36715]: pgmap v9810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:03 smithi183 ceph-mon[37273]: pgmap v9810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:03 smithi144 ceph-mon[32129]: pgmap v9810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:05 smithi160 ceph-mon[36715]: pgmap v9811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:06.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:05 smithi183 ceph-mon[37273]: pgmap v9811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:05 smithi144 ceph-mon[32129]: pgmap v9811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17649.938008 secs 2024-07-26T16:00:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17649.938008 secs 2024-07-26T16:00:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17649.938008 secs 2024-07-26T16:00:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:07 smithi160 ceph-mon[36715]: pgmap v9812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:08.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:07 smithi183 ceph-mon[37273]: pgmap v9812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:07 smithi144 ceph-mon[32129]: pgmap v9812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:09 smithi160 ceph-mon[36715]: pgmap v9813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:10.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:09 smithi183 ceph-mon[37273]: pgmap v9813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:09 smithi144 ceph-mon[32129]: pgmap v9813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:11 smithi160 ceph-mon[36715]: pgmap v9814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17654.938133 secs 2024-07-26T16:00:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:11 smithi183 ceph-mon[37273]: pgmap v9814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17654.938133 secs 2024-07-26T16:00:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:11 smithi144 ceph-mon[32129]: pgmap v9814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17654.938133 secs 2024-07-26T16:00:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:12.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:00:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:12.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:00:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:12.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:00:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:12.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:00:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:12.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:13 smithi160 ceph-mon[36715]: pgmap v9815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:13 smithi183 ceph-mon[37273]: pgmap v9815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:14.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:13 smithi144 ceph-mon[32129]: pgmap v9815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:15 smithi183 ceph-mon[37273]: pgmap v9816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:15 smithi144 ceph-mon[32129]: pgmap v9816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:15 smithi160 ceph-mon[36715]: pgmap v9816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17659.938271 secs 2024-07-26T16:00:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17659.938271 secs 2024-07-26T16:00:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17659.938271 secs 2024-07-26T16:00:18.125 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:17 smithi144 ceph-mon[32129]: pgmap v9817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:18.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:17 smithi183 ceph-mon[37273]: pgmap v9817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:17 smithi160 ceph-mon[36715]: pgmap v9817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:19 smithi183 ceph-mon[37273]: pgmap v9818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:19 smithi144 ceph-mon[32129]: pgmap v9818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:19 smithi160 ceph-mon[36715]: pgmap v9818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:21 smithi183 ceph-mon[37273]: pgmap v9819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17664.938376 secs 2024-07-26T16:00:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:21 smithi144 ceph-mon[32129]: pgmap v9819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17664.938376 secs 2024-07-26T16:00:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:21 smithi160 ceph-mon[36715]: pgmap v9819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17664.938376 secs 2024-07-26T16:00:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:23 smithi183 ceph-mon[37273]: pgmap v9820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:24.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:23 smithi144 ceph-mon[32129]: pgmap v9820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:23 smithi160 ceph-mon[36715]: pgmap v9820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:25 smithi183 ceph-mon[37273]: pgmap v9821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:25 smithi144 ceph-mon[32129]: pgmap v9821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:25 smithi160 ceph-mon[36715]: pgmap v9821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17669.938487 secs 2024-07-26T16:00:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17669.938487 secs 2024-07-26T16:00:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17669.938487 secs 2024-07-26T16:00:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:27.752+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:00:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:27.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:00:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:27.753+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:00:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:27.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:00:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:27.753+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:28.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:27 smithi183 ceph-mon[37273]: pgmap v9822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:27 smithi144 ceph-mon[32129]: pgmap v9822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:27 smithi160 ceph-mon[36715]: pgmap v9822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:29 smithi183 ceph-mon[37273]: pgmap v9823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:29 smithi144 ceph-mon[32129]: pgmap v9823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:29 smithi160 ceph-mon[36715]: pgmap v9823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:32.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:31 smithi183 ceph-mon[37273]: pgmap v9824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17674.938603 secs 2024-07-26T16:00:32.158 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:31 smithi144 ceph-mon[32129]: pgmap v9824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:32.158 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17674.938603 secs 2024-07-26T16:00:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:31 smithi160 ceph-mon[36715]: pgmap v9824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17674.938603 secs 2024-07-26T16:00:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:33 smithi183 ceph-mon[37273]: pgmap v9825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:33 smithi144 ceph-mon[32129]: pgmap v9825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:33 smithi160 ceph-mon[36715]: pgmap v9825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:35 smithi183 ceph-mon[37273]: pgmap v9826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:35 smithi144 ceph-mon[32129]: pgmap v9826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:35 smithi160 ceph-mon[36715]: pgmap v9826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17679.938723 secs 2024-07-26T16:00:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17679.938723 secs 2024-07-26T16:00:37.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17679.938723 secs 2024-07-26T16:00:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:37 smithi144 ceph-mon[32129]: pgmap v9827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:38.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:37 smithi183 ceph-mon[37273]: pgmap v9827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:37 smithi160 ceph-mon[36715]: pgmap v9827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:39 smithi144 ceph-mon[32129]: pgmap v9828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:39 smithi160 ceph-mon[36715]: pgmap v9828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:39 smithi183 ceph-mon[37273]: pgmap v9828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:41 smithi144 ceph-mon[32129]: pgmap v9829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17684.938846 secs 2024-07-26T16:00:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:41 smithi160 ceph-mon[36715]: pgmap v9829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17684.938846 secs 2024-07-26T16:00:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:41 smithi183 ceph-mon[37273]: pgmap v9829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17684.938846 secs 2024-07-26T16:00:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:42.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:00:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:42.756+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:00:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:42.756+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:00:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:42.756+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:00:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:42.757+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:43 smithi144 ceph-mon[32129]: pgmap v9830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:43 smithi160 ceph-mon[36715]: pgmap v9830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:43 smithi183 ceph-mon[37273]: pgmap v9830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:00:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:45 smithi144 ceph-mon[32129]: pgmap v9831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:45 smithi160 ceph-mon[36715]: pgmap v9831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:46.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:45 smithi183 ceph-mon[37273]: pgmap v9831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17689.938999 secs 2024-07-26T16:00:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17689.938999 secs 2024-07-26T16:00:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17689.938999 secs 2024-07-26T16:00:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:47 smithi144 ceph-mon[32129]: pgmap v9832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:47 smithi160 ceph-mon[36715]: pgmap v9832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:47 smithi183 ceph-mon[37273]: pgmap v9832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:00:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:49 smithi144 ceph-mon[32129]: pgmap v9833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:00:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:49 smithi160 ceph-mon[36715]: pgmap v9833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:00:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:49 smithi183 ceph-mon[37273]: pgmap v9833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:00:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:00:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:50 smithi144 ceph-mon[32129]: pgmap v9834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:00:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:50 smithi144 ceph-mon[32129]: pgmap v9835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:00:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:50 smithi160 ceph-mon[36715]: pgmap v9834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:00:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:50 smithi160 ceph-mon[36715]: pgmap v9835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:00:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:50 smithi183 ceph-mon[37273]: pgmap v9834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:00:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:50 smithi183 ceph-mon[37273]: pgmap v9835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:00:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17694.939122 secs 2024-07-26T16:00:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:51 smithi160 ceph-mon[36715]: pgmap v9836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17694.939122 secs 2024-07-26T16:00:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:51 smithi183 ceph-mon[37273]: pgmap v9836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17694.939122 secs 2024-07-26T16:00:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:51 smithi144 ceph-mon[32129]: pgmap v9836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:54 smithi144 ceph-mon[32129]: pgmap v9837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:00:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:54 smithi160 ceph-mon[36715]: pgmap v9837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:00:55.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:54 smithi183 ceph-mon[37273]: pgmap v9837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:00:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:56 smithi144 ceph-mon[32129]: pgmap v9838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17699.939190 secs 2024-07-26T16:00:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:56 smithi160 ceph-mon[36715]: pgmap v9838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17699.939190 secs 2024-07-26T16:00:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:56 smithi183 ceph-mon[37273]: pgmap v9838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:00:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17699.939190 secs 2024-07-26T16:00:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:57.772+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:00:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:57.773+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:00:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:57.773+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:00:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:57.773+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:00:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:00:57.774+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:58.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:00:58 smithi144 ceph-mon[32129]: pgmap v9839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:00:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:00:58 smithi160 ceph-mon[36715]: pgmap v9839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:00:59.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:00:58 smithi183 ceph-mon[37273]: pgmap v9839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:00 smithi144 ceph-mon[32129]: pgmap v9840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:01:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:00 smithi160 ceph-mon[36715]: pgmap v9840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:01:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:00 smithi183 ceph-mon[37273]: pgmap v9840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:01:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17704.939336 secs 2024-07-26T16:01:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17704.939336 secs 2024-07-26T16:01:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17704.939336 secs 2024-07-26T16:01:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:02 smithi160 ceph-mon[36715]: pgmap v9841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:02 smithi183 ceph-mon[37273]: pgmap v9841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:02 smithi144 ceph-mon[32129]: pgmap v9841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:04 smithi160 ceph-mon[36715]: pgmap v9842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:05.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:04 smithi183 ceph-mon[37273]: pgmap v9842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:04 smithi144 ceph-mon[32129]: pgmap v9842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:06 smithi160 ceph-mon[36715]: pgmap v9843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17709.939453 secs 2024-07-26T16:01:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:06 smithi183 ceph-mon[37273]: pgmap v9843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17709.939453 secs 2024-07-26T16:01:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:06 smithi144 ceph-mon[32129]: pgmap v9843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17709.939453 secs 2024-07-26T16:01:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:08 smithi160 ceph-mon[36715]: pgmap v9844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:09.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:08 smithi183 ceph-mon[37273]: pgmap v9844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:09.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:08 smithi144 ceph-mon[32129]: pgmap v9844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:10 smithi160 ceph-mon[36715]: pgmap v9845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:10 smithi183 ceph-mon[37273]: pgmap v9845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:11.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:10 smithi144 ceph-mon[32129]: pgmap v9845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17714.939575 secs 2024-07-26T16:01:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17714.939575 secs 2024-07-26T16:01:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17714.939575 secs 2024-07-26T16:01:12.791 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:12.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:12.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:12.789+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:12.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:12.789+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:12 smithi160 ceph-mon[36715]: pgmap v9846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:12 smithi183 ceph-mon[37273]: pgmap v9846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:12 smithi144 ceph-mon[32129]: pgmap v9846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:14 smithi160 ceph-mon[36715]: pgmap v9847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:14 smithi183 ceph-mon[37273]: pgmap v9847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:14 smithi144 ceph-mon[32129]: pgmap v9847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:16 smithi160 ceph-mon[36715]: pgmap v9848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17719.939701 secs 2024-07-26T16:01:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:16 smithi183 ceph-mon[37273]: pgmap v9848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17719.939701 secs 2024-07-26T16:01:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:16 smithi144 ceph-mon[32129]: pgmap v9848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17719.939701 secs 2024-07-26T16:01:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:18 smithi160 ceph-mon[36715]: pgmap v9849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:18 smithi183 ceph-mon[37273]: pgmap v9849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:18 smithi144 ceph-mon[32129]: pgmap v9849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:20 smithi160 ceph-mon[36715]: pgmap v9850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:20 smithi183 ceph-mon[37273]: pgmap v9850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:20 smithi144 ceph-mon[32129]: pgmap v9850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17724.939825 secs 2024-07-26T16:01:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17724.939825 secs 2024-07-26T16:01:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17724.939825 secs 2024-07-26T16:01:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:22 smithi160 ceph-mon[36715]: pgmap v9851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:22 smithi183 ceph-mon[37273]: pgmap v9851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:22 smithi144 ceph-mon[32129]: pgmap v9851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:24 smithi160 ceph-mon[36715]: pgmap v9852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:24 smithi183 ceph-mon[37273]: pgmap v9852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:24 smithi144 ceph-mon[32129]: pgmap v9852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:26 smithi183 ceph-mon[37273]: pgmap v9853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17729.939970 secs 2024-07-26T16:01:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:26 smithi144 ceph-mon[32129]: pgmap v9853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17729.939970 secs 2024-07-26T16:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:26 smithi160 ceph-mon[36715]: pgmap v9853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17729.939970 secs 2024-07-26T16:01:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:27.806+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:01:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:27.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:01:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:27.807+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:01:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:27.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:01:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:27.807+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:28 smithi183 ceph-mon[37273]: pgmap v9854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:28 smithi144 ceph-mon[32129]: pgmap v9854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:28 smithi160 ceph-mon[36715]: pgmap v9854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:30 smithi183 ceph-mon[37273]: pgmap v9855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:30 smithi144 ceph-mon[32129]: pgmap v9855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:30 smithi160 ceph-mon[36715]: pgmap v9855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17734.940110 secs 2024-07-26T16:01:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17734.940110 secs 2024-07-26T16:01:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17734.940110 secs 2024-07-26T16:01:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:32 smithi183 ceph-mon[37273]: pgmap v9856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:32 smithi144 ceph-mon[32129]: pgmap v9856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:32 smithi160 ceph-mon[36715]: pgmap v9856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:34 smithi183 ceph-mon[37273]: pgmap v9857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:34 smithi144 ceph-mon[32129]: pgmap v9857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:34 smithi160 ceph-mon[36715]: pgmap v9857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:36 smithi183 ceph-mon[37273]: pgmap v9858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17739.940247 secs 2024-07-26T16:01:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:36 smithi144 ceph-mon[32129]: pgmap v9858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17739.940247 secs 2024-07-26T16:01:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:36 smithi160 ceph-mon[36715]: pgmap v9858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17739.940247 secs 2024-07-26T16:01:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:38 smithi183 ceph-mon[37273]: pgmap v9859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:38 smithi144 ceph-mon[32129]: pgmap v9859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:38 smithi160 ceph-mon[36715]: pgmap v9859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:40 smithi183 ceph-mon[37273]: pgmap v9860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:40 smithi144 ceph-mon[32129]: pgmap v9860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:40 smithi160 ceph-mon[36715]: pgmap v9860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17744.940369 secs 2024-07-26T16:01:42.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17744.940369 secs 2024-07-26T16:01:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17744.940369 secs 2024-07-26T16:01:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:42.823+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:01:42.825 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:42.824+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:01:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:42.824+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:01:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:42.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:01:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:42.824+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:42 smithi183 ceph-mon[37273]: pgmap v9861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:43.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:42 smithi144 ceph-mon[32129]: pgmap v9861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:42 smithi160 ceph-mon[36715]: pgmap v9861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:01:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:44 smithi183 ceph-mon[37273]: pgmap v9862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:44 smithi144 ceph-mon[32129]: pgmap v9862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:44 smithi160 ceph-mon[36715]: pgmap v9862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:46 smithi183 ceph-mon[37273]: pgmap v9863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17749.940493 secs 2024-07-26T16:01:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:46 smithi144 ceph-mon[32129]: pgmap v9863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17749.940493 secs 2024-07-26T16:01:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:46 smithi160 ceph-mon[36715]: pgmap v9863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:47.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17749.940493 secs 2024-07-26T16:01:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:48 smithi144 ceph-mon[32129]: pgmap v9864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:48 smithi160 ceph-mon[36715]: pgmap v9864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:48 smithi183 ceph-mon[37273]: pgmap v9864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:01:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:50 smithi144 ceph-mon[32129]: pgmap v9865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:50 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:01:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:50 smithi160 ceph-mon[36715]: pgmap v9865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:50 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:01:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:50 smithi183 ceph-mon[37273]: pgmap v9865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:01:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:50 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:01:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:51 smithi144 ceph-mon[32129]: pgmap v9866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:51 smithi144 ceph-mon[32129]: pgmap v9867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17754.940558 secs 2024-07-26T16:01:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:51 smithi160 ceph-mon[36715]: pgmap v9866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:51 smithi160 ceph-mon[36715]: pgmap v9867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17754.940558 secs 2024-07-26T16:01:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:51 smithi183 ceph-mon[37273]: pgmap v9866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:51 smithi183 ceph-mon[37273]: pgmap v9867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17754.940558 secs 2024-07-26T16:01:54.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:53 smithi144 ceph-mon[32129]: pgmap v9868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:53 smithi160 ceph-mon[36715]: pgmap v9868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:53 smithi183 ceph-mon[37273]: pgmap v9868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:55 smithi144 ceph-mon[32129]: pgmap v9869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:55 smithi160 ceph-mon[36715]: pgmap v9869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:55 smithi183 ceph-mon[37273]: pgmap v9869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:01:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17759.940689 secs 2024-07-26T16:01:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17759.940689 secs 2024-07-26T16:01:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17759.940689 secs 2024-07-26T16:01:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:57.837+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:01:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:57.838+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:01:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:57.838+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:01:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:57.838+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:01:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:01:57.838+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:57 smithi144 ceph-mon[32129]: pgmap v9870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:57 smithi160 ceph-mon[36715]: pgmap v9870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:01:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:57 smithi183 ceph-mon[37273]: pgmap v9870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:01:59 smithi144 ceph-mon[32129]: pgmap v9871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:02:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:01:59 smithi160 ceph-mon[36715]: pgmap v9871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:02:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:01:59 smithi183 ceph-mon[37273]: pgmap v9871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:02:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:01 smithi144 ceph-mon[32129]: pgmap v9872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:02:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17764.940775 secs 2024-07-26T16:02:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:01 smithi160 ceph-mon[36715]: pgmap v9872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:02:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17764.940775 secs 2024-07-26T16:02:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:01 smithi183 ceph-mon[37273]: pgmap v9872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:02:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17764.940775 secs 2024-07-26T16:02:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:03 smithi144 ceph-mon[32129]: pgmap v9873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:03 smithi160 ceph-mon[36715]: pgmap v9873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:03 smithi183 ceph-mon[37273]: pgmap v9873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:05 smithi144 ceph-mon[32129]: pgmap v9874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:05 smithi160 ceph-mon[36715]: pgmap v9874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:05 smithi183 ceph-mon[37273]: pgmap v9874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17769.940891 secs 2024-07-26T16:02:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17769.940891 secs 2024-07-26T16:02:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17769.940891 secs 2024-07-26T16:02:08.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:07 smithi183 ceph-mon[37273]: pgmap v9875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:07 smithi160 ceph-mon[36715]: pgmap v9875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:07 smithi144 ceph-mon[32129]: pgmap v9875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:08 smithi160 ceph-mon[36715]: pgmap v9876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:08 smithi183 ceph-mon[37273]: pgmap v9876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:08 smithi144 ceph-mon[32129]: pgmap v9876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:11 smithi144 ceph-mon[32129]: pgmap v9877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17774.940956 secs 2024-07-26T16:02:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:11 smithi160 ceph-mon[36715]: pgmap v9877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17774.940956 secs 2024-07-26T16:02:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:11 smithi183 ceph-mon[37273]: pgmap v9877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17774.940956 secs 2024-07-26T16:02:12.855 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:12.853+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:02:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:12.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:02:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:12.854+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:02:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:12.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:02:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:12.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:13 smithi144 ceph-mon[32129]: pgmap v9878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:13 smithi160 ceph-mon[36715]: pgmap v9878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:13 smithi183 ceph-mon[37273]: pgmap v9878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:15 smithi144 ceph-mon[32129]: pgmap v9879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:15 smithi160 ceph-mon[36715]: pgmap v9879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:15 smithi183 ceph-mon[37273]: pgmap v9879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17779.941075 secs 2024-07-26T16:02:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17779.941075 secs 2024-07-26T16:02:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17779.941075 secs 2024-07-26T16:02:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:17 smithi160 ceph-mon[36715]: pgmap v9880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:17 smithi183 ceph-mon[37273]: pgmap v9880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:17 smithi144 ceph-mon[32129]: pgmap v9880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:18 smithi160 ceph-mon[36715]: pgmap v9881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:19.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:18 smithi144 ceph-mon[32129]: pgmap v9881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:18 smithi183 ceph-mon[37273]: pgmap v9881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:21 smithi144 ceph-mon[32129]: pgmap v9882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17784.941187 secs 2024-07-26T16:02:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:21 smithi160 ceph-mon[36715]: pgmap v9882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17784.941187 secs 2024-07-26T16:02:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:21 smithi183 ceph-mon[37273]: pgmap v9882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17784.941187 secs 2024-07-26T16:02:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:23 smithi160 ceph-mon[36715]: pgmap v9883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:23 smithi183 ceph-mon[37273]: pgmap v9883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:23 smithi144 ceph-mon[32129]: pgmap v9883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:25 smithi160 ceph-mon[36715]: pgmap v9884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:25 smithi183 ceph-mon[37273]: pgmap v9884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:25 smithi144 ceph-mon[32129]: pgmap v9884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17789.941321 secs 2024-07-26T16:02:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17789.941321 secs 2024-07-26T16:02:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17789.941321 secs 2024-07-26T16:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:27.870+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:02:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:27.871+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:02:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:27.871+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:02:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:27.871+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:02:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:27.871+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:27 smithi160 ceph-mon[36715]: pgmap v9885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:28.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:27 smithi183 ceph-mon[37273]: pgmap v9885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:27 smithi144 ceph-mon[32129]: pgmap v9885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:28 smithi160 ceph-mon[36715]: pgmap v9886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:28 smithi183 ceph-mon[37273]: pgmap v9886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:28 smithi144 ceph-mon[32129]: pgmap v9886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:31 smithi160 ceph-mon[36715]: pgmap v9887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:32.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17794.941443 secs 2024-07-26T16:02:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:31 smithi183 ceph-mon[37273]: pgmap v9887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17794.941443 secs 2024-07-26T16:02:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:31 smithi144 ceph-mon[32129]: pgmap v9887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17794.941443 secs 2024-07-26T16:02:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:33 smithi160 ceph-mon[36715]: pgmap v9888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:33 smithi183 ceph-mon[37273]: pgmap v9888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:33 smithi144 ceph-mon[32129]: pgmap v9888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:35 smithi160 ceph-mon[36715]: pgmap v9889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:35 smithi183 ceph-mon[37273]: pgmap v9889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:35 smithi144 ceph-mon[32129]: pgmap v9889: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17799.941562 secs 2024-07-26T16:02:36.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17799.941562 secs 2024-07-26T16:02:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17799.941562 secs 2024-07-26T16:02:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:37 smithi183 ceph-mon[37273]: pgmap v9890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:37 smithi144 ceph-mon[32129]: pgmap v9890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:37 smithi160 ceph-mon[36715]: pgmap v9890: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:39 smithi160 ceph-mon[36715]: pgmap v9891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:40.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:39 smithi183 ceph-mon[37273]: pgmap v9891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:39 smithi144 ceph-mon[32129]: pgmap v9891: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:41 smithi160 ceph-mon[36715]: pgmap v9892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17804.941685 secs 2024-07-26T16:02:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:41 smithi183 ceph-mon[37273]: pgmap v9892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17804.941685 secs 2024-07-26T16:02:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:41 smithi144 ceph-mon[32129]: pgmap v9892: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17804.941685 secs 2024-07-26T16:02:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:42.886+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:02:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:42.887+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:02:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:42.887+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:02:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:42.887+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:02:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:42.887+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:42 smithi160 ceph-mon[36715]: pgmap v9893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:42 smithi183 ceph-mon[37273]: pgmap v9893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:42 smithi144 ceph-mon[32129]: pgmap v9893: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:02:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:45 smithi160 ceph-mon[36715]: pgmap v9894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:45 smithi183 ceph-mon[37273]: pgmap v9894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:45 smithi144 ceph-mon[32129]: pgmap v9894: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17809.941805 secs 2024-07-26T16:02:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17809.941805 secs 2024-07-26T16:02:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17809.941805 secs 2024-07-26T16:02:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:47 smithi160 ceph-mon[36715]: pgmap v9895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:47 smithi183 ceph-mon[37273]: pgmap v9895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:47 smithi144 ceph-mon[32129]: pgmap v9895: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:02:49.244 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.243+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T16:02:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.243+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T16:02:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.244+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f74800065d0 con 0x7f74b0109990 2024-07-26T16:02:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.244+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:02:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.244+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7478006250 con 0x563c6c4182f0 2024-07-26T16:02:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.244+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:02:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.245+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T16:02:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:49.245+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T16:02:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:48 smithi160 ceph-mon[36715]: pgmap v9896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:49.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:48 smithi144 ceph-mon[32129]: pgmap v9896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:49.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:48 smithi183 ceph-mon[37273]: pgmap v9896: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:51 smithi144 ceph-mon[32129]: pgmap v9897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17814.941926 secs 2024-07-26T16:02:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:51 smithi160 ceph-mon[36715]: pgmap v9897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17814.941926 secs 2024-07-26T16:02:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:51 smithi183 ceph-mon[37273]: pgmap v9897: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:02:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17814.941926 secs 2024-07-26T16:02:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:52 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:02:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:52 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:02:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:52 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:02:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:53 smithi160 ceph-mon[36715]: pgmap v9898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:02:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:53 smithi183 ceph-mon[37273]: pgmap v9898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:02:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:53 smithi144 ceph-mon[32129]: pgmap v9898: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:02:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:55 smithi160 ceph-mon[36715]: pgmap v9899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:55.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:55 smithi183 ceph-mon[37273]: pgmap v9899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:55 smithi144 ceph-mon[32129]: pgmap v9899: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17819.942078 secs 2024-07-26T16:02:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:57 smithi183 ceph-mon[37273]: pgmap v9900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17819.942078 secs 2024-07-26T16:02:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:57 smithi144 ceph-mon[32129]: pgmap v9900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17819.942078 secs 2024-07-26T16:02:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:57 smithi160 ceph-mon[36715]: pgmap v9900: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:02:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:57.903+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:02:57.906 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:57.904+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:02:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:57.904+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:02:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:57.904+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:02:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:02:57.905+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:59.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:02:59 smithi144 ceph-mon[32129]: pgmap v9901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:02:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:02:59 smithi160 ceph-mon[36715]: pgmap v9901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:02:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:02:59 smithi183 ceph-mon[37273]: pgmap v9901: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:01 smithi144 ceph-mon[32129]: pgmap v9902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:03:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:01 smithi160 ceph-mon[36715]: pgmap v9902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:03:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:01 smithi183 ceph-mon[37273]: pgmap v9902: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:03:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17824.942210 secs 2024-07-26T16:03:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17824.942210 secs 2024-07-26T16:03:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17824.942210 secs 2024-07-26T16:03:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:03 smithi144 ceph-mon[32129]: pgmap v9903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:03 smithi160 ceph-mon[36715]: pgmap v9903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:03 smithi183 ceph-mon[37273]: pgmap v9903: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:05 smithi144 ceph-mon[32129]: pgmap v9904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:05 smithi160 ceph-mon[36715]: pgmap v9904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:05 smithi183 ceph-mon[37273]: pgmap v9904: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17829.942271 secs 2024-07-26T16:03:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:07 smithi144 ceph-mon[32129]: pgmap v9905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17829.942271 secs 2024-07-26T16:03:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:07 smithi160 ceph-mon[36715]: pgmap v9905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17829.942271 secs 2024-07-26T16:03:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:07 smithi183 ceph-mon[37273]: pgmap v9905: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:09 smithi144 ceph-mon[32129]: pgmap v9906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:09 smithi160 ceph-mon[36715]: pgmap v9906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:09 smithi183 ceph-mon[37273]: pgmap v9906: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:11.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:11 smithi183 ceph-mon[37273]: pgmap v9907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:11 smithi144 ceph-mon[32129]: pgmap v9907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:11 smithi160 ceph-mon[36715]: pgmap v9907: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:12.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17834.942377 secs 2024-07-26T16:03:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17834.942377 secs 2024-07-26T16:03:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17834.942377 secs 2024-07-26T16:03:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:12.920+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:03:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:12.922+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:03:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:12.922+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:03:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:12.922+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:03:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:12.922+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:13 smithi144 ceph-mon[32129]: pgmap v9908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:13 smithi160 ceph-mon[36715]: pgmap v9908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:13 smithi183 ceph-mon[37273]: pgmap v9908: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:15.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:15 smithi144 ceph-mon[32129]: pgmap v9909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:15 smithi160 ceph-mon[36715]: pgmap v9909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:15 smithi183 ceph-mon[37273]: pgmap v9909: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17839.942462 secs 2024-07-26T16:03:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:17 smithi144 ceph-mon[32129]: pgmap v9910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17839.942462 secs 2024-07-26T16:03:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:17 smithi160 ceph-mon[36715]: pgmap v9910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17839.942462 secs 2024-07-26T16:03:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:17 smithi183 ceph-mon[37273]: pgmap v9910: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:19 smithi144 ceph-mon[32129]: pgmap v9911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:19 smithi160 ceph-mon[36715]: pgmap v9911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:19 smithi183 ceph-mon[37273]: pgmap v9911: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:21 smithi144 ceph-mon[32129]: pgmap v9912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:21 smithi160 ceph-mon[36715]: pgmap v9912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:21 smithi183 ceph-mon[37273]: pgmap v9912: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17844.942586 secs 2024-07-26T16:03:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17844.942586 secs 2024-07-26T16:03:22.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17844.942586 secs 2024-07-26T16:03:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:23 smithi144 ceph-mon[32129]: pgmap v9913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:23 smithi160 ceph-mon[36715]: pgmap v9913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:23 smithi183 ceph-mon[37273]: pgmap v9913: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:25 smithi160 ceph-mon[36715]: pgmap v9914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:25 smithi183 ceph-mon[37273]: pgmap v9914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:25 smithi144 ceph-mon[32129]: pgmap v9914: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17849.942706 secs 2024-07-26T16:03:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17849.942706 secs 2024-07-26T16:03:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17849.942706 secs 2024-07-26T16:03:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:27.934+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:03:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:27.935+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:03:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:27.935+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:03:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:27.935+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:03:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:27.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:27 smithi160 ceph-mon[36715]: pgmap v9915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:27 smithi183 ceph-mon[37273]: pgmap v9915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:27 smithi144 ceph-mon[32129]: pgmap v9915: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:29 smithi160 ceph-mon[36715]: pgmap v9916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:29 smithi183 ceph-mon[37273]: pgmap v9916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:29 smithi144 ceph-mon[32129]: pgmap v9916: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:31 smithi160 ceph-mon[36715]: pgmap v9917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17854.942791 secs 2024-07-26T16:03:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:31 smithi183 ceph-mon[37273]: pgmap v9917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17854.942791 secs 2024-07-26T16:03:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:31 smithi144 ceph-mon[32129]: pgmap v9917: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17854.942791 secs 2024-07-26T16:03:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:33 smithi160 ceph-mon[36715]: pgmap v9918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:33 smithi183 ceph-mon[37273]: pgmap v9918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:33 smithi144 ceph-mon[32129]: pgmap v9918: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:35 smithi160 ceph-mon[36715]: pgmap v9919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:35 smithi183 ceph-mon[37273]: pgmap v9919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:35 smithi144 ceph-mon[32129]: pgmap v9919: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17859.942927 secs 2024-07-26T16:03:37.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17859.942927 secs 2024-07-26T16:03:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17859.942927 secs 2024-07-26T16:03:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:37 smithi160 ceph-mon[36715]: pgmap v9920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:38.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:37 smithi183 ceph-mon[37273]: pgmap v9920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:37 smithi144 ceph-mon[32129]: pgmap v9920: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:39 smithi160 ceph-mon[36715]: pgmap v9921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:39 smithi183 ceph-mon[37273]: pgmap v9921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:39 smithi144 ceph-mon[32129]: pgmap v9921: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:41 smithi160 ceph-mon[36715]: pgmap v9922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17864.942990 secs 2024-07-26T16:03:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:41 smithi183 ceph-mon[37273]: pgmap v9922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17864.942990 secs 2024-07-26T16:03:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:41 smithi144 ceph-mon[32129]: pgmap v9922: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17864.942990 secs 2024-07-26T16:03:42.943 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:42.941+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:03:42.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:42.943+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:03:42.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:42.943+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:03:42.944 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:42.943+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:03:42.945 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:42.943+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:43 smithi160 ceph-mon[36715]: pgmap v9923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:43 smithi183 ceph-mon[37273]: pgmap v9923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:43 smithi144 ceph-mon[32129]: pgmap v9923: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:45 smithi160 ceph-mon[36715]: pgmap v9924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:45 smithi183 ceph-mon[37273]: pgmap v9924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:45 smithi144 ceph-mon[32129]: pgmap v9924: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17869.943139 secs 2024-07-26T16:03:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17869.943139 secs 2024-07-26T16:03:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17869.943139 secs 2024-07-26T16:03:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:47 smithi183 ceph-mon[37273]: pgmap v9925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:47 smithi144 ceph-mon[32129]: pgmap v9925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:47 smithi160 ceph-mon[36715]: pgmap v9925: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:49 smithi183 ceph-mon[37273]: pgmap v9926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:49 smithi144 ceph-mon[32129]: pgmap v9926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:49 smithi160 ceph-mon[36715]: pgmap v9926: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:03:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:51 smithi183 ceph-mon[37273]: pgmap v9927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17874.943263 secs 2024-07-26T16:03:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:51 smithi144 ceph-mon[32129]: pgmap v9927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17874.943263 secs 2024-07-26T16:03:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:51 smithi160 ceph-mon[36715]: pgmap v9927: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:03:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17874.943263 secs 2024-07-26T16:03:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:52 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:52 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:52 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:54.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:53 smithi144 ceph-mon[32129]: pgmap v9928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:54.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:54.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:53 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:54.123 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:53 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:03:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:53 smithi183 ceph-mon[37273]: pgmap v9928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:53 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:53 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:03:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:53 smithi160 ceph-mon[36715]: pgmap v9928: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:03:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:53 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:53 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:03:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:54 smithi183 ceph-mon[37273]: pgmap v9929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:54 smithi183 ceph-mon[37273]: pgmap v9930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:03:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:54 smithi144 ceph-mon[32129]: pgmap v9929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:54 smithi144 ceph-mon[32129]: pgmap v9930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:03:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:54 smithi160 ceph-mon[36715]: pgmap v9929: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:03:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:54 smithi160 ceph-mon[36715]: pgmap v9930: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:03:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:56 smithi183 ceph-mon[37273]: pgmap v9931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:03:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17879.943389 secs 2024-07-26T16:03:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:56 smithi144 ceph-mon[32129]: pgmap v9931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:03:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17879.943389 secs 2024-07-26T16:03:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:56 smithi160 ceph-mon[36715]: pgmap v9931: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:03:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17879.943389 secs 2024-07-26T16:03:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:57.959+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:03:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:57.960+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:03:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:57.960+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:03:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:57.960+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:03:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:03:57.960+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:03:58 smithi183 ceph-mon[37273]: pgmap v9932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:03:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:03:58 smithi144 ceph-mon[32129]: pgmap v9932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:03:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:03:58 smithi160 ceph-mon[36715]: pgmap v9932: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:00 smithi183 ceph-mon[37273]: pgmap v9933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:00 smithi144 ceph-mon[32129]: pgmap v9933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:00 smithi160 ceph-mon[36715]: pgmap v9933: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17884.943489 secs 2024-07-26T16:04:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17884.943489 secs 2024-07-26T16:04:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17884.943489 secs 2024-07-26T16:04:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:02 smithi144 ceph-mon[32129]: pgmap v9934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:02 smithi160 ceph-mon[36715]: pgmap v9934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:02 smithi183 ceph-mon[37273]: pgmap v9934: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:04:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:04 smithi144 ceph-mon[32129]: pgmap v9935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:04:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:04 smithi160 ceph-mon[36715]: pgmap v9935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:04:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:04 smithi183 ceph-mon[37273]: pgmap v9935: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:04:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:06 smithi144 ceph-mon[32129]: pgmap v9936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17889.943576 secs 2024-07-26T16:04:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:06 smithi160 ceph-mon[36715]: pgmap v9936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17889.943576 secs 2024-07-26T16:04:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:06 smithi183 ceph-mon[37273]: pgmap v9936: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17889.943576 secs 2024-07-26T16:04:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:08 smithi144 ceph-mon[32129]: pgmap v9937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:08 smithi160 ceph-mon[36715]: pgmap v9937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:08 smithi183 ceph-mon[37273]: pgmap v9937: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:11.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:10 smithi183 ceph-mon[37273]: pgmap v9938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:10 smithi144 ceph-mon[32129]: pgmap v9938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:10 smithi160 ceph-mon[36715]: pgmap v9938: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17894.943694 secs 2024-07-26T16:04:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17894.943694 secs 2024-07-26T16:04:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17894.943694 secs 2024-07-26T16:04:12.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:12.968+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:04:12.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:12.969+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:04:12.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:12.969+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:04:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:12.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:04:12.972 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:12.969+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:13.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:12 smithi160 ceph-mon[36715]: pgmap v9939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:12 smithi144 ceph-mon[32129]: pgmap v9939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:12 smithi183 ceph-mon[37273]: pgmap v9939: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:14 smithi144 ceph-mon[32129]: pgmap v9940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:14 smithi160 ceph-mon[36715]: pgmap v9940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:14 smithi183 ceph-mon[37273]: pgmap v9940: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:16 smithi144 ceph-mon[32129]: pgmap v9941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17899.943819 secs 2024-07-26T16:04:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:16 smithi160 ceph-mon[36715]: pgmap v9941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17899.943819 secs 2024-07-26T16:04:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:16 smithi183 ceph-mon[37273]: pgmap v9941: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17899.943819 secs 2024-07-26T16:04:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:18 smithi144 ceph-mon[32129]: pgmap v9942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:18 smithi160 ceph-mon[36715]: pgmap v9942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:18 smithi183 ceph-mon[37273]: pgmap v9942: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:20 smithi144 ceph-mon[32129]: pgmap v9943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:20 smithi160 ceph-mon[36715]: pgmap v9943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:20 smithi183 ceph-mon[37273]: pgmap v9943: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17904.943950 secs 2024-07-26T16:04:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17904.943950 secs 2024-07-26T16:04:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17904.943950 secs 2024-07-26T16:04:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:22 smithi144 ceph-mon[32129]: pgmap v9944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:22 smithi160 ceph-mon[36715]: pgmap v9944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:22 smithi183 ceph-mon[37273]: pgmap v9944: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:24 smithi144 ceph-mon[32129]: pgmap v9945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:24 smithi160 ceph-mon[36715]: pgmap v9945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:24 smithi183 ceph-mon[37273]: pgmap v9945: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:26 smithi160 ceph-mon[36715]: pgmap v9946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17909.944089 secs 2024-07-26T16:04:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:26 smithi183 ceph-mon[37273]: pgmap v9946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17909.944089 secs 2024-07-26T16:04:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:26 smithi144 ceph-mon[32129]: pgmap v9946: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17909.944089 secs 2024-07-26T16:04:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:27.985+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:04:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:27.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:04:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:27.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:04:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:27.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:04:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:27.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:28 smithi160 ceph-mon[36715]: pgmap v9947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:28 smithi183 ceph-mon[37273]: pgmap v9947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:28 smithi144 ceph-mon[32129]: pgmap v9947: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:30 smithi160 ceph-mon[36715]: pgmap v9948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:30 smithi183 ceph-mon[37273]: pgmap v9948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:30 smithi144 ceph-mon[32129]: pgmap v9948: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17914.944214 secs 2024-07-26T16:04:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17914.944214 secs 2024-07-26T16:04:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17914.944214 secs 2024-07-26T16:04:33.313 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:32 smithi160 ceph-mon[36715]: pgmap v9949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:32 smithi183 ceph-mon[37273]: pgmap v9949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:32 smithi144 ceph-mon[32129]: pgmap v9949: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:35 smithi160 ceph-mon[36715]: pgmap v9950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:35 smithi183 ceph-mon[37273]: pgmap v9950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:35.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:35 smithi144 ceph-mon[32129]: pgmap v9950: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:37 smithi160 ceph-mon[36715]: pgmap v9951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17919.944327 secs 2024-07-26T16:04:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:37 smithi183 ceph-mon[37273]: pgmap v9951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17919.944327 secs 2024-07-26T16:04:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:37 smithi144 ceph-mon[32129]: pgmap v9951: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17919.944327 secs 2024-07-26T16:04:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:38 smithi160 ceph-mon[36715]: pgmap v9952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:38 smithi183 ceph-mon[37273]: pgmap v9952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:38 smithi144 ceph-mon[32129]: pgmap v9952: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:40 smithi183 ceph-mon[37273]: pgmap v9953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:40 smithi144 ceph-mon[32129]: pgmap v9953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:40 smithi160 ceph-mon[36715]: pgmap v9953: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17924.944444 secs 2024-07-26T16:04:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17924.944444 secs 2024-07-26T16:04:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17924.944444 secs 2024-07-26T16:04:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:42 smithi183 ceph-mon[37273]: pgmap v9954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:42.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:42 smithi144 ceph-mon[32129]: pgmap v9954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:42 smithi160 ceph-mon[36715]: pgmap v9954: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:43.001+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:04:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:43.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:04:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:43.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:04:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:43.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:04:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:43.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:44.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:44 smithi160 ceph-mon[36715]: pgmap v9955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:44.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:44 smithi183 ceph-mon[37273]: pgmap v9955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:44.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:44 smithi144 ceph-mon[32129]: pgmap v9955: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:46 smithi183 ceph-mon[37273]: pgmap v9956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17929.944552 secs 2024-07-26T16:04:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:46 smithi144 ceph-mon[32129]: pgmap v9956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17929.944552 secs 2024-07-26T16:04:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:46 smithi160 ceph-mon[36715]: pgmap v9956: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17929.944552 secs 2024-07-26T16:04:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:48 smithi183 ceph-mon[37273]: pgmap v9957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:48 smithi144 ceph-mon[32129]: pgmap v9957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:48 smithi160 ceph-mon[36715]: pgmap v9957: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:50 smithi183 ceph-mon[37273]: pgmap v9958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:50 smithi144 ceph-mon[32129]: pgmap v9958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:50 smithi160 ceph-mon[36715]: pgmap v9958: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:04:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17934.944673 secs 2024-07-26T16:04:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17934.944673 secs 2024-07-26T16:04:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17934.944673 secs 2024-07-26T16:04:52.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:52 smithi144 ceph-mon[32129]: pgmap v9959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:52 smithi160 ceph-mon[36715]: pgmap v9959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:52 smithi183 ceph-mon[37273]: pgmap v9959: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:54.713 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:54 smithi160 ceph-mon[36715]: pgmap v9960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:54.713 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:54 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:54 smithi144 ceph-mon[32129]: pgmap v9960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:54.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:54 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:54 smithi183 ceph-mon[37273]: pgmap v9960: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:04:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:54 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:56 smithi144 ceph-mon[32129]: pgmap v9961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T16:04:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17939.944765 secs 2024-07-26T16:04:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:56 smithi160 ceph-mon[36715]: pgmap v9961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T16:04:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17939.944765 secs 2024-07-26T16:04:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:56 smithi183 ceph-mon[37273]: pgmap v9961: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T16:04:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17939.944765 secs 2024-07-26T16:04:58.019 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:58.017+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:04:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:58.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:04:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:58.018+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:04:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:58.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:04:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:04:58.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:58 smithi183 ceph-mon[37273]: pgmap v9962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:04:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:04:58 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:04:58.673 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:58 smithi144 ceph-mon[32129]: pgmap v9962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:58.673 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:04:58.673 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:04:58 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:04:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:58 smithi160 ceph-mon[36715]: pgmap v9962: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:04:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:04:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:04:58 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: pgmap v9963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: pgmap v9964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:00.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: pgmap v9965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:05:00.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: pgmap v9963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:00.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:05:00.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:05:00.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:00.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:00.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: pgmap v9964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:00.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: pgmap v9965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:05:00.972 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: pgmap v9963: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:01.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: pgmap v9964: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:01.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: pgmap v9965: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:05:01.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:05:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17944.944911 secs 2024-07-26T16:05:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17944.944911 secs 2024-07-26T16:05:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17944.944911 secs 2024-07-26T16:05:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:02 smithi183 ceph-mon[37273]: pgmap v9966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T16:05:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:02 smithi144 ceph-mon[32129]: pgmap v9966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T16:05:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:02 smithi160 ceph-mon[36715]: pgmap v9966: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T16:05:04.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:04 smithi144 ceph-mon[32129]: pgmap v9967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:05:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:04 smithi160 ceph-mon[36715]: pgmap v9967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:05:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:04 smithi183 ceph-mon[37273]: pgmap v9967: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:05:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:06 smithi144 ceph-mon[32129]: pgmap v9968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T16:05:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17949.945051 secs 2024-07-26T16:05:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:06 smithi160 ceph-mon[36715]: pgmap v9968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T16:05:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17949.945051 secs 2024-07-26T16:05:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:06 smithi183 ceph-mon[37273]: pgmap v9968: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T16:05:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17949.945051 secs 2024-07-26T16:05:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:08 smithi144 ceph-mon[32129]: pgmap v9969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:05:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:08 smithi160 ceph-mon[36715]: pgmap v9969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:05:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:08 smithi183 ceph-mon[37273]: pgmap v9969: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:05:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:10 smithi144 ceph-mon[32129]: pgmap v9970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:10 smithi160 ceph-mon[36715]: pgmap v9970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:10 smithi183 ceph-mon[37273]: pgmap v9970: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17954.945188 secs 2024-07-26T16:05:12.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17954.945188 secs 2024-07-26T16:05:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17954.945188 secs 2024-07-26T16:05:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:12 smithi144 ceph-mon[32129]: pgmap v9971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:13.034+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:05:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:13.035+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:05:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:13.035+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:05:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:13.035+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:05:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:13.035+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:12 smithi160 ceph-mon[36715]: pgmap v9971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:12 smithi183 ceph-mon[37273]: pgmap v9971: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:14 smithi144 ceph-mon[32129]: pgmap v9972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:14 smithi160 ceph-mon[36715]: pgmap v9972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:14 smithi183 ceph-mon[37273]: pgmap v9972: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:16 smithi144 ceph-mon[32129]: pgmap v9973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17959.945273 secs 2024-07-26T16:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:16 smithi160 ceph-mon[36715]: pgmap v9973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17959.945273 secs 2024-07-26T16:05:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:16 smithi183 ceph-mon[37273]: pgmap v9973: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:05:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17959.945273 secs 2024-07-26T16:05:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:18 smithi144 ceph-mon[32129]: pgmap v9974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:05:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:18 smithi160 ceph-mon[36715]: pgmap v9974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:05:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:18 smithi183 ceph-mon[37273]: pgmap v9974: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:05:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:20 smithi144 ceph-mon[32129]: pgmap v9975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T16:05:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:20 smithi160 ceph-mon[36715]: pgmap v9975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T16:05:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:20 smithi183 ceph-mon[37273]: pgmap v9975: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T16:05:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17964.945371 secs 2024-07-26T16:05:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17964.945371 secs 2024-07-26T16:05:22.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17964.945371 secs 2024-07-26T16:05:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:22 smithi144 ceph-mon[32129]: pgmap v9976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:22 smithi160 ceph-mon[36715]: pgmap v9976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:22 smithi183 ceph-mon[37273]: pgmap v9976: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:05:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:24 smithi160 ceph-mon[36715]: pgmap v9977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:05:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:24 smithi183 ceph-mon[37273]: pgmap v9977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:05:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:24 smithi144 ceph-mon[32129]: pgmap v9977: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:05:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:26 smithi160 ceph-mon[36715]: pgmap v9978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:05:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17969.945465 secs 2024-07-26T16:05:27.070 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:26 smithi144 ceph-mon[32129]: pgmap v9978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:05:27.070 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17969.945465 secs 2024-07-26T16:05:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:26 smithi183 ceph-mon[37273]: pgmap v9978: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:05:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17969.945465 secs 2024-07-26T16:05:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:28.039+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:05:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:28.041+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:05:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:28.041+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:05:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:28.041+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:05:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:28.041+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:28 smithi160 ceph-mon[36715]: pgmap v9979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:28 smithi183 ceph-mon[37273]: pgmap v9979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:28 smithi144 ceph-mon[32129]: pgmap v9979: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:30 smithi160 ceph-mon[36715]: pgmap v9980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:30 smithi183 ceph-mon[37273]: pgmap v9980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:31.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:30 smithi144 ceph-mon[32129]: pgmap v9980: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17974.945592 secs 2024-07-26T16:05:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17974.945592 secs 2024-07-26T16:05:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17974.945592 secs 2024-07-26T16:05:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:32 smithi160 ceph-mon[36715]: pgmap v9981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:32 smithi183 ceph-mon[37273]: pgmap v9981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:32 smithi144 ceph-mon[32129]: pgmap v9981: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:34 smithi160 ceph-mon[36715]: pgmap v9982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:34 smithi183 ceph-mon[37273]: pgmap v9982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:34 smithi144 ceph-mon[32129]: pgmap v9982: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:36 smithi160 ceph-mon[36715]: pgmap v9983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17979.945710 secs 2024-07-26T16:05:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:36 smithi183 ceph-mon[37273]: pgmap v9983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17979.945710 secs 2024-07-26T16:05:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:36 smithi144 ceph-mon[32129]: pgmap v9983: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17979.945710 secs 2024-07-26T16:05:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:38 smithi160 ceph-mon[36715]: pgmap v9984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:38 smithi183 ceph-mon[37273]: pgmap v9984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:38 smithi144 ceph-mon[32129]: pgmap v9984: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:40 smithi160 ceph-mon[36715]: pgmap v9985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:40 smithi183 ceph-mon[37273]: pgmap v9985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:41.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:40 smithi144 ceph-mon[32129]: pgmap v9985: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17984.945850 secs 2024-07-26T16:05:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17984.945850 secs 2024-07-26T16:05:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17984.945850 secs 2024-07-26T16:05:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:43.057+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:05:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:43.058+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:05:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:43.058+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:05:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:43.058+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:05:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:43.058+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:42 smithi183 ceph-mon[37273]: pgmap v9986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:42 smithi144 ceph-mon[32129]: pgmap v9986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:42 smithi160 ceph-mon[36715]: pgmap v9986: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:45.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:44 smithi160 ceph-mon[36715]: pgmap v9987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:44 smithi183 ceph-mon[37273]: pgmap v9987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:44 smithi144 ceph-mon[32129]: pgmap v9987: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T16:05:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:05:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a803e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:05:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005c90 tx=0x7f74a803e040 comp rx=0 tx=0).stop 2024-07-26T16:05:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b55e0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T16:05:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 msgr2=0x7f74900b55e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:05:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:05:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.158+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:05:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.159+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:05:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:46.161+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5b20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f74a80319b0 tx=0x7f74a805a000 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:05:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:46 smithi183 ceph-mon[37273]: pgmap v9988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17989.945979 secs 2024-07-26T16:05:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:46 smithi144 ceph-mon[32129]: pgmap v9988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17989.945979 secs 2024-07-26T16:05:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:46 smithi160 ceph-mon[36715]: pgmap v9988: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17989.945979 secs 2024-07-26T16:05:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:48 smithi183 ceph-mon[37273]: pgmap v9989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:48 smithi144 ceph-mon[32129]: pgmap v9989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:49.275+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:05:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:49.275+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006540 con 0x563c6c4182f0 2024-07-26T16:05:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:49.275+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:05:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:49.275+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:05:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:48 smithi160 ceph-mon[36715]: pgmap v9989: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:50 smithi183 ceph-mon[37273]: pgmap v9990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:50 smithi144 ceph-mon[32129]: pgmap v9990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:50 smithi160 ceph-mon[36715]: pgmap v9990: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17994.946120 secs 2024-07-26T16:05:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17994.946120 secs 2024-07-26T16:05:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17994.946120 secs 2024-07-26T16:05:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:52 smithi183 ceph-mon[37273]: pgmap v9991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:52 smithi144 ceph-mon[32129]: pgmap v9991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:52 smithi160 ceph-mon[36715]: pgmap v9991: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:05:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:54 smithi183 ceph-mon[37273]: pgmap v9992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:54 smithi144 ceph-mon[32129]: pgmap v9992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:54 smithi160 ceph-mon[36715]: pgmap v9992: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:56 smithi183 ceph-mon[37273]: pgmap v9993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 17999.946243 secs 2024-07-26T16:05:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:56 smithi144 ceph-mon[32129]: pgmap v9993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 17999.946243 secs 2024-07-26T16:05:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:56 smithi160 ceph-mon[36715]: pgmap v9993: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:05:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 17999.946243 secs 2024-07-26T16:05:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:58.065+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:05:58.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:58.067+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:05:58.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:58.067+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:05:58.068 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:58.067+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:05:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:05:58.067+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:58 smithi183 ceph-mon[37273]: pgmap v9994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:58 smithi144 ceph-mon[32129]: pgmap v9994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:05:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:58 smithi160 ceph-mon[36715]: pgmap v9994: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:05:59 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:05:59 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:05:59 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: pgmap v9995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:00 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: pgmap v9995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:00 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: pgmap v9995: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:00 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:02.041 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:01 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:06:02.041 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:01 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:06:02.041 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18004.946340 secs 2024-07-26T16:06:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:01 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:06:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:01 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:06:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18004.946340 secs 2024-07-26T16:06:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:01 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:06:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:01 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:06:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18004.946340 secs 2024-07-26T16:06:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:02 smithi183 ceph-mon[37273]: pgmap v9996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:02 smithi144 ceph-mon[32129]: pgmap v9996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:02 smithi160 ceph-mon[36715]: pgmap v9996: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:06:04.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:06:04.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:04.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:04.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: pgmap v9997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:04.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: pgmap v9998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:04.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:06:04.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:04.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:04.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: pgmap v9997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:04.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: pgmap v9998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:04.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: pgmap v9997: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: pgmap v9998: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:04.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:06:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:06 smithi144 ceph-mon[32129]: pgmap v9999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18009.946441 secs 2024-07-26T16:06:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:06 smithi160 ceph-mon[36715]: pgmap v9999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18009.946441 secs 2024-07-26T16:06:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:06 smithi183 ceph-mon[37273]: pgmap v9999: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18009.946441 secs 2024-07-26T16:06:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:08 smithi144 ceph-mon[32129]: pgmap v10000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:08 smithi160 ceph-mon[36715]: pgmap v10000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:08 smithi183 ceph-mon[37273]: pgmap v10000: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:10.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:10 smithi144 ceph-mon[32129]: pgmap v10001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:10 smithi160 ceph-mon[36715]: pgmap v10001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:10 smithi183 ceph-mon[37273]: pgmap v10001: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18014.946560 secs 2024-07-26T16:06:12.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18014.946560 secs 2024-07-26T16:06:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18014.946560 secs 2024-07-26T16:06:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:12 smithi144 ceph-mon[32129]: pgmap v10002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:13.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:12 smithi183 ceph-mon[37273]: pgmap v10002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:12 smithi160 ceph-mon[36715]: pgmap v10002: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:06:13.079 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:13.077+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:06:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:13.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:06:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:13.078+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:06:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:13.078+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:06:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:13.079+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:14 smithi144 ceph-mon[32129]: pgmap v10003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:06:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:14 smithi160 ceph-mon[36715]: pgmap v10003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:06:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:14 smithi183 ceph-mon[37273]: pgmap v10003: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:06:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:16 smithi144 ceph-mon[32129]: pgmap v10004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18019.946627 secs 2024-07-26T16:06:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:16 smithi160 ceph-mon[36715]: pgmap v10004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18019.946627 secs 2024-07-26T16:06:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:16 smithi183 ceph-mon[37273]: pgmap v10004: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18019.946627 secs 2024-07-26T16:06:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:18 smithi144 ceph-mon[32129]: pgmap v10005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:18 smithi160 ceph-mon[36715]: pgmap v10005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:18 smithi183 ceph-mon[37273]: pgmap v10005: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:20.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:20 smithi144 ceph-mon[32129]: pgmap v10006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:20 smithi160 ceph-mon[36715]: pgmap v10006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:20 smithi183 ceph-mon[37273]: pgmap v10006: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18024.946725 secs 2024-07-26T16:06:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18024.946725 secs 2024-07-26T16:06:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18024.946725 secs 2024-07-26T16:06:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:22 smithi160 ceph-mon[36715]: pgmap v10007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:23.088 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:22 smithi183 ceph-mon[37273]: pgmap v10007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:22 smithi144 ceph-mon[32129]: pgmap v10007: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:24 smithi160 ceph-mon[36715]: pgmap v10008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:24 smithi183 ceph-mon[37273]: pgmap v10008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:25.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:24 smithi144 ceph-mon[32129]: pgmap v10008: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:26 smithi160 ceph-mon[36715]: pgmap v10009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18029.946853 secs 2024-07-26T16:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:26 smithi183 ceph-mon[37273]: pgmap v10009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18029.946853 secs 2024-07-26T16:06:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:26 smithi144 ceph-mon[32129]: pgmap v10009: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18029.946853 secs 2024-07-26T16:06:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:28.094+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:06:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:28.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:06:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:28.095+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:06:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:28.095+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:06:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:28.096+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:28 smithi160 ceph-mon[36715]: pgmap v10010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:28 smithi183 ceph-mon[37273]: pgmap v10010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:28 smithi144 ceph-mon[32129]: pgmap v10010: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:30 smithi160 ceph-mon[36715]: pgmap v10011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:30 smithi183 ceph-mon[37273]: pgmap v10011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:30 smithi144 ceph-mon[32129]: pgmap v10011: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18034.946996 secs 2024-07-26T16:06:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18034.946996 secs 2024-07-26T16:06:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18034.946996 secs 2024-07-26T16:06:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:32 smithi183 ceph-mon[37273]: pgmap v10012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:32 smithi144 ceph-mon[32129]: pgmap v10012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:32 smithi160 ceph-mon[36715]: pgmap v10012: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:34 smithi183 ceph-mon[37273]: pgmap v10013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:35.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:34 smithi144 ceph-mon[32129]: pgmap v10013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:34 smithi160 ceph-mon[36715]: pgmap v10013: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:36 smithi183 ceph-mon[37273]: pgmap v10014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18039.947145 secs 2024-07-26T16:06:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:36 smithi144 ceph-mon[32129]: pgmap v10014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18039.947145 secs 2024-07-26T16:06:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:36 smithi160 ceph-mon[36715]: pgmap v10014: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18039.947145 secs 2024-07-26T16:06:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:38 smithi183 ceph-mon[37273]: pgmap v10015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:38 smithi144 ceph-mon[32129]: pgmap v10015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:38 smithi160 ceph-mon[36715]: pgmap v10015: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:40 smithi183 ceph-mon[37273]: pgmap v10016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:41.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:40 smithi144 ceph-mon[32129]: pgmap v10016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:41.314 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:40 smithi160 ceph-mon[36715]: pgmap v10016: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18044.947269 secs 2024-07-26T16:06:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18044.947269 secs 2024-07-26T16:06:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18044.947269 secs 2024-07-26T16:06:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:43.111+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:06:43.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:43.113+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:06:43.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:43.113+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:06:43.114 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:43.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:06:43.115 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:43.113+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:42 smithi183 ceph-mon[37273]: pgmap v10017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:42 smithi144 ceph-mon[32129]: pgmap v10017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:42 smithi160 ceph-mon[36715]: pgmap v10017: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:44 smithi183 ceph-mon[37273]: pgmap v10018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:44 smithi144 ceph-mon[32129]: pgmap v10018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:44 smithi160 ceph-mon[36715]: pgmap v10018: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:46 smithi183 ceph-mon[37273]: pgmap v10019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18049.947391 secs 2024-07-26T16:06:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:46 smithi144 ceph-mon[32129]: pgmap v10019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18049.947391 secs 2024-07-26T16:06:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:46 smithi160 ceph-mon[36715]: pgmap v10019: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18049.947391 secs 2024-07-26T16:06:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:48 smithi183 ceph-mon[37273]: pgmap v10020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:49.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:48 smithi144 ceph-mon[32129]: pgmap v10020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:48 smithi160 ceph-mon[36715]: pgmap v10020: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:50 smithi183 ceph-mon[37273]: pgmap v10021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:51.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:50 smithi144 ceph-mon[32129]: pgmap v10021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:50 smithi160 ceph-mon[36715]: pgmap v10021: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18054.947473 secs 2024-07-26T16:06:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18054.947473 secs 2024-07-26T16:06:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18054.947473 secs 2024-07-26T16:06:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:52 smithi144 ceph-mon[32129]: pgmap v10022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:52 smithi160 ceph-mon[36715]: pgmap v10022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:52 smithi183 ceph-mon[37273]: pgmap v10022: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:06:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:54 smithi144 ceph-mon[32129]: pgmap v10023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:54 smithi160 ceph-mon[36715]: pgmap v10023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:54 smithi183 ceph-mon[37273]: pgmap v10023: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:56 smithi144 ceph-mon[32129]: pgmap v10024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18059.947555 secs 2024-07-26T16:06:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:56 smithi160 ceph-mon[36715]: pgmap v10024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18059.947555 secs 2024-07-26T16:06:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:56 smithi183 ceph-mon[37273]: pgmap v10024: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:06:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18059.947555 secs 2024-07-26T16:06:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:58.128+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:06:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:58.129+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:06:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:58.129+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:06:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:58.129+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:06:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:06:58.129+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:06:58 smithi144 ceph-mon[32129]: pgmap v10025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:06:58 smithi160 ceph-mon[36715]: pgmap v10025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:06:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:06:58 smithi183 ceph-mon[37273]: pgmap v10025: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:00 smithi144 ceph-mon[32129]: pgmap v10026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:00 smithi160 ceph-mon[36715]: pgmap v10026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:00 smithi183 ceph-mon[37273]: pgmap v10026: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:02.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18064.947618 secs 2024-07-26T16:07:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18064.947618 secs 2024-07-26T16:07:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18064.947618 secs 2024-07-26T16:07:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:02 smithi144 ceph-mon[32129]: pgmap v10027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:02 smithi160 ceph-mon[36715]: pgmap v10027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:02 smithi183 ceph-mon[37273]: pgmap v10027: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:03 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:03 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:03 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:04 smithi144 ceph-mon[32129]: pgmap v10028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:04 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:04 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:07:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:04 smithi160 ceph-mon[36715]: pgmap v10028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:04 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:04 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:07:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:04 smithi183 ceph-mon[37273]: pgmap v10028: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:04 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:04 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:07:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:05 smithi144 ceph-mon[32129]: pgmap v10029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:07:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:05 smithi144 ceph-mon[32129]: pgmap v10030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:07:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:05 smithi160 ceph-mon[36715]: pgmap v10029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:07:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:05 smithi160 ceph-mon[36715]: pgmap v10030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:07:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:05 smithi183 ceph-mon[37273]: pgmap v10029: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:07:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:05 smithi183 ceph-mon[37273]: pgmap v10030: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:07:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18069.947763 secs 2024-07-26T16:07:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18069.947763 secs 2024-07-26T16:07:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18069.947763 secs 2024-07-26T16:07:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:07 smithi144 ceph-mon[32129]: pgmap v10031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:07:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:07 smithi160 ceph-mon[36715]: pgmap v10031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:07:08.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:07 smithi183 ceph-mon[37273]: pgmap v10031: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:07:10.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:09 smithi144 ceph-mon[32129]: pgmap v10032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:07:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:09 smithi160 ceph-mon[36715]: pgmap v10032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:07:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:09 smithi183 ceph-mon[37273]: pgmap v10032: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:07:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:11 smithi160 ceph-mon[36715]: pgmap v10033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:07:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18074.947902 secs 2024-07-26T16:07:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:11 smithi183 ceph-mon[37273]: pgmap v10033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:07:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18074.947902 secs 2024-07-26T16:07:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:11 smithi144 ceph-mon[32129]: pgmap v10033: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:07:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18074.947902 secs 2024-07-26T16:07:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:13.145+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:07:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:13.146+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:07:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:13.146+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:07:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:13.146+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:07:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:13.146+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:13 smithi160 ceph-mon[36715]: pgmap v10034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:07:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:13 smithi183 ceph-mon[37273]: pgmap v10034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:07:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:13 smithi144 ceph-mon[32129]: pgmap v10034: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:07:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:15 smithi160 ceph-mon[36715]: pgmap v10035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:07:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:15 smithi183 ceph-mon[37273]: pgmap v10035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:07:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:15 smithi144 ceph-mon[32129]: pgmap v10035: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:07:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18079.948039 secs 2024-07-26T16:07:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18079.948039 secs 2024-07-26T16:07:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18079.948039 secs 2024-07-26T16:07:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:17 smithi160 ceph-mon[36715]: pgmap v10036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:17 smithi183 ceph-mon[37273]: pgmap v10036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:17 smithi144 ceph-mon[32129]: pgmap v10036: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:20 smithi160 ceph-mon[36715]: pgmap v10037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:20 smithi183 ceph-mon[37273]: pgmap v10037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:20 smithi144 ceph-mon[32129]: pgmap v10037: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:22.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:22 smithi144 ceph-mon[32129]: pgmap v10038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:22.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18084.948137 secs 2024-07-26T16:07:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:22 smithi160 ceph-mon[36715]: pgmap v10038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18084.948137 secs 2024-07-26T16:07:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:22 smithi183 ceph-mon[37273]: pgmap v10038: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18084.948137 secs 2024-07-26T16:07:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:24 smithi160 ceph-mon[36715]: pgmap v10039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:24 smithi183 ceph-mon[37273]: pgmap v10039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:24 smithi144 ceph-mon[32129]: pgmap v10039: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:26 smithi160 ceph-mon[36715]: pgmap v10040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:26 smithi183 ceph-mon[37273]: pgmap v10040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:26 smithi144 ceph-mon[32129]: pgmap v10040: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18089.948269 secs 2024-07-26T16:07:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18089.948269 secs 2024-07-26T16:07:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18089.948269 secs 2024-07-26T16:07:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:28.162+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:07:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:28.163+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:07:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:28.163+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:07:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:28.163+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:07:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:28.163+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:28 smithi160 ceph-mon[36715]: pgmap v10041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:28 smithi183 ceph-mon[37273]: pgmap v10041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:28 smithi144 ceph-mon[32129]: pgmap v10041: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:30 smithi160 ceph-mon[36715]: pgmap v10042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:30 smithi183 ceph-mon[37273]: pgmap v10042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:30 smithi144 ceph-mon[32129]: pgmap v10042: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:31 smithi160 ceph-mon[36715]: pgmap v10043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:31 smithi183 ceph-mon[37273]: pgmap v10043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:31 smithi144 ceph-mon[32129]: pgmap v10043: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:32.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18094.948386 secs 2024-07-26T16:07:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18094.948386 secs 2024-07-26T16:07:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18094.948386 secs 2024-07-26T16:07:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:33 smithi183 ceph-mon[37273]: pgmap v10044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:33 smithi144 ceph-mon[32129]: pgmap v10044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:33 smithi160 ceph-mon[36715]: pgmap v10044: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:35 smithi183 ceph-mon[37273]: pgmap v10045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:35 smithi144 ceph-mon[32129]: pgmap v10045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:35 smithi160 ceph-mon[36715]: pgmap v10045: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18099.948506 secs 2024-07-26T16:07:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18099.948506 secs 2024-07-26T16:07:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18099.948506 secs 2024-07-26T16:07:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:37 smithi144 ceph-mon[32129]: pgmap v10046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:37 smithi160 ceph-mon[36715]: pgmap v10046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:37 smithi183 ceph-mon[37273]: pgmap v10046: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:40.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:39 smithi144 ceph-mon[32129]: pgmap v10047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:39 smithi160 ceph-mon[36715]: pgmap v10047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:39 smithi183 ceph-mon[37273]: pgmap v10047: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:41 smithi183 ceph-mon[37273]: pgmap v10048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18104.948620 secs 2024-07-26T16:07:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:41 smithi144 ceph-mon[32129]: pgmap v10048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18104.948620 secs 2024-07-26T16:07:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:41 smithi160 ceph-mon[36715]: pgmap v10048: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18104.948620 secs 2024-07-26T16:07:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:43.179+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:07:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:43.180+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:07:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:43.180+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:07:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:43.180+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:07:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:43.180+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:43 smithi144 ceph-mon[32129]: pgmap v10049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:43 smithi160 ceph-mon[36715]: pgmap v10049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:43 smithi183 ceph-mon[37273]: pgmap v10049: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:45 smithi144 ceph-mon[32129]: pgmap v10050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:45 smithi160 ceph-mon[36715]: pgmap v10050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:45 smithi183 ceph-mon[37273]: pgmap v10050: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:47.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18109.948706 secs 2024-07-26T16:07:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18109.948706 secs 2024-07-26T16:07:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18109.948706 secs 2024-07-26T16:07:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:47 smithi144 ceph-mon[32129]: pgmap v10051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:47 smithi160 ceph-mon[36715]: pgmap v10051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:47 smithi183 ceph-mon[37273]: pgmap v10051: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:49 smithi144 ceph-mon[32129]: pgmap v10052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:49 smithi160 ceph-mon[36715]: pgmap v10052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:49 smithi183 ceph-mon[37273]: pgmap v10052: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:51 smithi144 ceph-mon[32129]: pgmap v10053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18114.948827 secs 2024-07-26T16:07:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:51 smithi160 ceph-mon[36715]: pgmap v10053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18114.948827 secs 2024-07-26T16:07:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:51 smithi183 ceph-mon[37273]: pgmap v10053: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18114.948827 secs 2024-07-26T16:07:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:53 smithi144 ceph-mon[32129]: pgmap v10054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:53 smithi160 ceph-mon[36715]: pgmap v10054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:53 smithi183 ceph-mon[37273]: pgmap v10054: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:07:56.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:55 smithi144 ceph-mon[32129]: pgmap v10055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:55 smithi160 ceph-mon[36715]: pgmap v10055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:55 smithi183 ceph-mon[37273]: pgmap v10055: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:07:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18119.948910 secs 2024-07-26T16:07:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18119.948910 secs 2024-07-26T16:07:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18119.948910 secs 2024-07-26T16:07:58.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:58.184+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:07:58.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:58.184+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:07:58.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:58.184+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:07:58.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:58.184+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:07:58.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:07:58.185+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:57 smithi160 ceph-mon[36715]: pgmap v10056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:57 smithi183 ceph-mon[37273]: pgmap v10056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:07:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:57 smithi144 ceph-mon[32129]: pgmap v10056: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:07:59 smithi160 ceph-mon[36715]: pgmap v10057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:00.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:07:59 smithi183 ceph-mon[37273]: pgmap v10057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:07:59 smithi144 ceph-mon[32129]: pgmap v10057: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:02.272 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:01 smithi144 ceph-mon[32129]: pgmap v10058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:02.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18124.949061 secs 2024-07-26T16:08:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:01 smithi160 ceph-mon[36715]: pgmap v10058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18124.949061 secs 2024-07-26T16:08:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:01 smithi183 ceph-mon[37273]: pgmap v10058: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18124.949061 secs 2024-07-26T16:08:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:03 smithi160 ceph-mon[36715]: pgmap v10059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:03 smithi183 ceph-mon[37273]: pgmap v10059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:03 smithi144 ceph-mon[32129]: pgmap v10059: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:05 smithi160 ceph-mon[36715]: pgmap v10060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:05 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:05 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:05 smithi183 ceph-mon[37273]: pgmap v10060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:05 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:05 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:06.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:05 smithi144 ceph-mon[32129]: pgmap v10060: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:05 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:05 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18129.949200 secs 2024-07-26T16:08:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:07 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18129.949200 secs 2024-07-26T16:08:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:07 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18129.949200 secs 2024-07-26T16:08:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:07 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:08:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:08 smithi160 ceph-mon[36715]: pgmap v10061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:08 smithi160 ceph-mon[36715]: pgmap v10062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:08 smithi183 ceph-mon[37273]: pgmap v10061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:08 smithi183 ceph-mon[37273]: pgmap v10062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:08 smithi144 ceph-mon[32129]: pgmap v10061: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:08 smithi144 ceph-mon[32129]: pgmap v10062: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:10 smithi160 ceph-mon[36715]: pgmap v10063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:10 smithi183 ceph-mon[37273]: pgmap v10063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:10.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:10 smithi144 ceph-mon[32129]: pgmap v10063: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:12 smithi160 ceph-mon[36715]: pgmap v10064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18134.949276 secs 2024-07-26T16:08:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:12 smithi183 ceph-mon[37273]: pgmap v10064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18134.949276 secs 2024-07-26T16:08:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:12 smithi144 ceph-mon[32129]: pgmap v10064: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18134.949276 secs 2024-07-26T16:08:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:13.200+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:08:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:13.201+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:08:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:13.201+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:08:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:13.201+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:08:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:13.201+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:14 smithi160 ceph-mon[36715]: pgmap v10065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:14 smithi183 ceph-mon[37273]: pgmap v10065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:14 smithi144 ceph-mon[32129]: pgmap v10065: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:16 smithi160 ceph-mon[36715]: pgmap v10066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:16 smithi183 ceph-mon[37273]: pgmap v10066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:16 smithi144 ceph-mon[32129]: pgmap v10066: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:08:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18139.949349 secs 2024-07-26T16:08:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18139.949349 secs 2024-07-26T16:08:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18139.949349 secs 2024-07-26T16:08:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:18 smithi160 ceph-mon[36715]: pgmap v10067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:08:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:18 smithi183 ceph-mon[37273]: pgmap v10067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:08:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:18 smithi144 ceph-mon[32129]: pgmap v10067: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:08:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:20 smithi160 ceph-mon[36715]: pgmap v10068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:20 smithi183 ceph-mon[37273]: pgmap v10068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:20 smithi144 ceph-mon[32129]: pgmap v10068: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:21 smithi183 ceph-mon[37273]: pgmap v10069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:21 smithi144 ceph-mon[32129]: pgmap v10069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:21 smithi160 ceph-mon[36715]: pgmap v10069: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:22.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18144.949466 secs 2024-07-26T16:08:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18144.949466 secs 2024-07-26T16:08:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18144.949466 secs 2024-07-26T16:08:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:23 smithi183 ceph-mon[37273]: pgmap v10070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:23 smithi144 ceph-mon[32129]: pgmap v10070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:23 smithi160 ceph-mon[36715]: pgmap v10070: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:25 smithi183 ceph-mon[37273]: pgmap v10071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:26.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:25 smithi144 ceph-mon[32129]: pgmap v10071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:25 smithi160 ceph-mon[36715]: pgmap v10071: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18149.949621 secs 2024-07-26T16:08:27.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18149.949621 secs 2024-07-26T16:08:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18149.949621 secs 2024-07-26T16:08:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:28.217+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:08:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:27 smithi144 ceph-mon[32129]: pgmap v10072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:28.218+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:08:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:28.218+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:08:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:28.218+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:08:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:28.219+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:27 smithi160 ceph-mon[36715]: pgmap v10072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:27 smithi183 ceph-mon[37273]: pgmap v10072: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:30.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:29 smithi144 ceph-mon[32129]: pgmap v10073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:29 smithi160 ceph-mon[36715]: pgmap v10073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:29 smithi183 ceph-mon[37273]: pgmap v10073: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:31 smithi144 ceph-mon[32129]: pgmap v10074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18154.949743 secs 2024-07-26T16:08:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:31 smithi160 ceph-mon[36715]: pgmap v10074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18154.949743 secs 2024-07-26T16:08:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:31 smithi183 ceph-mon[37273]: pgmap v10074: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18154.949743 secs 2024-07-26T16:08:34.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:33 smithi144 ceph-mon[32129]: pgmap v10075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:34.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:33 smithi160 ceph-mon[36715]: pgmap v10075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:33 smithi183 ceph-mon[37273]: pgmap v10075: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:35 smithi144 ceph-mon[32129]: pgmap v10076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:35 smithi160 ceph-mon[36715]: pgmap v10076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:35 smithi183 ceph-mon[37273]: pgmap v10076: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:37.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18159.949864 secs 2024-07-26T16:08:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18159.949864 secs 2024-07-26T16:08:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18159.949864 secs 2024-07-26T16:08:38.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:37 smithi144 ceph-mon[32129]: pgmap v10077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:37 smithi160 ceph-mon[36715]: pgmap v10077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:37 smithi183 ceph-mon[37273]: pgmap v10077: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:39 smithi144 ceph-mon[32129]: pgmap v10078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:39 smithi160 ceph-mon[36715]: pgmap v10078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:39 smithi183 ceph-mon[37273]: pgmap v10078: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:41 smithi183 ceph-mon[37273]: pgmap v10079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18164.950008 secs 2024-07-26T16:08:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:41 smithi144 ceph-mon[32129]: pgmap v10079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18164.950008 secs 2024-07-26T16:08:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:41 smithi160 ceph-mon[36715]: pgmap v10079: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18164.950008 secs 2024-07-26T16:08:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:43.230+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:08:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:43.231+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:08:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:43.231+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:08:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:43.231+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:08:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:43.231+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:43 smithi144 ceph-mon[32129]: pgmap v10080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:43 smithi160 ceph-mon[36715]: pgmap v10080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:43 smithi183 ceph-mon[37273]: pgmap v10080: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:46.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:45 smithi144 ceph-mon[32129]: pgmap v10081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:45 smithi160 ceph-mon[36715]: pgmap v10081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:45 smithi183 ceph-mon[37273]: pgmap v10081: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:47.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18169.950124 secs 2024-07-26T16:08:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18169.950124 secs 2024-07-26T16:08:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18169.950124 secs 2024-07-26T16:08:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:47 smithi144 ceph-mon[32129]: pgmap v10082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:47 smithi160 ceph-mon[36715]: pgmap v10082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:47 smithi183 ceph-mon[37273]: pgmap v10082: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:49.307+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:08:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:49.307+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006540 con 0x563c6c4182f0 2024-07-26T16:08:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:49.307+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d100 con 0x563c6c4182f0 2024-07-26T16:08:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:49.307+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:08:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:49 smithi160 ceph-mon[36715]: pgmap v10083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:49 smithi183 ceph-mon[37273]: pgmap v10083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:49 smithi144 ceph-mon[32129]: pgmap v10083: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:52.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:51 smithi144 ceph-mon[32129]: pgmap v10084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:52.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18174.950242 secs 2024-07-26T16:08:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:51 smithi160 ceph-mon[36715]: pgmap v10084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18174.950242 secs 2024-07-26T16:08:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:51 smithi183 ceph-mon[37273]: pgmap v10084: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18174.950242 secs 2024-07-26T16:08:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:53 smithi160 ceph-mon[36715]: pgmap v10085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:53 smithi183 ceph-mon[37273]: pgmap v10085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:54.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:53 smithi144 ceph-mon[32129]: pgmap v10085: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:08:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:55 smithi160 ceph-mon[36715]: pgmap v10086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:55 smithi183 ceph-mon[37273]: pgmap v10086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:56.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:55 smithi144 ceph-mon[32129]: pgmap v10086: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:08:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18179.950359 secs 2024-07-26T16:08:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18179.950359 secs 2024-07-26T16:08:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18179.950359 secs 2024-07-26T16:08:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:58.246+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:08:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:58.247+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:08:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:58.247+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:08:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:58.247+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:08:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:08:58.247+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:08:58 smithi160 ceph-mon[36715]: pgmap v10087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:08:58 smithi183 ceph-mon[37273]: pgmap v10087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:08:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:08:58 smithi144 ceph-mon[32129]: pgmap v10087: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:00 smithi160 ceph-mon[36715]: pgmap v10088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:00 smithi183 ceph-mon[37273]: pgmap v10088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:00 smithi144 ceph-mon[32129]: pgmap v10088: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:02 smithi160 ceph-mon[36715]: pgmap v10089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18184.950457 secs 2024-07-26T16:09:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:02 smithi183 ceph-mon[37273]: pgmap v10089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18184.950457 secs 2024-07-26T16:09:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:02 smithi144 ceph-mon[32129]: pgmap v10089: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18184.950457 secs 2024-07-26T16:09:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:04 smithi160 ceph-mon[36715]: pgmap v10090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:04 smithi183 ceph-mon[37273]: pgmap v10090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:04 smithi144 ceph-mon[32129]: pgmap v10090: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:06 smithi160 ceph-mon[36715]: pgmap v10091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:06 smithi183 ceph-mon[37273]: pgmap v10091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:06 smithi144 ceph-mon[32129]: pgmap v10091: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18189.950556 secs 2024-07-26T16:09:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:07 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18189.950556 secs 2024-07-26T16:09:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:07 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18189.950556 secs 2024-07-26T16:09:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:07 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:08.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:08 smithi183 ceph-mon[37273]: pgmap v10092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:08.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:08 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:08 smithi160 ceph-mon[36715]: pgmap v10092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:08 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.431 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:08 smithi144 ceph-mon[32129]: pgmap v10092: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:08.431 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.431 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.431 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:08.431 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:08 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:09 smithi144 ceph-mon[32129]: pgmap v10093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:09 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:09 smithi160 ceph-mon[36715]: pgmap v10093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:09 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:09 smithi183 ceph-mon[37273]: pgmap v10093: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:09 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:11 smithi160 ceph-mon[36715]: pgmap v10094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18194.950675 secs 2024-07-26T16:09:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:11 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:11 smithi183 ceph-mon[37273]: pgmap v10094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18194.950675 secs 2024-07-26T16:09:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:11 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:11 smithi144 ceph-mon[32129]: pgmap v10094: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18194.950675 secs 2024-07-26T16:09:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:11 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:13.264+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:09:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:13.265+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:09:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:13.265+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:09:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:13.265+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:09:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:13.265+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:12 smithi160 ceph-mon[36715]: pgmap v10095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:09:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:12 smithi160 ceph-mon[36715]: pgmap v10096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:12 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:12 smithi183 ceph-mon[37273]: pgmap v10095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:12 smithi183 ceph-mon[37273]: pgmap v10096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:12 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:12 smithi144 ceph-mon[32129]: pgmap v10095: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:09:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:12 smithi144 ceph-mon[32129]: pgmap v10096: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:12 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:09:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:14 smithi183 ceph-mon[37273]: pgmap v10097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:14 smithi144 ceph-mon[32129]: pgmap v10097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:14 smithi160 ceph-mon[36715]: pgmap v10097: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:16 smithi183 ceph-mon[37273]: pgmap v10098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:16 smithi144 ceph-mon[32129]: pgmap v10098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:16 smithi160 ceph-mon[36715]: pgmap v10098: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18199.950800 secs 2024-07-26T16:09:17.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18199.950800 secs 2024-07-26T16:09:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18199.950800 secs 2024-07-26T16:09:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:18 smithi183 ceph-mon[37273]: pgmap v10099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:18 smithi144 ceph-mon[32129]: pgmap v10099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:18 smithi160 ceph-mon[36715]: pgmap v10099: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:20 smithi183 ceph-mon[37273]: pgmap v10100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:09:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:20 smithi144 ceph-mon[32129]: pgmap v10100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:09:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:20 smithi160 ceph-mon[36715]: pgmap v10100: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:09:21.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18204.950946 secs 2024-07-26T16:09:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18204.950946 secs 2024-07-26T16:09:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18204.950946 secs 2024-07-26T16:09:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:22 smithi144 ceph-mon[32129]: pgmap v10101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:22 smithi160 ceph-mon[36715]: pgmap v10101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:22 smithi183 ceph-mon[37273]: pgmap v10101: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:24 smithi183 ceph-mon[37273]: pgmap v10102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:24 smithi144 ceph-mon[32129]: pgmap v10102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:24 smithi160 ceph-mon[36715]: pgmap v10102: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:26 smithi183 ceph-mon[37273]: pgmap v10103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:26 smithi144 ceph-mon[32129]: pgmap v10103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:26 smithi160 ceph-mon[36715]: pgmap v10103: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18209.951073 secs 2024-07-26T16:09:27.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18209.951073 secs 2024-07-26T16:09:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18209.951073 secs 2024-07-26T16:09:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:28.269+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:09:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:28.270+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:09:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:28.270+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:09:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:28.270+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:09:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:28.270+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:28 smithi144 ceph-mon[32129]: pgmap v10104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:28 smithi160 ceph-mon[36715]: pgmap v10104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:28 smithi183 ceph-mon[37273]: pgmap v10104: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:30 smithi183 ceph-mon[37273]: pgmap v10105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:09:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:30 smithi144 ceph-mon[32129]: pgmap v10105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:09:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:30 smithi160 ceph-mon[36715]: pgmap v10105: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:09:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18214.951195 secs 2024-07-26T16:09:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18214.951195 secs 2024-07-26T16:09:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18214.951195 secs 2024-07-26T16:09:32.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:32 smithi144 ceph-mon[32129]: pgmap v10106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:32 smithi160 ceph-mon[36715]: pgmap v10106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:32 smithi183 ceph-mon[37273]: pgmap v10106: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:34 smithi183 ceph-mon[37273]: pgmap v10107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:34 smithi144 ceph-mon[32129]: pgmap v10107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:34 smithi160 ceph-mon[36715]: pgmap v10107: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:09:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:36 smithi144 ceph-mon[32129]: pgmap v10108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:36 smithi183 ceph-mon[37273]: pgmap v10108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:36 smithi160 ceph-mon[36715]: pgmap v10108: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18219.951259 secs 2024-07-26T16:09:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18219.951259 secs 2024-07-26T16:09:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18219.951259 secs 2024-07-26T16:09:38.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:38 smithi183 ceph-mon[37273]: pgmap v10109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:38 smithi144 ceph-mon[32129]: pgmap v10109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:38 smithi160 ceph-mon[36715]: pgmap v10109: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:40 smithi183 ceph-mon[37273]: pgmap v10110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:40 smithi144 ceph-mon[32129]: pgmap v10110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:40 smithi160 ceph-mon[36715]: pgmap v10110: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:41.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18224.951383 secs 2024-07-26T16:09:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18224.951383 secs 2024-07-26T16:09:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18224.951383 secs 2024-07-26T16:09:43.288 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:43.286+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:09:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:43.287+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:09:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:43.287+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:09:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:43.287+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:09:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:43.287+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:43.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:43 smithi183 ceph-mon[37273]: pgmap v10111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:43 smithi144 ceph-mon[32129]: pgmap v10111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:43.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:43 smithi160 ceph-mon[36715]: pgmap v10111: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:45 smithi144 ceph-mon[32129]: pgmap v10112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:45 smithi160 ceph-mon[36715]: pgmap v10112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:45 smithi183 ceph-mon[37273]: pgmap v10112: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18229.951478 secs 2024-07-26T16:09:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18229.951478 secs 2024-07-26T16:09:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18229.951478 secs 2024-07-26T16:09:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:47 smithi144 ceph-mon[32129]: pgmap v10113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:47 smithi160 ceph-mon[36715]: pgmap v10113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:47 smithi183 ceph-mon[37273]: pgmap v10113: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:49 smithi144 ceph-mon[32129]: pgmap v10114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:49 smithi160 ceph-mon[36715]: pgmap v10114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:49 smithi183 ceph-mon[37273]: pgmap v10114: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:51 smithi144 ceph-mon[32129]: pgmap v10115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18234.951606 secs 2024-07-26T16:09:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:51 smithi160 ceph-mon[36715]: pgmap v10115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18234.951606 secs 2024-07-26T16:09:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:51 smithi183 ceph-mon[37273]: pgmap v10115: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:09:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18234.951606 secs 2024-07-26T16:09:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:53 smithi160 ceph-mon[36715]: pgmap v10116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:53 smithi183 ceph-mon[37273]: pgmap v10116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:53.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:53 smithi144 ceph-mon[32129]: pgmap v10116: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:09:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:55 smithi160 ceph-mon[36715]: pgmap v10117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:55 smithi183 ceph-mon[37273]: pgmap v10117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:55 smithi144 ceph-mon[32129]: pgmap v10117: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18239.951671 secs 2024-07-26T16:09:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18239.951671 secs 2024-07-26T16:09:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18239.951671 secs 2024-07-26T16:09:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:57 smithi160 ceph-mon[36715]: pgmap v10118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:57 smithi183 ceph-mon[37273]: pgmap v10118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:57.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:57 smithi144 ceph-mon[32129]: pgmap v10118: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:58.302+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:09:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:58.303+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:09:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:58.303+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:09:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:58.303+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:09:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:09:58.303+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:09:59 smithi160 ceph-mon[36715]: pgmap v10119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:59.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:09:59 smithi183 ceph-mon[37273]: pgmap v10119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:09:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:09:59 smithi144 ceph-mon[32129]: pgmap v10119: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:10:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:10:00.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:10:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:01 smithi160 ceph-mon[36715]: pgmap v10120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18244.951801 secs 2024-07-26T16:10:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:01 smithi183 ceph-mon[37273]: pgmap v10120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18244.951801 secs 2024-07-26T16:10:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:01 smithi144 ceph-mon[32129]: pgmap v10120: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18244.951801 secs 2024-07-26T16:10:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:03 smithi160 ceph-mon[36715]: pgmap v10121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:03.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:03 smithi183 ceph-mon[37273]: pgmap v10121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:03 smithi144 ceph-mon[32129]: pgmap v10121: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:05 smithi160 ceph-mon[36715]: pgmap v10122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:05.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:05 smithi183 ceph-mon[37273]: pgmap v10122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:05.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:05 smithi144 ceph-mon[32129]: pgmap v10122: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18249.951926 secs 2024-07-26T16:10:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18249.951926 secs 2024-07-26T16:10:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18249.951926 secs 2024-07-26T16:10:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:07 smithi160 ceph-mon[36715]: pgmap v10123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:07 smithi183 ceph-mon[37273]: pgmap v10123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:07 smithi144 ceph-mon[32129]: pgmap v10123: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:09 smithi160 ceph-mon[36715]: pgmap v10124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:09.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:09 smithi183 ceph-mon[37273]: pgmap v10124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:09.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:09 smithi144 ceph-mon[32129]: pgmap v10124: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:11 smithi160 ceph-mon[36715]: pgmap v10125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:11.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18254.952072 secs 2024-07-26T16:10:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:11 smithi183 ceph-mon[37273]: pgmap v10125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18254.952072 secs 2024-07-26T16:10:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:11 smithi144 ceph-mon[32129]: pgmap v10125: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18254.952072 secs 2024-07-26T16:10:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:12 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:12 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:12 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:13.319+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:10:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:13.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:10:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:13.320+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:10:13.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:13.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:10:13.322 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:13.320+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:13 smithi183 ceph-mon[37273]: pgmap v10126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:13.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:13 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:10:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:13 smithi144 ceph-mon[32129]: pgmap v10126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:13 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:10:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:13 smithi160 ceph-mon[36715]: pgmap v10126: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:13 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:10:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:14 smithi183 ceph-mon[37273]: pgmap v10127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:14 smithi183 ceph-mon[37273]: pgmap v10128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:14 smithi144 ceph-mon[32129]: pgmap v10127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:14.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:14 smithi144 ceph-mon[32129]: pgmap v10128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:14 smithi160 ceph-mon[36715]: pgmap v10127: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:14 smithi160 ceph-mon[36715]: pgmap v10128: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:16 smithi183 ceph-mon[37273]: pgmap v10129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:10:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18259.952202 secs 2024-07-26T16:10:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:16 smithi144 ceph-mon[32129]: pgmap v10129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:10:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18259.952202 secs 2024-07-26T16:10:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:16 smithi160 ceph-mon[36715]: pgmap v10129: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:10:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18259.952202 secs 2024-07-26T16:10:18.861 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:18 smithi160 ceph-mon[36715]: pgmap v10130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:10:18.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:18 smithi183 ceph-mon[37273]: pgmap v10130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:10:18.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:18 smithi144 ceph-mon[32129]: pgmap v10130: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:10:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:20 smithi183 ceph-mon[37273]: pgmap v10131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:20 smithi144 ceph-mon[32129]: pgmap v10131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:20 smithi160 ceph-mon[36715]: pgmap v10131: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18264.952351 secs 2024-07-26T16:10:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18264.952351 secs 2024-07-26T16:10:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18264.952351 secs 2024-07-26T16:10:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:22 smithi183 ceph-mon[37273]: pgmap v10132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:22 smithi144 ceph-mon[32129]: pgmap v10132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:22 smithi160 ceph-mon[36715]: pgmap v10132: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:10:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:24 smithi183 ceph-mon[37273]: pgmap v10133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:10:24.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:24 smithi144 ceph-mon[32129]: pgmap v10133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:10:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:24 smithi160 ceph-mon[36715]: pgmap v10133: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:10:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:26 smithi183 ceph-mon[37273]: pgmap v10134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18269.952476 secs 2024-07-26T16:10:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:26 smithi144 ceph-mon[32129]: pgmap v10134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18269.952476 secs 2024-07-26T16:10:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:26 smithi160 ceph-mon[36715]: pgmap v10134: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18269.952476 secs 2024-07-26T16:10:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:28.325+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:10:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:28.326+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:10:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:28.326+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:10:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:28.326+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:10:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:28.326+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:28 smithi144 ceph-mon[32129]: pgmap v10135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:28 smithi160 ceph-mon[36715]: pgmap v10135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:28 smithi183 ceph-mon[37273]: pgmap v10135: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:30 smithi144 ceph-mon[32129]: pgmap v10136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:30 smithi160 ceph-mon[36715]: pgmap v10136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:30 smithi183 ceph-mon[37273]: pgmap v10136: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18274.952610 secs 2024-07-26T16:10:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18274.952610 secs 2024-07-26T16:10:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18274.952610 secs 2024-07-26T16:10:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:32 smithi144 ceph-mon[32129]: pgmap v10137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:32 smithi160 ceph-mon[36715]: pgmap v10137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:32 smithi183 ceph-mon[37273]: pgmap v10137: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:34 smithi144 ceph-mon[32129]: pgmap v10138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:34 smithi160 ceph-mon[36715]: pgmap v10138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:34 smithi183 ceph-mon[37273]: pgmap v10138: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:36 smithi144 ceph-mon[32129]: pgmap v10139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18279.952743 secs 2024-07-26T16:10:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:36 smithi160 ceph-mon[36715]: pgmap v10139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18279.952743 secs 2024-07-26T16:10:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:36 smithi183 ceph-mon[37273]: pgmap v10139: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18279.952743 secs 2024-07-26T16:10:38.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:38 smithi144 ceph-mon[32129]: pgmap v10140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:38 smithi160 ceph-mon[36715]: pgmap v10140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:38 smithi183 ceph-mon[37273]: pgmap v10140: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:40.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:40 smithi144 ceph-mon[32129]: pgmap v10141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:40 smithi160 ceph-mon[36715]: pgmap v10141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:41.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:40 smithi183 ceph-mon[37273]: pgmap v10141: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:41.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18284.952872 secs 2024-07-26T16:10:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18284.952872 secs 2024-07-26T16:10:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18284.952872 secs 2024-07-26T16:10:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:42 smithi144 ceph-mon[32129]: pgmap v10142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:43.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:42 smithi183 ceph-mon[37273]: pgmap v10142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:42 smithi160 ceph-mon[36715]: pgmap v10142: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:43.342+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:10:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:43.343+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:10:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:43.343+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:10:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:43.343+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:10:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:43.343+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:44 smithi144 ceph-mon[32129]: pgmap v10143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:44 smithi160 ceph-mon[36715]: pgmap v10143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:44 smithi183 ceph-mon[37273]: pgmap v10143: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:46 smithi144 ceph-mon[32129]: pgmap v10144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18289.952959 secs 2024-07-26T16:10:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:46 smithi160 ceph-mon[36715]: pgmap v10144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18289.952959 secs 2024-07-26T16:10:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:46 smithi183 ceph-mon[37273]: pgmap v10144: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18289.952959 secs 2024-07-26T16:10:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:48 smithi144 ceph-mon[32129]: pgmap v10145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:48 smithi160 ceph-mon[36715]: pgmap v10145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:48 smithi183 ceph-mon[37273]: pgmap v10145: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:50 smithi144 ceph-mon[32129]: pgmap v10146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:51.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:50 smithi160 ceph-mon[36715]: pgmap v10146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:50 smithi183 ceph-mon[37273]: pgmap v10146: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18294.953101 secs 2024-07-26T16:10:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18294.953101 secs 2024-07-26T16:10:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18294.953101 secs 2024-07-26T16:10:53.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:52 smithi160 ceph-mon[36715]: pgmap v10147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:53.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:52 smithi183 ceph-mon[37273]: pgmap v10147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:53.214 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:52 smithi144 ceph-mon[32129]: pgmap v10147: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:10:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:54 smithi160 ceph-mon[36715]: pgmap v10148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:55.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:54 smithi183 ceph-mon[37273]: pgmap v10148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:54 smithi144 ceph-mon[32129]: pgmap v10148: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:56 smithi160 ceph-mon[36715]: pgmap v10149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18299.953236 secs 2024-07-26T16:10:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:56 smithi183 ceph-mon[37273]: pgmap v10149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18299.953236 secs 2024-07-26T16:10:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:56 smithi144 ceph-mon[32129]: pgmap v10149: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:10:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18299.953236 secs 2024-07-26T16:10:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:58.358+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:10:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:58.359+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:10:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:58.359+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:10:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:58.359+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:10:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:10:58.359+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:59.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:10:58 smithi160 ceph-mon[36715]: pgmap v10150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:59.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:10:58 smithi183 ceph-mon[37273]: pgmap v10150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:10:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:10:58 smithi144 ceph-mon[32129]: pgmap v10150: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:01.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:00 smithi160 ceph-mon[36715]: pgmap v10151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:01.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:00 smithi183 ceph-mon[37273]: pgmap v10151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:00 smithi144 ceph-mon[32129]: pgmap v10151: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18304.953368 secs 2024-07-26T16:11:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18304.953368 secs 2024-07-26T16:11:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18304.953368 secs 2024-07-26T16:11:03.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:02 smithi160 ceph-mon[36715]: pgmap v10152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:03.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:02 smithi183 ceph-mon[37273]: pgmap v10152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:03.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:02 smithi144 ceph-mon[32129]: pgmap v10152: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:04.330+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006540 con 0x563c6c4182f0 2024-07-26T16:11:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:04.330+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:11:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:04.331+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006540 con 0x563c6c4182f0 2024-07-26T16:11:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:04.331+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T16:11:05.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:04 smithi160 ceph-mon[36715]: pgmap v10153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:05.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:04 smithi183 ceph-mon[37273]: pgmap v10153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:05.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:04 smithi144 ceph-mon[32129]: pgmap v10153: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:06 smithi160 ceph-mon[36715]: pgmap v10154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18309.953455 secs 2024-07-26T16:11:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:06 smithi183 ceph-mon[37273]: pgmap v10154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18309.953455 secs 2024-07-26T16:11:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:06 smithi144 ceph-mon[32129]: pgmap v10154: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18309.953455 secs 2024-07-26T16:11:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:08 smithi160 ceph-mon[36715]: pgmap v10155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:09.086 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:08 smithi144 ceph-mon[32129]: pgmap v10155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:09.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:08 smithi183 ceph-mon[37273]: pgmap v10155: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:10 smithi160 ceph-mon[36715]: pgmap v10156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:11.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:10 smithi183 ceph-mon[37273]: pgmap v10156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:10 smithi144 ceph-mon[32129]: pgmap v10156: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18314.953535 secs 2024-07-26T16:11:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18314.953535 secs 2024-07-26T16:11:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18314.953535 secs 2024-07-26T16:11:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:12 smithi160 ceph-mon[36715]: pgmap v10157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:13.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:12 smithi183 ceph-mon[37273]: pgmap v10157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:13.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:12 smithi144 ceph-mon[32129]: pgmap v10157: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:13.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:13.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:11:13.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:13.377+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:11:13.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:13.377+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:11:13.379 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:13.377+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:11:13.379 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:13.377+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:13 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:13 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:13 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:14 smithi183 ceph-mon[37273]: pgmap v10158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:14 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:14 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:11:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:14 smithi144 ceph-mon[32129]: pgmap v10158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:14 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:15.221 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:14 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:11:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:14 smithi160 ceph-mon[36715]: pgmap v10158: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:14 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:14 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:11:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:15 smithi183 ceph-mon[37273]: pgmap v10159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:11:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:15 smithi183 ceph-mon[37273]: pgmap v10160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:11:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:15 smithi144 ceph-mon[32129]: pgmap v10159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:11:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:15 smithi144 ceph-mon[32129]: pgmap v10160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:11:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:15 smithi160 ceph-mon[36715]: pgmap v10159: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:11:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:15 smithi160 ceph-mon[36715]: pgmap v10160: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:11:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18319.953619 secs 2024-07-26T16:11:17.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18319.953619 secs 2024-07-26T16:11:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18319.953619 secs 2024-07-26T16:11:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:17 smithi183 ceph-mon[37273]: pgmap v10161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:11:18.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:17 smithi160 ceph-mon[36715]: pgmap v10161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:11:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:17 smithi144 ceph-mon[32129]: pgmap v10161: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:11:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:19 smithi183 ceph-mon[37273]: pgmap v10162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:11:20.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:19 smithi144 ceph-mon[32129]: pgmap v10162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:11:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:19 smithi160 ceph-mon[36715]: pgmap v10162: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:11:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:21 smithi183 ceph-mon[37273]: pgmap v10163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:11:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18324.953745 secs 2024-07-26T16:11:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:21 smithi144 ceph-mon[32129]: pgmap v10163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:11:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18324.953745 secs 2024-07-26T16:11:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:21 smithi160 ceph-mon[36715]: pgmap v10163: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:11:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18324.953745 secs 2024-07-26T16:11:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:23 smithi183 ceph-mon[37273]: pgmap v10164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:11:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:23 smithi144 ceph-mon[32129]: pgmap v10164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:11:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:23 smithi160 ceph-mon[36715]: pgmap v10164: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:11:26.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:25 smithi144 ceph-mon[32129]: pgmap v10165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:11:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:25 smithi183 ceph-mon[37273]: pgmap v10165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:11:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:25 smithi160 ceph-mon[36715]: pgmap v10165: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:11:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18329.953891 secs 2024-07-26T16:11:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18329.953891 secs 2024-07-26T16:11:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18329.953891 secs 2024-07-26T16:11:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:27 smithi183 ceph-mon[37273]: pgmap v10166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:28.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:27 smithi144 ceph-mon[32129]: pgmap v10166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:27 smithi160 ceph-mon[36715]: pgmap v10166: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:28.386+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:11:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:28.387+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:11:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:28.387+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:11:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:28.387+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:11:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:28.387+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:29 smithi144 ceph-mon[32129]: pgmap v10167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:29 smithi160 ceph-mon[36715]: pgmap v10167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:29 smithi183 ceph-mon[37273]: pgmap v10167: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:31 smithi144 ceph-mon[32129]: pgmap v10168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18334.954008 secs 2024-07-26T16:11:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:31 smithi160 ceph-mon[36715]: pgmap v10168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18334.954008 secs 2024-07-26T16:11:32.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:31 smithi183 ceph-mon[37273]: pgmap v10168: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:32.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18334.954008 secs 2024-07-26T16:11:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:33 smithi144 ceph-mon[32129]: pgmap v10169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:33 smithi160 ceph-mon[36715]: pgmap v10169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:33 smithi183 ceph-mon[37273]: pgmap v10169: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:36.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:35 smithi144 ceph-mon[32129]: pgmap v10170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:35 smithi160 ceph-mon[36715]: pgmap v10170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:35 smithi183 ceph-mon[37273]: pgmap v10170: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18339.954075 secs 2024-07-26T16:11:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18339.954075 secs 2024-07-26T16:11:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18339.954075 secs 2024-07-26T16:11:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:37 smithi160 ceph-mon[36715]: pgmap v10171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:37 smithi183 ceph-mon[37273]: pgmap v10171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:37 smithi144 ceph-mon[32129]: pgmap v10171: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:39 smithi160 ceph-mon[36715]: pgmap v10172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:40.318 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:39 smithi144 ceph-mon[32129]: pgmap v10172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:39 smithi183 ceph-mon[37273]: pgmap v10172: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:41 smithi160 ceph-mon[36715]: pgmap v10173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18344.954191 secs 2024-07-26T16:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:41 smithi183 ceph-mon[37273]: pgmap v10173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18344.954191 secs 2024-07-26T16:11:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:41 smithi144 ceph-mon[32129]: pgmap v10173: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18344.954191 secs 2024-07-26T16:11:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:43.391+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:11:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:43.392+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:11:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:43.392+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:11:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:43.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:11:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:43.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:43 smithi160 ceph-mon[36715]: pgmap v10174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:43 smithi183 ceph-mon[37273]: pgmap v10174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:43 smithi144 ceph-mon[32129]: pgmap v10174: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:45 smithi160 ceph-mon[36715]: pgmap v10175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:45 smithi183 ceph-mon[37273]: pgmap v10175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:46.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:45 smithi144 ceph-mon[32129]: pgmap v10175: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18349.954320 secs 2024-07-26T16:11:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18349.954320 secs 2024-07-26T16:11:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18349.954320 secs 2024-07-26T16:11:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:48 smithi160 ceph-mon[36715]: pgmap v10176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:48 smithi183 ceph-mon[37273]: pgmap v10176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:48 smithi144 ceph-mon[32129]: pgmap v10176: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:49.338+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T16:11:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:49.338+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:11:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:49.339+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480004740 con 0x7f74b0109990 2024-07-26T16:11:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:49.339+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:11:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:50 smithi160 ceph-mon[36715]: pgmap v10177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:50 smithi183 ceph-mon[37273]: pgmap v10177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:50 smithi144 ceph-mon[32129]: pgmap v10177: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:52 smithi160 ceph-mon[36715]: pgmap v10178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18354.954351 secs 2024-07-26T16:11:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:52 smithi183 ceph-mon[37273]: pgmap v10178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18354.954351 secs 2024-07-26T16:11:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:52 smithi144 ceph-mon[32129]: pgmap v10178: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18354.954351 secs 2024-07-26T16:11:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:54 smithi160 ceph-mon[36715]: pgmap v10179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:54 smithi183 ceph-mon[37273]: pgmap v10179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:54 smithi144 ceph-mon[32129]: pgmap v10179: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:11:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:56 smithi160 ceph-mon[36715]: pgmap v10180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:56 smithi183 ceph-mon[37273]: pgmap v10180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:56 smithi144 ceph-mon[32129]: pgmap v10180: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:11:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18359.954430 secs 2024-07-26T16:11:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18359.954430 secs 2024-07-26T16:11:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18359.954430 secs 2024-07-26T16:11:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:11:58 smithi160 ceph-mon[36715]: pgmap v10181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:58.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:11:58 smithi183 ceph-mon[37273]: pgmap v10181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:11:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:58.395+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:11:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:58.396+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:11:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:58.396+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:11:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:58.396+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:11:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:11:58.396+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:11:58 smithi144 ceph-mon[32129]: pgmap v10181: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:00 smithi160 ceph-mon[36715]: pgmap v10182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:00 smithi183 ceph-mon[37273]: pgmap v10182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:00 smithi144 ceph-mon[32129]: pgmap v10182: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:02 smithi183 ceph-mon[37273]: pgmap v10183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18364.954560 secs 2024-07-26T16:12:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:02 smithi144 ceph-mon[32129]: pgmap v10183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18364.954560 secs 2024-07-26T16:12:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:02 smithi160 ceph-mon[36715]: pgmap v10183: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18364.954560 secs 2024-07-26T16:12:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:04 smithi183 ceph-mon[37273]: pgmap v10184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:04 smithi144 ceph-mon[32129]: pgmap v10184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:04 smithi160 ceph-mon[36715]: pgmap v10184: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:06 smithi183 ceph-mon[37273]: pgmap v10185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:06 smithi144 ceph-mon[32129]: pgmap v10185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:06 smithi160 ceph-mon[36715]: pgmap v10185: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18369.954644 secs 2024-07-26T16:12:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18369.954644 secs 2024-07-26T16:12:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18369.954644 secs 2024-07-26T16:12:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:08 smithi183 ceph-mon[37273]: pgmap v10186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:08 smithi144 ceph-mon[32129]: pgmap v10186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:08 smithi160 ceph-mon[36715]: pgmap v10186: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:10 smithi183 ceph-mon[37273]: pgmap v10187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:10 smithi144 ceph-mon[32129]: pgmap v10187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:10 smithi160 ceph-mon[36715]: pgmap v10187: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:12 smithi183 ceph-mon[37273]: pgmap v10188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18374.954774 secs 2024-07-26T16:12:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:12 smithi144 ceph-mon[32129]: pgmap v10188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18374.954774 secs 2024-07-26T16:12:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:12 smithi160 ceph-mon[36715]: pgmap v10188: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18374.954774 secs 2024-07-26T16:12:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:13.410+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:12:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:13.411+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:12:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:13.411+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:12:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:13.411+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:12:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:13.411+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:13 smithi144 ceph-mon[32129]: pgmap v10189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:13 smithi160 ceph-mon[36715]: pgmap v10189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:13 smithi183 ceph-mon[37273]: pgmap v10189: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:15 smithi183 ceph-mon[37273]: pgmap v10190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:15 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:15 smithi144 ceph-mon[32129]: pgmap v10190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:15 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:15 smithi160 ceph-mon[36715]: pgmap v10190: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:15 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:16 smithi183 ceph-mon[37273]: pgmap v10191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:16 smithi183 ceph-mon[37273]: pgmap v10192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:16 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:12:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18379.954836 secs 2024-07-26T16:12:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:16 smithi144 ceph-mon[32129]: pgmap v10191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:16 smithi144 ceph-mon[32129]: pgmap v10192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:16 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:12:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18379.954836 secs 2024-07-26T16:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:16 smithi160 ceph-mon[36715]: pgmap v10191: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:16 smithi160 ceph-mon[36715]: pgmap v10192: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:16 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:12:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18379.954836 secs 2024-07-26T16:12:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:18 smithi160 ceph-mon[36715]: pgmap v10193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:12:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:18 smithi183 ceph-mon[37273]: pgmap v10193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:12:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:18 smithi144 ceph-mon[32129]: pgmap v10193: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:12:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:20 smithi183 ceph-mon[37273]: pgmap v10194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:12:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:20 smithi144 ceph-mon[32129]: pgmap v10194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:12:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:20 smithi160 ceph-mon[36715]: pgmap v10194: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:12:22.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18384.954971 secs 2024-07-26T16:12:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18384.954971 secs 2024-07-26T16:12:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18384.954971 secs 2024-07-26T16:12:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:22 smithi183 ceph-mon[37273]: pgmap v10195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:23.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:22 smithi144 ceph-mon[32129]: pgmap v10195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:22 smithi160 ceph-mon[36715]: pgmap v10195: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:12:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:24 smithi183 ceph-mon[37273]: pgmap v10196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:12:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:24 smithi144 ceph-mon[32129]: pgmap v10196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:12:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:24 smithi160 ceph-mon[36715]: pgmap v10196: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:12:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:26 smithi183 ceph-mon[37273]: pgmap v10197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:12:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18389.955054 secs 2024-07-26T16:12:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:26 smithi144 ceph-mon[32129]: pgmap v10197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:12:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18389.955054 secs 2024-07-26T16:12:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:26 smithi160 ceph-mon[36715]: pgmap v10197: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:12:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18389.955054 secs 2024-07-26T16:12:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:28.427+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:12:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:28.427+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:12:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:28.427+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:12:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:28.427+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:12:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:28.428+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:28 smithi183 ceph-mon[37273]: pgmap v10198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:29.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:28 smithi144 ceph-mon[32129]: pgmap v10198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:28 smithi160 ceph-mon[36715]: pgmap v10198: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:31.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:30 smithi183 ceph-mon[37273]: pgmap v10199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:31.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:30 smithi144 ceph-mon[32129]: pgmap v10199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:30 smithi160 ceph-mon[36715]: pgmap v10199: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18394.955169 secs 2024-07-26T16:12:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18394.955169 secs 2024-07-26T16:12:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18394.955169 secs 2024-07-26T16:12:33.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:32 smithi183 ceph-mon[37273]: pgmap v10200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:33.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:32 smithi144 ceph-mon[32129]: pgmap v10200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:32 smithi160 ceph-mon[36715]: pgmap v10200: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:34 smithi183 ceph-mon[37273]: pgmap v10201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:35.223 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:34 smithi144 ceph-mon[32129]: pgmap v10201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:34 smithi160 ceph-mon[36715]: pgmap v10201: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:36 smithi183 ceph-mon[37273]: pgmap v10202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18399.955260 secs 2024-07-26T16:12:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:36 smithi144 ceph-mon[32129]: pgmap v10202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18399.955260 secs 2024-07-26T16:12:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:36 smithi160 ceph-mon[36715]: pgmap v10202: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18399.955260 secs 2024-07-26T16:12:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:38 smithi183 ceph-mon[37273]: pgmap v10203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:38 smithi144 ceph-mon[32129]: pgmap v10203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:38 smithi160 ceph-mon[36715]: pgmap v10203: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:40 smithi144 ceph-mon[32129]: pgmap v10204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:40 smithi160 ceph-mon[36715]: pgmap v10204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:40 smithi183 ceph-mon[37273]: pgmap v10204: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:42.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18404.955386 secs 2024-07-26T16:12:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18404.955386 secs 2024-07-26T16:12:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18404.955386 secs 2024-07-26T16:12:43.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:42 smithi144 ceph-mon[32129]: pgmap v10205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:42 smithi160 ceph-mon[36715]: pgmap v10205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:42 smithi183 ceph-mon[37273]: pgmap v10205: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:43.443+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:12:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:43.444+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:12:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:43.444+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:12:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:43.444+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:12:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:43.444+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:45.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:44 smithi183 ceph-mon[37273]: pgmap v10206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:45.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:44 smithi144 ceph-mon[32129]: pgmap v10206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:44 smithi160 ceph-mon[36715]: pgmap v10206: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:46 smithi144 ceph-mon[32129]: pgmap v10207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18409.955448 secs 2024-07-26T16:12:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:46 smithi160 ceph-mon[36715]: pgmap v10207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18409.955448 secs 2024-07-26T16:12:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:46 smithi183 ceph-mon[37273]: pgmap v10207: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18409.955448 secs 2024-07-26T16:12:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:48 smithi144 ceph-mon[32129]: pgmap v10208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:48 smithi160 ceph-mon[36715]: pgmap v10208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:48 smithi183 ceph-mon[37273]: pgmap v10208: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:50 smithi144 ceph-mon[32129]: pgmap v10209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:50 smithi160 ceph-mon[36715]: pgmap v10209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:50 smithi183 ceph-mon[37273]: pgmap v10209: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18414.955536 secs 2024-07-26T16:12:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18414.955536 secs 2024-07-26T16:12:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18414.955536 secs 2024-07-26T16:12:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:52 smithi144 ceph-mon[32129]: pgmap v10210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:52 smithi160 ceph-mon[36715]: pgmap v10210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:52 smithi183 ceph-mon[37273]: pgmap v10210: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:12:55.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:54 smithi144 ceph-mon[32129]: pgmap v10211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:54 smithi160 ceph-mon[36715]: pgmap v10211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:54 smithi183 ceph-mon[37273]: pgmap v10211: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:56 smithi144 ceph-mon[32129]: pgmap v10212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18419.955596 secs 2024-07-26T16:12:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:56 smithi160 ceph-mon[36715]: pgmap v10212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18419.955596 secs 2024-07-26T16:12:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:56 smithi183 ceph-mon[37273]: pgmap v10212: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:12:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18419.955596 secs 2024-07-26T16:12:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:58.458+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:12:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:58.459+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:12:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:58.459+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:12:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:58.459+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:12:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:12:58.459+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:12:58 smithi144 ceph-mon[32129]: pgmap v10213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:12:58 smithi160 ceph-mon[36715]: pgmap v10213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:12:59.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:12:58 smithi183 ceph-mon[37273]: pgmap v10213: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:00 smithi144 ceph-mon[32129]: pgmap v10214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:00 smithi160 ceph-mon[36715]: pgmap v10214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:00 smithi183 ceph-mon[37273]: pgmap v10214: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18424.955685 secs 2024-07-26T16:13:02.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18424.955685 secs 2024-07-26T16:13:02.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18424.955685 secs 2024-07-26T16:13:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:02 smithi160 ceph-mon[36715]: pgmap v10215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:02 smithi183 ceph-mon[37273]: pgmap v10215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:02 smithi144 ceph-mon[32129]: pgmap v10215: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:04 smithi160 ceph-mon[36715]: pgmap v10216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:04 smithi183 ceph-mon[37273]: pgmap v10216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:05.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:04 smithi144 ceph-mon[32129]: pgmap v10216: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:06 smithi160 ceph-mon[36715]: pgmap v10217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18429.955817 secs 2024-07-26T16:13:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:06 smithi183 ceph-mon[37273]: pgmap v10217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18429.955817 secs 2024-07-26T16:13:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:06 smithi144 ceph-mon[32129]: pgmap v10217: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18429.955817 secs 2024-07-26T16:13:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:08 smithi160 ceph-mon[36715]: pgmap v10218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:08 smithi183 ceph-mon[37273]: pgmap v10218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:08 smithi144 ceph-mon[32129]: pgmap v10218: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:10 smithi160 ceph-mon[36715]: pgmap v10219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:10 smithi183 ceph-mon[37273]: pgmap v10219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:11.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:10 smithi144 ceph-mon[32129]: pgmap v10219: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18434.955940 secs 2024-07-26T16:13:12.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18434.955940 secs 2024-07-26T16:13:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18434.955940 secs 2024-07-26T16:13:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:13 smithi160 ceph-mon[36715]: pgmap v10220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:13 smithi183 ceph-mon[37273]: pgmap v10220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:13.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:13 smithi144 ceph-mon[32129]: pgmap v10220: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:13.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:13:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:13.477+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:13:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:13.477+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:13:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:13.477+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:13:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:13.477+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:15.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:15 smithi144 ceph-mon[32129]: pgmap v10221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:15 smithi160 ceph-mon[36715]: pgmap v10221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:15 smithi183 ceph-mon[37273]: pgmap v10221: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:16 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:16 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:16 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:17 smithi160 ceph-mon[36715]: pgmap v10222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18439.956004 secs 2024-07-26T16:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:17 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:17 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:13:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:17 smithi183 ceph-mon[37273]: pgmap v10222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18439.956004 secs 2024-07-26T16:13:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:17 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:17 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:13:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:17 smithi144 ceph-mon[32129]: pgmap v10222: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18439.956004 secs 2024-07-26T16:13:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:17 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:17 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:13:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:18 smithi160 ceph-mon[36715]: pgmap v10223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:13:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:18 smithi160 ceph-mon[36715]: pgmap v10224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:13:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:18 smithi183 ceph-mon[37273]: pgmap v10223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:13:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:18 smithi183 ceph-mon[37273]: pgmap v10224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:13:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:18 smithi144 ceph-mon[32129]: pgmap v10223: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:13:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:18 smithi144 ceph-mon[32129]: pgmap v10224: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:13:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:20 smithi160 ceph-mon[36715]: pgmap v10225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:13:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:20 smithi183 ceph-mon[37273]: pgmap v10225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:13:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:20 smithi144 ceph-mon[32129]: pgmap v10225: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:13:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:22 smithi160 ceph-mon[36715]: pgmap v10226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:13:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18444.956127 secs 2024-07-26T16:13:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:22 smithi183 ceph-mon[37273]: pgmap v10226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:13:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18444.956127 secs 2024-07-26T16:13:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:22 smithi144 ceph-mon[32129]: pgmap v10226: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:13:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18444.956127 secs 2024-07-26T16:13:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:24 smithi183 ceph-mon[37273]: pgmap v10227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:24 smithi144 ceph-mon[32129]: pgmap v10227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:24 smithi160 ceph-mon[36715]: pgmap v10227: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:26 smithi183 ceph-mon[37273]: pgmap v10228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:26 smithi144 ceph-mon[32129]: pgmap v10228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:26 smithi160 ceph-mon[36715]: pgmap v10228: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:27.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18449.956253 secs 2024-07-26T16:13:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18449.956253 secs 2024-07-26T16:13:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18449.956253 secs 2024-07-26T16:13:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:28 smithi183 ceph-mon[37273]: pgmap v10229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:28 smithi144 ceph-mon[32129]: pgmap v10229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:28.494 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:28.493+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:13:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:28.494+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:13:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:28.494+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:13:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:28.494+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:13:28.495 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:28.494+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:28 smithi160 ceph-mon[36715]: pgmap v10229: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:30 smithi144 ceph-mon[32129]: pgmap v10230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:30 smithi183 ceph-mon[37273]: pgmap v10230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:30 smithi160 ceph-mon[36715]: pgmap v10230: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:32 smithi183 ceph-mon[37273]: pgmap v10231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18454.956374 secs 2024-07-26T16:13:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:32 smithi144 ceph-mon[32129]: pgmap v10231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18454.956374 secs 2024-07-26T16:13:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:32 smithi160 ceph-mon[36715]: pgmap v10231: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18454.956374 secs 2024-07-26T16:13:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:34 smithi183 ceph-mon[37273]: pgmap v10232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:34 smithi144 ceph-mon[32129]: pgmap v10232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:34 smithi160 ceph-mon[36715]: pgmap v10232: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:36 smithi183 ceph-mon[37273]: pgmap v10233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:36 smithi144 ceph-mon[32129]: pgmap v10233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:36 smithi160 ceph-mon[36715]: pgmap v10233: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18459.956492 secs 2024-07-26T16:13:37.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18459.956492 secs 2024-07-26T16:13:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18459.956492 secs 2024-07-26T16:13:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:38 smithi183 ceph-mon[37273]: pgmap v10234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:38.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:38 smithi144 ceph-mon[32129]: pgmap v10234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:38 smithi160 ceph-mon[36715]: pgmap v10234: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:13:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:39 smithi144 ceph-mon[32129]: pgmap v10235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:39 smithi160 ceph-mon[36715]: pgmap v10235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:39 smithi183 ceph-mon[37273]: pgmap v10235: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:42 smithi160 ceph-mon[36715]: pgmap v10236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18464.956611 secs 2024-07-26T16:13:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:42 smithi183 ceph-mon[37273]: pgmap v10236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18464.956611 secs 2024-07-26T16:13:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:42 smithi144 ceph-mon[32129]: pgmap v10236: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18464.956611 secs 2024-07-26T16:13:43.503 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:43.501+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:13:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:43.502+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:13:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:43.502+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:13:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:43.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:13:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:43.502+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:44 smithi160 ceph-mon[36715]: pgmap v10237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:44 smithi183 ceph-mon[37273]: pgmap v10237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:44 smithi144 ceph-mon[32129]: pgmap v10237: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:13:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:46 smithi160 ceph-mon[36715]: pgmap v10238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:46 smithi183 ceph-mon[37273]: pgmap v10238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:46 smithi144 ceph-mon[32129]: pgmap v10238: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:13:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18469.956723 secs 2024-07-26T16:13:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18469.956723 secs 2024-07-26T16:13:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18469.956723 secs 2024-07-26T16:13:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:48 smithi160 ceph-mon[36715]: pgmap v10239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:48 smithi183 ceph-mon[37273]: pgmap v10239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:48 smithi144 ceph-mon[32129]: pgmap v10239: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:50 smithi160 ceph-mon[36715]: pgmap v10240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:50 smithi183 ceph-mon[37273]: pgmap v10240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:50 smithi144 ceph-mon[32129]: pgmap v10240: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:52 smithi183 ceph-mon[37273]: pgmap v10241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18474.956850 secs 2024-07-26T16:13:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:52 smithi144 ceph-mon[32129]: pgmap v10241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18474.956850 secs 2024-07-26T16:13:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:52 smithi160 ceph-mon[36715]: pgmap v10241: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18474.956850 secs 2024-07-26T16:13:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:54 smithi183 ceph-mon[37273]: pgmap v10242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:54 smithi144 ceph-mon[32129]: pgmap v10242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:54 smithi160 ceph-mon[36715]: pgmap v10242: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:13:56.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:56 smithi183 ceph-mon[37273]: pgmap v10243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:56 smithi144 ceph-mon[32129]: pgmap v10243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:56 smithi160 ceph-mon[36715]: pgmap v10243: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:13:57.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18479.956972 secs 2024-07-26T16:13:57.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18479.956972 secs 2024-07-26T16:13:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18479.956972 secs 2024-07-26T16:13:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:13:58 smithi183 ceph-mon[37273]: pgmap v10244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:13:58 smithi144 ceph-mon[32129]: pgmap v10244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:13:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:58.517+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:13:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:58.518+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:13:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:58.518+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:13:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:58.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:13:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:13:58.518+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:13:58 smithi160 ceph-mon[36715]: pgmap v10244: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:14:00.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:00 smithi183 ceph-mon[37273]: pgmap v10245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:00.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:00 smithi144 ceph-mon[32129]: pgmap v10245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:00 smithi160 ceph-mon[36715]: pgmap v10245: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18484.957097 secs 2024-07-26T16:14:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:02 smithi183 ceph-mon[37273]: pgmap v10246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18484.957097 secs 2024-07-26T16:14:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:02 smithi144 ceph-mon[32129]: pgmap v10246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18484.957097 secs 2024-07-26T16:14:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:02 smithi160 ceph-mon[36715]: pgmap v10246: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:04 smithi183 ceph-mon[37273]: pgmap v10247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:04.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:04 smithi144 ceph-mon[32129]: pgmap v10247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:04 smithi160 ceph-mon[36715]: pgmap v10247: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:06 smithi183 ceph-mon[37273]: pgmap v10248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:06 smithi144 ceph-mon[32129]: pgmap v10248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:06 smithi160 ceph-mon[36715]: pgmap v10248: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18489.957192 secs 2024-07-26T16:14:07.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18489.957192 secs 2024-07-26T16:14:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18489.957192 secs 2024-07-26T16:14:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:08 smithi183 ceph-mon[37273]: pgmap v10249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:14:08.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:08 smithi144 ceph-mon[32129]: pgmap v10249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:14:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:08 smithi160 ceph-mon[36715]: pgmap v10249: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:14:09.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:09 smithi144 ceph-mon[32129]: pgmap v10250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:09 smithi160 ceph-mon[36715]: pgmap v10250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:09 smithi183 ceph-mon[37273]: pgmap v10250: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18494.957333 secs 2024-07-26T16:14:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:12 smithi160 ceph-mon[36715]: pgmap v10251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18494.957333 secs 2024-07-26T16:14:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:12 smithi183 ceph-mon[37273]: pgmap v10251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18494.957333 secs 2024-07-26T16:14:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:12 smithi144 ceph-mon[32129]: pgmap v10251: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:13.523+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:14:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:13.524+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:14:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:13.524+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:14:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:13.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:14:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:13.524+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:14 smithi160 ceph-mon[36715]: pgmap v10252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:14 smithi183 ceph-mon[37273]: pgmap v10252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:14 smithi144 ceph-mon[32129]: pgmap v10252: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:16 smithi160 ceph-mon[36715]: pgmap v10253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:16 smithi183 ceph-mon[37273]: pgmap v10253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:16 smithi144 ceph-mon[32129]: pgmap v10253: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:17.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18499.957432 secs 2024-07-26T16:14:17.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18499.957432 secs 2024-07-26T16:14:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18499.957432 secs 2024-07-26T16:14:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:18 smithi183 ceph-mon[37273]: pgmap v10254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:14:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:18 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:18 smithi144 ceph-mon[32129]: pgmap v10254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:14:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:18 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:18 smithi160 ceph-mon[36715]: pgmap v10254: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:14:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:18 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:19 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:14:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:19 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:14:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:19 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:14:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:20 smithi183 ceph-mon[37273]: pgmap v10255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:20 smithi183 ceph-mon[37273]: pgmap v10256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:20.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:20 smithi160 ceph-mon[36715]: pgmap v10255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:20.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:20 smithi160 ceph-mon[36715]: pgmap v10256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:20 smithi144 ceph-mon[32129]: pgmap v10255: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:20.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:20 smithi144 ceph-mon[32129]: pgmap v10256: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:22 smithi183 ceph-mon[37273]: pgmap v10257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18504.957576 secs 2024-07-26T16:14:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:22 smithi144 ceph-mon[32129]: pgmap v10257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18504.957576 secs 2024-07-26T16:14:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:22 smithi160 ceph-mon[36715]: pgmap v10257: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18504.957576 secs 2024-07-26T16:14:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:24 smithi183 ceph-mon[37273]: pgmap v10258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:24 smithi144 ceph-mon[32129]: pgmap v10258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:24 smithi160 ceph-mon[36715]: pgmap v10258: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:26 smithi183 ceph-mon[37273]: pgmap v10259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:14:26.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:26 smithi144 ceph-mon[32129]: pgmap v10259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:14:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:26 smithi160 ceph-mon[36715]: pgmap v10259: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:14:27.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18509.957693 secs 2024-07-26T16:14:27.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18509.957693 secs 2024-07-26T16:14:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18509.957693 secs 2024-07-26T16:14:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:28 smithi183 ceph-mon[37273]: pgmap v10260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:28 smithi144 ceph-mon[32129]: pgmap v10260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:28.532+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:14:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:28.533+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:14:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:28.533+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:14:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:28.533+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:14:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:28.533+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:28 smithi160 ceph-mon[36715]: pgmap v10260: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:14:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:30 smithi183 ceph-mon[37273]: pgmap v10261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:14:30.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:30 smithi144 ceph-mon[32129]: pgmap v10261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:14:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:30 smithi160 ceph-mon[36715]: pgmap v10261: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:14:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:32 smithi183 ceph-mon[37273]: pgmap v10262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18514.957815 secs 2024-07-26T16:14:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:32 smithi144 ceph-mon[32129]: pgmap v10262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18514.957815 secs 2024-07-26T16:14:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:32 smithi160 ceph-mon[36715]: pgmap v10262: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18514.957815 secs 2024-07-26T16:14:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:33 smithi144 ceph-mon[32129]: pgmap v10263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:33 smithi160 ceph-mon[36715]: pgmap v10263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:33 smithi183 ceph-mon[37273]: pgmap v10263: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:35 smithi160 ceph-mon[36715]: pgmap v10264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:35 smithi183 ceph-mon[37273]: pgmap v10264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:35 smithi144 ceph-mon[32129]: pgmap v10264: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18519.957936 secs 2024-07-26T16:14:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:37 smithi160 ceph-mon[36715]: pgmap v10265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18519.957936 secs 2024-07-26T16:14:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:37 smithi183 ceph-mon[37273]: pgmap v10265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18519.957936 secs 2024-07-26T16:14:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:37 smithi144 ceph-mon[32129]: pgmap v10265: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:39 smithi160 ceph-mon[36715]: pgmap v10266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:39 smithi183 ceph-mon[37273]: pgmap v10266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:39 smithi144 ceph-mon[32129]: pgmap v10266: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:41 smithi160 ceph-mon[36715]: pgmap v10267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:41 smithi183 ceph-mon[37273]: pgmap v10267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:41 smithi144 ceph-mon[32129]: pgmap v10267: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18524.958049 secs 2024-07-26T16:14:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18524.958049 secs 2024-07-26T16:14:42.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18524.958049 secs 2024-07-26T16:14:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:43.549+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:14:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:43.550+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:14:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:43.550+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:14:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:43.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:14:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:43.550+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:43 smithi160 ceph-mon[36715]: pgmap v10268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:43 smithi183 ceph-mon[37273]: pgmap v10268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:43 smithi144 ceph-mon[32129]: pgmap v10268: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:45 smithi160 ceph-mon[36715]: pgmap v10269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:45 smithi183 ceph-mon[37273]: pgmap v10269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:45.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:45 smithi144 ceph-mon[32129]: pgmap v10269: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18529.958167 secs 2024-07-26T16:14:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:47 smithi144 ceph-mon[32129]: pgmap v10270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18529.958167 secs 2024-07-26T16:14:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:47 smithi160 ceph-mon[36715]: pgmap v10270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18529.958167 secs 2024-07-26T16:14:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:47 smithi183 ceph-mon[37273]: pgmap v10270: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:49.371+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:14:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:49.371+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T16:14:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:49.371+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:14:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:49.371+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T16:14:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:49 smithi160 ceph-mon[36715]: pgmap v10271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:49 smithi183 ceph-mon[37273]: pgmap v10271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:49 smithi144 ceph-mon[32129]: pgmap v10271: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:51 smithi160 ceph-mon[36715]: pgmap v10272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:51 smithi183 ceph-mon[37273]: pgmap v10272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:51 smithi144 ceph-mon[32129]: pgmap v10272: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18534.958305 secs 2024-07-26T16:14:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18534.958305 secs 2024-07-26T16:14:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18534.958305 secs 2024-07-26T16:14:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:53 smithi160 ceph-mon[36715]: pgmap v10273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:53 smithi183 ceph-mon[37273]: pgmap v10273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:53 smithi144 ceph-mon[32129]: pgmap v10273: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:14:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:55 smithi183 ceph-mon[37273]: pgmap v10274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:55 smithi144 ceph-mon[32129]: pgmap v10274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:55.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:55 smithi160 ceph-mon[36715]: pgmap v10274: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18539.958355 secs 2024-07-26T16:14:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:57 smithi183 ceph-mon[37273]: pgmap v10275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18539.958355 secs 2024-07-26T16:14:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:57 smithi144 ceph-mon[32129]: pgmap v10275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18539.958355 secs 2024-07-26T16:14:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:57 smithi160 ceph-mon[36715]: pgmap v10275: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:58.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:14:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:58.566+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:14:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:58.566+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:14:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:58.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:14:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:14:58.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:14:59 smithi183 ceph-mon[37273]: pgmap v10276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:14:59 smithi144 ceph-mon[32129]: pgmap v10276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:14:59.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:14:59 smithi160 ceph-mon[36715]: pgmap v10276: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:01 smithi183 ceph-mon[37273]: pgmap v10277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:01 smithi144 ceph-mon[32129]: pgmap v10277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:01 smithi160 ceph-mon[36715]: pgmap v10277: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18544.958476 secs 2024-07-26T16:15:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18544.958476 secs 2024-07-26T16:15:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18544.958476 secs 2024-07-26T16:15:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:03 smithi183 ceph-mon[37273]: pgmap v10278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:03.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:03 smithi144 ceph-mon[32129]: pgmap v10278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:03 smithi160 ceph-mon[36715]: pgmap v10278: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:05 smithi183 ceph-mon[37273]: pgmap v10279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:05.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:05 smithi144 ceph-mon[32129]: pgmap v10279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:05 smithi160 ceph-mon[36715]: pgmap v10279: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18549.958593 secs 2024-07-26T16:15:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:07 smithi183 ceph-mon[37273]: pgmap v10280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18549.958593 secs 2024-07-26T16:15:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:07 smithi144 ceph-mon[32129]: pgmap v10280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18549.958593 secs 2024-07-26T16:15:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:07 smithi160 ceph-mon[36715]: pgmap v10280: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:10 smithi160 ceph-mon[36715]: pgmap v10281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:10 smithi183 ceph-mon[37273]: pgmap v10281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:10 smithi144 ceph-mon[32129]: pgmap v10281: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:12 smithi160 ceph-mon[36715]: pgmap v10282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18554.958716 secs 2024-07-26T16:15:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:12 smithi183 ceph-mon[37273]: pgmap v10282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18554.958716 secs 2024-07-26T16:15:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:12 smithi144 ceph-mon[32129]: pgmap v10282: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18554.958716 secs 2024-07-26T16:15:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:13.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:15:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:13.584+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:15:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:13.584+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:15:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:13.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:15:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:13.584+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:14 smithi183 ceph-mon[37273]: pgmap v10283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:14.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:14 smithi144 ceph-mon[32129]: pgmap v10283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:14 smithi160 ceph-mon[36715]: pgmap v10283: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:16 smithi183 ceph-mon[37273]: pgmap v10284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:16.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:16 smithi144 ceph-mon[32129]: pgmap v10284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:16 smithi160 ceph-mon[36715]: pgmap v10284: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18559.958837 secs 2024-07-26T16:15:17.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18559.958837 secs 2024-07-26T16:15:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18559.958837 secs 2024-07-26T16:15:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:18 smithi183 ceph-mon[37273]: pgmap v10285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:18.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:18 smithi144 ceph-mon[32129]: pgmap v10285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:18 smithi160 ceph-mon[36715]: pgmap v10285: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:19.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: pgmap v10286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:20.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:20.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:15:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: pgmap v10286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:20.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:15:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: pgmap v10286: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:20.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:15:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:21 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:15:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:21 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:15:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:21 smithi183 ceph-mon[37273]: pgmap v10287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:15:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:21 smithi183 ceph-mon[37273]: pgmap v10288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:21.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:21 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:15:21.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:21 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:15:21.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:21 smithi160 ceph-mon[36715]: pgmap v10287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:15:21.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:21 smithi160 ceph-mon[36715]: pgmap v10288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:21.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:21 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:15:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:21 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:15:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:21 smithi144 ceph-mon[32129]: pgmap v10287: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:15:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:21 smithi144 ceph-mon[32129]: pgmap v10288: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18564.958960 secs 2024-07-26T16:15:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18564.958960 secs 2024-07-26T16:15:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18564.958960 secs 2024-07-26T16:15:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:23 smithi183 ceph-mon[37273]: pgmap v10289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:23 smithi144 ceph-mon[32129]: pgmap v10289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:23 smithi160 ceph-mon[36715]: pgmap v10289: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:25 smithi183 ceph-mon[37273]: pgmap v10290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:25 smithi144 ceph-mon[32129]: pgmap v10290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:25 smithi160 ceph-mon[36715]: pgmap v10290: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:27 smithi183 ceph-mon[37273]: pgmap v10291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18569.959086 secs 2024-07-26T16:15:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:27 smithi144 ceph-mon[32129]: pgmap v10291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18569.959086 secs 2024-07-26T16:15:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:27 smithi160 ceph-mon[36715]: pgmap v10291: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:15:27.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18569.959086 secs 2024-07-26T16:15:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:28.594+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:15:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:28.595+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:15:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:28.595+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:15:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:28.595+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:15:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:28.595+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:29 smithi144 ceph-mon[32129]: pgmap v10292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:29 smithi160 ceph-mon[36715]: pgmap v10292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:29.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:29 smithi183 ceph-mon[37273]: pgmap v10292: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:31 smithi144 ceph-mon[32129]: pgmap v10293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:31 smithi160 ceph-mon[36715]: pgmap v10293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:31 smithi183 ceph-mon[37273]: pgmap v10293: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18574.959202 secs 2024-07-26T16:15:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18574.959202 secs 2024-07-26T16:15:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18574.959202 secs 2024-07-26T16:15:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:33 smithi144 ceph-mon[32129]: pgmap v10294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:33 smithi160 ceph-mon[36715]: pgmap v10294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:33 smithi183 ceph-mon[37273]: pgmap v10294: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:34.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:34 smithi144 ceph-mon[32129]: pgmap v10295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:34 smithi160 ceph-mon[36715]: pgmap v10295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:34 smithi183 ceph-mon[37273]: pgmap v10295: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:36 smithi144 ceph-mon[32129]: pgmap v10296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18579.959323 secs 2024-07-26T16:15:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:36 smithi160 ceph-mon[36715]: pgmap v10296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18579.959323 secs 2024-07-26T16:15:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:36 smithi183 ceph-mon[37273]: pgmap v10296: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18579.959323 secs 2024-07-26T16:15:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:38 smithi144 ceph-mon[32129]: pgmap v10297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:38 smithi160 ceph-mon[36715]: pgmap v10297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:38 smithi183 ceph-mon[37273]: pgmap v10297: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:40 smithi144 ceph-mon[32129]: pgmap v10298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:40 smithi160 ceph-mon[36715]: pgmap v10298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:40.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:40 smithi183 ceph-mon[37273]: pgmap v10298: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:41.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18584.959444 secs 2024-07-26T16:15:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18584.959444 secs 2024-07-26T16:15:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18584.959444 secs 2024-07-26T16:15:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:42 smithi160 ceph-mon[36715]: pgmap v10299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:42.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:42 smithi183 ceph-mon[37273]: pgmap v10299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:42 smithi144 ceph-mon[32129]: pgmap v10299: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:43.598+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:15:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:43.599+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:15:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:43.599+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:15:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:43.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:15:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:43.599+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:44 smithi160 ceph-mon[36715]: pgmap v10300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:44 smithi183 ceph-mon[37273]: pgmap v10300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:44.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:44 smithi144 ceph-mon[32129]: pgmap v10300: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:46 smithi160 ceph-mon[36715]: pgmap v10301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18589.959567 secs 2024-07-26T16:15:46.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:46 smithi183 ceph-mon[37273]: pgmap v10301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18589.959567 secs 2024-07-26T16:15:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:46 smithi144 ceph-mon[32129]: pgmap v10301: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18589.959567 secs 2024-07-26T16:15:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:48 smithi160 ceph-mon[36715]: pgmap v10302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:48 smithi183 ceph-mon[37273]: pgmap v10302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:48.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:48 smithi144 ceph-mon[32129]: pgmap v10302: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:15:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:50 smithi160 ceph-mon[36715]: pgmap v10303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:50.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:50 smithi183 ceph-mon[37273]: pgmap v10303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:50.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:50 smithi144 ceph-mon[32129]: pgmap v10303: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:15:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18594.959688 secs 2024-07-26T16:15:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18594.959688 secs 2024-07-26T16:15:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18594.959688 secs 2024-07-26T16:15:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:52 smithi160 ceph-mon[36715]: pgmap v10304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:52 smithi183 ceph-mon[37273]: pgmap v10304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:52 smithi144 ceph-mon[32129]: pgmap v10304: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:15:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:54 smithi160 ceph-mon[36715]: pgmap v10305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:54.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:54 smithi183 ceph-mon[37273]: pgmap v10305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:54.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:54 smithi144 ceph-mon[32129]: pgmap v10305: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:56 smithi160 ceph-mon[36715]: pgmap v10306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18599.959803 secs 2024-07-26T16:15:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:56 smithi183 ceph-mon[37273]: pgmap v10306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18599.959803 secs 2024-07-26T16:15:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:56 smithi144 ceph-mon[32129]: pgmap v10306: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:15:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18599.959803 secs 2024-07-26T16:15:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:58.611+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:15:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:58.612+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:15:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:58.612+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:15:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:58.612+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:15:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:15:58.612+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:15:58 smithi160 ceph-mon[36715]: pgmap v10307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:15:58 smithi183 ceph-mon[37273]: pgmap v10307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:15:58.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:15:58 smithi144 ceph-mon[32129]: pgmap v10307: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:00 smithi160 ceph-mon[36715]: pgmap v10308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:16:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:00 smithi183 ceph-mon[37273]: pgmap v10308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:16:00.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:00 smithi144 ceph-mon[32129]: pgmap v10308: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:16:01.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18604.959922 secs 2024-07-26T16:16:01.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18604.959922 secs 2024-07-26T16:16:01.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18604.959922 secs 2024-07-26T16:16:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:02 smithi160 ceph-mon[36715]: pgmap v10309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:02 smithi183 ceph-mon[37273]: pgmap v10309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:02.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:02 smithi144 ceph-mon[32129]: pgmap v10309: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:04 smithi160 ceph-mon[36715]: pgmap v10310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:04 smithi183 ceph-mon[37273]: pgmap v10310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:04 smithi144 ceph-mon[32129]: pgmap v10310: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:06 smithi183 ceph-mon[37273]: pgmap v10311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18609.960067 secs 2024-07-26T16:16:06.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:06 smithi144 ceph-mon[32129]: pgmap v10311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18609.960067 secs 2024-07-26T16:16:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:06 smithi160 ceph-mon[36715]: pgmap v10311: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18609.960067 secs 2024-07-26T16:16:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:08 smithi183 ceph-mon[37273]: pgmap v10312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:08.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:08 smithi144 ceph-mon[32129]: pgmap v10312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:09.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:08 smithi160 ceph-mon[36715]: pgmap v10312: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:10 smithi183 ceph-mon[37273]: pgmap v10313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:10 smithi144 ceph-mon[32129]: pgmap v10313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:11.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:10 smithi160 ceph-mon[36715]: pgmap v10313: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18614.960209 secs 2024-07-26T16:16:11.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18614.960209 secs 2024-07-26T16:16:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18614.960209 secs 2024-07-26T16:16:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:12 smithi183 ceph-mon[37273]: pgmap v10314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:12 smithi144 ceph-mon[32129]: pgmap v10314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:12 smithi160 ceph-mon[36715]: pgmap v10314: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:13.629+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:16:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:13.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:16:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:13.630+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:16:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:13.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:16:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:13.630+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:14 smithi183 ceph-mon[37273]: pgmap v10315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:14 smithi144 ceph-mon[32129]: pgmap v10315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:14 smithi160 ceph-mon[36715]: pgmap v10315: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:16 smithi183 ceph-mon[37273]: pgmap v10316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18619.960273 secs 2024-07-26T16:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:16 smithi144 ceph-mon[32129]: pgmap v10316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18619.960273 secs 2024-07-26T16:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:16 smithi160 ceph-mon[36715]: pgmap v10316: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18619.960273 secs 2024-07-26T16:16:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:18 smithi183 ceph-mon[37273]: pgmap v10317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:18 smithi144 ceph-mon[32129]: pgmap v10317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:18 smithi160 ceph-mon[36715]: pgmap v10317: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:19.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:19 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:19.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:19 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:19 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: pgmap v10318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:20.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:20 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:16:20.895 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: pgmap v10318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:20.895 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.895 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.896 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:20.897 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:20 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:16:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: pgmap v10318: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:21.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:20 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:16:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:21 smithi183 ceph-mon[37273]: pgmap v10319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:21 smithi183 ceph-mon[37273]: pgmap v10320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18624.960394 secs 2024-07-26T16:16:21.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:21 smithi160 ceph-mon[36715]: pgmap v10319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:21.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:21 smithi160 ceph-mon[36715]: pgmap v10320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:21.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18624.960394 secs 2024-07-26T16:16:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:21 smithi144 ceph-mon[32129]: pgmap v10319: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:21 smithi144 ceph-mon[32129]: pgmap v10320: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18624.960394 secs 2024-07-26T16:16:23.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:23 smithi183 ceph-mon[37273]: pgmap v10321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:23.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:23 smithi144 ceph-mon[32129]: pgmap v10321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:23 smithi160 ceph-mon[36715]: pgmap v10321: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:25.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:25 smithi144 ceph-mon[32129]: pgmap v10322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:25 smithi160 ceph-mon[36715]: pgmap v10322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:25 smithi183 ceph-mon[37273]: pgmap v10322: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:16:26.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18629.960513 secs 2024-07-26T16:16:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18629.960513 secs 2024-07-26T16:16:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18629.960513 secs 2024-07-26T16:16:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:27 smithi144 ceph-mon[32129]: pgmap v10323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:27 smithi160 ceph-mon[36715]: pgmap v10323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:27 smithi183 ceph-mon[37273]: pgmap v10323: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:16:28.636 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:28.634+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:16:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:28.635+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:16:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:28.635+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:16:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:28.635+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:16:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:28.635+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:29.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:29 smithi144 ceph-mon[32129]: pgmap v10324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:16:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:29 smithi160 ceph-mon[36715]: pgmap v10324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:16:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:29 smithi183 ceph-mon[37273]: pgmap v10324: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:16:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:31 smithi144 ceph-mon[32129]: pgmap v10325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:31.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18634.960638 secs 2024-07-26T16:16:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:31 smithi160 ceph-mon[36715]: pgmap v10325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18634.960638 secs 2024-07-26T16:16:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:31 smithi183 ceph-mon[37273]: pgmap v10325: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:16:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18634.960638 secs 2024-07-26T16:16:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:33 smithi144 ceph-mon[32129]: pgmap v10326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:33 smithi160 ceph-mon[36715]: pgmap v10326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:33 smithi183 ceph-mon[37273]: pgmap v10326: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:35 smithi144 ceph-mon[32129]: pgmap v10327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:35 smithi160 ceph-mon[36715]: pgmap v10327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:35 smithi183 ceph-mon[37273]: pgmap v10327: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18639.960787 secs 2024-07-26T16:16:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18639.960787 secs 2024-07-26T16:16:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18639.960787 secs 2024-07-26T16:16:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:37 smithi144 ceph-mon[32129]: pgmap v10328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:37 smithi160 ceph-mon[36715]: pgmap v10328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:37 smithi183 ceph-mon[37273]: pgmap v10328: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:39 smithi144 ceph-mon[32129]: pgmap v10329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:39 smithi160 ceph-mon[36715]: pgmap v10329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:39 smithi183 ceph-mon[37273]: pgmap v10329: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:41 smithi144 ceph-mon[32129]: pgmap v10330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18644.960902 secs 2024-07-26T16:16:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:41 smithi160 ceph-mon[36715]: pgmap v10330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18644.960902 secs 2024-07-26T16:16:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:41 smithi183 ceph-mon[37273]: pgmap v10330: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18644.960902 secs 2024-07-26T16:16:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:43.651+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:16:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:43.652+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:16:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:43.652+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:16:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:43.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:16:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:43.652+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:43 smithi144 ceph-mon[32129]: pgmap v10331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:43 smithi160 ceph-mon[36715]: pgmap v10331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:43 smithi183 ceph-mon[37273]: pgmap v10331: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:45.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:45 smithi144 ceph-mon[32129]: pgmap v10332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:45 smithi160 ceph-mon[36715]: pgmap v10332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:45 smithi183 ceph-mon[37273]: pgmap v10332: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18649.961016 secs 2024-07-26T16:16:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18649.961016 secs 2024-07-26T16:16:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18649.961016 secs 2024-07-26T16:16:48.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:47 smithi183 ceph-mon[37273]: pgmap v10333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:47 smithi160 ceph-mon[36715]: pgmap v10333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:48.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:47 smithi144 ceph-mon[32129]: pgmap v10333: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:49 smithi160 ceph-mon[36715]: pgmap v10334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:49 smithi183 ceph-mon[37273]: pgmap v10334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:49 smithi144 ceph-mon[32129]: pgmap v10334: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:51 smithi160 ceph-mon[36715]: pgmap v10335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18654.961139 secs 2024-07-26T16:16:52.126 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:51 smithi144 ceph-mon[32129]: pgmap v10335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:52.126 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18654.961139 secs 2024-07-26T16:16:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:51 smithi183 ceph-mon[37273]: pgmap v10335: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18654.961139 secs 2024-07-26T16:16:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:53 smithi160 ceph-mon[36715]: pgmap v10336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:53 smithi183 ceph-mon[37273]: pgmap v10336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:53 smithi144 ceph-mon[32129]: pgmap v10336: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:16:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:55 smithi160 ceph-mon[36715]: pgmap v10337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:55 smithi183 ceph-mon[37273]: pgmap v10337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:55 smithi144 ceph-mon[32129]: pgmap v10337: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:16:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18659.961259 secs 2024-07-26T16:16:57.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18659.961259 secs 2024-07-26T16:16:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18659.961259 secs 2024-07-26T16:16:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:57 smithi160 ceph-mon[36715]: pgmap v10338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:57 smithi183 ceph-mon[37273]: pgmap v10338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:58.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:57 smithi144 ceph-mon[32129]: pgmap v10338: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:16:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:58.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:16:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:58.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:16:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:58.653+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:16:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:58.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:16:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:16:58.653+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:16:59 smithi160 ceph-mon[36715]: pgmap v10339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:16:59 smithi183 ceph-mon[37273]: pgmap v10339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:16:59 smithi144 ceph-mon[32129]: pgmap v10339: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:01 smithi160 ceph-mon[36715]: pgmap v10340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18664.961306 secs 2024-07-26T16:17:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:01 smithi183 ceph-mon[37273]: pgmap v10340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18664.961306 secs 2024-07-26T16:17:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:01 smithi144 ceph-mon[32129]: pgmap v10340: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18664.961306 secs 2024-07-26T16:17:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:03 smithi160 ceph-mon[36715]: pgmap v10341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:03 smithi183 ceph-mon[37273]: pgmap v10341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:04.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:03 smithi144 ceph-mon[32129]: pgmap v10341: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:05 smithi160 ceph-mon[36715]: pgmap v10342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:05 smithi183 ceph-mon[37273]: pgmap v10342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:06.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:05 smithi144 ceph-mon[32129]: pgmap v10342: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:07.139 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18669.961421 secs 2024-07-26T16:17:07.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18669.961421 secs 2024-07-26T16:17:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18669.961421 secs 2024-07-26T16:17:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:07 smithi183 ceph-mon[37273]: pgmap v10343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:07 smithi144 ceph-mon[32129]: pgmap v10343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:07 smithi160 ceph-mon[36715]: pgmap v10343: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:09 smithi183 ceph-mon[37273]: pgmap v10344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:09 smithi144 ceph-mon[32129]: pgmap v10344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:09 smithi160 ceph-mon[36715]: pgmap v10344: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:11 smithi183 ceph-mon[37273]: pgmap v10345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18674.961508 secs 2024-07-26T16:17:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:11 smithi144 ceph-mon[32129]: pgmap v10345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18674.961508 secs 2024-07-26T16:17:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:11 smithi160 ceph-mon[36715]: pgmap v10345: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18674.961508 secs 2024-07-26T16:17:13.672 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:13.670+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:17:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:13.671+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:17:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:13.671+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:17:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:13.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:17:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:13.671+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:13 smithi183 ceph-mon[37273]: pgmap v10346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:13 smithi144 ceph-mon[32129]: pgmap v10346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:13 smithi160 ceph-mon[36715]: pgmap v10346: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:15 smithi183 ceph-mon[37273]: pgmap v10347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:15 smithi144 ceph-mon[32129]: pgmap v10347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:15 smithi160 ceph-mon[36715]: pgmap v10347: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18679.961632 secs 2024-07-26T16:17:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18679.961632 secs 2024-07-26T16:17:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18679.961632 secs 2024-07-26T16:17:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:17 smithi183 ceph-mon[37273]: pgmap v10348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:18.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:17 smithi144 ceph-mon[32129]: pgmap v10348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:17 smithi160 ceph-mon[36715]: pgmap v10348: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:19 smithi183 ceph-mon[37273]: pgmap v10349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:19 smithi144 ceph-mon[32129]: pgmap v10349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:19 smithi160 ceph-mon[36715]: pgmap v10349: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:20 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:21.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:20 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:21.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:20 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:21 smithi183 ceph-mon[37273]: pgmap v10350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18684.961706 secs 2024-07-26T16:17:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:21 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:21 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:17:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:21 smithi144 ceph-mon[32129]: pgmap v10350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18684.961706 secs 2024-07-26T16:17:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:21 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:21 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:21 smithi160 ceph-mon[36715]: pgmap v10350: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18684.961706 secs 2024-07-26T16:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:21 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:21 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:17:23.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:22 smithi144 ceph-mon[32129]: pgmap v10351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:17:23.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:22 smithi144 ceph-mon[32129]: pgmap v10352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:17:23.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:22 smithi183 ceph-mon[37273]: pgmap v10351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:17:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:22 smithi183 ceph-mon[37273]: pgmap v10352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:17:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:22 smithi160 ceph-mon[36715]: pgmap v10351: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:17:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:22 smithi160 ceph-mon[36715]: pgmap v10352: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:17:25.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:24 smithi183 ceph-mon[37273]: pgmap v10353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:17:25.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:24 smithi144 ceph-mon[32129]: pgmap v10353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:17:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:24 smithi160 ceph-mon[36715]: pgmap v10353: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:17:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:26 smithi183 ceph-mon[37273]: pgmap v10354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:17:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18689.961831 secs 2024-07-26T16:17:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:26 smithi144 ceph-mon[32129]: pgmap v10354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:17:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18689.961831 secs 2024-07-26T16:17:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:26 smithi160 ceph-mon[36715]: pgmap v10354: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:17:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18689.961831 secs 2024-07-26T16:17:28.688 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:28.687+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:17:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:28.688+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:17:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:28.688+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:17:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:28.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:17:28.689 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:28.688+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:28 smithi144 ceph-mon[32129]: pgmap v10355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:17:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:28 smithi160 ceph-mon[36715]: pgmap v10355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:17:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:28 smithi183 ceph-mon[37273]: pgmap v10355: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:17:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:31 smithi183 ceph-mon[37273]: pgmap v10356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:17:31.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:31 smithi144 ceph-mon[32129]: pgmap v10356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:17:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:31 smithi160 ceph-mon[36715]: pgmap v10356: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:17:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18694.961956 secs 2024-07-26T16:17:32.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18694.961956 secs 2024-07-26T16:17:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18694.961956 secs 2024-07-26T16:17:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:33 smithi183 ceph-mon[37273]: pgmap v10357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:17:33.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:33 smithi144 ceph-mon[32129]: pgmap v10357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:17:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:33 smithi160 ceph-mon[36715]: pgmap v10357: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:17:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:35 smithi144 ceph-mon[32129]: pgmap v10358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:35 smithi160 ceph-mon[36715]: pgmap v10358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:35 smithi183 ceph-mon[37273]: pgmap v10358: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:37 smithi144 ceph-mon[32129]: pgmap v10359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18699.962072 secs 2024-07-26T16:17:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:37 smithi160 ceph-mon[36715]: pgmap v10359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18699.962072 secs 2024-07-26T16:17:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:37 smithi183 ceph-mon[37273]: pgmap v10359: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18699.962072 secs 2024-07-26T16:17:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:39 smithi144 ceph-mon[32129]: pgmap v10360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:39 smithi160 ceph-mon[36715]: pgmap v10360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:39 smithi183 ceph-mon[37273]: pgmap v10360: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:41 smithi144 ceph-mon[32129]: pgmap v10361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:41 smithi160 ceph-mon[36715]: pgmap v10361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:41 smithi183 ceph-mon[37273]: pgmap v10361: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18704.962198 secs 2024-07-26T16:17:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18704.962198 secs 2024-07-26T16:17:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18704.962198 secs 2024-07-26T16:17:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:43 smithi144 ceph-mon[32129]: pgmap v10362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:43 smithi160 ceph-mon[36715]: pgmap v10362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:43 smithi183 ceph-mon[37273]: pgmap v10362: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:43.705 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:43.704+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:17:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:43.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:17:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:43.704+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:17:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:43.705+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:17:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:43.705+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:45 smithi144 ceph-mon[32129]: pgmap v10363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:45 smithi160 ceph-mon[36715]: pgmap v10363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:45 smithi183 ceph-mon[37273]: pgmap v10363: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:46 smithi160 ceph-mon[36715]: pgmap v10364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:46 smithi183 ceph-mon[37273]: pgmap v10364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:46.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:46 smithi144 ceph-mon[32129]: pgmap v10364: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18709.962338 secs 2024-07-26T16:17:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18709.962338 secs 2024-07-26T16:17:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18709.962338 secs 2024-07-26T16:17:48.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:48 smithi183 ceph-mon[37273]: pgmap v10365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:48 smithi160 ceph-mon[36715]: pgmap v10365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:48 smithi144 ceph-mon[32129]: pgmap v10365: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:49.401+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:17:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:49.401+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:17:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:49.402+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:17:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:49.402+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:17:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:50 smithi183 ceph-mon[37273]: pgmap v10366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:50 smithi144 ceph-mon[32129]: pgmap v10366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:51.262 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:50 smithi160 ceph-mon[36715]: pgmap v10366: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18714.962466 secs 2024-07-26T16:17:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18714.962466 secs 2024-07-26T16:17:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18714.962466 secs 2024-07-26T16:17:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:52 smithi144 ceph-mon[32129]: pgmap v10367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:52 smithi160 ceph-mon[36715]: pgmap v10367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:52 smithi183 ceph-mon[37273]: pgmap v10367: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:17:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:54 smithi144 ceph-mon[32129]: pgmap v10368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:54 smithi160 ceph-mon[36715]: pgmap v10368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:54 smithi183 ceph-mon[37273]: pgmap v10368: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:56 smithi144 ceph-mon[32129]: pgmap v10369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18719.962587 secs 2024-07-26T16:17:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:56 smithi160 ceph-mon[36715]: pgmap v10369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18719.962587 secs 2024-07-26T16:17:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:56 smithi183 ceph-mon[37273]: pgmap v10369: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:17:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18719.962587 secs 2024-07-26T16:17:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:58.720+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:17:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:58.721+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:17:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:58.721+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:17:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:58.721+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:17:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:17:58.721+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:59.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:17:58 smithi144 ceph-mon[32129]: pgmap v10370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:17:58 smithi160 ceph-mon[36715]: pgmap v10370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:17:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:17:58 smithi183 ceph-mon[37273]: pgmap v10370: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:00 smithi144 ceph-mon[32129]: pgmap v10371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:00 smithi160 ceph-mon[36715]: pgmap v10371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:00 smithi183 ceph-mon[37273]: pgmap v10371: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18724.962718 secs 2024-07-26T16:18:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18724.962718 secs 2024-07-26T16:18:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18724.962718 secs 2024-07-26T16:18:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:02 smithi144 ceph-mon[32129]: pgmap v10372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:02 smithi160 ceph-mon[36715]: pgmap v10372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:02 smithi183 ceph-mon[37273]: pgmap v10372: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:04 smithi144 ceph-mon[32129]: pgmap v10373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:04 smithi160 ceph-mon[36715]: pgmap v10373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:04 smithi183 ceph-mon[37273]: pgmap v10373: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:06 smithi144 ceph-mon[32129]: pgmap v10374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18729.962845 secs 2024-07-26T16:18:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:06 smithi160 ceph-mon[36715]: pgmap v10374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18729.962845 secs 2024-07-26T16:18:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:06 smithi183 ceph-mon[37273]: pgmap v10374: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18729.962845 secs 2024-07-26T16:18:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:08 smithi144 ceph-mon[32129]: pgmap v10375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:08 smithi160 ceph-mon[36715]: pgmap v10375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:09.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:08 smithi183 ceph-mon[37273]: pgmap v10375: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:10 smithi144 ceph-mon[32129]: pgmap v10376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:10 smithi160 ceph-mon[36715]: pgmap v10376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:10 smithi183 ceph-mon[37273]: pgmap v10376: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18734.962986 secs 2024-07-26T16:18:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18734.962986 secs 2024-07-26T16:18:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18734.962986 secs 2024-07-26T16:18:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:12 smithi144 ceph-mon[32129]: pgmap v10377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:12 smithi160 ceph-mon[36715]: pgmap v10377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:12 smithi183 ceph-mon[37273]: pgmap v10377: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:13.737+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:18:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:13.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:18:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:13.738+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:18:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:13.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:18:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:13.738+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:14 smithi160 ceph-mon[36715]: pgmap v10378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:14 smithi183 ceph-mon[37273]: pgmap v10378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:14 smithi144 ceph-mon[32129]: pgmap v10378: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:16 smithi160 ceph-mon[36715]: pgmap v10379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18739.963077 secs 2024-07-26T16:18:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:16 smithi183 ceph-mon[37273]: pgmap v10379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18739.963077 secs 2024-07-26T16:18:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:16 smithi144 ceph-mon[32129]: pgmap v10379: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18739.963077 secs 2024-07-26T16:18:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:18 smithi160 ceph-mon[36715]: pgmap v10380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:18 smithi183 ceph-mon[37273]: pgmap v10380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:18 smithi144 ceph-mon[32129]: pgmap v10380: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:20 smithi160 ceph-mon[36715]: pgmap v10381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:20 smithi183 ceph-mon[37273]: pgmap v10381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:20 smithi144 ceph-mon[32129]: pgmap v10381: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18744.963152 secs 2024-07-26T16:18:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18744.963152 secs 2024-07-26T16:18:22.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18744.963152 secs 2024-07-26T16:18:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:23 smithi160 ceph-mon[36715]: pgmap v10382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:23 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:23 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:23 smithi183 ceph-mon[37273]: pgmap v10382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:23 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:23 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:23.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:23 smithi144 ceph-mon[32129]: pgmap v10382: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:23 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:23 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:24.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:18:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:25 smithi160 ceph-mon[36715]: pgmap v10383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:25 smithi160 ceph-mon[36715]: pgmap v10384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:25.310 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:25 smithi144 ceph-mon[32129]: pgmap v10383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:25.310 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:25 smithi144 ceph-mon[32129]: pgmap v10384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:25 smithi183 ceph-mon[37273]: pgmap v10383: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:25 smithi183 ceph-mon[37273]: pgmap v10384: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:27 smithi160 ceph-mon[36715]: pgmap v10385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18749.963286 secs 2024-07-26T16:18:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:27 smithi183 ceph-mon[37273]: pgmap v10385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18749.963286 secs 2024-07-26T16:18:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:27 smithi144 ceph-mon[32129]: pgmap v10385: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18749.963286 secs 2024-07-26T16:18:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:28.754+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:18:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:28.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:18:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:28.755+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:18:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:28.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:18:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:28.755+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:29 smithi160 ceph-mon[36715]: pgmap v10386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:29 smithi183 ceph-mon[37273]: pgmap v10386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:29.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:29 smithi144 ceph-mon[32129]: pgmap v10386: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:31 smithi160 ceph-mon[36715]: pgmap v10387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:31 smithi183 ceph-mon[37273]: pgmap v10387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:31 smithi144 ceph-mon[32129]: pgmap v10387: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18754.963409 secs 2024-07-26T16:18:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18754.963409 secs 2024-07-26T16:18:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18754.963409 secs 2024-07-26T16:18:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:33 smithi160 ceph-mon[36715]: pgmap v10388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:33 smithi183 ceph-mon[37273]: pgmap v10388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:33 smithi144 ceph-mon[32129]: pgmap v10388: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:18:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:35 smithi183 ceph-mon[37273]: pgmap v10389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:18:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:35 smithi144 ceph-mon[32129]: pgmap v10389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:18:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:35 smithi160 ceph-mon[36715]: pgmap v10389: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:18:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:37 smithi183 ceph-mon[37273]: pgmap v10390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18759.963527 secs 2024-07-26T16:18:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:37 smithi144 ceph-mon[32129]: pgmap v10390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18759.963527 secs 2024-07-26T16:18:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:37 smithi160 ceph-mon[36715]: pgmap v10390: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18759.963527 secs 2024-07-26T16:18:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:39 smithi183 ceph-mon[37273]: pgmap v10391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:39 smithi144 ceph-mon[32129]: pgmap v10391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:39 smithi160 ceph-mon[36715]: pgmap v10391: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:41 smithi183 ceph-mon[37273]: pgmap v10392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:41 smithi144 ceph-mon[32129]: pgmap v10392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:41 smithi160 ceph-mon[36715]: pgmap v10392: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18764.963639 secs 2024-07-26T16:18:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18764.963639 secs 2024-07-26T16:18:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18764.963639 secs 2024-07-26T16:18:43.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:43 smithi183 ceph-mon[37273]: pgmap v10393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:43.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:43 smithi144 ceph-mon[32129]: pgmap v10393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:43 smithi160 ceph-mon[36715]: pgmap v10393: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:43.770+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:18:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:43.771+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:18:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:43.771+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:18:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:43.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:18:43.773 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:43.771+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:45 smithi183 ceph-mon[37273]: pgmap v10394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:45.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:45 smithi144 ceph-mon[32129]: pgmap v10394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:45 smithi160 ceph-mon[36715]: pgmap v10394: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:47 smithi183 ceph-mon[37273]: pgmap v10395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18769.963766 secs 2024-07-26T16:18:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:47 smithi144 ceph-mon[32129]: pgmap v10395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18769.963766 secs 2024-07-26T16:18:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:47 smithi160 ceph-mon[36715]: pgmap v10395: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18769.963766 secs 2024-07-26T16:18:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:49 smithi183 ceph-mon[37273]: pgmap v10396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:49 smithi144 ceph-mon[32129]: pgmap v10396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:49 smithi160 ceph-mon[36715]: pgmap v10396: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:51 smithi183 ceph-mon[37273]: pgmap v10397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:51 smithi144 ceph-mon[32129]: pgmap v10397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:51 smithi160 ceph-mon[36715]: pgmap v10397: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18774.963887 secs 2024-07-26T16:18:52.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18774.963887 secs 2024-07-26T16:18:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18774.963887 secs 2024-07-26T16:18:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:53 smithi183 ceph-mon[37273]: pgmap v10398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:53 smithi144 ceph-mon[32129]: pgmap v10398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:53 smithi160 ceph-mon[36715]: pgmap v10398: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:18:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:55 smithi144 ceph-mon[32129]: pgmap v10399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:55 smithi160 ceph-mon[36715]: pgmap v10399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:55 smithi183 ceph-mon[37273]: pgmap v10399: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:57 smithi144 ceph-mon[32129]: pgmap v10400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18779.964010 secs 2024-07-26T16:18:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:57 smithi160 ceph-mon[36715]: pgmap v10400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18779.964010 secs 2024-07-26T16:18:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:57 smithi183 ceph-mon[37273]: pgmap v10400: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:18:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18779.964010 secs 2024-07-26T16:18:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:58.787+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:18:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:58.788+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:18:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:58.788+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:18:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:58.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:18:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:18:58.788+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:18:59 smithi144 ceph-mon[32129]: pgmap v10401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:18:59 smithi160 ceph-mon[36715]: pgmap v10401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:18:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:18:59 smithi183 ceph-mon[37273]: pgmap v10401: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:01.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:01 smithi144 ceph-mon[32129]: pgmap v10402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:01 smithi160 ceph-mon[36715]: pgmap v10402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:01 smithi183 ceph-mon[37273]: pgmap v10402: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18784.964125 secs 2024-07-26T16:19:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18784.964125 secs 2024-07-26T16:19:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18784.964125 secs 2024-07-26T16:19:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:03 smithi144 ceph-mon[32129]: pgmap v10403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:03 smithi160 ceph-mon[36715]: pgmap v10403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:03 smithi183 ceph-mon[37273]: pgmap v10403: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:05 smithi144 ceph-mon[32129]: pgmap v10404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:05 smithi160 ceph-mon[36715]: pgmap v10404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:05 smithi183 ceph-mon[37273]: pgmap v10404: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:07 smithi144 ceph-mon[32129]: pgmap v10405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18789.964266 secs 2024-07-26T16:19:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:07 smithi160 ceph-mon[36715]: pgmap v10405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18789.964266 secs 2024-07-26T16:19:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:07 smithi183 ceph-mon[37273]: pgmap v10405: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18789.964266 secs 2024-07-26T16:19:09.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:09 smithi144 ceph-mon[32129]: pgmap v10406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:09 smithi160 ceph-mon[36715]: pgmap v10406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:09 smithi183 ceph-mon[37273]: pgmap v10406: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:10 smithi160 ceph-mon[36715]: pgmap v10407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:10 smithi183 ceph-mon[37273]: pgmap v10407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:10 smithi144 ceph-mon[32129]: pgmap v10407: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:11.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18794.964396 secs 2024-07-26T16:19:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18794.964396 secs 2024-07-26T16:19:11.902 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18794.964396 secs 2024-07-26T16:19:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:12 smithi160 ceph-mon[36715]: pgmap v10408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:12 smithi183 ceph-mon[37273]: pgmap v10408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:12 smithi144 ceph-mon[32129]: pgmap v10408: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:13.805 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:13.803+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:19:13.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:13.804+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:19:13.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:13.804+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:19:13.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:13.804+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:19:13.806 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:13.804+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:15.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:14 smithi160 ceph-mon[36715]: pgmap v10409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:15.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:14 smithi183 ceph-mon[37273]: pgmap v10409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:15.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:14 smithi144 ceph-mon[32129]: pgmap v10409: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:16 smithi160 ceph-mon[36715]: pgmap v10410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18799.964503 secs 2024-07-26T16:19:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:16 smithi183 ceph-mon[37273]: pgmap v10410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18799.964503 secs 2024-07-26T16:19:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:16 smithi144 ceph-mon[32129]: pgmap v10410: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18799.964503 secs 2024-07-26T16:19:19.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:18 smithi160 ceph-mon[36715]: pgmap v10411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:19.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:18 smithi183 ceph-mon[37273]: pgmap v10411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:19.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:18 smithi144 ceph-mon[32129]: pgmap v10411: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:19.418+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:19:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:19.418+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:19:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:19.419+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:19:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:19.419+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:19:21.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:20 smithi160 ceph-mon[36715]: pgmap v10412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:21.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:20 smithi183 ceph-mon[37273]: pgmap v10412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:21.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:20 smithi144 ceph-mon[32129]: pgmap v10412: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18804.964625 secs 2024-07-26T16:19:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18804.964625 secs 2024-07-26T16:19:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18804.964625 secs 2024-07-26T16:19:23.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:22 smithi160 ceph-mon[36715]: pgmap v10413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:23.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:22 smithi183 ceph-mon[37273]: pgmap v10413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:23.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:22 smithi144 ceph-mon[32129]: pgmap v10413: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:25.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:24 smithi183 ceph-mon[37273]: pgmap v10414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:25.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:24 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:25.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.034 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:24 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:24 smithi160 ceph-mon[36715]: pgmap v10414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:25.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:24 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:25.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:24 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.198 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:24 smithi144 ceph-mon[32129]: pgmap v10414: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:25.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:24 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:25.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:25.199 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:24 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:26 smithi144 ceph-mon[32129]: pgmap v10415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18809.964712 secs 2024-07-26T16:19:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:26 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:26 smithi160 ceph-mon[36715]: pgmap v10415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18809.964712 secs 2024-07-26T16:19:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:26 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:26 smithi183 ceph-mon[37273]: pgmap v10415: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18809.964712 secs 2024-07-26T16:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:26 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:28.821 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:28.819+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:19:28.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:28.820+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:19:28.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:28.820+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:19:28.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:28.820+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:19:28.822 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:28.820+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:28 smithi183 ceph-mon[37273]: pgmap v10416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:29.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:28 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:28 smithi144 ceph-mon[32129]: pgmap v10416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:29.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:28 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:28 smithi160 ceph-mon[36715]: pgmap v10416: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:28 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:29 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:29 smithi183 ceph-mon[37273]: pgmap v10417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:29 smithi183 ceph-mon[37273]: pgmap v10418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:19:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:29 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:29 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:29 smithi144 ceph-mon[32129]: pgmap v10417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:29 smithi144 ceph-mon[32129]: pgmap v10418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:19:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:29 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:29 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:29 smithi160 ceph-mon[36715]: pgmap v10417: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:29 smithi160 ceph-mon[36715]: pgmap v10418: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:19:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:29 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:19:32.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:31 smithi144 ceph-mon[32129]: pgmap v10419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18814.964830 secs 2024-07-26T16:19:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:31 smithi160 ceph-mon[36715]: pgmap v10419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18814.964830 secs 2024-07-26T16:19:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:31 smithi183 ceph-mon[37273]: pgmap v10419: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18814.964830 secs 2024-07-26T16:19:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:33 smithi144 ceph-mon[32129]: pgmap v10420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:19:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:33 smithi160 ceph-mon[36715]: pgmap v10420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:19:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:33 smithi183 ceph-mon[37273]: pgmap v10420: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:19:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:35 smithi144 ceph-mon[32129]: pgmap v10421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:19:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:35 smithi160 ceph-mon[36715]: pgmap v10421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:19:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:35 smithi183 ceph-mon[37273]: pgmap v10421: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:19:37.222 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18819.964928 secs 2024-07-26T16:19:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18819.964928 secs 2024-07-26T16:19:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18819.964928 secs 2024-07-26T16:19:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:37 smithi160 ceph-mon[36715]: pgmap v10422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:37 smithi183 ceph-mon[37273]: pgmap v10422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:37 smithi144 ceph-mon[32129]: pgmap v10422: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:19:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:39 smithi160 ceph-mon[36715]: pgmap v10423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:19:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:39 smithi183 ceph-mon[37273]: pgmap v10423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:19:40.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:39 smithi144 ceph-mon[32129]: pgmap v10423: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:19:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:41 smithi160 ceph-mon[36715]: pgmap v10424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18824.965031 secs 2024-07-26T16:19:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:41 smithi183 ceph-mon[37273]: pgmap v10424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18824.965031 secs 2024-07-26T16:19:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:41 smithi144 ceph-mon[32129]: pgmap v10424: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18824.965031 secs 2024-07-26T16:19:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:43.829+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:19:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:43.830+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:19:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:43.830+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:19:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:43.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:19:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:43.830+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:43 smithi160 ceph-mon[36715]: pgmap v10425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:44.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:43 smithi183 ceph-mon[37273]: pgmap v10425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:44.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:43 smithi144 ceph-mon[32129]: pgmap v10425: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:45 smithi160 ceph-mon[36715]: pgmap v10426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:45 smithi183 ceph-mon[37273]: pgmap v10426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:45 smithi144 ceph-mon[32129]: pgmap v10426: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18829.965183 secs 2024-07-26T16:19:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18829.965183 secs 2024-07-26T16:19:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18829.965183 secs 2024-07-26T16:19:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:48 smithi160 ceph-mon[36715]: pgmap v10427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:48.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:48 smithi183 ceph-mon[37273]: pgmap v10427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:48.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:48 smithi144 ceph-mon[32129]: pgmap v10427: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:50 smithi160 ceph-mon[36715]: pgmap v10428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:50 smithi183 ceph-mon[37273]: pgmap v10428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:50 smithi144 ceph-mon[32129]: pgmap v10428: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:52 smithi160 ceph-mon[36715]: pgmap v10429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:52.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18834.965309 secs 2024-07-26T16:19:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:52 smithi183 ceph-mon[37273]: pgmap v10429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18834.965309 secs 2024-07-26T16:19:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:52 smithi144 ceph-mon[32129]: pgmap v10429: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18834.965309 secs 2024-07-26T16:19:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:54 smithi160 ceph-mon[36715]: pgmap v10430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:54 smithi183 ceph-mon[37273]: pgmap v10430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:54 smithi144 ceph-mon[32129]: pgmap v10430: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:19:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:56 smithi144 ceph-mon[32129]: pgmap v10431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:56 smithi160 ceph-mon[36715]: pgmap v10431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:56 smithi183 ceph-mon[37273]: pgmap v10431: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:57.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18839.965433 secs 2024-07-26T16:19:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18839.965433 secs 2024-07-26T16:19:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18839.965433 secs 2024-07-26T16:19:58.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:58 smithi144 ceph-mon[32129]: pgmap v10432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:58 smithi160 ceph-mon[36715]: pgmap v10432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:58 smithi183 ceph-mon[37273]: pgmap v10432: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:19:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:58.845+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:19:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:58.846+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:19:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:58.846+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:19:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:58.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:19:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:19:58.846+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:59.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:19:59 smithi144 ceph-mon[32129]: pgmap v10433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:19:59 smithi160 ceph-mon[36715]: pgmap v10433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:19:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:19:59 smithi183 ceph-mon[37273]: pgmap v10433: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:20:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:20:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:20:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:01 smithi160 ceph-mon[36715]: pgmap v10434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:01 smithi183 ceph-mon[37273]: pgmap v10434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:01.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:01 smithi144 ceph-mon[32129]: pgmap v10434: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18844.965556 secs 2024-07-26T16:20:02.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18844.965556 secs 2024-07-26T16:20:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18844.965556 secs 2024-07-26T16:20:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:03 smithi160 ceph-mon[36715]: pgmap v10435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:03 smithi183 ceph-mon[37273]: pgmap v10435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:03 smithi144 ceph-mon[32129]: pgmap v10435: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:05 smithi144 ceph-mon[32129]: pgmap v10436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:05 smithi160 ceph-mon[36715]: pgmap v10436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:05 smithi183 ceph-mon[37273]: pgmap v10436: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18849.965679 secs 2024-07-26T16:20:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18849.965679 secs 2024-07-26T16:20:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18849.965679 secs 2024-07-26T16:20:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:07 smithi160 ceph-mon[36715]: pgmap v10437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:07 smithi183 ceph-mon[37273]: pgmap v10437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:07 smithi144 ceph-mon[32129]: pgmap v10437: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:09 smithi160 ceph-mon[36715]: pgmap v10438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:09 smithi183 ceph-mon[37273]: pgmap v10438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:09 smithi144 ceph-mon[32129]: pgmap v10438: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:11 smithi160 ceph-mon[36715]: pgmap v10439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:12.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18854.965744 secs 2024-07-26T16:20:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:11 smithi183 ceph-mon[37273]: pgmap v10439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18854.965744 secs 2024-07-26T16:20:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:11 smithi144 ceph-mon[32129]: pgmap v10439: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18854.965744 secs 2024-07-26T16:20:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:13.854+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:20:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:13.855+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:20:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:13.855+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:20:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:13.855+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:20:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:13.855+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:14.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:13 smithi144 ceph-mon[32129]: pgmap v10440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:13 smithi160 ceph-mon[36715]: pgmap v10440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:13 smithi183 ceph-mon[37273]: pgmap v10440: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:16 smithi160 ceph-mon[36715]: pgmap v10441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:16 smithi183 ceph-mon[37273]: pgmap v10441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:16 smithi144 ceph-mon[32129]: pgmap v10441: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18859.965866 secs 2024-07-26T16:20:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18859.965866 secs 2024-07-26T16:20:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18859.965866 secs 2024-07-26T16:20:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:18 smithi160 ceph-mon[36715]: pgmap v10442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:18 smithi183 ceph-mon[37273]: pgmap v10442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:18.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:18 smithi144 ceph-mon[32129]: pgmap v10442: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:20 smithi144 ceph-mon[32129]: pgmap v10443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:20 smithi160 ceph-mon[36715]: pgmap v10443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:20 smithi183 ceph-mon[37273]: pgmap v10443: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:22 smithi160 ceph-mon[36715]: pgmap v10444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18864.965984 secs 2024-07-26T16:20:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:22 smithi183 ceph-mon[37273]: pgmap v10444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18864.965984 secs 2024-07-26T16:20:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:22 smithi144 ceph-mon[32129]: pgmap v10444: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18864.965984 secs 2024-07-26T16:20:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:24 smithi160 ceph-mon[36715]: pgmap v10445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:24 smithi183 ceph-mon[37273]: pgmap v10445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:24.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:24 smithi144 ceph-mon[32129]: pgmap v10445: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:26 smithi160 ceph-mon[36715]: pgmap v10446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:26 smithi183 ceph-mon[37273]: pgmap v10446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:26 smithi144 ceph-mon[32129]: pgmap v10446: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18869.966104 secs 2024-07-26T16:20:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18869.966104 secs 2024-07-26T16:20:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18869.966104 secs 2024-07-26T16:20:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:28 smithi183 ceph-mon[37273]: pgmap v10447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:28.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:28 smithi144 ceph-mon[32129]: pgmap v10447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:28 smithi160 ceph-mon[36715]: pgmap v10447: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:28.867+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:20:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:28.867+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:20:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:28.867+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:20:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:28.867+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:20:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:28.868+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:30 smithi183 ceph-mon[37273]: pgmap v10448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:30 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:30.466 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:30 smithi144 ceph-mon[32129]: pgmap v10448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:30.467 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:30 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:30 smithi160 ceph-mon[36715]: pgmap v10448: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:30 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:31 smithi183 ceph-mon[37273]: pgmap v10449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:20:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:31 smithi183 ceph-mon[37273]: pgmap v10450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:20:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:31 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:20:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:31 smithi144 ceph-mon[32129]: pgmap v10449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:20:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:31 smithi144 ceph-mon[32129]: pgmap v10450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:20:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:31 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:20:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:31 smithi160 ceph-mon[36715]: pgmap v10449: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:20:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:31 smithi160 ceph-mon[36715]: pgmap v10450: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:20:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:31 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:20:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18874.966197 secs 2024-07-26T16:20:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18874.966197 secs 2024-07-26T16:20:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18874.966197 secs 2024-07-26T16:20:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:33 smithi183 ceph-mon[37273]: pgmap v10451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:20:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:33 smithi144 ceph-mon[32129]: pgmap v10451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:20:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:33 smithi160 ceph-mon[36715]: pgmap v10451: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:20:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:35 smithi144 ceph-mon[32129]: pgmap v10452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:20:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:35 smithi160 ceph-mon[36715]: pgmap v10452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:20:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:35 smithi183 ceph-mon[37273]: pgmap v10452: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:20:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18879.966330 secs 2024-07-26T16:20:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:37 smithi144 ceph-mon[32129]: pgmap v10453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:20:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18879.966330 secs 2024-07-26T16:20:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:37 smithi160 ceph-mon[36715]: pgmap v10453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:20:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18879.966330 secs 2024-07-26T16:20:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:37 smithi183 ceph-mon[37273]: pgmap v10453: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:20:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:39 smithi144 ceph-mon[32129]: pgmap v10454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:20:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:39 smithi160 ceph-mon[36715]: pgmap v10454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:20:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:39 smithi183 ceph-mon[37273]: pgmap v10454: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:20:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:41 smithi144 ceph-mon[32129]: pgmap v10455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:20:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:41 smithi160 ceph-mon[36715]: pgmap v10455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:20:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:41 smithi183 ceph-mon[37273]: pgmap v10455: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:20:42.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18884.966434 secs 2024-07-26T16:20:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18884.966434 secs 2024-07-26T16:20:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18884.966434 secs 2024-07-26T16:20:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:43 smithi144 ceph-mon[32129]: pgmap v10456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:43 smithi160 ceph-mon[36715]: pgmap v10456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:43 smithi183 ceph-mon[37273]: pgmap v10456: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:43.882+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:20:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:43.882+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:20:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:43.883+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:20:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:43.883+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:20:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:43.883+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:45 smithi144 ceph-mon[32129]: pgmap v10457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:45 smithi160 ceph-mon[36715]: pgmap v10457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:45 smithi183 ceph-mon[37273]: pgmap v10457: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T16:20:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5b20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f74a80319b0 tx=0x7f74a805a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5b20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f74a80319b0 tx=0x7f74a805a000 comp rx=0 tx=0).stop 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74af400640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5b20 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.144:6800/2154879590, fault. 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 msgr2=0x7f74900b5b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b51d0 0x7f74900b5b20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74b4a00640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.162+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:20:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:46.164+0000 7f74af400640 1 --2- 172.21.15.144:0/108036915 >> [v2:172.21.15.144:6800/2154879590,v1:172.21.15.144:6801/2154879590] conn(0x7f74900b4dc0 0x7f74900b55e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005e30 tx=0x7f74a8059050 comp rx=0 tx=0).ready entity=mgr.24109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:20:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18889.966544 secs 2024-07-26T16:20:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:47 smithi144 ceph-mon[32129]: pgmap v10458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18889.966544 secs 2024-07-26T16:20:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:47 smithi160 ceph-mon[36715]: pgmap v10458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18889.966544 secs 2024-07-26T16:20:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:47 smithi183 ceph-mon[37273]: pgmap v10458: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:49.433+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:20:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:49.433+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:20:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:49.433+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:20:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:49.433+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:20:49.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:49 smithi144 ceph-mon[32129]: pgmap v10459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:49 smithi160 ceph-mon[36715]: pgmap v10459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:49 smithi183 ceph-mon[37273]: pgmap v10459: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:50.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:50 smithi183 ceph-mon[37273]: pgmap v10460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:50 smithi160 ceph-mon[36715]: pgmap v10460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:50 smithi144 ceph-mon[32129]: pgmap v10460: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18894.966666 secs 2024-07-26T16:20:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18894.966666 secs 2024-07-26T16:20:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18894.966666 secs 2024-07-26T16:20:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:52 smithi160 ceph-mon[36715]: pgmap v10461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:52.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:52 smithi183 ceph-mon[37273]: pgmap v10461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:52.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:52 smithi144 ceph-mon[32129]: pgmap v10461: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:20:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:55 smithi144 ceph-mon[32129]: pgmap v10462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:55 smithi160 ceph-mon[36715]: pgmap v10462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:55 smithi183 ceph-mon[37273]: pgmap v10462: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18899.966772 secs 2024-07-26T16:20:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:57 smithi144 ceph-mon[32129]: pgmap v10463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18899.966772 secs 2024-07-26T16:20:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:57 smithi160 ceph-mon[36715]: pgmap v10463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18899.966772 secs 2024-07-26T16:20:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:57 smithi183 ceph-mon[37273]: pgmap v10463: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:20:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:58.884+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:20:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:58.885+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:20:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:58.885+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:20:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:58.885+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:20:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:20:58.885+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:20:59 smithi144 ceph-mon[32129]: pgmap v10464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:20:59 smithi160 ceph-mon[36715]: pgmap v10464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:20:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:20:59 smithi183 ceph-mon[37273]: pgmap v10464: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:01 smithi144 ceph-mon[32129]: pgmap v10465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:01 smithi160 ceph-mon[36715]: pgmap v10465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:01 smithi183 ceph-mon[37273]: pgmap v10465: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18904.966888 secs 2024-07-26T16:21:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18904.966888 secs 2024-07-26T16:21:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18904.966888 secs 2024-07-26T16:21:03.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:03 smithi144 ceph-mon[32129]: pgmap v10466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:03 smithi160 ceph-mon[36715]: pgmap v10466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:03 smithi183 ceph-mon[37273]: pgmap v10466: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:05 smithi144 ceph-mon[32129]: pgmap v10467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:05 smithi160 ceph-mon[36715]: pgmap v10467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:05 smithi183 ceph-mon[37273]: pgmap v10467: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18909.967023 secs 2024-07-26T16:21:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:07 smithi144 ceph-mon[32129]: pgmap v10468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18909.967023 secs 2024-07-26T16:21:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:07 smithi160 ceph-mon[36715]: pgmap v10468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18909.967023 secs 2024-07-26T16:21:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:07 smithi183 ceph-mon[37273]: pgmap v10468: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:08 smithi160 ceph-mon[36715]: pgmap v10469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:08 smithi183 ceph-mon[37273]: pgmap v10469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:08 smithi144 ceph-mon[32129]: pgmap v10469: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:11 smithi144 ceph-mon[32129]: pgmap v10470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:11 smithi160 ceph-mon[36715]: pgmap v10470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:11 smithi183 ceph-mon[37273]: pgmap v10470: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:12.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18914.967161 secs 2024-07-26T16:21:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18914.967161 secs 2024-07-26T16:21:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18914.967161 secs 2024-07-26T16:21:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:13 smithi144 ceph-mon[32129]: pgmap v10471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:13 smithi160 ceph-mon[36715]: pgmap v10471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:13 smithi183 ceph-mon[37273]: pgmap v10471: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:13.901+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:21:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:13.902+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:21:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:13.902+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:21:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:13.902+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:21:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:13.902+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:15 smithi144 ceph-mon[32129]: pgmap v10472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:15 smithi160 ceph-mon[36715]: pgmap v10472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:15 smithi183 ceph-mon[37273]: pgmap v10472: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18919.967278 secs 2024-07-26T16:21:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:17 smithi144 ceph-mon[32129]: pgmap v10473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18919.967278 secs 2024-07-26T16:21:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:17 smithi160 ceph-mon[36715]: pgmap v10473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18919.967278 secs 2024-07-26T16:21:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:17 smithi183 ceph-mon[37273]: pgmap v10473: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:19.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:19 smithi144 ceph-mon[32129]: pgmap v10474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:19 smithi160 ceph-mon[36715]: pgmap v10474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:19 smithi183 ceph-mon[37273]: pgmap v10474: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:20.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:20 smithi183 ceph-mon[37273]: pgmap v10475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:20 smithi160 ceph-mon[36715]: pgmap v10475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:20 smithi144 ceph-mon[32129]: pgmap v10475: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:21.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18924.967397 secs 2024-07-26T16:21:21.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18924.967397 secs 2024-07-26T16:21:21.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18924.967397 secs 2024-07-26T16:21:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:22 smithi160 ceph-mon[36715]: pgmap v10476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:22 smithi183 ceph-mon[37273]: pgmap v10476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:22.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:22 smithi144 ceph-mon[32129]: pgmap v10476: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:25 smithi144 ceph-mon[32129]: pgmap v10477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:25 smithi160 ceph-mon[36715]: pgmap v10477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:25 smithi183 ceph-mon[37273]: pgmap v10477: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18929.967486 secs 2024-07-26T16:21:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:27 smithi144 ceph-mon[32129]: pgmap v10478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18929.967486 secs 2024-07-26T16:21:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:27 smithi160 ceph-mon[36715]: pgmap v10478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18929.967486 secs 2024-07-26T16:21:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:27 smithi183 ceph-mon[37273]: pgmap v10478: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:28.918+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:21:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:28.919+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:21:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:28.919+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:21:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:28.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:21:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:28.919+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:29 smithi144 ceph-mon[32129]: pgmap v10479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:29 smithi160 ceph-mon[36715]: pgmap v10479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:29 smithi183 ceph-mon[37273]: pgmap v10479: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:31 smithi144 ceph-mon[32129]: pgmap v10480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:31 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:31 smithi160 ceph-mon[36715]: pgmap v10480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:31 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:31 smithi183 ceph-mon[37273]: pgmap v10480: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:31 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18934.967601 secs 2024-07-26T16:21:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:21:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:21:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:32 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:21:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18934.967601 secs 2024-07-26T16:21:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:21:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:21:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:32 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:21:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18934.967601 secs 2024-07-26T16:21:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:21:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:21:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:32 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:21:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:33 smithi160 ceph-mon[36715]: pgmap v10481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:33.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:33 smithi160 ceph-mon[36715]: pgmap v10482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:33 smithi183 ceph-mon[37273]: pgmap v10481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:33 smithi183 ceph-mon[37273]: pgmap v10482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:33 smithi144 ceph-mon[32129]: pgmap v10481: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:33 smithi144 ceph-mon[32129]: pgmap v10482: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:34.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:34 smithi160 ceph-mon[36715]: pgmap v10483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:34.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:34 smithi183 ceph-mon[37273]: pgmap v10483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:34 smithi144 ceph-mon[32129]: pgmap v10483: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:36.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:36 smithi183 ceph-mon[37273]: pgmap v10484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:36.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:36 smithi144 ceph-mon[32129]: pgmap v10484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:36 smithi160 ceph-mon[36715]: pgmap v10484: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18939.967726 secs 2024-07-26T16:21:37.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18939.967726 secs 2024-07-26T16:21:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18939.967726 secs 2024-07-26T16:21:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:38 smithi183 ceph-mon[37273]: pgmap v10485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:38 smithi144 ceph-mon[32129]: pgmap v10485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:38 smithi160 ceph-mon[36715]: pgmap v10485: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:40.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:40 smithi183 ceph-mon[37273]: pgmap v10486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:40.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:40 smithi144 ceph-mon[32129]: pgmap v10486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:40.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:40 smithi160 ceph-mon[36715]: pgmap v10486: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18944.967810 secs 2024-07-26T16:21:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:42 smithi183 ceph-mon[37273]: pgmap v10487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:21:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18944.967810 secs 2024-07-26T16:21:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:42 smithi144 ceph-mon[32129]: pgmap v10487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:21:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18944.967810 secs 2024-07-26T16:21:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:42 smithi160 ceph-mon[36715]: pgmap v10487: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:21:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:43.935+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:21:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:43.936+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:21:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:43.936+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:21:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:43.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:21:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:43.936+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:44 smithi144 ceph-mon[32129]: pgmap v10488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:44.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:44 smithi160 ceph-mon[36715]: pgmap v10488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:44.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:44 smithi183 ceph-mon[37273]: pgmap v10488: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:46 smithi144 ceph-mon[32129]: pgmap v10489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:46.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:46 smithi160 ceph-mon[36715]: pgmap v10489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:46 smithi183 ceph-mon[37273]: pgmap v10489: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18949.967926 secs 2024-07-26T16:21:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18949.967926 secs 2024-07-26T16:21:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18949.967926 secs 2024-07-26T16:21:48.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:48 smithi144 ceph-mon[32129]: pgmap v10490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:48 smithi160 ceph-mon[36715]: pgmap v10490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:48 smithi183 ceph-mon[37273]: pgmap v10490: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:50 smithi160 ceph-mon[36715]: pgmap v10491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:50 smithi183 ceph-mon[37273]: pgmap v10491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:50 smithi144 ceph-mon[32129]: pgmap v10491: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18954.968063 secs 2024-07-26T16:21:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:52 smithi183 ceph-mon[37273]: pgmap v10492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18954.968063 secs 2024-07-26T16:21:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:52 smithi144 ceph-mon[32129]: pgmap v10492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18954.968063 secs 2024-07-26T16:21:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:52 smithi160 ceph-mon[36715]: pgmap v10492: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:21:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:54 smithi183 ceph-mon[37273]: pgmap v10493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:54 smithi144 ceph-mon[32129]: pgmap v10493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:54 smithi160 ceph-mon[36715]: pgmap v10493: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:56 smithi183 ceph-mon[37273]: pgmap v10494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:56 smithi144 ceph-mon[32129]: pgmap v10494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:56 smithi160 ceph-mon[36715]: pgmap v10494: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:21:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18959.968179 secs 2024-07-26T16:21:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18959.968179 secs 2024-07-26T16:21:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18959.968179 secs 2024-07-26T16:21:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:21:58 smithi144 ceph-mon[32129]: pgmap v10495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:21:58 smithi160 ceph-mon[36715]: pgmap v10495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:58.886 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:21:58 smithi183 ceph-mon[37273]: pgmap v10495: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:21:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:58.951+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:21:58.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:58.953+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:21:58.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:58.953+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:21:58.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:58.953+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:21:58.954 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:21:58.953+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:00.722 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:00 smithi144 ceph-mon[32129]: pgmap v10496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:00 smithi160 ceph-mon[36715]: pgmap v10496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:00 smithi183 ceph-mon[37273]: pgmap v10496: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18964.968318 secs 2024-07-26T16:22:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:02 smithi144 ceph-mon[32129]: pgmap v10497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18964.968318 secs 2024-07-26T16:22:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:02 smithi160 ceph-mon[36715]: pgmap v10497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18964.968318 secs 2024-07-26T16:22:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:02 smithi183 ceph-mon[37273]: pgmap v10497: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:04 smithi144 ceph-mon[32129]: pgmap v10498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:04 smithi160 ceph-mon[36715]: pgmap v10498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:04 smithi183 ceph-mon[37273]: pgmap v10498: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:06 smithi144 ceph-mon[32129]: pgmap v10499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:06 smithi160 ceph-mon[36715]: pgmap v10499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:06 smithi183 ceph-mon[37273]: pgmap v10499: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:07.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18969.968437 secs 2024-07-26T16:22:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18969.968437 secs 2024-07-26T16:22:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18969.968437 secs 2024-07-26T16:22:08.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:08 smithi144 ceph-mon[32129]: pgmap v10500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:08 smithi160 ceph-mon[36715]: pgmap v10500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:08 smithi183 ceph-mon[37273]: pgmap v10500: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:10 smithi144 ceph-mon[32129]: pgmap v10501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:10 smithi160 ceph-mon[36715]: pgmap v10501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:10 smithi183 ceph-mon[37273]: pgmap v10501: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:12.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18974.968556 secs 2024-07-26T16:22:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:12 smithi144 ceph-mon[32129]: pgmap v10502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18974.968556 secs 2024-07-26T16:22:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:12 smithi160 ceph-mon[36715]: pgmap v10502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18974.968556 secs 2024-07-26T16:22:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:12 smithi183 ceph-mon[37273]: pgmap v10502: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:13.968+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:22:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:13.970+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:22:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:13.970+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:22:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:13.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:22:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:13.970+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:14 smithi144 ceph-mon[32129]: pgmap v10503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:14 smithi160 ceph-mon[36715]: pgmap v10503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:14 smithi183 ceph-mon[37273]: pgmap v10503: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:16 smithi144 ceph-mon[32129]: pgmap v10504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:16 smithi160 ceph-mon[36715]: pgmap v10504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:16 smithi183 ceph-mon[37273]: pgmap v10504: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18979.968684 secs 2024-07-26T16:22:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18979.968684 secs 2024-07-26T16:22:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18979.968684 secs 2024-07-26T16:22:18.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:18 smithi144 ceph-mon[32129]: pgmap v10505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:18 smithi160 ceph-mon[36715]: pgmap v10505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:18 smithi183 ceph-mon[37273]: pgmap v10505: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:20 smithi160 ceph-mon[36715]: pgmap v10506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:20 smithi183 ceph-mon[37273]: pgmap v10506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:20 smithi144 ceph-mon[32129]: pgmap v10506: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18984.968733 secs 2024-07-26T16:22:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:22 smithi160 ceph-mon[36715]: pgmap v10507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18984.968733 secs 2024-07-26T16:22:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:22 smithi183 ceph-mon[37273]: pgmap v10507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18984.968733 secs 2024-07-26T16:22:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:22 smithi144 ceph-mon[32129]: pgmap v10507: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:24 smithi160 ceph-mon[36715]: pgmap v10508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:24 smithi183 ceph-mon[37273]: pgmap v10508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:24 smithi144 ceph-mon[32129]: pgmap v10508: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:26 smithi160 ceph-mon[36715]: pgmap v10509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18989.968887 secs 2024-07-26T16:22:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:26 smithi183 ceph-mon[37273]: pgmap v10509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18989.968887 secs 2024-07-26T16:22:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:26 smithi144 ceph-mon[32129]: pgmap v10509: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18989.968887 secs 2024-07-26T16:22:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:28 smithi160 ceph-mon[36715]: pgmap v10510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:28 smithi183 ceph-mon[37273]: pgmap v10510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:28 smithi144 ceph-mon[32129]: pgmap v10510: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:28.987 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:28.985+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:22:28.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:28.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:22:28.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:28.986+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:22:28.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:28.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:22:28.988 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:28.986+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:30 smithi160 ceph-mon[36715]: pgmap v10511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:22:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:30 smithi183 ceph-mon[37273]: pgmap v10511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:22:30.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:30 smithi144 ceph-mon[32129]: pgmap v10511: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:22:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18994.968979 secs 2024-07-26T16:22:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18994.968979 secs 2024-07-26T16:22:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18994.968979 secs 2024-07-26T16:22:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:32 smithi160 ceph-mon[36715]: pgmap v10512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:22:32.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:32 smithi144 ceph-mon[32129]: pgmap v10512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:32.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:22:32.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:32 smithi183 ceph-mon[37273]: pgmap v10512: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:22:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:33 smithi160 ceph-mon[36715]: pgmap v10513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:33.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:33 smithi160 ceph-mon[36715]: pgmap v10514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:33.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:22:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:33 smithi183 ceph-mon[37273]: pgmap v10513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:33 smithi183 ceph-mon[37273]: pgmap v10514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:33.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:22:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:33 smithi144 ceph-mon[32129]: pgmap v10513: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:33 smithi144 ceph-mon[32129]: pgmap v10514: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:33.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:22:35.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:35 smithi183 ceph-mon[37273]: pgmap v10515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:35.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:35 smithi144 ceph-mon[32129]: pgmap v10515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:35 smithi160 ceph-mon[36715]: pgmap v10515: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:22:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 18999.969097 secs 2024-07-26T16:22:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 18999.969097 secs 2024-07-26T16:22:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 18999.969097 secs 2024-07-26T16:22:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:37 smithi183 ceph-mon[37273]: pgmap v10516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:22:37.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:37 smithi144 ceph-mon[32129]: pgmap v10516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:22:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:37 smithi160 ceph-mon[36715]: pgmap v10516: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:22:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:39 smithi183 ceph-mon[37273]: pgmap v10517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:22:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:39 smithi144 ceph-mon[32129]: pgmap v10517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:22:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:39 smithi160 ceph-mon[36715]: pgmap v10517: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:22:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:41 smithi183 ceph-mon[37273]: pgmap v10518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:22:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19004.969189 secs 2024-07-26T16:22:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:41 smithi144 ceph-mon[32129]: pgmap v10518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:22:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19004.969189 secs 2024-07-26T16:22:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:41 smithi160 ceph-mon[36715]: pgmap v10518: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:22:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19004.969189 secs 2024-07-26T16:22:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:43 smithi183 ceph-mon[37273]: pgmap v10519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:43 smithi144 ceph-mon[32129]: pgmap v10519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:44.003 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:44.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:22:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:44.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:22:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:44.002+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:22:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:44.002+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:22:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:44.003+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:43 smithi160 ceph-mon[36715]: pgmap v10519: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:22:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:45 smithi183 ceph-mon[37273]: pgmap v10520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:45.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:45 smithi144 ceph-mon[32129]: pgmap v10520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:45 smithi160 ceph-mon[36715]: pgmap v10520: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19009.969314 secs 2024-07-26T16:22:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19009.969314 secs 2024-07-26T16:22:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19009.969314 secs 2024-07-26T16:22:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:47 smithi144 ceph-mon[32129]: pgmap v10521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:47 smithi160 ceph-mon[36715]: pgmap v10521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:47 smithi183 ceph-mon[37273]: pgmap v10521: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:49 smithi144 ceph-mon[32129]: pgmap v10522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:49 smithi160 ceph-mon[36715]: pgmap v10522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:49 smithi183 ceph-mon[37273]: pgmap v10522: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:51.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:51 smithi183 ceph-mon[37273]: pgmap v10523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:51.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19014.969426 secs 2024-07-26T16:22:51.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:51 smithi144 ceph-mon[32129]: pgmap v10523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19014.969426 secs 2024-07-26T16:22:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:51 smithi160 ceph-mon[36715]: pgmap v10523: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19014.969426 secs 2024-07-26T16:22:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:53 smithi144 ceph-mon[32129]: pgmap v10524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:53 smithi160 ceph-mon[36715]: pgmap v10524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:53 smithi183 ceph-mon[37273]: pgmap v10524: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:22:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:55 smithi144 ceph-mon[32129]: pgmap v10525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:55 smithi160 ceph-mon[36715]: pgmap v10525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:55 smithi183 ceph-mon[37273]: pgmap v10525: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:22:56.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19019.969551 secs 2024-07-26T16:22:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19019.969551 secs 2024-07-26T16:22:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19019.969551 secs 2024-07-26T16:22:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:57 smithi144 ceph-mon[32129]: pgmap v10526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:57 smithi160 ceph-mon[36715]: pgmap v10526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:57 smithi183 ceph-mon[37273]: pgmap v10526: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:22:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:59.018+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:22:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:59.019+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:22:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:59.019+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:22:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:59.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:22:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:22:59.019+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:22:59 smithi144 ceph-mon[32129]: pgmap v10527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:00.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:22:59 smithi160 ceph-mon[36715]: pgmap v10527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:22:59 smithi183 ceph-mon[37273]: pgmap v10527: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:01 smithi144 ceph-mon[32129]: pgmap v10528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19024.969657 secs 2024-07-26T16:23:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:01 smithi160 ceph-mon[36715]: pgmap v10528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19024.969657 secs 2024-07-26T16:23:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:01 smithi183 ceph-mon[37273]: pgmap v10528: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19024.969657 secs 2024-07-26T16:23:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:03 smithi144 ceph-mon[32129]: pgmap v10529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:03 smithi160 ceph-mon[36715]: pgmap v10529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:03 smithi183 ceph-mon[37273]: pgmap v10529: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:05 smithi144 ceph-mon[32129]: pgmap v10530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:05 smithi160 ceph-mon[36715]: pgmap v10530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:05 smithi183 ceph-mon[37273]: pgmap v10530: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19029.969779 secs 2024-07-26T16:23:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19029.969779 secs 2024-07-26T16:23:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19029.969779 secs 2024-07-26T16:23:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:07 smithi160 ceph-mon[36715]: pgmap v10531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:07 smithi183 ceph-mon[37273]: pgmap v10531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:08.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:07 smithi144 ceph-mon[32129]: pgmap v10531: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:09 smithi160 ceph-mon[36715]: pgmap v10532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:09 smithi183 ceph-mon[37273]: pgmap v10532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:09 smithi144 ceph-mon[32129]: pgmap v10532: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:11 smithi160 ceph-mon[36715]: pgmap v10533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19034.969905 secs 2024-07-26T16:23:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:11 smithi183 ceph-mon[37273]: pgmap v10533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19034.969905 secs 2024-07-26T16:23:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:11 smithi144 ceph-mon[32129]: pgmap v10533: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19034.969905 secs 2024-07-26T16:23:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:14.035+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:23:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:14.036+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:23:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:14.036+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:23:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:14.036+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:23:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:14.036+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:13 smithi160 ceph-mon[36715]: pgmap v10534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:13 smithi183 ceph-mon[37273]: pgmap v10534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:13 smithi144 ceph-mon[32129]: pgmap v10534: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:15 smithi160 ceph-mon[36715]: pgmap v10535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:15 smithi183 ceph-mon[37273]: pgmap v10535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:15 smithi144 ceph-mon[32129]: pgmap v10535: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19039.970020 secs 2024-07-26T16:23:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19039.970020 secs 2024-07-26T16:23:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19039.970020 secs 2024-07-26T16:23:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:17 smithi160 ceph-mon[36715]: pgmap v10536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:17 smithi183 ceph-mon[37273]: pgmap v10536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:17 smithi144 ceph-mon[32129]: pgmap v10536: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:19 smithi160 ceph-mon[36715]: pgmap v10537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:19 smithi183 ceph-mon[37273]: pgmap v10537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:19 smithi144 ceph-mon[32129]: pgmap v10537: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:21 smithi160 ceph-mon[36715]: pgmap v10538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19044.970146 secs 2024-07-26T16:23:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:21 smithi183 ceph-mon[37273]: pgmap v10538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19044.970146 secs 2024-07-26T16:23:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:21 smithi144 ceph-mon[32129]: pgmap v10538: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19044.970146 secs 2024-07-26T16:23:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:23 smithi160 ceph-mon[36715]: pgmap v10539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:23 smithi183 ceph-mon[37273]: pgmap v10539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:23 smithi144 ceph-mon[32129]: pgmap v10539: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:25 smithi160 ceph-mon[36715]: pgmap v10540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:25 smithi183 ceph-mon[37273]: pgmap v10540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:25 smithi144 ceph-mon[32129]: pgmap v10540: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19049.970285 secs 2024-07-26T16:23:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19049.970285 secs 2024-07-26T16:23:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19049.970285 secs 2024-07-26T16:23:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:27 smithi160 ceph-mon[36715]: pgmap v10541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:27 smithi183 ceph-mon[37273]: pgmap v10541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:27 smithi144 ceph-mon[32129]: pgmap v10541: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:29.043+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:23:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:29.044+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:23:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:29.044+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:23:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:29.044+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:23:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:29.045+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:29 smithi160 ceph-mon[36715]: pgmap v10542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:29 smithi183 ceph-mon[37273]: pgmap v10542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:29 smithi144 ceph-mon[32129]: pgmap v10542: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:31 smithi160 ceph-mon[36715]: pgmap v10543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19054.970397 secs 2024-07-26T16:23:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:31 smithi183 ceph-mon[37273]: pgmap v10543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19054.970397 secs 2024-07-26T16:23:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:31 smithi144 ceph-mon[32129]: pgmap v10543: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19054.970397 secs 2024-07-26T16:23:33.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:32 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:33.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:32 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:32 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:33 smithi183 ceph-mon[37273]: pgmap v10544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:33 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:33 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:23:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:33 smithi144 ceph-mon[32129]: pgmap v10544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:33 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:33 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:23:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:33 smithi160 ceph-mon[36715]: pgmap v10544: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:33 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:33 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:23:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:34 smithi183 ceph-mon[37273]: pgmap v10545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:34 smithi183 ceph-mon[37273]: pgmap v10546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:23:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:34 smithi144 ceph-mon[32129]: pgmap v10545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:35.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:34 smithi144 ceph-mon[32129]: pgmap v10546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:23:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:34 smithi160 ceph-mon[36715]: pgmap v10545: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:35.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:34 smithi160 ceph-mon[36715]: pgmap v10546: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:23:37.118 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:36 smithi144 ceph-mon[32129]: pgmap v10547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:37.119 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19059.970512 secs 2024-07-26T16:23:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:36 smithi183 ceph-mon[37273]: pgmap v10547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19059.970512 secs 2024-07-26T16:23:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:36 smithi160 ceph-mon[36715]: pgmap v10547: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19059.970512 secs 2024-07-26T16:23:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:38 smithi183 ceph-mon[37273]: pgmap v10548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:23:39.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:38 smithi144 ceph-mon[32129]: pgmap v10548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:23:39.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:38 smithi160 ceph-mon[36715]: pgmap v10548: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:23:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:40 smithi183 ceph-mon[37273]: pgmap v10549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:23:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:40 smithi144 ceph-mon[32129]: pgmap v10549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:23:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:40 smithi160 ceph-mon[36715]: pgmap v10549: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:23:42.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19064.970592 secs 2024-07-26T16:23:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19064.970592 secs 2024-07-26T16:23:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19064.970592 secs 2024-07-26T16:23:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:42 smithi183 ceph-mon[37273]: pgmap v10550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:43.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:42 smithi144 ceph-mon[32129]: pgmap v10550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:43.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:42 smithi160 ceph-mon[36715]: pgmap v10550: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:23:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:44.045+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:23:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:44.046+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:23:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:44.046+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:23:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:44.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:23:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:44.046+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:44 smithi183 ceph-mon[37273]: pgmap v10551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:23:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:44 smithi144 ceph-mon[32129]: pgmap v10551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:23:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:44 smithi160 ceph-mon[36715]: pgmap v10551: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:23:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:46 smithi183 ceph-mon[37273]: pgmap v10552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19069.970695 secs 2024-07-26T16:23:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:46 smithi144 ceph-mon[32129]: pgmap v10552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19069.970695 secs 2024-07-26T16:23:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:46 smithi160 ceph-mon[36715]: pgmap v10552: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19069.970695 secs 2024-07-26T16:23:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:48 smithi183 ceph-mon[37273]: pgmap v10553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:48 smithi144 ceph-mon[32129]: pgmap v10553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:48 smithi160 ceph-mon[36715]: pgmap v10553: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:49.464+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:23:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:49.464+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:23:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:49.465+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x7f74b0109990 2024-07-26T16:23:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:49.465+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:23:51.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:50 smithi183 ceph-mon[37273]: pgmap v10554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:50 smithi144 ceph-mon[32129]: pgmap v10554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:50 smithi160 ceph-mon[36715]: pgmap v10554: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:52.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19074.970763 secs 2024-07-26T16:23:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19074.970763 secs 2024-07-26T16:23:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19074.970763 secs 2024-07-26T16:23:53.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:52 smithi160 ceph-mon[36715]: pgmap v10555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:53.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:52 smithi144 ceph-mon[32129]: pgmap v10555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:52 smithi183 ceph-mon[37273]: pgmap v10555: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:23:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:54 smithi144 ceph-mon[32129]: pgmap v10556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:54 smithi160 ceph-mon[36715]: pgmap v10556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:54 smithi183 ceph-mon[37273]: pgmap v10556: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:56 smithi144 ceph-mon[32129]: pgmap v10557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19079.970909 secs 2024-07-26T16:23:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:56 smithi160 ceph-mon[36715]: pgmap v10557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19079.970909 secs 2024-07-26T16:23:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:56 smithi183 ceph-mon[37273]: pgmap v10557: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:23:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19079.970909 secs 2024-07-26T16:23:59.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:59.063+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:23:59.065 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:59.064+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:23:59.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:59.064+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:23:59.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:59.064+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:23:59.066 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:23:59.064+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:59.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:23:58 smithi144 ceph-mon[32129]: pgmap v10558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:23:58 smithi160 ceph-mon[36715]: pgmap v10558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:23:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:23:58 smithi183 ceph-mon[37273]: pgmap v10558: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:01.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:00 smithi144 ceph-mon[32129]: pgmap v10559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:01.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:00 smithi160 ceph-mon[36715]: pgmap v10559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:00 smithi183 ceph-mon[37273]: pgmap v10559: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19084.971031 secs 2024-07-26T16:24:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19084.971031 secs 2024-07-26T16:24:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19084.971031 secs 2024-07-26T16:24:03.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:02 smithi144 ceph-mon[32129]: pgmap v10560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:02 smithi160 ceph-mon[36715]: pgmap v10560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:02 smithi183 ceph-mon[37273]: pgmap v10560: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:05.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:04 smithi144 ceph-mon[32129]: pgmap v10561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:04 smithi160 ceph-mon[36715]: pgmap v10561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:04 smithi183 ceph-mon[37273]: pgmap v10561: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:06 smithi144 ceph-mon[32129]: pgmap v10562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19089.971170 secs 2024-07-26T16:24:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:06 smithi160 ceph-mon[36715]: pgmap v10562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19089.971170 secs 2024-07-26T16:24:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:06 smithi183 ceph-mon[37273]: pgmap v10562: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19089.971170 secs 2024-07-26T16:24:09.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:08 smithi144 ceph-mon[32129]: pgmap v10563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:09.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:08 smithi160 ceph-mon[36715]: pgmap v10563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:08 smithi183 ceph-mon[37273]: pgmap v10563: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:11.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:10 smithi144 ceph-mon[32129]: pgmap v10564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:10 smithi160 ceph-mon[36715]: pgmap v10564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:10 smithi183 ceph-mon[37273]: pgmap v10564: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:12.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19094.971237 secs 2024-07-26T16:24:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19094.971237 secs 2024-07-26T16:24:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19094.971237 secs 2024-07-26T16:24:13.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:12 smithi144 ceph-mon[32129]: pgmap v10565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:12 smithi160 ceph-mon[36715]: pgmap v10565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:12 smithi183 ceph-mon[37273]: pgmap v10565: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:14.080+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:24:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:14.081+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:24:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:14.081+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:24:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:14.081+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:24:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:14.081+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:14 smithi160 ceph-mon[36715]: pgmap v10566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:14 smithi183 ceph-mon[37273]: pgmap v10566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:14 smithi144 ceph-mon[32129]: pgmap v10566: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:16 smithi160 ceph-mon[36715]: pgmap v10567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19099.971356 secs 2024-07-26T16:24:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:16 smithi183 ceph-mon[37273]: pgmap v10567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19099.971356 secs 2024-07-26T16:24:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:16 smithi144 ceph-mon[32129]: pgmap v10567: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19099.971356 secs 2024-07-26T16:24:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:18 smithi160 ceph-mon[36715]: pgmap v10568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:18 smithi183 ceph-mon[37273]: pgmap v10568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:18 smithi144 ceph-mon[32129]: pgmap v10568: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:20 smithi160 ceph-mon[36715]: pgmap v10569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:20 smithi183 ceph-mon[37273]: pgmap v10569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:20 smithi144 ceph-mon[32129]: pgmap v10569: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:22.286 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19104.971420 secs 2024-07-26T16:24:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19104.971420 secs 2024-07-26T16:24:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19104.971420 secs 2024-07-26T16:24:23.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:23 smithi160 ceph-mon[36715]: pgmap v10570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:23 smithi183 ceph-mon[37273]: pgmap v10570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:23 smithi144 ceph-mon[32129]: pgmap v10570: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:25.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:25 smithi160 ceph-mon[36715]: pgmap v10571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:25 smithi183 ceph-mon[37273]: pgmap v10571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:25 smithi144 ceph-mon[32129]: pgmap v10571: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:27 smithi160 ceph-mon[36715]: pgmap v10572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19109.971539 secs 2024-07-26T16:24:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:27 smithi183 ceph-mon[37273]: pgmap v10572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19109.971539 secs 2024-07-26T16:24:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:27 smithi144 ceph-mon[32129]: pgmap v10572: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19109.971539 secs 2024-07-26T16:24:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:29.097+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:24:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:29.098+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:24:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:29.098+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:24:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:29.098+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:24:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:29.098+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:29.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:29 smithi160 ceph-mon[36715]: pgmap v10573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:29 smithi183 ceph-mon[37273]: pgmap v10573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:29 smithi144 ceph-mon[32129]: pgmap v10573: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:31.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:31 smithi160 ceph-mon[36715]: pgmap v10574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:31 smithi183 ceph-mon[37273]: pgmap v10574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:31 smithi144 ceph-mon[32129]: pgmap v10574: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19114.971605 secs 2024-07-26T16:24:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19114.971605 secs 2024-07-26T16:24:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19114.971605 secs 2024-07-26T16:24:33.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:33 smithi160 ceph-mon[36715]: pgmap v10575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:33.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:33 smithi144 ceph-mon[32129]: pgmap v10575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:33 smithi183 ceph-mon[37273]: pgmap v10575: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:34 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:34 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:34 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:35 smithi183 ceph-mon[37273]: pgmap v10576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:35 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:24:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:35 smithi144 ceph-mon[32129]: pgmap v10576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:35 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:24:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:35 smithi160 ceph-mon[36715]: pgmap v10576: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:35.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:35 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:24:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:36 smithi183 ceph-mon[37273]: pgmap v10577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:24:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:36 smithi183 ceph-mon[37273]: pgmap v10578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:24:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:36 smithi144 ceph-mon[32129]: pgmap v10577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:24:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:36 smithi144 ceph-mon[32129]: pgmap v10578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:24:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:36 smithi160 ceph-mon[36715]: pgmap v10577: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:24:36.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:36 smithi160 ceph-mon[36715]: pgmap v10578: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:24:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19119.971697 secs 2024-07-26T16:24:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19119.971697 secs 2024-07-26T16:24:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19119.971697 secs 2024-07-26T16:24:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:38 smithi183 ceph-mon[37273]: pgmap v10579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:24:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:38 smithi144 ceph-mon[32129]: pgmap v10579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:24:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:38 smithi160 ceph-mon[36715]: pgmap v10579: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:24:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:40 smithi183 ceph-mon[37273]: pgmap v10580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:24:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:40 smithi144 ceph-mon[32129]: pgmap v10580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:24:40.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:40 smithi160 ceph-mon[36715]: pgmap v10580: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:24:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:42 smithi183 ceph-mon[37273]: pgmap v10581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19124.971818 secs 2024-07-26T16:24:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:42 smithi144 ceph-mon[32129]: pgmap v10581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19124.971818 secs 2024-07-26T16:24:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:42 smithi160 ceph-mon[36715]: pgmap v10581: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19124.971818 secs 2024-07-26T16:24:44.105 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:44.102+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:24:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:44.103+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:24:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:44.103+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:24:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:44.103+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:24:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:44.103+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:44 smithi183 ceph-mon[37273]: pgmap v10582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:24:44.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:44 smithi144 ceph-mon[32129]: pgmap v10582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:24:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:44 smithi160 ceph-mon[36715]: pgmap v10582: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:24:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:46 smithi183 ceph-mon[37273]: pgmap v10583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:24:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:46 smithi144 ceph-mon[32129]: pgmap v10583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:24:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:46 smithi160 ceph-mon[36715]: pgmap v10583: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T16:24:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19129.971938 secs 2024-07-26T16:24:47.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19129.971938 secs 2024-07-26T16:24:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19129.971938 secs 2024-07-26T16:24:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:48 smithi183 ceph-mon[37273]: pgmap v10584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:24:48.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:48 smithi144 ceph-mon[32129]: pgmap v10584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:24:48.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:48 smithi160 ceph-mon[36715]: pgmap v10584: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:24:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:50 smithi183 ceph-mon[37273]: pgmap v10585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:24:50.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:50 smithi144 ceph-mon[32129]: pgmap v10585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:24:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:50 smithi160 ceph-mon[36715]: pgmap v10585: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:24:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:52 smithi183 ceph-mon[37273]: pgmap v10586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:24:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19134.972076 secs 2024-07-26T16:24:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:52 smithi144 ceph-mon[32129]: pgmap v10586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:24:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19134.972076 secs 2024-07-26T16:24:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:52 smithi160 ceph-mon[36715]: pgmap v10586: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:24:52.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19134.972076 secs 2024-07-26T16:24:54.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:54 smithi183 ceph-mon[37273]: pgmap v10587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:54 smithi144 ceph-mon[32129]: pgmap v10587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:54 smithi160 ceph-mon[36715]: pgmap v10587: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:24:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:56 smithi144 ceph-mon[32129]: pgmap v10588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:56 smithi160 ceph-mon[36715]: pgmap v10588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:56 smithi183 ceph-mon[37273]: pgmap v10588: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:24:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19139.972170 secs 2024-07-26T16:24:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19139.972170 secs 2024-07-26T16:24:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19139.972170 secs 2024-07-26T16:24:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:24:58 smithi144 ceph-mon[32129]: pgmap v10589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:24:58 smithi160 ceph-mon[36715]: pgmap v10589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:24:58 smithi183 ceph-mon[37273]: pgmap v10589: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:24:59.122 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:59.120+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:24:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:59.121+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:24:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:59.121+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:24:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:59.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:24:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:24:59.121+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:00 smithi144 ceph-mon[32129]: pgmap v10590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:00 smithi160 ceph-mon[36715]: pgmap v10590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:00 smithi183 ceph-mon[37273]: pgmap v10590: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:02 smithi144 ceph-mon[32129]: pgmap v10591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19144.972279 secs 2024-07-26T16:25:02.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:02 smithi160 ceph-mon[36715]: pgmap v10591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:02.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19144.972279 secs 2024-07-26T16:25:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:02 smithi183 ceph-mon[37273]: pgmap v10591: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19144.972279 secs 2024-07-26T16:25:04.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:04 smithi144 ceph-mon[32129]: pgmap v10592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:04 smithi160 ceph-mon[36715]: pgmap v10592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:04 smithi183 ceph-mon[37273]: pgmap v10592: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:06.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:06 smithi144 ceph-mon[32129]: pgmap v10593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:06 smithi160 ceph-mon[36715]: pgmap v10593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:06 smithi183 ceph-mon[37273]: pgmap v10593: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19149.972306 secs 2024-07-26T16:25:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19149.972306 secs 2024-07-26T16:25:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19149.972306 secs 2024-07-26T16:25:08.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:08 smithi144 ceph-mon[32129]: pgmap v10594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:08 smithi160 ceph-mon[36715]: pgmap v10594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:08 smithi183 ceph-mon[37273]: pgmap v10594: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:10.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:10 smithi144 ceph-mon[32129]: pgmap v10595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:10 smithi160 ceph-mon[36715]: pgmap v10595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:10 smithi183 ceph-mon[37273]: pgmap v10595: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:12 smithi144 ceph-mon[32129]: pgmap v10596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19154.972456 secs 2024-07-26T16:25:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:12 smithi160 ceph-mon[36715]: pgmap v10596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19154.972456 secs 2024-07-26T16:25:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:12 smithi183 ceph-mon[37273]: pgmap v10596: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:12.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19154.972456 secs 2024-07-26T16:25:14.139 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:14.137+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:25:14.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:14.138+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:25:14.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:14.138+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:25:14.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:14.138+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:25:14.140 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:14.138+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:14 smithi144 ceph-mon[32129]: pgmap v10597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:14 smithi160 ceph-mon[36715]: pgmap v10597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:14 smithi183 ceph-mon[37273]: pgmap v10597: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:16.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:16 smithi144 ceph-mon[32129]: pgmap v10598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:16 smithi160 ceph-mon[36715]: pgmap v10598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:16 smithi183 ceph-mon[37273]: pgmap v10598: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19159.972581 secs 2024-07-26T16:25:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19159.972581 secs 2024-07-26T16:25:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19159.972581 secs 2024-07-26T16:25:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:18 smithi160 ceph-mon[36715]: pgmap v10599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:18 smithi183 ceph-mon[37273]: pgmap v10599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:18 smithi144 ceph-mon[32129]: pgmap v10599: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:20 smithi160 ceph-mon[36715]: pgmap v10600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:20 smithi183 ceph-mon[37273]: pgmap v10600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:20.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:20 smithi144 ceph-mon[32129]: pgmap v10600: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:22 smithi160 ceph-mon[36715]: pgmap v10601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19164.972662 secs 2024-07-26T16:25:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:22 smithi183 ceph-mon[37273]: pgmap v10601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19164.972662 secs 2024-07-26T16:25:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:22 smithi144 ceph-mon[32129]: pgmap v10601: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19164.972662 secs 2024-07-26T16:25:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:24 smithi160 ceph-mon[36715]: pgmap v10602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:24 smithi183 ceph-mon[37273]: pgmap v10602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:24 smithi144 ceph-mon[32129]: pgmap v10602: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:26 smithi144 ceph-mon[32129]: pgmap v10603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:26 smithi160 ceph-mon[36715]: pgmap v10603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:26 smithi183 ceph-mon[37273]: pgmap v10603: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19169.972792 secs 2024-07-26T16:25:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19169.972792 secs 2024-07-26T16:25:27.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19169.972792 secs 2024-07-26T16:25:28.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:28 smithi160 ceph-mon[36715]: pgmap v10604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:28 smithi183 ceph-mon[37273]: pgmap v10604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:28.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:28 smithi144 ceph-mon[32129]: pgmap v10604: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:29.154+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:25:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:29.155+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:25:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:29.155+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:25:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:29.155+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:25:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:29.156+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:30 smithi160 ceph-mon[36715]: pgmap v10605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:30 smithi183 ceph-mon[37273]: pgmap v10605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:30 smithi144 ceph-mon[32129]: pgmap v10605: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:31 smithi160 ceph-mon[36715]: pgmap v10606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:31 smithi183 ceph-mon[37273]: pgmap v10606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:31 smithi144 ceph-mon[32129]: pgmap v10606: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19174.972863 secs 2024-07-26T16:25:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19174.972863 secs 2024-07-26T16:25:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19174.972863 secs 2024-07-26T16:25:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:33 smithi160 ceph-mon[36715]: pgmap v10607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:33 smithi183 ceph-mon[37273]: pgmap v10607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:33 smithi144 ceph-mon[32129]: pgmap v10607: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:35 smithi144 ceph-mon[32129]: pgmap v10608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:35 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:35 smithi160 ceph-mon[36715]: pgmap v10608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:35 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:35 smithi183 ceph-mon[37273]: pgmap v10608: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:35 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19179.972978 secs 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:37.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:37.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: pgmap v10609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:37.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: pgmap v10610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:37.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:36 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:25:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19179.972978 secs 2024-07-26T16:25:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: pgmap v10609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: pgmap v10610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:37.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:36 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:25:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19179.972978 secs 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi144 to 176.4M 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi144 to 185029017: error parsing value: Value '185029017' is below minimum 939524096 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:37.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: pgmap v10609: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:37.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: pgmap v10610: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:37.472 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:36 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:25:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:39 smithi144 ceph-mon[32129]: pgmap v10611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:39 smithi160 ceph-mon[36715]: pgmap v10611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:39 smithi183 ceph-mon[37273]: pgmap v10611: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:41 smithi144 ceph-mon[32129]: pgmap v10612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:25:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:41 smithi160 ceph-mon[36715]: pgmap v10612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:25:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:41 smithi183 ceph-mon[37273]: pgmap v10612: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:25:42.371 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19184.973101 secs 2024-07-26T16:25:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19184.973101 secs 2024-07-26T16:25:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19184.973101 secs 2024-07-26T16:25:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:43 smithi144 ceph-mon[32129]: pgmap v10613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:43 smithi160 ceph-mon[36715]: pgmap v10613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:43 smithi183 ceph-mon[37273]: pgmap v10613: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:44.169 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:44.166+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:25:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:44.168+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:25:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:44.168+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:25:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:44.168+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:25:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:44.168+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:45 smithi144 ceph-mon[32129]: pgmap v10614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:45.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:45 smithi160 ceph-mon[36715]: pgmap v10614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:45 smithi183 ceph-mon[37273]: pgmap v10614: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:25:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19189.973232 secs 2024-07-26T16:25:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:47 smithi144 ceph-mon[32129]: pgmap v10615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:25:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19189.973232 secs 2024-07-26T16:25:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:47 smithi160 ceph-mon[36715]: pgmap v10615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:25:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19189.973232 secs 2024-07-26T16:25:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:47 smithi183 ceph-mon[37273]: pgmap v10615: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:25:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:49 smithi144 ceph-mon[32129]: pgmap v10616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:49 smithi160 ceph-mon[36715]: pgmap v10616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:49 smithi183 ceph-mon[37273]: pgmap v10616: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:51 smithi144 ceph-mon[32129]: pgmap v10617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:51 smithi160 ceph-mon[36715]: pgmap v10617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:51 smithi183 ceph-mon[37273]: pgmap v10617: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:25:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19194.973327 secs 2024-07-26T16:25:52.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19194.973327 secs 2024-07-26T16:25:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19194.973327 secs 2024-07-26T16:25:53.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:53 smithi144 ceph-mon[32129]: pgmap v10618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:53 smithi160 ceph-mon[36715]: pgmap v10618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:53 smithi183 ceph-mon[37273]: pgmap v10618: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:55.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:55 smithi144 ceph-mon[32129]: pgmap v10619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:55 smithi160 ceph-mon[36715]: pgmap v10619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:55 smithi183 ceph-mon[37273]: pgmap v10619: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19199.973450 secs 2024-07-26T16:25:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:57 smithi144 ceph-mon[32129]: pgmap v10620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19199.973450 secs 2024-07-26T16:25:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:57 smithi160 ceph-mon[36715]: pgmap v10620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19199.973450 secs 2024-07-26T16:25:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:57 smithi183 ceph-mon[37273]: pgmap v10620: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:25:59.186 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:59.184+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:25:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:59.186+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:25:59.187 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:59.186+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:25:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:59.186+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:25:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:25:59.186+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:25:59 smithi160 ceph-mon[36715]: pgmap v10621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:25:59 smithi183 ceph-mon[37273]: pgmap v10621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:25:59.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:25:59 smithi144 ceph-mon[32129]: pgmap v10621: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:01 smithi160 ceph-mon[36715]: pgmap v10622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:01 smithi183 ceph-mon[37273]: pgmap v10622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:01 smithi144 ceph-mon[32129]: pgmap v10622: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19204.973568 secs 2024-07-26T16:26:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19204.973568 secs 2024-07-26T16:26:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19204.973568 secs 2024-07-26T16:26:03.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:03 smithi160 ceph-mon[36715]: pgmap v10623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:03 smithi183 ceph-mon[37273]: pgmap v10623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:03 smithi144 ceph-mon[32129]: pgmap v10623: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:05.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:05 smithi160 ceph-mon[36715]: pgmap v10624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:05 smithi183 ceph-mon[37273]: pgmap v10624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:05.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:05 smithi144 ceph-mon[32129]: pgmap v10624: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:07.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19209.973691 secs 2024-07-26T16:26:07.523 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:07 smithi144 ceph-mon[32129]: pgmap v10625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19209.973691 secs 2024-07-26T16:26:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:07 smithi160 ceph-mon[36715]: pgmap v10625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19209.973691 secs 2024-07-26T16:26:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:07 smithi183 ceph-mon[37273]: pgmap v10625: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:09.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:09 smithi160 ceph-mon[36715]: pgmap v10626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:09 smithi183 ceph-mon[37273]: pgmap v10626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:09.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:09 smithi144 ceph-mon[32129]: pgmap v10626: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:11.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:11 smithi160 ceph-mon[36715]: pgmap v10627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:11.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:11 smithi183 ceph-mon[37273]: pgmap v10627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:11.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:11 smithi144 ceph-mon[32129]: pgmap v10627: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19214.973833 secs 2024-07-26T16:26:12.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19214.973833 secs 2024-07-26T16:26:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19214.973833 secs 2024-07-26T16:26:13.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:13 smithi160 ceph-mon[36715]: pgmap v10628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:13.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:13 smithi183 ceph-mon[37273]: pgmap v10628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:13.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:13 smithi144 ceph-mon[32129]: pgmap v10628: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:14.202+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:26:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:14.202+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:26:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:14.202+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:26:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:14.202+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:26:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:14.203+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:14 smithi160 ceph-mon[36715]: pgmap v10629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:14 smithi183 ceph-mon[37273]: pgmap v10629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:14 smithi144 ceph-mon[32129]: pgmap v10629: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19219.973948 secs 2024-07-26T16:26:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:17 smithi144 ceph-mon[32129]: pgmap v10630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19219.973948 secs 2024-07-26T16:26:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:17 smithi160 ceph-mon[36715]: pgmap v10630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19219.973948 secs 2024-07-26T16:26:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:17 smithi183 ceph-mon[37273]: pgmap v10630: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:19 smithi144 ceph-mon[32129]: pgmap v10631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:19 smithi160 ceph-mon[36715]: pgmap v10631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:19 smithi183 ceph-mon[37273]: pgmap v10631: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:21 smithi144 ceph-mon[32129]: pgmap v10632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:21 smithi160 ceph-mon[36715]: pgmap v10632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:21 smithi183 ceph-mon[37273]: pgmap v10632: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19224.974068 secs 2024-07-26T16:26:22.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19224.974068 secs 2024-07-26T16:26:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19224.974068 secs 2024-07-26T16:26:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:23 smithi144 ceph-mon[32129]: pgmap v10633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:23 smithi160 ceph-mon[36715]: pgmap v10633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:23 smithi183 ceph-mon[37273]: pgmap v10633: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:25 smithi144 ceph-mon[32129]: pgmap v10634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:25 smithi160 ceph-mon[36715]: pgmap v10634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:25 smithi183 ceph-mon[37273]: pgmap v10634: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19229.974185 secs 2024-07-26T16:26:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:27 smithi144 ceph-mon[32129]: pgmap v10635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19229.974185 secs 2024-07-26T16:26:27.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:27 smithi160 ceph-mon[36715]: pgmap v10635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19229.974185 secs 2024-07-26T16:26:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:27 smithi183 ceph-mon[37273]: pgmap v10635: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:29.210+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:26:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:29.211+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:26:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:29.211+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:26:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:29.211+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:26:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:29.211+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:29 smithi144 ceph-mon[32129]: pgmap v10636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:29 smithi160 ceph-mon[36715]: pgmap v10636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:29 smithi183 ceph-mon[37273]: pgmap v10636: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:31.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:31 smithi160 ceph-mon[36715]: pgmap v10637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:31 smithi183 ceph-mon[37273]: pgmap v10637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:31.830 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:31 smithi144 ceph-mon[32129]: pgmap v10637: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19234.974323 secs 2024-07-26T16:26:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19234.974323 secs 2024-07-26T16:26:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19234.974323 secs 2024-07-26T16:26:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:33 smithi160 ceph-mon[36715]: pgmap v10638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:33 smithi183 ceph-mon[37273]: pgmap v10638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:33.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:33 smithi144 ceph-mon[32129]: pgmap v10638: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:35 smithi160 ceph-mon[36715]: pgmap v10639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:35.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:35 smithi183 ceph-mon[37273]: pgmap v10639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:35.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:35 smithi144 ceph-mon[32129]: pgmap v10639: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19239.974440 secs 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: pgmap v10640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:37 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19239.974440 secs 2024-07-26T16:26:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: pgmap v10640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:37 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19239.974440 secs 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: pgmap v10640: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:37.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:37 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:26:38.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:38 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:26:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:26:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:26:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:26:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:26:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:38.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:38.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:38 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:26:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi160 to 176.2M 2024-07-26T16:26:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-07-26T16:26:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi160 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T16:26:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: Unable to set osd_memory_target on smithi183 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-26T16:26:38.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:38.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:38.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:38 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:26:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:39 smithi160 ceph-mon[36715]: pgmap v10641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:26:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:39 smithi160 ceph-mon[36715]: pgmap v10642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:26:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:39 smithi183 ceph-mon[37273]: pgmap v10641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:26:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:39 smithi183 ceph-mon[37273]: pgmap v10642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:26:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:39 smithi144 ceph-mon[32129]: pgmap v10641: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:26:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:39 smithi144 ceph-mon[32129]: pgmap v10642: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:26:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:41 smithi160 ceph-mon[36715]: pgmap v10643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:41 smithi183 ceph-mon[37273]: pgmap v10643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:41 smithi144 ceph-mon[32129]: pgmap v10643: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19244.974523 secs 2024-07-26T16:26:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19244.974523 secs 2024-07-26T16:26:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19244.974523 secs 2024-07-26T16:26:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:43 smithi160 ceph-mon[36715]: pgmap v10644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:43 smithi183 ceph-mon[37273]: pgmap v10644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:43.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:43 smithi144 ceph-mon[32129]: pgmap v10644: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:44.226+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:26:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:44.227+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:26:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:44.227+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:26:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:44.227+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:26:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:44.227+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:44.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:44 smithi160 ceph-mon[36715]: pgmap v10645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:26:44.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:44 smithi183 ceph-mon[37273]: pgmap v10645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:26:44.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:44 smithi144 ceph-mon[32129]: pgmap v10645: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:26:46.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:46 smithi160 ceph-mon[36715]: pgmap v10646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:46.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:46 smithi183 ceph-mon[37273]: pgmap v10646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:46.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:46 smithi144 ceph-mon[32129]: pgmap v10646: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:26:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19249.974627 secs 2024-07-26T16:26:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19249.974627 secs 2024-07-26T16:26:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19249.974627 secs 2024-07-26T16:26:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:48 smithi183 ceph-mon[37273]: pgmap v10647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:26:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:48 smithi144 ceph-mon[32129]: pgmap v10647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:26:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:48 smithi160 ceph-mon[36715]: pgmap v10647: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:26:49.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:49.496+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:26:49.498 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:49.496+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480004740 con 0x7f74b0109990 2024-07-26T16:26:49.499 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:49.496+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7478006520 con 0x7f74b0109990 2024-07-26T16:26:49.499 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:49.496+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:26:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:50 smithi183 ceph-mon[37273]: pgmap v10648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:50.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:50 smithi144 ceph-mon[32129]: pgmap v10648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:50.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:50 smithi160 ceph-mon[36715]: pgmap v10648: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19254.974729 secs 2024-07-26T16:26:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:52 smithi183 ceph-mon[37273]: pgmap v10649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19254.974729 secs 2024-07-26T16:26:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:52 smithi144 ceph-mon[32129]: pgmap v10649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19254.974729 secs 2024-07-26T16:26:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:52 smithi160 ceph-mon[36715]: pgmap v10649: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:26:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:54 smithi183 ceph-mon[37273]: pgmap v10650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:54 smithi144 ceph-mon[32129]: pgmap v10650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:54 smithi160 ceph-mon[36715]: pgmap v10650: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:56.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:56 smithi160 ceph-mon[36715]: pgmap v10651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:56 smithi183 ceph-mon[37273]: pgmap v10651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:56 smithi144 ceph-mon[32129]: pgmap v10651: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:26:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19259.974850 secs 2024-07-26T16:26:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19259.974850 secs 2024-07-26T16:26:57.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19259.974850 secs 2024-07-26T16:26:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:26:58 smithi183 ceph-mon[37273]: pgmap v10652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:26:58 smithi144 ceph-mon[32129]: pgmap v10652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:26:58 smithi160 ceph-mon[36715]: pgmap v10652: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:26:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:59.244+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:26:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:59.245+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:26:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:59.245+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:26:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:59.245+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:26:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:26:59.245+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:00 smithi183 ceph-mon[37273]: pgmap v10653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:00 smithi144 ceph-mon[32129]: pgmap v10653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:00 smithi160 ceph-mon[36715]: pgmap v10653: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19264.974966 secs 2024-07-26T16:27:02.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:02 smithi183 ceph-mon[37273]: pgmap v10654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19264.974966 secs 2024-07-26T16:27:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:02 smithi144 ceph-mon[32129]: pgmap v10654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19264.974966 secs 2024-07-26T16:27:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:02 smithi160 ceph-mon[36715]: pgmap v10654: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:04 smithi183 ceph-mon[37273]: pgmap v10655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:04 smithi144 ceph-mon[32129]: pgmap v10655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:04 smithi160 ceph-mon[36715]: pgmap v10655: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:06 smithi183 ceph-mon[37273]: pgmap v10656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:06 smithi144 ceph-mon[32129]: pgmap v10656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:06 smithi160 ceph-mon[36715]: pgmap v10656: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19269.975092 secs 2024-07-26T16:27:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19269.975092 secs 2024-07-26T16:27:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19269.975092 secs 2024-07-26T16:27:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:08 smithi144 ceph-mon[32129]: pgmap v10657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:08 smithi160 ceph-mon[36715]: pgmap v10657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:08 smithi183 ceph-mon[37273]: pgmap v10657: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:10 smithi144 ceph-mon[32129]: pgmap v10658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:10 smithi160 ceph-mon[36715]: pgmap v10658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:10 smithi183 ceph-mon[37273]: pgmap v10658: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19274.975208 secs 2024-07-26T16:27:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:12 smithi144 ceph-mon[32129]: pgmap v10659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19274.975208 secs 2024-07-26T16:27:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:12 smithi160 ceph-mon[36715]: pgmap v10659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19274.975208 secs 2024-07-26T16:27:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:12 smithi183 ceph-mon[37273]: pgmap v10659: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:14.261+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:27:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:14.262+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:27:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:14.262+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:27:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:14.262+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:27:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:14.262+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:14.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:14 smithi144 ceph-mon[32129]: pgmap v10660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:14.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:14 smithi160 ceph-mon[36715]: pgmap v10660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:14 smithi183 ceph-mon[37273]: pgmap v10660: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:16 smithi144 ceph-mon[32129]: pgmap v10661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:16 smithi160 ceph-mon[36715]: pgmap v10661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:16 smithi183 ceph-mon[37273]: pgmap v10661: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19279.975286 secs 2024-07-26T16:27:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19279.975286 secs 2024-07-26T16:27:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19279.975286 secs 2024-07-26T16:27:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:18 smithi144 ceph-mon[32129]: pgmap v10662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:18 smithi160 ceph-mon[36715]: pgmap v10662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:18 smithi183 ceph-mon[37273]: pgmap v10662: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:20 smithi144 ceph-mon[32129]: pgmap v10663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:20 smithi160 ceph-mon[36715]: pgmap v10663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:20 smithi183 ceph-mon[37273]: pgmap v10663: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19284.975398 secs 2024-07-26T16:27:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:22 smithi144 ceph-mon[32129]: pgmap v10664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19284.975398 secs 2024-07-26T16:27:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:22 smithi160 ceph-mon[36715]: pgmap v10664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19284.975398 secs 2024-07-26T16:27:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:22 smithi183 ceph-mon[37273]: pgmap v10664: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:24 smithi144 ceph-mon[32129]: pgmap v10665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:24 smithi160 ceph-mon[36715]: pgmap v10665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:24 smithi183 ceph-mon[37273]: pgmap v10665: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:26 smithi144 ceph-mon[32129]: pgmap v10666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:26 smithi160 ceph-mon[36715]: pgmap v10666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:26 smithi183 ceph-mon[37273]: pgmap v10666: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19289.975551 secs 2024-07-26T16:27:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19289.975551 secs 2024-07-26T16:27:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19289.975551 secs 2024-07-26T16:27:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:28 smithi160 ceph-mon[36715]: pgmap v10667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:28 smithi183 ceph-mon[37273]: pgmap v10667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:28 smithi144 ceph-mon[32129]: pgmap v10667: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:29.278+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:27:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:29.279+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:27:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:29.279+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:27:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:29.279+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:27:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:29.279+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:30 smithi160 ceph-mon[36715]: pgmap v10668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:30 smithi183 ceph-mon[37273]: pgmap v10668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:30 smithi144 ceph-mon[32129]: pgmap v10668: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19294.975635 secs 2024-07-26T16:27:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:32 smithi160 ceph-mon[36715]: pgmap v10669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19294.975635 secs 2024-07-26T16:27:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:32 smithi183 ceph-mon[37273]: pgmap v10669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19294.975635 secs 2024-07-26T16:27:32.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:32 smithi144 ceph-mon[32129]: pgmap v10669: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:34.505+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:27:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:34.505+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:27:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:34.505+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7478006520 con 0x7f74b0109990 2024-07-26T16:27:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:34.505+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7480003690 con 0x563c6c4182f0 2024-07-26T16:27:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:34 smithi160 ceph-mon[36715]: pgmap v10670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:34 smithi183 ceph-mon[37273]: pgmap v10670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:34 smithi144 ceph-mon[32129]: pgmap v10670: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:36.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:36 smithi160 ceph-mon[36715]: pgmap v10671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:36.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:36 smithi183 ceph-mon[37273]: pgmap v10671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:36.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:36 smithi144 ceph-mon[32129]: pgmap v10671: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:37.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19299.975753 secs 2024-07-26T16:27:37.871 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19299.975753 secs 2024-07-26T16:27:37.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19299.975753 secs 2024-07-26T16:27:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:38 smithi160 ceph-mon[36715]: pgmap v10672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:38.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:38 smithi183 ceph-mon[37273]: pgmap v10672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:38.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:38 smithi144 ceph-mon[32129]: pgmap v10672: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:38.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:39 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:39.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:39 smithi160 ceph-mon[36715]: pgmap v10673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:39.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:39 smithi160 ceph-mon[36715]: pgmap v10674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:39.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:27:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:39 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:39 smithi183 ceph-mon[37273]: pgmap v10673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:39 smithi183 ceph-mon[37273]: pgmap v10674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:39.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:27:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:39 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:39 smithi144 ceph-mon[32129]: pgmap v10673: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:39 smithi144 ceph-mon[32129]: pgmap v10674: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:39.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:27:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:41 smithi183 ceph-mon[37273]: pgmap v10675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:41.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19304.975876 secs 2024-07-26T16:27:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:41 smithi144 ceph-mon[32129]: pgmap v10675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19304.975876 secs 2024-07-26T16:27:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:41 smithi160 ceph-mon[36715]: pgmap v10675: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19304.975876 secs 2024-07-26T16:27:43.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:43 smithi183 ceph-mon[37273]: pgmap v10676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:27:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:43 smithi144 ceph-mon[32129]: pgmap v10676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:27:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:43 smithi160 ceph-mon[36715]: pgmap v10676: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:27:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:44.294+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:27:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:44.295+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:27:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:44.295+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:27:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:44.295+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:27:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:44.295+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:45 smithi183 ceph-mon[37273]: pgmap v10677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:27:45.950 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:45 smithi144 ceph-mon[32129]: pgmap v10677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:27:46.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:45 smithi160 ceph-mon[36715]: pgmap v10677: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:27:46.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19309.975997 secs 2024-07-26T16:27:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19309.975997 secs 2024-07-26T16:27:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19309.975997 secs 2024-07-26T16:27:47.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:47 smithi183 ceph-mon[37273]: pgmap v10678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:47.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:47 smithi144 ceph-mon[32129]: pgmap v10678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:47 smithi160 ceph-mon[36715]: pgmap v10678: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:49.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:49 smithi183 ceph-mon[37273]: pgmap v10679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:27:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:49 smithi144 ceph-mon[32129]: pgmap v10679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:27:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:49 smithi160 ceph-mon[36715]: pgmap v10679: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:27:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:51 smithi183 ceph-mon[37273]: pgmap v10680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:51.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19314.976116 secs 2024-07-26T16:27:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:51 smithi144 ceph-mon[32129]: pgmap v10680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19314.976116 secs 2024-07-26T16:27:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:51 smithi160 ceph-mon[36715]: pgmap v10680: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19314.976116 secs 2024-07-26T16:27:53.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:53 smithi183 ceph-mon[37273]: pgmap v10681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:53 smithi144 ceph-mon[32129]: pgmap v10681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:53 smithi160 ceph-mon[36715]: pgmap v10681: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:27:55.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:55 smithi183 ceph-mon[37273]: pgmap v10682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:55 smithi144 ceph-mon[32129]: pgmap v10682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:55 smithi160 ceph-mon[36715]: pgmap v10682: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:27:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19319.976233 secs 2024-07-26T16:27:56.961 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19319.976233 secs 2024-07-26T16:27:56.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19319.976233 secs 2024-07-26T16:27:57.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:57 smithi144 ceph-mon[32129]: pgmap v10683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:57 smithi160 ceph-mon[36715]: pgmap v10683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:57 smithi183 ceph-mon[37273]: pgmap v10683: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:27:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:59.309+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:27:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:59.310+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:27:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:59.310+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:27:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:59.310+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:27:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:27:59.310+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:59.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:27:59 smithi144 ceph-mon[32129]: pgmap v10684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:27:59 smithi160 ceph-mon[36715]: pgmap v10684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:27:59 smithi183 ceph-mon[37273]: pgmap v10684: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:01 smithi144 ceph-mon[32129]: pgmap v10685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:01.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19324.976362 secs 2024-07-26T16:28:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:01 smithi160 ceph-mon[36715]: pgmap v10685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19324.976362 secs 2024-07-26T16:28:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:01 smithi183 ceph-mon[37273]: pgmap v10685: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19324.976362 secs 2024-07-26T16:28:03.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:03 smithi144 ceph-mon[32129]: pgmap v10686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:03 smithi160 ceph-mon[36715]: pgmap v10686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:03 smithi183 ceph-mon[37273]: pgmap v10686: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:05.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:05 smithi144 ceph-mon[32129]: pgmap v10687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:05 smithi160 ceph-mon[36715]: pgmap v10687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:05 smithi183 ceph-mon[37273]: pgmap v10687: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19329.976483 secs 2024-07-26T16:28:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19329.976483 secs 2024-07-26T16:28:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19329.976483 secs 2024-07-26T16:28:07.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:07 smithi144 ceph-mon[32129]: pgmap v10688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:07 smithi160 ceph-mon[36715]: pgmap v10688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:07 smithi183 ceph-mon[37273]: pgmap v10688: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:09.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:09 smithi144 ceph-mon[32129]: pgmap v10689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:09 smithi160 ceph-mon[36715]: pgmap v10689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:09 smithi183 ceph-mon[37273]: pgmap v10689: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:11 smithi144 ceph-mon[32129]: pgmap v10690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19334.976600 secs 2024-07-26T16:28:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:11 smithi160 ceph-mon[36715]: pgmap v10690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19334.976600 secs 2024-07-26T16:28:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:11 smithi183 ceph-mon[37273]: pgmap v10690: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19334.976600 secs 2024-07-26T16:28:13.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:13 smithi144 ceph-mon[32129]: pgmap v10691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:13 smithi160 ceph-mon[36715]: pgmap v10691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:13 smithi183 ceph-mon[37273]: pgmap v10691: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:14.326+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:28:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:14.327+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:28:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:14.327+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:28:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:14.327+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:28:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:14.327+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:15.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:15 smithi144 ceph-mon[32129]: pgmap v10692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:15 smithi160 ceph-mon[36715]: pgmap v10692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:15 smithi183 ceph-mon[37273]: pgmap v10692: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19339.976696 secs 2024-07-26T16:28:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19339.976696 secs 2024-07-26T16:28:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19339.976696 secs 2024-07-26T16:28:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:17 smithi160 ceph-mon[36715]: pgmap v10693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:17 smithi183 ceph-mon[37273]: pgmap v10693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:17 smithi144 ceph-mon[32129]: pgmap v10693: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:20.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:19 smithi160 ceph-mon[36715]: pgmap v10694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:19 smithi183 ceph-mon[37273]: pgmap v10694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:19 smithi144 ceph-mon[32129]: pgmap v10694: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:22.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:21 smithi160 ceph-mon[36715]: pgmap v10695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:22.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19344.976813 secs 2024-07-26T16:28:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:21 smithi183 ceph-mon[37273]: pgmap v10695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19344.976813 secs 2024-07-26T16:28:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:21 smithi144 ceph-mon[32129]: pgmap v10695: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19344.976813 secs 2024-07-26T16:28:24.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:23 smithi160 ceph-mon[36715]: pgmap v10696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:23 smithi183 ceph-mon[37273]: pgmap v10696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:23 smithi144 ceph-mon[32129]: pgmap v10696: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:26.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:25 smithi160 ceph-mon[36715]: pgmap v10697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:25 smithi183 ceph-mon[37273]: pgmap v10697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:25 smithi144 ceph-mon[32129]: pgmap v10697: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:27.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19349.976934 secs 2024-07-26T16:28:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19349.976934 secs 2024-07-26T16:28:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19349.976934 secs 2024-07-26T16:28:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:27 smithi160 ceph-mon[36715]: pgmap v10698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:27 smithi183 ceph-mon[37273]: pgmap v10698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:27 smithi144 ceph-mon[32129]: pgmap v10698: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:29.333+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:28:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:29.334+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:28:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:29.334+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:28:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:29.334+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:28:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:29.334+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:29 smithi160 ceph-mon[36715]: pgmap v10699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:29 smithi183 ceph-mon[37273]: pgmap v10699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:29 smithi144 ceph-mon[32129]: pgmap v10699: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:31 smithi160 ceph-mon[36715]: pgmap v10700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19354.977057 secs 2024-07-26T16:28:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:31 smithi183 ceph-mon[37273]: pgmap v10700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19354.977057 secs 2024-07-26T16:28:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:31 smithi144 ceph-mon[32129]: pgmap v10700: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19354.977057 secs 2024-07-26T16:28:34.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:33 smithi160 ceph-mon[36715]: pgmap v10701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:33 smithi183 ceph-mon[37273]: pgmap v10701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:33 smithi144 ceph-mon[32129]: pgmap v10701: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:36.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:35 smithi160 ceph-mon[36715]: pgmap v10702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:35 smithi183 ceph-mon[37273]: pgmap v10702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:35 smithi144 ceph-mon[32129]: pgmap v10702: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19359.977194 secs 2024-07-26T16:28:37.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19359.977194 secs 2024-07-26T16:28:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19359.977194 secs 2024-07-26T16:28:38.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:37 smithi160 ceph-mon[36715]: pgmap v10703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:37 smithi183 ceph-mon[37273]: pgmap v10703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:37 smithi144 ceph-mon[32129]: pgmap v10703: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:38 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:38 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:39.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:38 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:40.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:39 smithi160 ceph-mon[36715]: pgmap v10704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:40.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:40.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:39 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:39 smithi183 ceph-mon[37273]: pgmap v10704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:39 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:39 smithi144 ceph-mon[32129]: pgmap v10704: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:39 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:40 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:40 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:40 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:41.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:40 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:40 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:40 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:28:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:41 smithi183 ceph-mon[37273]: pgmap v10705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:41 smithi183 ceph-mon[37273]: pgmap v10706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19364.977303 secs 2024-07-26T16:28:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:41 smithi144 ceph-mon[32129]: pgmap v10705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:41 smithi144 ceph-mon[32129]: pgmap v10706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19364.977303 secs 2024-07-26T16:28:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:41 smithi160 ceph-mon[36715]: pgmap v10705: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:41 smithi160 ceph-mon[36715]: pgmap v10706: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:42.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19364.977303 secs 2024-07-26T16:28:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:43 smithi183 ceph-mon[37273]: pgmap v10707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:43 smithi144 ceph-mon[32129]: pgmap v10707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:43 smithi160 ceph-mon[36715]: pgmap v10707: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:44.352 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:44.349+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:28:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:44.350+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:28:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:44.350+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:28:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:44.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:28:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:44.350+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:46.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:45 smithi183 ceph-mon[37273]: pgmap v10708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:45 smithi144 ceph-mon[32129]: pgmap v10708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:45 smithi160 ceph-mon[36715]: pgmap v10708: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19369.977400 secs 2024-07-26T16:28:47.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19369.977400 secs 2024-07-26T16:28:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19369.977400 secs 2024-07-26T16:28:48.122 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:47 smithi144 ceph-mon[32129]: pgmap v10709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:47 smithi183 ceph-mon[37273]: pgmap v10709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:47 smithi160 ceph-mon[36715]: pgmap v10709: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:49 smithi183 ceph-mon[37273]: pgmap v10710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:49 smithi144 ceph-mon[32129]: pgmap v10710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:49 smithi160 ceph-mon[36715]: pgmap v10710: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:28:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:51 smithi183 ceph-mon[37273]: pgmap v10711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:28:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19374.977523 secs 2024-07-26T16:28:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:51 smithi144 ceph-mon[32129]: pgmap v10711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:28:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19374.977523 secs 2024-07-26T16:28:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:51 smithi160 ceph-mon[36715]: pgmap v10711: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:28:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19374.977523 secs 2024-07-26T16:28:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:53 smithi183 ceph-mon[37273]: pgmap v10712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:54.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:53 smithi144 ceph-mon[32129]: pgmap v10712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:54.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:53 smithi160 ceph-mon[36715]: pgmap v10712: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:28:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:55 smithi183 ceph-mon[37273]: pgmap v10713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:56.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:55 smithi144 ceph-mon[32129]: pgmap v10713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:56.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:55 smithi160 ceph-mon[36715]: pgmap v10713: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:28:57.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19379.977645 secs 2024-07-26T16:28:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19379.977645 secs 2024-07-26T16:28:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19379.977645 secs 2024-07-26T16:28:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:57 smithi144 ceph-mon[32129]: pgmap v10714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:58.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:57 smithi160 ceph-mon[36715]: pgmap v10714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:58.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:57 smithi183 ceph-mon[37273]: pgmap v10714: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:28:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:59.367+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:28:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:59.368+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:28:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:59.368+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:28:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:59.368+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:28:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:28:59.368+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:28:59 smithi144 ceph-mon[32129]: pgmap v10715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:00.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:28:59 smithi160 ceph-mon[36715]: pgmap v10715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:00.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:28:59 smithi183 ceph-mon[37273]: pgmap v10715: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:01 smithi144 ceph-mon[32129]: pgmap v10716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19384.977765 secs 2024-07-26T16:29:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:01 smithi160 ceph-mon[36715]: pgmap v10716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19384.977765 secs 2024-07-26T16:29:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:01 smithi183 ceph-mon[37273]: pgmap v10716: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19384.977765 secs 2024-07-26T16:29:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:03 smithi144 ceph-mon[32129]: pgmap v10717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:04.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:03 smithi160 ceph-mon[36715]: pgmap v10717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:04.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:03 smithi183 ceph-mon[37273]: pgmap v10717: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:05 smithi144 ceph-mon[32129]: pgmap v10718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:06.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:05 smithi160 ceph-mon[36715]: pgmap v10718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:06.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:05 smithi183 ceph-mon[37273]: pgmap v10718: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19389.977873 secs 2024-07-26T16:29:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19389.977873 secs 2024-07-26T16:29:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19389.977873 secs 2024-07-26T16:29:08.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:07 smithi144 ceph-mon[32129]: pgmap v10719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:08.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:07 smithi160 ceph-mon[36715]: pgmap v10719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:08.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:07 smithi183 ceph-mon[37273]: pgmap v10719: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:10.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:09 smithi144 ceph-mon[32129]: pgmap v10720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:10.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:09 smithi160 ceph-mon[36715]: pgmap v10720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:10.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:09 smithi183 ceph-mon[37273]: pgmap v10720: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:11 smithi144 ceph-mon[32129]: pgmap v10721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19394.977994 secs 2024-07-26T16:29:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:11 smithi160 ceph-mon[36715]: pgmap v10721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19394.977994 secs 2024-07-26T16:29:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:11 smithi183 ceph-mon[37273]: pgmap v10721: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19394.977994 secs 2024-07-26T16:29:14.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:13 smithi144 ceph-mon[32129]: pgmap v10722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:14.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:13 smithi160 ceph-mon[36715]: pgmap v10722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:14.375+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:29:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:14.376+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:29:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:14.376+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:29:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:14.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:29:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:14.376+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:14.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:13 smithi183 ceph-mon[37273]: pgmap v10722: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:16.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:15 smithi144 ceph-mon[32129]: pgmap v10723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:16.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:15 smithi160 ceph-mon[36715]: pgmap v10723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:16.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:15 smithi183 ceph-mon[37273]: pgmap v10723: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19399.978116 secs 2024-07-26T16:29:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19399.978116 secs 2024-07-26T16:29:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19399.978116 secs 2024-07-26T16:29:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:17 smithi144 ceph-mon[32129]: pgmap v10724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:18.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:17 smithi160 ceph-mon[36715]: pgmap v10724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:18.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:17 smithi183 ceph-mon[37273]: pgmap v10724: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:19 smithi144 ceph-mon[32129]: pgmap v10725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:19 smithi160 ceph-mon[36715]: pgmap v10725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:20.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:19 smithi183 ceph-mon[37273]: pgmap v10725: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:21 smithi160 ceph-mon[36715]: pgmap v10726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19404.978255 secs 2024-07-26T16:29:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:21 smithi183 ceph-mon[37273]: pgmap v10726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19404.978255 secs 2024-07-26T16:29:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:21 smithi144 ceph-mon[32129]: pgmap v10726: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19404.978255 secs 2024-07-26T16:29:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:23 smithi160 ceph-mon[36715]: pgmap v10727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:24.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:23 smithi183 ceph-mon[37273]: pgmap v10727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:24.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:23 smithi144 ceph-mon[32129]: pgmap v10727: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:25 smithi160 ceph-mon[36715]: pgmap v10728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:26.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:25 smithi183 ceph-mon[37273]: pgmap v10728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:26.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:25 smithi144 ceph-mon[32129]: pgmap v10728: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:27.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19409.978369 secs 2024-07-26T16:29:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19409.978369 secs 2024-07-26T16:29:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19409.978369 secs 2024-07-26T16:29:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:27 smithi160 ceph-mon[36715]: pgmap v10729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:28.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:27 smithi183 ceph-mon[37273]: pgmap v10729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:28.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:27 smithi144 ceph-mon[32129]: pgmap v10729: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:29.392+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:29:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:29.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:29:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:29.393+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:29:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:29.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:29:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:29.393+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:30 smithi160 ceph-mon[36715]: pgmap v10730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:30.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:30 smithi183 ceph-mon[37273]: pgmap v10730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:30.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:30 smithi144 ceph-mon[32129]: pgmap v10730: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:32 smithi160 ceph-mon[36715]: pgmap v10731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19414.978493 secs 2024-07-26T16:29:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:32 smithi183 ceph-mon[37273]: pgmap v10731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19414.978493 secs 2024-07-26T16:29:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:32 smithi144 ceph-mon[32129]: pgmap v10731: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19414.978493 secs 2024-07-26T16:29:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:34 smithi160 ceph-mon[36715]: pgmap v10732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:34.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:34 smithi183 ceph-mon[37273]: pgmap v10732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:34.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:34 smithi144 ceph-mon[32129]: pgmap v10732: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:36 smithi160 ceph-mon[36715]: pgmap v10733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:36.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:36 smithi183 ceph-mon[37273]: pgmap v10733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:36.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:36 smithi144 ceph-mon[32129]: pgmap v10733: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19419.978612 secs 2024-07-26T16:29:37.372 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19419.978612 secs 2024-07-26T16:29:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19419.978612 secs 2024-07-26T16:29:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:38 smithi160 ceph-mon[36715]: pgmap v10734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:38.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:38 smithi183 ceph-mon[37273]: pgmap v10734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:38.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:38 smithi144 ceph-mon[32129]: pgmap v10734: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:40 smithi160 ceph-mon[36715]: pgmap v10735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:40.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:40 smithi183 ceph-mon[37273]: pgmap v10735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:40.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:40 smithi144 ceph-mon[32129]: pgmap v10735: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:41 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:41 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:41 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:41 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:41 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:41.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:41 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:42 smithi160 ceph-mon[36715]: pgmap v10736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19424.978678 secs 2024-07-26T16:29:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:42 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:42 smithi183 ceph-mon[37273]: pgmap v10736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19424.978678 secs 2024-07-26T16:29:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:42 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:42 smithi144 ceph-mon[32129]: pgmap v10736: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19424.978678 secs 2024-07-26T16:29:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:42 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:43.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: pgmap v10737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:43.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:43 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: pgmap v10737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:43 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: pgmap v10737: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:43 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:44.394+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:29:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:44.395+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:29:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:44.395+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:29:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:44.395+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:29:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:44.395+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:44 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:44 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:45.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:44 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:45 smithi160 ceph-mon[36715]: pgmap v10738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:45 smithi183 ceph-mon[37273]: pgmap v10738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:46.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:45 smithi144 ceph-mon[32129]: pgmap v10738: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19429.978805 secs 2024-07-26T16:29:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:29:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: pgmap v10739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: pgmap v10740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:47.471 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:47 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19429.978805 secs 2024-07-26T16:29:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:29:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: pgmap v10739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: pgmap v10740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:47.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:47 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19429.978805 secs 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: pgmap v10739: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: pgmap v10740: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:47 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:29:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:49.527+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:29:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:49.527+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:29:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:49.528+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:29:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:49.528+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:29:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:49 smithi160 ceph-mon[36715]: pgmap v10741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:49 smithi183 ceph-mon[37273]: pgmap v10741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:49 smithi144 ceph-mon[32129]: pgmap v10741: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:51 smithi160 ceph-mon[36715]: pgmap v10742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:29:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:51 smithi183 ceph-mon[37273]: pgmap v10742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:29:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:51 smithi144 ceph-mon[32129]: pgmap v10742: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T16:29:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19434.978928 secs 2024-07-26T16:29:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19434.978928 secs 2024-07-26T16:29:52.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19434.978928 secs 2024-07-26T16:29:53.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:53 smithi160 ceph-mon[36715]: pgmap v10743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:53 smithi183 ceph-mon[37273]: pgmap v10743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:53 smithi144 ceph-mon[32129]: pgmap v10743: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:55.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:55 smithi160 ceph-mon[36715]: pgmap v10744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:55.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:55 smithi183 ceph-mon[37273]: pgmap v10744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:55.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:55 smithi144 ceph-mon[32129]: pgmap v10744: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19439.979024 secs 2024-07-26T16:29:57.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:57 smithi160 ceph-mon[36715]: pgmap v10745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:29:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19439.979024 secs 2024-07-26T16:29:57.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:57 smithi183 ceph-mon[37273]: pgmap v10745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:29:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19439.979024 secs 2024-07-26T16:29:57.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:57 smithi144 ceph-mon[32129]: pgmap v10745: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:29:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:59.398+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:29:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:59.399+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:29:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:59.399+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:29:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:59.399+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:29:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:29:59.399+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:59.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:29:59 smithi160 ceph-mon[36715]: pgmap v10746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:59.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:29:59 smithi183 ceph-mon[37273]: pgmap v10746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:29:59.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:29:59 smithi144 ceph-mon[32129]: pgmap v10746: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:00 smithi160 ceph-mon[36715]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:30:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:00 smithi183 ceph-mon[37273]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:30:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:00 smithi144 ceph-mon[32129]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:30:01.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:01 smithi160 ceph-mon[36715]: pgmap v10747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:01 smithi183 ceph-mon[37273]: pgmap v10747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:01.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:01 smithi144 ceph-mon[32129]: pgmap v10747: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19444.979168 secs 2024-07-26T16:30:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19444.979168 secs 2024-07-26T16:30:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19444.979168 secs 2024-07-26T16:30:03.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:03 smithi183 ceph-mon[37273]: pgmap v10748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:03.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:03 smithi144 ceph-mon[32129]: pgmap v10748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:03.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:03 smithi160 ceph-mon[36715]: pgmap v10748: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:05 smithi144 ceph-mon[32129]: pgmap v10749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:05.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:05 smithi183 ceph-mon[37273]: pgmap v10749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:05.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:05 smithi160 ceph-mon[36715]: pgmap v10749: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19449.979285 secs 2024-07-26T16:30:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:07 smithi183 ceph-mon[37273]: pgmap v10750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19449.979285 secs 2024-07-26T16:30:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:07 smithi144 ceph-mon[32129]: pgmap v10750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19449.979285 secs 2024-07-26T16:30:07.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:07 smithi160 ceph-mon[36715]: pgmap v10750: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:09.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:09 smithi183 ceph-mon[37273]: pgmap v10751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:09.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:09 smithi144 ceph-mon[32129]: pgmap v10751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:09.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:09 smithi160 ceph-mon[36715]: pgmap v10751: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:10 smithi183 ceph-mon[37273]: pgmap v10752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:10.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:10 smithi144 ceph-mon[32129]: pgmap v10752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:10 smithi160 ceph-mon[36715]: pgmap v10752: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:11.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19454.979403 secs 2024-07-26T16:30:11.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19454.979403 secs 2024-07-26T16:30:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19454.979403 secs 2024-07-26T16:30:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:12 smithi183 ceph-mon[37273]: pgmap v10753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:12 smithi144 ceph-mon[32129]: pgmap v10753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:13.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:12 smithi160 ceph-mon[36715]: pgmap v10753: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:14.403 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:14.401+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:30:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:14.402+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:30:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:14.402+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:30:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:14.402+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:30:14.404 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:14.402+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:15.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:15 smithi160 ceph-mon[36715]: pgmap v10754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:15.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:15 smithi183 ceph-mon[37273]: pgmap v10754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:15.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:15 smithi144 ceph-mon[32129]: pgmap v10754: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19459.979524 secs 2024-07-26T16:30:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:17 smithi160 ceph-mon[36715]: pgmap v10755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19459.979524 secs 2024-07-26T16:30:17.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:17 smithi183 ceph-mon[37273]: pgmap v10755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19459.979524 secs 2024-07-26T16:30:17.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:17 smithi144 ceph-mon[32129]: pgmap v10755: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:19.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:19 smithi160 ceph-mon[36715]: pgmap v10756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:19.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:19 smithi183 ceph-mon[37273]: pgmap v10756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:19.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:19 smithi144 ceph-mon[32129]: pgmap v10756: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:21 smithi144 ceph-mon[32129]: pgmap v10757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:21.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:21 smithi160 ceph-mon[36715]: pgmap v10757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:21.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:21 smithi183 ceph-mon[37273]: pgmap v10757: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19464.979649 secs 2024-07-26T16:30:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19464.979649 secs 2024-07-26T16:30:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19464.979649 secs 2024-07-26T16:30:23.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:23 smithi160 ceph-mon[36715]: pgmap v10758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:23.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:23 smithi183 ceph-mon[37273]: pgmap v10758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:23.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:23 smithi144 ceph-mon[32129]: pgmap v10758: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:25 smithi160 ceph-mon[36715]: pgmap v10759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:25.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:25 smithi183 ceph-mon[37273]: pgmap v10759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:25.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:25 smithi144 ceph-mon[32129]: pgmap v10759: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19469.979749 secs 2024-07-26T16:30:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:27 smithi183 ceph-mon[37273]: pgmap v10760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19469.979749 secs 2024-07-26T16:30:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:27 smithi144 ceph-mon[32129]: pgmap v10760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19469.979749 secs 2024-07-26T16:30:27.965 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:27 smithi160 ceph-mon[36715]: pgmap v10760: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:29.410+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:30:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:29.411+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:30:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:29.411+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:30:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:29.411+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:30:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:29.411+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:29.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:29 smithi183 ceph-mon[37273]: pgmap v10761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:29.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:29 smithi144 ceph-mon[32129]: pgmap v10761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:29.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:29 smithi160 ceph-mon[36715]: pgmap v10761: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:31.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:31 smithi183 ceph-mon[37273]: pgmap v10762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:31.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:31 smithi144 ceph-mon[32129]: pgmap v10762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:31.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:31 smithi160 ceph-mon[36715]: pgmap v10762: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19474.979844 secs 2024-07-26T16:30:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19474.979844 secs 2024-07-26T16:30:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19474.979844 secs 2024-07-26T16:30:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:33 smithi144 ceph-mon[32129]: pgmap v10763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:33.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:33 smithi183 ceph-mon[37273]: pgmap v10763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:33.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:33 smithi160 ceph-mon[36715]: pgmap v10763: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:34.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:34 smithi144 ceph-mon[32129]: pgmap v10764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:34.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:34 smithi160 ceph-mon[36715]: pgmap v10764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:34 smithi183 ceph-mon[37273]: pgmap v10764: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19479.979934 secs 2024-07-26T16:30:37.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:37 smithi160 ceph-mon[36715]: pgmap v10765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19479.979934 secs 2024-07-26T16:30:37.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:37 smithi183 ceph-mon[37273]: pgmap v10765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19479.979934 secs 2024-07-26T16:30:37.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:37 smithi144 ceph-mon[32129]: pgmap v10765: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:39 smithi160 ceph-mon[36715]: pgmap v10766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:39.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:39 smithi183 ceph-mon[37273]: pgmap v10766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:39.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:39 smithi144 ceph-mon[32129]: pgmap v10766: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:41 smithi160 ceph-mon[36715]: pgmap v10767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:41.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:41 smithi183 ceph-mon[37273]: pgmap v10767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:41.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:41 smithi144 ceph-mon[32129]: pgmap v10767: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:30:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19484.980030 secs 2024-07-26T16:30:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19484.980030 secs 2024-07-26T16:30:42.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19484.980030 secs 2024-07-26T16:30:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:43 smithi160 ceph-mon[36715]: pgmap v10768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:43 smithi183 ceph-mon[37273]: pgmap v10768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:43.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:43 smithi144 ceph-mon[32129]: pgmap v10768: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:44.415+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:30:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:44.415+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:30:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:44.415+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:30:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:44.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:30:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:44.416+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:45 smithi183 ceph-mon[37273]: pgmap v10769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:45.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:45 smithi144 ceph-mon[32129]: pgmap v10769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:45.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:45 smithi160 ceph-mon[36715]: pgmap v10769: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:30:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19489.980168 secs 2024-07-26T16:30:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:47 smithi183 ceph-mon[37273]: pgmap v10770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19489.980168 secs 2024-07-26T16:30:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:47 smithi144 ceph-mon[32129]: pgmap v10770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:47.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:47.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19489.980168 secs 2024-07-26T16:30:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:47 smithi160 ceph-mon[36715]: pgmap v10770: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:30:47.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:48.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:30:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:48.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:30:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:48.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:30:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:49 smithi183 ceph-mon[37273]: pgmap v10771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:30:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:49 smithi183 ceph-mon[37273]: pgmap v10772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:30:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:49 smithi144 ceph-mon[32129]: pgmap v10771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:30:49.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:49 smithi144 ceph-mon[32129]: pgmap v10772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:30:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:49 smithi160 ceph-mon[36715]: pgmap v10771: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T16:30:49.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:49 smithi160 ceph-mon[36715]: pgmap v10772: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:30:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:51 smithi183 ceph-mon[37273]: pgmap v10773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:51.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:51 smithi144 ceph-mon[32129]: pgmap v10773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:51.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:51 smithi160 ceph-mon[36715]: pgmap v10773: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:52.621 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19494.980308 secs 2024-07-26T16:30:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19494.980308 secs 2024-07-26T16:30:52.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19494.980308 secs 2024-07-26T16:30:53.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:53 smithi183 ceph-mon[37273]: pgmap v10774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:53.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:53 smithi144 ceph-mon[32129]: pgmap v10774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:53.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:53 smithi160 ceph-mon[36715]: pgmap v10774: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:54 smithi183 ceph-mon[37273]: pgmap v10775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:30:54.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:54 smithi144 ceph-mon[32129]: pgmap v10775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:30:54.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:54 smithi160 ceph-mon[36715]: pgmap v10775: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:30:56.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:56 smithi144 ceph-mon[32129]: pgmap v10776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:56.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:56 smithi160 ceph-mon[36715]: pgmap v10776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:56.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:56 smithi183 ceph-mon[37273]: pgmap v10776: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:30:57.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19499.980427 secs 2024-07-26T16:30:57.719 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19499.980427 secs 2024-07-26T16:30:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19499.980427 secs 2024-07-26T16:30:58.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:30:58 smithi144 ceph-mon[32129]: pgmap v10777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:30:58.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:30:58 smithi160 ceph-mon[36715]: pgmap v10777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:30:58.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:30:58 smithi183 ceph-mon[37273]: pgmap v10777: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:30:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:59.432+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:30:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:59.433+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:30:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:59.433+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:30:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:59.433+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:30:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:30:59.433+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:00.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:00 smithi144 ceph-mon[32129]: pgmap v10778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:00 smithi160 ceph-mon[36715]: pgmap v10778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:00.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:00 smithi183 ceph-mon[37273]: pgmap v10778: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19504.980547 secs 2024-07-26T16:31:02.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:02 smithi144 ceph-mon[32129]: pgmap v10779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19504.980547 secs 2024-07-26T16:31:02.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:02 smithi160 ceph-mon[36715]: pgmap v10779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19504.980547 secs 2024-07-26T16:31:02.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:02 smithi183 ceph-mon[37273]: pgmap v10779: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:04.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:04 smithi160 ceph-mon[36715]: pgmap v10780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:04.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:04 smithi183 ceph-mon[37273]: pgmap v10780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:04.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:04 smithi144 ceph-mon[32129]: pgmap v10780: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:06.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:06 smithi160 ceph-mon[36715]: pgmap v10781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:06.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:06 smithi183 ceph-mon[37273]: pgmap v10781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:06.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:06 smithi144 ceph-mon[32129]: pgmap v10781: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:07.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19509.980674 secs 2024-07-26T16:31:07.872 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19509.980674 secs 2024-07-26T16:31:07.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19509.980674 secs 2024-07-26T16:31:08.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:08 smithi160 ceph-mon[36715]: pgmap v10782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:08.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:08 smithi183 ceph-mon[37273]: pgmap v10782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:08.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:08 smithi144 ceph-mon[32129]: pgmap v10782: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:10.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:10 smithi160 ceph-mon[36715]: pgmap v10783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:10.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:10 smithi183 ceph-mon[37273]: pgmap v10783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:10.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:10 smithi144 ceph-mon[32129]: pgmap v10783: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19514.980751 secs 2024-07-26T16:31:12.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:12 smithi160 ceph-mon[36715]: pgmap v10784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19514.980751 secs 2024-07-26T16:31:12.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:12 smithi183 ceph-mon[37273]: pgmap v10784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:12.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19514.980751 secs 2024-07-26T16:31:12.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:12 smithi144 ceph-mon[32129]: pgmap v10784: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:14.449+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:31:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:14.450+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:31:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:14.450+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:31:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:14.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:31:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:14.450+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:14.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:14 smithi160 ceph-mon[36715]: pgmap v10785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:14.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:14 smithi183 ceph-mon[37273]: pgmap v10785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:14.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:14 smithi144 ceph-mon[32129]: pgmap v10785: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:16.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:16 smithi160 ceph-mon[36715]: pgmap v10786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:16.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:16 smithi183 ceph-mon[37273]: pgmap v10786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:16.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:16 smithi144 ceph-mon[32129]: pgmap v10786: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:17.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19519.980876 secs 2024-07-26T16:31:17.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19519.980876 secs 2024-07-26T16:31:17.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19519.980876 secs 2024-07-26T16:31:18.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:18 smithi160 ceph-mon[36715]: pgmap v10787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:18.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:18 smithi183 ceph-mon[37273]: pgmap v10787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:18.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:18 smithi144 ceph-mon[32129]: pgmap v10787: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:20.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:20 smithi160 ceph-mon[36715]: pgmap v10788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:20.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:20 smithi183 ceph-mon[37273]: pgmap v10788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:20.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:20 smithi144 ceph-mon[32129]: pgmap v10788: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:22.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19524.980995 secs 2024-07-26T16:31:22.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:22 smithi160 ceph-mon[36715]: pgmap v10789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19524.980995 secs 2024-07-26T16:31:22.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:22 smithi183 ceph-mon[37273]: pgmap v10789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19524.980995 secs 2024-07-26T16:31:22.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:22 smithi144 ceph-mon[32129]: pgmap v10789: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:24.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:24 smithi160 ceph-mon[36715]: pgmap v10790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:24.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:24 smithi183 ceph-mon[37273]: pgmap v10790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:24.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:24 smithi144 ceph-mon[32129]: pgmap v10790: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:26 smithi160 ceph-mon[36715]: pgmap v10791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:26.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:26 smithi183 ceph-mon[37273]: pgmap v10791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:26.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:26 smithi144 ceph-mon[32129]: pgmap v10791: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:27.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19529.981135 secs 2024-07-26T16:31:27.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19529.981135 secs 2024-07-26T16:31:28.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19529.981135 secs 2024-07-26T16:31:28.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:28 smithi183 ceph-mon[37273]: pgmap v10792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:28.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:28 smithi144 ceph-mon[32129]: pgmap v10792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:29.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:28 smithi160 ceph-mon[36715]: pgmap v10792: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:29.466+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:31:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:29.467+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:31:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:29.467+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:31:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:29.467+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:31:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:29.467+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:30.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:30 smithi183 ceph-mon[37273]: pgmap v10793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:30.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:30 smithi144 ceph-mon[32129]: pgmap v10793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:31.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:30 smithi160 ceph-mon[36715]: pgmap v10793: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:31.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19534.981251 secs 2024-07-26T16:31:31.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19534.981251 secs 2024-07-26T16:31:32.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19534.981251 secs 2024-07-26T16:31:32.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:32 smithi183 ceph-mon[37273]: pgmap v10794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:32.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:32 smithi144 ceph-mon[32129]: pgmap v10794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:33.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:32 smithi160 ceph-mon[36715]: pgmap v10794: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:34.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:34 smithi183 ceph-mon[37273]: pgmap v10795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:34.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:34 smithi144 ceph-mon[32129]: pgmap v10795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:35.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:34 smithi160 ceph-mon[36715]: pgmap v10795: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:36 smithi144 ceph-mon[32129]: pgmap v10796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:36.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19539.981371 secs 2024-07-26T16:31:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:36 smithi160 ceph-mon[36715]: pgmap v10796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:37.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19539.981371 secs 2024-07-26T16:31:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:36 smithi183 ceph-mon[37273]: pgmap v10796: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19539.981371 secs 2024-07-26T16:31:38.910 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:38 smithi144 ceph-mon[32129]: pgmap v10797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:39.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:38 smithi160 ceph-mon[36715]: pgmap v10797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:39.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:38 smithi183 ceph-mon[37273]: pgmap v10797: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:40.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:40 smithi144 ceph-mon[32129]: pgmap v10798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:41.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:40 smithi160 ceph-mon[36715]: pgmap v10798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:41.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:40 smithi183 ceph-mon[37273]: pgmap v10798: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:41.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19544.981434 secs 2024-07-26T16:31:42.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19544.981434 secs 2024-07-26T16:31:42.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19544.981434 secs 2024-07-26T16:31:42.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:42 smithi144 ceph-mon[32129]: pgmap v10799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:43.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:42 smithi160 ceph-mon[36715]: pgmap v10799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:43.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:42 smithi183 ceph-mon[37273]: pgmap v10799: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:31:44.479 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:44.476+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:31:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:44.477+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:31:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:44.477+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:31:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:44.477+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:31:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:44.478+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:44.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:44 smithi144 ceph-mon[32129]: pgmap v10800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:45.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:44 smithi160 ceph-mon[36715]: pgmap v10800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:45.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:44 smithi183 ceph-mon[37273]: pgmap v10800: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:46 smithi144 ceph-mon[32129]: pgmap v10801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:46.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19549.981555 secs 2024-07-26T16:31:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:46 smithi160 ceph-mon[36715]: pgmap v10801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:47.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19549.981555 secs 2024-07-26T16:31:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:46 smithi183 ceph-mon[37273]: pgmap v10801: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:31:47.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19549.981555 secs 2024-07-26T16:31:47.969 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:47 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:48.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:47 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:48.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:47 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:48 smithi144 ceph-mon[32129]: pgmap v10802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:48.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:48 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:31:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:48 smithi160 ceph-mon[36715]: pgmap v10802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:49.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:49.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:48 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:31:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:48 smithi183 ceph-mon[37273]: pgmap v10802: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:31:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:49.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:48 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:31:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:49 smithi144 ceph-mon[32129]: pgmap v10803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:49.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:49 smithi144 ceph-mon[32129]: pgmap v10804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:31:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:49 smithi160 ceph-mon[36715]: pgmap v10803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:50.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:49 smithi160 ceph-mon[36715]: pgmap v10804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:31:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:49 smithi183 ceph-mon[37273]: pgmap v10803: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:50.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:49 smithi183 ceph-mon[37273]: pgmap v10804: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:31:51.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:51 smithi144 ceph-mon[32129]: pgmap v10805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:51.971 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19554.981699 secs 2024-07-26T16:31:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:51 smithi160 ceph-mon[36715]: pgmap v10805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:52.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19554.981699 secs 2024-07-26T16:31:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:51 smithi183 ceph-mon[37273]: pgmap v10805: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:52.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19554.981699 secs 2024-07-26T16:31:53.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:53 smithi144 ceph-mon[32129]: pgmap v10806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:31:54.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:53 smithi160 ceph-mon[36715]: pgmap v10806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:31:54.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:53 smithi183 ceph-mon[37273]: pgmap v10806: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:31:55.970 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:55 smithi144 ceph-mon[32129]: pgmap v10807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:31:56.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:55 smithi160 ceph-mon[36715]: pgmap v10807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:31:56.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:55 smithi183 ceph-mon[37273]: pgmap v10807: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:31:57.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19559.981832 secs 2024-07-26T16:31:57.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19559.981832 secs 2024-07-26T16:31:57.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19559.981832 secs 2024-07-26T16:31:58.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:57 smithi160 ceph-mon[36715]: pgmap v10808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:58.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:57 smithi183 ceph-mon[37273]: pgmap v10808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:58.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:57 smithi144 ceph-mon[32129]: pgmap v10808: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:31:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:59.494+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:31:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:59.495+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:31:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:59.495+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:31:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:59.495+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:31:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:31:59.496+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:31:59 smithi160 ceph-mon[36715]: pgmap v10809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:32:00.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:31:59 smithi183 ceph-mon[37273]: pgmap v10809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:32:00.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:31:59 smithi144 ceph-mon[32129]: pgmap v10809: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:32:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:01 smithi160 ceph-mon[36715]: pgmap v10810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:02.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19564.981954 secs 2024-07-26T16:32:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:01 smithi183 ceph-mon[37273]: pgmap v10810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:02.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19564.981954 secs 2024-07-26T16:32:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:01 smithi144 ceph-mon[32129]: pgmap v10810: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:02.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19564.981954 secs 2024-07-26T16:32:04.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:03 smithi160 ceph-mon[36715]: pgmap v10811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:04.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:03 smithi183 ceph-mon[37273]: pgmap v10811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:04.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:03 smithi144 ceph-mon[32129]: pgmap v10811: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:06.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:05 smithi160 ceph-mon[36715]: pgmap v10812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:06.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:05 smithi183 ceph-mon[37273]: pgmap v10812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:06.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:05 smithi144 ceph-mon[32129]: pgmap v10812: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:07.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:06 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19569.982068 secs 2024-07-26T16:32:07.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:06 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19569.982068 secs 2024-07-26T16:32:07.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:06 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19569.982068 secs 2024-07-26T16:32:08.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:07 smithi160 ceph-mon[36715]: pgmap v10813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:08.121 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:07 smithi144 ceph-mon[32129]: pgmap v10813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:08.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:07 smithi183 ceph-mon[37273]: pgmap v10813: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:10.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:09 smithi160 ceph-mon[36715]: pgmap v10814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:10.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:09 smithi183 ceph-mon[37273]: pgmap v10814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:10.142 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:09 smithi144 ceph-mon[32129]: pgmap v10814: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:11 smithi160 ceph-mon[36715]: pgmap v10815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:12.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:11 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19574.982185 secs 2024-07-26T16:32:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:11 smithi183 ceph-mon[37273]: pgmap v10815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:12.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:11 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19574.982185 secs 2024-07-26T16:32:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:11 smithi144 ceph-mon[32129]: pgmap v10815: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:12.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:11 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19574.982185 secs 2024-07-26T16:32:14.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:13 smithi160 ceph-mon[36715]: pgmap v10816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:14.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:13 smithi183 ceph-mon[37273]: pgmap v10816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:14.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:13 smithi144 ceph-mon[32129]: pgmap v10816: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:14.511+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:32:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:14.512+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:32:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:14.512+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:32:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:14.512+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:32:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:14.512+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:16.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:15 smithi160 ceph-mon[36715]: pgmap v10817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:16.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:15 smithi183 ceph-mon[37273]: pgmap v10817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:16.219 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:15 smithi144 ceph-mon[32129]: pgmap v10817: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:17.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:16 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19579.982332 secs 2024-07-26T16:32:17.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:16 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19579.982332 secs 2024-07-26T16:32:17.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:16 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19579.982332 secs 2024-07-26T16:32:18.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:17 smithi160 ceph-mon[36715]: pgmap v10818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:18.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:17 smithi183 ceph-mon[37273]: pgmap v10818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:18.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:17 smithi144 ceph-mon[32129]: pgmap v10818: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:20.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:19 smithi183 ceph-mon[37273]: pgmap v10819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:20.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:19 smithi144 ceph-mon[32129]: pgmap v10819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:20.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:19 smithi160 ceph-mon[36715]: pgmap v10819: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:21 smithi183 ceph-mon[37273]: pgmap v10820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:22.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:21 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19584.982452 secs 2024-07-26T16:32:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:21 smithi144 ceph-mon[32129]: pgmap v10820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:22.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:21 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19584.982452 secs 2024-07-26T16:32:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:21 smithi160 ceph-mon[36715]: pgmap v10820: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:22.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:21 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19584.982452 secs 2024-07-26T16:32:24.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:23 smithi183 ceph-mon[37273]: pgmap v10821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:24.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:23 smithi144 ceph-mon[32129]: pgmap v10821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:24.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:23 smithi160 ceph-mon[36715]: pgmap v10821: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:26.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:25 smithi183 ceph-mon[37273]: pgmap v10822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:26.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:25 smithi144 ceph-mon[32129]: pgmap v10822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:26.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:25 smithi160 ceph-mon[36715]: pgmap v10822: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:27.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:26 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19589.982538 secs 2024-07-26T16:32:27.211 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:26 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19589.982538 secs 2024-07-26T16:32:27.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:26 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19589.982538 secs 2024-07-26T16:32:28.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:27 smithi183 ceph-mon[37273]: pgmap v10823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:28.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:27 smithi144 ceph-mon[32129]: pgmap v10823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:28.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:27 smithi160 ceph-mon[36715]: pgmap v10823: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:29.528+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:32:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:29.529+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:32:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:29.529+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:32:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:29.529+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:32:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:29.529+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:30.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:29 smithi183 ceph-mon[37273]: pgmap v10824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:30.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:29 smithi144 ceph-mon[32129]: pgmap v10824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:29 smithi160 ceph-mon[36715]: pgmap v10824: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:31 smithi183 ceph-mon[37273]: pgmap v10825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:32.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:31 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19594.982616 secs 2024-07-26T16:32:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:31 smithi144 ceph-mon[32129]: pgmap v10825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:32.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:31 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19594.982616 secs 2024-07-26T16:32:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:31 smithi160 ceph-mon[36715]: pgmap v10825: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:32.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:31 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19594.982616 secs 2024-07-26T16:32:34.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:33 smithi183 ceph-mon[37273]: pgmap v10826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:34.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:33 smithi144 ceph-mon[32129]: pgmap v10826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:34.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:33 smithi160 ceph-mon[36715]: pgmap v10826: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:36.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:35 smithi183 ceph-mon[37273]: pgmap v10827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:36.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:35 smithi144 ceph-mon[32129]: pgmap v10827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:36.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:35 smithi160 ceph-mon[36715]: pgmap v10827: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:37.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:36 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19599.982682 secs 2024-07-26T16:32:37.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:36 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19599.982682 secs 2024-07-26T16:32:37.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:36 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19599.982682 secs 2024-07-26T16:32:38.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:37 smithi183 ceph-mon[37273]: pgmap v10828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:38.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:37 smithi144 ceph-mon[32129]: pgmap v10828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:38.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:37 smithi160 ceph-mon[36715]: pgmap v10828: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:40.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:39 smithi183 ceph-mon[37273]: pgmap v10829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:40.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:39 smithi144 ceph-mon[32129]: pgmap v10829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:40.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:39 smithi160 ceph-mon[36715]: pgmap v10829: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:41 smithi144 ceph-mon[32129]: pgmap v10830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:42.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:41 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19604.982766 secs 2024-07-26T16:32:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:41 smithi160 ceph-mon[36715]: pgmap v10830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:42.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:41 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19604.982766 secs 2024-07-26T16:32:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:41 smithi183 ceph-mon[37273]: pgmap v10830: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:42.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:41 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19604.982766 secs 2024-07-26T16:32:44.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:43 smithi144 ceph-mon[32129]: pgmap v10831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:44.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:43 smithi160 ceph-mon[36715]: pgmap v10831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:44.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:43 smithi183 ceph-mon[37273]: pgmap v10831: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:32:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:44.543+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:32:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:44.544+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:32:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:44.544+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:32:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:44.544+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:32:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:44.544+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:46.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:45 smithi144 ceph-mon[32129]: pgmap v10832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:46.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:45 smithi160 ceph-mon[36715]: pgmap v10832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:46.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:45 smithi183 ceph-mon[37273]: pgmap v10832: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:47.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:46 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19609.982902 secs 2024-07-26T16:32:47.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:46 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19609.982902 secs 2024-07-26T16:32:47.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:46 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19609.982902 secs 2024-07-26T16:32:48.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:47 smithi144 ceph-mon[32129]: pgmap v10833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:48.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:47 smithi160 ceph-mon[36715]: pgmap v10833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:48.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:47 smithi183 ceph-mon[37273]: pgmap v10833: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:32:49.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:48 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:49.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:48 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:49.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:48 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:49.560+0000 7f74a6a00640 1 -- 172.21.15.144:0/108036915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:32:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:49.560+0000 7f7497400640 1 -- 172.21.15.144:0/3349571383 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7478006520 con 0x563c6c4182f0 2024-07-26T16:32:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:49.560+0000 7f74b4a00640 1 -- 172.21.15.144:0/108036915 <== mon.2 v2:172.21.15.160:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7480003690 con 0x7f74b0109990 2024-07-26T16:32:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:49.560+0000 7f749cc00640 1 -- 172.21.15.144:0/3349571383 <== mon.0 v2:172.21.15.144:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f74a009d070 con 0x563c6c4182f0 2024-07-26T16:32:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:49 smithi144 ceph-mon[32129]: pgmap v10834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:49 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:50.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:49 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:32:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:49 smithi160 ceph-mon[36715]: pgmap v10834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:49 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:50.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:49 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:32:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:49 smithi183 ceph-mon[37273]: pgmap v10834: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:32:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:49 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:50.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:49 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:32:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:50 smithi144 ceph-mon[32129]: pgmap v10835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:32:51.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:50 smithi144 ceph-mon[32129]: pgmap v10836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:32:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:50 smithi160 ceph-mon[36715]: pgmap v10835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:32:51.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:50 smithi160 ceph-mon[36715]: pgmap v10836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:32:51.384 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:50 smithi183 ceph-mon[37273]: pgmap v10835: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T16:32:51.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:50 smithi183 ceph-mon[37273]: pgmap v10836: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:32:52.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:51 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19614.983012 secs 2024-07-26T16:32:52.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:51 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19614.983012 secs 2024-07-26T16:32:52.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:51 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19614.983012 secs 2024-07-26T16:32:53.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:52 smithi144 ceph-mon[32129]: pgmap v10837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:32:53.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:52 smithi160 ceph-mon[36715]: pgmap v10837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:32:53.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:52 smithi183 ceph-mon[37273]: pgmap v10837: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:32:55.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:54 smithi144 ceph-mon[32129]: pgmap v10838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:32:55.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:54 smithi160 ceph-mon[36715]: pgmap v10838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:32:55.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:54 smithi183 ceph-mon[37273]: pgmap v10838: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:32:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:56 smithi144 ceph-mon[32129]: pgmap v10839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:32:57.220 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:56 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19619.983130 secs 2024-07-26T16:32:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:56 smithi160 ceph-mon[36715]: pgmap v10839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:32:57.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:56 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19619.983130 secs 2024-07-26T16:32:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:56 smithi183 ceph-mon[37273]: pgmap v10839: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:32:57.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:56 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19619.983130 secs 2024-07-26T16:32:59.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:32:58 smithi160 ceph-mon[36715]: pgmap v10840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:32:59.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:32:58 smithi183 ceph-mon[37273]: pgmap v10840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:32:59.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:32:58 smithi144 ceph-mon[32129]: pgmap v10840: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:32:59.563 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:59.561+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:32:59.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:59.562+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:32:59.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:59.562+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:32:59.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:59.562+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:32:59.564 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:32:59.562+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:00 smithi160 ceph-mon[36715]: pgmap v10841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:33:01.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:00 smithi183 ceph-mon[37273]: pgmap v10841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:33:01.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:00 smithi144 ceph-mon[32129]: pgmap v10841: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:33:02.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:01 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19624.983259 secs 2024-07-26T16:33:02.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:01 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19624.983259 secs 2024-07-26T16:33:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:01 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19624.983259 secs 2024-07-26T16:33:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:02 smithi160 ceph-mon[36715]: pgmap v10842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:03.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:02 smithi183 ceph-mon[37273]: pgmap v10842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:03.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:02 smithi144 ceph-mon[32129]: pgmap v10842: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:05.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:04 smithi160 ceph-mon[36715]: pgmap v10843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:05.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:04 smithi183 ceph-mon[37273]: pgmap v10843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:05.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:04 smithi144 ceph-mon[32129]: pgmap v10843: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:07 smithi160 ceph-mon[36715]: pgmap v10844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:07.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19629.983324 secs 2024-07-26T16:33:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:07 smithi183 ceph-mon[37273]: pgmap v10844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:07.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19629.983324 secs 2024-07-26T16:33:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:07 smithi144 ceph-mon[32129]: pgmap v10844: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:07.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19629.983324 secs 2024-07-26T16:33:09.273 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:09 smithi144 ceph-mon[32129]: pgmap v10845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:09.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:09 smithi160 ceph-mon[36715]: pgmap v10845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:09.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:09 smithi183 ceph-mon[37273]: pgmap v10845: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:11.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:11 smithi160 ceph-mon[36715]: pgmap v10846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:11.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:11 smithi183 ceph-mon[37273]: pgmap v10846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:11.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:11 smithi144 ceph-mon[32129]: pgmap v10846: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:12.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19634.983439 secs 2024-07-26T16:33:12.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19634.983439 secs 2024-07-26T16:33:12.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19634.983439 secs 2024-07-26T16:33:13.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:13 smithi160 ceph-mon[36715]: pgmap v10847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:13.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:13 smithi183 ceph-mon[37273]: pgmap v10847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:13.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:13 smithi144 ceph-mon[32129]: pgmap v10847: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:14.565+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:33:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:14.566+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:33:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:14.566+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:33:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:14.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:33:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:14.566+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:15.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:15 smithi160 ceph-mon[36715]: pgmap v10848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:15.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:15 smithi183 ceph-mon[37273]: pgmap v10848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:15.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:15 smithi144 ceph-mon[32129]: pgmap v10848: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:17 smithi160 ceph-mon[36715]: pgmap v10849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:17.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19639.983550 secs 2024-07-26T16:33:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:17 smithi183 ceph-mon[37273]: pgmap v10849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:17.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19639.983550 secs 2024-07-26T16:33:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:17 smithi144 ceph-mon[32129]: pgmap v10849: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:17.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19639.983550 secs 2024-07-26T16:33:19.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:19 smithi160 ceph-mon[36715]: pgmap v10850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:19.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:19 smithi183 ceph-mon[37273]: pgmap v10850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:19.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:19 smithi144 ceph-mon[32129]: pgmap v10850: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:21.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:21 smithi160 ceph-mon[36715]: pgmap v10851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:21.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:21 smithi183 ceph-mon[37273]: pgmap v10851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:21.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:21 smithi144 ceph-mon[32129]: pgmap v10851: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:22.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19644.983661 secs 2024-07-26T16:33:22.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19644.983661 secs 2024-07-26T16:33:22.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19644.983661 secs 2024-07-26T16:33:23.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:23 smithi183 ceph-mon[37273]: pgmap v10852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:23.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:23 smithi144 ceph-mon[32129]: pgmap v10852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:23.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:23 smithi160 ceph-mon[36715]: pgmap v10852: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:25.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:25 smithi183 ceph-mon[37273]: pgmap v10853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:25.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:25 smithi144 ceph-mon[32129]: pgmap v10853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:25.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:25 smithi160 ceph-mon[36715]: pgmap v10853: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:27.362 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:27 smithi160 ceph-mon[36715]: pgmap v10854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:27.362 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19649.983778 secs 2024-07-26T16:33:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:27 smithi183 ceph-mon[37273]: pgmap v10854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:27.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19649.983778 secs 2024-07-26T16:33:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:27 smithi144 ceph-mon[32129]: pgmap v10854: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:27.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19649.983778 secs 2024-07-26T16:33:29.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:29 smithi183 ceph-mon[37273]: pgmap v10855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:29.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:29 smithi144 ceph-mon[32129]: pgmap v10855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:29.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:29 smithi160 ceph-mon[36715]: pgmap v10855: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:29.582+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:33:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:29.583+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:33:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:29.583+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:33:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:29.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:33:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:29.583+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:31.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:31 smithi183 ceph-mon[37273]: pgmap v10856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:31.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:31 smithi144 ceph-mon[32129]: pgmap v10856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:31.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:31 smithi160 ceph-mon[36715]: pgmap v10856: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:32.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19654.983903 secs 2024-07-26T16:33:32.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19654.983903 secs 2024-07-26T16:33:32.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19654.983903 secs 2024-07-26T16:33:33.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:33 smithi183 ceph-mon[37273]: pgmap v10857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:33.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:33 smithi144 ceph-mon[32129]: pgmap v10857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:33.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:33 smithi160 ceph-mon[36715]: pgmap v10857: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:35.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:35 smithi183 ceph-mon[37273]: pgmap v10858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:35.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:35 smithi144 ceph-mon[32129]: pgmap v10858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:35.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:35 smithi160 ceph-mon[36715]: pgmap v10858: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:37 smithi183 ceph-mon[37273]: pgmap v10859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:37.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:37 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19659.984020 secs 2024-07-26T16:33:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:37 smithi144 ceph-mon[32129]: pgmap v10859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:37.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:37 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19659.984020 secs 2024-07-26T16:33:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:37 smithi160 ceph-mon[36715]: pgmap v10859: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:37.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:37 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19659.984020 secs 2024-07-26T16:33:39.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:39 smithi183 ceph-mon[37273]: pgmap v10860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:39.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:39 smithi144 ceph-mon[32129]: pgmap v10860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:39.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:39 smithi160 ceph-mon[36715]: pgmap v10860: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:41.385 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:41 smithi183 ceph-mon[37273]: pgmap v10861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:41.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:41 smithi144 ceph-mon[32129]: pgmap v10861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:41.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:41 smithi160 ceph-mon[36715]: pgmap v10861: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:42.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:42 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19664.984162 secs 2024-07-26T16:33:42.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:42 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19664.984162 secs 2024-07-26T16:33:42.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:42 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19664.984162 secs 2024-07-26T16:33:43.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:43 smithi144 ceph-mon[32129]: pgmap v10862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:43.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:43 smithi160 ceph-mon[36715]: pgmap v10862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:43.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:43 smithi183 ceph-mon[37273]: pgmap v10862: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:33:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:44.591+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:33:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:44.592+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:33:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:44.592+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:33:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:44.592+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:33:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:44.592+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:45.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:45 smithi144 ceph-mon[32129]: pgmap v10863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:45.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:45 smithi160 ceph-mon[36715]: pgmap v10863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:45.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:45 smithi183 ceph-mon[37273]: pgmap v10863: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:47 smithi144 ceph-mon[32129]: pgmap v10864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:47.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:47 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19669.984286 secs 2024-07-26T16:33:47.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:47 smithi160 ceph-mon[36715]: pgmap v10864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:47.572 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:47 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19669.984286 secs 2024-07-26T16:33:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:47 smithi183 ceph-mon[37273]: pgmap v10864: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:47.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:47 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19669.984286 secs 2024-07-26T16:33:49.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:49 smithi144 ceph-mon[32129]: pgmap v10865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:49.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:49 smithi160 ceph-mon[36715]: pgmap v10865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:49.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:49 smithi183 ceph-mon[37273]: pgmap v10865: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:33:50.469 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:50 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:50.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:50 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:50.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:50 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:51 smithi144 ceph-mon[32129]: pgmap v10866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:51 smithi144 ceph-mon[32129]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:51.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:51 smithi144 ceph-mon[32129]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:33:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:51 smithi160 ceph-mon[36715]: pgmap v10866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:51 smithi160 ceph-mon[36715]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:51 smithi160 ceph-mon[36715]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:33:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:51 smithi183 ceph-mon[37273]: pgmap v10866: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:33:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:51 smithi183 ceph-mon[37273]: from='mgr.24109 172.21.15.144:0/3308676231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:51.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:51 smithi183 ceph-mon[37273]: from='mgr.24109 ' entity='mgr.x' 2024-07-26T16:33:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:52 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19674.984304 secs 2024-07-26T16:33:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:52 smithi144 ceph-mon[32129]: pgmap v10867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:52.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:52 smithi144 ceph-mon[32129]: pgmap v10868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:52 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19674.984304 secs 2024-07-26T16:33:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:52 smithi160 ceph-mon[36715]: pgmap v10867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:52.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:52 smithi160 ceph-mon[36715]: pgmap v10868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:52 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19674.984304 secs 2024-07-26T16:33:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:52 smithi183 ceph-mon[37273]: pgmap v10867: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:52.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:52 smithi183 ceph-mon[37273]: pgmap v10868: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:54.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:54 smithi144 ceph-mon[32129]: pgmap v10869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:33:54.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:54 smithi160 ceph-mon[36715]: pgmap v10869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:33:54.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:54 smithi183 ceph-mon[37273]: pgmap v10869: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:33:56.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:56 smithi144 ceph-mon[32129]: pgmap v10870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:33:56.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:56 smithi160 ceph-mon[36715]: pgmap v10870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:33:56.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:56 smithi183 ceph-mon[37273]: pgmap v10870: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:33:57.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:57 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19679.984424 secs 2024-07-26T16:33:57.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:57 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19679.984424 secs 2024-07-26T16:33:57.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:57 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19679.984424 secs 2024-07-26T16:33:58.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:33:58 smithi144 ceph-mon[32129]: pgmap v10871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:58.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:33:58 smithi160 ceph-mon[36715]: pgmap v10871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:58.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:33:58 smithi183 ceph-mon[37273]: pgmap v10871: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T16:33:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:59.609+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:33:59.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:59.610+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:33:59.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:59.610+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:33:59.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:59.610+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:33:59.611 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:33:59.610+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:00.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:00 smithi144 ceph-mon[32129]: pgmap v10872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:34:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:00 smithi160 ceph-mon[36715]: pgmap v10872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:34:00.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:00 smithi183 ceph-mon[37273]: pgmap v10872: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T16:34:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:02 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19684.984554 secs 2024-07-26T16:34:02.470 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:02 smithi144 ceph-mon[32129]: pgmap v10873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:02 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19684.984554 secs 2024-07-26T16:34:02.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:02 smithi160 ceph-mon[36715]: pgmap v10873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:02 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19684.984554 secs 2024-07-26T16:34:02.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:02 smithi183 ceph-mon[37273]: pgmap v10873: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:04.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:04 smithi160 ceph-mon[36715]: pgmap v10874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:34:04.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:04 smithi183 ceph-mon[37273]: pgmap v10874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:34:04.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:04 smithi144 ceph-mon[32129]: pgmap v10874: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:34:06.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:06 smithi160 ceph-mon[36715]: pgmap v10875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:06.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:06 smithi183 ceph-mon[37273]: pgmap v10875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:06.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:06 smithi144 ceph-mon[32129]: pgmap v10875: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:07.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:07 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19689.984668 secs 2024-07-26T16:34:07.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:07 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19689.984668 secs 2024-07-26T16:34:07.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:07 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19689.984668 secs 2024-07-26T16:34:08.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:08 smithi160 ceph-mon[36715]: pgmap v10876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:34:08.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:08 smithi183 ceph-mon[37273]: pgmap v10876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:34:08.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:08 smithi144 ceph-mon[32129]: pgmap v10876: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:34:10.522 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:10 smithi144 ceph-mon[32129]: pgmap v10877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:10.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:10 smithi160 ceph-mon[36715]: pgmap v10877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:10.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:10 smithi183 ceph-mon[37273]: pgmap v10877: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:12 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19694.984753 secs 2024-07-26T16:34:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:12 smithi160 ceph-mon[36715]: pgmap v10878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:12 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19694.984753 secs 2024-07-26T16:34:12.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:12 smithi183 ceph-mon[37273]: pgmap v10878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:12 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19694.984753 secs 2024-07-26T16:34:12.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:12 smithi144 ceph-mon[32129]: pgmap v10878: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:14.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:14 smithi160 ceph-mon[36715]: pgmap v10879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:34:14.616 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:14.614+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:34:14.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:14.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:34:14.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:14.616+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:34:14.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:14.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:34:14.617 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:14.616+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:14.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:14 smithi183 ceph-mon[37273]: pgmap v10879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:34:14.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:14 smithi144 ceph-mon[32129]: pgmap v10879: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T16:34:16.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:16 smithi160 ceph-mon[36715]: pgmap v10880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:34:16.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:16 smithi183 ceph-mon[37273]: pgmap v10880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:34:16.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:16 smithi144 ceph-mon[32129]: pgmap v10880: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T16:34:17.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:17 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19699.984867 secs 2024-07-26T16:34:17.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:17 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19699.984867 secs 2024-07-26T16:34:17.721 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:17 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19699.984867 secs 2024-07-26T16:34:18.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:18 smithi160 ceph-mon[36715]: pgmap v10881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:18.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:18 smithi183 ceph-mon[37273]: pgmap v10881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:18.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:18 smithi144 ceph-mon[32129]: pgmap v10881: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:20.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:20 smithi160 ceph-mon[36715]: pgmap v10882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:20.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:20 smithi183 ceph-mon[37273]: pgmap v10882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:20.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:20 smithi144 ceph-mon[32129]: pgmap v10882: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T16:34:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:22 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19704.984990 secs 2024-07-26T16:34:22.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:22 smithi160 ceph-mon[36715]: pgmap v10883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:22 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19704.984990 secs 2024-07-26T16:34:22.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:22 smithi183 ceph-mon[37273]: pgmap v10883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:22 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19704.984990 secs 2024-07-26T16:34:22.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:22 smithi144 ceph-mon[32129]: pgmap v10883: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:24.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:24 smithi160 ceph-mon[36715]: pgmap v10884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:34:24.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:24 smithi183 ceph-mon[37273]: pgmap v10884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:34:24.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:24 smithi144 ceph-mon[32129]: pgmap v10884: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T16:34:26.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:26 smithi160 ceph-mon[36715]: pgmap v10885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:26.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:26 smithi183 ceph-mon[37273]: pgmap v10885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:26.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:26 smithi144 ceph-mon[32129]: pgmap v10885: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:27.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:27 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19709.985084 secs 2024-07-26T16:34:27.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:27 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19709.985084 secs 2024-07-26T16:34:27.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:27 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19709.985084 secs 2024-07-26T16:34:28.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:28 smithi183 ceph-mon[37273]: pgmap v10886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:34:28.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:28 smithi144 ceph-mon[32129]: pgmap v10886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:34:28.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:28 smithi160 ceph-mon[36715]: pgmap v10886: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T16:34:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:29.632+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:34:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:29.632+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T16:34:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:29.632+0000 7f74a7e00640 1 -- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 msgr2=0x563c6c802760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T16:34:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:29.632+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:34:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi144.front.sepia.ceph.com.smithi144.stderr:2024-07-26T16:34:29.633+0000 7f74a7e00640 1 --2- 172.21.15.144:0/3349571383 >> [v2:172.21.15.160:6838/1074049609,v1:172.21.15.160:6839/1074049609] conn(0x563c6c3ffd80 0x563c6c802760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:30.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:30 smithi183 ceph-mon[37273]: pgmap v10887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:30.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:30 smithi144 ceph-mon[32129]: pgmap v10887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:30.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:30 smithi160 ceph-mon[36715]: pgmap v10887: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:32 smithi183 ceph-mon[37273]: 2 slow requests, 0 included below; oldest blocked for > 19714.985229 secs 2024-07-26T16:34:32.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 26 16:34:32 smithi183 ceph-mon[37273]: pgmap v10888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:32 smithi144 ceph-mon[32129]: 2 slow requests, 0 included below; oldest blocked for > 19714.985229 secs 2024-07-26T16:34:32.720 INFO:journalctl@ceph.mon.a.smithi144.stdout:Jul 26 16:34:32 smithi144 ceph-mon[32129]: pgmap v10888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:32 smithi160 ceph-mon[36715]: 2 slow requests, 0 included below; oldest blocked for > 19714.985229 secs 2024-07-26T16:34:32.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Jul 26 16:34:32 smithi160 ceph-mon[36715]: pgmap v10888: 129 pgs: 129 active+clean; 2.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T16:34:33.448 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-26T16:34:33.448 DEBUG:teuthology.task.console_log:Killing console logger for smithi144 2024-07-26T16:34:33.449 DEBUG:teuthology.task.console_log:Killing console logger for smithi160 2024-07-26T16:34:33.450 DEBUG:teuthology.task.console_log:Killing console logger for smithi183 2024-07-26T16:34:33.450 DEBUG:teuthology.exit:Finished running handlers